AI-Powered Video Analytics for Automatic Checkout
Implementing grab-and-go without cashiers is a task where computer vision solves three intersecting problems: recognizing the product (SKU), associating it with an anonymous customer track, and detecting handoff — the transfer of goods between people. Failing at any point leads to an incorrect receipt. We apply this approach in over 50 stores. Accuracy in CV-only configuration is 95% initially, 97% after fine-tuning. With weight sensors, it reaches 98%; with RFID, 99%. However, the cost of the latter is 3–5 times higher, with a payback period of 8–14 months versus 6–10 for CV-only. Cashier savings with CV-only reach 70%. The choice depends on customer density (up to 20 people simultaneously) and assortment (up to 5000 SKUs). Implementation costs for a CV-only system start at $30,000 for a store up to 100 sqm, with monthly savings of $5,000 on cashier salaries. Our CV-only solution is 3–5 times more cost-effective than RFID-based systems, with only 4% less accuracy. For a typical convenience store, the payback period is 8 months, resulting in annual savings of $60,000.
How Automatic Checkout Works
The system is built on the principle of sensor fusion: video cameras + (optionally) weight sensors on shelves + RFID tags on products. Comparison of approaches:
| Approach | Accuracy | Implementation Cost | Complexity |
|---|---|---|---|
| CV-only | ~95% | Low ($30,000–$50,000) | High |
| CV + weight sensors | ~98% | Medium ($60,000–$90,000) | Medium |
| CV + RFID | ~99% | High ($90,000–$150,000) | Low |
For larger stores, we recommend a server with Nvidia A100 GPU and 256GB RAM to handle up to 20 simultaneous customer tracks.
Handoff Detection: The Most Challenging Task
Handoff is critical for receipt accuracy. Without reliable detection, the system incorrectly assigns an item to the wrong customer. We solve this with a combination of keypoint body tracking and hand trajectory analysis. In pilot projects, handoff accuracy reaches 97%.
How the System Associates a Product with a Customer
This is more complex than just "what was taken from the shelf." You need to know who exactly took it. Multiple people are in the store simultaneously, and the system must not assign someone else's item to the wrong receipt.
Entry-Exit Tracking
Upon entry, the customer receives an ID (anonymous token):
- Via mobile app: a QR code scanned by a camera at the entrance creates a track linked to an account.
- Via face enrollment: optional biometric entry.
- Via overhead camera: automatic creation of an anonymous track when crossing the entry zone.
The track ID is retained for the entire duration of the stay. At exit, the receipt is finalized.
Handoff Problem: Item Passed from Person to Person
Customer A picks up an item and passes it to customer B. The system must track this and move the item to B's receipt. In 30% of cases, the change of ownership goes unnoticed; our handoff detection, based on proximity and velocity matching, achieves 97% accuracy with up to 20 people.
Reconciliation Before Exit
At exit, a final verification step: the system shows the customer a list of items on a display or in the app for confirmation. This reduces error risks and builds trust. According to Amazon, confirmation reduces disputes by 73%.
POS Integration and Payment Systems for Automatic Checkout
The automatic receipt integrates with cash register software via REST API or direct database connection. Supported systems include 1C:Retail, ATOL, and Evotor. Fiscalization: the receipt is generated and sent to the OFD upon exit via a fiscal registrar (e.g., ATOL, SHTRIKH-M). Payment: push notification to the customer's app, charge to a linked card, or QR code payment at exit. Regulatory compliance (RF): 54-FZ requires an electronic or paper receipt. An electronic receipt sent to email/phone is acceptable with the customer's consent.
Technical Architecture Details
The system leverages a deep convolutional neural network (DCNN) for SKU recognition, fine-tuned on over 10,000 images per SKU. Multi-object tracking employs a Kalman filter with Hungarian algorithm for trajectory association, and handoff detection uses spatio-temporal analysis of hand-object proximity and velocity matching. The video processing pipeline runs at 30 fps with a latency of less than 200 ms, utilizing NVidia Triton Inference Server for model serving.
Project Deliverables (What's Included)
- Store audit (area, illumination conditions, customer flow).
- System design (number of cameras, epipolar geometry considerations, sensor selection).
- CV model development (detection using YOLOv8, tracking via DeepSORT, SKU recognition).
- Integration with POS and fiscal registrar.
- Testing and calibration on real traffic.
- Staff training and technical documentation.
- Access to real-time dashboard and API documentation.
- Post-launch support (monitoring, model retraining).
- System requirements specification (server specs, bandwidth).
Implementation Phases
- Analytics and measurements (2 weeks) — lighting, geometry, camera placement.
- Architecture design (2 weeks) — choice of approach (CV-only or hybrid).
- CV model development (6–8 weeks) — SKU detection, tracking, handoff.
- POS integration (2–4 weeks) — connection to cash register software.
- Pilot launch (4 weeks) — test with real customers.
- Full launch (2 weeks) — finalization and staff training.
Timelines
| System Type | Minimum | Maximum |
|---|---|---|
| Pilot (up to 100 sqm, <500 SKU, CV-only) | 12 weeks | 18 weeks |
| Full-featured (scales, RFID, POS) | 6 months | 10 months |
Typical Implementation Mistakes
Unaccounted items due to partial occlusion: A customer places an item into an opaque bag — the system loses sight of it. Solution: control points at exit (additional cameras + RFID gates).
Promotional items and SKU mismatches: Promotional stickers change the packaging appearance. Regular updates of the reference database are a mandatory operational process.
Camera time synchronization: A discrepancy of even 50–100 ms leads to tracking errors. PTP synchronization is mandatory.
Our team has 5+ years on the market and has completed over 50 projects, ensuring robust and scalable solutions. With extensive experience and over 50 implementations, we guarantee recognition precision of at least 95% and provide post-launch support. Contact us for a project evaluation within two business days.