One Camera, Four Sensors: AI Tracks Warehouse Packages in Real Time

A single overhead camera running on edge hardware can now perform four separate conveyor-monitoring jobs at once—counting packages, spotting doubles, detecting reversals, and flagging stalled parcels—according to a new study published in Frontiers in Artificial Intelligence. The system, developed by researchers at Amazon’s Science and Advanced Concepts team, replaces the tangle of photoeyes, break-beam counters, and proximity sensors that warehouses traditionally install for each individual task with one shared computer vision pipeline. In an operational evaluation at a working sortation facility, throughput counting reached 98.05% accuracy across 513 manually counted packages, while doubles detection achieved 96.0% precision and reversal detection 96.6% precision.

The core idea is what the authors call a virtual sensor: a software-defined trigger line or detection zone evaluated geometrically against tracked package trajectories. Instead of bolting new hardware onto a conveyor for every monitoring need, operators define spatial primitives in a configuration file, and the same detection and tracking pass serves all of them. Three of the four functions—throughput counting, doubles detection, and reversal detection—are derived directly from continuous, identity-preserving package trajectories. The fourth, dwelling detection, reuses the same per-frame detections for a periodic zone-occupancy check without any tracking state at all. Every function is therefore served by one detector pass per processed frame, with no task-specific sensing hardware and no redundant inference pipelines.

The perception backbone combines a fine-tuned RF-DETR Small detector—a transformer-based architecture chosen for its balance of accuracy and inference speed—with ByteTrack multi-object tracking. ByteTrack was selected because its association strategy incorporates low-confidence detection boxes rather than discarding them, which improves tracking continuity for packages temporarily obscured by motion blur or partial occlusion, a common condition on fast-moving conveyors. The detector was fine-tuned on a proprietary dataset of 1,920-by-1,080 images collected from overhead cameras across multiple warehouse sites, split 80/10/10 into training, validation, and test sets. On the held-out test set, the model attained an mAP@50 of 96.5%, precision of 98.3%, recall of 94.7%, and an F1 score of 0.965, recognizing two classes: boxes and envelopes.

The authors are candid about an important caveat in those detection numbers. Because the dataset was split randomly at the image level rather than by camera or site, images from the same cameras—including the evaluation facility—appear in all three subsets. The test figures therefore characterize performance under acquisition conditions related to those seen during training, rather than demonstrating generalization to unseen cameras or sites. A grouped evaluation holding out entire cameras or sites is left to future work, and the team notes that cross-site generalization remains unproven.

On the hardware side, each induction point is equipped with an NVIDIA A1000 GPU operating within an 8 GiB VRAM budget, running FP16 inference to fit the model while maintaining real-time throughput. The RTSP video stream arrives at 30 frames per second, but full-rate inference would exceed the computational budget, so every second frame is processed, yielding an effective rate of 15 FPS. End-to-end per-frame latency—detection, tracking, and all virtual sensor evaluations—is 67 milliseconds. Over an approximately 19-hour production window of 59,791 telemetry samples, each device sustained the 15 FPS target, consuming a median resident memory of 2.0 GB, under 1 GB of GPU memory, and roughly 1.2 CPU cores, with no event clips dropped.

Throughput counting works by detecting when a tracked package centroid crosses a virtual trigger line for the first time, with a set of already-counted track identities preventing double counting from jitter near the line boundary. Evaluated against a human annotator on five production videos of roughly five minutes each, the system counted 503 packages against 513 ground-truth packages, with all errors being undercounts and two videos achieving perfect accuracy. Counts are aggregated over fixed intervals and streamed to a database for real-time dashboards.

Doubles detection—identifying two or more packages inducted side by side, a primary cause of downstream jams—uses a two-line state machine: an upstream line flags candidates, and a downstream line determines whether the facility’s quality control reversed them into the reject bin or they escaped onto the sortation loop. Across 500 manually reviewed production events, precision was 99.6% for reversed doubles and 92.4% for escaped doubles, for an overall 96.0%. Event recall, however, was estimated separately on approximately six hours of continuously annotated video: of 30 annotated doubles events, only 20 were detected, a recall of 66.7% with a wide 95% Wilson confidence interval of 48.8% to 80.8%. All ten misses were perception-level failures involving stacked or overlapping envelopes and boxes that the detector resolved as a single bounding box, leaving no second detection for the trajectory logic to use.

Reversal detection analyzes trajectory displacement and velocity direction: a package is classified as reversed when it has traveled a meaningful distance from its origin, returned significantly from its farthest observed position, and its velocity vector aligns with the return direction. Of 500 reviewed reversal alerts, 483 were confirmed, yielding 96.6% operational precision—but 16 of the 17 false positives were operator-induced reversals, workers manually pulling packages backward, which the algorithm correctly detected as direction changes. The raw algorithmic precision was therefore 99.8%. Recall on the same six hours of annotated video was 88.9% (95% Wilson CI 71.9% to 96.1%), with the three missed reversals being genuine failures of the reversal criteria rather than detection errors, indicating room for improvement in the displacement and alignment thresholds.

Dwelling detection, which identifies packages accumulated in the reject bin, is the simplest module: once every 60 seconds it checks whether any detection centroid falls within a predefined dwelling-zone polygon, a stateless test robust to track identity discontinuities. Evaluated on 500 manually labeled snapshot images—182 dwelling events and 318 clear zones—it achieved 100% precision, meaning zero false alarms for operators, with 96.15% recall reflecting seven missed events, an F1 score of 98.04%, and 98.60% accuracy.

The researchers frame the system as a potential alternative or complement to physical sensors for selected non-safety-critical monitoring functions, not a proven replacement. No side-by-side comparison with photoeyes was performed, and cameras introduce their own failure modes, including occlusion, lighting changes, lens contamination, and detector errors. The evaluation was conducted at a single facility on a single conveyor type with a standardized camera installation, and the tracking stage itself was not independently evaluated against annotated ground-truth tracks, so identity switches and trajectory fragmentation could propagate into the downstream metrics. Privacy is handled on-device: a separate person-detection model blurs any frames containing people before storage or upload, and no unfiltered video ever leaves the edge. With its high-precision, reconfigurable design and richer trajectory metadata than binary sensors can offer, the framework points toward warehouses where adding a new monitoring capability requires nothing more than a coordinate update and a few lines of geometric logic.

Subject of Research: A unified edge computer vision framework that derives four conveyor-monitoring functions from one overhead camera at a warehouse sortation facility.

Article Title: AI-driven warehouse automation: a unified edge computer vision framework for real-time package tracking and conveyor monitoring

Article References: Mahyoub, M., Pandey, A., Halawa, F., & Mohammed, R. (2026). AI-driven warehouse automation: a unified edge computer vision framework for real-time package tracking and conveyor monitoring. Frontiers in Artificial Intelligence, 9, Article 1917595. https://doi.org/10.3389/frai.2026.1917595

Image Credits: AI Generated

DOI: 10.3389/frai.2026.1917595

Keywords: computer vision, warehouse automation, object tracking, edge computing, virtual sensors, conveyor monitoring, RF-DETR, ByteTrack, trajectory analysis, sortation facility, multi-object tracking, package detection