Tiny neural networks sharpen vehicle tracking while keeping Kalman filters intact

Engineers at Daimler Truck and the University of Tübingen have built three miniature neural networks that slot into a classical Kalman-filter tracking pipeline and measurably improve it. The networks, named SPENT, SANT and MANTa, each contain fewer than 50,000 trainable parameters, run in real time on ordinary desktop hardware, and were evaluated on the KITTI tracking benchmark using vehicle tracks of cars and vans. According to the study, published in the journal Autonomous Intelligent Systems, the prediction network cut the root mean square error of state prediction by more than half compared with a standard Kalman filter, while the two association networks reached up to 95 percent assignment accuracy on the data they were designed for.

Multi-object tracking is the computational backbone of advanced driver assistance systems, which are installed in millions of vehicles worldwide. In the widely used tracking-by-detection paradigm, a tracker fuses sensor detections into consistent object tracks over time. Every time a sensor delivers a new measurement, the system must decide which existing track, if any, that measurement belongs to, update the track’s properties, and initialize new tracks when no association can be made. The vast majority of production systems rely on the Kalman filter or its variants, prized for their modularity and interpretability, but the heuristics and manual parameter tuning they require can make the software cumbersome to maintain and difficult to extend across different vehicle models and sensor suites.

The new work attacks that weakness with a data-driven approach that deliberately preserves the classical architecture. The researchers designed SPENT, the Single-Prediction Network, to replace the heuristic motion model that a Kalman filter normally uses to predict where tracked objects will be at the next timestamp. SANT, the Single-Association Network, assigns a single incoming sensor object to one of the existing tracks without relying on a predefined distance metric or the Hungarian algorithm, the classical assignment procedure that minimizes a distance measure but only considers current observations. MANTa, the Multi-Association Network, goes further and jointly associates multiple sensor objects with multiple tracks in a single operational step, covering scenarios from one object to many tracks, many objects to one track, many objects to many tracks, and many objects to no tracks at all.

Each network is built around recurrent layers that carry temporal information. SPENT uses a Long Short-Term Memory layer whose hidden states act as an information repository for each tracked object, updated at every timestamp from incoming measurement data, followed by batch normalization, a ReLU activation, dropout regularization and a fully connected output layer. The network is trained with a mean squared error loss on ground-truth vehicle tracks extracted from the KITTI dataset, where 635 tracks were extracted and filtered down to 624 by removing tracks shorter than three frames. Track lengths ranged from 4 to 643 frames. State values were normalized to zero mean and unit variance, and sequences were pre-padded with zeros and sorted by length before batching to minimize padding noise during training.

SANT and MANTa were trained on synthetic ground truth generated from the same KITTI tracks, because genuine hand-labelled association data does not exist for this problem. The researchers randomly selected tracks and used the next state vector as the incoming sensor object, distorting it with zero-mean Gaussian noise of up to three percent standard deviation per dimension to simulate typical measurement uncertainty in automotive perception. The best-performing association architecture proved to be a bidirectional LSTM layer that reads the sequence of tracks and the incoming object from both ends, followed by a fully connected layer and a softmax output that produces class probabilities over the possible assignments. For MANTa, up to 16 tracks each receive their own softmax stack with 18 possible assignment classes, and the outputs are concatenated into a single 288-element vector.

The experimental results are striking in their specificity. On a testing set of 31 tracks, a Kalman filter framework implemented by the Daimler Truck Research Group achieved a root mean square error of 0.066. SPENT, using identical datasets, reduced that error to 0.029, a reduction of more than 50 percent. For positional predictions, the average deviations were 42 centimeters on the x-axis and 23 centimeters on the y-axis. SANT achieved 95 percent accuracy on a testing dataset of 391 samples, representing 5 percent of the total dataset of 7,827 association samples. MANTa, trained on the same data, also reached 95 percent accuracy for the six most frequently occurring association sets, those with one to six tracks per timestamp, but dropped to 14 percent for scenarios with more tracks, which were rare in the training data.

That polarized performance reflects the structure of the extracted KITTI data rather than a fundamental limitation of the approach. Timestamps containing one to six tracks constitute 81.5 percent of the samples, or 6,374 of 7,827, and exactly one track accounts for 29.9 percent, or 2,315 samples. Only 18.5 percent of samples, 1,448 in total, contain more than six tracks, making the dataset unbalanced and learning the association for seven or more tracks harder. Across the entire dataset, including cars and vans, MANTa achieved an average association accuracy of 80 percent. The researchers attribute the limitation primarily to the characteristics of the extracted data and note that tests on a reduced dataset with one to six tracks per timestamp demonstrated the network’s multi-association capability.

Speed matters as much as accuracy for embedded automotive systems, which must provide immediate state prediction directly after an object’s first detection and cannot use offline methods that look at future sensor data. On a desktop CPU without AI acceleration hardware, an Intel Core i7-13850HX running single-threaded in FP32 precision, SPENT required approximately 0.5 milliseconds per track, or 8.0 milliseconds for 16 concurrent tracks. SANT needed about 1.0 millisecond per association for up to 16 tracks, and MANTa performed a joint multi-object association in approximately 3.5 milliseconds total. Normalized over all tracks in a typical tracking cycle with up to 16 tracks, that corresponds to roughly 0.2 milliseconds per track.

A key design decision is that the networks do not take over the entire tracker. SPENT outputs only the predicted state mean, while uncertainty propagation and measurement updates remain within the classical Kalman filter framework. This makes the modules drop-in compatible with many existing trackers and preserves the modularity and interpretability that make Kalman-based systems attractive in practice. Each network can be trained and evaluated independently of the others, and the track management layer, which deletes stale tracks and sends confirmed tracks to higher software components, remains untouched. The researchers argue that this philosophy improves maintainability and adaptability, because the data-driven components can be fine-tuned for specific vehicle configurations without rewriting the whole pipeline.

The authors position their work as a foundation for future driver assistance research, highlighting the potential of data-driven software development to overcome the limitations of classical algorithms that in practice often need heuristics. They outline three directions for follow-up work: extending validation beyond KITTI to additional datasets to assess transferability across different sensor configurations and traffic scenarios, quantifying uncertainties to improve decision making, and investigating multitasking networks to optimize context-dependent tracking of multiple objects. Because the models were developed within Daimler Truck AG internal research, model parameters are not directly downloadable, though requests can be sent to the authors and all used datasets are publicly available. The study demonstrates that small, task-specific neural modules can substantially improve tracking accuracy and robustness without sacrificing the modularity, interpretability or real-time constraints that automotive deployment demands.

Subject of Research: Tiny neural networks integrated into a Kalman-filter multi-object tracking pipeline for automotive driver assistance systems

Article Title: Tiny neural networks for multi-object tracking in a modular Kalman framework

Article References: Holz, C., Bader, C., Enzweiler, M., & Drüppel, M. (2026). Tiny neural networks for multi-object tracking in a modular Kalman framework. Autonomous Intelligent Systems, 6(1), Article 6. https://doi.org/10.1007/s43684-026-00127-2

Image Credits: AI Generated

DOI: 10.1007/s43684-026-00127-2

Keywords: multi-object tracking, Kalman filter, neural networks, LSTM, automotive perception, driver assistance systems, KITTI benchmark, data association, state prediction, embedded systems, tracking-by-detection, recurrent networks