Bio-inspired cameras and AI help drivers detect pedestrians and obstacles faster

It’s every driver’s nightmare: a pedestrian stepping out in front of the car seemingly out of nowhere, leaving only a fraction of a second to brake or steer the wheel and avoid the worst. Some cars now have camera systems that can alert the driver or activate emergency braking. But these systems are not yet fast or reliable enough, and they will need to improve dramatically if they are to be used in autonomous vehicles where there is no human behind the wheel.

Quicker detection using less computational power

Now, Daniel Gehrig and Davide Scaramuzza from the Department of Informatics at the University of Zurich (UZH) have combined a novel bio-inspired camera with AI to develop a system that can detect obstacles around a car much quicker than current systems and using less computational power. The study is published in this week’s issue of Nature.

Most current cameras are frame-based, meaning they take snapshots at regular intervals. Those currently used for driver assistance on cars typically capture 30 to 50 frames per second and an artificial neural network can be trained to recognize objects in their images – pedestrians, bikes, and other cars. “But if something happens during the 20 or 30 milliseconds between two snapshots, the camera may see it too late. The solution would be increasing the frame rate, but that translates into more data that needs to be processed in real-time and more computational power,” says Daniel Gehrig, first author of the paper.

Combining the best of two camera types with AI

Event cameras are a recent innovation based on a different principle. Instead of a constant frame rate, they have smart pixels that record information every time they detect fast movements. “This way, they have no blind spot between frames, which allows them to detect obstacles more quickly. They are also called neuromorphic cameras because they mimic how human eyes perceive images”, says Davide Scaramuzza, head of the Robotics and Perception Group. But they have their own shortcomings: they can miss things that move slowly and their images are not easily converted into the kind of data that is used to train the AI algorithm.

Gehrig and Scaramuzza came up with a hybrid system that combines the best of both worlds: It includes a standard camera that collects 20 images per second, a relatively low frame rate compared to the ones currently in use. Its images are processed by an AI system, called a convolutional neural network, that is trained to recognize cars or pedestrians. The data from the event camera is coupled to a different type of AI system, called an asynchronous graph neural network, which is particularly apt for analyzing 3-D data that change over time. Detections from the event camera are used to anticipate detections by the standard camera and also boost its performance. “The result is a visual detector that can detect objects just as quickly as a standard camera taking 5,000 images per second would do but requires the same bandwidth as a standard 50-frame-per-second camera”, says Daniel Gehrig.

One hundred times faster detections using less data

The team tested their system against the best cameras and visual algorithms currently on the automotive market, finding that it leads to one hundred times faster detections while reducing the amount of data that must be transmitted between the camera and the onboard computer as well as the computational power needed to process the images without affecting accuracy. Crucially, the system can effectively detect cars and pedestrians that enter the field of view between two subsequent frames of the standard camera, providing additional safety for both the driver and traffic participants – which can make a huge difference, especially at high speeds.

According to the scientists, the method could be made even more powerful in the future by integrating cameras with LiDAR sensors, like the ones used on self-driving cars. “Hybrid systems like this could be crucial to allow autonomous driving, guaranteeing safety without leading to a substantial growth of data and computational power,” says Davide Scaramuzza.