Penn engineers build F³, a common software layer for event cameras

Why event cameras have been hard to use

Event cameras imitate the biological retina. Instead of recording full images at regular intervals, such as 30 frames per second, they continuously respond to changes in light intensity. According to Penn Engineering, this has long promised faster and more reliable robot perception than conventional cameras allow.

The catch is the data. It looks like a stream of points rather than a series of recognizable images, and interpreting it has traditionally required a patchwork of specialized algorithms, each built for a single task. Pratik Chaudhari, Associate Professor in Electrical and Systems Engineering and senior author of the paper, says that "without the ability to remove redundant information," the data from an event camera is "far too noisy and voluminous to use directly."

Richeek Das, a doctoral student and the paper's first author, points to the biological comparison: the retina has cells beyond rods and cones that detect motion and color or suppress redundant information, while event cameras have lacked a general equivalent.

Teaching a network to predict

The team first tried mathematical techniques for summarizing visual information, hoping to build the event-camera equivalent of the Fast Fourier Transform. After months of experimentation, they could not find an approach that reliably went from the camera's vast information rate to a picture of a scene's layout and the motion within it.

They then trained a neural network to find its own way of summarizing the signals, using prediction as the training goal. "To accurately anticipate future events based on past ones, the network must learn to represent both the structure of a scene and the motion taking place within it," says Das. This avoids designing a separate approach for every task.

The first implementation was too slow for real-time use. The researchers noticed that roughly 95% of the possible positions in the data contain no events at all, so they restructured the calculations to run only where events actually occur.

Results across cars, legged robots and drones

The researchers tested F³ on object recognition, distance estimation and motion tracking, using data from cars, four-legged robots and flying drones. Environments ranged from urban streets at night to crowded indoor spaces. According to the team, F³-based approaches achieved state-of-the-art performance on multiple robotic perception tasks.

Algorithms using F³ were roughly two to five times faster than previous approaches. F³ itself processed high-definition event-camera data 120 times per second, or 440 times per second at lower Video Graphics Array (VGA) resolution.

Penn contrasts this generality with Sony AI's Ace, a ping-pong-playing robot unveiled earlier this year that uses event cameras to track the speed and spin of a ball. Its vision system was built specifically for table tennis, which illustrates the need for custom software per application. Kostas Daniilidis, a co-author, says F³ gives event cameras "a common language," turning their signals into a format that software already developed for conventional cameras can be adapted to use.

What comes next

The team is already using F³ to build autonomous drones that can navigate cluttered and poorly lit environments with event cameras, working with doctoral students Dexter Ong and Ioannis Asmanis. Daniilidis says F³ "has become a key component of our pipeline."

Longer term, the researchers envision building F³ directly into the circuitry of event cameras, where it could work like a camera driver and automatically convert raw signals into a format existing computer-vision software can use. "Our goal is to make using an event camera as straightforward as using a conventional one," says Chaudhari.

If that works as described, engineers would no longer need to rewrite software each time a robot takes on a new task, which could make event cameras practical for platforms such as self-driving cars and drones. The study was funded in part by the National Science Foundation and the Internet of Things for Agriculture Engineering Research Center.