What is an event-based camera?

While the world of photography is more approachable now than ever, it hasn’t changed as much as one would expect since its inception in the 1800s. Of course, modern cameras are fast, compact, and don’t rely on film rolls. But the general idea of how cameras work remains the same, Light enters a sensor, this information is saved and a picture is formed.

The most notable breakthrough in recent camera technology comes in the form of event cameras, also known as neuromorphic cameras or DVS (Dynamic Vision Sensor). An event camera approaches the concept of photography from a completely different angle, in fact, it doesn’t really “capture” images as much as it “sees” them.

Where traditional photography only provides static information, event-based cameras actively interpret the images in front of them. These cameras are designed to work as our eyes do. They react in real-time to the changes in front of them which means they can transmit much more data in a smaller window of time. But to truly understand event cameras we need to take a closer look at how they work.

Reference: Samuel Bryner, Guillermo Gallego, Henri Rebecq, Davide Scaramuzza Event-based, Direct Camera Tracking from a Photometric 3D Map using Nonlinear Optimization, IEEE International Conference on Robotics and Automation (ICRA), 2019

How do event cameras work?

The most important thing to understand about event cameras is the peculiarities behind their sensors. Like we mentioned above these cameras aren’t designed to capture a single moment in time, but instead, they aim to replicate our traditional interpretation of sight. This means that event cameras need to be able to react to changes as they happen.

The main sensor or chip of an event camera contains hundreds of thousands of miniature pixels, which react to the light they receive at fixed intervals, usually a matter of microseconds. Each pixel interprets the image in front of them based on how it reacts to light. If the pixel recognizes that the amount of light it’s receiving has changed it proceeds to change the internal value, and if it doesn’t see any change, it remains as usual.

What this means is that instead of capturing an instant, these cameras record an accurate estimate of motion. Since every pixel is independent and is constantly responding event cameras provide a more accurate representation of reality and show us the minuscule changes that happen to the environment almost in real-time.

Reference: Samsung SmartThings Vision Event Camera. Image attribution: Samsung Electronics Australia Pty Ltd

Event camera vs Standard camera

By now we’ve taken a look at the inner mechanics behind event cameras, but how different are they really to traditional cameras at the end of the day? As it turns out they have next to nothing in common.

The biggest difference between the two lies in how they “capture” information. As we just saw event cameras don’t capture instants in the form of fixed images, instead they continuously react to every change in their environment and record this information as they experience it. In standard cameras on the other hand a shutter needs to open to allow light to enter the sensor, at which point the camera records photography of what it’s in front of it.

In other words, standard cameras always record fixed and flat images of the world in front of them. Even film is just a long series of joined photos, just recorded at an even faster pace. Event cameras don’t take a photo in the traditional sense; they don’t even have a shutter. Instead, they record the tiny variances in the image in front of them to offer a more authentic and genuine interpretation of the world around them.

Similarly, while standard cameras offer their information directly as photography the data that event cameras provide require further interpretation. Event cameras don’t offer a flat image but instead a set of coordinates and information tied to each pixel in a given frame of time. So specialized software is required to properly interpret the information compiled by an event camera.

Reference: Robot vision group Imperial College London

The main perks

There’s no denying that event-based vision is very different from traditional photography, but how is this advantageous? What are the real benefits that come from using this technology?

The first advantage that event cameras offer comes in the form of data transfer. While the internal sensor of an event camera is complex, the information it registers and sends is deceptively simple. The only thing an event camera needs to report is if a pixel experienced a change in light. If there’s no change, it doesn’t need to send any new information, and if it does it only needs to register that since the actual image reconstruction is usually handled by the receiver. This means that the data feed of an event camera is surprisingly light, and this is important now that photography files only get larger and larger.

Responsiveness is also a major consideration when it comes to event-based vision. Event cameras can react by the microsecond, in other words, those tiny pixels are sending information every one-millionth of a second. Traditional cameras might be becoming faster, but there are mechanical limitations to shutter speeds. Since event cameras don’t have this limitation, their speed is unbeatable.

Reference: Metavision Intelligence Suit is the Event-Based Vision software developed by Prophesee.

And last but not least the information event-based cameras offer is just more comprehensive. There’s much more information to obtain from the feedback of event-based vision, and this allows the technology to find more practical applications, particularly in the field of robotics.

Applications of Technology

Event cameras have a lot of potential applications in modern technology. and while they still aren’t an industry-standard across multiple fields, early adopters have shown the true versatility of this invention. Today we’ll focus on three of the most important areas where event-based vision can improve on existing technologies.

Reference: Samsung SmartThings Vision Event Camera. Image attribution: Samsung Electronics Australia Pty Ltd

Tracking

Tracking is generally speaking the most obvious application of event-based vision, however, it’s important to understand why this is the case. Traditional tracking technology and software are complex, and this comes down to the fact that the system needs to understand the concept of “motion”.

Like we cleared up before even film is nothing more than a fast series of still images. And while recording such information is no longer a complex task in our world, being able to interpret motion in it to perform tracking is a completely different task. Tracking software is complex because it needs to add a function that isn’t native to traditional photography, but this isn’t the case for event cameras.

Event cameras are always detecting changes in their environment, and the information they record isn’t a static picture. By default, the data feed of an event camera is a representation of the movement and changes in front of it. This means that the light-sensitive pixels of event cameras offer a simple way to track moving objects, as in a sense they are always doing so.

Robotics

Event-based vision is one of the most exciting developments in the field of computer vision and this is major news for the world of robotics. Traditionally one of the main problems with robotics has been the limitations that a lack of sight produces in the field. For humans, sight is something most individuals take for granted, but the act of observing and interpreting the world is incredibly complex and not easily replicated.

While event cameras are not the industry standard yet, it has a lot of potential in the field of high-speed robotics. A DVS captures information at a much faster pace, meaning that it provides robots with far more accurate data. This provides a significant boost in efficiency for recording tasks as well as navigation since they’ll be able to recognize distances much faster and prevent collisions.

Reference: Drone dodgeball, demonstration by the University of Zurich Robotics and Perception Group

Similarly, the improved speed of information input would allow robots to interact in more accurate ways with external objects. Event cameras are promising when it comes to robots with manipulators who must be able to interpret depth and distance continuously to be able to properly grasp and move objects. Event-based vision will allow for robotics to start manipulating their environment more efficiently, a major barrier in the current market.

Localization

Event-based vision also has important applications in the field of localization due to the possibilities that event camera SLAM offers. SLAM or Simultaneous Localization And Mapping is a technology that allows agents to map and navigate new spaces simultaneously, and it’s not only responsible for speeding up mapping processes all over the world but also is essential to allow automated robots to navigate any environment. If you have ever heard of automated cars or seen a Roomba in motion that only happened thanks to the practical applications of SLAM.

However, SLAM has traditionally proved to be a demanding technology particularly when it comes to processing power. This means that a lot of energy and resources have to be spent to allow automated agents to navigate using SLAM, something that proper application of event-based technology can circumvent.

Like we mentioned before event cameras don’t transmit heavy data. Not only do they limit themselves to sending information only when relevant change has been noticed, but that information is compressed in the first place. When coupled with its speed an event camera can allow a SLAM system to interpret its surroundings at high speeds while consuming fewer resources. This means that event cameras are a reliable option both for traditional mapping and high-speed localization, the main fields that rely on SLAM.

Conclusion

While event-based vision can sound deceptively simple there’s no denying the practical applications of the technology. Unlike classic photography event cameras provide a way to constantly gather information on their surroundings thanks to their light-sensitive pixels. In short, they offer a way to truly interpret and observe images mechanically instead of simply recording a still frame.

While event-based cameras have yet to become an industry-standard in most fields, the technology has already proved its relevance in the form of all its potential applications. Even in its early stages event cameras already offer a simpler way to deal with tracking, and provide a very powerful alternative for computer vision.

Right now, event-based vision is one of the most exciting breakthroughs in the world of cameras, and the practical applications for this technology are only bound to increase as more companies choose to adopt it.

Event-Based Cameras Dioram

At the moment the situation with event cameras is a classic chicken-and-egg problem. We need more sophisticated production level hardware, but to start investing into this tech big hardware sensors manufacturers need solid applied benefits, which are hard to obtain without next-gen hardware.

We have conducted intensive research with event-cameras prototypes and can’t wait to get our hands on some production sensors so we could implement this amazing tech in our Dioram SLAM One.