Ubicept wants half the world’s cameras to see things differently
Computer vision could be much faster and better if we skipped the concept of still images and instead looked directly at the data stream from a camera. At least that’s the theory on which the latest idea from the MIT media lab, Ubicept, works.
Most computer vision applications work the same way: a camera captures an image (or a rapid sequence of images in the case of video). These still images are sent to a computer, which then performs an analysis to find out what’s in the frame. Sounds too easy.
But there’s a problem: this paradigm assumes that creating static images is a good idea. Since people used to see photos and videos, this may seem reasonable. Computers don’t care though, and Ubicept thinks it can make computer vision a lot better and more reliable by ignoring the idea of frames.
The company itself is a collaboration between the co-founders. Sebastian Bauer is the company’s CEO and a postdoctoral researcher at the University of Wisconsin where he worked on Lidar systems. Tristan Swedish is now the CTO of Ubicept. Prior to that, she was a Research Associate and received an MS and a Ph.D. Eight years as a student at the MIT Media Lab.
“There are 45 billion cameras in the world, and most of them take images and videos that aren’t actually seen by a human being,” Bauer explains. “These cameras are primarily for observation, so systems can make decisions based on that observation. Think of autonomous driving, for example, as a system that revolves around pedestrian detection. There are all those studies that show that pedestrian detection works really well in broad daylight, but performs particularly poorly in low light. Other examples are cameras for industrial classification, inspection and quality control. All of these cameras are used for automated decision making. They work well in well-lit areas or in daylight. However, problems arise in low light, especially when it comes to fast movements.
The company’s solution is to ignore the “freeze frame” as a source of truth for computer vision and instead measure the individual photons that directly hit an image sensor. This can be achieved with a single photon avalanche diode array (or SPAD array, among friends). This raw data stream can be fed into a field programmable gate array (FPGA, a highly specialized type of processor) and then analyzed by computer vision algorithms.
The newly formed company demonstrated its technology at CES in Las Vegas in January and has some pretty bold plans for the future of imaging.
“Our vision is to have the technology in at least 10% of cameras in the next five years and in at least 50% of cameras in the next 10 years,” predicts Bauer. “If you detect every photon with a very high temporal resolution, you are doing what nature allows. And you see the benefits, like the high-quality videos on our site that put everything else to shame.”
TechCrunch saw the technology in action at a recent demo in Boston and wanted to explore how the technology works and what impact it has on computer vision and AI applications.
A new way of looking
Digital cameras generally work by capturing a frame exposure by “counting” the number of photons that hit each pixel on the sensor over a period of time. At the end of the period, all those photons are multiplied and you have a still image. If nothing moves in the frame, this works great, but the “if nothing moves” is a big warning, especially when it comes to computer vision. As it turns out, everything is in constant motion as you try to use cameras to make decisions.
Of course, the raw data still allows the company to blend the stream of photons into frames, creating sharp videos with no motion blur. Perhaps most excitingly, doing away with the idea of frames meant the Ubicept team could take the raw data and analyze it directly. Here’s an example video showing the big difference this can make in practice:
Source: La Neta Neta

Jason Jack is an experienced technology journalist and author at The Nation View. With a background in computer science and engineering, he has a deep understanding of the latest technology trends and developments. He writes about a wide range of technology topics, including artificial intelligence, machine learning, software development, and cybersecurity.