‘Unmaking’ a move: Correcting motion blur in single-photon images

Imaging technology has come a long way since the beginning of photography in the mid-19th century. Now, many state-of-the-art cameras for demanding applications rely on mechanisms that are considerably different from those in consumer-oriented devices. One of these cameras employs what is known as “single-photon imaging,” which can produce vastly superior results in dark conditions and fast dynamic scenes. But how does single-photon imaging differ from conventional imaging?

When taking a picture with a regular CMOS camera, like the ones on smartphones, the camera sensor is open to a large influx of photons during a predefined exposure time. Each pixel in the sensor grid outputs an analog value that depends on the number of photons that hit that pixel during exposure. However, this type of imaging has few ways to deal with moving objects; the movement of the object has to be much slower than the exposure time to avoid blurring. In contrast, single-photon cameras capture a rapid burst of consecutive frames with very short individual exposure times. These frames are binary–a grid of 1s and 0s that respectively indicate whether one photon arrived at each pixel or not during exposure. To reconstruct an actual picture from these binary frames (or bit planes), many of them have to be processed into a single non-binary image. This can be achieved by assigning different levels of brightness to all the pixels in the grid, depending on how many of the bit planes had a “1” for each pixel.

Besides its higher speed, the completely digital nature of single-photon imaging allows for designing clever image reconstruction algorithms that can make up for technical limitations or difficult scenarios. At Tokyo University of Science, Japan, Professor Takayuki Hamamoto has been leading a research team focused on taking the capabilities of single-photon imaging further. In the latest study by Prof. Hamamoto and his team, which was published in IEEE Access, they developed a highly effective algorithm to fix the blurring caused by motion in the imaged objects, as well as common blurring of the entire image such as that caused by the shaking of the camera.

Their approach addresses many limitations of existing deblurring techniques for single-photon imaging, which produce low-quality pictures when multiple objects in the scene are moving at different speeds and dynamically overlapping each other. Instead of adjusting the entire image according to the estimated motion of a single object or on the basis of spatial regions where the object is considered to be moving, the proposed method employs a more versatile strategy.

First, a motion estimation algorithm tracks the movement of individual pixels through statistical evaluations on how bit values change over time (over different bit planes). In this way, as demonstrated experimentally by the researchers, the motion of individual objects can be accurately estimated. “Our tests show that the proposed motion estimation technique produced results with errors of less than one pixel, even in dark conditions with few incident photons,” remarks Prof. Hamamoto.

The team then developed a deblurring algorithm that uses the results of the motion estimation step. This second algorithm groups pixels with a similar motion together, thereby identifying in each bit plane separate objects moving at different speeds. This allows for deblurring each region of the image independently according to the motions of objects that pass through it. Using simulations, the researchers showed that their strategy produced very crisp and high-quality images, even in low-light dynamic scenes crowded with objects coursing at disparate velocities.

Overall, the results of this study aptly showcase how greatly single-photon imaging can be improved if one gets down to developing effective image processing techniques. “Methods for obtaining crisp images in photon-limited situations would be useful in several fields, including medicine, security, and science. Our approach will hopefully lead to new technology for high-quality imaging in dark environments, like outer space, and super-slow recording that will far exceed the capabilities of today’s fastest cameras,” says Prof. Hamamoto. He also states that even consumer-level cameras might timely benefit from progress in single-photon imaging.

We are certainly getting closer to a new era in digital photography, and studies like this one are crucial for paving the wave towards that future!

###

About The Tokyo University of Science

Tokyo University of Science (TUS) is a well-known and respected university, and the largest science-specialized private research university in Japan, with four campuses in central Tokyo and its suburbs and in Hokkaido. Established in 1881, the university has continually contributed to Japan’s development in science through inculcating the love for science in researchers, technicians, and educators.

With a mission of “Creating science and technology for the harmonious development of nature, human beings, and society”, TUS has undertaken a wide range of research from basic to applied science. TUS has embraced a multidisciplinary approach to research and undertaken intensive study in some of today’s most vital fields. TUS is a meritocracy where the best in science is recognized and nurtured. It is the only private university in Japan that has produced a Nobel Prize winner and the only private university in Asia to produce Nobel Prize winners within the natural sciences field.

Website: https://www.tus.ac.jp/en/mediarelations/

About Professor Takayuki Hamamoto from Tokyo University of Science

Takayuki Hamamoto received Bachelor’s and Master’s degrees from Tokyo University of Science in 1992 and 1994, respectively. He proceeded to get a doctoral degree in electrical engineering from The University of Tokyo in 1997. He is currently a Professor at Tokyo University of Science and leads the Hamamoto Lab, which focuses on image processing and coding, imaging sensors, and very-large-scale integration. He has over 300 papers to his name and received multiple “best paper” and “best poster” awards from various workshops and symposiums. Prof. Hamamoto is also a member of the IEEE.

Funding information

This work was supported by Japan Society for the Promotion of Science (JSPS) KAKENHI under Grant JP19K12025 and Grant 20K19829.