Your Smartphone’s LiDAR Can Now See Around Corners

Researchers turned faint reflected light into rough 3D views of hidden objects.

The same kind of LiDAR sensor that helps some phones measure depth may have learned a new trick: seeing what is not directly in front of it.

Researchers led by Siddharth Somasundaram at the MIT Media Lab showed that a smartphone-grade LiDAR sensor can reconstruct hidden 3D objects, track motion around a corner and even help locate a camera in space using objects outside its direct view. The system does not produce a hidden-room photograph. It recovers sparse shapes and motion from light so faint that earlier versions of the technology required specialized lab equipment.

The result is an ordinary smartphone that can sense around blind corners. Adjusting the same tech for self-driving cars could mean detecting a pedestrian before a driver or camera sees them. For robots, it could mean navigating cluttered rooms. For augmented-reality headsets, it could mean keeping track of a user’s hands after they leave the headset’s field of view.

“The most exciting part of this work to me is that we took a capability that used to require a specialized $50,000 imaging setup and put it into the hands of people in robotics, AR/VR, and beyond,” Somasundaram said in a press release.

How a Wall Becomes a Dim Mirror

LiDAR works by firing laser pulses and measuring how long the light takes to return. Because light travels very fast, the timing has to be extraordinarily precise. Modern consumer LiDAR sensors can measure time-of-flight at picosecond scales, short enough to register differences of centimeters.

Normally, a LiDAR sensor maps only what it sees directly. But light does not stop there. Some of it hits a wall or floor, scatters toward a hidden object, bounces back to the wall and then returns to the sensor. That path is indirect and the return signal is weak. Still, it carries information.

Researchers call this non-line-of-sight imaging. The idea has been around for years. Earlier demonstrations used expensive ultrafast lasers, sensitive detectors and careful calibration. In the new paper, the researchers asked whether the raw, noisy signals from a cheap consumer sensor could be made useful.

Their answer was yes, but only by changing the way the data are treated.

×

Get smarter every day...

Stay ahead with ZME Science and subscribe.

Daily Newsletter
The science you need to know, every weekday.

Weekly Newsletter
A week in science, all in one place. Sends every Sunday.
No spam, ever. Unsubscribe anytime. Review our Privacy Policy.

Thank you! One more thing...

Please check your inbox and confirm your subscription.

The team used a portable smartphone-grade LiDAR system with roughly 100 pixels, each combining a laser emitter and a single-photon detector. A single frame was too noisy and low-resolution to reveal much. So the researchers combined many frames, turning the natural movement of a handheld device into an advantage.

The method, called motion-induced aperture sampling, borrows from two consumer technologies. Smartphone cameras already combine bursts of images to improve low-light photos. Synthetic aperture radar uses motion to make a sensor act like a larger one. Here, the researchers used motion to gather many weak glimpses of a hidden scene and fuse them into a stronger estimate.

“Once we developed algorithms that could combine information across those measurements, the hidden signals started to emerge much more clearly,” Somasundaram told IEEE Spectrum.

What the System Actually Saw

The team demonstrated three main abilities.

First, it reconstructed the 3D shapes of static hidden objects. Second, it tracked moving objects of known shape, including multiple hidden objects at once. Third, it used hidden objects as landmarks to help determine where the camera was.

The resulting images are not crisp photos you would expect to see from a camera. They’re more like rough 3D forms, probability clouds and motion tracks. In one experiment, the system reconstructed a hidden U-shaped object. In another, it tracked objects and a bouncing ball around a corner. The researchers also showed a hand-tracking demonstration using retroreflective gloves, similar in spirit to motion-capture markers.

“What I found most surprising is that these consumer lidar systems are able to capture any useful signal from around a corner at all,” Somasundaram told IEEE Spectrum. “The amount of light that reaches the sensor after multiple bounces is incredibly small, and these devices were never designed with this kind of imaging in mind. Seeing that there is enough information present to reconstruct and track hidden objects was very exciting for us.”

That faintness explains why the system worked best with reflective objects, though the team also tested everyday diffuse surfaces. In those cases, the results were worse because much less light returned to the sensor.

Beyond Phones

The immediate breakthrough is not that your phone can suddenly spy around corners. The important shift is that a capability once confined to optics labs may now work on inexpensive, widely available hardware.

The paper argues that earlier non-line-of-sight systems often required bulky, costly equipment and careful setup. By contrast, the new demonstrations used off-the-shelf hardware costing less than $100 and no difficult physical calibration. The researchers also publicly released code and data for the work.

“We think the most important implication is the democratization of the technology,” Somasundaram told IEEE Spectrum. “When technologies like this become accessible, people often discover applications far beyond what the original researchers imagined.”

The list of possible uses is long. A warehouse robot might sense movement in an aisle before turning into it. A home robot might use a hidden chair or table as a landmark when blank walls offer few visual clues. An AR headset might infer where a user’s hands are even when they are temporarily outside the cameras’ view.

RelatedPosts

This simple code can increase your smartphone battery life by 16% – and yes, it’s free
Constantly checking your phone reduces wellbeing, makes you less mindful
Your smartphone can tell when you’re drunk — and this might save lives
Laser technology reveals huge medieval cities hidden in Cambodian jungle

Autonomous driving is another obvious possibility, though the gap between a lab demonstration and a safety-critical car system is large. Roads are brighter and more dynamic than controlled indoor experiments.

“As LiDARs become more common, I think this could lead to entirely new forms of machine vision and spatial perception,” Somasundaram said.

Jessica Rosenworcel, executive director of the MIT Media Lab, framed the work as the realization of a long-running idea. “Now that vision is arriving in consumer devices, with implications we’re only beginning to explore,” she said.

The findings appeared in Nature.