News Column

"Method and System for Generating Structured Light with Spatio-Temporal Patterns for 3D Scene Reconstruction" in Patent Application Approval Process

February 27, 2014

By a News Reporter-Staff News Editor at Politics & Government Week -- A patent application by the inventors Taguchi, Yuichi (Arlington, MA); Agrawal, Amit (Somerville, MA); Tuzel, Oncel (Cambridge, MA), filed on July 31, 2012, was made available online on February 13, 2014, according to news reporting originating from Washington, D.C., by VerticalNews correspondents.

This patent application has not been assigned to a company or institution.

The following quote was obtained by the news editors from the background information supplied by the inventors: "Projecting structured light (SL) onto scenes is one of the most reliable techniques for shape measurement and 3D scene reconstruction in computer vision and robotics applications. The correspondence problem present in stereo vision is simplified by projecting known patterns from a projector onto a scene, which is then imaged with a camera. For each pixel in the acquired image, the corresponding projector row or column is obtained by decoding the acquired patterns, followed by a ray-plane intersection to determine a 3D point.

"Coded patterns, including Gray codes, are widely used to provide high quality reconstructions for static scenes. SL methods, using an N-bit binary Gray code pattern, project N binary images onto a scene to uniquely encode 2.sup.N projector columns or rows. Using N acquired images, each pixel is decoded by identifying the corresponding projector column or row.

"Single-shot SL methods project a single pattern that allows per-image reconstruction, and thus can be used for dynamic scenes and deforming objects. However, such methods decrease the spatial resolution and perform poorly near depth discontinuities, e.g., thin structures and edges, because a contiguous spatial neighborhood of pixels is required to perform the decoding for each pixel.

"More importantly, prior art single-shot methods project the same pattern repeatedly for each time instant. Even if the scene is static, or if parts of the scene are slowly moving, prior art methods still decrease the spatial resolution as if the entire scene is dynamic. Thus, conventional single-shot methods are not motion-sensitive.

"Furthermore, conventional single-shot methods typically reconstruct depths at sparse feature points such as edges, intensity peaks of color stripes, and 2D grid points, by using complex mechanisms and heuristics for decoding.

"Single-Shot Structured Light

"Single-shot structured light with spatial multiplexing can use 1D and 2D patterns to decode an image. A 1D De Bruijn sequence, having a window uniqueness property, is projected onto a scene to be decoded or reconstructed. The De Bruijn sequence enables unique decoding if a small spatial window of symbols is detected near a pixel. Color stripe patterns can be used to generate De Bruijn sequences when more than two symbols are required. Examples of 2D patterns include grid patterns and M-arrays, and perfect sub-maps using various geometric shapes and colors.

"One technique uses a low cost 3D sensor for computer vision and human-computer interaction applications. That technique projects an infrared random dot 2D pattern as the single-shot pattern, which is acquired using an infrared camera. The matching is done per image, and depth maps for a dynamic scene can be obtained in real time. However, the depth maps are noisy especially near depth discontinuities.

"All of the above methods project the same pattern for every image, process each image independently, and are not motion-sensitive.

"Another method registers the above depth maps and then reconstructs a static 3D scene with a higher quality compared to raw depth maps.

"Spatio-Temporal Decoding

"Structured light patterns that are spatio-temporally decodable are known. However, conventional methods require disconnected windows of pixels for decoding, and thus do not reduce the effective size of spatial neighborhood required for decoding using all patterns.

"The spatial resolution can also be improved by shifting a single-shot color stripe pattern one pixel at a time and analyzing the temporal profile for each pixel using all of the shifted patterns. However, that method is not hierarchical. That method requires the entire scene to be static during the projection of all shifted patterns to reduce the spatial neighborhood to a single pixel.

"Flexible voxels enable a spatio-temporal resolution trade-off for reconstructing a video depending on the motion of each pixel.

"Adaptive Window Matching

"Spatio-temporal windows have been used for stereo processing to improve the matching quality. However, for stereo processing, the size of the window is typically fixed for every pixel, or regular box shaped windows are used."

In addition to the background information obtained for this patent application, VerticalNews journalists also obtained the inventors' summary information for this patent application: "Single-shot structured light (SL) methods enable 3D shape determination and reconstruction of dynamic scenes. However, conventional methods decrease the spatial resolution and perform poorly near depth discontinuities, such as edges.

"Conventional single-shot methods project the same pattern repeatedly, which reduces the spatial resolution even if the scene is static or has slow moving parts.

"The embodiments of the invention provide a method and system for reconstructing a 3D shape of a scene using a structured light system that projects a set of sequentially shifted striped patterns that is decodable spatially and temporally.

"The method enables single-shot 3D reconstruction with any of the projected patterns by using spatial windows. Moreover, the sequence of patterns progressively improves the reconstruction quality near depth discontinuities by also using temporal windows. In other words, the more patterns that are used, the more accurate the reconstructions.

"The method enables motion-sensitive reconstruction for each pixel. An optimal spatio-temporal window is automatically selected depending on the scene structure, motion, and the number of available images. This significantly reduces the number of pixels near discontinuities where depth cannot be obtained with conventional single-shot methods.

"The decoding uses an adaptive window matching that includes 1D spatial windows, and temporal windows. The adaptation is according to spatial and temporal characteristics of a scene onto which the patterns are projected.

"The invention has a number of advantages for a variety of scenarios, including thin structures, dynamic scenes, and scenes containing both static and dynamic regions.


"FIG. 1A is a schematic of a prior art striped structured light pattern;

"FIG. 1B is a schematic of a set of sequentially shifted, striped structured light patterns using a De Bruijn sequence with spatial and temporal variations according to embodiments of the invention;

"FIG. 1C is a table of spatial window size s required at each image for decoding using different numbers of images (frames) and patterns;

"FIG. 2 is a schematic of motion-sensitive spatio-temporal window selection as a function of time and space according to embodiments of the invention;

"FIG. 3 is a schematic of 1D shifted windows according to embodiments of the invention;

"FIGS. 4A-4B are block and 3D diagrams of a method and system for reconstructing a scene using structured light with spatio-temporal decodable patterns according to embodiments of the invention; and

"FIG. 5 is a block diagram of a method for generating the pattern of FIG. 1B according to embodiments of the invention."

URL and more information on this patent application, see: Taguchi, Yuichi; Agrawal, Amit; Tuzel, Oncel. Method and System for Generating Structured Light with Spatio-Temporal Patterns for 3D Scene Reconstruction. Filed July 31, 2012 and posted February 13, 2014. Patent URL:

Keywords for this news article include: Patents.

Our reports deliver fact-based news of research and discoveries from around the world. Copyright 2014, NewsRx LLC

For more stories covering arts and entertainment, please see HispanicBusiness' Arts & Entertainment Channel

Source: Politics & Government Week

Story Tools