News Column

Patent Application Titled "Method and Apparatus for 2d to 3d Conversion Using Panorama Image" Published Online

May 22, 2014



By a News Reporter-Staff News Editor at Politics & Government Week -- According to news reporting originating from Washington, D.C., by VerticalNews journalists, a patent application by the inventors Noh, Jun Yong (Daejeon, KR); Choi, Sung Woo (Daejeon, KR); Blanco Ribera, Roger (Daejeon, KR); Kim, Young Hui (Gyeonggi-do, KR); Lee, Jung Jin (Daejeon, KR), filed on May 30, 2013, was made available online on May 8, 2014.

No assignee for this patent application has been made.

Reporters obtained the following quote from the background information supplied by the inventors: "The present disclosure relates to a method and apparatus for 2D to 3D conversion, and more particularly, to a method and apparatus for 2D to 3D conversion using a panorama image.

"In these days, as the popularity of 3-dimensional (hereinafter, referred to as 3D) stereoscopic movies has increased, the number of contents made with 3D images has rapidly increased. However, in order to make a 3D image, two synchronized cameras are generally fixed to a stereo camera rig for photographing. However, making a 3D image using a stereo camera is not easy since various kinds of hardware such as cameras should be accurately corrected and post processes are demanded to ensure the level of difficulty for the control of the stereo camera rig and give convenience to spectators. As an alternative to solve the above problems, a technique of making a stereo image by converting a 2D image to a 3D image is being utilized. The 2D to 3D conversion is very useful since an existing 2D image may be converted into and reproduced as a 3D image.

"The technique for converting a 2D image to a 3D image produces a stereo image pair corresponding to each of at least one single image. In order to generate a stereo image pair, a method of estimating suitable depth information of an image is well known in the art. If a depth map based on the depth information on an image is available, a stereo image pair may be generated by pixel translation of a single view sequence according to a depth value calculated at each location on the image. A method of estimating a depth of a monocular image or sequence based on a depth cue such as motion, fog or focus is currently being utilized and automated. However, this image is composed of a plurality of images frames, different from a single image, and depth maps respectively corresponding to the image frames should be organically connected with each other in a soft manner. Therefore, a 3D image obtained by an automated conversion method is inferior to the specialized high quality conversion demanded in the entertainment industry. Therefore, in order to make a 3D image with high quality, manual intervention for correcting a depth estimated by an automation method is utilized or an entire depth map is manually generated. However, this means that a very large amount of manual works should be performed.

"Generally, in the case conversion quality should be ensured, a worker for 3D conversion should make manual inputs at every several frames or at every frame. In addition, for rotoscoping foreground objects, suitable depth painting is demanded in some cases. In addition, if consistency over time of an estimated depth map is demanded for the overall image sequence, the conversion work becomes more complex."

In addition to obtaining background information on this patent application, VerticalNews editors also obtained the inventors' summary information for this patent application: "An embodiment of the present disclosure is directed to providing a method for 2D to 3D conversion using a panorama image, in which a user records scribbles at a single panorama image corresponding to a plurality of image frames to generate depth information of an original image frame, thereby greatly reducing a workload of a 3D conversion worker.

"The present disclosure is also directed to providing an apparatus for 2D to 3D conversion using a panorama image.

"In one aspect of the present disclosure, there is provided an apparatus for 2D to 3D conversion using a panorama image, which includes: an image receiving unit for receiving and storing an input image; a user interface for receiving an input of a user who performs a 3D conversion work; a panorama image generating unit for extracting feature points of a plurality of images which compose an image sequence of the input image and warping and combining the plurality of images based on the extracted feature points to generate a single panorama image; a depth setting unit for recording scribbles including depth information in at least one of a plurality of pixels of the panorama image in response to the input of the user received through the user interface; a depth information propagating unit for calculating depth values of other pixels based on a depth value of the depth information of the at least one pixel in which the scribbles are recorded, to calculate depth values of all pixels of the panorama image and generate a panorama image depth map; a depth information remapping unit for mapping a depth value with respect to each of the plurality of images by using the depth map of the panorama image to generate an individual image depth map; and a stereo image generating unit for generating a stereo image pair for each of the plurality of images by using the individual image depth map and generating a stereo image by using the generated stereo image pair.

"The panorama image generating unit may include a reference image selecting unit for selecting a reference image among the plurality of images according to a preset manner; a feature point tracking unit for extracting feature points from the plurality of images and tracking the feature points extracted from each of the plurality of images to be matched with feature points of the reference image; an image warping unit for warping images other than the reference image among the plurality of images according to the tracked feature points; and an image accumulating unit for accumulatively matching the plurality of warped images with the reference image based on the feature points to generate a single panorama image.

"The panorama image generating unit may further include a confidence map generating unit for generating a confidence map by evaluating confidence of each of the plurality of pixels of the panorama image according to a preset manner.

"The reference image selecting unit may select a single image among the plurality of images as the reference image in response to a command of the user applied through the user interface.

"The apparatus may further include a color information analyzing unit for analyzing color information of each of the plurality of pixels of the panorama image and transmitting the color information to the depth information propagating unit.

"The depth information propagating unit may calculate the depth values of all pixels of the panorama image by combining the depth information of the at least one pixel in which the scribbles are recorded, with the color information.

"The depth information remapping unit may generate the individual image depth map by combining the depth map of the panorama image with the confidence map and thus performing a local image optimization work.

"In another aspect of the present disclosure, there is also provided a method for 2D to 3D conversion using a panorama image, performed by an apparatus for 2D to 3D conversion which includes an image receiving unit, a user interface, a panorama image generating unit, a depth setting unit, a depth information propagating unit, a depth information remapping unit and a stereo image generating unit, the method including: by the image receiving unit, receiving and storing an input image; by the panorama image generating unit, extracting feature points of a plurality of images which compose an image sequence of the input image and warping and combining the plurality of images based on the extracted feature points to generate a single panorama image; by the depth setting unit, recording scribbles including depth information in at least one of a plurality of pixels of the panorama image in response to the input of the user received through the user interface; by the depth information propagating unit, calculating depth values of other pixels based on a depth value of the depth information of the at least one pixel in which the scribbles are recorded, to calculate depth values of all pixels of the panorama image and generate a panorama image depth map; by the depth information remapping unit, mapping a depth value with respect to each of the plurality of images by using the depth map of the panorama image to generate an individual image depth map; and by the stereo image generating unit, generating a stereo image pair for each of the plurality of images by using the individual image depth map and generating a stereo image by using the generated stereo image pair.

"The generating of a panorama image may include selecting a reference image among the plurality of images according to a preset manner; extracting feature points from the plurality of images and tracking the feature points extracted from each of the plurality of images to be matched with feature points of the reference image; warping images other than the reference image among the plurality of images according to the tracked feature points; and accumulatively matching the plurality of warped images with the reference image based on the feature points.

"The generating of a panorama image may further include generating a confidence map by evaluating confidence of each of the plurality of pixels of the panorama image according to a preset manner.

"The selecting of a reference image may select a single image among the plurality of images as the reference image in response to a command of the user applied through the user interface.

"The apparatus for 2D to 3D conversion may further include a color information analyzing unit, and the method for 2D to 3D conversion may further include analyzing color information of each of the plurality of pixels of the panorama image and transmitting the color information to the depth information propagating unit.

"The generating of a panorama image depth map may calculate the depth values of all pixels of the panorama image by combining the depth information of the at least one pixel in which the scribbles are recorded, with the color information.

"The generating of an individual image depth map may generate the individual image depth map by combining the depth map of the panorama image with the confidence map and thus performing a local image optimization work.

"Therefore, the apparatus for 2D to 3D conversion using a panorama image according to the present disclosure converts an image composed of an image sequence into a single panorama image, designates depth information to the converted panorama image by means of scribbles of a worker, and then if the designated depth information is propagated to the entire panorama image to generate a depth map, remaps the depth map to the image sequence to generate a stereo image. Therefore, even though the worker performs manual works only to a single panorama image, a high quality 3D stereo image may be obtained. For this reason, it is possible to greatly reduce manual works of a 3D conversion worker and generate 3D stereo images which are organically connected in a soft manner according to time. In addition, since a perfect panorama image is not needed, the present disclosure may be easily applied to relatively free camera motions in comparison to the existing techniques.

BRIEF DESCRIPTION OF THE DRAWINGS

"The above and other features and advantages will become more apparent to those of ordinary skill in the art by describing in detail exemplary embodiments with reference to the attached drawings, in which:

"FIG. 1 shows an apparatus for 2D to 3D conversion using a panorama image according to the present disclosure;

"FIG. 2 shows a method for 2D to 3D conversion using a panorama image according to the present disclosure;

"FIG. 3 shows an example to which a confidence map is applied;

"FIG. 4 shows an example to which worker scribbles are applied;

"FIG. 5 shows an example of a depth map calculated using color of a panorama image;

"FIG. 6 comparatively shows mapping results before and after local image recognition optimization;

"FIG. 7 shows an example of depth scaling;

"FIG. 8 shows an example to which the method for 2D to 3D conversion using a panorama image as shown in FIG. 2 is applied;

"FIG. 9 shows an experimental example of a 2D to 3D conversion process according to the movement of a camera;

"FIG. 10 shows another example of the 2D to 3D conversion process according to the movement of a camera; and

"FIG. 11 shows an example of an image converted into 3D according to the present disclosure.

"In the following description, the same or similar elements are labeled with the same or similar reference numbers."

For more information, see this patent application: Noh, Jun Yong; Choi, Sung Woo; Blanco Ribera, Roger; Kim, Young Hui; Lee, Jung Jin. Method and Apparatus for 2d to 3d Conversion Using Panorama Image. Filed May 30, 2013 and posted May 8, 2014. Patent URL: http://appft.uspto.gov/netacgi/nph-Parser?Sect1=PTO2&Sect2=HITOFF&u=%2Fnetahtml%2FPTO%2Fsearch-adv.html&r=4877&p=98&f=G&l=50&d=PG01&S1=20140501.PD.&OS=PD/20140501&RS=PD/20140501

Keywords for this news article include: Patents.

Our reports deliver fact-based news of research and discoveries from around the world. Copyright 2014, NewsRx LLC


For more stories covering arts and entertainment, please see HispanicBusiness' Arts & Entertainment Channel



Source: Politics & Government Week


Story Tools






HispanicBusiness.com Facebook Linkedin Twitter RSS Feed Email Alerts & Newsletters