News Column

"Three-Dimensional Modeling from Single Photographs" in Patent Application Approval Process

September 4, 2014

By a News Reporter-Staff News Editor at Politics & Government Week -- A patent application by the inventors COHEN-OR, Daniel (Hod-HaSharon, IL); SHAMIR, Ariel (Jerusalem, IL); CHEN, Tao (Shandong, CN), filed on February 11, 2014, was made available online on August 21, 2014, according to news reporting originating from Washington, D.C., by VerticalNews correspondents.

This patent application is assigned to Ramot at Tel-Aviv University Ltd.

The following quote was obtained by the news editors from the background information supplied by the inventors: "The present invention, in some embodiments thereof, relates to three-dimensional modeling from single photographs and, more particularly but not exclusively to modeling of manmade objects with straightforward geometry.

"The creation and modeling of 3D objects has always been a difficult task even for professionals. First, a mental idea of what the model should look like needs to be formed. This conceptual stage requires creativity and inspiration. Then, the idea needs to be implemented by a series of actions using various geometric modeling tools. These steps take time, demand very high proficiency, and a fair amount of skill. By modeling objects from existing photographs one can first alleviate the mental stage. Second, it allows much simpler modeling that can also borrow textures from the image. This forms at least an initial base model that can later be edited and refined. In addition, such abilities can be utilized for manipulating the images themselves using 3D. An example of a suitable object is shown in FIG. 1, in which the left image shows the object, and the right image shows the object after minor rotation. The result is to leave a black hole in the image.

"Extracting three dimensional models from a single photo is still a long way from realization at the current state of technology, as it involves numerous complex tasks: the target object must be separated from its background, and its 3D pose, shape and structure should be recognized from its projection. These tasks are difficult since they require some degree of semantically understanding the object. To alleviate this problem, complex 3D models can be partitioned into simpler parts, but identifying object parts also requires semantic understanding and is difficult to perform automatically. Moreover, once decomposing a 3D shape to parts, the relations between these parts should also be understood and maintained in the final composition."

In addition to the background information obtained for this patent application, VerticalNews journalists also obtained the inventors' summary information for this patent application: "The present embodiments provide a method and apparatus for extracting three-dimensional information of objects in single photographs by providing a user with interactivity to draw a cross-section for a part of the object and then sweep the cross section over the part of the object to which it applies. Unlike certain of the above cited works, the present embodiments may focus on the modeling of a single subject that is observed in a photograph and not the whole scene.

"The computer then fits the cross-section to the object outline of which it is aware and once all parts of the object have been addressed in this way the computer is able to generate a three-dimensional model of the object, which can then be rotated, or used in animations or in any other way.

"Thus, in the present embodiments, the original object is not restricted, as with Xu et al, to prestored shapes. Rather, the embodiments work on geometric primitives, so that any shape that can be deconstructed into geometric primitives can be reconstructed into a 3D object. The reconstructed object is thus composed of these generic primitives, providing larger scope and flexibility.

"The prior art teaches snapping, and separately teaches sweeping. The present embodiments combine sweeping and snapping to provide automatic alignment of the primitives into an overall object.

"According to an aspect of some embodiments of the present invention there is provided a method of obtaining a three-dimensional digital model of an artificial object made up of a plurality of geometric primitives, the artificial object being in a single two-dimensional photograph, the method comprising:

"defining a two-dimensional outline of the artificial object within the photograph;

"interactively allowing a user to define cross-sectional profiles of successive ones of the geometric primitives, the cross-sectional profiles defining a third dimension;

"interactively allowing a user to provide sweep input to sweep respective defined cross-sectional profiles over an extent of a corresponding one of the geometric primitives within the image, the sweeping generating successive three-dimensional model primitives from existing detected edges of the corresponding geometric primitives and the sweeping of the respective profile; and

"aligning the plurality of three-dimensional model primitives to form the three-dimensional model.

"The method may comprise interactively allowing the user to explicitly define three dimensions of the geometric primitive using three sweep motions, wherein a first two of the three sweeps define a first and second dimension of the cross-sectional profile and a third sweep defines a main axis of the geometric primitive.

"The method may comprise, upon the user sweeping the two-dimensional profile over a respective one of the geometric primitives, dynamically adjusting the two-dimensional profile using a pictorial context on the photograph and automatically snapping photograph lines to the profile.

"In an embodiment, the snapping allows the three-dimensional model to include three-dimensional primitives that adhere to the object in the photographs, while maintaining global constraints between the plurality of three-dimensional model primitives composing the object.

"The method may comprise optimizing the global constraints while taking into account the snapping and the sweep input.

"The method may comprise a post snapping fit improvement of better fitting the primitive to the image, the better fitting comprising searching for transformations within .+-.10% of primitive size, that create a better fit of the primitive's projection to the profile.

"In an embodiment, the defining the two dimensional outline comprises edge detecting.

"An embodiment may comprise estimating a field of view angle from which the photograph was taken in order to estimate and compensate for distortion of the primitives within the photograph.

"An embodiment may comprise using relationships between the primitives in order to define global constraints for the object.

"An embodiment may comprise obtaining geo-semantic relations between the primitives to define the three-dimensional digital model, and encoding the relations as part of the model.

"An embodiment may comprise inserting the three-dimensional digital model into a second photograph.

"The method may comprise extracting a texture from the photograph and applying the texture to sides of the three-dimensional model not visible in the photograph.

"In an embodiment, the defining the cross-sectional profiles comprises defining a shape and then distorting the shape to correspond to a three-dimensional orientation angle.

"The method may comprise applying different constraints to different parts respectively of a given one of the geometric primitives, or locally modifying different parts respectively of a given one of the geometric primitives.

"The method may comprise snapping the first two user sweep motions to the photograph lines, using the endpoints of the first two user sweep motions along with an anchor point on a respective primitive to create three-dimensional orthogonal system for a respective primitive.

"The method may comprise supporting a constraint, the constraint being one member of the group consisting of: parallelism, orthogonality, collinear axis endpoints, overlapping axis endpoints, coplanar axis endpoints and coplanar axes, and for the member testing whether a pair of components is close to satisfying the member, and if the member is satisfied or close to satisfied then adding the constraint to a respective one of the primitives.

"In the method, aligning the three dimensional primitives may comprise finding an initial position for all primitives together by changing only their depth to adhere to geo-semantic constraints, followed by modifying shapes shape of the primitives.

"The present embodiments may include a user interface for carrying out the above method. The user interface may comprise an outline view of a current photograph on which view to carry out interactive sweeping to define cross sections of respective primitives and on which to snap the cross-sections. The user interface may further comprise a solid model view and a texture view respectively of the current photograph, and selectability for user selection between different basic cross-sectional shapes.

"According to a second aspect of the present invention there may be provided a method of digitally forming a three-dimensional geometric primitive from a two-dimensional geometric primitive from a two-dimensional photograph, comprising:

"interactively obtaining user input to draw a two-dimensional cross section of the primitive and then using further user input to sweep the cross-section over a length of the primitive.

"A geometric primitive is a part of an object whose cross section does not change, or which does not change discontinuously. That is to say the part is a geometric primitive if it has a cross section that remains constant or changes continuously along the length of the part.

"According to a third aspect of the present invention there is provided a method of forming a derivation of a photograph, the photograph incorporating a two dimensional representation of a three-dimensional object, the two-dimensional representation being a rotation of an original two-dimensional representation, the rotation being formed by:

"carrying out the method described hereinabove to form a three-dimensional model of the original two-dimensional representation;

"rotating the three-dimensional model; and

"projecting the rotated three-dimensional model onto a two-dimensional surface to form the derivation.

"Unless otherwise defined, all technical and/or scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the invention pertains. Although methods and materials similar or equivalent to those described herein can be used in the practice or testing of embodiments of the invention, exemplary methods and/or materials are described below. In case of conflict, the patent specification, including definitions, will control. In addition, the materials, methods, and examples are illustrative only and are not intended to be necessarily limiting.

"Implementation of the method and/or system of embodiments of the invention can involve performing or completing selected tasks manually, automatically, or a combination thereof. Moreover, according to actual instrumentation and equipment of embodiments of the method and/or system of the invention, several selected tasks could be implemented by hardware, by software or by firmware or by a combination thereof using an operating system.

"For example, hardware for performing selected tasks according to embodiments of the invention could be implemented as a chip or a circuit. As software, selected tasks according to embodiments of the invention could be implemented as a plurality of software instructions being executed by a computer using any suitable operating system. In an exemplary embodiment of the invention, one or more tasks according to exemplary embodiments of method and/or system as described herein are performed by a data processor, such as a computing platform for executing a plurality of instructions. The data processor may include a volatile memory for storing instructions and/or data and/or a non-volatile storage, for example, a magnetic hard-disk, flash memory and/or removable media, for storing instructions and/or data. A network connection may be provided and a display and/or a user input device such as a keyboard or mouse may be available as necessary.


"The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee.

"Some embodiments of the invention are herein described, by way of example only, with reference to the accompanying drawings. With specific reference now to the drawings in detail, it is stressed that the particulars shown are by way of example and for purposes of illustrative discussion of embodiments of the invention. In this regard, the description taken with the drawings makes apparent to those skilled in the art how embodiments of the invention may be practiced.

"In the drawings:

"FIG. 1A is a simplified flow chart illustrating a procedure for forming a 3D model from a single 2D photograph according to an embodiment of the present invention;

"FIG. 1B is a simplified diagram showing an object being extracted from a 2D photograph for modeling;

"FIGS. 2A-2F schematically illustrate the various stages of extracting the object, modeling, and reinserting a rotated version of the image back into the original photograph, according to embodiments of the present invention;

"FIGS. 3A-3E are simplified diagrams illustrating drawing a two-dimensional profile of a primitive and sweeping the profile over a curved axis of the primitive, the profile snapping to the successively shrinking edges of the primitive, according to embodiments of the present invention;

"FIG. 4 is a simplified diagram illustrating a series of graphic primitives and their representation as a series of three sweeps respectively, according to embodiments of the present invention;

"FIGS. 5A-5B are simplified diagrams illustrating alignment of different primitives based on axis points, according to embodiments of the present invention;

"FIG. 6 is a simplified diagram illustrating the use of sweeps for representation of cubes according to embodiments of the present invention;

"FIGS. 7A-7E are a series of photographs in a top row, from which objects are extracted, modeled and manipulated in a second row and then replaced in the original photograph in the third row according to embodiments of the present invention;

"FIGS. 8A-8D illustrate how parts can be taken from different images to deal with lack of detail or occlusion of parts in one or other of the images according to embodiments of the present invention;

"FIG. 9 shows four series of three images, in each of which a detail from an original object is replicated according to embodiments of the present invention;

"FIG. 10 shows two series of photographs in which an object in the leftmost image in each series is modified in different ways according to embodiments of the present invention;

"FIG. 11 is a collage made up of objects from individual photographs, the collage generated according to embodiments of the present invention; and

"FIG. 12 is a simplified diagram showing the generation of 3D models according to embodiments of the present invention from originating sketches."

URL and more information on this patent application, see: COHEN-OR, Daniel; SHAMIR, Ariel; CHEN, Tao. Three-Dimensional Modeling from Single Photographs. Filed February 11, 2014 and posted August 21, 2014. Patent URL:

Keywords for this news article include: Software, Ramot at Tel-Aviv University Ltd..

Our reports deliver fact-based news of research and discoveries from around the world. Copyright 2014, NewsRx LLC

For more stories covering arts and entertainment, please see HispanicBusiness' Arts & Entertainment Channel

Source: Politics & Government Week

Story Tools Facebook Linkedin Twitter RSS Feed Email Alerts & Newsletters