News Column

Patent Issued for Portable Electronic and Method of Processing a Series of Frames

September 11, 2014



By a News Reporter-Staff News Editor at Computer Weekly News -- According to news reporting originating from Alexandria, Virginia, by VerticalNews journalists, a patent by the inventors Wernersson, Mats (Helsingborg, SE); Heringslack, Henrik (Lomma, SE), filed on July 23, 2012, was published online on August 26, 2014.

The assignee for this patent, patent number 8818097, is Sony Corporation (Tokyo, JP).

Reporters obtained the following quote from the background information supplied by the inventors: "A wide variety of portable electronic devices nowadays is provided with components for capturing image data. Examples for such electronic devices include portable communication devices which may include a camera. For illustration, modern mobile telephones or personal digital assistants may be equipped with a camera. With increasing performance of the image capturing components of such devices, it becomes increasingly desirable to perform additional image processing on the captured images.

"Owing to the computational complexity of some image processing functions, one approach to address the above need of providing additional image processing features is to interface the portable electronic equipment with a computer. The computer may execute dedicated software for processing images captured by the portable electronic equipment. Because captured images must first be transferred to the computer for further processing, this approach may be inconvenient for a user.

"Another approach is to implement additional image processing functions in the portable electronic device. For illustration, functions for forming composite images may be provided in the portable electronic device. Such functions may include steps such as computing a difference image of a first image having an object of interest positioned in front of a background and of a second image only showing the background, to thereby identify the outline of the object of interest. While such operations require only basic image processing functions, such as computing difference images, the dedicated user which are required to determine the object outline may be inconvenient. Also, techniques which require a background to remain essentially the same while successive exposures are being recorded may be of limited value to a wide variety of scenarios where the background is not stationary, as is the case when at least part of the background shows moving objects."

In addition to obtaining background information on this patent, VerticalNews editors also obtained the inventors' summary information for this patent: "Accordingly, there is a continued need in the art for a portable electronic equipment and for a method which address some of the above shortcomings. In particular, there is a continued need in the art for a portable electronic equipment and for a method which allows a series of frames captured by the portable electronic equipment to be processed without requiring the frames to be transferred to an external computer. There is also a continued need in the art for portable electronic equipment and for a method which is configured to form composite images from a series of images, without requiring a background to remain stationary while the series of frames is being captured.

"According to an aspect, a portable electronic equipment is provided. The portable electronic equipment comprises an image sensor and a processing device. The image sensor is configured to capture a series of frames in a time-sequential manner. The processing device is coupled to the image sensor and is configured to process the series of frames. The processing device is configured to perform an image segmentation for respectively each frame of at least a subset of the series, in order to respectively identify a section of the frame which represents a pre-determined object. The processing device respectively performs the image segmentation based on a depth map for the respective frame. The processing device is configured to superimpose the identified sections to generate a sequence image.

"The portable electronic equipment provides an image processing feature which allows a sequence image to be formed from a series of frames. Thereby, a sequence image may be generated which shows an object movement in a stroboscopic manner. The series of frames may, for example, be captured with a pre-determined repetition rate, such as in a continuous exposure mode. By using depth maps for the frames in the image segmentation, a distance-selective sequence image may be generated. An object which moves approximately at a distance relative to the portable electronic equipment may respectively be identified in plural frames, using the depth maps, and a sequence image showing the different object positions in one image may be generated.

"The processing device may be configured to establish a distance between the given object and the portable electronic equipment using a depth map for a given frame of the series. The processing device may be configured to identify the section representing the given object in another frame of the series based on both the established distance and the depth map for the other frame. This allows the distance information to be used when generating the sequence image, so that an object may be tracked through plural frames using the depth maps.

"The processing device may be configured to determine whether the pixels of the depth map for the other frame correspond to distances which lie within a distance range, which distance range is determined as a function of the established distance. Thereby, object tracking through plural frames may be facilitated, and variations in the object distance from the portable electronic equipment which may occur while the series is being shot may be accounted for.

"The processing device may be configured to identify, for respectively each frame of at least the subset, the section representing the given object such that a distance between the given object and the portable electronic equipment as specified by the depth maps corresponds to the established distance. This allows a distance-selective sequence image to be generated, where an object moving at the established distance is shown in a stroboscopic manner.

"The distance between the given object and the portable electronic equipment may correspond to a distance of a focal plane from the portable electronic equipment.

"When an image sensor is used which allows digital focusing or digital re-focusing to be performed, the position of the focal plane may be set equal to the established distance and the frames may be re-computed based on this position of the focal plane, prior to forming the sequence image.

"The portable electronic equipment may further comprise an input interface configured to receive a user input indicating the given object in the given frame. The processing device may be coupled to the input interface and may be configured to establish the distance between the given object and the portable electronic equipment as a function of the user input. If plural objects can be identified, based on the depth map for the given frame, the input interface allows the user to indicate for which one of the different objects the sequence image is to be generated, showing the movement of the object selected by the user. The input device may include a touch-sensitive or proximity-sensing display.

"The processing device may be configured to superimpose the identified sections as layers onto one of the frames to generate the sequence image. The processing device may be configured to superimpose the identified sections as layers onto a background image which may be different from each one of the frames of the sequence.

"The portable electronic equipment may further comprise an optical output device. The processing device may be coupled to the optical output device and may be configured to control the optical output device to output the generated sequence image.

"The image sensor may include a plurality of image sensor sections, each of the image sensor sections being respectively configured to capture an image. Each one of the image sensor sections may have a dedicated focusing optics. The plural image sensor sections may respectively be provided on separate charge-coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor. The plural image sensor sections may be an array camera. Plural image sensor sections may be combined on one CCD or CMOS sensor. Digital refocusing or computation of depth maps can be performed using outputs of the plural image sensor sections.

"The processing device may be configured to process, for each frame of at least the subset, images captured by the plurality of image sensor sections to compute the respective frame and to compute the depth map for the respective frame.

"The image sensor and processing device may be mounted within a housing of the portable electronic equipment.

"According to another aspect, a method of processing a series of frames captured by an image sensor of a portable electronic equipment is provided. In the method, for respectively each frame of at least a subset of the series, a section of the frame is identified, the section representing a given object. Image segmentation is respectively performed based on a depth map for the respective frame to identify the section. The identified sections are superimposed to generate a sequence image.

"A distance between the given object and the portable electronic equipment may be established using a depth map for a given frame of the series. The established distance may be used when identifying the section representing the given object in another frame of the series, based on both the established distance and the depth map for the other frame. Thereby, information from plural depth maps may be combined to identify the section in the other frame.

"Pixels of the depth map for the other frame may be compared to the established distance to identify the section representing the given object in the other frame.

"The method may comprise determining whether pixels of the depth map for the other frame correspond to distances which lie within a distance range, the distance range being determined as a function of the established distance.

"For plural frames of the series, the section representing the given object may respectively be identified such that a distance between the given object and the portable electronic equipment corresponds to the established distance. The established distance may be equal to a distance between a focal plane of the image sensor and the portable electronic equipment. Digital ref-focusing may be performed to set the distance between the focal plane of the image sensor and the digital electronic equipment equal to the established distance.

"The portable electronic equipment may comprise an input interface configured to receive a user input indicating the given object in the given frame. The distance between the object and the portable electronic equipment may be established in response to the user input.

"The identified sections may be superimposed as layers onto one of the frames to generate the sequence image.

"The method may comprise outputting the generated sequence image over an optical output device of the portable electronic equipment.

"The method may comprise capturing the series of frames by an image sensor of the portable electronic equipment.

"The image sensor may include a plurality of image sensor sections, each of the image sensor sections being respectively configured to capture an image. The series of frames and the depth maps for the frames may be computed from the output signals of the plural image sensor sections.

"The method may be performed by the portable electronic equipment of any one aspect or embodiment.

"According to yet another embodiment, there is provided a non-transitory storage medium storing instruction code which, when executed by a processor of a portable electronic equipment, directs the portable electronic equipment to perform the method of any one aspect or embodiment.

"It is to be understood that the features mentioned above and features yet to be explained below can be used not only in the respective combinations indicated, but also in other combinations or in isolation, without departing from the scope of the present invention. Features of the above-mentioned aspects and embodiments may be combined with each other in other embodiments."

For more information, see this patent: Wernersson, Mats; Heringslack, Henrik. Portable Electronic and Method of Processing a Series of Frames. U.S. Patent Number 8818097, filed July 23, 2012, and published online on August 26, 2014. Patent URL: http://patft.uspto.gov/netacgi/nph-Parser?Sect1=PTO1&Sect2=HITOFF&d=PALL&p=1&u=%2Fnetahtml%2FPTO%2Fsrchnum.htm&r=1&f=G&l=50&s1=8818097.PN.&OS=PN/8818097RS=PN/8818097

Keywords for this news article include: Sony Corporation.

Our reports deliver fact-based news of research and discoveries from around the world. Copyright 2014, NewsRx LLC


For more stories covering the world of technology, please see HispanicBusiness' Tech Channel



Source: Computer Weekly News


Story Tools






HispanicBusiness.com Facebook Linkedin Twitter RSS Feed Email Alerts & Newsletters