News Column

Patent Issued for Portable Remote Control Device Enabling Three-Dimensional User Interaction with at Least One Appliance

July 23, 2014

By a News Reporter-Staff News Editor at Electronics Newsweekly -- From Alexandria, Virginia, VerticalNews journalists report that a patent by the inventor Rafii, Abbas (Palo Alto, CA), filed on June 29, 2012, was published online on July 8, 2014.

The patent's assignee for patent number 8773512 is Aquifi, Inc. (Palo Alto, CA).

News editors obtained the following quote from the background information supplied by the inventors: "Remote control of an electronic appliance by a user is often accomplished using a handheld device with a number of buttons, e.g., a TV remote control. As the user presses a button, an associated wireless signal is emitted to remotely control the appliance, perhaps changing channels, or volume level. But such mechanisms are quantized in the sense that their remote control output command sets are rather limited by the number of discrete remote control buttons.

"Some remote control devices include a touch screen that the user contacts to control a remote device. But such remote control device can only detect user interactions that contact the touch screen, and cannot detect user gestures made in three-dimensional space away from the touch screen surface. Some touch screen devices employ electromagnetic and/or capacitive technologies that try to sense presence of an object close to the touch screen. This proximity requirement tends to restrict sensing natural ad hoc gestures made in three-dimensional space.

"What is needed is a method and portable or handheld remote control device to sense user interaction, including natural interface gesture interaction, in a three-dimensional zone to remotely control at least one appliance. Typically the depth of this zone will be less than the arm length (.ltoreq.1 m) of the user, which is a comfortable range distance for the device. The remote control device is preferably implemented with low power consumption generic inexpensive components that have small form factor. The remote control device should process data to reduce imaging ambiguities and substantially expedite processing of a sparse number of three-dimensional data points needed to identify user gestures, including natural user interactions with the device. Transmitted commands from the remote control device system should meet applicable industry standards of accuracy and resolution.

"The present invention provides such systems and methods for implementing such systems."

As a supplement to the background information on this patent, VerticalNews correspondents also obtained the inventor's summary information for this patent: "The present invention provides a handheld or portable remote control device that is used to sense user gestures including natural gestures made within a three-dimensional hover zone. The device preferably includes an acquisition system, a processor system, and an input/output system with transceiver capability.

"The acquisition system captures user position and movements within the hover zone, using at least one and preferably at least two two-dimensional cameras operationally coupled in a grid to optically capture user images two-dimensional (x,y) data. Each camera has an image plane that defines (x,y) coordinates, which plane is substantially parallel to the front service of the device. As used herein, a z-axis is defined normal to the x-y plane, and projects outwardly therefrom in the direction of the user. As the remote control device is typically handheld, a range of operation comfortable to the user would be out to about an arm's length, e.g., typically z.ltoreq.1 m from the device. In another embodiment the acquisition system includes a time-of-flight (TOF) system that can acquire (x,y,z) three-dimensional data directly from a user in its hover zone. Structured light may be used as a three-dimensional acquisition system to detect gestures, assuming reliably precise operation over a z.ltoreq.1 m or so range is attained at an acceptable implementation and power consumption cost. In addition to an acquisition system, a device according to the present invention further includes a signal processing system that processes data captured by the acquisition system. The signal processing system may be distributed in that it can be disposed within the device housing or, in some embodiments, disposed at least partially external to the device housing, e.g., in a cloud or in a remotely located processor system. Any such externally disposed processing system communicates preferably wirelessly with the remotely located appliance, and with the remote controller device, directly or indirectly through the appliance. The device input/output system includes a transceiver that can communicate wirelessly with appliance(s) to be controlled, and with any externally disposed signal processor system(s).

"When the acquisition system includes two-dimensional camera(s), each camera captures from its own vantage point on the device gestures made by the user within the three-dimensional hover zone. This hover zone is defined by the interaction of the three-dimensional fields of view (FOV) of each camera in the grid of cameras. Cameras can be placed such that the hover zone grazes the user-facing surface of the device so as to enable user interaction directly on the device surface (x,y,0) as well as within the hover zone (x,y,z>0). In preferred embodiments the acquisition system includes at least two two-dimensional cameras, and the processor system receives substantially simultaneously two-dimensional image data from each camera. The processor system executes at least one algorithm that intelligently discards typically at least 99% of the incoming data, while extracting a relatively few so-called (x,y,z) landmark points on the imaged user that suffice to identify gesture(s). Exemplary landmark points on a hand might be the hand centroid, each finger tip, etc. Imaged user position and movements, as indicated by the locus of such (x,y,z) locations, are interpreted by the processor system as user gestures including natural user gestures.

"Whether processor system components are disposed within or external to the device housing, processor system analysis preferably models each two-dimensional camera to emulate pinhole optics, which justifies use of rectified images and epipolar line geometry. These techniques enable rapid elimination of nearly all of the data being processed and rapid resolution of data ambiguities. The result is that a sparse set of perhaps only a dozen (x,y,z) landmark points on the imaged user are determined, substantially in real-time and on demand, using low power consuming, inexpensive processors.

"The processor system can translate a recognized user gesture into a user-intended command. If the processor system is disposed within the device housing, the device input/output transceiver system wirelessly broadcasts the commands to the remote appliance that the user gestures were intended to control. In embodiments in which at least some of the processor system is disposed external to the device housing, unprocessed (x,y) image data from the acquisition system can be transmitted (via the input/output transceiver system) to externally disposed processor system(s), but at the cost of high bandwidth. More preferably, housed within the remote control device housing is at least some of the processor system such that some pre-processing of the raw (x,y) camera data is carried out and transmitted with a narrower bandwidth to at least one externally disposed processing system. The externally disposed processing system then processes the transmitted data, discerns the landmark points, identifies gestures, and then transmits appropriate commands to the appliance that the user wishes to control. In some embodiments the externally disposed processing system can transmit commands to the device, whose input/output transceiver system then broadcasts commands to the appliance in question.

"In some embodiments, the remote control device includes a display screen that can replicate at least portions of imagery on the device being controlled by user gestures, such device being perhaps a smart TV, a computer display, etc. The user can view and interact with such images displayed on the handheld device, to control the appliance. If desired, the user can also interact with the device while viewing a display on the appliance. Without limitation, appliances that may be remotely controlled with the present invention include a TV, a computer system, a set top box, a game console, a kiosk, a signage unit, as well as 'dumb' appliances such as a lighting system, a door lock system, etc.

"In another embodiment, the acquisition system is a time-of-flight (TOF) system that determines depth distance to the user by comparing phase shift between emitted optical energy and such energy reflected back from the user. The TOF system creates true three-dimensional data, albeit a great deal of such data, representing locations on the user. Such data is discerned by the processor system as user-made gestures that are used to remotely control the appliance using the transceiver system in the handheld remote control device.

"Other features and advantages of the invention will appear from the following description in which the preferred embodiments have been set forth in detail, in conjunction with their accompanying drawings."

For additional information on this patent, see: Rafii, Abbas. Portable Remote Control Device Enabling Three-Dimensional User Interaction with at Least One Appliance. U.S. Patent Number 8773512, filed June 29, 2012, and published online on July 8, 2014. Patent URL:

Keywords for this news article include: Aquifi Inc., Electronics, Signal Processing.

Our reports deliver fact-based news of research and discoveries from around the world. Copyright 2014, NewsRx LLC

For more stories covering the world of technology, please see HispanicBusiness' Tech Channel

Source: Electronics Newsweekly

Story Tools Facebook Linkedin Twitter RSS Feed Email Alerts & Newsletters