The first, "User Interface for Presenting Search Results for Multiple Regions of a Visual Query," describes how an overall visual query, such as a photograph, screen shot, or video frame is first submitted to a computing system, which then breaks it up into discrete pieces, possibly according to type. Each piece is then sent to a specific search system implementing a distinct visual query search process. A plurality of results is generated and an interactive results document is created and sent back. This document has one or more visual identifier for each piece of the overall picture with a selectable link to a particular result for that piece. The visual identifier may be a box or a translucent label which could be color-coded by type.
Even more provocative, though, is the second patent, "Facial Recognition With Social Network Aiding," which describes a computer-implemented method of processing facial image queries using social media data pertaining to the requester in order to obtain results. This is accomplished by identifying one or more individuals associated with potential image matches and measuring social connectivity between that image and the requester. These social metrics may be obtained from communication applications, social networking applications, calendar applications and other collaborative applications. Next, an ordered list of persons is generated by ranking the visual similarity between the queried image and potential image matches. Finally, results from the list are sent to the requester.
Google has a couple of ideas for protecting the privacy of Facebook users and the like. First, Google contemplates allowing facial recognition capabilities only to the person positively identified in a picture. A second possibility is based upon obtaining permission. Once a person has been positively identified, a request might be sent to that person asking to allow their image to be returned in future searches conducted by people within their social network.
The ideal of mining social network data to improve facial recognition has been explored by others, too, including researchers at Harvard, who published a paper on the topic in 2008 concluding the technology had promise. (Autotagging Facebook: Social Network Context Improves Photo Annotation).
Critics point to darker possibilities, though. Dictators and governments are already identifying people based upon photos posted in social media connected to their real names. After the Iranian protests were over in 2009, the government is known to have gone through Flickr, collected photos of protesters then published them on government websites.
Nonetheless, Google continues to develop such technology amid its broader overall effort to add visual and social networking capabilities to its search technology.