Camera Arrangements With Backlighting Detection And Methods Of Using Same

What is this patent about?

’605 is related to the field of multi-camera arrays and image processing, particularly systems designed for telepresence and virtual reality applications. The background involves capturing images of a scene from multiple viewpoints to reconstruct a virtual viewpoint image, creating a sense of telepresence. A challenge arises when superimposing virtual reality elements onto these real-scene images, as inconsistencies in illumination can make the virtual elements appear unnatural.

The underlying idea behind ’605 is to capture not only the images of a real scene but also data representing the real-world illumination impinging on that scene. This illumination data is then used to model the lighting of virtual objects that are to be superimposed on the real scene. By matching the virtual object's lighting to the real scene's lighting, the combined image appears more realistic and natural.

The claims of ’605 focus on a camera system comprising multiple cameras, each transmitting image data through its own output connection. Data communication circuitry then merges these data streams, and at least one processing chip combines the data into a single merged data set. Crucially, the individual camera data represents images taken with different camera parameters, and the merged data can be displayed as a single, coherent image or stored in a single image file.

In practice, the system uses an array of cameras, each potentially with different exposure settings or viewpoints, to capture a scene. Simultaneously, illumination data capture units , which may include photosensitive chips or even dedicated cameras, record the ambient lighting conditions. This illumination data is then used to render virtual objects with matching lighting, ensuring they blend seamlessly into the real scene when the images are combined. The system can also adapt to changing lighting conditions by capturing successive sets of illumination data.

The key differentiation from prior approaches lies in the simultaneous capture and utilization of real-world illumination data. Traditional virtual reality systems often rely on assumed or manually configured light sources, which can lead to mismatches when combined with real-world imagery. By capturing and using actual illumination data, ’605 enables a more realistic and immersive experience, particularly in applications where virtual and real elements are combined, such as in training simulations or entertainment systems. This approach ensures that virtual objects respond to lighting changes in the real environment, maintaining visual consistency.