Stereoscopic 3D Camera For Virtual Reality Experience

Patent No. US10979693 (titled "Stereoscopic 3D Camera For Virtual Reality Experience") was filed by Bluwhale Ai Inc on Feb 28, 2018.

What is this patent about?

’693 is related to the field of stereoscopic imaging and virtual reality. It addresses the problem of capturing and displaying high-quality 3D video, particularly with wide fields of view, for immersive VR experiences. The background includes challenges in manufacturing consistent stereoscopic camera systems, compensating for lens distortions, and stabilizing video captured by moving cameras. Existing VR systems often struggle with these issues, leading to a less realistic and comfortable user experience.

The underlying idea behind ’693 is to enhance the VR experience by improving the quality and stability of stereoscopic 3D video. This is achieved through a combination of techniques, including using wide-angle lenses to capture a large field of view, embedding calibration data into the video stream to correct for manufacturing variances, and applying video stabilization algorithms to compensate for camera motion. The core insight is that by addressing these issues, a more immersive and realistic VR experience can be created.

The claims of ’693 focus on methods for processing stereoscopic video data to improve its quality and stability. Specifically, claim 1 covers a method for mapping stereoscopic data from a 3D space to a 2D space and filtering motion data. This involves obtaining a video stream from a stereoscopic camera, identifying a reference frame and surrounding frames, characterizing motion by comparing frames, filtering the motion, calculating matrices based on the filtered motion and camera parameters, and applying these matrices to the reference frame to obtain a modified, stabilized frame. Claim 5 is similar, but more general, and covers video playback.

In practice, the invention uses a stereoscopic camera equipped with two wide-angle lenses, each paired with an image sensor. The camera captures video streams along with metadata about the camera's parameters and any detected motion. This metadata is then used to correct for lens distortions and stabilize the video. The motion filtering process analyzes frames before and after a reference frame to estimate and compensate for unwanted camera movement, resulting in a smoother and more stable video playback.

The invention differentiates itself from prior approaches by embedding calibration data directly into the video stream, allowing for real-time correction of lens distortions and manufacturing variances. Furthermore, the video stabilization algorithm uses a combination of frame analysis and motion sensor data to achieve a high level of stability, even when the camera is in motion. This combination of features results in a more immersive and comfortable VR experience, with a wider field of view and reduced motion sickness.

How does this patent fit in bigger picture?

Technical landscape at the time

In the mid-2010s when ’693 was filed, stereoscopic 3D video capture at a wide field of view was typically implemented using two cameras with wide-angle lenses, at a time when processing power and storage capacity were sufficient to handle the large data streams. At that time, virtual reality headsets commonly relied on motion sensors to track head movements and adjust the displayed view accordingly, when hardware or software constraints made real-time processing of high-resolution 3D video a non-trivial engineering challenge.

Novelty and Inventive Step

The examiner approved the application because the allowable claims, which relate to mapping stereoscopic data from a three-dimensional space to a two-dimensional space and filtering motion data, were not taught by the prior art references of record. The claims involve identifying a reference frame, comparing sets of frames before and after the reference frame to characterize motion, filtering the motion, calculating matrices based on the filtered motion, and applying these matrices to obtain a modified reference frame.

Claims

This patent includes 11 claims, with claims 1 and 5 being independent. The independent claims are directed to methods for mapping stereoscopic data and filtering motion data, and methods for video playback, respectively, both involving matrix operations on video frames. The dependent claims generally elaborate on the independent claims by specifying variations and additional features related to the number of frames, cameras, and camera parameters.

Key Claim Terms New

Definitions of key terms used in the patent claims.

Term (Source)Support for SpecificationInterpretation
Filtered motion
(Claim 1, Claim 5)
“The device simultaneously records two videos from two different perspectives using the two digital image sensors. Because of the wide-angle lenses, the videos are captured with a wide field of views (FOVs), e.g., 180 degrees. Such a field of view is wider than the field of view of human eyes. For example, humans' binocular visions typically have a field of view of 114 degrees (horizontally).”Motion data that has been processed to remove unwanted components, resulting in a value corresponding to the reference frame.
First matrix
(Claim 1)
“The device plays back the cropped 3D images or videos with a narrower field of view on the left and right portions of the display screen. The left eye of the user is in front of the left portion of the display screen, and views the left channel of the cropped 3D images or videos. Similarly, the right eye of the user is in front of the right portion of the display, and views the right channel of the 3D images or videos.”A matrix calculated based on the filtered motion.
Second matrix
(Claim 1)
“When the user moves the head to a different position or orientation, the motion sensor detects the movement. Based on the new head position or orientation, the device determines new positions (or new sizes as well) of the cropping windows and generates in real time the cropped 3D images or videos. Therefore, the field of view experienced by the user eyes changes correspondingly as the user moves the head.”A matrix representing a mapping from a three-dimensional space to a two-dimensional space associated with a plane of the stereoscopic camera in the absence of motion.
Stereoscopic camera
(Claim 1, Claim 5)
“The disclosed embodiments disclose a stereoscopic device (also referred to simply as the “device”) that captures three-dimensional (3D) images and videos with a wide field of view and provides a virtual reality (VR) experience by immersing a user in a simulated environment using the captured 3D images or videos. In some embodiments, the stereoscopic device includes two wide-angle lenses (e.g., fish-eye lenses) to captures 3d images and videos with a wide field of view.”A camera that captures stereoscopic video streams, providing a three-dimensional effect.
Third matrix
(Claim 1)
“In this way, the user can move the head to look freely in different directions at different parts of the 3D image or 3D video within the wide field of view (e.g. 180 degrees). Because the field of view of the captured 3D image or 3D video (prior to cropping) is larger than the field of view of the human vision, the user experiences a sense of presence in a virtual environment created based on the captured 3D image or 3D video.”A matrix that is an inverse of the first matrix.

Patent Family

Patent Family

File Wrapper

The dossier documents provide a comprehensive record of the patent's prosecution history - including filings, correspondence, and decisions made by patent offices - and are crucial for understanding the patent's legal journey and any challenges it may have faced during examination.

  • Date

    Description

  • Get instant alerts for new documents

US10979693

BLUWHALE AI INC
Application Number
US15908692
Filing Date
Feb 28, 2018
Status
Granted
Expiry Date
Aug 24, 2036
External Links
Slate, USPTO, Google Patents