Rolling shutter reduction based on motion sensors

What is this patent about?

'140 is related to the field of image processing, specifically addressing the problem of rolling shutter distortion in images and videos captured by devices using CMOS sensors. This distortion arises when the sensor reads out different rows of the image at different times, and the camera or subject moves during the readout process. This is a common issue in modern electronic devices like smartphones and tablets, which often use CMOS sensors due to their low cost and power consumption.

The underlying idea behind '140 is to use motion sensor data (e.g., from a gyroscope or accelerometer) to estimate and compensate for the motion that occurs during the image sensor's readout process. Instead of analyzing the image content itself, the invention leverages the timestamped motion data to calculate a corrective transformation for each row or segment of the image, effectively counteracting the distortion caused by the rolling shutter effect. This approach allows for real-time or near-real-time correction without requiring computationally intensive image analysis.

The claims of '140 focus on a method, apparatus, and storage medium for reducing rolling shutter distortion. The core steps involve estimating the motion of a base row within the image frame, determining the motion difference between each other row and the base row, generating a perspective transform for each row based on this motion difference, and then applying these transforms to the image data to produce a corrected frame. The independent claims emphasize using motion sensor data to drive these corrections.

In practice, the invention works by first capturing a video frame and simultaneously recording motion data from the device's sensors. The system then divides the image into rows and selects a base row, often the center row. By interpolating the motion sensor data, the system estimates the motion of each row relative to the base row. This relative motion is then used to calculate a perspective transformation matrix for each row, which is applied to correct the geometric distortion. Finally, the corrected rows are assembled to form the final, distortion-reduced image.

This approach differs from prior art methods that rely on analyzing the image content itself to detect and correct rolling shutter distortion. By using motion sensor data, '140 avoids the computational complexity of image analysis, enabling real-time or near-real-time correction. Furthermore, the use of a base row and the calculation of relative motion simplifies the process, making it more efficient and suitable for implementation on devices with limited processing power. The timestamped correlation of motion and image data is key to the invention's effectiveness.