From drones to handheld devices, the rising demand for video cameras has made them ubiquitous, constantly driving down size and cost while pushing up resolution and overall quality. One of the main challenges in this field is stabilizing the image to generate clear, smooth footage. In this post, I would like to discuss the challenges that stabilization poses, and the pros and cons of existing solutions. I'll also give a brief technical overview of CEVA's software-based stabilization solution here.
Why Do We Need Image Stabilization?
Smartphones are designed to be lightweight and easy to carry around and operate with just one hand. This basic design requirement is detrimental to steady photography. In order to take clear pictures, even more so to capture sharp video, it helps to have a heavy, sturdy device that will move around as little as possible when shooting. The added stability of a tripod or other support is rarely available when shooting from a smartphone, however. And in addition to the natural movement of the hand, deliberate actions like zooming, panning or rotation also create unwanted jitter, which then needs to be differentiated and filtered from desired movement.
Besides smartphones and other handheld devices, video camera functions are becoming prevalent in many other consumer products, each bringing its own unique stabilization challenges. Wearable devices, for example, are even lighter (therefore shakier) than phones, making the task of stabilization even more difficult. Wearables that support augmented reality, such as Google Glass, must have a stable source image in order for features like object detection to function correctly. In the automotive industry, autonomous driving technologies like ADAS (advanced driver assistance systems) need to deal with a camera moving at high speeds while capturing objects moving just as fast. Aerial cameras deployed in drones have the added complexity of a yet another axis of movement, plus the additional vibrations of motors and propellers (Figure 1).
Figure 1. Drone with a camera connected to a gimbal (source: Pixabay)
All these shakes and jitters, in all of these applications, need to be smoothed out in order to enjoy high quality video, as well as to identify critical objects and perform complex tasks based on this information.
Stabilizing Ultra-High-Definition Video Without Prematurely Draining the Battery
One of the emerging challenges of video stabilization is to support UHD (ultra-high definition) video at 30 FPS (frames per second) and even 60 FPS. The solution must also be scalable to support the different resolution modes available on the device. These requirements require significant processing capabilities. But at the same time, the need for low power usage and longer battery life demands efficient performance; bandwidth, power consumption and processing time all need to be kept to a minimum. Another challenge is to overcome rolling shutter distortion, which is introduced by a typical CMOS sensor during the scanning of the image. Finally, the stabilization must take into account at least 4-axis motion: left-right, up-down, forward-backward and rotation on one axis, while drones exhibit 6-axis motion, which includes rotation on all axes (Figure 2).
Figure 2. Depiction of 6-axis motion (source: public domain)
Comparing Stabilization Approaches: Optical vs. Digital
Existing technologies that deal with stabilization issues can be divided into two broad categories: OIS (optical image stabilization and DVS (digital video stabilization), the latter also known as EIS (electronic image stabilization). OIS mechanisms employ hardware components inside the camera module, which serve to minimize the vibrations and shakiness. These solutions provide good performance, but only target high-end devices because of the related costs.
Digital stabilizers can be embedded in silicon or implemented as programmable software, and can be tuned to work with different lenses and devices. Digital stabilizers are a low-cost solution that provide good performance and are not limited to high-end devices, but can be used in low and mid-range devices as well.
The following video shows a comparison between the iPhone 6s, which uses DVS, and the iPhone 6s Plus, which uses OIS.
While both of these phones are on the high end of today's range of available smartphone video cameras and both demonstrate excellent video resolution, it's clear that the iPhone 6s Plus delivers much smoother panning and less shaking. These advancements are due to the OIS technology present in the 6s Plus.
However, OIS inclusion is part of the reason that the iPhone 6s Plus is also significantly larger, heavier and more expensive than its conventional iPhone 6s counterpart. Another big advantage for digital video stabilization is that the alternative OIS mechanism needs to be tailored for each camera module with a specific lens. This tailoring requires very sensitive calibration, which can be easily be skewed if the device is dropped or otherwise receives some sort of impact.
Conversely, if the stabilization is performed digitally via programmable software, calibration can easily be performed for different camera modules and is also not as prone to damage. The same calibration software can be used for many different lenses and devices. The software can also be easily maintained and improved, with features added via firmware updates, without replacing any hardware. Software-based stabilization can also be combined with other software modules such as Super-Resolution, HDR or video encoding. Since there is no standard for DVS, the implementation serves as a strong differentiator between manufacturers, each of which continually strives to offer better solutions. And a software-programmable solution also enables rapid and easy adaptation, thus shortening time to market.
CEVA DVS Solution: Algorithm Flow
CEVA’s solution for DVS is comprised of two phases, motion estimation and motion correction, each consisting of 3 steps (Figure 3). The first step, feature detection, is implemented using a Harris corner detection algorithm. Pyramid creation is also performed at the same time, for the hierarchical motion search in the following stage. The next step, feature tracking, is performed using the KLT (Kanade-Lucas-Tomasi) optical flow algorithm, and also includes smart refresh for lost features. The camera motion model is estimated by the RANSAC (random sample consensus) algorithm; the estimated model describes camera motion in four axes.
Figure 3. CEVA DVS solution flow (source: CEVA)
Once the estimation model is ready, the correction phase commences. The first step in this phase is motion smoothing. It is performed using a Kalman filter, which smoothes the motion in all four axes. The smoothing might normally introduce black margins at the edges of the output video, but these are eliminated by a logical block that adjusts the compensation parameters. Next, global motion and rolling shutter distortion in the frame are corrected by applying an affine transformation to each frame. The transformation parameters are calculated so as to remove undesired shaky motion and distortion.
The result is a software module for real-time video stabilization that is fully optimized for smooth and clear video output. Our solution supports various lighting conditions and runs at very low power consumption, performing stabilization of 4K-resolution video at 30 fps while using less than 40 mW on the CEVA-XM4.
A Flexible Solution Can Prevent a Shaky Future
For the foreseeable future, it seems we will be seeing more and more video cameras built into more and more devices, with increasing frame resolutions and rates and shakier use-case scenarios. The diversity of devices, lenses, resolutions and other variables in this exciting and evolving market requires a flexible and scalable solution that can be implemented quickly and updated as necessary. In these conditions, a digital programmable module for video stabilization can be the best solution.
To learn more about this topic and view demonstrations of CEVA’S DVS technology, visit http://www.ceva-dsp.com/CEVA-XM4.
By Liran Bar
Director of Product Marketing, CEVA
