This article was originally published at Avnet's website. It is reprinted here with the permission of Avnet.
By Jim Beneke
Vice President, Global Technical Marketing
Avnet Electronics Marketing
Most of us have heard about Google Glass. Many of us have seen the Microsoft Kinect for the Xbox 360 video game console. Some of us may even have a car with a rear view camera, pedestrian detection or a lane departure warning system. What you may not realize is that all of these devices have something in common – embedded vision. What exactly is embedded vision, why will it revolutionize many of the products and systems that exist today and why should you care?
In basic terms, embedded vision is the combination of an image sensor or camera with some sort of embedded processing system. Both of these elements have existed for more than 40 years in some form or fashion, yet only recently have they come together in such a way as to enable an entire new paradigm of machines that see. What we are observing today is the result of a convergence in lower power, higher performance, smaller size and lower cost in the key elements that make up vision-enabled systems. Combine this with improvements and breakthroughs in software algorithms and data manipulation techniques, and the result is the dramatic acceleration and adoption of embedded vision.
Elements of an Embedded Vision System
Describing something as an embedded vision system is a very broad definition encompassing many different features or capabilities. However, most vision based systems tend to include some variation of the following functions:
- Image Sensor/Camera: Today many embedded vision systems make use of CMOS image sensors. Dramatic improvements in these sensors have taken place since they were first invented at NASA’s Jet Propulsion Laboratory in 1995. Resolution in terms of pixels and speed in terms of frames per second are two areas where CMOS sensors have been able to leverage Moore’s Law to their advantage. Power and cost have been decreasing over the years as well, which is a primary driver in the expansion of embedded vision applications. ON Semiconductor is one Avnet Electronics Marketing supplier that offers a family of image sensors targeted at industrial applications. Specialized cameras that include image sensors are also being developed, providing unique capabilities over a standard image or video capture. Time-of-Flight (ToF) sensors are one example since they use the speed of light to determine distances of various points in an image relative to the sensor. These sensors are finding uses in gesture recognition, object classification and automotive safety.
- Processor: Since we are talking about embedded vision, we are implying the use of embedded processors verses PCs or workstations. This is an important distinction because only in the last 15 years has the performance of these processors reached a level where they could adequately handle real-time video. What has emerged over the last 10 years is a group of specialized processors that implement unique architectures or dedicated accelerators specific to image and video processing. General purpose Digital-Signal-Processors (DSPs) are giving way to highly optimized video processors capable of performing very efficient pixel-based processing and frame-based processing. These are often combined with ARM® cores which provide the higher level processing or intelligence in the system as well as system management and connectivity functions. Just like the CMOS image sensor, the embedded vision processor is leveraging Moore’s Law and making significant improvements in processing capability, reduced power, higher integration and reduced costs. Companies like Analog Devices, Freescale, Texas Instruments and Xilinx all offer processing solutions tailored for embedded video applications.
- Memory: Processing data from the image sensor or camera often requires the storage of either all or some parts of the video data as it streams through the system. The density of the memory is less a driver than the IO data bandwidth between the processor and the memory. In the past, specialized memory such as video RAM was required to maintain the performance needs of the processing system. Although effective, these memories included a cost premium. With speed and density advances in DDR memory driven by the PC industry, we can now use standard DDR2/DDR3 devices, yielding significant cost saving in the overall system budget.
- Software/Algorithms: Having the required hardware pieces is only half the battle. Software and more specifically, specialized vision algorithms are required to manipulate and analyze the flood of incoming video data. In 2000, the process for developing and implementing these algorithms changed. Spun from an Intel Research initiative started in 1999, Open Source Computer Vision Library or OpenCV (www.opencv.org) was released to the public in 2000 as an optimized, open source library of C/C++ functions centered on vision-based applications. Since then, periodic releases of the OpenCV library has resulted in additional functionality and further optimizations to the various vision algorithms, making them easier to port and run on embedded processors. In addition to the free, open source OpenCV functions, commercially developed vision libraries offer an alternative option. Many 3rd parties offer specialized vision and video processing solutions for various applications. A model-based design methodology from MathWorks provides another option with a complete system-level approach to designing embedded vision systems. The Mathworks’ tools support everything from system modeling and simulation to automatic code generation and hardware validation.
Embedded Vision Applications
Vision-enabled and embedded vision systems have been around for a number of years. Anything with a camera could be classified as vision-enabled. However, over the next three to five years, it is very likely you will begin to see an explosion of embedded vision applications. Driven by the advances in sensors, processors and software, these systems will leverage the price, performance and power advantages in an exponential fashion. Nearly every market will be impacted by the technology, from industrial, medical, automotive, to consumer,aerospace/defense and security. More importantly, there will be a range of embedded vision systems offered, depending on the processing performance needs and supportable product cost.
Not all applications require the biggest and fastest devices to solve a problem. If you want to track the movement of a person in a room, you can likely achieve this with a low resolution (VGA) style image sensor and a processing system analyzing movement at a couple of frames per second. Likewise on the high-end, the introduction of image sensors that can output hundreds of megapixel resolutions or thousands of frames per second, are enabling high-precision machine control and inspection equipment that was unheard of just 5 years ago.
Bar code scanning will soon employ embedded vision to better find and read bar codes. Other systems will identify items that have no bar codes on them and determine what they are through the use of vision analytics. Security systems will no longer use keys and cards to identify users, but will instead recognize users and owners by capturing an image of their face. Safety systems will be more accurate and faster. Appliances and instruments will no longer require you to press buttons or touch them, but instead will see you and respond to your hand movements or gestures. Robotics will become smarter and more autonomous as they can better identify objects and their surroundings. Cars will continue to become safer and easier to drive through the use of specialized embedded vision systems both in-cabin and external.
Conclusion
Embedded vision is growing. Many next-generation products will include some sort of vision capability to detect, recognize, analyze, categorize, or track objects or people. At Avnet, we offer all the pieces to help you get started. From the image sensor or camera to the embedded processor and memory sub-system, we sell and support the leaders in this evolving technology.
We are one of the founding members of the Embedded Vision Alliance (www.embeddedvisionalliance.com) where you can obtain a wealth of information, tutorials and videos related to embedded vision. We also offer Avnet created development kits that support embedded vision design and prototyping. The MicroZed™ Embedded Vision Development Kit builds on the MicroZed SOM by providing a video specific carrier card. The kit includes hardware, software and IP components necessary for the development of custom video applications. The development kit includes the MicroZed 7020 SOM, as well as the Embedded Vision Carrier Card, which includes on-board HDMI input/output interfaces and a camera connector for optional camera modules. The FinBoard™ Embedded Vision Development Kit (www.finboard.org) provides a low-cost, vision optimized Analog Devices Blackfin® BF609 processor in a complete kit with software development tools, debugger and numerous reference designs. FinBoard is ideal for creating low-to-mid range machine vision, security and video analytics solutions. Another option is Avnet’s ZedBoard (www.zedboard.org) which is targeted at higher performance embedded vision applications with its Xilinx Zynq®-7000 All Programmable SoC. Be part of this next revolution and start exploring the endless possibilities that can enhance and differentiate your future products. The future of machines that “see” is happening now. For more information, the latest tools, kits and trainings, visit the Avnet Embedded Vision site.
