By Gaurav Agarwal, Frank Brill, Bruce Flinchbaugh, Branislav Kisacanin, Mukesh Kumar, and Jacek Stachurski
Texas Instruments
This is a reprint of a Texas Instruments-published white paper, which is also available here (2.1 MB PDF).
Overview
When a driver starts a car, he doesn’t think about starting an intelligent analytics system; sometimes, that’s precisely what he’s doing. In the future, we will encounter intelligent systems more often as embedded analytics is added to applications such as automotive vision, security and surveillance systems, industrial and factory automation, and a host of other consumer applications.
Texas Instruments Incorporated (TI) has been innovating in embedded analytics for more than 20 years, blending real-world sensor driving technologies like video and audio with embedded processors and analytics algorithms. TI provides software libraries and development tools to make these intelligent applications fast and easy to develop.
Now, high-performance, programmable and low-power digital signal processors (DSPs) are providing the foundation for a new wave of embedded analytics systems capable of gathering data on their own, processing it in real time, reaching conclusions and taking actions.
Abstract
This white paper explains how TI, together with members of the TI Design Network, are today empowering leading-edge embedded analytics systems in some of the most prominent application areas, including automotive, surveillance, access control and industrial inspection systems, as well as many emerging applications, including digital signage, gaming and robotics.
What is “embedded analytics”?
Embedded analytics technology unites embedded systems and the human senses to enable systems to analyze information and make intelligent decisions. Although embedded analytics technology appeals to a wide range of industries, there is a set of technical characteristics that most embedded analytics applications share. They are:
- Diverse algorithms: Embedded analytics draws on a myriad of mathematical, statistical, signal and image-processing techniques. It combines these with machine learning, pattern recognition and other types of algorithms. The way in which these algorithms are combined tends to be unique to the application, and each of the algorithms usually needs to be adjusted a bit. This makes programmable processors and flexible software, often in the form of re-usable software libraries, very important.
- Fast processing, predictable latency: Embedded analytics generates a tremendous computational load that must be processed in real time. Also, time allocated for processing must be bounded and deterministic. Otherwise, the timing of the system is thrown off. Advanced architectures with parallelism help in this regard.
- Data throughput: Practically all embedded analytics applications involve some form of extreme data throughput. Huge amounts of data are brought into the system from sensors, cameras, microphones and other input devices. This data must be processed quickly, and the results, often involving huge amounts of data, must be output just as rapidly. To maintain data throughput, embedded analytics systems need advanced solutions like hierarchical memory organization, advanced direct memory access (DMA) controllers and wide memory interfaces.
- Low power consumption: Many applications of embedded analytics are mobile or deeply embedded systems that may or may not have access to the power grid. Low power drain is often a must-have.
- Cost: Many systems with embedded analytics – such as IP security cameras, smart TVs and games – are cost sensitive, yet the technical requirements are considerable. Balancing the two is a challenge.
Automotive embedded analytics for Advanced Driver Assistance Systems (ADAS)
First introduced into the automotive market more than a decade ago, embedded analytics has become widespread to the point where it is a “must-have” feature on many cars. Outside and in the vehicle, TI’s DSPs, particularly the TMS320C6000TM DSP platform, enable the various vision and audio processing subsystems that form a vehicle’s embedded analytics system (Figure 1).
Figure 1. ADAS enables the car to assist the driver in avoiding dangers on the road.
Many, but not all, of the vision processing subsystems in automobiles are outward facing. That is, image sensors monitor the space around a car and perform a wide variety of analytics functions intended to assist the driver, protect the vehicle from possible damage, and safeguard objects and pedestrians in the roadway. For example, several vision-based subsystems, widely known as Advanced Driver Assistance Systems (ADAS), process the field of vision in front of the car and provide information directly to the driver. These subsystems include a lane departure warning system, which warns drivers when the vehicle begins to move out of its lane; high-beam assist, which adjusts the level of the car’s headlights automatically when the lights from an approaching vehicle are detected; traffic sign recognition, which ensures that drivers don’t miss speed limit changes and other important road signs; forward collision warning to help drivers avoid front-end collisions; and an object detection capability that can automatically take countermeasures to avoid pedestrians or obstructions.
Other types of ADAS systems can assist with parking maneuvers, monitor the entire area around the car as well as the driver’s rear- and side-view blind spots to provide warnings, sound alarms or automatic evasive actions, and offer night vision functionality based on infrared sensors. In many cars on the road today, an adaptive cruise control system with embedded analytics will automatically detect other vehicles based on vision or radar data, calculate the distance and adjust the speed of the car to maintain a pre-determined distance.
TI’s DaVinciTM video processors, including DM81x video processors, are key to enabling ADAS technology. The parallel architecture of these processors can handle many vision algorithms with the short latency necessary for these safety applications. In addition, the processors’ high performance is balanced by the sub-3-Watt power budget, a must-have for automotive applications. In the future, TI’s smart multicore, automotive grade (AEC-Q100) OMAPTM processors will unleash the high-performance and low-power capabilities necessary for collecting, analyzing and displaying information and warnings in real time.
Inside the vehicle, embedded analytics enables various hands-free voice recognition control systems for the vehicle’s infotainment system. For more than 30 years, TI has been in research and development of speech-recognition technology, and a portion of this research has been donated to the open source community in the form of the TI Embedded Speech Recognizer (TIesr).
TIesr is a medium-size speech recognition system intended for embedded applications in automotive, industrial controls, consumer products, appliances and other market segments that require that the speech recognition and analytic processing are performed locally in the device itself. It should be noted that some large-size, more powerful speech recognition/analytic applications are not true embedded systems. In certain cases, these types of applications will utilize a communication link and perform much of the processing remotely, often in a cloud computing client or server application.
Embedded analytics in the automotive industry will continue to evolve as new techniques are investigated and developed, and as technology providers like TI continue to innovate with low-power, programmable single and multicore DSPs and the tools that facilitate their rapid deployment. Three-dimensional (3D) vision systems, for example, are becoming an integral part of automotive embedded analytics. In recent years, extensive research has been compiled on stereoscopic vision, which deploys two cameras. Other vision-related techniques like structured light and time-of-flight systems could be employed with embedded vision algorithms that leverage 3D sensor measurements to solve problems requiring higher precision.
Security and surveillance embedded analytics
Security and surveillance systems have also incorporated embedded analytics for quite some time. Initially, analytics was employed in conjunction with data compression/decompression algorithms to optimize the communication bandwidth associated with security systems. This led to greater penetration of embedded analytics and, specifically, vision-related analytics for automated real-time monitoring applications of property and infrastructure, traffic conditions and others. In addition, a significant amount of off-line video analytics has been implemented for forensics purposes.
Besides vision analytics, sound-processing technologies are bringing embedded audio analytics to security applications as well. Alarms can be triggered by sounds of aggression, explosions, sirens, collisions, breakins and other sounds of trouble. Multiple microphones or sound sensors in surveillance applications are also implemented to analyze and determine where the source of certain sounds is located or the direction from which the sounds are coming.
In addition to vision- or sound-only implementations of analytics in security applications, embedded analytics has brought these two sensory technologies together in certain systems.
In sound-assisted video analytics (SAVA), audio analytics inspect the sound scene of a surveyed environment and provide additional information about activities not readily discerned from video. A system could detect glass breaking, and as a result of embedded analytics, a surveillance camera might be redirected to the region of interest where the sound originated. Or, the sound of an intrusion might trigger an increased resolution of certain cameras for better images. Also, audio annotation may help determine the relevance of a large amount of recorded surveillance video. Sound identification may warn of potential security risks even when they are partially obstructed or hidden, or before they appear within the camera’s field-of-view. Taking advantage of the complementary aspects of video and audio provides a powerful framework that can lead to system robustness for enhanced alarm detection rates.
Security systems that require embedded analytics can leverage many of the capabilities provided by TI’s C6000TM DSPs, DaVinciTM video processors and other system-on-chip (SoC) devices. In addition to their low power and powerful processing capabilities, these programmable devices are architected for high-bandwidth data movement. A comprehensive tools environment specific to embedded analytics ensures rapid development cycles and an accelerated time-to-market.
TI’s DaVinci DMVAx video processors are equipped with capabilities targeted at embedded analytic security applications. Some of these capabilities include integrated video analytics acceleration, the industry’s first vision co-processor, an image co-processor and a complete video processing subsystem capable of face detection, video stabilization, noise filtering and other functions. Based on an ARM9TM core, TI’s DMVAx processors are supported by TI’s Smart Analytics, which includes five fundamental embedded analytics functions: camera tamper detection; intelligent motion detection; trip zone, which detects and analyzes objects moving from one zone to another; object counting; and streaming metadata, which tracks and tags objects on a frame-to-frame basis (Figure 2).
Figure 2. Smart analytics are embedded on TI’s DaVinci DMVAx video processors.
An integral part of the DMVAx processors’ embedded analytics capabilities is TI’s smart codec technology for improving codec efficiencies in analytic applications. For example, smart codec technology might function in concert with face detection to allot more bits to the face in an image and thereby achieve higher resolution for this region of interest (Figure 3).
Figure 3. DMVA2 block diagram.
TI offers reference designs for digital cameras with Internet Protocol (IP) connectivity that simplify development and allow designers to concentrate on adding features that will differentiate their products from the competition. These reference designs are based on TI’s DaVinci video processors, including the DMVAx, as well as an IP camera software suite. TI’s Digital Media Video Analytics Library (DMVAL) contains much of the base functionality needed to assemble an embedded analytic security system. Another building block for embedded analytic applications, TI’s Vision Library (VLIB), accelerates the development of vision subsystems in embedded analytic systems for security, automotive and others.
TI’s TMS320C674x DSPs are ideal for audio analytics. The processor offers the floating- and fixed-point capabilities and parallel architecture needed for real-time processing of audio analytics algorithms, but with low power consumption and at a low cost.
Access control
Many biometric characteristics are used to verify identity, including hand and face geometry, retinal scans and fingerprint analysis. For example, fingerprint scanners are used for identity verification at public safety facilities, on cell phones and laptops, at health care facilities and even at the local gym to enable quick and easy access to personal information and secure buildings and to keep everyone else out.
Systems that process these applications take a “picture” of the hand, face, retina or fingerprint, analyze the image for biometric data, and store this data in a database used for future matching. These applications must often be ultra-low-power when they are on mobile electronics like cell phones and laptops. Slightly more performance is necessary to obtain the image of and perform processing on faces, irises and retinas (Figure 4).
Figure 4. Block diagram of fingerprint process system.
TI’s TMS320C55x ultra-low-power DSPs are ideal for residential or commercial fingerprint recognition systems. They fulfill the need for less than two seconds of recognition time for a system with a 100-user fingerprint template. Since the power consumption is the 16-bit DSP industry’s lowest, users only need to change the battery of battery-powered systems every few months. TI offers the C5515 DSP Fingerprint Development Kit to simplify development of this application. For face recognition, iris recognition and other higher performance biometrics applications, TI’s C674x DSPs and OMAP-L138 DSP+ARM® processor are ideal.
Industrial embedded analytics
Control systems, factory automation, robotics, automated optical inspection, currency inspection, traffic management and many other types of industrial systems incorporate various aspects of embedded analytics. Often, machine vision is central to these industrial systems, but many also include a range of sensor inputs not found in other types of embedded analytic applications, such as pressure, temperature, motion, sound and other sensors.
The ongoing and seemingly constant advancements in low-power yet high-performance DSPs have enabled greater levels of intelligence in all aspects of industrial embedded analytics utilizing machine vision. As a consequence, the cameras on the factory floor and the centralized vision processing systems they are connected to are all able to function as powerful platforms for additional analytics processing. A smart camera, for instance, might perform some of the image enhancement and refinement functions locally that had previously been performed in the central vision processing system. Then, the smart camera could analyze the image and respond to it by zooming in or out, or turning for a better angle. And since the central vision processing systems are not constrained by the low power budgets or small enclosures of smart cameras, multiple single and multicore DSPs can be added to the centralized image-processing subsystem to support high-order embedded analytics like 3D object analysis, surface texture analysis and more. See Figure 5 for a diagram of a typical industrial imaging system.
Figure 5. Typical industrial imaging system.
In industrial embedded analytic applications, the scalable processing power and both fixed- and floating-point capabilities of TI’s TMS320C66x multicore DSPs give these low-power and programmable devices the characteristics required by smart cameras, vision-processing systems and other rugged processing platforms. A host of software tools and libraries, including TI’s Multicore Software Development Kits (MCSDKs) also streamlines development.
TI’s C66x DSPs integrate one to eight C66x DSP cores and are based on TI’s scalable KeyStone multicore architecture. They have a wide array of peripherals integrated on-chip, including very high throughput interfaces to FPGAs and CPLDs that accelerate system design and reduce system cost. Combining the KeyStone architecture with extensive memory resources ensures that each processing core will function at its fullest.
C66x DSPs are well suited to a wide variety of industrial applications, including optical defect inspection, part identification, high-speed barcode readers, color inspection, optical character readers (OCR), traffic management, currency inspection and high-end industrial printer/scanners.
Emerging embedded analytic applications
As an enabling technology, embedded analytics is so adaptable and malleable that it can emerge and be deployed in surprisingly unrelated and disparate places. Frequently, its appearance is unexpected. Typically, it disrupts the status quo in an application segment and takes it to a higher and more exciting level.
Embedded analytics is the engine behind robotics, augmented reality and a range of new natural user interfaces incorporating 2D or 3D gesture recognition and/or depth sensing. These capabilities play into a wide array of applications as varied as video games, medical imaging, home automation, smart TVs, e-commerce, digital signage and unmanned vehicles. The impetus underlying many of these emerging applications is simply to give machines a certain ability to analyze and respond to the real world around them. 2D and 3D vision analysis is an important capability in this regard because it moves computer vision closer to human vision.
Embedded analytics for 2D vision analysis can bring about new interactive and natural user interfaces for computers, appliances, industrial machines and other devices. For example, instead of relying on a mouse to move the cursor on a PC screen, users are able to control their computers with several hand gestures. Of course, adding the third dimension to vision analysis is considerably more complex, but it opens the door to many new applications, some of which have yet to be invented.
3D vision analysis will extend many applications that today deploy 2D vision. For example, today’s 2D hand gesture recognition can morph into a full-body tracking interface. Microsoft’s Kinect is a good example. The fastest ever consumer adoption of an embedded analytic vision technology, Kinect allows players to interact with a computer without accessories. The computer, which in Kinect’s case is the Xbox 360, perceives the players and calculates the body pose from 3D information (Figure 6).
Figure 6. Screenshot from TI’s body-tracking demo (using third-party algorithm on TI’s DaVinciTM DM3730 video processor).
Digital signage is another example of an emerging embedded analytics application. Not just a static digital advertisement, digital signage with embedded analytics is able to read the person reading the sign. Inside a retail store, such a sign will serve up an ad targeted at the demographic group of the reader.
A broad range of TI processors are adept at 2D or 3D processing tasks for a variety of applications. For 2D processing for hand tracking and other low-level applications, TI’s SitaraTM AM335x and AM37x ARM® microprocessors are a good fit. For applications requiring full-body tracking or tracking multiple users, TI’s DaVinci DM3730 and DM8148 video processors, as well as the smart multicore OMAPTM mobile applications processors, offer a variety of performance options and capabilities.
Getting smart with embedded analytics
Embedded analytics is reframing how technology is encountered in everyday life. In the past, a problem would be brought to a computer, where answers would be dispensed, and in the end, a human being would decide on a solution. Now, embedded analytics is moving digital-processing technology to the problem, and the system determining a solution. The technology challenges enabling embedded analytic applications are as diverse and as unique as the problems being solved. Fortunately, the embedded processor innovations from TI are meeting these challenges head on.
The sheer diversity of emerging embedded analytics applications demands a broad range of embedded processors to meet all requirements. TI’s breadth of embedded analytics processors, software and tools; additional hardware and software support from its extensive Design Network; and years of leadership in automotive, security and industrial analytics will continue to help systems “get smart” by enabling embedded analytics for new applications.
Sidebar: A good listener: TIesr
A robust and efficient open source speech recognizer, the TI Embedded Speech Recognizer targets embedded platforms with a simple, easy-to-use application programming interface (API). Capable of adapting to changing noise environments and various microphones, the downloadable TIesr balances memory requirements and processing power with its speech-recognition capabilities and robustness.
Sidebar: Hearing is believing
The ecosystem that has grown around TI’s embedded analytic technologies includes third-party companies that are developing breakthrough audio solutions.
Audio Analytic has developed a range of analytics, each detecting a specific class of sound, used individually or in combinations to address particular applications and security scenarios.
For example, detecting breaking glass or car alarms can add significant value to premises or property protection applications. Aggression and gunshot detection provide increased staff protection in lone worker locations or other public safety and potentially hostile situations such as hospital A&E, prisons or police-custody centers. Also, keyword detection allows monitoring stations to be alerted when members of staff require assistance through use of designated security keywords.
Learn more: www.audioanalytic.com
Sidebar: Easy to image-ine
TI’s Design Network includes several companies that provide hardware and software design and optimization services for imaging applications based on TI’s processors.
eInfochips’ product design services and IP portfolio reduce development time, cost and risk for developers of industrial and video surveillance analytics applications and beyond.
Learn more: eInfochips’ Video Analytics Daughter card developed around TI’s DaVinciTM DM6435 video processor and Video Analytics Services.
D3 Engineering provides a fast, low-risk path through embedded product development. Building on proven DesignCoreTM modules and application software libraries, D3 Engineering speeds design through launch of embedded systems for digital video and analytics, digital power management, and precision motion control.
Learn more: www.D3Engineering.com
