By Peter Voss and Benno Kusstatscher, Automotive Segment Team
Analog Devices
This article was originally published by Elektronik Magazine. It is reprinted here with the permission of the original publisher.
Abstract
Car manufacturers (OEM‘s) and system suppliers (Tier 1s) do agree: Advanced Driver Assistance Systems (ADAS) will see a steep growth in the years to come. One driving factor for sure is the increased safety awareness and the desire for more driving comfort on the customer side. But first and foremost Euro NCAP’s tightened safety requirements will boost installed ADAS equipment from single digit take rates to almost 100% over the next years. Hence it’s to no surprise that the call for commercially viable solution is on. This is to satisfy deployments as high volume standard equipment vs. the moderate customer option business as seen today. Analog Devices recently announced a new ADAS Processor Family, the Blackfin® ADSP-BF60x series which was specifically developed for this now emerging mass market requirement. The first two processors of this new family are available now to service this new requirement for camera based solutions.
Introduction
Depending on the ADAS tasks to solve, different types of sensors are being deployed in systems today. For the environmental near and far-field monitoring around the vehicle, Radar-, Ultrasonic-, LIDAR-, PMD-, Camera- and Night camera sensors are used either as single or multi sensor systems. The latter do combine the sensor data to achieve more accurate results as desired. Camera systems will further see an additional boost looking inside the car for analyzing the driver’s state. Driver situation and prediction analysis will be used to better filter the various warning messages an ADAS system can generate in the future. The goal is not to overflow the driver with irrelevant warnings. If the system would detect that a sporty driver is concentrating on the traffic in front of him, a premature warning of a car he is following might be very annoying. However, knowing that the driver is distracted while playing with his mobile phone or simply is about to fall asleep, an early enough warning or even braking action in the same situation might be very appropriate.
ADAS systems providing the extended view outside of the car while detecting and classifying objects plus systems determining the driver state are all aiming at increasing road safety and will soon become part of the standard equipment in new cars. Increased driving comfort and economical aspects like controlling the timely gear shifts before an incline will additionally drive customer acceptance and will enable car manufacturers to market optional functionality in addition to the standard ADAS equipment.
ADAS Deployments Fueled by Euro NCAP
Euro NCAP tests and publically reports the safety of vehicles marketed in Europe. Transparent test methods are used while four categories are considered: Adult Occupant protection, Child occupant protection, Pedestrian protection and safety assist (e.g. seatbelt reminder). As a result Euro NCAP awards up to five safety stars which then will be published on their web site. Each safety level (one to five stars) requires a minimum number of points in each of the four categories and in addition a minimum number of total points. It is expected that by 2017 cars will not be able to achieve the desired five star rating without ADAS present and that car manufacturers will therefore include at least one ADAS system as standard equipment by that time. While looking at the Euro NCAP rating scheme and comparing the changes over the years, one get a feel for the current priorities. Between 2011 and 2012, requirements to achieve the minimum points for pedestrian protection were upped by 50%. This boots the importance for ADAS cameras due to their ability to not only detect and classify objects as pedestrians but they also have the capability to do the same with partially obscured pedestrians as required by the rating schemes.
Cameras Make the Standard Equipment
Camera based ADAS is not new, the technology had some time to mature over the past years but was mostly seen as customer option on premium brand cars. The accumulated knowhow is now very useful while going after the emerging Euro NCAP requirements. However, a rethinking has to happen as the standard equipment business brings the commercial aspect right into the focus. While in the past, complex and high performance systems were developed the task at hand is now a different one. ADAS systems must now support the “just required” functionality at the appropriate commercial level. System providers and component suppliers like Analog Devices are now asked to walk a fine line between enabling commercially attractive solutions while maintaining the freedom for OEM’s to differentiate.
Due to the early engagement in driver assistance systems and the continued investment into this emerging technology, Analog Devices started in time with the development of dedicated ADAS solutions and is now sampling the first two processors focusing on camera based systems. The specification for these ADAS Camera centric processors targeted from the start the total cost of ownership (ToC) without taking the flexibility out of the hands of the system suppliers and OEM’s. Next to the required programmability and enough processing power, lowest power consumption in class was achieved in order to keep the thermal design manageable. Support for functional safety following ISO26262 and the availability of an application oriented development environment with optimized vision processing libraries helps to design a total system with “time to market” and low risk in mind.
Optimum Performance on System Level
The ADSP-BF60x family reduces the overall cost of a five function system of up to 30%. The ADSP-BF609 (processing up to Mega Pixel format) and the ADSP-BF608 (processing up to VGA format) support up to five concurrent Vision functions with up to 30 processed frames per second. With the <1.3W at ambient temperature of 105°C the ADSP-BF60x offer the lowest power consumption figures in their class.
To enable this, Analog Devices deployed a straight forward but unique concept. It is based on two Blackfin cores as they are used in production ADAS systems today. However, algorithms that can’t be modeled by software in a highly efficient and economical way have been implemented as hardware engines instead, resulting in a highly configurable tool box of vision processing units. Analog Devices calls this the Pipelined Vision Processor (PVP1) which is now part of the new ADSP-BF60x processors. Although, all implemented on low-power process technology, further innovation was needed to also tackle those consumers dominating power dissipation of modern designs more than logic circuits do: external memory (DDR) interfaces. Lowest power dissipation is achieved by distribution of processing and by smart utilization of moderate memory bandwidth. In addition, the Blackfin architecture has been enriched by a number of hardware blocks to address functional safety requirements.
Efficient Data Streaming
Many chip architectures receive data from the video sensor and save them frame by frame into the external memories, such as DDR, just to read them back frame by frame (slightly delayed). Multi-core architectures even tend to multiply that data movement for the only benefit of letting the cores identify the region of interests (ROIs) in every frame. The ADSP-BF60x concept avoids such power consuming transport of video data. Still, the full frame is stored in DDR2 memory, but there is no need to read the entire frame ever back. As soon as received by the device the incoming data is multi-casted to the Pipelined Vision Processor (PVP1) which preprocesses the data directly on its way in.
Figure 1: Vision Processing with ADSP-BF60x Processors
As shown in Figure 1 the so-called PVP1 Camera Pipes can generate up to three intermediate preprocessing results like edge map, integrals or re-quantification by non-linear thresholds. On top, it can provide status information like histograms. Not a single instruction has to be executed on the Blackfin cores and not a single byte has to be transferred over the DDR bus in order to achieve these results. Additionally, the PVP1 uses dense data formats when outputting intermediate results and hence, most of times the results fit into on-chip L1 and L2 memories.
From here software running on the Blackfin cores can take it further. The results provided by the PVP1 Camera Pipes enable software to identify ROIs efficiently. Now, only the ROIs need to be read back from external memory for further analysis. They can be either read by DMA, by the cores or by the so-called PVP1 Memory Pipe. The latter can further analyze or massage the ROI data, say it filters it or scales it to the preferred template size. Again, the histogram results come for free. If the dense output of the Memory Pipe is stored into on-chip memories, the Blackfin cores can now locally classify the ROIs and, if needed, verify them against raw data or against respective region in previous frames as stored in DDR2 memory.
Pipelined Vision Processor (PVP1) – A Closer Look
The PVP1 can process up to four data streams, three Camera Pipes and one Memory Pipe, not yet counting histogram status outputs. As shown in Figure 2 the PVP1 tool box consists of twelve processing blocks, which are optimized for various vision processing steps.
Figure2: Pipelined Vision Processor (PVP1)
Typically, the Memory Pipe utilizes one or multiple processing blocks on demand to analyze a series of ROIs and may change operators on the fly. Camera Pipes tend to process the full frame in parallel manner. Incoming data can be multi-casted to multiple processing branches. The user has full flexibility in interconnecting operators to form the pipes. One possible configuration is shown in Figure 3:
Figure 3: Example Configuration for Object and Pedestrian Detection
The traditional Canny-structure can be identified in the figure. It consists of a low-pass filter with Gaussian coefficients, enhanced Sobel filters with a 5×5 matrix, Cartesian-to-Polar conversion and a Non-Maxima Suppression (NMS) stage. Remaining maxima might be passed through a Threshold block to reduce bit resolution or to run-length compress the output for least memory loading. If post-processed by software, say with a Hough-Transform algorithm, line detection and lane keep assist strategies can be derived from this edge map.
If the gradients are forwarded to an Integral block (Histogram of Gradients, HoG) half of the required processing for Pedestrian Detection has already been accomplished. The square of the low-passed pixels may feed the other Integral block, which then can operate in regular (SAT) or diagonal mode (RSAT). It can reduce the frame resolution at the output if high-resolution is not needed, like in the case of shadow detection.
The shown example configuration demonstrates how lane keep assist as well as pedestrian and vehicle detection can be implemented on a single chip in efficient manner, while the MIPS load of the Blackfin cores is still kept at a moderate level. Unused MIPS and the not-yet-used Memory Pipe functionality can be utilized to integrate head lamp control and/or traffic sign detection on top. All of this is enables by an ADSP-BF60x device at less than 1.3W at 105°C ambient temperature and can therefore be easily installed behind the central mirror in the car next to the image sensor.
Affordable Driver Assistance
At Analog Devices, ADAS solutions for Vision and Radar Systems are a focus. With the new ADAS Processor models ADSP-BF609 and ADSP-BF608, camera based driver assistance is becoming commercial attractive for mass deployments. ADAS Camera systems no longer need to be a privilege for premium models. Euro NCAP’s tighten safety requirements will indirectly fuels the acceptance of ADAS Vision systems. Similar trends are observed worldwide. The ADAS Vision processors from Analog Devices do support Day- and Night vision systems analyzing the surroundings of the car as well as driver monitoring systems to detect the state of the driver.
