This blog post was originally published at STMicroelectronics’ website. It is reprinted here with the permission of STMicroelectronics.
While driver-monitoring systems are not new, Leopard Imaging’s cabin camera modules at the Embedded Vision Summit in May represented a milestone because they inaugurated a path to democratization. The ST Authorized Partner already makes several in-cabin camera modules with the VG5761. However, by adopting ST’s VB56G4A, Leopard Imaging took advantage of the sensor’s backside-illuminated global shutter to work with a broader range of lighting conditions, thus improving the accuracy of machine learning algorithms while reducing overall power consumption. By giving more choices to car makers that can now use modules with or without backside illumination, Leopard Imaging ensures that driver-monitoring systems (DMS) and occupant-monitoring systems (OMS) become mainstream.
A 20-year-old concept
The modern conception of a driver-monitoring system is at least 20 years old. In a paper presented in 2004 at the IEEE International Conference on Systems, Man, and Cybernetics, researchers at the University of Paderborn in Germany and Queensland University of Technology in Australia1 used sensors to track eyes and blinking to determine if a driver was tired or distracted. The research used more commonly-found devices than previous attempts, which made the paper unique because it opened the door to mainstream applications. Yet, 20 years later, DMS and OMS remain rare exceptions in consumer vehicles. Let us, therefore, understand what was holding the technology back and what makes Leopard Imaging’s presentation at the Embedded Vision Summit unique.
Lesson 1: Using new devices like the VG56G4A to fit more environments
Building a unique expertise
While Leopard Imaging has almost 15 years of expertise in this field, the member of the ST Partner Program also explained that the availability of the VG56G4A created new opportunities that they were able to leverage thanks to their existing expertise. Indeed, the company already offers modules without illumination, such as those using the VG5761 from ST. Over the years, they’ve helped engineers use those modules behind the steering wheel and install lighting somewhere else. The challenge is that the setup only works in certain cabins. Hence, one lesson Leopard Imaging shared is that finding ways to tailor solutions to their environment is crucial. In a nutshell, innovation is about making technology relevant in more spaces.
Leveraging new technologies
Thanks to its 940-nm near-infrared laser, the VB56G4A provides illumination without a distracting red glow. The wavelength is also immune to sunlight, meaning the technology works in any light condition, which wasn’t the case previously. Moreover, the use of VCSEL illumination comes with the ability to use narrow 940-nm filters that provide a better immunity against infrared from sunlight.
In other considerations, the presence of a global shutter, instead of a rolling shutter, means that it’s possible to detect fast lateral movements and provide more precise edge detection thanks to the elimination of image artifacts. Additionally, the presence of backside illumination means that it’s possible to install modules in previously unreachable cabin areas. Consequently, Leopard Imaging showcased the LI-ST-VB56G4A-DMS-GMSL2 and LI-ST-VB56G4A-MONO-DMS-ADJ at the Embedded Vision Summit.
The LI-ST-VB56G4A-MONO-DMS-ADJ from Leopard Imaging
The LI-ST-VB56G4A-DMS-GMSL2
Moving from niche to mainstream
As Leopard Imaging itself demonstrates, DMS and OMS were possible before. The ST Authorized Partner has been working on the technology and has had solutions for years. However, the two new modules are highly symbolic because they enable greater customization, performance, and efficiency at a lower price, thanks partly to a much more straightforward implementation. The company thus built on its expertise to create solutions that fit many more use cases. As a result, the new modules based on the VG56G4A are a milestone not because they make DMS and OMS possible but because they make them mainstream.
Lesson 2: Using Leopard Imaging to adopt new frameworks and comply with new regulations
From heuristic to machine learning
Another milestone contributing to the democratization of DMS and OMS is the increased accessibility to machine learning at the edge. Leopard Imaging shared how they work with their customers to help them gather training data with their modules. Interestingly, the ST Authorized Partner is noticing a move from a heuristic-based approach to DMS and OMS to machine learning because it ensures developers can account for more use cases without using more resources. It also means cutting down on development costs. Instead of accruing decades of expertise to refine the formula that will govern the application, machine learning algorithms move the decision-making process to a neural network that can more easily be improved over time.
New industry requirements
Leopard Imaging also explained that DMS and OMS are crossing a critical threshold in mindshare thanks to new legislative initiatives. In 2021, American senators proposed new laws that would study the benefits of DMS and mandate its installation in new vehicles. The European Parliament has been even more proactive and already passed regulations in 2019 mandating DMS by mid-2024, and a European safety group is considering requiring DMS to issue five-star ratings. Consequently, Leopard Imaging has been working with solution makers to ensure they understand the new regulations and meet the new requirements. As the ST Authorized Partner demonstrated, DMS and OMS are being democratized because the company is solving upcoming industry challenges.
- T. Brandt, R. Stemmer and A. Rakotonirainy, “Affordable visual driver monitoring system for fatigue and monotony,” 2004 IEEE International Conference on Systems, Man and Cybernetics (IEEE Cat. No.04CH37583), The Hague, Netherlands, 2004, pp. 6451-6456 vol.7, doi: 10.1109/ICSMC.2004.1401415.
