This blog post was originally published at Qualcomm’s website. It is reprinted here with the permission of Qualcomm.
Mobile robotics developers equip their machines to behave autonomously in the real world by generating facility maps, localizing within them and understanding the geometry of their surroundings. Machines like autonomous mobile robots (AMR), automated guided vehicles (AGV) and service robots must perceive and learn on the fly, especially in fast-changing industrial environments. They need the intelligence to navigate around obstacles, humans and other machines, in variable light conditions both indoors and outdoors.
But few robots have that level of intelligent autonomy. The founders of RGo Robotics believed that providing better visual perception can generate a new wave of intelligent solutions and stimulate innovation in robotics. They saw a market opportunity in developing that visual perception technology for mobile robots. Their Perception Engine allows mobile machines to see and understand the world around them.
And now, RGo has implemented their Perception Engine on Qualcomm Robotics SoCs. The combined solution enables manufacturers to quickly develop smarter robots with reliable localization that leads to a better understanding of physical surroundings.
This post looks under the hood at RGo’s Perception Engine and at the work of integrating the software with the development kit.
Perception Engine – Intelligent autonomy based on vision and AI
Perception Engine includes two main software components:
- Vision-based localization – RGo has built Perception Engine around sensor fusion, AI and geometrical computer vision to attain the level of localization robustness needed for fully autonomous robots. Among its layers of intelligence are algorithms like visual-inertial-semantic simultaneous localization and mapping (VIS-SLAM), a learning vision system, an advanced sensor fusion engine and cognitive algorithms. Most appealing to developers is that the engine can run as part of a single-camera reference design, and at a level of computing power appropriate for even low-cost robots.
- Obstacle detection – Tuned to the specific needs of the mobile robot, the engine uses cameras to detect any obstacle at sub-centimeter accuracy. That includes objects as small as cable and overhead objects such as shelves. It detects the ground and can tell the robot how and where to drive, consistent with the machine’s particular ability to maneuver.
Perception Engine is suited to the toughest cases for robotics, including the problems of reflective or dark surfaces, cliffs, glass walls, forklift forks and very low obstacles.
Implementing Perception Engine on Qualcomm Robotics RB5 development kit
On the landscape of robotic platforms, the Qualcomm Robotics RB5 development kit stood out to RGo as an ideal target. The kit provides ample compute power for Perception Engine, it is affordable and it is ready to deploy at scale, including for advanced AI applications. Its small size and low power consumption compare favorably with Intel CPUs or competing platforms, and are well suited to edge applications with intense computer vision functions.
RGo’s extensive experience with mobile embedded products from Qualcomm Technologies convinced them that the implementation effort would require considerable effort but ultimately be successful.
Vision-based localization – Modifying the mezzanine board
Robust localization requires a dedicated wide-angle camera and an inertial measuring unit (IMU). As an additional depth camera for obstacle detection, RGo chose the Intel RealSense, which is natively supported by the Qualcomm Robotics RB5. To connect the additional wide-angle camera and IMU, RGo engineers decided to modify the standard Qualcomm Robotics RB5 navigation mezzanine board and add the required connectors, as shown below.
But integrating the IMU posed an architectural dilemma in that sensors on the Qualcomm Robotics RB5 are usually connected through the Sensor Hub. RGo engineers preferred to control the sensor from the CPU on the Qualcomm Robotics RB5 using the Linux driver provided by the IMU manufacturer. (The rework in the photo at left is for passing the IMU interrupts to CPU GPIOs instead of passing them to the Sensor Hub GPIOs.)
Most of RGo’s implementation work consisted of designing and bringing up the board. That included developing and testing the drivers for the camera, EEPROM and IMU, and working with the board support package (BSP) for the Robotics RB5.
Software – Yocto, imagers and CamX
For both vision-based localization and obstacle detection, most of the workload is on the CPU, with some GPU optimization. The software does not run in real time per se; it runs in Linux. However, it requires control of the timing to ensure that pose estimates are provided with minimal latency.
RGo engineers focused on software development in several areas:
- Building the Qualcomm Robotics RB5 environment – There are two options for building: the default Linux Embedded (Yocto) and Linux Ubuntu (Frankenbuild) operating systems. Both are generated by a poky (containing the OpenEmbedded system) build. The support team at Qualcomm Technologies pointed RGo to the bitbake scripts, which they used to build the Yocto environment.
- Connecting the camera – RGo engineers found this process surprisingly easy, taking only a few days to get the camera up and running. The build supports multiple imagers (cameras) from different manufacturers and at different image settings and resolutions. That facilitates camera bring-up, and RGo was even able to activate two cameras simultaneously. Qualcomm Technologies provides software for this in its BSP.
- Creating a camera driver – To create a camera driver binary for accessing camera sensor frames, RGo used the CamX framework, which contains a well-designed CHI-CDK abstraction layer with the necessary files and scripts. They built their camera driver around a series of shared libraries, kernel devicetree files and XML files, which they easily accessed to configure the driver. That spared them the far more laborious task of opening up and modifying Linux kernel drivers, a common practice with other hardware platforms. CamX also allowed them to access the camera EEPROM, and they access the camera itself through Video4Linux (V4L).
Results – High performance, ease of maintenance and low power consumption
Perception Engine runs on the Qualcomm Robotics RB5 even better than RGo engineers had initially expected – on par with competing platfroms while consuming a fraction of the power. The system achieves centimeter-level position accuracy at velocities far beyond the maximal AMR speed, processing thousands of frames per minute and providing reliable pose estimates at up to 100Hz. The simple passive thermal design of the platform reduces cost and size and simplifies integration with an industrial robot. The Qualcomm Robotics RB5 comes with enough RAM for any reasonable use case and its built-in connectivity (4G/5G, Bluetooth, 802.11a/b/g/n/ax/ac Wave 2) saved the engineers a lot of time and effort.
The engineers also noted these advantages for development:
- The mezzanine-card model makes it easy to add hardware to the platform, as RGo decided to do.
- Being able to work in two different OSes was helpful. RGo used Ubuntu for developing on the Qualcomm Robotics RB5 and for running demos of Perception Engine on the platform. For their commercial product they can also use the smaller, embedded Yocto OS.
- The engineers overcame their strong, initial misgivings about installing a new camera to a Qualcomm Technologies product. Their previous experience showed that integration at that level is nearly impossible without support from Qualcomm Technologies’ experts. Hence their surprise to discover that CamX uses XML files for automatic driver creation instead of requiring a driver rewrite, and their relief at its ease of use. For imagers and configurations that are already supported and not exceptionally unique, CamX greatly simplifies the process.
After designing the modifications to the Qualcomm Robotics RB5 navigation mezzanine board, RGo followed our recommendation and had the original manufacturer of the Qualcomm Robotics RB5 build it. That became the longest-lead task of the implementation. In retrospect, RGo believes they would have saved time and lowered project costs had they designed and produced their own small adapter card instead of modifying the mezzanine card.
Next steps
“The Qualcomm Robotics RB5 can cover a much wider range of cost and performance variants compared to other products,” says Amir Bousani, co-founder and CEO of RGo Robotics. “With Qualcomm Technologies’ wide variety of processors, the Qualcomm RB5 Robotics development kit is a good starting point, allowing developers to switch later to a different processor for cost-sensitive applications. It’s based on the Qualcomm QRB5165 chipset, with a powerful CPU, GPU, neural processing unit (NPU), image signal processor (ISP) and digital signal processor (DSP) for real-time computer vision applications. It has all the interfaces a robotic platform needs for camera, motor control, connectivity and on-board AI.
“We believe that RGo’s Perception Engine running on Qualcomm Technologies’ cost- and power-efficient SOCs is a big step toward the next wave of intelligent automation. There is great market potential for AI-powered mobile machines that see and understand their surroundings and that can scale to a wide variety of applications.”
Learn more about RGo Robotics and its approach to intelligent autonomy at rgorobotics.ai.
Want to discuss the integration and join fellow IoT developers? Join our Developer Discord community.
Vivek Hegde
Product Manager, Qualcomm Technologies, Inc.
