What is an RGB-IR Camera, and How Does It Work?

This blog post was originally published at e-con Systems’ website. It is reprinted here with the permission of e-con Systems.

RGB-IR cameras leverage a Color Array Filter (CFA) with dedicated pixels for visible and IR light – eliminating the need for a mechanical switch and preventing color corruption. Get a closer look into the working principle of RGB-IR cameras and their key benefits.

Most of the modern color cameras come with a CFA (Color Filter Array) with Bayer BGGR pattern. But the sensitivity of the RGB channel is such that it can capture both the visible and IR wavelength. This results in the reduced quality of the final RGB image output owing to color corruption. Also, this makes it difficult to measure the intensity of IR light in the captured image.

The traditional approach to overcome this was to use an IR cut filter for daytime imaging that would prevent any IR light from falling on the sensor. During the night, the IR cut filter is removed mechanically to allow IR light to come in to assist with low light imaging. However, this mechanical component leads to wear and tear over time and adversely affects the lifetime of a camera system.

An RGB-IR camera helps to overcome these challenges faced in an embedded camera system. An RGB-IR camera uses a new type of CFA with dedicated pixels for both visible and IR light. This way, images in both the visible and IR spectrum can be captured without having to use a mechanical switch, at the same time preventing any form of color corruption. In this article, we look at the working principle of an RGB-IR camera, its benefits, and some key embedded vision applications where it is recommended over normal cameras.

How does an RGB-IR camera work?

As mentioned before, the standard Bayer CFA comes with a BGGR pattern. The below image represents the pixels in the Bayer format.

Figure 1: Bayer pattern

But an RGB-IR camera comes with an additional set of pixels that are dedicated to allow only light in the IR spectrum to pass through them. Presence of these pixels facilitates multi-band imaging. Many sensor manufacturers have developed this new CFA that comes with a combination or R,G,B, and IR pixels as shown in the image below:

Figure 2: RGB-IR filter pattern

Image Signal Processor (ISP)

On the ISP end, you have to implement an algorithm to separate the image data that contains both RGB and IR into two different frames. You have to interpolate the processed RGB output and IR frame simultaneously. It should also have an algorithm to subtract the IR contamination on the RGB channels to output the right colors. In addition, the ISP should have the capability to output only the processed RGB or IR frames as per the host system requirements.

Benefits of using an RGB-IR camera

Given below are some of the benefits of an RGB-IR camera:

Easily adapts to changing day and night conditions. This is very useful in applications that require round the clock image and video capture.
Avoids the mechanical filter to switch between visible and IR imaging – increasing application lifetime and robustness
Provides a dedicated IR channel to separate the visible and IR image data cleanly. This improves the quality of the RGB output by helping to measure the amount of IR light in the RGB component of the image accurately and do proper color correction.

How to leverage visible and IR imaging CFA

Having an RGB-IR filter alone is not enough to effectively use the imaging technique. It is important to choose the right camera components such as the sensor, lens, and ISP (Image Signal Processor) that support RGB-IR imaging.

Let us look at each of them in detail.

Sensor

You have to choose a sensor with IR sensitivity pixels on the CFA. Both onsemi and OmniVision offer various sensors that come equipped with RGB-IR imaging capabilities.

Have a look at the below image to understand the pixel pattern of a sensor that supports RGB-IR imaging.

Figure 3: Pixel of an RGB-IR sensor

Optics

Generally, most lenses for color cameras come with an IR cut filter to block wavelength above 650nm. To facilitate RGB-IR imaging, it is pertinent to use a dual-band pass filter – allowing 400-650nm as well as 800-950nm for the IR wavelength.

Given below is the transmission diagram of a dual band pass filter:

Figure 4: Dual band pass filter

Popular embedded vision use cases of RGB-IR cameras

Biometric (facial and iris) recognition

A vast majority of biometric and access control systems require facial and iris recognition for the purpose of identify verification. This demands the ability to do visible and IR imaging smoothly irrespective of the lighting conditions of the environment. These applications also need to produce images without any color corruption for accurate recognition and analysis. Hence, an RGB-IR camera gains an edge over traditional color cameras if color accuracy and lifetime of the camera are two of the highest priorities for a biometric and access control system.

Advanced 24X7 security

With RGB-IR cameras, security and smart surveillance applications can overcome the challenge of color inaccuracies – which hinder the detection of objects or people. No matter the lighting condition – day or night – these cameras use an RGB-IR sensor and a dual band pass filter to capture high-quality images that help extract accurate information for analysis.

Automated Number Plate Recognition (ANPR)

ANPR applications need to detect letters, symbols, and colors of registration plates in varying lighting conditions. Using an RGB-IR camera, ANPR systems can reliably capture visible and IR images with an increased lifetime and accuracy.

To learn more about how to choose the right sensor for your ANPR system, have a look at the article How to choose the right image sensor for Automatic Number Plate Recognition (ANPR).

Also, if you are planning to develop an ALPR system with Raspberry Pi 4, the article Automatic number plate recognition with e-con Systems’ global shutter camera and Raspberry Pi 4 will be a good read.

e-con Systems and RGB-IR cameras: What’s happening now?

e-con Systems is currently developing a camera module based on an RGB-IR sensor to address the problems faced by customers in application that require both visible and infrared (IR) imaging. The camera comes with the right combination of sensor, optics, and ISP for seamless RGB-IR imaging.

At e-con Systems, we also know that building lighting-agnostic applications for biometric recognition or ALPR can be a complex process. Our deep experience in customization ensures that you can reduce design lifecycles that typically take a long time.

Keep an eye on our website and social media profiles to get the latest updates on what’s happening on the RGB-IR camera development front at e-con Systems.

We hope this blog helped you understand the intricacies of why RGB-IR cameras are a blessing when it comes to certain embedded vision applications.

RGB-IR cameras from e-con Systems

Prabu Kumar
Chief Technology Officer and Head of Camera Products, e-con Systems

If you're building AI or vision-enabled products, you've come to the right place.