Real-Time Vehicle Detection with MobileNet SSD and Xailient

This blog post was originally published at Xailient’s website. It is reprinted here with the permission of Xailient.

From vehicle counting and smart parking systems to Autonomous Driving Assistant Systems, the demand for detecting cars, buses, and motorbikes is increasing and soon will be as common of an application as face detection.

And of course, they need to run real-time to be usable in most real-world applications, because who will rely on an Autonomous Driving Assistant Systems if it cannot detect cars in front of us while driving.

In this post, I will show you how you can implement your own car detector using pre-trained models that are available for download: MobileNet SSD and Xailient Car Detector.

Before diving deep into the implementation, let’s gets a bit familiar and know about these models. But feel free to skip to the code and results if you wish.

MobileNet SSD

MobileNet is a light-weight deep neural network architecture designed for mobiles and embedded vision applications.

MobileNet architecture [1]

In many real-world applications such as a self-driving car, the recognition tasks need to be carried out in a timely fashion on a computationally limited device. To fulfill this requirement, MobileNet was developed in 2017.

The core layers of MobileNet is built on depth-wise separable filters. The first layer, which is a full convolution, is an exception.

To learn further about MobileNet, please refer to the paper.

Around the same time (2016), SSD: Single Shot detector was also developed by Google Research team to cater the need for models that can run real-time on embedded devices without a significant trade-off in accuracy.

SSD Architecture [2]

Single Shot object detection or SSD takes one single shot to detect multiple objects within the image. The SSD approach is based on a feed-forward convolutional network that produces a fixed-size collection of bounding boxes and scores for the presence of object class instances in those boxes.

It’s composed of two parts:

1. Extract feature maps, and
2. Apply convolution filter to detect objects

SSD is designed to be independent of the base network, and so it can run on top of any base networks such as VGG, YOLO, MobileNet.

In the original paper, Wei Liu and team used VGG-16 network as the base to extract feature maps.

To learn further about SSD, please refer to the paper.

To further tackle the practical limitations of running high resource and power-consuming neural networks on low-end devices in real-time applications, MobileNet was integrated into the SSD framework. So, when MobileNet is used as the base network in the SSD, it became MobileNet SSD.

MobileNet SSD overview [7]

The MobileNet SSD method was first trained on the COCO dataset and was then fine-tuned on PASCAL VOC reaching 72.7% mAP (mean average precision).

MobileSSD for Real-time Car Detection

Step 1: Download pre-trained MobileNetSSD Caffe model and prototxt.

We’ll use a MobileNet pre-trained downloaded from https://github.com/chuanqi305/MobileNet-SSD/ that was trained in Caffe-SSD framework.

Download the pre-trained MobileNet SSD model and prototxt from here.
MobileNetSSD_deploy.caffemodel
MobileNetSSD_deploy.prototxt

Step 2: Implement Code to use MobileNet SSD

import time
import cv2 as cv
import numpy as np
import math

# load our serialized model from disk
print("Load MobileNetSSD model")

prototxt_path = "MobileNetSSD_deploy.prototxt"
model_path = "MobileNetSSD_deploy.caffemodel"

# initialize the list of class labels MobileNet SSD was trained to detect
CLASSES = ["background", "aeroplane", "bicycle", "bird", "boat",
    "bottle", "bus", "car", "cat", "chair", "cow", "diningtable",
    "dog", "horse", "motorbike", "person", "pottedplant", "sheep",
    "sofa", "train", "tvmonitor"]

net = cv.dnn.readNetFromCaffe(prototxt_path, model_path)

def process_frame_MobileNetSSD(next_frame):
    rgb = cv.cvtColor(next_frame, cv.COLOR_BGR2RGB)
    (H, W) = next_frame.shape[:2]

    # convert the frame to a blob and pass the blob through the
    # network and obtain the detections
    blob = cv.dnn.blobFromImage(next_frame, size=(300, 300), ddepth=cv.CV_8U)
    net.setInput(blob, scalefactor=1.0/127.5, mean=[127.5, 127.5, 127.5])
    detections = net.forward()

    # loop over the detections
    for i in np.arange(0, detections.shape[2]):
        # extract the confidence (i.e., probability) associated
        # with the prediction
        confidence = detections[0, 0, i, 2]
        # filter out weak detections by ensuring the `confidence`
        # is greater than the minimum confidence
        if confidence > 0.7:
            # extract the index of the class label from the
            # detections list
            idx = int(detections[0, 0, i, 1])
            # if the class label is not a car, ignore it
            if CLASSES[idx] != "car":
                continue
            # compute the (x, y)-coordinates of the bounding box
            # for the object
            box = detections[0, 0, i, 3:7] * np.array([W, H, W, H])
            (startX, startY, endX, endY) = box.astype("int")
            
            cv.rectangle(next_frame, (startX, startY), (endX, endY), (0, 255, 0), 3)

    return next_frame

def VehicheDetection_UsingMobileNetSSD(filename):
    cap = cv.VideoCapture(filename)

    # Write output file
    frame_width = int(cap.get(cv.CAP_PROP_FRAME_WIDTH))
    frame_height = int(cap.get(cv.CAP_PROP_FRAME_HEIGHT))

    # Define the codec and create VideoWriter object
    fps = 20
    size = (int(frame_width),int(frame_height))
    fourcc = cv.VideoWriter_fourcc('m','p','4','v')
    out = cv.VideoWriter()
    success = out.open('output_mobilenetssd.mov', fourcc, fps, size, True)

    frame_count = 0

    # start timer
    t1 = time.time()

    while True:
        ret, next_frame = cap.read() # Reads the next video frame into memory
        
        if ret == False: break

        frame_count += 1
        next_frame = process_frame_MobileNetSSD(next_frame)
        
        # write frame
        out.write(next_frame)
        
        key = cv.waitKey(50)
        
        if key == 27: # Hit ESC key to stop
            break
    
    
    # end timer
    t2 = time.time()

    # calculate FPS
    fps = str( float(frame_count / float(t2 - t1))) + ' FPS'

    print("/MobileNetSSD Car Detector")
    print("Frames processed: {}".format(frame_count))
    print("Elapsed time: {:.2f}".format(float(t2 - t1)))
    print("FPS: {}".format(fps))

    cap.release()
    cv.destroyAllWindows()
    out.release()

Because we want to use it for real-time application, lets calculate the frames it processes per second as well.

(Parts of this code is inspired from PyImageSearch blog.)

Experiments:

I ran the above code on two different devices:

1. On my dev machine, which is Lenovo Yoga 920 with Ubuntu18.04 operating system.
2. On low-cost, resource-constrained device, which is Raspberry Pi 3B+ with Raspbian Buster operating system.

Results:

MobileNet SSD Results.

On my dev machine, Lenovo Yoga, with MobileNet SSD, I got an inference speed of 23.3 FPS and when I ran RaspberryPi 3B+, the inference speed was 0.9 FPS, using all 4 cores.

Pretty dramatic. This experiment shows that if you have a powerful device to run the MobileNetSSD, it performs well and will serve the real-time requirement.  But if your application is targeted to be deployed on a computationally limited IoT/embedded device such as the Raspberry Pi, this does not seem to be a good fit for a real-time application.

Xailient

Xailient model uses selective attention approach to perform detection. It is inspired by the working mechanism of the human eye.

Xailient models are optimized to run on low power devices that are memory and resource-constrained.

Now let’s see how Xailient Pre-trained Car detector performs.

Xailient Car Detector for Real-time Car Detection

Step-1: Download pre-trained Car Detector model.

We’ll use a Xailient’s pre-trained car detector model downloaded from console.xailient.com.

Step 2: Implement Code to use Xailient Car detector mode

import time
import cv2 as cv
import numpy as np
import math
from xailient import dnn

# initialize Xailient model
print("Initialize Xailient model")
THRESHOLD = 0.6 # Value between 0 and 1 for confidence score
detectum = dnn.Detector()


def process_frame_xailient(next_frame):
    
    _, bboxes = detectum.process_frame(next_frame, THRESHOLD) # Extract bbox coords

    # Loop through list (if empty this will be skipped) and overlay green bboxes
    # Format of bboxes is: xmin, ymin (top left), xmax, ymax (bottom right)
    for i in bboxes:
        cv.rectangle(next_frame, (i[0], i[1]), (i[2], i[3]), (0, 255, 0), 3)

    return next_frame
def VehicheDetection_UsingXailient(filename):
    cap = cv.VideoCapture(filename)

    # Write output file
    frame_width = int(cap.get(cv.CAP_PROP_FRAME_WIDTH))
    frame_height = int(cap.get(cv.CAP_PROP_FRAME_HEIGHT))

    # Define the codec and create VideoWriter object
    fps = 20
    size = (int(frame_width),int(frame_height))
    fourcc = cv.VideoWriter_fourcc('m','p','4','v')
    out = cv.VideoWriter()
    success = out.open('output_xailient.mov', fourcc, fps, size, True)

    frame_count = 0

    # start timer
    t1 = time.time()

    while True:
        ret, next_frame = cap.read() # Reads the next video frame into memory
        
        if ret == False: break

        frame_count += 1
        next_frame = process_frame_xailient(next_frame)

        # write frame
        out.write(next_frame)

        key = cv.waitKey(50)
        
        if key == 27: # Hit ESC key to stop
            break
    
    

    # end timer
    t2 = time.time()

    # calculate FPS
    fps = str( float(frame_count / float(t2 - t1))) + ' FPS'

    print("/nXailient Car Detector")
    print("Frames processed: {}".format(frame_count))
    print("Elapsed time: {:.2f}".format(float(t2 - t1)))
    print("FPS: {}".format(fps))

    cap.release()
    cv.destroyAllWindows()
    out.release()

Experiments:

I ran the above code the same two sets of devices:

1. On my dev machine, which is Lenovo Yoga 920 with Ubuntu18.04 operating system.
2. On a low-cost, resource-constrained device, which is Raspberry Pi 3B+ with Raspbian Buster operating system.

Results:

Xailient Car Detection Results.

On dev machine, there is a slight improvement on inference speed when using Xailient Car Detector even when only 1 core is used. On Raspberry Pi, however, Xailient processes 8x more frames per second with a single core.

Summarizing the results of both models:

MobileNetSSD vs Xailient

The video I used for this experiment was downloaded from Pexels.com

In this post, we looked the need for real-time detection models, briefly introduced MobileNet, SSD, MobileNetSSD and Xailient, all of which were developed to solve the same challenge: to run detection models on low-powered, resource-constrained IoT/embedded devices with a right balance of speed and accuracy. We used pre-trained MobileNetSSD and Xailient car detector models and performed experiments on two separate devices: dev machine and a low-cost IoT device. Results show a slight improvement in speed of Xailient Car detector over MobileNetSSD, in the dev machine and a significant improvement in the low-cost IoT device, even when only 1 core was used.

If you want to extend your car detection application to car tracking and speed estimation, here’s a very good blog by PyImageSearch.

References

1. Howard, Andrew G., et al. “Mobilenets: Efficient convolutional neural networks for mobile vision applications.” arXiv preprint arXiv:1704.04861 (2017).
2. Liu, Wei, et al. “Ssd: Single shot multibox detector.” European conference on computer vision. Springer, Cham, 2016.
3. https://www.pyimagesearch.com/2019/12/02/opencv-vehicle-detection-tracking-and-speed-estimation/
4. https://honingds.com/blog/ssd-single-shot-object-detection-mobilenet-opencv/
5. https://github.com/chuanqi305/MobileNet-SSD
6. https://mc.ai/object-detection-with-ssd-and-mobilenet/
7. https://machinethink.net/blog/mobilenet-v2/#:~:text=SSD%20is%20designed%20to%20be,detection%20portion%20of%20the%20network.

Sabina Pokhrel
Customer Success AI Engineer, Xailient