[ad_1]

*Real-world implementation of ADAS-CAS with LIDAR-Camera Sensor Fusion, using RPi, RPLIDAR A1, Pi Cam V2, LED SHIM, NCS 2, Speaker, and battery.*

India accounts for only 1% of total vehicles in the world. However, World Bank’s survey reports **11% of** **global** **road death happens in India,** exposing the dire need** to enhance road safety**. Most of the developing countries like India pose a set of challenges, unique on their own. These include **chaotic traffic, outdated vehicles, lack of pedestrian lanes and zebra crossings,** animals crossing the road, and the like. Needless to say, most vehicles don’t have advanced driver-assist features nor can they afford to upgrade the car for better safety.

Against this backdrop, **this solution aims to augment even the least expensive cars in India with an** **ultra-cheap ADAS Level 0, i.e. collision avoidance and smart surround-view.** Modern cars with a **forward-collision warning (FCW) system or autonomous emergency braking (AEB)** are very expensive, but **we can augment such functions on old cars, at a low cost.**

**The complete source code of this solution is available ****here**

The idea is to use a battery-powered Pi connected to a LIDAR, Pi Cam, LED SHIM, and NCS 2, mounted on the car bonnet to perceive frontal objects with their depth & direction. This **not only enables a** **forward-collision warning system but also** **smart driver assistance that gives alerts about traffic signs** **or pedestrians, walking along the roadside**,** or crossing the road**.

LIDAR uses laser beams to **compute distance based on reflection time.**

Cameras generally have higher resolution than LiDAR but cameras have a limited FOV and can’t estimate distance. While rotating **LIDAR** has a **360° **field of view, **Pi Cam** has only **62×48 degrees** **Horizontal x Vertical FoV. **As we deal with multiple sensors here, we need to employ **visual fusion techniques** to integrate the sensor output, that is to get the distance and angle of an obstacle in front of the vehicle. Let’s first discuss the theoretical foundation of sensor fusion before hands-on implementation.

Each sensor has its own advantages and disadvantages. Take, for instance, RADARs are low in resolution, but are good at measurement without a line of sight. In an autonomous car, often a **combination of LiDARs, RADARs, and Cameras **are used to perceive the environment. This way we can compensate for the disadvantages, by combining the advantages of all sensors.

**Camera:**excellent to understand a scene or to**classify objects****LIDAR**: excellent to**estimate distances**using**pulsed laser waves****RADAR:**can measure**the speed**of obstacles using**Doppler Effect**

The camera is a 2D Sensor from which features like bounding boxes, traffic lights, lane divisions can be identified. LIDAR is a 3D Sensor that outputs a set of point clouds. The **fusion technique finds a correspondence between points detected by LIDAR and** points detected by the **camera**. To use LiDARs and Cameras in unison to build ADAS, **the 3D sensor output needs to be fused with 2D sensor output** by doing the following steps.

1.** Project the LiDAR point clouds **(3D) onto the 2D image.

2. **Do object detection **using an algorithm like YOLOv4.

3. **Match the ROI **to find the interested LiDAR projected points.

By doing the 3 steps above, **the surrounding** **objects would** **be** **measured and classified using LIDAR-Camera fusion.**

When a raw image from a cam is merged with raw data from RADAR or LIDAR then it’s called** Low-Level Fusion or Early Fusion**. In Late Fusion, detection is done before the fusion. Note that there are many challenges to project the 3D LIDAR point cloud on a 2D image. The **relative orientation and translation between the two sensors must be considered** in performing fusion.

**Rotation:**The**coordinate system of LIDAR and Camera can be different**. Distance on the LIDAR may be on the z-axis, while it is x-axis on the camera.**We need to apply rotation**on the LIDAR point cloud to make the coordinate system the same, i.e.**multiply each LIDAR point with the****Rotation matrix.**

**Translation:**In an autonomous car, the LIDAR can be at the center top and the camera on the sides. The**position of LIDAR and camera in each installation can be different.**Based on the relative sensor position,**translate the LIDAR Points by multiplying with a Translation matrix.****Stereo Rectification:**For stereo camera setup, we**need to****do Stereo Rectification****to make the left and right images co-planar.**Thus, we need to multiply with matrix R0 to align everything along the**horizontal Epipolar line**.**Intrinsic calibration:**Calibration is the step where you tell your camera how to convert a point in the 3D world into a pixel. To account for this, we need to multiply with an intrinsic calibration matrix containing factory calibrated values.

To sum it up, **we need to multiply LIDAR points with all the 4 matrices **to project on the camera image.

**To project a point X in 3D onto a point Y in 2D,**

*P = Camera Intrinsic Calibration matrix**R0 = Stereo Rectification matrix**R|t = Rotation & Translation to go from LIDAR to Camera**X = Point in 3D space**Y = Point in 2D Image*

Note that we have combined both the rigid body transformations, rotation, and translation, in one matrix, R|t. Putting it together, the 3 matrices, P, R0, and R|t account for** extrinsic and intrinsic calibration** to project LIDAR points onto the camera image. However, the **matrix values highly depend on our custom sensor installation.**

This is just one piece of the puzzle. Our aim is to augment any cheap car with an **end-to-end collision avoidance system** **and smart surround view. **This would include our choice of sensors, sensor positions, data capture, custom visual fusion, and object detection, coupled with a data analysis node, to do synchronization across sensors in order to trigger driver-assist warnings to avoid danger.

First, we need to assemble the Pi with RPLIDAR A1, Pi Cam, LED SHIM, and NCS 2. **2D LIDAR is used instead of 3D LIDAR **as we aim to make the gadget, **cheapest** possible. The unit is powered by a 5V 3A 10K mAH battery. For ease of assembly, **a LIDAR mount is 3D printed **and attached to the RPi. Part of the mount design is taken from the STL files obtained from here

Connect RPLIDAR A1 with the USB adapter that is connected to the Pi USB using a micro-USB cable. LIDAR’s adapter provides power and converts LIDAR’s internal UART serial interface to a USB interface. Use an Aux-to-Aux cable to connect RPi to speakers. Due to physical constraints, an LED SHIM is used instead of Blinkt to signal warning messages. While **the total cost of the ADAS gadget is just around US$ 150–200,** one may have to shell out at least **$10–20K more,** **to get a car model with such advanced features.**

Let’s imagine, a 3D LIDAR is connected the same way as above. First, we will try to solve **3D LIDAR-Camera Sensor Fusion** on the above gadget. Then we will see the **variation for 2D LIDAR-Camera Fusion**, so as to make it work on RPLIDAR A1.

It is clear from the above discussion that we need to do** rotation, translation, stereo rectification**,** and intrinsic calibration **to project LIDAR points on the image. We will try to apply the above formula based on the custom gadget that we built.

From the above image, you can estimate the Pi Cam is 10 mm below the LIDAR scan plane. i.e. a **translation of [0, -10, 0]** along the 3D-axis. Consider Velodyne HDL-64E as our 3D LIDAR, which** requires 180° rotation** to align the coordinate system with Pi Cam. **We can compute the** **R|t matrix **now.

As we use a **monocular camera** here, the stereo rectification matrix will be an **identity matrix. **We can make the intrinsic calibration matrix based on the **hardware spec of Pi Cam V2.**

**For **the **Raspberry Pi V2 camera,**

- Focal Length (FL) = 3.04 mm
- FL Pixels = focal length * sx, where sx = real world to pixels ratio
- Focal Length * sx = 3.04mm * (1/ 0.00112 mm per px) = 2714.3 px

Due to a **mismatch in shape, the matrices cannot be multiplied**. To make it work, we **need to transition from Euclidean to Homogeneous coordinates** by adding 0’s and 1’s as the last row or column. After doing the multiplication we need to **convert back to Homogeneous **coordinates.

You can see the **3D LIDAR-CAM sensor fusion projection output** after applying the projection formula on the 3D point cloud. The **input sensor data** from 360° Velodyne HDL-64E and camera is downloaded [9] and fed in.

[ad_2]

Source link