3D Object Detection from Fisheye Images without Training Images.

Real-world problem solution for Autonomous Driving

Brief

Implemented a solution directly from a published paper to predict 3D objects from fisheye cameras without training images, using our vanilla model trained on rectilinear images. Tasks included:

Python implementation;

Porting to C++;

Integration in the main AI stack.

The implementation has reduced the R&D time significantly as I did not train on any extra images. I utilized the existing 3D object detector.

Methodology

Problem: Perspective images are shift-invariant, which means that an object moving along the X and Y axes with constant Z will result in the same appearance model and size. This is not true for fisheye images where an object with the same depth will have a different appearance model and size when translating along the X and Y axes.

Solution: The solution is to transform fisheye images into cylindrical projections which are shift-invariant. Then utilizing a pre-trained 3D object detector model and passing the transformed images to predict 3D objects. Another post-process transformation is applied to get the final 3D position in the real world.

Implementation steps:

Transform fisheye images into cylindrical projections

Inference on cylindrical Images using a pre-trained 3d Object detector

Convert the detected 3D bounding box from 3D cylindrical space to real 3D space

Original Paper

Citation

Plaut, E., Ben Yaacov, E., & El Shlomo, B. (2021). 3D object detection from a single fisheye image without a single fisheye training image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 3659-3667).