Implemented a solution directly from a published paper to predict 3D objects from fisheye cameras without training images, using our vanilla model trained on rectilinear images. Tasks included:

Python implementation;

Porting to C++;

Integration in the main AI stack.

The implementation has reduced the R&D time significantly as I did not train on any extra images. I utilized the existing 3D object detector.

Problem: Perspective images are shift-invariant, which means that an object moving along the X and Y axes with constant Z will result in the same appearance model and size. This is not true for fisheye images where an object with the same depth will have a different appearance model and size when translating along the X and Y axes.

Solution: The solution is to transform fisheye images into cylindrical projections which are shift-invariant. Then utilizing a pre-trained 3D object detector model and passing the transformed images to predict 3D objects. Another post-process transformation is applied to get the final 3D position in the real world.

Implementation steps:

Transform fisheye images into cylindrical projections

Inference on cylindrical Images using a pre-trained 3d Object detector

Convert the detected 3D bounding box from 3D cylindrical space to real 3D space