Written by Michael Wyrley-Birch, one of our excellent summer interns.
Over the past few weeks, I have been experimenting with different methods of real time 6D pose estimation for generic objects to allow for realistic digital versions of products to be mapped onto low fidelity prototypes.
The different methods include:
- Using ArUco markers
- Using a generic object tracking algorithm and point clouds.
- Using Retroreflective material applied to the object
Aruco Markers
Different ArUco markers were placed around the object and OpenCV’s aruco package was used to get the pose estimation for each one. This did not give excellent results, was very visually intrusive and caused a lot of problems as they all needed to be very flat on the prototype.
Using a Generic Object Tracker
The SiamMask generic object tracker was used generate 2D bounding boxes and masks of the prototype. These then allowed me to segment the point cloud generated from the camera so it only contains the surface of the tracked object.
The first frames point cloud is used as a reference for all subsequent transformations of the object. For every subsequent frame a new point cloud is generated and compared to the original using ICP, this gives the translation and rotation needed to move the new one onto the old one, giving us the 6D pose estimation. If the amount of overlap is relatively small then the new point cloud is added to the reference one allowing for more accurate ICP registration in future.
Using retroreflective material
A series of retroreflective dots are placed around the object which can be highlighted using an IR camera. These are detected and a convex hull algorithm is used to calculate a mask which can be used in the same way as the previous method to generate the 6d pose estimation.