Gesture-controlled CAD

This project took place as part of a summer internship in 2021.

The initial goal of the project was to replicate the outcome of previous applications of hand tracking in Computer Aided Design. Namely the navigation interface demonstrated by Elon Musk and SpaceX (link) as well as an integration demonstrated by Autodesk (link). The development of this platform using the Fusion API and Leap Motion SDK would allow for the lab to conduct further research in this area in future.

Once previous applications had been replicated, new ones could then be developed, built upon and their suitability investigated.

The Gesture Control System

Data from the Leap Motion device was sent to a Fusion 360 add-in to create a real time interface. This was achieved by constantly updating a CSV file that could be read by the Fusion API. This was sufficient in this case, but a more robust solution may be required for more complex interfaces.

Visualisation applications were achieved by movement of the camera within Fusion 360. The most successful of which was the adjustment of the roll, yaw and zoom of the model simultaneously. These parameters could be manipulated either by using one hand – the roll and yaw of the hand and height of the hand controlling the zoom – or two hands where the orientation of the model is as if it were between the two hands; this was an incredibly intuitive way to interact with the model.

Next was geometry creation, this was a much harder challenge as very little work had been previously conducted in this area. The nearest was a Virtual Reality CAD project that used a game engine to create geometry and sent the data to the Fusion API for a model to be created. (link) This goal for this project was to not only send the data to Fusion but provide a real time interface to visualise result.

The application of this interface is most useful in the early geometric design phase, where the rough geometry and structure of the model is created. The ability to quickly express concepts leads to fast iteration and accelerated solution-finding.

There is a balance between workflow benefit and accuracy. Geometry could be created step-by-step but provide little benefit over a mouse whereas the final outcome allows a body’s size and location to be chosen with one gesture. The disadvantage of this is that it is hard to balance all these variables at once. It is especially hard due to the 3D geometry from a 3D gesture being displayed on a 2D screen with fixed camera angle. A future participant study could investigate the nuances of this and help to design a more intuitive interface.

Towards the end of this project, a paper was published discussing the best gestures for a CAD interface. (link) Many of the ideas in this paper had already been implemented but it helped to refine the outcome and provide a framework to work from. This influenced the decision to first generate a unit body at the origin and then use a pinch gesture with two hands to rescale and reposition the body. Once positioned, the pinch gesture could be released, and the body turned green to show confirmation.

The final outcome is usable and, in some cases, results in faster model creation than with a mouse and keyboard. There are, however, a few drawbacks that must be considered and solved for it to be a truly useful interface. The responsiveness of the interface is inconsistent, this can be frustrating with hands held in the air for an extended period of time. Occasionally, one hand may be occluded by the other so that the Leap Controller cannot detect it. Finally, there is a relatively small working area making it difficult to create larger and more complex models.

Next Steps for the Project

There is plenty of opportunity to optimise the code, particularly in the communication between the Leap Python script and the Fusion API. Gesture suitability could also be investigated further using the aforementioned paper as a framework (link) and a participant study. This project serves proof of concept to demonstrate the feasibility of such an interface.

To solve the problem of occluded hands, multiple Leap Motion controllers could be used. This is something that is becoming more widespread, but it is unlikely that the Leap Python SDK will have support for it as it is already multiple versions behind the SDK for Game Engines such as Unity. This project could involve interaction with a game engine and result in some interesting VR applications with CAD. (link)

There is potential for future Fusion API development as it was not designed for real time interfaces such as this. If it can be demonstrated to be of use, there is the motivation for certain aspects of the API to be improved by Autodesk for a more streamlined interface.

This project has demonstrated the feasibility and limitations of a gesture-based CAD interface. It will hopefully spark further research that can lead to a novel interface that makes interaction with CAD software more accessible and more efficient.

It has been a great experience to collaborate in person in the lab after working on remote projects for the past year. Working on something that is new, exciting and has real applications is why I chose a career in engineering, and this has shown me that working in industry is not the only way to do this. It has given me an interesting insight into the field of research and the challenges that come with it.

Lloyd Cleary-Richards


Lloyd Cleary-Richards


Chris Snider

Senior Lecturer

Blog Posts

See all related posts here