Markie

Human-computer interaction plays a crucial role in daily activities, underscoring the importance of developing intuitive, adaptable, and precise interaction mechanisms. In this work, we present a novel vision-based input device that leverages computer vision techniques and fiducial object pose estimation to enable seamless user interaction with digital systems. This input device aims to capture a wide range of user actions and translate them into system commands, offering a natural and adaptable means of interaction.

This work is developed within the framework of the UNITE project, which specifically aims to foster inclusion in education by integrating advanced technological solutions. To achieve this, we designed an interactive experience where users can manipulate and interact with different 3D objects using our developed input device. By providing an accessible and engaging alternative to conventional input methods, this system can reduce the barriers encountered by students with diverse needs.

A preliminary user study conducted with several participants demonstrated high satisfaction, emphasizing the system's fluidity, intuitiveness, and ease of use. These findings suggest that vision-based input devices hold significant potential for enhancing interaction paradigms in both educational and general-purpose computing environments. Furthermore, due to its adaptability and ability to track user interactions in real-time, the proposed system could be seamlessly integrated into new scenarios such as virtual reality or extended reality environments, providing an intuitive and immersive interaction modality for diverse applications.

3D-printed fiducial objects are tracked in real time by a standard, uncalibrated webcam, no external sensors, markers on the walls, or camera calibration step required. C++ and OpenCV detect the object's visible faces frame by frame, fuse them into a single 6DoF pose estimate, and reproject it onto an on-screen cursor.

That pose stream is what users actually manipulate: rotating the physical object rotates the 3D model on screen, tilting or moving it pans and scales the view, and swapping between differently shaped objects switches which model is active, all rendered live in a Godot 4 interface, so each physical gesture maps directly onto a system command without any keyboard, mouse, or touch input.

Guiding the experience is Markie, a virtual assistant modeled in Blender who narrates each action in real time and speaks four languages (English, Spanish, Italian, and Turkish) to keep the interaction accessible and easy to follow.