Mars Rover Autonomous Keyboard Typing
Vision-guided autonomous keyboard typing for Mars Rover missions: YOLOv8 detection with PaddleOCR key recognition, up to 30% faster on-rover inference.
From camera to keypress
Scroll to watch it unfold. Each beat is one documented step of the project.
- 01
Camera frame
The rover's onboard camera sees the keyboard.
- 02
Keyboard detection
A YOLOv8 model finds the keyboard in real time.
- 03
Key recognition
PaddleOCR reads the key labels.
- 04
Angle mapping
The chosen key is mapped to arm angles.
- 05
Keypress
The arm presses the key.
- 06
Outcome
The rover's arm presses the right key without a human aiming the camera.
- 07
Unreadable key
With no confident read, no keypress is sent.
An illustration of how the system works, drawn from the write-up below. It is not recorded output.
Overview
Part of the BRACU Mongol-Tori Mars Rover team's vision-guided autonomy work, this prototype lets the rover's onboard vision system find and read a physical keyboard so commands can be entered without a human aiming a camera by hand — a small piece of the same object-detection and autonomous-vision pipeline that improved the rover's on-board inference performance by up to 30% for the University Rover Challenge.
Approach
A YOLOv8 model handles real-time keyboard detection in the camera frame, and PaddleOCR reads the individual keys once the keyboard region is localized, so the system knows both where a key is and what it says. Detected key positions are angle-mapped to the rover arm's reference frame so a keypress can be executed as a physical motion rather than just a bounding box.
By the numbers
Highlights
- Real-time Detection
- Angle Mapping
- Rover Integration
Tech stack & key skills
Core tools, methods and skills demonstrated in this project: