AI Human Activity Recognition for On-board BAS Experiments
“ISRO needs an offline AI assistant that watches astronauts perform science experiments on space stations through fixed cameras. Because communication with Earth has heavy delays, this software must run locally, track hand-object interactions, verify if experiment steps are done in the right order, and speak out audio alerts if a step is skipped.”
Deconstructed Problem Blueprint
WHAT TO BUILD?
An offline-first Edge AI software application with computer vision (Human Activity Recognition, Hand-Object Interaction, Pose Estimation), text-to-speech voice alerts, a structured text logger, and a local monitoring GUI dashboard.
WHO IS IT FOR?
Primary Beneficiaries: Astronauts on space missions (BAS / lunar stations) and ISRO scientists/payload managers on Earth monitoring mission logs post-mission.
WHY DOES IT EXIST?
Real-time communication delays with Earth make immediate ground-based troubleshooting impossible. Astronauts operating in microgravity with complex protocols can miss steps or alter sequences, compromising high-cost space experiments.
INPUT → PROCESS → OUTPUT ARCHITECTURE PIPELINE
Live video feed from a fixed payload camera, predefined experiment protocol configuration files (e.g., JSON detailing steps like pick red box -> pick yellow box), custom synthetic local training dataset.
Local edge-based inference running object detection (YOLO), hand-object interaction tracking, state-machine validation matching current visual state against experiment protocol sequence, and text-to-speech generation.
Real-time voice alerts for out-of-sequence actions or skipped steps, next-step audio/visual suggestions, timestamped structured log files (.txt/.json) of experiment outcomes, local video recording storage, and an interactive GUI dashboard streaming to a local IP.
CONSTRAINTS
- Must run entirely offline on edge hardware without relying on cloud internet connectivity
- Low compute footprint optimized for resource-constrained edge devices (Jetson Nano/Orin or local laptop)
- Must handle unique microgravity orientations where traditional 'up/down' floor references do not apply
- Low-latency processing to provide immediate feedback during fast-moving actions
JURY METRICS
- Experiment step classification accuracy and sequence state-machine precision
- Inference latency (frames per second on edge hardware)
- False alarm rate on out-of-sequence alerts
- Clarity and responsiveness of voice alerts and GUI usability
CORE BOTTLENECK
Building a robust state-transition logic that accurately handles ambiguous hand-object interactions and occlusions in real-time without cloud assistance.
Why You Should Choose This PS (Pros)
- Extremely high prestige and relevance being an ISRO problem statement
- Clear, well-defined deliverables with a tangible demo (webcam + colored boxes simulation)
- Strong wow factor through live edge video processing, voice synthesis, and automated logging
Challenges & Risks to Watch (Cons)
- Requires building custom local training data (recording yourself with webcams moving boxes)
- Complex integration of multiple AI components (object detection, spatial tracking, state machine)
- Crowded category if multiple teams choose space tech without unique edge features
Recommended Tech Stack & Grand Finale Features
Killer Features to Impress the SIH Jury
View Official SIH 2026 Portal Raw Text▼
Background As humanity aims for space missions such as BAS and lunar missions, real-time ground support becomes impossible due to communication delays. An AI-based HAR system acts as an on-board assistant that supports the execution of scientific experiments, ensuring the success of science beyond Earth's orbit.
In the space environment, AI-based HAR system may act as mission-critical support for astronauts. By tracking astronaut movements and activities in real time, HAR ensures scientific experiments and related protocols are executed flawlessly without requiring constant, high-bandwidth communication with mission control.
Description Challenge is to design and train an AI model that recognizes and validates the sequence of a pre-defined experiment using human activity recognition techniques.
Standalone operation: Space stations operate on restricted data bandwidth to Earth. Rather than streaming raw video to ground control, data is processed locally at the 'edge.' Inputs are given from fixed-payload cameras.
Dataset generation to train model for object detection, pose estimation and hand-object interaction based on the steps of the experiment.
Optional: Another challenge is that Standard 2D or ground-based 3D posture models fail because astronauts do not have a fixed 'up' or 'down' orientation. The AI model should use orientation-agnostic 3D Human Mesh Recovery (HMR) to track the astronaut’s body relative to the payload rack, not the floor.