On-device Visual Perception for Light-weight Browser Agents
“We are building a smart browser extension that looks at your computer screen using lightweight AI to understand what you are doing. Before sending any data to a remote cloud server for complex thinking, it automatically hides private details like passwords and personal info. The server then tells the browser what actions to take, allowing AI assistants to help you safely without leaking your confidential data.”
Deconstructed Problem Blueprint
WHAT TO BUILD?
A privacy-preserving browser extension and supporting backend server architecture that executes client-side vision processing, dynamic PII masking, and cloud-based VLM/LLM reasoning for automated task execution.
WHO IS IT FOR?
End-users who want AI workflow automation without compromising sensitive data, and enterprise or government organizations (like ISRO) managing sensitive internal digital workflows.
WHY DOES IT EXIST?
Current agentic AI pipelines require uploading raw screen states and visual context directly to cloud servers, creating severe security and privacy risks regarding accidental exposure of personal or classified data.
INPUT → PROCESS → OUTPUT ARCHITECTURE PIPELINE
Live browser screen capture frames, DOM element coordinates, user task prompts, and open-source vision-language model configurations.
Client-side WebGPU-accelerated Vision Transformer (ViT) screen analysis, dynamic bounding-box detection of sensitive PII/passwords, local redaction/blurring, and transmission of anonymized payloads to a server-side Vision-Language Model (VLM) for task planning.
Sanitized visual snapshots transmitted to the cloud, and precise actionable UI commands returned to the browser extension (e.g., 'click_element', 'scroll_down', 'type_text') for automated browser execution.
CONSTRAINTS
- Client-side inference must run smoothly within browser resource limitations using WebGPU/ONNX Runtime Web.
- Redaction of PII, passwords, and faces must happen in real-time with high precision before any network transmission.
- Low end-to-end latency between screen state reading, server decision, and local client execution.
JURY METRICS
- Accuracy of visual context understanding from the screen (25%)
- Recall and precision for detection of sensitive/PII data (20%)
- Precision of local redaction/masking (20%)
- Client-side resource utilization such as CPU/GPU memory and usage (20%)
- Overall end-to-end latency of the provided task (15%)
CORE BOTTLENECK
Achieving real-time, low-latency client-side vision processing using WebGPU/ONNX without freezing the browser tab or exceeding low resource limits while accurately detecting layout structures.
Why You Should Choose This PS (Pros)
- Addresses an extremely relevant and cutting-edge domain in AI (Edge-Cloud hybrid architectures and privacy-first agents).
- Clear, quantifiable evaluation metrics provided directly in the problem statement.
- Strong potential for high visual impact during the live prototype demo.
Challenges & Risks to Watch (Cons)
- Running Vision Transformers client-side inside a browser extension can be heavy and prone to compatibility issues across different GPUs.
- Synchronizing client-side DOM parsing with visual bounding boxes requires robust multi-modal alignment.
Recommended Tech Stack & Grand Finale Features
Killer Features to Impress the SIH Jury
View Official SIH 2026 Portal Raw Text▼
Background AI agents are becoming omnipresent in the current era and can play an important role in our digital interactions. If an agentic AI pipeline has access to our visual context, screen states, they can assist users in complex workflows and automate many tasks. Most of the agentic AI pipelines are deployed on server side which limits the type to data that a user can share with it. It would open a new dimension of possibilities, if a local agent is deployed on user machine particularly browser which can eliminate the need to share the sensitive data with the server. Local system generally has fewer resources than server and is unable to host a full-fledged pipeline therefore only the non-sensitive data such as structure of the screen, application fields etc can be sent to server for processing.
Modern browser APIs (such as WebGPU and WebAssembly) and local inference libraries (like ONNX Runtime Web and Transformers.js) have unlocked the ability to run lightweight machine learning models directly on the client. The aim is to bridge these two environments: leveraging the reasoning power of cloud or server based AI while strictly enforcing data privacy at the client side.
Description Participants are required to build a privacy-preserving vision agent which runs on browser. This involves implementing a client-side architecture where a local Vision Transformer (ViT) or equivalent computer vision model 'reads' the user's screen and takes decision based on that. If it requires the visual context to be sent to server, it shall sanitize the sensitive/PII data using DOM tags or any other method, before any network request is made. It should dynamically detect and redact sensitive elements. For example, blurring faces, blacking out passwords, and masking PII etc. Only this anonymized, unidentifiable data should be transmitted to the central server which should be aware for this redaction scheme and can process data accordingly. The server will then process the sanitized context and return actionable commands for the browser agent to execute. Participants must balance the trade-offs between inference latency and the accuracy.
Expected Solution A successful submission should include a working prototype consisting of client side extension and server that demonstrates the following:
Client-side (extension/JS) running in popular browsers (chrome, Firefox) components:
Server-side implementation components:
Evaluation will be done on the following metrics:
1-Accuracy of visual context from screen – 25% 2-Recall and precision for detection of sensitive/PII data – 20% 3-Precision of redaction – 20% 4-Client side resource utilization – 20% 5-Overall end-to-end latency of the provided task -15%