Home/Problem Statements/SIH26056

SIH26056SoftwareSmart Automation

₹1,50,000 Prize (₹75k Joint)

•

Deadline: 20 September 2026

•

Submitted Ideas: 0/500

Development of a Real-time Airfare Price Index for India through Automated Web Scraping of Airline and Online Travel Aggregator Portals for Augmentation of the Consumer Price Index (CPI).

Ministry / Organization: MoSPI

Department: Data Informatics & Innovation Division (DIID)

SIH Official Portal Dataset Repository

Technical Complexity

8/10

Dataset Accessibility

8/10

36h Hackathon Viability

9/10

Step 0 • Zero-Jargon Layman Summary

“The government needs to track how air ticket prices change every day to accurately calculate retail inflation (CPI) in India. Right now, officials collect prices manually, which misses dynamic online price surges, so we need to build an automated system to scrape, clean, and index flight fares from major airlines and travel websites.”

The DecodeX PS Decoder™

Deconstructed Problem Blueprint

Step 1

WHAT TO BUILD?

An end-to-end automated web-scraping platform and analytics dashboard that extracts airfares, normalizes data, and calculates a real-time Airfare Price Index (APIx) across major Indian domestic routes.

Step 2

WHO IS IT FOR?

MoSPI (Ministry of Statistics and Programme Implementation) and the National Statistical Office (NSO) are the owners/users, while the Reserve Bank of India (RBI) consumes the index for monetary policy. Everyday citizens benefit through more accurate inflation metrics.

Step 3

WHY DOES IT EXIST?

Over 90% of tickets are bought online with dynamic pricing fluctuating 200-400% daily based on demand, advance booking, and fuel costs. Manual surveys are obsolete, slow, and fail to capture what consumers actually pay.

Step 4

INPUT → PROCESS → OUTPUT ARCHITECTURE PIPELINE

01 • Inputs & Telemetry

Scheduled scraping configurations, list of representative city-pairs (DEL-BOM, etc.), advance-purchase windows (T+1 to T+45), and raw HTML/JSON responses from airline sites and OTAs.

02 • Processing & AI Core

Headless browser scraping with proxy/IP rotation, parsing and schema normalization, data cleaning (handling sold-out flights, separating base fares from taxes/fees), and index construction using weighted route baskets.

03 • Actionable Deliverables

A cleaned time-series airfare database, route-wise heatmaps, lead-time elasticity curves, and a REST API for NSO/RBI consumption.

Step 5

CONSTRAINTS

Must comply with website terms of service and robots.txt using ethical rate-limiting
Must handle complex JavaScript rendering, anti-bot mechanisms, and dynamic CAPTCHAs
Must accurately separate base fare from user development fees and taxes

Step 6

JURY METRICS

Scraping success rate and uptime under anti-bot constraints
Accuracy of data extraction and de-duplication pipeline
Performance and reliability of the index-construction calculation module
UI/UX clarity and responsiveness of the dashboard for policymakers

Step 7

CORE BOTTLENECK

Bypassing sophisticated airline/OTA anti-bot firewalls (Cloudflare, Akamai) without getting IP addresses permanently blocked or flagged.

Why You Should Choose This PS (Pros)

Extremely high national economic importance with direct government utility (MoSPI & RBI)
Clear deliverables: scraping engine, database, calculation formula, and dashboard
Great potential for a live visual dashboard that instantly wows judges

Challenges & Risks to Watch (Cons)

Airlines frequently update their frontend DOM structures, breaking brittle scrapers
Aggressive anti-scraping protections can cause high failure rates during live evaluation
Requires careful handling of proxy pools and rate-limiting

Hackathon Blueprint

Recommended Tech Stack & Grand Finale Features

Frontend & Mobile

React.jsTailwind CSSRecharts or Chart.js for data visualization

Backend & APIs

FastAPI (Python)Celery for background schedulingPlaywright / Selenium for browser automation

AI / ML & Models

Regex and rule-based NLP normalizersOutlier detection using Interquartile Range (IQR) or Isolation Forests

Database & Cloud

PostgreSQLRedis for task queues and caching

Killer Features to Impress the SIH Jury

An automated fallback mechanism that switches between direct airline parsing and OTA aggregation if an airline blocks a request

An anomaly detection alert system that flags sudden 50%+ price spikes ahead of festival seasons

A transparent index audit trail showing exact weighting formulas and raw-to-clean data lineage

View Official SIH 2026 Portal Raw Text

▼

•Background The Consumer Price Index (CPI) released by the National Statistical Office (NSO), Ministry of Statistics and Programme Implementation (MoSPI), is the primary measure of retail inflation in India and is used by the Reserve Bank of India (RBI) for setting monetary policy under the flexible inflation-targeting framework. The current CPI framework, however, collects 'Transport and Communication' sub-group prices, including air travel fares, primarily through manual price-collection from a limited set of outlets and ticketing offices. With over 90% of domestic air tickets in India now sold online through airline websites and Online Travel Aggregators (OTAs) such as MakeMyTrip, Yatra, EaseMyTrip, Cleartrip, Ixigo and Goibibo, manual collection no longer captures the highly dynamic, route-specific, and time-sensitive pricing that Indian consumers actually face. Airfares in India follow dynamic pricing where the same sector can vary by 200-400% within a single day depending on advance-booking window, day-of-week, demand surges, festival seasons and fuel-price-linked surcharges. There is therefore an urgent need for an automated, scalable and high-frequency data-collection system that mirrors what a real Indian traveller pays.

•Detailed Description The problem statement envisages development of an end-to-end software platform that automatically web-scrapes airfare data from major Indian airline websites (IndiGo, Air India, Air India Express, Akasa Air, SpiceJet) and leading OTAs, cleans and normalises the collected price quotes, and computes a Real-time Airfare Price Index (APIx) at daily, weekly and monthly frequencies. The system shall maintain a basket of representative city-pairs (such as DEL-BOM, DEL-BLR, BOM-BLR, DEL-CCU, BLR-HYD, MAA-DEL, etc.) selected on the basis of DGCA passenger-traffic data, and shall capture fares for multiple advance-purchase windows (T+1, T+7, T+15, T+30, T+45 days). Scraping must handle JavaScript-rendered pages, dynamic CAPTCHAs, anti-bot measures, IP rotation, and session management while remaining compliant with the robots.txt and terms of service of source websites, with appropriate rate-limiting and ethical-scraping safeguards. The collected raw quotes shall be passed through a data-cleaning pipeline that removes outliers, handles missing values, accounts for cancellations/sold-out flights, and separates base fare from taxes, user-development fee and convenience charges. The dashboard must visualise price trends, sector-wise heatmaps, lead-time elasticity curves, and provide an API that the NSO and RBI can consume.

•Expected Solution A working software prototype consisting of (a) a robust, ethically-designed multi-source web-scraping engine using Python (Scrapy/Selenium/Playwright) capable of scheduled daily extraction from airline portals; (b) a cleaned and de-duplicated airfare database with metadata such as origin, destination, carrier, advance-purchase window, fare-class, base fare, taxes and total fare; (c) an index-construction module based on PSD given routes and weights; (d) a web-based interactive dashboard showing the daily Airfare Price Index. The solution must include documentation, automated testing, and demonstrate at least 30 days of back-tested results against publicly available DGCA monthly average-fare data.