Universal Log Pre-processing Framework
“Organizations collect massive amounts of digital activity records from hundreds of different devices, but they all speak different 'languages' making them hard to analyze. This project asks you to build a universal translator that converts all these messy, unique records into one standard format so security teams can easily spot threats.”
Deconstructed Problem Blueprint
WHAT TO BUILD?
A high-performance, containerized data pipeline framework that ingests, parses, normalizes, and stores heterogeneous log data into a unified, queryable schema.
WHO IS IT FOR?
National Technical Research Organisation (NTRO), cybersecurity analysts, SOC (Security Operations Center) teams, and IT infrastructure managers.
WHY DOES IT EXIST?
Current security tools struggle because every vendor uses a different log format. Manually writing parsers for every new device is slow, error-prone, and creates blind spots that hackers exploit.
INPUT → PROCESS → OUTPUT ARCHITECTURE PIPELINE
Raw log streams (Syslog, JSON, XML, CSV, CEF, LEEF) from network devices, servers, cloud services, and IoT endpoints.
Ingestion via message queues (Kafka/NATS), schema-agnostic parsing using regex/NLP, normalization to a Common Event Format (CEF/ECS), and metadata tagging for traceability.
A unified, searchable data stream ready for SIEM/Data Lakes, complete with a dashboard for log health and audit trails.
CONSTRAINTS
- Must be deployable in air-gapped (offline) environments.
- Must handle massive throughput (billions of events/day).
- Must preserve original raw logs for forensic integrity.
- Must be vendor-agnostic and containerized (Docker/K8s).
JURY METRICS
- Parsing latency (ms per log event).
- Throughput (events per second per node).
- Schema normalization accuracy (percentage of fields correctly mapped).
- Resource efficiency (CPU/RAM usage per 10k logs/sec).
CORE BOTTLENECK
Designing a 'Universal' schema that is flexible enough to handle future, unknown log formats without requiring a full code rewrite every time a new device is added.
Why You Should Choose This PS (Pros)
- High strategic value for national security agencies.
- Clear, well-defined technical requirements.
- Excellent potential for a scalable, enterprise-grade product.
Challenges & Risks to Watch (Cons)
- High performance requirements make it difficult to demo 'billions of events' in a 36-hour hackathon.
- Requires deep knowledge of networking protocols and log structures.
Recommended Tech Stack & Grand Finale Features
Killer Features to Impress the SIH Jury
View Official SIH 2026 Portal Raw Text▼
The diversity of log structures creates significant challenges in centralized monitoring, security operations, compliance reporting, incident investigation, and threat analytics. Security teams often spend substantial effort developing source-specific parsers and normalization rules before the data can be effectively utilized by SIEM, data lake, or machine learning platforms.
As organizations adopt hybrid, multi-cloud, and AI-driven environments, the need for a universal and extensible log standard that can accommodate both current and future data sources have become increasingly critical.
The framework should support diverse event sources while preserving the original event data for forensic and compliance purposes. It should transform heterogeneous logs into a unified schema that enables consistent analytics, correlation, visualization, threat hunting, anomaly detection, and machine learning applications.
The framework must be scalable, extensible, vendor-agnostic, and suitable for deployment in Big Data environments handling billions of events per day.
a) Preserve complete raw event data without information loss.
b) Extract and parse source-specific attributes.
c) Normalize fields into a common event taxonomy.
d) Maintain traceability between normalized and original events.
e) Plug-and-play on boarding of new log sources.
f) Unified visibility across enterprise environments.
g) Efficient SIEM and Data Lake integration.
h) AI/ML-ready security and operational analytics.
i) Reduced parser development effort.
j) The solution shall be deployable in an air-gapped network.
k) Solution may be packaged in a container for making it platform independent.