Data Engineer
About the Role
Company: IDEA Foundation
Location: Panchkula, Haryana (travel to client site as required)
Job Type: Full-time
Salary: ₹1,00,000 – ₹1,20,000 per month
Experience Required: Minimum 4 years
Department: AI & Data Engineering
Reports To: Project Manager / AI-ML Architect
About the Role
IDEA Foundation is hiring an experienced Data Engineer for a high-impact government AI modernization project.
This role involves building the data backbone for multiple AI modules, including:
- Intelligent document processing
- Multilingual translation
- Financial forecasting
- Vendor scoring
- LLM-powered citizen response generation
These systems will be integrated with SAP S/4HANA through a cloud-hosted microservices architecture.
As the Data Engineer, you will own the complete data layer — from building ETL pipelines and feature pipelines to managing data infrastructure that supports real-time AI inference and model training.
This is a hands-on role requiring strong coding, data engineering, and cloud experience.
Key ResponsibilitiesETL Pipeline Development & Data Integration
- Design, build, and maintain ETL pipelines using Apache Airflow
- Extract enterprise data from SAP S/4HANA via OData APIs, CDS views, and SAP Integration Suite
- Build ingestion workflows for:
- Financial data
- Procurement data
- Citizen case management data
- Implement delta extraction mechanisms to pull only changed records
- Develop pipelines to extract and prepare data from legacy systems, file shares, and spreadsheets before SAP migration is complete
Data Preparation for AI/ML Models
- Convert raw enterprise data into ML-ready feature sets for:
- Forecasting
- Vendor scoring
- Anomaly detection
- Build and maintain both:
- Online feature pipelines for real-time inference
- Offline feature pipelines for batch retraining
- Normalize dates, currencies, and business entities into standardized formats for model input
Data Infrastructure & Storage Management
- Manage cloud-based data infrastructure on Azure
- Work with:
- PostgreSQL
- Blob storage
- Queue services
- Maintain traceability of AI outputs to source data, model version, and timestamps
- Implement data versioning, archival policies, and lifecycle management for compliance
- Set up and manage Qdrant or similar vector databases for knowledge retrieval systems
Document & Content Processing Pipeline
- Build document pipelines that move files from SAP into Azure Document Intelligence / OCR
- Process extracted structured data for downstream AI workflows
- Develop ingestion pipelines for SOPs, circulars, and policy documents
- Handle text extraction, chunking, embeddings, and vector database loading
Scheduling, Monitoring & Data Quality
- Configure and manage Airflow DAGs for:
- Scheduled ETL runs
- Data refresh
- Model retraining
- Rescoring jobs
- Implement automated data quality checks:
- Schema validation
- Null detection
- Outlier checks
- Drift monitoring
- Build dashboards and alerts for pipeline health, freshness, and latency
- Support infrastructure automation for training workloads
Required Skills & QualificationsMust Have
- Minimum 4 years of hands-on experience in data engineering
- Strong proficiency in Python for data engineering
- Experience with Apache Airflow
- Strong knowledge of SQL and relational databases (PostgreSQL preferred)
- Experience with Azure or AWS
- Familiarity with REST APIs and OData
- Understanding of data modelling and feature engineering
- Experience with Git and CI/CD workflows
Strongly Preferred
- Experience with SAP data extraction (S/4HANA, ECC, CDS views, OData, BAPIs/RFCs, SAP Integration Suite)
- Familiarity with SAP modules such as:
- FI/CO
- MM
- SD
- Experience with vector databases like Qdrant, Pinecone, or Weaviate
- Exposure to Azure Document Intelligence or related Azure AI services
- Experience in government or public sector projects
- Working knowledge of Docker
- Familiarity with data governance, PII handling, and compliance practices
What You Will Build
In the first 8 months, you will help build:
- ETL pipelines for financial and procurement data from SAP
- Data pipelines for Forecasting, Vendor Scoring, and Anomaly Detection models
- A document processing pipeline using OCR and structured extraction
- A vector knowledge base for SOPs and policy documents
- Airflow-based workflows for retraining, data refresh, and system monitoring
Why Join IDEA Foundation?
- Work on a live, large-scale government AI deployment
- Gain exposure to enterprise AI systems across OCR, NLP, predictive analytics, and LLMs
- Work with SAP S/4HANA integration, a highly valuable skill set
- Build systems that directly improve government operations and citizen services
How to Apply
Send your resume and a brief note on your relevant experience to: hr@ideafoundation.in
Please include links to any GitHub projects, open-source contributions, or technical work that demonstrate your data engineering experience.
Job Type: Full-time
Pay: ₹100,000.00 - ₹120,000.00 per month
Work Location: In person