TixelJobs
S
Scaler Studiosvia Indeed

Data Engineer

HR, INPosted 8mo ago
Data EngineerMid LevelFull-time#python#llm#nlp#docker#aws#azure

About the Role

Company: IDEA Foundation
Location: Panchkula, Haryana (travel to client site as required)
Job Type: Full-time
Salary: ₹1,00,000 – ₹1,20,000 per month
Experience Required: Minimum 4 years
Department: AI & Data Engineering
Reports To: Project Manager / AI-ML Architect

About the Role

IDEA Foundation is hiring an experienced Data Engineer for a high-impact government AI modernization project.

This role involves building the data backbone for multiple AI modules, including:

  • Intelligent document processing
  • Multilingual translation
  • Financial forecasting
  • Vendor scoring
  • LLM-powered citizen response generation

These systems will be integrated with SAP S/4HANA through a cloud-hosted microservices architecture.

As the Data Engineer, you will own the complete data layer — from building ETL pipelines and feature pipelines to managing data infrastructure that supports real-time AI inference and model training.

This is a hands-on role requiring strong coding, data engineering, and cloud experience.

Key ResponsibilitiesETL Pipeline Development & Data Integration

  • Design, build, and maintain ETL pipelines using Apache Airflow
  • Extract enterprise data from SAP S/4HANA via OData APIs, CDS views, and SAP Integration Suite
  • Build ingestion workflows for:
  • Financial data
  • Procurement data
  • Citizen case management data
  • Implement delta extraction mechanisms to pull only changed records
  • Develop pipelines to extract and prepare data from legacy systems, file shares, and spreadsheets before SAP migration is complete

Data Preparation for AI/ML Models

  • Convert raw enterprise data into ML-ready feature sets for:
  • Forecasting
  • Vendor scoring
  • Anomaly detection
  • Build and maintain both:
  • Online feature pipelines for real-time inference
  • Offline feature pipelines for batch retraining
  • Normalize dates, currencies, and business entities into standardized formats for model input

Data Infrastructure & Storage Management

  • Manage cloud-based data infrastructure on Azure
  • Work with:
  • PostgreSQL
  • Blob storage
  • Queue services
  • Maintain traceability of AI outputs to source data, model version, and timestamps
  • Implement data versioning, archival policies, and lifecycle management for compliance
  • Set up and manage Qdrant or similar vector databases for knowledge retrieval systems

Document & Content Processing Pipeline

  • Build document pipelines that move files from SAP into Azure Document Intelligence / OCR
  • Process extracted structured data for downstream AI workflows
  • Develop ingestion pipelines for SOPs, circulars, and policy documents
  • Handle text extraction, chunking, embeddings, and vector database loading

Scheduling, Monitoring & Data Quality

  • Configure and manage Airflow DAGs for:
  • Scheduled ETL runs
  • Data refresh
  • Model retraining
  • Rescoring jobs
  • Implement automated data quality checks:
  • Schema validation
  • Null detection
  • Outlier checks
  • Drift monitoring
  • Build dashboards and alerts for pipeline health, freshness, and latency
  • Support infrastructure automation for training workloads

Required Skills & QualificationsMust Have

  • Minimum 4 years of hands-on experience in data engineering
  • Strong proficiency in Python for data engineering
  • Experience with Apache Airflow
  • Strong knowledge of SQL and relational databases (PostgreSQL preferred)
  • Experience with Azure or AWS
  • Familiarity with REST APIs and OData
  • Understanding of data modelling and feature engineering
  • Experience with Git and CI/CD workflows

Strongly Preferred

  • Experience with SAP data extraction (S/4HANA, ECC, CDS views, OData, BAPIs/RFCs, SAP Integration Suite)
  • Familiarity with SAP modules such as:
  • FI/CO
  • MM
  • SD
  • Experience with vector databases like Qdrant, Pinecone, or Weaviate
  • Exposure to Azure Document Intelligence or related Azure AI services
  • Experience in government or public sector projects
  • Working knowledge of Docker
  • Familiarity with data governance, PII handling, and compliance practices

What You Will Build

In the first 8 months, you will help build:

  • ETL pipelines for financial and procurement data from SAP
  • Data pipelines for Forecasting, Vendor Scoring, and Anomaly Detection models
  • A document processing pipeline using OCR and structured extraction
  • A vector knowledge base for SOPs and policy documents
  • Airflow-based workflows for retraining, data refresh, and system monitoring

Why Join IDEA Foundation?

  • Work on a live, large-scale government AI deployment
  • Gain exposure to enterprise AI systems across OCR, NLP, predictive analytics, and LLMs
  • Work with SAP S/4HANA integration, a highly valuable skill set
  • Build systems that directly improve government operations and citizen services

How to Apply

Send your resume and a brief note on your relevant experience to: hr@ideafoundation.in

Please include links to any GitHub projects, open-source contributions, or technical work that demonstrate your data engineering experience.

Job Type: Full-time

Pay: ₹100,000.00 - ₹120,000.00 per month

Work Location: In person

Share