W
Web Spidersvia muse
Lead Data / ML Engineer
REMOTEPosted 1w ago
MLOpsLeadFull-time#remote
About the Role
We are looking for a Lead Data / ML Engineer to take end\-to\-end technical ownership of a large\-scale data and ML platform on AWS. You will design and build PySpark pipelines, bring ML models and LLM and agent\-based capabilities into production, and own what you build from first design through to running it reliably in production. This is a hands\-on role. Architecture and leadership experience is welcome, but you will still write code every day, and you will be the person others turn to when a pipeline breaks, a job slows down, or a design needs a second opinion.
##### **Location**
Kolkata (Rajarhat–Newtown)
##### **Type**
Full Time
##### **Department**
Data Science \& Engineering
#### **About the Role:**
We are looking for a Lead Data / ML Engineer to take end\-to\-end technical ownership of a large\-scale data and ML platform on AWS. You will design and build PySpark pipelines, bring ML models and LLM and agent\-based capabilities into production, and own what you build from first design through to running it reliably in production.
This is a hands\-on role. Architecture and leadership experience is welcome, but you will still write code every day, and you will be the person others turn to when a pipeline breaks, a job slows down, or a design needs a second opinion.
#### **About the Work:**
You will work with high\-volume consumer transaction data, turning it into insights and predictive models that help understand and anticipate consumer behaviour. The work spans large\-scale data processing, feature engineering, machine learning, and applying LLMs and agents to enrich and make sense of the data.
**Experience:** 7\+ Years
**Location:** Kolkata (Rajarhat–Newtown)
**Mode of Working:** WFO / Hybrid / Remote
**Timing:** Ability to work in the US Eastern time zone. This may be relaxed to half day IST and half day US EST based on project needs.
#### **What We Are Looking For:**
* End\-to\-end ownership: you take a problem from design to production and stay accountable for it once it is live
* Hands\-on depth: you write, review and debug production code yourself, not only design it
* Troubleshooting leadership: you help teammates diagnose failing jobs, performance problems and data issues, and leave them better at solving the next one
* Practical judgement on LLMs and agents: you know where they add real value in a data pipeline and how to measure whether they are working
#### **Key Responsibilities:**
* Design, build and optimize large\-scale ETL/ELT pipelines in PySpark on AWS (EMR, Glue), with strong attention to partitioning, skew, shuffle and cost.
* Productionize ML models and feature pipelines, including training data pipelines, batch and real\-time inference, and model serving on SageMaker.
* Build LLM and agent\-powered capabilities into data workflows, such as document and entity extraction, classification and enrichment at scale, RAG pipelines, and tool\-calling agents, along with evaluation and monitoring of their output quality.
* Orchestrate complex workflows with Apache Airflow (MWAA or self\-managed), ensuring reliability, observability and SLA adherence.
* Architect storage across S3 (data lake), Redshift (warehouse) and RDS, applying best practices for data modeling, partitioning, compression and cost optimization.
* Build containerized data services with Docker and ECS/Fargate, with CI/CD through GitHub Actions.
* Develop event\-driven and serverless processing with Lambda, SQS, SNS and EventBridge.
* Use AI coding assistants (Claude Code, Cursor, Kiro) to speed up delivery, and set the team's standards for reviewing and validating AI\-generated code.
* Be the go\-to person for production issues: lead root\-cause analysis on failing or slow jobs, data quality incidents and model problems, and help teammates debug their own work.
* Review code, guide design decisions and help engineers on the team grow technically.
* Implement data quality, monitoring and alerting
##### **Location**
Kolkata (Rajarhat–Newtown)
##### **Type**
Full Time
##### **Department**
Data Science \& Engineering
#### **About the Role:**
We are looking for a Lead Data / ML Engineer to take end\-to\-end technical ownership of a large\-scale data and ML platform on AWS. You will design and build PySpark pipelines, bring ML models and LLM and agent\-based capabilities into production, and own what you build from first design through to running it reliably in production.
This is a hands\-on role. Architecture and leadership experience is welcome, but you will still write code every day, and you will be the person others turn to when a pipeline breaks, a job slows down, or a design needs a second opinion.
#### **About the Work:**
You will work with high\-volume consumer transaction data, turning it into insights and predictive models that help understand and anticipate consumer behaviour. The work spans large\-scale data processing, feature engineering, machine learning, and applying LLMs and agents to enrich and make sense of the data.
**Experience:** 7\+ Years
**Location:** Kolkata (Rajarhat–Newtown)
**Mode of Working:** WFO / Hybrid / Remote
**Timing:** Ability to work in the US Eastern time zone. This may be relaxed to half day IST and half day US EST based on project needs.
#### **What We Are Looking For:**
* End\-to\-end ownership: you take a problem from design to production and stay accountable for it once it is live
* Hands\-on depth: you write, review and debug production code yourself, not only design it
* Troubleshooting leadership: you help teammates diagnose failing jobs, performance problems and data issues, and leave them better at solving the next one
* Practical judgement on LLMs and agents: you know where they add real value in a data pipeline and how to measure whether they are working
#### **Key Responsibilities:**
* Design, build and optimize large\-scale ETL/ELT pipelines in PySpark on AWS (EMR, Glue), with strong attention to partitioning, skew, shuffle and cost.
* Productionize ML models and feature pipelines, including training data pipelines, batch and real\-time inference, and model serving on SageMaker.
* Build LLM and agent\-powered capabilities into data workflows, such as document and entity extraction, classification and enrichment at scale, RAG pipelines, and tool\-calling agents, along with evaluation and monitoring of their output quality.
* Orchestrate complex workflows with Apache Airflow (MWAA or self\-managed), ensuring reliability, observability and SLA adherence.
* Architect storage across S3 (data lake), Redshift (warehouse) and RDS, applying best practices for data modeling, partitioning, compression and cost optimization.
* Build containerized data services with Docker and ECS/Fargate, with CI/CD through GitHub Actions.
* Develop event\-driven and serverless processing with Lambda, SQS, SNS and EventBridge.
* Use AI coding assistants (Claude Code, Cursor, Kiro) to speed up delivery, and set the team's standards for reviewing and validating AI\-generated code.
* Be the go\-to person for production issues: lead root\-cause analysis on failing or slow jobs, data quality incidents and model problems, and help teammates debug their own work.
* Review code, guide design decisions and help engineers on the team grow technically.
* Implement data quality, monitoring and alerting
Similar Jobs
2
2P Perfect Presentation• 1mo ago
Senior AI/ML engineer
Giza, Al Jizah, EgyptFull-time
P
Parser• 2mo ago
MLOps Technical Manager
Spain, England#SP-UK, Greater LondonFull-time
Z
Zenlayer• 2mo ago
AI Infrastructure Procurement Specialist
Diamond Bar, CaliforniaFull-time
A
Amidel• 6mo ago
AWS MLOps Engineer
Johannesburg, GautengFull-time