LLM R&D Specialist - Gen AI
About the Role
Engineering
Bengaluru
5 - 8 years experience
Responsibilities
The IndiaAI Mission is building India’s next-generation foundational models. As a Senior ML Engineer, you will design and optimize end-to-end LLM pre-training pipelines, build high-performance distributed training systems, and lead large-scale multilingual data creation/curation efforts. You will work with frameworks like PyTorch Distributed, DeepSpeed, Megatron-LM, and NVIDIA NeMo, and operate large GPU clusters orchestrated via SLURM.
This is a deeply technical, systems-heavy role ideal for engineers who have experience training large transformer models at scale.
Key Responsibilities
1. LLM Pre-Training & Model Engineering
- Build and optimize end-to-end LLM pre-training pipelines (7B–70B+).
- Implement and tune training using:
- PyTorch Distributed
- DeepSpeed (ZeRO, FSDP)
- Megatron-LM / Megatron-Core
- NVIDIA NeMo Framework
- XLA / JAX (nice to have)
- Optimize transformer model components: attention kernels, RoPE, KV-cache, tensor/pipeline parallelism, FlashAttention, MoE.
- Design training recipes, hyperparameters, and scaling strategies.
-
2. Distributed Training & Systems Optimization
- Manage and schedule large multi-node training jobs using SLURM.
- Optimize NCCL communication, RDMA/InfiniBand, NVLink/NVSwitch topologies.
- Diagnose and resolve distributed training failures (hangs, desyncs, GPU OOM, network issues).
- Implement mixed precision, gradient checkpointing, activation recompute for efficiency.
-
3. Data Creation, Curation & Tokenization
- Lead large-scale dataset creation pipelines for multilingual + Indian-language data.
- Build workflows for:
- Web-scale text extraction (Indian languages & domain data)
- Data cleaning, deduplication, normalization
- Quality filtering (heuristics, models, toxicity checks)
- Multi-stage data labeling & human-in-the-loop QA
- Implement tokenizers, vocabulary strategy, and data sharding at scale.
- Build streaming dataloaders optimized for massive throughput.
-
4. Infrastructure Collaboration
- Work with infra & HPC teams to ensure high GPU utilization.
- Design checkpointing strategies, artifact storage, and experiment tracking.
- Build dashboards for training metrics, throughput optimization, and data stats.
Required Skills
- 5+ years experience in ML engineering / distributed systems / LLM training.
- Hands-on experience with LLM pre-training (preferred: 7B+ models).
- Strong expertise with:
- PyTorch Distributed
- DeepSpeed (ZeRO, FSDP)
- Megatron-LM / Megatron-Core
- NVIDIA NeMo (Megatron + NeMo recipes)
- Deep understanding of transformer internals: attention, RoPE, positional encodings, tensor/pipeline parallelism, FlashAttention.
- Expertise with SLURM and large GPU clusters (A100/H100 class).
- Experience with multilingual / Indian-language dataset creation pipelines.
- Solid foundation in CUDA basics, NCCL, memory optimization.