TixelJobs
G
Gnani Innovationsvia Indeed

LLM R&D Specialist - Gen AI

KA, INPosted 8mo ago
NLP / LLMMid Level#pytorch#jax#llm#transformers

About the Role

Engineering

Bengaluru

5 - 8 years experience

Responsibilities

The IndiaAI Mission is building India’s next-generation foundational models. As a Senior ML Engineer, you will design and optimize end-to-end LLM pre-training pipelines, build high-performance distributed training systems, and lead large-scale multilingual data creation/curation efforts. You will work with frameworks like PyTorch Distributed, DeepSpeed, Megatron-LM, and NVIDIA NeMo, and operate large GPU clusters orchestrated via SLURM.


This is a deeply technical, systems-heavy role ideal for engineers who have experience training large transformer models at scale.


Key Responsibilities

1. LLM Pre-Training & Model Engineering

  • Build and optimize end-to-end LLM pre-training pipelines (7B–70B+).
  • Implement and tune training using:
  • PyTorch Distributed
  • DeepSpeed (ZeRO, FSDP)
  • Megatron-LM / Megatron-Core
  • NVIDIA NeMo Framework
  • XLA / JAX (nice to have)
  • Optimize transformer model components: attention kernels, RoPE, KV-cache, tensor/pipeline parallelism, FlashAttention, MoE.
  • Design training recipes, hyperparameters, and scaling strategies.

-

2. Distributed Training & Systems Optimization

  • Manage and schedule large multi-node training jobs using SLURM.
  • Optimize NCCL communication, RDMA/InfiniBand, NVLink/NVSwitch topologies.
  • Diagnose and resolve distributed training failures (hangs, desyncs, GPU OOM, network issues).
  • Implement mixed precision, gradient checkpointing, activation recompute for efficiency.

-

3. Data Creation, Curation & Tokenization

  • Lead large-scale dataset creation pipelines for multilingual + Indian-language data.
  • Build workflows for:
  • Web-scale text extraction (Indian languages & domain data)
  • Data cleaning, deduplication, normalization
  • Quality filtering (heuristics, models, toxicity checks)
  • Multi-stage data labeling & human-in-the-loop QA
  • Implement tokenizers, vocabulary strategy, and data sharding at scale.
  • Build streaming dataloaders optimized for massive throughput.

-

4. Infrastructure Collaboration

  • Work with infra & HPC teams to ensure high GPU utilization.
  • Design checkpointing strategies, artifact storage, and experiment tracking.
  • Build dashboards for training metrics, throughput optimization, and data stats.


Required Skills

  • 5+ years experience in ML engineering / distributed systems / LLM training.
  • Hands-on experience with LLM pre-training (preferred: 7B+ models).
  • Strong expertise with:
  • PyTorch Distributed
  • DeepSpeed (ZeRO, FSDP)
  • Megatron-LM / Megatron-Core
  • NVIDIA NeMo (Megatron + NeMo recipes)
  • Deep understanding of transformer internals: attention, RoPE, positional encodings, tensor/pipeline parallelism, FlashAttention.
  • Expertise with SLURM and large GPU clusters (A100/H100 class).
  • Experience with multilingual / Indian-language dataset creation pipelines.
  • Solid foundation in CUDA basics, NCCL, memory optimization.

Skills Required
Primary Skills
LLM
Applied AI/LLMs: RAG, structured outputs (Pydantic), retrieval eval (MRR/Hit-rate), query rewrite, prompt engineering, LLM orchestration (LangGraph)
Share