U
United Softwvia muse
Senior ML Systems Engineer, LLM Inference Performance on AWS Trainium
REMOTEPosted 2w ago
ML EngineerSeniorContract#remote
About the Role
**Role Name: Sr. ML Systems Engineer, LLM Inference Performance on AWS Trainium**
**Location: Canada (Remote)**
**Role overview**
· We are seeking a hands\-on machine learning systems engineer to optimize production\-grade large language model inference across current and next\-generation AWS Trainium platforms, including Trainium2 and Trainium3\.
· The work focuses on decode latency, throughput under representative load, price\-performance, long\-context and multi\-chip scaling, and reproducible benchmarking against leading GPU platforms.
· As part of a small, senior engineering team, you will own substantial workstreams end to end, from model\-level optimization through measurement, analysis, and documentation. You will communicate technical findings and business implications clearly to customer engineers and leadership.
· This is a hands\-on, customer\-facing role focused on delivering measurable production outcomes.
**Key responsibilities**
· Lead decode optimization. Improve decode latency, throughput, and cost on AWS Trainium platforms, validating correctness at every step. Initial priorities include speculative decoding and native multi\-token prediction (MTP) for Qwen\-family models in vLLM on AWS Neuron.
· Benchmark and assess performance rigorously. Design and run matched, reproducible tests against leading GPU platforms, controlling prompts, concurrency, cache state, and warm\-up. Deliver latency\-throughput Pareto frontiers and price\-performance comparisons aligned with customer service\-level objectives.
· Lead cross\-generation optimization. Benchmark and migrate workloads across Trainium generations, including Trainium2 and Trainium3\. Scale serving on multi\-chip Trainium systems for long\-context workloads, including contexts of 100K tokens or more, and apply architecture\-specific optimizations.
· Extend model coverage. Onboard additional architectures, including hybrid state\-space and mixture\-of\-experts models, reusing and generalizing existing implementations where practical.
· Enable customer ownership. Create runbooks, implementation guides, and benchmark documentation, and support hands\-on workshops so customer engineers can operate, reproduce, and tune the stack independently.
**What success looks like**
· Decode latency and serving cost improve measurably while maintaining model quality.
· Performance results are reproducible, correctly scoped, and defensible across hardware platforms.
· Final benchmarks reflect representative customer workloads, not synthetic peak numbers alone.
· Customer engineers can reproduce the environment, run the benchmarks, and continue optimization independently.
**Basic qualifications**
· 5\+ years of experience in machine learning or systems engineering, including 2\+ years focused on production model inference or serving.
· Strong proficiency in Python and PyTorch, with the ability to read and modify model implementations involving attention, mixture\-of\-experts routing, and KV\-cache management.
· Hands\-on experience with an LLM serving framework such as vLLM, TensorRT\-LLM, SGLang, or Triton Inference Server.
· Deep understanding of LLM inference performance, including prefill and decode, continuous batching, and tensor and expert parallelism, and their impact on latency, throughput, and cost.
· Experience profiling and optimizing accelerator workloads at the framework, compiler, or kernel level, and designing rigorous performance experiments.
· Strong written and verbal communication skills, with the ability to explain engineering trade\-offs to technical audiences and business impact to senior stakeholders.
**Preferred qualifications**
· Experience with AWS Trainium or Inferentia and the AWS Neuron SDK, including NxD Inference, vLLM on Neuron, or Neuron Kernel Interface (NKI), or with migrating workloads across accelerator generations.
· Experience implementing or evaluating speculative decoding, such as draft\-model approaches, Medus
**Location: Canada (Remote)**
**Role overview**
· We are seeking a hands\-on machine learning systems engineer to optimize production\-grade large language model inference across current and next\-generation AWS Trainium platforms, including Trainium2 and Trainium3\.
· The work focuses on decode latency, throughput under representative load, price\-performance, long\-context and multi\-chip scaling, and reproducible benchmarking against leading GPU platforms.
· As part of a small, senior engineering team, you will own substantial workstreams end to end, from model\-level optimization through measurement, analysis, and documentation. You will communicate technical findings and business implications clearly to customer engineers and leadership.
· This is a hands\-on, customer\-facing role focused on delivering measurable production outcomes.
**Key responsibilities**
· Lead decode optimization. Improve decode latency, throughput, and cost on AWS Trainium platforms, validating correctness at every step. Initial priorities include speculative decoding and native multi\-token prediction (MTP) for Qwen\-family models in vLLM on AWS Neuron.
· Benchmark and assess performance rigorously. Design and run matched, reproducible tests against leading GPU platforms, controlling prompts, concurrency, cache state, and warm\-up. Deliver latency\-throughput Pareto frontiers and price\-performance comparisons aligned with customer service\-level objectives.
· Lead cross\-generation optimization. Benchmark and migrate workloads across Trainium generations, including Trainium2 and Trainium3\. Scale serving on multi\-chip Trainium systems for long\-context workloads, including contexts of 100K tokens or more, and apply architecture\-specific optimizations.
· Extend model coverage. Onboard additional architectures, including hybrid state\-space and mixture\-of\-experts models, reusing and generalizing existing implementations where practical.
· Enable customer ownership. Create runbooks, implementation guides, and benchmark documentation, and support hands\-on workshops so customer engineers can operate, reproduce, and tune the stack independently.
**What success looks like**
· Decode latency and serving cost improve measurably while maintaining model quality.
· Performance results are reproducible, correctly scoped, and defensible across hardware platforms.
· Final benchmarks reflect representative customer workloads, not synthetic peak numbers alone.
· Customer engineers can reproduce the environment, run the benchmarks, and continue optimization independently.
**Basic qualifications**
· 5\+ years of experience in machine learning or systems engineering, including 2\+ years focused on production model inference or serving.
· Strong proficiency in Python and PyTorch, with the ability to read and modify model implementations involving attention, mixture\-of\-experts routing, and KV\-cache management.
· Hands\-on experience with an LLM serving framework such as vLLM, TensorRT\-LLM, SGLang, or Triton Inference Server.
· Deep understanding of LLM inference performance, including prefill and decode, continuous batching, and tensor and expert parallelism, and their impact on latency, throughput, and cost.
· Experience profiling and optimizing accelerator workloads at the framework, compiler, or kernel level, and designing rigorous performance experiments.
· Strong written and verbal communication skills, with the ability to explain engineering trade\-offs to technical audiences and business impact to senior stakeholders.
**Preferred qualifications**
· Experience with AWS Trainium or Inferentia and the AWS Neuron SDK, including NxD Inference, vLLM on Neuron, or Neuron Kernel Interface (NKI), or with migrating workloads across accelerator generations.
· Experience implementing or evaluating speculative decoding, such as draft\-model approaches, Medus
Similar Jobs
C
Capital One• 1mo ago
Machine Learning Engineer 4 (IC)
McLean, VAFull-time
W
Waystone• 6mo ago
AI Engineer
Leeds, West YorkshireFull-time
A
Aderant• 1mo ago
AI Engineer – Center of Excellence
Atlanta, GAFull-time
S
Signifyd• 1mo ago
Senior Machine Learning Engineer I // II
Chicago, ILFull-time