Y
yalvia muse
Senior ML Engineer - Inference and Model Serving
TS, INPosted 3w ago
ML EngineerSeniorFull-time
About the Role
**Role overview**
We are hiring a senior ML engineer to own how our language and speech models run in production. Our
agents depend on fast, reliable inference: every spoken reply passes through speech recognition, a
language model and speech synthesis within a tight latency budget. You will serve these models on
GPUs at the lowest cost per call without losing quality, build the security and metering layers around
them, and automate how models move from registry to production. You will work with the architect on
model\-serving interfaces and with the AI engineers who build agents on top. **Key responsibilities** **Language model serving*** Serve open\-weight language models with engines such as vLLM, TensorRT\-LLM or SGLang, tuned
for both time to first token and throughput.* Apply continuous batching, prefix caching, speculative decoding and multi\-LoRA serving to cut
latency and cost per request.* Manage GPU memory with PagedAttention, FlashAttention, KV\-cache quantisation and tensor
or pipeline parallelism, to maximise batch size and context length without out\-of\-memory
failures.* Quantise models (INT8, INT4, FP8\) and choose precision per model based on measured quality
and speed.
Speech model serving* Serve streaming speech recognition and speech synthesis models on GPUs, with low time to first
partial transcript and time to first audio.* Batch and schedule many concurrent audio streams on shared GPUs without letting one stream
delay another.* Work with AI engineers to optimise end\-to\-end latency of the speech pipeline, from audio
chunking to model execution to streaming output.* Place speech and language models on GPUs to balance latency, utilisation and cost when they
run side by side. **Performance and quality*** Profile models and serving stacks with tools such as NVIDIA Nsight Systems and Nsight Compute,
then find and fix bottlenecks.* Use compilers and runtimes such as TensorRT, TVM or PyTorch Inductor to fuse operations and
reduce memory traffic.* Check every optimisation against quality evals — task accuracy, word error rate and voice
quality — and block any change that makes a model faster but worse.* Benchmark candidate models and hardware on latency, throughput, quality and cost, and
recommend what to serve. **Scale, cost and deployment*** Autoscale GPU capacity with traffic, plan capacity for peaks, and track cost per request, per call
and per tenant.* Deploy to cloud, on\-premises and restricted\-network environments, including air\-gapped installs
for enterprise clients.* Build fallback paths across model sizes, replicas and providers, so traffic keeps flowing when a
model or node fails. **Access, metering and MLOps*** Design secure model endpoints with OAuth2, JWT and role\-based access control, and enforce
strict multi\-tenant isolation.* Build high\-throughput metering for input and output tokens, audio seconds and GPU time,
feeding rate limits, billing and cost allocation.* Build CI/CD pipelines that profile, regression\-test and deploy models from registries such as
MLflow or Hugging Face Hub into Kubernetes.* Monitor latency, throughput, GPU utilisation and errors with OpenTelemetry and Prometheus,
and respond to production incidents. **Required qualifications and skills*** Strong Python and working C\+\+ skills.
* Hands\-on experience serving language models in production with engines such as vLLM,
TensorRT\-LLM, SGLang or Triton Inference Server.* Experience serving or optimising speech or other streaming models under strict real\-time
latency budgets.* Solid understanding of GPU architecture — streaming multiprocessors, memory hierarchy,
asynchronous copies — and how it drives inference performance.* Practical experience with quantisation, batching strategies, KV\-cache management and model
parallelism.* Experience profiling and fixing performance problems with tools such as Nsight Systems or
PyTorch Profiler.* Experience running GPU workloads
We are hiring a senior ML engineer to own how our language and speech models run in production. Our
agents depend on fast, reliable inference: every spoken reply passes through speech recognition, a
language model and speech synthesis within a tight latency budget. You will serve these models on
GPUs at the lowest cost per call without losing quality, build the security and metering layers around
them, and automate how models move from registry to production. You will work with the architect on
model\-serving interfaces and with the AI engineers who build agents on top. **Key responsibilities** **Language model serving*** Serve open\-weight language models with engines such as vLLM, TensorRT\-LLM or SGLang, tuned
for both time to first token and throughput.* Apply continuous batching, prefix caching, speculative decoding and multi\-LoRA serving to cut
latency and cost per request.* Manage GPU memory with PagedAttention, FlashAttention, KV\-cache quantisation and tensor
or pipeline parallelism, to maximise batch size and context length without out\-of\-memory
failures.* Quantise models (INT8, INT4, FP8\) and choose precision per model based on measured quality
and speed.
Speech model serving* Serve streaming speech recognition and speech synthesis models on GPUs, with low time to first
partial transcript and time to first audio.* Batch and schedule many concurrent audio streams on shared GPUs without letting one stream
delay another.* Work with AI engineers to optimise end\-to\-end latency of the speech pipeline, from audio
chunking to model execution to streaming output.* Place speech and language models on GPUs to balance latency, utilisation and cost when they
run side by side. **Performance and quality*** Profile models and serving stacks with tools such as NVIDIA Nsight Systems and Nsight Compute,
then find and fix bottlenecks.* Use compilers and runtimes such as TensorRT, TVM or PyTorch Inductor to fuse operations and
reduce memory traffic.* Check every optimisation against quality evals — task accuracy, word error rate and voice
quality — and block any change that makes a model faster but worse.* Benchmark candidate models and hardware on latency, throughput, quality and cost, and
recommend what to serve. **Scale, cost and deployment*** Autoscale GPU capacity with traffic, plan capacity for peaks, and track cost per request, per call
and per tenant.* Deploy to cloud, on\-premises and restricted\-network environments, including air\-gapped installs
for enterprise clients.* Build fallback paths across model sizes, replicas and providers, so traffic keeps flowing when a
model or node fails. **Access, metering and MLOps*** Design secure model endpoints with OAuth2, JWT and role\-based access control, and enforce
strict multi\-tenant isolation.* Build high\-throughput metering for input and output tokens, audio seconds and GPU time,
feeding rate limits, billing and cost allocation.* Build CI/CD pipelines that profile, regression\-test and deploy models from registries such as
MLflow or Hugging Face Hub into Kubernetes.* Monitor latency, throughput, GPU utilisation and errors with OpenTelemetry and Prometheus,
and respond to production incidents. **Required qualifications and skills*** Strong Python and working C\+\+ skills.
* Hands\-on experience serving language models in production with engines such as vLLM,
TensorRT\-LLM, SGLang or Triton Inference Server.* Experience serving or optimising speech or other streaming models under strict real\-time
latency budgets.* Solid understanding of GPU architecture — streaming multiprocessors, memory hierarchy,
asynchronous copies — and how it drives inference performance.* Practical experience with quantisation, batching strategies, KV\-cache management and model
parallelism.* Experience profiling and fixing performance problems with tools such as Nsight Systems or
PyTorch Profiler.* Experience running GPU workloads
Similar Jobs
C
Capital One• 1mo ago
Machine Learning Engineer 4 (IC)
McLean, VAFull-time
W
Waystone• 6mo ago
AI Engineer
Leeds, West YorkshireFull-time
A
Aderant• 1mo ago
AI Engineer – Center of Excellence
Atlanta, GAFull-time
S
Signifyd• 1mo ago
Senior Machine Learning Engineer I // II
Chicago, ILFull-time