TixelJobs
K
Komodohealthvia Greenhouse

Staff Applied AI Engineer (Observability)

New York, NYPosted 6mo ago
ML EngineerStaff+Full-time

About the Role

We Breathe Life Into Data

At Komodo Health, our mission is to reduce the global burden of disease. And we believe that smarter use of data is essential to this mission. That’s why we built the Healthcare Map — the industry’s largest, most complete, precise view of the U.S. healthcare system — by combining de-identified, real-world patient data with innovative algorithms and decades of clinical experience. The Healthcare Map serves as our foundation for a powerful suite of software applications, helping us answer healthcare’s most complex questions for our partners. Across the healthcare ecosystem, we’re helping our clients unlock critical insights to track detailed patient behaviors and treatment patterns, identify gaps in care, address unmet patient needs, and reduce the global burden of disease. 

As we pursue these goals, it remains essential to us that we stay grounded in our values: be awesome, seek growth, deliver “wow,” and enjoy the ride. At Komodo, you will be joining a team of ambitious, supportive Dragons with diverse backgrounds but a shared passion to deliver on our mission to reduce the burden of disease — and enjoy the journey along the way.

The Opportunity at Komodo Health: 

Healthcare in the U.S. is fragmented and inefficient. Komodo Health is working to fix that—with data.

By building the most complete view of patient journeys across the U.S., Komodo enables life sciences companies, payers, and providers to make better decisions that directly improve patient outcomes.To unlock the full potential of this data, Komodo is investing in AI-native infrastructure—systems that make AI reliable, scalable, and deeply embedded into how products are built and used.

Mission of the Role:

We’re hiring Staff Applied AI Engineers to join a newly formed team focused on building the foundation that makes AI systems trustworthy at scale. This is a greenfield, backend-heavy systems role where you’ll define and build the core infrastructure that powers AI across the company. Your entry point is AI observability, evaluation, and reliability—but that is not the destination.

You’ll start by building the systems that make AI measurable and debuggable. Then you’ll expand into broader platform ownership: agent architectures, orchestration systems, and company-wide AI standards.

Looking back on your first 12 months at Komodo Health, you will have accomplished… 

  • Establish the observability foundation for AI systems across Komodo (logging, tracing, evaluation pipelines, feedback loops)
  • Define how the organization measures LLM performance and reliability (failure modes, hallucinations, drift, system degradation)
  • Build reusable infrastructure and frameworks adopted across multiple teams
  • Set standards for AI system quality, monitoring, and production readiness
  • Influence technical direction and platform decisions across the AI org
  • Expand into broader AI platform ownership (multi-agent systems, orchestration layers, shared services)
  • Mentor engineers in AI system design, debugging, and evaluation

What You’ll Own

AI Observability & Reliability Foundation (Initial Focus)

  • Design and build systems for:
    • Logging, tracing, and request visibility
    • Evaluation pipelines and benchmarking frameworks
    • Feedback loops for continuous system improvement
  • Develop approaches to detect hallucinations, model drift, performance regressions and system failures
  • Establish organization-wide standards for AI monitoring and evaluation

AI Platform Expansion (Ongoing Scope)

  • Build foundational AI systems: agent frameworks, orchestration and tool-calling systems, and shared infrastructure for LLM-powered applications
  • Contribute to company-wide AI architecture and standards
  • Make high-impact build vs. buy decisions for AI tooling and platforms
  • Partner cross-functionally to ensure systems are scalable, reliable, secure and production-ready

What you bring to Komodo Health (required):

  • Strong experience building production AI systems (not just prototypes)
  • Deep expertise in LLMs and applied AI systems: tool/function calling, agent orchestration and prompt systems
  • Hands-on experience with:
    • AI observability, evaluation, or reliability systems
    • Monitoring, debugging, and performance analysis in production
  • Strong backend engineering skills: Python, APIs, distributed systems, or platform architecture
  • Proven ability to operate in ambiguous, undefined environments
  • Track record of owning systems end-to-end and driving technical direction
  • You will operate as a strategic AI leader—setting patterns, driving adoption, and ensuring Komodo stays at the forefront of applied AI innovation.

Additional skills and experience we’d prioritize (nice to have)…

  • Experience building internal observability platforms and valuation frameworks for LLM systems.
  • Familiarity with: Request tracing, replay systems, or model monitoring
  • Experience leading large-scale distributed data and compute systems (e.g., Spark, Databricks, Snowflake).
  • Healthcare data experience (not required)

** Location flexible to NYC or SF hybrid, and remote

The pay range for each

Share