N
nanvia muse
Claude AI Senior Engineer w/ Databricks
REMOTE$80 - $90/hrPosted 2w ago
ML EngineerSeniorContract#remote
About the Role
Hi,
We have a 12–18\-month requirement for **Claude AI Senior Engineer w/ Databricks** for the location \- **REMOTE.**
**ONLY GC/USC**
Job Title: **Claude AI Senior Engineer w/ Databricks**
Location: **REMOTE**
Job Type: Contract 12\-18 months (possibilities of further extension)
Hourly Rate: $DOE
**Build Claude applications and agents**
* Implement RAG, document intelligence, and multi\-step agents with the Claude Developer Platform (Messages API, tool use, structured outputs, prompt caching, thinking, citations, Files API, batch processing), the Claude Agent SDK, and Agent Skills.
* Build and integrate MCP servers that expose Unity Catalog tables, Lakebase, and client APIs (FHIR, claims, CRM) as governed tools with least privilege.
* Call Claude natively on Databricks through Foundation Model APIs and AI Gateway (Anthropic Messages API and OpenAI\-compatible endpoints), or through Amazon Bedrock, Google Cloud (Vertex AI), or Microsoft Foundry when the client requires it.
* Write prompts, tool schemas, and system instructions for healthcare tasks such as prior authorization review, clinical note summarization, claims and appeals triage, member and patient service, and medical coding support. Version them in the MLflow prompt registry.
* Use Claude Code for day\-to\-day engineering and codify repeatable workflows as Agent Skills.
**Engineer the Databricks data and AI platform**
* Build Lakeflow pipelines (Connect, Spark Declarative Pipelines, Jobs) in PySpark and SQL that land HL7 v2, FHIR R4, C\-CDA, X12 EDI (837, 835, 270/271, 278\), and EHR extracts into bronze, silver, and gold Delta tables.
* Build PHI de identification, tokenization, and masking steps. Tag and govern PHI with Unity Catalog row filters, column masks, ABAC, lineage, and audit.
* Build and tune Mosaic AI Vector Search indexes, chunking, and embedding pipelines. Deploy agents and models with Model Serving, Agent Framework, Agent Bricks, and Databricks Apps.
* Ship with CI/CD: Databricks Asset Bundles, GitHub Actions or Azure DevOps, unit and integration tests, and promotion across dev, test, and prod workspaces.
**Quality, safety, and operations**
* Build evaluation suites in MLflow 3 (tracing, LLM as judge scorers, golden datasets, regression tests). Gate releases on groundedness, PHI leakage, task success, latency, and cost per transaction.
* Implement guardrails, human\-in\-the\-loop checkpoints, rate limits, retries, fallbacks, and caching. Monitor endpoints with AI Gateway usage tracking, inference tables, and system tables.
* Optimize cost: route across Opus, Sonnet, and Haiku by task, apply prompt caching and batch processing, and right\-size serving endpoints, jobs, and serverless compute.
* Document designs and runbooks, take part in code and prompt reviews, and support production incidents on a rotation.
Must have**Claude and generative AI**
* 4\+ years building LLM applications in production, including 2 years of hands\-on Claude development (Messages API, tool use, structured outputs, prompt caching, thinking, batch processing).
* Hands\-on with the Claude Agent SDK and MCP: building or integrating MCP servers and tools, managing permissions and tool results.
* Practical RAG and agent evaluation experience: retrieval tuning, groundedness checks, test datasets, cost and latency measurement.
* Working knowledge of Claude on Databricks (Foundation Model APIs, AI Gateway) and at least one of Amazon Bedrock, Google Cloud (Vertex AI), or Microsoft Foundry.
**Databricks (deep, hands\-on)**
* 4\+ years of hands\-on Databricks engineering at enterprise scale.
* Expert in PySpark, Spark SQL, and Python. Delta Lake internals: liquid clustering, OPTIMIZE and VACUUM, change data feed, time travel. Streaming with Structured Streaming or Spark Declarative Pipelines.
* Unity Catalog: catalogs, schemas, volumes, row filters, column masks, ABAC, lineage, system tables, Delta Sharing.
* Mosaic AI: Model Serving (Foundation Model APIs, external model
We have a 12–18\-month requirement for **Claude AI Senior Engineer w/ Databricks** for the location \- **REMOTE.**
**ONLY GC/USC**
Job Title: **Claude AI Senior Engineer w/ Databricks**
Location: **REMOTE**
Job Type: Contract 12\-18 months (possibilities of further extension)
Hourly Rate: $DOE
**Build Claude applications and agents**
* Implement RAG, document intelligence, and multi\-step agents with the Claude Developer Platform (Messages API, tool use, structured outputs, prompt caching, thinking, citations, Files API, batch processing), the Claude Agent SDK, and Agent Skills.
* Build and integrate MCP servers that expose Unity Catalog tables, Lakebase, and client APIs (FHIR, claims, CRM) as governed tools with least privilege.
* Call Claude natively on Databricks through Foundation Model APIs and AI Gateway (Anthropic Messages API and OpenAI\-compatible endpoints), or through Amazon Bedrock, Google Cloud (Vertex AI), or Microsoft Foundry when the client requires it.
* Write prompts, tool schemas, and system instructions for healthcare tasks such as prior authorization review, clinical note summarization, claims and appeals triage, member and patient service, and medical coding support. Version them in the MLflow prompt registry.
* Use Claude Code for day\-to\-day engineering and codify repeatable workflows as Agent Skills.
**Engineer the Databricks data and AI platform**
* Build Lakeflow pipelines (Connect, Spark Declarative Pipelines, Jobs) in PySpark and SQL that land HL7 v2, FHIR R4, C\-CDA, X12 EDI (837, 835, 270/271, 278\), and EHR extracts into bronze, silver, and gold Delta tables.
* Build PHI de identification, tokenization, and masking steps. Tag and govern PHI with Unity Catalog row filters, column masks, ABAC, lineage, and audit.
* Build and tune Mosaic AI Vector Search indexes, chunking, and embedding pipelines. Deploy agents and models with Model Serving, Agent Framework, Agent Bricks, and Databricks Apps.
* Ship with CI/CD: Databricks Asset Bundles, GitHub Actions or Azure DevOps, unit and integration tests, and promotion across dev, test, and prod workspaces.
**Quality, safety, and operations**
* Build evaluation suites in MLflow 3 (tracing, LLM as judge scorers, golden datasets, regression tests). Gate releases on groundedness, PHI leakage, task success, latency, and cost per transaction.
* Implement guardrails, human\-in\-the\-loop checkpoints, rate limits, retries, fallbacks, and caching. Monitor endpoints with AI Gateway usage tracking, inference tables, and system tables.
* Optimize cost: route across Opus, Sonnet, and Haiku by task, apply prompt caching and batch processing, and right\-size serving endpoints, jobs, and serverless compute.
* Document designs and runbooks, take part in code and prompt reviews, and support production incidents on a rotation.
Must have**Claude and generative AI**
* 4\+ years building LLM applications in production, including 2 years of hands\-on Claude development (Messages API, tool use, structured outputs, prompt caching, thinking, batch processing).
* Hands\-on with the Claude Agent SDK and MCP: building or integrating MCP servers and tools, managing permissions and tool results.
* Practical RAG and agent evaluation experience: retrieval tuning, groundedness checks, test datasets, cost and latency measurement.
* Working knowledge of Claude on Databricks (Foundation Model APIs, AI Gateway) and at least one of Amazon Bedrock, Google Cloud (Vertex AI), or Microsoft Foundry.
**Databricks (deep, hands\-on)**
* 4\+ years of hands\-on Databricks engineering at enterprise scale.
* Expert in PySpark, Spark SQL, and Python. Delta Lake internals: liquid clustering, OPTIMIZE and VACUUM, change data feed, time travel. Streaming with Structured Streaming or Spark Declarative Pipelines.
* Unity Catalog: catalogs, schemas, volumes, row filters, column masks, ABAC, lineage, system tables, Delta Sharing.
* Mosaic AI: Model Serving (Foundation Model APIs, external model
Similar Jobs
C
Capital One• 1mo ago
Machine Learning Engineer 4 (IC)
McLean, VAFull-time
W
Waystone• 6mo ago
AI Engineer
Leeds, West YorkshireFull-time
A
Aderant• 1mo ago
AI Engineer – Center of Excellence
Atlanta, GAFull-time
S
Signifyd• 1mo ago
Senior Machine Learning Engineer I // II
Chicago, ILFull-time