Technical Director, Data Engineering

Diligent is looking for a Technical Director of Data Engineering to help design and build the next generation of data and AI infrastructure powering our products and internal platforms.

This is not a traditional management role. We are looking for an experienced builder; someone who still enjoys writing code, debugging distributed systems, evaluating frameworks, and getting hands-on with architecture and implementation.

You will work across large-scale ingestion pipelines, search systems, AI/LLM infrastructure, event-driven architectures, vector databases, analytics platforms, and real-time data processing. You should be equally comfortable discussing high-level architecture with senior leadership and diving into a failing Kubernetes pod or optimizing a Spark job.

The ideal candidate has strong opinions informed by real-world experience, understands tradeoffs deeply, and can move quickly without creating unnecessary complexity.

What You’ll Do

• Design and build scalable data platforms and distributed processing systems

• Develop modern ingestion, transformation, and retrieval pipelines for structured and unstructured data

• Build systems supporting AI/LLM applications, semantic search, RAG pipelines, vector search, and agentic workflows

• Work hands-on with engineering teams to implement production-grade solutions rather than producing slideware

• Evaluate and standardize frameworks, tooling, and infrastructure patterns across teams

• Improve performance, reliability, observability, and cost efficiency of data systems

• Partner with product and platform engineering teams to accelerate delivery of AI-native capabilities

• Drive pragmatic engineering decisions balancing speed, maintainability, and operational simplicity

• Mentor engineers technically through design reviews, architecture guidance, and pair debugging

• Help establish engineering standards around CI/CD, testing, data quality, monitoring, and operational excellence

What We’re Looking For

Strong Hands-On Engineering Experience

Candidates should have significant real-world experience building and operating production systems using many of the following:

Data & Distributed Systems

• Airflow

• Elasticsearch / OpenSearch

• Vector databases and semantic retrieval systems

• MongoDB, PostgreSQL, DynamoDB, or similar platforms

Cloud & Infrastructure

• AWS

• AWS CDK

• Serverless architectures

• Distributed observability and monitoring stacks

AI / Search / Modern Data Applications

• LLM integration patterns

• RAG architectures

• Embeddings and vector search

• MCP servers and AI orchestration frameworks

• LangChain, LlamaIndex, DSPy, or similar ecosystems

• AI evaluation, tracing, and observability tooling

• Search relevance and ranking systems

Backend Engineering

• Python strongly preferred

• Experience with Java, Go, or TypeScript is a plus

• API design and distributed service architectures

• Event-driven and asynchronous systems

The Right Candidate

• Still enjoys building and debugging systems directly

• Has strong technical depth, not just architectural vocabulary

• Comfortable operating in ambiguity and fast-moving environments

• Understands how to simplify systems instead of endlessly abstracting them

• Has experience modernizing legacy platforms and evolving architectures incrementally

• Can distinguish between engineering fundamentals and hype cycles

• Values shipping working systems over theoretical perfection

What Success Looks Like

• Engineering teams can move faster because the underlying platforms are reliable and scalable

• AI and data systems become production-grade rather than experimental prototypes

• Infrastructure costs and operational complexity are reduced through better architecture

• Search, ingestion, and retrieval systems improve significantly in performance and relevance

• Teams adopt consistent, maintainable technical patterns without excessive bureaucracy

Nice to Have

• Experience with large-scale news, document, or regulatory data pipelines

• Experience with search relevancy tuning and semantic retrieval

• Exposure to governance, compliance, risk, or enterprise SaaS platforms

• Familiarity with modern AI engineering workflows and agentic systems

• Experience supporting both startup-speed execution and enterprise-scale operations