Tech Lead Manager, AI Evaluations
About the Role
About the Role:
The bottleneck in building reliable and useful AI systems isn’t writing code anymore. It’s knowing what good looks like, applying that knowledge and experience from production usage into building rigorous evals, and using those evals to guide continuous product self-improvement loops.
As the Tech Lead Manager of AI Evaluations, you will lead a team of applied AI scientists and engineers and build the evaluation strategy and platform that helps Glean ship delightful and reliable AI experiences. You will make evaluation a continuous part of how we develop and operate models, harnesses, assistants, agents, and other AI products at Glean.
You’ll guide the team in translating emerging AI/ML research into reliable evaluation systems, build scalable, efficient and reusable eval infrastructure and trusted quality signals, and work across R&D, product, and marketing teams to turn evaluation results into better products and customer outcomes.
You will:
- Own the AI evaluation eng roadmap and technical direction end to end: from defining what success means and building the evaluation platform to integrating results into release and product decisions.
- Build scalable, reusable evaluation infrastructure for datasets, experiments, graders, trace capture and replay, result analysis, and dashboards. Make evaluations reproducible, versioned, cost-aware, and easy for product and engineering teams to use.
- Create and maintain representative evaluation sets from expert-curated examples, appropriately governed production signals, synthetic cases, and historical failures. Cover common workflows, edge cases, long-tail requests, adversarial inputs, and important user and domain slices; protect held-out acceptance sets from tuning leakage.
- Define a balanced quality measurement system for AI products, including correctness, groundedness, retrieval and context relevance, completeness, instruction following, citation quality, user preference, and task success. For agents, evaluate tool selection and arguments, multi-step trajectories, policy compliance, recovery, and final task state, not just the final response.