Freelance Senior Software Engineer | AI Evaluation

Company Intro

At Toloka AI we create data that powers leading GenAI models and innovations. We work with frontier labs, big tech, renowned AI startups, enterprises and non-profit research organizations worldwide. We use a combination of Experts + Crowd + Tech Platform to teach AI models to reason and evaluate their efficacy and safety. We have experts in more than 50 different domains—from doctors and lawyers to physicists and engineers and boast one of the most diverse global crowds, representing over 100 countries and speaking 40+ languages. We are a well-funded startup with an enviable portfolio of clients including Anthropic, Amazon, Microsoft, Poolside, Recraft, and Shopify.

Recently, we secured strategic investment led by Bezos Expeditions and Nebius Group with participation from Mikhail Parakhin, CTO of Shopify and board advisor to leading GenAI companies, who now serves as our Chairman of the Board. Our remote-first team is globally distributed around the world: USA, UK, the Netherlands, Serbia, and more.

About the Role

We're building a dataset to evaluate AI coding agents - how well a model handles real-world developer tasks. Essentially, we're building an olympiad for AI models: the better the tasks, the more precisely we see where models are weak, and the faster they improve.

What you'll actually do

You create tasks and the criteria to evaluate them. Each task consists of:

• An isolated environment - an emulation of a developer's workstation: a Linux machine with dev tools (terminal, CLI), MCP servers (repository, task tracker, messenger, documentation, etc.), and a real web application codebase;
• A task - a prompt, just like in real work (a ticket link, a messenger conversation, anything). The agent figures it out on its own - reads the ticket, checks discussions, searches docs, explores the code. All of that context is also part of the task and needs to be created and thought through;
• Tests and evaluation criteria - how to verify the agent solved the task correctly.

The work breaks down into two phases:

1. Building the virtual company - following a high-level plan, you create a company: codebase, infrastructure, context (conversations, documentation, tickets). The result is a realistic environment with development history.
2. Assembling and calibrating tasks - from intermediate states of the company, task drafts are assembled (based on a taxonomy). You turn a draft into a full task: write the prompt, define evaluation criteria, ensure the task is solvable and the evaluation is fair.

Then an AI agent receives the task and tries to solve it the way a developer would - and you evaluate the result.

What the work looks like in practice:

• Iterating with an AI agent on tests - they should catch real problems, not miss bad solutions, not break on good ones;
• Reviewing code written by agents;
• Analyzing why an agent failed or succeeded;
• Designing edge cases and adversarial scenarios.

A significant part of the work is done together with AI - it's very hard to create tasks that are hard for frontier models without using frontier models.

Why this is hard

1. Frontier models are already good at coding. Creating a task that genuinely challenges the best models is non-trivial - you need to understand where models fail and what reveals the difference between a good and a bad solution.
2. Tasks have many valid solutions. Tests must be neither too strict (breaking valid solutions) nor too lenient (passing bad ones).

Core tech stack

The virtual companies are built around a typical full-stack web application:

• Backend: Python, FastAPI
• Frontend: JavaScript/TypeScript, React
• Infrastructure: Docker, Postgres, Kafka, Redis

You don't need to be an expert in every item, but you should be comfortable reading and reasoning about code across the stack - that's what lets you design realistic tasks, write meaningful tests, and evaluate results.

Growth

You start with task and dataset work, and if it goes well, we expect you to move into a project team - with a broader scope and deeper involvement.

Requirements

• Strong software engineering experience: 8+ years in any modern programming language, with a willingness and ability to work primarily in Python in our projects;
• Hands-on experience using coding agents such as Claude Code and <