Evaluation Engineer, Applied AI

ABOUT MISTRAL Mistral provides full-stack AI solutions: from frontier models to developer tools, applications, and compute. We partner with enterprises tackling the hardest problems across high-stakes industries like finance, manufacturing, defense, healthcare, and the public sector, co-creating customized AI systems that they can run on their terms. We are a dynamic, collaborative team passionate about AI and its potential to transform society. Our diverse workforce thrives in competitive environments and is committed to driving innovation. Our teams are distributed between Europe, North America, Asia and the Middle East. We are creative, low-ego and team-spirited. THE ROLE This is a chance to build the frontier LLM evals that shape the next generations of flagship models. Could a model design a computer chip, or find the backdoor an attacker left in a network? Public benchmarks can't tell us, as they're generally not a good measure of real work. As an Evaluation Engineer, you'll own evals from start to finish. You'll run workshops with customers, industry experts and our customer-facing teams to understand their most important industry use cases (e.g., an engineer updating a part in CAD software), and help decide which are worth building first. You'll also look beyond today's use cases and design moonshot evals for work no model can do yet, so Mistral gets there first. Then you'll turn each one into a benchmark, often one where an agent has to do the work inside real software. You'll source the data, decide how to score it, and run it on every new Mistral model and on competitor models. When our models fall short, you'll work with the post-training teams to close the gap. When they catch up, you'll make the eval harder. You'll write up and present your results so senior leaders can act on them. They also give our go-to-market teams evidence of where Mistral leads. You'll learn new fields quickly. One month you might be learning how chip designers check