Research Engineer Evaluations

You will engineer sophisticated AI agents that can automatically assess the quality and human alignment of our generative design models.

This high-impact role focuses on building the practical systems that make cutting-edge research effective, to provide a rapid feedback loop that guides the future of design generation at Canva, ultimately empowering millions of users to create.

At the moment, this role is focused on:

• Agentic Evaluation Systems: Engineering autonomous AI agents that use Multimodal Large Language Models (MLLMs) to evaluate the quality, relevance, and human alignment of generated designs.
• Inference-Time Alignment: Mastering techniques that improve model outputs without full retraining, but by inference-based methods including prompt engineering, in-context learning and Retrieval-Augmented Generation (RAG).
• Model Benchmarking & Analysis: Building a rigorous framework to systematically benchmark internal and external quality understanding models, delivering clear, data-driven insights on human alignment.

Primary Responsibilities:

• Design, build, and optimize the infrastructure for an 'MLLM-as-a-Judge' evaluation system for scalable, automated feedback.
• Implement and experiment with inference-time alignment techniques (Prompt Engineering, RAG, ICL) to directly improve model output quality.
• Establish and manage a comprehensive benchmarking process to compare various foundation models on design-centric tasks.
• Analyze evaluation data to identify model failure modes and provide actionable recommendations to the research team.
• Collaborate with research scientists and ML engineers to integrate the agentic judge system into the model development lifecycle.
• Translate the latest research in LLM evaluation and agentic AI into practical, production-ready engineering solutions.