Google Launches Multimodal Knowledge Agent Benchmark

Google DeepMind has introduced the Multimodal Knowledge Agent benchmark in partnership with Artificial Analysis. The evaluation measures how AI models handle visually complex professional tasks that require cross-referencing blueprints, examining schematics, and producing reports or spreadsheets.

To build the benchmark, domain experts across 10 verticals and 83 occupations created a 200-task evaluation set. Each task replicates a multi-step project that takes a seasoned professional between three and eight hours to complete. The inputs include multimodal files paired with a rubric containing roughly 70 yes/no checks, weighted 75% on factual accuracy and 25% on visual design.

Models run inside an open-source agent harness called Stirrup using uniform tools, sandboxes, and turn budgets. An agentic autorater grades the deliverables by inspecting files, rendering pages, and checking formulas against the expert rubric. According to Google, domain experts agreed with the autorater's grades 96% of the time.

Artificial Analysis keeps the 200-task evaluation set private so tasks cannot leak into training data or search results. A separate set of 10 public tasks is available on Hugging Face for developers to inspect the format and test agents. Google reported that tested frontier models showed extreme efficiency differences on the same tasks, requiring between 30 and 130 turns on average to complete each task.

According to Google, Gemini 4 Carbon achieved second place in quality among tested models and completes tasks at 6x lower cost on average compared to the top-scoring model. The model is available via the Gemini API in Google AI Studio.