MMBU Challenge / Stanford MARVL / field report, updated 4 Oct 2026

Reading the scan is not the same as seeing it.

The MMBU Challenge asks a vision-language model an open question about a biomedical image, then asks it again, separately, what the image even is. A model that has learned to sound medical without looking gets found out on the second pass. This report starts from zero and ends with a plan for the two tracks we are entering.

Pass 2 asks for the context

Modality
Submodality
Specimen
Body part
Stain
Domain

35 submodalities
hidden test set

A drawn specimen, not patient data. Pass 1 scores the answer to the question (T). Pass 2 scores whether the model can name what it is looking at (C).

How a submission is scored

base score = 0.8 × T + 0.2 × C

Task answer, T 60 Context, C 60

60.0 48.0 from T, 12.0 from C

Move the sliders: these are made-up inputs, not results. An automated judge model grades both passes, and ties break on T.

Track 1 Open Frontier

No size cap. Pretraining, mid-training and post-training must come from the same organisation.
Track 2 Domain Adaptation Our focus

Adapt a disclosed base model toward medicine. The base score is multiplied by how far the adapted model beats its own base: (0.8T + 0.2C) × (adapted − base accuracy).
Track 3 Efficient Inference Our focus

At most 4B active parameters and 12B in total. The base score is scaled by an efficiency bonus. The challenge brief sets it at (P_max / P)^0.1, though the website still lists it as to be announced.