How to judge AI model strength: objective measures and human preference
Understand task success, end-to-end time and blinded preference in strength scores, with cost compared separately.
A model can be strong at one task and weak at another. Accurate answers, useful video, quick responses and predictable cost call for different evidence. Define the workload first, choose measures second, and apply weights only after the results are comparable.
Objective measures that can be checked
- Task success or accuracy: give models the same tasks and score them against answers or acceptance rules fixed in advance. Text tasks may check code or tool completion; speech tasks may check pronunciation of specified words.
- Failure rate and consistency: count failed requests, missing output and results that miss the specification. Record whether retries are included.
- End-to-end time: measure from submission until the complete result is available, including queues, generation, encoding and download. Time to first token answers a different question.
- Full task cost: use actual billable quantities for every charge in the same workload. A per-token input rate and a per-second video rate are not equivalent prices.
Category-specific tests can add fixed checks for object counts, placement and actions in images or videos; temporal flicker and subject consistency in video; and transcription error on generated speech using a fixed recognizer with human spot checks. These are proxy measures: their numbers can be reproduced, but they do not establish visual appeal or genuine intelligibility by themselves. The VBench paper separates temporal quality and subject consistency while noting the gap between automatic metrics and human perception. Microsoft’s speech evaluation note describes a word-level intelligibility measure checked by listeners.
Context length, supported resolution, frame rate and duration are also verifiable specifications. They describe available operating ranges, not performance on a shared task. Supporting 4K alone cannot be converted into a quality score.
Even objective measures depend on test conditions. The Artificial Analysis intelligence methodology publishes task sets and scoring rules. Its video methodology defines output settings and what generation time includes. Cite the benchmark version, model effort level and settings alongside a number.
Where human judgment belongs
Visual appeal, voice naturalness and clarity often require people to judge samples. Demo videos show examples, while user reports can reveal recurring problems. Both can be affected by cherry-picked samples, prompts and personal taste.
Blinded comparisons are more systematic: evaluators choose between outputs for the same prompt without seeing model identities. The image and text-to-speech benchmark methods explain this preference approach. Elo is a relative measure of human preference, not a task accuracy percentage or a 0–100 strength score. Overlapping confidence intervals also limit claims about which model is better.
A test video can carry both types of evidence. Whether a viewer likes the image is subjective; duration, resolution, request failures and an object count checked against a rule fixed beforehand can be verified. A user saying “this feels better” reports a preference. A report with request logs and reproduction steps can point to an objective failure, pending verification.
Four checks before weighting
- Same workload: compare text, image, video and speech separately. Text-to-video and image-to-video may need separate groups too.
- Same conditions: record model version, effort level, resolution, duration, language, prompt and provider.
- Independent dimensions: do not count one preference Elo twice under different labels such as visual quality and prompt adherence.
- Missing evidence: if failure rate, objective task outcomes or blinded preference data are absent, mark the model unranked rather than filling gaps with provider claims. Missing cost belongs in the separate pricing comparison.
Our four category rankings directly order listed models by an original metric from a named public benchmark, with the tested variant, source and check date. This is an order among models listed here, not the benchmark provider’s global rank. Each category page also shows proposed composite dimensions and weights. Weights are editorial choices, not natural constants. Raw results need a published conversion to 0–100 before weighting, and one source result cannot count twice. Price is excluded from strength and shown separately in the API pricing table; an API rate is not necessarily the full task cost. The available evidence is not a complete shared-task evaluation, so we have not published a composite strength score.
The formula is strength = Σ (dimension score on a 0–100 scale × dimension weight ÷ 100). For example, 80 points in a 30% dimension contributes 24 points. This illustrates the arithmetic; it is not a measured model result. Conversion scales, task lists, repeat counts and blinded reviewer numbers must be fixed and published before real results are entered.
What would qualify as an independent assessment?
Before publishing our own composite score, we need to publish the protocol rather than select tasks or adjust weights after seeing results. For each workload, fix prompts, source assets, model versions, endpoints, generation settings and rules for failures and retries. Run every model on shared tasks, retain raw outputs and request records, score objective checks against rules fixed in advance, and include failed requests in reliability denominators. Hide model identities, randomize presentation order and use multiple independent reviewers for preference judgments.
Set sample size against the desired precision, and report task counts, repeat counts, reviewer counts and uncertainty intervals. Overlapping intervals permit an order by point estimate, but do not establish a certain winner. Automatic video dimensions can draw on VBench, with human judgments checked separately. Elo for image and video is a relative preference measure within one arena, not an absolute quality percentage. Artificial Analysis’s video methodology also separates audio-enabled, silent and different generation tasks into distinct boards. Our current budget and samples do not support this independent assessment, so composite strength scores remain blank.