About Everypixel Research

Who we are

Our company started in 2005 as a production studio making photo, video and audio content at scale. Today the same team builds Everypixel, an AI workspace for generating images and video, and evaluates the models that power it.
New AI models come out almost every week. Most of them are not better for real creative work than the models people already use, and an announcement rarely tells you which ones are. So before a model goes into the product, we test it on the kind of tasks our users bring. Some models go in. Some stay out. This site publishes what we learn along the way, including the results that did not make a model look good.

How we test a model

Every comparison on this site follows the same method. The details that change from study to study, such as which models, how many raters and the dates, are listed in each study's own methodology section.

1. A fixed test set

A series of studies always reuses the same set of sources, so a result from one article can be compared with the next. For image generation, each test scene starts from a real photograph: the prompt is a written description of that photo, and the photo itself serves as the reference. For upscaling, the sources are a balanced mix of real photographs and AI-generated images across content categories such as faces, text and signage, materials, nature and urban scenes.
To compare new models with ones tested months earlier, we keep an anchor ladder: a set of reference models and real photographs whose scores are frozen. A new model is generated on the same prompts, compared against the anchors, and placed on the same scale.

2. Blind, side-by-side votes

Raters see two outputs made from the same prompt or the same source, side by side, with model names hidden. For each quality, they pick left, right or equal. Nothing is pre-selected, there is no time limit, and the side each model appears on is balanced across the panel to cancel position bias.
We use pairwise comparison instead of 1–10 scores for a practical reason. Our first benchmark used absolute scores, and strong models clustered near the top of the scale with no room to tell them apart. People answer "which of these two is better on this one property" far more consistently than they assign a number.

3. Separate qualities, not one overall vote

A single "overall" vote hides tradeoffs. An upscaler can win on detail and lose on naturalness, and a generator can follow the prompt and still look synthetic. Each task type gets its own scales, for example prompt adherence, believability as a photograph and aesthetics for image generation, and raters vote on each one separately.

4. Professional raters in different roles

The raters are members of the Everypixel production team. We describe them by role rather than by name, because each role looks for something different in an image:

Results are reported by role, so you can see whether a conclusion holds for every kind of professional reader or only for one. In published data, raters appear as anonymous IDs with their role kept.
Before a live session, raters go through a calibration round and work from a checklist of observable signs for each scale. No vote is ever thrown out. Votes from a rater who fails a quality check, for example by answering too fast or disagreeing with colleagues at a random-chance level, count at a reduced weight. The rules for this are written down before results are calculated.

5. Statistics that show uncertainty

Every study reports two kinds of numbers for each scale:

Win share: the raw percentage of votes won by each model, with ties reported separately.
Win probability: the chance that a rater prefers one model over the other, estimated with the Bradley-Terry model, a standard statistical model for paired comparisons, with 95% confidence intervals.

Intervals are calculated by resampling whole scenes, because votes on the same scene are not independent. Agreement between raters is measured with Gwet's AC1, which stays informative when votes lean heavily toward one model, a case where the more common Fleiss' kappa underreports agreement.

6. Confidence labels on every result

Each study carries labels on two levels, and the thresholds for them are fixed before any results are calculated:

The whole study is labeled standard or directional, depending on how much the raters agreed. A directional study holds for the test set as a whole, not for single scenes.
Each number is labeled verified, medium or low, depending on how wide its confidence interval is. Low-confidence numbers are not published as findings.

When a scale turns out to be contested, the study says so and the verdict rests on the other scales.

7. AI judges stay separate
Some studies also ask AI models the same pairwise questions. Their votes are stored separately and never mixed into human results. So far they agree with people where people agree with each other and fail where people do not. We wrote about this in Can an AI judge replace human image evaluation?

What we record for every run

For every model run we log the provider and endpoint, the model checkpoint as far as the provider discloses it, the code version of our evaluation tool, checksums of the test set, and for each image its seed, size, latency and cost. Images are only ever scaled down, never upscaled, and all of them are encoded the same way. A model that returns a smaller image than requested is logged as a defect.

What our results do not tell you

What the general public prefers. The raters are one professional production team working in stock, advertising and editorial content.

How a model performs on other content. Confidence intervals cover the scenes in our fixed categories, not other content types, other panels or a fresh batch of generations.

Speed, price and reliability. These are not part of the vote. When a study reports price, it gives the source and the date separately.

Full blinding against photography. Our in-house panel knows that some comparisons include real photographs, which is why images are always judged in pairs on named qualities and never in a "guess the source" format.

How to read a study

Each study opens with a short summary that answers the main question directly. If you are choosing a tool, read the methodology section next: it tells you when the test ran, which model versions were compared and how confident the result is. Model rankings shift with every update, so check the dates.

Data we publish

Every study ships with its raw data: per-pair votes, a summary for each scale, and breakdowns by content category and rater role. Vote and result tables are released under CC BY 4.0. Images are released for research and evaluation use only, because they include licensed reference photographs, Everypixel production images and generated images subject to each vendor's terms.

Independence and disclosure

Nobody pays for a place in our rankings, and model providers do not see results before publication. Everypixel offers several of the models we test in its own product, so we have a commercial interest in the results. Every study discloses which of the tested models Everypixel offers.

Corrections. If you find an error on this page or in any study, write to hello@everypixel.com. Changes to the method are dated and noted in the studies they affect.

Sources

- Bradley, R. A., and Terry, M. E. (1952). Rank analysis of incomplete block designs: I. The method of paired comparisons. *Biometrika*, 39(3/4), 324–345.
- Gwet, K. L. (2008). Computing inter-rater reliability and its variance in the presence of high agreement. *British Journal of Mathematical and Statistical Psychology*, 61(1), 29–48.
- Feinstein, A. R., and Cicchetti, D. V. (1990). High agreement but low kappa: I. The problems of two paradoxes. *Journal of Clinical Epidemiology*, 43(6), 543–549.