Barometer
Speed, reliability and preferences of the chat models, measured on real requests. No made-up score: rankings wait until there are enough votes.
How we measure
Time to the first word
The time between sending the request and the first word of the answer. We give the median over 7 days: half of the answers arrive faster.
Writing speed
The number of tokens written per second, once the answer has started. A token is a piece of a word, about 4 letters.
Success rate
The share of requests that got an answer, over 7 days. A model shows from 20 measured requests.
Preferences
They only come from blind duels: you vote without knowing the models. Duels where you choose the models do not count.
Preference score
The Bradley-Terry method turns duel wins into a score, with a 95% interval that shows the margin of uncertainty. It shows from 100 votes.
Anonymous data
We keep figures: duration, number of tokens, success, type of task. Never the text of requests or answers, nor who sent them.
Results
Overall quality of answers
To compare the quality of answers over a large number of votes, see compar:IA, the French government’s public comparison arena. Its datasets can be reused under the Licence Ouverte Etalab 2.0. aidock does not try to replace it.
Open compar:IA (new tab)Every vote makes the Barometer more reliable.
Two models answer the same request. You vote without knowing their names, then they show.