How we measure
Last updated
A score on its own says little: a 70 can be excellent or mediocre depending on who it is compared against. This page explains what we compare against, when we publish a relative position, and when we would rather publish none.
The thresholds and the window below are not written on this page: they are read from the same place the calculation reads them. If they change, they change here at the same time.
What a score is
The score is the percentage of questions answered correctly, from zero to a hundred. There is no weighting and no adjustment: it is the count. What can be missing is not the number but the context, which is why the cell that shows it distinguishes five cases rather than two.
- A number
- The assessment was completed and the count exists.
- A zero
- A zero is a score, not an absence. It is shown as a number, with its relative position beside it like any other: someone who got none right still sat the assessment.
- Pending
- The assessment was completed and the calculation is not available yet. This is neither a zero nor an absence.
- No score
- There is no completed assessment, so there is nothing to count. It is drawn as a dash, and anyone using a screen reader hears “no score”, not “dash”.
- Suppressed
- There is a score, but we chose not to publish a relative position, and the reason is visible beside it.
Who the comparison is against
Relative position is calculated against a cohort, and the cohort has three conditions at once. None of them is negotiable per screen: if one fails, there is no comparison.
- The same discipline
- A React assessment is compared with React assessments. Never across disciplines: the data layer does not offer that operation, not even to a screen that asks for it.
- The same scope
- Today the cohort is the whole platform: every completed assessment in that discipline, whoever sent the invitation. No company sees another’s scores; what the cohort uses is their count, not their records.
- The same window
- Only assessments from the last 12 months count. It is a rolling window: an assessment from 13 months ago drops out on its own, without anyone retiring it.
When we publish a number, and when we do not
The fewer assessments a cohort holds, the less can be claimed about it. Rather than always publishing a number and adding a caveat in small print, the calculation changes its answer.
| Assessments in the cohort | What is published |
|---|---|
| fewer than 20 | Nothing. “Sample too small” |
| 20 to 99 | The band only, drawn as a band |
| 100 or more | The full percentile |
The table does two jobs, not one. It avoids claiming a precision the sample does not support, and it stops the aggregate from pointing at anyone: in a small cohort, a relative position says more about one person than about a group. That is why the lower threshold is not a rounding choice that can be relaxed.
The number never travels alone
Wherever a percentile appears, its cohort, its n and its window appear attached to it. This is not a presentation flourish: the value cannot be requested on its own, so no surface — not the dashboard, not the record, not the PDF — can show the number without them.
The extremes are not published: the lowest percentile shown is P1 and the highest is P99. Over a finite sample, “better than nobody” and “better than everybody” are not claims we can stand behind.
When there is no relative position
There are three cases where a score exists and we still publish no comparison.
- Nothing. “Sample too small”
- The table does two jobs, not one. It avoids claiming a precision the sample does not support, and it stops the aggregate from pointing at anyone: in a small cohort, a relative position says more about one person than about a group. That is why the lower threshold is not a rounding choice that can be relaxed.
- Demo assessments
- They do not enter the aggregate, and they are excluded at source: it is not a filter every query has to remember, it is that the cohort never held them. A new query written without knowing the rule cannot be contaminated either.
- Nothing already issued is recalculated
- When a record is issued, its relative-position block is frozen with the date it was calculated. If the cohort grows afterwards, the PDF and the email keep saying what they said: a document whose contents change without notice is not a document.
What we do not publish, and why
We publish the method and who processes the data. There are two things we do not publish, and we would rather name them than let the gap be noticed.
- The exact thresholds of the supervision signals. Publishing them explains how to sit just below them, and whoever would benefit most is precisely whoever is trying to game them.
- Other companies’ scores. The cohort uses an anonymous aggregate count; no account can see, or infer, the result of an assessment it did not send.