Scoring named-entity recognition without fooling yourself
Most tokens in running text are outside any entity, so a tagger that predicts O
everywhere can reach 90% token accuracy. The useful measure is entity-level F1: a prediction
counts only when both the span and the type match the reference exactly.
Three ways to count
| Mode | A match requires | Use it for |
|---|---|---|
| Strict | Same boundaries and same type | Headline number, comparison with papers |
| Type only | Overlap and same type | Finding boundary disagreements |
| Boundary only | Same boundaries, any type | Finding confusable classes |
Things that distort the score
- Tagging scheme. Converting between BIO and BIOES before scoring changes nothing if done correctly and quietly breaks things if not.
- Tokenization mismatch. Align predictions to the reference tokens before comparing spans.
- Micro versus macro averaging. Micro F1 is dominated by the most frequent type; report per-type scores next to it.
- Entity overlap between train and test. Score seen and unseen entities separately to see how much is memorised.
I report strict micro F1, a per-type table, and the unseen-entity score. Together they fit in a few lines and answer most follow-up questions in advance.