My NLP Notes

A checklist for text normalization

2026-05-09

Normalization bugs are dull and expensive: two strings that look identical on screen do not compare equal, and a deduplication step silently keeps both. This is the order I apply.

  1. Decode strictly. Fail on invalid bytes instead of replacing them, so broken sources are noticed.
  2. Unicode form. NFC for text that people will read, NFKC when matching matters more than appearance.
  3. Invisible characters. Remove zero-width spaces, soft hyphens and byte-order marks.
  4. Whitespace. Map non-breaking and other exotic spaces to a plain space, then collapse runs.
  5. Quotes and dashes. Unify only if the downstream model was trained on unified text.
  6. Case. Lowercase last and only where the task allows it; casing carries signal for entities.
import unicodedata, re

def normalize(s: str) -> str:
    s = unicodedata.normalize("NFC", s)
    s = re.sub(r"[\u200b\u200c\u200d\u00ad\ufeff]", "", s)
    s = re.sub(r"\s+", " ", s)
    return s.strip()

Whatever the pipeline does, apply the same function at training and inference time and keep it under test. A dozen awkward strings in a unit test catch most regressions.

← All notes