NER Annotation: How to Build High-Quality Named Entity Recognition Datasets

Named Entity Recognition is one of the most critical NLP tasks — and its quality depends entirely on the annotation. This guide covers how to design NER annotation guidelines, manage entity boundary disputes, handle multilingual text, and measure inter-annotator agreement.

NER Annotation: How to Build High Quality Named Entity Recognition Datasets Named Entity Recognition is one of the most critical NLP tasks — and its quality depends entirely on the annotation. What Is NER and Why Does It Matter? NER identifies and classifies named entities in text — people, organisations, locations, dates, monetary values, and more. It's foundational for information extraction, question answering, and document understanding systems. Common Entity Types: PERSON, ORG, LOC, DATE, AMOUNT, PRODUCT Each entity type requires precise definition in your annotation guidelines to ensure consistency across annotators. Designing NER Annotation Guidelines: Boundary Rules, Nested Entities, Ambiguity The most common source of annotation error is ambiguity in guidelines — particularly around entity boundaries and nested entities. Clear examples and decision trees are essential. Multilingual NER Challenges: Code Switching, Script Changes, Transliteration Multilingual NER is significantly harder than single language annotation. Code switching (e.g., Hindi English sentences) and transliteration require specialised annotator training. Measuring Quality: IAA Scores and Adjudication Workflows Cohen's Kappa and F1 based IAA scores measure annotator agreement on entity spans and types. Disagreements must be adjudicated by senior reviewers, not ignored. Delivery Formats: CoNLL vs JSONL vs Custom Schema CoNLL format: Industry standard for sequence labeling tasks JSONL: Flexible, machine readable, easy to process programmatically Custom schema: Sometimes required for complex nested entity structures Case Study: eQOURSE NER Annotation for a FinTech Chatbot We annotated 500,000+ customer service utterances with 12 entity types for a FinTech NLU system, achieving 0.94 Cohen's Kappa. Build your NER dataset with eQOURSE