What is named entity recognition (NER)?
Named entity recognition (NER) is the task of finding the names of things in text and labeling each with a type, such as person, organization, or location. In "Tim Cook leads Apple from Cupertino", an NER system marks "Tim Cook" as a person, "Apple" as an organization, and "Cupertino" as a location. It is usually framed as sequence labeling: the model assigns one tag to every token.
NER turns unstructured text into structured records. Search engines use it to link queries to knowledge bases, news and finance tools use it to track which companies and people a story mentions, hospitals use it to pull drugs and diagnoses from clinical notes, and privacy tools use it to find and redact personal data. It is also a standard first step for relation extraction and question answering.
Key ideas
- NER finds spans of text that name specific entities and labels each span with a type.
- BIO tags turn span detection into one classification per token.
- Context decides ambiguous cases, such as Apple the company versus apple the fruit.
- Bidirectional models with a CRF layer, and later fine-tuned Transformers, became the standard approach.
- A confidence threshold trades precision against recall.
- Benchmarks score whole entities, so a boundary error counts as both a miss and a false alarm.
Common questions
What is the difference between NER and part-of-speech tagging?
Both assign a label to every token. Part-of-speech tagging labels grammatical roles such as noun and verb for every word, while NER labels only the tokens inside entity names and gives everything else O. A proper noun is often, but not always, part of an entity.
How do you do named entity recognition in Python?
spaCy ships pretrained pipelines that return entities with one call, and Hugging Face Transformers offers a token-classification pipeline with fine-tuned BERT-style models. For a custom entity set, both can be fine-tuned on a few hundred to a few thousand labeled sentences.
Can large language models do NER?
Yes. A prompt that lists the entity types and asks for the spans works reasonably well with no training data, which helps for new domains. Fine-tuned encoder models are still usually more accurate on a fixed type set, faster, and far cheaper to run on large volumes of text.
What is nested NER?
Some entities contain other entities. "Bank of China" is an organization that contains a location, and "University of California, Berkeley" contains two. Flat BIO tagging can only give each token one tag, so nested NER needs span-based or layered models.
Related explainers
- Word2Vec & Word Embeddings — Learn dense vector representations where similar words live near each other in space
- Recurrent Neural Networks — How RNNs read and understand sequences
- LSTMs — Gates and memory cells for long sequences
- BERT & Masked Language Modeling — Mask a word, predict it from both sides — how BERT learns deep bidirectional representations
- Text Classification & Sentiment — Teach a model to assign labels to text — from spam detection to sentiment analysis
- Model Evaluation & Metrics — Precision, recall, F1, ROC curves, and regression metrics
References
- Tjong Kim Sang & De Meulder (2003), "Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition"
- Collobert et al. (2011), "Natural Language Processing (Almost) from Scratch", JMLR
- Huang, Xu & Yu (2015), "Bidirectional LSTM-CRF Models for Sequence Tagging"
- Lample et al. (2016), "Neural Architectures for Named Entity Recognition", NAACL
- Devlin et al. (2019), "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding", NAACL
Last updated