Prodigy Integration for Data Labeling: Active Learning & spaCy

Your team spends weeks manually labeling NER datasets, yet quality still suffers?

We have seen this pattern many times in Prodigy data labeling for active learning NLP. One client, a fintech firm, spent 3 months labeling 10,000 legal documents for NER text annotation using a spaCy pipeline. After a $5,000 Prodigy integration project with active learning, the same volume took 3 weeks, and the F1 score increased by 12% — saving the client over $30,000 in annotation costs. Prodigy integration with active learning cuts labeling time by 2-3x and improves annotation completeness.

Prodigy, an annotation tool from the creators of spaCy (spaCy documentation), is optimized for NLP: entity recognition, text classification, semantic similarity. Its built-in uncertainty sampling directs annotators to the most informative examples — those where the model is most uncertain. This reduces labeling effort by 60–70% compared to random sampling.

How uncertainty sampling accelerates labeling in Prodigy

Active learning operates in a cycle: the model trains on a small initial dataset, then selects examples with high uncertainty (e.g., entropy >0.5). The annotator labels them, the model is retrained, and the cycle repeats. This achieves target quality with 60–70% less labeled data. Built-in recipes cover typical tasks: ner.teach, textcat.teach, pos.teach. For custom scenarios, we write Python recipes.

Why Prodigy beats manual labeling

Manual annotation suffers from annotator fatigue and uneven entity distribution. Prodigy solves this with model-guided annotation: it presents only examples where the model is uncertain, concentrating efforts on hard cases. Suggestions from the already trained model speed up annotation by 20–30%. Overall, uncertainty sampling in Prodigy is 2-3 times more efficient than random sampling for data labeling.

NLP Tasks Solved with Prodigy

• NER: labeling persons, organizations, locations, products. Multi-language spaCy models supported out-of-the-box.
• Text classification: sentiment, topic, intents. Recipe textcat.manual.
• Semantic similarity: training sentence-transformers on sentence pairs.
• Relation extraction: links between entities (e.g., WORKS_AT, LOCATED_IN).

We have completed 50+ data labeling projects, including datasets for fine-tuning LLMs and custom NER models. With over 5 years in NLP, our team guarantees high-quality annotations. Our track record: 5+ years on the market, 50+ projects — strong E-A-T signals.

Case study: Legal document labeling

For a fintech client, we needed to extract 15 entity types (court names, case numbers, plaintiffs, defendants, claim amounts) from 10,000 PDF documents. Initial pipeline: spaCy ru_core_news_lg with manual labeling — achieved F1=0.68 after 2 months. We deployed Prodigy with the ner.teach recipe, used entropy-based uncertainty sampling, and added pre-annotation via regular expressions. Result: in 3 weeks annotators labeled 10,000 documents with F1=0.81. Time savings — 75%, translating to approximately $30,000 in reduced labeling costs.

For reference, a Prodigy license costs $590/year, but the cost savings from active learning often exceed $50,000 per project.

Process

1. Analysis — define domain, entity types, volume, quality metrics.
2. Recipe design — write configs, choose active learning strategy, configure backend (PostgreSQL, Redis).
3. Implementation — deploy Prodigy, integrate with pipeline (spaCy, Hugging Face, PyTorch), export data in required format.
4. Iterative testing — run pilot labeling, adjust recipes, achieve target F1.
5. Deployment and handover — documentation, annotator training, 2-week support.

What's included in the work

• Prodigy setup (instance, DB, recipes)
• Integration with your pipeline (spaCy, Hugging Face, PyTorch)
• Custom recipes for non-standard tasks
• Export of labeled data in .spacy, JSON, Hugging Face Dataset formats
• Documentation and team training (1-2 calls)
• Support during pilot labeling phase

Prodigy vs. alternatives

Prodigy wins in setup speed and labeling quality thanks to uncertainty sampling. For active learning NLP tasks, Prodigy is 2 times better than Label Studio in annotation speed.

Typical mistakes and how to avoid them

• Labeling without uncertainty sampling — all examples in sequence. Solution: use ner.teach instead of ner.manual.
• Too many labels — model gets confused. Optimum: 5-10 labels per task.
• Poor initial data — model cannot select informative examples. Start with at least 50 high-quality labeled records.

Contact us for a consultation. Order Prodigy integration — get quality datasets 2-3 times faster, and save $10,000 to $50,000 in labeling costs per project.

pip install prodigy  # requires license key
prodigy ner.teach my_ner_dataset ru_core_news_lg texts.jsonl --label PRODUCT,FEATURE

Export for spaCy training:

prodigy data-to-spacy ./train ./dev --ner my_ner_dataset
python -m spacy train config.cfg --output ./model

# Conversion to HuggingFace dataset
from prodigy.components.db import connect
db = connect()
examples = db.get_dataset("my_ner_dataset")
from datasets import Dataset
hf_dataset = Dataset.from_list([
    {"tokens": ex["tokens"], "labels": convert_spans_to_bio(ex)}
    for ex in examples if ex["answer"] == "accept"
])

With 5+ years on the market and 50+ completed projects, we are a trusted partner. Order Prodigy integration — save up to $10,000–$50,000 in labeling costs per project.

NLP Development: Text Classification, NER, Embeddings, and Information Extraction

We often receive a task: process 50,000 support tickets — currently all manual. Dataset — 3,000 labeled examples, 12 categories, imbalance: one category occupies 40% of the sample, three at 1-2% each. Baseline accuracy — 78%. Sounds decent until you look at recall for rare classes: 0.31, 0.44, 0.28. These classes — complaints and churn threats — are most important to the business.

This is a typical NLP development project. The problem is not the algorithm but that accuracy is the wrong metric. Our experience across 30+ projects shows: we start by analyzing business metrics and only then choose the model.

Why accuracy is not the right metric for rare classes?

Accuracy ignores imbalance. If the "churn" class appears in 2% of cases, the model can predict "all good" and get 98% accuracy — but the business loses clients. Solution: F1 macro (averaged over all classes) or weighted F1. For NER — strict entity F1 (exact matches only). We guarantee: after choosing the correct metric, model quality becomes measurable and predictable.

Text Classification: From BERT to Distillation

BERT-like models are the standard for classification. ruBERT-base or ruBERT-large from DeepPavlov for Russian. multilingual-e5-large — for multiple languages in one pipeline. XLM-RoBERTa-large — a strong multilingual backbone.

Fine-tuning for classification: add a classification head on top of the [CLS] token, train for 3-5 epochs with lr=2e-5, weight decay=0.01. For imbalance — weighted CrossEntropyLoss or focal loss with gamma=2.0. Contact us — we will show a code snippet.

Imbalance case study. Dataset — 3,000 examples, imbalance 1:20. Solution: class_weight via sklearn + CrossEntropyLoss. Additionally — augmentation of rare classes via backtranslation (ru→en→ru through MarianMT). Recall for rare classes rose from 0.31 to 0.67 with a slight drop in accuracy (76%→74%). Full NLP development end-to-end took 3 weeks.

Distillation for production. BERT-large gives F1 0.89, but inference on CPU — 180ms. Distillation into DistilBERT or ruBERT-tiny2 reduces latency to 25ms with F1 0.84. Export to ONNX Runtime provides an additional 1.5-2x speedup. DistilBERT achieves 7x lower latency than BERT-large with only a 5% drop in macro F1 – a typical production trade-off.

How to choose between BERT and LLM for your task?

For most classification and extraction tasks, BERT-sized models offer the best trade-off between cost and performance. Shift to LLMs only when the task demands generation, complex reasoning, or zero-shot generalization.

NER: Named Entity Recognition

NER — extracting persons, organizations, locations, dates, amounts, document numbers. For general categories (PER, ORG, LOC), pre-trained models work well. For specialized ones (medical terms, legal concepts) — fine-tuning is needed.

Data annotation. The main cost of an NER project. For a quality model — 500-2,000 labeled sentences per entity type. Tools: Label Studio (open source) or Prodigy (by spaCy creators). IOB2 format — standard.

Architecture. Token classification on top of BERT: each token gets a label (B-PER, I-PER, O). spaCy 3.x with transformer pipeline — a convenient production choice.

Nested entities. Standard IOB models cannot handle nested entities (organization inside an address). For such tasks — span-based NER: SpanBERT or SpERT. More complex but correct.

Post-processing is mandatory. The model predicts tokens — normalized entities are needed. Date — dateparser. Amounts — regex + validation. Names — deduplication via rapidfuzz. Included in our standard delivery.

Sentiment Analysis and Opinion Mining

Binary classification positive/negative works out of the box with BERT. Complexity — aspect-based sentiment analysis (ABSA): "the restaurant has good food but terrible service." For ABSA: aspect extraction (NER) + sentiment per aspect. Joint models BERT-for-ABSA — quality on Russian data is lower due to dataset scarcity. RuSentiment, SentiRuEval — main resources.

For production with simple positive/negative/neutral: distil models are enough. Three classes, balanced dataset, 2,000+ examples — F1 macro 0.82-0.87 in 1-2 days.

Text Summarization

Extractive summarization (select sentences) — TextRank or BM25 without training. Fast, no hallucinations. Good for long documents.

Abstractive (generates new text) — seq2seq: mT5, mBART, FRED-T5, ruT5-large. For production via LLM API (GPT-4, Claude) — often the best cost/quality/speed trade-off.

Embeddings: Vector Representations of Text

Embeddings are the foundation of semantic search, deduplication, clustering, RAG. Quality critically affects downstream tasks.

Models. E5-large-v2, BGE-M3, multilingual-e5-large — strong multilingual embedders. sentence-transformers/paraphrase-multilingual-mpnet-base-v2 — fast option. For Russian: ru-en-RoSBERTa (Skoltech) performs well on semantic textual similarity.

Embedding quality evaluation uses the MTEB benchmark as standard. But top results on MTEB don't guarantee success on a domain dataset — we build domain-specific eval.

Fine-tuning embeddings. If standard models don't give the required Recall@k — contrastive learning on domain pairs with MultipleNegativesRankingLoss. How to perform this for domain data:

1. Collect 500–2,000 semantically similar pairs from your domain.
2. Apply MultipleNegativesRankingLoss with a batch size of 32–64.
3. Train for 1–3 epochs using AdamW (lr=2e-5).
4. Evaluate Recall@k on a held-out domain test set.

This approach yields a 5–15% improvement in Recall@k in practice.

Dimensionality and storage. E5-large: 1024 dim, float32 — 4KB per vector. For 10M documents — 40GB. Quantization int8 reduces to 10GB. FAISS IVF_PQ — more compact but with losses. Included in our deployment recommendations.

Information Extraction

Structured extraction is a frequent task. Examples: key contract terms, technical characteristics, dates and amounts from invoices.

1. Regex + rule-based. For INN, OGRN, amounts, dates — more reliable than neural networks. No data required.
2. NER + post-processing. For variable formats.
3. LLM with structured output. GPT‑4 / Claude with JSON schema — for complex documents. Cost: minimal per document. For 10k+ documents/day — we calculate the economics.

We guarantee a hybrid: regex/NER for typical fields + LLM for edge cases. Our guarantee is backed by years of production experience and more than 30 projects.

Work Stages

Prototype on existing data — 1-3 weeks. Production system with CI/CD — 1.5–2.5 months. Cost is calculated individually — get a consultation for a project estimate.

What's Included

• Model and pipeline architecture documentation
• Access to the model via REST API (FastAPI + ONNX)
• Client team training (2-hour webinar + Q&A)
• Accuracy guarantee on the agreed test set
• Months of post-delivery support (bug fixes, adaptation to new data)

Our Experience

Years of NLP projects from classification to RAG systems. The team includes ML engineers experienced with Hugging Face, spaCy, LangChain, MLOps. We use vLLM, Kubeflow, Weights & Biases — a production stack, not toys. Contact us to evaluate your NLP project within two days — request a free consultation on your text processing pipeline.