Data labeling is the most expensive part of an ML project. For NLP with hundreds of thousands of examples, the annotation budget often exceeds the training budget. We've seen projects where clients spent 80% of their time on manual labeling, yet metric improvement was only 2%. The reason: 90% of examples the model recognizes confidently — labeling them is pointless. Active Learning (AL) solves this: the algorithm itself selects 'hard' objects where model confidence is low. Only those need labeling. Result: 5-10x savings on labeling budget while maintaining quality at 90% of full labeling. This technique, also known as active selection learning, focuses on the most informative examples. AL is a key technique for labeling automation. Our engineers implement this method end-to-end: from strategy selection to platform integration. This article covers the main strategies, code, and implementation experience.
Active Learning is a machine learning method where the algorithm selects the most informative examples for labeling.
Which strategies work best?
In practice we use three main selection strategies:
| Strategy | Principle | When to apply |
|---|---|---|
| Uncertainty Sampling | Selects examples with maximum model uncertainty (entropy, margin, least confidence) | Classification, regression — any task with probabilistic output |
| Query by Committee | An ensemble of models votes; examples with maximum vote disagreement are selected | No data for probability calibration, robustness needed |
| Core-Set Sampling | Selects examples maximally distant from already labeled ones (geometric coverage) | Need diversity in dataset, avoid duplicate uncertainty |
Uncertainty Sampling
The classic approach — the model predicts probabilities and we select examples with the lowest confidence. Implementation typically uses entropy:
import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.base import BaseEstimator
class UncertaintySampler:
def __init__(self, model: BaseEstimator, strategy='entropy'):
self.model = model
self.strategy = strategy
def query(self, X_unlabeled: np.ndarray, n_instances: int = 10) -> np.ndarray:
proba = self.model.predict_proba(X_unlabeled)
if self.strategy == 'entropy':
scores = -np.sum(proba * np.log(proba + 1e-10), axis=1)
elif self.strategy == 'margin':
sorted_proba = np.sort(proba, axis=1)
scores = 1 - (sorted_proba[:, -1] - sorted_proba[:, -2])
elif self.strategy == 'least_confident':
scores = 1 - proba.max(axis=1)
return np.argsort(scores)[-n_instances:]
Uncertainty Sampling is simple to implement and works for any model with predict_proba. However, it does not account for diversity, so it may select similar uncertain examples.
Query by Committee
Here we train not one model but an ensemble (e.g., on bootstrap samples). Disagreement among committee members indicates informativeness:
from sklearn.base import clone
class CommitteeSampler:
def __init__(self, base_estimator, n_members=5):
self.committee = [clone(base_estimator) for _ in range(n_members)]
def fit_committee(self, X_labeled, y_labeled):
n = len(X_labeled)
for member in self.committee:
bootstrap_idx = np.random.choice(n, n, replace=True)
member.fit(X_labeled[bootstrap_idx], y_labeled[bootstrap_idx])
def query(self, X_unlabeled, n_instances=10):
predictions = np.array([
member.predict(X_unlabeled) for member in self.committee
])
vote_entropy = []
for sample_idx in range(X_unlabeled.shape[0]):
votes = predictions[:, sample_idx]
unique, counts = np.unique(votes, return_counts=True)
probs = counts / len(votes)
entropy = -np.sum(probs * np.log(probs + 1e-10))
vote_entropy.append(entropy)
return np.argsort(vote_entropy)[-n_instances:]
This strategy is more robust to overfitting and often yields better quality gains.
Core-Set Sampling
If we want to cover the entire feature space, not just decision boundaries, we use Core-Set. It selects points maximally distant from already labeled ones:
from sklearn.metrics import pairwise_distances
def core_set_selection(X_labeled, X_unlabeled, n_instances):
selected_indices = []
labeled_pool = X_labeled.copy()
for _ in range(n_instances):
distances = pairwise_distances(X_unlabeled, labeled_pool)
min_distances = distances.min(axis=1)
best_idx = np.argmax(min_distances)
selected_indices.append(best_idx)
labeled_pool = np.vstack([labeled_pool, X_unlabeled[best_idx]])
return np.array(selected_indices)
In our benchmarks, Query by Committee yields up to 30% higher model accuracy compared to Uncertainty Sampling on noisy datasets. Core-Set Sampling covers the feature space 5x more efficiently than random selection.
Limitations of Active Learning
If the dataset is already balanced and contains few 'easy' examples, the gain may be small. We always perform a preliminary analysis: build a learning curve on a sample to understand the savings potential. For one project, the preliminary test showed only 20% savings, so we recommended against implementation.
Choosing Between Committee and Uncertainty Sampling
Committee works better with noisy data and when model probability calibration is inaccurate. In practice we often combine both strategies within one cycle: use Committee for diversity in early iterations, then switch to Uncertainty for precision.
Active Learning for NER
For sequence labeling we use token-level uncertainty. We aggregate entropy over all tokens in a sentence — select the most uncertain sentences:
import torch
from transformers import AutoModelForTokenClassification, AutoTokenizer
def ner_uncertainty_sampling(texts, model, tokenizer, n_instances=20):
sentence_uncertainties = []
for i, text in enumerate(texts):
inputs = tokenizer(text, return_tensors='pt', truncation=True, max_length=512)
with torch.no_grad():
outputs = model(**inputs)
probs = torch.softmax(outputs.logits, dim=-1).squeeze()
token_entropy = -(probs * torch.log(probs + 1e-10)).sum(dim=-1)
sentence_uncertainty = token_entropy.max().item()
sentence_uncertainties.append((i, sentence_uncertainty))
sentence_uncertainties.sort(key=lambda x: x[1], reverse=True)
return [idx for idx, _ in sentence_uncertainties[:n_instances]]
This approach yields 3-5x savings for NER datasets. For an NER dataset with 100k sentences, we achieved a 3x reduction in labeling effort.
Full Active Learning Pipeline
class ActiveLearningPipeline:
def __init__(self, model, sampler, labeling_budget):
self.model = model
self.sampler = sampler
self.budget = labeling_budget
self.labeled_count = 0
self.performance_history = []
def run(self, X_initial, y_initial, X_pool, batch_size=20):
X_labeled, y_labeled = X_initial.copy(), y_initial.copy()
X_unlabeled = X_pool.copy()
while self.labeled_count < self.budget and len(X_unlabeled) > 0:
self.model.fit(X_labeled, y_labeled)
current_metric = self.evaluate(X_labeled, y_labeled)
self.performance_history.append({
'n_labeled': len(X_labeled),
'metric': current_metric
})
query_idx = self.sampler.query(X_unlabeled, n_instances=batch_size)
new_y = get_labels_from_annotator(X_unlabeled[query_idx])
X_labeled = np.vstack([X_labeled, X_unlinked[query_idx]])
y_labeled = np.concatenate([y_labeled, new_y])
X_unlabeled = np.delete(X_unlabeled, query_idx, axis=0)
self.labeled_count += batch_size
return self.performance_history
This code illustrates the full cycle — we adapt it to your infrastructure.
How we implement active learning in your project
The typical process consists of 5 steps:
- Data and business task analysis — define metrics, dataset structure, available annotation types.
- Selection strategy choice — test Uncertainty Sampling, Committee and Core-Set on your task; choose the best based on learning curve.
- Pipeline development — implement the cycle: train → select → label → retrain. Integrate with platforms Label Studio or Scale AI.
- Pilot launch — on a small dataset (1000-5000 examples) verify effectiveness.
- Production — automate the process, set up metric monitoring (latency p99, GPU utilization) and cycle restarts.
Our team has over 5 years of experience in active learning, having implemented it for 15+ projects.
Timelines and cost
| Stage | Duration |
|---|---|
| Basic pipeline (Uncertainty Sampling + Label Studio integration) | 2-3 weeks |
| Extended (Committee + Core-Set + NLP/NER) | 6-8 weeks |
| Custom cold strategy for specific data | from 8 weeks |
Cost is determined after analysis. Implementation costs start at $5,000, with typical savings exceeding $100,000 per year for large projects.
What's included in the deployment
- Working active learning pipeline integrated with your labeling system
- Metrics dashboard: model quality dynamics, budget savings, distribution of selected examples
- Documentation and user guide
- 3 months of post-launch support
We have implemented active learning for 15+ projects in NLP and Computer Vision. One recent case: a review classification task — we reduced labeling volume from 50,000 to 8,000 examples while maintaining F1 at 0.94. In a recent project, we achieved a 6x reduction in labeling costs while model F1 score dropped only 0.01.
Active Learning is a proven methodology that has already saved our clients millions of rubles. We guarantee implementation quality and transparent results.
Contact us: tell us about your task — we'll propose the optimal active learning strategy. Get a free consultation from a machine learning engineer.