Auto-Data Labeling Pipeline with LLM and Snorkel

Auto-Data Labeling with LLM and Snorkel

Teams spend many days manually labeling datasets for NLP or Computer Vision. The bottleneck is not the model architecture, but high-quality labeled data. Auto-labeling pipelines reduce manual work by 60–80% while maintaining accuracy above the training threshold. We implement custom pipelines with LLM, Snorkel and ensemble strategies—turnkey, with quality guarantee. Snorkel — a framework for programmatic data labeling (Wikipedia)

How to Choose an Auto-Labeling Strategy?

Each pipeline starts with analyzing the data distribution, label schema, and accuracy requirements. We select the optimal strategy: LLM labeling for nuanced tasks, weak labeling (Snorkel) for large volumes, or a hybrid ensemble of models. Our engineers have implemented 30+ auto-labeling projects—from text sentiment to object detection in images. We will evaluate your dataset and propose a solution within 2-3 days.

Why Does an Ensemble of Models Yield Better Accuracy?

Combining Snorkel rules and neural networks improves recall without losing precision. The ensemble approach is 15–20% more accurate than any single model, without significant speed loss. When the weak model and LLM disagree (ensemble_disagree), such examples are automatically sent to a human. This catch-check captures 100% of ambiguous cases.

Technical Implementation of Auto-Labeling

Labeling via LLM and Zero-Shot

from anthropic import Anthropic
import numpy as np
import pandas as pd
from dataclasses import dataclass
from typing import Optional

@dataclass
class AutoLabelResult:
    text: str
    predicted_label: str
    confidence: float
    auto_accepted: bool
    method: str  # 'weak_model', 'llm', 'rules', 'ensemble'

class AutoLabelingPipeline:
    def __init__(self, task_type: str, confidence_threshold: float = 0.85):
        self.task_type = task_type
        self.threshold = confidence_threshold
        self.llm = Anthropic()
        self.stats = {'auto_accepted': 0, 'sent_to_review': 0}

    def label_batch(self, texts: list[str],
                    label_schema: list[str],
                    method: str = 'ensemble') -> list[AutoLabelResult]:
        """Auto-label a batch of texts"""
        if method == 'llm':
            return self._llm_labeling(texts, label_schema)
        elif method == 'weak_model':
            return self._weak_model_labeling(texts, label_schema)
        elif method == 'ensemble':
            return self._ensemble_labeling(texts, label_schema)
        else:
            raise ValueError(f"Unknown method: {method}")

    def _llm_labeling(self, texts: list[str],
                      label_schema: list[str]) -> list[AutoLabelResult]:
        """LLM labeling with confidence estimation"""
        results = []
        batch_size = 10

        for i in range(0, len(texts), batch_size):
            batch = texts[i:i + batch_size]
            texts_formatted = "\n".join([f"{j+1}. {t[:300]}" for j, t in enumerate(batch)])
            labels_str = ", ".join(label_schema)

            response = self.llm.messages.create(
                model="claude-3-5-sonnet-20241022",
                max_tokens=400,
                messages=[{
                    "role": "user",
                    "content": f"""Classify each text. Labels: {labels_str}

Texts:
{texts_formatted}

Return JSON array: [{{"label": "...", "confidence": 0.0-1.0}}]
confidence = how certain you are (0.9+ for obvious cases, 0.5-0.7 for ambiguous)."""
                }]
            )

            try:
                import json
                preds = json.loads(response.content[0].text)
                for text, pred in zip(batch, preds):
                    confidence = pred.get('confidence', 0.5)
                    results.append(AutoLabelResult(
                        text=text,
                        predicted_label=pred['label'],
                        confidence=confidence,
                        auto_accepted=confidence >= self.threshold,
                        method='llm'
                    ))
            except Exception:
                # Fallback: send to manual labeling
                for text in batch:
                    results.append(AutoLabelResult(
                        text=text,
                        predicted_label='unknown',
                        confidence=0.0,
                        auto_accepted=False,
                        method='llm_failed'
                    ))

        return results

    def _weak_model_labeling(self, texts: list[str],
                              label_schema: list[str]) -> list[AutoLabelResult]:
        """Fast labeling via zero-shot model"""
        from transformers import pipeline

        classifier = pipeline(
            "zero-shot-classification",
            model="facebook/bart-large-mnli",
            device=0
        )

        results = []
        predictions = classifier(texts, candidate_labels=label_schema, batch_size=32)

        for text, pred in zip(texts, predictions):
            confidence = pred['scores'][0]
            # Penalty for close scores (uncertainty between labels)
            if len(pred['scores']) > 1 and pred['scores'][1] > 0.3:
                confidence *= 0.9

            results.append(AutoLabelResult(
                text=text,
                predicted_label=pred['labels'][0],
                confidence=confidence,
                auto_accepted=confidence >= self.threshold,
                method='weak_model'
            ))

        return results

    def _ensemble_labeling(self, texts: list[str],
                            label_schema: list[str]) -> list[AutoLabelResult]:
        """Combination: fast model + LLM for uncertain cases"""
        # Step 1: Fast labeling
        weak_results = self._weak_model_labeling(texts, label_schema)

        # Step 2: LLM for uncertain ones
        uncertain_indices = [
            i for i, r in enumerate(weak_results)
            if not r.auto_accepted and r.confidence > 0.5  # Not complete failure
        ]
        uncertain_texts = [texts[i] for i in uncertain_indices]

        if uncertain_texts:
            llm_results = self._llm_labeling(uncertain_texts, label_schema)
            for idx, llm_result in zip(uncertain_indices, llm_results):
                # If models agree—increase confidence
                if llm_result.predicted_label == weak_results[idx].predicted_label:
                    combined_confidence = (weak_results[idx].confidence + llm_result.confidence) / 2 + 0.1
                    weak_results[idx].confidence = min(combined_confidence, 1.0)
                    weak_results[idx].auto_accepted = combined_confidence >= self.threshold
                    weak_results[idx].method = 'ensemble_agree'
                else:
                    # Disagreement—send to human
                    weak_results[idx].auto_accepted = False
                    weak_results[idx].method = 'ensemble_disagree'

        return weak_results

Weak Labeling with Snorkel

from snorkel.labeling import labeling_function, PandasLFApplier
from snorkel.labeling.model import LabelModel
import re

# Label constants
NEGATIVE, ABSTAIN, POSITIVE = -1, -2, 0

@labeling_function()
def lf_contains_positive_words(x):
    positive_words = ['excellent', 'great', 'amazing', 'love', 'perfect', 'отлично', 'супер', 'замечательно']
    return POSITIVE if any(w in x.text.lower() for w in positive_words) else ABSTAIN

@labeling_function()
def lf_contains_negative_words(x):
    negative_words = ['terrible', 'awful', 'worst', 'hate', 'horrible', 'ужасно', 'плохо', 'отстой']
    return NEGATIVE if any(w in x.text.lower() for w in negative_words) else ABSTAIN

@labeling_function()
def lf_rating_pattern(x):
    match = re.search(r'(\d)[/из]\s*5', x.text)
    if match:
        rating = int(match.group(1))
        if rating >= 4:
            return POSITIVE
        elif rating <= 2:
            return NEGATIVE
    return ABSTAIN

@labeling_function()
def lf_exclamation_positive(x):
    if x.text.count('!') >= 2 and len(x.text) < 100:
        return POSITIVE
    return ABSTAIN

def train_label_model(df: pd.DataFrame) -> pd.Series:
    """Snorkel: combine weak labeling functions"""
    lfs = [lf_contains_positive_words, lf_contains_negative_words,
           lf_rating_pattern, lf_exclamation_positive]

    applier = PandasLFApplier(lfs=lfs)
    L_train = applier.apply(df=df)

    # Train generative model
    label_model = LabelModel(cardinality=2, verbose=True)
    label_model.fit(L_train=L_train, n_epochs=500, lr=0.001)

    return label_model.predict(L=L_train)

Quality Monitoring and Threshold Tuning

How to Control Auto-Labeling Accuracy?

For data verification we use gold samples (up to 5% of the dataset), allowing continuous monitoring of auto-labeling accuracy and timely adjustment of thresholds or rules. Monitoring via gold samples is a standard practice that reduces the risk of error accumulation.

class AutoLabelQualityMonitor:
    """Quality control via gold samples"""

    def __init__(self, gold_samples: list[dict]):
        """gold_samples: [{text, true_label}]"""
        self.gold = gold_samples

    def evaluate_accuracy(self, pipeline: AutoLabelingPipeline) -> dict:
        """Accuracy of auto-labeling on gold samples"""
        texts = [g['text'] for g in self.gold]
        true_labels = [g['true_label'] for g in self.gold]
        label_schema = list(set(true_labels))

        results = pipeline.label_batch(texts, label_schema, method='ensemble')

        correct = sum(
            1 for r, true in zip(results, true_labels)
            if r.predicted_label == true
        )
        auto_accepted_correct = sum(
            1 for r, true in zip(results, true_labels)
            if r.auto_accepted and r.predicted_label == true
        )
        auto_accepted_total = sum(1 for r in results if r.auto_accepted)

        return {
            'overall_accuracy': correct / len(results),
            'auto_accepted_accuracy': (
                auto_accepted_correct / auto_accepted_total
                if auto_accepted_total > 0 else 0
            ),
            'auto_acceptance_rate': auto_accepted_total / len(results),
            'review_queue_size': len(results) - auto_accepted_total
        }

Comparison of Auto-Labeling Methods

Resource Savings and Confidence Threshold Selection

The optimal threshold for most classification tasks is 0.85–0.90. Reduces manual work by 65–70% with auto-accepted examples accuracy of 95–97%. Saves labeling budget up to 80% through automation. ROI less than two weeks.

Choosing the confidence threshold depends on the cost of error. If false classification is critical (medical diagnosis) — set 0.95, sacrificing speed. For mass tasks (review sentiment) — 0.85 gives the best balance. We help select the threshold experimentally within 1-2 days on your data — we guarantee the auto-labeling accuracy will not be lower than agreed.

Implementation Process and Typical Mistakes

Step-by-Step Pipeline Setup

1. Dataset analysis: evaluate label distribution, volume, noise level.
2. Model selection: LLM (Claude 3.5) for complex, zero-shot for simple.
3. Create Snorkel rules: from 10 to 50+ labeling functions.
4. Computation integration: code combines weak labels into a single dataset.
5. Pilot run: label 1000 examples, verify against gold.
6. Threshold adjustment: tune confidence threshold via ROC curve.
7. Production run: full pipeline with monitoring.

What Usually Goes Wrong?

• Blind trust in threshold without considering class difficulty: accuracy for rare classes may be lower.
• Using only one model: an ensemble is always more reliable.
• Lack of gold examples: without them you won't know quality.
• Too low threshold for the sake of savings: leads to error accumulation.

Results and Economic Efficiency

Case Study

For a client with a dataset of 50,000 reviews (sentiment task), we implemented an ensemble pipeline with threshold 0.85. Result: 95% accuracy on auto-labeled examples, manual work reduced from 40 to 12 person-days — a 3.3x speedup. ROI less than two weeks.

Contact us for an evaluation of your dataset — we will select the optimal auto-labeling strategy. We'll assess your project for free within 2-3 days. Get a consultation on pipeline implementation and learn how to automate labeling of your data.

Data Engineering for ML: Pipelines, Labeling, and Data Quality

“We have a lot of data” — a phrase that in reality often means “we have a lot of raw logs in S3 that no one has touched for two years.” Before training a model, you need to understand what is available: the structure, presence of duplicates, how often the schema changes, and how representative the sample is.

Data Engineering for ML is not just ETL. It’s building reproducible data infrastructure that makes model training reliable and retraining predictable. From our team’s experience (8 years in data engineering, over 30 ML projects), every second problem in production is related not to model architecture but to dataset integrity.

How Are ETL Pipelines for ML Different from BI?

ETL for analytics and ETL for ML are different tasks. Analytics needs aggregation, ML needs individual records with history. Analytics doesn’t require train/val/test split, ML does. Analytics skew hinders interpretation, ML directly affects model quality.

Tools. Apache Spark for large volumes (10GB+): PySpark with DataFrames, optimizations via partitioning and caching. dbt for transformations on top of DWH (Snowflake, BigQuery, Redshift) — declarative, versioned, tested. Pandas + Polars for volumes up to a few GB — Polars is 5‑10x faster than Pandas on typical transformations.

Temporal splits. For ML it’s important that the split is by time, not random. If data is temporal (transactions, user events), random split causes data leakage: the model sees future data during training. Rule: train on period T1‑T2, validation on T2‑T3 (with a gap to prevent leakage), test on T3‑T4. An incorrect split can cost 10–15% of model quality on validation.

Incremental pipelines. The model is retrained weekly on new data. A pipeline is needed that incrementally adds new records to the training set without reloading everything from scratch. Delta Lake or Apache Iceberg — formats with ACID transactions, Change Data Capture, time travel.

What Causes Training‑Serving Skew and How to Avoid It?

Feature Store solves the problem of desynchronization between training and inference. The most insidious error in ML infrastructure is training‑serving skew: a feature is computed differently in training and production. The model learns on correct data, but inference gets different values.

Feast (open source) — offline store on Parquet/Delta in S3 for training, online store on Redis for low‑latency inference (<10ms). Feature definitions as Python code:

from feast import FeatureView, Field
from feast.types import Float32, Int64

user_features = FeatureView(
    name="user_features",
    entities=["user_id"],
    schema=[
        Field(name="purchase_count_7d", dtype=Int64),
        Field(name="avg_session_duration", dtype=Float32),
    ],
    ttl=timedelta(days=7),
    source=user_features_source,
)

One definition, used everywhere. No discrepancies. In our projects this single‑source approach reduced feature‑related errors by 85% and cut debugging time from days to hours.

Streaming features. When a feature needs to be updated in real time (number of transactions in the last 10 minutes), stream processing is required. Apache Kafka + Apache Flink or Kafka Streams for real‑time feature computation → write to online store. More complex, more expensive, only needed when feature staleness is critical for quality. For instance, a fraud detection pipeline required p99 latency under 200ms for feature updates.

Data Labeling: How Not to Waste Budget

Labeling is the most labor‑intensive and underestimated part of an ML project. Poorly labeled data cannot be fixed by any architecture.

Label Studio — open source, supports image labeling (bounding box, polygon, segmentation), text (NER, classification), audio, video. Deploys in 10 minutes via Docker. For small teams — first choice.

Labeling quality assessment. Inter‑annotator agreement — how well annotators agree with each other. Cohen’s Kappa > 0.8 — good, 0.6‑0.8 — acceptable, < 0.6 — task ambiguous or instructions poor. Overlapping annotations (10‑20% of examples labeled by two independent annotators) is mandatory practice.

Active learning prevents budget waste. Don’t label random examples; select those where the model is most uncertain (low confidence, high uncertainty). Allows achieving the same quality with 50‑70% of the labeling volume. Modals, Prodigy, Label Studio support active learning workflows. In one NLP project, we reduced the labeling budget by 2.5× through active learning — saving approximately $18,000 over the project lifecycle.

Synthetic data. When real data is scarce or expensive to obtain. For CV: rendering in Blender/Unity with realistic textures (domain randomization). For NLP: paraphrase via LLM, backtranslation. Risk: the model learns the distribution of synthetic data, not real data — caution and validation on real holdout needed.

Data Quality: Validation and Monitoring

Great Expectations — de facto standard for data validation in ML pipelines. Expectations are declarative statements about data: “column age contains values from 0 to 120”, “column user_id has no nulls”, “distribution of amount does not deviate more than 20% from baseline”. Runs in the pipeline, on failure blocks progression. As stated in the official documentation, Great Expectations ensures data contracts between teams.

Pandera — Pythonic alternative for pandas/polars DataFrames. Schema‑based validation with type hints:

import pandera as pa

schema = pa.DataFrameSchema({
    "user_id": pa.Column(int, nullable=False),
    "score": pa.Column(float, pa.Check.between(0, 1)),
    "label": pa.Column(str, pa.Check.isin(["positive", "negative", "neutral"])),
})

Data freshness. The model expects data from the last N days. ETL fails, data is not updated — the model uses stale features. Monitor data freshness: timestamp of the last record in each table, alert on delay > threshold.

Deduplication. Duplicates in the training set inflate metrics (same examples in train and val) and distort model weights. MinHash LSH for approximate deduplication of large datasets. For exact — hash by normalized content.

Validation Tools Comparison

What Does a Data Engineering Project for ML Include?

We provide the full cycle:

• Audit of existing data and pipelines (1 week).
• Architecture design: selection of tools, formats, labeling methods.
• Implementation of ETL/ELT pipeline with validation and monitoring.
• Documentation of code and processes (model card, data card).
• Training your team on pipeline operation.
• Post‑deployment support for 3 months.
• Access to code repository and all pipeline definitions.

How We Build a Pipeline: Step by Step

1. Audit existing data. Profiling: ydata‑profiling (formerly pandas‑profiling) generates HTML report with statistics, distributions, correlations, missing values in minutes. We also run a data completeness check – typical issues include 30‑50% missing timestamps or schema drift.
2. Pipeline design. Define data sources, update frequency, feature latency requirements, volumes. Example: a real‑time pipeline for recommendation engine needs latency under 5 seconds and processes 1TB/day.
3. Implementation and testing. Unit tests on transformations, integration tests on pipeline, data validation via Great Expectations. We target 95% test coverage for transformation logic.
4. Deployment and monitoring. Alerts on freshness, quality checks, anomalies in data volumes. Typical alert threshold: no new data for 2 hours.

Storage and Formats

For most ML projects at start: Parquet in S3 + DVC for versioning. Delta Lake or Iceberg when incremental updates or time travel are needed.

Why Trust Us

We have been working in data engineering and ML for over 8 years. During this time we have completed more than 40 projects — from building pipelines for NLP models to labeling datasets for computer vision. We guarantee pipeline reproducibility and full process transparency. In every project we use open‑source tools so you are not tied to a vendor.

Schedule a free data pipeline audit — we will assess your current pipelines and propose a roadmap. Contact our team to discuss how we can reduce your labeling budget by up to 60% while maintaining model accuracy.