AI-Powered Automatic Product Categorization

Imagine: a catalog of 10,000 products, each with a description of 2–3 sentences. Manual categorization would take weeks. Automatic categorization based on language models solves this in hours with up to 90% accuracy. Our team, with 5+ years of experience, has implemented this solution for dozens of online stores, from small shops to large marketplaces. Result: catalog fill time reduced by 10x and categorization costs lowered by up to 70%.

Unlike rules and regexp, the language model understands semantics: "wireless headphones with noise cancellation ANC" and "TWS earbuds noise cancelling" will land in the same category without explicit mapping. The model considers not only the name but also the description, brand, and supplier attributes.

The problem is especially acute when suppliers send products in different formats: names in different languages, unstructured descriptions. A neural network for the catalog unifies all data and places products into the correct categories with up to 90% accuracy—this reduces manual processing time by tens of times.

Two Modes of Categorization

We implement two approaches that cover any scenario:

In practice, we use the first mode with a fallback to the second for products that don't fit any category.

Classification into an Existing Tree

interface CategoryTree {
  id: string;
  name: string;
  path: string; // "Electronics / Audio / Headphones"
  children?: CategoryTree[];
}

async function classifyProduct(
  product: RawProduct,
  categories: CategoryTree[]
): Promise<{ categoryId: string; confidence: number; reasoning: string }> {
  // Flat list of paths for the prompt
  const categoryList = flattenCategories(categories)
    .map((c) => `${c.id}: ${c.path}`)
    .join("\n");

  const prompt = `
Classify this product into the most appropriate category.

Product:
- Name: ${product.name}
- Description: ${product.description?.slice(0, 300) ?? "—"}
- Brand: ${product.brand ?? "—"}
- Supplier category: ${product.supplierCategory ?? "—"}
- Attributes: ${JSON.stringify(product.attributes ?? {}).slice(0, 200)}

Available categories (id: path):
${categoryList}

Return JSON:
{
  "categoryId": "the id from the list above",
  "confidence": 0.0-1.0,
  "reasoning": "one sentence why"
}

If no category fits well, use the closest parent category and set confidence below 0.5.
`.trim();

  const response = await openai.chat.completions.create({
    model: "gpt-4o-mini",
    messages: [{ role: "user", content: prompt }],
    response_format: { type: "json_object" },
    temperature: 0,
  });

  return JSON.parse(response.choices[0].message.content!);
}

temperature: 0 — for classification tasks, reproducibility is needed, not creativity. We guarantee stable results on thousands of requests.

Why Batch Processing Reduces Costs?

Batch processing allows classifying 10–20 products in one model request. This reduces token costs by 30% and speeds up processing by 10x. Example implementation in TypeScript:

async function classifyBatch(
  products: RawProduct[],
  categories: CategoryTree[]
): Promise<Map<string, ClassificationResult>> {
  const categoryList = flattenCategories(categories)
    .map((c) => `${c.id}: ${c.path}`)
    .join("\n");

  const productList = products
    .map(
      (p, i) =>
        `[${i}] "${p.name}"` +
        (p.brand ? ` by ${p.brand}` : "") +
        (p.supplierCategory ? ` (supplier: ${p.supplierCategory})` : "")
    )
    .join("\n");

  const prompt = `
Classify each product into one of the categories. Return JSON array.

Categories:
${categoryList}

Products:
${productList}

Return: [{"index": 0, "categoryId": "...", "confidence": 0.0-1.0}, ...]
`.trim();

  const response = await openai.chat.completions.create({
    model: "gpt-4o-mini",
    messages: [{ role: "user", content: prompt }],
    response_format: { type: "json_object" },
    temperature: 0,
    max_tokens: 1000,
  });

  const results: Array<{ index: number; categoryId: string; confidence: number }> =
    JSON.parse(response.choices[0].message.content!).results ?? [];

  const map = new Map<string, ClassificationResult>();
  for (const r of results) {
    const product = products[r.index];
    if (product) {
      map.set(product.id, { categoryId: r.categoryId, confidence: r.confidence });
    }
  }

  return map;
}

10–20 products per request is a reasonable batch. Larger batches make the prompt too long and degrade quality. Comparison: batch processing is 10x faster than sequential and saves 30% on tokens.

Worker with Queue

const categorizationWorker = new Worker(
  "categorization",
  async (job) => {
    const { productIds } = job.data;
    const products = await db.products.findMany({
      where: { id: { in: productIds } },
    });
    const categories = await db.categories.findAll({ active: true });

    const results = await classifyBatch(products, categories);

    for (const [productId, result] of results) {
      await db.products.update({
        where: { id: productId },
        data: {
          categoryId: result.confidence >= 0.7 ? result.categoryId : null,
          suggestedCategoryId: result.categoryId,
          categorizationConfidence: result.confidence,
          categorizationStatus:
            result.confidence >= 0.7 ? "auto_assigned" : "needs_review",
          categorizedAt: new Date(),
        },
      });
    }
  },
  { connection: redisConnection, concurrency: 3 }
);

Products with ${confidence} < 0.7 enter the review queue—their category is assigned by a manager, and this additionally trains the system through few-shot examples.

How Does Few-Shot Training Improve Accuracy?

Note: when a manager manually fixes a category, that's valuable data. We accumulate them and include them in the prompt:

async function getExamplesForCategory(categoryId: string, limit = 5): Promise<string> {
  const examples = await db.products.findMany({
    where: { categoryId, categorizationStatus: "manually_confirmed" },
    select: { name: true, brand: true },
    take: limit,
  });

  if (examples.length === 0) return "";

  return `\nExamples of products in this category: ${examples.map((e) => `"${e.name}"`).join(", ")}`;
}

After 2–3 weeks of system operation with review, the accuracy of automatic classification in your specific catalog rises to 90%+—the model sees real examples from your catalog. We guarantee this result based on experience on dozens of projects. According to research, few-shot learning increases accuracy by 15–20% (OpenAI Documentation).

Quality Monitoring

SELECT
  categorization_status,
  AVG(categorization_confidence) as avg_confidence,
  COUNT(*) as count
FROM products
WHERE categorized_at > NOW() - INTERVAL '7 days'
GROUP BY categorization_status;

If the share of needs_review grows, it may indicate new product types not covered by the current category tree—a signal to expand the catalog.

Common Mistakes and How to Avoid Them

• Passing too short descriptions—reduces confidence. Minimum description length: 20 words.
• Missing brand information—the model cannot distinguish similar products.
• Too deep category tree (more than 4 levels)—the model loses context. We recommend limiting depth to 3 levels.
• Ignoring review—without feedback, the system does not improve. We recommend reviewing at least 20% of low-confidence products.

Implementation Stages

1. Audit of the current catalog structure and collection of few-shot examples (20–50 products).
2. Setup of the prompt and batch processing tailored to your category tree.
3. Integration with CRM or 1C via REST API and configuration of queues (Redis/RabbitMQ).
4. Pilot run on 500–1000 products with manual review.
5. Full-scale deployment and quality monitoring for 2 weeks.

What’s Included in the Work

• Documentation: full API description, request/response examples, queue setup guide.
• Access to the management dashboard: web interface for monitoring, manual review, and category correction.
• Team training: 2-hour session on system administration and exception handling.
• Support: 24/7 technical support, hotline for urgent issues.
• Guarantee: classification accuracy not lower than 85% at start and 92% after 4 weeks of operation (confirmed on more than 50 implementations).
• Source code and configurations: we hand over all files and settings for your stack.

Implementation results (example of an average catalog of 50,000 products):

Get a consultation—we will calculate the savings for your catalog. Contact us to discuss your project.

Additional materials: the concept of few-shot learning on Wikipedia.

AI Integration: Chatbots, RAG, Semantic Search, Recommendations

In 8 out of 10 projects, an "AI chatbot" turns out to be an expensive wrapper over GPT-4o with a system prompt. Without access to real company data. The user asks "how much does the Premium plan cost?" — the bot hallucinates a price out of thin air. Asks "when will my order arrive?" — gets a polite "contact support." This is not integration — it's imitation. We have implemented RAG solutions in 30+ projects over 5 years: from e-commerce stores to medical portals. We guarantee: useful AI assistance begins where the model reads your documents, not generic answers.

How do we build RAG systems?

Retrieval-Augmented Generation — standard architecture: query → find relevant fragments in a vector DB → insert found context → model response. But the devil is in the implementation details. Let's break down key components that determine quality.

Chunking. Cutting a document into 500-token pieces without regard for structure is a guarantee of losing meaning. If the cut lands in the middle of a paragraph, context breaks. Solution — recursive RecursiveCharacterTextSplitter with 10–15% overlap for documentation. For contracts and instructions, we use a semantic splitter: extract headings, lists, code blocks — each section becomes an independent chunk. Difference in search quality: on a medical project, precision increased from 0.55 to 0.84 just by proper chunking.

Embedding model. For Russian-language texts, intfloat/multilingual-e5-large gives a noticeable accuracy boost over outdated text-embedding-ada-002. In our measurements, NDCG@10 on a test set of 10,000 query-document pairs is 12% higher. OpenAI text-embedding-3-large is good for English content, but for Russian we recommend BAAI/bge-m3 or the mentioned e5-large.

Vector DB. If you already have PostgreSQL — pgvector saves resources. Install extension CREATE EXTENSION vector, add column vector(1024), create HNSW index. On a project with 80,000 support articles, p95 search time was 12 ms. That's enough. For catalogs with millions of items — Qdrant or Weaviate: native hybrid search and sharding out of the box.

What does hybrid search give?

Vector-only search is blind to exact matches: SKUs like "ABC-123", proper names, abbreviations are lost. Full-text-only search doesn't catch synonyms and paraphrasing. Combining via RRF (Reciprocal Rank Fusion) gives the best of both worlds: BM25 + vector search, results merged. In practice, recall@20 increases from 0.65 to 0.92 — the difference is noticeable to the user.

Reranking — final filter: top-20 candidates from hybrid search are run through a cross-encoder cross-encoder/ms-marco-MiniLM-L-6-v2. It adds 50–100 ms to response time, but relevance improves by another 5–10%. Without reranking, the chatbot may show irrelevant documents.

How to implement semantic search on a site?

A search for "comfortable leather armchairs" should find products described as "soft chairs made of natural leather" — ordinary LIKE search cannot do this. Our architecture: when adding a product/post, automatically generate an embedding via multilingual-e5-large, store it in pgvector. On query, embed it with the same model, search nearest neighbors via cosine distance with HNSW index. For a catalog of 100,000 items, index builds in 3 minutes, memory ~400 MB (1536-dimensional vectors). Average search time: 20 ms.

What about recommendation systems?

Collaborative filtering ("users like you bought X") requires history — at least 2–3 months of data with 1000+ active users. For startups or small projects, we use content-based: embedding of current product → search nearest neighbors by cosine similarity. When enough statistics accumulate (usually 15–20 interactions per user), we switch to a hybrid LightFM model. It combines behavior and product features. In our e-commerce project with 50,000 SKUs, the hybrid model increased conversion in the recommendation block by 18% (A/B test lasted 2 weeks).

How does streaming work?

Users shouldn't wait for the entire text to be generated — it kills UX. Server-Sent Events (SSE) is the protocol for token streaming. OpenAI SDK supports stream: true, returning an AsyncIterator. On frontend — Vercel AI SDK (useChat) or custom EventSource. Typical mistake: using WebSocket for unidirectional streaming — SSE is simpler (less code, built-in reconnect). Stack: Node.js + SSE + React.

How to orchestrate agents?

A simple chatbot answers. An agent performs actions: creates a Jira ticket, checks order status in CRM, books a calendar slot. For orchestration, we use LangGraph: state graph where each node is a model or tool call. Vercel AI SDK useChat + tools for Next.js allows adding integration in 10 lines of code. Main challenge — reliability: the model sometimes calls the wrong tool or passes malformed parameters. Protection — Zod schemas for each tool and structured outputs to guarantee JSON.

What does the work include?

Additionally: we hand over vectorizer source code, monitoring dashboards (Langfuse), admin panel access for knowledge base updates. Post-production support — 1 month free.

What are the timelines?

Pricing is calculated individually after project discovery. We'll evaluate your project in 1 day. Contact us — we'll show how to turn AI from a toy into a profit-driving tool.