REST API for Parser Bots: Management, Monitoring, Webhooks

When you manage multiple parsers manually, each launch via SSH takes 5 minutes. With 20 scripts, that's nearly 2 hours daily. Our REST API centralizes control: any parser launches in 30 seconds via HTTP. We've built such APIs for 30+ projects, managing up to 50 bots per client. After implementation, the time to launch a new parser dropped from 30 minutes to 30 seconds — 60x faster. A typical client saves $5,000/month in engineering hours.

Key Problems That REST API Solves

Chaotic Management of Multiple Scripts

Without an API, each parser is launched manually via SSH, logs are scattered, error restart is manual. Our API centralizes management: all operations via HTTP with unified authentication. With 50 parsers, time savings amount to 40 hours per month, cutting costs by 30%.

Real-time Monitoring

Parsers without monitoring crash unnoticed — data is not collected for hours. We added webhooks and status endpoints — each run is tracked in real time. On error, a notification arrives instantly via Telegram or Slack. Average response time drops from 2 hours to 2 minutes — 60x improvement.

Complexity of Integration with External Systems

REST API provides a single interface understandable by any system supporting HTTP. Launching parsers can be easily embedded into CI/CD: after deployment, data collection starts automatically. In one project, integration took 2 days instead of 2 weeks with manual setup — 5x faster.

Choosing the API Stack

Performance and development speed are key. We prefer FastAPI for its asynchronicity, automatic validation, and built-in OpenAPI documentation. As the official site states: FastAPI is a modern, fast (high-performance) web framework for building APIs with Python. Alternatives include Django (powerful admin) and Node.js (if the team is JS). Comparison:

In high-load projects, FastAPI delivers 2x the RPS of Django, with API response time under 50 ms. This means FastAPI is 2 times better than Django in terms of raw performance.

How to Connect a Parser to the API in Three Steps

Step 1: Register the parser. Send a POST request to /api/v1/scrapers with configuration: name, start URL, schedule (cron), proxy pool, and limits. Takes 1 minute. Step 2: Launch immediately or on schedule. Call POST /api/v1/scrapers/{id}/run. The API returns run_id. The parser runs in the background; status is tracked via GET or webhooks. Step 3: Set up monitoring. Subscribe to webhooks: specify URL and events (run.completed, run.failed). The system sends a notification automatically when the event occurs.

The entire setup process for a new parser takes no more than 2 minutes, whereas manual management requires 2 hours for writing scripts. Using the REST API is 10 times better than manual SSH launches.

Typical API Endpoints

We design APIs according to REST principles using modern frameworks like FastAPI. Basic endpoint structure:

POST   /api/v1/scrapers              — create a new scraper
GET    /api/v1/scrapers              — list scrapers
GET    /api/v1/scrapers/{id}         — scraper configuration
PATCH  /api/v1/scrapers/{id}         — update configuration
DELETE /api/v1/scrapers/{id}         — delete scraper

POST   /api/v1/scrapers/{id}/run     — launch immediately
POST   /api/v1/scrapers/{id}/stop    — stop running
GET    /api/v1/scrapers/{id}/status  — current status

GET    /api/v1/scrapers/{id}/runs    — run history
GET    /api/v1/scrapers/{id}/runs/{runId} — run details

GET    /api/v1/scrapers/{id}/results — parsing results

Example implementation in FastAPI with background tasks and validation:

from fastapi import FastAPI, HTTPException, BackgroundTasks
from pydantic import BaseModel
from typing import Optional

app = FastAPI()

class ScraperConfig(BaseModel):
    name:        str
    url:         str
    schedule:    Optional[str] = None  # cron expression
    proxy_pool:  Optional[str] = None
    rate_limit:  int = 5  # req/sec
    headers:     dict = {}

@app.post('/api/v1/scrapers', status_code=201)
async def create_scraper(config: ScraperConfig):
    scraper = await ScraperRepository.create(config.dict())
    if config.schedule:
        await Scheduler.register(scraper.id, config.schedule)
    return scraper

@app.post('/api/v1/scrapers/{scraper_id}/run')
async def run_scraper(scraper_id: int, background_tasks: BackgroundTasks):
    scraper = await ScraperRepository.get_or_404(scraper_id)
    if scraper.status == 'running':
        raise HTTPException(409, 'Scraper is already running')
    run = await ScraperRun.create(scraper_id=scraper_id, status='pending')
    background_tasks.add_task(execute_scraper, scraper, run.id)
    return {'run_id': run.id, 'status': 'started'}

@app.get('/api/v1/scrapers/{scraper_id}/status')
async def get_status(scraper_id: int):
    scraper  = await ScraperRepository.get_or_404(scraper_id)
    last_run = await ScraperRun.get_latest(scraper_id)
    return {
        'id':          scraper_id,
        'status':      last_run.status if last_run else 'idle',
        'last_run':    last_run.started_at if last_run else None,
        'items_count': last_run.items_collected if last_run else 0,
    }

Authentication and Security

API keys with access levels: read, write, admin. Keys are stored as hashes (bcrypt), transmitted in the Authorization: Bearer {key} header. We also support OAuth2 for corporate clients — this simplifies audit and increases security.

Webhooks for Notifications

Subscribe to events: run completion, errors, new data. Example webhook registration:

@app.post('/api/v1/webhooks')
async def create_webhook(url: str, events: list[str]):
    return await WebhookRepository.create(url=url, events=events)

What's Included in the Work (Deliverables)

• API architecture design tailored to your business processes.
• Implementation on FastAPI or another stack (Django, Node.js).
• Automatic OpenAPI documentation (Swagger).
• Authentication via API keys or OAuth2.
• Webhooks for key events.
• Deployment on your server or cloud (Docker Compose, Kubernetes).
• 30 days of free support after launch.
• Source code, API keys, and deployment scripts provided.
• Training session for your team (1 hour).

Our engineers have over 5 years of experience in developing such systems, certified in FastAPI and Kubernetes, with 30+ projects completed.

Timelines and Cost

Estimated timelines: 5–8 business days for basic functionality (CRUD, start/stop, status). Cost: from $2,000 (basic) to $3,500 (with webhooks and monitoring). Complex integrations may cost up to $8,000. For example, one client saved $3,000 in the first month after switching to our API. Write to us for a free evaluation — we'll give you a fixed price in 1 business day.

Comparing Profitability: REST API vs Manual Management

REST API provides much higher reliability and development speed. Automation reduces support costs by up to 40% compared to manual management. A typical project pays for itself in 2–3 months, saving $5,000/month.

Order your turnkey REST API development today. Get a consultation on your API architecture — we'll help reduce integration time by 3x. Contact us for a free quote with no obligation.

API Development with REST, GraphQL, WebSocket, and tRPC

A client comes to us with a Postman collection of 200 endpoints and says: 'Everything works, but the frontend is slow.' We open the Network tab — 47 sequential requests to load one dashboard page. Each one waits for the previous. This is not a server speed issue — it's an API architecture problem. With 10 years on the market, we've redesigned dozens of such integrations, and we guarantee: the right protocol and contract solve the problem at its root.

When REST stops being enough

REST works well for simple CRUD operations. But as soon as a mobile app appears alongside the web interface, over-fetching begins: the mobile app requests /api/users/123 and gets a 4KB object, but only needs name and avatar. Multiply that by a list of 50 users — 200KB traffic instead of 8KB.

GraphQL solves this with selection sets. The client describes exactly the fields it needs, and the server returns only those. On a project with React Native + Next.js, we migrated from REST to Apollo Server: payload size on the main screen dropped from 340KB to 28KB — a 92% traffic savings. Our certified engineers confirm: the typical pain when adopting GraphQL is N+1 query. A resolver for the author field on a post calls SELECT * FROM users WHERE id = ? for each post in the list. On a page with 20 posts — 21 database queries. Solved with DataLoader — it batches queries and turns them into one SELECT * FROM users WHERE id IN (...).

What is tRPC and how is it better than REST/GraphQL?

If the entire stack is TypeScript (Next.js + Node/Bun), tRPC removes a whole layer of problems. You define a procedure on the server — the client gets full type-safety automatically, without code generation and without Swagger. Renamed a field in the Zod schema — TypeScript highlights all places on the frontend where it's used. tRPC reduces code by 2 times compared to REST + Swagger + openapi-typescript: no need to maintain a separate specification and generate types — everything is inferred from runtime validators. However, tRPC is not suitable if the API is consumed by third-party clients or mobile apps in other languages — in such cases we use GraphQL or REST with OpenAPI specification.

WebSocket and real-time: when SSE, when WS?

HTTP polling every 5 seconds is an illusion of real-time with up to 5 seconds delay and useless server load. For chats, live notifications, collaborative editing — WebSocket or Server-Sent Events. SSE is a one-way stream from server to client, works over ordinary HTTP, automatically reconnects. Suitable for notifications, data streaming, progress bars. WebSocket is bidirectional, needed for chats and collaborative features. Experience shows: 80% of 'real-time' tasks are solved with SSE, not WebSocket — fewer infrastructure complexities.

A typical mistake: opening a WebSocket connection for each page component. On one project, the dashboard opened 12 parallel WS connections. The correct approach is one connection manager at the application level, subscriptions through it. In our work results, we always transfer the connection scheme and a ready solution.

Swagger / OpenAPI as a contract

Documentation written after the fact becomes outdated the day after release. We write the OpenAPI 3.1 specification before development starts; it becomes the contract between frontend and backend. The frontend generates types via openapi-typescript, the backend validates incoming data using generated schemas. Contract deviation from implementation is caught on CI, not during review. For Laravel — l5-swagger or dedoc/scramble. For Node.js — @fastify/swagger or Zod + zod-to-openapi.

How to properly authenticate an API?

JWT with long-lived access tokens without rotation is a source of problems when compromised. The correct scheme: access token for 15 minutes, refresh token for 30 days with rotation on each use. Refresh token stored in an httpOnly cookie, access token in memory (not in localStorage). For inter-service communication — API Keys with scope limitations or mTLS. OAuth 2.0 with PKCE for public clients (SPA, mobile).

How to handle versioning and backward compatibility?

Breaking changes in an API without versioning break clients. Three approaches we use in projects:

We guarantee backward compatibility through automated checks (oasdiff) on CI.

How we develop APIs: step-by-step plan

1. Analysis — audit of current integrations, data schema compilation, protocol selection (REST/GraphQL/tRPC/WebSocket).
2. Contract design — OpenAPI or SDL (GraphQL) before the first line of code.
3. Development — implementation per contract, unit tests for each endpoint.
4. Load testing — k6: 500 virtual users, 10 minutes, p95 latency ≤ 200ms.
5. Deployment — CI/CD with backward compatibility check, automatic documentation publication.
6. Team training — handover of Postman collection or Playground, connection instructions.

What is included (deliverables)

• OpenAPI 3.1 specification (or SDL for GraphQL).
• Generated client types for TypeScript / Dart / Kotlin.
• Set of automated tests covering all endpoints (unit + integration).
• Load tests (k6) and report (p50/p95/p99 latency, RPS).
• Documentation in Swagger UI / Redoc / GraphiQL.
• Team training (2–4 hour workshop).
• Support for 30 days after delivery (per contract).

Our experience

• 10+ years in the API development market.
• 200+ completed projects (REST, GraphQL, WebSocket, tRPC).
• 50+ certified engineers (AWS, Kubernetes, API Design).
• Traffic savings averaging 85% when migrating from REST to GraphQL for mobile apps.
• 100% backward compatibility — not a single broken client in the last 3 years.

Timeline

API development for a typical SaaS project with 30–50 endpoints: from 3 to 8 weeks depending on business logic complexity and number of external integrations. Migration of an existing REST API to GraphQL: from 2 to 6 weeks. Adding a WebSocket layer to an existing backend: from 1 to 3 weeks. Cost is calculated individually after an audit. Get a consultation — contact us to discuss your project.