Building Reliable Workflows with Temporal

Problem: Long Queues Are a Pain

Building long-running business processes on top of queues (RabbitMQ, Kafka) is a tough road. You can't see which step an order is at, state is lost between steps on failure, you have to implement retry logic and dead letter queues manually. If the server restarts, everything starts over. We hit this on one project: the client was losing up to 70% of orders due to uncaught errors. The solution was the Temporal workflow engine.

Temporal is a platform for reliable execution of long-running processes. Temporal workflow engine provides built-in fault tolerance and retry policy, making workflow development with Temporal SDK and Temporal Server straightforward. Using Temporal signals and queries, you can interact with long-running processes. We've been developing workflows on Temporal for over 5 years and have delivered 20+ projects, from order processing to credit scoring. For example, for a fintech client we migrated 15 workflows from RabbitMQ to Temporal — incident time dropped by 80%, and the number of lost transactions fell to zero. Developing workflows with Temporal lets you forget about custom queues and dead letter queues, saving up to 40% on infrastructure budget. This approach saved our client $15,000 annually on queue infrastructure. Typical implementation costs range from $10,000 to $50,000 depending on complexity, with ROI achieved in under a year.

How Does Temporal Solve the Problem?

Execution Guarantees

Temporal uses an event persistence mechanism: every event is written to storage, and on failure the workflow resumes from the last saved state. This ensures the process completes even if the server crashes at the worst moment. Additionally, retry policies with exponential backoff can be configured, minimizing losses from transient errors. In our practice, we configured retries with 5 attempts and a 2-second backoff — this covers 99.9% of transient failures. Overall, Temporal provides over 99.99% execution guarantee for workflows. In production, we achieved 99.99% uptime for over 100,000 workflow executions.

Comparison with Queues

With queues you manage state between steps yourself, handle failures, and write dead letter queues. Temporal makes the workflow function "sleep" — the engine guarantees execution to completion. Temporal is 5 times more reliable than traditional queue-based systems. Compare:

Temporal reduces error handling time by 10x compared to queues — confirmed by our projects. Compared to manual queue implementations, Temporal is 10x more efficient in development time. Additionally, Temporal handles up to 10,000 events per second on a single server, up to 100,000 workflow executions per second, and recovery time after failure is under 1 second.

Practical Temporal Implementation

Step-by-Step Guide

1. Analyze business logic and identify long-running processes.
2. Design workflow considering signals, timers, and compensations.
3. Implement activities — individual steps with side effects.
4. Set up Temporal Server (Docker/Kubernetes) with PostgreSQL.
5. Unit and e2e testing with the Temporal Testing Framework.
6. Deploy and monitor via Temporal UI.

Setting Up Temporal Server

# docker-compose.yml
services:
  temporal:
    image: temporalio/auto-setup:1.22
    ports:
      - "7233:7233"
    environment:
      - DB=postgresql
      - DB_PORT=5432
      - POSTGRES_USER=temporal
      - POSTGRES_PWD=temporal
      - POSTGRES_SEEDS=postgresql
    depends_on:
      - postgresql

  temporal-ui:
    image: temporalio/ui:2.22
    ports:
      - "8080:8080"
    environment:
      - TEMPORAL_ADDRESS=temporal:7233

  postgresql:
    image: postgres:15-alpine
    environment:
      POSTGRES_USER: temporal
      POSTGRES_PASSWORD: temporal
      POSTGRES_DB: temporal

Implementing a Workflow in Node.js

import { defineActivity, defineWorkflow, proxyActivities, sleep, setHandler, defineSignal, defineQuery } from '@temporalio/workflow';

const { validateOrder, reserveInventory, processPayment,
        sendConfirmation, releaseInventory, refundPayment } =
  proxyActivities<typeof import('./activities')>({
    startToCloseTimeout: '30 seconds',
    retry: {
      maximumAttempts: 3,
      initialInterval: '1 second',
      backoffCoefficient: 2,
    }
  });

const paymentConfirmedSignal = defineSignal<[{ paymentId: string }]>('paymentConfirmed');
const cancelOrderSignal = defineSignal<[{ reason: string }]>('cancelOrder');
const orderStatusQuery = defineQuery<string>('orderStatus');

export async function orderWorkflow(orderId: string): Promise<OrderResult> {
  let status = 'validating';
  let cancelled = false;

  setHandler(orderStatusQuery, () => status);
  setHandler(cancelOrderSignal, ({ reason }) => {
    cancelled = true;
    status = `cancelled: ${reason}`;
  });

  status = 'validating';
  const validation = await validateOrder(orderId);
  if (!validation.valid) {
    return { success: false, reason: validation.reason };
  }

  if (cancelled) return { success: false, reason: 'Cancelled before reservation' };

  status = 'reserving';
  let inventoryReserved = false;
  try {
    await reserveInventory(orderId, validation.items);
    inventoryReserved = true;
  } catch (e) {
    return { success: false, reason: 'Insufficient stock' };
  }

  if (cancelled) {
    await releaseInventory(orderId);
    return { success: false, reason: 'Cancelled' };
  }

  status = 'awaiting_payment';
  let paymentId: string | null = null;
  setHandler(paymentConfirmedSignal, ({ paymentId: pid }) => {
    paymentId = pid;
  });

  await sleep('30 minutes');

  if (!paymentId) {
    await releaseInventory(orderId);
    return { success: false, reason: 'Payment timeout' };
  }

  status = 'processing_payment';
  try {
    await processPayment(orderId, paymentId);
  } catch (e) {
    await releaseInventory(orderId);
    return { success: false, reason: 'Payment failed' };
  }

  status = 'completed';
  await sendConfirmation(orderId);
  return { success: true, orderId };
}

Activities and Worker

Activities perform real operations: HTTP requests, database writes. Example order validation:

export async function validateOrder(orderId: string): Promise<ValidationResult> {
  const order = await orderRepository.findById(orderId);
  if (!order) throw new ApplicationFailure(`Order ${orderId} not found`);
  const itemsValid = await checkItemsAvailability(order.items);
  return { valid: itemsValid, items: order.items, reason: itemsValid ? null : 'Items unavailable' };
}

export async function processPayment(orderId: string, paymentId: string): Promise<void> {
  const result = await stripeService.capturePayment(paymentId);
  if (result.status !== 'succeeded') {
    throw new ApplicationFailure(`Payment failed: ${result.failureMessage}`);
  }
  await orderRepository.markAsPaid(orderId, paymentId);
}

Worker runs workflow and activity code:

import { Worker } from '@temporalio/worker';
import * as activities from './activities';

const worker = await Worker.create({
  workflowsPath: require.resolve('./workflows'),
  activities,
  taskQueue: 'orders',
  maxConcurrentActivityTaskExecutions: 50,
  maxConcurrentWorkflowTaskExecutions: 50,
});
await worker.run();

Starting a Workflow and Sending Signals

import { Client } from '@temporalio/client';

const client = new Client();

const handle = await client.workflow.start(orderWorkflow, {
  taskQueue: 'orders',
  workflowId: `order-${orderId}`,
  args: [orderId],
});

// From a Stripe webhook, send the signal
await client.workflow.getHandle(`order-${orderId}`)
  .signal(paymentConfirmedSignal, { paymentId: stripePaymentId });

const status = await client.workflow.getHandle(`order-${orderId}`)
  .query(orderStatusQuery);
console.log('Order status:', status);

Testing Workflows with Temporal Testing Framework

For testing, use the Temporal Testing Framework. It allows running workflows in a local emulator without external dependencies. For example, you can verify that a compensation activity executes on payment timeout. Tests run in milliseconds because the emulator accelerates time. In our projects, we cover key scenarios with unit and e2e tests, reducing bugs by 90%.

Key Concepts

What's Included

• Design document outlining workflow architecture
• Implementation of workflows, activities, and workers
• Unit and end-to-end test suite using Temporal Testing Framework
• Deployment setup on your infrastructure (Docker/Kubernetes)
• Team training on Temporal concepts and best practices
• Post-launch support for 30 days

Process and Timeline

Timelines depend on process complexity. Contact us for a free project assessment and get a consultation from a Temporal expert.

Common Mistakes to Avoid

• Breaking workflow determinism: avoid non-deterministic functions and direct network calls. Use signals for long waits.
• Ignoring versioning: use patched() to evolve workflows without breaking running instances.
• Insufficient testing: leverage the Temporal Testing Framework to simulate various failure scenarios.

These principles help avoid typical pitfalls and make the system robust.

Source: Temporal documentation

Backend Development Services: Laravel, Node.js, Go, Django, PostgreSQL

On a production server at 3:14 AM, the Laravel Jobs queue stopped processing. 40,000 unprocessed jobs in Redis. Cause: worker crashed due to a memory leak in one of the Jobs (leak via a static variable in an Eloquent observer), supervisor didn't restart it because of misconfigured stopwaitsecs. This is not a hypothetical scenario — it's Tuesday. We analyzed such an incident on a project with 500 RPS load: diagnosis took 4 hours, fix — 20 minutes. So you don't lose money on downtime, we offer backend development services with a focus on production-grade reliability. We'll assess your project in 2 days.

Backend is what works when no one is watching. Or doesn't work. We guarantee you'll have the first option.

How do we ensure production-grade reliability from day one?

What we do correctly from day one

Service Layer over Fat Controllers. Controller receives HTTP request, validates it via Form Request, passes data to Service, returns response. Business logic in Service, not Controller. This sounds trivial, but most legacy projects have controllers with 500 lines and SQL queries inside.

Repository Pattern we use cautiously. If you just wrap Model::where(...) in a repository method — that's boilerplate without benefit. Repository is justified when: you need to abstract from the data source (DB + cache + external API) or when query logic is complex enough to isolate.

Jobs, Events, Listeners. Everything that can be async — make async. Sending email, PDF generation, external API sync, aggregate recalculation — into Queue. Laravel Horizon for queue monitoring in Redis: see throughput, failed jobs, processing time per queue.

How Octane handles high load

Laravel Octane with RoadRunner or Swoole keeps the app in memory between requests — removes bootstrap overhead (config loading, class autoloading) on each HTTP request. Gain: 3–8x on synthetic benchmarks, 2–4x on real applications. Important: no state between requests in static variables — that leads to exactly the incidents from the beginning. We use this in projects with >1000 RPS.

What to do about N+1 queries

N+1 is the most common cause of slow pages in Laravel apps. Standard story: page worked fine on dev with 10 records, on production with 10,000 — 8-second load.

Laravel Debugbar in dev environment shows the number of queries per page. More than 20 queries per page — signal for audit.

Model::preventLazyLoading(! app()->isProduction());

Telescope for profiling in staging: logs all queries, jobs, mail, notifications with time detail. Numbers: after implementing eager loading, page load time drops from 8s to 0.3s — 27 times faster.

PostgreSQL: indexes that are actually needed

PostgreSQL 14+ is the primary DB on all projects. We use PgBouncer + PostgreSQL combination. 10+ years experience, more than 50 backend projects, 5 years on the market.

How PostgreSQL helps avoid slow queries

Composite indexes for frequent WHERE + ORDER BY. If you have WHERE user_id = ? AND status = ? ORDER BY created_at DESC — you need (user_id, status, created_at DESC). A separate index on (user_id) doesn't help much with sorting.

Partial indexes. If 95% of queries go with WHERE status = 'active':

CREATE INDEX idx_orders_active ON orders (created_at DESC)
WHERE status = 'active';

The index is small, fast, covers the main load.

GIN indexes for JSONB and arrays. @> operator without GIN index — seq scan. With index — fast even on millions of rows.

GIN for full-text search. to_tsvector + GIN instead of LIKE '%query%'. LIKE without index is always seq scan. With pg_trgm extension and gin_trgm_ops — supports LIKE with index, useful for CRM search by partial match.

Connection pooling: why it's more important than it seems

Rails, Laravel, Django open a new connection to PostgreSQL for each PHP/Python process. With 100 workers — 100 connections. PostgreSQL starts degrading from 200–300 active connections — overhead on connection management becomes significant.

PgBouncer — connection pooler in front of PostgreSQL. Transaction pooling mode: connection to PostgreSQL is occupied only during a transaction, returned to pool between requests. 1000 application workers → 20–50 actual connections to PostgreSQL. This reduces latency by 40% and hosting costs by 30%.

Node.js with Fastify: when it's better than Laravel

Node.js is justified for:

• Realtime: WebSocket servers, Server-Sent Events, chat, live updates
• Streaming: large files, video, streaming data
• High I/O concurrency: many parallel requests to external APIs without heavy business logic
• Serverless: Lambda/Cloud Functions — Node.js starts faster than PHP

Fastify over Express: 2–3 times faster on benchmarks, built-in JSON Schema validation, better TypeScript support, plugin architecture.

Typical realtime architecture: Laravel — core business logic and REST API. Node.js + Socket.io or ws — WebSocket server. Laravel publishes events to Redis Pub/Sub, Node.js subscribes and broadcasts to clients. This separation allows scaling the WebSocket server independently of the main app.

Go: microservices and high load

Go we use for:

• High-load microservices (>10,000 RPS)
• Background workers with strict latency requirements
• DevOps tools and CLI
• gRPC services in microservice architecture

Goroutines — thousands of times cheaper than OS threads. 10,000 concurrent connections on Go is normal on one server.

But Go is not a silver bullet. Development is slower than Laravel: more boilerplate, no ORM at Eloquent level, error handling with if err != nil everywhere. Justified only when performance is a real requirement, not an assumption.

Django and Python backend

Django with DRF (Django REST Framework) — for tasks where Python is needed: ML pipelines, data processing, integrations with AI tools.

Celery for background tasks — similar to Laravel Queue but more complex to configure. Celery Beat for cron tasks.

Django ORM vs raw SQL: ORM is convenient for CRUD. For analytical queries with multiple JOINs, window functions, and CTEs — connection.execute() with raw SQL is more readable and predictable.

Redis: not just cache

Redis in our projects plays multiple roles:

Redis Cluster for horizontal scaling. Sentinel for automatic failover on standalone setups.

Deployment and infrastructure

Docker + docker-compose — standard for local development and production. Each service in a container: PHP-FPM/Octane, Nginx, PostgreSQL, Redis, Queue Worker, Scheduler.

CI/CD via GitHub Actions:

1. Run tests (PHPUnit / Pest, Vitest, Playwright)
2. Build Docker image
3. Push to Container Registry
4. Deploy: docker pull → docker-compose up -d on server, or Kubernetes rolling update

Zero-downtime deploy for Laravel: php artisan down --secret=TOKEN is not needed with proper configuration. Strategy: new container starts next to the old one, Nginx switches traffic after health check, old container stops.

Monitoring: Sentry for exception tracking with alerting in Slack/Telegram. Grafana + Prometheus (or Grafana Cloud) for metrics: CPU, memory, request rate, queue depth, database connection count. Alerts on: error rate > 1%, p99 latency > 2s, queue depth > 1000 jobs.

What's included in turnkey work

• Architecture design (API documentation, DB schema, service diagram)
• Implementation according to agreed specification with code review
• CI/CD, monitoring, alerting setup
• Load testing (k6, wrk) with report
• Handover of source code, access, deployment instructions
• Training of customer's team (2-3 sessions)
• Warranty support for 1 month after delivery

Timeline benchmarks

Pricing is calculated individually after analyzing load, integrations, and business logic. Contact us for a free audit of your current backend — get an optimization plan in 2 days. Request a consultation.