When most people think about AI systems, they think about prompts, models, and responses. But once you start building real products, the interesting problems move elsewhere. The hard part is not always generating an answer. The hard part is making sure the system behaves reliably when requests fail, retries happen, jobs arrive out of order, users change their minds, and downstream services become slow or unavailable.

That is where event-driven design becomes powerful.

In this article, I want to explore how I think about production AI systems when they are built around events, queues, state machines, and background workers instead of a single synchronous request-response path. This is the kind of architecture that becomes necessary once your product needs to be dependable, scalable, and easy to operate in the real world.

Why event-driven systems matter in AI

Many AI applications begin as simple synchronous flows.

A user submits input. The model processes it. A result is returned.

That works for small demos. But real products often need more than that. They need:

  • asynchronous processing

  • retry handling

  • long-running workflows

  • audit trails

  • background execution

  • graceful failure recovery

  • status tracking

  • task coordination across multiple services

If you do everything synchronously, the system becomes fragile very quickly. A slow model call can block the user experience. A temporary outage can break the whole workflow. A downstream API delay can cause timeouts. And if there is no durable state, it becomes hard to know what happened.

Event-driven architecture solves a lot of this by turning the system into a sequence of durable actions instead of one big fragile request.

My core principle: every important action should have a state

One of the biggest lessons I have learned is that reliable systems are stateful systems. If something matters, it should exist somewhere as a tracked state, not just as an in-memory assumption.

That means tracking things like:

  • job status

  • retry count

  • workflow stage

  • error state

  • external service response

  • human review state

  • final output version

If you cannot tell where a job is in the pipeline, you cannot operate the system confidently.

A simple state model

from enum import Enum

class JobStatus(str, Enum):
    PENDING = "pending"
    PROCESSING = "processing"
    RETRYING = "retrying"
    COMPLETED = "completed"
    FAILED = "failed"

This looks basic, but it gives you a shared language for the whole workflow.

Start with a durable job record

Whenever I build an AI workflow, I like to store a durable record of the task before processing begins. That record becomes the source of truth for the workflow.

Example job schema

from datetime import datetime
from typing import Optional
from dataclasses import dataclass

@dataclass
class AIJob:
    job_id: str
    user_id: str
    input_text: str
    status: str
    attempts: int = 0
    error_message: Optional[str] = None
    created_at: datetime = datetime.utcnow()
    updated_at: datetime = datetime.utcnow()

In a real system, this would live in a database, not just memory. The point is to make state durable, inspectable, and recoverable.

Idempotency is not optional

In event-driven systems, retries happen. Messages can be delivered more than once. Workers can crash mid-process. External APIs can fail after partially completing work.

That is why idempotency is essential.

If a job is processed twice, it should not create duplicate output or inconsistent state.

Example idempotent handler

def process_job(job_id, db):
    job = db.get_job(job_id)

    if job.status == "completed":
        return job.result

    if job.status == "processing":
        return "Job already in progress"

    db.update_job(job_id, status="processing")

    result = run_model(job.input_text)

    db.save_result(job_id, result)
    db.update_job(job_id, status="completed")

    return result

The handler checks state before acting. That is a simple but important form of safety.

Background workers give you breathing room

A major advantage of event-driven design is that the user does not have to wait for every step in the browser or API request. You can accept the request quickly, persist the work, and process it in the background.

That improves:

  • latency perception

  • reliability

  • throughput

  • user experience

Simplified queue pattern

def enqueue_job(queue, job):
    queue.push({
        "job_id": job.job_id,
        "user_id": job.user_id,
        "input_text": job.input_text
    })

Then a worker picks up the job later.

def worker_loop(queue, db):
    while True:
        message = queue.pop()
        if not message:
            continue

        job_id = message["job_id"]
        process_job(job_id, db)

This separation is useful because the request path stays fast while the work path handles complexity.

Retries need policy, not guesswork

Retries are useful, but blind retries are dangerous. If a failure is permanent, retrying repeatedly just burns time and resources.

I usually think about retries using these categories:

  • transient failures

  • rate limits

  • timeout errors

  • invalid input

  • downstream service outages

  • internal processing errors

Each one should have a different response.

Retry with backoff

import time

def retry_with_backoff(fn, max_attempts=5):
    delay = 1

    for attempt in range(max_attempts):
        try:
            return fn()
        except Exception as e:
            if attempt == max_attempts - 1:
                raise e
            time.sleep(delay)
            delay *= 2

This is a simple form of exponential backoff. In production, you would likely make it more selective and better instrumented.

Not every failure should be retried

One of the most useful distinctions in workflow design is the difference between retryable and non-retryable failures.

Retryable

  • timeouts

  • temporary service unavailability

  • throttling

  • network errors

Non-retryable

  • malformed input

  • missing required fields

  • permission failures

  • invalid business rules

If the system knows the difference, it can behave much more intelligently.

class RetryableError(Exception):
    pass

class ValidationError(Exception):
    pass

That distinction helps both automation and debugging.

Observability is part of the product

A lot of teams treat logs and metrics as operational extras. I think they are core product features. If you cannot see what your system is doing, you cannot improve it with confidence.

For AI workflows, I like to track:

  • job created

  • job started

  • job retried

  • job completed

  • job failed

  • model latency

  • external API latency

  • retry count

  • queue time

  • worker processing time

Example logging pattern

import logging

logger = logging.getLogger(__name__)

def log_job_start(job_id):
    logger.info(f"job_started job_id={job_id}")

def log_job_complete(job_id, latency_ms):
    logger.info(f"job_completed job_id={job_id} latency_ms={latency_ms}")

In a real environment, those logs become crucial for debugging workflows that span multiple systems.

Human in the loop can be part of the flow

Not every AI output should be fully automated. Some workflows benefit from human review before final action. That is especially true when the result is high impact, ambiguous, or externally visible.

In an event-driven system, human review can be modeled as just another state.

Example flow

submitted
  -> processing
  -> needs_review
  -> approved
  -> completed

This allows the workflow to continue without forcing everything into a fully automatic path.

Example review checkpoint

def submit_for_review(job_id, db):
    db.update_job(job_id, status="needs_review")

That state can later be resumed once a human confirms it.

State machines make workflow logic easier to reason about

As workflows grow, simple if-statements can become messy. A state machine gives you a cleaner way to define allowed transitions.

Example transitions

valid_transitions = {
    "pending": ["processing", "failed"],
    "processing": ["completed", "retrying", "failed"],
    "retrying": ["processing", "failed"],
    "needs_review": ["processing", "completed", "failed"],
}

This makes the workflow easier to audit and reduces accidental invalid transitions.

Why this matters for AI products specifically

AI workflows are often less deterministic than traditional application flows. The model may return different outputs for similar inputs. External tools may behave differently from one run to the next. Retrieval may vary. Timeouts may happen.

That means the system around the model matters just as much as the model itself.

If the workflow is weak, the product feels unreliable even when the model is good.

If the workflow is strong, the product can tolerate uncertainty much better.

That is why I think serious AI engineering should spend as much time on orchestration as on model selection.

A simple end-to-end pattern

Here is a stripped-down version of a production-style flow:

def submit_request(db, queue, user_id, input_text):
    job = create_job(db, user_id, input_text)
    enqueue_job(queue, job)
    return {"job_id": job.job_id, "status": job.status}

Then:

def run_worker_job(job_id, db):
    try:
        db.update_job(job_id, status="processing")
        result = run_model_for_job(job_id, db)
        db.save_result(job_id, result)
        db.update_job(job_id, status="completed")
    except RetryableError as e:
        db.increment_attempts(job_id)
        db.update_job(job_id, status="retrying", error_message=str(e))
        raise
    except Exception as e:
        db.update_job(job_id, status="failed", error_message=str(e))
        raise

This is not a complete framework, but it shows the pattern clearly: persist state, process asynchronously, track failures, and make transitions explicit.

The trade-offs are real

Event-driven systems are powerful, but they do add complexity.

Costs

  • more infrastructure

  • more state management

  • harder debugging if observability is poor

  • more moving parts

  • more design work up front

Benefits

  • better reliability

  • better scalability

  • better recovery from failure

  • better handling of long-running tasks

  • better support for human review and retries

For me, that trade-off is usually worth it when the workflow matters enough to be operationally serious.

What I would optimise first

If I were building a production AI workflow from scratch, I would focus on these things first:

  • durable state

  • idempotent handlers

  • clear retry logic

  • structured logging

  • explicit job status

  • safe failure handling

  • observability from day one

That foundation makes every future improvement easier.

Final thoughts

The most important lesson I have learned about production AI systems is that the model is only one piece of the puzzle. The workflow around the model is what makes the system dependable.

If you want AI products that can survive real users, real failures, and real scale, you need to think in terms of events, state, retries, and recoverability. That is where reliability is built.

For me, event-driven design is not just an engineering preference. It is a way of making AI systems behave like real products instead of fragile demos.