Event-driven architecture in practice, for SaaS teams
Event-driven architecture is a design where components communicate by publishing events – facts about something that happened, like “invoice.paid” – instead of calling each other directly. Producers don’t know who consumes their events; consumers react in their own time. It decouples parts of a system and absorbs load spikes, at the cost of eventual consistency and harder debugging.
Most explanations stop at the diagram with boxes and arrows. The work is in the details: choosing between a queue and a stream, not losing events when a transaction rolls back, and handling the same event arriving twice. That’s what this guide covers, with the patterns I use in SaaS systems and the cases where I’d avoid events entirely.
Three things people mean by “event-driven”
Martin Fowler pointed out that “event-driven” covers several different patterns, and mixing them up causes most of the confusion. The ones that matter in practice:
- Event notification. The event says something happened and carries little data – an ID and a type. Consumers call back for details if they need them. Simple, but it can create a burst of follow-up calls to the producer.
- Event-carried state transfer. The event carries the data consumers need, so they can keep their own copy and never call back. Better decoupling and resilience, at the price of duplicated data and larger events.
- Event sourcing. The event log is the system of record; current state is derived by replaying events. Powerful for audit-heavy domains like ledgers, but a big commitment that changes how you model everything. Don’t adopt it by accident.
Most SaaS systems need the first two and not the third. When someone proposes “going event-driven”, ask which of these they mean.
Queues vs streams
The transport choice shapes everything downstream, and the two families behave very differently:
- Queues (SQS, RabbitMQ, Sidekiq or Celery on Redis, Google Pub/Sub in its usual mode): each message is delivered to one consumer in a group, and it’s removed once acknowledged. Great for distributing work – send this email, render this PDF, sync this record. Scaling is adding workers. Replaying old messages usually isn’t possible.
- Streams or logs (Kafka, Kinesis, Redpanda, Redis Streams): events are appended to a durable, ordered log and kept for a retention period. Each consumer group tracks its own position, so many independent consumers can read the same events, and a new consumer can replay history. Ordering is guaranteed within a partition, not across the whole topic.
- Fan-out sits between them: SNS to multiple SQS queues, or a RabbitMQ exchange bound to several queues, gives each subscriber its own queue of the same events.
My default for a SaaS team without a platform group: a managed queue (or the job system you already run) for work distribution, plus fan-out for the handful of domain events several parts of the system care about. Reach for Kafka when you have many consumers, need replay, or are moving serious volume – and have someone to operate it, or pay for a managed service.
The dual-write problem and the outbox pattern
The most common bug in event-driven systems is the dual write. Code saves a record to the database, then publishes an event to the broker. If the process crashes between the two, or the publish fails, the database says the subscription was created and nothing downstream ever hears about it. Publish first and the opposite happens: an event for a record that rolled back.
The transactional outbox pattern fixes this. Instead of publishing directly:
- In the same database transaction as the business change, insert a row into an outbox table: event ID, type, payload, created_at.
- A separate relay process reads unpublished outbox rows in order, publishes them to the broker, and marks them sent.
- If the relay crashes after publishing but before marking, it publishes again on restart. So delivery is at-least-once, which is why consumers must be idempotent.
- Clean up old outbox rows on a schedule.
The relay can be a polling job (SELECT … FOR UPDATE SKIP LOCKED works well in Postgres) or change data capture reading the database’s write-ahead log, with Debezium as the usual tool. Polling is simpler and enough for most SaaS volumes.
Idempotent consumers
Most brokers give you at-least-once delivery by default. Kafka transactions and SQS FIFO deduplication narrow the gap, but neither covers side effects outside the broker, so exactly-once processing is still something you build at the consumer. Assume every event will eventually arrive twice, and design for it:
- Give every event a unique ID at creation (in the outbox row), and keep it through every retry.
- Record processed IDs in a processed_events table with a unique constraint, inserted in the same transaction as the consumer’s side effects. A duplicate hits the constraint and is skipped.
- Prefer naturally idempotent operations – “set status to paid” rather than “increment paid count”, upserts keyed on a business ID.
- Make external side effects idempotent too. Pass idempotency keys to payment APIs; Stripe supports them for exactly this reason. This is a big part of handling Stripe webhooks correctly, where retries and duplicates are normal.
Ordering, retries and dead letters
- Ordering: only guarantee it where you need it, and scope it. In Kafka, partition by the entity key (account_id, subscription_id) so events for one entity arrive in order. In queues, either use FIFO queues with a group key or make consumers tolerate out-of-order events by checking a version number or timestamp and ignoring stale updates.
- Retries with backoff: transient failures (a timeout calling a third party) should retry with exponential backoff and jitter. Permanent failures (invalid payload) should not retry forever.
- Dead-letter queues: after a set number of attempts, move the event aside, alert someone, and keep processing the rest. A DLQ nobody monitors is a silent data-loss mechanism.
- Poison messages in streams: one bad event at the head of a partition blocks everything behind it. Catch, park it in a DLQ topic, and move on.
- Schema evolution: events are a public contract. Add fields, don’t rename or remove them; version the event type when the meaning changes. A schema registry helps once several teams publish.
A concrete SaaS example
A subscription is upgraded. In a synchronous design, the upgrade endpoint updates the plan, changes entitlements, emails the customer, updates the CRM, recalculates usage limits and posts to Slack. Any one of those failing or being slow breaks the upgrade.
In an event-driven design, the endpoint updates the plan and entitlements in one transaction and writes a subscription.upgraded event to the outbox. Consumers then send the email, sync the CRM and notify sales independently. The CRM being down delays the CRM update, nothing else. The customer gets their upgrade instantly.
Notice what stayed synchronous: entitlements. The customer expects to use the new features immediately, so that change belongs in the transaction. Deciding what must be consistent now and what can be eventually consistent is the core design skill in event-driven architecture.
When not to use event-driven architecture
- When the caller needs the answer. Validation, authorization and pricing calculations are request–response. Modeling them as events with a reply queue adds latency and complexity for nothing.
- When the work must be atomic. If two changes must succeed or fail together, keep them in one database transaction in one service.
- For a small team with one app. Background jobs and in-process events inside a modular monolith give you most of the decoupling with none of the broker operations.
- When nobody will own observability. Without correlation IDs, consumer lag metrics and DLQ alerts, an event-driven system fails silently. If you can’t staff that, stay synchronous.
- To avoid deciding boundaries. Events between badly drawn services just make the coupling asynchronous and harder to see.
Observability you need from day one
Propagate a correlation ID from the originating request through every event and consumer log line. Track consumer lag (how far behind each consumer is) and alert on it. Track DLQ depth. Record the event ID in every log line so you can trace a single event through the system. With those four, most incidents become a query rather than an investigation.
If you’re designing an event-driven system or untangling one that’s gone wrong – lost events, duplicate charges, consumers hours behind – that’s squarely in my software architecture consulting work. For the bigger question of whether to split services at all, start with microservices vs monolith.
Questions
What is event-driven architecture?
A software design where components communicate by publishing events – records of something that happened – rather than calling each other directly. Consumers subscribe to the events they care about and react independently. It decouples components and absorbs load spikes, at the cost of eventual consistency and more complex debugging.
What is the outbox pattern?
A way to publish events reliably alongside a database change. The event is written to an outbox table in the same transaction as the business data, and a separate relay publishes it to the broker. This avoids losing events or publishing events for rolled-back changes. Delivery is at-least-once, so consumers must be idempotent.
What is the difference between a message queue and an event stream?
A queue delivers each message to one consumer and deletes it once acknowledged, which suits distributing work. A stream such as Kafka keeps events in an ordered, durable log for a retention period, so many consumer groups can read the same events independently and replay history.
How do you make event consumers idempotent?
Give each event a unique ID, record processed IDs in a table with a unique constraint inside the same transaction as the consumer’s changes, and skip duplicates. Prefer operations that are naturally idempotent, such as upserts and setting a status, and pass idempotency keys to external APIs.
Is Kafka required for event-driven architecture?
No. Many SaaS systems run event-driven designs on a managed queue, a fan-out service, Redis Streams or their existing background job system. Kafka is worth it when you need many independent consumers, replay of history or very high throughput, and can operate it or pay for a managed version.
Losing events or processing them twice?
In a two-week sprint I’ll trace your event flows with the team, add an outbox and idempotent consumers where they matter most, and leave the rest written down.
GMT+2 · Full EU overlap · US East mornings