The outbox pattern and the crash in between

When you build a calendar application, creating a booking is rarely just a database operation.

Imagine this simple flow:

Bob creates a calendar booking with Alice.

When the booking is created, we need to do at least two things:

1. Save the booking in the database.
2. Notify Alice that she has been invited.

At first glance, this sounds straightforward.

But once these operations cross system boundaries, things get surprisingly interesting.

What happens if the booking is saved successfully, but the application crashes before the email is sent?

What if the email is sent successfully, but the application crashes before it records that fact?

What if two workers process the same event at the same time?

And most importantly:

How do you make retries safe when you can’t guarantee exactly-once execution?

This is where the outbox pattern comes in.

And we ended up implementing it with three separate database-level mechanisms, each solving a different problem.

The problem: database + external side effect

Let’s start with the naïve implementation.

Suppose the API receives a request to create a booking:

POST /bookings

        |
        v

Save booking to DB

        |
        v

Send notification email

        |
        v

Alice receives the invitation

Looks fine.

But there is a dangerous gap between those two operations.

Consider this:

1. Save booking
2. COMMIT
3. 💥 Application crashes
4. Send email never happens

Now the booking exists, but Alice never receives her notification.

So perhaps we should send the email first?

That creates the opposite problem:

1. Send email
2. Alice receives invitation
3. Database transaction fails

Now Alice has received an invitation for a booking that doesn’t actually exist.

Neither ordering gives us a reliable system.

The fundamental problem is that the database transaction and the external side effect cannot be committed atomically.

We can’t do:

BEGIN TRANSACTION

save booking
send email

COMMIT

because the SMTP server is not participating in our database transaction.

So we need another approach.

Enter the outbox pattern

The core idea is surprisingly simple:

Instead of sending the external message as part of the request, record the intention to send it in the same database transaction as the business operation.

The transaction becomes:

BEGIN;

INSERT INTO bookings (...);

INSERT INTO outbox_events (
    id,
    event_type,
    payload
) VALUES (
    'event-abc',
    'BOOKING_CONFIRMED',
    '{ ... }'
);

COMMIT;

Now we have:

bookings
----------------
booking-123


outbox_events
----------------
event-abc
BOOKING_CONFIRMED
PENDING

The crucial property is that both writes happen in the same database transaction.

Therefore, we get:

Booking exists
      +
Outbox event exists

or:

Neither exists

We no longer have the dangerous state where the booking exists but our system has forgotten that it needs to send a notification.

The outbox row is essentially a durable promise:

“This business event happened, and somebody needs to process its side effects.”

But who sends the event?

We don’t want the HTTP request to sit around waiting for email delivery.

Instead, a background worker periodically reads pending outbox events.

Conceptually:

                 Database
                    |
                    v
             outbox_events
                    |
                    v
              Outbox Worker
                    |
                    v
                dispatch()
                    |
                    v
              Notification
                 Service
                    |
                    v
                 SMTP
                    |
                    v
                  Alice

This gives us another useful property:

The booking request doesn’t have to synchronously perform every downstream side effect.

The work becomes asynchronous and retryable.

But now we have another problem.

Problem #2: what if two workers pick the same event?

Imagine we’re running two instances of the worker:

Worker A
Worker B

Both wake up at roughly the same time.

If both execute a simple:

SELECT * FROM outbox_events
WHERE status = 'PENDING';

they could both see:

event-abc

and both try to process it.

That could produce:

Worker A → send email
Worker B → send email

Alice gets two emails.

Not good.

SELECT FOR UPDATE SKIP LOCKED

So we use database row locking when claiming work:

SELECT ...
FROM outbox_events
WHERE status = 'PENDING'
FOR UPDATE SKIP LOCKED;

The mental model is:

                 event-abc
                    |
          ┌─────────┴─────────┐
          |                   |
       Worker A            Worker B
          |                   |
       CLAIMED              SKIPPED

Worker A locks the row.

Worker B doesn’t wait for it. SKIP LOCKED tells it to move on and find other available work.

This is excellent for preventing concurrent workers from grabbing the same work.

But there’s a subtle distinction:

SKIP LOCKED is an optimization for concurrency. It is not our ultimate correctness guarantee.

Why?

Because locks don’t protect us from crashes and retries.

Suppose Worker A claims the event and then dies.

Later, the event can be picked up again.

So we need another layer.

Layer 2: processed_events

We maintain a processed_events table.

This is where idempotency enters the picture.

When processing an outbox event, we attempt:

tryInsert(eventId)

The table has a unique constraint on the event ID, and the insert behaves like:

INSERT INTO processed_events (event_id, processed_at)
VALUES ('event-abc', ...)
ON CONFLICT DO NOTHING;

Then the worker checks the result.

Conceptually:

tryInsert("event-abc")

        |
        +---- inserted = 1
        |          |
        |          v
        |       dispatch()
        |
        +---- inserted = 0
                   |
                   v
             Already processed
             Skip dispatch

The first attempt:

event-abc
    |
    v
INSERT succeeds
    |
    v
dispatch()

A later retry:

event-abc
    |
    v
INSERT conflicts
    |
    v
inserted = 0
    |
    v
Don't dispatch again

This gives us a durable correctness guarantee:

A successfully processed outbox event should not be dispatched again.

This is different from SKIP LOCKED.

SKIP LOCKED prevents two workers from processing the same row at the same time.

processed_events protects us when the same event comes back later.

That’s why we can think about them like this:

SKIP LOCKED
    ↓
Concurrency optimization


processed_events
    ↓
Correctness guarantee

But we’re still not done.

The crash that changes everything

Let’s say our outbox event is:

event-abc

and it represents:

BOOKING_CONFIRMED

The worker processes it.

It inserts event-abc into processed_events.

Then it calls:

dispatch(event)

which eventually sends an email through SMTP.

Now imagine:

processed_events insert
        |
        v
dispatch()
        |
        v
SMTP accepts email
        |
        v
Alice receives email
        |
        v
💥 The process crashes

What happens when the service restarts?

This is where things become subtle.

The surrounding outbox transaction rolls back, so the event can be retried.

The worker processes event-abc again.

Because the processed_events row was never committed, the insert succeeds again.

Therefore:

tryInsert(event-abc)
        |
        v
SUCCESS
        |
        v
dispatch(event)

So processed_events doesn’t prevent this retry.

And if dispatch() simply sent the email again, Alice would get two emails.

This is exactly the kind of failure that makes distributed systems interesting.

We need another layer.

Layer 3: booking_notification_sends

This is where the implementation goes beyond the simplest description of the outbox pattern.

We maintain a separate table:

booking_notification_sends

The important unique key is:

(outbox_event_id, recipient_email, event_type)

So instead of asking:

“Has this event been processed?”

we ask something more specific:

“Have we already claimed this particular notification for this particular recipient?”

For example:

event-abc
alice@example.com
BOOKING_CONFIRMED

is one unique notification.

This distinction matters because one calendar event can have multiple recipients.

For example:

event-abc → Alice
event-abc → Bob
event-abc → Charlie

We want to deduplicate per recipient, not globally per event.

Claim before send

Now the notification flow looks like this:

claim(
    event-abc,
    alice@example.com,
    BOOKING_CONFIRMED
)

        |
        v

      COMMIT

        |
        v

    sendMail()

        |
        v

      SMTP

The claim is committed in its own REQUIRES_NEW transaction before we call SMTP.

This ordering is extremely important.

Let’s go back to our crash scenario.

First attempt

claim()
   |
   v
COMMIT
   |
   v
sendMail()
   |
   v
Alice receives email
   |
   v
💥 The process crashes

Now the outbox event can be retried.

The retry eventually reaches:

claim(
    event-abc,
    alice@example.com,
    BOOKING_CONFIRMED
)

But the database responds:

Already exists.

Because:

(event-abc, alice@example.com, BOOKING_CONFIRMED)

was already claimed.

So:

claim() = false

and we skip the SMTP call.

The result:

First attempt:
    claim → SUCCESS
    send email → SUCCESS
    crash

Retry:
    processed_events → retry allowed
    dispatch() → runs
    claim → FALSE
    send email → SKIPPED

Alice receives exactly one email.

This is the key insight:

The layer that protects the recipient from duplicate email is booking_notification_sends, not processed_events.

Why not just use processed_events?

Because the two tables answer different questions.

processed_events asks:

Have we already processed this event?

Its identity is essentially:

event-abc

The notification deduplication table asks:

Have we already claimed this particular notification for this particular recipient?

Its identity is:

event-abc
+
alice@example.com
+
BOOKING_CONFIRMED

That’s a much finer-grained identity.

This allows a single event to safely notify multiple recipients:

event-abc + Alice     → claimed
event-abc + Bob       → claimed
event-abc + Charlie   → claimed

without one recipient’s processing interfering with another’s.

But there is a tradeoff

This design doesn’t magically create exactly-once delivery.

In fact, it deliberately makes a different tradeoff.

Consider this sequence:

claim()
   |
   v
COMMIT
   |
   v
💥 Process dies
   |
   X
sendMail() never happens

The claim has already been committed.

So when the event is retried:

claim()
   |
   v
FALSE
   |
   v
Skip email

Alice never receives the email.

We’ve traded one type of failure for another.

Instead of risking:

duplicate email

we risk:

missing email

This happens because we claim before we send.

If SMTP doesn’t support an idempotency key, there is no atomic transaction spanning:

our database
+
SMTP provider

We can’t perfectly know whether the SMTP handoff happened if our process dies at exactly the wrong moment.

So the bias is toward:

At-most-once notification delivery per recipient, rather than risking duplicate notifications.

For calendar invitations, that’s a defensible product decision.

A duplicate calendar invitation can be confusing or annoying, while a missing email may be recoverable through the calendar itself or another notification mechanism.

But it’s important to recognize that this is a tradeoff, not a free guarantee.

What happens when SMTP actually fails?

There is another case that’s much easier.

Suppose:

claim()
   |
   v
COMMIT
   |
   v
sendMail()
   |
   v
SMTP ERROR

This is a failure we can detect.

The code releases the notification claim:

sendMail()
    |
    X
  ERROR
    |
    v
release claim
    |
    v
retry later

The outbox worker can then retry with exponential backoff.

Eventually, if the event exceeds the configured maximum number of attempts, it lands in a dead-letter state.

So the system distinguishes between:

Known failure
    ↓
Release claim
    ↓
Retry

and:

Unknown outcome
(process died)
    ↓
Claim remains
    ↓
Don't risk duplicate email

That distinction is subtle, but very important.

One more piece: the stuck-row sweep

There’s a detail that’s easy to leave out and load-bearing when you do.

When a worker claims a row, it marks it as PROCESSING. If that worker dies mid-flight, nothing moves the row back:

Worker claims row
       |
       v
   PROCESSING
       |
       v
   💥 Worker dies
       |
       v
   PROCESSING  ← forever

The polling query only looks for PENDING rows, so the event is never retried. All the retry logic above simply never runs.

So a periodic sweep resets rows that have been PROCESSING longer than a timeout:

PROCESSING for > timeout
       |
       v
   back to PENDING
       |
       v
   worker picks it up

Rows that have already exhausted their attempts are failed permanently instead of reset — otherwise the sweep and the worker would hand the same doomed event back and forth forever.

The complete mental model

At this point, we can describe the entire architecture in one picture:

                 BOOKING REQUEST
                       |
                       v
              ┌─────────────────┐
              │    Database     │
              │                 │
              │    Booking      │
              │       +         │
              │  Outbox Event   │
              └────────┬────────┘
                       |
                 SAME TRANSACTION
                       |
                       v
              ┌─────────────────┐
              │ outbox_events   │
              │                 │
              │ "Something      │
              │  needs to       │
              │  happen"        │
              └────────┬────────┘
                       |
                       v
                Outbox Worker
                       |
                       v
             SELECT ... FOR UPDATE
                  SKIP LOCKED
                       |
                       |  Prevent concurrent
                       |  workers from grabbing
                       |  same work
                       v
              ┌─────────────────┐
              │ processed_events│
              │                 │
              │ "Have I already │
              │  dispatched this│
              │  event?"        │
              └────────┬────────┘
                       |
                       v
                   dispatch()
                       |
                       v
        ┌──────────────────────────────┐
        │ booking_notification_sends   │
        │                              │
        │ "Have I already claimed      │
        │  this notification for       │
        │  this recipient?"            │
        └──────────────┬───────────────┘
                       |
                       | claim committed
                       v
                    sendMail()
                       |
                       v
                 SMTP / ZeptoMail
                       |
                       v
                     Alice

And each layer has one job:

The bigger lesson

The most important lesson we took from implementing this is that “exactly once” is usually not a single feature you turn on.

It’s a collection of carefully chosen guarantees.

Put together:

Database transaction
    +
Outbox
    +
SKIP LOCKED
    +
processed_events
    +
Per-recipient notification claims
    +
Retry / backoff
    +
Dead-letter handling

Together, these mechanisms make the system reliable despite crashes, retries, concurrent workers, and external service failures.

But they don’t make failures disappear.

They make the failures predictable.

And that’s perhaps the most useful way to think about the outbox pattern:

The goal isn’t to make failure impossible. The goal is to make failure recoverable without corrupting the business state or producing unacceptable side effects.

For a calendar, that means we can tolerate workers crashing, messages being retried, SMTP failures, and multiple worker instances running concurrently — while keeping the calendar and its notifications in a consistent, predictable state.