Praveen

Order Lifecycle State Machine

One Seam for Money and Stock: The Order Lifecycle Behind Zesty

Six different callers can change a restaurant order. Here is the transaction boundary they all go through, the invariants it enforces, and why side effects live in an outbox instead of the request.

Praveen Gonthina

12 min read

On this page8 sections

A restaurant order is not a row you update. It is a small, hostile distributed system that happens to fit on one screen.

By dinner service the same order can be touched by a guest phone that scanned a QR code, a kitchen display, a counter tablet, a manager's laptop, a mobile client replaying a queue it built while the WiFi was down, and a payment webhook arriving from a provider minutes after the customer walked out. Every one of them wants to change state. Most of them are allowed to, under some conditions, and each change moves money or stock or both.

I build Zesty with my team, and the single most valuable architectural decision in that product was refusing to let any of those callers write order state directly. This is what an order lifecycle state machine actually buys you, what it enforces, and what it costs.

The problem is not the happy path

Anyone can ship create, serve, and pay. The trouble is the transitions nobody demos.

A bill gets printed and then a table wants dessert. An order is voided after the kitchen already cooked it. A payment webhook arrives twice because the provider retried. A waiter's phone was offline for twenty minutes and now wants to apply three actions in the order they actually happened. A refund is issued the next morning against a shift that was reconciled and closed last night.

Where restaurant order bugs actually live

After settlement

Someone edits totals on an order that has already been paid.

During a void

Stock is restored for food that was already served and eaten.

On replay

An offline queue applies the same action twice and the till is wrong.

Across roles

The UI hides a button the API still happily accepts.

At shift close

A late refund lands against a shift that was already reconciled.

In the webhook

The provider retries, and the order is marked paid twice.

None of those are exotic. All of them are cheap to cause when six callers each hold their own copy of the rules.

One door, named intents

The fix is structural, not clever. In Zesty, post-creation state changes do not happen through scattered update calls. Callers submit a named intent through a trust-specific adapter, and one lifecycle component decides.

It exposes exactly two behaviours.

The first loads order facts in batches and returns a schema-versioned projection for each order: a description of the order and, for every intent the system knows about, whether it is currently allowed and why not if it is denied. Read models embed that projection, so a client can render labels and controls without reconstructing policy or firing a query per order.

The second executes one command. It serialises a single order, deduplicates the command ID, validates the expected revision against current facts, and commits the state change and all required effects atomically. The outcome reports whether the command was applied or deduplicated, and returns the new revision, the order, the projection, and the durable effect identifiers.

What every command carries

Organisation and order ID

Scope

No ambient tenant from a session

Command ID

Deduplication

The same ID applied twice changes nothing

Expected revision

Conflict detection

Stale means reject, not retry

Named intent

What the caller wants

Serve, bill, settle, void, refund

Occurrence time

When it really happened

Not when the server heard about it

Captured shift

Which till owns this

Survives a later shift close

The projection is the part teams skip, and it is the part that pays for itself. Because the server returns an explicit allowed or denied outcome for every known intent with a stable reason code, the client does not reimplement policy to decide which buttons to draw. The kitchen display, the counter, the guest page, and the mobile app all read their controls from the same answer. A rule change ships once.

It also fails closed. Every known intent has an explicit outcome. There is no "unknown, probably fine" branch, because that branch is where money leaks.

The invariants are the product

The rules that matter are not features. Writing them down as invariants rather than as conditionals scattered through handlers is what makes them testable and what makes an argument about correctness resolvable.

These are the ones I would carry into any system that moves money and stock together.

  1. Paid is derived, never asserted. Paid state comes from successful net settlement, for every order channel. Marketplace and webhook adapters must supply trusted external settlement evidence; they cannot set paid directly. No caller gets to simply claim that money arrived.
  2. Settled orders stop changing. Once settlement is pending or successful, item, discount, tax, fee, and charge edits that change the total are denied. Not hidden in the UI. Denied at the seam.
  3. Compensation follows the food, not the database. Voiding restores deducted inventory only from received or accepted states. After that the ingredients are genuinely gone, so a later void retains consumption.
  4. Reopening re-deducts, once. Reopening requires an unsettled cancelled order. If the void compensated inventory, reopening re-deducts it in the same transaction and only if stock is actually available. A void that retained consumption is not deducted twice.
  5. Refunds are money, not inventory. A refund cannot exceed successful net settlement, never restores inventory automatically, and reconciles customer spend and loyalty against retained revenue.
  6. Everything commits together. Order status, totals, payments, table-session settlement, inventory, actor fields, audit records, profit facts, command deduplication rows, and outbox rows commit in one serialisable transaction, or none of them do.
  7. A stale revision is a conflict. It is never an implicit retry against whatever state the order happens to be in now.
  8. Projections are complete. Every known intent has an explicit allowed or denied outcome with a stable reason code.

Point three is the one that surprises people. "Cancel should put the stock back" sounds obviously right until a cancelled order means a plate already carried to a table. Compensation is a business decision about physical reality, not a database rollback.

Traps this design exists to close

Trusting the client

Problem: A caller marks an order paid because its own checkout screen succeeded.

Better move: Derive paid state from settlement evidence, and make external adapters prove it.

Last write wins

Problem: Two staff act at once and the second silently overwrites the first.

Better move: Carry an expected revision. A stale revision is a conflict the caller must resolve.

Retry means duplicate

Problem: A flaky network turns one tap into two settlements.

Better move: Client-generated command IDs, recorded server-side, returning the original outcome.

Effects outside the transaction

Problem: Stock is deducted, the notification send fails, and someone unwinds it by hand.

Better move: Write effects to an outbox in the same transaction and deliver them separately.

UI as authorization

Problem: A hidden button is treated as a permission check.

Better move: Resolve membership, branch, permissions, and shift server-side before execution.

Offline as a special case

Problem: A separate code path for replayed actions, with its own subtly different rules.

Better move: Replay through the same seam, resolving local IDs first and rechecking authority.

Trust adapters, one seam

Different callers deserve different authentication and wildly different trust, but not different policy.

The protected lifecycle route resolves the caller's current membership, effective branch, permissions, and captured shift before execution. Guest-token and OTP ordering from a scanned table QR is a separate authentication adapter. Marketplace ingestion is another. Payment webhooks are another again. All of them authenticate differently, and all post-creation state changes cross the same lifecycle seam.

That split is worth being explicit about. The adapter answers "who is this and what may they attempt." The lifecycle answers "is that legal for this order right now, and what happens if it is." Collapsing those two questions into one layer is how you end up with a webhook handler that quietly knows how to adjust inventory.

Side effects belong in an outbox

Committing state and firing side effects in the same breath is the classic way to lose one of them. Redis and notification providers cannot participate in a Postgres transaction, so you have to choose where the gap goes.

We wrote the decision down, including what we rejected.

Where to put the delivery gap

Rejected
  • Publish before commit: announces state that may roll back
  • Publish after commit: loses effects if the process exits between
  • Process-local emitter: cannot cross API replicas, no recovery on restart
  • Deliver inside the request: couples response time to the provider
Chosen
  • Outbox row written in the same serialisable transaction
  • A dedicated worker claims ready rows and delivers them
  • Redis publish and subscribe as the shared live transport
  • Failures become a queue you can inspect on Monday

Concretely: lifecycle execution writes one outbox message per live event or notification effect, inside the transaction that changed the order. A dedicated order worker then claims ready rows with row-level skip-locked selection, prevents a later revision for an order from overtaking an unfinished earlier one, renews its claim through bounded leases, retries with backoff, and dead-letters effects that exhaust their retries. Live order events publish through Redis so every API replica and every subscriber shares one transport. Staff and customer notification adapters receive the durable effect ID and use it as their idempotency key.

The consequences are worth stating plainly, because they are the price.

  • A committed command has durable delivery intent even if the API process dies immediately afterwards.
  • Delivery is at least once, so every consumer must treat the effect ID as an idempotency key. There is no exactly-once to hide behind.
  • Deployment now includes a worker, an outbox table, and Redis, all of which must be monitored together. Production readiness fails if the events Redis connection is missing or unreachable, which is deliberate: silently degrading to no live updates is worse than refusing to start.
  • Dead-lettered rows require operational review. They are not discarded.

That last point is the operational win. Delivery failure becomes visible and recoverable instead of silent. A dead-letter queue someone looks at on Monday beats a customer telling you their receipt never arrived.

Offline replay is where the design gets tested

The mobile client keeps status, serve-and-bill, and collect-payment commands in durable order when the network is gone. That is easy to describe and hard to get right, because replay is not just re-sending.

On replay, local order IDs have to resolve to server IDs first, since the order may have been created offline too. Membership and branch authority are rechecked at replay time, because a staff member's permissions may have changed in the interval. But the shift captured at the moment the action actually happened is retained, so a payment collected at 9pm does not land in the next morning's till. Conflicts on revision or on captured shift go to manual review rather than being force-applied.

That last choice matters more than it looks. The tempting move is to make replay always succeed. The correct move is to let it fail loudly into a queue a human owns, because a silently reconciled till is worse than an obviously broken one.

What this costs

I would not pretend this is free.

The lifecycle component is the most complex part of the codebase and needs the most tests. Adding a transition means adding it to the projection, the policy, and the test matrix, not just wiring a button. Every caller has to be adapted rather than allowed to write directly, including the ones added later under deadline pressure. Serialisable transactions and per-order serialisation put a ceiling on concurrent writes to a single order, which is fine for restaurants and would not be fine for something else.

It is worth it exactly when the record controls something physical or financial. For a blog CMS this is absurd overhead. For orders, inventory, gate passes, or invoices, the alternative is a category of bug you cannot test your way out of after the fact.

Before you let a second caller write to the same record

Policy

  • The named transitions are written down, not implied by endpoints
  • Every transition has an explicit allowed or denied answer with a reason code
  • Rules about money and stock are stated as invariants you can test
  • Compensation is a documented product decision, per transition

Mechanics

  • Commands carry an idempotency key recorded server-side
  • Commands carry the revision the caller believed it was acting on
  • State and effects commit in one transaction, effects via an outbox
  • Delivery is assumed at least once, and consumers are idempotent
  • Authorization is resolved server-side before execution, every time

The general lesson

Nothing here is specific to restaurants. The same shape shows up in Trax gate passes, where a transfer must not be cleared twice by two security staff, and in Soma invoicing, where a document that has been issued cannot quietly change its totals.

The question to ask of any record in your system is simple: how many places can change this, and what physical or financial fact does each change imply? If the answer is more than one place and the implication is real, you do not have a table. You have a lifecycle, and it deserves one door.

For how these services reach production and stay verifiable, read how I actually deploy. For the app that drives most of these commands from a phone, see what shipping Flutter and Expo taught me.

Give it one seam, and every later argument about correctness happens in one file instead of six.

Frequently asked questions

One authoritative place that decides whether a requested change to an order is allowed and commits that change together with every side effect it implies. Instead of each screen writing its own updates, callers submit a named intent such as serve, bill, settle, void, or refund, and a single component answers yes or no and performs the whole transaction.

Building a SaaS product?

Tell me what you're shipping. I'll help you scope an MVP that can grow into a real product.