Ivan Ball-llovera

↑↓ to navigate Enter to open "…" all these words ANDOR to combine

Architecture Decision Record

ADR-054: Choreographed Saga Compensation with a Reconciliation Backstop

Status

Accepted (2026-07-25). Amended (2026-07-28): Store's reconciliation sweep now derives from PeriodicBackgroundService, so the shared-loop and adoption paragraphs are rewritten and the duplicated-scaffolding trade-off is dropped; the scope/IUnitOfWork and maxReplicas citations are corrected. Amended (2026-08-01): SafeDomainEventHandler<TDomainEvent> logs and rethrows, so the redelivery bullet, the throwing-is-correct rationale and the redelivery trade-off are corrected: the framework packages no swallow, the one hand-rolled swallow is not an instance of the base class, and the redelivery blast radius is the whole save's local batch. Amended (2026-09-01): an order line whose inventory row is missing is no longer silently skipped, so the best-effort trade-off is rewritten around the returned unmatched ids and the handler warning; the RestoreInventory and maxReplicas citations are corrected. Revised (2026-09-11): the reconciliation sweep now runs two passes per cycle, so the sweep bullet records the second one (expiring stranded unpaid orders straight to Cancelled so the existing compensation releases their stock) alongside the Stripe reconciliation it already described; the PaymentReconciliationSettings path, the CheckOutHandler transaction range, the index, appsettings.json, maxReplicas and every PaymentReconciliationService citation are re-anchored. Revised again (2026-09-11): unpaid-order expiry is now armed per order at checkout, so the sweep bullet, the "no new infrastructure" rationale and the bounded-inconsistency trade-off record the scheduled Sales.ExpireUnpaidOrder deadline that runs ahead of the sweep, and the sweep's expiry pass is described as the backstop it now is; every PaymentReconciliationService, CheckOutHandler and OrderPaymentFailedSagaHandler citation is re-anchored, and the two sentences describing that saga handler as an inline notification are corrected. See the Revision below. Revised (2026-09-19): CancelOrderHandler no longer performs the guarded transition and nothing else, so the compensation bullet records the one compensating action that now runs inside the command (retiring the provider's checkout session) and the already-paid case that refuses the cancellation outright; the CancelOrderHandler citations are re-anchored with it. Revised (2026-09-25): a new trade-off records that OrderCancelledSagaHandler returns early, saving nothing, when an inventory invariant rejects an increase (a return rather than a throw, so the event is not redelivered); the checkout deadline is armed through a CheckOutDeadlineScheduler collaborator the transaction delegate calls, so the Revision (2026-09-11) cites it; the adoption paragraph points to ADC's one compensating step (the session-asset orphan-blob delete, ADR-123); every Store citation is re-anchored (CheckOutHandler, OrderCancelledSagaHandler, OrderPaymentFailedSagaHandler, Order, InventoryRestorationDomainService, appsettings.json, maxReplicas). Revised (2026-10-01): the restoration marker and the stock increase now commit in one transaction, marker first, with the stock moved by an atomic relative update, so the Decision and Rationale bullets are corrected and a trade-off records the discarded failed Result; CancelOrderHandler retires the session through PaymentSessionRetirement, the reconciliation pass is gated by PaymentProof, and PeriodicBackgroundService moved with Common now subclassing it; two Revision (2026-09-11) statements are marked as no longer holding; every current-state citation is re-anchored. See the Revision below. Revised 2026-10-07: anchors refreshed after the v1.233.0 release. Revised 2026-10-21: anchors refreshed after the Common and Store changes that moved DbContextFactory, OutboxProcessor, appsettings.json and main.bicep lines. Revised 2026-10-26: anchors refreshed after the Common and Store changes that moved DomainEventSaveChangesInterceptor, OutboxProcessor, CheckOutHandler, appsettings.json, main.bicep, ProcessPaymentWebhookHandler and VerifyPaymentHandler lines.

Context

Checkout spans a boundary no transaction covers. CheckOutHandler commits the order insert, the cart transition and the atomic conditional stock decrements in one local transaction (MMCA.Store/Source/Modules/Sales/MMCA.Store.Sales.Application/ShoppingCarts/UseCases/CheckOut/CheckOutHandler.cs:122-181, run through ExecuteInTransactionAsync at :80-82), but the money moves at Stripe and the confirmation arrives later, as a webhook, from outside the database. Two failure shapes follow directly:

  • A later step invalidates an earlier one. Cancelling an order leaves stock that checkout already committed. The decrement cannot be rolled back, because its transaction closed minutes ago.
  • The external confirmation never arrives. A dropped webhook (endpoint outage past Stripe's retry window, a misconfigured secret) leaves the order in PaymentInitiated forever with its stock held.

Neighbouring records answer adjacent questions and deliberately not this one. ADR-003 gets an event out of the process at least once. ADR-021 stops a broker redelivery from being applied twice at the consume edge. ADR-006 decides database-per-service and records "no cross-database transactions, no two-phase commit" only as a cost it accepts. None of them says what a multi-step workflow does when step two fails or step three never reports back. That is the decision recorded here: consistency without a distributed transaction.

Decision

Multi-step workflows are choreographed sagas: each step raises a domain event, and the follow-up or compensating action lives in its own handler. A periodic reconciliation sweep is the saga-timeout backstop for steps that depend on an external system.

  • Compensation is a domain-event handler, and the one exception has to run before the commit. CancelOrderHandler performs the guarded transition and saves (.../Orders/UseCases/Cancel/CancelOrderHandler.cs:60-81, the transition at :80), and owns exactly one compensating action itself: retiring the payment provider's checkout session (CancelOrderHandler.cs:74, RetirePaymentSessionAsync at CancelOrderHandler.cs:104-140). That call runs only for an order still in PaymentInitiated with a session id (CancelOrderHandler.cs:106-109); it goes through the shared PaymentSessionRetirement.RetireAsync (CancelOrderHandler.cs:116-117), which reads the session's authoritative status, expires the hosted page, and re-reads the status when the expire fails (.../Orders/PaymentSessionRetirement.cs:49-85; the same helper serves .../UseCases/Pay/PayOrderHandler.cs:69). When either read finds the session already paid, the handler refuses the cancellation with OrderCancellationErrorCodes.PaymentAlreadyCompleted instead of transitioning (CancelOrderHandler.cs:124-132, the code at .../Cancel/OrderCancellationErrorCodes.cs:18), so a paid order is left alone for a refund. Every other outcome (already expired, session gone, provider unreachable) is a Warning and the cancellation proceeds, the hosted page having been expired best-effort inside the helper (CancelOrderHandler.cs:119-122 and :134-137). It sits in the command rather than a saga step because a saga step runs after the commit and could only discover the money once the order was already Cancelled and a manual refund was the only remedy left. Everything that compensates after the fact is still its own handler: restoring stock is OrderCancelledSagaHandler : IDomainEventHandler<OrderCancelled> (.../Orders/Saga/OrderCancelledSagaHandler.cs:40-42) and notifying the customer of a failed payment is OrderPaymentFailedSagaHandler (.../Orders/Saga/OrderPaymentFailedSagaHandler.cs:22-24). A new compensating action is a new handler unless it has to decide whether the commit may happen at all.
  • Each handler runs in its own DI scope. Domain-event handlers are registered as singletons (MMCA.Common/Source/Core/MMCA.Common.Application/DependencyInjection.ModuleScanning.cs:54-56), so every one opens its own scope through IServiceScopeFactory (OrderCancelledSagaHandler.cs:49, OrderPaymentFailedSagaHandler.cs:31, .../Infrastructure/Payments/Reconciliation/PaymentReconciliationService.cs:235,254,300), and the ones that persist resolve their own IUnitOfWork inside it (OrderCancelledSagaHandler.cs:50, PaymentReconciliationService.cs:236,255,301). OrderPaymentFailedSagaHandler is the exception that shows the rule: it resolves the internal-command scheduler and little else in its scope (OrderPaymentFailedSagaHandler.cs:32) and writes no aggregate of its own, because both halves of its compensation (the "your payment did not go through" notification and the re-armed payment deadline) are queue rows the scheduler persists. Compensation that does write therefore commits on its own, after the originating save, rather than joining the transaction it is compensating for.
  • Idempotency is a persisted marker committed in the SAME transaction as the compensating writes. Order.InventoryRestored (.../Domain/Orders/Order.cs:116-122) is the marker; MarkInventoryRestored refuses a second call and refuses a non-cancelled order (Order.cs:571-594). The whole restoration runs as one ExecuteInTransactionAsync delegate (OrderCancelledSagaHandler.cs:55-57). The handler checks the marker first (OrderCancelledSagaHandler.cs:84-88) and validates the increases through a pure domain service (.../Domain/Inventory/InventoryRestorationDomainService.cs:24-51) against a non-tracking snapshot whose in-memory increase is never saved (OrderCancelledSagaHandler.cs:93-108). It then saves the marker first, so a concurrent delivery fails on the order row before it touches any stock (OrderCancelledSagaHandler.cs:130-139), and moves the stock through IInventoryAllocationService.IncrementAsync (OrderCancelledSagaHandler.cs:141-156), a relative atomic ExecuteUpdate (.../Infrastructure/Inventory/InventoryAllocationService.cs:97,127). A failed Result from the delegate rolls the whole unit back, marker included (MMCA.Common/.../DbContexts/Factory/DbContextFactory.cs:641-647). Same database, one transaction: the marker cannot exist without the writes it guards, and the writes cannot land unmarked.
  • Redelivery is the retry mechanism. A failing in-process handler leaves its outbox row unprocessed (MMCA.Common/.../Interceptors/DomainEventSaveChangesInterceptor.cs:371-406, the catch at :390-401) and the OutboxProcessor re-dispatches the pure domain event on a later cycle (MMCA.Common/.../Outbox/Processing/OutboxProcessor.cs:581, the dispatch at :792), with the bounded retries, backoff and dead-lettering ADR-003 already defines (OutboxProcessor.cs:593-605,629-638,1067-1068). The framework packages that failure mode and only that one: SafeDomainEventHandler<TDomainEvent> runs the subclass inside an exception filter whose LogAndRethrow writes one error line and always returns false, so the exception keeps propagating, with OperationCanceledException excluded because a host shutdown is not a delivery failure (MMCA.Common/.../DomainEvents/SafeDomainEventHandler.cs:36-70). What it standardizes is the log line, not a swallow. No production handler in the four repos derives from it today: its only subclass is the test double in its own unit tests (MMCA.Common/Tests/Core/MMCA.Common.Application.Tests/DomainEvents/SafeDomainEventHandlerTests.cs:124). A handler whose work is a pure side effect and must not force a redelivery therefore owns that decision alone: OrderPaymentFailedSagaHandler implements IDomainEventHandler<OrderPaymentFailed> directly (OrderPaymentFailedSagaHandler.cs:22-24) and turns each of its two failure modes into a log line rather than a throw: an Error when the notification could not be scheduled (OrderPaymentFailedSagaHandler.cs:38-41, message at OrderPaymentFailedSagaHandler.cs:100-103) and a Warning when the payment deadline could not be re-armed (OrderPaymentFailedSagaHandler.cs:86-90, message at OrderPaymentFailedSagaHandler.cs:108-111).
  • Concurrent redeliveries are serialized by the RowVersion concurrency token. Every auditable entity carries one (MMCA.Common/.../Entities/AuditableBaseEntity.cs:53), configured as a concurrency token on every non-owned auditable type (MMCA.Common/.../DbContexts/ApplicationDbContext.cs:588-608, ADR-035). Two deliveries that both pass the marker check carry the same original token into their update: one commits, the other gets DbUpdateConcurrencyException and its outbox retry then finds the committed marker and skips.
  • A periodic sweep drives the transitions a lost webhook would have, and backstops the orders no webhook will ever speak for. PaymentReconciliationService (.../Infrastructure/Payments/Reconciliation/PaymentReconciliationService.cs:69) is registered as a hosted service by the Sales module's infrastructure (.../Infrastructure/DependencyInjection.cs:42) and runs two passes per cycle against one age cutoff (PaymentReconciliationService.cs:120-131). Both passes select the oldest matching orders, ordered, bounded and projected to ids entirely in SQL through one shared helper (PaymentReconciliationService.cs:230-246) over a dedicated filtered index (.../Persistence/EntityConfiguration/OrderConfiguration.cs:70-76). The reconciliation pass takes orders sitting in PaymentInitiated with a Stripe session (PaymentReconciliationService.cs:152-157), asks Stripe for the session's authoritative status, and applies the matching transition: a paid session that PaymentProof accepts as payment of this order (amount and currency) to MarkAsPaid, expired to MarkAsPaymentFailed, still open to nothing (PaymentReconciliationService.cs:350-373). A paid session the proof refuses changes nothing and is logged at Warning with what an operator needs to reconcile it by hand, the same refusal the webhook applies (PaymentReconciliationService.cs:343-348,380-400). The expiry pass takes unpaid orders with no session to ask about, PendingPayment or PaymentFailed past the same cutoff (PaymentReconciliationService.cs:199-203), which hold stock checkout already committed and which nothing else can release. It calls no provider: it drives them to Cancelled through MarkAsCancelled (PaymentReconciliationService.cs:274), the terminal state OrderCancelledSagaHandler already listens on, so the compensation and its marker return the stock exactly once. Cancelled rather than PaymentFailed because PaymentFailed is retryable, so such an order keeps its stock for the whole retry window and loses it only when the window passes without a retry (PaymentReconciliationService.cs:43-49). Expiry runs before reconciliation in the cycle so an order the reconciliation pass just moved into PaymentFailed is not cancelled in the same cycle (PaymentReconciliationService.cs:125-130). The sweep is configuration-gated (.../Payments/Reconciliation/PaymentReconciliationSettings.cs:30, defaults of a 10-minute interval, a 30-minute stuck age and a 50-order batch, carried in MMCA.Store/Source/Services/MMCA.Store.Sales.Service/appsettings.json:104-109).
  • The sweep gets no private path into the aggregate, and loses races on purpose. It calls the same guarded transitions as the webhook handler (.../Orders/UseCases/ProcessPaymentWebhook/ProcessPaymentWebhookHandler.cs:106,169) and the client-initiated check (.../Orders/UseCases/VerifyPayment/VerifyPaymentHandler.cs:71). Both passes reload each order tracked in its own scope and re-check the status under the fresh load (PaymentReconciliationService.cs:311-315 for reconciliation, PaymentReconciliationService.cs:264-270 for expiry), and a DbUpdateConcurrencyException from a webhook or a customer that won the race is logged and skipped, not retried (PaymentReconciliationService.cs:335-340, PaymentReconciliationService.cs:286-291).

The loop shape is deliberate and shared, and it lives in the framework rather than in the sweep. MMCA.Common ships it as an abstract base class, PeriodicBackgroundService (MMCA.Common/Source/Core/MMCA.Common.Infrastructure/Hosting/Background/PeriodicBackgroundService.cs:20-50), whose ExecuteAsync owns the enablement gate, the startup delay, the per-cycle try/catch that never kills the loop, and every wait through TimeProvider (PeriodicBackgroundService.cs:53-95). PaymentReconciliationService derives from it (PaymentReconciliationService.cs:75) and overrides only the three parts that are its own: Interval, read from configuration (PaymentReconciliationService.cs:82); IsEnabled, which distinguishes "the toggle is off" from "Stripe is not configured" before refusing to run (PaymentReconciliationService.cs:90-108); and ExecuteCycleAsync, which delegates to the internally visible ReconcileOnceAsync so one cycle is testable without the timer (PaymentReconciliationService.cs:111-112). It is the base class's only subclass outside MMCA.Common. Inside the framework the base also carries Common's own periodic jobs (.../Persistence/Outbox/Administration/OutboxCleanupService.cs:49, .../Persistence/Auth/RefreshSessionCleanupService.cs:53, .../Persistence/InternalCommands/Administration/InternalCommandCleanupService.cs:46, .../Persistence/Auth/PermissionGrantRefreshService.cs:36) and the test double in its own unit tests (MMCA.Common/Tests/Core/MMCA.Common.Infrastructure.Tests/Scheduling/PeriodicBackgroundServiceTests.cs:104).

Adoption is one module. This pattern lives in MMCA.Store's Sales module only: the two saga handlers and the one reconciliation sweep above. MMCA.ADC and MMCA.Helpdesk have no compensating saga handler and no reconciliation sweep today. The nearest things in ADC are two compensating steps, not a saga. The first: when a session-asset upload has put its bytes in storage but the row does not follow, UploadSessionAssetHandler schedules the durable Conference.DeleteSessionAssetBlob.v1 internal command to remove the orphaned blob (MMCA.ADC/Source/Modules/Conference/MMCA.ADC.Conference.Application/SessionAssets/UseCases/UploadFile/UploadSessionAssetHandler.cs:112-116 and :126-133, the schedule at :178-195, the command at .../SessionAssets/UseCases/DeleteSessionAssetBlob/DeleteSessionAssetBlobInternalCommand.cs:28-29). The second is the avatar upload: when an avatar's bytes are in storage but the user row does not follow, SetUserAvatarHandler schedules the durable DeleteAvatarBlob internal command for the orphaned blob (SetUserAvatarHandler.cs:80 and :87, the schedule at :187, in MMCA.ADC/Source/Modules/Identity/MMCA.ADC.Identity.Application/Users/UseCases), recorded in ADR-045. Those behaviours are recorded in ADR-123 and ADR-045 and ride ADR-114's queue, not this record's saga machinery. The record exists because the mechanism (compensate, mark, reconcile) is the framework's stated answer to cross-boundary consistency, not because it is broadly adopted.

Rationale

  • No two-phase commit is available, and none is wanted. Transactions are per data source and best-effort sequential (ADR-006), and an external payment provider cannot enlist in a database transaction at all. Compensation plus reconciliation is the only mechanism left, so the decision is how to make it disciplined rather than whether to use it.
  • Choreography fits a workflow whose state is already on the aggregate. Order.Status plus Order.InventoryRestored are the saga state. An orchestrator would add a state machine and a persistence store to track what the aggregate already records.
  • A marker committed with its writes beats a marker written after them. Because both land in one transaction against one database, idempotency is a database invariant rather than handler discipline. This is strictly stronger than ADR-021's inbox, which records the message after the handlers succeed and therefore keeps a narrow crash window open.
  • Throwing is the correct failure mode. The outbox already owns retry policy, backoff and dead-lettering, so a handler that cannot complete should fail loudly and let redelivery re-run it rather than invent local retries. The framework's one packaged base class now says the same thing in code: it logs and rethrows (SafeDomainEventHandler.cs:36-47), because a swallowing base made the "retried via the outbox" promise false. A handler that reported success had its outbox row marked processed, so nothing retried and the side effect was lost with only a log line to show for it. Opting out of redelivery is therefore an explicit, per-handler catch, not a base class.
  • The Stripe pass needs no new infrastructure; expiry has earned some. For an order stuck in PaymentInitiated the authoritative answer already lives at the provider, so a bounded, indexed poll plus the existing guarded transitions is enough there: no timer message, no saga-timeout store. Expiry is a different question, because nothing outside the database has to be asked, and it is now armed per order on ADR-114's internal-command queue rather than waited for by a scan (see the Revision below). The sweep keeps running both passes, one as the primary path and one as a backstop.
  • Reusing the domain transitions keeps one state machine. The sweep cannot reach a state the webhook could not, because it goes through the same Result-returning methods (ADR-013).

Trade-offs

  • Inconsistency is bounded, not eliminated. Between the cancellation commit and the compensation commit, stock is held against a cancelled order. Between a dropped webhook and the sweep, an order sits in PaymentInitiated with stock held for up to the stuck age plus one poll interval (30 plus 10 minutes at the shipped defaults), and an order that never reaches Stripe at all is cancelled by its own scheduled deadline as the window closes, or by the expiry pass up to one poll interval later when that deadline was never written. That window is the price of not having a distributed transaction.
  • Compensation is best-effort per line, and names what it could not restore. RestoreInventory skips an order line whose InventoryItem row is missing (InventoryRestorationDomainService.cs:33-38), but the skip is not silent: the unmatched variant ids are collected and returned to the caller (InventoryRestorationDomainService.cs:28,37,50), and the handler logs them at Warning naming each one (OrderCancelledSagaHandler.cs:117-123, message at OrderCancelledSagaHandler.cs:171). The marker still commits anyway, so that quantity is never restored and nothing retries it: withholding the marker would re-apply every matched increase on the next redelivery, which is a double restore (OrderCancelledSagaHandler.cs:125-129). The miss is narrow because the handler reads inventory with ignoreQueryFilters: true (OrderCancelledSagaHandler.cs:103), so a soft-deleted row still counts and a missing row means the row never existed. The stock is still lost; what changed is that the loss is recorded rather than invisible.
  • An invariant rejection abandons the compensation instead of retrying it. When any matched increase is refused by an inventory invariant, RestoreInventory returns a failure carrying every rejected error (InventoryRestorationDomainService.cs:41-49), and the handler returns early with nothing saved: the rejected codes go to a Warning, and neither a partial restoration nor the marker reaches the database (OrderCancelledSagaHandler.cs:109-115, message at OrderCancelledSagaHandler.cs:174). A refused MarkInventoryRestored ends the same way (OrderCancelledSagaHandler.cs:130-135). A failed IncrementAsync after the marker save also returns a failed Result with a Warning (OrderCancelledSagaHandler.cs:148-155), which rolls the marker back with the transaction (DbContextFactory.cs:641-647). All three are returns, not throws: the handler discards the transaction's Result (OrderCancelledSagaHandler.cs:55), so the event counts as handled and its outbox row is marked processed. Nothing redelivers it, and that order's stock stays unrestored until someone acts on the Warning.
  • Redelivery re-runs every handler of the event, not the failed one. The dispatcher iterates handlers sequentially with no per-handler isolation (MMCA.Common/.../Services/DomainEventDispatcher.cs:76-85), so one throwing handler also skips the handlers after it, and a redelivery re-runs the ones that already succeeded. The blast radius is wider than the one event: the interceptor dispatches every local event of a save in a single call and marks their outbox rows processed only afterwards (MMCA.Common/.../Interceptors/DomainEventSaveChangesInterceptor.cs:375-385), so a throw skips that mark for the whole batch and every local event written by that save is redelivered, not just the one whose handler failed. Every handler on a shared event must therefore be idempotent against both a repeat of its own event and a repeat of every sibling event of the same save, or must swallow its failures itself.
  • The sweep is not replica-leased. The outbox processor claims rows with a lease before working them (ADR-003); the sweep takes no such claim, so at the configured maxReplicas: 2 (MMCA.Store/infra/main.bicep:1960, the salesApp scale block) two replicas can pick the same stuck order and each spend a Stripe status call. Correctness holds through the concurrency token; the duplicated external call does not deduplicate.
  • Every compensating action needs its own marker. There is no generic mechanism: the ADR-021 inbox dedups broker messages between services, not in-process handler re-runs. A second compensating action means a second persisted marker or a naturally idempotent operation, decided by the author.

Revision (2026-09-11): the unpaid-order deadline moves onto the order, and the sweep becomes its backstop

Expiring an unpaid order is no longer something a scan notices. Every order now arms its own deadline, and the sweep's expiry pass stays exactly where it was, underneath.

CheckOutHandler arms one ExpireUnpaidOrderInternalCommand(orderId) (MMCA.Store/Source/Modules/Sales/MMCA.Store.Sales.Application/Orders/InternalCommands/ExpireUnpaidOrderInternalCommand.cs:23-24) to run at now plus UnpaidOrderExpirySettings.Minutes, through IInternalCommandScheduler (MMCA.Common/Source/Core/MMCA.Common.Application/InternalCommands/IInternalCommandScheduler.cs:16, ADR-114's queue), from inside the same transaction that inserts the order and commits the stock: the transaction delegate calls its CheckOutDeadlineScheduler collaborator (.../ShoppingCarts/UseCases/CheckOut/CheckOutHandler.cs:167-171), whose ArmAsync does the scheduling (.../CheckOut/CheckOutDeadlineScheduler.cs:49-72, the schedule at :57-60). Inside, not after: "there is an order" and "there is a deadline for it" have to be one fact, or a crash between two commits leaves stock held with nothing scheduled to release it. The arming happens after the save within the delegate (CheckOutHandler.cs:165), because the order id is store-generated, and it is followed by a second SaveChangesAsync (CheckOutDeadlineScheduler.cs:68-71), because ExecuteInTransactionAsync commits without flushing pending changes first: the scheduler only enrolled its row on the enlisted context, so without that save the deadline would be discarded at commit. A scheduling failure is a Warning and nothing further (CheckOutDeadlineScheduler.cs:62-66, message at CheckOutDeadlineScheduler.cs:33-36); it never fails the checkout, because the deadline is an optimization over a sweep that still runs, and losing a completed checkout over an unwritten queue row would trade a slow stock release for a lost sale.

OrderPaymentFailedSagaHandler re-arms the same command from now after every failed payment attempt (.../Orders/Saga/OrderPaymentFailedSagaHandler.cs:67-93, the schedule at :81-84), so a customer whose card was declined gets the whole window again rather than the remainder of the old one. The earlier row armed at checkout stays in the queue and runs as a no-op, because the due instant is re-derived rather than trusted.

ExpireUnpaidOrderInternalCommandHandler is where that re-derivation lives (.../Orders/InternalCommands/ExpireUnpaidOrderInternalCommandHandler.cs:41-98). It reloads the order and returns success untouched if the order is gone or has left PendingPayment/PaymentFailed (:55-65), then recomputes the deadline as (LastModifiedOn ?? CreatedOn) plus Minutes and does nothing if that instant is still in the future (:67-74). That is the sweep's own rule applied to a single order, and it is why the payload is the order id alone: a row armed half an hour ago says nothing about what the order is doing now, so what is stored decides. When the order is genuinely due the handler calls the same guarded MarkAsCancelled the sweep calls (:78-83), and OrderCancelledSagaHandler returns the stock through the marker this record already describes. A concurrency conflict on the save is a lost race to a webhook that moved the order between the load and the save, so the row completes successfully rather than retrying into the same conflict (:85-93), and the provider exception is classified through IConcurrencyConflictDetector because the Application layer references no data provider.

Both paths are configuration-gated and are meant to agree. UnpaidOrderExpirySettings (.../Orders/UnpaidOrderExpirySettings.cs:25-43, Enabled defaulting to true and Minutes to 30) is bound and validated on start beside the sweep's own settings (.../MMCA.Store.Sales.API/SalesModule.cs:59-67), and PaymentReconciliation:StuckAgeMinutes carries the same 30 (PaymentReconciliationSettings.cs:48, MMCA.Store/Source/Services/MMCA.Store.Sales.Service/appsettings.json:96-105). Turning Enabled off falls back to the sweep alone, which is slower and still correct.

The sweep stays, and its expiry pass is unchanged. A deadline is a row, and a row can fail to be written (a scheduling failure at checkout is logged, never fatal), can be dead-lettered by the internal-command processor, or can simply never have existed for an order created before the mechanism did. Those cases are rare, which is precisely the argument for a backstop rather than against one: one bounded, indexed query per cycle covers all three without anyone having to decide in advance which one occurred. The Stripe half of the sweep is untouched and could not become a deadline anyway, because its work is a live provider call whose answer is only knowable at the moment it is made, and the failure it exists for (a prolonged webhook outage) strands a backlog, which is the shape batching is right for (PaymentReconciliationService.cs:50-59). Both paths converge on the same guarded transition and the same OrderCancelled compensation, so whichever arrives first wins and the other finds nothing to do.

What the deadline buys is timing, and a quieter sweep. An unpaid order is now cancelled as its window closes rather than up to one poll interval later (10 minutes at the shipped defaults), which is the difference between holding a basket for 30 minutes and holding it for 30 minutes give or take the next cycle. The common case also stops doing work: the expiry pass still runs on its interval, but on a healthy system it now selects nothing, because the orders it used to collect have already cancelled themselves.

Revision (2026-10-01)

The idempotency guarantee is now one transaction rather than one SaveChanges. OrderCancelledSagaHandler runs the whole restoration as a single ExecuteInTransactionAsync delegate (OrderCancelledSagaHandler.cs:55-57), saves the marker first so a concurrent delivery fails on the order row before any stock moves (:130-139), and moves the stock through IInventoryAllocationService.IncrementAsync, a relative atomic ExecuteUpdate (:141-156, InventoryAllocationService.cs:97,127). InventoryRestorationDomainService still decides which lines match and whether every increase is allowed, but its in-memory increase lands on a non-tracking snapshot and is never saved (OrderCancelledSagaHandler.cs:93-108). The Decision bullet and the matching Rationale bullet are corrected. A new trade-off case follows: an IncrementAsync failure returns a failed Result, which rolls the marker back (MMCA.Common/.../DbContexts/Factory/DbContextFactory.cs:644-650), but the handler discards that Result (OrderCancelledSagaHandler.cs:55), so the event is not redelivered, although the code comment at :53-54 says a later delivery can still restore.

Three other statements moved. CancelOrderHandler retires the session through the shared PaymentSessionRetirement.RetireAsync (PaymentSessionRetirement.cs:49-85), which adds a re-read on a failed expire, so a session paid in that window is also refused. The reconciliation pass marks a paid session paid only when PaymentProof accepts it for this order, and logs a refused one at Warning (PaymentReconciliationService.cs:350-400). PeriodicBackgroundService moved to Hosting/Background/ and Common's own periodic jobs now derive from it, so the sweep is its only subclass outside MMCA.Common rather than in all four repos.

Two statements in the Revision (2026-09-11) above no longer hold and are left there as the record of that date, and its citations keep the line numbers of that date and are not re-anchored. CheckOutDeadlineScheduler performs no second SaveChangesAsync (ArmAsync at CheckOutDeadlineScheduler.cs:47-64): Common's ExecuteInTransactionAsync now saves the enrolled internal-command rows just before the commit (DbContextFactory.cs:654,705-731), as the scheduler's remarks state (CheckOutDeadlineScheduler.cs:14-19). And the sweep's stuck age no longer merely is meant to agree with the unpaid-order window: PaymentReconciliationSettingsValidator, registered at .../MMCA.Store.Sales.API/SalesModule.cs:73-74 beside ValidateOnStart (SalesModule.cs:66-69), refuses to boot unless StuckAgeMinutes equals UnpaidOrderExpiry:Minutes (PaymentReconciliationSettingsValidator.cs:33-37). Every other citation in the current-state sections is re-anchored (CheckOutHandler, OrderCancelledSagaHandler, CancelOrderHandler, PaymentReconciliationService, appsettings.json, ProcessPaymentWebhookHandler, VerifyPaymentHandler, DomainEventSaveChangesInterceptor, OutboxProcessor, ApplicationDbContext, maxReplicas, UploadSessionAssetHandler).

Revision (2026-10-06)

No behaviour changed; this pass only re-anchors.

  • Current-state citations moved for CancelOrderHandler (MutateAsync now :60-81, retirement :104-140), PayOrderHandler (:69), ProcessPaymentWebhookHandler (:110,175), DbContextFactory (the failed-Result rollback now :615-621), OutboxProcessor, PeriodicBackgroundService, OutboxCleanupService, InternalCommandCleanupService and maxReplicas (main.bicep:1860); the avatar orphan delete also cites its schedule (SetUserAvatarHandler.cs:187).
  • Facts the earlier Revisions record that still hold, at their current locations: the enrolled internal-command rows are flushed before commit by FlushEnrolledCommandsBeforeCommitAsync (called at DbContextFactory.cs:625, defined at :681-707); UnpaidOrderExpirySettings is bound and validated on start at SalesModule.cs:61-64, PaymentReconciliationSettings at :66-69, and PaymentReconciliationSettingsValidator is registered at :74.
  • Every other citation in the current-state sections was re-verified against current source.

Revision (2026-10-07)

Re-verified against current source. No behaviour changed: a failing in-process handler still leaves its local outbox rows unprocessed, the interceptor still dispatches every local event of a save in one call before marking any row processed, and the OutboxProcessor still re-dispatches with bounded retries, backoff and dead-lettering. Only the Common line numbers moved.

  1. Anchors re-verified against current source: DomainEventSaveChangesInterceptor (FlushStateAsync now :369-400, the catch that logs, releases the local lease and signals at :383-394, the dispatch and OutboxFinalizer.MarkProcessedAsync at :373-378) and OutboxProcessor (the per-row DeliverAsync call at :520, its dispatcher.DispatchAsync at :694, the retry increment and backoff re-lease at :534-544, the retries-exhausted dead-letter metric at :568-578, ComputeRetryBackoffSeconds at :858-859). The Revision (2026-09-11) citations of CheckOutHandler keep the line numbers of that date, as the Revision (2026-10-01) states.

Revision (2026-10-21)

Re-verified against current source. No behaviour changed: a failed Result from the transaction delegate still rolls the whole unit back, the enrolled internal-command rows are still flushed just before the commit, the OutboxProcessor still redelivers with bounded retries, backoff and dead-lettering, the shipped defaults are unchanged (unpaid-order window 30 minutes, a 10-minute poll, a 30-minute stuck age, a 50-order batch), and maxReplicas is still 2. Only line numbers moved.

  1. Anchors re-verified against current source: DbContextFactory (the failed-Result rollback now :641-647, FlushEnrolledCommandsBeforeCommitAsync called at :651 and defined at :713), OutboxProcessor (the per-row DeliverAsync call at :579, its dispatcher.DispatchAsync at :790, the retry increment and backoff re-lease at :593-603, the retries-exhausted dead-letter metric at :627-636, ComputeRetryBackoffSeconds at :1062-1063), the Sales.Service appsettings.json (UnpaidOrderExpiry at :101-104, PaymentReconciliation at :105-110), maxReplicas: 2 in the salesApp scale block (main.bicep:1954), SalesModule (UnpaidOrderExpirySettings bound at :61-64, PaymentReconciliationSettings at :66-69, PaymentReconciliationSettingsValidator registered at :73-74, its equality check at PaymentReconciliationSettingsValidator.cs:33-37) and CheckOutHandler (the transaction at :80, deadlineScheduler.ArmAsync at :180). The citations inside the dated Revisions above keep the line numbers of their dates.

Revision (2026-10-26)

Re-verified against current source. No behaviour changed: a failing in-process handler still leaves its local outbox rows unprocessed (the interceptor marks them processed only after the dispatch succeeds, and its catch logs, releases the local lease and signals), the OutboxProcessor still redelivers with bounded retries, backoff and dead-lettering, the shipped defaults are unchanged (unpaid-order window 30 minutes, a 10-minute poll, a 30-minute stuck age, a 50-order batch), maxReplicas is still 2, and the sweep still calls the same guarded transitions as the webhook and the client-initiated check. Only line numbers moved.

  1. Anchors re-verified against current source: DomainEventSaveChangesInterceptor (FlushStateAsync now :371-406, the dispatch at :375-381, OutboxFinalizer.MarkProcessedAsync at :385, the catch at :390-401), OutboxProcessor (the per-row DeliverAsync call at :581, its dispatcher.DispatchAsync at :792, the retry increment and backoff re-lease at :593-605, the retries-exhausted dead-letter metric at :629-638, ComputeRetryBackoffSeconds at :1067-1068), the Sales.Service appsettings.json (UnpaidOrderExpiry at :100-103, PaymentReconciliation at :104-109), maxReplicas: 2 in the salesApp scale block (MMCA.Store/infra/main.bicep:1960), CheckOutHandler (WriteAsync at :122-181, the transaction still at :80-82, deadlineScheduler.ArmAsync at :178), ProcessPaymentWebhookHandler (MarkAsPaid at :106, MarkAsPaymentFailed at :169) and VerifyPaymentHandler (MarkAsPaid at :71). The citations inside the dated Revisions above keep the line numbers of their dates.

ADR-003 (the outbox delivery and retry this leans on for compensation redelivery; this record says what the redelivered handler must do), ADR-006 (which accepts "no cross-database transaction" as a cost without saying how it is paid; this is the payment), ADR-021 (dedup of broker redeliveries between services, complementary to the aggregate marker that dedups in-process handler re-runs), ADR-035 (the RowVersion token that serializes concurrent compensations and lets the sweep lose a race to a webhook), ADR-014 (the command pipeline whose commit compensation runs after), ADR-052 (in-process background work, the hosted-service family this sweep belongs to, with a fixed-interval poll instead of a queue drain), ADR-013 (the Result-returning guarded transitions both the sweep and the per-order deadline reuse rather than bypassing), ADR-114 (the durable internal-command queue the per-order unpaid-order deadline is scheduled on), ADR-086 (the orchestrated alternative to the choreography decided here: deferred, with the shape it would take and the trigger that would build it, and with this record's reconciliation sweep remaining underneath it).

← Back to Architecture Decision Records View source on GitHub ↗