Dealer Auto-Order Pipeline

The problem I inherited

Submission was synchronous and inline in the HTTP request, one blocking call per order item, no retry. A single slow or erroring distributor failed the dealer's entire order.

Dealer POS, nightly

scheduled auto-order run

Admin, manual

on-demand trigger

Roughly 3,000 dealers fire in the same nightly window, so this is a burst, not a smooth stream

Request path: synchronousfast, deterministic, no external calls

Authenticated API endpoint

bearer-token auth, accepts a product and quantity list

Validate products and quantities

Create order, type AUTO, status PENDING

written before any external call

Resolve eligible distributors

auto-order enabled + relationship active, in the dealer's priority order

Greedy split across distributors

walk priority order, allocate against available inventory, roll the unmet remainder to the next

Group order items by distributor

one external call per distributor, not per item

Stash payload in Redis

Given a TTL, and the key embeds the order id so a stuck submission can be replayed

Dispatch job to the orders queue

Return to dealer: order accepted

async boundary: acceptance decoupled from fulfillment

Concurrency: the queue absorbs the burst

Redis-backed queue fans out to many workers across autoscaled EC2 instances. Each job is reserved by exactly one worker, so distribution itself is safe.

workerEC2

count flexes with load

Queued workeroff the request path, queue retry semantics

Read payload back from Redis

Submit purchase order per distributor

Distributor integration layer

About a dozen vendors, and the transports vary: FTP flat file, positional CSV, multi-step SOAP, REST/JSON

Confirmation events written back

fulfillment event history per order item

Status derived at read time

dealer polls a check endpoint

Daily recovery loopdetect, dedup, resubmit, report

Scan errored fulfillments in a date window

Walk the fulfillment-event history: the concurrency guard

Is it a genuine error, and has it already been resubmitted? This is what stops concurrent workers and a scheduled scan from double-sending a real purchase order

Queue an automatic resubmit

order-item hash detects duplicates

Residual daily report

runs after the resubmit, so it surfaces only what genuinely could not be recovered

Why this shape

Accept fast, fulfill async. The dealer's order is durable the moment it is persisted. Nothing about a flaky vendor can fail acceptance.

Same shape as acknowledging a webhook immediately and processing it on a queue.

Durability

Payload in Redis with a generous TTL, keyed by order id, so a stuck or failed submission can be found and replayed rather than lost.

Fewer fragile calls

Grouping by distributor instead of one call per item cut the number of external calls, and each one is the unit that retries.

Who arbitrates the race

Two dealers could allocate the same distributor stock at the same moment and both pass locally.

Local inventory was a hint. The distributor's confirmation was the source of truth, so the vendor arbitrated the race, not us.

Recovery, not just alerting

Errored orders used to sit until a human noticed. The loop detects, verifies against event history, and resubmits, and the report shows only the residue.

The event-history check is the idempotency guard. It is what stops a double-send.

What I'd add today

Explicit HTTP timeouts on every vendor call. A real idempotency key on submission rather than inferring from status bookkeeping.

Per-distributor rate limiting so concurrent workers do not stampede one vendor, and no-overlap on the scheduled scan.