Back to selected work

AwaLedger

A financial backend and detached AI worker designed as one distributed system, with durable state before delivery and idempotency at every retry boundary.

FastAPIPostgreSQLGCP Pub/SubRedisGemini

The boundary

Financial document processing and model-assisted classification are slower and less predictable than transactional API work. Running them inside the request path would couple user-facing reliability to a broker, model provider, and callback network.

The system instead commits business state and an outbox event together, then lets a detached worker perform AI processing and report typed results through a secured callback.

System map

Four boundaries have distinct failure behavior: the database transaction, message delivery, AI work, and result callback.

Reliability decisions

  1. Use a transactional outbox. Financial state and the event announcing it commit together. Publishing can retry independently without inventing state.
  2. Classify delivery failures. The worker distinguishes malformed or permanent input from transient model, network, and callback failures before choosing ack or nack.
  3. Lock by event identity. A Redis lock suppresses concurrent duplicate work when Pub/Sub redelivers the same event.
  4. Require typed model output. Pydantic contracts keep classification and business-insight payloads bounded before they cross the callback boundary.
  5. Make callbacks idempotent and authenticated. Stable event identity prevents repeated callbacks from duplicating stored results; secrets and sample financial data never enter logs or public examples.

Engineering outcomes

Fast request path

User-facing transactions do not wait for model processing or callback delivery.

Durable handoff

An event cannot exist without its committed source state, and unpublished outbox rows remain retryable.

Duplicate resistance

Redis locks and callback idempotency defend both sides of at-least-once delivery.

Replaceable intelligence

The worker owns model integration while the backend owns financial truth and result acceptance.