Architecture

Flux separates four concerns with four different failure modes: the systems that collect, the system that coordinates, the system that writes, and the systems that read.

The data path #

cloud sources → collectors → analytical writer → validated snapshot → serving
                     ↑              ↑                    ↑
                     └──────── control plane ────────────┘
                        (leases, claims, retries, ledger)

Collectors run independently on their own schedules — inventory, cost, recommendations, policy, telemetry, billing exports. One being slow or throttled does not hold up the others.

The control plane is a transactional database holding coordination state: which job owns what, retry eligibility, throttle timing, and which snapshot version is currently approved. It carries no analytical data.

A single writer owns every analytical mutation. Nothing else writes to the analytical store, which removes an entire class of concurrency bug.

Publication takes a consistent copy, validates it independently, checksums it, and only then marks it approved.

Serving instances each hold their own read-only copy of the approved snapshot. They never touch the writer’s database.

Why reads never wait on ingestion #

This is the property everything else is arranged to produce.

Collection can run flat out — importing hundreds of thousands of charge records — while dashboards stay responsive, because readers and the writer are not contending for the same thing. Readers serve a published copy.

It also makes horizontal scale-out possible: any number of serving instances can hold the same versioned copy, because none of them coordinate with each other or with the writer.

Publication and validation #

A snapshot becomes live only after passing every gate:

1. The analytical store is checkpointed and copied
2. The copy is opened independently, read-only
3. Required tables and views must exist
4. Report smoke queries must execute
5. Row counts are recorded
6. A SHA-256 checksum is computed
7. The copy is uploaded to private storage
8. Only then is the version marked approved

A candidate failing any gate is recorded as rejected and discarded. The previous approved version keeps serving. There is no state where a bad dataset is live.

Each serving instance independently verifies the checksum and smoke-tests its downloaded copy before swapping to it, and in-flight requests finish against the previous version.

Coordination #

Jobs coordinate through the control plane rather than through files:

• Advisory leases ensure one owner per job, and release automatically if a process dies
• Expiring claims mean work orphaned by a crash is picked up by the next worker rather than stranded
• Shared throttle state paces requests across every process, so parallel collectors cannot collectively breach an API quota
• An apply ledger with idempotency keys and payload checksums means a duplicate delivery cannot double-count

Retention #

Approved snapshots are retained on a tiered policy — the most recent versions plus one per day for a rolling window. Because each is a validated, checksummed copy of the whole analytical store, they serve as the analytical backup as well as the serving artifact.

Failure behaviour #

Failure	Behaviour
A source is throttled	Recorded, retried first next run, coverage marked partial
A collector fails	Other sources still land; last-good data preserved
A candidate snapshot fails validation	Rejected and logged; previous version keeps serving
A download is corrupt	Checksum mismatch, discarded, instance keeps current version
The writer dies mid-publication	No partial approval; leases release; next run recovers
A worker crashes mid-job	Claim expires; another worker resumes

Every one of these is exercised deliberately rather than assumed.