Case Study 02 · DirtBid

Real-Time Geospatial and Earthwork Estimation Pipeline

An event-driven pipeline that ingests site plans and geospatial data, computes cut-and-fill volumes, and produces bid-ready estimates with every step traceable and retryable.

Client
DirtBid
Industry
Construction bidding and earthwork
System
Event-driven estimation pipeline
Discipline
Workflow pipelines, cloud and DevOps
Headline result
99.4% pipeline uptime

The challenge

What was breaking.

Earthwork estimation is where construction bids are won or lost, and it is brutally manual. Estimators receive site plans in inconsistent formats, trace surfaces by hand, compute cut-and-fill volumes in spreadsheets, and reconcile the result with haul distances, material types, and equipment rates. A single bid can take days, and a single transcription error can swing the margin.

The platform needed to turn that process into a pipeline: accept plans and geospatial inputs as they arrive, run the geometry and volume calculations automatically, and present estimators with a bid-ready number and the evidence behind it. It also had to be reliable enough to run during bid deadlines, when every job is urgent and a stalled queue is a lost contract.

The solution

What we built.

InsightSol built an event-driven processing pipeline with a typed ingestion boundary, a queue-backed worker fleet, and a results store that keeps every intermediate artifact. Plans and geospatial files are validated on arrival, normalized into a common surface representation, and processed through stages for surface reconstruction, cut-and-fill computation, haul analysis, and estimate assembly. Each stage is idempotent and independently retryable, so a failure in haul analysis never forces the geometry to recompute.

Reliability was engineered rather than hoped for. Every stage reports health and throughput to a telemetry layer, failed jobs land in a dead-letter queue with full context, and a reconciliation job verifies that every accepted input produced an estimate or an explained failure. Estimators see progress in real time through WebSocket updates and can inspect the surfaces and volumes behind any number before it goes into a bid.

Headline result

99.4%

pipeline uptime across a full production quarter

Stack

PythonPostgreSQLPostGISRedisDockerKubernetesAWSTypeScriptNext.jsWebSockets

System architecture

The components and the path a request takes through them.

Components

  • Ingestion boundary

    Schema-validated uploads and API intake, normalized into a common geospatial representation

  • Worker fleet

    Stateless containerized workers pulling from queues, autoscaled on queue depth

  • Processing stages

    Surface reconstruction, cut-and-fill volumes, haul analysis, estimate assembly, each idempotent

  • Results store

    PostgreSQL with PostGIS for geometry, object storage for artifacts, versioned per run

  • Real-time layer

    WebSocket progress and results pushed to the estimator's workspace

  • Reliability layer

    Retries with backoff, dead-letter queue, reconciliation job, health and throughput telemetry

Request flow

  1. 1Plan or geospatial file arrives through upload or API and is validated
  2. 2Normalization produces a common surface representation
  3. 3Geometry stage reconstructs surfaces and computes cut-and-fill volumes
  4. 4Haul and material stages compute distances, quantities, and rates
  5. 5Estimate assembly produces the bid-ready number with evidence
  6. 6Estimator reviews live; artifacts and run history are stored

Quantified results

What was measured in production.

  • 99.4 percent pipeline uptime measured across a full production quarter, including bid-deadline peaks
  • Every accepted input is accounted for by a reconciliation job: an estimate or an explained, replayable failure
  • Estimators inspect the surfaces and volumes behind any number, replacing spreadsheet transcription with traceable computation
  • Horizontal scaling on queue depth absorbs deadline surges without manual intervention

Engineering lessons

01

Make every stage idempotent and independently retryable

A failure in haul analysis should never force the geometry to recompute, and in this pipeline it never did. Stage isolation is what kept deadline-day retries cheap and predictable.

02

Reconciliation is the real uptime metric

Counting every accepted input against either an estimate or an explained failure is what let the team trust the 99.4 percent figure. Uptime without reconciliation only tells you the servers were on.

03

Autoscale on queue depth, not on CPU

Bid deadlines arrive as bursts of work. Queue depth is the signal that tracks those bursts, so workers scale before latency is visible to an estimator rather than after.

Want the architecture behind your own numbers?

The audit produces the same diagrams and unit economics this system started with, for your workflow and your data.

  • Read-only access under NDA
  • Written review and diagrams
  • Fixed-price roadmap