Skip to content
TRUETOPO TRUETOPO TRUETOPO Docs

Data Pipelines

Pipeline architecture

Section titled “Pipeline architecture”

A typical pipeline flows as follows:

Raw Data  →  Download Script  →  Transform Script  →  GeoJSON  →  Vector Tiles  →  R2 Bucket

Technologies:

  • Python — Ingestion, normalization, and orchestration
  • GDAL — Geospatial transforms and format conversion
  • Tippecanoe — Vector tile generation (GeoJSON → PMTiles / MBTiles)
  • GitHub Actions — Scheduled runs, validation, and upload

Example: GNIS pipeline

Section titled “Example: GNIS pipeline”
  1. Download — Fetch GNIS dataset (e.g. Alaska or national subset).
  2. Filter — Restrict to Alaska (or target region).
  3. Convert — Output normalized GeoJSON with consistent schema.
  4. Generate vector tiles — Run Tippecanoe to produce PMTiles or MBTiles.
  5. Upload to R2 — Push to the appropriate bucket (e.g. truetopo-tiles or data-truetopo).

Pipeline config and workflow definitions live in the TrueTopo data catalog repository (schedules, validation steps, and upload process are managed via CI/CD).

Validation

Section titled “Validation”

Pipelines include validation (schema checks, geometry validity, required fields) before upload. Failed validation blocks deployment so only valid data reaches production.