Live demo · data pipeline · 2026

A data pipeline you can run: the USGS earthquake feed

A five-step ETL job that runs on this site’s server when you press Run: it fetches the USGS earthquake feed you choose, reshapes it into a table, validates every record, builds a report and publishes the cleaned data as CSV or GeoJSON. The log shows what each step did and the run’s total server time.

Year
2026
5
steps, each with its own log lines
60 s
shared cache for each feed under 1.5 MB, so visitors don’t hammer USGS

Results

  • Runs only when a visitor presses Run; nothing runs on page load or on a timer.
  • Bad records are counted and reported in the log, not dropped silently.
  • Rate limited per visitor and overall, keyed on hashed addresses; no visitor data is stored.

My role

Designed and built it: the server job, its REST endpoints and the page.

Stack

  • PHP 8
  • WordPress REST API
  • USGS GeoJSON feeds
  • Record validation
  • CSV & GeoJSON output
  • Canvas map

Run it

Running on this server. Also on Live demos.

Source feed: 2.5_day.geojson · USGS Earthquake Hazards Program

Waiting for you. Pick your settings and press Run pipeline.
Not run yet

What a run does

  1. Fetch. Picks the smallest USGS summary feed that holds your magnitude and time window, with one retry. A feed under 1.5 MB is cached and shared for 60 seconds, the feeds’ own update interval.
  2. Reshape. Turns each GeoJSON feature into a nine-field row (id, magnitude, place, time, longitude, latitude, depth, tsunami flag, link), then keeps the rows at or above your magnitude and inside your region’s bounding box.
  3. Validate. Checks coordinates, magnitude and a unique id on every row; rejected rows are counted in the log.
  4. Load. Sorts the clean rows by magnitude and builds the run’s report: the count, the largest quake, the six most recent and up to 1,500 points for the map.
  5. Publish. Up to 3,000 cleaned rows, largest first, become the site’s public dataset, downloadable as CSV or GeoJSON, with the recent runs alongside.

Why it’s built this way

The USGS summary feeds are GeoJSON, regenerated every minute. The job notes when the copy it read was generated, shares a cached copy of each feed under 1.5 MB for 60 seconds, and retries a failed fetch once before it stops the run. The region filter is a bounding-box test on longitude and latitude, which is why the region names say approximate.

The filters pass rows with missing values on to validation instead of quietly dropping them. There a row is rejected for a missing coordinate, a latitude beyond ±90° or longitude beyond ±180°, a magnitude that isn’t a number, or a blank or repeated event id, and the log reports how many were rejected.

Every row in the download keeps its USGS event id and link, the GeoJSON carries depth as each point’s third coordinate, and each download is tied to the run that produced it for up to an hour, so the file comes from the same rows as the map and report.