Automation · data operations · 2019 – 2021

Running a county GIS on about 200 Python scripts

About 200 scheduled Python scripts kept Yavapai County’s GIS current: parcel imports from the assessor, weekly updates from 11 cities and towns, geodatabase maintenance, QA/QC checks and regular exports to 9-1-1, dispatch, the state DOT and an imagery provider.

Organization
Yavapai County
Year
2019 – 2021
~200
scheduled Python scripts
11
cities and towns on automated weekly updates

Results

  • Parcels, city and town updates and county imports loaded on a schedule.
  • Shared modules, so a fix in one place reached every script that used it.
  • Regular exports to 9-1-1, dispatch, the state DOT and an imagery provider.
  • An onboarding guide for new GIS staff.

Stack

  • Python 2.7 and 3
  • ArcPy
  • pyodbc
  • SQL Server enterprise geodatabase
  • FME
  • SSIS
  • PowerShell
  • Windows Task Scheduler
  • ArcGIS Server, Portal and ArcGIS Online
Diagram in three stages. In: assessor parcels daily and weekly, weekly updates from 11 cities and towns, and public works and permit imports all feed one set of about 200 scheduled Python scripts, built on shared modules and running QA/QC checks on road centerlines and addresses. Out: the scripts maintain the county geodatabase and send regular exports to 9-1-1 and dispatch, and to the state DOT and an imagery provider.
Data comes in from the assessor, cities and towns and county departments, runs through about 200 scheduled Python scripts and goes out to the county geodatabase and partner agencies.

What I inherited

Some of these systems dated back to 2013. I moved them to the cloud, then upgraded them.

What ran

  • Parcels. Daily and weekly imports from the assessor.
  • Cities and towns. Weekly update and delete scripts for 11 cities and towns, plus integration workflows for independent agencies.
  • County departments. Scheduled imports from public works and permitting.
  • Geodatabase maintenance. Compress, reindex, and reconcile and post on the enterprise geodatabase.
  • Exports. Regular deliveries to the sheriff’s dispatch system, the state 9-1-1 program, the state DOT and an aerial imagery provider.
  • Reporting. Scripts that reported on ArcGIS Online and Portal content, and a tool that republishes feature services in place.

Built to be maintained

The scripts shared one library of modules, in Python 2.7 and Python 3 versions, so a fix there reached every script that used it. Windows Task Scheduler ran them, alongside FME, SSIS and PowerShell.

I documented the server migration and wrote an onboarding guide for new GIS staff.

Related QA/QC work

The road centerlines and addresses that 9-1-1 depends on had checks of their own: the centerline checks and change reports and the fishbone address analysis.