Load / Write Data

Load data

Load canonical survey tables from file paths into memory.

Algorithm

1. Iterate through each table name and file path in input_paths
2. Validate file path exists, providing helpful error message with broken path component
3. Load data based on file extension:
   • .csv/.tsv → polars.read_csv()
   • .parquet → polars.read_parquet()
   • .shp/.shp.zip/.geojson → geopandas.read_file()
4. Return dictionary of loaded tables

Notes

• All CSV/Parquet files loaded as Polars DataFrames for performance
• Geospatial files loaded as GeoPandas GeoDataFrames
• Path validation helps diagnose configuration errors

Example

- name: load_data
  params:
    input_paths:
      households: data/hh.csv
      persons: data/person.csv
      days: data/day.csv
      unlinked_trips: data/trip.csv

Settings

Key	Type	Default	Meaning
input_paths	mapping	required	Dictionary mapping table names to file paths. Supported formats: CSV, TSV, Parquet, Shapefile, GeoJSON.

Write data

Write canonical survey tables to output file paths.

Algorithm

1. If validate_input=True, validate each table using canonical data models
2. For each table in output_paths:
   • Retrieve DataFrame from canonical_data
   • Create parent directories if needed
   • Write data based on file extension:
     • .csv → DataFrame.write_csv()
     • .parquet → DataFrame.write_parquet()
     • .shp/.shp.zip/.geojson → GeoDataFrame.to_file()
     • .txt → Path.write_text()
3. If enum_codebook_path is set, discover all LabeledEnum types from the models for the written tables and write a codebook workbook.
4. Log completion status

Notes

• Validation ensures output conforms to canonical data schemas
• Automatic directory creation prevents path errors
• Supports multiple output formats for flexibility

Example

- name: write_data
  params:
    write_only_canonical: true
    output_paths:
      households: output/households.csv
      persons: output/persons.csv
      days: output/days.csv
      linked_trips: output/linked_trips.csv
      tours: output/tours.csv

Settings

Key	Type	Default	Meaning
output_paths	mapping	required	Dictionary mapping table names to output file paths.
write_only_canonical	true / false or null	null	Keep only columns the pipeline stands behind -- declared model fields plus the config-named columns steps register (zone ids, pre-imputation stashes). False dumps every column on the frame, which is what you want for debugging. Required: there is no sensible default, and silently choosing one would either drop columns a project expected or ship hundreds it did not.
create_dirs	true / false	true	Whether to create parent directories (default: True).
enum_codebook_path	text or null	null	Optional path for an .xlsx enum codebook. When provided, a workbook is written with one worksheet per LabeledEnum found in the models for the written tables. Each sheet contains Value, Label, and Value Label columns. Must end in .xlsx.

Code: processing.read_write