Skip to content

Data Models

Pydantic data models provide validation and type checking for survey data processing.

Overview

Data models represent individual records (rows) and define:

  • Required and optional fields
  • Field validation rules and constraints
  • Foreign key relationships between tables

Which steps need a field is not part of the model. Each step declares the columns it reads and writes in its own @step(requires=..., produces=...); see Column Requirements for the matrix built from them.

Models use Pydantic's BaseModel with custom field validators to ensure data quality throughout the processing pipeline.

Key Features

Field Validation

Each field includes validation rules: 

age: AgeCategory = schema_field()
home_lat: float = schema_field(ge=-90, le=90)

Foreign Key Relationships

Models enforce referential integrity: 

hh_id: int = schema_field(
    ge=1,
    fk_to="households.hh_id",
    required_child=True,
)

Pipeline Step Requirements

A step names the fields it needs, and the ones it creates: 

@step(
    requires={"persons": {"person_id", "age", "employment", "student", "school_type"}},
    produces={"tours": {"tour_id", "joint_tour_id"}},
)
def extract_tours(...): ...
A column whose name the config chooses is written as a pattern, such as usable_<profile>.

Usage Example

from data_canon.models.survey import PersonModel

person = PersonModel(
    person_id=1,
    hh_id=100,
    person_num=1,
    age=AgeCategory.AGE_35_TO_44,
    gender=Gender.FEMALE,
    employment=Employment.EMPLOYED_FULLTIME,
    student=Student.NONSTUDENT,
    # ... other fields
)

Survey Data Models

Core data models used in the processing pipeline for households, persons, days, trips, and tours.

data_canon.models.survey.HouseholdModel pydantic-model

Household attributes (minimal for tour building).

Fields:

hh_id pydantic-field 

hh_id: int

home_lat pydantic-field 

home_lat: float

home_lon pydantic-field 

home_lon: float

residence_rent_own pydantic-field 

residence_rent_own: ResidenceRentOwn

residence_type pydantic-field 

residence_type: ResidenceType

income pydantic-field 

income: int | None = None

income_bin pydantic-field 

income_bin: IncomeBroad

num_vehicles pydantic-field 

num_vehicles: int

survey_complete pydantic-field 

survey_complete: bool

data_canon.models.survey.PersonModel pydantic-model

Person attributes for tour building.

Fields:

person_id pydantic-field 

person_id: int

hh_id pydantic-field 

hh_id: int

person_num pydantic-field 

person_num: int

age pydantic-field 

age: AgeCategory

gender pydantic-field 

gender: Gender

work_lat pydantic-field 

work_lat: float | None

work_lon pydantic-field 

work_lon: float | None

school_lat pydantic-field 

school_lat: float | None

school_lon pydantic-field 

school_lon: float | None

industry pydantic-field 

industry: Industry | None = None

occupation pydantic-field 

occupation: Occupation | None = None

job_type pydantic-field 

job_type: JobType | None = None

employment pydantic-field 

employment: Employment

student pydantic-field 

student: Student

school_type pydantic-field 

school_type: SchoolType | None

work_park pydantic-field 

work_park: WorkParking | None

work_mode pydantic-field 

work_mode: Mode | None

race pydantic-field 

race: Race | None = None

ethnicity pydantic-field 

ethnicity: Ethnicity | None = None

telework_freq pydantic-field 

telework_freq: CommuteFreq | None = None

commute_freq pydantic-field 

commute_freq: CommuteFreq | None = None

education pydantic-field 

education: Education | None = None

commute_subsidy_provide_free_parking pydantic-field 

commute_subsidy_provide_free_parking: BooleanYesNo | None = None

commute_subsidy_provide_discounted_parking pydantic-field 

commute_subsidy_provide_discounted_parking: BooleanYesNo | None = (
    None
)

commute_subsidy_use_free_parking pydantic-field 

commute_subsidy_use_free_parking: BooleanYesNo | None = None

commute_subsidy_use_discounted_parking pydantic-field 

commute_subsidy_use_discounted_parking: BooleanYesNo | None = None

is_proxy pydantic-field 

is_proxy: bool | None = None

surveyable pydantic-field 

surveyable: bool | None = None

The survey could collect this person's travel at all. Unrelated household members (e.g. roommates) are enumerated for household composition and weighting but file no travel, and the vendor gives them no day rows whatsoever. Read by the completeness cascade, which excludes them from the household-day reductions so they cannot veto a date they were never asked about, and by the persons->days required-child constraint. Null counts as surveyable.

num_days_complete pydantic-field 

num_days_complete: int = 0

survey_complete pydantic-field 

survey_complete: bool | None = None

data_canon.models.survey.PersonDayModel pydantic-model

Daily activity pattern summary with clear purpose-specific counts.

Fields:

person_id pydantic-field 

person_id: int

day_id pydantic-field 

day_id: int

hh_id pydantic-field 

hh_id: int

travel_date pydantic-field 

travel_date: datetime

travel_dow pydantic-field 

travel_dow: TravelDow

begin_day pydantic-field 

begin_day: BeginEndDay | None = None

Where the respondent said the day began. Null where the survey did not ask. HOME or OTHER_HOME puts a home of theirs at the day's first trip origin; away from every reported home, that is another home.

end_day pydantic-field 

end_day: BeginEndDay | None = None

Where the respondent said the day ended. Null where the survey did not ask. HOME or OTHER_HOME puts a home of theirs at the day's last trip destination; away from every reported home, that is another home.

survey_complete pydantic-field 

survey_complete: bool | None = False

hh_day_survey_complete pydantic-field 

hh_day_survey_complete: bool | None = None

Household-day coherence: every member of the household reported a complete day on this travel_date. Stamped by the cascade_completeness step as an ALL reduction over member-days. Under a profile that asks for whole household-days, a day is only usable within one.

data_canon.models.survey.UnlinkedTripModel pydantic-model

Trip data model for validation.

Fields:

Validators:

unlinked_trip_id pydantic-field 

unlinked_trip_id: int

day_id pydantic-field 

day_id: int

person_id pydantic-field 

person_id: int

hh_id pydantic-field 

hh_id: int

linked_trip_id pydantic-field 

linked_trip_id: int

linked_trip_num pydantic-field 

linked_trip_num: int | None = None

Sequence number of the linked trip this segment belongs to, within its person-day.

tour_id pydantic-field 

tour_id: int | None

joint_tour_id pydantic-field 

joint_tour_id: int | None = None

The joint tour this segment's tour belongs to, if any.

o_lon pydantic-field 

o_lon: float

o_lat pydantic-field 

o_lat: float

d_lon pydantic-field 

d_lon: float

d_lat pydantic-field 

d_lat: float

o_purpose pydantic-field 

o_purpose: Purpose

d_purpose pydantic-field 

d_purpose: Purpose

o_purpose_category pydantic-field 

o_purpose_category: PurposeCategory

d_purpose_category pydantic-field 

d_purpose_category: PurposeCategory

mode_type pydantic-field 

mode_type: ModeType

mode_1 pydantic-field 

mode_1: Mode | None

mode_2 pydantic-field 

mode_2: Mode | None

mode_3 pydantic-field 

mode_3: Mode | None

mode_4 pydantic-field 

mode_4: Mode | None

tnc_type pydantic-field 

tnc_type: TNCType | None

duration_minutes pydantic-field 

duration_minutes: float

distance_meters pydantic-field 

distance_meters: float

depart_time pydantic-field 

depart_time: datetime | None

arrive_time pydantic-field 

arrive_time: datetime | None

num_travelers pydantic-field 

num_travelers: int

survey_complete pydantic-field 

survey_complete: bool | None = None

validate_arrival_after_departure pydantic-validator 

validate_arrival_after_departure() -> UnlinkedTripModel

Ensure arrive_time is after depart_time.

Raises:

Type Description
ValueError

If arrival time is before or equal to departure time

data_canon.models.survey.LinkedTripModel pydantic-model

Linked Trip data model for validation.

Fields:

day_id pydantic-field 

day_id: int

person_id pydantic-field 

person_id: int

hh_id pydantic-field 

hh_id: int

linked_trip_id pydantic-field 

linked_trip_id: int

joint_trip_id pydantic-field 

joint_trip_id: int | None = None

tour_id pydantic-field 

tour_id: int

travel_dow pydantic-field 

travel_dow: TravelDow

o_purpose pydantic-field 

o_purpose: Purpose

o_purpose_category pydantic-field 

o_purpose_category: PurposeCategory

o_lat pydantic-field 

o_lat: float

o_lon pydantic-field 

o_lon: float

d_purpose pydantic-field 

d_purpose: Purpose

d_purpose_category pydantic-field 

d_purpose_category: PurposeCategory

d_lat pydantic-field 

d_lat: float

d_lon pydantic-field 

d_lon: float

mode_type pydantic-field 

mode_type: ModeType

driver pydantic-field 

driver: Driver

num_travelers pydantic-field 

num_travelers: int

access_mode pydantic-field 

access_mode: AccessEgressMode | None = None

egress_mode pydantic-field 

egress_mode: AccessEgressMode | None = None

duration_minutes pydantic-field 

duration_minutes: float

distance_meters pydantic-field 

distance_meters: float

depart_time pydantic-field 

depart_time: datetime

arrive_time pydantic-field 

arrive_time: datetime

d_activity_duration pydantic-field 

d_activity_duration: int

Whole minutes spent at the trip destination before the next departure (activity duration at the destination). Sentinel values: -1 when the destination is home; -2 for the last trip of a person-day (no subsequent departure).

tour_direction pydantic-field 

tour_direction: TourDirection

tour_num pydantic-field 

tour_num: int | None = None

The trip's tour, numbered within its person-day.

subtour_num pydantic-field 

subtour_num: int | None = None

Which at-work/at-school subtour of the parent tour this trip belongs to; 0 when the trip is on the parent tour itself.

parent_tour_id pydantic-field 

parent_tour_id: int | None = None

The containing tour for a subtour trip. Primary tours self-reference.

tour_purpose pydantic-field 

tour_purpose: PurposeCategory | None = None

The primary purpose of the tour this trip belongs to.

joint_tour_id pydantic-field 

joint_tour_id: int | None = None

The joint tour this trip's tour belongs to, if any.

linked_trip_num pydantic-field 

linked_trip_num: int | None = None

This linked trip's sequence number within its person-day.

num_segments pydantic-field 

num_segments: int | None = None

Unlinked trip segments merged into this linked trip. Greater than one where access/egress or mode-change legs were joined.

travel_duration_minutes pydantic-field 

travel_duration_minutes: int | None = None

Minutes in motion, summed across the merged segments.

dwell_duration_minutes pydantic-field 

dwell_duration_minutes: int | None = None

Minutes waiting between the merged segments (e.g. a transfer), summed over each gap from one segment's arrival to the next segment's departure. Zero for a single-segment trip, which has no gap to wait in. Not exactly duration_minutes minus travel_duration_minutes: those two round separately, so the identity holds only to within a minute.

o_location_type pydantic-field 

o_location_type: LocationType

Classified location type of the trip origin (Home/Work/School/Other).

d_location_type pydantic-field 

d_location_type: LocationType

Classified location type of the trip destination (Home/Work/School/Other).

survey_complete pydantic-field 

survey_complete: bool | None = None

data_canon.models.survey.TourModel pydantic-model

Tour-level records with clear, descriptive schema_field names.

Fields:

Validators:

tour_id pydantic-field 

tour_id: int

hh_id pydantic-field 

hh_id: int

person_id pydantic-field 

person_id: int

day_id pydantic-field 

day_id: int

tour_num pydantic-field 

tour_num: int

subtour_num pydantic-field 

subtour_num: int

parent_tour_id pydantic-field 

parent_tour_id: int

joint_tour_id pydantic-field 

joint_tour_id: int | None = None

tour_purpose pydantic-field 

tour_purpose: PurposeCategory | None = None

The tour's primary activity: the highest-priority purpose among its destinations, ranked by the person's category (a worker's work trip outranks their shopping stop). Null where no stop qualifies -- every candidate was the return leg to the anchor or a mode change. On a closed tour that is NO_DESTINATION; on a partial one the missing half is the reason, so tour_data_quality names the open end instead.

tour_type pydantic-field 

tour_type: TourType

What the tour is anchored on: home for a home-based tour, the workplace or campus for a subtour. Orthogonal to tour_category, which grades how completely the tour reaches that anchor.

tour_category pydantic-field 

tour_category: TourCategory

tour_data_quality pydantic-field 

tour_data_quality: TourDataQuality

What is structurally wrong with this tour, or VALID. Derived from the tour's own trips only -- never from survey completeness or household coherence, which have their own columns. Codes 2-4 mirror tour_category value for value. Not itself a gate: which codes are admissible is a usability profile's decision.

trip_count pydantic-field 

trip_count: int

Linked trips in the tour. Stop counts are a formatter concern: they are a modelling convention, and a directional count has to come from the trips anyway.

origin_depart_time pydantic-field 

origin_depart_time: datetime

origin_arrive_time pydantic-field 

origin_arrive_time: datetime

dest_arrive_time pydantic-field 

dest_arrive_time: datetime | None = None

dest_depart_time pydantic-field 

dest_depart_time: datetime | None = None

origin_linked_trip_id pydantic-field 

origin_linked_trip_id: int

dest_linked_trip_id pydantic-field 

dest_linked_trip_id: int | None = None

o_lat pydantic-field 

o_lat: float

o_lon pydantic-field 

o_lon: float

d_lat pydantic-field 

d_lat: float

d_lon pydantic-field 

d_lon: float

o_location_type pydantic-field 

o_location_type: LocationType

d_location_type pydantic-field 

d_location_type: LocationType

tour_mode pydantic-field 

tour_mode: ModeType

outbound_mode pydantic-field 

outbound_mode: ModeType | None

inbound_mode pydantic-field 

inbound_mode: ModeType | None

num_travelers pydantic-field 

num_travelers: int = 1

survey_complete pydantic-field 

survey_complete: bool | None = None

validate_complete_tours pydantic-validator 

validate_complete_tours() -> TourModel

Validate that a tour with a primary destination describes it fully.

A null tour_purpose is a legitimate outcome, not an error: a tour whose every candidate stop was the return leg or a mode change has no activity to anchor on, and extraction records that as TourDataQuality.NO_DESTINATION. What must hold is the converse -- once a tour has a purpose, it reached a primary destination, so the arrival and the trip it was drawn from cannot be missing.

Departure is not in that set. A tour only leaves its destination if the diary kept watching: one cut at the diary edge, or resuming the next day, arrived somewhere and stopped. tour_category already records which end is open, so a departure is required exactly of the tours that closed -- demanding it of the rest would reject a partial tour for being partial, which is what the quality codes exist to describe.

data_canon.models.survey.JointTripModel pydantic-model

Joint trip group containing multiple linked trips from same household.

Represents a detected shared trip where several household members travelled together. This is a linking table: it records which member trips form the group, and nothing about where or when they went. Those facts stay on the member trips, which hold them first-hand.

Collapsing the members into one place or time means choosing a rule -- an average names a point none of them reported, a representative privileges one of them -- and which rule is right depends on what the answer is for. That choice belongs to whoever is asking, so it is made downstream rather than frozen here. Mirrors JointTourModel.

This table is an overlay on linked_trips, not a partition of it: every member trip is still a row there in its own right. See joint_trip_weight.

Fields:

joint_trip_id pydantic-field 

joint_trip_id: int

hh_id pydantic-field 

hh_id: int

travel_date pydantic-field 

travel_date: datetime

The household date this grouping happened on, shared by every member's day. Not any one member's day_id: a grouping belongs to the household's date, and a member's own day can be unusable while the others' are not.

num_joint_travelers pydantic-field 

num_joint_travelers: int

Detected group size: how many member trips the clique detection put in this joint trip. Surveyed household members only - the reported travel party, which may include people outside the survey, is num_travelers on the member trips. Not a weighting quantity: it does not track which members the weighting later excluded.

survey_complete pydantic-field 

survey_complete: bool | None = None

data_canon.models.survey.JointTourModel pydantic-model

Joint tour group containing the tours of several household members.

A joint tour is a meta-entity: the participants' individual tours remain the real records and keep their own tour_weight. This table exists so the group can be counted and weighted once, rather than inferred by grouping tours on joint_tour_id at every point of use.

Its weight is the SUM of its member tours -- person-tours, matching joint_trips. Membership requires at least two participants -- a group of one is not joint.

Fields:

joint_tour_id pydantic-field 

joint_tour_id: int

hh_id pydantic-field 

hh_id: int

travel_date pydantic-field 

travel_date: datetime

The household date this grouping happened on, shared by every member's day. Not any one member's day_id: a grouping belongs to the household's date, and a member's own day can be unusable while the others' are not.

num_participants pydantic-field 

num_participants: int

Group size: household members on this joint tour, whether or not each carries weight. Not a weighting quantity - it does not track which members the weighting excluded.

survey_complete pydantic-field 

survey_complete: bool | None = None

data_canon.models.survey.HabitualLocationModel pydantic-model

A place a person habitually uses: their home, workplace or school.

This is the tall replacement for the wide scalar work_lat/work_lon, school_lat/school_lon columns: one row per known location, so a person can hold multiple locations of the same kind (e.g. a primary workplace plus a recurring alternate worksite) with per-row provenance.

The table is deliberately not a catalogue of every place a person went — discretionary stops are not habitual locations and do not appear here. What distinguishes a habitual location is the person's standing relationship to it, which is what source and is_primary record.

location_num numbers locations of the same kind for a person: reported ones first, primary at 1, then the rest in the order the person was first seen there. An alternate worksite is therefore just a WORK row with location_num > 1 and source = OBSERVED — there is no separate ALTERNATE_WORK kind. An "other home" (second home, other parent's house) is likewise a HOME row with location_num > 1. Homes are never inferred from travel: every home is one the respondent reported or named as where a day began or ended.

Survey cleaning delivers the table holding the reported locations only — the survey's home, work and school coordinates plus any further ones, such as a vendor's second home — numbered and identified. The detect_habitual_locations step appends the observed ones after them and never changes a delivered row. Each reported primary repeats its coordinate columns (home_lat/lon on households, work_*/school_* on persons), and validation holds the two equal.

Per-day use of a location lives in HabitualLocationDayModel, keyed on habitual_location_id. A location exists here whether or not it was visited during the diary period — a reported workplace belonging to someone who teleworked all week is still their workplace — which is why the two are separate tables.

Fields:

habitual_location_id pydantic-field 

habitual_location_id: int

Identifier for this location, formed as person_id * 1000 + location_type * 100 + location_num.

person_id pydantic-field 

person_id: int

location_type pydantic-field 

location_type: LocationType

Kind of location (Home/Work/School/Other).

location_num pydantic-field 

location_num: int

Sequence number of this location within its kind for the person (1 = primary when a primary is known).

is_primary pydantic-field 

is_primary: bool | None = None

Whether this is the person's primary location of its kind, as stated in the survey. None when primacy is unknown (e.g. a multi-site worker with no single usual workplace).

lat pydantic-field 

lat: float

lon pydantic-field 

lon: float

source pydantic-field 

source: LocationSource

How this location was established (Reported/Observed/Imputed).

data_canon.models.survey.HabitualLocationDayModel pydantic-model

One person-day's use of one habitual location.

The per-day companion to HabitualLocationModel: a row exists for every day the person was present at that location. Rows are recorded for all habitual locations, reported ones included, so that "did this person visit their usual workplace on day D" is a lookup rather than a re-derivation from trips.

These are presence facts, not conclusions: the table records how long somebody was somewhere and how the day related to it, never which location "was" the day's anchor. That judgement belongs to whichever consumer is making it, and different consumers may weigh these facts differently (by duration, by recurrence, by a purpose hierarchy).

Fields:

habitual_location_id pydantic-field 

habitual_location_id: int

person_id pydantic-field 

person_id: int

day_id pydantic-field 

day_id: int

location_type pydantic-field 

location_type: LocationType

Kind of location, denormalized from the parent row for filtering.

dwell_minutes pydantic-field 

dwell_minutes: float | None = None

Total measurable minutes spent at this location on this day. None when no visit that day had a measurable duration, which is not the same as zero: a stay that ran past the last trip of the day has no recorded end, and arrivals home carry no duration at all (see LinkedTripModel.d_activity_duration sentinels).

n_visits pydantic-field 

n_visits: int

Number of separate arrivals at this location on this day, counted whether or not the stay had a measurable duration.

is_day_start pydantic-field 

is_day_start: bool

Whether the person's first trip of the day departed from here.

is_day_end pydantic-field 

is_day_end: bool

Whether the person's last trip of the day arrived here, i.e. they were still here when the diary day ended.

data_canon.models.weighting.HouseholdWeightingModel pydantic-model

Weighting geography and seed weight for one household.

Fields:

hh_id pydantic-field 

hh_id: int

study_geoid pydantic-field 

study_geoid: str | None = None

Geography the household is assigned to for weighting (e.g. county FIPS).

ctrl_geoid pydantic-field 

ctrl_geoid: str | None = None

The control geography study_geoid maps into. Several study geographies can share one control zone where sample is too thin to balance them separately.

bg_geo_id pydantic-field 

bg_geo_id: str | None = None

Census block group GEOID containing the home location.

Travel Model-formatted Data Models

DaySim Models

Output file format models for the DaySim activity-based travel demand model.

CTRAMP Models

Output file format models for the CT-RAMP travel demand model.