Completeness & Usability

The single place the canonical completeness and usability flags are computed. "Flag definitions" below carries the definitions, the derivation-order diagram, and the per-level rule table; the pipeline step (cascade_completeness) stamps them once so every downstream consumer only reads the result.

Two kinds of flag, computed side by side:

• survey_complete — survey reporting completeness (vendor-provided at the day level, possibly adjusted by the project cleaner), cascaded through household ↔ person ↔ day ↔ trip/tour so it is internally consistent. One answer per run, never configurable.
• one column per usability profile — the subset of survey_complete that profile admits. A project names its profiles in config, so several standards can coexist in one run and a column's meaning reads off the config.

A profile answers four axes, all explicitly, none with a default:

axis	values	meaning
tour_closes_at	primary_home	only a tour that returns to the home it left
	any_home	also one closing at another home of this person's
	anywhere	also the diary-edge and day-split open ends
household_day_needs	all_members	every surveyable member's day complete on that date
	nothing	no household-date requirement
zone_coverage	a zone_name that add_zone_ids builds	every location on the record has a zone in that system
	none	no geographic requirement
joint_tours	fully_joint	everyone on a joint tour shares every leg, as activity-based models draw it; work, school and escort tours are never joint
	partial	the joint groups the survey detected, as they are

No setting admits a tour with missing data (NO_DESTINATION, SPATIAL_GAP): those are not open ends, and no tolerance makes a missing leg present.

Example

- name: cascade_completeness
  params:
    usability_profiles:
      ctramp:
        tour_closes_at: primary_home
        household_day_needs: all_members
        zone_coverage: TAZ1454
        joint_tours: fully_joint
      analysis:
        tour_closes_at: anywhere
        household_day_needs: nothing
        zone_coverage: none
        joint_tours: partial

One profile, several column families

A profile is a name. Every column belonging to it is that name suffixed onto a family, so the family says what kind of thing a column is and the suffix says whose:

family	column	written by
usable_	usable_ctramp	the per-record verdict on every table, joint tables included, from this step
hh_day_	hh_day_ctramp	whether every surveyable member's day passed, that date
joint_tour_id_, joint_trip_id_	joint_tour_id_ctramp	the joint group a record is on for that profile, null where the profile splits it off
hh_weight_, day_weight_, …	hh_weight_ctramp	the weight fitted over that profile's universe
base_weight_	base_weight_ctramp	the seed weight its fit started from

Two profiles can therefore never write the same column, whatever they are called: the families have disjoint prefixes, and a name is unique within one.

survey reporting completeness sits outside all of it. survey_complete and hh_day_survey_complete record what the survey collected rather than what a model will take, so they carry no family prefix and are reserved — a profile may not take either name.

Downstream consumers name the profiles they honour: the weighting takes weight_profiles, one weight set per profile, and the CT-RAMP and DaySim formatters each take usability_profile. None re-derive the verdict: the column is read, and its absence raises.

That one name is the whole address. It resolves to usable_<profile> for the records a consumer keeps and to hh_weight_<profile> for the weights it reads, so there is no second setting that could disagree with the first. A consumer naming a profile the weighting was not asked to fit raises rather than falling back to an unsuffixed column, since that column describes some other universe, or none.

Flag definitions

Canonical completeness and model-usability logic.

Two kinds of flag roll up from two atomic facts -- whether each day was surveyed (days.survey_complete, which the project cleaner sets from the day's surveyed trips or a declared no-travel day), and each tour's structural validity:

• survey_complete -- was it reported? Rolls up from complete days. One answer per run, never narrowed by model criteria and never configurable: if it is wrong, the fix belongs upstream in the project cleaner.
• one column per usability profile -- can the model consume it? Fuses each tour's structure with its household-day coherence, then gates the rest.

A profile states a standard on four axes -- which home has to close a tour, what the household-date has to show, which zone system has to be able to address the record, and what a joint tour has to be -- and a run may stamp several, so different consumers can hold different standards. A joint-tour model needs whole households and fully joint tours; a trip-level estimation does not, and needs no model geography at all.

Every profile is named in config and answers every axis, so a column's meaning reads off the config without knowing a base rule, and no verdict appears that nobody asked for. Consumers name the profiles they read (a formatter's usability_profile, the weighting's weight_profiles), and survey_complete is always available as the floor beneath all of them.

Each flag comes from one place in the tree -- its direction -- via a counting operation (the op column below). Naming the direction makes it clear that household-day and the joint entities are the same kind of thing: cross-person groupings, not parent/child.

direction	value comes from	example
self	measured on the record	a trip never filled out
up	aggregate your own children	person with zero complete days
down	inherit your parent's verdict	good trip on a dropped tour
lateral	aggregate a cross-person group	one member skips a day, whole date fails

The flow, and the rule on each line:

COMPLETE -- rolls up from complete days     op        rule
------------------------------------------------------------------------
person-day ...........................   direct    days.survey_complete, from the cleaner
                                                    (all trips surveyed, or no-travel)
 ├ trip / tour .......................   inherit   its own value AND its day's
 ├ person ............................   >=1       has >=1 complete day
 └ household-day .....................   ALL       all surveyable members complete that date
    └ household ......................   >=1       has >=1 complete household-day

USABLE, UP (_cascade_up) -- counts from the tour fuse   op   rule
------------------------------------------------------------------------
household ...........................   >=1       has >=1 usable household-day
 ├ household-day ....................   ALL       all surveyable members' days usable that date
 └ person ...........................   >=1       has >=1 usable day
    └ day ...........................   >=1       >=1 usable tour (or a no-travel day)
       └ tour .......................   fuse      complete AND admitted-quality AND
                                                     hh-day complete AND has a zone
                                                     (the last three are the axes)

USABLE, DOWN (_cascade_down) -- inherits from the household   op   rule
------------------------------------------------------------------------
household -> person -> day -> tour ..   inherit   unusable if the one it belongs to is
 └ linked trip / unlinked trip ......   inherit   takes its tour's verdict

JOINT (flag_joint) -- judged beside the chain, after it     op   rule
------------------------------------------------------------------------
joint tour ..........................   >=2       >=2 tours on it, and its
                                                  household usable
joint trip ..........................   >=2       >=2 linked trips on it, likewise;
                                                  judged again after rejected joint
                                                  tours void their ids

JOINT IDS -- joint_tour_id_<profile> on tours and both trip tables, and
joint_trip_id_<profile> on linked trips: the bare id where the record is on it,
null where it is not. Decided first within flag_joint, read by the groupings
above, the weighting and the formatters; the bare id is the survey's own
answer and is never edited.
------------------------------------------------------------------------
partial .............................   direct    usable and in a survey-detected group
fully_joint .........................   class     usable, not work/school/escort, and
                                                  made the same joint trips as the
                                                  others (largest class of identical
                                                  legs); then the tour needs >=2 such
                                                  people and one purpose, or everyone
                                                  on it travels alone

declared no-travel: num_reasons_no_travel >= 1, OR proxy_complete (a proxy filled the day in --
  this is how children, who file no trips themselves, still get a complete day).
surveyable: persons whose travel the survey could collect at all. Unsurveyable persons
  (unrelated members, e.g. roommates) have no day rows in the vendor data; where a source
  carries any, they neither veto the household-day ALL reductions nor inherit their verdict.
VALID feeds the fuse: trips -> linked trips -> tour, home-to-home, no missing legs.
op: ALL / >=1 / >=2 = quantity gate (count members vs threshold);
    direct = measured; inherit = take a neighbour's verdict; fuse = AND of conditions

Because each level reads its neighbours, the derivation order is load-bearing; getting it wrong fails loudly (an unflagged member table raises), never silently.

This package is the one place the logic lives. The cascade_completeness pipeline step runs :func:cascade_complete once and then :func:stamp_usable per profile; every downstream consumer only reads the resulting flags.

Modules:

• profiles: what a profile is, and how config states one.
• survey_complete: the reporting cascade, once per run.
• household_day: the household-date reduction, for both kinds of flag.
• zone_coverage: whether a record's locations have a zone.
• usable_tours: the tour fuse, where a verdict is decided.
• usable_cascade: that verdict, walked up and then down every table.
• joint_groupings: which records are on a joint tour or joint trip, per profile.
• kept_records: which records survive a verdict once references are restored.
• descriptions: what each stamped column means, in words.
• cascade_completeness: the pipeline step.

The step

Cascade survey_complete through the hierarchy and stamp one column per profile.

Walks both kinds of flag across every table so they are internally consistent with whatever was set upstream (the vendor's day-level completeness, any manual adjustments in the project cleaner, and the structural verdicts from tour extraction):

• survey_complete -- survey reporting completeness, cascaded hh <-> person <-> day <-> trip/tour. Never narrowed by model criteria, and never configurable: if it is wrong the fix belongs upstream in the cleaner.
• one column per usability profile -- the subset of survey_complete that profile admits.

A profile states its standard on four axes: which home has to close a tour, what the household-date has to show, which zone system must hold every location, and what a joint tour has to be. Several can be stamped in one run so different consumers can hold different standards -- a joint-tour model needs whole households and fully joint tours, a trip-level estimation does not:

usability_profiles:
  ctramp:
    tour_closes_at: primary_home
    household_day_needs: all_members
    zone_coverage: taz
    joint_tours: fully_joint
  analysis:
    tour_closes_at: anywhere
    household_day_needs: nothing
    zone_coverage: none
    joint_tours: partial

Nothing is implicit: every column stamped is named in config and every profile answers every axis. Consumers then choose which to honour by name (weight_profiles in the weighting, usability_profile in each formatter), and none of them re-derive completeness or tour validity -- the formatters read the column and raise if it is absent. survey_complete is always available as the floor without being declared.

See stamp_usable for the per-level rules and parse_usability_profiles for the vocabulary.

Settings

Key	Type	Default	Meaning
usability_profiles	mapping or null	null	Profile name -> {tour_closes_at, household_day_needs, zone_coverage, joint_tours}. Required, and every profile answers every axis: there is no default at either level, so a verdict never appears unasked and a column's meaning is never implicit.

Code: processing.completeness