processing.weighting

Survey weighting module.

This module provides two pipeline options for attaching expansion weights to survey tables:

add_existing_weights -- load pre-computed weights from CSV files and join them to tables, optionally deriving missing weights by propagating values through the survey hierarchy.
compute_weights -- compute weights from scratch using PUMS / ACS microdata as population controls via maximum-entropy balancing.

The compute_weights step orchestrates the full pipeline — see compute_weights.py for the detailed step-by-step description.

Weight hierarchy

Every weight is written once per profile, suffixed with its name (hh_weight_<profile>, person_weight_<profile>, ...):

hh_weight
  └─ person_weight        (household's weight, shared among its usable persons)
      └─ day_weight        (split: a person's usable days sum to their weight)
          └─ unlinked_trip_weight  (day's weight, shared within the day)
              ├─ linked_trip_weight   (mean agg via linked_trip_id)
              │   └─ joint_trip_weight (SUM agg via joint_trip_id)
              └─ tour_weight          (mean agg via tour_id)
                  └─ joint_tour_weight (SUM agg via joint_tour_id)

Module structure

weighting/
├── existing_weights.py       # attach pre-computed weights
├── existing/                 # its settings and redistribution onto usable records
├── compute_weights.py        # single @step() entry point for full weighting pipeline
├── core/                     # WeightingPipeline, the hierarchy, propagation, settings
├── controls/                 # control variable definitions & registry
├── data_prep/                # PUMS I/O, control totals, survey seed, geography
├── balancing/                # balancer, base weights, importance, merges
├── diagnostics/              # interactive HTML report (Plotly + Jinja2)
└── validation/               # post-balancing sanity checks

processing.weighting.compute_weights

Entry point for the weighting pipeline step.

Orchestrates the full weighting pipeline in the following stages:

A. Geography Crosswalk

Setup -- register controls from YAML config, build the geographic crosswalk (translating between Census PUMAs and the project's custom weighting geography using block-group population as the intermediary), and prepare the sample plan.

B. Control Data Preparation

PUMS recoding -- load PUMS 1-year microdata and recode into the YAML-configured variable bins used by the controls.
Merges -- apply any user-specified category merges (global or zone-specific) to both controls and the survey incidence table.
Control aggregation -- apply the crosswalk to PUMS and aggregate into marginal control totals per zone.

C. Survey Seed Preparation

Survey recoding -- recode canonical survey variables into the same bin / group categories as the PUMS controls.
Null imputation -- fill null-induced zeros in the survey incidence table with RF-predicted fractional class probabilities.
Zone assignment -- assign survey households to weighting zones via the geographic crosswalk.

D. Maximum-Entropy Balancing

Importance resolution -- compute per-control importance weights from replicate-weight MOE or explicit YAML overrides.
Balancing -- build per-household incidence vectors and fit weights using a numba-accelerated max-entropy balancer (vendored from PopulationSim), independently per zone, once per profile.
Weight propagation -- propagate each profile's hh_weight_<profile> to all canonical tables (persons, days, trips, tours).

E. Diagnostics & Validation

Diagnostics -- generate one self-contained interactive HTML report for the run, comparing every profile fitted and carrying each one's convergence, fit, weight-quality and weight-cascade diagnostics.
Validation -- run sanity checks on the final weights and control totals. The fit against the controls is logged; the hierarchy identities, the joint-table sums and weight placement raise when they fail. None of it is in the HTML report -- check the pipeline log.

compute_weights

compute_weights(
    census: CensusConfig,
    geography: GeographyConfig,
    balancing: BalancingConfig,
    controls: list[dict],
    diagnostics: DiagnosticsConfig,
    weight_profiles: list[str],
    max_unplaceable_share: float,
    sample_plan: str | None,
    strict_survey_nulls: bool = False,
    *,
    pipeline_cache: PipelineCache | None = None,
    canonical_data: CanonicalData | None = None,
    households: pl.DataFrame | None = None,
    persons: pl.DataFrame | None = None,
    days: pl.DataFrame | None = None,
    unlinked_trips: pl.DataFrame | None = None,
    linked_trips: pl.DataFrame | None = None,
    joint_trips: pl.DataFrame | None = None,
    tours: pl.DataFrame | None = None,
    joint_tours: pl.DataFrame | None = None
) -> dict[str, pl.DataFrame]

Compute expansion weights from PUMS controls and propagate to all tables.

Constructs a WeightingPipeline; full documentation of the algorithm, configuration, and diagnostics is included in that class.

Parameters:

Name	Type	Description	Default
`census`	`CensusConfig`	Where the Census controls come from. See `CensusConfig`.	required
`geography`	`GeographyConfig`	The zones weights are balanced over. See `GeographyConfig`.	required
`balancing`	`BalancingConfig`	Solver bounds and control importance. See `BalancingConfig`.	required
`controls`	`list[dict]`	The control list, each by registry name with optional `importance` and `merge`.	required
`diagnostics`	`DiagnosticsConfig`	The report. See `DiagnosticsConfig`.	required
`weight_profiles`	`list[str]`	Usability profiles to weight, one fit each; a single profile is a list of one.	required
`max_unplaceable_share`	`float`	The most of the survey whose homes may fall outside the control geography before the step raises.	required
`sample_plan`	`str \| None`	Path to the sample plan for base weights; null for none.	required
`strict_survey_nulls`	`bool`	Raise on survey nulls in control variables rather than imputing them.	`False`
`pipeline_cache`	`PipelineCache \| None`	Supplied by the pipeline.	`None`
`canonical_data`	`CanonicalData \| None`	Supplied by the pipeline.	`None`
`households`	`pl.DataFrame \| None`	Households.	`None`
`persons`	`pl.DataFrame \| None`	Persons.	`None`
`days`	`pl.DataFrame \| None`	Days.	`None`
`unlinked_trips`	`pl.DataFrame \| None`	Unlinked trips.	`None`
`linked_trips`	`pl.DataFrame \| None`	Linked trips.	`None`
`joint_trips`	`pl.DataFrame \| None`	Joint trips.	`None`
`tours`	`pl.DataFrame \| None`	Tours.	`None`
`joint_tours`	`pl.DataFrame \| None`	Joint tours.	`None`

processing.weighting.existing_weights

Attach pre-computed weights to survey data tables.

This module provides the add_existing_weights pipeline step, which loads weight CSV files and joins them to the corresponding canonical tables. Optionally, it can derive missing weights by propagating values through the survey hierarchy.

Core algorithm

Phase 1 -- Load and Join Weights

For each provided weight config:
1. Validate the config key against allowed table types.
2. Load the weight CSV from weight_path.
3. Validate required ID and weight columns exist.
4. Handle ID column name mismatches (rename if needed).
5. Left-join weights to the table on the ID column.

Phase 2 -- Move weight off dropped records, per profile

Households, then persons within their household, then each person's days, then each day's trips. See the Existing Weights docs page for the rules and worked examples.

Phase 3 -- Derive Missing Weights (when `derive_missing_weights=True`)

Down the hierarchy for household → person → day → unlinked_trip: validate parent has weights, then share the parent's weight among its usable records (a person's usable days split their weight, rather than each carrying all of it).
Aggregated weights for linked_trip and tour: the mean of their members' final weights (excluding nulls and zeros). The joint levels sum instead -- see Agg.

Gap detection: raises an error if a middle-tier weight is missing (e.g. household + trip weights provided but person/day weights are not).

add_existing_weights

add_existing_weights(
    weights: dict[str, ExistingWeightConfig | dict],
    existing: ExistingWeightsConfig,
    canonical_data: CanonicalData | None = None,
    households: pl.DataFrame | None = None,
    persons: pl.DataFrame | None = None,
    days: pl.DataFrame | None = None,
    unlinked_trips: pl.DataFrame | None = None,
    linked_trips: pl.DataFrame | None = None,
    tours: pl.DataFrame | None = None,
    joint_trips: pl.DataFrame | None = None,
    joint_tours: pl.DataFrame | None = None,
) -> dict[str, pl.DataFrame]

Attach existing weights to the data.

Loads weights from provided files and attaches them to the data.

For any tables that exist, and do not have weights provided, we can optionally derive missing weights from the next logical upstream table.

For example, if only household weights exist, persons share their household's weight and each person's days split that person's weight. If trip (unlinked) weights exist but not linked trips or tours, those are the mean of their trips.

If a "middle" weight is missing, an error will be raised if derive_missing_weights is True. For example, if household and trip weights are provided, but not person or day weights, an error will be raised as this likely indicates a misconfiguration.

Weight hierarchy logic

hh_weight
  └─ person_weight        (carry forward via hh_id)
      └─ day_weight        (split: person_weight / n_usable_days)
          └─ unlinked_trip_weight  (carry forward via day_id)
              ├─ linked_trip_weight   (mean agg via linked_trip_id)
              │   └─ joint_trip_weight (SUM agg via joint_trip_id)
              └─ tour_weight          (mean agg via tour_id)
                  └─ joint_tour_weight (SUM agg via joint_tour_id)

The shape is declared once in HIERARCHY; this step only supplies the weights it is given and derives the rest from it.

Supplied weights move household -> person -> day -> trip, each level only within the level above it; the rules, the arithmetic and worked examples are on the Existing Weights docs page. In short: households scale by one factor per household size; persons copy their household, a dropped person's share staying inside it; a person's days carry exactly that person's weight; a trip is scaled by its day's factor; linked trips and tours are the mean of their final trips.

Checked before the step returns, per profile, raising on failure:

the household total equals the supplied total;
hh_weight x num_persons == sum(person_weight) for every household;
each person's days sum to that person's weight, when both are supplied;
each derived level against the identity it was derived by;
no record grew past its max_weight_factor;
no dropped record carries weight, and the formatter's own record selection keeps every weight.

The person total is logged against the supplied total rather than held: size groups hold it exactly, except where the top group joins sizes. Not held, by design: the trip, linked trip and tour totals.