Skip to content

processing.weighting.core

Shared machinery behind the two weighting entry points.

Neither module here is a pipeline step. The steps live one level up -- compute_weights (balance from controls) and add_existing_weights (attach supplied weights) -- and both are built from:

  • hierarchy -- the weight hierarchy, declared once.
  • propagation -- the walk that moves weights along it.
  • specs -- configuration and result types.
  • pipeline -- orchestration for the computed path.

Kept import-free so it cannot close a cycle with the sub-packages that read it.

processing.weighting.core.hierarchy

The canonical weight hierarchy, declared once.

Every table that carries a weight is declared here exactly once -- its weight column, where the weight comes from, and the group within which that weight is conserved. Everything else is a projection of that list, so the shape of the hierarchy cannot drift between the propagation (processing.weighting.core.propagation), the supplied-weight path and the checksum.

Weights disaggregate down from households and aggregate back up into groupings; both directions are edges of one tree.

Table Weight Derivation
households hh_weight Anchor -- from the balancer, or supplied
persons person_weight Down from hh_weight via hh_id
days day_weight Split: person_weight / n_usable_days
unlinked unlinked_trip_weight Down from day_weight via day_id
linked linked_trip_weight Up: mean of member unlinked_trip_weight
tours tour_weight Up: mean of member linked_trip_weight
joint trips joint_trip_weight Up: sum of member linked_trip_weight
joint tours joint_tour_weight Up: sum of member tour_weight

Two units

Most weights count one row of their own table. The joint levels instead sum their members, so they count person-trips -- the travel of the whole party, not the joint occasion -- and are therefore an overlay on the member table, never a partition of it: sum(linked) + sum(joint) double counts.

To count occasions instead, divide by the number of members that carried weight -- counted from the member table, not read off the grouping::

events = joint_trip_weight / n_member_trips_with_weight

A party-size column is not that divisor: it counts travellers, including any the weighting excluded.

One column set per profile

A run may weight several usability profiles, each fitted to its own universe. The columns above are then suffixed with the profile's name, verbatim and unabridged (hh_weight_ctramp), so a column names the universe it expands and no rule is needed to read one back. Only [weight_col_for][processing.weighting.core .hierarchy.weight_col_for] and its siblings spell that suffix; Level.weight_col stays the base name, which is what keeps id_col derivable.

HIERARCHY module-attribute

HIERARCHY: tuple[Level, ...] = (
    Level(
        table="households",
        weight_col="hh_weight",
        means="Households represented by this record.",
    ),
    Level(
        table="persons",
        weight_col="person_weight",
        means="Persons represented by this record.",
        parent="households",
        flow=Flow.DOWN,
        scope="hh_id",
    ),
    Level(
        table="days",
        weight_col="day_weight",
        means="Person-days represented by this record. A person's usable days sum to their person weight, the average-day convention, so a day is a share of that person rather than a count of its own.",
        parent="persons",
        flow=Flow.DOWN,
        scope="person_id",
        split=True,
    ),
    Level(
        table="unlinked_trips",
        weight_col="unlinked_trip_weight",
        means="Unlinked trips (single vehicle or mode segments) represented.",
        parent="days",
        flow=Flow.DOWN,
        scope="day_id",
    ),
    Level(
        table="linked_trips",
        weight_col="linked_trip_weight",
        means="Linked trips (whole origin-to-destination journeys) represented.",
        parent="unlinked_trips",
        flow=Flow.UP,
        scope="day_id",
    ),
    Level(
        table="joint_trips",
        weight_col="joint_trip_weight",
        means="Person-trips represented: the SUM of the member linked trip weights, so the record stands for its whole party. Same unit as linked_trips, which this table OVERLAYS rather than partitions, so summing both double counts.",
        parent="linked_trips",
        flow=Flow.UP,
        scope="hh_id",
        agg=Agg.SUM,
    ),
    Level(
        table="tours",
        weight_col="tour_weight",
        means="Tours represented by this record.",
        parent="linked_trips",
        flow=Flow.UP,
        scope="day_id",
    ),
    Level(
        table="joint_tours",
        weight_col="joint_tour_weight",
        means="Person-tours represented: the SUM of the member tour weights. Same unit as tours, which this table OVERLAYS rather than partitions, so summing both double counts.",
        parent="tours",
        flow=Flow.UP,
        scope="hh_id",
        agg=Agg.SUM,
    ),
)

LEVELS module-attribute

LEVELS: dict[str, Level] = {
    (level.table): level for level in HIERARCHY
}

TABLE_NAMES module-attribute

TABLE_NAMES: list[str] = [(level.table) for level in HIERARCHY]

WEIGHT_COLUMNS module-attribute

WEIGHT_COLUMNS: dict[str, str] = weight_columns_for(None)

WEIGHT_CONFIG_MAPPING module-attribute

WEIGHT_CONFIG_MAPPING: dict[str, tuple[str, str, str]] = {
    (level.weight_col): (
        level.table,
        level.id_col,
        level.weight_col,
    )
    for level in HIERARCHY
}

Supplied-weight config key -> (table, id column, weight column).

The config key is the weight column: one name for one thing, so a config cannot name a level the hierarchy does not have.

SEED_TABLE module-attribute

SEED_TABLE = 'household_weights'

Table the weighting's working columns move to before the data is handed back.

Flow

Direction a weight moves along a hierarchy edge.

DOWN class-attribute instance-attribute

DOWN = auto()

Disaggregate: the parent's weight is shared out among its children.

UP class-attribute instance-attribute

UP = auto()

Aggregate: a grouping combines the weights of its usable members.

Agg

How an UP edge combines its members' weights.

MEAN class-attribute instance-attribute

MEAN = auto()

The grouping counts once: one linked trip, one tour. Its weight is the mean of its usable members, so the weight stays in the unit of its own row.

SUM class-attribute instance-attribute

SUM = auto()

The grouping carries the combined weight of its members -- person-trips, not occasions. This puts it in the member table's unit while overlapping it, so summing both double counts.

Level dataclass

One table in the weight hierarchy.

Only what cannot be derived is declared: id_col and key follow mechanically from weight_col and the edge, so they are properties -- there is no way to spell them inconsistently.

Attributes:

Name Type Description
table str

Canonical table name.

weight_col str

Weight column this level owns. Names the level -- id_col is read off it, and it is the key naming this level in the supplied-weights config.

means str

What one unit of this weight counts, for the delivered codebook. Carried here rather than on a model field because the column's name comes from config once a run weights profiles separately, and a reader of the joint levels in particular needs telling: they are an overlay on their member table, not a partition of it.

parent str | None

Table the weight is derived from; None marks the anchor.

flow Flow | None

Which way the weight moves along the edge.

scope str | None

Group within which this weight is conserved.

split bool

If True, the parent's weight is divided equally among its usable children (parent_weight / n_usable) instead of copied to each. Days use this -- see processing.weighting.core.propagation.

agg Agg

How an UP edge combines its members. DOWN edges must leave it at the default.

table instance-attribute

table: str

weight_col instance-attribute

weight_col: str

means instance-attribute

means: str

parent class-attribute instance-attribute

parent: str | None = None

flow class-attribute instance-attribute

flow: Flow | None = None

scope class-attribute instance-attribute

scope: str | None = None

split class-attribute instance-attribute

split: bool = False

agg class-attribute instance-attribute

agg: Agg = Agg.MEAN

id_col property

id_col: str

Primary key of the table, e.g. tour_weight -> tour_id.

key property

key: str | None

The id column carried on this side of the edge.

DOWN edges carry the parent's id, identifying the record the weight comes from; UP edges carry their own, identifying the grouping a member belongs to.

weight_col_for

weight_col_for(profile: str | None) -> str

This level's weight column under profile.

The suffix is applied here and nowhere else, so weight_col -- and therefore id_col and key, which are read off it -- keeps its base name however many profiles a run weights.

member_key_for

member_key_for(
    usability_flag_col: str | None, columns: Collection[str]
) -> str

The column on the source table saying which grouping a record is on.

The joint groupings differ by profile: joint_tour_id_ctramp is the bare joint_tour_id where the tour is on it for ctramp and null where it is not, so summing by it counts exactly the tours that profile keeps together. Where the profile's copy is absent -- a single un-suffixed run, survey_complete, or a level with no per-profile copy -- the bare key is the answer.

Parameters:

Name Type Description Default
usability_flag_col str | None

The flag the weighting is gated on, from which the profile is read.

required
columns Collection[str]

Columns present on the source table.

required

Returns:

Type Description
str

The column to group the source by. Its values are the bare key's.

weight_col_for

weight_col_for(base: str, profile: str | None) -> str

The hierarchy weight column base as written under profile.

Parameters:

Name Type Description Default
base str

A declared weight column, e.g. "hh_weight".

required
profile str | None

Usability profile the fit was gated on, or None for a single un-suffixed set.

required

Raises:

Type Description
ValueError

If base is not a weight column the hierarchy declares. Checked so a caller cannot coin a column the hierarchy has no level for -- the suffix rule is only trustworthy if the stem is.

seed_col_for

seed_col_for(base: str, profile: str | None) -> str

The pre-balancing column base as written under profile.

Separate from weight_col_for because these columns belong to no level: they describe the seed a fit was solved from, not a weight the hierarchy propagates.

Raises:

Type Description
ValueError

If base is not a declared seed column.

weight_columns_for

weight_columns_for(profile: str | None) -> dict[str, str]

Map each table to its weight column under profile.

describe_weight_columns

describe_weight_columns(
    profiles: tuple[str | None, ...],
) -> dict[str, dict[str, GeneratedColumn]]

Describe and constrain every weight column a set of fits writes.

These columns are not model fields. A run that weights profiles separately names them from config, and even the un-suffixed set is a profile's weights with the label left off -- there is no un-profiled weight. So the promise a field would have carried is made here instead: what the number counts, and that it cannot be negative.

Parameters:

Name Type Description Default
profiles tuple[str | None, ...]

The profiles weighted. None means the un-suffixed set.

required

Returns:

Type Description
dict[str, dict[str, GeneratedColumn]]

One entry per canonical table that gained columns.

levels_with_flow

levels_with_flow(flow: Flow) -> list[Level]

Return the levels whose weight moves in flow, in hierarchy order.

processing.weighting.core.pipeline

Top-level weighting pipeline step.

Orchestrates the full weighting pipeline via WeightingPipeline:

  1. Setup — parse YAML config → specs, target names, merges, importance. Cross-tab controls are registered with pre-merged dimensions so the enum reflects the effective cell count.
  2. Data fetching — load PUMS (API or files); receive survey tables.
  3. Conformance — recode both PUMS and survey through identical control expressions → same control-column schema.
  4. Incidence pivot — unified pivoter produces identical {ctrl}__{member} column layout for both datasets.
  5. Zone assignment — crosswalk assigns study_geoid and ctrl_geoid to survey HHs (point-in-polygon) and allocates PUMS weights to target zones. Zone groups (if configured) are applied inside the crosswalk so ctrl_geoid is ready for balancing.
  6. 1-D merges — global merges collapse incidence columns symmetrically on both tables (originals dropped). Zone-specific merges add merged columns (originals kept) and modify control totals for the specified zones after aggregation.
  7. Control totals — aggregate PUMS incidence into target totals per zone.
  8. Balancer — base weights → max-entropy balancing → weight propagation.

Design decisions

  • Balancer — PopulationSim's numba balancer (np_balancer_numba), vendored in balancing/_np_balancer.py: a pure @njit function taking numpy arrays, so PopulationSim itself is not a dependency.
  • Geography columns — three distinct levels: PUMA (raw Census PUMA), study_geoid (crosswalk target zones from the user's polygon file), and ctrl_geoid (balancing geography, equal to study_geoid unless zone groups are configured). Downstream balancing always uses ctrl_geoid; diagnostics/maps use study_geoid for spatial detail.
  • Symmetric incidence — both the survey sample and the PUMS universe are first recoded and pivoted into incidence tables with identical column layouts. Geography, merges, and crosstabs are applied after incidence construction, keeping the recode/pivot logic independent of geography.

Algorithm

Find weight vector w closest to seed weights w₀ (KL-divergence) subject to marginal constraints:

min Σᵢ wᵢ ln(wᵢ / w₀ᵢ) s.t. A w = t, wᵢ ≥ 0

where A is the incidence matrix and t is the target totals vector. Runs independently per control geography zone (zones are parallelisable).

WeightingPipeline

Stateful weighting pipeline.

Separates configuration (frozen at __init__) from intermediate state (built up phase-by-phase). Phase methods are designed to be called in sequence from the @step() compute_weights entry-point; each stores results as instance attributes.

Usage

pipeline = WeightingPipeline(
    controls=..., config=..., data=..., balancing=..., importance=...
)

pipeline.setup()
pipeline.fetch_pums()

pipeline.recode_and_pivot()
pipeline.assign_zones()
pipeline.apply_merges()

pipeline.aggregate_totals()
pipeline.resolve_importance()

pipeline.fit_all(diagnostics=...)  # one fit per configured profile

One fit per profile

Everything above fit_all describes the survey and the controls, so it is shared. Everything from the seed onward belongs to a profile: which households are admitted, what they were balanced to, and the weight columns written. Each fit's products live on its own ProfileFit, and its columns carry the profile name as a suffix.

A profile is fitted, never derived from another profile's fit. Reusing one profile's household weights for a different universe produces numbers indistinguishable from balanced ones that match no control total; the honest way to impose weights from outside is add_existing_weights, which says so in its name.

crosswalk instance-attribute

crosswalk: PumaCrosswalk

pums_hh instance-attribute

pums_hh: pl.DataFrame

pums_per instance-attribute

pums_per: pl.DataFrame

survey_bundle instance-attribute

survey_bundle: IncidenceBundle

pums_bundle instance-attribute

pums_bundle: IncidenceBundle

pums_incidence instance-attribute

pums_incidence: pl.DataFrame

control_totals instance-attribute

control_totals: ControlTotals

resolved_importance instance-attribute

resolved_importance: dict[str, float]

controls instance-attribute

controls = controls

config instance-attribute

config = config

data instance-attribute

data = data

balancing instance-attribute

balancing = balancing

importance_cfg instance-attribute

importance_cfg = importance

control_moe instance-attribute

control_moe: pl.DataFrame | None = None

fits instance-attribute

fits: dict[str | None, ProfileFit] = {}

cache_dir property

cache_dir: Path | None

Weighting sub-directory of the pipeline cache (computed from config).

setup

setup() -> None

Register crosstabs, resolve control instances, build crosswalk.

fetch_pums

fetch_pums() -> None

Load PUMS microdata from local files or the Census API.

recode_and_pivot

recode_and_pivot() -> None

Recode both datasets and build incidence tables.

Two parallel streams — PUMS (complete Census microdata) and survey (travel-diary seed) — are recoded through identical control expressions, then pivoted into incidence tables with the same {ctrl}__{member} column layout. The PUMS bundle is completed first because it serves as the training set for RF-based fractional imputation of null survey values.

Stream diagram::

PUMS HH/PER ─► recode ─► pivot ─► IncidenceBundle (pums)
                                           │
                                           ▼  (training data)
Survey HH/PER ─► recode ─► pivot ─► IncidenceBundle (survey)
                                           │
                                           ▼  (per profile)
                                      `build_seed`

The survey bundle is left unfiltered and un-imputed: both depend on which profile's universe is being fitted, and one run may fit several.

assign_zones

assign_zones() -> None

Point-in-polygon zone assignment + optional block-group assignment.

Mutates self.data.households in place (adds geo columns) and allocates self.pums_incidence across zones. Each seed takes its geo columns off self.data.households in build_seed, since a seed only exists once a profile has been named.

apply_merges

apply_merges() -> None

Apply 1-D merges to the PUMS incidence table.

Cross-tab merges are applied at registration time (pre-merge into the enum), so only 1-D merges need post-pivot application here. The seed side is merged in build_seed, symmetrically and once per profile.

aggregate_totals

aggregate_totals() -> None

Aggregate PUMS incidence into per-zone control totals.

resolve_importance

resolve_importance() -> None

Build the final importance dict (MOE-based, explicit overrides, or default).

build_seed

build_seed(profile: str | None) -> ProfileFit

Gate, impute, place and merge one profile's seed.

The profile's own flag decides who enters the seed, so the balancer spreads each zone's population over exactly the households that will keep a weight. Gating afterwards instead deletes fitted mass that nothing re-spreads, households being the hierarchy anchor, and the survey then expands to less than the population it was fitted to.

Parameters:

Name Type Description Default
profile str | None

Usability profile to fit.

required

Returns:

Type Description
ProfileFit

The seed and its metadata, ready to balance.

Raises:

Type Description
ValueError

If the gating column is absent from households.

balance_fit

balance_fit(fit: ProfileFit) -> None

Compute base weights and balance one seed, in place on fit.

A non-converged zone is recorded rather than raised on, so every profile still produces a diagnostics report to debug from. fit_all raises once at the end, naming each profile that failed.

propagate_fit

propagate_fit(fit: ProfileFit) -> None

Attach one fit's weights to households and propagate them down.

The only place a profile suffix is applied to canonical data: the balancer emits the base column name, knowing nothing about profiles, and the rename happens here as it is joined on.

fit

fit(profile: str | None) -> ProfileFit

Build, balance and propagate one profile's weights.

Reporting is deliberately not done here: the report describes the run and needs every fit, so fit_all writes it once they are all in.

fit_all

fit_all(
    *, diagnostics: DiagnosticsConfig
) -> dict[str | None, ProfileFit]

Run one fit per configured profile, then raise if any failed to converge.

Every profile is attempted and the report written before raising, so a convergence failure in one costs a diagnosis of the others rather than hiding it -- including the failure itself, which the report names.

Raises:

Type Description
RuntimeError

If any profile left zones unconverged.

processing.weighting.core.propagation

Walk the weight hierarchy, moving weights between canonical tables.

Used by both the compute_weights (computed) and add_existing_weights (supplied) steps. The shape being walked is declared in processing.weighting.core.hierarchy; this module only moves weights along it.

The distribution rule

A record is admitted by a usability flag -- one of the usability profiles stamped by the cascade_completeness step (see processing.completeness), so the weighted sample matches the records that profile's consumer actually keeps. Pass survey_complete instead to weight the whole valid survey, including partial and overnight tours.

Unusable records carry no weight, but their population does not vanish: it is carried by the usable records in the same scope. Give each record a claim equal to its parent's weight -- what it would carry if usable -- and the usable claims share the scope's whole claim:

w = claim x sum(claims in scope) / sum(claims of usable in scope)

Because every record in a scope is scaled by the same factor, the relative weights inside a household are untouched, and the conservation identity holds by construction:

sum(weight in scope) == sum(claim in scope)

Days: the average-day split

Days use a different DOWN rule (split=True): a person's usable days jointly represent that person once, not once each --

day_weight = person_weight / n_usable_days        (usable days; 0 otherwise)
sum(day_weight over a person's days) == person_weight

the vendor's own convention (their weighted days are exactly person_weight / n), with our usability flag deciding which days count. Summing day weights therefore yields persons on an average day, not person-days.

Conservation is strictly within the person -- never pooled across a household: pooling would siphon day-weight from members with fewer usable days toward members with more, silently distorting person-day totals while still passing any household-level checksum. A person with no usable day keeps their person weight (person weights stay calibrated to the person controls) but contributes nothing at the day level; that shortfall is reported by the checksum, never rescaled away. Unsurveyable persons (e.g. unrelated household members) carry no day rows at all, matching the vendor's day table.

Where a whole scope is unusable there is no denominator anywhere and the weight is genuinely stranded. That is reported as a shortfall, never silently rescaled.

drop_unsuffixed_weights

drop_unsuffixed_weights(
    tables: dict[str, pl.DataFrame | None],
) -> None

Remove each level's bare weight column, in place.

A table must not offer both hh_weight and hh_weight_<profile>: a reader has no way to tell which is authoritative, and the bare one is stale by construction -- either a placeholder from the loader or a previous run's output that this run did not rewrite.

collect_tables

collect_tables(
    **tables: pl.DataFrame | None,
) -> dict[str, pl.DataFrame | None]

Bundle canonical tables into a dict keyed by table name.

Every table in the hierarchy is present, defaulting to None, so callers can pass whatever subset they hold.

Parameters:

Name Type Description Default
**tables pl.DataFrame | None

Canonical tables by name.

{}

Returns:

Type Description
dict[str, pl.DataFrame | None]

One entry per hierarchy table, None where not supplied.

Raises:

Type Description
TypeError

If a name is not part of the hierarchy.

safe_join_weight

safe_join_weight(
    df: pl.DataFrame, weight_df: pl.DataFrame, on: str
) -> pl.DataFrame

Left-join weight_df onto df, dropping any pre-existing weight columns.

is_usable

is_usable(usability_flag_col: str) -> pl.Expr

True when a record may carry weight; a null flag counts as unusable.

seed_admits

seed_admits(usability_flag_col: str) -> pl.Expr

True when a household belongs in the balancer's seed for this profile.

The profile's own flag, floored by survey_complete. Every profile is a subset of survey_complete by construction, so the floor changes no number -- it states the invariant where the seed is chosen, and covers a hand-written flag that is not.

The same expression decides the seed and the post-balancing zeroing. Were the two to differ, a household the seed rejected could keep a null where the output promises a zero.

distribute_within_scope

distribute_within_scope(
    df: pl.DataFrame,
    *,
    weight_col: str,
    scope: str,
    usability_flag_col: str,
    table: str
) -> pl.DataFrame

Share each scope's claim among its usable records, returning a new frame.

Each record arrives holding its claim -- the weight it would carry if it were usable. Usable records then split the scope's whole claim in proportion to their own, so the scope's total is conserved and every record in it is scaled by the same factor::

w = claim x sum(claims in scope) / sum(claims of usable in scope)

A scope with no usable record has no denominator; its claim is stranded and logged, for the checksum to report as a shortfall.

Parameters:

Name Type Description Default
df pl.DataFrame

Table holding its claim in weight_col.

required
weight_col str

Weight column, replaced in place by the distributed weight.

required
scope str

Column naming the group within which the claim is conserved.

required
usability_flag_col str

Boolean column deciding which records carry weight.

required
table str

Table name, for logging.

required

Returns:

Type Description
pl.DataFrame

df with weight_col distributed across the usable records.

split_among_usable

split_among_usable(
    df: pl.DataFrame,
    *,
    weight_col: str,
    key: str,
    usability_flag_col: str | None,
    table: str
) -> pl.DataFrame

Divide each parent's weight equally among its usable children, in place.

The average-day rule: a parent's children jointly represent the parent once, not once each ::

w = parent_weight / n_usable_children    (usable children; 0 otherwise)

so the usable children of every parent sum exactly to the parent's weight. This is the vendor's own day-weight convention (day_weight = person_weight / n_weighted_days), with our usability flag deciding which children count. There is no cross-parent pooling: a parent whose children are all unusable is a reported shortfall, never re-homed onto other parents' children.

Parameters:

Name Type Description Default
df pl.DataFrame

Child table holding each row's parent weight in weight_col.

required
weight_col str

Weight column, replaced in place by the split weight.

required
key str

Column identifying the parent record.

required
usability_flag_col str | None

Boolean column deciding which children carry weight, or None to split among all children.

required
table str

Table name, for logging.

required

Returns:

Type Description
pl.DataFrame

df with weight_col split among each parent's usable children.

propagate_weights

propagate_weights(
    tables: dict[str, pl.DataFrame | None],
    has_weight: dict[str, str],
    *,
    skip: set[str] | None = None,
    usability_flag_col: str | None,
    profile: str | None = None
) -> None

Walk the hierarchy, deriving every weight that is not already supplied.

Modifies tables and has_weight in place. Levels are visited in declaration order, so each parent is weighted before its children.

Parameters:

Name Type Description Default
tables dict[str, pl.DataFrame | None]

Mutable dict of table_name → DataFrame (or None).

required
has_weight dict[str, str]

Mutable dict tracking which tables already have a weight column and its name. E.g. {"households": "hh_weight"}.

required
skip set[str] | None

Table names to leave alone (e.g. externally supplied weights).

None
usability_flag_col str | None

Boolean column deciding which records may carry weight -- a usability profile stamped upstream. Pass survey_complete to weight the whole valid survey including partial/overnight tours, or None to give every record its claim regardless of usability (split levels still divide the parent weight equally, over all children).

required
profile str | None

Usability profile these weights belong to, which suffixes every column written. None writes the un-suffixed set. A parent's column is read from has_weight, never re-derived, so two profiles may be propagated over the same tables without reading each other's numbers.

None

non_null_tables

non_null_tables(
    tables: dict[str, pl.DataFrame | None],
) -> dict[str, pl.DataFrame]

Return only the non-None entries from a tables dict.

processing.weighting.core.run_report

The weighting report for one run: where it goes, what it compares, what it says.

One self-contained HTML document per run, written after every profile is fitted. See generate_diagnostics for its sections.

generate_diagnostics

generate_diagnostics(
    pipeline: WeightingPipeline, diagnostics: DiagnosticsConfig
) -> None

Write one pipeline-contained interactive HTML report for the whole run.

Covers the weighting from geographic crosswalk through balancer convergence to what carries weight at every level below the household. Opens in any browser with no external dependencies beyond Plotly, loaded from CDN.

One document, not one per profile. The crosswalk geometry, control totals and PUMS incidence are built once in setup() however many profiles are fitted -- and on a regional run that geometry is the great majority of the file -- so writing a report per fit re-embedded the same polygons once per profile. Only the balancer's output actually varies, and it sits behind a profile toggle.

Report sections
  1. Profile Comparison — one row per fit: seed size, weight sums by level, ESS%, CV, MAPE and convergence. The only view that needs every fit at once, and so the only one no per-fit report could produce.
  2. Crosswalk Map — choropleth showing how PUMAs overlap the target zones, with allocation weights per PUMA-zone pair and each profile's seed count in the zone tooltip.
  3. Weight Cascade — per level, what the profile admitted and what carries weight, every profile side by side; the redistribution each survivor absorbs, how each weight set's lower levels descend from the level above, the day-split identity, and control-geography coverage.
  4. Weight Set Comparer (optional) — any two weight sets compared pairwise at any level, this run's fits against each other or against sets named in diagnostics.compare_weights. Omitted when there is only one set to show.
  5. Fractional Seed Imputation (per profile) — null rate per control in the seed incidence, and the quality of the PUMS-trained model that filled it.
  6. Balancer Performance (per profile) — per-zone convergence, target fit (MAPE, P90, Max), CV and ESS%.
  7. Weight Quality (per profile) — per-zone weight and expansion factor statistics, with violin plots.
  8. Expansion Factor Calibration (per profile, optional) — MAPE vs CV across the expansion_factor_grid values. Only included when the grid was set in the weighting config.
  9. Target Fit (% Error) (per profile) — diverging bars per control category per zone, with PUMS replicate-weight whiskers.
  10. Unweighted Cell Counts (per profile) — seed counts per control category per zone, flagging sparse cells.

Numbers are assigned to the sections a run actually renders, so an optional section that is absent takes its number with it rather than leaving a gap.

Parameters

diagnostics: The diagnostics block of the weighting config.

``output_path`` is the destination for the HTML file, a string
(including template paths from the YAML). When null the file is
written to ``<cache_dir>/diagnostics.html`` (or
``./weighting/diagnostics.html`` if no cache directory is
configured).  No profile suffix: a run writes one file however many
profiles it fits.

``compare_weights`` names weight sets that arrived with the survey,
to compare this run's fits against.  See
[`reference_weight_sets`][processing.weighting.diagnostics.comparison
.reference_weight_sets] for why they are named rather than found.

processing.weighting.core.specs

Data structures for the weighting pipeline.

Single source of truth for all dataclasses and named tuples used across data preparation, balancing, validation, and diagnostics.

Kept as a leaf module with minimal imports to avoid circular dependencies.

ControlSpec dataclass

Specification for a single weighting control.

Attributes:

Name Type Description
name str

Registry name (must exist in CONTROLS).

importance float | None

Explicit importance weight for the balancer. None means use the default (100 for normal controls, 1000 for structural) or the MOE-derived value when moe_based_importance is enabled.

dimensions list[str] | None

For cross-tab controls only: list of dimension control names (e.g. ["h_size", "h_income"]). None for standard controls.

merges dict | None

For cross-tab controls only: per-dimension merge specs applied at registration time. None for standard controls.

name instance-attribute

name: str

importance class-attribute instance-attribute

importance: float | None = None

dimensions class-attribute instance-attribute

dimensions: list[str] | None = None

merges class-attribute instance-attribute

merges: dict | None = None

ControlTotals dataclass

Result of PUMS control-total aggregation.

Attributes:

Name Type Description
totals pl.DataFrame

Tidy frame with columns: [geo_id, control_name, category, target_total]

pums_hh_count int

Total PUMS housing unit records (before weighting).

pums_person_count int

Total PUMS person records.

geo_ids list[str]

Unique geography IDs in the totals.

totals instance-attribute

totals: pl.DataFrame

pums_hh_count instance-attribute

pums_hh_count: int

pums_person_count instance-attribute

pums_person_count: int

geo_ids instance-attribute

geo_ids: list[str]

IncidenceBundle dataclass

All intermediate and final products of incidence-table construction.

Built by build_incidence_table and available for downstream consumers (balancer, fractional imputation, diagnostics).

Attributes:

Name Type Description
incidence pl.DataFrame

Combined household-level incidence table — one row per HH with {ctrl}__{member} columns for all controls (HH + person aggregated). This is what the balancer operates on.

person_pivot pl.DataFrame

Per-person 0/1 indicator table — one row per person with {p_ctrl}__{member} columns. Useful for person-level classification (fractional imputation of null persons).

household_pivot pl.DataFrame

HH-only 0/1 indicator table — one row per HH with {h_ctrl}__{member} columns. Useful for HH-level classification without person-count noise.

incidence instance-attribute

incidence: pl.DataFrame

person_pivot instance-attribute

person_pivot: pl.DataFrame

household_pivot instance-attribute

household_pivot: pl.DataFrame

filter_households

filter_households(
    hh_ids: list | pl.Series, hh_id_col: str = "hh_id"
) -> IncidenceBundle

Return a new bundle keeping only the given household IDs.

ImputationSummary dataclass

Per-control fractional seed imputation metadata for diagnostics.

Describes what fill_null_incidence filled in the seed incidence matrix -- never the survey imputation step, which writes discrete values to canonical data and is reported separately.

One entry per control target, regardless of whether any filling was needed. Controls with no nulls have n_null == 0 and None for the RF metrics.

control instance-attribute

control: str

level instance-attribute

level: str

n_total instance-attribute

n_total: int

n_null instance-attribute

n_null: int

log_loss class-attribute instance-attribute

log_loss: float | None = None

f1_macro class-attribute instance-attribute

f1_macro: float | None = None

PUMSSource dataclass

Configuration for PUMS data source.

Attributes:

Name Type Description
state_fips str

Two-digit FIPS code for the state (e.g. "06" for California).

pums_year int

ACS 1-year PUMS vintage (e.g. 2022).

puma_ids list[str] | None

Optional list of PUMA codes to fetch. If None, fetches all PUMAs in the state (can be large).

state_fips instance-attribute

state_fips: str

pums_year instance-attribute

pums_year: int

puma_ids class-attribute instance-attribute

puma_ids: list[str] | None = None

MergeSpec dataclass

Category-merge specification for one control.

Supports both 1D (standard) and N-D (cross-tab) merges.

Attributes:

Name Type Description
control str

Registry name (e.g. "p_employment" or "h_income_x_size").

groups dict[str, list[str] | dict[str, list[str]]]

Merge specifications. Each key is a merged label, each value is either:

  • 1D merge (list): Base member names to combine. E.g. {"employed": ["employed_full", "employed_part"]}

  • N-D merge (dict): Dimension name -> member names for cross-tabs. E.g. {"low_income_large": {"h_income": ["inc_under_25k", "inc_25k_to_50k"], "h_size": ["size_4", "size_5"]}}

The N-D merge creates a single merged cell from the cartesian product of all specified dimension members.

zones list[str] | None

If set, apply this merge only to these geo IDs. None means apply globally (all zones).

Example

1D merge:

MergeSpec( ... control="p_employment", ... groups={"employed": ["employed_full", "employed_part"]}, ... )

N-D merge for cross-tab:

MergeSpec( ... control="h_income_x_size", ... groups={ ... "low_income_large": { ... "h_income": ["inc_under_25k", "inc_25k_to_50k"], ... "h_size": ["size_4", "size_5", "size_6"], ... } ... }, ... )

control instance-attribute

control: str

groups instance-attribute

groups: dict[str, list[str] | dict[str, list[str]]]

zones class-attribute instance-attribute

zones: list[str] | None = None

CensusConfig pydantic-model

Where the Census controls come from: the state, the PUMS year, and optional local files.

Config:

  • extra: forbid

Fields:

model_config class-attribute instance-attribute

model_config = ConfigDict(extra='forbid')

state_fips pydantic-field

state_fips: str

Two-digit state FIPS code, e.g. '06'.

pums_year pydantic-field

pums_year: int

ACS PUMS 1-year vintage the controls are built from.

pums_households pydantic-field

pums_households: str | None = None

Local PUMS household file; fetched from the Census API when not given.

pums_persons pydantic-field

pums_persons: str | None = None

Local PUMS person file; fetched from the Census API when not given.

ReferenceWeightSetConfig pydantic-model

One weight set that arrived with the survey, named for the report's comparer.

Config:

  • extra: forbid

Fields:

model_config class-attribute instance-attribute

model_config = ConfigDict(extra='forbid')

label pydantic-field

label: str

How the report names the set.

columns pydantic-field

columns: dict[str, str]

Table -> the column holding this set's weight on it.

DiagnosticsConfig pydantic-model

The weighting report: where it goes, what it compares, and the grid search.

Config:

  • extra: forbid

Fields:

model_config class-attribute instance-attribute

model_config = ConfigDict(extra='forbid')

output_path pydantic-field

output_path: str | None

Where the HTML report is written; null for the cache directory.

expansion_factor_grid pydantic-field

expansion_factor_grid: list[float] | None

max_expansion_factor values to re-run balancing at, charting fit against weight quality; null to skip the grid search.

compare_weights pydantic-field

compare_weights: dict[str, ReferenceWeightSetConfig] | None = None

Weight sets supplied with the survey, compared against this run's fits.

BalancingConfig pydantic-model

Maximum-entropy balancer configuration, and how hard each control pulls.

Controls Newton-Raphson iteration bounds, weight expansion limits, parallel execution and the importance given to controls without their own. Consumed by balance_weights, grid_search_expansion_factor, and the pipeline orchestrator.

Config:

  • extra: forbid

Fields:

model_config class-attribute instance-attribute

model_config = ConfigDict(extra='forbid')

max_expansion_factor pydantic-field

max_expansion_factor: float

Upper bound on a weight, as a multiple of its seed weight.

min_expansion_factor pydantic-field

min_expansion_factor: float

Lower bound on a weight, as a multiple of its seed weight.

min_weight pydantic-field

min_weight: float | None

Absolute weight floor; null for none.

max_weight pydantic-field

max_weight: float | None = None

Absolute weight ceiling; none unless given.

max_iterations pydantic-field

max_iterations: int

Newton-Raphson iteration limit per zone.

n_workers pydantic-field

n_workers: int

Zones (or grid points) balanced in parallel.

moe_based_importance pydantic-field

moe_based_importance: bool

Derive each control's importance from its PUMS replicate-weight MOE.

default_importance pydantic-field

default_importance: float

Importance for a control without its own, when not MOE-derived.

ImportanceConfig dataclass

Importance weighting configuration.

Determines how per-control importance weights are computed:

  • explicit — YAML-declared overrides (always highest precedence).
  • moe_based — derive importance from PUMS replicate-weight CVs.
  • default — fallback value when neither of the above applies.

explicit class-attribute instance-attribute

explicit: dict[str, float] = field(default_factory=dict)

moe_based class-attribute instance-attribute

moe_based: bool = False

default class-attribute instance-attribute

default: float = 100.0

WeightingConfig dataclass

Top-level configuration for the weighting pipeline.

Groups the parameters that are not already covered by ControlRegistryConfig, BalancingConfig, or ImportanceConfig — primarily Census/geography settings and pipeline plumbing.

geography instance-attribute

geography: GeographyConfig

state_fips instance-attribute

state_fips: str

pums_year instance-attribute

pums_year: int

weight_profiles class-attribute instance-attribute

weight_profiles: tuple[str, ...] = ()

max_unplaceable_share class-attribute instance-attribute

max_unplaceable_share: float = 0.01

pums_households class-attribute instance-attribute

pums_households: str | None = None

pums_persons class-attribute instance-attribute

pums_persons: str | None = None

sample_plan class-attribute instance-attribute

sample_plan: str | None = None

cache_dir class-attribute instance-attribute

cache_dir: Path | None = None

expansion_factor_grid class-attribute instance-attribute

expansion_factor_grid: list[float] | None = None

strict_survey_nulls class-attribute instance-attribute

strict_survey_nulls: bool = False

fitted_profiles property

fitted_profiles: tuple[str, ...]

One entry per balancing run.

flag_for

flag_for(profile: str) -> str

The verdict column the fit for profile is gated on.

A profile names both its gate and its weights -- usable_<profile> and hh_weight_<profile> -- so there is no second setting to disagree with the first.

ControlRegistryConfig dataclass

Parsed control definitions, merge specs, and derived target names.

Built via from_yaml from the YAML controls block.

specs instance-attribute

specs: list[ControlSpec]

target_names instance-attribute

target_names: list[str]

crosstab_merges instance-attribute

crosstab_merges: list[MergeSpec]

merges_1d instance-attribute

merges_1d: list[MergeSpec]

importance_overrides property

importance_overrides: dict[str, float]

Explicit importance overrides from YAML.

all_merges property

all_merges: list[MergeSpec]

All merge specs (crosstab + 1-D) for diagnostics.

from_yaml classmethod

from_yaml(controls: list[dict]) -> ControlRegistryConfig

Parse the YAML controls block into a config object.

GridPoint dataclass

Aggregate metrics for one expansion-factor grid point.

max_expansion_factor instance-attribute

max_expansion_factor: float

converged_zones instance-attribute

converged_zones: int

total_zones instance-attribute

total_zones: int

mape instance-attribute

mape: float

p90 instance-attribute

p90: float

max_error instance-attribute

max_error: float

cv instance-attribute

cv: float

ess_pct instance-attribute

ess_pct: float

SamplePlan dataclass

Stratified sampling plan mapping Census block groups to segments.

Each row represents a Census block group. Block groups that share the same sample_segment are treated as a single stratum for initial-weight computation: base_weight = segment_bg_pop / segment_n_responses.

Block-group population totals are sourced from the crosswalk (PumaCrosswalk.block_group_populations), not from this table.

Attributes:

Name Type Description
strata pl.DataFrame

Required columns:

  • bg_geo_id (str) — 12-character Census block-group FIPS code.
  • sample_segment (str) — sampling-stratum label. All block groups sharing a segment get the same base weight.

strata instance-attribute

strata: pl.DataFrame

ZoneStatus

Per-zone convergence diagnostics.

geo_id instance-attribute

geo_id: str

converged instance-attribute

converged: bool

iterations instance-attribute

iterations: int

delta instance-attribute

delta: float

max_gamma_diff instance-attribute

max_gamma_diff: float

ZoneInput

Pre-built numpy arrays for a single geography zone.

hh_ids instance-attribute

hh_ids: pl.Series

incidence instance-attribute

incidence: np.ndarray

initial instance-attribute

initial: np.ndarray

lb instance-attribute

lb: np.ndarray

ub instance-attribute

ub: np.ndarray

targets instance-attribute

targets: np.ndarray

importance instance-attribute

importance: np.ndarray

master_idx instance-attribute

master_idx: int

max_iterations instance-attribute

max_iterations: int

geo_id instance-attribute

geo_id: str

verbose class-attribute instance-attribute

verbose: bool = True

ZoneResult

Balancer output for a single geography zone.

weights instance-attribute

weights: pl.DataFrame

status instance-attribute

status: ZoneStatus

GeographyCoverage

How much of one profile's universe the control geography can place.

A household the control polygons do not contain belongs to no balancing zone, so no fit can give it a weight. That is a bound on what the weighting can answer, not a defect in the household, and it is counted so a reader sees the bound rather than inferring it from nulls.

profile instance-attribute

profile: str | None

n_universe instance-attribute

n_universe: int

n_placed instance-attribute

n_placed: int

n_unplaceable property

n_unplaceable: int

Households in the profile that the control geography cannot place.

unplaceable_share property

unplaceable_share: float

Unplaceable households as a share of the profile's universe.

ProfileFit dataclass

Everything one balancing run produced.

A run may fit several profiles over the same survey. Holding each fit's products here rather than on the pipeline keeps them from overwriting one another -- otherwise the last profile's seed silently becomes the one the diagnostics describe and the checks read.

Attributes:

Name Type Description
profile str | None

Usability profile fitted, or None for a single un-suffixed set. Suffixes every weight column this fit writes.

usability_flag_col str

The column that gated the seed and will gate the propagation. Equal to profile whenever a profile was named.

seed_incidence pl.DataFrame

The seed this fit was solved from, carrying its own base_weight once balancing has run.

pre_imputation_incidence pl.DataFrame

The same seed before null imputation, for the diagnostics report.

imputation_summary list[ImputationSummary]

Per-control imputation metadata for this seed.

coverage GeographyCoverage

How much of the profile's universe had a control geography.

weights pl.DataFrame | None

Balancer output, keyed by hh_id, carrying the base weight column name -- the suffix is applied where it is joined on.

statuses list[ZoneStatus]

Per-zone convergence diagnostics.

grid_results list[GridPoint] | None

Expansion-factor grid points, when a grid was configured.

profile instance-attribute

profile: str | None

usability_flag_col instance-attribute

usability_flag_col: str

seed_incidence instance-attribute

seed_incidence: pl.DataFrame

pre_imputation_incidence instance-attribute

pre_imputation_incidence: pl.DataFrame

imputation_summary instance-attribute

imputation_summary: list[ImputationSummary]

coverage instance-attribute

weights class-attribute instance-attribute

weights: pl.DataFrame | None = None

statuses class-attribute instance-attribute

statuses: list[ZoneStatus] = field(default_factory=list)

grid_results class-attribute instance-attribute

grid_results: list[GridPoint] | None = None

unconverged_zones property

unconverged_zones: int

Zones this fit failed to converge.

resolve_fitted_profiles

resolve_fitted_profiles(
    weight_profiles: tuple[str, ...] | list[str] | None,
) -> tuple[str, ...]

The profiles to weight: one entry per weight column set to be written.

Shared by both weighting entry points so they cannot disagree about what a profile list is supposed to mean.

Parameters:

Name Type Description Default
weight_profiles tuple[str, ...] | list[str] | None

Profiles to weight, each writing its own suffixed column set. A single profile is a list of one.

required

Returns:

Type Description
tuple[str, ...]

One entry per profile, in the order given.

Raises:

Type Description
ValueError

If no profile is given, or a profile is named twice.