processing.weighting.core
Shared machinery behind the two weighting entry points.
Neither module here is a pipeline step. The steps live one level up --
compute_weights (balance from
controls) and add_existing_weights
(attach supplied weights) -- and both are built from:
hierarchy-- the weight hierarchy, declared once.propagation-- the walk that moves weights along it.specs-- configuration and result types.pipeline-- orchestration for the computed path.
Kept import-free so it cannot close a cycle with the sub-packages that read it.
processing.weighting.core.hierarchy
The canonical weight hierarchy, declared once.
Every table that carries a weight is declared here exactly once -- its weight
column, where the weight comes from, and the group within which that weight is
conserved. Everything else is a projection of that list, so the shape of the
hierarchy cannot drift between the propagation (processing.weighting.core.propagation), the supplied-weight path and the checksum.
Weights disaggregate down from households and aggregate back up into groupings; both directions are edges of one tree.
| Table | Weight | Derivation |
|---|---|---|
| households | hh_weight |
Anchor -- from the balancer, or supplied |
| persons | person_weight |
Down from hh_weight via hh_id |
| days | day_weight |
Split: person_weight / n_usable_days |
| unlinked | unlinked_trip_weight |
Down from day_weight via day_id |
| linked | linked_trip_weight |
Up: mean of member unlinked_trip_weight |
| tours | tour_weight |
Up: mean of member linked_trip_weight |
| joint trips | joint_trip_weight |
Up: sum of member linked_trip_weight |
| joint tours | joint_tour_weight |
Up: sum of member tour_weight |
Two units
Most weights count one row of their own table. The joint levels instead sum
their members, so they count person-trips -- the travel of the whole party,
not the joint occasion -- and are therefore an overlay on the member table,
never a partition of it: sum(linked) + sum(joint) double counts.
To count occasions instead, divide by the number of members that carried weight -- counted from the member table, not read off the grouping::
events = joint_trip_weight / n_member_trips_with_weight
A party-size column is not that divisor: it counts travellers, including any the weighting excluded.
One column set per profile
A run may weight several usability profiles, each fitted to its own universe. The
columns above are then suffixed with the profile's name, verbatim and unabridged
(hh_weight_ctramp), so a column names the universe it expands and no
rule is needed to read one back. Only [weight_col_for][processing.weighting.core
.hierarchy.weight_col_for] and its siblings spell that suffix; Level.weight_col
stays the base name, which is what keeps id_col derivable.
HIERARCHY
module-attribute
HIERARCHY: tuple[Level, ...] = (
Level(
table="households",
weight_col="hh_weight",
means="Households represented by this record.",
),
Level(
table="persons",
weight_col="person_weight",
means="Persons represented by this record.",
parent="households",
flow=Flow.DOWN,
scope="hh_id",
),
Level(
table="days",
weight_col="day_weight",
means="Person-days represented by this record. A person's usable days sum to their person weight, the average-day convention, so a day is a share of that person rather than a count of its own.",
parent="persons",
flow=Flow.DOWN,
scope="person_id",
split=True,
),
Level(
table="unlinked_trips",
weight_col="unlinked_trip_weight",
means="Unlinked trips (single vehicle or mode segments) represented.",
parent="days",
flow=Flow.DOWN,
scope="day_id",
),
Level(
table="linked_trips",
weight_col="linked_trip_weight",
means="Linked trips (whole origin-to-destination journeys) represented.",
parent="unlinked_trips",
flow=Flow.UP,
scope="day_id",
),
Level(
table="joint_trips",
weight_col="joint_trip_weight",
means="Person-trips represented: the SUM of the member linked trip weights, so the record stands for its whole party. Same unit as linked_trips, which this table OVERLAYS rather than partitions, so summing both double counts.",
parent="linked_trips",
flow=Flow.UP,
scope="hh_id",
agg=Agg.SUM,
),
Level(
table="tours",
weight_col="tour_weight",
means="Tours represented by this record.",
parent="linked_trips",
flow=Flow.UP,
scope="day_id",
),
Level(
table="joint_tours",
weight_col="joint_tour_weight",
means="Person-tours represented: the SUM of the member tour weights. Same unit as tours, which this table OVERLAYS rather than partitions, so summing both double counts.",
parent="tours",
flow=Flow.UP,
scope="hh_id",
agg=Agg.SUM,
),
)
WEIGHT_CONFIG_MAPPING
module-attribute
WEIGHT_CONFIG_MAPPING: dict[str, tuple[str, str, str]] = {
(level.weight_col): (
level.table,
level.id_col,
level.weight_col,
)
for level in HIERARCHY
}
Supplied-weight config key -> (table, id column, weight column).
The config key is the weight column: one name for one thing, so a config cannot name a level the hierarchy does not have.
SEED_TABLE
module-attribute
SEED_TABLE = 'household_weights'
Table the weighting's working columns move to before the data is handed back.
Flow
Direction a weight moves along a hierarchy edge.
DOWN
class-attribute
instance-attribute
DOWN = auto()
Disaggregate: the parent's weight is shared out among its children.
UP
class-attribute
instance-attribute
UP = auto()
Aggregate: a grouping combines the weights of its usable members.
Agg
How an UP edge combines its members' weights.
MEAN
class-attribute
instance-attribute
MEAN = auto()
The grouping counts once: one linked trip, one tour. Its weight is the mean of its usable members, so the weight stays in the unit of its own row.
SUM
class-attribute
instance-attribute
SUM = auto()
The grouping carries the combined weight of its members -- person-trips, not occasions. This puts it in the member table's unit while overlapping it, so summing both double counts.
Level
dataclass
One table in the weight hierarchy.
Only what cannot be derived is declared: id_col and key follow
mechanically from weight_col and the edge, so they are properties --
there is no way to spell them inconsistently.
Attributes:
| Name | Type | Description |
|---|---|---|
table |
str
|
Canonical table name. |
weight_col |
str
|
Weight column this level owns. Names the level -- |
means |
str
|
What one unit of this weight counts, for the delivered codebook. Carried here rather than on a model field because the column's name comes from config once a run weights profiles separately, and a reader of the joint levels in particular needs telling: they are an overlay on their member table, not a partition of it. |
parent |
str | None
|
Table the weight is derived from; None marks the anchor. |
flow |
Flow | None
|
Which way the weight moves along the edge. |
scope |
str | None
|
Group within which this weight is conserved. |
split |
bool
|
If True, the parent's weight is divided equally among its usable
children ( |
agg |
Agg
|
How an |
table
instance-attribute
table: str
weight_col
instance-attribute
weight_col: str
means
instance-attribute
means: str
parent
class-attribute
instance-attribute
parent: str | None = None
scope
class-attribute
instance-attribute
scope: str | None = None
split
class-attribute
instance-attribute
split: bool = False
id_col
property
id_col: str
Primary key of the table, e.g. tour_weight -> tour_id.
key
property
key: str | None
The id column carried on this side of the edge.
DOWN edges carry the parent's id, identifying the record the weight
comes from; UP edges carry their own, identifying the grouping a
member belongs to.
weight_col_for
weight_col_for(profile: str | None) -> str
This level's weight column under profile.
The suffix is applied here and nowhere else, so weight_col -- and
therefore id_col and key, which are read off it -- keeps its base
name however many profiles a run weights.
member_key_for
member_key_for(
usability_flag_col: str | None, columns: Collection[str]
) -> str
The column on the source table saying which grouping a record is on.
The joint groupings differ by profile: joint_tour_id_ctramp is the
bare joint_tour_id where the tour is on it for ctramp and null where
it is not, so summing by it counts exactly the tours that profile keeps
together. Where the profile's copy is absent -- a single un-suffixed
run, survey_complete, or a level with no per-profile copy -- the
bare key is the answer.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
usability_flag_col
|
str | None
|
The flag the weighting is gated on, from which the profile is read. |
required |
columns
|
Collection[str]
|
Columns present on the source table. |
required |
Returns:
| Type | Description |
|---|---|
str
|
The column to group the source by. Its values are the bare key's. |
weight_col_for
weight_col_for(base: str, profile: str | None) -> str
The hierarchy weight column base as written under profile.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
base
|
str
|
A declared weight column, e.g. |
required |
profile
|
str | None
|
Usability profile the fit was gated on, or None for a single un-suffixed set. |
required |
Raises:
| Type | Description |
|---|---|
ValueError
|
If base is not a weight column the hierarchy declares. Checked so a caller cannot coin a column the hierarchy has no level for -- the suffix rule is only trustworthy if the stem is. |
seed_col_for
seed_col_for(base: str, profile: str | None) -> str
The pre-balancing column base as written under profile.
Separate from weight_col_for
because these columns belong to no level: they describe the seed a fit was
solved from, not a weight the hierarchy propagates.
Raises:
| Type | Description |
|---|---|
ValueError
|
If base is not a declared seed column. |
weight_columns_for
weight_columns_for(profile: str | None) -> dict[str, str]
Map each table to its weight column under profile.
describe_weight_columns
describe_weight_columns(
profiles: tuple[str | None, ...],
) -> dict[str, dict[str, GeneratedColumn]]
Describe and constrain every weight column a set of fits writes.
These columns are not model fields. A run that weights profiles separately names them from config, and even the un-suffixed set is a profile's weights with the label left off -- there is no un-profiled weight. So the promise a field would have carried is made here instead: what the number counts, and that it cannot be negative.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
profiles
|
tuple[str | None, ...]
|
The profiles weighted. |
required |
Returns:
| Type | Description |
|---|---|
dict[str, dict[str, GeneratedColumn]]
|
One entry per canonical table that gained columns. |
processing.weighting.core.pipeline
Top-level weighting pipeline step.
Orchestrates the full weighting pipeline via
WeightingPipeline:
- Setup — parse YAML config → specs, target names, merges, importance. Cross-tab controls are registered with pre-merged dimensions so the enum reflects the effective cell count.
- Data fetching — load PUMS (API or files); receive survey tables.
- Conformance — recode both PUMS and survey through identical control expressions → same control-column schema.
- Incidence pivot — unified pivoter produces identical
{ctrl}__{member}column layout for both datasets. - Zone assignment — crosswalk assigns
study_geoidandctrl_geoidto survey HHs (point-in-polygon) and allocates PUMS weights to target zones. Zone groups (if configured) are applied inside the crosswalk soctrl_geoidis ready for balancing. - 1-D merges — global merges collapse incidence columns symmetrically on both tables (originals dropped). Zone-specific merges add merged columns (originals kept) and modify control totals for the specified zones after aggregation.
- Control totals — aggregate PUMS incidence into target totals per zone.
- Balancer — base weights → max-entropy balancing → weight propagation.
Design decisions
- Balancer — PopulationSim's numba balancer (
np_balancer_numba), vendored inbalancing/_np_balancer.py: a pure@njitfunction taking numpy arrays, so PopulationSim itself is not a dependency. - Geography columns — three distinct levels:
PUMA(raw Census PUMA),study_geoid(crosswalk target zones from the user's polygon file), andctrl_geoid(balancing geography, equal tostudy_geoidunless zone groups are configured). Downstream balancing always usesctrl_geoid; diagnostics/maps usestudy_geoidfor spatial detail. - Symmetric incidence — both the survey sample and the PUMS universe are first recoded and pivoted into incidence tables with identical column layouts. Geography, merges, and crosstabs are applied after incidence construction, keeping the recode/pivot logic independent of geography.
Algorithm
Find weight vector w closest to seed weights w₀ (KL-divergence) subject to marginal constraints:
min Σᵢ wᵢ ln(wᵢ / w₀ᵢ) s.t. A w = t, wᵢ ≥ 0
where A is the incidence matrix and t is the target totals vector. Runs independently per control geography zone (zones are parallelisable).
WeightingPipeline
Stateful weighting pipeline.
Separates configuration (frozen at __init__) from intermediate
state (built up phase-by-phase). Phase methods are designed to be
called in sequence from the
@step() compute_weights
entry-point; each stores results as instance attributes.
Usage
pipeline = WeightingPipeline(
controls=..., config=..., data=..., balancing=..., importance=...
)
pipeline.setup()
pipeline.fetch_pums()
pipeline.recode_and_pivot()
pipeline.assign_zones()
pipeline.apply_merges()
pipeline.aggregate_totals()
pipeline.resolve_importance()
pipeline.fit_all(diagnostics=...) # one fit per configured profile
One fit per profile
Everything above fit_all describes the survey and the controls, so it is
shared. Everything from the seed onward belongs to a profile: which
households are admitted, what they were balanced to, and the weight columns
written. Each fit's products live on its own
ProfileFit, and its columns
carry the profile name as a suffix.
A profile is fitted, never derived from another profile's fit. Reusing one
profile's household weights for a different universe produces numbers
indistinguishable from balanced ones that match no control total; the honest
way to impose weights from outside is
add_existing_weights,
which says so in its name.
pums_hh
instance-attribute
pums_hh: pl.DataFrame
pums_per
instance-attribute
pums_per: pl.DataFrame
pums_incidence
instance-attribute
pums_incidence: pl.DataFrame
resolved_importance
instance-attribute
resolved_importance: dict[str, float]
controls
instance-attribute
controls = controls
config
instance-attribute
config = config
data
instance-attribute
data = data
balancing
instance-attribute
balancing = balancing
importance_cfg
instance-attribute
importance_cfg = importance
control_moe
instance-attribute
control_moe: pl.DataFrame | None = None
cache_dir
property
cache_dir: Path | None
Weighting sub-directory of the pipeline cache (computed from config).
setup
setup() -> None
Register crosstabs, resolve control instances, build crosswalk.
fetch_pums
fetch_pums() -> None
Load PUMS microdata from local files or the Census API.
recode_and_pivot
recode_and_pivot() -> None
Recode both datasets and build incidence tables.
Two parallel streams — PUMS (complete Census microdata) and
survey (travel-diary seed) — are recoded through identical
control expressions, then pivoted into incidence tables with the
same {ctrl}__{member} column layout. The PUMS bundle is
completed first because it serves as the training set for
RF-based fractional imputation of null survey values.
Stream diagram::
PUMS HH/PER ─► recode ─► pivot ─► IncidenceBundle (pums)
│
▼ (training data)
Survey HH/PER ─► recode ─► pivot ─► IncidenceBundle (survey)
│
▼ (per profile)
`build_seed`
The survey bundle is left unfiltered and un-imputed: both depend on which profile's universe is being fitted, and one run may fit several.
assign_zones
assign_zones() -> None
Point-in-polygon zone assignment + optional block-group assignment.
Mutates self.data.households in place (adds geo columns) and
allocates self.pums_incidence across zones. Each seed takes its geo
columns off self.data.households in build_seed, since a seed only
exists once a profile has been named.
apply_merges
apply_merges() -> None
Apply 1-D merges to the PUMS incidence table.
Cross-tab merges are applied at registration time (pre-merge into
the enum), so only 1-D merges need post-pivot application here. The seed
side is merged in build_seed, symmetrically and once per profile.
aggregate_totals
aggregate_totals() -> None
Aggregate PUMS incidence into per-zone control totals.
resolve_importance
resolve_importance() -> None
Build the final importance dict (MOE-based, explicit overrides, or default).
build_seed
build_seed(profile: str | None) -> ProfileFit
Gate, impute, place and merge one profile's seed.
The profile's own flag decides who enters the seed, so the balancer spreads each zone's population over exactly the households that will keep a weight. Gating afterwards instead deletes fitted mass that nothing re-spreads, households being the hierarchy anchor, and the survey then expands to less than the population it was fitted to.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
profile
|
str | None
|
Usability profile to fit. |
required |
Returns:
| Type | Description |
|---|---|
ProfileFit
|
The seed and its metadata, ready to balance. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If the gating column is absent from households. |
balance_fit
balance_fit(fit: ProfileFit) -> None
Compute base weights and balance one seed, in place on fit.
A non-converged zone is recorded rather than raised on, so every profile
still produces a diagnostics report to debug from. fit_all raises once
at the end, naming each profile that failed.
propagate_fit
propagate_fit(fit: ProfileFit) -> None
Attach one fit's weights to households and propagate them down.
The only place a profile suffix is applied to canonical data: the balancer emits the base column name, knowing nothing about profiles, and the rename happens here as it is joined on.
fit
fit(profile: str | None) -> ProfileFit
Build, balance and propagate one profile's weights.
Reporting is deliberately not done here: the report describes the run
and needs every fit, so fit_all writes it once they are all in.
fit_all
fit_all(
*, diagnostics: DiagnosticsConfig
) -> dict[str | None, ProfileFit]
Run one fit per configured profile, then raise if any failed to converge.
Every profile is attempted and the report written before raising, so a convergence failure in one costs a diagnosis of the others rather than hiding it -- including the failure itself, which the report names.
Raises:
| Type | Description |
|---|---|
RuntimeError
|
If any profile left zones unconverged. |
processing.weighting.core.propagation
Walk the weight hierarchy, moving weights between canonical tables.
Used by both the compute_weights (computed) and add_existing_weights (supplied)
steps. The shape being walked is declared in processing.weighting.core.hierarchy; this module only moves weights along it.
The distribution rule
A record is admitted by a usability flag -- one of the usability profiles
stamped by the cascade_completeness step (see processing.completeness), so the weighted sample matches the records that
profile's consumer actually keeps. Pass survey_complete instead to weight the whole
valid survey, including partial and overnight tours.
Unusable records carry no weight, but their population does not vanish: it is carried by the usable records in the same scope. Give each record a claim equal to its parent's weight -- what it would carry if usable -- and the usable claims share the scope's whole claim:
w = claim x sum(claims in scope) / sum(claims of usable in scope)
Because every record in a scope is scaled by the same factor, the relative weights inside a household are untouched, and the conservation identity holds by construction:
sum(weight in scope) == sum(claim in scope)
Days: the average-day split
Days use a different DOWN rule (split=True): a person's usable days jointly
represent that person once, not once each --
day_weight = person_weight / n_usable_days (usable days; 0 otherwise)
sum(day_weight over a person's days) == person_weight
the vendor's own convention (their weighted days are exactly person_weight /
n), with our usability flag deciding which days count. Summing day weights
therefore yields persons on an average day, not person-days.
Conservation is strictly within the person -- never pooled across a household: pooling would siphon day-weight from members with fewer usable days toward members with more, silently distorting person-day totals while still passing any household-level checksum. A person with no usable day keeps their person weight (person weights stay calibrated to the person controls) but contributes nothing at the day level; that shortfall is reported by the checksum, never rescaled away. Unsurveyable persons (e.g. unrelated household members) carry no day rows at all, matching the vendor's day table.
Where a whole scope is unusable there is no denominator anywhere and the weight is genuinely stranded. That is reported as a shortfall, never silently rescaled.
drop_unsuffixed_weights
drop_unsuffixed_weights(
tables: dict[str, pl.DataFrame | None],
) -> None
Remove each level's bare weight column, in place.
A table must not offer both hh_weight and hh_weight_<profile>: a
reader has no way to tell which is authoritative, and the bare one is stale
by construction -- either a placeholder from the loader or a previous run's
output that this run did not rewrite.
collect_tables
collect_tables(
**tables: pl.DataFrame | None,
) -> dict[str, pl.DataFrame | None]
Bundle canonical tables into a dict keyed by table name.
Every table in the hierarchy is present, defaulting to None, so callers can pass whatever subset they hold.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
**tables
|
pl.DataFrame | None
|
Canonical tables by name. |
{}
|
Returns:
| Type | Description |
|---|---|
dict[str, pl.DataFrame | None]
|
One entry per hierarchy table, None where not supplied. |
Raises:
| Type | Description |
|---|---|
TypeError
|
If a name is not part of the hierarchy. |
safe_join_weight
safe_join_weight(
df: pl.DataFrame, weight_df: pl.DataFrame, on: str
) -> pl.DataFrame
Left-join weight_df onto df, dropping any pre-existing weight columns.
is_usable
is_usable(usability_flag_col: str) -> pl.Expr
True when a record may carry weight; a null flag counts as unusable.
seed_admits
seed_admits(usability_flag_col: str) -> pl.Expr
True when a household belongs in the balancer's seed for this profile.
The profile's own flag, floored by survey_complete. Every profile is a subset of
survey_complete by construction, so the floor changes no number -- it states the
invariant where the seed is chosen, and covers a hand-written flag that is
not.
The same expression decides the seed and the post-balancing zeroing. Were the two to differ, a household the seed rejected could keep a null where the output promises a zero.
distribute_within_scope
distribute_within_scope(
df: pl.DataFrame,
*,
weight_col: str,
scope: str,
usability_flag_col: str,
table: str
) -> pl.DataFrame
Share each scope's claim among its usable records, returning a new frame.
Each record arrives holding its claim -- the weight it would carry if it were usable. Usable records then split the scope's whole claim in proportion to their own, so the scope's total is conserved and every record in it is scaled by the same factor::
w = claim x sum(claims in scope) / sum(claims of usable in scope)
A scope with no usable record has no denominator; its claim is stranded and logged, for the checksum to report as a shortfall.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
df
|
pl.DataFrame
|
Table holding its claim in weight_col. |
required |
weight_col
|
str
|
Weight column, replaced in place by the distributed weight. |
required |
scope
|
str
|
Column naming the group within which the claim is conserved. |
required |
usability_flag_col
|
str
|
Boolean column deciding which records carry weight. |
required |
table
|
str
|
Table name, for logging. |
required |
Returns:
| Type | Description |
|---|---|
pl.DataFrame
|
df with weight_col distributed across the usable records. |
split_among_usable
split_among_usable(
df: pl.DataFrame,
*,
weight_col: str,
key: str,
usability_flag_col: str | None,
table: str
) -> pl.DataFrame
Divide each parent's weight equally among its usable children, in place.
The average-day rule: a parent's children jointly represent the parent once, not once each ::
w = parent_weight / n_usable_children (usable children; 0 otherwise)
so the usable children of every parent sum exactly to the parent's weight.
This is the vendor's own day-weight convention (day_weight =
person_weight / n_weighted_days), with our usability flag deciding which
children count. There is no cross-parent pooling: a parent whose
children are all unusable is a reported shortfall, never re-homed onto
other parents' children.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
df
|
pl.DataFrame
|
Child table holding each row's parent weight in weight_col. |
required |
weight_col
|
str
|
Weight column, replaced in place by the split weight. |
required |
key
|
str
|
Column identifying the parent record. |
required |
usability_flag_col
|
str | None
|
Boolean column deciding which children carry weight, or None to split among all children. |
required |
table
|
str
|
Table name, for logging. |
required |
Returns:
| Type | Description |
|---|---|
pl.DataFrame
|
df with weight_col split among each parent's usable children. |
propagate_weights
propagate_weights(
tables: dict[str, pl.DataFrame | None],
has_weight: dict[str, str],
*,
skip: set[str] | None = None,
usability_flag_col: str | None,
profile: str | None = None
) -> None
Walk the hierarchy, deriving every weight that is not already supplied.
Modifies tables and has_weight in place. Levels are visited in declaration order, so each parent is weighted before its children.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
tables
|
dict[str, pl.DataFrame | None]
|
Mutable dict of table_name → DataFrame (or None). |
required |
has_weight
|
dict[str, str]
|
Mutable dict tracking which tables already have a weight
column and its name. E.g. |
required |
skip
|
set[str] | None
|
Table names to leave alone (e.g. externally supplied weights). |
None
|
usability_flag_col
|
str | None
|
Boolean column deciding which records may carry
weight -- a usability profile stamped upstream. Pass |
required |
profile
|
str | None
|
Usability profile these weights belong to, which suffixes every column written. None writes the un-suffixed set. A parent's column is read from has_weight, never re-derived, so two profiles may be propagated over the same tables without reading each other's numbers. |
None
|
non_null_tables
non_null_tables(
tables: dict[str, pl.DataFrame | None],
) -> dict[str, pl.DataFrame]
Return only the non-None entries from a tables dict.
processing.weighting.core.run_report
The weighting report for one run: where it goes, what it compares, what it says.
One self-contained HTML document per run, written after every profile is fitted.
See generate_diagnostics for its sections.
generate_diagnostics
generate_diagnostics(
pipeline: WeightingPipeline, diagnostics: DiagnosticsConfig
) -> None
Write one pipeline-contained interactive HTML report for the whole run.
Covers the weighting from geographic crosswalk through balancer convergence to what carries weight at every level below the household. Opens in any browser with no external dependencies beyond Plotly, loaded from CDN.
One document, not one per profile. The crosswalk geometry, control
totals and PUMS incidence are built once in setup() however many
profiles are fitted -- and on a regional run that geometry is the great
majority of the file -- so writing a report per fit re-embedded the same
polygons once per profile. Only the balancer's output actually varies,
and it sits behind a profile toggle.
Report sections
- Profile Comparison — one row per fit: seed size, weight sums by level, ESS%, CV, MAPE and convergence. The only view that needs every fit at once, and so the only one no per-fit report could produce.
- Crosswalk Map — choropleth showing how PUMAs overlap the target zones, with allocation weights per PUMA-zone pair and each profile's seed count in the zone tooltip.
- Weight Cascade — per level, what the profile admitted and what carries weight, every profile side by side; the redistribution each survivor absorbs, how each weight set's lower levels descend from the level above, the day-split identity, and control-geography coverage.
- Weight Set Comparer (optional) — any two weight sets compared
pairwise at any level, this run's fits against each other or against
sets named in
diagnostics.compare_weights. Omitted when there is only one set to show. - Fractional Seed Imputation (per profile) — null rate per control in the seed incidence, and the quality of the PUMS-trained model that filled it.
- Balancer Performance (per profile) — per-zone convergence, target fit (MAPE, P90, Max), CV and ESS%.
- Weight Quality (per profile) — per-zone weight and expansion factor statistics, with violin plots.
- Expansion Factor Calibration (per profile, optional) — MAPE vs
CV across the
expansion_factor_gridvalues. Only included when the grid was set in the weighting config. - Target Fit (% Error) (per profile) — diverging bars per control category per zone, with PUMS replicate-weight whiskers.
- Unweighted Cell Counts (per profile) — seed counts per control category per zone, flagging sparse cells.
Numbers are assigned to the sections a run actually renders, so an optional section that is absent takes its number with it rather than leaving a gap.
Parameters
diagnostics:
The diagnostics block of the weighting config.
``output_path`` is the destination for the HTML file, a string
(including template paths from the YAML). When null the file is
written to ``<cache_dir>/diagnostics.html`` (or
``./weighting/diagnostics.html`` if no cache directory is
configured). No profile suffix: a run writes one file however many
profiles it fits.
``compare_weights`` names weight sets that arrived with the survey,
to compare this run's fits against. See
[`reference_weight_sets`][processing.weighting.diagnostics.comparison
.reference_weight_sets] for why they are named rather than found.
processing.weighting.core.specs
Data structures for the weighting pipeline.
Single source of truth for all dataclasses and named tuples used across data preparation, balancing, validation, and diagnostics.
Kept as a leaf module with minimal imports to avoid circular dependencies.
ControlSpec
dataclass
Specification for a single weighting control.
Attributes:
| Name | Type | Description |
|---|---|---|
name |
str
|
Registry name (must exist in |
importance |
float | None
|
Explicit importance weight for the balancer. |
dimensions |
list[str] | None
|
For cross-tab controls only: list of dimension control names
(e.g. |
merges |
dict | None
|
For cross-tab controls only: per-dimension merge specs applied
at registration time. |
name
instance-attribute
name: str
importance
class-attribute
instance-attribute
importance: float | None = None
dimensions
class-attribute
instance-attribute
dimensions: list[str] | None = None
merges
class-attribute
instance-attribute
merges: dict | None = None
ControlTotals
dataclass
Result of PUMS control-total aggregation.
Attributes:
| Name | Type | Description |
|---|---|---|
totals |
pl.DataFrame
|
Tidy frame with columns: [geo_id, control_name, category, target_total] |
pums_hh_count |
int
|
Total PUMS housing unit records (before weighting). |
pums_person_count |
int
|
Total PUMS person records. |
geo_ids |
list[str]
|
Unique geography IDs in the totals. |
totals
instance-attribute
totals: pl.DataFrame
pums_hh_count
instance-attribute
pums_hh_count: int
pums_person_count
instance-attribute
pums_person_count: int
geo_ids
instance-attribute
geo_ids: list[str]
IncidenceBundle
dataclass
All intermediate and final products of incidence-table construction.
Built by
build_incidence_table
and available for downstream consumers (balancer, fractional imputation, diagnostics).
Attributes:
| Name | Type | Description |
|---|---|---|
incidence |
pl.DataFrame
|
Combined household-level incidence table — one row per HH with
|
person_pivot |
pl.DataFrame
|
Per-person 0/1 indicator table — one row per person with
|
household_pivot |
pl.DataFrame
|
HH-only 0/1 indicator table — one row per HH with
|
incidence
instance-attribute
incidence: pl.DataFrame
person_pivot
instance-attribute
person_pivot: pl.DataFrame
household_pivot
instance-attribute
household_pivot: pl.DataFrame
filter_households
filter_households(
hh_ids: list | pl.Series, hh_id_col: str = "hh_id"
) -> IncidenceBundle
Return a new bundle keeping only the given household IDs.
ImputationSummary
dataclass
Per-control fractional seed imputation metadata for diagnostics.
Describes what fill_null_incidence
filled in the seed incidence matrix -- never the survey imputation
step, which writes discrete values to canonical data and is reported
separately.
One entry per control target, regardless of whether any filling was
needed. Controls with no nulls have n_null == 0 and None
for the RF metrics.
control
instance-attribute
control: str
level
instance-attribute
level: str
n_total
instance-attribute
n_total: int
n_null
instance-attribute
n_null: int
log_loss
class-attribute
instance-attribute
log_loss: float | None = None
f1_macro
class-attribute
instance-attribute
f1_macro: float | None = None
PUMSSource
dataclass
Configuration for PUMS data source.
Attributes:
| Name | Type | Description |
|---|---|---|
state_fips |
str
|
Two-digit FIPS code for the state (e.g. "06" for California). |
pums_year |
int
|
ACS 1-year PUMS vintage (e.g. 2022). |
puma_ids |
list[str] | None
|
Optional list of PUMA codes to fetch. If None, fetches all PUMAs in the state (can be large). |
state_fips
instance-attribute
state_fips: str
pums_year
instance-attribute
pums_year: int
puma_ids
class-attribute
instance-attribute
puma_ids: list[str] | None = None
MergeSpec
dataclass
Category-merge specification for one control.
Supports both 1D (standard) and N-D (cross-tab) merges.
Attributes:
| Name | Type | Description |
|---|---|---|
control |
str
|
Registry name (e.g. |
groups |
dict[str, list[str] | dict[str, list[str]]]
|
Merge specifications. Each key is a merged label, each value is either:
The N-D merge creates a single merged cell from the cartesian product of all specified dimension members. |
zones |
list[str] | None
|
If set, apply this merge only to these geo IDs.
|
Example
1D merge:
MergeSpec( ... control="p_employment", ... groups={"employed": ["employed_full", "employed_part"]}, ... )
N-D merge for cross-tab:
MergeSpec( ... control="h_income_x_size", ... groups={ ... "low_income_large": { ... "h_income": ["inc_under_25k", "inc_25k_to_50k"], ... "h_size": ["size_4", "size_5", "size_6"], ... } ... }, ... )
control
instance-attribute
control: str
groups
instance-attribute
groups: dict[str, list[str] | dict[str, list[str]]]
zones
class-attribute
instance-attribute
zones: list[str] | None = None
CensusConfig
pydantic-model
Where the Census controls come from: the state, the PUMS year, and optional local files.
Config:
extra:forbid
Fields:
-
state_fips(str) -
pums_year(int) -
pums_households(str | None) -
pums_persons(str | None)
model_config
class-attribute
instance-attribute
model_config = ConfigDict(extra='forbid')
state_fips
pydantic-field
state_fips: str
Two-digit state FIPS code, e.g. '06'.
pums_year
pydantic-field
pums_year: int
ACS PUMS 1-year vintage the controls are built from.
pums_households
pydantic-field
pums_households: str | None = None
Local PUMS household file; fetched from the Census API when not given.
pums_persons
pydantic-field
pums_persons: str | None = None
Local PUMS person file; fetched from the Census API when not given.
ReferenceWeightSetConfig
pydantic-model
One weight set that arrived with the survey, named for the report's comparer.
Config:
extra:forbid
Fields:
model_config
class-attribute
instance-attribute
model_config = ConfigDict(extra='forbid')
label
pydantic-field
label: str
How the report names the set.
columns
pydantic-field
columns: dict[str, str]
Table -> the column holding this set's weight on it.
DiagnosticsConfig
pydantic-model
The weighting report: where it goes, what it compares, and the grid search.
Config:
extra:forbid
Fields:
-
output_path(str | None) -
expansion_factor_grid(list[float] | None) -
compare_weights(dict[str, ReferenceWeightSetConfig] | None)
model_config
class-attribute
instance-attribute
model_config = ConfigDict(extra='forbid')
output_path
pydantic-field
output_path: str | None
Where the HTML report is written; null for the cache directory.
expansion_factor_grid
pydantic-field
expansion_factor_grid: list[float] | None
max_expansion_factor values to re-run balancing at, charting fit against weight quality; null to skip the grid search.
compare_weights
pydantic-field
compare_weights: dict[str, ReferenceWeightSetConfig] | None = None
Weight sets supplied with the survey, compared against this run's fits.
BalancingConfig
pydantic-model
Maximum-entropy balancer configuration, and how hard each control pulls.
Controls Newton-Raphson iteration bounds, weight expansion limits, parallel
execution and the importance given to controls without their own. Consumed by
balance_weights,
grid_search_expansion_factor,
and the pipeline orchestrator.
Config:
extra:forbid
Fields:
-
max_expansion_factor(float) -
min_expansion_factor(float) -
min_weight(float | None) -
max_weight(float | None) -
max_iterations(int) -
n_workers(int) -
moe_based_importance(bool) -
default_importance(float)
model_config
class-attribute
instance-attribute
model_config = ConfigDict(extra='forbid')
max_expansion_factor
pydantic-field
max_expansion_factor: float
Upper bound on a weight, as a multiple of its seed weight.
min_expansion_factor
pydantic-field
min_expansion_factor: float
Lower bound on a weight, as a multiple of its seed weight.
min_weight
pydantic-field
min_weight: float | None
Absolute weight floor; null for none.
max_weight
pydantic-field
max_weight: float | None = None
Absolute weight ceiling; none unless given.
max_iterations
pydantic-field
max_iterations: int
Newton-Raphson iteration limit per zone.
n_workers
pydantic-field
n_workers: int
Zones (or grid points) balanced in parallel.
moe_based_importance
pydantic-field
moe_based_importance: bool
Derive each control's importance from its PUMS replicate-weight MOE.
default_importance
pydantic-field
default_importance: float
Importance for a control without its own, when not MOE-derived.
ImportanceConfig
dataclass
Importance weighting configuration.
Determines how per-control importance weights are computed:
explicit— YAML-declared overrides (always highest precedence).moe_based— derive importance from PUMS replicate-weight CVs.default— fallback value when neither of the above applies.
explicit
class-attribute
instance-attribute
explicit: dict[str, float] = field(default_factory=dict)
moe_based
class-attribute
instance-attribute
moe_based: bool = False
default
class-attribute
instance-attribute
default: float = 100.0
WeightingConfig
dataclass
Top-level configuration for the weighting pipeline.
Groups the parameters that are not already covered by
ControlRegistryConfig,
BalancingConfig, or
ImportanceConfig
— primarily Census/geography settings and pipeline plumbing.
state_fips
instance-attribute
state_fips: str
pums_year
instance-attribute
pums_year: int
weight_profiles
class-attribute
instance-attribute
weight_profiles: tuple[str, ...] = ()
max_unplaceable_share
class-attribute
instance-attribute
max_unplaceable_share: float = 0.01
pums_households
class-attribute
instance-attribute
pums_households: str | None = None
pums_persons
class-attribute
instance-attribute
pums_persons: str | None = None
sample_plan
class-attribute
instance-attribute
sample_plan: str | None = None
cache_dir
class-attribute
instance-attribute
cache_dir: Path | None = None
expansion_factor_grid
class-attribute
instance-attribute
expansion_factor_grid: list[float] | None = None
strict_survey_nulls
class-attribute
instance-attribute
strict_survey_nulls: bool = False
fitted_profiles
property
fitted_profiles: tuple[str, ...]
One entry per balancing run.
flag_for
flag_for(profile: str) -> str
The verdict column the fit for profile is gated on.
A profile names both its gate and its weights -- usable_<profile> and
hh_weight_<profile> -- so there is no second setting to disagree with
the first.
ControlRegistryConfig
dataclass
Parsed control definitions, merge specs, and derived target names.
Built via from_yaml
from the YAML controls block.
target_names
instance-attribute
target_names: list[str]
importance_overrides
property
importance_overrides: dict[str, float]
Explicit importance overrides from YAML.
from_yaml
classmethod
from_yaml(controls: list[dict]) -> ControlRegistryConfig
Parse the YAML controls block into a config object.
GridPoint
dataclass
Aggregate metrics for one expansion-factor grid point.
max_expansion_factor
instance-attribute
max_expansion_factor: float
converged_zones
instance-attribute
converged_zones: int
total_zones
instance-attribute
total_zones: int
mape
instance-attribute
mape: float
p90
instance-attribute
p90: float
max_error
instance-attribute
max_error: float
cv
instance-attribute
cv: float
ess_pct
instance-attribute
ess_pct: float
SamplePlan
dataclass
Stratified sampling plan mapping Census block groups to segments.
Each row represents a Census block group. Block groups that share
the same sample_segment are treated as a single stratum for
initial-weight computation:
base_weight = segment_bg_pop / segment_n_responses.
Block-group population totals are sourced from the crosswalk
(PumaCrosswalk.block_group_populations), not from this table.
Attributes:
| Name | Type | Description |
|---|---|---|
strata |
pl.DataFrame
|
Required columns:
|
strata
instance-attribute
strata: pl.DataFrame
ZoneStatus
Per-zone convergence diagnostics.
geo_id
instance-attribute
geo_id: str
converged
instance-attribute
converged: bool
iterations
instance-attribute
iterations: int
delta
instance-attribute
delta: float
max_gamma_diff
instance-attribute
max_gamma_diff: float
ZoneInput
Pre-built numpy arrays for a single geography zone.
hh_ids
instance-attribute
hh_ids: pl.Series
incidence
instance-attribute
incidence: np.ndarray
initial
instance-attribute
initial: np.ndarray
lb
instance-attribute
lb: np.ndarray
ub
instance-attribute
ub: np.ndarray
targets
instance-attribute
targets: np.ndarray
importance
instance-attribute
importance: np.ndarray
master_idx
instance-attribute
master_idx: int
max_iterations
instance-attribute
max_iterations: int
geo_id
instance-attribute
geo_id: str
verbose
class-attribute
instance-attribute
verbose: bool = True
ZoneResult
Balancer output for a single geography zone.
weights
instance-attribute
weights: pl.DataFrame
GeographyCoverage
How much of one profile's universe the control geography can place.
A household the control polygons do not contain belongs to no balancing zone, so no fit can give it a weight. That is a bound on what the weighting can answer, not a defect in the household, and it is counted so a reader sees the bound rather than inferring it from nulls.
profile
instance-attribute
profile: str | None
n_universe
instance-attribute
n_universe: int
n_placed
instance-attribute
n_placed: int
n_unplaceable
property
n_unplaceable: int
Households in the profile that the control geography cannot place.
unplaceable_share
property
unplaceable_share: float
Unplaceable households as a share of the profile's universe.
ProfileFit
dataclass
Everything one balancing run produced.
A run may fit several profiles over the same survey. Holding each fit's products here rather than on the pipeline keeps them from overwriting one another -- otherwise the last profile's seed silently becomes the one the diagnostics describe and the checks read.
Attributes:
| Name | Type | Description |
|---|---|---|
profile |
str | None
|
Usability profile fitted, or None for a single un-suffixed set. Suffixes every weight column this fit writes. |
usability_flag_col |
str
|
The column that gated the seed and will gate the propagation. Equal to profile whenever a profile was named. |
seed_incidence |
pl.DataFrame
|
The seed this fit was solved from, carrying its own
|
pre_imputation_incidence |
pl.DataFrame
|
The same seed before null imputation, for the diagnostics report. |
imputation_summary |
list[ImputationSummary]
|
Per-control imputation metadata for this seed. |
coverage |
GeographyCoverage
|
How much of the profile's universe had a control geography. |
weights |
pl.DataFrame | None
|
Balancer output, keyed by |
statuses |
list[ZoneStatus]
|
Per-zone convergence diagnostics. |
grid_results |
list[GridPoint] | None
|
Expansion-factor grid points, when a grid was configured. |
profile
instance-attribute
profile: str | None
usability_flag_col
instance-attribute
usability_flag_col: str
seed_incidence
instance-attribute
seed_incidence: pl.DataFrame
pre_imputation_incidence
instance-attribute
pre_imputation_incidence: pl.DataFrame
weights
class-attribute
instance-attribute
weights: pl.DataFrame | None = None
statuses
class-attribute
instance-attribute
statuses: list[ZoneStatus] = field(default_factory=list)
unconverged_zones
property
unconverged_zones: int
Zones this fit failed to converge.
resolve_fitted_profiles
resolve_fitted_profiles(
weight_profiles: tuple[str, ...] | list[str] | None,
) -> tuple[str, ...]
The profiles to weight: one entry per weight column set to be written.
Shared by both weighting entry points so they cannot disagree about what a profile list is supposed to mean.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
weight_profiles
|
tuple[str, ...] | list[str] | None
|
Profiles to weight, each writing its own suffixed column set. A single profile is a list of one. |
required |
Returns:
| Type | Description |
|---|---|
tuple[str, ...]
|
One entry per profile, in the order given. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If no profile is given, or a profile is named twice. |