Compute Weights

This path computes new expansion weights from PUMS controls and survey seed data.

It is the full weighting workflow, including:

data preparation and PUMS fetching
control recoding and shared incidence construction
survey null imputation
zone assignment and control merges
control aggregation and importance resolution
max-entropy balancing
weight propagation, diagnostics, and validation

Both PUMS (Census microdata) and Survey (travel-diary seed) are conformed to the same incidence schema: one row per household, with shared 1-D controls and cross-tab targets expanded into a common set of incidence columns. That shared representation is what allows the same geography and merge logic to operate on both datasets.

%%{init: {'theme': 'base', 'themeVariables': {
  'background': '#fcfaf6',
  'primaryTextColor': '#22313f',
  'primaryBorderColor': '#9f8b6d',
  'lineColor': '#6b7280',
  'fontFamily': 'IBM Plex Sans, Segoe UI, sans-serif'
}}}%%
flowchart TD
  A("Setup\n• Register controls\n• Resolve 1-D targets\n• Resolve cross-tab targets\n• Build crosswalk\n• Fetch PUMS")

  subgraph inputs ["**Normalize to shared incidence schema**"]
    direction LR
    P0(["PUMS HH + Person"]) --> P1("Recode + pivot") --> PI[["PUMS incidence"]]
    S0(["Survey HH + Person"]) --> S1("Recode + pivot") --> SI[["Survey incidence"]]
  end

  NI("Null imputation on survey only\nRF trained on PUMS incidence")

  subgraph shared_stage ["**Shared incidence transforms**"]
    direction TD
    G1("Zone assignment\n• assign survey HHs\n• allocate PUMAs to control zones")
    G2("Create cross-tab targets")
    G3("Apply merges\ncollapse merged categories")
    G1 --> G2
    G2 --> G3
  end

  subgraph outputs ["**Divergent downstream roles**"]
    direction LR
    PZ[["PUMS incidence\nwith geography + merges"]] --> CT[["Control totals\naggregate PUMS by zone"]]
    PZ --> IM[["Importance\nMOE from PUMS or explicit config"]]
    SZ[["Seed incidence\nwith geography + merges"]]
  end

  B("Max-entropy balancing\nseed incidence × control totals × importance")
  O(["Weight propagation\ndiagnostics\nvalidation"])

  A --> P0
  A --> S0
  PI -. "Train RF Model" .-> NI
  SI --> NI
  PI --> G1
  NI --> G1
  G3 --> PZ
  G3 --> SZ
  SZ --> B
  CT --> B
  IM --> B
  B --> O

  classDef setup fill:#dce7f5,stroke:#557aa3,color:#1d2c3c,stroke-width:2px;
  classDef source fill:#f8f1e5,stroke:#c7a977,color:#3a3126,stroke-width:1.5px;
  classDef action fill:#efe7d8,stroke:#bca07a,color:#2f2a22,stroke-width:1.5px;
  classDef pums fill:#d9ecf7,stroke:#5b8fb9,color:#17324d,stroke-width:1.5px;
  classDef survey fill:#f8dfc9,stroke:#cf8c52,color:#4a2b15,stroke-width:1.5px;
  classDef shared fill:#e3efe2,stroke:#7ca16f,color:#233821,stroke-width:1.5px;
  classDef balance fill:#dcefe6,stroke:#4f7b6c,color:#183229,stroke-width:2px;
  classDef output fill:#f3dca2,stroke:#b6903b,color:#47360f,stroke-width:1.5px;

  class A setup;
  class P0,S0 source;
  class P1,S1,NI action;
  class PI,PZ,CT,IM pums;
  class SI,SZ survey;
  class G1,G2,G3 shared;
  class B balance;
  class O output;

  style inputs fill:#f9f4ec,stroke:#ccb99d,stroke-width:2px,color:#3a3126;
  style shared_stage fill:#edf5ec,stroke:#9db798,stroke-width:2px,color:#233821;
  style outputs fill:#f9f4ec,stroke:#ccb99d,stroke-width:2px,color:#3a3126;

The important split is not that PUMS and survey run as fully parallel pipelines. It is that they are both transformed into the same incidence format. From there, PUMS incidence is used to produce control totals and MOE-based importance, while the survey incidence becomes the seed to be reweighted. Null imputation is applied only to the survey side; PUMS is used as the training source for that step.

The step

Compute expansion weights from PUMS controls and propagate to all tables.

Constructs a WeightingPipeline; full documentation of the algorithm, configuration, and diagnostics is included in that class.

One fit per profile

A run weights each profile named in weight_profiles separately, gating its seed on that profile's own flag, and writes one column set per fit suffixed with the profile's name. Each fit's products are held on its own ProfileFit rather than on the pipeline, so several fits over one survey cannot overwrite each other's seed, weights or diagnostics.

Example

- name: compute_weights
  params:
    weight_profiles: [ctramp, daysim]
    sample_plan: data/sample_plan.csv
    max_unplaceable_share: 0.01
    census:
      state_fips: "06"
      pums_year: 2023
    geography:
      target_zones:
        file: zones/counties.shp
        id_field: fipco
      resolution: 50        # meters
      min_allocation: 0.01  # drop PUMA slivers below 1%
      zone_groups:
        "San Francisco": ["075"]
        "Alameda": ["001"]
    balancing:
      max_expansion_factor: 5.0
      min_expansion_factor: 0.01
      min_weight: 1
      max_iterations: 10000
      n_workers: 8
      moe_based_importance: true
      default_importance: 100
    controls:
      - name: h_total
        importance: 200
      - name: h_size
        merge:
          size_4_plus: [size_4, size_5, size_6, size_7, size_8, size_9, size_10_plus]
      - name: p_gender
    diagnostics:
      output_path: output/weighting_diagnostics.html
      expansion_factor_grid: [2, 3, 5, 10]

Settings

Key	Type	Default	Meaning
`controls`	list	required	The control list, each by registry name with optional `importance` and `merge`.
`weight_profiles`	list	required	Usability profiles to weight, one fit each; a single profile is a list of one.
`max_unplaceable_share`	number	required	The most of the survey whose homes may fall outside the control geography before the step raises.
`sample_plan`	text or null	required	Path to the sample plan for base weights; null for none.
`strict_survey_nulls`	true / false	`false`	Raise on survey nulls in control variables rather than imputing them.

`census`

Key	Type	Default	Meaning
`state_fips`	text	required	Two-digit state FIPS code, e.g. '06'.
`pums_year`	whole number	required	ACS PUMS 1-year vintage the controls are built from.
`pums_households`	text or null	`null`	Local PUMS household file; fetched from the Census API when not given.
`pums_persons`	text or null	`null`	Local PUMS person file; fetched from the Census API when not given.

`geography`

Key	Type	Default
`target_zones`	block (see `geography.target_zones`)	required
`resolution`	whole number	required
`min_allocation`	number	required
`zone_groups`	mapping or null	required

`geography.target_zones`

Key	Type	Default	Meaning
`file`	text	required
`id_field`	text or null	required	Zone-ID column; null for one study-area boundary.

`balancing`

Key	Type	Default	Meaning
`max_expansion_factor`	number	required	Upper bound on a weight, as a multiple of its seed weight.
`min_expansion_factor`	number	required	Lower bound on a weight, as a multiple of its seed weight.
`min_weight`	number or null	required	Absolute weight floor; null for none.
`max_weight`	number or null	`null`	Absolute weight ceiling; none unless given.
`max_iterations`	whole number	required	Newton-Raphson iteration limit per zone.
`n_workers`	whole number	required	Zones (or grid points) balanced in parallel.
`moe_based_importance`	true / false	required	Derive each control's importance from its PUMS replicate-weight MOE.
`default_importance`	number	required	Importance for a control without its own, when not MOE-derived.

`diagnostics`

Key	Type	Default	Meaning
`output_path`	text or null	required	Where the HTML report is written; null for the cache directory.
`expansion_factor_grid`	list or null	required	max_expansion_factor values to re-run balancing at, charting fit against weight quality; null to skip the grid search.
`compare_weights`	mapping or null	`null`	Weight sets supplied with the survey, compared against this run's fits.

A block's keys may also sit flat under params:; see Configuring a step.

Code: processing.weighting

Compute Weights

The step

One fit per profile

Example

Settings

census

geography

geography.target_zones

balancing

diagnostics

Related Topics

`census`

`geography`

`geography.target_zones`

`balancing`

`diagnostics`