Skip to content

Diagnostics

Diagnostics sub-package — HTML report generation for weighting results.

Produces one self-contained interactive HTML report per run (Plotly + Jinja2, no external dependencies), covering every profile the run fitted. The crosswalk geometry, control totals and PUMS incidence are built once however many profiles are fitted -- and on a regional run that geometry is the great majority of the file -- so one document per run embeds them once rather than once per profile.

A sidebar outlines the document and holds the profile toggle. Every statistic is defined, with its formula, in a Definitions section at the top (see glossary), and abbreviated headers link to their entry. Each section shows a one-line summary, with the longer guidance collapsed beneath it.

Run-level sections, always visible

  1. Profile Comparison — one row per fit: seed size, weight sums by level, ESS%, CV, max/median, MAPE and convergence. Two profiles matching exactly means two balancing runs produced one answer.
  2. Crosswalk Map — geographic crosswalk visualization, with every profile's seed count in each zone's tooltip. Boundaries are simplified to a few metres.
  3. Weight Cascade — per level, what each profile admitted and what carries weight, all profiles side by side; plus the redistribution each survivor absorbs, whether each weight set's lower levels were copied, redistributed or adjusted per record, the day-split identity, and control-geography coverage.
  4. Weight Set Comparer — any two weight sets compared at any level: this run's fits against each other, or against weights supplied with the survey and named in diagnostics.compare_weights. Run-level because a comparison names two profiles. Shows a scatter with a reduced-major-axis fit, the median, p90 and maximum fold difference, rank correlation and R-squared, and the spread of ln(A/B) by decile. No verdict is drawn: a comparison is not a validation, and the section says so above the plot.

Per-profile sections, behind the profile toggle

  1. Fractional Seed Imputation — per-control null rate in the seed incidence matrix and the quality of the PUMS-trained model that filled it. Shown only when something was filled. This describes the seed handed to the balancer, not the survey imputation pipeline step.
  2. Balancer Performance — per-zone convergence status, target fit (MAPE, P90, Max), CV and ESS%.
  3. Weight Quality — per-zone weight and expansion-factor statistics, with violin plots of final_weight / base_weight.
  4. Expansion Factor Calibration — MAPE vs CV across a grid of max_expansion_factor values. Enabled by setting expansion_factor_grid in the weighting config.
  5. Target Fit (% Error) — diverging bar charts per control category per zone, with PUMS replicate-weight whiskers.
  6. Unweighted Cell Counts (Data Sparsity) — seed counts per control category per zone.

Each per-profile heading names its profile, so a screenshot cropped out of the document still says which fit it describes.

Section numbers are assigned to the sections a run actually renders, so they run consecutively from 1 whether or not the optional ones are present -- the numbers above are a full run. A per-profile section is numbered once for the whole document rather than per pane: the number is an address into the report, so a profile that omits a section leaves a gap in its own pane rather than shifting every heading below it when the reader presses the toggle.

Configuration (YAML)

diagnostics:
  output_path: "{{ output_dir }}/weighting_diagnostics.html"
  compare_weights:            # optional; weight sets supplied with the survey
    vendor:
      label: vendor
      columns: {households: hh_weight, unlinked_trips: trip_weight}

When output_path is omitted the report is written to <cache_dir>/diagnostics.html. One file per run, with no profile suffix.

report

Report orchestration: assemble sections and render the Jinja2 template.

Entry point is generate_report, which collects every fit a run produced, builds Plotly figures and HTML tables via the sibling modules (charts, crosswalk_map, data, tables, profile_tables, comparison, glossary), then renders them into a single .html file using a bundled Jinja2 template.

One document per run, not per fit

The crosswalk geometry, the control totals and the PUMS incidence are properties of the run: they are built once in setup() however many profiles are fitted. Only the balancer's output varies per profile. Writing one report per fit therefore re-embedded the same polygons once per profile -- on a Bay Area run that geometry is 95% of the file -- so the report is assembled once, with the per-fit sections behind a profile toggle.

What sits where follows from what can be compared on screen:

  • Run-level, always visible -- the definitions, the profile comparison, the map, the weight-cascade tables and the weight set comparer. These are small enough to show every profile at once, and a labelled column cannot be misread the way a toggled chart can.
  • Per-profile, toggled -- the per-zone tables and every figure. Three profiles of a per-zone fit grid will not fit side by side, so they are switched rather than juxtaposed, and each carries its profile in the heading so a cropped screenshot still says what it shows.

charts

Plotly chart builders for the diagnostics report.

crosswalk_map

The crosswalk map: how PUMAs overlap the target zones, for the weighting report.

data

Data transformations for the diagnostics report.

comparison

Pairwise comparison of weight sets, at every level of the hierarchy.

A weight set is one complete assignment of weights across the hierarchy: a profile this run fitted, or a set that arrived with the survey and is being used as a reference. Any two of them can be compared at any level where both carry a column.

What a comparison can and cannot say

Weights from two methods are not expected to agree record by record. Balancing matches marginal totals, and many different weight vectors satisfy the same margins equally well, so two methods can both be correct and still disagree on nearly every household. Low correlation is not evidence that either is wrong, and high correlation is not evidence that either is right. Nothing here is a validation, and the report says so above the plot rather than in a footnote.

What the comparison can show is how far apart two sets are, whether they agree on which records are heavy, and whether the gap is even across the range of weights or widens at one end. The report shows those numbers and leaves the judgement to the analyst.

The fit line

Least squares is the wrong fit for two weight sets. It treats one set as known and the other as noisy, so ln(A) on ln(B) and ln(B) on ln(A) give slopes that are not reciprocals, and a weak correlation drags both toward zero. On bats_2023 the household fit of our weights on the vendor's gave 0.42, the reverse fit implied 1.74, and the spreads actually differ by a factor of 0.86. The reduced major axis is symmetric and does not dilute: its slope is the ratio of the two spreads, signed by the correlation.

glossary

Definitions of every statistic the diagnostics report uses.

One source for both places a definition appears: the Definitions section at the top of the report, and the hover text on a column header or tile that uses the statistic. A header links to its entry, so the hover is a shortcut to the reference rather than the only place the definition lives.

profile_tables

The report's per-profile tables: how each fit compares, and what carries weight.

One row or column per profile, so the fits can be read side by side: seed and weight sums, the cascade of usable records, redistribution, the day split, control-geography coverage and how lower-level weights descend from the level above.

tables

HTML table builders for the diagnostics report.

This module generates five of the tables displayed in the diagnostics HTML report:

  1. Balancer Performance Table — Per-zone convergence status, target-fit metrics (MAPE, P90, Max), and weight quality metrics (CV, ESS%). Combines all key performance indicators into a single comprehensive table.

  2. Weight Quality Table — Per-zone weight distribution statistics (mean, median, std, min, max) and expansion factor stats (min/max/mean/median EF ratio).

  3. Unweighted Cell Counts — Data sparsity matrix showing unweighted sample counts per control category per zone, with optional PUMS-weighted percentages for context.

  4. Crosswalk Summary Table — Zone → HH samples mapping with optional zone group aggregation.

  5. Imputation Summary Table — Per control, how many seed rows had an empty incidence block, and the cross-validated log-loss and F1 of the PUMS-trained model that filled them.

All table builders return raw HTML strings suitable for Jinja2 template insertion, built with the helpers in table_html. The per-profile tables are in profile_tables.

Code: processing.weighting.diagnostics