Skip to content

processing.weighting.existing

Machinery for add_existing_weights.

Its settings, and moving supplied weights onto usable records.

processing.weighting.existing.config

Settings for add_existing_weights: each supplied weight file, and the step's own.

ExistingWeightConfig pydantic-model

Configuration for a single weight file.

Most fields use sensible defaults - typically you only need to provide weight_path. Override weight_id_col only if your weight file uses different ID column names. Override weight_col only if your weight file uses different weight column names.

Examples:

Basic usage with defaults:

>>> ExistingWeightConfig(config_key="hh_weight", weight_path="hh_weights.csv")
# Uses: weight_id_col="hh_id", weight_col="hh_weight"

Override weight column name:

>>> ExistingWeightConfig(
...     config_key="person_weight",
...     weight_path="person_weights.csv",
...     weight_col="final_weight"  # Weight file uses "final_weight" not "person_weight"
... )

Override ID column name (external weight file with non-canonical IDs):

>>> ExistingWeightConfig(
...     config_key="unlinked_trip_weight",
...     weight_path="trip_weights.csv",
...     weight_id_col="trip_id",  # Weight file uses "trip_id" not "unlinked_trip_id"
...     weight_col="trip_weight" # Weight file uses "trip_weight" not "unlinked_trip_weight"
... )

Attributes:

Name Type Description
config_key str

Key identifying the weight type (e.g., 'hh_weight')
weight_path Path

Path to the CSV file containing weights
weight_id_col str | None

ID column name in the weight file. Defaults to canonical table ID.
weight_col str | None

Weight column name in the weight file. Defaults to canonical weight column.

Fields:

Validators:

config_key pydantic-field 

config_key: str

weight_path pydantic-field 

weight_path: Path

weight_id_col pydantic-field 

weight_id_col: str | None = None

Defaults to canonical table ID if not provided

weight_col pydantic-field 

weight_col: str | None = None

Defaults to canonical weight column if not provided

table_name property 

table_name: str

Get the table name for this weight config.

table_id_col property 

table_id_col: str

Get the canonical table ID column.

canonical_weight_col property 

canonical_weight_col: str

Get the canonical weight column name for this table.

validate_config_key pydantic-validator 

validate_config_key(v: str) -> str

Validate that config_key is one of the allowed values.

validate_path_exists pydantic-validator 

validate_path_exists(v: Path) -> Path

Validate that the weight file exists.

set_defaults_from_mapping pydantic-validator 

set_defaults_from_mapping() -> ExistingWeightConfig

Set default values for weight_id_col and weight_col from the mapping.

joined_weight_col 

joined_weight_col(profile: str) -> str

The column the supplied weight lands in under profile.

validate_columns 

validate_columns(
    weight_df: pl.DataFrame, table_df: pl.DataFrame
) -> None

Validate that required columns exist in weight file and table.

Parameters:

Name Type Description Default
weight_df pl.DataFrame

The weight DataFrame loaded from weight_path

 required
table_df pl.DataFrame

The survey data table to join weights to

 required

Raises:

Type Description
ValueError

If required columns are missing

ExistingWeightsConfig pydantic-model

Settings for add_existing_weights: how supplied weights move onto usable records.

Config:

  • extra: forbid

Fields:

model_config class-attribute instance-attribute 

model_config = ConfigDict(extra='forbid')

preserve_travel_totals pydantic-field 

preserve_travel_totals: bool

Households are always held to their total and persons to their household, as known populations, and a person's days carry that person's weight. Every unlinked trip moves with its day, by the day's final weight over its supplied weight. Trips and tours are not targets, so whether a dropped trip's weight moves is a choice. true: a kept day's dropped trips hand their weight to its kept trips, in proportion, so the day's trips keep their total. false: every kept trip keeps its own weight times its day's factor, and a dropped one's weight is gone. Either way, linked trips and tours are the mean of their final trips, and joint trips and joint tours the sum of their kept members.

household_size_top_group pydantic-field 

household_size_top_group: int

Households are scaled by one factor per household size (number of persons), so dropping more large households than small ones does not move the person total. Sizes from this one up share a factor, which keeps a size with few households from carrying an extreme one.

weight_profiles pydantic-field 

weight_profiles: list[str]

Usability profiles to weight, each writing its own column set suffixed with the profile name; a single profile is a list of one. Each is stamped upstream by cascade_completeness; name survey_complete to weight the whole valid survey including partial and overnight tours. The supplied files are read once and moved onto each profile's usable records rather than simply zeroed, so every set holds the same household and person totals -- the vendor's anchor cannot be re-balanced from here.

derive_missing_weights pydantic-field 

derive_missing_weights: bool

Whether to derive weights for tables without supplied weight files.

max_weight_factor pydantic-field 

max_weight_factor: dict[str, float] | None = None

Optional limits, table -> the most a record's final weight may be, as a multiple of its own supplied weight. A table left out has no limit. The step raises naming the records past it. Each table named must have a supplied weight.

attach_supplied 

attach_supplied(
    tables: dict[str, pl.DataFrame | None],
    supplied: dict[str, pl.DataFrame],
    configs: dict[str, ExistingWeightConfig],
    profile: str | None,
) -> dict[str, str]

Join each supplied weight under its column for profile, in place.

Returns:

Type Description
dict[str, str]

table -> the weight column that is actually there, which is what every
dict[str, str]

later step reads the weight by.

processing.weighting.existing.population

Supplied weights onto usable households, persons and days.

Households scale by one factor per household size; persons copy their household, a dropped person's share staying inside it; a person's days carry exactly that person's weight.

SCALE_TOL module-attribute 

SCALE_TOL = 1e-09

POPULATION_TABLES module-attribute 

POPULATION_TABLES = frozenset({'households', 'persons', 'days'})

CARRIES module-attribute 

CARRIES = '_can_carry_weight'

apply_usability 

apply_usability(
    tables: dict[str, pl.DataFrame | None],
    has_weight: dict[str, str],
    *,
    usability_flag_col: str
) -> dict[str, float]

Redistribute supplied weights onto usable records, preserving each total.

Unlike the computed path, the vendor's anchor cannot be re-balanced: their weights already sum to their population estimate, so dropping records must not shrink that total. Each supplied weight is first conserved within its declared scope (distribute_within_scope); whatever that leaves stranded — scopes with no usable record at all — is then closed with one table-wide factor. Modifies tables in place.

Parameters:

Name Type Description Default
tables dict[str, pl.DataFrame | None]

Mutable dict of table_name → DataFrame (or None).

 required
has_weight dict[str, str]

Dict of table_name → weight column name.

 required
usability_flag_col str

Boolean column deciding which records may carry weight -- a usability profile stamped upstream, or survey_complete to weight the whole valid survey.

 required

Returns:

Type Description
dict[str, float]

table -> the supplied total, for the placement check to hold each
dict[str, float]

table to.

mark_persons_who_can_carry 

mark_persons_who_can_carry(
    tables: dict[str, pl.DataFrame | None],
    has_weight: dict[str, str],
    supplied: Collection[str],
    *,
    usability_flag_col: str
) -> None

Add the CARRIES verdict to persons, in place.

Days take weight in proportion to the supplier's, so a day weighted 0 stays 0. A usable person whose supplier-weighted days were all dropped has no day to put their weight on, so they count as a dropped person: their weight goes to the household's other persons. A household with no person who can carry weight counts as a dropped household. Their own verdicts are untouched -- they stay usable, with weight 0.

Raises:

Type Description
ValueError

If there is no persons table, or it carries no verdict. Household size is the number of persons, so there is no size without them.

gate_households_by_size 

gate_households_by_size(
    tables: dict[str, pl.DataFrame | None],
    has_weight: dict[str, str],
    *,
    usability_flag_col: str,
    top_group: int
) -> float

Scale kept households by one factor per household size, in place.

Size is the number of persons; sizes from top_group up share one group. Within each group::

household = supplied x (group's supplied total / group's kept supplied total)

Because every household in a group has the same size, this holds the group's person total as well as its household total -- one factor for all sizes would not, when the dropped households are larger or smaller than the kept ones. A household is kept when it is usable and has a person who can carry weight.

Returns:

Type Description
float

The supplied household total, for the placement check to hold.

Raises:

Type Description
ValueError

If a size group carries supplied weight but has no kept household to put it on.

persons_follow_household 

persons_follow_household(
    tables: dict[str, pl.DataFrame | None],
    has_weight: dict[str, str],
) -> None

Give each household's weight to its persons who can carry it, in place.

A household stands for all its persons, so its persons together carry its weight times its number of persons. Those who can carry share it in proportion to their supplied weights::

person = household x n_persons x supplied / sum(supplied of those who can carry)

so a dropped person's share stays in the household, and household weight x number of persons = the sum of its persons' weights.

follow_parent 

follow_parent(
    tables: dict[str, pl.DataFrame | None],
    has_weight: dict[str, str],
    table: str,
    *,
    usability_flag_col: str
) -> None

Share each parent's final weight among table's usable records, in place.

Each record takes its parent's weight in proportion to its own claim::

w = claim x parent_final / sum(usable claims under the parent)

so a parent's usable records sum to the parent, and a record the supplier weighted 0 stays 0. A parent whose final weight is positive but that has no usable claim under it is stranded: its weight reaches no record here, and is logged for the totals check to report.

processing.weighting.existing.travel

Supplied weights onto usable trips and tours, and the checks after redistribution.

A trip moves with its day; linked trips and tours are the mean of their final trips, and joint trips and joint tours the sum of their kept members.

TRAVEL_TABLES module-attribute 

TRAVEL_TABLES = frozenset(
    {"unlinked_trips", "linked_trips", "tours"}
)

JOINT_TABLES module-attribute 

JOINT_TABLES = frozenset({'joint_trips', 'joint_tours'})

gate_travel 

gate_travel(
    tables: dict[str, pl.DataFrame | None],
    has_weight: dict[str, str],
    *,
    usability_flag_col: str,
    preserve_travel_totals: bool
) -> dict[str, float]

Gate the travel tables, each record starting from its pre-gate claim, in place.

Used only where a travel weight cannot follow its day: a linked trip or tour weight supplied as a file, or unlinked trips supplied without day weights. When days and unlinked trips are both supplied, the trips follow their day ([trips_follow_their_day]) and this does not see them.

A claim is the weight a record would carry if every trip and tour were usable: the supplied weight, or the weight derived from its usable day.

With preserve_travel_totals, a dropped record's claim moves to the kept records on its day and anything left stranded closes region-wide, so each table keeps its claim total. Without it, a kept record keeps exactly its claim and a dropped one carries nothing.

Returns:

Type Description
dict[str, float]

table -> the claim total the placement check holds it to; empty when
dict[str, float]

the totals are not preserved.

check_weight_factors 

check_weight_factors(
    tables: dict[str, pl.DataFrame | None],
    has_weight: dict[str, str],
    supplied: dict[str, pl.DataFrame],
    limits: dict[str, float],
) -> None

Raise if any record's weight grew past its limit times its supplied weight.

The factor is a record's final weight over its own supplied weight, so it counts everything that moved weight onto it: a household's size factor, a person taking a dropped member's share, a day taking a dropped day's.

log_usable_supplied_zero 

log_usable_supplied_zero(
    tables: dict[str, pl.DataFrame | None],
    has_weight: dict[str, str],
    *,
    usability_flag_col: str
) -> None

Count the usable records the supplier gave no weight.

Supplied weights are used as given, so such a record stays at 0. The count is how far this profile's universe is wider than the supplier's.

trips_follow_their_day 

trips_follow_their_day(
    tables: dict[str, pl.DataFrame | None],
    has_weight: dict[str, str],
    supplied_days: pl.DataFrame,
    *,
    usability_flag_col: str,
    preserve_travel_totals: bool
) -> None

Scale each unlinked trip by its day's factor, in place.

A day's factor is its final weight over its supplied weight, so a trip moves exactly as its day did -- a dropped day's weight went to the person's other days, and the trips on those days carry it::

w = claim x day_final / day_supplied

With preserve_travel_totals, a kept day's dropped trips also hand their claim to its kept trips, in proportion::

w x sum(claims on the day) / sum(usable claims on the day)

Raises:

Type Description
ValueError

If a trip carries a supplied weight but its day was supplied 0 or not at all -- there is no factor to apply, and guessing one would invent the supplier's intent.

warn_null_weights 

warn_null_weights(
    tables: dict[str, pl.DataFrame | None], profile: str | None
) -> None

Report weights that came out null -- a record no supplied total covered.