Imputation

Data imputation for missing values in travel diary survey data.

This module provides configurable imputation using established statistical methods. It is designed as a pipeline step that operates on any combination of canonical survey tables (households, persons, days, trips, tours).

Supported methods

• KNN -- single-column imputation via K-Nearest Neighbors similarity matching. Best for isolated missing fields where similar records exist.
• Random Forest -- single-column imputation using a supervised RF model that auto-selects classifier vs. regressor based on the column type. Best for complex non-linear relationships or mixed feature types.
• MICE -- multi-column imputation via Multiple Imputation by Chained Equations. Best for correlated variables (e.g. depart/arrive/duration).

Additional capabilities

• Pre-imputation stash -- optional {column}_preimputed columns that preserve the original value (null, PNTA, MISSING, etc.) before imputation, enabling downstream equity analysis and imputation auditing.
• Quality validation -- optional k-fold cross-validation that masks known values, re-imputes them, and reports accuracy / RMSE metrics.
• Method comparison -- head-to-head benchmark of KNN, RF, and MICE on every imputed column using the same folds and feature sets.
• Cross-table features -- join_tables pulls parent-table columns and auto-generates within-household mode features; aggregate_from pivots child rows up to a parent.

Typical pipeline position

steps:
  - name: load_data
  - name: clean_<project>
  - name: imputation        # ← after cleaning, before linking
  - name: link_trips
  - name: detect_habitual_locations
  - name: detect_joint_trips
  - name: extract_tours

Supported relationships for joining tables

These parent → child relationships are supported for join_tables

Child table	Parent table	Join key
persons	households	hh_id
days	persons / households	person_id / hh_id
unlinked_trips	days / persons / households	day_id / person_id / hh_id
linked_trips	days / persons / households	day_id / person_id / hh_id
tours	persons / households	person_id / hh_id

Missing-data assumptions

• KNN assumes similar records (by feature distance) share similar values.
• MICE assumes Missing At Random (MAR): missingness may depend on observed values but not on the missing value itself.
• If data is Missing Not At Random (MNAR), results may be biased.

Current limitations

• No stratified imputation (no group_by option for within-group models).
• No support for exogenous data sources (PUMS, land use data).
• High-cardinality one-hot encoding can slow MICE convergence -- move ordinal/count variables to numeric_features to mitigate.

References

• van Buuren, S. & Groothuis-Oudshoorn, K. (2011). mice: Multivariate Imputation by Chained Equations in R. Journal of Statistical Software, 45(3), 1-67.
• Troyanskaya, O. et al. (2001). Missing value estimation methods for DNA microarrays. Bioinformatics, 17(6), 520-525.

The step

Impute missing values using KNN, Random Forest, and/or MICE methods.

Each config block specifies its method (knn, rf, or mice) along with the method-specific parameters. Configs are grouped by method and executed in a fixed order (KNN → RF → MICE) across all tables so that later phases can benefit from values filled in earlier phases.

Handling Missing Values with Enum Labels

Survey data often uses special codes for missing values (e.g. 995 for "Missing Response", 999 for "Prefer not to answer"). Use enum member names (labels) rather than raw numeric values in the config::

missing_values: [MISSING, PNTA]   # enum labels, not 995/999

The module automatically:

1. Maps the table name to the appropriate codebook module (e.g. households → data_canon.codebook.households).
2. Finds the enum class whose canonical_field_name matches the target column (e.g. income_broad → IncomeBroad).
3. Resolves enum member names to their values (e.g. MISSING → 995).
4. Replaces those values with null before imputation.

For MICE with multiple columns, missing_values can be a dict mapping each column to its own labels, or a single list applied to all columns::

# Per-column
missing_values:
  race: [MISSING]
  ethnicity: [MISSING, PNTA]

# Shared
missing_values: [MISSING, PNTA]   # applied to all columns

Cross-Table Features (join_tables): By default only features from the same table are used. Adding join_tables to a config block pulls columns from parent tables via left-join on known foreign keys, which can significantly improve quality.

Behaviour:

1. Columns from the specified parent table(s) are joined onto the
   child table (e.g. ``persons`` ← ``households`` via ``hh_id``).
2. For each target column a ``hh_mode_{column}`` feature is
   auto-generated — the mode of that column among *other* household
   members (exclude-self).  This captures within-household
   correlation (e.g. siblings sharing race/ethnicity).
3. Auto-generated ``hh_mode_*`` columns are appended to
   ``categorical_features`` automatically.
4. After imputation all joined/aggregated columns are stripped;
   the output schema is unchanged.

Example::

    impute_columns:
      persons:
        - method: knn
          column: gender
          n_neighbors: 5
          join_tables: [households]
          categorical_features: [age, employment, income_bin, residence_type]
          #                                       ^^^^^^^^^^  ^^^^^^^^^^^^^^
          #                              columns from the households table

Child-to-Parent Aggregation (aggregate_from): The reverse of join_tables: aggregate child rows up to a parent table. Useful when imputing parent-level fields that depend on household composition (e.g. predicting household income from the employment/education mix of its members).

For each child table and each field listed under ``pivot_count``, the
module groups child rows by the parent's FK and creates one column per
unique value, counting occurrences.  Generated columns are named
``{child_table}_count_{field}={value}`` and are automatically added to
``numeric_features``.  After imputation, all generated columns are
stripped.

Example::

    impute_columns:
      households:
        - method: mice
          columns: [income_bin]
          aggregate_from:
            persons:
              pivot_count: [employment, education, student]
          categorical_features: [residence_type, residence_rent_own]
          max_iter: 10

Example config

.. code-block:: yaml

impute_columns:
  households:
    - method: knn
      column: income_broad
      missing_values: [MISSING, PNTA]
      n_neighbors: 5
      neighbor_weights: distance
      numeric_features: [num_persons, num_vehicles, num_workers]
  persons:
    - method: knn
      column: gender
      missing_values: [MISSING]
      n_neighbors: 5
      join_tables: [households]
      numeric_features: [age]
      categorical_features: [relationship, employment, income_bin]
    - method: rf
      column: education
      missing_values: [MISSING]
      n_estimators: 200
      max_depth: 15
      numeric_features: [age]
      categorical_features: [employment, occupation]
    - method: mice
      columns: [race, ethnicity]
      missing_values:
        race: [MISSING]
        ethnicity: [MISSING, PNTA]
      join_tables: [households]
      max_iter: 10
      numeric_features: [age]
random_state: 42
stash_preimputed: true
validate_imputation:
  enabled: true
  n_folds: 5
  sample_pct: 5.0

Example

Each table lists its own rules; a rule names a column, a method (knn, rf or mice), the values to treat as missing, and the features to predict it from.

Example

- name: imputation
  params:
    impute_columns:
      households:
        - column: income_bin
          method: rf
          missing_values: [MISSING, PNTA]
          categorical_features: [residence_type, residence_rent_own]
          aggregate_from:
            persons:
              pivot_count: [employment, education, student]
      persons:
        - column: gender
          method: rf
          missing_values: [MISSING, PNTA]
          join_tables: [households]
          numeric_features: [num_trips]
          categorical_features: [age, employment, education, income_bin]
    stash_preimputed: true
    random_state: 42
    validate_imputation:
      enabled: true
      n_folds: 5
      sample_pct: 5.0

Settings

Key	Type	Default	Meaning
impute_columns	mapping or null	null	Dict mapping table names to list of imputation configs. Every config dict must include a method key (knn, rf, or mice). The remaining keys are method-specific:
stash_preimputed	true / false	true	Whether to create {column}_preimputed columns that preserve the original value before imputation (default: True). The stashed column retains sentinel codes (e.g. PNTA 999, MISSING 995) and nulls so downstream consumers can distinguish the reason a value was imputed. To recover the old boolean flag: col_preimputed.is_null() & col.is_not_null().
random_state	whole number or null	null	Random seed for reproducibility across all imputation.
validate_imputation	mapping or null	null	Optional k-fold check of imputation accuracy against known values.

Feature Importance

When using the Random Forest method, the imputation module automatically reports which features contributed most to the model's predictions. This helps you verify that the model is learning from sensible predictors and catch potential issues like data leakage or uninformative features.

How it works (Mean Decrease in Impurity)

Each tree in a Random Forest repeatedly splits data on the feature that best reduces impurity — Gini impurity for classifiers, variance for regressors. The total impurity reduction attributable to a feature, summed across all splits in all trees and normalised so that importances sum to 1.0, is called Mean Decrease in Impurity (MDI).

MDI is computed for free during model.fit() — no extra passes over the data are required. The imputation module extracts it from model.feature_importances_, maps column indices back to human-readable names, and aggregates one-hot encoded dummies back to their parent categorical feature (e.g. gender=1 + gender=2 → gender).

Interpreting the output

The returned feature_importance dict maps feature names to their normalised importance score. Example:

Feature	Importance
employment	0.312
education	0.241
age	0.189
occupation	0.127
hh_mode_income_bin	0.065
…	…

An importance of 0.31 means ~31% of the model's total impurity reduction came from splits on that feature. Higher values indicate features the model relies on the most.

What to look for:

• Sensible top features — age and employment should matter for income; trip_id should not.
• Low-importance features — candidates for removal to speed up imputation without affecting quality.
• Surprising features — may indicate data leakage (e.g. a downstream field that shouldn't be available at imputation time).

MDI caveats

MDI is biased toward high-cardinality features: a feature with 50 unique values gets more splitting opportunities than one with 2, inflating its apparent importance. The module mitigates this by aggregating one-hot columns back to their parent feature, but the bias can still affect comparisons between features of very different cardinality.

For more robust ranking, permutation importance (shuffle a feature, measure accuracy drop) is planned as a future Tier 2 enhancement gated behind a config flag.

Comparison to statistical significance

Aspect	Statistical significance (p-values)	Feature importance (MDI)
Question answered	"Is this feature's effect distinguishable from zero?"	"How much does this feature contribute to prediction accuracy?"
Sample size sensitivity	Very sensitive — large N makes trivially small effects "significant"	Less sensitive — a useless feature stays near zero regardless of N
Nonlinearity	Only captured if you specify the right model form	Inherently captures nonlinear and interaction effects
Collinearity	Inflates standard errors, masks individual predictors	Spreads importance across correlated features
Threshold	Arbitrary α (usually 0.05)	No hard threshold — it's a ranking, not a binary yes/no

A feature can be statistically significant (p < 0.001) yet have near-zero predictive importance (tiny real effect + large sample). Conversely, a feature can have high importance with no p-value because there's no parametric model to test against. For imputation, importance is the better lens — you want to know "does including this feature actually improve the imputed values?"

Code: processing.imputation