processing.joint_trips

Joint trip processing utilities.

This module provides functionality for detecting shared trips among household members using spatiotemporal similarity matching and graph-based clique detection.

processing.joint_trips.aggregation

Aggregation and validation helpers for joint trip detection.

This module builds the joint_trips table from detected cliques and validates results against reported num_travelers field.

household_date_of

household_date_of(
    members: pl.DataFrame, days: pl.DataFrame, key: str
) -> pl.DataFrame

Return each grouping's travel_date, read from its members' days.

A joint trip or joint tour is shared by several people, so it belongs to the household's date, not to any one member's day. Every member's day must fall on the same date; a grouping whose members sit on two dates, or on a day the days table does not hold, raises rather than picking one.

Parameters:

Name	Type	Description	Default
`members`	`pl.DataFrame`	Member records with key and `day_id`.	required
`days`	`pl.DataFrame`	Days with `day_id` and `travel_date`.	required
`key`	`str`	The grouping id, e.g. `joint_trip_id`.	required

Returns:

Type	Description
`pl.DataFrame`	One row per key with its `travel_date`.

Raises:

Type	Description
`ValueError`	If a grouping's members span more than one travel date, or a member's day is missing from days.

build_joint_trips_table

build_joint_trips_table(
    linked_trips: pl.DataFrame,
    joint_trip_assignments: pl.DataFrame,
    days: pl.DataFrame,
) -> pl.DataFrame

Build the joint_trips linking table from detected groups.

One row per joint_trip_id, recording which household and travel date the group belongs to and how many member trips it holds. Where and when they travelled stays on the member trips: collapsing those into one place or time is a choice of rule, and it is left to whoever needs the answer.

Parameters:

Name	Type	Description	Default
`linked_trips`	`pl.DataFrame`	Original trips DataFrame, carrying person_id	required
`joint_trip_assignments`	`pl.DataFrame`	Output from detect_disjoint_cliques with linked_trip_id and joint_trip_id columns	required
`days`	`pl.DataFrame`	Days with `day_id` and `travel_date`; see :func:`household_date_of`.	required

Returns:

Type	Description
`pl.DataFrame`	DataFrame with joint_trips schema (one row per joint_trip_id)

validate_against_num_travelers

validate_against_num_travelers(
    linked_trips: pl.DataFrame,
    joint_trip_assignments: pl.DataFrame,
    log_discrepancies: bool,
) -> None

Validate detected joint trips against reported num_travelers field.

Compares detected joint trip group sizes with the num_travelers field reported in the survey. Logs INFO-level summary and optional DEBUG-level discrepancies.

Parameters:

Name	Type	Description	Default
`linked_trips`	`pl.DataFrame`	Original trips DataFrame with num_travelers column	required
`joint_trip_assignments`	`pl.DataFrame`	Detected assignments with joint_trip_id	required
`log_discrepancies`	`bool`	If True, log DEBUG-level details for each mismatch	required

processing.joint_trips.clique_detection

Graph-based clique detection for joint trip identification.

This module uses NetworkX to construct weighted similarity graphs and find maximal disjoint cliques representing groups of mutually matching trips. Cliques are ranked by match quality (Mahalanobis distance), temporal overlap, and size to ensure optimal joint trip assignments.

detect_disjoint_cliques

detect_disjoint_cliques(
    filtered_pairs: pl.DataFrame, all_trip_ids: pl.Series
) -> tuple[pl.DataFrame, list[list[int]]]

Detect maximal disjoint cliques with quality-based ranking.

Finds all maximal cliques and resolves conflicts by ranking cliques based on match quality (Mahalanobis distance), temporal overlap, and size. Ensures each trip belongs to exactly one joint trip group.

When trips appear in multiple cliques, the best-quality clique wins. Losing cliques have their edges pruned and are flagged for review.

Parameters:

Name	Type	Description	Default
`filtered_pairs`	`pl.DataFrame`	DataFrame with trip pairs that passed similarity threshold. Must contain linked_trip_id and linked_trip_id_b. Optional: mahalanobis_distance, temporal_overlap_min for weights.	required
`all_trip_ids`	`pl.Series`	Series of all linked_trip_ids to include in output (even those not in any joint trip)	required

Returns:

Type	Description
`pl.DataFrame`	Tuple of:
`list[list[int]]`	DataFrame with columns: linked_trip_id: Trip identifier clique_id: Sequential clique group ID (null for non-joint trips)
`tuple[pl.DataFrame, list[list[int]]]`	List of flagged cliques (conflicts resolved by pruning)

processing.joint_trips.detect_joint_trips

Joint trip detection step for identifying shared household trips.

This module implements the main pipeline step for detecting joint trips where multiple household members travel together.

detect_joint_trips

detect_joint_trips(
    linked_trips: pl.DataFrame,
    households: pl.DataFrame,
    days: pl.DataFrame,
    matching: JointTripConfig,
) -> dict[str, pl.DataFrame]

Detect joint trips among household members using similarity matching.

Identifies trips where multiple household members traveled together by comparing origin-destination-time similarity using either strict buffer thresholds or Mahalanobis distance.

Parameters:

Name	Type	Description	Default
`linked_trips`	`pl.DataFrame`	Journey records with coordinates and timing. Required columns: linked_trip_id, hh_id, person_id, o/d coordinates, depart/arrive times.	required
`households`	`pl.DataFrame`	Household table for pre-filtering.	required
`days`	`pl.DataFrame`	Days with `travel_date`; trips are paired only within one household date, and each joint trip records it.	required
`matching`	`JointTripConfig`	How trips are matched: the method and its thresholds, from the config's `matching:` block. See `JointTripConfig`.	required

Returns:

Type	Description
`dict[str, pl.DataFrame]`	Dictionary containing: - linked_trips: Original trips with added joint_trip_id column - joint_trips: Aggregated table of shared trips with participant lists

Algorithm

Phase 1: Household Pre-filtering

Filter to households with 2+ members who took trips
Reduces search space to only households where joint trips are possible

Phase 2: Pairwise Distance Calculation

Within each multi-person household, compute pairwise distances between all trip combinations using 4D space:
- Origin coordinates (o_lon, o_lat)
- Destination coordinates (d_lon, d_lat)
- Departure time
- Arrival time
Store distances in condensed matrix format for efficiency

Phase 3: Similarity Filtering

Buffer Method:

Filter trip pairs where:
- Spatial distance (haversine) ≤ space_threshold_meters for both origin AND destination
- Absolute time difference ≤ time_threshold_minutes for both departure AND arrival
Simple, interpretable thresholds

Mahalanobis Method:

Calculate statistical distance using covariance matrix:
- Accounts for correlated variations in space/time
- Keeps pairs whose distance is below a cutoff set by confidence_level (see JointTripConfig.get_distance_threshold)
More sophisticated, calibrated to actual joint trip patterns
Can capture joint trips more flexibly than fixed thresholds

Phase 4: Clique Detection

Build graph where nodes = trips, edges = similar trip pairs
Detect maximal cliques (groups of mutually-similar trips)
Handle overlapping cliques by selecting disjoint set with maximum coverage
Each clique represents one joint trip event
Ensures transitivity: if A travels with B, and B with C, then A,B,C form one joint trip

Phase 5: Joint Trip Aggregation

Assign unique joint_trip_id to each clique
Create joint_trips table, one row per joint trip:
- joint_trip_id, hh_id, travel_date
- num_joint_travelers: number of member linked trips
- survey_complete: true only if every member trip is complete
Mark each member linked trip with its joint_trip_id
Validate against reported num_travelers field if available

Notes

Only compares trips within one household date (joint trips across households, or across the survey's day boundary, not detected)
Mahalanobis method requires calibrated covariance matrix (see scripts/calibrate_joint_trip_covariance.py)
Clique detection prevents false positives from partial or coincidental matches
Handles survey reporting errors where respondents over/under-report number of travelers
Non-joint trips retain joint_trip_id = NULL

processing.joint_trips.joint_trip_configs

Configuration models and calibration utilities for joint trip detection.

ALGORITHM DESIGN NOTES:

This module implements joint trip detection using two methods:

BUFFER METHOD (debugging/calibration): Strict AND across four dimensions: origin distance, destination distance, departure time, arrival time. Uses Haversine for spatial matching and absolute time differences. All four must pass thresholds.
MAHALANOBIS METHOD (production): Statistical distance on 4D vector [Δorigin_m, Δdest_m, Δdepart_min, Δarrive_min]. Supports diagonal covariance (assumes independence) or full covariance matrix (captures correlations). More robust to GPS noise via compensation.

CALIBRATION WORKFLOW:

Run detection with method='buffer' to find high-confidence joint trips
Use estimate_covariance_from_detected_pairs() to analyze buffer results
Helper scans threshold range and recommends optimal distance_threshold
Copy diagonal or full covariance to config for production runs
Run detection with method='mahalanobis' using calibrated parameters

TEMPORAL OVERLAP:

Trips must have overlapping time windows to be joint (travelers physically together). Pre-filter: max(depart_A, depart_B) <= min(arrive_A, arrive_B)

JointTripConfig `pydantic-model`

Configuration for joint trip detection.

Attributes:

Name	Type	Description
`method`	`Literal['buffer', 'mahalanobis']`	Detection method ('buffer' for strict thresholds, 'mahalanobis' for statistical distance)
`time_threshold_minutes`	`float`	Maximum time difference for buffer method
`space_threshold_meters`	`float`	Maximum spatial distance for buffer method
`covariance`	`Sequence[float \| int] \| Sequence[Sequence[float \| int]] \| None`	Diagonal variances (list of 4) or full covariance matrix (4x4 list of lists) in units [m^2, m^2, min^2, min^2]. Required for mahalanobis; null for buffer. The empirical variance of BATS 2023 household joint trips is about [7000, 7000, 20, 20]: origin ~84m std (7142 m² var), dest ~84m (7089 m²), depart ~4.7min (22 min²), arrive ~4.6min (21 min²).
`confidence_level`	`float`	Strictness of the mahalanobis cutoff. Higher values keep fewer pairs. It is not the share of joint pairs kept; see `get_distance_threshold`.
`log_discrepancies`	`bool`	If True, log DEBUG-level details when detected joint trip size differs from reported num_travelers field

Config:

extra: forbid

Fields:

method (Literal['buffer', 'mahalanobis'])
time_threshold_minutes (float)
space_threshold_meters (float)
covariance (Sequence[float | int] | Sequence[Sequence[float | int]] | None)
confidence_level (float)
log_discrepancies (bool)

Validators:

validate_covariance → covariance
validate_method_params

model_config `class-attribute` `instance-attribute`

model_config = ConfigDict(extra='forbid')

method `pydantic-field`

method: Literal['buffer', 'mahalanobis']

Detection method: buffer (strict) or mahalanobis (statistical)

time_threshold_minutes `pydantic-field`

time_threshold_minutes: float = 15.0

Maximum time difference in minutes for buffer method

space_threshold_meters `pydantic-field`

space_threshold_meters: float = 100.0

Maximum spatial distance in meters for buffer method

covariance `pydantic-field`

covariance: (
    Sequence[float | int] | Sequence[Sequence[float | int]] | None
)

Diagonal (4 values) or full (4x4) covariance matrix in units [origin_m^2, dest_m^2, depart_min^2, arrive_min^2]. Required for mahalanobis; null for buffer. The empirical variance of BATS 2023 household joint trips is about diagonal [7000, 7000, 20, 20]: origin ~84m std dev (7142 m² var), dest ~84m (7089 m²), depart ~4.7min (22 min²), arrive ~4.6min (21 min²).

confidence_level `pydantic-field`

confidence_level: float

Strictness of the mahalanobis cutoff (mahalanobis method only). Higher = stricter. It reads backward from the usual meaning of confidence and is not the share of joint pairs kept. See get_distance_threshold.

log_discrepancies `pydantic-field`

log_discrepancies: bool = False

If True, log DEBUG-level details when detected joint trip size differs from reported num_travelers

ConfigDict

Pydantic model configuration.

arbitrary_types_allowed `class-attribute` `instance-attribute`

arbitrary_types_allowed = True

validate_covariance `pydantic-validator`

validate_covariance(
    v: Sequence[float] | Sequence[Sequence[float]] | None,
) -> Sequence[float] | Sequence[Sequence[float]] | None

Validate covariance shape and values.

validate_method_params `pydantic-validator`

validate_method_params() -> JointTripConfig

Ensure required parameters are provided for each method.

get_distance_threshold

get_distance_threshold() -> float

Convert confidence_level to the cutoff the mahalanobis filter uses.

Returns chi2.ppf(1 - confidence_level, df=4). Two things make this read backward:

The level is inverted. Usually "90% confidence" would keep 90% of true matches; here a higher level gives a lower cutoff.
apply_mahalanobis_filter compares the cutoff to the distance d, while the chi-squared distribution describes d**2.

If the covariance were exactly the noise between true joint pairs, the share of them kept would be chi2.cdf(cutoff**2, df=4):

confidence_level	cutoff	true matches kept
0.95	0.711	~3%
0.90	1.064	~11%
0.75	1.923	~55%
0.50	3.357	~98%

A covariance wider than that noise loosens the cutoff in real terms. On BATS 2023, covariance [10000, 10000, 100, 100] at 0.90 found 35,021 joint trips against 35,628 from the buffer method (15 min / 100 m), with 98% of matched pairs in common.

Returns:

Type	Description
`float`	The cutoff on the Mahalanobis distance `d`.

estimate_covariance_from_detected_pairs

estimate_covariance_from_detected_pairs(
    joint_trips: pl.DataFrame,
    linked_trips: pl.DataFrame,
    threshold_range: list[float] | None = None,
) -> dict

Estimate covariance from buffer-detected joint trip pairs.

This calibration helper analyzes joint trips detected using the buffer method to compute covariance parameters for the Mahalanobis method. It also scans a range of distance thresholds to recommend optimal values.

Workflow

Run detect_joint_trips with method='buffer'
Pass output to this function
Copy returned diagonal or full covariance to config
Use recommended_threshold for distance_threshold
Run detect_joint_trips with method='mahalanobis'

Parameters:

Name	Type	Description	Default
`joint_trips`	`pl.DataFrame`	Output from detect_joint_trips (joint_trips table)	required
`linked_trips`	`pl.DataFrame`	Original linked_trips DataFrame with coordinates/times	required
`threshold_range`	`list[float] \| None`	List of distance thresholds to scan for recommendation. Defaults to [1.5, 2.0, 2.5, 3.0, 3.5]	`None`

Returns:

Type Description

dict

Dictionary containing: - diagonal: List of 4 variances [origin_m^2, dest_m^2, depart_min^2, arrive_min^2] - full: 4x4 covariance matrix as list of lists - correlation: 4x4 correlation matrix for inspecting relationships - threshold_analysis: DataFrame with columns [threshold, detected_pairs, avg_clique_size, buffer_recall, marginal_increase] - recommended_threshold: Float, optimal threshold based on inflection

Raises:

Type	Description
`ValueError`	If joint_trips or linked_trips missing required columns

processing.joint_trips.similarity

Similarity computation helpers for joint trip detection.

This module provides functions to compute pairwise distances between trips and apply filtering logic for both buffer and Mahalanobis methods.

compute_pairwise_distances

compute_pairwise_distances(
    trip_pairs: pl.DataFrame,
) -> pl.DataFrame

Compute spatial and temporal distances for trip pairs.

Takes self-joined trip pairs and computes: - origin_dist_m: Haversine distance between origins (meters) - dest_dist_m: Haversine distance between destinations (meters) - depart_diff_min: Absolute difference in departure times (minutes) - arrive_diff_min: Absolute difference in arrival times (minutes)

Parameters:

Name	Type	Description	Default
`trip_pairs`	`pl.DataFrame`	DataFrame with columns from self-join: - o_lat, o_lon, d_lat, d_lon (first trip) - o_lat_b, o_lon_b, d_lat_b, d_lon_b (second trip) - depart_time, arrive_time (first trip) - depart_time_b, arrive_time_b (second trip)	required

Returns:

Type	Description
`pl.DataFrame`	Input DataFrame with four additional columns: origin_dist_m, dest_dist_m, depart_diff_min, arrive_diff_min

apply_buffer_filter

apply_buffer_filter(
    trip_pairs: pl.DataFrame,
    space_threshold_meters: float,
    time_threshold_minutes: float,
) -> pl.DataFrame

Apply strict buffer filtering to trip pairs.

Uses strict AND logic: all four dimensions (origin, destination, depart, arrive) must pass their respective thresholds.

Parameters:

Name	Type	Description	Default
`trip_pairs`	`pl.DataFrame`	DataFrame with distance columns from compute_pairwise_distances	required
`space_threshold_meters`	`float`	Maximum spatial distance in meters	required
`time_threshold_minutes`	`float`	Maximum time difference in minutes	required

Returns:

Type	Description
`pl.DataFrame`	Filtered DataFrame containing only pairs passing all four thresholds

apply_mahalanobis_filter

apply_mahalanobis_filter(
    trip_pairs: pl.DataFrame,
    covariance: list[float] | list[list[float]],
    distance_threshold: float,
) -> pl.DataFrame

Apply Mahalanobis distance filtering to trip pairs.

Computes statistical distance on 4D vector [origin_m, dest_m, depart_min, arrive_min] using provided covariance. Supports diagonal (list of 4) or full (4x4 matrix) covariance.

Parameters:

Name	Type	Description	Default
`trip_pairs`	`pl.DataFrame`	DataFrame with distance columns from compute_pairwise_distances	required
`covariance`	`list[float] \| list[list[float]]`	Diagonal variances (4 values) or full covariance (4x4)	required
`distance_threshold`	`float`	Maximum Mahalanobis distance (chi-squared threshold)	required

Returns:

Type	Description
`pl.DataFrame`	Filtered DataFrame containing only pairs below distance threshold

Raises:

Type	Description
`ValueError`	If covariance matrix is singular (condition number > 1000)

processing.joint_trips

processing.joint_trips.aggregation

household_date_of

build_joint_trips_table

validate_against_num_travelers

processing.joint_trips.clique_detection

detect_disjoint_cliques

processing.joint_trips.detect_joint_trips

detect_joint_trips

Phase 1: Household Pre-filtering

Phase 2: Pairwise Distance Calculation

Phase 3: Similarity Filtering

Buffer Method:

Mahalanobis Method:

Phase 4: Clique Detection

Phase 5: Joint Trip Aggregation

processing.joint_trips.joint_trip_configs

ALGORITHM DESIGN NOTES:

CALIBRATION WORKFLOW:

TEMPORAL OVERLAP:

JointTripConfig pydantic-model

model_config class-attribute instance-attribute

method pydantic-field

time_threshold_minutes pydantic-field

space_threshold_meters pydantic-field

covariance pydantic-field

confidence_level pydantic-field

log_discrepancies pydantic-field

ConfigDict

arbitrary_types_allowed class-attribute instance-attribute

validate_covariance pydantic-validator

validate_method_params pydantic-validator

get_distance_threshold

estimate_covariance_from_detected_pairs

processing.joint_trips.similarity

compute_pairwise_distances

apply_buffer_filter

apply_mahalanobis_filter

JointTripConfig `pydantic-model`

model_config `class-attribute` `instance-attribute`

method `pydantic-field`

time_threshold_minutes `pydantic-field`

space_threshold_meters `pydantic-field`

covariance `pydantic-field`

confidence_level `pydantic-field`

log_discrepancies `pydantic-field`

arbitrary_types_allowed `class-attribute` `instance-attribute`

validate_covariance `pydantic-validator`

validate_method_params `pydantic-validator`