Kuriko Iwai ·

How to Solve Time-Varying Confounding & Mediator Bias in Longitudinal ML Pipelines

[Series] Causal Machine Learning - 3. De-Biasing Sequential Telemetry and Multi-Period Decisions

Time-varying trap (or time-varying confounding) indicates a complex scenario in sequential, multi-period data where a variable measured over time acts simultaneously as a mediator and a confounder.

Below diagram shows how system states evolve over a multi-period horizon:

Figure A. Longitudinal Directed Acyclic Graph (DAG) showing time-varying confounding and telemetry dual-roles in enterprise employee retention (Created by Kuriko IWAI)

Let us consider an enterprise employee retention pipeline across two time steps.

The figure include key components:

• Confounder X: The employee's background and professional experience.

• Initial intervention A_0: Day1 onboarding program assignment.

• Month-1 telemetry M_1: Engagement of the employee (M_1 = 1 indicates high engagement).

• Month-1 intervention A_1: Manager's check-ins in Month 1.

• Final outcome Y: Retention of the employee at the end of temporal horizon (i.e., T = 12 months).

◼ The Dual-Role of Telemetry - Mediator AND Confounder

Here, M_1 is a mediator because the initial intervention improves it.

For instance, if an employee had good onboarding experience, they'd be likely to engage more (A_0 = 1 → M_1 = 1).

But M_1 is also a confounder because a sudden drop in engagement could trigger A_1 (manager's check-ins) and directly lower retention Y.

M_1 is directly impacted by the past interventions, but also impacts on future interventions, and hence, the final outcome.

Because M_1 plays both roles over time, static feature-flattening models fail to isolate the true causal effect of this continuous sequences.

In static ML, we only worry about baseline confounder X.

But in sequential systems we saw in the previous section, past interventions pollute future states.

This makes static ML face a structural dilemma.

◼ The True Causal Effect

Imagine we have 1,000 software engineers in our staffing pipeline.

When we split them into two groups:

1. Group 1. (500 engineers) Receives intensive onboarding (A_0 = 1), and

2. Group 2. (500 engineers) Receives standard onboarding (A_0 = 0).

Below diagram demonstrates how the true causal effect is measured for retention at temporal horizon T:

Figure B-1. True total causal effect of multi-period interventions on long-term retention (Created by Kuriko IWAI)

Among 500 engineers under the intensive onboarding (upper row, Figure B-1), 400 engineers had high engagement at Month 1, and 320 engineers were retained.

The remaining 100 developers had low engagement, but 40 engineers were retained.

So, in total, the intensive onboarding achieved 72% of retention rate; compared to 38% of standard onboarding (bottom row, Figure B-1), boosting the retention rate by 34 points.

◼ Mediator-blocking Bias

Native ML models control for M_1 by including M_1 in the feature set.

However, this accidentally blocks the indirect causal path of the initial intervention A_0:

Figure B-2. Mediator-blocking bias misestimates causal effect due to covariate over-adjustment (Created by Kuriko IWAI).

Figure B-2 demonstrates mediator-blocking bias (or over-adjustment bias) where when an ML model conditioned on M_1, it completely misses the benefit of intensive onboarding (A_0 = 1) which drives 80% of developers into high engagement (M_1 = 1), instead of just 20%.

▫ Mathematical Background of Mediator-blocking Bias

Suppose our data science team attempts to evaluate the retention strategy by fitting a standard multi-variable regression model:

where β's represent weights assigned to each variable and k represents temporal horizon such as k=12.

Eq. 1 includes M_1 because we correctly identify that M_1 is a confounder for A_1.

This, however, blocks the indirect causal path from A_0 to Y, dampening weight β_0 which measures the Controlled Direct Effect (CDE) of A_0 on the final outcome.

On the other hand, the model attributes credit for M_1, even though A_0 actually caused M_1, assigning a large value to β_1.

Stakeholders looking at large β_1 might say: "Month 1 engagement (M_1) drives employee retention. Let's focus on engagement.", missing the fact that M_1 was a result of the onboarding program (A_0).

Table 1. Ground Truth vs. Naive ML Conditioning on Time-Varying Mediators (Created by Kuriko IWAI)

◼ Selection Bias

When we exclude M_1 from the feature set, the estimation of the secondary intervention A_1 suffers from uncontrolled baseline confounding.

Selection bias occurs in two situations:

1. Confounder selection bias: Failing to control for baseline X when assigning onboarding, so the model analyzes non-randomly selected samples.

2. Collider stratification bias: Conditioning on a variable that creates a fake correlation between two independent variables; for example, evaluating manager interventions (A_1) by filtering the dataset strictly to low-engagement developers (M_1 = 0).

◼ Confounder Selection Bias

Suppose Senior Engineers (X = 1) inherently retain at higher rates (80%), but managers preferentially send Junior Engineers (X = 0) to the intensive onboarding program (A_0 = 1) because they need more help:

• Senior Engineers (X = 1, 500 total): Naturally retained at 80%. Only 100 were assigned to the intensive program.

• Junior Engineers (X = 0, 500 total): Naturally retained at 40%. 400 were assigned to the intensive program.

Below diagram demonstrates confounder selection bias:

Figure C-1. Confounder selection bias resulting from non-random treatment assignment across baseline demographics (Created by Kuriko IWAI)

When the analyst simply compares retention rates after the intensive and standard onboarding program, standard has much higher rate (72%, right, Figure C-1) than intensive (24%, middle, Figure C-1).

But the truth is that the sample receiving the intensive program was heavily biased toward Junior Engineers who naturally leave at higher rates.

Here, intensive onboarding wasn't failing at all—it was handed a fundamentally harder population: Junior Engineers.

◼ Collider Stratification Bias

Collider stratification bias is a subtle form of selection bias where conditioning on a middle variable (M_1) artificially creates an inverted correlation between two unlinked factors.

Suppose two unlinked factors cause low engagement (M_1 = 0):

1. Standard onboarding (A_0 = 0), and

2. Personal burnout / external hardship (U = 1).

Suppose leadership asks:

By restricting the analysis strictly to low-engaging engineers (M_1 = 0), the model confuses that the intensive onboarding causes low engagement.

But in reality, these engineers had low engagement not because of poor onboarding, but because of personal burnout (U = 1, left, Figure C-2).

Figure C-2. Collider stratification bias induced by conditioning on intermediate telemetry affected by unobserved confounders (Created by Kuriko IWAI).

Here, M_1 is a collider because two separate factors (A_0 and U) point directly into it.

And U represents factors like personal family emergencies, mental fatigue, bad team dynamics, or sudden external hardships.

In real-world enterprise telemetry, U is rarely logged or quantifiable.

Because U remains unobserved and hence, omitted from the regression model, filtering on M_1 = 0 forces A_0 to act as a proxy of U.

So, how can we know if we have recorded enough history to calculate true cause-and-effect?

Sequential exchangeability tackles these challenges by structurally completing the causal pipeline, extending Judea Pearl's backdoor criterion across time t:

where \bar{H}_t represents historical trajectory up to time step t:

where \bar{A}_{t-1} and \bar{M}_t represent a complete sequence of past interventions up to step t-1 and post-treatment intermediate states up to step t, respectively.

In this concept, the decision to assign intervention A_t is independent of the potential counterfactual outcome Y(\bar{a}), indicating that there's no unobserved confounders affecting intervention A_t at any stage of the pipeline, once historical trajectory is defined.

◼ How Sequential Exchangeability Solves Both Biases

Sequential exchangeability is the theoretical backbone that powers Inverse Probability of Treatment Weighting (IPTW) and Marginal Structural Models (MSMs).

▫ Eliminating Selection Bias

First, it eliminates selection bias at every step.

IPTW calculates time-varying weights:

These weights construct a synthetic pseudo-population where intervention A_t is pseudo-randomized relative to past history.

This can strip away the selection bias because A_t is no longer determined by past history \bar{H}_t, breaking the confounding links that caused the bias in the first place.

▫ Eliminating Mediator-blocking bias

Instead of putting M_1 directly into the regression model, sequential exchangeability uses M_1 only to construct the propensity weights W_t.

For a two-stage longitudinal pipeline with initial intervention A_0, intermediate state M_1, and secondary treatment A_1, the cumulative unstabilized weight W is the product of the inverse propensity scores at each time step:

where:

• Weight at time t=0 (W_0) for individual i inversely models the probability of receiving initial intervention A_0 given baseline confounder X, and

• Weight at time t=1 (W_1) for individual i inversely models the probability of receiving secondary treatment A_1 given the full prior trajectory \bar{H}_1 = {X, A_0, M_1}.

This unstabilized weight can produce extreme values (large variance) if some intervention probabilities are very small.

In practice, we use Stabilized Weights (SW), which replace the numerator 1 in Eq. 3.2 with the probability of the current intervention conditioned only on past intervention history:

where A_0 and A_1 represent interventions at t = 0 and t = 1 respectively.

Eq 3.3.1 can be generalized with T time steps:

Once SW_i (Eq. 3.3.2) is calculated for every individual, we can fit a weighted regression model for final outcome Y using only the intervention history variables (A_0, A_1), completely excluding M_1:

Because M_1 was used to construct SW_i (included in the past history), the sample is reweighted to break the confounding links, while leaving the mediator path (A_0 -> M_1 -> Y) open.

Now, β_1 successfully captures the true total causal effect (+34pt) of the intensive onboarding program.

Also, the model does not stratify on post-treatment status, preventing A_0 from becoming a proxy for unobserved factors (U).

This section demonstrates how an outcome changes when evaluating a continuous policy sequence versus evaluating single-step isolated decisions.

◼ Model Selection

I'll select a following models:

1. Standard Regression

• Serves as the baseline benchmark to demonstrate traditional model failure modes.

• Shows what happens when we either ignore confounders (inducing baseline selection bias) or include intermediate variables directly in the feature space.

2. Static IPS

• Corrects for single-point baseline confounding at time t=0 (X -> A_0).

• Re-weights the population so intervention and control groups look balanced at baseline.

3. Long Short-Term Memory (LSTM)

• Acts as a flexible, non-linear propensity estimator for complex, high-dimensional longitudinal histories.

• Captures non-linear dependencies, sequential patterns, and long-term history across time steps without requiring manual feature engineering of historical trajectories.

4. MSM

• Represents the gold-standard causal model for longitudinal data with time-varying confounding and mediation.

• Uses intermediate variables strictly to build sequential IPTW weights while leaving M_1 out of the final outcome regression.

◼ Interventional Policy Evaluation in Action

Now, let us compare the four models.

First, I'll define the synthetic data:

1import torch
2import numpy as np
3import pandas as pd
4
5# set seed for reproducibility
6np.random.seed(42)
7torch.manual_seed(42)
8
9N = 2000
10
11# X: experience level (1: senior, 0: junior)
12X = np.random.binomial(1, 0.5, N)
13
14# A_0: onboarding (Juniors targeted more heavily)
15p_A0 = 1 / (1 + np.exp(-(-0.8 * X + 0.2)))
16A0 = np.random.binomial(1, p_A0)
17
18# U: unobserved elements (personal burnout)
19U = np.random.binomial(1, 0.3, N)
20
21# M1: month-1 engagement driven by A_0 and U
22p_M1 = 1 / (1 + np.exp(-(1.2 * A0 - 2.0 * U + 0.5)))
23M1 = np.random.binomial(1, p_M1)
24
25# A1: secondary check-in (target heavly for low engagement M_1=0)
26p_A1 = 1 / (1 + np.exp(-(-1.5 * M1 + 0.5 * X)))
27A1 = np.random.binomial(1, p_A1)
28
29# Y: final retention outcome
30y_logit = -0.5 + 0.8 * A0 + 1.2 * M1 + 0.6 * A1 + 1.0 * X - 2.5 * U
31p_Y = 1 / (1 + np.exp(-y_logit))
32Y = np.random.binomial(1, p_Y)
33
34# create a pandas df
35df = pd.DataFrame({"X": X, "A0": A0, "M1": M1, "A1": A1, "Y": Y})
36

Next, I'll define and train the models:

1import numpy as np
2import pandas as pd
3import statsmodels.api as sm
4import statsmodels.formula.api as smf
5import torch
6import torch.nn as nn
7
8# 1. standard regression
9model_standard = smf.logit("Y ~ A0 + M1 + A1 + X", data=df).fit(disp=0)
10
11
12# 2. static IPS
13ps_model_a0 = smf.logit("A0 ~ X", data=df).fit(disp=0)
14df["ps_a0"] = ps_model_a0.predict(df)
15df["w_static"] = np.where(df["A0"] == 1, 1 / df["ps_a0"], 1 / (1 - df["ps_a0"]))
16
17# fit outcome model with static weights (excluding M1)
18model_static_ips = smf.wls(
19    "Y ~ A0 + A1", data=df, weights=df["w_static"]
20).fit()
21
22
23# 3. lmst
24class LongitudinalPropensityLSTM(nn.Module):
25
26    def __init__(self, input_dim, hidden_dim):
27        super().__init__()
28        self.lstm = nn.LSTM(input_dim, hidden_dim, batch_first=True)
29        self.fc = nn.Linear(hidden_dim, 1)
30
31    def forward(self, x):
32        lstm_out, _ = self.lstm(x)
33        # Output probability for treatment at each time step
34        probs = torch.sigmoid(self.fc(lstm_out))
35        return probs
36
37# format time steps: [t0_features (X), t1_features (X, A0, M1)]
38X_t0 = np.column_stack([df["X"], np.zeros(N), np.zeros(N)])
39X_t1 = np.column_stack([df["X"], df["A0"], df["M1"]])
40seq_data = torch.tensor(
41    np.stack([X_t0, X_t1], axis=1), dtype=torch.float32
42)  # shape: (N, 2, 3)
43
44lstm_model = LongitudinalPropensityLSTM(input_dim=3, hidden_dim=8)
45predicted_probs = lstm_model(seq_data).squeeze().detach().numpy()
46
47
48# 4. msm
49# numerators (P(A_t | A_{t-1}))
50num_a0 = smf.logit("A0 ~ 1", data=df).fit(disp=0).predict(df)
51num_a1 = smf.logit("A1 ~ A0", data=df).fit(disp=0).predict(df)
52
53p_num_a0 = np.where(df["A0"] == 1, num_a0, 1 - num_a0)
54p_num_a1 = np.where(df["A1"] == 1, num_a1, 1 - num_a1)
55
56# denominators (P(A_t | History_t))
57den_a0 = ps_model_a0.predict(df)
58den_a1 = smf.logit("A1 ~ A0 + M1 + X", data=df).fit(disp=0).predict(df)
59
60p_den_a0 = np.where(df["A0"] == 1, den_a0, 1 - den_a0)
61p_den_a1 = np.where(df["A1"] == 1, den_a1, 1 - den_a1)
62
63# cumulative stabilized weights
64df["sw"] = (p_num_a0 / p_den_a0) * (p_num_a1 / p_den_a1)
65
66# fit msm on pseudo-population
67msm = smf.wls("Y ~ A0 + A1", data=df, weights=df["sw"]).fit()
68Results
69

◼ Results

I estimated the initial onboarding and first manager check-in effects (β^​A_0​​ and β^​A_1​​), and measured biases from the ground truth.

The below table compares the results:

Table 2. Numerical Causal Model Validation and Bias Comparison Matrix (Created by Kuriko IWAI)

▫ Standard Regression

Standard regression suffers mediator blocking bias (-0.58) as well as collider stratification (-0.48).

Conditioning on M_1 in the feature space forces A_1 to act as a proxy on unobserved elements (U).

▫ Static IPS

Static IPS successfully re-weights A_0, making the bias near zero, but causes severe bias on A_1 (-0.42) because it omits M_1 from t = 1 propensity scores.

▫ LSTM

Without causal re-weighting, LSTM relies on conditional probability P(Y | A, H), underestimating early actions (-0.45). Then, it over-inflates the manager's check-ins (+0.28), confusing high managerial focus on risk profiles for treatment efficacy.

▫ MSM

MSM successfully recovers both ground truth targets.

MSM can re-weights the observational trajectory using sequential propensity scores.

It uses intermediate states (M_1) strictly to construct time-varying weights, then removes it from the final outcome model.

Here is the summary of key production health check metrics:

Table 3. Production Health Check and Operational Risk Matrix for Longitudinal Causal Models (Created by Kuriko IWAI)

Understanding dual-role variables and sequential exchangeability provides the structural framework for multi-period causal modeling.

But mathematical graph theory is useless if your underlying database telemetry is a chaotic mess.

In Part 4, we will step out of the math and into the data engineering trenches to build a timestamp architecture that enforces chronological ordering in production.

Fixing Selection Bias in Enterprise Data with Inverse Propensity Scoring (IPS)

[Series] Causal Machine Learning - 2. De-Biasing Historical Data Logs (without Expensive A/B Tests)

The absolute cleanest way to solve our project routing dilemma is a Randomized Controlled Trial (RCT).

An RCT is a scientific experiment where baseline biases are eliminated by design using a perfectly random allocation mechanism.

◼ Core Mechanism

Below diagram illustrates its mechanism:

Figure B. Workflow diagram of a Randomized Control Trial (RCT) showing candidate pool randomization into balanced treatment and control groups (Created by Kuriko IWAI).

In an RCT, a pool of subjects is split into a treatment group (A_0 = 1) and a control group (A_0 = 0), following:

• Control Group: A baseline group that receives no intervention (e.g., keeping developers on standard maintenance).

• Randomization: Allocation is completely randomized. All potential baseline biases—whether they stem from a candidate's university pedigree, years of experience, or technical stack—are distributed evenly across both groups.

• Blinding: To further mitigate psychological or observer biases, subjects (single-blind) or both subjects and researchers (double-blind) are kept unaware of group assignments.

By balancing these baseline traits perfectly, RCTs provide the surest means of achieving two critical benchmarks of research quality:

• Internal Validity: Ensuring that the relationships found within the study data reflect true, unconfounded causal relationships in the target population.

• External Validity: Ensuring that the results calculated by the model can be confidently applied to similar populations in different real-world settings.

◼ Applied to the Use Case: The Live Hiring RCT

If we ran a pristine RCT in our engineering org, we would take a fresh pool of 1,000 incoming developers.

Regardless of whether they went to an Ivy League university or a self-paced coding bootcamp, we would flip a coin:

• Heads (A_0 = 1) → Sent directly to a Tier-1 Cloud Architecture project.

• Tails (A_0 = 0) → Sent directly to standard maintenance.

Because both groups are structurally identical at day zero, any variance in our final 1-year retention outcome (Y) could be confidently attributed to the project assignment (A_0) alone.

While the live hiring RCT yields perfectly clean data, running it in a production environment hits an immediate operational wall.

◼ Why Live Hiring Experiments Fail Operationally

There are two main reasons that the experiment fails in production:

1. Severe operational risk and

2. Ethical constraints.

▫ 1. Severe Operational Risk and Costs

Forcing a randomized assignment policy is hazardous.

If we randomly assign an under-qualified team to a critical tier-1 infrastructure project just to gather unconfounded data, we'd risk catastrophic downtime and lost revenues.

▫ 2. Ethical Constraints

It can be highly disruptive to intentionally strip career-advancing, beneficial treatments like premier projects away from a control group simply to satisfy an algorithmic requirement.

◼ Alternative Analytical Studies

Because of these operational frictions, we must pivot to alternative study designs.

The below diagram shows the breakdown of analytical study designs:

Figure C. Taxonomy chart of analytical study designs categorizing experimental variants and observational frameworks like cohort, case-control, and cross-sectional studies.

▫ A. Experimental Variants

• Non-Randomized Controlled Trials: We actively manipulate the assignment policy, but rely on non-random criteria (like team tenure or hiring dates) to form groups.

  • Use Case Example: We roll out an automated Tier-1 project routing pilot exclusively to our Tokyo office, while keeping our New York office on legacy manager routing.

  • The Causal Flaw: Without a random coin flip, the baseline traits ($X$) of the Tokyo engineering culture differ systemically from New York, leaving the door open for hidden regional confounders.

▫ B. Observational Frameworks

When we cannot manipulate project assignments at all, groups form naturally in historical operational logs.

• Cohort Studies: 

  • Isolates a group of developers right at their hire date, record their initial baseline context, and actively follow them forward over a 12-month period.

  • Use Case Example: Tracking developers as they naturally land Tier-1 projects or standard maintenance based on legacy human manager preferences.

• Case-Control Studies: 

  • Instead of waiting 12 months, identifies two terminal groups: the “Cases” (developers who left the company within 1 year) and the “Controls” (developers who stayed). We then analyze their past logs to check what project exposures and baseline traits drove the split.

  • Use Case Example: This is a massive time-saver for studying rare infrastructure failures or extreme corporate attrition, though we must heavily adjust for survivorship bias.

• Cross-Sectional Studies (Snapshots): 

  • Measures exposure and outcomes at the exact same time.

  • Use Case Example: Pulling a data dump of our entire global directory today to see which developers are currently on Tier-1 projects and matching that to their current Q3 happiness score. It provides a quick operational snapshot but cannot establish a clear chronological sequence of events.

▫ Comparing RCTs and Observational Alternatives

To see exactly how these choices impact our causal pipeline, let us map our project routing challenge across these methodologies:

Table 1. Methodological Trade-offs: Randomized Experiments vs. Observational Cohorts

Because running RCT in our use case is operationally too risky, we must debias our historical data logs synthetically.

Inverse Propensity Scoring (IPS) is a causal inference framework designed to estimate the true causal effect of an action when running a live RCT is impractical.

The below diagram illustrates its core concept:

Figure D. Distribution graph demonstrating how Inverse Propensity Scoring (IPS) re-weights skewed observational data into a balanced synthetic sample (Created by Kuriko IWAI)

Figure D shows that IPS can synthetically adjust the balances by re-weighting the original population with selection biases.

In our use case, when an Ivy League graduate (red circles, Figure D) had a 95% chance of being selected for a Tier-1 project based on their pedigree, their data point is down-weighted (IPW = 1.05 < 20).

Conversely, if an equally capable engineer from a non-traditional background (orange circle, Figure D) was selected for a Tier-1 project against all historical odds (a 5% probability in reality), their data point is up-weighted by a factor of 20 (IPW = 20).

This synthetic adjustment balances the distribution of baseline confounder X to look statistically identical across both groups.

◼ The Exchangeability of IPS

The exchangeability of IPS indicates its ability to transform a biased historical log into a clean, pseudo population where treatment and control groups can be fairly compared without risking live revenues.

This is critical to causal inference.

Causal inference attempts to answer prescriptive questions such as:

"What will happen to our retention rate if we change our active policy and assign this specific cohort to a Tier-1 project?",

But in reality, causal inference faces counterfactuals where it cannot observe what would happen when Engineer A were assigned to the control group (A_0 = 0), and vice versa.

So, without IPS, causal inference collapses because confounding (high-performing profile) simultaneously causes a higher likelihood of being assigned to the treatment group (A_0 = 1), and can naturally exhibit higher retention (Y).

1                  [Confounder: Engineer with Pedigree (X)]
2                                 /          \
3                                /            \
4                               v              v
5[Project Assignment (A0)] ----------> [1-Year Retention (Y)]
6

Figure E. Causal loop diagram showing baseline confounder X directly influencing both the treatment action A_0 and the retention outcome Y, resulting in entangled correlation and causation (Created by Kuriko IWAI)

IPS can mathematically strip away this dilemma.

To understand how IPS achieves synthetic randomization, let us look at the underlying mathematical framework.

◼ Core Variables

IPS has three key variables:

• X: Baseline confounder. A vector of observed baseline confounders. In the hiring context, X can be a jobseeker's professional background.

• A_0 ∈ {0, 1}: An intervention. The binary treatment indicator represents the primary initial action. For instance, A_0 = 1 if assigned to a complex Tier-1 project, A_0 = 0 if assigned to standard maintenance.

• Y: The target outcome. For instance, retention status or promotion after 1 year. Y(1), Y(0 indicate the potential outcomes (counterfactuals) if the jobseeker were un/treated by the intervention A_0 (e.g., assigned a complex project), respectively.

◼ Propensity Score

The propensity score e(X) is the conditional probability of receiving the treatment given the baseline confounder X:

where A_0 = 1 represents that an individual (jobseeker) is assigned to an intervention A_0.

In an enterprise ML pipeline, this score is typically estimated using a calibrated binary classifier such as Logistic Regression or LightGBM trained on historical logs to predict A_0 from X.

◼ Estimating IPS

To isolate the true structural impact of our action, the Average Treatment Effect (ATE) can measure the average total causal impact of the policy change across the entire population.

Mathematically, ATE is denoted:

Here, E[Y(1)] is the expected value of the outcome if every single individual in the population had been forced into the treatment group (A_0 = 1).

In the hiring context, this represents a situation where all jobseekers (employees) are assigned to the competitive project A_0 = 1 for instance.

On the other hand, E[Y(0)] is the expected value of the outcome when A_0 = 0, meaning all jobseekers are assigned to a simple, baseline project.

Because we can never simultaneously observe both Y(1) and Y(0) for a single individual, the IPS estimator allows us to calculate these unobserved counterfactual expectations using only observed data by applying Horvitz-Thompson inverse weights:

Where:

• A_0 ∈ {0, 1} is the observed treatment assignment for a given individual.

• Y is the actual observed outcome.

• e(X) is the estimated propensity score for that individual.

• A_0⋅Y / e(X) zero-out anyone who did not receive the treatment (A_0 = 0) and dynamically amplifies the outcomes of individuals who received the treatment despite a low historical probability.

◼ Computing Causal Weight

To operationalize this, the IPS pipeline computes an individual causal weight (w_i) for every single data point in our cohort:

For example:

For Engineer A: Ivy League Grad, 5 YOE

• Because managers actively favor this profile for top tasks, their estimated propensity score is highly skewed: e(X_A) = 0.95.

• They were assigned a Tier-1 project (A_A = 1).

For Engineer B: Non-traditional background, 1 YOE

• Because managers usually bypass this profile, their propensity score is low: e(X_B) = 0.05.

• Due to an emergency staff shortage, they were assigned a Tier-1 project anyway (A_B = 1).

Eq. 1.4 proves that the causal weight for Engineer A remains almost one:

while Engineer B's data point is scaled up by a factor of 20:

As Figure D shows, this re-weighting can evenly distribute population to the treatment and control group in the pseudo-population.

▫ Defining Selection Bias in Math

In the original historical logs, the conditional probability of seeing a specific profile X with a specific project assignment A_0 is governed by the standard product rule of probability:

Because human managers prefer certain backgrounds, the assignment probability P(X, A_0) changes drastically depending on X.

As we saw in the previous example,

• For Engineer A, P(A_0 = 1 | X) = 0.95.

• For Engineer B, P(A_0 = 1 | X) = 0.05.

Selection bias indicates that the confounder X and treatments A_0 are statistically dependent.

▫ Mitigating Selection Bias

Securing statistic independence between the confounder X and treatments A_0 can mitigate selection biases.

Let's look closely at how Eq 1.4 treats a person who actually received the treatment (A_i = 1).

For anyone in the treatment group, the second half of Eq 1.4 drops out because (1 - A_i) = 0.

Their individual weight simplifies to:

Now, let's observe what happens when we create a pseudo-population by multiplying our original observational probability distribution P(X, A_0=1) by this exact mathematical weight:

Substitute the definitions from Eq 2.1 and Eq 1.4 into this equation:

▫

By running the exact same math for the control group (A_0 = 0) and combine both branches back together, the new, synthetically weighted joint distribution of the dataset simplifies to:

Eq. 2.3 forces X and A_0 to become independent; the baseline features balance perfectly across both treatment branches.

This converts the heavily skewed observational logs into a clean synthetic sandbox.

Let’s implement this de-biasing pipeline in Python.

We will simulate a confounded corporate dataset where legacy managers over-indexed on an employee’s technical background (X) when assigning them to Tier-1 projects (T).

We will showcase how a standard model overestimates the impact of the project assignment, and how our IPS pipeline recovers the ground truth.

1import numpy as np
2import pandas as pd
3import sklearn.linear_model as lm
4import statsmodels.api as sm
5import statsmodels.formula.api as smf
6
7# simulate a dataset with a confounder (X), treatment assignment (A_0), and outcome (Y)
8np.random.seed(42)
9N = 12000
10
11X = np.random.normal(0, 1, N)
12
13prob_A = 1 / (1 + np.exp(- (1.8 * X)))
14A_0 = np.random.binomial(1, prob_A)
15
16
17true_beta_A0 = 0.6
18prob_Y = 1 / (1 + np.exp(- (-1.0 + true_beta_A0 * A_0 + 1.2 * X)))
19Y = np.random.binomial(1, prob_Y)
20
21df = pd.DataFrame({'X': X, 'A_0': A_0, 'Y': Y})
22
23# native estimation of the effect of A_0 on Y without any adjustment for confounding
24naive_model = smf.logit("Y ~ A_0", data=df).fit(disp=0)
25naive_beta = naive_model.params['A_0']
26
27# ips pipeline - estimating propensity scores using logistic regression
28propensity_model = lm.LogisticRegression(penalty=None)
29propensity_model.fit(df[['X']], df['A_0'])
30df['propensity_score'] = propensity_model.predict_proba(df[['X']])[:, 1]
31
32# check positivity violation: propensity scores should be bounded away from 0 and 1
33min_score, max_score = df['propensity_score'].min(), df['propensity_score'].max()
34print(f"Propensity Score Range: [{min_score:.4f}, {max_score:.4f}]")
35
36# compute ipw
37df['weight'] = df['A_0'] / df['propensity_score'] + (1 - df['A_0']) / (1 - df['propensity_score'])
38df['id'] = df.index
39
40# fit a GEE model to estimate the causal effect of A_0 on Y using the computed weights
41ips_model = smf.gee(
42    "Y ~ A_0", 
43    groups="id", 
44    data=df, 
45    family=sm.families.Binomial(link=sm.families.links.Logit()), 
46    weights=df['weight']
47).fit()
48ips_beta = ips_model.params['A_0']
49
50

◼ Results

▫ Propensity Score Range: [0.0011, 0.9990]

The score range shows the lowest and highest probabilities the model calculated for a developer getting assigned to a Tier-1 project.

The range is incredibly extreme from almost zero to 1; a score of 0.0011 means a developer had a 0.11% chance of getting a Tier-1 project but got it anyway, while 0.9990 means they had a 99.9% chance.

This tells us that management's selection bias was extremely severe, creating a massive imbalance in who got selected for top projects.

▫ Intervention (treatment assignment) A_0

1. The ground truth: A_0 = 0.60

This indicates the actual, unconfounded impact of a Tier-1 project assignment to 1-year retention Y.

In a perfect world like a clean RCT experiment, giving a developer a Tier-1 project increases their baseline retention metrics by exactly 0.60.

This is the benchmark number the causal model is trying to find.

2. Naive observation by standard ML (logistic regression): A_0 = 1.6974

This is what a standard ML model sees if we just feed it raw historical data without correcting any biases.

The number is massively inflated (1.6947 vs the true 0.60).

Because managers historically gave Tier-1 projects to the highest-performing engineers, a standard model incorrectly credits the project for high retention, when in reality, those elite engineers were already highly likely to stay regardless of their assignment.

3. IPS re-weighted causal inference: A_0 = 0.4415

This is the calculation from the corrected causal inference pipeline after using IPS to strip away historical management bias.

It successfully broke the bias, moving drastically closer to the real-world truth of 0.60.

Note: It is slightly lower than 0.60 because of the extreme probability weights we saw in step 1, which introduce statistical noise, and because of how math behaves when dealing with log-odds binary outcomes.

While IPS is mathematically competitive, its real-world deployment faces a roadblock: the positivity violation.

Formally, the IPS math requires to secure non-zero probabilities

for every unique combination of baseline confounders X.

This indicates that all individuals in the original population must land in either the treatment or control group.

▫ Hitting the Wall: Strict Corporate Rules

But is it realistic?

Imagine our engineering organization in the use case enforces a rigid corporate rule:

When the propensity model evaluates this junior subpopulation X_{junior}, the historical probability of treatment drops to absolute zero:

due to the new rule.

So, when the IPS pipeline attempts to calculate the counterfactual expectation for the treatment group, the denominator in the formula collapses:

making it impossible to run causal estimation.

As Eq. 3.3 shows, when a subpopulation has a probability of exactly 0 or 1, IPS fails completely.

In other words, no optimization and tuning efforts can recover a causal signal from an environment where alternative actions were never attempted.

One cannot simulate the strategy because we have zero historical data points to balance.

◼ How to Overcome the Positivity Violation in Production

When hitting a deterministic corporate policy, we'd deploy two primary mitigation strategies to save the pipeline:

▫ Option 1. Trimming and Restricting the Causal Manifold

The most pragmatic engineering solution is to redefine the scope of your causal inquiry.

If e(X_{junior}) = 0, we must explicitly remove junior engineers from the model's dataset entirely - and include only mid-level and senior engineers where true managerial discretion exists (0 < e(X) < 1).

This allows the model to regain mathematical stability, but we sacrifice scope.

"We can optimize routing for experienced engineers, but we cannot mathematically evaluate counterfactuals for junior engineers because alternative choices were never attempted."

▫ Option 2. Utilizing Dynamic Overlap via System Shocks (Natural Experiment)

If historical exceptions exist that break the rigid corporate rule, we can use them as counterfactual data whose e(X) > 0 is secured.

For example, a massive cloud outage occurred last quarter forced managers to temporarily pull junior engineers onto Tier-1 response teams.

This can be used to calculate valid weights for junior engineers.

By applying IPS, we successfully forced our historical log data to simulate an unbiased, randomized experiment.

We decouple an engineer's entry-level background from the true, isolated impact of their project placement.

But this brings us to the ultimate caveat of IPS: IPS is a static snapshot game.

◼ The Time Varying Trap

IPS assumes all baseline traits X are measured at day one, and the intervention A_0 is a single, permanent choice.

But in reality, corporate environments are continuous, evolving timelines.

Consider this common production feedback loop:

1. A junior engineer is hired.

2. At Month 3, they perform exceptionally well on a small task (time-varying confounder).

3. Because of this success, the manager overrides standard policy and promotes them early to a Tier-1 team (time-varying treatment).

4. Their performance at Month 6 dictates their final 1-year retention Y = 1.

When past treatments affect future confounders, standard static IPS is completely blindfolded.

To solve this challenge, I'll explore Time-Varying Confounding and the Descent into G-Estimation in Part 3.

◼ Causal Machine Learning Series

• Part 1. Why Machine Learning Demands Causal Inference - Rethinking the Predictive Paradigm

• Part 2. De-Biasing Historical Data Logs (without Expensive A/B Tests)

Why Machine Learning Demands Causal Inference

[Series] Causal Machine Learning - 1. Rethinking the Predictive Paradigm

Causal inference is the formal statistical and mathematical framework dedicated to uncovering, quantifying, and predicting the independent, directional effects of specific variables (treatments) on targeted outcomes.

The below diagram illustrates how causal inference works, compared to standard, predictive machine learning:

Figure 1. Symmetrical Statistical Associations in Predictive ML vs. Directional Causal Inference Mechanics

Rather than treating data as a collection of symmetrical statistical associations (Left, Figure 1), causal inference models the underlying physical mechanisms and data-generating processes of the system (Right, Figure 1).

◼ Structural Causal Model (SCM) Components

To differentiate standard ML (correlation) from causal inference, we define a chronologically ordered directed acyclic graph (DAG) across four variables:

1. Baseline Confounder X

2. Intervention A_0 ∈ {0, 1}

3. Mediator M ∈ {0, 1}

4. Outcome Y ∈ {0, 1}

To anchor these concepts, we map each variable to a real-world hiring scenario.

▫ 1. Baseline Confounder X

Baseline confounder (confounder) refers to the pre-existing context or characteristics before any action is taken:

where:

• f_X is a deterministic, non-parametric structural mapping function.

• ε_X is mutually independent, unobserved background noise terms (exogenous variables) drawn from an arbitrary joint probability space.

Hiring context: The jobseeker’s background (e.g., years of experience, tech stack proficiency).

Causal mechanic: Acts as a confounder by simultaneously biasing the assignment choice (X → A_0) and inherently increasing candidate market mobility(X → Y), making them more likely to churn regardless of the intervention.

▫ 2. Intervention A_0 ∈ {0, 1}

The intervention refers to the initial action taken by human or other events:

Where:

• f_A is a deterministic, non-parametric structural mapping function.

• ε_A is mutually independent, unobserved background noise terms (exogenous variables) drawn from an arbitrary joint probability space.

Hiring context: The hiring manager's decision to hire (1) or reject (0).

Causal Mechanic: The decision is driven by the confounder X (the jobseeker's background) and unobserved management heuristics ε_A such as the vibe check, impromptu deep-dive, and even mood and fatigue.

For example, the manager might think, "They have the technical skills, but their communication felt a bit hesitant during the system design round."

That hesitation isn't a standardized metric; it's a subjective interpretation (ε_A).

Or, if the manager is interviewing the candidate at 4:30 PM on a Friday after a brutal production outage, their decision threshold might be entirely different than it would be at 10:00 AM on a Tuesday.

▫ 3. Mediator M ∈ {0, 1}

The meditator refers to an incident happened after the intervention:

Where:

• f_M is a deterministic, non-parametric structural mapping function.

• ε_M is mutually independent, unobserved background noise terms (exogenous variables) drawn from an arbitrary joint probability space.

Hiring context: The specific project assignment given to the employee after they join the company.

Causal mechanic: The intervention (the hiring decision) triggers a behavioral shift (A_0 → M), capturing the indirect effect of the treatment.

▫ 4. Outcome Y ∈ {0, 1}

The outcome is the consequence we'd observe after the intervention and mediator:

Where:

• f_Y is a deterministic, non-parametric structural mapping function.

• ε_Y is mutually independent, unobserved background noise terms (exogenous variables) drawn from an arbitrary joint probability space.

Hiring context: Promotion within the first year.

Causal mechanic: The terminal node endogenous to the system, structurally determined by the direct effects of the baseline context, the intervention, and the mediator.

The below diagram illustrate standard ML approach (top) and three causal inference approaches (bottom):

Figure 2. Structural comparison displaying a standard flattened machine learning input layer versus causal directed acyclic graph frameworks. (Created by Kuriko IWAI)

◼ Statistical View of Standard ML

As shown in Figure 2, standard ML treats confounders X, intervention A_0, and mediator M equally as a flat layer of independent input features.

These features all point directly to the outcome Y, even when the true casual reality (left, Figure 2) has an open backdoor path between the intervention A_0 and the outcome Y.

This makes the optimizer shift weight to whatever features yield the highest immediate impact to the outcome, instead of isolating the true impact of the intervention.

Mathematically, standard ML throws observed features into the conditioning set of the conditional probabilities such that:

Where:

• Y, A_0, M, and X is the outcome, intervention, mediator, confounder, respectively.

• P(Y | ...) denotes the conditional probability distribution of the outcome Y given the conditioning set: A_0, M, and X.

And because the mediator M occurs in close temporal proximity to the outcome (Y = 1, indicating the contract termination), the observational data exhibits a strong conditional dependency such that:

Where:

• Y = 1 indicates the final outcome, the contract termination.

• M = Drop is the mediator, indicating a severe decline in communication response rate.

• ≈ 1.0 indicates the empirical conditional probability approaching unity due to tight chronological and statistical correlation between M and Y.

Consequently, the optimizer assigns maximum weight to the mediator M, concluding that "Communication degradation is the primary driver of turnover".

The challenge here is that an operational policy would misallocate capital toward the symptom (communication degradation) rather than the root cause.

For instance, leadership might decide to automate HR alerts to improve response rates, while leaving the true mechanism, the project mismatch entirely unaddressed.

◼ The Feature Flattening Trap

This fundamental failure of a standard ML in decision-making contexts stems from the feature flattening trap.

Eq. 1.1 denotes that the model completely ignores the structural dependencies and directional relationships between features by flattening them.

This induces two primary structural failures in production:

1. Mediator conditioning, and

2. Weight stealing.

▫ Challenge 1. Mediator Conditioning - Confusing Symptoms with Causes

By flattening the feature space, the model optimizes solely for predictive accuracy on the observational manifold.

It cannot distinguish between an upstream cause, the intervention A_0 and a downstream symptom, the mediator M.

Because M shares a high mutual information metric with Y due to its temporal proximity, the model treats the symptom as the lever.

▫ Challenge 2. Weight Stealing - The Statistical Masking

During back propagation, the optimizer attempts to identify the path of least resistance.

Because the mediator M sits closer to Y on the causal path, it absorbs the gradient updates during training.

Concurrently, because the model cannot map the directional flow X → A_0, the baseline confounder X leaks spurious correlation.

And the resulting parameters β's show a catastrophic distortion such that:

Where:

• β_{M} is the estimated coefficient assigned by the model to the mediator M (communication drop).

• β_{X} is the estimated coefficient assigned by the model to the confounder X.

• β_{A_0} is the estimated coefficient assigned by the model to the primary action lever A_0 (project assignment).

Eq. 1.3. shows that the true cause - intervention A_0 - is masked entirely by a downstream symptom, the meditator M and an upstream context, the confounder X.

Causal inference transitions the paradigm from passive observational conditioning to active, counterfactual intervention.

Unlike standard ML, the causal inference approach maps out the exact structural mechanics of how variables interact using Directed Acyclic Graphs (DAGs).

In this section, I'll explore three frameworks as shown in Figure 2:

• Standard DAG.

• Controlled Direct Effect (CDE).

• Total Causal Effect (TCE).

Standard DAG establishes the foundational network of cause and effect where confounders X influence both the intervention A_0 and the mediator M.

The framework can acknowledge the baseline confounder X such as the human manager's bias affect who gets the treatment, and that the treatment flows through a downstream mediator M before hitting the outcome Y.

Mathematically, standard DAG is denoted as observational joint distribution factorization:

Where:

• P(X, A_0, M, Y): The joint observational probability density function over all variables in the system.

• P(X): The marginal distribution of the baseline confounder X such as a candidate's background context.

• P(A_0 | X): Human bias. The conditional probability representing historical human policy. This models how heavily a manager's assignment action A_0 is influenced by the candidate's background context X.

• P(M | A_0): The conditional probability tracking the downstream transition from action to mediator (e.g., the likelihood of a communication drop given a specific project match style).

• P(Y | A_0, M, X): The conditional probability mapping the final outcome based on all preceding factors combined.

Eq. 2.1 represents the joint probability before any intervention, reflecting the historical data-generating process including human biases exactly as it exists in the data logs.

◼ Pros and Cons - Use Cases

• Can serve as the structural hypothesis, before mapping out any domain knowledge.

• Purely describes the status quo historical system. No mechanism to predict what happens if the network is disrupted by changing corporate policies.

Controlled Direct Effect (CDE) tackles this limitation of standard DAGs by isolating the direct impact of the intervention on the outcome by stripping away any influence from the mediator.

Mathematically, the Controlled Direct Effect (CDE) is defined as:

Where:

• CDE(m) is the Controlled Direct Effect evaluated at a fixed mediator state m.

• E[Y | ...] is the mathematical expectation operator under the designated interventional distributions.

• do(A_0 = 1) and do(A_0 = 0) represent Judea Pearl’s interventional operators which set the treatment to active and baseline control states respectively.

• do(M = m) denotes forcing the mediator to a static value m via a joint, secondary intervention.

Eq. 2.2 performs graph surgery on standard DAGs where the mediator M is held as a constant m using do-operator do(M=m).

▫ The Career Progress Scenario

In our tech pipeline, we want to isolate the Controlled Direct Effect (CDE) of the initial hiring decision (A_0) on a first-year promotion (Y), independent of the subsequent project assignment (M).

Imagine leadership wants to know: Does the raw signaling value of a high-bar hiring decision (A_0) inherently accelerate an employee's promotion velocity (Y), even if they are assigned to a low-visibility, maintenance-heavy project (M)?

To measure this via CDE, we enforce two simultaneous interventions:

1. Change the assignment policy: do(A_0 = 1) vs. do(A_0 = 0)

2. Hold M as constant m by invoking do(M = no_drop).

The second intervention can be a situation (m) where the company introduces a strict corporate policy that every new hire is uniformly assigned to the exact same baseline project, regardless of their background (X) or how highly they were rated during the hiring loop (A_0).

By mathematically locking M in place via do(M = m), we effectively sever the causal link from the hiring decision through the project assignment (from A_0 → M → Y to A_0 → Y).

◼ Pros and Cons

• Best to assess if certain action has an independent path to the outcome.

• Forces an artificial environment where the human behavior which acts as the mediator M is frozen while the policy A_0 is shifting, which is unrealistic in most real-world cases.

▫

Total Causal Effect (TCE) can tackle the challenge of CDE by measuring the entire, unfiltered impact of the intervention on the final outcome Y.

In TCE, the mediator is allowed to flow naturally with the causal signal, instead of being blocked.

Mathematically, TCE can be denoted as the net change in outcome expectation generated across all downstream pathways:

Where:

• E[Y | do(A_0 = 1)] is the expected value of the outcome Y in a counterfactual world where the entire population is actively forced to receive the intervention (A_0 = 1 or A_0 = 0)

• do(A=a) represents Judea Pearl’s interventional operator.

For example, A_0 = 1 refers to the situation where every candidate is hired, whereas in A_0 = 0, every candidate is not hired.

This includes the meditator M block which directs to the outcome Y.

▫ Use Cases

• Best for policy optimization and strategic ROI planning.

• Can simulate the true, real-world impact of policy updates using historical data logs (No need to run an expensive A/B test).

▫ The Unconfoundedness Trap

Unconfoundedness trap is the situation where TCE assumes that all confounders are observed and adjusted.

This works against TCE especially when some unobserved feature influences both the policy and the outcome because TCE cannot capture this hidden feature in the first place.

For example, in the hiring / career progress example, a major hidden confounder could be an employee's hidden motivation or personal resilience level.

Highly motivated employees naturally seek out and volunteer for the most complex project because they want to prove themselves (hence, A_0 = 1).

Those candidates are inherently more likely to promote (hence, Y = 1) because of their strong work ethic and motivation.

This unobserved trait influences both the policy assignment and the outcome.

But here's the challenge.

Because the personal motivation metric isn't captured in the confounder, the causal model cannot adjust for it.

As a result, the TCE calculation falls into the unconfoundedness trap: it will absorb this hidden bias and incorrectly credit the low turnover rate entirely to the project assignment policy, creating a skewed ROI metric for leadership.

Causal inference isolates the interventional distribution such that:

Where:

• Y is the random variable representing the terminal target outcome.

• A_0 is the actionable treatment variable (e.g., project assignment policy).

• a ∈ {0, 1} is the specific counterfactual assignment value enforced by the intervention.

• do(A=a) is Judea Pearl’s interventional operator.

◼ Judea Pearl’s Do-operator

Judea Pearl’s do-operator (Judea Pearl, et. al, 2009) provides the mathematical syntax required to formalize this distinction by separating "seeing" (passive observation) from "doing" (active intervention).

Standard ML operates on the observational distribution denoted in Eq. 1.1, which conditions on the subpopulation that happened to receive treatment A = a under the historical, non-randomized policy.

The do-operator, denoted as do(A = a), represents a hypothetical, physical intervention that overrides the natural data-generating mechanism.

It forces the variable A to take the exact value a for the entire population, irrespective of historical tendencies.

For example, suppose we decide to programmatically overwrite a human manager.

We instantiate a strict routing policy that forces every single employee in the pipeline into a high-pressure project, completely ignoring their resume prestige, their historical background, or the managers' personal preferences.

Mathematically, we evaluate the interventional distribution:

Where do(A_0 = 1) represents the programmatic mutation of the system where we exogenously force the assignment A_0 to 1 for the entire incoming population (employees).

▫ The Enterprise Unlock: Off-Policy Evaluation (OPE)

Shifting from P(Y | A_0) to P(Y | do(A_0)) enables counterfactual Off-Policy Evaluation (OPE).

Instead of deploying an unverified algorithm to production and risking live revenue for instance, a causally identified model allows engineers to leverage biased historical logs to answer the counterfactual optimization problem:

What would our retention curve look like if we had executed policy A1 instead of the legacy human-driven policy A_old?

◼ Identification via the Backdoor Criterion

To recover the true interventional distribution from purely observational logs, we must locate an adjustment set Z that satisfies Pearl's Backdoor Criterion relative to the causal pair (A_0, Y).

Pearl’s Backdoor Criterion is the structural graph theory used to determine exactly which variables must be controlled for to isolate a true causal effect.

It provides the algorithmic rules to select a conditioning set of variables Z that blocks all spurious correlations without accidentally destroying or masking the true causal signal.

▫ The Two Rules of the Backdoor Criterion

A set of variables Z satisfies the backdoor criterion relative to an intervention A and an outcome Y that it passes two strict topological conditions:

• Rule 1: No Downstream Mediators. No variable in Z can be a descendant of A. If a variable M is caused by A, meaning it sits downstream on the causal path like A → M → Y, M is a mediator or a symptom. If you control for it, you block the very effect you are trying to measure, which is what triggers the feature flattening trap.

• Rule 2: Block All Upstream Leaks (The Backdoor Paths) Z must block every path between A and Y that contains an arrow pointing into A. An arrow pointing into A such as N → A → Y represents an upstream cause or a confounder (like human bias or background environment) impacting A. These paths allow statistical information to flow backwards from A, through the confounder, and into Y, creating an illusion of causality.

▫ The Backdoor Adjustment Formula

Once an adjustment set Z is mathematically validated via the Backdoor Criterion, we can safely rewrite the do-operator using standard, observable conditional probabilities:

By applying Eq. 4.1 to our specific staffing pipeline topology, M is structurally excluded, and X represents the support space of our baseline confounders—the continuous and discrete backdoor adjustments yield:

Where:

• X is the complete support space of the baseline confounder X (e.g., the distribution of talent and prestige strata across the candidate pool).

• P(Y | A_0 = a, X = x) is the standard conditional observational probability of the outcome given specific real-world realizations of the treatment action and background confounder.

• P(X = x) or dP(X = x) is the marginal observational probability weight of each confounder stratum, used to re-weight the conditional estimates uniformly across the entire population.

▫ The Enterprise Unlock

Mathematically, Eq. 4.2.2 forces the historical data to simulate a counterfactual world where the assignment action A_0 was distributed completely at random across the background environment X (e.g., all candidates are assigned to complex projects).

This eliminates selection bias and enables engineers to execute a clean, synthetic A/B test directly out of old data logs, completely neutralizing the feature flattening trap.

Let us simulate an enterprise dataset characterized by our structural causal topology.

After defining a confounded dataset, the script runs both standard ML and causal inference, and visualizes the profound estimation bias that occurs in production frameworks:

1import numpy as np
2import pandas as pd
3import statsmodels.api as sm
4import statsmodels.formula.api as smf
5
6
7## 1. define a confounded dataset
8np.random.seed(42)
9N = 10000
10
11# X: baseline context confounder (e.g., talent score)
12X = np.random.normal(0, 1, N)
13
14# A0: match action (historical human managers over-index on X when making matches)
15prob_A0 = 1 / (1 + np.exp(- (1.5 * X)))
16A0 = np.random.binomial(1, prob_A0)
17
18# M: downstream mediator (communication dropout - a bad match A0=1 causes communication dropouts)
19prob_M = 1 / (1 + np.exp(- (-2.0 + 2.5 * A0)))
20M = np.random.binomial(1, prob_M)
21
22# Y: true project exit - driven directly by the bad match AND the communication dropout. True total effect on Y is a combination of direct and mediated pathways
23prob_Y = 1 / (1 + np.exp(- (-3.0 + 1.2 * A0 + 1.8 * M + 0.5 * X)))
24Y = np.random.binomial(1, prob_Y)
25
26# create a dataframe
27df = pd.DataFrame({'X': X, 'A0': A0, 'M': M, 'Y': Y})
28
29
30# 2. standard ML approach (feature flattening)
31## throw the mediator M into the feature matrix.
32standard_model = smf.logit("Y ~ A0 + M + X", data=df).fit(disp=0)
33
34
35# 3. causal inference
36## structurally leave mediator M out of the regression
37causal_model = smf.logit("Y ~ A0 + X", data=df).fit(disp=0)
38
39
40# 4. visualize estimation bias
41models_data = {
42    'Model Type': [
43        'Standard ML (Flattened)', 
44        'Causal Backdoor Adjustment'
45    ],
46    'Estimated Beta (A0)': [
47        standard_model.params['A0'],
48        causal_model.params['A0']
49    ],
50    'CI_lower': [
51        standard_model.conf_int().loc['A0', 0],
52        causal_model.conf_int().loc['A0', 0]
53    ],
54    'CI_upper': [
55        standard_model.conf_int().loc['A0', 1],
56        causal_model.conf_int().loc['A0', 1]
57    ]
58}
59
60results_df = pd.DataFrame(models_data)
61

◼ The Simulation Results: Proof of the Trap

By running the Python script, we generate a dataset where the initial match decision (A_0) has both a direct impact on project success and an indirect impact by causing a downstream communication dropout (M).

When we run both a standard machine learning and causal inference, the resulting coefficients expose a massive blind spot.

▫ The Results - Empirical Breakdown of the Coefficients

Table 1. Empirical Parameter Comparison: Structural Ground Truth vs. Observational Feature-Flattened Frameworks

The Model Illusion

The Standard ML model will report a highly deceptive feature importance profile.

It will convince humans that a drop in communication (M = 1.81) is nearly twice as critical to fix than the actual matching process (A_0 = 1.12).

The Downstream Value Drain

If leadership uses the standard model's outputs to plan investments, the company will pour capital into downstream band-aids such as automated Slack nudges or communication alerts.

These interventions will ultimately fail because they leave the true, high-leverage driver—the broken matching habit—completely unaddressed.

The Causal Unlock

By utilizing Pearl's Backdoor Criterion and blocking only the baseline confounder (X), the Causal model uncovers the true structural coefficient of 2.00.

This gives leadership an accurate, non-confounded ROI metric to justify rewriting the entire onboarding and assignment policy.

While the Backdoor Criterion provides the structural blueprint to escape the Feature Flattening Trap, implementing it on real-world enterprise architectures introduces an immediate engineering bottleneck.

In our Python simulation, I utilized a clean, low-dimensional confounder X.

But in production, the baseline confounder X is high-dimensional.

Attempting a naive, exact stratification on such an expansive space triggers the curse of dimensionality where the data splits into empty strata, and math collapses.

Worse yet, real-world historical data is plagued by selection bias and institutional behavioral cloning.

Because legacy human managers systematically over-indexed on specific shortcuts (like assigning all Ivy League graduates exclusively to premier accounts), there are vast regions of the feature space where alternative choices were never attempted.

To run a true offline evaluation, we must find a way to re-weight our dirty observational data to create a balanced, equitable foundation.

◼ Up Next — Part 2: De-Biasing Historic Logs

▫ Synthetic Randomization via Inverse Propensity Scoring (IPS)

In the next installment of this series, we will transition from pure structural graph theory to scalable algorithmic execution.

I will explore how to use Inverse Propensity Scoring (IPS) to isolate and neutralize day-one human selection shortcuts.

◼ References

• Judea Pearl, Causal inference in statistics: An overview

Architecting Semantic Chunking Pipelines for High-Performance RAG

Master critical chunking strategies for RAG to enhance retrieval accuracy and context retention.

Chunking is the process of breaking down large bodies of text into smaller, manageable pieces (chunks) before they are converted into mathematical representations called embeddings.

The below diagram illustrate the data ingestion pipeline and how chunking plays a key role in the pipeline:

Figure A. Technical diagram of the RAG data ingestion pipeline illustrating the flow from raw data to chunking, embedding, and high-dimensional vector storage (Created by Kuriko IWAI)

The process begins with raw, unstructured data in various formats like text (PDFs, docs), images, audio, and video.

The chunking process happens after the retrieval (second box, Figure A) where the data is broken down into smaller pieces called chunks.

By splitting a long document into smaller paragraphs or segments, the system can later retrieve only the specific part that answers a user's question, rather than the entire file.

Then each chunk is passed through an embedding model (an AI algorithm) that converts the content into a vector embedding, a long list of numbers (coordinates) that represent the semantic meaning of the chunk.

Lastly, these embeddings are stored in a vector store (database).

The vector space in Figure A has three dimensions for demonstration purpose, but usually the space is high-dimensional.

These data points are stored close together when they are considered related.

For example, if we have a chunk about "Golden Retrievers" and another about "Labradors," they will be mathematically near each other in the database, allowing the LLM to find relevant information almost instantly.

◼ Why Chunking Matters

Chunking matters for the following reasons:

• Relevance: Small chunks ensure that RAG can retrieve the exact piece from the large document.

• Cost efficiency: Processing smaller, targeted snippets saves on tokens and computation time.

• Context retention: Well-chunked data maintains enough surrounding information to enable the LLM to comprehend the context on why and how.

Overall, chunking helps the system to retrieve the most relevant context to the user query, while saving input tokens (and fit the context into the LLM's context window).

Selecting a chunking strategy is not a one-size-fits-all decision.

This section explores five major chunking strategies:

• Fixed-size chunking.

• Recursive character chunking.

• Document-specific chunking.

• Semantic chunking.

• Parent–Child (Hierarchical) chunking.

To understand how these methods diverge in practice, we will apply each to a sample text regarding Solar Energy Infrastructure.

▫ Sample Text

I'll use the sample text to see how each chunking strategy splits the text:

1text = "Solar panels, or photovoltaic cells, convert sunlight into electricity. This process happens at the atomic level. Some materials exhibit a property known as the photoelectric effect. This causes them to absorb photons and release electrons. Beyond the cells, an inverter is required to convert DC to AC. Large-scale solar farms also require battery storage systems to manage peak load during non-sunny hours."
2

◼ Fixed-Size Chunking

The fixed-size chunking is the most straightforward approach to define a specific number of characters or tokens per chunk.

The below diagram illustrates how it works:

Figure B. Visualization of fixed-size chunking showing the sliding window mechanism with defined chunk size and yellow-highlighted overlap section (Created by Kuriko IWAI)

For example, the chunk size (Green cells, Figure B) and chunk overlap (or called sliding windows) (yellow cells, Figure B) are set to 500 tokens / 50 tokens respectively.

The overlap ensures that context isn't lost if a key sentence is split in half.

▫ Pros

Computationally affordable and easy to implement.

▫ Best For

• Quick prototyping.

• Handling simple text.

• General use cases where speed is prioritized over granular semantic accuracy.

▫ Practical Implementation

The CharacterTextSplitter class from the langchain_text_splitters library can split the sample text:

1from langchain_text_splitters import CharacterTextSplitter
2
3# fixed size chunking
4fixed_splitter = CharacterTextSplitter(
5    separator="",
6    chunk_size=100,
7    chunk_overlap=20
8)
9
10fixed_chunks = fixed_splitter.split_text(text)
11

The configuration shows 100 tokens for each chunk with 20 tokens overlapped.

▫ Resulting Chunks:

Each chunk contains exactly 100 tokens:

• Chunk 1: "Solar panels, or photovoltaic cells, convert sunlight into electricity. This process happens at the a"

• Chunk 2: "ns at the atomic level. Some materials exhibit a property known as the photoelectric effect. This ca"

In this method, a word like "atomic" is sliced.

Although computationally fastest, the method can create semantic noise as the LLM receives partial words, which can degrade the quality of the generated response.

◼ Recursive Character Chunking

The recursive character text splitting uses a hierarchy of separators to find a natural breaking point, instead of cutting text at a hard character limit.

The below diagram illustrates how the method works:

Figure C. Diagram of recursive character splitting logic demonstrating the hierarchical priority of separators like newlines and periods to preserve sentence integrity (Created by Kuriko IWAI)

The method first attempts to split by the most significant separator, period, and then moves down the list for commas and spaces, until the chunk size requirement (pink box, Figure C) is met.

▫ Pros

• Keeps related ideas together better than fixed-size splitting.

▫ Best For

• Maintaining the integrity of paragraphs and sentences.

• Articles and blogs.

▫ Practical Implementation

The RecursiveCharacterTextSplitter class from the langchain_text_splitters library can split the sample text:

1from langchain_text_splitters import RecursiveCharacterTextSplitter
2
3recursive_splitter = RecursiveCharacterTextSplitter(
4    chunk_size=100,
5    chunk_overlap=20,
6    separators=["\n\n", "\n", " ", ""]
7)
8
9recursive_chunks = recursive_splitter.split_text(text)
10

The configuration shows 100 tokens for each chunk with 20 tokens overlapped.

The splitters have priority of double lines, single lines, double spaces, and single spaces.

▫ Resulting Chunks

Each chunk contains exactly 100 tokens such that:

• Chunk 1: 'Solar panels, or photovoltaic cells, convert sunlight into electricity.'

• Chunk 2: 'This process happens at the atomic level. Some materials exhibit a property'

Compared to the fixed-size chunking method, the recursive splitter identifies the period at the end of the first sentence and stops there, preserving grammatical integrity.

This makes the method more readable for the LLM.

◼ Document-Specific Chunking

The document-specific chunking respects the inherent format of the file types like Markdown, HTML, LaTeX, or Code.

For example:

• Markdown: Splits by headers (#, ##, ###).

• HTML: Splits by html tags, comma, or dots.

• Code: Splits by function or class definitions.

▫ Pros

• Preserves the logical hierarchy of the document.

▫ Best For

• Highly structured technical documentation like PDF reports, manuals.

• Codebases.

▫ Practical Implementation

The MarkdownHeaderTextSplitter class from the langchain_text_splitter library can define the splitters and split the document:

1from langchain_text_splitter import MarkdownHeaderTextSplitter
2
3markdown_document = """
4# Solar Energy Guide
5
6## The Physics
7Solar panels, or photovoltaic cells, convert sunlight into electricity. 
8This process happens at the atomic level. 
9Some materials exhibit the photoelectric effect.
10
11## The Hardware
12Beyond the cells, an inverter is required to convert DC to AC. 
13Large-scale solar farms also require battery storage systems.
14
15### Maintenance
16Regular cleaning of panels ensures maximum photon absorption.
17"""
18
19# define the headers
20headers_to_split_on = [
21    ("#", "Header 1"),
22    ("##", "Header 2"),
23    ("###", "Header 3"),
24]
25
26# initialize the splitter
27markdown_splitter = MarkdownHeaderTextSplitter(headers_to_split_on=headers_to_split_on)
28
29# split
30md_header_splits = markdown_splitter.split_text(markdown_document)
31

▫ Resulting Chunks:

Each chunk has been sliced by the headers defined in the code snippets:

Chunk 1.

• Content: Solar panels, or photovoltaic cells, convert sunlight into electricity. This pro...

• Metadata: {'Header 1': 'Solar Energy Guide', 'Header 2': 'The Physics'}

Chunk 2.

• Content: Beyond the cells, an inverter is required to convert DC to AC. Large-scale solar...

• Metadata: {'Header 1': 'Solar Energy Guide', 'Header 2': 'The Hardware'}

Chunk 3.

• Content: Regular cleaning of panels ensures maximum photon absorption....

• Metadata: {'Header 1': 'Solar Energy Guide', 'Header 2': 'The Hardware', 'Header 3': 'Maintenance'}

The method can leverage the author's intent, ensuring that "The Physics" and "The Hardware" are never accidentally blended into the same chunk, which is vital for technical manuals.

◼ Semantic Chunking

The semantic chunking is a more advanced technique that calculates the distance of the meaning to determine where a topic changes.

The method looks at the cosine similarity between the embeddings of consecutive sentences.

When the similarity drops below a certain threshold, it assumes a new topic has started and creates a break to move onto a new chunk.

▫ Pros

• High retrieval accuracy because chunks represent complete ideas.

▫ Best For

• Academic paper.

• Narrative-heavy documents.

• Long-form essays (paragraphs don't align with the topic shift).

▫ Practical Implementation

I'll first create vector embeddings using the SentenceTransformer class from the sentence_transformers library.

Then, split the embeddings into smaller chunks by calculating cosign similarity scores between embeddings.

1from sklearn.metrics.pairwise import cosine_similarity
2from sentence_transformers import SentenceTransformer
3
4# split the text into sentences
5sentences = split_into_sentences(text)
6
7# create vector embeddings 
8model = SentenceTransformer(MODEL)
9embeddings = model.encode(sentences)
10
11# compute cosine similarity to split into chunks
12chunks = []
13current_chunk = [sentences[0]]
14for i in range(1, len(sentences)):
15    prev_emb = embeddings[i - 1].reshape(1, -1)
16    curr_emb = embeddings[i].reshape(1, -1)
17
18    similarity = cosine_similarity(prev_emb, curr_emb)[0][0]
19    print(f"Similarity between sentence {i-1} and {i}: {similarity:.3f}")
20
21    # if similarity drops → meaning shift → split
22    if similarity < similarity_threshold:
23        chunks.append(" ".join(current_chunk))
24        current_chunk = [sentences[i]]
25    else:
26        current_chunk.append(sentences[i])
27
28# add last chunk
29if current_chunk:
30    chunks.append(" ".join(current_chunk))
31
32return chunks
33

▫ Resulting Chunks

In this method, the system notices a shift in meaning between "release electrons" and "Beyond the cells."

• Chunk 1. Solar panels, or photovoltaic cells, convert sunlight into electricity.

  • Cosign similarity: 0.196

• Chunk 2. This process happens at the atomic level.

  • Cosign similarity: 0.313

• Chunk 3. Some materials exhibit a property known as the photoelectric effect.

  • Cosign similarity: 0.546

• Chunk 4. This causes them to absorb photons and release electrons.

  • Cosign similarity: 0.266

• Chunk 5. Beyond the cells, an inverter is required to convert DC to AC.

  • Cosign similarity: 0.336

• Chunk 6. Large-scale solar farms also require battery storage systems to manage peak load during non-sunny hours.

Semantic chunking recognizes that the topic changed from physics to engineering and forces a split.

◼ Parent–Child (Hierarchical) Chunking

The parent-child (hierarchical) chunking involves storing two versions of the same data: a small child chunk for searching and a larger parent chunk for context.

The system first searches against small, highly specific chunks (e.g., 100 tokens).

Then, once a match is found, it retrieves the larger surrounding parent document (e.g., 1000 tokens) to provide to the LLM.

▫ Pros

• Avoids the lost in the middle problem by giving the LLM plenty of background info.

▫ Best For

• Enterprise-grade RAG systems.

• Balancing high-precision search with comprehensive context.

▫ Practical Implementation

I'll first create parent chunk which contains all the sample text, and then create child chunks which contains vector embeddings:

1import re, uuid
2from sentence_transformers import SentenceTransformer
3
4# create parent chunk
5parent_chunk = {
6    "id": str(uuid.uuid4()),
7    "text": text.strip() # the entire sample text
8}
9
10
11# load embedding model
12model = SentenceTransformer("all-MiniLM-L6-v2")
13
14def create_child_chunks(parent_chunk):
15    sentences = split_into_sentences(parent_chunk["text"])
16    
17    children = []
18    for sentence in sentences:
19        child = {
20            "id": str(uuid.uuid4()),
21            "parent_id": parent_chunk["id"],
22            "text": sentence,
23            "embedding": model.encode(sentence)
24        }
25        children.append(child)
26    return children
27
28child_chunks = create_child_chunks(parent_chunk)
29

▫ Results

Parent context:
Solar panels, or photovoltaic cells, convert sunlight into electricity.
This process happens at the atomic level.
Some materials exhibit a property known as the photoelectric effect.
This causes them to absorb photons and release electrons.
Beyond the cells, an inverter is required to convert DC to AC.
Large-scale solar farms also require battery storage systems to manage peak load during non-sunny hours.

Matched child sentence:
Large-scale solar farms also require battery storage systems to manage peak load during non-sunny hours.

In the process, the retriever pulls the parent chunk when it finds that the query matches some vector embedding, instead of returning just the matched sentence.

This enables the LLM to receive the full paragraph and assess the relationship between solar panels, inverters, and battery storage. In other words, it explains the "Why" (infrastructure) rather than just the "What" (batteries).

Chunking is not just slicing data.

With a proper strategy, it can work as semantic optimization.

Proper grouping ensures that when a user asks a question, the vector database returns a coherent piece of information rather than a fragmented snippet that leaves the AI guessing.

Here are key strategies to consider when it comes to choosing the optimal chunking strategies.

◼ Implementation Roadmap: Choosing Optimal Strategy

▫ 1. Identify Patterns & Logical Structures

Look for repetitions, sequences, or inherent connections in the data.

In a technical context, this means identifying document headers, Markdown tags, or paragraph breaks to ensure a chunk doesn't cut off in the middle of a vital sentence or thought.

• The key question:

• Strategy to choose: Document-specific (structure-aware) chunking.

Use parsers that respect the document's native format (e.g., Markdown, HTML, or LaTeX) to split data at logical boundaries rather than arbitrary character counts.

▫ 2. Maintain Semantic Context (The Mnemonic for AI)

Just as humans use mnemonics to link ideas, AI systems use overlapping chunks.

By including a small portion of the previous chunk at the start of the next one, you create a narrative bridge that prevents the model from losing the broader context of the data.

• The key question to ask:

• Strategy to choose: Fixed-sized chunking with sliding window (overlapping).

Implementing a context window of 10–20% overlap between chunks ensures that the end of one chunk and the beginning of the next share enough connective tissue to maintain semantic flow.

▫ 3. Prioritize Categorical Grouping

Organize information by category or hierarchy (e.g., grouping a grocery list by "produce" or "dairy").

• The key question to ask:

• Strategy to choose: Recursive Character Splitting.

Start with a large separator (like a double newline) and progressively move to smaller separators (space, character) until the desired chunk size is reached. This keeps neighboring ideas in the same bucket.

◼ Optimize for Retrieval & Relevance

Lastly, in either chunking strategy we choose, it is best to regularly test the chunk size against real-world queries because if chunks are too small, they lack context; if they are too large, they introduce noise that can confuse the LLM.

The key question one can ask is:

And if this is the case, experimental benchmarking would work the best.

The experimental benchmarking runs chunking with different chunk sizes (e.g., 256, 512, and 1024 tokens), and evaluates each of them using metrics like Hit Rate or MRR (Mean Reciprocal Rank).

It allows one to determine which size consistently yields the most accurate answers for a task in hand.