Are Observational Causal Studies Hard To Get Right?

In last week's post Ron Kohavi cited a 2022 Facebook Ads paper to demonstrate that despite using massive data, observational methods can lead to biased estimates of causal effects.

The authors show that two "observational" methods: stratified propensity score matching (SPSM) and double machine learning (DML) lead to significant overestimation of causal effects of ads on conversion compared to a reference A/B test (6x and 2.9x respectively)

Assuming the reference A/B test has been conducted and analyzed correctly, these numbers indeed don't look good for SPSM and DML.

I agree with Ronny that these results are a "massive overestimation", especially when it comes to SPSM.

Yet...

More Is Better. Or Is It?

I have a hard time agreeing with Ronny's diagnosis that "there was an enormous effort (on part of the authors) to address selection bias with 5,000 user-level features (yes, five thousand)."

Here's why:

The authors correctly identify in their paper that DML in order to work correctly requires two assumptions to be fulfilled: unconfoudedness and positivity.

Both are basic causal inference assumptions, required by a vast majority of standard causal estimators.

However, what the authors miss is that these are not the only assumptions required to get unbiased estimates from DML.

Unconfoundedness is one of the assumptions required to obtain causal identification for interventions under the standard structural causal model framework.

But causal identification is not only about the variables we should include in the model.

It's also about ones that we should avoid using.

When we're interested in the total causal effect of, let's say, X on Y, we should avoid using in our model variables that are the "middlemen" between X and Y, so-called mediators.

We should also avoid using variables that are descendants of the treatment and the outcome (or, for that matter, descendants of any pair made up of a mediator, the outcome, or the treatment).

The authors seem not to take this fact into account.

In fact, they explicitly state that in their modeling they used features like "estimated probability of conversion given exposure", which seems like an opener of a (noisy) collider path between a mediator and the outcome (a seeming paradox: the better the estimate of this probability, the more bias we have!)

It also seems that among four categories of features mentioned by the authors there are at least some features that would mediate the impact of ads on conversion, even if only partially (I assume this would also hold if we change the estimand from ATE to ATT).

Both, colliders and mediators introduce bias in causal estimates that can be arbitrarily more severe than the benefit of bias reduction coming from controlling for confounders (or their proxies).

What I read as an unfortunate implicit conclusion from both, the paper and Ronny's post, is that the number of variables we include in causal models is a somewhat reliable indicator of the effort put into causal modeling.

Practice seems to support what we know from modern causal theory: models with a small(er) number of well selected features almost always outperform models with a massive set of features without any selection.

The Narrow Temptation Of Extrapolation Or...

The idea of "the more features the better" in causal modeling seem to be an extrapolation from predictive modeling and the improvements we often (though not always) observe in in-distribution performance after adding more predictors. to our machine learning models.

As it turns out, this idea does not extrapolate very well to causal modeling.

My dream is that one day we'll start teaching causality as a framework, rather than a set of loosely connected methods, procedures and tricks.

I briefly outlined this idea in Lecture 9 of Causal Secrets, and I believe it can immunize us against popular misconceptions about causal modeling, which, along with the increase in popularity of causal modeling itself, seem to be on the rise.

...The Broader View

My belief is that looking at causality as a framework gives us more opportunities to honestly evaluate which methods and techniques can help us answer the questions we care about.

Does this make causal inference, and particularly observational causal inference easy to get right?

In my opinion, not necessarily.

Causality is difficult!

Yet, the framework perspective allows us to approach causal modeling more thoughtfully and answer questions that are often impossible to address, or even invisible, when we choose to have only a partial view:

• What can I do when I cannot randomize the treatment and cannot guarantee there are no hidden confounders in the system I'm modeling?
• When and why can design-of-experiments (DoE) be a better choice than an A/B test?
• Which causal questions cannot be reliably answered with an RCT and why?
• Is it possible to combine observational and experimental data to answer more causal questions or answer existing ones with more precision?

Closing

I've never met a person for whom understanding causal identification or DoE would make them a worse experimenter, or someone who learned more about experimentation and became a worse causal data scientist.

I've met many, who by learning them as a part of a bigger framework were able to answer questions they were never able to answer before.

I prepared a notebook for you, demonstrating how controlling for different types of variables impacts the magnitude of bias in DML.