Causal Inference for Text and Image Outcomes: Discovering the Most Affected Feature

Introduction

Traditional causal inference focuses on scalar outcomes such as recovery status or revenue change, where the average treatment effect (ATE) is defined as the difference between potential outcomes. When the outcome is a document, an open‑ended questionnaire response, or a medical image, subtraction is undefined and researchers often lack a pre‑specified metric to evaluate the effect.

Problem Definition

The paper defines the problem for unstructured outcomes by assuming observed data contain covariates, a binary treatment, and a representation of the outcome (e.g., text embeddings or image vectors). Under the standard causal assumptions of consistency, ignorability, and overlap, the causal contrast for any bounded scoring function f is the expected difference E[f(Y^1)] - E[f(Y^0)]. The “max‑contrast feature” is the function in a candidate class that maximizes this contrast, thereby identifying the aspect of the outcome most altered by the treatment without pre‑defining it.

Identification and Estimation

Given the assumptions, the contrast for a fixed f is identifiable via propensity‑score weighting to align covariate distributions between treatment groups. Extending to a whole function class, the authors derive an estimable objective that searches for the f with the largest weighted difference. In practice they parameterize f with neural networks, bound its output to [0,1], estimate propensity scores, and optimize the weighted objective using cross‑fitting. The paper also sketches asymptotic normality and efficiency conditions for statistical inference.

Paired Feature Learning for Unstructured Treatment and Outcome

When both treatment and outcome are unstructured (e.g., a prompt and a generated image), a single scoring function is insufficient. The authors introduce paired scoring functions g_T for the treatment and g_Y for the outcome. The objective seeks the pair that yields the strongest causal contrast after controlling for covariates. To avoid learning spurious correlations, they construct negative‑control outcomes by mismatching treatments with outcomes that share covariates but not the true causal link, forcing the model to capture genuine treatment‑result relationships.

Empirical Studies

Text outcomes – formalness experiment: In a global setting the learned feature assigns higher scores to formal sentences after treatment; in a context‑dependent setting the direction flips (higher scores for informal text in entertainment contexts and higher scores for formal text in family‑advice contexts). This demonstrates that the method captures context‑specific treatment effects rather than a fixed formalness classifier.

Text outcomes – toxicity experiment: The learned feature gives higher scores to non‑toxic text globally, but the direction reverses in high‑conflict versus mild‑support contexts, illustrating the risk of a single, context‑agnostic interpretation.

Multi‑attribute text experiment: By varying whether treatment changes formalness, punctuation, or both, the max‑contrast feature recovers the correct combination of attributes, confirming that it does not collapse to a single pre‑defined label.

Image outcomes – cell‑image blur experiment: Using cell microscopy images, the learned feature score correlates positively with image blur while controlling for cell count. Nudging the representation along the learned direction increases blur without altering cell content, validating the method on visual data.

Paired feature learning – treatment & result both unstructured: A controlled experiment shows that only the first treatment coordinate and the first result coordinate exhibit systematic relationships; all other dimensions behave as noise, confirming that the paired functions isolate the true causal direction.

News‑title generation experiment: Formal prompts receive higher treatment‑side scores, and the generated headlines also receive higher outcome‑side scores across entertainment and political topics, indicating that the method captures the causal link between prompt formalness and headline formalness rather than merely topic differences.

Discussion and Limitations

The max‑contrast feature turns an open‑ended causal query on unstructured data into a tractable feature‑search problem, eliminating the need for hand‑crafted metrics. The approach depends on the richness of the representation space; if key attributes are not captured, the method cannot recover them. Interpreting the learned feature requires additional analysis—high scores cannot be automatically equated with a semantic label. The standard causal assumptions (ignorability, overlap) remain essential; unobserved confounders cannot be resolved solely by using unstructured representations.