Wednesday, August 20, 2008

Interaction effects

Interesting new post in a really uncreatively named blog ("Statistical Modeling, Causal Inference, and Social Science") that I found recently on interaction effects. The author is critiquing a post by Jeremy Freese that essentially cautions against over-interpretation of interactions.

I think the point the author makes is obvious - you never want to go fishing for significant terms and stake your claim on whatever pops up out of the hundreds of regressions that you run. Of course this applies as readily to interaction effects as it does to anything else included in the model. But I think the author misses a major point that Freese makes - and that is understanding when to expect insignificant interaction effects. Freese suggests that when the categories you are interacting involve very small groups, the risk of pulling out spurious effects increases dramatically. This is because any single sub-group member with an unusual outcome can skew the effect much more easily.

I also find it interesting that they're only talking about interaction effects as a method of sub-group analysis. I've always been most interested in interactions of two continuous variables, rather than a dichotomous and a dummy variable - looking at a sort of cross-effect. I know if you want to look at the difference of an effect between men and women or blacks and whites, that's just not appropriate - but I imagine that the significance of these interactions is substantially more reliable.

We're working on a report for the Assistant Secretary for Planning and Evaluation right now, that not only has a host of interactions - but also has interactions in a two stage model. Now that is REALLY tough to get your head around. You'd think it would be easy - just an extension of a one-stage model... not really.

1 comment:

jeremy said...

Yes, the problem is specifically when there is a small group AND the small group has a larger estimated effect than the smaller group