1. Are interaction effects important? Yes! If anything, I believe interaction effects are more important than most quantitative social scientists do, because I believe “main effects” are a very useful fiction but fictive nonetheless. Anything that affects anyone affects different people differently. And there are reasons why different causes affect some people more than others, which means there are ubiquitious interactions Out There to be found.
2. In behavioral genetics, there is a distinction between “biological” and “statistical” interaction. Roughly, biological interaction refers to an actual substantive process out in the world, and statistical interaction is what you can observe with regression models on population data. You can statistical interaction in the absence of biological interaction and vice versa. I wish a distinction between “substantive” and “statistical” interaction would diffuse more broadly, and substantive interaction is what social scientists interested in causal inference ultimately need to be focused on.
3. The problem is not that interaction effects are not important, but that they are very tricky. Part of the trickiness is how easy it is to be deluded into thinking interaction effects are replicably real when they are not. There are deeper issues, though. One:
Imagine that 30% of white men and 50% of white women have read a novel in the last year (I’ve no idea what the actual percentages are). Now imagine that 15% of black men have read a novel in the last year. What % of black women would correspond to their being no interaction between race and gender? 25%? 35%? In my neck of the woods, the most commonly used model implies a null hypothesis of 29.5%. There’s a lot of unreflected faith in the substantive fidelity of logistic regression to take deviations from 29.5% as worthy of little asterisks and publication.
4. What I was pointing to in my post is the question of how one goes about deciding what published interaction effects to believe, especially in areas where data limitations make replication sparse (and, for that matter, often not actually replication, in that there is not some transcendant true parameter that one can pursue with samples on different populations). Three things:
a. Does the theoretical explanation seem like something came up with after the fact? (It’s good to practice post hoc explanation–no, not in print–so that one has a better ability to detect it in other work.)
b. Does the data include several related measures that it seems like the author could have used but did not? Worst here is when an an interaction is based on a survey item that is part of a set of items in a big study that were intended to measure a similar construct, especially if the paper makes no mention of the existence of the other items. (This is one reason ‘distance breeds enchantment’ with datasets; that is, the more one learns first-hand about secondary datasets, often the less one comes to believe findings from it.)
c. The phenomenon referred to in my earlier post. The interaction is a group difference in the effect of a continuous variable on an outcome, where the group with the larger effect is also the group much smaller in size. In evaluating such interactions, at the very least figure out how small the smaller group is, and think of that number as basically the N on which base one’s judgment about how to regard the finding.