There’s lots to say about the recent article by Mark Regnerus on outcomes of adults who remember a parent having had a same-sex relationship and the other articles and commentaries surrounding it in the journal, and much has already been said. The bottom line is that this is bad science, it is not about same-sex or gay parenting, and strong but circumstantial evidence suggests its main reason for being is to provide ammunition to right-wing activists against LGBT rights. In this (long!) post I offer my evaluation of the scientific merit of the paper as well as the politics surrounding the papers’ funding, publication, spin, and evaluation.
Background: The so-called “no-differences” claim
The logic of Regnerus’s study is motivated by what he calls the “no-differences” position, the claim that there are no differences between children raised by same-sex parents and those raised by opposite-sex parents. Stacey and Biblarz offered a good criticism of this research program back in 2001. The prior article in this issue of SSR, by Loren Marks with respect to the American Psychological Association’s 2005 brief on same-sex parenting, provides an extended criticism of the scientific process that led to the adoption of that statement. (To my non-specialist eye, that statement is similar to the one from the American Academy of Pediatrics some years ago, in which [full disclosure] my mother, Ellen Perrin, was a key player.) The APA statement holds that:
Not a single study has found children of lesbian or gay parents to be disadvantaged in any significant respect relative to children of heterosexual parents.
Marks takes this statement to task by evaluating the various studies examined for the brief and arguing that none of them is sufficiently scientifically rigorous to justify that summary claim. Regnerus does similarly at the outset of his article, though more sloppily:
Suffice it to say that versions of the phrase “no differences” have been employed in a wide variety of studies, reports, depositions, books, and articles since 2000.
Both Marks and Regnerus make a key epistemological error, whether intended or not. The APA and AAP reports conclude essentially that there is no evidence of systematic difference, and Marks and Regnerus treat this as if the reports conclude that there is conclusive evidence of lack of systematic difference. But they do not, and this is the first dishonesty of the articles. Rejecting a hypothesis is not the same thing as proving a null hypothesis.
Regnerus’s article is framed as an early report on a new dataset, the New Family Structures Survey, which is also shown in postliterate style here. The study uses a Knowledge Networks sample of about 3,000 respondents born between about 1981 and 1994. It asks about a variety of characteristics of interest as well as for information about the respondents’ families of origin, including a diary in four-month blocks of with whom the respondent lived during his/her childhood between birth and age 18. It also asks:
From when you were born until age 18 (or until you left home to be on your own), did either of your parents ever have a romantic relationship with someone of the same sex?
Respondents who answered “yes” were classified as children of lesbian mothers (“LM”) or gay fathers (“GF”) (depending on which parent they recalled having such a relationship), and compared to respondents from “Intact Biological Families,” (“IBF”) as well as to respondents from adopted, divorced, single-parent, and step-family environments. These categories were treated as mutually exclusive (more on that below).
The article’s single biggest weakness is this definition of same-sex parents, which in turn undermines much of the rest of the study. The definition is not just that a parent ever had a same-sex romantic relationship while the respondent was under 18 (which would be bad enough as an indicator of “gay or lesbian parent”), but that the respondent had to know about that relationship and choose to report it on the survey. So the conceptual definition of “same-sex couples” is thoroughly invalid, as some proportion of these parents was certainly not raising children in a same-sex household. In fact, virtually none of the respondents were actually raised in a same-sex household. The measurement is also subject to substantial and unknown recall bias, as respondents who have encountered later-in-life problems and/or have strained relationships with parents may be more likely to remember same-sex “affairs” and potentially even make them up in retrospect. The article is very careless in its use of language around this point, often referring to “gay men” and “lesbians” when it means “parents whose adult children reported that they had ever had a romantic relationship with someone of the same sex.”
The categories used are, as the article acknowledges, not mutually exclusive in real life, but treated that way for analytical purposes. But their non-mutual-exclusivity is precisely the point: people who fall into the LM and GF categories may quite plausibly also be in any other category as well, and indeed most of them almost certainly were, so the analytical strategy amounts to manufacturing a virtual world that doesn’t fit the data at all, then analyzing that world. Why do it this way? Well, “gay and lesbian parents, as well as heterosexual adoptive parents, can be challenging to identify and locate.” The article goes to great lengths to explain how hard they worked to find the necessary respondents. But trying really hard doesn’t help when the conceptual category is fatally flawed. The parents discussed aren’t, by and large, gay and lesbian parents!
The family diary data collected for NFSS and described in the article would go a long way toward evaluating who the parents were and what the associated family structures were, but those data aren’t used or presented here. So Regnerus had available to him data that would actually address the question of the family structures and relationship statuses of the respondents’ families of origin, but chose not to use them and instead to use a conceptually fatally flawed measure to create fictional groups. I will address some theses on why he may have made that decision below, but strictly on the science, had I been asked to review the paper I would have said there is no point in publishing this analysis when the same data would provide the opportunity for a non-fictional analysis. Gary Gates has made precisely this point as well.
One of the outcome measures, respondent’s sexual preference, is dichotomized as “100% heterosexual (vs. anything else)” (p. 758). This seems designed to show big differences, since kids with nontraditional parents may be more likely at least to question sexuality. No discussion is offered of whether the analysis is robust to relaxing the 100% rule.
There’s a lot made of the relationship between LMs and receipt of public assistance, both in family of origin and currently. The obvious question, even granting the bizarre definition of LMs, is causality – some alternative hypotheses would include that (a) women fleeing abusive husbands sometimes enter romantic relationships with other women; or that (b) given women’s subordinate status particularly with respect to employment, lesbian working-class households have a harder time financially. And yes, the article does make causal claims with this outcome, at least in the conclusion (see below). Perhaps more interestingly, looking at the likelihood of a respondent receiving public assistance based on his/her family of origin having done so: that likelihood decreases by 41% for IBFs (from 17% to 10%) but by 44% for LMs and by nearly 60% for GFs, suggesting that poverty is a more temporary condition for these respondents than for IBFs.
In the “Summary of Differences” section on 764, there’s an attempt to catalog the number of possible between-group differences we might find, out of a total of 279 (the original 40 plus 239 additional). The article offers a count of how many such differences are found (at p < .05) between GFs and everyone else and LMs and everyone else. Recognize, though, that at p < .05, we’d expect to find 5% significant differences by chance alone. Apparently nothing was done to correct for the likelihood of chance associations even though 279 is an awful lot of dependent variables!
In the last two pages we turn to the conclusion, where Regnerus just can’t help but make causal claims, even though elsewhere he calls “implying causation here—to parental sexual orientation or anything else, for that matter—… a bridge too far.” First is the idea that “diminished context of kin altruism” is the overarching causal principle that connects various non-IBF families to the various pathologies he delineates. Finally, and incongruously, the last paragraph leaves behind same-sex parents altogether and expands to suggest that family breakdown in general is responsible for “heightened dependence on public health organizations, federal and state public assistance, ” etc. This is clearly a causal statement, entirely unsupported by his evidence.
The alternative hypothesis with reference to confounding is so well rehearsed that it almost doesn’t bear repeating, but here goes. Even if we accept the empirical claims to association between LM and GF families and the outcomes measured here, if the reason why IBF outcomes are “better” is because parents who “belong” in IBFs stay there, but those who don’t, for whatever reason, leave, then increasing participation in IBF families relative to other family forms will not produce these “better” outcomes. For example, if a biological-parent heterosexual couple stays together even though there is abuse, or even though there is no sexual attraction because one or both members of the couple is gay, it does not follow that the outcomes will look the same as those of happily-married, loving couples.
The science of the article, therefore, fails for two major reasons, either of which is probably a fatal flaw on its own:
- First, the measurement of the key independent variable (having a gay or lesbian parent) is conceptually incorrect and subject to confounding through recall bias;
- Second, the analytical strategy assumes mutual exclusivity among synthetic categories that are certainly not mutually exclusive.
As far as I can tell, a perfectly adequate theory that is thoroughly consistent even with this rather strange analysis is as follows. Social policy is based around supporting the IBF. There are substantial costs–financial, emotional, and opportunity–to any lifestyle other than that. Those costs vary by time, place, and specifics, but each is marginalized in important ways. If we are to believe that the LM category means anything at all, apparently lesbian relationships are particularly associated with marginalization, again both financially and socially, and so children who remember their mothers having lesbian relationships are somewhat more likely also to report a variety of other marginal sequelae. This is consistent with Regnerus’s statement on 764, referring to children adopted by strangers: “Given that such adoptions are typically the result of considerable self-selection, it should not surprise that they display fewer differences with IBFs.” This theory, of course, directly contradicts the so-called “diminished context of kin altruism” theory discussed above, and could reasonably be taken as evidence in favor of same-sex marriage and open same-sex parenting, since each of these would tend to result in “considerable self-selection,” including into adoption. Others have made this point, including Will Saletan and Ilana Yurkiewicz.
Returning, then, to the “no-differences” hypothesis, which theoretically motivated the article and which Regnerus, in his self-styled FAQ, says he thought was “quirky”: Following this article, there is no more evidence than there was before of any systematic deficits for children of gay and lesbian parents. The article therefore consists of bad science that offers no insight into gay or lesbian parenting, the ostensible reason for its existence.
Why, then, was it published at all, and why has it received such attention? In the rest of this post I offer some thoughts on the political ecology of the article and its reception.
Several critics have noted that the research was funded by large grants from two very conservative sources, the Bradley Foundation and the Witherspoon Institute. In general, I do not believe that “follow the money” is a particularly useful way of evaluating either research or politics. Indeed, in many cases I think criticizing the funding source is a way of making an argument ad hominem, that is, discussing who’s saying something instead of what’s being said.
That said, I think there are some important issues here. First, what was the foundations’ goal in making the grants? The article says that they weren’t involved in the research design, and Regnerus’s FAQ just says: “And the Ford Foundation is a pretty liberal [institution]“, drawing the parallel. But unlike Ford, both these institutions have explicit political purposes in contemporary politics; see, for example, Bradley’s list of 2010 public policy “research” grantees, the vast majority of which are not academic research by any stretch. In other words: the foundations must have had some reason to expect that NFSS would provide ammunition for their political agendas, since they essentially only fund things that underwrite those agendas. Regnerus’s personally-stated views and track record of promoting “traditional” sexual morality are probably part of what made the funders comfortable with him, his prior most-famous item being a silly essay in Slate on sex markets on campuses, which was in turn the subject of a fabulous take-down on Crooked Timber.
Regnerus has said that government funders–generally considered the most prestigious, rigorous sources of research funding–“don’t want to touch this stuff,” “I actually don’t think a study like this would fly at NICHD or NSF.” This seems unlikely to me (see, e.g., this NIH RFA). Was there any attempt to garner support from sources without an ax to grind? I think it’s a better bet that the research design wouldn’t have passed peer review at a serious science funder such as NIH or NSF, and Regnerus preferred not to have to modify the research design in order to do so. One prominent scholar told me:
I was asked to serve on this panel [for NFSS] but declined, explaining to Regnerus that it was clear that the design of the study was biased toward finding bad outcomes for LGBT parenting. Unless there was a possibility that the basic design could be altered (which he indicated there was not), I said that I could not participate. I also questioned his qualifications for serving as a PI on this study.
In my view, all this is evidence that the study was designed to achieve results that could be spun in favor of the funders’ chosen outcomes. If I’m right about that, then the critique is not due to the funders per se, but rather due to the scientific decisions that made the funders comfortable with offering the funding.
Journal review/timing irregularities
Several critics noted that the turnaround time between submission and publication was extremely fast for social science, and way out of whack with the equivalent times for other articles in the same journal and issue. I asked the journal editor, Jim Wright, about this, and he said:
The ‘average’ time between submission and acceptance that is being bandied about in the blogosphere includes the time authors sit on their R&Rs and is therefore misleading. We always ask reviewers to get their reviews back within four to six weeks. Some do, many don’t…. Regnerus’s reviewers were quick, and so was he. (For the record, it was obvious to me that the Regnerus submission had been heavily reviewed and rewritten many times before it was submitted, as one would expect when dealing with such a hyper-charged issue.)
SSR is a respected journal (I’m proud to have published there), and Wright an accomplished scholar and editor, but I do think the speed of the reviews is so unusual as to be worth further consideration. SSR is unusual among social science journals in using single-blind review, meaning the reviewers see who the author of the submission is, and the editorial board contains at least two members (Brad Wilcox and Chris Ellison) who are generally sympathetic to the political leanings of the project and the article. Were potential reviewers primed ahead of time, either by the author or others? Were they in some way connected to the project already? Wright told me:
As you know, peer review is an anonymous process. There were seven reviews of the two papers, only one of which, so far as I know, had previous ties to the project….
One reviewer refereed both papers. And in case it matters, it now appears that two of these six had prior ties to the Regnerus project, although I was only aware of one of these at the time.
Good to know, but it does seem to me that even one reviewer with previous ties to the project is a conflict of interest on the part of the reviewer.
Philip Cohen and Neal Caren have pointed out that the timeline is not just fast but virtually impossible, since the date the final data were delivered to the project was 23 days after the original submission and just 5 days before the revised submission. Regnerus says in a response to Cohen’s post that he just submitted based on incomplete data, then added the final data in at revision time because “the story didn’t change.” This seems particularly surprising, though, given editor Wright’s observation that the manuscript had clearly been through much editing and revision before submission.
All this is to say that the publication process is very unusual for social science, but I don’t see any “smoking gun” to suggest that the journal did anything wrong. However, given the poor quality of the article itself, its reviewers clearly were asleep at the wheel. Jim Wright also told me he plans to publish “serious, civil, science-based critiques” on the article in a future issue of SSR.
Based on all the above, I think it’s safe to say the main point of the article was to provide an opportunity for political spin, which Regnerus wasted no time in launching. From a philosophy-of-science perspective, what a scholar who favors a particular hypothesis–as Regnerus clearly does–should do is to try hard to falsify that hypothesis. That, in turn, would make the surviving hypothesis that much more believable. Regnerus is, of course, far from alone in not having taken that course of action, but the fact that he didn’t lends credence to the theory that the article’s goal was to provide the funders and allies with political ammunition. As I suggested above, I think this is likely one of the reasons for choosing the fictional approach using the ever-had-a-same-sex-relationship variable instead of carrying out a serious analysis that actually addressed the real issues using data that were available in the same dataset.
The right-wing blogosphere wasted no time trumpeting the findings, at National Review (also here), Fox News, and the Heritage Foundation, among others. Many of these articles are cataloged on the NFSS homepage. Among those headlined, quite a number refer to “same-sex parents” and “gay parenting,” both of which are literally inaccurate even on the study’s own terms. Others refer to “gay” parents, which is also surely inaccurate but it’s an inaccuracy drawn from the article itself so I suppose it’s slightly more justified.
There are also lots of good, substantive, and solid critiques of the research from both political and social scientific angles, including by Philip Cohen, CJ Pascoe, Christine Woodman, and the Box Turtle Bulletin. The overwhelming sense of the journalistic findings as well (see, e.g., The New Yorker, The New Republic and Slate) are that the research is slipshod and, even if we were to accept the research, the gratuitous right-wing spin is certainly not supported. The position of Will Saletan–a thoughtful, hardly left-wing, commentator–is actually that the findings provide support for same-sex marriage, since the instability that is the likely mechanism for any demonstrated deficits would be mitigated by removing legal barriers to same-sex families. Editor Jim Wright’s “standard take,” which he sent me, is similar:
The children studied in this survey were raised in an era when it was legally impossible for their parents to form normal marital unions, when gay people were subjected to hostilities and prejudices of the worst imaginable sort, and when their children could expect to be stereotyped and vilified by their peers and others. The hypothesis that these children would not suffer lasting effects from this sort of social environment seems implausible in the extreme. I do not see that is damaging either to the parents or children to call attention to the formidable difficulties gay parents must have faced (and still face) in trying to raise their children, or to the consequences for these children that are still detectable years and even decades later. To the contrary, these strike me as precisely the realities that must be acknowledged and faced if we are ever to progress beyond our current heteronormative bigotries.
This position, of course, is utterly absent from both Regnerus’s spin–which is, essentially, that two-parent, married, opposite-sex families are the gold standard for family life and are causally related to children’s success–and from the right-wing echo chamber that has picked up the article and run with it. The defenses of the critiques have been utterly predictable. Heritage calls them “liberal intolerance,” preferring to consider it entirely as an ideological contest instead of reading the substance of the critiques. Regnerus, having begun and licensed the spin, retreats to a “who, me?” stance:
there’s no ‘Christian’ approach to sampling or ‘Catholic’ way of crunching numbers. Any trained methodologist, data manager, and statistician can locate the same patterns I reported.
But, of course, any serious sociologist would recognize that these patterns, as reported, are meaningless and misleading.
Finally, a group of sociologists, generally of religion, and who generally tend to agree with Regnerus on social matters, offer a defense of Regnerus on the grounds that, well, other studies are bad too. They call their response “social scientific,” but it is only such insofar as the authors are social scientists; no actual social science is brought to bear on the matter. They do point to an interesting, forthcoming study by Daniel Potter, which does a far better job addressing similar questions–academic achievement by children of same-sex parents–using different data. The defenders say Potter’s study “comes to conclusions that parallel those of Regnerus’s study,” but in fact the Potter study shows that all the differences observed in children of same-sex couples are fully explained by differences in family instability! In other words: stable families are good for kids, and the sexual orientation of their parents doesn’t matter. Imagine the headlines if that had been Regnerus’s finding!