constructing quantitative data

I don’t envy those researchers who collect and analyze retrospective family history surveys. The data can be a real mess.

Mark Regnerus recently released the raw data from his New Family Structure Study (NFSS). I thought that many researchers would want to reanalyze the study, so I attempted to put together a Stata do file that replicates the original analyses that were published in Social Science Research. I’ve also posted my version of the full regression tables, which weren’t in the original article. The article is quite clear on how most of the items were constructed so it wasn’t too difficult to put this together for most of the variables.

The items that were most difficult were those dealing with family structure growing up. I haven’t been able to get my numbers to match Regnerus’s for all the different types of families. Largely this is because the item being measured is quite messy with multiple overlapping categories. If someone else can figure out how Regnerus got his numbers, specifically the step-parent and single parent categories, please send me the code and I’ll incorporate it in my publicly available do file.

An additional problem when analyzing the data is that an individual’s answers are often quite contradictory.  While sometimes this can create sociologically interesting questions, like under what conditions do people change their self-identified ethnic/racial classification, other times you just want a decent operationalization of the concept, like how many respondents lived with two moms?

Regnurus’s first article didn’t really address all the complexities involved with measuring family types. He relied on a single question for same-sex relations, whether a parent, “ever have a romantic relationship with someone of the same sex” (Question S7). 162 respondents said this applied to just their mom and 61 respondents said this applied to just their dad, and 11 said this applied to both parents.

One of the criticisms leveled at Regnerus was that this was a bad operationalization of same-sex parenting. Regnerus took this criticism seriously and, in a follow-up analysis, further split the group of people who reported having a mother involved in a same-sex relationship into those who had “spent time in residence with mother’s same-sex romantic partner (N = 85)” and those who “never lived with mother’s same-sex romantic partner (N = 90)” (page 1372).

This number is presumably based on the calendar that respondents were asked to fill out listing what years they lived with what adults when growing up (Q21). On that calendar, 85 people reported living at least four months with their “mother’s girlfriend/partner.” However—and this is where it gets tricky—a different question (S8) asked, “Did you ever live with your mother while she was in a romantic relationship with another woman?” Eight people who reported in the calendar that they lived with their mother’s girlfriend answered no to this question.

Looking at the annual calendar data for when a person reports living with their biological mother, 23 of the 85 respondents report at least one year when they lived with their mother’s girlfriend and not their biological mother. This could be because of the complexity of family life or it could be miss-clicks. For example, 4 people report living for a time with their biological dad and their mother’s girlfriend but not their mother.

It’s been widely noted that the Regnerus sample includes only two people who spent their whole lives with two moms. I think this estimate might be too high by half. Two respondents checked that they “always” lived with their “mother’s girlfriend/partner.”  However, one (case id #11118) of these two also reported he never lived with his mom while she was in a same-sex relationship (S8).  The other (#11825) reported that in addition to living with her mom and her mom’s girlfriend her whole life, she was living at least four months a year with her biological father her whole life and another unspecified person, complicating this case as well.

By my calculation, only 59 of the 85 people Regnerus counts as having, “spent time in residence with mother’s same-sex romantic partner” actually report ever living with their mom while she was dating a woman and report living with their mom and her girlfriend in the same year. In total, at most 1/3 (68 out of 236) of the people coded in the original article as having a “lesbian mother” or “gay father” report living with a parent and his/her partner while the parent were in a same-sex romantic relationship.

For some people, this messy data is likely more evidence of the instability associated with many adult relationships. Others might blame the fact that the data was gathered over the web or that a national survey is the wrong way to collect data when the category of interest is less than .5% of the population.

Another interpretation is that these retrospective questions are hard to answer and that social scientists should be quite wary in interpreting the results. For example, one of Regnerus’s interesting findings was that the children of father’s who had relationships with men are more likely to vote than others. In the weighted survey data, they are about 40% more likely to say yes to, “Did you vote in the last presidential election, in 2008?” (Q110). This effect, however, is largely driven by the fact that people were much more likely to say yes to this if they were polled during the 2012 Presidential primary season, and the NFSS didn’t interview people whose parents were married the whole time they were growing up during this last phase of the data collection.

Looking at just the subset of people who reported that their parents weren’t together their whole childhood, those polled in the final wave of the survey (based on case id number) are significantly more likely to report voting in 2008 than those who were polled earlier, with reported participation jumping from 52% to an implausible 79%. Since this late period of polling was conducted during a time in which only some states were holding elections, and those elections were relatively low-turnout Republican primaries and caucuses, it is unlikely that the effect is caused by people reporting that they voted in 2012.  Instead, it suggests to me that when everyone is talking about voting, people are likely to say they voted, even if they didn’t.

If current context so powerfully impacts reporting of relatively recent and uncontroversial events, it is hard to believe it has no effect on recall and reporting of events, attitudes and behaviors from childhood and adolescence.