As I have admitted before, I am a terrible electronic file-keeper. If I was to count up the minutes I have wasted in the last 15 years searching for files that should have been easy to find or typing and retyping Stata code that would have (and should have) been a simple do-file or doing web searches for things that I read that I thought I wanted to include in lectures or powerpoints or articles but couldn’t place, I fear I would discover many months of my life wasted as a result of my organizational ineptitude.
For a long while, these bad habits only affected me (and the occasional collaborator). It was my wasted time and effort. Now, though, expectations are changing and this type of disorganization can make or break a career. I think about my dissertation data and related files, strewn about floppy disks and disparate folders, and I feel both shame and fear. What would I do if someone asked for all my work? Would I be able to produce what they would need, to explain how things are coded, to share the program I ran in the lab? If it weren’t for all the detail that I actually include in my dissertation – significantly more detail than I normally include in my work given page and time constraints – could I replicate what I had done? More importantly, could someone else?
When I’ve occasionally asked others over the years if they’ll share notes or data and they have been unable to because of changing technology or office moves or some other excuse, I haven’t batted an eye. That was then; this is now. Given recent controversies (and gross errors) in academia, and a related shift toward more transparency in research and data-sharing in the sciences, effectively managing workflow (and making more than the final product available to others) has transformed from a useful skill to a fundamental job requirement for both qualitative and quantitative scholars.
I am trying to do better in my own life and I hope to get to my students before they have time to acquire bad habits. To help students learn to manage workflow effectively, I would like to incorporate relevant techniques into our graduate training. Scott Long’s book, The Workflow of Data Analysis Using Stata, recommended by a colleague and mentioned previously here, is an excellent place to start. Long covers planning, organizing, and documenting – from naming files to writing research logs – in addition to how to write do-files and automate much of your work. With his insight, files are clean and orderly and ready to share. I highly recommend this as a place for anyone interested in changing their ways – or learning the trade – to start and I wish I had the time to offer a course like he does or to incorporate the entire book into our Proseminar or Stats sequence. In the meantime, and as a complement, I am on the hunt for other resources, helpful tips, and ideas about how best to weave these topics into graduate education. Do readers have examples of where they’ve learned these habits, or where they wish they had? Best practices for qualitative researchers, to complement the approaches geared toward quantitative methods? Alternatives to full-fledged courses as way to teach these things (e.g., summer reading groups)?