Hey, long time no see. While we’ve been apart, there’s arisen a bit of a dustup in my little corner of the world about the Data Access and Research Transparency (DA-RT) initiative. In a nutshell, DA-RT represents a movement to continued discussion, implementation, and fine-tuning of standards regarding how social science research is produced and shared amongst scholars and the broader community.
In (quite belated) response, this petition dated November 3rd, 2015, requests a delay in the implementation of “DA-RT until more widespread consultation can be accomplished at, for instance, the regional meetings this year, and the organized section meetings and panels and workshops at the 2016 annual meeting.”
With the background set, a disclosure/explanation is in order: I am a coeditor of the Journal of Theoretical Politics, and hence a co-signatory on the DA-RT Journal Editors’ Transparency Statement (JETS). That’s basically why I’m writing this, particularly once one reads the petition twice and realizes that, its length and detail notwithstanding, it is entirely unclear to whom the petition is directed (other than “colleagues”).
In practical terms, is this a petition to
- Journal editors?
- Journal publishers?
- Journals’ editorial boards?
- Journal reviewers?
- The governing bodies of the various political science associations?
- Political scientists in general?
In the spirit of this blog and my own view of the world, I’ll be clear:
the absence of a clearly named target of the petition is absolutely and definitively telling: this is not a serious (or at least well thought-out) plea. Full. stop.
Delay, delay, delay. Without impugning any of the signers of the petition, it is clear to me that the petition is classic and barely disguised foot-dragging. This petition, as drafted, will do nothing to further serious dialogue about the issues at hand. Rather, it draws a (sadly, frequently and unnecessarily drawn) line in the sand between quantitative and qualitative analyses in the social sciences.
Transparency is hard for everybody. The petition states that “Achieving transparency in analytic procedures may be relatively straightforward for quantitative methods executed via software code.” Sure, it might be. But it need not be. Difficulties with implementing transparency are qualitatively common to all forms of analysis: formal, quantitative, and qualitative. Formal analysis can depend on methods, proofs, or arguments that are obscure or opaque even to many scholars. Along the same lines, both quantitative and qualitative methods can be difficult to convey in a parsimonious fashion. Finally, both quantitative and qualitative analyses can bring up questions about how to preserve anonymity of subjects, maintain incentives for the collection of new data (“embargoing”), etc.
Let’s keep talking…at, you know, some place and some time. Each of the above issues is difficult to deal with, of course. But rather than acknowledging this (clear) reality and putting something productive forward, the petition instead suggests that “we” should delay implementation
“until more widespread consultation can be accomplished at, for instance, the regional meetings this year, and the organized section meetings and panels and workshops at the 2016 annual meeting. Postponing the date of implementation will allow a discipline-wide consideration of the principles of data access and research transparency and how they should be put into practice.”
To understand why this is foot-dragging, note first this “Response by the DA-RT organizers to Discussions and Debates at the 2015 APSA Meeting” (henceforth “the Response”). Seriously, if you’re already here in this post, you should take the time to read it. It’s not that long, but it’s got a lot of information.
Finished reading it? Good. Let’s move on to what I think is the money shot of the Response, and it’s adroitly situated right in the opening:
At the 2015 Annual Meeting of the American Political Science Association in San Francisco, DA-RT and JETS were a central topic at several meetings. There were multiple workshops, roundtables, and ad hoc discussions. In addition, transparency was debated at several of the organized section business meetings. As a result, conversations about openness took place on almost every day of the Annual Meeting. As facilitators of a now five-year long dialogue on openness, we were of course delighted that the topic received such a wide airing. (Emphasis added and doubled.)
All that said, the petition asks for more discussions: “discussions” that are neither organized nor even clearly described. Just a vague call for “let’s talk some more at some of those meetings that we’ll all be at in the next year or so.”
But, wait…to stop piling on and return to the facts as stipulated by both the Response and the petition itself: such discussions have been going on for the past 5 years.
Yes, it’s tough. But the sky isn’t falling. Look, both sides of the debate are filled by smart and well-meaning scholars. Is the topic at hand—implementing the right kind(s) of transparency in research—a hard task? Yes. …And all involved acknowledge that, even if only because denying it would be ridiculous.
Any Good Transparency Standard Requires and Relies Upon Context. Why is this a hard task? Because there’s no perfect answer. Transparency is a beguiling concept, especially to scholars. To beguile implies at least a strong possibility of deception (which is ironic) and the allure of transparency fits this bill, precisely because “transparency” is like obscenity: you know it when you see it, because when you see it, you can account for the context. If a statue of a nude person is made of marble, it’s totally okay: not obscene. If you withhold data because the IRB (or contract, law) requires you to do so, or because revealing it would put people in harm’s way, that’s okay: still transparent. Just tell the editor(s) and reviewers (and, by extension) readers why. This is a collaborative enterprise, this search for knowledge and betterment. In the end, we’re in this together.
Look, This Ain’t A Democracy. Finally, and I think most importantly, note that editors can and do impose policies about topics like this. Simply put, the petition is silly because journals and their editors do (and should) have discretion: that’s why we don’t have one big “JOURNAL OF RESEARCH” that everybody publishes in.
More specifically, and as the Response states,
It is important to note that JETS does not create new powers for journal editors. Instead, it asks them to clarify or articulate decisions they are already making or attempting to manage. Journal editors have had, and will continue to have, broad discretion to choose what they will and will not publish and their basis for doing so. (Emphasis added…twice.)
This isn’t about quantitative versus qualitative. The petition draws a false, and all too commonly drawn, line in the sand. The Response—and clear thinking—makes clear that neither the issue of transparency nor reproducibility differentially impinges on scholars due to the nature of their data or their method. Data is data, method is method. Sure, the implementation details of how best to achieve transparency will vary from one study to another—but this is based on the subject, not the nature of the data or method. A method is something that can be done…you know…methodically. That doesn’t require numbers. Write down your method. Share your data to degree that is legally and ethically possible. Stop being fearful. If none of that works, ask the editor for an exception. If all of those steps fail…publish it somewhere else. You can be like John Fogerty, Trent Reznor, or Prince.
This petition is cynical. In the end, there’s no fire in that barn: somebody else is just blowing a lot of smoke from behind it. The petition is a manipulative force both playing upon and probably driven by fear. Hopefully either the Response or maybe even even this post makes clear that this fear is unwarranted.
In the end, “haters gonna hate,” and, as a corollary, “editors gonna reject.”
Neither the DA-RT initiative, nor the petition, will change either of those truisms.
With that, I leave you with this.
Pingback: Political Scientists Trying to Delay Research Transparency | Political Science Replication
Pingback: Perspectives | Dialogue on DA-RT
Hi John – We want to clarify a few things.
You’re right that we did not make clear in the letter who we meant by “Dear Colleagues” because we were imagining them reading it and were writing to them. In the cover letter we do say that the letter “will be sent to the JETS editors” (JETS being the DA-RT acronym for the editors’ statement). It’s understandable that someone reading only the letter itself it would be confused.
On the issue of “delay, delay, delay,” “foot-dragging,” and “let’s keep talking at, you know, some place and some time,” and the point that there has been a “now five-year long dialogue on openness”: we can see how anyone involved in this already long process would be frustrated.
It was only about a year ago, in October 2014, however, that the editors of several journals committed themselves to implement (by January 2016) a set of new policies that they had crafted at a meeting at the University of Michigan. That decision changed the ground considerably, because before this, the prevailing situation was one in which the researchers themselves took the responsibility for following the APSA Guide to Professional Ethics in Political Science (2012). It was only at the Michigan meeting that editors took the responsibility for defining and enforcing a particular interpretation of transparency principles.
The point of the proposed delay is not to extend the period of discussion indefinitely. It is designed only to allow for some discussion and deliberation on what exactly this means for authors.
Responding to the new situation in which editors would be policing their statement, the most recent annual meeting produced considerable controversy on many questions related to the statement. At least one editor expressed publicly his desire for greater input from the qualitative and multi-method communities regarding appropriate guidelines for these types of research. The controversy at the meeting put many issues on the table, but it was not structured to produce much clarity even as to which features of the issue might provoke fundamental conflict among members of the association and which features (such as the priority of respect for human subjects) might produce considerable consensus.
In the coming months, particularly through the process of deliberation being developed by Tim Büthe and Alan Jacobs, which was authorized at the meeting of the Qualitative and Multi-Methods Research (QMMR) section at APSA 2015, we expect that these issues will be greatly clarified. By January 15, 2016, the date announced for implementation of the journal editors’ statement, however, the reasons for and against different interpretations of the issues will not be even relatively fully articulated. A delay will allow structured deliberation over the next months and then relevant panels at the next annual meeting will clarify for the profession what is at stake and how various issues might be resolved to the at least partial satisfaction of the most affected.
On the thought that the letter will “do nothing to further serious dialogue about the issues at hand,” we think that the delay will make possible just that kind of dialogue through the QMMR process, which we believe holds great promise.
On drawing “a (sadly, frequently and unnecessarily drawn) line in the sand between quantitative and qualitative analyses in the social sciences,” we could not agree with you more that this line is far too frequently and unnecessarily drawn. This is one reason the Qualitative and Multi-Methods Research section emphasizes the concept of “multi-methods.” We do think that some interpretations of what transparency “requires” of authors poses special hurdles for certain kinds of research. The excellent Qualitative and Multi-Methods Research newsletter addresses some of these issues. But more broadly we tend to agree with Chris Blattman’s observations about the ways in which qualitative and quantitative scholarship work together.
On “just tell the editor(s) and reviewers why,” and (later) “if all of those steps fail… publish it somewhere else”: some of the papers in the QMMR newsletter demonstrate the problems created by so dramatically shifting the burden of proof to the researcher without well articulated guidelines from the profession. Journal editors are performing a hugely important task for the profession; they volunteer vast amounts of their own time for the collective. Given the heavy workload this imposes, it is easy for them to forget the enormous power they wield. Because of the important role that publication in the top journals plays in promotion and tenure, they are gatekeepers whose judgement calls shape the profession. And while it is possible for individual authors to push back if they think that what is being asked of them goes beyond the appropriate, these are negotiations under circumstances of significant power asymmetry. The author who has just spent all this time on this journal and who needs the publication (this goes especially for junior scholars) is in a much weaker position than the editor who is swamped with submissions.
On the idea that “in the end,” “editors gonna reject” anyway, of course they will. We hope, however, that they will reject in a way that is more, rather than less, informed. That is the reason for our request for a delay in implementation. For more information, check out our webpage http://dialogueondart.org.
Nancy Hirschmann (U of Penn), Mala Htun (U of New Mexico), Jane Mansbridge (Harvard), Kathleen Thelen (MIT), Elisabeth Wood (Yale)
Thanks for the reply. I hope my second post further clarifies my concerns, as an editor who has “signed” the statement, with the implications of delay. I don’t think the implementation date is the right place to focus for promoting the types of discussions that I think we all want. I definitely favor continued discussions of how we should promote transparency while upholding our other obligations. I think delaying implementation will not promote those discussions but rather muddy the waters even more. This will specifically harm junior scholars and scholars who have fewer resources, as well as differentially advantage scholars according to their methodological approach. I see the impacts of delay being potentially quite ironic, in a very sad way.
That said, I want to thank you for your efforts—we don’t agree on the details at this point, but I definitely believe that we are in pursuit of the same goals.
Pingback: One Thing Leads to Another: “Delaying“ DA-RT Standards to Discuss Better DA-RT Standards Will Be Ironic | The Math Of Politics
Pingback: DA-RT, TOP and Rolling Back Transparency | RKWRICE
Pingback: Different tools, shared standards: The debate about DA-RT | Politics, Science, Political Science
Pingback: Weekly Links | Political Violence @ a Glance