How Would We Know If Scholarship Matters?
-- Posted by Neil H. Buchanan
This past weekend, I attended a conference, "Legal Scholarship We Like and Why It Matters," which was held at the University of Miami School of Law. The conference celebrated the fifth anniversary of Jotwell - The Journal of Things We Like (Lots), to which I contribute one article each year. (My Dorf on Law post discussing my most recent piece is here. The first paragraph includes links to previous years' writings.)
The initial Call for Papers for the Jotwell conference provided three broad categories for paper topics, with two questions under each category. I chose "III. Improving the World: Legal Scholarship and Its Influence," and Question 5: "What makes legal scholarship influential? Note that influence is not necessarily the same as 'greatness'. Also, influence has many possible meanings, encompassing influence within or outside the academy." My task, then, was to write a paper that describes how legal scholarship can improve the world, and that further describes how to determine what has been influential.
My written piece is now available online. (Jotwell emphasizes brevity and clarity, so my piece is roughly 2600 words, with no footnotes.) Here, I want to summarize my oral comments at the conference, in which I tried to describe ways to think about the basic question: How would a scholar know that her scholarship has had a practical impact on the world?
It is important to emphasize that my question could be incorrectly read to suggest that I think that "practical impact" is the only way that scholarship can "improve the world." I most definitely and emphatically reject that notion. I strongly believe in "basic research" across all fields, in which any possible practical impacts of the work are, at best, not obvious to anyone, and probably will not be obvious for a long time. Not just for the scholars themselves, but for society as a whole, "impractical scholarship" is important and should be supported.
For my purposes, therefore, I am simply asking about a subset of valuable scholarship -- scholarship that arguably has had an important effect on real-world outcomes, especially (in the case of legal scholarship) policy outcomes. How would we know if or when a piece of scholarship (or a body of work) has affected the world? In my paper, I note in passing several well-known examples of scholarship that has undeniably changed the legal/policy landscape, including among others Catherine MacKinnon's work on gender equality that all but single handedly created the law of sexual harassment, hostile work environment causes of action, and so on; and Elizabeth Warren's work on personal bankruptcies, which ultimately led (in large part through dogged political advocacy on now-Senator Warren's part) to the creation of the Consumer Financial Protection Bureau (CFPB).
Those examples (and others), however, merely make cameo appearances at the end of my article. The bulk of the article is devoted to trying to understand how to determine whether my own work (co-authored with Professor Dorf), from 2011 onward, regarding the federal debt ceiling has "mattered" in some sense. I focus on that example in part, of course, because it is something that I care deeply about and with which I am intimately familiar. More importantly, however, I think that my being a co-author of these pieces allows me to think about how our work might actually matter -- or not.
And it really might not matter (at least in the practical sense to which I am limiting the discussion here). After all, the first-best response that Professor Dorf and I would have hoped for -- a statement by President Obama that our argument had convinced him that he would not allow a default on federal obligations, even if the Republicans were to refuse to increase the debt ceiling -- has not happened. Even with the prospect of another standoff over the debt ceiling looming in Spring 2015 (a prospect that is reduced, but not eliminated, by the post-election happy talk from Mitch McConnell and other Republicans about "responsible governing"), it is unlikely that the White House will ever take the Buchanan-Dorf position as its own.
Of course, that was true for both MacKinnon and Warren as well. Neither of them, I am sure, are happy with the state of the law now, despite the important changes that they helped to bring about. Then-Professor Warren's work, in particular, was followed not by a recognition of the realities of consumer bankruptcies and their devastating effects (especially on women and children), but by passage of a draconian bankruptcy law that made matters worse on nearly every dimension that Warren cares about. Even the CFPB has been hobbled by Republicans, so that Warren's signature practical accomplishment is deeply disappointing, even though it is an important achievement.
The point is that "practical impact" is not the same as "changing things exactly as the author wishes." Which brings us back to the Buchanan-Dorf line of work. As I note in the article, another way to infer that our work might have had an impact is that the Republicans started to act in ways that suggested that they were worried about our arguments. (And we really were the only people making those arguments, so this was a case in which it is not difficult to figure out whose arguments matter.) We had argued that one of the most important constitutional constraints on the President was that he is required to minimize the amount of legislative-like actions that he takes, which in this case means that he is not permitted to pick and choose which legal obligations the government will pay, and which he will choose to ignore.
Republicans had been arguing that the President can "prioritize" payments to the holders of Treasury bonds, so that there would be no "default," because the only people who would be stiffed by the government would not be people who loaned money to the government, but "merely" people who had been legally promised payments on a specific date and in a specific amount. There are plenty of holes in that line of reasoning, which Professor Dorf and I have discussed at length, but the key point here is that House Republicans ended up introducing bills that would allow the President to prioritize bondholders over other obligees. If those bills had passed, however, they would have amounted to Congressional action to change spending obligations, not unilateral Presidential action, which would have satisfied our constitutional analysis. In our formulation, the prioritization bill would have amounted to a new spending law, meaning that there would be no "trilemma," because the taxing, spending, and borrowing laws would then be consistent with one another. No presidential choices would be required at all.
Therefore, even though our first-best outcome did not happen, there is evidence suggesting that our scholarship had a practical impact. In fact, it is arguable that this impact might actually have been more important than a Rose Garden ceremony in which the President puts his arms around Professor Dorf and me and endorses our argument. If Republicans were in any way pulling back from their brinksmanship strategy, because they were losing the war of ideas, then that would count as "impact." I do not mean to overstate the case, but only to say that there are ways in which scholarship can change politicians' choices and strategies, which might well have happened in this case.
A different way of thinking about the impact of scholarship is in its effect on the scholarly conversation. I noted in my spoken remarks in Miami that the "law and economics movement" had had a great deal of effect on in-group scholarship, but that the evidence of its effect on practical outcomes is rather thin, to say the least. (Have, for example, economic theories justifying a "punitive multiple" changed the computations of punitive damages?) In that sense, then, it is possible to have a huge scholarly impact but virtually no real-world importance.
The Buchanan-Dorf line of articles is arguably an example of the opposite case -- scholarship that is "practically important" but that has (thus far) had essentially no impact on the scholarly conversation. As Professor Dorf and I have noted many times, no one (and certainly not the law professors who are most often quoted in news reports, denigrating our conclusions) has actually produced any scholarship that engages with our analysis. I should note that this Thursday (11/13), my Dorf on Law post will discuss a forthcoming law review article that attempts to engage with our arguments; but as readers will see there, the disagreement ends up being less important than it might seem. In any event, one response in two years would normally be taken as evidence of "no impact" on scholarship.
There is, however, another possibility. (I concede that this will sound arrogant, but that is not my point.) What if our arguments are so convincing that other scholars have simply said, in essence, "Nothing more need be said"? Then, the impact of the line of scholarship would not be measured in the usual ways -- especially citation counts -- but by the utter silence that has ensued.
Of course, there is a danger in reading meaning into non-events. Some scholarship is ignored because it says nothing interesting, or because the arguments do not make sense. On the other hand, some "high impact" scholarship is simply nothing more than absurd, provocative assertions that are so ridiculous that everyone cites them. ("For a crazy argument to the contrary, see ... ") As everyone knows, citations (and certainly downloads) are a very unsatisfying way to measure impact.
More broadly, my argument here is that the practical impact of scholarly work is generally very difficult to track. The Buchanan-Dorf work is a helpful prism in evaluating these questions, precisely because it was so unique and because it dealt with an issue of almost literal economic life or death. Even there, however, the ways in which we might measure how scholarship matters are muddy, to say the least.
Finally, a comment by Professor Donna Coker at the conference deserves mention. She noted that one never knows when, or where, any piece of scholarsip might have an important effect. In part, she is arguing that "absence of evidence is not evidence of absence," because one might not always know where to look to find evidence of scholarly effects on real-world outcomes. Moreover, there is no guarantee that something that seems to have had no impact thus far will not ultimately matter a great deal. The lag time of Elizabeth Warren's real-world impact, after all, is measured in decades.
In the end, the interesting questions are not ultimately about the articles that Professor Dorf and I have written. Those are merely a window into the broader issues. What I find most interesting is how difficult it is to know, even if one limits one's ambitions to "having practical effects in the real world," whether and how scholarship matters.
This past weekend, I attended a conference, "Legal Scholarship We Like and Why It Matters," which was held at the University of Miami School of Law. The conference celebrated the fifth anniversary of Jotwell - The Journal of Things We Like (Lots), to which I contribute one article each year. (My Dorf on Law post discussing my most recent piece is here. The first paragraph includes links to previous years' writings.)
The initial Call for Papers for the Jotwell conference provided three broad categories for paper topics, with two questions under each category. I chose "III. Improving the World: Legal Scholarship and Its Influence," and Question 5: "What makes legal scholarship influential? Note that influence is not necessarily the same as 'greatness'. Also, influence has many possible meanings, encompassing influence within or outside the academy." My task, then, was to write a paper that describes how legal scholarship can improve the world, and that further describes how to determine what has been influential.
My written piece is now available online. (Jotwell emphasizes brevity and clarity, so my piece is roughly 2600 words, with no footnotes.) Here, I want to summarize my oral comments at the conference, in which I tried to describe ways to think about the basic question: How would a scholar know that her scholarship has had a practical impact on the world?
It is important to emphasize that my question could be incorrectly read to suggest that I think that "practical impact" is the only way that scholarship can "improve the world." I most definitely and emphatically reject that notion. I strongly believe in "basic research" across all fields, in which any possible practical impacts of the work are, at best, not obvious to anyone, and probably will not be obvious for a long time. Not just for the scholars themselves, but for society as a whole, "impractical scholarship" is important and should be supported.
For my purposes, therefore, I am simply asking about a subset of valuable scholarship -- scholarship that arguably has had an important effect on real-world outcomes, especially (in the case of legal scholarship) policy outcomes. How would we know if or when a piece of scholarship (or a body of work) has affected the world? In my paper, I note in passing several well-known examples of scholarship that has undeniably changed the legal/policy landscape, including among others Catherine MacKinnon's work on gender equality that all but single handedly created the law of sexual harassment, hostile work environment causes of action, and so on; and Elizabeth Warren's work on personal bankruptcies, which ultimately led (in large part through dogged political advocacy on now-Senator Warren's part) to the creation of the Consumer Financial Protection Bureau (CFPB).
Those examples (and others), however, merely make cameo appearances at the end of my article. The bulk of the article is devoted to trying to understand how to determine whether my own work (co-authored with Professor Dorf), from 2011 onward, regarding the federal debt ceiling has "mattered" in some sense. I focus on that example in part, of course, because it is something that I care deeply about and with which I am intimately familiar. More importantly, however, I think that my being a co-author of these pieces allows me to think about how our work might actually matter -- or not.
And it really might not matter (at least in the practical sense to which I am limiting the discussion here). After all, the first-best response that Professor Dorf and I would have hoped for -- a statement by President Obama that our argument had convinced him that he would not allow a default on federal obligations, even if the Republicans were to refuse to increase the debt ceiling -- has not happened. Even with the prospect of another standoff over the debt ceiling looming in Spring 2015 (a prospect that is reduced, but not eliminated, by the post-election happy talk from Mitch McConnell and other Republicans about "responsible governing"), it is unlikely that the White House will ever take the Buchanan-Dorf position as its own.
Of course, that was true for both MacKinnon and Warren as well. Neither of them, I am sure, are happy with the state of the law now, despite the important changes that they helped to bring about. Then-Professor Warren's work, in particular, was followed not by a recognition of the realities of consumer bankruptcies and their devastating effects (especially on women and children), but by passage of a draconian bankruptcy law that made matters worse on nearly every dimension that Warren cares about. Even the CFPB has been hobbled by Republicans, so that Warren's signature practical accomplishment is deeply disappointing, even though it is an important achievement.
The point is that "practical impact" is not the same as "changing things exactly as the author wishes." Which brings us back to the Buchanan-Dorf line of work. As I note in the article, another way to infer that our work might have had an impact is that the Republicans started to act in ways that suggested that they were worried about our arguments. (And we really were the only people making those arguments, so this was a case in which it is not difficult to figure out whose arguments matter.) We had argued that one of the most important constitutional constraints on the President was that he is required to minimize the amount of legislative-like actions that he takes, which in this case means that he is not permitted to pick and choose which legal obligations the government will pay, and which he will choose to ignore.
Republicans had been arguing that the President can "prioritize" payments to the holders of Treasury bonds, so that there would be no "default," because the only people who would be stiffed by the government would not be people who loaned money to the government, but "merely" people who had been legally promised payments on a specific date and in a specific amount. There are plenty of holes in that line of reasoning, which Professor Dorf and I have discussed at length, but the key point here is that House Republicans ended up introducing bills that would allow the President to prioritize bondholders over other obligees. If those bills had passed, however, they would have amounted to Congressional action to change spending obligations, not unilateral Presidential action, which would have satisfied our constitutional analysis. In our formulation, the prioritization bill would have amounted to a new spending law, meaning that there would be no "trilemma," because the taxing, spending, and borrowing laws would then be consistent with one another. No presidential choices would be required at all.
Therefore, even though our first-best outcome did not happen, there is evidence suggesting that our scholarship had a practical impact. In fact, it is arguable that this impact might actually have been more important than a Rose Garden ceremony in which the President puts his arms around Professor Dorf and me and endorses our argument. If Republicans were in any way pulling back from their brinksmanship strategy, because they were losing the war of ideas, then that would count as "impact." I do not mean to overstate the case, but only to say that there are ways in which scholarship can change politicians' choices and strategies, which might well have happened in this case.
A different way of thinking about the impact of scholarship is in its effect on the scholarly conversation. I noted in my spoken remarks in Miami that the "law and economics movement" had had a great deal of effect on in-group scholarship, but that the evidence of its effect on practical outcomes is rather thin, to say the least. (Have, for example, economic theories justifying a "punitive multiple" changed the computations of punitive damages?) In that sense, then, it is possible to have a huge scholarly impact but virtually no real-world importance.
The Buchanan-Dorf line of articles is arguably an example of the opposite case -- scholarship that is "practically important" but that has (thus far) had essentially no impact on the scholarly conversation. As Professor Dorf and I have noted many times, no one (and certainly not the law professors who are most often quoted in news reports, denigrating our conclusions) has actually produced any scholarship that engages with our analysis. I should note that this Thursday (11/13), my Dorf on Law post will discuss a forthcoming law review article that attempts to engage with our arguments; but as readers will see there, the disagreement ends up being less important than it might seem. In any event, one response in two years would normally be taken as evidence of "no impact" on scholarship.
There is, however, another possibility. (I concede that this will sound arrogant, but that is not my point.) What if our arguments are so convincing that other scholars have simply said, in essence, "Nothing more need be said"? Then, the impact of the line of scholarship would not be measured in the usual ways -- especially citation counts -- but by the utter silence that has ensued.
Of course, there is a danger in reading meaning into non-events. Some scholarship is ignored because it says nothing interesting, or because the arguments do not make sense. On the other hand, some "high impact" scholarship is simply nothing more than absurd, provocative assertions that are so ridiculous that everyone cites them. ("For a crazy argument to the contrary, see ... ") As everyone knows, citations (and certainly downloads) are a very unsatisfying way to measure impact.
More broadly, my argument here is that the practical impact of scholarly work is generally very difficult to track. The Buchanan-Dorf work is a helpful prism in evaluating these questions, precisely because it was so unique and because it dealt with an issue of almost literal economic life or death. Even there, however, the ways in which we might measure how scholarship matters are muddy, to say the least.
Finally, a comment by Professor Donna Coker at the conference deserves mention. She noted that one never knows when, or where, any piece of scholarsip might have an important effect. In part, she is arguing that "absence of evidence is not evidence of absence," because one might not always know where to look to find evidence of scholarly effects on real-world outcomes. Moreover, there is no guarantee that something that seems to have had no impact thus far will not ultimately matter a great deal. The lag time of Elizabeth Warren's real-world impact, after all, is measured in decades.
In the end, the interesting questions are not ultimately about the articles that Professor Dorf and I have written. Those are merely a window into the broader issues. What I find most interesting is how difficult it is to know, even if one limits one's ambitions to "having practical effects in the real world," whether and how scholarship matters.