Isn't a Little Legal Assistance Better Than None At All? Maybe Not
-- Posted by Neil H. Buchanan
One of the little-known duties of tenured faculty is to review the scholarship of our colleagues, for the purposes of various internal reviews, consideration of submitted articles and books for scholarly journals and academic publishers, and so on. This often seems like a burden, in that it distracts from one's own teaching and writing, and the scholarship to be reviewed is frequently in fields of law with which one is not intimately familiar. Often, however, the opportunity to see what is happening outside of one's own corner of the academic universe can be an unexpected pleasure.
Last week, I reviewed an article by one of my GW Law colleagues, Professor Jessica K. Steinberg, who teaches in our clinics. Prior to her arrival at GW a few years ago, she was on the legal staff of Stanford's Community Law Clinic, working on landlord-tenant disputes and other issues involving low-income clients. Some of that work inspired her to write In Pursuit of Justice? Case Outcomes and the Delivery of Unbundled Legal Services, 18 Geo. J. of Poverty Law & Policy (2011). That article is a pleasure to read, and it raises some interesting questions about the delivery of legal aid services and the difficult choices that well-meaning advocates must make in a time of ever-tighter budgets for low-income legal assistance.
Professor Steinberg's article is, I think, an outstanding example of the value of marrying careful legal analysis with data-driven inquiry. The article sets out a series of questions about the efficacy of the provision of “unbundled” legal services to the poor, which Professor Steinberg describes as the ethically-dubious (but increasingly popular) efforts in many jurisdictions to give some basic legal assistance to poor clients, usually at the initial stages of a dispute, but without the follow-through (to the litigation phase and after, if necessary) that is the norm for legal representation of the non-poor.
The movement to provide unbundled services, Steinberg explains, is a well-motivated effort to extract the most benefit possible from the very limited pool of legal resources that is available for non-paying clients. In other words, this is not a conservative-liberal dispute, but rather a question about how to grapple with the consequences of conservatives' having won the budget wars.
Given the increasingly severe limitations on legal aid budgets over the last few decades, a new idea emerged: If we cannot give everyone the full "bundle" of legal assistance that a lawyer would typically be expected to provide, perhaps we should instead allow lawyers to provide only small doses of legal advice, on the plausible theory that a little bit of legal assistance is better than none at all. For example, in a deeply resource-constrained environment, one might well believe that giving each of five clients one-fifth of the legal services that could instead be devoted to one client is likely to result in “more justice” in the aggregate, with four people who would almost surely have lost their cases now having at least a fighting chance to win. Some will still lose, but the odds will have moved in their favor.
I confess that I find the intuition behind this theory to be quite strong – and, other than the concerns about legal ethics (which, again, normally require a lawyer to provide full soup-to-nuts representation to her clients), Professor Steinberg reports that many in the poverty law community have embraced the strategy. The core of the idea is that an untutored person, intimidated by the legal system, would not even know where to begin to pursue her case, and would thus be likely to give up without taking even the most basic measures that might have worked to her benefit. If a legal aid lawyer could only provide the client with, say, an hour’s worth of advice, offering even the most basic information about how to file forms and what to expect as the process unfolds, it seems intuitively obvious that at least some clients would find that they could then win on their own.
The visual image that comes to mind is teaching a child to ride a bicycle. With just a little bit of explanation, some encouragement and confidence building, and a few seconds of having an adult run alongside the bicycle, most children find that staying upright is easier than they thought it would be. Sure, some will still fall and skin their knees, but the success rate is high. Especially at the earliest stages of a legal problem, it seems similarly likely that a lawyer's brief intervention is all that would be necessary, for many disputes.
Professor Steinberg questions that presumption, suggesting that we at least need to be aware of the possibility that the legal process might actually require good representation throughout, such that even a person who is assisted in starting the process will lose at the next stage, or the stage after that. To use a completely different metaphor, legal representation might be like a bridge across a river. If the river is a mile wide, and we build only one-tenth of a mile of bridge, people who try to cross the river will still eventually get wet, and many will be swept away.
Having challenged the conventional wisdom, showing that what looked like a clearly optimal strategy might not be such an obvious winner, Professor Steinberg properly concludes that this is a question ripe for empirical inquiry. Can we track whether people who receive unbundled services experience measurably better outcomes than those who receive no aid at all (and, obviously as a related hypothesis, worse outcomes than those who receive the full bundle of legal assistance)?
In the core section of her article, Professor Steinberg reports preliminary results of a modest empirical study, in which she “reviewed and analyzed all 474 evictions filed in San Mateo County from May 19, 2009 to August 7, 2009, plus 20 eviction cases handled by the Stanford Community Law Clinic between September 2007 and May 2009” (article at 480). Her findings are surprising and sobering. She concludes that, although some intermediate outcomes improve for the recipients of unbundled services, the outcomes that ultimately matter – evictions, and money paid to (or received from) landlords – stay the same. The clients, it seems, really are led to the edge of the metaphorical unfinished bridge, at which point they inevitably plummet into the river. (I confess that I have tried to figure out a way to combine my metaphors, imagining a lawyer pushing a poor client on a bike off the edge of an unfinished bridge, but sometimes metaphors must simply to be abandoned, once they have served their purpose. I think it was when I was going to add sharks to the waters below that I knew things had gone too far.)
Professor Steinberg's statistical analysis is straightforward, comparing the experiences of the different groups of parties with respect to outcome-based questions (successfully avoiding eviction, money exchanged between landlords and tenants), showing that there is no statistically significant difference (based on chi-square tests) between the outcomes of people who receive no assistance versus those who receive unbundled assistance. Professor Steinberg then compares the interim success of the different groups, showing that the recipients of unbundled services do avoid some procedural errors, but that ultimately it does not matter.
She also demonstrates that the small, lucky group of people who do receive the full bundle of services really does experience dramatically better outcomes than either of the other two groups. She does not emphasize that point, I suspect because that outcome seems so obvious, but it is important to note (as I discuss below) that we are not talking about a client base that loses all the time, no matter whether they have lawyers or not.
With the dataset that Professor Steinberg assembled, there is no reason to go for overkill with a multiple regression analysis, and her approach gleans plenty of interesting information from those data. The more fundamental question is whether her data can be treated statistically as if they were from a randomized experiment, which is what we implicitly assume when we apply various tests of statistical significance. If the data are not actually randomized, then it might be the case that we could learn nothing at all from the analysis.
Appropriately, Professor Steinberg explains that the data could be tainted by self-selection bias. Essentially, she suggests that there could be a good reason that the recipients of unbundled services ultimately lose their cases: Their cases might be losers! It is possible, after all, that the people who are completely unrepresented do not seek representation because they have figured out a good way to proceed, and there is an obvious merits-based argument on which they could win. Perhaps only those people who see no good arguments on their own then apply for unbundled representation. If that were the case, it would actually be an extraordinary affirmation of the power of unbundled legal services that the partially lawyered clients could achieve outcomes as good as (or, more accurately, no worse than) the people who actually had good and obvious arguments in their favor in the first place.
Professor Steinberg acknowledges (even highlights) this potential problem, but she argues (and I agree) that the results of her statistical comparisons are still useful. She writes: “The purpose of the study was not to reach incontrovertible or generalizable conclusions about the provision of unbundled legal services, but instead to use the resources available, and an ethically-feasible methodological design …, to make a preliminary assessment of the efficacy of one iteration of the unbundled model” (article at 457) (emphasis added). Having acknowledged that the non-randomization of the data offers an alternative hypothesis to explain her findings, she has set the stage for further research, providing “at least some evidence-backed information about [unbundled legal services’] impact on procedural and substantive justice for litigants” (id.).
I would analogize this to the “burden-switching” rule in Title VII litigation. If we see an employment category in which 90% of the workers are men, then the employer is obligated to provide a non-discriminatory explanation for the disparity. If such an explanation is provided, then further inquiry is necessary. Similarly, in empirical research, one does not say, “Well, you have not taken everything into account in this single study, so we will simply ignore your findings.” Instead, when we identify a statistically interesting result, we hypothesize about possible alternative explanations, precisely because the initial finding piques our curiosity.
As an alternative to Professor Steinberg’s findings, imagine that she had instead found statistical evidence that the partially represented clients won as often as fully represented clients do, which is four times higher in the case of evictions than completely unrepresented parties’ results. (See article at 483, Table 1). That would be such a strong confirmation of the “bicycle analogy” above that it would lead to a very different research agenda than her actual findings suggest. More importantly, for people who must make decisions about real-life policies in the face of incomplete information, this alternative set of findings would have provided extraordinarily strong reason to redirect resources into unbundled services.
This difference in the legal outcomes for the fully lawyered clients also offers a reason to think that the self-selection possibility described above is not biasing the results. After all, if the idea is that the people who do not bother to go to the clinic are the ones with the stronger cases, then the people who walk through the clinic doors all have presumptively weaker cases. If the full bundle of legal services is so much more successful than unbundled services (and no services at all), however, then the problem cannot be that the cases cannot be won. We might then reasonably infer that it really is the provision of bundled services that changes the outcomes so dramatically.
In any event, Professor Steinberg does not at all over-claim in the paper, and she fully acknowledges that this is the beginning of a potentially important inquiry, not the end. Even so, her findings would be extremely important for any policymaker to take into account when allocating scarce legal aid resources, at least offering reasons for skepticism in the face of suggestions to expand unbundled legal services.
In short, this article is only as imperfect as every other empirical study inevitably must be. I am often skeptical of what now counts as "empirical legal studies," because so much of that work (like the work on which it is modeled from Economics departments) is technique-obsessed, focusing on the latest statistical fads rather than thinking carefully about the underlying policy questions. Because of Professor Steinberg's article, we know more about an important policy question than we used to know. That is the ultimate test of good scholarship.
One of the little-known duties of tenured faculty is to review the scholarship of our colleagues, for the purposes of various internal reviews, consideration of submitted articles and books for scholarly journals and academic publishers, and so on. This often seems like a burden, in that it distracts from one's own teaching and writing, and the scholarship to be reviewed is frequently in fields of law with which one is not intimately familiar. Often, however, the opportunity to see what is happening outside of one's own corner of the academic universe can be an unexpected pleasure.
Last week, I reviewed an article by one of my GW Law colleagues, Professor Jessica K. Steinberg, who teaches in our clinics. Prior to her arrival at GW a few years ago, she was on the legal staff of Stanford's Community Law Clinic, working on landlord-tenant disputes and other issues involving low-income clients. Some of that work inspired her to write In Pursuit of Justice? Case Outcomes and the Delivery of Unbundled Legal Services, 18 Geo. J. of Poverty Law & Policy (2011). That article is a pleasure to read, and it raises some interesting questions about the delivery of legal aid services and the difficult choices that well-meaning advocates must make in a time of ever-tighter budgets for low-income legal assistance.
Professor Steinberg's article is, I think, an outstanding example of the value of marrying careful legal analysis with data-driven inquiry. The article sets out a series of questions about the efficacy of the provision of “unbundled” legal services to the poor, which Professor Steinberg describes as the ethically-dubious (but increasingly popular) efforts in many jurisdictions to give some basic legal assistance to poor clients, usually at the initial stages of a dispute, but without the follow-through (to the litigation phase and after, if necessary) that is the norm for legal representation of the non-poor.
The movement to provide unbundled services, Steinberg explains, is a well-motivated effort to extract the most benefit possible from the very limited pool of legal resources that is available for non-paying clients. In other words, this is not a conservative-liberal dispute, but rather a question about how to grapple with the consequences of conservatives' having won the budget wars.
Given the increasingly severe limitations on legal aid budgets over the last few decades, a new idea emerged: If we cannot give everyone the full "bundle" of legal assistance that a lawyer would typically be expected to provide, perhaps we should instead allow lawyers to provide only small doses of legal advice, on the plausible theory that a little bit of legal assistance is better than none at all. For example, in a deeply resource-constrained environment, one might well believe that giving each of five clients one-fifth of the legal services that could instead be devoted to one client is likely to result in “more justice” in the aggregate, with four people who would almost surely have lost their cases now having at least a fighting chance to win. Some will still lose, but the odds will have moved in their favor.
I confess that I find the intuition behind this theory to be quite strong – and, other than the concerns about legal ethics (which, again, normally require a lawyer to provide full soup-to-nuts representation to her clients), Professor Steinberg reports that many in the poverty law community have embraced the strategy. The core of the idea is that an untutored person, intimidated by the legal system, would not even know where to begin to pursue her case, and would thus be likely to give up without taking even the most basic measures that might have worked to her benefit. If a legal aid lawyer could only provide the client with, say, an hour’s worth of advice, offering even the most basic information about how to file forms and what to expect as the process unfolds, it seems intuitively obvious that at least some clients would find that they could then win on their own.
The visual image that comes to mind is teaching a child to ride a bicycle. With just a little bit of explanation, some encouragement and confidence building, and a few seconds of having an adult run alongside the bicycle, most children find that staying upright is easier than they thought it would be. Sure, some will still fall and skin their knees, but the success rate is high. Especially at the earliest stages of a legal problem, it seems similarly likely that a lawyer's brief intervention is all that would be necessary, for many disputes.
Professor Steinberg questions that presumption, suggesting that we at least need to be aware of the possibility that the legal process might actually require good representation throughout, such that even a person who is assisted in starting the process will lose at the next stage, or the stage after that. To use a completely different metaphor, legal representation might be like a bridge across a river. If the river is a mile wide, and we build only one-tenth of a mile of bridge, people who try to cross the river will still eventually get wet, and many will be swept away.
Having challenged the conventional wisdom, showing that what looked like a clearly optimal strategy might not be such an obvious winner, Professor Steinberg properly concludes that this is a question ripe for empirical inquiry. Can we track whether people who receive unbundled services experience measurably better outcomes than those who receive no aid at all (and, obviously as a related hypothesis, worse outcomes than those who receive the full bundle of legal assistance)?
In the core section of her article, Professor Steinberg reports preliminary results of a modest empirical study, in which she “reviewed and analyzed all 474 evictions filed in San Mateo County from May 19, 2009 to August 7, 2009, plus 20 eviction cases handled by the Stanford Community Law Clinic between September 2007 and May 2009” (article at 480). Her findings are surprising and sobering. She concludes that, although some intermediate outcomes improve for the recipients of unbundled services, the outcomes that ultimately matter – evictions, and money paid to (or received from) landlords – stay the same. The clients, it seems, really are led to the edge of the metaphorical unfinished bridge, at which point they inevitably plummet into the river. (I confess that I have tried to figure out a way to combine my metaphors, imagining a lawyer pushing a poor client on a bike off the edge of an unfinished bridge, but sometimes metaphors must simply to be abandoned, once they have served their purpose. I think it was when I was going to add sharks to the waters below that I knew things had gone too far.)
Professor Steinberg's statistical analysis is straightforward, comparing the experiences of the different groups of parties with respect to outcome-based questions (successfully avoiding eviction, money exchanged between landlords and tenants), showing that there is no statistically significant difference (based on chi-square tests) between the outcomes of people who receive no assistance versus those who receive unbundled assistance. Professor Steinberg then compares the interim success of the different groups, showing that the recipients of unbundled services do avoid some procedural errors, but that ultimately it does not matter.
She also demonstrates that the small, lucky group of people who do receive the full bundle of services really does experience dramatically better outcomes than either of the other two groups. She does not emphasize that point, I suspect because that outcome seems so obvious, but it is important to note (as I discuss below) that we are not talking about a client base that loses all the time, no matter whether they have lawyers or not.
With the dataset that Professor Steinberg assembled, there is no reason to go for overkill with a multiple regression analysis, and her approach gleans plenty of interesting information from those data. The more fundamental question is whether her data can be treated statistically as if they were from a randomized experiment, which is what we implicitly assume when we apply various tests of statistical significance. If the data are not actually randomized, then it might be the case that we could learn nothing at all from the analysis.
Appropriately, Professor Steinberg explains that the data could be tainted by self-selection bias. Essentially, she suggests that there could be a good reason that the recipients of unbundled services ultimately lose their cases: Their cases might be losers! It is possible, after all, that the people who are completely unrepresented do not seek representation because they have figured out a good way to proceed, and there is an obvious merits-based argument on which they could win. Perhaps only those people who see no good arguments on their own then apply for unbundled representation. If that were the case, it would actually be an extraordinary affirmation of the power of unbundled legal services that the partially lawyered clients could achieve outcomes as good as (or, more accurately, no worse than) the people who actually had good and obvious arguments in their favor in the first place.
Professor Steinberg acknowledges (even highlights) this potential problem, but she argues (and I agree) that the results of her statistical comparisons are still useful. She writes: “The purpose of the study was not to reach incontrovertible or generalizable conclusions about the provision of unbundled legal services, but instead to use the resources available, and an ethically-feasible methodological design …, to make a preliminary assessment of the efficacy of one iteration of the unbundled model” (article at 457) (emphasis added). Having acknowledged that the non-randomization of the data offers an alternative hypothesis to explain her findings, she has set the stage for further research, providing “at least some evidence-backed information about [unbundled legal services’] impact on procedural and substantive justice for litigants” (id.).
I would analogize this to the “burden-switching” rule in Title VII litigation. If we see an employment category in which 90% of the workers are men, then the employer is obligated to provide a non-discriminatory explanation for the disparity. If such an explanation is provided, then further inquiry is necessary. Similarly, in empirical research, one does not say, “Well, you have not taken everything into account in this single study, so we will simply ignore your findings.” Instead, when we identify a statistically interesting result, we hypothesize about possible alternative explanations, precisely because the initial finding piques our curiosity.
As an alternative to Professor Steinberg’s findings, imagine that she had instead found statistical evidence that the partially represented clients won as often as fully represented clients do, which is four times higher in the case of evictions than completely unrepresented parties’ results. (See article at 483, Table 1). That would be such a strong confirmation of the “bicycle analogy” above that it would lead to a very different research agenda than her actual findings suggest. More importantly, for people who must make decisions about real-life policies in the face of incomplete information, this alternative set of findings would have provided extraordinarily strong reason to redirect resources into unbundled services.
This difference in the legal outcomes for the fully lawyered clients also offers a reason to think that the self-selection possibility described above is not biasing the results. After all, if the idea is that the people who do not bother to go to the clinic are the ones with the stronger cases, then the people who walk through the clinic doors all have presumptively weaker cases. If the full bundle of legal services is so much more successful than unbundled services (and no services at all), however, then the problem cannot be that the cases cannot be won. We might then reasonably infer that it really is the provision of bundled services that changes the outcomes so dramatically.
In any event, Professor Steinberg does not at all over-claim in the paper, and she fully acknowledges that this is the beginning of a potentially important inquiry, not the end. Even so, her findings would be extremely important for any policymaker to take into account when allocating scarce legal aid resources, at least offering reasons for skepticism in the face of suggestions to expand unbundled legal services.
In short, this article is only as imperfect as every other empirical study inevitably must be. I am often skeptical of what now counts as "empirical legal studies," because so much of that work (like the work on which it is modeled from Economics departments) is technique-obsessed, focusing on the latest statistical fads rather than thinking carefully about the underlying policy questions. Because of Professor Steinberg's article, we know more about an important policy question than we used to know. That is the ultimate test of good scholarship.