What constitutes “good output” for a law school, and how do you measure it?

The ABA Journal reports that the part of the ABA that accredits law schools is due to adopt sweeping changes that will radically change the ways it evaluates the quality of the education individual law schools deliver. Most significantly, the ABA will “move away from evaluating law schools on the basis of criteria that measure ‘input’-such things as faculty size, budget and physical plant. Instead, the Legal Education Section would evaluate law schools more heavily on the basis of ‘outcome’ measures. The essential difference is that outcome measures would focus on what students actually take away from their educational experience at a particular law school rather than what the school teaches, and how . . . .”

There is one HUGE question, however, still to be resolved: “Speaking on Friday’s panel, [committee vice chair Margaret Martin] Barry and her fellow committee members said the greatest challenge is to determine the best ways to measure outcome.” 

I’ve got a suggestion on where they should look: Back in March, as I wrote,  the New York Times reported that professors at the University of California, Berkeley, have studied what makes lawyers (not law students or law professors) effective and “have come up with a test that they say is better at predicting success in” practicing law than is the LSAT.  The study concluded, as I’ve long been convinced, that “LSAT scores . . . ‘were not particularly useful’ in predicting lawyer effectiveness’. . .”  What does the new test consider factors that contribute to lawyerly effectiveness?

“[T]he ability to write, manage stress, listen, research the law and solve problems.”

When the ABA starts testing law school graduates on those abilities I’ll be convinced the ABA genuinely is measuring “outcomes.” Somehow, though, I suspect that they’ll focus on instead on salaries, clerkships, and professorships, outcomes determined to a significant degree by people with vested interests in the status quo.

The financial crisis is an opportunity for innovation in legal practice and law schools.

This blog is supposed to be about law and innovation — both the ways law affects innovative and creative endeavors and the ways creativity informs the practice of law.  I’m not sure where I go t the nerve to believe I have something worth saying on these matters, but it is gratifying when I find out that I’m not entirely empty-headed.  The New York Times yesterday published an editorial observing that the economic downturn is hitting the legal profession just as profoundly as it is hitting any other occupation.  As the Times notes, The “American Lawyer is calling it ‘the fire this time’ and warning that big firms may be hurtling toward ‘a paradigm-shifting, blood-in-the-suites’ future.” The thrust of the editorial, though, is that crisis is an opportunity for change and that the legal profession is much in need of change: “The silver lining, if there is one, is that the legal world may be inspired to draw blueprints for the 21st century.”

I am not at all happy with the job market, especially for my students.  They are talented, well educated, and hard working people who will do a lot of good for their clients.  But I am gratified that the types of changes the Times are ones I’ve long believed are important.  I do believe the U.S. legal system is a brilliant embodiment of practical justice, but it’s biggest defect in achieving justice is its cost.  It is appalling that achieving any sort of justice against any adversary willing to fight you (regardless of the merits of his cause) will invariably cost you enormously.  The fact someone whose position has little or no merit can make you expend enormous amounts of money to prove the demerits of his claims undermines justice by tilting the entire system radically in favor of the more wealthy members of our society. The internet has brought home this phenomenon to artists who want to make their work available and to people who want to post their family videos, but it will be one familiar to anyone who has called upon the legal system or been dragged into it, whether through divorce, the need to obtain payment from one’s customers, the need to get relief from economic or physical threat, or any of the myriad other ways one might need to call upon legal process to attain justice.

Two ways the economy will force down the costs of using lawyers are (1) the salaries of lawyers at the top end (which the Times notes begin at $160,000 at the wealthiest law firms) will have to be reduced and (2) the reduction of money available to spend on lawyers will mean “more leverage to push . . . for successful outcomes” on the part of those who have traditionally been less well financed.

Moreover, law firms will have to change their billing practices, replacing the “billable hour,” Law firms also, of course, will have to come up with more efficient ways of delivering their services.

The Times also suggests the economic crisis might require law schools to “become more serious about curricular reform,” in particular by “including more focus on practical skills.”  I could not agree more — making clear the inextricable bond between legal theory and legal practice has been central to my work as a law professor.  I don’t understand how you can teach law without understanding how it works.  My conviction is evidenced, I hope, by the school i’ve chosen to teach at this year and (at least) next, the University of Detroit Mercy Law School.

But I am skeptical of the power of the economy to change law schools.  Lawyers and judges have for a long time called for  law schools to focus more on training lawyers (rather than teaching legal theory in a way that makes sense primarily to law professors, not lawyers or judges), and still the changes have been very, very slow and very, very minor.  Law schools do not look to their success at training lawyers to guide their curriculur decisions; rather, they principally look to a rankings system that rewards law schools that admit students most like the students at the schools that are already the highest ranked. That’s a formula to entrench the status quo, not a formula for change. Nor does the critique from within law schools of the most influential rankings system really do much to solve the institutional deference to the status quo.  Brian Leiter, a law professor at the University of Chicago, publishes his own rankings and regularly critizises the most influential rankings, but even he relies principally in judging law schools on the “scholarly reputation” of faculty (which is largely based on where the professors teach and the law reviews in which they publish, both of which are merely indicators of how well those professors fit the prevailing view of quality) and student undergraduate grade point averages and LSAT scores (both of which correlate to success as law students, not as lawyers).

It cannot hurt, though, that the New York Times has joined the chorus calling for law school’s to focus their curricula more on the practice of law.

Best bonds: AIG? Greatest Fascist Dictator: Adolf Hitler? Best Law School: ?????

What is it with the human thirst for numerical rankings, for judging one thing better than another even when the comparisons are known to be completely arbitrary or, at best, based on judgments so subjective regarding criteria so limited as to render the rankings nothing more than crude subjective judgments disguised as hard data?

Woody Allen fittingly complained:

What’s with all these awards? They’re always giving out awards. Best Fascist Dictator: Adolf Hitler.

But it’s not just the thirst for the rankings.  It’s basing one’s actions on rankings as if they have profound meaning despite their lack of meaning.

As I wrote recently, US News and World Report’s rankings of law schools are determined largely by the LSAT scores and undergraduate grade point averages of the students each law school admits even though those scores and averages bear no correlation to success as a lawyer; rather, they correlate only to success in law school, which, again, bears no meaningful correlation to success as a lawyer (as would not surprise most lawyers but, I would guess, would surprise most non-lawyers, including law students and law professors who have not practiced extensively).

Yet an overwhelming number of law applicants rely on the US News rankings.  Even more depressingly, an overwhelming number of law faculties make their educational decisions to improve those rankings, not to improve the way they educate law students to be lawyers. (As I also pointed out, Detroit Mercy, where I am currently a visiting professor and where I will continue in that capacity next year while remaining on leave from Case Western Reserve, is a rare exception to this rule.)

And today, reading in the New York Times about Moody’s, I realized another reason the US News rankings are so useless and their importance so poisonous to legal education.  It is because the US News rankings are accepted, followed, and never questioned in a way meaningful enough to threaten their influence. There is therefore little incentive to make judgments on a law school’s quality based on judgments independent of those rankings.

Moody’s is one of the private companies that rate corporate bonds.  When a corporation sells bonds to raise money (simply put, they borrow money from the purchasers of the bonds and pay back the loan at the interest rate called for by the bond), Moody’s issues “grades” to the bonds that predict the likelihood the corporation will pay back the loan.  “Junk” bonds are so-called because they are bonds issued by companies that are at high risk of being unable to pay the purchaser of the bond when payment is due.  In other words, junk bonds are “sub-prime” bonds.  Why do people loan money to companies or homeowners despite the high risk the borrowers will default?  Because those borrowers have to pay a higher interest rate.  The high interest rate on the loans that are repaid makes up for the loans that aren’t paid back.

The unconscionable innacuracy of Moody’s rankings, however, has played a major role in our financial crisis.  As the Times points out:

Moody’s rated Lehman Brothers’ debt A2, putting it squarely in the investment-grade range, days before the company filed for bankruptcy. And Moody’s gave the senior unsecured debt of the American International Group, the insurance behemoth, an Aa3 rating – which is even stronger than A2 – the week before the government had to step in and take over the company in September as part of what has become a $170 billion bailout.

Moody’s and the other major ratings companies also “put their seals of approval on countless subprime mortgage-related securities now commonly described as toxic.”

There are numerous reasons to the ratings companies were bound to fail, but the Times article brought up an interesting one I had never considered before.  There is little incentive to question anyone who is paid to judge the the quality of something unless and until the accuracy of those judgments is put to the test. As Frank Psrtnoy, a law professor at the Universitiy of San Diego and a former derivatives trader, explains it:

Imagine if you had a rabbi and said, “All the laws of kosher depend on whether this rabbi decides if food is kosher or not.”  If the rules say “You have to use this rabbi,”  he could be totally wrong and it won’t affect the value of his franchise.

In other words, if you wanted kosher food, you’d buy food approved by that rabbi and never question his judgment unless and until the accuracy of his judgments was threatened in a meaningful way. US News is that rabbi.  It has become the principal judge of law school quality and it doesn’t matter whether its judgments are legitimate or not.  Students buy its rankings guides, law faculties and deans make decisions driven solely by the desire to meet the criteria US News employs, and applicants and legal academia continue to make their educational decisions based on the criteria employed by US News rather than on their own judgments.

It’s a terrible situation, and particularly ironic when it comes to legal education.  Lawyers every day, every moment, make judgments and decisions based on incomplete, subjective, and biased information.  You can only consider the circumstances under which those decisions are made inadequate, however, if you believe it is ever possible to have all the information you would want and if all that information could be stripped of the distortions inherent in the limitations of human perception.  Making decisions based on incomplete, subjective, and biased information is what life is about. That doesn’t mean there aren’t better and worse judgments; it merely means that one can never be certain, that there is always risk, that almost every important decision one makes in one’s life cannot be reduced to a choice between black and white, right and wrong, #1 and #2.  Lawyers make their living making such difficult decisions and judgments.  The legal situations where there are clear answers don’t require lawyers, and if lawyers become involved they certainly don’t make much of a living answering those questions.

Yet law school applicants and law professors act as if the judgment that one law school is better than another can be reduced to a comparison of hard numbers, and that, therefore, those numbers should be the determinant of their actions.  They’re being as stupid as the investors in Lehman Brothers and AIG were in relying on Moody’s.

And the Times article mentions one other fact that bears on this point.  Warren Buffett — the man “known as the Oracle of Omaha,” the daddy we turn to to guide us out of our financial pit, “the closest thing that the United States economy has to a life coach”  — owns 20% of Moody’s.  But you know what?  In making his investment decsions he doesn’t rely at all on Moody’s ratings.  He has his own research department.  He makes his own judgments.  I wish more college graduates did the same.  And it maddens me beyond measure that most law professors don’t.

The making of a lawyer

Most law schools are odd places.  I suspect most people outside the law believe a law school’s principal mission is to train lawyers.  I am a law professor, and I happen to believe that too.  But I am a very odd duck within law school academia.  Practice for twelve years and partnership in a top nationwide firm is of very little value as a qualifaction to be a law professor.  Rather, the valuable assets among law school faculty are articles published in journals edited by students and rarely read by lawyers.  Most law school classes address theory (or “doctrine”) in a manner remarkably removed from its real world application.  Qualifying for law school rests to a significant degree, perhaps primarily, on an applicant’s score on the LSAT test, which may correlate to success in law school but bears little relationship to one’ effectiveness as a lawyer.  To add to the gap between law school and legal practice, the principal criterion underlying the rankings on which law schools and applicants rely to rate the quality of a law school is the median LSAT score of the school’s students.  Those rankings provide a tremendous incentive for a law school to act in ways intended to accept applicant’s with higher LSAT scores — scores that don’t correlate to effectiveness as a lawyer — at the expense of acting in ways that increase the effectiveness of its graduates as lawyers.

So I am thrilled to read in yesterday’s New York Times that professors at the University of California, Berkeley, have studied what makes lawyers (not law students or law professors) effective and “have come up with a test that they say is better at predicting success in” practicing law than is the LSAT.  The study concluded, as I’ve long been convinced, that “LSAT scores . . . ‘were not particularly useful’ in predicting lawyer effectiveness’. . .”  What does the new test consider factors that contribute to lawyerly effectiveness?

“[T]he ability to write, manage stress, listen, research the law and solve problems.”

I am also not surprised to read that the new test is no better than the LSAT at predicting how well participants would do in law school.  As I wrote above, there is far too great a gap between most law school instruction and the actual practice to consider a test that measures effectiveness in the latter able to test effectiveness in the former.

I wish all my students would read this post.  They’ve been dealing with a considerable degree of stress of late that they blame on me and the problems I’ve given them to try to solve — problems that are down and dirty real life problems lawyers face — and they’ve been complaining a lot.  One student in my Contracts course yesterday complainied that online discussion boards made clear to him that students at other schools were covering a lot more “theory” than I am.  I looked at him a little in surprise.  That’s the whole point of my teaching.  And it’s the whole point of the rather unusual curriculum at the school where I am a visiting professor, the University of Detroit Mercy Law School, where it has been recognized that theory and practice are inextricably intertwined and that each can only be understood in the law in relation to one another.  Thus, the school offers a “revolutionary new curriculum . . . [that] complements traditional theory- and doctrine-based coursework with practical learning, providing a solid transition between law school and a legal career.

But it’s hard to teach students to manage stress, listen, and solve problems.  First, it mean subjecting them to the stress of solving problems they do not know the solutions to in advance because what lawyers do is solve problems they don’t know the solutions to in advance.  No one enjoys stress.  I like to think that the students realize the stress I am subjecting to them is not one intended to or that will break them.  It’s school.  As I’ve always told them, in law school we hurl you into the water to see if you can swim, but the water’s only about 4 feet deep, so when you can’t swim, you just get on your feet, come back, try to figure out what went wrong, and then try again.  It’s when you’re a lawyer trying to solve problems you don’t know the solutions to in advance that the stress can be truly overwhelming, especially if you have not been at all prepared for it.