Experts, Crowds and Algorithms – AI Machine Learns to Drive Using Crowdteaching


The example above is an algorithmic system that enhanced by the use of crowd based teaching.  It is a useful example of the creativity employed by those in the machine learning research community. It is also instructive (at broader level) of the cutting edge approaches used in all of predictive analytics / machine learning.

In discussing legal prediction or the application of predictive analytics in law, we often try to start by highlighting The Three Forms of (Legal) Prediction: Experts, Crowds and Algorithms. These are really the only streams of intelligence that one can use to forecast anything.  Historically, in the law – experts centered forecasting has almost exclusively dominated the industry.  In virtually every field of human endeavor, there have been improvements (sometimes small to sometimes large) in forecasting which have been driven in the move from experts to ensembles (i.e. mixtures of these respective streams of intelligence – experts, crowds + algorithms).

Through our company LexPredict and in our research, we have been working toward building such ensemble models across a wide range of topics.  In addition, we have engaged in a public display of these ideas through Fantasy SCOTUS, our SCOTUS prediction algorithm and through the identification of non-traditional experts (i.e. our superforecasters which — unlike most lawyers — are folks that have actually been benchmarked in their predictive performance). Finally, we have demonstrated the usefulness of SCOTUS prediction in a narrow subset of cases that actually move the securities market.

Fin (Legal) Tech – Law’s Future from Finance’s Past (Katz + Bommartio)

Econometrics (hereinafter Causal Inference) versus Machine Learning

Perhaps some hyperbolic language in here but the basic idea is still intact … for law+economics / empirical legal studies – the causal inference versus machine learning point is expressed in detail in this paper called “Quantitative Legal Prediction.”  Mike Bommarito and I have made this point in these slides, these slides, these slides, etc.   Mike and I also make this point on Day 1 of our Legal Analytics Class (which really could be called  “machine learning for lawyers”).

Legal Analytics – Introduction to the Course – Professors Daniel Martin Katz + Michael J Bommarito

Here is an introductory slide deck from “Legal Analytics” which is a course that Mike Bommarito and I are teaching this semester. Relevant legal applications include predictive coding in e-discovery (i.e. classification), early case assessment and overall case prediction, pricing and staff forecasting, prediction of judicial behavior, etc.

As I have written in my recent article in Emory Law Journal – we are moving into an era of data driven law practice. This course is a direct response to demands from relevant industry stakeholders. For a large number of prediction tasks … humans + machines > humans or machines working alone.

We believe this is the first ever Machine Learning Course offered to law students and it our goal to help develop the first wave of human capital trained to thrive as this this new data driven era takes hold.  Richard Susskind likes to highlight this famous quote from Wayne Gretzky … “A good hockey player plays where the puck is. A great hockey player plays where the puck is going to be.”

3 Thoughts on E-Discovery in 2015 and Beyond – LegalTechNYC 2013 – ( Daniel Martin Katz + Michael J. Bommarito II )


The focus of my panel was “E-Discovery in 2015 and Beyond.” My Panel included: The Honorable Faith Hochberg, United States District Court for the District of New Jersey; Joe Looby, FTI Technology & Dawn Hall, FTI Consulting. As was true last year, I was the only Law Professor asked to speak at an event which draws more than 12,000 attendees from many of the law divisions of the Fortune 500, many of the law firms in the AmLaw100 / NLJ 250 and the large number of emerging legal technology companies which as Bill Henderson noted are not really being held back by Rule 5.4.

Rock / Paper / Scissors – Man v. Machine (as t→∞ you are not likely to win) [via NY Times]

From the site … “A truly random game of Rock-Paper-Scissors would result in a statistical tie with each player winning, tying and losing one-third of the time …  However, people are not truly random and thus can be studied and analyzed. While this computer won’t win all rounds, over time it can exploit a person’s tendencies and patterns to gain an advantage over its opponent.

Computers mimic human reasoning by building on simple rules and statistical averages. Test your strategy against the computer in this rock-paper-scissors game illustrating basic artificial intelligence. Choose from two different modes: novice, where the computer learns to play from scratch, and veteran, where the computer pits over 200,000 rounds of previous experience against you.”

Time to dust off your random seedpseudorandom number generators … good luck!

IBM Watson on Jeopardy – Scoreboard: Watson 2, Humans 0 [via CNN]

Update: For his thoughts on possible implications in the market for legal services, check out Larry Ribstein’s post “Lawyer’s in Jeopardy” over at Truth on the Market. In a related vein, check out today’s WSJ Is Your Job an Endangered Species? The subtitle reads: “Technology is eating jobs—and not just obvious ones like toll takers and phone operators. Lawyers and doctors are at risk as well.”