E-Discovery Insights – Clearwell Systems, Inc.

2009 TREC Legal Track Sheds Light on Search Efficacy in

Electronic Discovery

by Venkat Ranganon July 27th, 2010

In one of my previous posts, I had discussed the value and importance of TREC to the legal
community. Clearwell Systems has been a TREC participant for the last two years, and believes
in working with the rest of the participants in advancing the collective knowledge of electronic
discovery-related information retrieval methodologies. TREC’s work has been conducted in the
context of annual workshops, and is organized in the form of specific tracks. For legal
professionals, the TREC Legal Track is the most relevant, and track organizers have just released
the much-awaited overview of the 2009 workshop. I will summarize the key results from the
study and its broader implications.

The overview paper is now available and covers the design of the two tasks within the track –
the Interactive task and the Batch task. The Interactive Task is very relevant for the legal
community, since it is designed specifically for analyzing the task of producing specific records
in response to a “discovery request”. As noted in the paper, 15 teams participated, including 10
commercial teams, up from three teams in 2008. The 2009 study was also the first time an
email collection (based on Enron emails released by FERC) was used.

The Interactive Task involves a “mock complaint” and seven different topics, with each topic
described in the form of a general information request. Several teams participated by choosing
one or more topics and submitting responsive documents for each. These were then assessed
using a mathematically sound sampling and estimation methodology, and effectiveness metrics
were computed for each team.

The critical summary measure is F1, a combination of precision (estimate of false positives) and
recall (estimate of false negatives). Overall, the highest F1 measure achieved across six of the
seven topics was very good, as evidenced by values from 0.614 to 0.840. As an example, an F1
measure of 0.840 was achieved with a Recall of 0.778 and Precision of 0.912. This implies that
the information request was satisfied with very few false positives (8.8%) and few false
negatives (22.2%). Having a high precision implies that your reviewers will be reviewing fewer
irrelevant documents, hence reducing your review workload and review costs. A high recall
ensures that very few documents were missed, so your case teams can be confident that all the
facts of the case are examined.

It’s always important to look not only at the results, but the costs incurred when achieving said
results. We can break this into the costs that each team incurred, and the costs that
assessment and topic authorities incurred. Unfortunately, the study did not track the amount of
resources each team expended, so we will have to leave that as a possible improvement for a
future study. To get a view of the second cost, a review of the tabulation of team interactions
with topic authorities (Figure 3 of the overview paper) is helpful. In this study, the topic

To read more visit www.clearwellsystems.com/e-discovery-blog/ 1

E-Discovery Insights – Clearwell Systems, Inc.

authority plays the role of a case expert. The numbers show that for some topics, a highly
acceptable F-measure (over 0.75) was achieved even with interactions of 100 minutes, well
below the 600 minutes allocated for each team. This would indicate that the teams were able
to understand and construct meaningful searches with very reasonable amount of involvement
of a case expert.

The other interesting conclusion is that there is value in selecting a corpus containing
attachments. The study found that attachments increased the value of responsiveness by
measuring the “document to attachment” ratios. For the responsive set, this ratio was a
significantly higher value of 4.8 (i.e., responsive document families had, on average, one
message and 3.8 attachments), while the entire population had this ratio at 2.2. This suggests
that using the Enron corpus that contained attachments was a very good decision.

Of course, the most revealing, controversial finding is with respect to the Assessment and
Adjudication phase of the project. As noted in the overview paper’s section 2.3.4, the rate of
success of appeals was significant, ranging from 82% to 97%. In other words, the initial sample
assessments were reversed in an astonishingly large number of cases. One could argue that the
appealed documents were carefully selected, but that argument is weakened by the varying
number of appeals by participating teams, and the success rate for even the teams with larger
number of submissions. As noted in the paper, the teams that invested greater amounts of
resources in the appeals phase benefited proportionately in the levels of improvement of their
final precision and recall numbers. I know that constructing appeals can consume a lot of
resources since, in addition to the normal information retrieval task, you are required to
provide a convincing argument for reversing an initial judgment. This becomes very much a
review exercise, not unlike the traditional manual review that the broader legal industry has
been struggling with. For example, our own appeals budget was limited, forcing us to sample
the appealed documents and select only a few. The outcome of this is that un-appealed
documents are all assessed as relevant, which is unsubstantiated by the large number of
appeals. In the final analysis, section 2.4.2 illustrates a salient indicator of success – teams that
had a positive and useful interaction with the topic authority had the greatest success of initial
assessments as well as success in appeals, and the ones that leveraged this for the greatest
number of appeals had reported the greatest F-measure.

The 2009 study saw a significant increase in participation from commercial teams. My own
personal observation is that unlike academic teams, commercial teams tend to evaluate their
participation in TREC projects through the narrow prism of short-term return on investment.
While there is value in contributing to the community, I am sure each team is asked to justify
the benefits of participation to their management. Some would argue that the full benefit is not
realized because of the restrictions placed on dissemination of results within the broader
community, especially in the area of marketing the results. I am sure every commercial
participant would want to promote their performance, and highlight how their technology and
methodology was superior. Given that such direct comparisons are not permitted, the ability to
market your results is severely curtailed. The potential for comparative ediscovery analysis could be a

To read more visit www.clearwellsystems.com/e-discovery-blog/ 2

E-Discovery Insights – Clearwell Systems, Inc.

powerful motivator for all participating teams to invest more in the exercise, with the final
outcome that the community benefits.

As I noted in my previous post, the legal e-discovery profession needs an independent authority
that can challenge vendor claims and provide objective validation of one of the most complex
areas of e-discovery search and information retrieval. TREC has stepped in and served that
need very effectively. And, this has been deservedly noticed by the people that matter –
Justices of cases involving electronic discovery, expressing their opinions regarding
“reasonableness” with respect to cost-shifting, adverse inference, motion to dismiss and other
judgments.

A study of such magnitude is bound to have certain flaws, and these are documented in Section
2.5. Leaving aside these shortcomings, the TREC Legal Track effort is immensely useful for both
participants and consumers/users of legal technologies and services. The value offered to the
community by such studies is well captured in the companion report, titled the Economic
Impact Assessment of NIST’s TREC program. As the TREC coordinators are rolling out their new
TREC 2010 Legal Track tasks, it is obvious that continued improvements in both the design and
execution will make it even more attractive for all participants. Clearwell Systems is committed
to the overall goals of TREC and intends to continue their involvement in the TREC 2010 Legal
Track projects.

To read more visit www.clearwellsystems.com/e-discovery-blog/ 3