McConnell Dowell Constructors (Aust) Pty Ltd v Santam Ltd (No 2)
[2017] VSC 640
•22 September 2017
| IN THE SUPREME COURT OF VICTORIA | Not Restricted |
AT MELBOURNE
COMMERCIAL COURT
TECHNOLOGY ENGINEERING AND CONSTRUCTION LIST
S ECI 2014 000497
| McCONNELL DOWELL CONSTRUCTORS (AUST) PTY LTD (ACN: 002 929 017) | Plaintiff |
| v | |
| SANTAM LTD (REGISTRATION NUMBER 1918/001680/06) | First Defendant |
---
JUDGE: | VICKERY J |
WHERE HELD: | Melbourne |
DATE OF HEARING: | 15 September 2017 |
DATE OF JUDGMENT: | 22 September 2017 |
CASE MAY BE CITED AS: | McConnell Dowell Constructors (Aust) Pty Ltd v Santam Ltd & Ors (No 2) |
MEDIUM NEUTRAL CITATION: | [2017] VSC 640 |
---
PRACTICE AND PROCEDURE – Appointment of special referee – Questions as to discovery of documents and inspection of documents in a large document case referred to special referee – Reference conducted as a facilitation process rather than an adversarial process – Adoption of special referee’s report – Special referee’s report adopted in part – Further questions put to the special referee to give his opinion and report thereon – O 50 of the Supreme Court (General Civil Procedure) Rules 2015 (Vic) – Validation process in TAR considered.
PRACTICE AND PROCEDURE – Discovery in a large document case – Use of predictive coding technology – Technology Assisted Review (‘TAR’) – Orders for TAR made in accordance with the recommendations of the special referee following a facilitation process conducted with the parties in a reference under O 50 of the Supreme Court (General Civil Procedure) Rules 2015 (Vic) – Special referee’s report adopted in part – Further questions put to the special referee to give his opinion and report thereon – Validation process in TAR considered.
---
APPEARANCES: | Counsel | Solicitors |
| For the Plaintiff | Mr T J Margetts QC & Ms A J Golding | Baker McKenzie |
| For the First Defendant | Ms P A Neskovcin QC & Mr B G Mason | Holding Redlich |
HIS HONOUR:
Background
On 2 December 2016 the Court delivered its judgment in this proceeding on the use of Technology Assisted Review (‘TAR’ or predictive coding technology) to assist the Plaintiff (‘McConnell Dowell’) in complying with its discovery obligations in the proceeding, which had generated a very large volume of documents for review.[1]
[1]McConnell Dowell Constructors (Aust) Pty Ltd v Santam Ltd & Ors(No 1) [2016] VSC 734 (‘McConnell Dowell (No 1)’).
Simply stated, TAR is a type of software that can be trained by a human to distinguish between relevant and non-relevant documents.[2]
[2] Pyrrho Investments Limited v MWB Property Limited [2016] EWHC 256 (Ch) [19]–[24].
In 2014, McConnell Dowell issued proceedings in the Technology Engineering and Construction (TEC) list against the First Defendant, (‘Santam’), the Second Defendant, QBE Underwriting Ltd as managing agent for QBE Syndicate 386 and QBE Syndicate 1886 and the Third Defendant, Liberty Mutual Insurance Company.
The proceeding involves a large claim involving tens of millions if not hundreds of millions of dollars arising from the design and construction of a natural gas pipeline in Queensland.
The construction contract at the centre of this case and an associated arbitration generated approximately 4 million potentially relevant documents. These have been scanned and are available in a searchable electronic format. McConnell Dowell has reduced the number of documents said to be relevant to the present proceeding from approximately 4 million to approximately 1,400,000.
The use of TAR was found to be likely to dramatically reduce this number and contain it further within reasonable and manageable bounds.[3] It was conversely found that traditional manual discovery of the Plaintiff’s documents is not likely to be either cost effective or proportionate.[4]
[3]McConnell Dowell (No 1) [2016] VSC 734 [3].
[4]Ibid [7].
For these reasons, on 8 September 2016, in accordance with s 7(2)(c)(ii) of the Civil Procedure Act 2010 (Vic), the Court made an order for the appointment of a Special referee pursuant to Order 50 of the Supreme Court (General Civil Procedure) Rules 2015 (Vic) (the ‘Rules’) to conduct a reference for the purposes of answering questions as to the appropriate management of discovery in the proceeding and to deliver a report to the Court on those questions.
The appointment of the Special Referee in this case has proved to be a particularly useful practice, not only from the perspective of the managing Judge, but also for the parties. For the Court, the Special Referee has provided a monitoring facility for the managing Judge as to progress achieved in the use of TAR by way of reports from the referee from time to time. The reference has also served to bring to the attention of the parties and their legal advisers the opportunity for the application of TAR in the proceeding and to facilitate something of a ‘due diligence’ exercise in aid of making the commercial decision to invest in the process. The Special Referee is also in a position to assist with the development of an agreed TAR protocol adopted by the parties and to facilitate any necessary changes to the protocol as the process proceeds.
On 7 June 2017 the Special Referee, pursuant to rr 50.01(2)(b) and 50.03 of the Rules, filed a report with Court detailing the results which had been achieved using TAR.
To this point, the parties engaged in a collaborative approach in the use of TAR, as prescribed in their agreed protocol. This involved each firm of solicitors reviewing the documents generated in the training rounds with a view to jointly training the algorithm to select relevant documents from the potential pool of discoverable documents (the ‘discovery collection’). The process involved input from senior solicitors for each party engaged in the case working with IT experts. The training rounds essentially involved reviewing groups of sample documents in each round to assess the relevance of each document, making a coding decision, and conferring with a view to resolving any coding differences. Thus the approach undertaken by parties in this TAR process applied a different paradigm to the standard discovery processes.
In his report of 7 June 2017, the Special Referee recommended that the TAR process continue for at least the next 3 training rounds, but without using the collaborative process prescribed in the agreed protocol. The parties accepted the recommendation in the report and three further training rounds were undertaken by the legal team for the Plaintiff, without collaboration with and review by the Defendants. Between 9 and 22 June 2017, and in accordance with the recommendations of the Special Referee, the solicitors for the Plaintiff undertook three further training rounds in conjunction with the Plaintiff’s IT expert, FTI Consulting.
The proceeding as against the Second Defendant was settled and the proceeding against it was dismissed by order made on 26 June 2017.
The proceeding as against the Third Defendant was settled and the proceeding against it was dismissed by order made on 4 September 2017.
This leaves the proceeding on foot only as against the First Defendant.
By report dated 14 July 2017 (the ‘14 July Report’) the Special Referee advised the Court of the progress that has been achieved in the TAR process since 7 June and made recommendations concerning the continued use of the TAR process to complete the Plaintiff’s discovery.
On 15 September 2017, the parties made submissions to the Court on whether or not the 14 July Report should be adopted pursuant to r.50.04 of the Rules. While the Plaintiff accepted the conclusions arrived at in the report, the First Defendant submitted that the report should not be adopted.
These reasons concern the question as to whether the recommendations of the Special Referee in the 14 July Report should or should not be adopted.
Measures of TAR
By way of further background relevant to this application, there are two measures commonly used to assess the accuracy of the TAR document review process in fulfilling a party’s discovery obligations —‘recall’ and ‘precision’.
Recall measures how many of the relevant documents in the discovery collection have been found. For example, a 40 per cent recall rate means that 40 per cent of all relevant documents in a discovery collection have been found, and 60 per cent have been missed.
Precision measures the level of non-relevant documents found in a discovery collection. Precision measures how many of the documents retrieved are actually relevant. For example, a 65 per cent precision rate means that 65 per cent of the documents retrieved are relevant, while 35 per cent of those documents have been misidentified as relevant.
As explained by Judge Andrew Peck in the Da Silva Moore case:[5]
The objective of review in ediscovery is to identify as many relevant documents as possible, while reviewing as few non-relevant documents as possible. Recall is the fraction of relevant documents identified during a review; precision is the fraction of identified documents that are relevant. Thus, recall is a measure of completeness, while precision is a measure of accuracy or correctness.
[5] Da Silva Moore et al v Publicis Groupe et al [24 February 2012] 11 Civ. 1279 (ALC) (AJP) (S.D.N.Y), 17. Judge Peck’s decision was upheld by Judge Andrew L. Carter, Jr, United States District Court, Southern District of New York.in Da Silva Moore et al v. Publicis Groupe et al 11 Civ.1279 (ALC) (AJP) (S.D.N.Y. [26 April 2012].
A TAR document review process can achieve either high recall or high precision, but rarely both simultaneously. An effort to improve the performance of one factor generally causes the performance of the other to drop. This is often referred to as the ‘Precision-Recall Tradeoff’.[6]
[6]See Maura R Grossman and Gordon V Cormack, ‘The Grossman-Cormack Glossary of Technology-Assisted Review’ (2014) 7 Federal Courts Law Review 85, 104.
Findings and recommendations of the Special Referee
The central findings made by the Special Referee and his consideration of the submissions of the parties before him were:
6.3The training rounds results are equivocal. On the one hand training round 6 produced a substantial increase in the precision rate (from 22-32% to 38-52%). This produced a dramatic drop in the number of documents generated 304,494 (approx.) to 181,745. Had this improvement continued I would have recommended further training. However training rounds 7 and 8 did not produce an increase in the precision rate. In fact the rate dropped to 33-45% and 33-46%.
6.4The plaintiff is required to undertake a reasonable search for documents. It discovered 1.2 million documents after conducting keyword and Boolean searches. It has provided those documents (subject to privilege claims) to the defendants in electronic format. This has enabled the defendants to conduct searches of the documents. However the size of the 1.2 million document database, which includes so many relevant documents, has made the searches very difficult.
6.5The parties have co-operated as required by the Supreme Court Rules and Standard Operating Procedures to identify a proportionate regime for the discovery. A linear review of the documents has been rejected by Justice Vickery as a proportionate search mechanism. Keyword and other searches failed to substantially reduce the size of the database. It is in this context I recommended and the court approve the use of TAR.
6.6Unfortunately the protocols recommended by the parties IT experts have not been as successful as the parties wanted or expected. Nevertheless the use of keyword searches to reduce the size of the original database and then the application of TAR has resulted in the database of potential documents be reduced from approximately 4 million documents to approximately 208,000 documents.
6.7I am required to consider whether there likely to be any increase in the precision rate if further training rounds occur. The submissions filed by the parties highlight the dilemma. The plaintiff says that after the time-consuming and expensive process carried out in accordance with the Joint Protocol and for rounds 6-8 it is unlikely that there will be any improvement and therefore any further training will not be productive. Therefore it is not cost-effective and reasonable to undertake the searches. The validation process is only applicable if the TAR process has stabilised. This may never occur. The defendant says that the precision rate is too low and the algorithm has not stabilised. The plaintiff should be required to undertake further training rounds or at the very least be required to undertake the training if I recommended that TAR continue.
6.8 There is merit in both arguments.
6.9 …
6.10Given the difficulties which have been experienced I conclude that it is unlikely there will be any dramatic improvement in the precision rate which would warrant further training rounds. In these circumstances validation as set out in the original joint protocol is not warranted or likely to reduce the number of documents in the database. Further training rounds will also increase the cost and the delay in the preparation of this matter for trial. As J Forrest J as [sic] stated the resources should be allocated for the trial, not the discovery fight.
Recommendations
The primary recommendations made by the Special referee were:
7.1In my opinion it is not proportionate for the Plaintiff to undertake any further training rounds of the TAR algorithm. The plaintiff cannot be expected to continue a process where there is little or no likely improvement to the TAR algorithm.
7.2 I recommend that:
7.2.1the plaintiff is not required to undertake any further training of the TAR algorithm;
7.2.2the plaintiff is not required to prepare a validation set of documents as prescribed in the original joint protocol;
7.2.3the plaintiff produce within 14 days a list of documents projected as positive by the TAR model at a recall of 80%
7.2.4upon production of the list of documents the plaintiff shall be deemed to have undertaken a reasonable search to locate discoverable documents.
The Special Referee also recommended an alternative regime in the event that the Court did not adopt his primary recommendations and the Plaintiff is required to review additional documents using TAR.
Legal principles for the adoption of a Special Referee’s report
In Kilpatrick Green Pty Ltd v Leading Synthetics Pty Ltd (‘Kilpatrick Green’),[7] Gillard J dealt with the reception of a referee’s report the subject of challenge in an engineering case.
[7][1998] VicSC 291 (5 June 1998).
After examining the jurisdiction of the Court in exercising its powers under O 50,[8] his Honour traced the history of the provision before turning to consider the question as to how a Court should approach the reception and adoption of a referee’s report, noting a distinction where a determination of a point of law is the subject of a challenge by a party, and where a finding of fact is under challenge. Gillard J approved and adopted the general approach stated by Giles J of the Supreme Court of New South Wales in Leighton Contractors Pty Ltd v C E Heath Underwriting and Agency services Ltd & Ors (‘Leighton Contractors’)[9] which he treated as ‘an appropriate guide’:[10]
More shortly, a party dissatisfied with a referee’s report is not entitled to have the judge before whom it comes to reconsider and determine afresh all issues whether of facts or law which it would wish to contest. Nor does consideration of the report involve an appeal. Rather the judge has a discretion to exercise, a discretion which would normally be exercised by reconsidering a question of law or the application of legal standards to established facts, but otherwise may fall to be exercised having regard to matters such as the nature of the complaints, the type of litigation involved, and the length and complexity of the proceedings before the referee. Patent misapprehension of the evidence, or perversity or manifest unreasonableness in fact finding, would ordinarily preclude relevant adoption of or action upon a report, but a report may be adopted or acted upon even if upon reconsideration of the evidence the judge might have reached a conclusion different to that of the referee, In general, where there is shown to be evidence available to support a referee’s finding of fact, or where the issue involves a choice between conflicting evidence, in the exercise of the discretion the judge will not reconsider disputed questions of fact. But it is always a question of judicial discretion, exercised in a manner consistent with the object and purpose of the Rules and the place they play in the administration of justice according to law.
[8]Ibid 4–9.
[9](1996) 12 BCL 415, 418.
[10]Kilpatrick Green [1998] VicSC 291 (5 June 1998) 7.
The law on the reception of reports from referees has received more recent attention at appellate level in Victoria, where the observations of Giles J in Leighton Contractors, applied by Gillard J in Kilpatrick Green were expanded upon.
In Wenco Industrial Pty Ltd v W Industries Pty Ltd (‘Wenco’),[11] the Court of Appeal set out nine propositions derived from cases which had considered the principles applicable to the exercise of the court’s discretion in adopting, varying or rejecting a report of a special referee.[12] It is instructive to reproduce that part of the reasons for judgment:
[11][2009] VSCA 191; (2009) 25 VR 119 (Redlich and Bongiorno JJA and Beach AJA).
[12]Ibid 126–127 [17]. Their Honours drew largely from the distillation of the principles elucidated by McDougall J in Chocolate Factory Apartments Ltd v Westpoint Finance Pty Ltd [2005] NSWSC 784 [7].
Adopting the report: the principles to be applied
The approach to be taken in considering whether to adopt the report of a referee, has been the subject of extensive consideration by courts in different jurisdictions. Although the underlying rules are not always the same,[13] the following propositions can be extracted from the cases. They provide a general guide as to how the question of the adoption of a referee’s report should be approached:
[13]Re Markbys Renaissance Pty Ltd [1999] 3 VR 851, 859 [23].
(a)First, in exercising the power conferred by r 50.04 to adopt the report of a special referee, the Court has a wide power which is to be exercised ‘as the interests of justice require’. This broad mandate should not be the subject of restrictions laid down in advance of judges exercising it.[14] Subject to what follows, it is undesirable to attempt closely to confine the manner in which the discretion is to be exercised.
(b)Secondly, the purpose of rules 50.01 and 50.04 is to provide, where the interests of justice so require, a form of partial resolution of disputes alternative to orthodox litigation. Further, that purpose would be frustrated if the reference were to be treated as ‘some kind of warm-up for the real contest’.[15]
(c)Thirdly, insofar as the subject matter of dissatisfaction with a report is a question of law, or the application of legal standards to established facts, a proper exercise of discretion requires the judge to consider and determine that matter afresh.[16]
(d)Fourthly, where a report shows a thorough, analytical and scientific approach to the assessment of the subject matter of the reference, the Court would have a disposition towards acceptance of the report, for to do otherwise would be to negate both the purpose and the facility of referring complex technical issues to independent experts for inquiry and report.[17]
(e)Fifthly, if the referee’s report reveals some error of principle, absence or excess of jurisdiction, patent misapprehension of the evidence or perversity or manifest unreasonableness in fact finding, that would ordinarily be a reason for rejection. In this context, patent misapprehension of the evidence refers to a lack of understanding of the evidence as distinct from the according to particular aspects of it different weight; and perversity or manifest unreasonableness mean a conclusion that no reasonable tribunal of fact could have reached. The test denoted by these phrases is more stringent than ‘unsafe and unsatisfactory’.[18]
(f)Sixthly, generally, the referee’s findings of fact should not be re-agitated in the Court. The Court will not reconsider disputed questions of fact where there is factual material sufficient to entitle the referee to reach the conclusions he or she did, particularly where the disputed questions are in a technical area in which the referee enjoys an appropriate expertise. Thus, the Court will not ordinarily interfere with findings of fact by a referee where the referee has based his or her findings upon a choice between conflicting evidence.[19]
(g)Seventhly, the purpose of r 50.01 and r 50.04 would be frustrated if the Court were required to reconsider disputed questions of fact in circumstances where it is conceded that there was material on which the conclusions could be based.
(h)Eighthly, the Court is entitled to consider the futility and cost of re-litigating an issue determined by the referee where the parties have had ample opportunity to place before the referee such evidence and submissions as they desire.[20]
(i)Ninthly, even if it were shown that the Court might have reached a different conclusion in some respect from that of the referee, it would not ordinarily be (in the absence of any of the matters referred to in sub para (e) above) a proper exercise of the discretion conferred by r 50.04 to allow matters agitated before the referee to be re-explored so as to lead to qualification or rejection of the report.[21]
[14]Chocolate Factory Apartments Ltd v Westpoint Finance Pty Ltd [2005] NSWSC 784 [7].
[15]Ibid.
[16]Ibid.
[17]Ibid.
[18]Ibid.
[19]Ibid.
[20]Ibid.
[21]Ibid. See also Super Pty Ltd v SJP Formwork (Australia) Pty Ltd 29 NSWLR 549; Chloride Batteries Australia Ltd v Glendale Chemical Products Pty ltd (1988) 17 NSWLR 60; White Constructions (NT) Pty Ltd v Commonwealth (1990) 7 BCL 193; Foxman Holdings Pty Ltd v NMBE Pty Ltd (1994) 38 NSWLR 615; Nicholls v Stamer [1980] VR 479; Westpoint Management Ltd v Chocolate Factory Apartments Ltd [2007] NSWCA 253; Bellevarde Constructions Pty Ltd v CPC Energy Pty Ltd [2008] NSWCA 228; Re Markbys Renaissance Pty Ltd [1999] 3 VR 851 and Plumley v Adguage Pty Ltd (1998) 29 ACSR 315 (Plumley was the appeal in Re Markbys Renaissance – Markbys Renaissance having been decided on 16 June 1995, notwithstanding its appearance in [1999] 3 VR).
It appears that no subsequent decision has queried or qualified the correctness of these observations and neither party sought to do so in the present application.
Submissions of the First Defendant
The First Defendant submitted that, in the interests of justice, the Court should decline to adopt the 14 July Report and should order the Plaintiff to conduct two additional training rounds and undertake the validation process.
The First Defendant set out the nub of its case in its written reply submission, where six grounds were relied upon.
Validation issue
First, it was submitted that the 14 July Report misunderstood the function of the validation process, and Plaintiff’s submissions repeated this error.[22] The role of the validation process is not to reduce the volume of potentially discoverable documents. Its function is to verify that the TAR model is accurately reporting the parameters within which it is operating and is defensible.[23] Users of the TAR model cannot know with any certainty whether the model is operating within the reported parameters until this step is completed.[24]
First discretionary matter
[22]Paragraphs 3(b) and 9(f)(iii) of the Plaintiff’s submissions.
[23][10] of the supplementary affidavit of Justin Barry Smith sworn 12 September 2017 and exhibit JBS-1, being a Ringtail document from its webpage.
[24][22]-[26] of the affidavit of Justin Barry Smith sworn 29 August 2017, and [9]-[13] of Mr Smith’s supplementary affidavit sworn 12 September 2017.
Secondly, a matter which goes to the Court’s discretion was advanced. It was put that the Plaintiff has not discharged its discovery obligations.[25] The First Defendant should not have to suffer the burden of trawling through the large mass of substantially irrelevant documents to identify those relating to the issues in this proceeding.[26] The Protocol providing for the TAR process was intended to overcome this issue, and the parties have invested considerable time and resources to implement that process. For these reasons, the TAR process should continue.
Procedural matter
[25]In his report dated 23 November 2016 at [41], the Special Referee opined that the manner in which the Plaintiff then sought to discharge its discovery obligations breached the objects of the Civil Procedure Act 2010 (Vic).
[26]See McConnell Dowell Constructors (Aust) Pty Ltd v Santam Ltd (No. 1) [2016] VSC 734, [5]-[7] (per Vickery J).
Thirdly, it was submitted that the proposal to abandon the validation process without the First Defendant’s consent was not properly in issue before the Special Referee.[27] That proposal was raised in the Plaintiff’s submissions to the Special Referee dated 13 July 2017 at 10.40 am.[28] The First Defendant had filed its submissions at 10.38 am.[29] The technical evidence the Plaintiff presented in support of this submission to abandon the validation process was incorrect because, for the reasons detailed above, it misstated the validation process’ function. The First Defendant did not have a reasonable opportunity to place before the Special Referee expert material in response before the pre-determined date on which the Special Referee was to issue his report. The First Defendant has now submitted that responding material in this application. That material should be preferred because it properly appreciates the validation process’ function.[30] It is also based on reasonable and substantiated assessments of the time required to complete the validation process,[31] in contrast to what was said to be ‘the exaggerated estimates on which the Plaintiff relies’.[32]
Alleged erroneous assessment of the evidence
[27]Paragraph 9(g) of the Plaintiff’s submissions.
[28]Annexure 3 to the Special Referee’s report dated 14 July 2017, at Exhibit KAH-1 to the affidavit of Kathryn Anne Howard sworn 29 August 2017.
[29]Annexure 4 to the Special Referee’s report dated 14 July 2017, at Exhibit KAH-1 to the affidavit of Kathryn Anne Howard sworn 29 August 2017.
[30][22]-[26] of Mr Smith’s affidavit sworn 29 August 2017 and [9]-[13] of Mr Smith’s affidavit sworn 12 September 2017.
[31][4]-[6] of Mr Smith’s affidavit sworn 12 September 2017.
[32][26]-[27] of the affidavit of Anthony Brendan Whelan sworn 7 September 2017.
Fourthly, it was contended that the Special Referee appeared to rely upon the Plaintiff’s expert’s evidence when making his recommendations regarding the utility of conducting further training rounds, and his analysis when doing so was erroneous.[33] The First Defendant pointed to the Special Referee noting ‘the plaintiff says that after the time-consuming and expensive process carried out in accordance with the Joint Protocol for rounds 6–8 it is unlikely that there will be any improvement and therefore any further training will not be productive.’[34] He considered that this submission had ‘merit’.[35] However, it was put by the First Defendant that the Special Referee disregarded the fact that training rounds 6 to 8 were completed in no more than 9, 7 and 6 days, respectively.[36] He also failed to place sufficient weight on the precision rate results achieved in round 6 or consider whether that rate could be achieved through further training. It was further contended that the Plaintiff’s submission also disregarded the fact that the Protocol anticipated 10 to 15 training rounds.[37] Accordingly, it was submitted that the factual material before the Special referee did not support his conclusion on this point.
Alleged application of the incorrect tests
[33]Paragraphs 3(b) and 9(f)(iii) of the Plaintiff’s submissions.
[34][6.7] of the Special Referee’s report dated 14 July 2017 at Exhibit KAH-1 to Ms Howard’s affidavit sworn 29 August 2017.
[35][6.8] of the Special Referee’s report dated 14 July 2017 at Exhibit KAH-1 to Ms Howard’s affidavit sworn 29 August 2017.
[36]Annexure 1 to the Special Referee’s report dated 14 July 2017 at Exhibit KAH-1 to Ms Howard’s affidavit sworn 29 August 2017.
[37]Paragraph 3.4 of the Protocol, at Annexure A to the Court’s orders made on 1 December 2016.
Fifthly, it was contended by the First Defendant that the Special Referee made another error because he applied the incorrect test for discovery by ‘requiring’ that the TAR model display a ‘dramatic improvement’ before further training rounds would be appropriate.[38] It was submitted that this analysis failed to appreciate or consider that occasionally the performance of a TAR model can stall, and this can only be verified by further training rounds.[39]
[38][6.10] of the Special Referee’s report dated 14 July 2017 at Exhibit KAH-1 to Ms Howard’s affidavit sworn 29 August 2017 (emphasis added). The test for a ‘reasonable search’ for the purposes of r 29.01.1 of the Supreme Court (General Civil Procedure) Rules 2015 (Vic) should not require that the improvement be ‘dramatic’ or ‘substantial’ before a further training round is necessary.
[39][21] of Mr Smith’s affidavit sworn 29 August 2017.
It was also put that the Special Referee failed to appreciate or consider that a relatively small improvement in the performance of the TAR model can significantly reduce the volume of discovery to be reviewed manually.[40]
Second discretionary matter
[40][8] of Mr Smith’s affidavit sworn 12 September 2017.
Sixthly, the First Defendant responded to the submission of the Plaintiff that it had already commenced applying the TAR model over its privilege list.[41] The First Defendant responded with the submission that the extent to which the Plaintiff had embarked upon this exercise was not explained, and the affidavits on which it relied contained no evidence of any wasted costs. In any event, it was submitted that it was difficult to see how such costs were wasted because any documents manually reviewed for privilege to this point can be excluded from any further document review processes.
Third discretionary matter
[41][14] of the Plaintiff’s submissions.
Finally it was submitted by the First Defendant that the results of the TAR process had improved during the last three training rounds. These three rounds were conducted by the Plaintiff rather than the collaborative process initially engaged. It was put that the improved results may be attributable to having only one reviewer coding documents which were achieved efficiently (that is, approximately one week per round).
The validation process
Validation of the TAR process is an important step to satisfy the Court and other parties as to the adequacy and accuracy of the discovery which has been undertaken and to avoid the potential for dispute as to the validity of the process.
The objective is to provide transparent validation of the results generated by the TAR application.
It is a system of validation or quality control. However, the validation process is not a process which in itself can improve either the recall rate or the precision rate of the TAR algorithm.
A recent example which illustrates the importance of validation where TAR is used, is an order in a Federal Court proceeding made by Murphy J in Money Max Pty Ltd v QBE Insurance Group Ltd.[42] The Court ordered the respondent, which had used TAR for the purposes of giving discovery to the applicant, to provide a report to the applicant describing the manner in which the respondent had applied the technology for the purposes of giving discovery and the results of the application of TAR. This was a form of validation of the TAR process used by the respondent in the course of its discovery.
[42]Federal Court of Australia (Murphy J) VID 513/2015 [7 November 2016].
Judge Andrew Peck in his opinion in the Da Silva Moore case[43] said this about the validation process:
An important aspect of cooperation is transparency in the discovery process … such transparency allows the opposing counsel (and the Court) to be more comfortable with computer assisted review, reducing fears about the so-called "black box" of the technology. This Court highly recommends that counsel in future cases be willing to at least discuss, if not agree to, such transparency in the computer-assisted review process.
[43] Da Silva Moore et al v Publicis Groupe et al 11 Civ. 1279 (ALC) (AJP) (S.D.N.Y) [24 February 2012] at p.23. Judge Peck’s decision was upheld by Judge Andrew L. Carter, Jr, United States District Court, Southern District of New York.in Da Silva Moore et al v Publicis Groupe et al 11 Civ 1279 (ALC) (AJP) (S.D.N.Y.) [26 April 2012].
Quality assurance achieved through an acceptable validation process is a key element in the use of TAR for document review.
Conclusions
I arrive at the following conclusions on the reception of the Special Referee’s Report.
Validation issue
At paragraph 6.7 of the 14 July Report the Special Referee recited the principal submissions advanced by the parties.
In the course of this exercise, the Special Referee noted ‘[t]he validation process is only applicable if the TAR process has stabilised. This may never occur’. However, it is apparent that this was not recorded as an express finding made by the Special Referee. Rather, it was, in its context, a recitation of a submission made by the Plaintiff.
However, in paragraph 6.10 the Special Referee observed:
Given the difficulties which have been experienced I conclude that it is unlikely there will be any dramatic improvement in the precision rate which would warrant further training rounds. In these circumstances validation as set out in the original joint protocol is not warranted or likely to reduce the number of documents in the database. Further training rounds will also increase the cost and the delay in the preparation of this matter for trial. As J Forrest J as stated the resources should be allocated for the trial, not the discovery fight.
[Emphasis added]
In one respect, the finding emphasised above was correct. The validation was indeed not likely to reduce the number of documents in the database. This was not the function of validation.
However, this did not mean that validation was not warranted. For the reasons which follow, I find that validation is clearly warranted.
Nevertheless, I am not satisfied that this reasoning in the 14 July Report resulted in any substantive error in a principal recommendation he made, being recommendation 7.1.1, namely recommending that the Plaintiff should not be required to undertake any further training of the TAR algorithm.
For the reasons which follow, I find that this particular recommendation should be adopted by the Court.
First discretionary matter
As to the first discretionary matter, I am not satisfied that this factor was something which the Special Referee did not take into account.
The Special Referee noted in paragraph 6.7 the thrust of the First Defendant’s submission in the following terms:
The defendant says that the precision rate is too low and the algorithm has not stabilised. The plaintiff should be required to undertake further training rounds or at the very least be required to undertake the training if I recommended that TAR continue.
The Special Referee then concluded in paragraph 6.8 that there was ‘merit in both arguments’. In other words, he considered the arguments of the parties and expert evidence there relied upon, including the material advanced by of the First Defendant, which he considered to have had merit.
Further, the Special Referee also annexed to his report both the submissions advanced by the Plaintiff and those of the First Defendant.
I conclude that the Special Referee took the material submitted by the First Defendant into took into account in making his recommendations.
I am satisfied that the Special Referee appropriately weighed the matters before him in determining that the Plaintiff should not be required to undertake any further training of the TAR algorithm.
I find no error on this ground which would warrant not accepting the Special Referee’s Report in relation to his recommendation 7.1.1.
Procedural matter
I accept the submissions raised by the First Defendant that it did not have the opportunity to advance a case as to whether the validation process should cease, as found by the Special Referee in paragraph 7.2.2.
Even though the validation process proposed to be abandoned was that prescribed in the ‘original protocol’, which contemplated a collaborative review process, I am satisfied that the First Defendant was not given the opportunity to make submissions or provide evidence to the Special Referee before he delivered his report to the Court as to the following matters:
(a)whether the validation process prescribed in the original protocol should continue to apply up until the time when the collaborative process ceased;
(b)what new process of validation (if any) should apply following the cessation of the collaborative process; and
(c)what, if any, recommendations should be made in the light of matters (a) and (b) above, which bear upon the recommendations made in paragraphs 7.2.3 and 7.2.4 of the 14 July Report.
It is to be noted that the parties and the Special Referee were working to a very tight timetable. The Plaintiff first put forward its proposal for the validation process to be abandoned in submissions to the Special Referee (copied to the First Defendant’s lawyer) on 13 July 2017 sent by email at 10:40 am.[44] The First Defendant had already filed its submissions at 10:38 am.[45]
[44]Annexure 3 to the Special Referee’s report dated 14 July 2017, at Exhibit KAH-1 to the affidavit of Kathryn Anne Howard sworn 29 August 2017.
[45]Annexure 4 to the Special Referee’s report dated 14 July 2017, at Exhibit KAH-1 to the affidavit of Kathryn Anne Howard sworn 29 August 2017.
Accordingly, in the interests of justice, I have determined not to adopt paragraphs 7.2.2; 7.2.3 and 7.2.4 of the Special Referees’ Report.
Given the importance of the validation process in the use of TAR for document review, in the interests of justice, it would be appropriate to order that the following questions be referred to the Special Referee for his opinion and report:
(a)whether the validation process prescribed in the original protocol should continue to apply up until the time when the collaborative process ceased;
(b)what new process of validation (if any) should apply following the cessation of the collaborative process; and
(c)what, if any, recommendations should be made in the light of matters (a) and (b) above, which bear upon the recommendations made in paragraphs 7.2.3 and 7.2.4 of the 14 July Report.
I will make an order to this effect.
Alleged erroneous assessment of the evidence and alleged application of the incorrect tests
This ground turns on whether the Special Referee, having allegedly disregarded one factor said to be relevant, failed to place sufficient weight on the precision rate results achieved in round 6 or consider whether that rate could be improved through further training.
The Special Referee had before him the expert evidence put on by the Plaintiff (FTI Consulting) which was annexed to his report.
FTI Consulting concluded that:
Taking the above into account, it is our opinion that additional review is unlikely to yield substantial improvement over the results already obtained.
The Special Referee also had before him the expert evidence put on by the First Defendant (Law in Order), a summary of which was also annexed to his report in the First Defendant’s submissions.
Law in Order concluded that:
It is Law in Order’s view that the model has not stabilised so as to warrant concluding further review rounds. The precision is still low, meaning that more than half of the ~218,000 documents identified by the TAR will likely be a false positive.
(…)
Accordingly, it is Law in Order’s view that it is premature to conclude the TAR.
Law in Order recommend that two further training rounds be completed, following the standard procedure of 1,000 documents per training round.
The Special Referee was also entitled to draw on his own expertise in the area of TAR discovery in his assessment of the competing views of the experts. The Special Referee based his findings upon a choice between conflicting expert evidence. He was entitled to do. The Court should not in this case interfere with those findings.
Further, insofar as it was contended by the First Defendant that the Special Referee fell into error because he:
(a) applied the incorrect test for discovery by ‘requiring’ that the TAR model display a ‘dramatic improvement’ before further training rounds would be appropriate; and
(b) failed to appreciate or consider that occasionally the performance of a TAR model can stall, and this can only be verified by further training rounds.
I am not satisfied that the Special Referee in fact applied such a test. What the Special Referee decided in this respect in paragraph 6.10 of his report was that:
Given the difficulties which have been experienced I conclude that it is unlikely there will be any dramatic improvement in the precision rate which would warrant further training rounds. In these circumstances validation as set out in the original joint protocol is not warranted or likely to reduce the number of documents in the database. Further training rounds will also increase the cost and the delay in the preparation of this matter for trial. As J Forrest J as [sic] stated the resources should be allocated for the trial, not the discovery fight.
Looking at the 14 July Report overall, I am satisfied that in paragraph 6.10 cited above the Special Referee was doing no more than expressing his conclusion that, balanced against the cost of the Plaintiff conducting further training rounds and the delay involved in progressing the matter to trial, there was no sufficient likelihood of achieving an improvement in the precision rate sufficient to justify the cost of further training rounds and the consequent delay in completing the discovery process in readiness for trial.
This was a decision which the Special Referee was duly called to make and express his opinion upon. I find no reason to interfere with his recommendation 7.2.1 on this basis.
It was also put that the Special Referee failed to appreciate or consider that a relatively small improvement in the performance of the TAR model can significantly reduce the volume of discovery to be reviewed manually.
The 14 July Report exhibits a balanced approach to the assessment of the subject matter of the expert evidence and the cost factors involved in the Plaintiff either progressing with TAR or not. TAR has already successfully worked to substantially reduce the discovery collection from approximately 1.4 million documents to approximately 218,000. This is a very considerable inroad into reducing the number of documents which need to be manually reviewed or reviewed by other electronic means. As found by the Special Referee, in my assessment there are clear risks in achieving any further reduction sufficient to warrant progressing further with TAR in this case. In undertaking the assessment as a proportional exercise, the costs and benefits of proceeding further with TAR has not been established by the First Defendant to any extent sufficient to warrant disturbing the assessment of the Special Referee.
I add that, if I was to determine the matter afresh, I would have reached the same conclusion.
Applying the principles in Wenco cited above, in the interests of justice, and in the exercise of the Court’ discretion I will adopt paragraph 7.2.1 of the 14 July Report and order that the Plaintiff is not required to undertake any further training of the TAR algorithm.
Second discretionary matter
In arriving at my determination of this application I do not take into account or give any weight to the fact that the Plaintiff, apparently, already commenced applying the TAR model over its privilege list.
Third discretionary matter
Further, in arriving at my determination of this application I do not accord any weight to the position advanced by the First Defendant that improved results achieved during the last three training rounds may be attributable to having only one reviewer coding documents. This is, with respect, a speculative submission which is not sufficient to disturb my finding that the Plaintiff should not be required to undertake any further training of the TAR algorithm.
Orders
It will be ordered that:
1.Paragraph 7.2.1 of the 14 July Report be adopted by the Court with the result that the Plaintiff should not be required to undertake any further training of the TAR algorithm.
2.Paragraphs 7.2.2, 7.2.3 and 7.2.4 of the 14 July Report are not adopted by the Court.
3.The following questions are referred to the Special Referee for his opinion and report:
(a)whether the validation process prescribed in the original protocol should continue to apply up until the time when the collaborative process ceased;
(b)what new process of validation (if any) should apply following the cessation of the collaborative process; and
(c)what, if any, recommendations should be made in the light of matters (a) and (b) above, which bear upon the recommendations made in paragraphs 7.2.3 and 7.2.4 of the 14 July Report.
On the question of costs, by reason of the novelty of the subject matter of this application and the fact that both parties have achieved a measure of success, I will order that:
1. The costs of this application be costs in the cause.
---
0
4
0