Linguis International Institute of Language and Culture Limited v New Zealand Qualifications Authority
[2016] NZHC 691
•14 April 2016
IN THE HIGH COURT OF NEW ZEALAND AUCKLAND REGISTRY
CIV-2016-404-158 [2016] NZHC 691
BETWEEN LINGUIS INTERNATIONAL
INSTITUTE OF LANGUAGE AND CULTURE LIMITED
Plaintiff
AND
THE NEW ZEALAND QUALIFICATIONS AUTHORITY Defendant
Hearing: 7 and 8 April 2016 Appearances:
B O'Callahan and M Chen for plaintiff
R Scott and M Cavanaugh for defendantJudgment:
14 April 2016
JUDGMENT OF LANG J
[on application for judicial review]
This judgment was delivered by me on 14 April 2016 at 3.30 pm, pursuant to Rule 11.5 of the High Court Rules.
Registrar/Deputy Registrar
Date……………
LINGUIS INTERNATIONAL INSTITUTE OF LANGUAGE AND CULTURE LTD v THE NEW ZEALAND QUALIFICATIONS AUTHORITY [2016] NZHC 691 [14 April 2016]
[1] Linguis International Institute of Language and Culture Limited (Linguis) is a private training establishment (PTE) registered under s 233 and Part 19 of the Education Act 1989 (the Act). It provides tertiary education to students predominantly from India. It offers diploma courses in business and general English, together with certificates in English across various levels. It operates from two premises, one in Auckland and one in Christchurch.
[2] The NZQA (the Authority) is a Crown entity created and empowered under s
246A of the Act. It is the primary quality assurance body for tertiary education in New Zealand, and has statutory obligations to oversee, monitor and review performance and accreditation standards in respect of tertiary education providers including PTE’s such as Linguis.
[3] In November 2014 the Authority commenced a review process designed to enable it to express the level of its confidence in Linguis’ educational performance and capability in relation to self-assessment. At the end of that process the Authority produced a final report on 3 February 2016. This stated that the Authority was not confident about either aspect of Linguis’ performance.
[4] In the present proceeding Linguis seeks judicial review of several of the conclusions reached in the final report and aspects of the procedure followed before the report was issued.
[5] The Authority now accepts that there were shortcomings in the final stage of the process used to produce the report. As a result, it acknowledges that the report must be set aside. The Court is now required to determine the terms upon which the review process is to be completed.
Statutory framework
[6] In order to understand the issues the present proceeding raises, it is necessary to have some understanding of the relevant statutory framework. I am indebted to counsel for their succinct summaries of this, which I largely adopt.
[7] The Authority was required to undertake the review process in accordance with the External Evaluation and Review Rules 2013 (EER Rules). These were promulgated under s 253 of the Act, and create a framework within which the Authority carries out periodic External Evaluation Reviews (EER’s) of PTE’s on a regular basis. EER’s involve a sampling process that enables the Authority to issue reports containing “Statements of Confidence” in respect of a PTE’s performance and capability in self-assessment. The latter refers to how well the PTE understands its own performance, and how it uses this understanding to improve the educational services and facilities that it provides.
[8] In order to achieve certainty and consistency in the review process the Authority issued a detailed “Policy and Guidelines for the Conduct of External Evaluation and Review” document (the Policy and Guidelines document) on 1
September 2009. This prescribes in detail the procedure to be followed when an EER is undertaken. The Policy and Guidelines document identifies key evaluative questions that need to be considered in relation to each area that is the focus of the review. When answers to these have been obtained, the review team must synthesise them into organisational level judgments regarding the educational performance and capability in self-assessment of the PTE that is the subject of the review. Once the review team has produced a draft report, the PTE is given an opportunity to make submissions in respect of it.
[9] The review process results in a categorisation of the PTE from 1 (highest) to
4 (lowest). The category assigned to a PTE as a result of the EER produces several consequences, one of the most important of which is the frequency of future EERs. Category 4 institutions are reviewed every six to 12 months, Category 3 institutions every 12 to 24 months, and Category 1 and Category 2 every four years. The
categorisation of a PTE is published on the Authority’s website along with final
reports.
[10] The Authority publishes the final report and categorisation on its website so that it is accessible to the public. Prior to publication, a PTE may apply on specified grounds for the appointment of an independent reviewer to reconsider the Statements of Confidence expressed in the report. Should that occur, publication of the report will not occur until such time as the Authority has received the recommendations of the independent reviewer.
[11] The statutory scheme has several purposes. First, it ensures that accreditation standards are maintained consistently on a nationwide basis. Secondly, it provides the public with access to the categorisation and history of the institution through the EER reports. This enables staff and students to make an informed choice as to their place of employment or study. Thirdly, it provides a yardstick against which the institutions can measure their performance against objective standards.
Background to the present proceeding
[12] The first EER that the Authority conducted in respect of Linguis culminated in a report published on 27 October 2010. This classified Linguis as a Category 1 institution.
[13] The present proceeding arises out of the second EER that the Authority commenced in 2014. After setting the scope of the review, the review team conducted on-site visits on 17, 18 and 19 November 2014. The review team then issued a first draft report on 23 December 2014 and provided Linguis with a copy for its comments. Linguis provided its comments in respect of the draft report on 13
March 2015.
[14] The team then returned to the site on 8 May 2015. During this visit, two members of the review team sampled 24 assessments handed in by students to check to see whether they contained plagiarism. Following the on-site visit, the review team issued a second draft EER report on 11 August 2015. This proposed a Category 4 classification. Linguis provided the review team with its comments in
respect of the second draft report within the time given to it. A first final report was then issued on 3 September 2015. This continued to propose a Category 4 classification.
[15] Linguis exercised its right under r 9 of the EER Rules to apply for a reconsideration of the report on the basis that there were alleged process failures and errors of fact and/or judgment. The Authority then appointed an independent reviewer, Mr David Andrews, to conduct the reconsideration process. Mr Andrews provided Linguis with a draft reconsideration report on 8 December 2015 and Linguis provided its comments in respect of that report on 22 January 2016. Mr Andrews then issued a final reconsideration report and the Authority issued a second final EER report on 3 February 2016.
[16] Linguis immediately issued the present proceeding, together with an application for interim relief preventing publication of the final EER report. The interim position has been resolved through undertakings given by the Authority pending determination of this proceeding.
Grounds of review
[17] Linguis advances two grounds of review. The first is that the final report contains numerous errors of fact that materially affected the ultimate findings and classification made by the review team. The second is that the reconsideration process undertaken by Mr Andrews was flawed because he deliberately put to one side and failed to take into account several aspects of the material that Linguis asked him to take into account.
[18] For Linguis, Mr O'Callahan originally submitted that the Court should set aside the EER report and direct the Authority to take no further steps to complete the review process. During the course of the hearing Mr O'Callahan advised me that Linguis had modified its stance. It now asks the Court to set aside the final report and to direct the review team to reconsider its findings and expressions of confidence taking into account the Court’s conclusions in relation to both grounds of review. This would require the review team to participate in further evaluative discussions with Linguis about matters that remain in dispute.
[19] The Authority resists the first ground based on errors of fact, and contends that there is no justification for the Court to review the second final report on that basis. It concedes the second ground, however, because it accepts that Mr Andrews put to one side factual material that he ought to have taken into account. He did so in the mistaken view that a set of guidelines recently issued by the Authority prohibited him from taking that material into account.
[20] The Authority submits that the Court should set aside the second final EER report and the reconsideration report. It should then direct Mr Andrews to recommence the reconsideration process taking into account the material that he did not consider when he prepared his reconsideration report.
First ground: reviewable error of fact
[21] The traditional view, espoused in cases such as Lewis v Wilson & Horton Ltd1 and Daganyasi v Minister of Immigration,2 is that an error of fact will only be reviewable where it goes to the jurisdiction of the decision making body to make the decision, or where it renders the decision in question unreasonable or based on an
error of law. This approach has been applied recently at first instance in cases such as Diagnostic Medlab Ltd v Auckland District Health Board3 and Mary Moodie Family Trust Board Inc v Attorney-General.4
[22] Linguis submits that the jurisdiction to review decisions based on errors of fact is wider than these cases suggest. Mr O'Callahan points out that the Courts have also been prepared to intervene to correct errors of fact made in circumstances where the decision maker has not sufficiently informed itself on facts central to the decision
it was required to make.5 The Courts have also been prepared to intervene where
findings of fact have been reached through an unfair process or through a flawed reasoning process.6 In addition, judicial review has been held to be available where
findings of fact are inconsistent, self-contradictory and/or logically flawed.7
1 Lewis v Wilson & Horton Ltd [2000] 3 NZLR 546 (CA).
2 Daganyasi v Minister of Immigration [1980] 2 NZLR 130 (CA).
3 Diagnostic Medlab Ltd v Auckland District Health Board HC Auckland CIV 2006 4040 4274,
20 March 2007.
4 Mary Moodie Family Trust Board v Attorney-General [2015] NZHC 365 at [122].
5 CREEDNZ Inc v Governor-General [1981] 1 NZLR 172 (CA) at [200].
6 Re Erebus Royal Commission; Air New Zealand v Mahon [1983] NZLR 662 (PC) at 681;
[23] The starting point in this context is to determine whether Linguis is able to point to any factual errors that underpin the conclusions of the final version of the draft report.
The alleged factual errors
[24] In his written submissions Mr O’Callahan contended that the following factual findings in the final report constituted reviewable errors of fact:
(a) The finding that there were deficiencies in the English proficiency of students accepted into Linguis’ programmes, when in fact Linguis’ policy was more stringent than the Authority’s requirements. To the extent that some students did not meet Linguis’ current requirements, those students had been accepted under the Authority’s requirements during a short period before Linguis made its own requirements more stringent;
(b)The finding that rapid growth in student numbers resulted in Linguis maintaining inappropriate and overcrowded premises that negatively affected the student environment in learning and teaching, when in fact the period of overcrowding was for a limited period and had been resolved prior to the first visits in November 2015. In addition, there was no evidence of learning outcomes being affected during the period in which it persisted;
(c) The finding that Linguis’ processes for monitoring and reporting attendances were inconsistent and that some records were unreliable, when in fact the issue was for a limited period and had been resolved prior to the first visits and there was no evidence of learning outcomes
being affected during the period in which it persisted.
Roussel Uclaf Australia Pty Ltd v Pharmaceutical Management Agency Ltd HC Wellington
CP9/96, 13 August 1997.
7 Re Erebus Royal Commission, above n 6, at 681; Lalli v Attorney-General [2009] NZAR (HC)
at [76].
(d)The finding that evidence of a high degree of undetected plagiarism in student’s work undermined the validity of achievement data when there was no probative evidence of this;
(e) The finding that Linguis did not have adequate systems or methods for detecting plagiarism, when in fact it used systems and methods recommended by the Authority.
(f) The finding that the use of summative appraisal as a teaching method learning tool was inappropriate, when that approach to teaching is acknowledged as good teaching practice by the Authority’s own publications.
[25] I propose to deal with each of these in turn, although not in the order set above.
Deficiencies in English proficiency
[26] This issue has its genesis in several observations made in the report to the effect that there were significant issues with students not having sufficient proficiency in English to undertake their proposed course of study.
[27] In order to understand this issue it is necessary to briefly outline the means by which Linguis has endeavoured to ensure that students recruited from overseas are sufficiently proficient in English to undertake the courses for which they are accepted. Linguis relies upon agents in India to market the educational services that it offers and to screen candidates for admission. Up until June 2014 the principal screening tool was an English language proficiency test that Linguis has developed and provided to its agents. Candidates for admission were required to pass this test, which was usually administered by its overseas agents. Alternatively, it would accept candidates who had spent at least five years attending educational institutions in India where English was the language used for teaching purposes.
[28] During the first half of 2014, however, Linguis experienced an unprecedented increase in the number of students who were able to meet its entry requirements. As
a result, there was a dramatic increase in the number of students arriving from overseas at both of its sites. From 26 June 2014 candidates for admission were required to achieve a prescribed score in the internationally recognised IELTS examination for English proficiency.
[29] When the review team spoke to staff members on-site, they learned that large numbers of students had arrived in New Zealand during 2014 with a very limited command of English. These students were having considerable difficulty coping with the courses in which they were enrolled. Although English language assistance was offered to students, the review team gained the impression that very few students were taking advantage of the offer.
[30] The final report contains the following passage in the Summary of Results section:
·English language levels remain a significant barrier to learner achievement for some currently enrolled students, despite changes to entry requirements having been introduced midway through 2014.
[31] The “Findings” section of the report also refers to the fact that the review team had heard about “the barriers to achievement posed by low English language levels among many students in the Auckland campus”. Later in the same section, the report records that “the low English language levels of some current students also present a significant challenge for teachers, and although additional English language assistance is available, very few students take up the opportunity”.
[32] The same section of the report also states that “a review of a random sample of student files indicates that changes to English language entry requirements have not been consistently implemented”. Linguis takes issue with this particular statement. It says the statement relates to a small number of students who were accepted for admission after June 2014 although they had not passed the required IELTS test. Linguis acknowledges that this occurred, but says that it had earlier offered these students placements before the new language requirements came into effect.
[33] I consider this particular issue to be a distraction. Even if Linguis’ understanding of the position is correct, the impugned comment forms a tiny part of a large section of the report. In addition, the review team ultimately rated Linguis’ performance and capability in self-assessment in relation to that particular key evaluative question as “Good”. It is unlikely to have played any significant role in the ultimate conclusions that the review team made.
[34] The important conclusions for present purposes are those contained in the Summary of Results section of the report, because they directly inform the review team’s statements of confidence in Linguis’ performance and capability in self- assessment. The review team’s conclusion in the Summary of Results was that English language levels remain a significant barrier to learner achievement for some currently enrolled students despite the alteration in Linguis’ entry requirements. There was ample material to support that conclusion, and there is nothing to suggest it was wrong. It follows that the statement cannot be regarded as a reviewable error of fact.
The negative effects produced by the rapid increase in student numbers
[35] This issue also arises as a result of the rapid increase in student numbers during 2014. The statements in the Summary of Results section with which Linguis
takes issue are as follows:
· Linguis has experienced high rates of growth over recent years. This growth has not been well-managed, at least during 2014, resulting in large classes, overcrowding and inadequate physical and staff (sic). The poor quality of the environment has negatively affected teaching and learning, as well as being non-compliant with the PTE’s own quality management system. ·
New delivery sites have now been established and additional staff recruited. Timetabling changes have improved the management of class sizes, although they have also resulted in a compressed teaching timetable which includes 10-hour days for a small number of students and teachers.
[36]
The
evidence demonstrates overwhelmingly that these conclusions are
factually correct. The facilities in both Auckland and Christchurch were placed under very considerable pressure for several months by the rapid increase in student
numbers during 2014. During this period classroom facilities were not big enough to cope with the influx of students and toilet facilities were also inadequate. Class numbers reached an unacceptably high level, and in some cases classrooms were only able to be used because a high level of absenteeism reduced the number of students to a level where desks were available for all students. These types of issues were obviously unsatisfactory, and must have had a negative impact on the learning experience of the students.
[37] Linguis candidly acknowledged these problems to the review team and also explained the measures taken to deal with them. These included the acquisition of further facilities (delivery sites) and the introduction of flexible teaching hours. Some students were asked to attend classes for ten hours on two days of the week so that classrooms could be used by other students on the other three days.
[38] In his closing submissions Mr O'Callahan attempted to overcome this hurdle by submitting that the tone of the observations made in the paragraph under the first bullet point was unduly negative. He submitted that they should have been balanced by setting out the steps that Linguis had taken to deal with the issues that it faced. This submission overlooks the observations made in the paragraph under the second bullet point.
[39] I do not consider that either of these paragraphs can realistically be described as containing reviewable errors of fact.
Inadequate and inconsistent monitoring of attendances
[40] This issue is not mentioned in the Summary of Issues but is the subject of the following comment in the “Findings” section of the second final report under the heading “How well are learners guided and supported?”:
While Linguis has clear policies and guidelines for guidance and support of students, the application of these is inconsistent. A review of a random sample of student files indicates that changes to English language entry requirements have not been consistent implemented. It also appears that processes for monitoring and reporting attendance are inconsistently followed, and that some records are unreliable.
[41] For the Authority, Ms Scott has directed me to a large body of evidence suggesting that the attendance of students at class was not monitored on a uniform or consistent basis. There is reference in this material to attendance not being checked at all on at least one occasion. At other times attendance checks were conducted on numerous different bases. These ranged from hourly checks to checks carried out two to three times per day and at one stage up to seven times each day. The relevance of this was that there appears to have been a high level of absenteeism, a matter that could only be established through reliable and consistent checks being carried out.
[42] Two points can be made about this ground of review. The first is that the impugned statement consists of a single sentence in a lengthy report. It does not appear in the Summary of Results section, and Linguis can scarcely suggest that it assumed any significance in terms of the ultimate conclusions as to confidence reached in the report. The second is that there is no evidence to suggest that the statement is factually incorrect. It therefore does not qualify as a reviewable error of fact.
Plagiarism
[43] This issue arises because of the following paragraphs in the Summary of
Results section of the final report:
·There is also widespread evidence of systematic plagiarism among students enrolled in the business programmes. NZQA found evidence of high rates of reported plagiarism (gathered through interviews and document reviews) and high rates of undetected plagiarism (based on sampling of marked assessments).
·The scale of this plagiarism brings into question the reliability of the reported figures on educational achievement, and the processes leading to them. Some Linguis academic staff seemed unclear about what constitutes plagiarism, which in itself is likely to limit Linguis’ ability to resolve the problem fully and satisfactorily.
[44] In addition, Linguis relies upon observations made in the “Findings” section of the report in relation to the key evaluation question “How well do learners achieve?” The rating for performance under this head was “Poor”, whilst the rating for capability in self-assessment was “Adequate”. The report stated:
1.1 How well do learners achieve?
The rating for performance in relation to this key evaluation question is
Poor.
The rating for capability in self-assessment for this key evaluation question is Adequate.
Linguis reports consistently high rates of overall qualification completion for the business programmes (refer Table 1). Achievement rates at the Auckland campus have improved each year since it was established in 2012. In Christchurch there is some variation in completion rates, which is attributed in part to the impact of the Christchurch earthquakes in 2011, although this explanation does not address the particularly low rates of completion of the level 6 diploma in 2011 and 2012.
Table 1. Qualification completion rates for the business diplomas at
Linguis, by campus, from 2011 to end term (3 September) 2014
2011
2012
2013
2014
Auck
Chch
Auck
Chch
Auck
Chch
Auck
Chch
National Diploma in Business (Level 5)
N/A
96%
86%
87%
90%
95%
95%
95%
National Diploma in Business (Level 6)
N/A
33%
81%
30%
96%
94%
98%
94%
Diploma in Business (Level 7)
N/A
N/A
91%
94%
91%
89%
99%
89%
All programmes
N/A
80%
86%
83%
92%
94%
97%
94%
Although the reported rates of achievement overall are quite strong, the evaluation team finds that these rates are questionable. Many interviewees and Linguis documents identified very high plagiarism rates as an issue, particularly in the level 5 diploma. This was confirmed by a sampling of assessments. Rates of plagiarism quoted in interviews and documents varied between 20 and 50 per cent. NZQA notes that most assessments are submitted in hard copy and Linguis largely relies on manual identification of plagiarised material, which is less reliable than the use of specialised software. Assignments sighted by the evaluation team did not comply with Linguis’ own policy on the use of APA referencing and yet they did not contain any assessor comments on referencing or plagiarism. There is no data available to substantiate the assurances from Linguis that steps taken to raise awareness among students and to increase staff ability to detect incidences of cheating and plagiarism have been effective.
[45] Mr O'Callahan submits, and the evidence confirms, that plagiarism is an issue that all educational institutions, and particularly those at tertiary level, are required to confront. Plagiarism in this context is the use of material taken from other sources without referencing or attributing the material to its source.
[46] Linguis relies under this head on the fact that the review team has used its conclusion regarding the existence of high rates of plagiarism by Linguis students to undermine the validity of the reported rates of achievement. The report acknowledged that the reported rates of achievement overall were quite strong. It considered, however, that the existence of high rates of plagiarism meant that the reported rates of achievement were questionable.
[47] Mr O'Callahan points out, correctly in my view, that the existence of plagiarism should only affect the reliability of reported rates of achievement where the plagiarism has not been detected. In common with other educational institutions Linguis does not accept and will not mark assessments or assignments where tutors discover that plagiarism has occurred. For that reason detected plagiarism cannot undermine the reported rates of achievement. Those figures will only be undermined where there is reason to believe that a significant amount of plagiarism is not being detected.
[48] In the present case the review team obtained its information from several sources. The first was from documents that Linguis provided to the review team at the beginning of the review. These included Course Summary Evaluation Reports prepared by Linguis tutors in respect of the courses they had taught. The review team also interviewed staff members during the on-site visits. There are numerous references to plagiarism in the notes taken during the course of these interviews and in the evaluation reports prepared by the course tutors.
[49] It is clear from these documents that staff members knew that Linguis students were regularly using plagiarised material in assessments that they handed in to be marked. Staff members attributed this in part to the fact that plagiarism is apparently not regarded as an issue in India, and Indian students often do not immediately appreciate that it is unacceptable in this country. Linguis has
endeavoured to deal with this issue by emphasising the unacceptability of plagiarism in the orientation sessions that are held when students first arrive. Linguis has also emphasised to tutors that they should not tolerate plagiarism when they encounter it.
[50] The third source from which the review team gained information about plagiarism was through the 24 assessments that were examined for plagiarism during the on-site visit on 8 May 2015. Evidence about this was given by Ms Anna Castelle, who has considerable experience in carrying out EER’s. Ms Castelle says that after arriving at the Linguis premises the members of the review team were directed to a large room where they met Linguis representatives. After the initial discussion ended Ms Castelle and another member of the review team asked to look at student assessments. They were then taken to a storage room that was full of boxes.
[51] Ms Castelle says that she and her colleague then looked through the boxes to find assessments that had been marked prior to November 2014. They decided to examine assessments containing continuous, or essay-type, writing rather than short answers. In addition, they decided to examine assessments that did not contain any references attributing material within the assessment to outside sources.
[52] Ms Castelle and her colleague then took eight to ten boxes from the storage room to the main meeting room where they examined the contents. In all they chose
24 assessments to examine for plagiarism. Ms Castelle says that she considered that these contained language and grammatical structure beyond the level to be expected from Level 5 or 6 students for whom English was a second language. Ms Castelle and her colleague then selected sections of the assessments and entered them into Google on the laptop computer they had brought to the site. This exercise immediately revealed that all but one of the assessments contained passages that had been taken virtually without alteration from an outside source without attribution being made. None of the assessments contained any indication that the plagiarised material had been discovered by the tutors who had marked them.
[53] Ms Castelle then asked Linguis staff to take copies of the 24 assessments so that she could take them back to her office. Once this had been done Ms Castelle left the 24 assessments in a pile on the table of the main meeting room.
[54] Mr O'Callahan advances several criticisms of the process adopted by Ms Castelle and her colleague and the conclusions reached as a result of that process. First, he submits that the selection of assessments could not realistically be described as random because of several factors. First, he challenges the criteria that Ms Castelle applied when selecting the assessments to be examined. Mr O'Callahan contends that her decision to only examine assessments that did not contain any references meant that the sample would inevitably produce an extremely high level of plagiarism. Secondly, he points out that the assessments that were examined had only been marked by four tutors, none of whom Linguis now employs. Thirdly, Linguis students submit approximately 50,000 assessments per year. Mr O'Callahan suggests that an examination of just 24 assessments is not sufficient to produce any reliable conclusions regarding the overall level of undetected plagiarism by Linguis students. Furthermore, only seven of the assessments achieved a passing grade. As a result, those were the only assessments that had the ability to undermine Linguis’ reported levels of achievement.
[55] I accept these submissions as far as they go, but they need to be considered having regard to the purpose of the EER process and the manner in which it is carried out. As Ms Scott points out, the process is not designed to enable the Authority to make a definitive judgment regarding the issues of performance and capability in self-assessment. Rather, the ultimate report only goes so far as to express statements as to confidence in relation to those issues. This reflects the fact that the review team uses observations made by the institution’s staff as its primary source of information, supplemented as necessary by sampling of material that may be relevant to any issues of concern arising during the review.
[56] The Policy and Guidelines document issued by the Authority summarises these observations succinctly in the following passage:
It is not possible, nor defensible, to make a judgement about the overall performance of the organisation based on the sample of focus areas. This is
because the sample is unlikely to be sufficiently representative or statistically significant. However, the sample is sufficient to support the conclusion about how confident the evaluation team is in the TEO’s performance.
[57] In the present case plagiarism was an important issue from the outset because Linguis’ staff consistently raised it in their written evaluations and in their on-site discussions with the review team. Although these concerns no doubt arose because of the extent to which tutors had detected instances of plagiarism, they also raised a wider issue as to the extent to which plagiarism was not being detected. The evidence suggests that Linguis had the capability to use computer software to detect plagiarism but most tutors elected to check for plagiarism manually. It is unrealistic to suggest that checks of this type would detect every instance of plagiarism. I therefore consider that the extent to which tutors were detecting plagiarism leads inevitably to the conclusion that a significant body of plagiarism was also not being detected. The examination of the 24 assessments tended to confirm this because there was nothing to indicate that the plagiarism in those assessments had been detected by the marking tutors regardless of whether the assessments passed or failed.
[58] I therefore do not consider that the comments made in the report about plagiarism can realistically be described as reviewable errors of fact. They were conclusions that were available to the review team based on all of the information that they had accumulated.
[59] During the hearing Mr O'Callahan also submitted that the process adopted by the review team in relation to the 24 assessments was unfair. He argued that Linguis has never had a proper opportunity to make submissions to the review team about the shortcomings in the sampling and reasoning processes adopted by Ms Castelle and her colleague. He contended that Linguis ought to have a further opportunity to engage in an evaluative discussion with the review team regarding these issues.
[60] I am not prepared to accept that submission for two reasons. First, I consider that Linguis has already had ample opportunity to put its side of the story in relation to the 24 assessments. It knew that Ms Castelle had carried out an examination of those assessments on 8 May 2015 because staff members watched it occur. At the
closing meeting on 8 May 2015 Ms Castelle also advised Linguis personnel of the results of her examination. At that point Linguis could have carried out its own examination of the assessments if it wished to do so. It could also have raised the issues that it now wishes to raise in the submissions that it made to the review team after 8 May 2015.
[61] Secondly, the sampling exercise does not stand on its own. It must be viewed in the light of the information provided to the review team by Linguis tutors. This was overwhelmingly to the effect that plagiarism by Linguis students was rife. For the reasons I have already given I do not consider that these comments raise only a concern about the levels of detected plagiarism. They also raise a concern about undetected plagiarism. The sampling process merely confirmed the validity of that concern. Even if the overall validity of the sampling process and reasoning is called into question, the concerns expressed by Linguis staff remain a valid basis for the observations about plagiarism made in the final report.
[62] This ground of review fails as a result.
The high level of “re-sits”and late assessments
[63] This issue arises because the first draft EER report issued on 23 December
2014 contained the following passage in the Summary of Results section:
Linguis reports high levels of qualification completion over the previous three years. In response to two years of poor external moderation results, Linguis has sought to increase staff capability in assessment practice, including a greater focus on reducing academic misconduct. However, the reported incidence of plagiarism remains high. Despite changes to entry requirements being introduced midway through 2014, English language levels remain a significant barrier to learner achievement for some currently enrolled students. Many students require multiple attempts to pass assessments (reflecting Linguis’ reliance on the use of summative assessment as a learning tool) and/or students submit assessments very late. Taken together, these issues reduce confidence in the validity of student achievement rates.
(Emphasis added)
[64] The first draft report also contained the following statement in the “Findings”
section in relation to the key evaluative question “How well do learners achieve?”:
The high qualification completion rates are not consistent with some other evidence of student progress. … In addition, very few students are passing assessments on their first attempt; many need three or four attempts to pass.
[65] In the same section, under the heading “How effective is the teaching?”, the
report states:
The quality of teaching practice at Linguis is variable. … Students often submit assessments late (sometimes long after the delivery of relevant course material) and a very high rate of re-sits and resubmissions (despite the re-use of tests in some cases) does not reflect good teaching practice. Nor do these practices necessarily help maintain academic standards, as was suggested by some staff.
[66] When Linguis made submissions about the draft report, it challenged these findings, and in particular the implied criticism of the use of the summative teaching method. It pointed out that the Authority’s Policy and Guidelines document expressly recognises the use of that method. The summative method does not involve the grading of assessments in the same way that students at a university might have assignments or examinations marked on a percentage basis or graded from A to F. Instead, the summative method works on an Achieve or Not Achieve basis according to a prescribed unit standard. An assessment will either achieve the required unit standard or it will not.
[67] Linguis relies in this context on the following passages from the Policy and
Guidelines document:
Reassessment and fairness
One-chance assessment is not appropriate in this kind of assessment. Students are being assessed against the written standard, not against each other or against a set task. So there is every justification for giving learners further chance to show what they can achieve.
In the past, when one-chance assessment was the norm, it was considered unfair to allow a student to have a second try at a test or hand in an improved essay or bake another cake. When only a limited number could ‘pass’ or where students were compared with each other, it would have been seen as unfair on the other students.
Fair assessment becomes a matter of giving every learner ample opportunity to provide evidence about what they can achieve – especially if they do not succeed the first time but improve later. Each learner should be given the same opportunities to provide further evidence. If this means having access to resources, then you will need to make equitable provision. You could
make it the learner’s responsibility to provide the evidence – if you guarantee to accept the evidence they offer – then many of these aspects of fairness are eased.
[68] Linguis submits that this passage is a clear endorsement of the summative teaching method, and contends that the review team erred in fact by stating that it was not good teaching practice.
[69] The review team did not engage directly with Linguis regarding this issue. However, the second version of the report issued on 11 August 2015 omitted the references to it set out above at [64] and [65]. This suggests that the review team accepted the validity of Linguis’ concern about the issue. Surprisingly, however, the report retained the passage set out above at [66]. The Summary of Results section also stated that “the evaluation team identified serious failures in key aspects of educational practice”. Linguis is concerned that this refers at least in part to the fact that it uses the summative teaching method.
[70] I consider Linguis’ concerns under this head to be valid. The issue of re-sits and late assessments was clearly an important one from the outset, because it was listed as an issue of concern in a handwritten evaluation form that the review team’s Lead Evaluator, Ms Joanne Milne, completed shortly after the first on-site visit. This was the first written step in the process by which the answers given to key evaluation questions are synthesised into statements of confidence. It is also difficult to see why the review team was prepared to remove two of the references to the issue from its second report but not the third. Ms Milne has filed a lengthy affidavit in this proceeding, but she did not use that opportunity to explain why the review team took that step.
[71] The statement of claim alleges that the remaining reference to re-sits and late assessments is an error of fact. I do not consider it fits within that category of reviewable error because it may well be factually correct. The Policy and Guidelines document does not state that it is appropriate, for example, to permit students to have numerous opportunities to re-submit assessments. Nor does it expressly authorise the submission of late assessments.
[72] My concern is that the review team’s concern about the issue may have influenced the final outcome in circumstances where Linguis has never been told the reasons underlying the concern. As a result, Linguis has not had an opportunity to make submissions to the review team as to why the concern might be misplaced. That is an error of process, and if necessary I would have given Linguis leave to amend its statement of claim to reflect it. Given the fact that the report is to be set aside by consent, however, I consider that I can provide an appropriate remedy to deal with this issue in the ancillary directions that I am required to make.
[73] I consider that the issue needs to be dealt with by the review team rather than as part of the reconsideration process. The review team is the appropriate body to explain why it decided to retain one of the references to re-sits and late assessments in the report, and the weight that it gave to that issue.
[74] This means that it is not necessary for me to consider whether the Reconsideration Guidelines that the Authority has recently issued are ultra vires because they are in conflict with the EER Rules. That may have been an issue in the present case, because the new guidelines expressly exclude from the reconsideration process any challenge based on alleged error of judgment in relation to a matter that is not contained in the Summary of Results section of the report. The guidelines also state that the reconsideration process does not extend to a disagreement by a PTE with evaluative findings made in the main body of the report and not in the Summary of Results. The remaining statement relating to re-sits and late assessments was obviously an evaluative judgment, and it appeared outside the Summary of Results section of the report. As a result, the guidelines would have prohibited it from being part of the reconsideration process.
Second ground of review: procedural errors in the reconsideration process
[75] Given that this ground of review is conceded, it is not necessary to say more about it than has been set out earlier in this judgment.8
8 At [19].
Directions
[76] Both parties accept that the reconsideration report and the second final report issued on 3 February 2016 must be set aside, and I make orders to that effect by consent.
[77] My conclusion in relation to the first ground of review means that it will not be possible for the review to return to the reconsideration stage. It needs to return to an earlier point in the EER process so that the review team can explain why it considers the issue of re-sits and late assessments to remain relevant. Linguis should then have an opportunity to make submissions in relation to that issue. If the review team concludes that the sole remaining reference to the issue should be removed from the report, it will need to go on to consider whether any of its ultimate conclusions need to be reconsidered.
[78] I therefore direct the review team to provide a further draft report expressly stating why it considers the issue of re-sits and late assessments to be a matter of concern having regard to the statements the Authority has made in the Policy and Guidelines document. Linguis will have 14 days to provide submissions in response. The review team is then produce a final report within 14 days of receiving submissions from Linguis. Thereafter the procedure set out in the EER Rules is to govern the completion of the EER process.
[79] In the event that Linguis seeks reconsideration of the report under r 9, the person appointed to carry out that process must take into account the material that Mr Andrews erroneously excluded from the scope of his reconsideration of the first final report together with such other material as he or she considers to be appropriate.
[80] I reserve leave to both parties to seek further directions on 48 hours notice should that be necessary to implement the directions set out above.
Costs
[81] Linguis has succeeded in having the two reports set aside, but has failed in relation to all but one of the grounds it advanced in relation to the first ground of review. If counsel cannot reach agreement regarding costs, Linguis shall file and serve a succinct memorandum setting out the orders that it seeks no later than 6 May
2016. The Authority is to file and serve a brief memorandum in response within 14 days of receiving that memorandum. I will then deal with the issue of costs on the
papers.
Lang J
Solicitors:
Kirkland Morrison O’Callahan & Ho, Auckland
McElroys, Auckland
1
1
1