CQMS Pty Ltd v Motion Metrics International Corp
[2023] APO 2
•18 January 2023
IP AUSTRALIA
AUSTRALIAN PATENT OFFICE
CQMS Pty Ltd v Motion Metrics International Corp [2023] APO 2
Patent Application: 2016265876
Title:Method and apparatus for locating a wear part in an image of an operating implement
Patent Applicant: Motion Metrics International Corp
Opponent:CQMS Pty Ltd
Delegate:B. Norman
Decision Date: 18 January 2023
Hearing Date: 5 August 2021, via teleconference
Catchwords: PATENTS – wear part monitoring – s59 opposition – inventive step, support, clarity, clear and complete disclosure, manner of manufacture, utility – opponent relying on applicant’s evidence – stepwise analysis, determining a wear part’s condition – failure to establish features as CGK-– costs apportioned as a result of amendments during opposition proceedings – opposition unsuccessful
Representation: Patent attorneys for the applicant: Malcolm Bell and Ross McFarlane of Phillips Ormonde Fitzpatrick
Patent attorneys for the opponent: Ian Finch, Adam Luxton, and James Rowland of James & Wells
IP AUSTRALIA
AUSTRALIAN PATENT OFFICE
Patent Application: 2016265876
Title:Method and apparatus for locating a wear part in an image of an operating implement
Patent Applicant: Motion Metrics International Corp
Date of Decision: 18 January 2023
DECISION
The opposition is unsuccessful as none of the grounds have been made out.
Subject to appeal, I direct that the application proceed to grant.
I award costs against the applicant according to Schedule 8 of the Regulations up to the date the amendments were allowed, namely 12 February 2021, and from 12 February 2021 I award costs against the opponent according to Schedule 8 of the Regulations.
REASONS FOR DECISION
BACKGROUND
Australia application 2016265876 (‘the application’) is the national phase entry of PCT/CA2016/000144, which was filed 13 May 2016 by Motion Metrics International Corp (‘the applicant’). The application claims priority from US provisional application 62/162,203, thus having an earliest priority date of 15 May 2015.
The PCT application entered national phase on 22 November 2017, and the applicant requested examination on 18 May 2018.
The applicant filed amendments in anticipation of examination, and a first examination report was issued 21 February 2019 identifying claims that lacked novelty and inventive step, and also identifying some clarity issues.
The applicant amended the specification further on 1 May 2019 overcoming the objections, and acceptance was published on 20 June 2019.
CQMS Pty Ltd (‘the opponent’) filed a notice of opposition on 19 September 2019, and their Statement of Grounds and Particulars (the SGP) on 19 December 2019.
The opponent filed their evidence in support (‘the EIS’) on 19 March 2020
The applicant was granted a 3 month extension of time under Reg 5.9 due to the COVID-19 outbreak, and filed their evidence in answer (‘the EIA’) on 21 September 2020.
The applicant further took the opportunity to file amendments to the claims (and description) on 19 October 2020, with a follow-up amendment on 20 November 2020. These were allowed on 12 February 2021, and consequently this decision is directed to towards these claims.
The opponent was also granted a 3 month extension of time under Reg 5.9 due to the COVID-19 outbreak and filed their evidence in reply (‘the EIR’) on 22 February 2021.
The opponent also requested to amend the SGP on 26 March 2021, which was subsequently allowed on the 14 April 2021.
The oral hearing was held on 5 August 2021 via teleconference.
APPLICABLE LAW
The Application was filed after 15 April 2013 and is governed by the Patents Act 1990 (the Act) and Patents Regulations 1991 as amended by the Intellectual Property Laws Amendment (Raising the Bar) Act 2012 (Raising the Bar). Thus, the standard of proof that applies in the present case is the balance of probabilities. Under subsection 60(3A) of the Act, if I am satisfied, on the balance of probabilities, that a ground of opposition to the grant of a patent exists, I may refuse the application.
EVIDENCE
The opponent’s evidence in support was a Declaration by Dr Nicholas Hillier signed 19 March 2020 (1st Hillier Declaration) with exhibits NH-1 to NH-11, and a declaration by Fraser Smith signed 19 March 2020 (Smith Declaration) with exhibits FS-1 to FS-3.
The applicant’s evidence in answer was a Declaration by Dr Rhys Newman signed 21 September 2020 (Newman Declaration) and exhibits RN-1 to RN-5.
The opponent’s evidence in reply was a Declaration by Dr Nicholas Hillier signed 18 February 2021 (2nd Hillier Declaration).
GROUNDS OF OPPOSITION
At the hearing and in their submissions, the opponent pursued invalidity on the grounds of Lack of Clarity, a Lack of Clear and Complete Disclosure, a Lack of Inventive Step, a Lack of Manner of Manufacture, and a Lack of Utility.
At the hearing and in their submissions the opponent relied on the following prior art documents:
·D1: US Patent No. 8,411,930 B2 to Ridley et al and assigned to Alberta Research Council Inc., issued on 2 April 2013.
·D3: US Publication No. 2002/0128790 A1 to Woodmansee, published 12 September 2002.
·D4: US Publication No. 2014/0105481 A1 to Hasselbusch et al and assigned to Caterpillar Inc., published 17 April 2014.
·D6: International Publication No. 2015/059457 A1 to Taylor et al and assigned to Wheelright Limited, published on 30 April 2015.
·D7: International Publication No. 2014/205231 A1 to Lee et al and assigned to The Regents of the University of Michigan, published on 24 December 2014.
·D12: M Egmont-Peterson et al “Image processing with neural networks – a review” Pattern Recognition Volume 35, Issue 10, October 2002, pages 2279-2301 published by Elsevier B.V. Matthew Browne “Convolutional Neural Networks for image processing: An application in Robot Vision” Proceedings of the 16th Australian Conference on Artificial Intelligence, Perth, Australia, December 3-5, 2003. >
In their outline of submissions and at the hearing, the applicant submitted that the opponent’s outline of submissions was not on those matters as particularised in their amended SGP.
On this point, I note that, firstly, the opponent makes submissions regarding clear and complete enough disclosure in reference to the undue burden on the skilled addressee in determining what constitutes a labelled set of training images[1]. Secondly, the opponent makes submissions regarding support of the claims regarding the technical contribution to the art with respect to the training of the neural network using labelled images[2]. Thirdly, the opponent makes submissions regarding manner of manufacture in reference to the claimed invention being an abstract idea[3]. These particulars do not appear in the amended SGP. I also note that none of these submissions by the opponent rely on any new evidence, but instead largely focus upon the evidence in answer of the applicant.
[1] Paragraph 56 of the opponent’s written submissions
[2] Paragraph 85 of the opponent’s written submissions
[3] Paragraph 143 of the opponent’s written submissions
I agree with the applicant that these arguments, which are not particularised by the opponent in their SGP, stand outside the scope of the opposition. Nonetheless, the applicant made submissions in response to the arguments at the hearing and in their written outline[4]. Considering that the opponent’s submissions rely on the existing evidence already filed, are pertinent to existing grounds of the opposition, and the applicant was given an opportunity to (and did) make submissions in response, I conclude that, on the balance, it is appropriate to consider them in my decision, pursuant to s60(3) if that proves necessary.
[4] Paragraph 71 et seq of the applicant’s written submissions
THE SPECIFICATION
Earth moving machinery is subject to extreme conditions, and their continued use results in wear and tear. As such various components are configured to be replaceable or to breakoff under load to prevent damage to the superstructure. Such components may be wear members/parts, also referred to as ground engaging tools (‘GET’). However, this configuration creates further problems in and of itself, particularly where the material being moved is to be processed by machinery, as lost, or broken off wear members can cause serious damage to mineral processing equipment[5].
[5] Page 1 of the Description as Amended
Machinery operators will visually inspect equipment to monitor the loss of wear members. But this is not always possible in some environment conditions and is always subject to human error. As such there is much interest in the industry towards means and ways of automatically monitoring wear members on earth moving equipment. Existing methods rely on physically tracking wear members and other means of directly monitoring the wear member’s presence.
The specification is directed towards a method of processing images of earth moving machinery, involving the use of a neural network, to identify and locate wear members and, in turn, to be able to ascertain their condition. In particular, the specification discloses the use of a convolutional neural network to process the images to determine if the wear part is present in the image and, thus, to ultimately enable the location, and identification, of a condition of the wear members on an operating implement in real time.
The specification discloses several embodiments of this invention. For example: where the images processed are a sequence of images of a moving operating element; where the images are processed to identify a subset of pixels to be furthered analysed; and where the analysis also determines the size or condition of the wear member. The images can be captured in the visible spectrum or in the infrared, which are then analysed based on their light intensity.
The condition of the wear member can be as simple as its mere presence (as opposed to its absence), or something more complex such as estimating the remaining portion or state of the wear member.
The specification gives some attention to the physical electronics[6] the method relies on, however there is nothing particularly noteworthy as to the image capturing apparatus as described. Similarly, the computational hardware as described is equally as generic. Instead, the specification is focused on the method of analysing the images to identify and monitor the wear members.
[6] Page 11 of the Description as Amended
The embodiment to which the amended claims are now directed is best depicted in Figure 16:
This outlines the broad operation of the embodied method. Where images are continually processed and analysed, and when there is a perceived issue, an alert is generated. In the above figure, images of an operating implement are captured in block 852, an example of which is shown below in Figure 8. Block 854 processes each of these images to locate a particular wear part within the image, as further described below with reference to Figures 7 and 8. If the wear part is detected in block 854, further analysis is undertaken to determine a dimensional attribute and condition of the wear part. If at any point the wear part is not located, or has an unsatisfactory condition, an alert can be generated.
A more detailed example of how the images are processed is seen in Figures 8 and 9. Figure 8 is depicted below:
In Figure 8 it can be seen how the image is processed in subsets of pixels as indicated by items 510. The image also highlights the challenges faced in that there are number of objects potentially present in an image, and that the operating element indicated as item 204 will also be moving within the images captured. Item 504 is a typical dump truck, whereas item 204 is the excavator bucket itself having wear members depicted by item 206.
These pixel subsets with their array of pixel weighting values (item 602) [7], used to determine the likely presence of a wear part, are depicted in Figure 9 below:
:
[7] Page 16 of the Description as Amended
Figure 7, depicted below shows one of the key steps of the method of the claimed embodiment:
In this part of the method, a portion of the image (pixel subset) which was shown earlier as Figure 9 is processed by the neural network to determine whether the subset meets the matching criterion for potentially containing a wear member. The location of this portion in the whole image is recorded and can be used to further process the images to determine dimensional attributes or the wear part condition.
Figure 14 depicted below shows another key step in the method locating the operating element, to further process for wear members.
This portion of the method is used to identify if the operating implement is in the image selection, so as to locate it, and, if present, further analyse portions of the image according to the method of Figure 7 (incorrectly referenced as ‘Fig.4’ in Figure 14) to identify if the wear members are present.
From here, the image portions determined to have wear members present can be further analysed to determine the dimensional attributes and condition of the individual wear parts as discussed earlier with reference to Figure 16.
The Invention as Claimed
Claim 1 is as follows (with integers I have assigned myself in the margin):
A method for locating and identifying a condition of a wear part in an image of an operating implement associated with heavy equipment, the method comprising:
(1)capturing images during operation of the heavy equipment, each image including a plurality of pixels, each pixel having an intensity value;
(2)processing the plurality of pixels for each image to determine whether the operating implement is present in the image, and if the operating implement is present in the image, restricting the plurality of pixels to pixels within a region of interest that includes the operating implement; and
(3)for each image in which the operating implement is present:
(a)selecting successive pixel subsets within the plurality of pixels;
(b)processing each pixel subset to determine whether pixel intensity values in the pixel subset meet a matching criterion indicating a likelihood that the pixel subset corresponds to the wear part, wherein processing each pixel subset comprises processing each pixel subset through a corresponding plurality of input nodes of a neural network, each input node having an assigned weight and being operable to produce a weighted output in response to the pixel intensity value; and
(c)wherein the matching criterion is based on processing a labeled set of training images during a training exercise prior to capturing the images, the labeled training set including images of various examples of wear parts, the training exercise being operable to determine the assigned weights for the plurality of input nodes; and
(d)for each pixel subset that meets the matching criterion, determining a dimensional attribute of the wear part within the pixel subset, and determining whether the condition of the wear part is satisfactory based on a predetermined criterion.
Claim 19 is as follows (with integers I have assigned myself in the margin):
An apparatus for locating a wear part in an image of an operating implement associated with heavy equipment, the apparatus comprising:
(1)an image sensor for capturing images during operation of the heavy equipment, at least one of the images including the operating equipment, each image including a plurality of pixels, each pixel having an intensity value;
a processor circuit operably configured to:
(2)process the plurality of pixels to determine whether the operating implement is present in the image, and if the operating implement is present in the image, restrict the plurality of pixels to pixels within a region of interest that includes the operating implement;
(3)for each image in which the operating implement is present:
(a)select successive pixel subsets within the region of interest;
(b)process each pixel subset to determine whether pixel intensity values in the pixel subset meet a matching criterion indicating a likelihood that the pixel subset corresponds to the wear part, each pixel subset being processed through a corresponding plurality of input nodes of a neural network, each input node having an assigned weight and being operable to produce a weighted output in response to the pixel intensity value;
(c)wherein the matching criterion is based on processing a labelled set of training images during a training exercise prior to capturing the images, the training exercise being operable to determine the assigned weights for the plurality of input nodes; and
(d)for each pixel subset that meets the matching criterion, determine a dimensional attribute of the wear part, and determine whether the condition of the wear part is satisfactory based on a pre-determined criterion.
Both claims 1 and 19 can be broken down into generally the same integer features which is useful. Claim 1 defines the method of the invention, whereas claim 19 is an apparatus applying this method in practice.
Integer 1 defines capturing an image in at least a greyscale, as represented by the intensity value.
Integer 2 defines step of locating an operating element (e.g., a digger bucket) within the aforementioned image, which can be done by any means inclusive of conventional image analysis and neural networks. This location is then followed by a restriction of the image to a region of interest, e.g., to crop the image.
Integer 3 covers off processing the images via a neural network according to the main method of the invention, by carrying out steps 3a, 3b, 3c and 3d.
Integer 3a defines that for each image, a series of pixel subsets within the restricted plurality of pixels is selected for analysis.
Integer 3b defines that it is a neutral network that is being used to determine whether the subset corresponds to a wear part (e.g., tooth) in the aforementioned image, using a matching criterion. The neural network outputs this as a weighted result.
Integer 3c defines the neural network as having been previously trained with a series of labelled training images.
The final integer 3d defines determining a dimensional attribute based on the intensity values of the subset processed in 3b and deemed to correspond to a wear part, and to determine whether the wear part’s condition is satisfactory based on a predetermined criterion.
When I move to the appended claims, these further define the method and apparatus. Claims 2-18 are appended to claim 1, whereas claim 20 is appended to claim 19.
Claim 2 defines that integer 3b’s processing of the pixel intensity involves generating a histogram of oriented gradients.
Claim 3 further defines integer 3b in that the weighted outputs are put through a plurality of hidden nodes in the neural network to produce a weighted output. Claim 4 further defines that the hidden nodes are in layers and claim 5 further defines that one of the layers is a memory layer for capturing and processing the images as a sequence of images. Claim 6 also further defines that the weight outputs from the hidden nodes of claim 3 are received at output nodes.
Claim 7 defines that the step of 3b of determining whether pixel intensity values in the pixel subset meet the matching criterion comprises determining whether the weighted output exceeds a reference threshold.
Claim 8 defines that the pixel subsets are processed using a convolutional neural network. Claim 9 further defines that the convolution uses a sparse kernel which is where rows and columns of zeros are inserted into the convolution kernel to separate entries in the kernel. Claim 10 further defines the neural network has a pooling layer. Claim 11 further defines that the processing using the convolutional neural network includes resampling the image and training images. Claim 12 further defines a memory layer in the neural network which uses previous processing results to configure the neural network for subsequent processing of further images.
Claim 13 further defines integer 3c in that the training set of images are labelled as to whether they have, or have not, got a wear part.
Claim 14 further defines integer 2 in that the matching criterion for determining whether there is an operating implement is also based on processing a labelled set of training images.
Claim 15 further defines integer 3a such that the pixel subsets can be a predetermined size or based on the captured image.
Claim 16 further defines integer 3b such that it involves calculating a product of both the assigned weights and the pixel intensity, and that the matching criterion is based on whether this product exceeds a threshold.
Claim 17 defines that integer 1 involves an image sensor working in either visible or infrared spectrum.
Claim 18 defines integer 3d as also including calculating a predicted time to failure based on a rate of wear of a wear part.
Claim 20 further defines that the image sensor works in either visible or infrared spectrum.
THE PERSON SKILLED IN THE ART
The person skilled in the art (PSA) was considered in Root Quality Pty Ltd v Root Control Technologies Pty Ltd [2000] FCA 980; at [70].
“He is the person to whom the patent is addressed and who must construe it. He is the person whose knowledge will determine whether a patent is novel. He is the person who will judge whether a patent is obvious.”
However, the PSA is not a real person, but an artificial construct that is used as a tool of analysis which is used to make the determination as noted in AstraZeneca AB v Apotex Pty Ltd [2015] HCA 30 at [23]:
“The notional person is not an avatar for expert witnesses whose testimony is accepted by the court. It is a pale shadow of a real person – a tool of analysis which guides the court in determining, by reference to expert and other evidence, whether an invention as claimed does not involve an inventive step.”
The parties made submissions regarding who the hypothetical skilled addressee is that the application was addressing.
The applicant’s written submissions were along the lines as to which of the experts in this opposition was the relevant PSA[8], seeking to promote their expert over the opponent’s.
[8] Paragraphs 42-50 of the applicant’s written Submission.
The opponent’s submissions were that the PSA was one or more of a mining and/or mechanical and/or electrical and/or software engineer(s) with knowledge of, and experience in, heavy equipment, monitoring wear parts and image processing algorithms[9].
[9] Paragraphs 28 and 29 of the opponents Written Submissions.
The specification states the importance of monitoring wear members on earth moving equipment, and the need to improve existing ‘Camera based monitoring systems’[10]. Thus, I consider that the skilled addressee would be a manufacturer of wear part monitoring systems. Along the lines of the opponent’s submissions, this would be a team of engineers who had knowledge of, and experience in, heavy equipment, monitoring wear parts and image processing algorithms inclusive of neural networks. To the extent that a particular expert is to be preferred in their evidence over another, I will address this if and where required in the decision.
SECTION 40
[10] Page 2 of the Description as amended
Clarity
Subsection 40(3) of the Act requires that the claim or claims of a specification are clear. While the rules of construction for an Australian patent specification are well summarized in Decor Corp v Dart Industries [1988] FCA 399; 13 IPR 385, the correct application of these rules to the construction of claims was discussed by Bennett J in H Lundbeck A/S v Alphapharm Pty Ltd [2009] FCAFC 70; 81 IPR 228 at [118] – [120]:
“the words in a claim should be read through the eyes of the skilled addressee in the context in which they appear ... while the claims define the monopoly claimed in the words of the patentee's choosing, the specification should be read as a whole ... it is not permissible to read into a claim an additional integer or limitation to vary or qualify the claim by reference to the body of the specification ... terms in the claim which are unclear may be defined or clarified by reference to the body of the specification.”
63. The requirement that the claims are clear is understood to be satisfied if a person could ascertain “whether or not what he proposes to do falls within the ambit of the claim”[11]. To paraphrase, if one can work out whether what they are doing with their product or method is inside of the scope of the claim, then a claim can be considered as clear. Importantly, the description and drawings can be useful for resolving ambiguity in the wording of a claim. However, as noted in Minnesota Mining And Manufacturing Co. v. Beiersdorf (Australia) Ltd [1980] HCA 9 at [46] by Aickin J, a claim does not lack clarity because it uses inexact language or is difficult to construe if it provides a “workable standard” suitable to the intended use
[11] Monsanto Co v Commissioner of Patents (1974) 48 ALJR 59 at [60]-[61]
As noted in Flexible Steel Lacing Company v Beltreco Ltd [2000] FCA 890 at [81]; cited with approval in Austal Ships Sales Pty Ltd v Stena Rederi Aktiebolag [2008] FCAFC 121 at [14]:
“The consideration is whether, on any reasonable view, the claim has meaning. In determining this, the expression in question must be understood in a practical, common sense manner.”
The opponent has opposed the claims of the application for lacking clarity, submitting that[12]:
“The claims lack clarity as to whether the steps of determining a dimensional attribute / the condition of the wear part are linked and/or carried out via a neural network; and…
…The claims lack clarity as to the meaning of the word ‘condition’… [in that the term ‘condition’] extends to a determination of the presence or absence of the wear part; or it is intended to be carried out only on those pixel subsets which meet a matching criterion indicating the likelihood that the subset corresponds to (the presence of) a wear part to determine the state of that wear part.”
[12] Paragraphs 89 and 90 of the opponent’s written submissions
The opponent’s submissions do not direct my attention to any evidence regarding the lack of clarity.
The applicant submits that there is no evidence to support the lack of clarity[13], and that the opponent’s expert Dr Hillier has no trouble understanding the claims. However, they submit that the claim should be construed as the condition being determined based on the dimensional attribute.
[13] Paragraphs 178 of the applicants written submissions
Firstly, in my analysis of the construction of the claims above, I have been able to give these terms and the claims meaning. However, I will address each of the opponent’s submissions directly.
I consider that it would be clear that, when Claims 1 and 19 are construed by the skilled addressee, that those pixel subsets that have been picked up by the matching criterion of the neural network in integer 3b are to be further analysed in integer 3d. The claim does not define how this analysis is being performed because this can be achieved either by conventional image analysis as would be known to the skilled addressee[14] or further processing in a neural network[15]. As such the claim is not limited to where the condition is determined by the neural network, as Dr Hillier appears to have recognised[16]. So, whilst broad in scope, there is no ambiguity in how this feature is being defined by the claim.
[14] Page 2 of the Specification
[15] Page 27 of the Specification
[16] Paragraph 16 of the 2nd Hillier Declaration
With respect to what is defined by the term ‘condition’ and whether this is inclusive of determination of presence, a plain reading of the claim indicates that for each pixel subset that meets a matching criterion, a determination of a dimensional attribute of “the wear part” within the pixel subset and whether the condition of “the wear part” is satisfactory is made based on a predetermined criterion. I consider that it would be clear to the skilled addressee that, by its plain meaning, the condition of a wear member in the claim is distinct to determining its mere presence or otherwise. It is “the wear part” in the pixel subset that is considered in terms of its dimensionality and, on plain reading, I consider it is this same wear part that is determined for its condition. Thus, in other words, I consider that the determination as to whether the condition of the wear part is satisfactory is a qualitative assessment, in contrast to a quantitative determination of its presence or absence[17]. In the specification one example of a condition is the comparing the size of the wear member[18].
[17] Paragraph 121 of the Newman Declaration
[18] Page 2 of the Specification
Whilst a skilled addressee would recognise that an absent wear member is not a satisfactory condition, I do not consider that the complete absence of the wear member falls within the scope of the “conditions” encompassed by the claim. The claims define that the determination of the condition of a wear member only occurs when the pixel subset has been selected by the matching criterion, thus considered to be containing a wear member. It would follow that the method cannot include the determining the condition of absent wear members as these would not have met the matching criterion for the pixel subset. This is also consistent with the preamble of the claim, which defines a method for ‘locating and identifying a condition of a wear part in an image….’ It is only after a wear member has been located, that its condition can be identified.
The applicant’s submissions that there was a lack of evidence establishing the term condition as unclear has no bearing on my above consideration. Whilst I disagree with the applicant’s submission that the condition should be construed in the claims as being determined from the dimensional attribute, it would none the less appear to be consistent with my above reasoning given a missing wear part would not have a dimensional attribute.
I thus prefer the opponent’s second proposed construction for the term “condition” in that determination is intended to be carried out only on those pixel subsets which meet a matching criterion indicating the likelihood that the subset corresponds to (the presence of) a wear part to determine the state of that wear part.
I am satisfied that the claims are clear.
Clear Enough and Complete Enough disclosure
The Raising the Bar Act introduced two new provisions to section 40: a requirement for disclosure and a requirement for support. The two concepts are closely connected. The disclosure requirements are found in subsection 40(2) under paragraphs (a) and (aa), which state that the specification must:
(a) disclose the invention in a manner which is clear enough and complete enough for the invention to be performed by a person skilled in the relevant art; and
(aa) disclose the best method known to the applicant of performing the invention;
The purpose of these provisions is explained in the Explanatory Memorandum to the Raising the Bar Act (the Explanatory Memorandum) pages 47 – 48:
“The item is intended to modify the wording of paragraph 40(2)(a) of the Act so as to require enablement across the full width of the claims, while adopting language that is consistent with that used in other jurisdictions. The wording in the amendment is similar to s 14(3) of the UK patents legislation, which has been interpreted as imposing this requirement. The wording is also similar to art 83 of the European Patent Convention, which has been interpreted with similar effect. The intention is that paragraph 40(2)(a) be given, as close as is practicable, the same effect as the corresponding provisions of UK legislation and the European Patent Convention.
A specification that provides a single example of the invention may satisfy the requirements, but only where the skilled person can extend the teaching of the specification to produce the invention across the full width of the claims, without undue burden, or the need for further invention.
However, it is expected to be more likely that, where the claims are broad, the specification will need to give a number of examples or describe alternative embodiments or variations extending over the full scope of the claims. This ensures that the monopoly extends only to that which could reasonably be said to be disclosed and no further.
If, on its face, the specification would appear to the skilled person to lack sufficient disclosure, the onus of establishing that the invention is described in enough detail lies with the applicant.”
The general approach to deciding sufficiency of disclosure was summarised succinctly by Lord Hoffmann in Kirin-Amgen Inc v Hoechst Marion Roussel Ltd [2005] RPC 9 (Kirin-Amgen) at [103]:
“The first step is to identify the invention and decide what it claims to enable the skilled man to do. Then one can ask whether the specification enables him to do it.”
Lord Hoffmann provided a more detailed discussion in Biogen Inc v Medeva plc [1996] UKHL 18; [1997] RPC 1 (Biogen) at [48]:
“...the specification must enable the invention to be performed to the full extent of the monopoly claimed. If the invention discloses a principle capable of general application, the claims may be in correspondingly general terms. The patentee need not show that he has proved its application in every individual instance. On the other hand, if the claims include a number of discrete methods or products, the patentee must enable the invention to be performed in respect of each of them.”
This was also considered in detail in CSR Building Products Limited v United States Gypsum Company [2015] APO 72 (‘CSR’). The delegate explained at [95], that in order to decide whether sufficiency of disclosure has been satisfied according to section 40(2), one must:
i) construe the claims to determine the scope of invention as claimed,
ii) construe the description to determine what it discloses to the person skilled in the art, and
iii) decide whether the specification provides an enabling disclosure of all the things that fall within the scope of the claims.
The opponent submits that the “secret sauce” to implement the claimed invention successfully lies in the selection and labelling of the dataset used to train the neural network, for which the specification is deficient in detail. In particular[19]:
“…the curation and labelling of these ‘many thousands if not millions’ of images at ‘significant economic cost’ to train the neural network in order to implement the claimed invention, for which the ‘eventual resulting accuracy…[would be] far from certain’, represents an undue burden to the PSA.”
[19] Paragraph 68 of the opponent’s written submissions
More broadly, the opponent submits that:
“..the Application does not provide clear guidance telling the PSA how to label the dataset of training images to train the claimed neural network – especially as regards assessing the condition of a wear part. The Application is also lacking in any detail how to adjust the training exercise so as to achieve with certitude the claimed process. In fact, the Application does not even provide a starting point: how to assemble the dataset of training images for use with the training process.”
The opponent points to the evidence of the applicant’s expert Dr Newman to support this submission. It is important to note that Dr Newman’s evidence was given in a specific context, which was with respect to obviousness of the claimed invention. Specifically, Dr Newman’s evidence was with respect to uncertainty that neural networks could be used to determine the condition of wear components. I will address this context later in more detail as part of my consideration.
The applicant submits Dr Newman’s evidence is not applicable for these grounds because it was provided for the ground of obviousness and not the ground of sufficiency, and that the opponent offers no other evidence that the specification does not provide an enabling disclosure.
I do not agree with the applicant’s general submission that Dr Newman’s evidence is not applicable to this ground. The role of the expert in giving evidence is to answer questions of fact. If the facts are relevant to my consideration of a different ground of the opposition, I can see no reason why I should disregard it. Similarly, evidence given by Dr Hillier in the context of obviousness also cannot be disregarded.
Scope of the invention claimed
I have previously construed the claims above, but I now focus on the scope of the contended features of Claim 1 and 19.
Claim 1 explicitly defines the labelled images are used to train the matching criterion of integer 3b, a criterion in the neural network used to detect if a wear part is present in the pixel subset. When the wear part is detected, the pixel subset is further processed to determine the wear part’s dimensional attribute and condition. The claim does not define how the dimensional attribute in integer 3d is determined, only that the condition is based on a predetermined criterion, and that both occurs after the neural network has identified that a wear part is present in the pixel subset.
Similarly, the processing of the pixels in integer 2 to determine whether the operating implement is present is also not specified.
This means that the scope of the claims is inclusive of making the determinations in integers 2 and 3d with or without a neural network, as described in potential embodiments[20]. This means that the determination of the dimensional attribute of a wear part or its condition (integer 3d) is inclusive of using, for that determination: the existing neural network, a separate neural network, or any other conventional means of image analysis. A similar conclusion applies for integer 2. A consequence, for example, is that using a neural network for the determination of the condition of a wear part based on a predetermined criterion (integer 3d), after determining the presence of the operating implement also using a neural network (integer 2), is within the scope of the claim.
[20] Page 27 of the Specification
What is disclosed by the specification.
The bulk of the specification is directed to the configuration of the neural network which will detect wear parts in the images using a matching criterion. The specification also addresses other aspects of implementing the invention, such as hardware devices.
To use a neural network for image analysis, the neural network needs to be trained with labelled images. The labelling of images for training neural networks consists of associating data with the images to be used for the training[21]. For detection of a singular presence, this can be as simple as assigning a binary value. The specification discusses the training under the heading of ‘Generating matching criterion’[22].
[21] Paragraph 20 of the 1st Hillier Declaration
[22] Page 18 of the Specification
The labelling process is described as a supervised learning which is performed by using bounding boxes on an image to identify teeth in the image, which was shown in Figure 12 (not reproduced). The image portions can be saved in a group folder on a drive.
Furthermore, the specification also discloses an example of the software for labelling of pixel subsets containing wear parts shown in Figure12. The specification describes a training (i.e. supervised learning) process for the neural network. This includes the use of bounding boxes[23] on images to identify individual teeth. The specification also describes how to manipulate the scale of images for additional training inputs. Lastly, the specification also describes using the machine operator to help train the neural network by identifying which regions contain wear parts and regions which do not.
[23] Page 18 and 19 of the Specification.
With respect to determining the presence of the operating implement the specification describes steps analogous to, and with specific reference to, steps disclose for using the neural network to determine the presence of the wear parts. Again, this is achieved by using a matching criterion that has been obtained using appropriately labelled images[24].
[24] Page 23 of the Specification
With respect to determining a dimensional attribute, the specification describes how this can be done by using reference distances in the images[25] to convert pixels into real world dimensions. Several examples of objects and distances are given that can be used as references for this conversion. From this dimensional attribute, the specification describes that a condition can be determined with reference to historical values previously determined or reference dimensions. The specification notes that a neural network can also be trained to determine a direction of the operating implement.
[25] Page 27 of the Specification
Does the specification enable all of the things that fall within the scope of the claims?
The opponent’s submissions have put much emphasis on the need for disclosure of image labelling and their curation. I did not find this helpful, as it puts the cart before the horse. It is more appropriate to firstly consider what is being claimed that is not disclosed by the specification, and whether this places an undue burden on the skilled addressee in order to work the full scope of the claimed invention.
With respect to the training of the neural network for the matching criterion to identify wear parts, and the labelling of images for this purpose, this has been described by the specification as I have discussed above. Whilst the evidence of Dr Newman suggests that there can be complexity in the labelling of images, this was with respect to identifying multiple wear parts in single image. As such, I can find no relevant evidence to suggest that that training the neural network to identify the condition of individual wear parts is not clearly disclosed and enabled by the specification.
With respect to the training of a neural network for the matching criterion to identify operating implements, and the labelling of images for this purpose, this has been described by the specification as I have discussed above. There is no evidence to suggest that this or conventional means of determining the presence of the operating implement would be an undue burden on the skilled addressee.
With respect to the determination of the dimensional attributes and the condition of the wear parts by conventional image analysis, this would also appear to be disclosed as I have discussed above. The evidence of Dr Newman does not discuss such image analysis in this way. The evidence of Dr Hillier, which is in the context of the claimed invention being obvious, would suggest that this is a matter of routine to implement for the skilled addressee[26]. Therefore, it appears that this is also clearly disclosed and enabled by the specification.
[26] Paragraph 22 of the 2nd Hillier declaration.
What I was unable to find was any disclosure as to how to determine dimensional attributes or the condition of the wear part using a neural network. The only other reference to using neural networks in the method described, other than detecting the wear part and/or operating implement, is a brief mention of the applicability of neural networks to determine the direction of the operating implement[27]. I could find no disclosure of using a neural network to determine a dimensional attribute or condition of the wear part. To determine whether using the absence of this disclosure places an undue burden, I should consider the evidence from both experts.
[27] Page 27 of the Specification
The opponent submits that Dr Newman’s evidence supports their argument that, in the absence of sufficient disclosure, there is an undue burden for a skilled addressee to determine the dimensional attribute or condition using a neural network.[28]. Dr Newman’s evidence regarding obviousness goes to great lengths to explain the challenges that would be faced by a skilled addressee seeking to use neural networks to detect wear parts in images, and states that determining their condition would add further challenge.
[28] Paragraph 55 of the opponent’s written submissions
It is Dr Newman’s evidence on these challenges that comes the closest to addressing the question of whether there is a burden on the skilled addressee to determine the dimension attributes and condition using a neural network after the wear part is detected. Dr Newman makes the statement that[29].:
“The system also needs to be able to cope with subtleties such as partially missing teeth (broken not worn) which is sometimes difficult for even a human reviewer to determine. The method disclosed in the Opposed Specification involves not only locating wear parts but determining their condition. The question being posed to the system is therefore much more complicated than simply whether the tooth is ’missing’ or ‘present’”
[29] Paragraph 121 of the Newman Declaration
Dr Newman’s evidence would suggest that to determine dimensional attributes or condition is more complicated than simply detecting that wear parts are present. As I have discussed above, the specification does not explicitly disclose how to determine this with a neural network. But the example he uses is with respect to the processing of whole images by the neural network, to detect multiple wear parts in the image, and in the context of whether it would be obvious to implement this. Therefore, Dr Newman’s comments on this complication are directed to the scenario when a whole image is being used, rather than a pixel subset in a region of interest, and not the method being claimed. Because this is not what is claimed, it is thus not within the scope to be enabled. As such, any perceived complexity or challenge for that example is not relevant to my consideration.
With respect to neural networks in general, Dr Newman makes the remark that there “... is no standard methodology of using neural networks to locate and assess the conditions of [wear] parts” in the closing paragraph of his declaration[30]. Dr Hillier’s evidence has a more positive outlook on using a neural network to find the location of wear parts in an image, but makes no statements as to the feasibility of determining their condition with a neural network.
[30] Paragraph 173 of the Newman declaration
With respect to assessing the condition via neural network, Dr Hillier stated in his evidence[31]:
“It appears as though Dr Newman is of the view that the solution presented by the opposed application relates to using a neural network to assess the condition of a wear part. For instance, at Para 172 ‘The invention involves taking a particular stepwise approach to first locating and then assessing the condition of a wear part, this latter function achieved using a neural network to replicate the expert human opinion that is expressed implicitly in a set of manually annotated training images.’ However, there is no suggestion in the specification that the latter function of ‘assessing the condition of a wear part’ is achieved using a neural network. The neural network is used to determine the condition of the wear part (length) via defining a bounding box surrounding a wear part in an image that previously used a neural network to determine that the image contains an operating implement. The size of this bounding box is then compared, without the use of a neural network, to a predetermined criterion to achieve the assessment of satisfactory condition of the wear part.” (underlining and bolding in original)
[31] Paragraph 8 of the 2nd Hillier declaration
Dr Hillier’s statement would appear to confirm that the specification does not explicitly disclose how to determine the condition of a wear part via a neural network. But there is no evidence from Dr Hillier to suggest implementing this would be burdensome for the skilled addressee.
So, whilst the evidence overall would suggest that utilising a neural network to determine the condition of a wear part member can be complex, it does not go so far as to suggest that it will always be complex irrespective of the preceding steps taken. Similarly, this applies to training a neural network to determine the condition after such steps. The evidence of Dr Newman only goes so far as to establish that doing such analysis on whole images, rather than as part of the stepwise method, may be burdensome.
As such, I cannot be satisfied that is more likely than not that this complexity will carry over to using a neural network to determine the condition after the other steps of the method have been followed to detect and locate the wear part. The opponent bears the onus to establish that there will be an undue burden for these embodiments within the scope of the claim. The mere possibility of a burden is not enough to establish the claims are not enabled.
Therefore, I cannot conclude the scope of the claimed invention for Claims 1 and 19 is not enabled. The appended claims further limit the scope of the claims. Therefore, it follows I can also not conclude that they are also not enabled by the specification.
It follows that I am satisfied that the specification does provides a disclosure of the invention that is clear enough and complete enough for the invention to be performed by a person skilled in the art.
Support
The requirement for support lies in subsection 40(3), which states:
“The claim or claims must be clear and succinct and supported by matter disclosed in the specification.”
This provision was introduced as a replacement for the former requirement of fair basis. The purpose of this change is also explained in the Explanatory Memorandum at page 49:
“This item is intended to align the Australian requirement with overseas jurisdictions' requirements (such as the UK). Overseas case law and administrative decisions in respect of the ‘support’ requirement will be available to Australian courts and administrative decision-makers to assist in interpreting the new provision.”
Both the concept of sufficiency and support relate to the need for an enabling disclosure of the invention, as explained in Generics (UK) Ltd v H Lundbeck A/S [2009] UKHL 12; [2009] RPC 13 (Generics), a decision of the House of Lords, by Lord Walker of Gestingthorpe, at paragraph [20]:
“The disclosure must be such as to enable the invention to be performed (that is, to be carried out if it is a process, or to be made if it is a product) to the full extent of the claims.”
Despite the similarity of these two concepts, they each require distinct tests which must be conducted separately according to their own criteria.
In Merck Sharp & Dohme Corporation v Wyeth LLC (No 3) [2020] FCA 1477 (Merck), Burley J explored the requirement of support at [546]-[547]:
“In CSR Building Products Ltd v United States Gypsum Company [2015] APO 72, Dr S D Barker adopted the summary provided by Aldous J in Schering Biotech at 252 – 253, which has been often followed in the United Kingdom (emphasis added):
‘...to decide whether the claims are supported by the description it is necessary to ascertain what is the invention which is specified in the claims and then compare that with the invention which has been described in the specification. Thereafter the court’s task is to decide whether the invention in the claims is supported by the description. I do not believe that the mere mention in the specification of features appearing in the claim will necessarily be a sufficient support. The word “support” means more than that and requires the description to be the base which can fairly entitle the patentee to a monopoly of the width claimed.’
That approach encapsulates broadly the claim support obligation under s 40(3). To it may be added the requirement that the technical contribution to the art must be ascertained. Where it is a product, it is that which must be supported in the sense that the technical contribution to the art disclosed by the specification must justify the breadth of the monopoly claimed.”
This was also considered in detail in CSR, which was referred to with approval in Merck. The delegate explained at [115], in order to decide whether that the claim or claims are supported by matter disclosed in the specification, one must:
i)construe the claims to determine the scope of the invention as claimed,
ii)construe the description to determine the technical contribution to the art, and
iii)decide whether the claims are supported by the technical contribution to the art.
The opponent’s submissions are that the claims extend beyond the technical contribution to the art, in particular with regard to: the training of the neural network in general, the training to determine a condition, breadth of “condition”, and whether the criterion for the condition is based on dimensions. The opponent submits that there are insufficient details as to how to assemble, curate and label the training set of images, particularly where the neural network is determining the condition. The opponent also submits that the specification only discloses determining the condition from the dimensional aspects, and not other methods that might be encompassed by the term ‘condition.’
The applicant submits that opponent has provided no evidence that the claims lack support, and that the evidence of Dr Newman is not related to the invention claimed.
I note that the submissions of both parties are along the same lines as those discussed above for sufficiency.
What is the invention as claimed?
I have previously construed the scope of the claims. As discussed above, broadly claims 1 and 19 are directed to a stepwise method for locating a wear part and determining its condition comprising:
· capturing an image of an operating implement with wear members;
· processing the image to detect the operating implement, and its location and thus region of interest in the image;
· processing the region of interest by analysing a smaller portion (pixel subsets) of the image to detect wear members;
· using a neural network to determine if the image meets the matching criterion for a wear part being present, where the matching criterion is developed from training the neural network with labelled images; and
· analysing those subsets with wear part presence to determine a dimensional attribute and determine the condition of the wear part based on a pre-determined criterion.
The appended claims further narrow the scope of the independent claims.
What is the technical contribution to the art?
Dr Newman describes the stepwise method as being what he considers the technical contribution of the specification[32]. Dr Newman describes the method’s approach as “segment the problem into a set of simpler problems that were less likely to generate false positives or false negatives”[33]. He subsequently describes that[34]:
“..both the processing of the images and the training of the neural network is done on individual wear parts (e.g. the teeth) rather than on the total image. In this way, the method significantly reduces the need to collect and label an unrealistically large number of images that would otherwise be required.”
[32] Paragraph 139-141 of the Newman declaration
[33] Paragraph 128 of the Newman Declaration
[34] Paragraph 140 of the Newman Declaration
As I understand it Dr Newman’s take on the technical contribution is on how the application is analysing the images with the neural network, which can be characterised by the following discussion. Firstly, rather than detecting a wear part or multiple wear parts in an image, the image is progressively scanned via a smaller subset of pixels to both detect and locate the wear part. The specification also describes an additional preliminary step of locating the operating implement thus reducing the portion of image to be analysed. This can be done by a method analogous to detecting the wear parts. The method seeks to reduce the computational resources required, as well as the accuracy of the system in locating wear parts.
Having located the wear part(s), analysis can be performed to determine the dimensional attribute and whether the condition of the wear part is satisfactory. The methods for doing this determination do not appear to comprise any advancement over the art as they are not disclosed in any detail. In addition, the specification also discusses how to specifically configure the neural networks to make them more suitable for use to detect the wear parts using the stepwise method.
Dr Hillier’s evidence is directed to the scope of the claims, rather than what is taught by the specification. I could not find any evidence that could be inferred as to what Dr Hillier considers the contribution to be from the specification. His evidence was directed to what he considered was not disclosed by the specification but fell within the scope of the claim. The fact that there is no disclosure of certain features, however, does not mean that there must be a lack of support.
Upon review of the specification, I am of the view that the technical contribution is the combination of preliminary steps to reduce the portion of the image that needs to be analysed by the neural network, thus making it possible to detect and locate the wear part(s) in the image of the operating implement using the neural network. This accords with the technical contribution as identified by Dr Newman.
Are the claims supported?
The consideration is whether the claims are broader than justified by the contribution of the specification. The technical contribution to the art lies in the preliminary steps that allow a neural network to be used in the process for detecting, locating, and determining dimensions and conditions of the wear parts in an image. It is quite clear that the claims reflect this technical contribution.
In my view, the evidence does not suggest there to be an undue burden or a lack of enabling disclosure. As discussed in connection with s40(2)(a), the specification describes configuration of the neural networks to be used and provides basis for generally determining the presence of an operating element, using a neural network to determine presence of wear parts, and subsequently determining dimensionality and/or condition of a wear part using known techniques.
Therefore, I am satisfied on balance that the claimed invention is supported.
INVENTIVE STEP
Relevant Law.
It is a requirement of subsection 18(1) of the Act that the invention, so far as claimed in any claim, involves an inventive step. Subsection 7(2) states that an invention is taken to involve an inventive step unless it would have been obvious to a person skilled in the art in the light of the common general knowledge, considered alone or together with the prior art as specified in subsection 7(3).
Prior art information is information that is part of the prior art base, and the prior art base includes information in a document that is publicly available, and information made publicly available through doing an act. Once the common general knowledge and prior art information have been identified, the question is whether the claimed invention would have been obvious. Various verbal tests have been set out to explain this question. In Wellcome Foundation Ltd v V.R. Laboratories (Aust.) Pty Ltd (1981) 148 CLR 262 Aickin J stated:
“The test is whether the hypothetical addressee faced with the same problem would have taken as a matter of routine whatever steps might have led from the prior art to the invention, whether they be the steps of the inventor or not.”
The High Court in Aktiebolaget Hässle v Alphapharm Pty Ltd (“Alphapharm”) [2002] HCA 59 at [51] – [53] also approved the approach taken in Olin Mathieson Chemical Corporation v Biorex Laboratories Ltd [1970] RPC 157 in which Graham J at [187] had posed the reformulated Cripps question:
“Would the notional research group at the relevant date in all the circumstances directly be led as a matter of course to try [the claimed invention] in the expectation that it might well produce a useful [desired result]?”
In AstraZeneca AB v Apotex Pty Ltd [2014] FCAFC 99 (“AstraZeneca”) the Full Court at [203] held that in formulating the problem it is not permissible to incorporate information that is not available to the person skilled in the art either as common general knowledge or information available under subsection 7(3).
In relation to what level of inventiveness is required to sustain a patent, the Full Federal Court in Garford Pty Ltd v Dywidag Systems International Pty Ltd [2015] FCAFC 6 stated at [44] as follows:
“The inventiveness required to sustain a patent for a claimed invention is quite small. A ‘scintilla’ of inventiveness is all that is required: Alphapharm at [195]. However, there must still be ‘some difficulty overcome, some barrier crossed’ (per Lockhart J in RD Werner & Co Inc v Bailey Aluminium Products Pty Ltd [1989] FCA 57; (1989) 25 FCR 565 at 574) or some contribution to the art “beyond the skill of the calling” (Allsop Inc v Bintang Ltd [1989] FCA 297; (1989) 15 IPR 686 at 701)”.
Common general knowledge was described by the High Court in Lockwood Security Products Pty Ltd v Doric Products Pty Ltd [No 2] [2007] HCA 21; (2007) 235 CLR 173 at [55] in the following terms:
“the background knowledge and experience which is available to all in the trade in considering the making of new products, or the making of improvements in old”.
In order to qualify as common general knowledge, it is not necessary that the knowledge be memorised by the PSA. It is sufficient if it is material in the field in which the PSA is working which he or she knows exists and to which he or she would refer as a matter of course (ICI Chemicals & Polymers Ltd v Lubrizol Corp Inc [1999] FCA 345; (1999) 45 IPR 577 at [112] per Emmett J; ICI Chemicals & Polymers Ltd v Lubrizol Corp Inc [2000] FCA 1349; (2000) 106 FCR 214 at [57]; see Bodkin C, Patent Law in Australia (2nd ed, Thomson Reuters, 2014) at [4110]).
Submissions
The opponent’s submissions on the ground of inventive step were primarily directed towards the claimed invention being obvious light of the common general knowledge (‘CGK’). They also argued the invention to be obvious in light of either of D4 or D13 when combined with the CGK. I have discussed who the notional skilled addressee is above. It follows that I need to establish the common general knowledge and the problem to be addressed in order to assess the opponent’s contentions
The problem to be solved
The applicant submits the problem addressed by the application is the identification of worn wear parts in GET equipment[35].
[35] Paragraph 52 of the applicant’s written submissions
The opponent made no submission as to what it considers to be the problem faced by skilled addressee.
As best understood, the specification characterises the problem to be solved as being how to monitor wear parts on an operating implement associated with heavy equipment, and the automatic monitoring of their condition using image based sensors[36].
[36] Page 2 of the specification.
It is appropriate to consider whether these problems accord with the principles of AstraZeneca.
Whilst Dr Hillier’s evidence indicates that vision based systems (i.e., image processing) for monitoring the presence of wear parts were commercially available before the priority date[37], he does not indicate that these were also used for monitoring the condition of the wear part[38]. The only means of monitoring the condition of wear parts indicated by Dr Hillier were physical means directly on the wear part and not automatic[39]. I also note that Dr Hillier indicates that, whilst he suspected the applicant was using neural networks, he was not aware of anyone else using them for monitoring wear parts[40].
[37] Paragraph 17 & 19 of NH-1
[38] Paragraph 21 of NH-1
[39] Paragraph 21 of NH-1
[40] Paragraph 23-24 of the 1st Hillier Declaration
I consider that the applicant’s characterisation of the problem overly broad, noting that Dr Hillier gives evidence of other means of identifying the condition of wear parts.
Because it appears in the evidence above that it was not known to monitor the condition of wear parts from images, as opposed to monitoring their presence, I consider it appropriate to characterise the problem as the one in the specification.
There were no submissions on the appropriateness of adopting the problem solution approach to assess inventive step. I see no reason why this approach should not be used here, noting that I do not take the problem as I have defined it to not be in line with the principles discussed at [203] of AstraZeneca.
The common general knowledge (CGK)
The opponent provided a list of what it considered was CGK in the art in their written submissions[41], and points to both the specification and the evidence of Dr Newman to support those statements. The opponent’s submissions took an approach of dividing the CGK up into that which pertains to wear part monitoring and that which pertains to neural networks. I have previously considered that the skilled addressee would be aware of how neural networks function for image analysis even though, as discussed above, it does not appear to be CGK that these could be successfully applied to monitoring wear parts for either presence or condition.
[41] Paragraphs 30-35 of the opponent’s written submissions
With respect to what the opponent submits is the CGK of wear part monitoring in general, this can be summarised as follows: earthmoving equipment has wear parts; they can get damaged and lost; the importance of monitoring them; and that there are imaging processing systems commercially available that could monitor their presence and absence. They put forward the specification itself as evidence of this[42]. In so far as the skilled addressee would be expected to have a peripheral knowledge of neural networks and image processing in general, the opponent’s submissions can be summarised as follows: that it was CGK as to how neural networks are used to analyse images, in particular the usage of pixel intensity and the layers of neurons in the network. The opponent relied on the evidence of Dr Newman to substantiate this[43].
[42] Pages 1, 2 & 33 of the Specification
[43] Paragraphs 12-48 of the Newman Declaration
Unsurprisingly the applicant did not challenge these as being CGK in their submissions. There is nothing controversial or surprising in what the opponent was submitting as CGK. I am satisfied that these aspects fall within the CGK of the skilled addressee.
However, with respect to obviousness the opponent makes two more specific submissions on what was is CGK to the skilled addressee.
Firstly, they submit that the ‘stepwise method’ is known and part of the CGK. They point to D1 and D12[44] as evidence of this[45], and make submissions as to how the claims should be construed. They also submit that Dr Newman’s construction of the claim is wrong, in particular how he construes the initial step of identifying the operating implement before the wear part.
Secondly, the opponent submits that the analysis of wear part condition is also known in the art. They rely on the evidence of Dr Hillier and submit that prior art documents D1, D3, D4 and D6 disclose this[46].
[44] Also annexed as NH-8 in the evidence in support
[45] Paragraph 127 of the opponent’s written submissions
[46] Paragraph 130 of of the opponent’s written submissions
The stepwise method
The opponent submits that starting the method with a process that looks for pictures with an implement in them, only analyses those parts of those images which have an implement in them for the presence of a wear part, and only determines the condition of the wear part when the wear part is detected, is known to the skilled addressee. It is this which I will refer to as the ‘stepwise method.’
For the claimed invention to be obvious when the CGK alone is combined, it is not relevant that this stepwise method is arguably disclosed in D1. What is relevant in the evidence is what the experts considered to be the CGK with respect to utilising a stepwise method of image analysis. This may be confirmed by reference to the evidence of the experts and their reference to publications.
Neither Dr Hillier nor Dr Newman suggest that D1 formed part of CGK of the skilled addressee. Dr Hillier never discusses the document in any detail[47][48], and Dr Newman only gives a brief summary of what he considered it to disclose or fail to disclose[49]. I do not take D1 to be CGK as such.
[47] Paragraph 42 of the 1st Hillier Declaration
[49] Paragraph 158 of the Newman Declaration
Dr Hillier discusses D12 in his first declaration with respect to the claims as accepted. He notes it comprises a summary of the state of the art in image processing 14 years before the priority date. He notes that some terminology has changed[50], and the artificial neural networks are simply referred to as neural networks. Dr Hillier considers D12 to exemplify how well known using neural networks for image processing was at the time of its publication in 2002[51].
[50] Paragraph 35 of the 1st Hillier Declaration.
[51] Paragraphs 33 & 37 of the 1st Hillier Declaration.
With respect to the stepwise method and ‘segmentation’ as described by Dr Hillier, he directs attention to sections 3.3 and 3.4 of D12 regarding those earlier claims of the application (at acceptance) that defined the feature of detecting if an operating implement is present in the image before processing the image to detect a wear part. Dr Hillier does not address the stepwise feature in his evidence in reply which was after the claims had been amended.
In Dr Newman’s evidence, he also relies on D12 to support some of his own assertions regarding the complexity of using neural networks for image analysis. He does not discuss whether he considered this document as being representative of what was CGK to the skilled addressee. He only states that considers it does not “… disclose or suggest the use of neural networks in the particular application of locating and then assessing the condition of wear parts in GET…”[52]
[52] Paragraph 153 of the Newman Declaration
Importantly, I do not need to consider whether and what aspects of D12 are representative of the CGK of the skilled addressee around neural networks because I do not construe it as disclosing the stepwise method, as I will now explain.
D12 is an article reviewing more than two hundred applications of neural networks in image processing, discussing what the authors consider is the present and possible future role of neural networks. It provides an explanation of how neural networks can be used to process images, and the types of tasks they can be expected to perform in the analysis. With reference to the term “segmentation,” D12 teaches that this is the partitioning of an image into parts that are coherent according to some criterion. When considered as a classification task, the purpose of segmentation is to assign labels to individual pixels or voxels. This is the division of an image by boundary.
The examples in D12 to which Dr Hillier draws attention, ‘the identification of surfaces’ or ‘deciding if a pixel is inside or outside a segment’, would appear to demonstrate that it was well known to use a neural network to perform the first part of the step of integer 2 in the claims per se – that is, determining which part in the image was the operating implement. But I do not follow how this supports that the stepwise approach of the method was CGK – that is identifying the operating implement first, then looking for a wear part, and once detected by the neural network only then determining the condition. Hence, even assuming D12 is CGK, it does not comprise all the steps of the claimed invention in suit.
The opponent also submits that “feature extraction” as discussed under section 3.2.2 of D12 is also evidence of the stepwise method. D12 describes feature extraction as any operation that extracts significant components from an image[53]. Neither Dr Hillier or Dr Newman discuss this part of D12, and I do not follow how this quite general disclosure of extracting features corresponds to the stepwise method of the claim.
[53] Section 2 of NH-8
D12 also discusses hierarchical segmentation approaches designed to combine neural networks on different abstraction levels[54] (original emphasis included):
“The guiding principles behind hierarchical approaches are specialisation and bottom up processing: one or more ANNs are dedicated to low level feature extraction/segmentation, and their results are combined at a higher abstraction level where another (neural) classifier performs the final image segmentation.” (emphasis in original)
[54] Section 3.3.1 of NH-8
It would appear that the stepwise approach adopted, which is top down processing, was against the guiding principle of the hierarchical approach.
It follows that, even in the general sense of using a neural network, it is not CGK to the skilled addressee that they should first detect the operating element and then the wear part using a neural network. I cannot be satisfied that taking a stepwise approach to image analysis with a neural network was CGK.
Determining condition
The opponent also submits that determining the condition of a wear part was CGK to the skilled addressee. To support this, they submit that D4 demonstrates that this was well known to the skilled addressee. The opponent further submits Dr Hillier’s evidence regarding D1, as well as the disclosure of D3 and D6 also supports this submission.[55]
[55] Paragraph 133 of the opponent’s written submissions.
Whilst, in a broad sense, it would be expected that determining a condition of a wear part would be CGK, what is more relevant for obviousness is whether it was CGK to determine the condition of a wear part by analysing an image of a wear part, and to do so after detecting the wear part with a neural network.
D1, D3, D4 and D6 are all patent publications. A publication does not automatically form part of the CGK, and patent documents are no exception to this. To establish a feature as being CGK, it must be supported by evidence such as endorsement by a relevant expert. Turning to Dr Hilliers comments on D1[56], he note that there is a statement in D1 of “constant monitoring of the bucket tooth line status in terms of teeth wear”. This is not a statement that suggest this was well known to a skilled addressee. I cannot find any evidence from either Dr Hillier or Dr Newman that D1, D3, D4 or D6 would be part of the CGK for a skilled addressee.
[56] Paragraph 7 of the 2nd Hillier Declaration
I could also find no evidence from either expert that it was well known to determine the condition, in contrast to presence, of a wear part by analysing the image. As such the opponent’s submissions that determining the condition of a wear part is CGK is unsupported by their evidence. In the absence of such evidence, I cannot be satisfied that is CGK to the skilled addressee to determine the condition of a wear part by analysing an image of the wear part, after detecting the wear part with a neural network.
Adding this to my considerations above regarding the stepwise method, I consider it clear that the opponent fails to establish as CGK a stepwise method of first identifying an operating implement then detecting the wear part and finally determining the condition once the wear part is detected. The opponent bears the onus to establish this was CGK to the skilled addressee.
Obviousness in light of common general knowledge alone
As discussed above, the use of a stepwise method in analysis and the determination of condition using this stepwise method were not established as being CGK. Thus, it cannot be said that the claimed invention is merely an obvious combination of CGK elements.
However, the opponent submits that because the ability of neural networks to detect wear parts is CGK, it would be obvious to apply a stepwise method to analysing the images and subsequently determine their condition.
It follows that I must consider if the skilled addressee, when faced with the problem of determining the condition of a wear part, could arrive at the claimed invention by routine means. Thus, the question to answer is whether, in solving the problem, they would be directed to utilise a stepwise method to apply the neural network to detect and locate a wear part and then determine the condition of the wear part when detected and located.
With respect to evidence that the skilled addressee might apply such a method, Dr Hillier’s evidence with respect to Claims 14 and 20 of the specification as accepted is the most relevant to the invention as presently claimed in Claims 1 and 19. Dr Hillier considers the approach of identifying the operating implement before the wear part is a common approach and supports this by reference to NH-8[57]. However, as discussed above with respect to the ‘stepwise method’, this would appear to be directly at odds with what NH-8 teaches with respect to bottom up processing for neural networks. I could find no evidence from Dr Hillier as to why the skilled addressee would be directed to take this approach
[57] Paragraph 36.18 of the 1st Hillier declaration.
Dr Newman does not give evidence to whether it is obvious or not to first detect an operating implement before using a neural network to detect the presence of a wear part. However, Dr Newman does explain what he considers to be the disclosure of D7 –which is an example of using a neural network to detect objects. I note that Dr Newman describes D7 as using what I consider to be a bottom up approach, i.e., finding the wear part before the operating implement[58]. I could not find a single example of using a ‘top down’ approach to neural network analysis in the evidence submitted.
[58] Paragraph 164 of the Newman declaration
I could find no evidence from Dr Hillier, in relation to Claim 20 of the accepted specification, to suggest that determining the condition of the wear part after detecting it via a neural network would be obvious. Dr Hillier’s comments were limited as to the feasibility of it, and that the skilled addressee could expect it to work[59]. In contrast, Dr Newman provided evidence that it would not have been obvious to the skilled addressee that this could be done because neural networks were not considered reliable or practical for detecting wear parts[60].
[59] Paragraph 17 of the 2nd Hillier Declaration
[60] Paragraph 126 of the Newman declaration.
Whilst it might seem obvious in hindsight, when I weigh up the evidence, I consider that it is highly unlikely that the skilled addressee would be directed to adopt a stepwise method to analyse images of wear parts. It would appear to go directly against the conventional approach to using neural networks.
But I also consider that it is unlikely that the skilled addressee would further be directed to also determine the condition after detecting the wear part with a neural network. Whilst the skilled addressee might recognise the possibility of success, I consider Dr Newman’s evidence quite compelling that the skilled addressee would expect that such an endeavour to be an impractical and unworkable solution in part due to the unreliability of the neural network.
As such, I can see no circumstance whereby the skilled addressee would be directed to arrive at the claimed solution in light of the CGK when faced with the problem. I can find no motivation to use a stepwise method with neural networks, or subsequently determining the condition of the wear part.
As such, I find the claimed invention inventive in light of CGK alone.
Obviousness in light of common general knowledge combined with D4 or D13
The opponent submits that the claims also lack an inventive step in light of D4 or D13[61] when combined with the CGK. Their submissions are that, for the reasons already given with respect to obviousness in light of CGK alone, the claims also lack an inventive step. I did not find the written submissions particularly helpful for my considerations, as no detail regarding the disclosure of these documents was given.
[61] Also annexed as NH-9 in the evidence in support
D4 is directed to monitoring wear parts on earth moving machinery, in particular the wear parts of the tracks. However, D4 is not limited to this application and discloses that its method can also be applied to work tools (e.g., a saw or drill) that have wear parts[62]. I consider that this falls within the scope of an operating implement, to the extent these could be characterised in a particular instance as heavy equipment.
[62] Paragraph 24 of D4
The disclosure of D4 is silent to the usage of neural networks, using a stepwise method, and determining the presence of the operating implement and wear part. I do not follow how it would be obvious to modify the method disclosed in D4 to fall within the scope of Claim 1 or Claim 19, and the opponent has provided no elaboration on this point.
D13 is a paper on the application of convolution neural networks for image processing. It is concerned with detecting cracks in a sewer pipe with an autonomous robot. Whilst it is using a convolutional neural network to detect the cracks, I cannot find any disclosure of a stepwise method to achieve this. Again, I do not follow how it would be obvious to modify the method disclosed in D13 to detect wear parts and determine their condition as defined by Claim 1 or Claim 19. No arguments have been provided by the opponent on this point.
It follows that it would not be obvious to modify either of these methods such they fall within the scope of the appended claims 2-18 and 20.
Summary on obviousness
As the claim is not obvious in light of the CGK alone or in combination with the prior art documents, I cannot be satisfied that the claimed invention defined by claims 1to 20 lacks an inventive step.
MANNER OF MANUFACTURE.
The statutory basis for manner of manufacture is found at s18(1)(a) of the Act which states:
“(1) Subject to subsection (2), an invention is a patentable invention for the purposes of a standard patent if the invention, so far as claimed in any claim:
(a) is a manner of manufacture within the meaning of section 6 of the Statute of Monopolies; and ...”
In National Research Development Corporation v Commissioner of Patents, [1959] HCA 67, (1959) 102 CLR 252 (“NRDC”), the High Court provided a statement of the law in this regard. At page 275:
“... a process, to fall within the limits of patentability which the context of the Statute of Monopolies has supplied, must be one that offers some advantage which is material, in the sense that the process belongs to a useful art as distinct from a fine art ...- that its value to the country is in the field of economic endeavour”.
The High Court though was not laying down a precise formulation that can be applied without detailed consideration and noted that a case-by-case approach should be taken. In D’Arcy v Myriad Genetics Inc. [2015] HCA 35 (“Myriad”), at [23]:
“This Court in NRDC did not prescribe a well-defined pathway for the development of the concept of ‘manner of manufacture’ in its application to unimagined technologies with unimagined characteristics and implications. Rather, it authorised a case-by-case methodology.”
In Myriad, Gageler and Nettle JJ. at [144] stressed the importance of having regard to the substance of the claimed invention, not simply the form of the claim:
“Whatever words have been used, the matter must be looked at as one of substance and effect must be given to the true nature of the claim.”
The principle of identifying the substance of the invention when determining whether a computer implemented method is patentable has been discussed in multiple decisions. For instance, Commissioner of Patents v RPL Central Pty. Ltd. [2015] FCAFC 177 (“RPL”) at [96] – [98]:
“A claimed invention must be examined to ascertain whether it is in substance a scheme or plan or whether it can broadly be described as an improvement in computer technology. The basis for the analysis starts with the fact that a business method, or mere scheme, is not, per se, patentable. The fact that it is a scheme or business method does not exclude it from properly being the subject of letters patent, but it must be more than that. There must be more than an abstract idea; it must involve the creation of an artificial state of affairs where the computer is integral to the invention, rather than a mere tool in which the invention is performed. Where the claimed invention is to a computerised business method, the invention must lie in that computerisation. It is not a patentable invention simply to ‘put’ a business method ‘into’ a computer to implement the business method using the computer for its well- known and understood functions.
Is the mere implementation of an abstract idea in a well-known machine sufficient to render patentable subject matter? Is the artificial effect that arises, because information is stored in RAM and there is communication over the Internet or wifi, sufficient? Does any physical effect give rise to a manner of manufacture? Are the mere presence of an artificial effect and economic utility, without more, sufficient to determine manner of manufacture?
It is not a question of stating precise guidelines but of deciding, in each case, whether the claimed invention, as a matter of substance not form, is properly the subject of a patent.”
RPL went further with regard to the role the computer plays stating at [107]:
“Simply putting a business method or scheme into a computer is not patentable unless there is an invention in the way in which the computer carries out the scheme or method.”
Similarly, the importance of recognising that there needs to be an improvement in the computer per se for an invention to be a manner of manufacture was brought up in in Research Affiliates LLC v Commissioner of Patents [2014] FCAFC 150 (“Research Affiliates”). The Full Court of the Federal Court noted a distinction between mere implementation of an abstract idea in a computer and implementation of the idea in a computer that created an improvement in the computer. At [103]:
“… there is a distinction, between mere implementation of an abstract idea in a computer and implementation of an abstract idea in a computer that creates an improvement in the computer.”
Thus, in relation to computer implemented inventions, it is necessary to look at the invention as a matter of substance, rather than as a matter of form. If the invention is as a matter of substance directed to patentable subject matter, then it follows that it is a manner of manufacture as defined by previous authorities.
The Courts have been consistent in their approach in ensuring that where a method is claimed, consideration is not just given to the words of the method but that there are physical computing integers involved and that it is the combination of interworking computing integers or advance in those that needs to be considered when working out the substance of the invention. At RPL [112]:
“Recognising that the claims are to a method and system comprising a combination of integers, it is necessary to understand where the inventiveness or ingenuity is said to lie. Turning to the integers of the invention as set out at [36] and [38] and summarised at [37] and [39] above, it is apparent that, other than the integers providing that the computer processes the criteria to generate corresponding questions and presents those questions to the user, the method does not include any steps that are outside the normal use of a computer.”
The recent Full Federal Court decisions in Commissioner of Patents v Rokt Pte Ltd [2020] FCAFC 86 (“Rokt 2”) and Encompass Corporation Pty Ltd v InfoTrack Pty Ltd [2019] FCAFC 161 (“Encompass”) confirmed and applied the principles from Research Affiliates and RPL. More recently the Full Court in Commissioner of Patents v Aristocrat Technologies Australia Pty Ltd [2021] FCAFC 202 (“Aristocrat ’21”), at [56] – [57] observed in relation to a claim defining an electronic gaming machine (EGM) with a particular feature game:
“What this purpose-specific but extremely common computer does is play the feature game. Consequently, the substance of the invention disclosed by Claim 1 is that feature game implemented on the computer which is an EGM. It is therefore a computer-implemented invention.
As we have already observed, integers 1.10-1.12 embody an abstract idea which may be characterised both as a set of rules defining a family of games and as a business scheme for increasing player interest in an EGM. As such its implementation in the computer which is an EGM cannot constitute patentable subject matter unless it represents an advance in computer technology.”
On appeal to the High Court, in Aristocrat Technologies Australia Pty Ltd v Commissioner of Patents [2022] HCA 29 the Court was evenly split, and via section 23(2)(a) of the Judiciary Act 1903 affirmed the Aristocrat ’21 decision, while appearing to confirm that an advance in computer technology is not a useful test for patentability. Additionally, the High Court appeared to confirm that the decisions of the Full Federal Court in RPL, Research Affiliates, and Encompass were correct.
The principles of law that apply to the present matter in the context of computer implementation appear to be well reflected in that summarised and generally accepted at [200]-[201] by Robertson J in Rokt Pte Ltd v Commissioner of Patents [2018] FCA 1988 (“Rokt 1”) at [189] as follows:
“17.1 The Court must decide, as matter of substance not form, whether the claimed invention is the proper subject-matter for a patent: RPL Central at [99]; Research Affiliates at [106], [117].
17.2 This requires consideration of both the claims of the Application and the invention described in the body of the specification: RPL Central at [114].
17.3 The assessment is not done mechanically. There are no precise guidelines or mathematical formula. It is ‘a question of understanding what has been the work of, the output of, and the result of, human ingenuity’ and then applying the developed principles: Research Affiliates at [116]. See further RPL Central at [112]:
Recognising that the claims are to a method and system comprising a combination of integers, it is necessary to understand where the inventiveness or ingenuity is said to lie ...
17.4 One well-settled principle is that a distinction exists between a technological innovation and a business innovation. A technological innovation is patentable. A business innovation is not: Research Affiliates at [94]; RPL Central at [100]. Consequently, a business method or scheme is not, per se, a proper subject for letters patent: RPL Central at [96]. Nor are abstract ideas, mere intellectual information or mere directions for use patentable: Research Affiliates at [101]; RPL Central at [100].
17.5 A computerised business method or scheme can, in some cases, be patentable. However, ‘[w]here the claimed invention is to a computerised business method, the invention must lie in that computerisation’: RPL Central at [96] (emphasis added). This requires ‘some ingenuity in the way in which the computer is used’: RPL Central at [104]. It is not a patentable invention ‘to simply “put” a business method “into” a computer to implement the business method using the computer for its well-known and understood functions’: RPL Central at [96]. In other words, if the ingenuity lies in the business method or scheme alone, the invention will not be patentable despite the computer-implementation.
17.6 Thus, a claimed invention must be examined to ascertain whether it is in substance a scheme or plan, or whether it can broadly be described as an improvement in computer technology: RPL Central at [96]. Contrary to [the applicant’s submissions at [49]], this is a binary distinction: the invention is either an unpatentable scheme or plan, or it is a patentable improvement in computer technology. In conducting the analysis, it is useful to:
17.6.1 ascertain whether the contribution to the claimed invention is technical in nature: RPL Central at [99], Research Affiliates at [114];
17.6.2 consider whether the invention solves a ‘technical’ problem within the computer or outside the computer: RPL Central at [99], Research Affiliates at [103];
17.6.3 consider whether the invention results in an improvement in the functioning of the computer, irrespective of the data being processed: RPL Central at [99], Research Affiliates at [118];
17.6.4 consider whether the invention requires merely ‘generic computer implementation’, as distinct from steps which are ‘foreign’ to the normal use of computers: RPL Central at [99], [102]; Research Affiliates at [101]; and
17.6.5 consider whether the computer is merely the intermediary, configured to carry out the method using program code for performing the method, but adding nothing to the substance of the idea: RPL Central at [99].”
The submissions
The opponent submits that the substance of the purported invention as claimed broadly resides in the abstract idea of determining the condition of a tooth or wear part from an image of a bucket[63]. The opponent further submits that this is an abstract idea and involves no more than the use of a generic computer to perform well known image processing techniques and generic neural networks to generate data. Therefore, the opponent considers there is no creation of an artificial state of affairs where the computer is integral to the purported invention, and the computer is no more than a tool upon which the purported invention is performed.
[63] Paragraph 143 of the opponent’s written submissions
Manner of manufacture consideration
A stepped approach for assessing manner of manufacture can be summarised as construing the claim, identifying the substance of the claimed invention, and then asking whether the substance of the claim lies within established principles of what constitutes a patentable invention.
The claimed invention
I have previously construed the scope of the claims as discussed above, broadly claims 1 and 19 are directed to a stepwise method for locating a wear part and determining its condition:
(1) Capturing an image of an operating implement with wear members,
(2) Processing the image to detect the operating implement, and its location and thus region of interest in the image
(3) Processing the region of interest by analysing a smaller portion (pixel subsets) of the image to detect wear members.
(4) Using a neural network to determine if the image meets the matching criterion for a wear part being present, where the matching criterion is developed from training the neural network with labelled images.
(5) Analysing those subsets with wear part presence to determine a dimensional attribute and determine the condition of the wear part based on a pre-determined criterion
Because the claim defines the usage of a neural network to process the images, it is required to be implemented on a computer of some type. In so far as claims 1 and 19 go, there is no limitation that the image capture is being done on the same computer processing the images.
The steps at integers 2 and 3 could be broadly construed to include the selection of pixel subsets being done by human input, curating the portion of the image to be manually fed to the neural network. However, I cannot envisage how the claimed use of the neural network can be performed without at least requiring a computer.
To determine the substance of the claims, it is useful to ask the questions laid out in Rokt 1 to identify where the substance of the claimed invention lies in contrast to the form of the claimed invention.
Does the invention as claimed solve a technical problem within the computer or outside the computer?
As has been discussed above, the claimed invention seeks to improve the detection and location of wear parts by capturing images so as to allow the determination of a dimensional attribute, and the condition, of wear parts. There is a technical problem in the sense that image analysis via neural network is prone to a number of potential errors and computational complexity[64]. This represents a technical problem within the computer. Furthermore, monitoring of wear parts represents a technical problem for the use of earthmoving equipment, as the condition of wear parts directly impacts on the functionality of earthmoving machinery. This latter problem is one that is residing outside the computer. As such the substance of the claim is clearly directed to the identification of wear parts and determining their condition, and thus overcoming problems associated with this task.
[64] Paragraph 122-124 of the Newman declaration
Is the contribution of the claimed invention technical in nature?
As I have discussed above under other grounds, the contribution lies in the steps of the method taken. The method detects the operating implement, before the image is analysed by a neural network to detect the wear part, and only then is the wear part’s condition determined. I do not consider that this is a mere scheme or abstract idea. The steps in the method seek to improve the method of image analysis to allow the wear parts to be detected and located by the neural network, and in turn allow the wear part condition to be determined. This is a specific technical solution to address a specific technical problem associated with detecting wear parts, for the purposes of ascertaining their condition, utilising a neural network for image analysis. It is clear that the substance of the invention is directed to the steps of the method pertaining to selecting the subset of pixel to be processed by the neural network.
The remaining questions of Rokt 1
From the considerations above, it would appear that the substance of the invention clearly lies in the steps of the method and how they improve the method of detecting and locating the wear parts. As such, it is quite clear that, on considering these, the substance will not lie in something different to what I have identified above. Thus, further elaboration on the considerations from Rokt 1 is unnecessary.
Substance of the invention
In considering the questions above, I would characterise the substance of the claims as being directed to a method of determining the condition of a wear part by taking an image of heavy equipment and first locating the operating implement and its region in the image, then selecting a smaller image to analysed by use of the neural network to detect if a wear part is present, and, if a wear part is detected, then determine its dimensional attribute(s) and condition.
As it is quite clear from the above questions, the substance of invention lies in the technical realm, noting that it allows the detection of wear parts by machine logic, and facilitates this by including preparatory steps to improve the processing of the images via a neural network. The substance of the claimed invention is not directed to unpatentable subject matter, but seeks to solve a technical problem by producing a technical result of making a qualitative assessment of wear parts that may have otherwise been unfeasible by machine logic.
It follows that I am satisfied that this is patentable subject matter. This ground of opposition fails.
Utility
It is a requirement of subsection 18(1) of the Act that the invention, so far as claimed in any claim, is useful. Subsection 7A further states that an invention is not to be considered useful unless a specific, substantial and credible use for the invention (so far as claimed) is disclosed in the complete specification. It also states that the disclosure in the complete specification must be sufficient for that specific, substantial, and credible use to be appreciated by a person skilled in the relevant art.
In Merck, at [432], [433], [436] and [449]; Burley J summarised the principles relating to utility:
“Section 18(1)(c) of the Patents Act provides that an invention is a patentable invention for the purposes of a standard patent if the invention, so far as claimed in any claim, is useful. Until recently, the requirement that an invention so far as claimed be “useful” within s 18(1)(c) was defined solely by reference to the common law development of that concept. The words in Lane Fox v Kensington & Knightsbridge Electric Lighting Co … [1892] 3 Ch 424 at 431 of Lindley LJ (with whom Lopes LJ agreed) set the scene (emphasis added):
The utility of the alleged invention depends not on whether by following the directions in the complete specification all the results now necessary for commercial success can be obtained, but on whether by such directions the effects which the patentee professed to produce could be produced, and on the practical utility of those effects.
What the patentee ‘professed to produce’ is to be ascertained by having regard to what is now routinely referred to as the ‘promise of the invention’ being the promise that the specification is said to make of the invention claimed: Rehm Pty Ltd v Websters Security Systems (International) Pty Ltd (1988) FCA 232; 81 ALR 79 at 84 and 96–97; 11 IPR 289 at 292 at 305–6 (Gummow J); Décor Corporation Pty Ltd v Dart Industries Inc (1988) FCA 682:13 IPR 385 at 394 (per Lockhart J). This is assessed as a matter of construction of the specification: see generally ESCO Corporation v Ronneby Road Pty Ltd [2018] FCAFC 46; 358 ALR 431 at [182] – [239] (Greenwood, Rares and Moshinsky JJ)
…
In each case it is necessary to consider the nature of the promise of the invention by reference to the specification and also whether that promise is met by that which is the subject of the claims. Often that enquiry gives rise to a question of claim construction: if a broad claim includes something that does not meet the promise of the invention, will it be invalid for want of utility? …
…
There is no dispute that a promise may be implied from language used in the specification.”
Moreover, Apotex Pty Ltd v AstraZeneca AB (No 4) [2013] FCA 162 at [352] Jagot J pointed out that lack of utility requires evidence, not just speculation:
“Ultimately, an asserted lack of utility must be established by appropriate evidence, not be mere speculation that the invention will not work or meet the promise set out in the specification.”
The opponent submits that the promise of the invention is automatically assessing the condition of a wear part in a production mining environment. They further submit, however, that the specification itself fails to disclose critical aspects of successful implementation of the claimed invention in a mining environment. Specifically, the specification fails to disclose how to overcome environmental factors and how to label the collected dataset so as to train the neural network to determine the condition of a wear part. They direct attention to the evidence of Dr Newman to support their submissions. The applicant submits there is no evidence to support the ground, as the evidence of Dr Newman is in regard to the difficulties in following the approach of whole image analysis by a neural network as suggested by Dr Hillier in his first declaration.
Consideration
I have previously discussed the claimed invention with respect to sufficiency and support. From the specification[65], the promise of the claimed invention clearly lies in the improved detection and location of wear parts in images, so as to permit the determination of condition and a dimensional attribute of the wear part. It also promises to reduce the calibration steps required for installing and commissioning the system. I must determine whether the claimed invention fails to meet these promises.
[65] Pages 32 and 33 of the accepted specification
A first consideration for utility would be to determine whether the invention as claimed is able to detect and locate a wear part in an image, and to reduce the calibration steps required for installing the system. The onus rests with the opponent to demonstrate that the invention cannot for everything that falls within the scope of the claims. With respect to the opponent’s submissions that there are environmental factors to be overcome, the evidence they rely on (being Dr Newman’s) does not support this. As discussed above for sufficiency and disclosure, Dr Newman’s evidence is directed to difficulties in attempting to detect multiple wear parts in a whole image. There is no evidence that the environmental factors suggested by the opponent would inhibit the claimed invention from working.
With respect the opponent’s submissions that there is no disclosure of how to train a neural network to determine the condition of a wear part, again the opponent relies on Dr Newman’s evidence, and it does not support this. Dr Newman’s evidence is directed to those difficulties in attempting to train a neural network to detect multiple wear parts in a whole image.
The opponent has provided no evidence that the claimed invention cannot be achieved, or that it fails to provide the promised benefit. Nor has the opponent shown any evidence that there are embodiments within the claims that are not useful and will not work. This reduces the opponent’s case one of mere speculation that the invention will not work.
This ground of opposition fails.
CONCLUSION
None of the grounds of opposition has been made out.
Subject to appeal, I direct that the application proceeds to grant.
COSTS
Costs normally follow the event. However, the opposition process has resulted in the applicant making significant amendments to the claims after the filing of the opponent’s evidence in support. In this sense the opponent can be seen as being partially successful despite the opposition per se being unsuccessful.
As a consequence, I award costs against the applicant according to Schedule 8 of the Regulations up to the date the amendments were allowed, and from the date the amendments were allowed I award costs against the opponent according to Schedule 8 of the Regulations.
B. Norman
Delegate of the Commissioner of Patents
[48] Paragraph 7 of the 2nd Hillier Declaration
0
27
0