SwimEye As v Coral Smart Pool Ltd
[2024] APO 9
•19 February 2024
IP AUSTRALIA
AUSTRALIAN PATENT OFFICE
SwimEye AS v Coral Smart Pool Ltd [2024] APO 9
Patent Application: 2017211712
Title:Methods and systems for drowning detection
Patent Applicant: Coral Smart Pool Ltd
Opponent:SwimEye AS
Delegate:Neil Miller
Decision Date: 19 February 2024
Hearing Date: Written submissions filed on 6 September 2023
Catchwords: PATENTS - Section 59 – opposition to grant of a patent – grounds of novelty and inventive step – support for the claims – methods and systems for drowning detection – deep learning neural networks – distinguishing humans from non-humans in underwater images – lack of evidence from the opponent – failure of opponent to discharge onus of proof – whether prior art publicly accessible – claims novel and inventive – opposition unsuccessful – ground of support raised by Delegate under section 60(3) – no award of costs
Representation: Patent attorney for the applicant: Dr Don Angus, Collison & Co
IP AUSTRALIA
AUSTRALIAN PATENT OFFICE
Patent Application: 2017211712
Title:Methods and systems for drowning detection
Patent Applicant: Coral Smart Pool Ltd
Date of Decision: 19 February 2024
DECISION
I consider the invention so far as claimed to be novel and to involve an inventive step.
Claims 1-14, 16 (in part) and 17-25 lack support.
However, the amendments as proposed on 23 November 2023 appear to address the support deficiency. Whether that is in fact the case will be determined via final determination after these amendments are allowed and incorporated.
The Applicant has two (2) months from the date of this decision to propose any further amendments it deems appropriate.
I make no award of costs.
REASONS FOR DECISION
Background
Australian patent application no. 2017211712 (“the application”) was filed on 19 January 2017 in the name of Coral Detection Systems Ltd. The application is the national phase entry of international application no. PCT/IL2017/050081 which claims priority from US application no 62/287,165 filed 26 January 2016.
The application was accepted on 20 September 2021. A first Notice of Opposition was filed on 5 January 2022, this notice contained no details concerning the opponent. A correct Notice of Opposition was filed on 6 January 2022, the notice identifying SwimEye AS (“the opponent”), as the interested party.
The application was assigned to Coral Smart Pool Ltd (“the applicant”) on 22 December 2021. The assignment was recorded on 5 January 2022. The opponent was advised of the change of applicant on 19 January 2022.
On 23 March 2022 the opponent filed material in support of the opposition. Following the submission of these materials the opponent was requested to confirm the whether the material was intended to be the Statement of Grounds and Particulars (“SGP”) and associated mentioned documents. If so, the material was not in the appropriate format. An SGP together with accompanying prior art documents in the appropriate format were subsequently filed on 5 April 2022. Following the submission of the SGP the opponent did not elect to file further evidence, consequently per the direction of 6 April 2022 the documents accompanying the SGP are treated as Evidence in Support (“EIS”).
The applicant filed Evidence in Answer (“EIA”) on 10 October 2022. No Evidence in Reply (“EIR”) was filed by the opponent. On 6 January 2023 the opponent filed additional materials, on a review of these materials I note that they are essentially a refiling of the materials provided on 23 March 2022.
On 9 February 2023 it was acknowledged in correspondence from IP Australia that the evidentiary stages of the proceedings were complete, and that the opposition could be dealt with on the basis of written submissions, rather than an oral hearing. The parties were then provided with 14 days to request an oral hearing. Following this a Delegate of the Commissioner in correspondence of 26 June 2023 made the following directions:
· Written submissions in support of the opponent’s opposition under s59 are due four (4) weeks from the date of this letter;
· Written submissions in answer to the submissions in support are due four (4) weeks from the date the Commissioner advises the Applicant submissions in support have been filed; and
· Written submissions in reply to the submissions in answer are due two (2) weeks from the date the Commissioner advises the Opponent submissions in answer have been filed.
The opponent did not elect to file any written submissions in support. On 27 July 2023 a Delegate of the Commissioner provided the opponent with additional time to file their written submissions in support. The Delegate also requested that the opponent provide an explanation as to why such submissions were not filed within the initial time frame provided. The Delegate advised that if there was no response, he would proceed to issue directions for the filing of submissions in answer by the Applicant.
The Delegate in the correspondence of 27 July 2023 also commented on the additional material filed on 6 January 2023. In the Delegate’s view it seemed that the opponent intended this material to be taken as evidence of some kind. The Delegate then advised the opponent that the periods for filing evidence had expired, and the additional materials did not constitute evidence and as such did not form part of the opposition. The only way the materials could form part of the opposition was for the opponent to either request under an extension of time to file evidence under Reg 5.9 or request the material be admitted using Reg 5.23.
The opponent did not respond to the Delegate’s letter of 27 July 2023. In correspondence of 7 August 2023 a Delegate noted the lack of response from the opponent and advised that the opposition would proceed on the basis that the opponent would not be filing any submissions in support of its opposition, nor would the opponent seek an extension of time to file evidence under Reg 5.9 or make a request for the material to be admitted under Reg 5.23. Consequently, the Delegate issued a direction maintaining the timetable for written submissions in answer and reply as noted above.
The applicant filed their written submissions in answer (“the applicant’s submissions”) on 6 September 2023. No written submissions in reply to the submissions in answer were filed by the opponent. Following this a Delegate gave the opponent a further opportunity to provide reply submissions with the final date for submissions being set down as 10 October 2023. The opponent again did not elect to provide any submissions in reply and as such the matter was then referred to a hearing officer for determination based on the material of record.
Following an initial consideration of this matter, I exercised my power under section 60(3) of the Patents Act to take into account any ground of opposition. To this end, I invited specific submissions from the parties regarding a matter under section 40(3) of the Patents Act1990 in relation to the ground of Support. 29 November 2023. The submissions were accompanied by a set of proposed amendments to the claims. The opponent did not file any submissions in relation to the issue of support. I will discuss these submissions where necessary in the decision.
Onus and Standard of Proof
The present application was filed after 15 April 2013. As such, the present application is governed by the Act as amended by the Intellectual Property Laws Amendment (Raising the Bar) Act 2012 (“the Raising the Bar Act”). Amendments to Sections 7, 40 and 60 of the Act apply to the present case as a consequence of Schedule 1, items 55(1)(d) and 55(4)(a), and Schedule 6, item 133(7)(d) of the Raising the Bar Act.
The standard of proof that applies in the present case is the balance of probabilities. Under subsection 60(3A), if I am satisfied, on the balance of probabilities, that a ground of opposition to the grant of a patent exists, I may refuse the present application. Nevertheless, it remains the case that, in proceedings such as these before the Commissioner, the onus rests with the opponent to clearly establish its case in reaching a conclusion on any issue.
The nature of the opposition
The SGP lists the following grounds of opposition:
·s18(1)(b)(i) – the invention as claimed lacks novelty;
·s18(1)(b)(ii) – the invention as claimed does not comprise an inventive step;
The evidence
The opponent submitted several prior art documents together with the SGP, these documents are as follows:
· Thesis from the University of Stavanger 2009, referenced as D1;
· An unverified translation of D1 into English, refenced as D1′;
· A photograph of the front page of physical copy of D1, referenced as D1′′;
· “A Hidden Markov Model–Based Approach for Recognizing Swimmer’s Behaviors in Swimming Pool”, Hsi-Lin Chen et al, referenced as D2; and
· “DEWS: A Live Visual Surveillance System for Early Drowning Detection at Pool”, How-Lung Eng et al, referenced as D3.
Outside of these documents that opponent has not sought to put on any additional evidence such as declaratory evidence from experts or evidence attesting to the state of the common general knowledge in the art.
The applicant’s EIA includes a declaration from Dr Tamar Avraham dated 5 October 2022 (“the Avraham declaration”). Dr Avraham is the Chief Technical Officer and co-founder of Coral Smart Pool Ltd. and has 25 years of experience in developing image processing and computer vision algorithms and software. Dr Avraham has worked on various computer vision and video analytics research projects in cooperation with major industrial companies (Avraham declaration at paragraph [4]).
The specification
The specification is directed to a system and method for the detection of drowning. The system is composed of a pool unit, a remote alarm unit and a control unit (specification at page 7 lines 27-29). The pool unit is responsible for capturing a series of underwater images for analysis. In a preferred embodiment the pool unit captures the series of underwater images via a single camera for smaller pools and multiple cameras maybe utilised in the case of larger pools (see specification at page 8 lines 16-29). The images are then analysed to determine potential human in water candidates in each image. These candidates are then tracked across the sequence of images to determine a drowning risk/event.
The process for determining drowning risk according to the instant invention is best understood with reference to Figure 15 reproduced below.
As shown the sequence of video frames (i.e. the sequence of underwater images) is compared to a “humans-in-water” model (step 151). The humans-in-water model is composed of a plurality of models, each representing a different posture of a human in water. Posture as defined in the specification refers to the way a human appears in the 2D projection in the image plane, not to the physical posture of a human in 3D space (specification at page 14 lines 12-16).
The result of the comparison is a plurality of humans-in-water candidates per image (step 152). The humans-in-water candidates may each belong to a different posture. Overlapping candidates can then be combined into a single candidate. This unification can be performed by providing each candidate with a confidence grading through processes such as non-maximal suppression or determining an average location, where the location of a new (unified) candidate is determined based on a weighting of each candidate and its confidence grading.
Once the series of humans-in-water candidates is determined a multi-object tracker is employed (step 153). The object tracker in this case is utilised to recognise a potential drowning risk. The tracker effectively looks for situations in which a human is submerged and does not move for predefined amount or for a predefined period. To do this the tracker assigns different tracks to different postures, these are known as active tracks. For each active track an estimate of the amount of movement of a candidate over a certain temporal window is determined. When the amount of movement is below a particular threshold or there is no movement within a predetermined time frame a drowning risk/event is determined. A warning, or an alarm may then be outputted depending on the severity of the drowning risk/event.
The claims
The specification as accepted includes a total of 25 claims, with claims 1, 17 and 25 being the independent claims. Claim 1 is directed to a method of detecting human drowning. Claim 17 is the corresponding system claim to that of claim 1, while claim 25 is directed to a computer program for performing the method of claim 1. Both claims 17 and 25 recite similar limitations to that of claim 1. Dependent claims 2-16, 18 and 19 recite additional limitations relating to the various learning algorithms, tracking techniques, use of clustering, use of weightings etc. and the type of pool in which the method/system is employed, for example an above ground residential pool. Dependent claims 20-24 are directed to various hardware arrangements including the use of solar panels etc.
Claim 1 as accepted is reproduced below:
1. A method of detecting human drowning, comprising, by a system comprising a processing unit:
performing detection of humans in a sequence of underwater images taken by a single
camera in an operating zone, for identifying humans-in-water candidates in the
underwater images, said detection using at least a deep learning neural network
differentiating between humans and non-humans in the sequence of underwater images,tracking humans-in-water candidates throughout this sequence, and
detecting human drowning risk,
wherein the system is operable to detect human drowning risk without calibration with
respect to said operating zone.Claim 1 is broadly directed to a method for drowning detection wherein a series of underwater images of an operating zone are analysed by a deep learning neural network (DNN) to identify humans-in-water candidates. The humans-in-water candidates are then tracked throughout the sequence to determine a drowning risk.
Person Skilled in the Art
The specification is to be construed through the eyes of the person skilled in the art being a notionally non-inventive skilled worker aware of the common general knowledge in the relevant field. In KD Kanopy Australasia Pty Ltd v Insta Image Pty Ltd [2007] FCA 481; 71 IPR 615 Kiefel J identified ([16]) the person skilled in the art (“PSA”) as:
“...a person acquainted with the surrounding circumstances of the state of the art and manufacture at the relevant time...They are likely to have a practical interest in the subject matter of the invention...and may often work in the art with which the invention is connected.”
In Root Quality Pty Ltd v Root Control Technologies Pty Ltd [2000] FCA 980; 9 IPR 225, Finkelstein J stated at [70]:
“He is the person to whom the patent is addressed and who must construe it. He is the person whose knowledge will determine whether a patent is novel. He is the person who will judge whether a patent is obvious.”
In AstraZeneca AB v Apotex Pty Ltd [2015] HCA 30 (at [23]), the High Court additionally noted that:
“The notional person is not an avatar for expert witnesses whose testimony is accepted by the court. It is a pale shadow of a real person – a tool of analysis which guides the court in determining, by reference to expert and other evidence, whether an invention as claimed does not involve an inventive step.”
On the face of the specification, I consider that the PSA would require knowledge of machine learning algorithms such as support vector machines, neural networks including deep convolutional neural networks, CNN (Convolutional neural networks), Decision Trees/Forests based technologies, HOG (Histogram of Gradients) based technology, DPM (Deformable Parts Model) based technologies or any other classification/modelling technology. The PSA would also require an understanding of various image processing and tracking techniques such as Kalman filter, Particle Filters, mean shift etc. The skilled person would also require some knowledge concerning various postures of humans in water and how such postures would appear in the 2D image plane including an awareness of certain tilts of the torso, the legs and arms, or any other descriptions associated with the posture of the body for particular swimming strokes/styles.
The opponent has not sought to comment as to what skills and knowledge should be attributed to the hypothetical skilled person. By contrast the applicant has submitted evidence from Dr Avraham, whom in their view represents a reasonable facsimile for the PSA. I note that Dr Avraham has 25 years of experience, in developing image processing and computer vision algorithms and software. Dr Avraham has also worked on various computer vision and video analytics research projects in cooperation with major industrial companies (per Avraham declaration at paragraph [4]). I also note that Dr Avraham does not seemingly have any expertise or knowledge concerning various postures of humans in water. Nonetheless Dr Avraham does possess a number of skills which align with the notional PSA as I have outlined above. As such Dr Avraham would seem well placed to provide evidence as to what may constitute the common general knowledge (“CGK”) in the art. I am, however, mindful of the fact that Dr Avraham is listed as a co-inventor and as such has a vested interest in the opposition.
Construction
The rules of construction for an Australian patent specification are well summarized in Decor Corp v Dart Industries [1988] FCA 399; 13 IPR 385, the correct application of these rules to the construction of claims was discussed by Bennett J in H Lundbeck A/S v Alphapharm Pty Ltd [2009] FCAFC 70; 81 IPR 228 at [118] – [120]:
“the words in a claim should be read through the eyes of the skilled addressee in the context in which they appear...while the claims define the monopoly claimed in the words of the patentee's choosing, the specification should be read as a whole ... it is not permissible to read into a claim an additional integer or limitation to vary or qualify the claim by reference to the body of the specification...terms in the claim which are unclear may be defined or clarified by reference to the body of the specification.”
I would also note what was said in Eli Lilly and Company Limited v Apotex Pty Ltd [2013] FCA 214 at [139]:
“It is well settled that the Court should, from the outset, approach the task of patent construction with a generous measure of common sense. The Court must place itself in the position of a person skilled in the relevant art, being the subject matter of the patent. From this perspective, the patent is to be read as a whole, in the context of the specification and in light of the prevailing common general knowledge and state of the relevant art at the priority date.”
While on the surface the interpretation of the claim would appear to be relatively straight forward there are a number of terms that require closer scrutiny, notably the phrases “underwater image” and “operating zone”.
Underwater image
The “Macquarie Dictionary Online, 2023, Macquarie Dictionary Publishers, an imprint of Pan Macmillan Australia Pty Ltd, defines the term underwater as “being or occurring under water”. In the present circumstances the term underwater image could therefore refer to an image taken underwater i.e. the image is captured below the waterline. Alternatively, the phrase could refer to an image of something underwater which does not preclude the image being captured above the waterline. The fact that the claim does not specify the positioning of the camera during the capture of the image does not favour one of the above interpretations over the other.
Under the general principles of construction, it is permissible to have recourse to the body of the specification to further qualify the meaning of the terms utilised in the claims. Looking at the specification numerous refences are made of both cameras and underwater cameras (see specification at pages 3, 4, 5, 8-10 and 12), of note is the discussion on page 8 lines 16-28 concerning the pool unit. At lines 16-18 of page 8 the specification simply refers that the unit includes at least one video camera, and in embodiments the system's cameras may enable a full visual coverage of the pool, each camera may cover a different part of the pool. The discussion at page 8 lines 20-28 then provides details as to the number of cameras and the positioning of the camera i.e. at the corner of the pool along the edges of the pool or positioned to supply full coverage of the pool. There is nothing in the passages provided on page 8 that indicates that the camera is positioned below the water line, rather the passages simply indicate that the camera or cameras need to be positioned to adequately cover the pool. Similarly, the discussion concerning the method for drowning detection as illustrated in Figure 4 refers to the system sensing underwater image from the video camera/s no reference is made as to the positioning of the camera.
In contrast to the above I note that the depiction of the pool unit according to Figures 3A-3C (reproduced below) shows that the camera is designed to be deployed below the waterline.
As shown in the above figures the camera (which I have denoted as C in Fig 3A) is positioned at the base of the arm (which I have denoted as A in Fig 3A). The arm extends over the edge of the pool so as to position the camera below the waterline. That is the camera per the embodiments of Figs 3A-3C is an underwater camera. I also note that the discussion of the method of Figure 7 also refers to the camera being an underwater camera (instant application at page 12 lines 9-14).
Considering the specification as a whole it does not appear to be limited to the use of an underwater camera for the capture of the underwater images. In light of this I am inclined to interpret the term underwater image broadly as simply meaning an image of something (i.e. an object or human) being underwater.
Operating zone
The term operating zone does not place any meaningful limitation as to the area that is covered by the camera. Operating zone is broad enough to cover any body of water. On a review of the specification reference is made to residential swimming pools including above ground and underground pools. Most notably the entire discussion relating to the preferred embodiments all include a pool unit that can be deployed in a swimming pool. The specification at page 8 also refers to varying the number of pool units deployed depending on the size of the pool i.e. small, medium or large pools. I note that the specification references different types of pools, including private/residential pools and commercial pools. However, I do not see any distinction being drawn based upon any of the physical parameters of such pools, with these examples merely being indicative of their apparent end usage. I am also conscious of the fact that unduly limiting the swimming pools in general may be viewed as reading in a gloss from the specification. All that the claims require is there be a camera that captures images of swimmers underwater. In the present circumstances the common-sense construction of the term operating zone is that it is referring to a body of water that contains swimmers. This is further supported by the fact that a number of dependent claims then seek to limit the application of the invention to above ground pools.
Taking the above into consideration it is apparent that claim 1 is directed to a method of detecting human drowning wherein a camera is utilised to capture a series of underwater images (i.e. images of objects below the waterline or surface) within a body of water containing swimmers. The images are then analysed using deep learning neural network to differentiate between humans and non-humans and the candidate objects denoted as human are then tracked throughout the series of images to detect a drowning risk.
Novelty
It is well established that the general test for lack of novelty is the reverse infringement test. The classic formulation of this test is that given by Aickin J in Meyers Taylor Pty Ltd v Vicarr Industries Ltd [1977] HCA 19; (1977) 137 CLR 228 at page 235:
“The basic test for anticipation or want of novelty is the same as that for infringement and generally one can ask oneself whether the alleged anticipation would, if the patent were valid, constitute an infringement”.
It follows that if a citation discloses all the integers of the claim, the claim will lack novelty. If the citation does not disclose all the integers of the claim, the claim will still lack novelty provided the citation discloses all the essential integers of the claim. However, if one or more of the essential integers are not disclosed in the citation, the claim is novel (see Nicaro Holdings v Martin Engineering [1990] FCA 40; 16 IPR 545 and Catnic Components Ltd v Hill and Smith (1982) RPC 183).
It is noted that General Tire & Rubber Company v The Firestone Tyre and Rubber Company Limited [1972] RPC 457 at 486 stated that:
“If … the prior publication contains a direction which is capable of being carried out in a manner which would infringe the patentee’s claim, but would be at least as likely to be carried out in a way which would not do so, the patentee’s claim will not have been anticipated, although it may fail on the ground of obviousness.”,
and that, in order to meet the test for anticipation, the prior art:
“…must contain clear and unmistakeable directions to do what the patentee claims to have invented…A signpost, however clear, upon the road to the patentee’s invention will not suffice. The prior inventor must be clearly shown to have planted his flag at the precise destination before the patentee.”
The opponent’s case for want of novelty is best articulated in the correspondence of 23 March 2022 which was subsequently refiled on 6 January 2023. Under the heading “Reasons for Opposition”, the opponent notes that Swimeye has used camera/cameras to obtain underwater pictures, these pictures are then used in combination with algorithms to detect humans drowning. The opponent’s work in this area also led them to commission the masters thesis that is the subject of D1-D1′′. On the basis of this information the opponent submits that the claims as currently presented lack novelty. Outside of the submissions of 22 March 2022 the opponent has not sought to provide any declaratory evidence or elected to provide additional written argument in support of their case for lack of novelty. Given the material submitted by the opponent in this case I am of the understanding that the opponent seeks to rely primarily on the disclosure of D1-D1′′, with D2 and D3 not being expressly pressed for the ground of lack of novelty.
Consideration of D1-D1′′
The first issue I have with D1, is that the opponent has not established with any degree of certainty that D1 was in fact published before the priority date of the present application. I note from the front page of D1 and D1′′, as filed with the SGP, that a date of 12 June 2009 is referenced. It is not apparent, however, on the face of the evidence provided whether this date can be treated as a publication date of the document. This date could in fact be the submission date of the thesis. If the date is the submission date, then this also raises the question of whether the information in D1 was publicly available.
The question of public availability was considered in Stanway Oyster Cylinders Pty Ltd v Clement Rex Marks [1996] FCA 1544, wherein the Federal Court at [7] observed in referring to the decision in Sunbeam Corporation v Morphy-Richards (Australia) Pty Ltd [1961] HCA 39, that the concept of information being “publicly available” involves the information being accessible to the public. The notion of accessible to the public has been considered in a number of authorities including Gadd v Mayor of Manchester (1892) 9 RPC 516 at page 527; Fomento v Mentmore (1956) RPC 87 at page 105 and Monsanto (Brignac's) Application (1971) RPC 153 at page 159 and has been taken to mean information that the public has or can acquire by consulting a source open to it, that is the material can be inspected "as of right" by the public. Moreover, it is sufficient that the information is available to a single person, provided that person is able to use the information freely without an obligation of confidence. In Nicator AB's Applications 7 IPR 504; [1986] APO 33, information contained in a document was still considered to be publicly available even in circumstances where the document could not be located based on certain information.
I also note that the front page of the thesis contains a reference to being open/confidential. What this appears to indicate is that the document in question could have been submitted in one of two states, the first being an open state which implies that the document was accessible to all as of the date of the paper’s submission i.e. accessible to the public as of June 2009. The second state being confidential means that the document was not available to the public on submission. There is no clear indication from the copy of D1 or D1′′ as to whether the material was submitted in confidence or not. The opponent states in the SGP that D1 was submitted without any obligation of confidentiality at the University of Stavanger, Norway, during the spring semester of 2009. I can appreciate that the document may well have been submitted without an obligation of confidentiality, however outside of the opponent’s statement as to this fact, no further evidence has been provided to establish that the document was in fact publicly accessible as of 12 June 2009. It may well be reasonable to surmise that the thesis in question may have been published or otherwise made available to the public at some point after 12 June 2009 and before the priority date. This is speculation on my part, as the evidence provided does not allow me to determine a date of publication of D1. Consequently, the opponent has not established the publication date of D1 and this calls into question whether D1 is a valid prior art reference for the purposes of novelty.
Putting the issue of publication to one side, D1 in my view still fails to anticipate claim 1 as currently presented. When I assess the disclosure of D1 through the lens of the translation (i.e. D1′) provided by the opponent, I note that D1′ firstly discusses the arrangement of the SwimEye system that is the subject of the research being conducted under the thesis. This is best understood with reference to Figure 1.1. of D1′ reproduced below.
As shown the SwimEye system considered in the thesis utilises one or more camcorders to capture images from the pool bottom. That is SwimEye captures a series of underwater images, these being images of the pool taken below the waterline. These images are then analysed to determine incidents of drowning. The analysis is done by counting the number of white pixels in a binary image and classifying the object in the image as human/fixed object if the number of pixels is greater than a given limit (D1′ at page 3 lines 1-3). The paper at page 2 lines 1- 5 discusses its motivation, namely to refine the detection of the SwimEye system by separating people/solid objects from noise. Noise in this context refers to air bubbles, reflections of the pool floor etc. The solution proposed by the paper is the use of a neural network to separate people/solid objects from noise within the sequence of images (D1′ page 3 line 3 - page 5 line 7).
On further review of D1′, I note that it is primarily focused on the removal of noise from the images, there being no distinction drawn between a human and a solid object within the images as such. In fact, under the system of D1′ humans and solid objects in an image are combined into a single class. This is best exemplified by the following passage of D1′ at page 24:
“In order to get even better classification, it is appropriate to try to distinguish people from the fixed objects. This task has merged people and solid objects into one class, separating them could lead to a more precise classification (emphasis added). In order to do this, one must define characteristics that best describes people. More analysis images are also needed where there is actually a human being in the picture.”
D1′ suggests that separating humans from solid objects could be performed to improve accuracy, however there is no indication that the system of D1′ in fact performs such a separation. The focus of D1′ remains on the use of a single class that includes both humans and solid objects (i.e. non-humans). Given that D1′ does not distinguish between humans and non-humans in the sequence of underwater images it follows that it also fails to disclose tracking humans-in-water candidates throughout the sequence of underwater images.
Additionally, I note that D1′ expressly discloses the use of a specific neural network structure notably a 2-5-2 structure. The 2-5-2 notation refers to a neural network structure having an input and output layer composed of 2 nodes and a single hidden layer having 5 nodes. In Dr Avraham’s view this neural network structure is different to that of the DNN utilised in the instant invention as claimed. This is explained at paragraphs [29]-[30] of the Avraham declaration where Dr Avraham opines that a deep neural network is an artificial neural network (ANN) with multiple layers between the input and output layers whereas the neural network of D1′ has a single hidden layer between the input and output layers. Based on this information I understand the difference between the DNN of the instant application and the neural network of D1′ to reside in the number of layers employed between the input and the output layers. Such a difference does not on its face seem to be significant to me, however the expert evidence points to there being a distinction between the two types of neural networks, and I note that the claims explicitly require a DNN.
Thus, on a review of D1′, it is clear that it fails to disclose the features of using a DNN to distinguish between humans and non-humans in a sequence of underwater images. D1′ also fails to disclose tracking humans-in-water candidates throughout the sequence of underwater images. Consequently, claim 1 is considered to be novel over the disclosure of D1′, it also follows that claims 17 and 25 are novel over the disclosure of D1 as they recite similar limitations to that claim 1.
Consideration of D2 and D3
As noted above the opponent does not seemingly wish to press D2 and D3 in relation to the ground of novelty. Nonetheless I will now address the disclosure of each of these documents in the interests of completeness.
The applicant advances that neither D2 nor D3 disclose the use of a DNN to identify humans-in-water candidates in the sequence of underwater images. As noted by the applicant at [48] of their written submissions D2 makes no mention of a DNN, a fact noted in the evidence of Dr Avraham. More specifically, at paragraph [44] of the Avraham declaration, Dr Avraham states that D2 is silent as to the use of a DNN and that D2 only refers to the use of Hidden Markov Models “HMMs”. Dr Avraham then clearly states that a HMM is not a DNN. Additionally, Dr Avraham notes at [45] of his declaration that the HMM of D2 is used to predict swimmer behaviour and not to differentiate between humans and non-humans in the image.
At paragraph [53] of the applicant’s written submissions it is noted that Dr Avraham considers the system of D3 to be of a similar ilk to that of D2 (per the Avraham declaration at [54]). Moreover at [57]-[59] of the Avraham declaration, Dr Avraham notes that D3 discloses the extraction of foreground blobs which have similar colours, and which differ from the water background. These blobs in Dr Avraham’s view can include any floating object, and not necessarily human candidates. As a consequence, D3 in Dr Avraham’s opinion fails to differentiate whether an object is human or non-human within the sequence of images.
I note that D2 is directed to use of a hidden Markov model for recognising specific behaviours of swimmers. The system of D2 takes a sequence of overhead images of swimmers in water (i.e. underwater images). The images are processed to produce a set of static swimmer-blobs. From there the system determines a posture type of a swimmer based on a set of feature vectors derived from static swimmer-blobs. As the behaviour of a swimmer for a particular stroke e.g. backstroke, breaststroke, freestyle etc. consists of repeated postures it is possible to generate a model representing various behaviour patterns. This model can then be translated into a simple code sequence (D2 page 2459 left hand column line 30 – page 2460 line 4). The code sequences are then utilised by the HMM to recognise particular swimmer behaviours within a sequence of images of swimmers within a swimming pool, the images being captured via one or more cameras positioned at various points above the swimming pool (see D2 at sections 3.3 and 3.4 and Figure 5 and 6 (reproduced below)).
D2 - Figure 6 HMM Recognition Process of a BehaviourD3 discloses a similar approach to that of D2 for determining swimmer behaviours. That is, D3 also utilises blob detection and classifications to model swimmer behaviours. The behaviour models are then utilised in conjunction with a HMM to recognise particular swimmer behaviours within sequences of live images of a pool that are captured by one or more cameras mounted above the pool (D3 Section II “A proposed DEWs system” Figures 1 (reproduced below) and 3, Section III-IV, Figures 8-14).
D3 – Fig. 1 Proposed DEWS system installed at an outdoor public swimming pool.
On inspection of D2 and D3, neither document expressly discloses the use of a DNN to distinguish between humans and non-humans in the sequence of images. Both D2 and D3 specifically focus on the determination of swimmer behaviours relating to the four main stroke types i.e. Butterfly, Backstroke, Breaststroke and Freestyle and other behaviours such as treading water, grasping lane rope, struggling etc. through blob analysis (D2 section 3, D3 section III). That is D2 and D3 look to classify the blob into one of the predetermined categories relating to swimmer behaviour, that is under the systems of D2 and D3 any blob matching the code book sequence for the given behaviour is classified as belonging to that class of behaviour. D2 and D3 are silent on whether any additional analysis of the blob is performed in order to determine if the object represented by the blob is human or non-human prior to classification.
D2 and D3 in my view fail to disclose the use of a DNN to distinguish humans from non-humans in a series of underwater images. I am therefore satisfied that claim 1 is novel over the disclosure of D2 and D3. With respect to claims 17 and 25 these claims recite similar limitations to that of claim 1 and it therefore follows that claims 17 and 25 are also novel over D2 and D3 for the reasons stated above.
Inventive step
The statutory basis for inventive step for this opposition is set out at s7(2) and s7(3) of the Act as it stood after commencement of the Raising the Bar Act, and is reproduced below:
(2) For the purposes of this Act, an invention is to be taken to involve an inventive step when compared with the prior art base unless the invention would have been obvious to a person skilled in the relevant art in the light of the common general knowledge as it existed (whether in or out of the patent area) before the priority date of the relevant claim, whether that knowledge is considered separately or together with the information mentioned in subsection (3).
(3) The information for the purposes of subsection (2) is:
(b) any single piece of prior art information; or
(b) a combination of any 2 or more pieces of prior art information that the skilled person mentioned in subsection (2) could, before the priority date of the relevant claim, be reasonably expected to have combined.
The question of obviousness has been extensively considered by the courts. In particular Aickin J stated in Wellcome Foundation Ltd. V VR Laboratories (Aust) Pty. Ltd. [1981] HCA 12 at [45]; [1981] HCA 12; (1981) 148 CLR 262 at 286 (“Wellcome Foundation”):
“The test is whether the hypothetical addressee faced with the same problem would have taken as a matter of routine whatever steps might have led from the prior art to the invention, whether they be the steps of the inventor or not.”
In Aktiebolaget Hassle v Alphapharm Pty Ltd [2002] HCA 59; 212 CLR 411 at [53], the High Court accepted the approach taken in Olin Mathieson Chemical Corporation v Biorex Laboratories Ltd [1970] RPC 157 at [187] where Graham J posed the reformulated Cripps question:
“Would the notional research group at the relevant date, in all the circumstances, ... directly be led as a matter of course to try [the claimed combination] in the expectation that it might well produce a [useful or better result]?” (emphasis in the original)
Moreover, the relevant authorities have also made it clear that the question of obviousness is a question of fact that is to be determined by evidence.
Assessment of inventive step
As previously noted, the opponent has not sought to offer any specific or declaratory evidence as to the state of the CGK in the art. This is a fact that was not lost on the applicant (per applicant’s written submissions at [66]). The applicant at [67] of their written submissions advanced that with or without CGK the opponent has failed to discharge their onus regarding lack of inventive step. I acknowledge that the opponent’s lack of engagement with these proceedings has significantly hampered their case. Nonetheless, the opponent has made a general assertion that that the claimed invention would be obvious in view of the disclosures of D1-D3 which requires due consideration.
Consideration of D1′
As I have noted above D1′ fails to disclose using a DNN to distinguish between humans and non-humans in a sequence of underwater images. As previously discussed, D1′ at page 24 lines 6-8 acknowledges that distinguishing people from fixed objects may lead to better classification. Such a statement could well be viewed as providing a teaching toward a process whereby humans are distinguished from non-humans within a sequence of images. D1′ then notes that to perform such a separation it would be necessary to define characteristics that best describes people, the process would also require additional analysis images where there is actually a human being in the picture. Such a statement could well be viewed as a suggestion that such processes distinguishing humans from solid object i.e. non- humans would be obvious to try. However, outside this passing reference D1′ provides no further technical detail as to how such processes could be implemented. Nor is there anything in the teaching of D1′ or the evidence before me to suggest that the use of a DNN as claimed to perform this task would be obvious to one of skill in the art.
In addition to the above I note that D1′ fails to teach or suggest tracking the human in water candidates through the sequence of images. D1′ suggests that it may well be possible to distinguish people from fixed objects, but it provides no teaching as to how this may be accomplished, let alone tracking a human candidate throughout the sequence of images. Moreover, given the lack of evidence as to the state of the CGK, it cannot be reasonably asserted that such a process formed part of the CGK in the art and consequently such a process would be obvious to the PSA in view of such a teaching.
There are also questions concerning whether the selection of a DNN over the artificial neural network (ANN) of D1′ is nothing more than a matter of routine design choice. Both are neural networks having a number of hidden layers between the input and output stages, although as I understand it based on the evidence of Dr Avraham, a DNN contains significantly more hidden layers than that of the ANN of D1′. While I appreciate that the DNN as claimed may be a more complex neural network than the ANN discussed in D1′, there is an argument that one type of neural network could well be substituted for another and that such a substitution would seem well within the remit of the PSA. In the present circumstances however, such an argument is not sustainable in light of the following. Firstly, I do not see it as being controversial to say the use of DNN’s was generally known. A simple Google search regarding DNNs shows that a vast amount of research on the topic has been conducted since the early 1990s, with DNNs seeing a rise in deployed in various endeavours such as speech recognition since the mid to late 2000s this being well before the priority date of the present application. However, I have no evidence that the use of DNN’s formed part of the CGK in the art as at the relevant date, particularly in the present context. Secondly, even if I am to take the view that DNN’s formed part of the CGK in the art, I have no evidence to support the view that proposed substitution could or would be made by the PSA. There is nothing in the teaching of D1′ nor in the evidence submitted in this case that would be lead a person of skill in the art as a matter of course to make the identified substitution.
Moreover, even if such a substitution were made there is nothing to suggest that the substitution would result in the claimed invention. As noted, D1′ does not expressly disclose distinguishing humans from non-humans; D1′ does acknowledge that such a process may well be possible but fails to provide any indication or further detail as to how this process could be performed. Consequently, there is nothing in the teaching of D1′ that would lead or otherwise motivate the PSA, having made the necessary substitution, to modify the relevant neural network to enable it to distinguish humans from non-humans. Based on the evidence before me it cannot be said that the substitution of the ANN of D1′ with a DNN as claimed would be obvious to one of skill in the art.
In light of the above the opponent has failed to establish that the use of a DNN to distinguish between humans and non-humans would be obvious to one of skill in the art. Nor has the opponent established that tracking the human in water candidates through the sequence of images would be obvious to the PSA in view of the teaching of D1′. I am therefore satisfied based on the information before me that the claimed invention is inventive over the disclosure of D1′.
Consideration of D2 and D3
The applicant in their written submissions advances that D2 and D3 teach away from the use of an underwater camera and performing detection of humans in a sequence of underwater images (applicant’s submissions at [47] and [53]). The applicant in their submissions also refers to various paragraphs in the Avraham declaration concerning Dr Avraham’s assessment of D2 and D3 notably the statements at paragraph [39] and [55] of the Avraham declaration.
At paragraph [39] of the Avraham declaration, Dr Avraham notes that D2 refers to the use of underwater cameras (D2 at 1. Introduction, third paragraph). Dr Avraham then states that the identified passage in D2 was taken from a paper entitled “A Vision-Based Approach to Early Detection of Drowning Incidents in Swimming Pools”, Wenmiao Lu et al. published in IEEE Transactions on Circuits and Systems for Video Technology, Vol. 14, no. 2, February 2004 (“the Lu publication”). Dr Avraham then notes that the Lu publication is highly critical of the usage of underwater cameras for the purposed of drowning detection. Dr Avraham then states (Avraham declaration at [40]) that it appears that D2 by way of its reference to the Lu publication discourages a PSA from using underwater cameras.
At paragraph [55] of the Avraham declaration, Dr Avraham also opines that D3 discourages the use of underwater cameras. More specifically in the first paragraph of Section II “A. Proposed DEWS System”, D3 considers underwater systems to be directed to the detection of later stages of drowning, whilst detecting a drowning event at its early stages where the victim is struggling at the water surface is an attractive alternative. It is this that then provides the motivation for D3 to explore the use of overhead cameras for this purpose. As a consequence, D3 teaches away from the use of underwater cameras.
While I appreciate the applicant’s position that neither D2 or D3, disclose or teach toward the use of an underwater camera, the argument is moot. As I have noted at [36] above neither the specification nor the claims are necessarily limited to the use of an underwater camera for the capture of the underwater images. Given the nature of the specification and claims, I had cause to construe the term underwater image broadly as simply meaning an image of something (i.e. an object or human) underwater. Such a construction being independent of the camera’s position with respect to the underwater object being imaged and is line with the plain meaning of the terms used in the compound phrase.
The issue of whether the claimed invention involves an inventive step over D2 or D3 does not in my view necessarily turn on aspects relating to the type of camera utilised or its positioning. As I have previously noted D2 and D3 fail to disclose a number of salient claim features. Notably neither D2 nor D3 expressly disclose the use of a DNN. Rather D2 and D3 favour the use of HMMs for the purposes of classifying particular swimmer behaviours. I do, however, note that D3 does make mention of a feed forward neural network and support vector machine, but this reference is in the form of comparative examples to highlight the superior performance of the proposed HMM solution used in conjunction with the researcher’s previously developed reduced model classifier (D3 page 206 right hand column, discussion of the results displayed in Table II).
As in the case of D1 discussed above, there is the argument that D2 and D3 teach the use of a particular machine learning algorithm that could well be substituted for another, such substitution being readily apparent to one of skill in the art. There seems to be support for this line of argument given the comparative examples of various forms of machine learning algorithms provided in D3. Again, given the lack of evidence concerning the state of the CGK this line of argument is unsustainable. There is nothing in the evidence before me to suggest that the selection of the DNN would be obvious to one of skill in the art. Moreover, there is nothing in D2 or D3 or the evidence presented in these proceedings that would motivate the PSA to make such a substitution.
Additionally, even if I were to consider that the PSA would make such a substitution, the substitution would still not address the fact that the systems of D2 or D3 fail to distinguish humans from non-human objects within the sequence of underwater images. D2 and D3 teach the use of blob extraction from time sequential images in order to assign the extracted blob a particular classification associated with a swimming behaviour; no determination is made in either system as to whether the blob is human or non-human. Moreover, there is nothing in the evidence presented before me to suggest that such processes formed part of the CGK as at the priority date.
Given the above discussion I am of the view that neither D2 nor D3 teach toward or suggest distinguishing humans from non-humans, as noted above both simply seek to assign the extracted blob to a classification associated with a swimming behaviour. Moreover, there is nothing in either document or in the evidence presented in this matter that would lead a PSA to implement such a process or otherwise suggest that such a process would be obvious to said PSA. I am therefore satisfied that the invention as defined in claims 1, 17 and 25, and therefore all the claims, involve an inventive step over the disclosures of D2 and D3 and the evidence before me.
S40(3) claims are fully supported.
S40(3) of the Act requires the claims be supported by matter disclosed in the specification. As was discussed in CSR Building Products Limited v United States Gypsum Company [2015] APO 72 at [115] in order to determine whether the requirements of support are satisfied it is necessary to:
·construe the claims to determine the scope of the invention as claimed,
·construe the description to determine the technical contribution to the art, and
·decide whether the claims are supported by the technical contribution to the art.
Such an approach was recently affirmed by the Federal Court in Merck Sharp & Dohme Corporation v Wyeth LLC (No 3) [2020] FCA 1477 at [546]-[547].
After consideration of the specification, I had reservations as to whether the independent claims 1, 17 and 25 were properly supported as required by s40(3). More specifically, the independent claims seemingly recite nothing more than general actions such as detecting and tracking. Outside the identification of a particular class of algorithm i.e. a deep learning neural network the claims do not recite any additional technical features for the performance of the recited functions.
On reading the specification it appears that the differentiating between humans and non-humans in the sequence of underwater images requires the use of postural features to associate an image with a particular subcategory/model of swimmer behaviour. Similarly, the specification discloses that the identification of a drowning risk is performed by determining movement within a threshold i.e. specific amount of motion or lack of motion within a temporal window. These aspects appear to be central to the contribution to the art made by the instant invention. Moreover, the above identified aspects do not appear to extend to a general principle of application across the whole scope of claims 1, 17 and 25 as currently presented. In other words, claims 1, 17 and 25 seemingly cover any and all means of differentiating between humans and non-humans in the sequence of underwater images and/or determining a drowning risk based on a sequence of images.
As previously noted, given my concerns regarding s40(3), I invited both parties to provide submissions on the support issues noted above. The applicant filed their submissions 29 November 2023. The submissions were accompanied by a set of proposed amendments to the claims. The opponent did not file any submissions in relation to the issue of support.
On a review of the applicant’s submissions of 29 November, I note that they have sought to directly address the question of whether use of postural features is central to the contribution made by the instant invention. I also note that the applicant’s arguments regarding the use of movement and temporal thresholds centre on the claims as proposed to be amended under s104 and not those under consideration in these proceedings.
Use of postural features
In their submissions of 29 November 2023, the applicant advanced that the trained deep learning algorithm is able to differentiate between humans and non-humans in an underwater image without requiring the use of postural features. The applicant then references the discussion in relation to Figures 5 and 6 at page 10 lines 1-9 and 23-31.
With respect to the discussion on Figure 5 the applicant notes that specification states that each underwater image is fed to the trained machine learning algorithm (i.e. the DNN). The machine learning algorithm then outputs humans-in-water candidates in the images, which comprise parts of the image that are detected as comprising a human. There is no mention in the discussion of Figure 5 as to the use of postural features to associate an image with a particular subcategory/model of swimmer behaviour.
The training of the DNN is then discussed with reference to Figure 6. Training of the algorithm includes using a training set with positive examples (including humans) and negative examples (without humans). The applicant again noted that training does not use postural features at all, and merely differentiates between positive examples and negative examples. Moreover, since the machine learning algorithm has not been trained based on postural features, it is clear that there is no requirement to use postural features to detect humans-in-water candidates.
To further support this point the applicant then sought to reference the discussion concerning Figures 8-12. The embodiments discussed in relation to Figures 8-12 use various models for different sub-categories (such as different human postures). However, these embodiments describe only optional features, which are not mandatory.
On a review of the discussion of Figure 5, it is apparent that detection of human in water candidates comprises a sequence of video frames (underwater images) being fed into the trained detector (the trained machine learning algorithm i.e. the DNN) the result of which is identification of humans-in-water candidates in the images, which comprise parts of the image that are detected as comprising a human. These are then tracked in order to detect drowning.
Looking at the discussion provided in relation to Figure 6 (reproduced below) and in particular the passages on page 10 line to page 11 line 5, it is clear that the algorithm is trained with positive and negative examples i.e. images containing and not containing humans respectively. The specification then goes onto explain that the result of this training stage is a model or a plurality of models for determining humans-in-water from images. These models may comprise one or more visual representations (such as a particular distribution of pixels) that indicate the fact that a human is present in the underwater image. The models are then used to detect humans-in-water candidates in the underwater images.
The following discussion concerning Figure 7 at page 12 lines 15-18 relates to associating the in-water candidates to additional subcategorises, for instance these additional subcategories may include additional information such as postural features. The specification then goes on to discuss Figures 8-12 which provide additional detail as to the use of postural features in associating the subcategories - that is postural features may be utilised to further refine the human in water models.
Given the applicant’s submissions and my review of the specification, it is apparent that the DNN of the instant invention is able to differentiate between humans and non-humans in an underwater image without requiring the use of postural features. I am inclined to agree with the applicant that the use of postural features is optional and does not need to be expressly recited in the independent claims in order to satisfy the requirements of s40(3).
Use of movement within a threshold
The applicant in their submissions states that it is their belief that the independent claims are clear, well defined and supported by the specification, however in order to expedite prosecution, the independent claims have been amended to address the issues raised in my correspondence of 26 October 2023. The applicant submits that the independent claims as now proposed do not cover any and all means of differentiating between humans and non-humans in the sequence of underwater images and/or determining a drowning risk based on a sequence of images. While it is appreciated that the proposed amendments seek to narrow the scope of the claims as accepted, it is the claims as accepted that are the subject of these proceedings. Consequently, the applicant’s commentary regarding the claims as proposed to be amended does not assist their case that the claims as they stand are properly supported.
Independent claims 1, 17 and 25 as accepted currently recite the features of tracking humans-in-water candidates throughout the sequence and detecting human drowning risk. The claims in question however do not recite any specific features for the performance of these processes. That is, the claims merely make a reference to a general process and do not place any meaningful limitation on how such processes are to be performed. By contrast the specification refers to tracking the human in water candidates via the use of a visual tracking algorithm or multi-object trackers to produce an active track which is then used to track objects that change in time and in location. For each active track an estimate of the amount of movement at a certain temporal window is estimated and in case of non-movement, or a movement smaller than a certain pre-defined amount for a pre-defined amount of time, that is associated with a drowning risk (specification at page 10 lines 11-16, page 17 line 33 – page 18 line 10).
Considering the above it is apparent that the claims lack a number of features which appear to be necessary to perform the invention. In the absence of any specific features relating to the performance of the tracking and detection stages the independent claims extend beyond the technical contribution. I therefore find that claims 1, 17 and 25 lack proper support. Dependent claims 2-14, 16 (when appended to claims 1-14) and 18-24 do not recite any additional limitations concerning the tracking of human in water candidates or detecting drowning risk based on motion. It therefore follows that claims 2-14, 16 (when appended to claims 1-14) and 18-24 also lack support.
Claim 15 introduces features relating to tracking the candidates throughout the sequence and detecting drowning risk based on a lack of motion. Claim 15 in my view is properly supported.
Although I have found that claims 1-14, 16 (in part) and 17-25 lack support, there would appear to be sufficient material in the specification that would enable the finding to be overcome by way suitable amendment. I also note that the amendments proposed by the applicant on 29 November 2023 appear to be directed towards remedying this finding.
Conclusion
The opponent has not established that claims 1, 17 and 25, being the independent claims, lack novelty in light of the disclosures of D1, D2 or D3, nor has the opponent established based on the evidence submitted in these proceedings that the invention as defined in claim 1, 17 and 25 lacks an inventive step. Since I have found independent claims 1, 17 and 25 to be both novel and inventive, it follows that their respective dependent claims are novel and inventive. Consequently, the opposition fails on all grounds pursued by the opponent.
I find that claims 1-14, 16 (in part) and 17-25 in their current form are not properly supported by the specification and cannot, in my view be granted in their present form. However, the amendments as proposed on 29 November 2023 appear to be directed towards remedying this finding. I direct that the amendments as proposed on 29 November 2023 be referred to an appropriate delegate for consideration. Upon allowance of the amendment, I will further consider whether the amendments do in fact overcome the deficiency as part of final determination of the opposition.
Although it may perhaps be superfluous, I also allow the Applicant two (2) months from the date of this decision to file any further amendments it deems appropriate. Final determination of the opposition will not occur prior to this two-month period and an incorporation of any further amendments that may be filed.
Costs
The opponent’s case is unsuccessful. The applicant submitted that the opponent should pay their costs. The normal approach is that costs should follow the event. However, in the present case, I have found that a ground of opposition not pursued by the opponent has been made out, namely that the claims as accepted do not comply with s40(3). In light of this I make no award as to costs, that is each party is to bear their own costs.
Neil Miller
Delegate of the Commissioner of Patents
0
14
4