Intuit Inc. - [2023] APO 48

IP AUSTRALIA
AUSTRALIAN PATENT OFFICE
Intuit Inc. [2023] APO 48

Patent Application: 2021202846
Title:Categorizing transaction records

Patent Applicant:                   Intuit Inc.
Delegate:  Dr N. R. Madsen – Deputy Commissioner of Patents
Decision Date:  18 September 2023
Hearing Date:  by written submissions filed 9 August 2023

Catchwords: PATENTS – section 45 – examiner’s objection – using multiple machine learning models in a vector space for categorisation of transactions – inventive step – manner of manufacture – examiner’s inventive step objection is sustainable – invention in substance appears to be directed to a mere scheme – opportunity to amend – remitted for further examination

Representation: Patent attorneys for the applicant: Davies Collison Cave Pty Ltd

IP AUSTRALIA
AUSTRALIAN PATENT OFFICE
Patent Application: 2021202846
Title:Categorizing transaction records

Patent Applicant: Intuit Inc.
Date of Decision: 18 September 2023
DECISION
The examiner’s objection that the claims lack an inventive step is sustainable. I consider independent claims 1, 8 and 15 are not inventive. The claims do not appear to be for a manner of manufacture, but I note that I am not making a formal finding in that regard.
I provide the applicant an opportunity to amend and provide six (6) months from the date of this decision to gain acceptance.
REASONS FOR DECISION
BACKGROUND

Intuit Inc. (“the applicant”) filed patent application 2021202846 on 5 May 2021 as a convention application associated to basic application US 17/217907, generating an earliest priority date of 30 March 2021. A first examination report issued on 5 May 2022 including objections under the grounds of manner of manufacture, clarity, support, and inventive step. The applicant filed a response with amendments and a second adverse report issued including the grounds of clarity, support and inventive step. Following this report, a further amendment and response was filed, after which time a third report issued containing only an inventive step objection. A hearing was requested on 5 April 2023.
The examination of the present application is governed by the Patents Act 1990 (“the Act”) as amended by the Intellectual Property Laws Amendment (Raising the Bar) Act 2012 (“the Raising the Bar Act”) as the application was filed after 15 April 2013. Thus, I must accept the application if satisfied on the balance of probabilities that the application complies with the Act. If I am not so satisfied, then I can refuse the application. Furthermore, the final date for acceptance of the application was 5 May 2023, however paragraph 13.4(1)(g) of the Patent Regulations 1991 is available to extend the time for gaining acceptance to 3 months (or longer if appropriate under sub-regulation 13.4(3)) from the date of the present decision.
SPECIFICATION
The specification begins by discussing that in order to properly assess the finances of an entity transactions need to be accurately categorised[1]. When transaction numbers are large, it helps to use a computer to automate the process of categorisation, with there being challenges when an entity is new and there are a limited number of categorised transactions to use as a guide to the analytical process[2].It is these disadvantages that the specification seeks to address through a useful alternative, this alternative being outlined in the summary of the invention in the form of consistory statements generally reflective of the claims[3].
[1] Specification as proposed to be amended at [0001]
[2] Specification as proposed to be amended at [0002]
[3] Specification as proposed to be amended at [0003] – [0005]
The specification provides a detailed description of the invention, initially specifically seeking to identify “technical” issues overcome by the invention[4].
One or more embodiments are directed to addressing a cold-start problem of an automated categorization engine categorizing transactions for new entities that have limited, if any, transactions categorized into a chart of accounts. Because of the lack of categorization, new entities have insufficient data to train a machine learning model to categorize transactions into accounts of a customized chart of accounts. Moreover, because of the customizations, millions of accounts exist creating a large classification problem (e.g., each account is a class in the classification problem).
One or more embodiments address the problems by converting the problem to a binary problem rather than a multi-class classification problem. Positive samples are positive associations between transactions and accounts, i.e., the actual transactions with the account to which the entity assigned the transaction. Negative samples are negative associations, i.e., actual transactions with the account to which the transaction not assigned. In this manner, transactions and accounts are paired features and have association scores defined. The benefit is that the number of unique accounts is not needed. Instead, interactions between transactions and accounts are learned explicitly.
From a more technical perspective, to overcome the above problems, one or more embodiments are directed to using a twin tower model to generate account recommendations for categorizing transactions. A twin tower model has two machine learning models (e.g., transaction model and accounting embedding model) that map to the same vector space. Namely, the vector output of both the transaction model and the accounting embedding model have the same number of dimensions and are trained such that the degree of similarity between the output vectors is representative of the level of match of the input. Thus, for the transaction model within the twin tower model, transaction information is used as input while, for the account embedding model within the twin tower model, account information is used as input.
[4] Specification as proposed to be amended at [0014] – [0016]
It is this later reference to a twin tower model that appears most relevant in the context of the presently claimed invention.
Discussion then turns to an embodiment[5]. Figure 1A is provided as a general representation of the system architecture employed to perform the present invention. The system (100) includes a series of devices including a user device which may be a desktop personal computer (PC), a smartphone, a tablet, etc. that is used by a user to access a webpage via a server through a user application. The user may be a person or set of people accessing the system of behalf of an entity such as a family, business organisation, nonprofit organisation. etc. A developer device may physically interact with the server in a similar way to the user device by controlling training and updating of machine learning models of the system. The repository (106) is referred to as any type of storage mechanism or device that includes functionality to store. On this I trust the reader would appreciate the different devices and means for storage of data in a computer environment that are available as standard in the art. A range of data is stored in the repository.
[5] Specification as proposed to be amended at [0021] – [0024]
After describing the context of Figure 1A the specification provides an overview of the machine learning models[6]. The specification discusses the general use of neural networks involving forward and back propagation, touching upon the uses of weights within layers and the fact that the neural networks may include one or more fully connected layers, convolutional neural network layers, recurrent neural network layers, or the like. To this point of the specification, it is plain to me that standard technological elements, and well-known and well understood neural network operation, are being discussed.
[6] Specification as proposed to be amended at [0036] – [0043]
Turning to the models themselves the specification points to the transaction model taking transaction information as input and encoding the transaction information with a pre-trained encoder, the output being a transaction vector. Similarly, the account embedding model encodes account information to generate an account vector. It is noted that the transaction model and account embedding model output vectors in the same vector space. As the specification puts it[7]:
Being in the same vector space, transaction vectors (output from the transaction model (132)) and account vectors (output from the account embedding model (144)) that are the same or similar in value will identify the same accounts while transaction vectors and account vectors that have different values will identify different accounts. In one embodiment, the transaction model (132) may be trained independently of other models and an account vector may be used as the training output for training the transaction model (132). Thus, directly using the vector space and values of the account vectors may be performed to train the transaction model (132) to generate transaction vectors with similar values.
[7] Specification as proposed to be amended at [0039]
The idea of vectors and vector spaces is well traversed in the field of mathematics and logic. A vector is a mathematical object having both magnitude and direction, and it is straight forward to understand this idea in the context of, for example, 3-D space, where a vector may be represented as a column having values for X, Y and Z co-ordinates. Any vectors that take this form exist in a 3-D vector space. Clearly the concept of the three spatial dimensions can be abstracted to consider sets of non-spatial parameters, or degrees of freedom, as constituting a vector space. With this in mind, a transaction vector or an account vector can be generated in a manner involving certain parameters/dimensions that can be compared like for like.
With this understanding, it is a simple step to appreciate that a match model can combine the outputs of the transaction model and account embedding model (i.e. mathematically compare the transaction vectors and account vectors) to create a match score. Specifically[8]:
In one or more embodiments, the match model (152) uses an element-wise product to combine the transaction vector output from the transaction model (132) and the account vector output from the account embedding model (144). The element-wise product may be an input to one of the multilayer perceptron (MLP) layers. The element-wise product is conceptually similar to a cosine similarity operator. The element-wise product encourages a behavior in which positively associated pairs of transactions and categories are embedded to similar locations, and negatively associated pairs are embedded far away from each other. The shared vector space for the transaction and account vectors further allows layers in each of the models to explore patterns and structure.
[8] Specification as proposed to be amended at [0041]
Having generated a match score between a transaction vector and account vectors, the highest match score may be used to identify and present to a user the most appropriate account for categorising transactions. In layman’s terms, I understand that the match model looks for patterns across the transaction and account vectors and in the situation of highest similarity, effectively suggests that transactions in question are associated with a particular known account. Schematic elements of methods of training transaction, account embedding, and match models are further described in Figures 1B and 1C, however I need not discuss these further at this stage.
Figure 2A depicts a process executing on a server to categorise transactions using machine learning models that is reflective of the claimed invention. It is largely self-explanatory.
In describing this figure, the specification[9] adds that a transaction record may be received from a client device through a web page and the method may provide recommendations for accounts which should be linked, also through a web page. In operation, the transaction model receives name data, name metadata, and transaction data and uses a multilayer neural network to generate a vector using this data. The match model will then select an appropriate account vector on the basis of the likelihood of a match between account vectors and the transaction vector[10].
[9] Specification as filed at [0045]
[10] Specification as filed at [0047]
Figure 2B provides details as to the steps involved in the generation of the transaction vector by a machine learning model, and Figure 2C provides details as to the steps involved in the combination of the account and transaction vectors in the matching model for generation of a score. They are presented similarly schematically to Figure 2A, each generally discussing logical/mathematic operations of the machine learning models in generating and processing the vectors to identify a match. Figure 2D is a similar schematic flow diagram identifying basic steps of training each of the models.
Figure 3 provides an example user interface that may be displayed on a client user device for operation of the claimed invention. This is not the focus of the presently claimed invention, so I need not discuss it further, though I note this appears to simply represent generic and well known user interface methods and techniques. Figures 4A and 4B provide for computing systems or groups of computing systems for performing the invention. They are clearly representative of standard, generic computer technology in the form of communicatively connected input, output and central devices, and a basic network representation. The latter part of the specification focuses heavily on a discussion of specific aspects of this well-known, well understood technology that I need not repeat.
It is plain to me that the present invention merely employs computer technology for its well-known and well understood functions, to implement the machine learning and processing methods. In this light, I find it difficult to see the substance of the invention as being some kind of technical adaptation or improvement to computer technology or solution to a technical problem as contrasted with a scheme for analysing transaction records and identifying a matching account identifier. I will discuss this matter further under the heading of manner of manufacture.
Claimed Invention
With their response to the second examination report, the applicant filed amendments to the claim dated 9 November 2022. In those amendments are 15 claims, 3 of which being independent. Independent claims 1, 8 and 15 as proposed to be amended are directed to methods and a system, with claim 8 being a system corresponding to method claim 1. Claims 1 and 15 are as follows:

1. A method comprising:
  receiving, by a server application, a transaction record;
encode the transaction record with a first machine learning model to obtain a transaction vector, wherein the transaction vector is in a same vector space as a plurality of account vectors;
           wherein the transaction vector is generated by training a transaction model to generate transaction vectors from the transaction records using an update function of the transaction model;
           selecting, by a second machine learning model executing in the server application, an account vector, from the plurality of account vectors, corresponding to the transaction vector, wherein the account vector is generated by training an account embedding model to generate account vectors from account identifiers using an update function of the account embedding model;
wherein selecting the account vector further comprises:
generating a set of match scores for a set of account vectors using the transaction vector and the set of account vectors; and
           selecting the account vector from a set of account vectors based on a match score for the account vector; and
           presenting an account identifier corresponding to the account vector for the transaction record.
15. A method comprising:
training a transaction model to generate a plurality of transaction vectors from a plurality of transaction records using an update function of the transaction model;
           training a match model to generate match scores from the plurality of transaction vectors and a plurality of account vectors using an update function of the match model;
           wherein the transaction vector is in a same vector space as a plurality of account vectors;
           generating a transaction vector, of the plurality of transaction vectors, with the transaction model from a transaction record of the plurality of transaction records; and
           generating a set of match scores for a set of account vectors, including the plurality of account vectors, using the match model.
I do not see any issues with allowability of the amendments and I do not see any particular challenges in understanding the features and scope of the invention of the independent claims as proposed to be amended on 9 November 2022.
With a focus on claim 15 initially, what is present at the highest level is a method where a transaction model is trained to generate transaction vectors from transaction records using some kind of updating function. This transaction model is subsequently used to generate a transaction vector from a transaction record. An updating function is also used to train a matching model to generate scores for matches between transaction and accounting vectors and as discussed in the specification, the transaction and accounting vectors are in the same vector space. The matching model then generates match scores for a set of account vectors. Fundamentally the invention of this claim accords closely with Figure 2A.
Claim 1 (and associated claim 8) are similar to claim 15 although additionally the claim includes the feature whereby account vectors are generated by training an account embedding model using an update function to generate account vectors from account identifiers. These claims thus include an additional/third machine learning exercise. Additionally, instead of simply generating match scores for account vectors, claims 1 and 8 select a particular account vector and present an account identifier corresponding to the relevant transaction record.
Dependent claims 2-7 and 9-14 further generally identify steps over and above the independent claims involving processing of particular data with machine learning models using various machine learning steps. Where necessary in this decision I will refer to those features more specifically.
INVENTIVE STEP
Legal Principles
The test for obviousness was provided by Justice Aicken in Wellcome Foundation Ltd v VR Laboratories (Aust) Pty Ltd[11] as follows:
“The test is whether the hypothetical addressee faced with the same problem would have taken as a matter of routine whatever steps might have led from the prior art to the invention, whether they be the steps of the inventor or not.”
[11] [1981] HCA 12 at [45]
The High Court in Aktiebolaget Hässle v Alphapharm Pty Ltd[12] approved this approach, in addition to that taken in Olin Mathieson Chemical Corporation v Biorex Laboratories Ltd[13]in which Graham J had posed the question:
“Would the notional research group at the relevant date in all the circumstances directly be led as a matter of course to try [the claimed invention] in the expectation that it might well produce a useful [desired result]?”
[12] [2002] HCA 59 at [51]-[53]
[13] [1970] RPC 157 at [187]
The usual approach to determining inventive step is the problem-solution approach. Once the problem has been formulated and the common general knowledge and the prior art base has been determined, the question of whether the claimed solution is obvious must be addressed.
Examiner’s Objection
In his third examination report the examiner objected under the ground of inventive step using both documents D1[14] and D5[15] as starting points for objection. In relation to document D1 the fundamental basis of objection in the report reads as follows:
D1 addresses the problem of determining a degree of similarity between ostensibly dissimilar objects, i.e., a transaction and an account, for the purposes of categorisation in circumstances where training data may be lacking. The solution provided involves the adaptation of a method commonly used in recommendation systems for calculating similarity (or ‘match score’), based on previously categorised data ([D1: p 9367, col 2, para 3, ‘Personalized transaction categorization assigns transaction to account according to maximum likelihood given company specific CoA accounts and the transactions that have been assigned to these CoA accounts so far’]).
The disclosure of D1 is therefore directed to towards solving the same problem, in the same field of application, as the currently claimed invention. Further, like the currently claimed invention, the proffered solution borrows techniques commonly applied in the context of recommender systems to determine a degree of similarity between a given transaction and an account, using data obtained from previous categorisations.”
[14] Lesner, C., et al., ‘Large scale personalized categorization of financial transactions’, 2019, Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33, No. 01, pages 9365-9372. (D1)
[15] ‘Alexander Backus: Categorizing financial transactions for personal finance. | PyData Eindhoven 2019’ Published on 30 December 2019 (D5)

Acknowledging the absence of disclosure of a number of features of the claims in document D1 the examiner states:
“In the current case, based on the teaching of D1 and common general knowledge of the particular efficacy of vector space embedding models (of the forms defined by the claims) in situations where training data is limited (as exemplified in prior art documents D2-D6), I consider that the skilled person would immediately and without the benefit of hindsight recognise the suitability of these models to the task of categorising transactions and be led to implement the claimed invention in the context of D1, in order to ameliorate its manifest deficiencies.”
Regarding document D5 the examiner states:
“…D5 discloses a method comprising:
receiving, by a server application, a transaction record; encode the transaction record with a first machine learning model to obtain a transaction vector, [D5: 20:58-21:04, ‘bag of words’ representation of a transaction record and ‘transaction embedding’; 28:18-28:22, pretraining of an embedding model]
wherein the transaction vector is in a same vector space as a plurality of account vectors; [D5: 21:00-21:24, embedding transaction and account data in a shared vector space via embedding functions (‘f’ for transactions and ‘g’ for categories/accounts)]
wherein the transaction vector is generated by training a transaction model to generate transaction vectors from the transaction records using an update function of the transaction model; [D5: 28:18-28:22, pretraining of an embedding model]
selecting, by a second machine learning model executing in the server application, an account vector, from the plurality of account vectors, corresponding to the transaction vector, [D5: 18:45-19:22, discusses the efficient learning of new mappings and a similarity function, based on previously obtained data]
wherein the account vectors are generated using an embedding model from account identifiers; [D5: 21:11-21:20, the ‘one-hot category indicator’ is a unique binary encoded account identifier, itself derived from a unique textual and/or numerical identifier]
wherein selecting the account vector further comprises:
generating a set of match scores for a set of account vectors using the transaction vector and the set of account vectors; and [D5: 21:55-22:46, cosine similarity and triplet loss function]
selecting the account vector from a set of account vectors based on a match score for the account vector; and [D5: 22:47-22:54, use the match scores to select the ‘most likely category’]
presenting an account identifier corresponding to the account vector for the transaction record. [D5: as there is a one-to-one correspondence between account vectors and identifiers, the selection of an account vector implies selection of the corresponding identifier]
D5 does not appear to disclose
wherein the account vector is generated by training an account embedding model using an update function of the account embedding model.
However, for the reasons given above, I do not consider that the inclusion of this feature furnishes the claim with an inventive step.”
These reasons being:
“I do not, however, consider that this feature can contribute to providing the claim with an inventive step. Supervised embedding techniques requiring training formed part of the common general knowledge of the art at the priority date. It was also common general knowledge at the priority date that such techniques provide performance advantages by extracting and encoding additional potentially useful information from the available training data. It would therefore have been an obvious choice, on the part of the skilled person, to train a supervised account embedding model in order encode additional relevant information into the embedded account vectors, wherever relevant training data was available.”
Applicant’s Submissions
The applicant’s submissions for the present matter are brief and refer to their earlier responses to examiner reports. They request that I essentially consider the matter on the papers, and I am content with this approach. To focus their arguments, the applicant has added that their main point of contention is that the relevant documents fail to describe an embodiment which resembles the claimed invention, and that they disagree what actions the person skilled in the art would be expected to perform. Regarding document D1 they argue that:
“…just because D1 may suggest that a technique exists does not mean that it would also be obvious to a person skilled in the art. Whilst we do not disagree that just because a number of possible solutions exist does not prevent a person skilled in the art from selecting a similar solution to the claimed invention, the Examiner must demonstrate why the skilled person will be directly led as a matter of course to at least try the solution employed by the claimed invention, with a reasonable expectation of success.”
And regarding document D5 they argue:
“…D5 only opens up the possibility that a person skilled in the art might, albeit with an impermissible degree of hindsight, and rejecting all other possible alternatives, produce something that might somewhat resemble the claimed invention.
The applicant notes that the question of obviousness must consider the selection of possibilities and rejection of others correctly identifying that the claimed combination of integers must be obvious. They point to Burley J’s observations in Vehicle Monitoring Systems Pty Ltd v SARB Management Group Pty Ltd[16] submitting:
“Burley J found that there is no lack of inventive step where ‘there is no single line of logic that leads to the invention as claimed such that it can be concluded that the combination arrived at was “very plain”, or obvious’ (at [212]). In that case, there was no ‘single line of logic’ even though the number of alternative solutions was ‘relatively limited’. We submit that there is no single line of logic leading from the cited art to make the particular combination of integers of the claimed invention 'very plain' or obvious. Similarly to the invention considered in VMS v SARB, a number of alternative solutions would have been possible, with a person skilled in the art having to ‘evaluate and then decide what direction to go in’”.
[16] [2020] FCA 408 at [211]
The applicant also discusses the common general knowledge identified by the examiner as present in a number of patent documents pointing out that generally, patent documents are not themselves an account of common general knowledge. I accept this general assertion.
Consideration
Document D1 + CGK
I first look to document D1. Document D1 is an academic paper written by employees of Intuit Inc. which focuses on solving problems of large-scale personalised categorisation of financial transactions. It begins by discussing similar problems to those addressed in the present application noting that a major part of financial accounting involves tracking and organising business transactions into categories. Such a task can quickly become time consuming and tedious, particularly when the range of possibilities of transactions and categories gets large. The paper discusses a system that uses machine learning to combine fragments of information from millions of users in a manner that allows accurate recommendation to specific account categories[17]. Document D1 poses that the categorisation of financial transactions can be viewed as a supervised classification problem[18] and that a well-studied problem that is adaptable to circumstances involving transactions and accounts is personalised tag recommendation where instead of personalised tag recommendations a personalised account is recommended[19].
[17] D1 at column 2 paragraph 1
[18] D1 at column 3 paragraph 3
[19] D1 at column 4 paragraph 6
With this background in mind, I turn to the discussion of the techniques used by document D1. Document D1 refers to an “Account Likelihood Ranking Model” which assigns transactions to an account according to maximum likelihood given company specific (CoA, Chart of Accounts) accounts and the transactions that have been assigned to the CoA’s so far. Where users have already categorised transactions, previous counterparties to CoA accounts assignments are used to guide further assignment. Where there is no previous assignment, a recommendation is made. To facilitate this, the system used arranges each transaction counterparty as an n-dimensional vector that is normalised with other counterparties[20]. Measures for similarity are calculated using various techniques[21].
[20] D1 at column 6 paragraphs 3 – 5
[21] D1 at columns 6 – 7
Document D1 discusses its experiments and the operation of a model. The document notes the building of an account likelihood ranking model whereby users have an opportunity to accept/correct how their transactions have been filed and their transactions are used to update the account likelihood ranking model next time it is rebuilt[22].
[22] D1 at column 10
Consistent with the findings of the examiner in the first examination report it appears clear to me that document D1 discloses the conversion of transaction records into transaction vectors and the matching of accounts to transactions with a machine learning model. On review of each of the examination reports and examination responses, it appears to me that the applicant agrees with this contention. What is not disclosed by document D1 in relation to the claimed invention is as follows (my emphasis):
·In relation to claim 15 while there is disclosure of the generation of transaction vectors and the use of an updatable matching model, there is no disclosure in document D1 of training a transaction model and generating the transaction vectors using an update function of a transaction model; and, arranging transaction vectors in a same vector space as a plurality of account vectors wherein the matching model generates matches between account and transaction vectors.
·In relation to claims 1 and 8, similarly to claim 15, there is no disclosure of the above features in addition to the failure of document D1 to disclose the account vector is generated by training an account embedding model to generate account vectors from account identifiers using an update function of the account embedding model.
To address the limited disclosure of document D1 the examiner throughout his reports essentially points out that the difference between the claimed invention and document D1 amounts to the implementation of the well-known tool of a twin tower machine learning recommendation methodology. He points to documents D2-D6 and argues that the skilled person would immediately and without the benefit of hindsight recognise the suitability of these models to the task of categorising transactions and be led to implement the claimed invention in the context of D1, in order to ameliorate its manifest deficiencies.
Turning to documents D2 – D6, I note the following.
·     D2[23] is a patent document that describes a machine learning system for semantic matching of queries and documents, whereby “two towers” of data in the same vector space to be matched are generated by updatable machine learning algorithms, and matching is performed by a machine learning matching model.
·     D3[24] is a paper from members of Carnegie Mellon University and Google studying embedding-based document retrieval models which involve “two-tower retrieval” in the form of two vector encodings in the same space, one each for queries and documents. Cosine similarity scoring functions are then used to identify similarities using the two towers of data.
·     D4[25] is a paper describing course recommendations technology at LinkedIn which uses two embedded machine learning models, one to embed data regarding the learner and the other to embed data relevant to courses. The technology involves an output layer of the model for generating recommendations. The model is trained iteratively.
·     D5[26] is a YouTube video which performs matching using account and transaction vectors using a matching model operating using cosine similarities. It notes that transaction records are encoded into the same vector space as account records.
·     D6[27] is a document discussing a general-purpose neural embedding model that can solve a wide variety of problems in the realm of classification and recommendation. It is a document written in 2017 by employees of Facebook AI Research. As discussed in the introduction it is noted that “StarSpace” embeds entities of different types into a vectorial embedding space, and in that common space compares them against each other. It learns to rank a set of entities, documents or objects given a query entity, document or object, where the query is not necessarily of the same type as the items in the set. Embedding data into the same vectorial space involves supervised machine learning models.
[23] US 20150074027 A1 see abstract, [0047], [0067] (D2)
[24] Chang, Wei-Cheng et al., ‘Pre-training Tasks for Embedding-based Large-scale Retrieval’, 2020, arXiv preprint arXiv:2002.03932v1 see abstract, page 1-2 (D3)
[25] Chaudhari, S et al., ‘A closer look at the AI behind course recommendations on LinkedIn Learning, Part 2’, [retrieved from internet on 03 May 2022]. < URL: > published on 19 October 2020 as per Wayback Machine see Figure 1. (D4)
[26] D5 at timestamps 21:04 - 22:47
[27] Wu, L., et al., 'StarSpace: Embed All The Things!', 2017, arXiv preprint arXiv:1709.03856v5 see Introduction and Model. (D6)
In response, the core elements of the applicant’s submissions appear as follows. Firstly, the applicant argues that just because a solution may be part of the common general knowledge, this is not sufficient. The Examiner must still demonstrate why the person skill in the art would also disregard all other possible solutions such that they would have been directly led as a matter of course to try claimed invention with a reasonable expectation of success. I interpret this along the lines that the person skilled in the art would not be directly led as a matter of routine to modify D1 with the knowledge of documents D2 – D4, whether these be representative of CGK or otherwise. Secondly, the applicant refers generally to documents D2 – D6 noting that they do not necessarily represent common general knowledge in the art. Specifically with reference to documents D2 – D4 they argue that these documents are directed towards different applications and solutions than D1.
First of all, I accept that the mere repeated identification of features in citations in itself falls short of a demonstration of common general knowledge. This point noted, there is logically a point at which the nature of disclosure of certain features across a collection of pieces of prior art becomes of probative value to the question of common general knowledge in the art. An examiner is not in a position to produce such evidence and therefore must formulate an opinion of what is common general knowledge on the basis of published information. The published information in documents D2 – D6 represents a corpus of information across a wide spectrum of media delivered to a wide audience[28]. They each point to the idea of a twin tower model in which certain data objects are mapped to the same vector space using parallel machine learning models for comparison using further machine learning techniques as being widely known. I am satisfied on balance that collectively the documents show at a high level that such a twin tower model is common general knowledge. It appears that the applicant does not take particular contention with this idea.
[28] See similar in Paul Andrew Cronk [2012] APO 28 at [31] to [33]
Specifically, to this point the applicant has argued that a solution’s presence in the art or as common general knowledge does not necessarily mean that one would be directly led from document D1 to the claimed invention. Turning to document D1, disclosed is a specific machine learning arrangement that does not appear to involve any machine learning driven embedding of data before matching is performed. Certain mathematical methods (including a “Kulczynski similarity index” and a “Jaccard index”) are used to perform matching of data, being matching of a counterparties vector with accounts, and it is not clear to me that these methods are of a nature such that the methodology discussed in D1 would as a matter of routine, be substituted by the elements of the twin tower modelling method. While twin tower recommendation systems do appear to be, on balance, common general knowledge, consistent with the submissions of the applicant, I consider D1 appears to teach away from application of such common general knowledge to the particular scenario described in document D1. More specifically, the data prepared and method of identifying similarities does not rely on data in the same vector space. I am not satisfied that vectorially embedded data in a twin tower model operation represents a mere routine variation to the methodology of D1.
I do not consider the objection to an inventive step whereby D1 is a starting point to be considered in the light of common general knowledge to be a sustainable objection.
An alternative starting point?
A key submission made by the applicant is with regard to documents D2, D3 and D4 as evidence of common general knowledge where they suggest that these citations are directed towards completely different applications and it is not apparent, from the disclosure of these documents, how a twin tower model may be suitable for categorising transactions. I accept that each of documents D2 – D4 do not make reference to transactions and accounts. D2 refers to semantic matching, D3 refers to queries and documents, and D4 refers to matching courses with learners on LinkedIn. I add that while no specific comments have been made in this regard by the applicant in relation to documents D5 and D6, document D5 refers to transactions and accounts and document D6 refers to a general purpose twin tower tool for content based recommendations and more generally, for ranking a set of entities, documents or objects given a query entity document or object is not necessarily the same type as the items in the set.[29]
[29] D6 at Introduction
Thus, prima facie it appears that at the very least, documents D5 and D6 may provide for a better starting point for consideration of inventive step than document D1. Document D5 is discussed further below and has been raised as the basis of an objection by the examiner. Document D6 appears to be a detailed document outlining the broad versatility of twin tower embedded classification/recommendation and matching models, that on its face, appears only to differ in disclosure from the claimed invention by specific mention of the nature of the data that is being classified. In other words, document D6 does not make mention of transactions and accounts but appears to suggest applicability generally to entity classification which would appear to involve any type of data entities possessing sufficient detail and relevance.
The specification outlines the problem relevant to the present application to be that there exists a cold-start problem of an automated categorization engine categorizing transactions for new entities that have limited, if any, transactions categorized into a chart of accounts. Because of the lack of categorization, new entities have insufficient data to train a machine learning model to categorize transactions into accounts of a customized chart of accounts. Moreover, because of the customizations, millions of accounts exist creating a large classification problem (e.g., each account is a class in the classification problem). It appears to me that document D6 may provide a general solution to this cold start problem, that is merely obvious to apply to any type of uncharacterised data where associations exist between input data and defined categories. The present specification does not appear to raise any specific challenges in the nature of transaction and account data that rise above any other general forms of data that may be matched using the twin tower model of D6. On this basis it appears to me that the examiner may consider whether an inventive step objection applies by for example using D6 as a starting point.

Briefly, I note that my observation above echoes the principles of a new use of a known contrivance under the doctrine of manner of manufacture. Relevant law is set down in Gadd & Mason v The Mayor etc. of Manchester[30], and was adopted by the High Court of Australia in Willmann v Petersen 2[31] CLR 1 at page 17. In the former case, Lindley LJ stated:
"1. A patent for the mere use of a known contrivance, without any additional ingenuity in overcoming fresh difficulties, is bad and cannot be supported. If the new use involved no ingenuity but is in manner and purpose analogous to the old use, although not quite the same, there is no invention; no manner of new manufacture within the meaning of the Statute of James. 2. On the other hand a patent for a new use of a known contrivance is good and can be supported if the new use involves practical difficulties which the patentee has been first to see and overcome by some ingenuity of his own."
[30] (1892) 9 RPC 516 at page 524
[31] CLR 1 at page 17
However as also noted in the Patent Examiner’s Manual of Practice and Procedure[32] such issues are in essence, akin to inventive step.
[32] >
These observations aside, noting my findings later in the decision I consider it appropriate that the application is returned to examination for further consideration.
D5 + CGK
Document D5 is a YouTube video first cited by the examiner in the second examination report. In that report, the examiner argued that all the features of the independent claims were disclosed, and those claims lacked novelty. The applicant’s response to this report was to add features to claims in the form of generating transaction and account vectors using trained models and an update function of those trained models, and to argue that there is no enabling disclosure in D5. Ultimately the applicant suggests that D5 does not describe a system capable of embedding entities of different types into the same vector space, and that D5 does not describe encoding transaction and account vectors using machine learning models. Beyond these points, I do not see any further substantive rebuttal to the examiner’s position in the third report.
Firstly, I turn to document D5 itself. The YouTube video is a presentation by a data scientist who has led development of an organised overview of an individual’s spending. He describes a money management app that automatically categorises transactions into spending categories (accounts) such as “expenses”, “housing”, “groceries”, etc. He discusses the use of machine learning to perform the task. In terms of specific function, key to the operation of the described method is the use of a model which uses similarity function mapping[33] to compare transactions with accounts, this being akin to the “second machine learning model” present in claim 1 and the “match model” that is present in claims 8 and 15. Also described is the encoding of transaction records with a first machine learning model to obtain a transaction vector, whereby pretraining is performed of an embedding model to form a “bag of words” representation of the transaction record[34]. These transaction vectors are discussed in the context of the “StarSpace” algorithm of Facebook (discussed above with respect to document D6) which relates to the embedding of different entities of different types into the same vector space for subsequent machine learnt matching[35]. It is in this context that transaction and category (account) vectors are disclosed as being embedded in the same vector space. Additionally, there is a disclosure where category (account) vectors are generated using an embedding model involving a “one-hot category indicator”[36]. In discussing the use of the “StarSpace” model, the presenter discusses that matches are generated for category and transaction vectors using a model involving cosine similarity and a triplet loss function[37] producing an identified most likely category[38].
[33] D5 at timestamp 18:45 – 19:22
[34] D5 at timestamps 20:58 – 21:04 and 28:18 – 28:22
[35] D5 at timestamp 21:00 – 21:24
[36] D5 at timestamp 21:11 – 21:20
[37] D5 at timestamp 21:55 – 22:46
[38] D5 at timestamp 22:47 – 22:54
In his third report, the examiner concedes that D5 does not disclose the training of the category embedding model using an update function but suggests that this would be an obvious step. I note that the applicant in their latest submissions does not specifically deal with this argument.
On the basis of the identification of features above, I consider it clear that there is a disclosure of relevant features of the independent claims. Notably, independent claim 15 does not include a feature related to generation of account embedding using an update function of an account embedding model. Thus, prima facie, document D5 in fact stands as a potential novelty disclosure for claim 15[39]. With this in mind I will first turn to the applicant’s proposition that document D5 is not an enabling disclosure as they argue it does not describe a system capable of embedding entities of different types into the same vector space.
[39] The examiner’s focus of objection in the third report with respect to D5 was independent claim 1, ultimately arguing that all the claims lacked an inventive step.
In the third examination report the examiner makes the following relevant comments.
“…the PSA would possess expertise in machine learning. The methods discussed in D5, including cosine similarity, bag-of-words representations, one hot-encoding representations, and embedding techniques in general, were all common general knowledge in the art at the priority date. Furthermore, at 23:02-24:05, D5 presents explicit code for performing a substantial portion of the disclosed method. The document therefore provides considerably more than a ‘general overview of how financial transactions can be categorised’, as the Applicant submits. If the Applicant disagrees with this assessment, it may expedite matters if they were to point out the particular deficiencies in the teaching of D5 that would prevent the skilled person from executing the disclosed categorisation model.”
I agree with the examiner. The applicant has not elaborated further to address these points. Furthermore, it is clear to me that the presenter in D5 describes developments in the context of transaction categorisation as a modification of the well described existing technology of “StarSpace” by Facebook, and also provides specific code enabling embedding of transaction records. The “StarSpace” technology is described in detail in the prior art as represented by document D6. I have no information before me to suggest that there would be any difficulty in applying the teaching of D5 such that it would not be enabling. Furthermore, given the broad and non-specific nature of the claims in terms of a system capable of embedding relevant entities, if I were to accept that there were any issues with enablement of the disclosure in D5, then it would appear likely that the present claims themselves would not be enabled across their full scope. Given the broad nature of the problems identified in the specification, none of which are directed to any specific challenges associated with the particular data being embedded, I proceed on the assumption that there would be no specific challenges.
Thus, I agree with the examiner as to the disclosure of document D5 a consequence of which being that claim 15 lacks an inventive step by way of its anticipation by document D5[40].
[40] I note it appears an objection to claim 15 lacking Novelty in accordance with s18(1)(b)(i) would also apply, although not raised by the examiner.
Turning to independent claims 1 and 8 which were the focus of the examiner’s most recent objection as I identified above, the examiner considers D5 differs from these claims by way of the account (category) vector being generated by the training of an account embedding model using an update function. Specifically, the examiner argues that:
“I do not, however, consider that this feature can contribute to providing the claim with an inventive step. Supervised embedding techniques requiring training formed part of the common general knowledge of the art at the priority date. It was also common general knowledge at the priority date that such techniques provide performance advantages by extracting and encoding additional potentially useful information from the available training data. It would therefore have been an obvious choice, on the part of the skilled person, to train a supervised account embedding model in order encode additional relevant information into the embedded account vectors, wherever relevant training data was available.”
The applicant has not engaged specifically with this objection. While document D5 does disclose an account embedding model in the form of a “one-hot category indicator”, the difference between document D5 and the claimed invention here is that there is no clearly disclosed training of this model using an update function.
I agree with the examiner. The prior art that has been cited by the examiner, including D2-D4 and D6, all point to machine learning techniques in the context of twin tower models that use trained machine learning models to embed different data into the same vector space for easy comparison (see references provided earlier for disclosure in D2, D3, D4, and D6). These documents generally point to the updatability/training of machine learning models as a basic function of supervised learning when embedding data to be compared into the same vector space using twin tower models. I consider the person skilled in the art, equipped with the relevant standard machine learning knowledge, would be directly led to this solution.
Therefore, I find that the examiner’s objection as to a lack of inventive step is maintainable. Notably, I consider that claim 15 is not inventive in light of D5 and agree that claims 1 and 8 lack an inventive step in light of D5 and CGK. I will not consider the dependent claims as the applicant has not engaged with any further features beyond the independent claims. The examiner has suggested that dependent claims merely add features that are common general knowledge in the art, and which therefore cannot contribute to providing an inventive step.
There is much detail in the specification, and it would appear that there is likely content that may overcome objections as to inventive step, potentially even in relation to a dependent claim. This is reason to provide a further opportunity for the applicant to file amendments for consideration in examination. As I note below, I also hold reservations as to the patentability of the claimed invention. I will discuss this further as an additional reason for providing an opportunity to amend and referring the matter back to examination.
MANNER OF MANUFACTURE
I will briefly address the issue of manner of manufacture noting that while initially raising objection under this ground at the first report stage, the examiner has considered the claims directed towards a manner of manufacture in the second and third reports. I refer to summaries of principles present in recent Patent Office decisions[41]. Suffice to say, one must consider the substance of the invention[42]:
[41] Mastercard International Incorporated [2023] APO 42, Block, Inc [2023] APO 34
[42] D’Arcy v Myriad Genetic Inc. [2015] HCA 35 at [144]
“Whatever words have been used, the matter must be looked at as one of substance and effect must be given to the true nature of the claim.”
I also note that relevant principles to apply are well articulated in Rokt 1[43].
[43] Rokt Pte Ltd v Commissioner of Patents [2018] FCA 1988 (“Rokt 1”) at [189]
“17.1 The Court must decide, as matter of substance not form, whether the claimed invention is the proper subject-matter for a patent: RPL Central at [99]; Research Affiliates at [106], [117].
17.2 This requires consideration of both the claims of the Application and the invention described in the body of the specification: RPL Central at [114].
17.3 The assessment is not done mechanically. There are no precise guidelines or mathematical formula. It is ‘a question of understanding what has been the work of, the output of, and the result of, human ingenuity’and then applying the developed principles: Research Affiliates at [116]. See further RPL Central at [112]:
Recognising that the claims are to a method and system comprising a combination of integers, it is necessary to understand where the inventiveness or ingenuity is said to lie ...
17.4 One well-settled principle is that a distinction exists between a technological innovation and a business innovation. A technological innovation is patentable. A business innovation is not: Research Affiliates at [94]; RPL Central at [100]. Consequently, a business method or scheme is not, per se, a proper subject for letters patent: RPL Central at [96]. Nor are abstract ideas, mere intellectual information or mere directions for use patentable: Research Affiliates at [101]; RPL Central at [100].
17.5 A computerised business method or scheme can, in some cases, be patentable. However, ‘[w]here the claimed invention is to a computerised business method, the invention must lie in that computerisation’: RPL Central at [96] (emphasis added). This requires ‘some ingenuity in the way in which the computer is used’: RPL Central at [104]. It is not a patentable invention ‘to simply “put” a business method “into” a computer to implement the business method using the computer for its well-known and understood functions’: RPL Central at [96]. In other words, if the ingenuity lies in the business method or scheme alone, the invention will not be patentable despite the computer-implementation.
17.6 Thus, a claimed invention must be examined to ascertain whether it is in substance a scheme or plan, or whether it can broadly be described as an improvement in computer technology: RPL Central at [96]. Contrary to [the applicant’s submissions at [49]], this is a binary distinction: the invention is either an unpatentable scheme or plan, or it is a patentable improvement in computer technology. In conducting the analysis, it is useful to:
17.6.1 ascertain whether the contribution to the claimed invention is technical in nature: RPL Central at [99], Research Affiliates at [114];
17.6.2 consider whether the invention solves a ‘technical’ problem within the computer or outside the computer: RPL Central at [99], Research Affiliates at [103];
17.6.3 consider whether the invention results in an improvement in the functioning of the computer, irrespective of the data being processed: RPL Central at [99], Research Affiliates at [118];
17.6.4 consider whether the invention requires merely ‘generic computer implementation’, as distinct from steps which are ‘foreign’ to the normal use of computers: RPL Central at [99], [102]; Research Affiliates at [101]; and
17.6.5 consider whether the computer is merely the intermediary, configured to carry out the method using program code for performing the method, but adding nothing to the substance of the idea: RPL Central at [99].”
The claimed invention clearly employs standard computer technology in the form of a server to implement the method, the computer technology being well depicted in Figures 4A and 4B and the method being exemplified in Figure 2A. In this sense the computer technology serves as an intermediary to implement the method. I consider that the substance of the invention before me is the computer implementation of an algorithmic model involving encoded transaction and account vectors that exist in the same vector space, these transaction and account vectors being generated by an updatable machine learning algorithm, wherein the vectors are matched using a matching model for identification of a relevant account using match scores.
The exercise of creating vectors using machine learning and matching vectors similarly using machine learning appears to me to be a mathematical exercise of comparison of data that does not appear to address a technical problem within or outside a computer. Instead, it appears to me to be a problem of abstract algorithmic analysis of data. The independent claims simply identify the use of “machine learning” models in the most generic of senses to represent any suitable mathematical process applied to relevant data. Such operations do not appear foreign to the normal use of computers.
Turning to the problems identified by the specification it is suggested that there is a “cold start” problem in the presence of limited data for new entities. The specification discusses the conversion of the problem to a binary problem rather than a multi-class classification problem. In their response to the first report the applicant argued that the invention solves a technical problem in the form of addressing this cold start issue by means of the twin tower model. They also pointed to analogy with the invention in IBM v Commissioner of Patents[44].
[44] IBM v Commissioner of Patents (1991) 22 FCR 218
I do not see the present invention addressing a technical problem. The problem is one of lack of data and effective logical similarity comparison between pieces of particular data. No limitations of computer technology appear to be addressed, and instead the substance of the invention appears to be a mathematical data analysis method (although a reasonably complex computer implemented one) for matching transactions with account identifiers. I do not see this as a technical solution to a technical problem. Furthermore, to the extent IBM v Commissioner of Patents is identified, I understand the invention in that matter was directed to steps that were foreign to the normal use of computers being the use of non-floating point (or integer) arithmetic in computer memory to generate a curve. The present invention merely employs “machine learning”, this not being foreign to the normal use of computers.
As such, it appears that the claimed invention is not directed to a not a manner of manufacture. The matter is referred back to the examination section for further consideration.
CONCLUSION
I consider objections of the examiner are sustainable. I consider on balance all independent claims are not inventive. I hold reservations as to whether the claimed invention is directed towards a manner of manufacture.
In accordance with paragraph 13.4(1)(g) and sub-regulation 13.4(3) of the Patent Regulations 1991 I provide six (6) months from the date of this decision for the applicant to gain acceptance.
Dr N. R. Madsen
Deputy Commissioner of Patents