Accenture Global Solutions Limited
[2022] APO 19
•25 March 2022
IP AUSTRALIA
AUSTRALIAN PATENT OFFICE
Accenture Global Solutions Limited [2022] APO 19
Patent Application: 2019203168
Title:METADATA-DRIVEN PROGRAM CODE GENERATION FOR CLINICAL DATA ANALYSIS
Patent Applicant: Accenture Global Solutions Limited
Delegate:Felix White
Decision Date: 25 March 2022
Hearing Date: Written submissions filed on 13 December 2021
Catchwords: PATENTS – section 45 – examiner’s objections – analysis of clinical trial data - whether invention is manner of manufacture – consideration of substance of the invention – substance is procedural generation of software code or preparation of software code in advance - no patentable subject matter identified in specification – application refused
Representation: Patent attorney for the applicant: Murray Trento & Associates
IP AUSTRALIA
AUSTRALIAN PATENT OFFICE
Patent Application: 2019203168
Title:METADATA-DRIVEN PROGRAM CODE GENERATION FOR CLINICAL DATA ANALYSIS
Patent Applicant: Accenture Global Solutions Limited
Date of Decision: 25 March 2022
DECISION
The claimed invention is not for a patentable manner of manufacture, and there is no prospect of this being overcome by amendment. Subject to appeal, I refuse the application.
REASONS FOR DECISION
Background
Australian patent application 2019203168 (“The application”) was filed on 6 May 2019 in the name of Accenture Global Solutions Limited (“The Applicant”), as a divisional application from 2017179795 (“The parent”). The parent had lapsed due to failing to gain acceptance by the final date of 17 May 2019, after receiving three adverse examination reports.
The applicant was directed to request examination on 21 May 2019 and examination was requested on 3 July 2019. The applicant filed amendments in anticipation of an examiner’s report on 4 August 2020, along with submissions relating to the then-recent Federal Court decisions Aristocrat Technologies Australia Pty Limited v Commissioner of Patents [2020] FCA 778[1] and Commissioner of Patents v Rokt Pte Ltd [2020] FCAFC 86.[2]
[1] At the time of writing this decision was overturned on appeal to the Full Federal Court and is pending an appeal to the High Court.
[2] An application for special leave to appeal this decision to the High Court was dismissed.
The application received three adverse examination reports dated 13 August 2020, 10 March 2021, and 30 June 2021. In each of these examination reports, the only ground of objection was lack of manner of manufacture.
The Applicant requested to be heard on 10 August 2021. The hearing was set by written submissions.
I wrote to the Applicant on 11 October 2021 providing a preliminary determination of the substance of the claimed invention and invited the Applicant to comment on my proposed construction. I will reproduce the text of this letter here - as will become evident, my preliminary determination of the substance of the claimed invention has not substantially changed.
The approach I intend to adopt (consistent with D'Arcy v Myriad Genetics Inc [2015] HCA 35 (Myriad), Commissioner of Patents v RPL Central Pty Ltd [2015] FCAFC 177 (RPL), Research Affiliates LLC v Commissioner of Patents [2014] FCAFC 150 and endorsed in the patent manual of practice and procedure will be to
·Construe the claim;
·Identify the substance of the claimed invention.
·Determine whether the substance of the claim lies within established principles of what does not constitute a patentable invention
This is fundamentally a question of construction. The following characterisations of the substance of the invention have been made by the examining delegates during the prosecution of either the present application or its parent 2017279795.
1st report in 2017279795 “a scheme of generating program code based on metadata to analyse clinical data and create artifacts (for example, reports)”
3rd report in 2017279795 “the contribution of the current invention lies in the automation (i.e., generating/ retrieving program code) to analyse clinical data and create artifact”
2nd report in this application: “the abstract set of steps, the scheme that utilises generic computer functionalities (for instance, metadata) for ‘generating’ program code to perform a predetermined operation on an input data set to create output ‘artifacts’ to be communicated for presentation through a user interface”
In the interests of providing the best possible opportunity for the Applicant to engage with this threshold question, I am providing my preliminary construction of claim 1 below (the other independent claims having substantially identical features) and my preliminary conclusion as to the substance of the claim, and I would invite the Applicant to comment on this in their submissions.
Key features are underlined, and my commentary on each is italicised. Bracketed references are to the instant description.
My understanding of claim 1 is that it is directed to a method carried out on a metadata structure which is composed of
A clinical trial data set. [0030] refers to checking consistency of blood sample results, [0032] refers to regulatory required data formats and [0034] mentions safety and efficacy data of treatment vs placebo control groups.
The metadata is “hierarchical” insofar as it relates to two or more of global, project or study level data. Hierarchy is mentioned at [0038] as a property of metadata 106A desired output format or “artifact” which could be e.g. a table or report
A desired data transformation “operation”
The “operation” includes a step of data checking. Data checking is mentioned at [0030] as identifying erroneous or anomalous data. E.g. a male identified as being pregnant or a blood sample with no matching blood results report. The output of data checking can be to remove the erroneous data or simply to include an error report in the final “artifact”
The “operation” involves conversion of data through two different data formats. From [0031] and [0032] it appears that the two formats are to comply with different regulatory reporting standards.
The metadata further includes information on how to carry out the operation, which can be in one of three forms:
Instructions for generating the operation code. This appears to be discussed at [0037] – where the code generation is deterministic, the code can be generated more efficiently than traditional systems where code is written manually. It seems implicit that these instructions would need some level of prior design to ensure that appropriate code is generated; OR
A reference to code stored in a repository; OR
The code for carrying out the operation itself.The claim then specifies that the operation is executed, such that an artifact such as a table or report is generated from the clinical data set for presentation to a user.
The stated advantage of this invention appears to be (from [0002]) that traditional methods of analysing clinical trial data involve writing bespoke software to analyse the data from each trial, which is an inefficient process.
The substance of the claimed invention, then, appears to me as preparing the analysis code in advance so that it can be stored as metadata and executed on the clinical trial data.
If this is the case, the question to be addressed is whether the concept of preparing the code in advance falls within the field of patentable inventions.
Alternatively, the Applicant may wish to present their own submissions as to the substance of the claimed invention.
The Applicant’s written submissions were filed on 13 December 2021.
The examination of the present application is governed by the Patents Act 1990 as amended by the Intellectual Property Laws Amendment (Raising the Bar) Act 2012 as the application was filed after 15 April 2013. Thus, the standard of proof that applies in the present case is the balance of probabilities (subsection 49(1)). I must accept the application if satisfied on the balance of probabilities that the application complies with the Act. If I am not so satisfied, then I can refuse the application.
The final date for acceptance of the application was 10 July 2021, however paragraph 13.4(1)(g) of the Patent Regulations 1991 is available to extend the time for gaining acceptance to 3 months (or longer if appropriate under sub-regulation 13.4(3)) from the date of the present decision
Field of the invention
The application is entitled “Metadata-driven program code generation for clinical data analysis”.
10.The specification sets out at [0002] that clinical trials can generate a large amount of data to be analyzed; “traditionally, software is manually written to analyze the data” of each trial, and it is not often possible to reuse standard reports from study to study “given that the studies may be sufficiently different in the data being analyzed”. As such the specification sets out that manually coding clinical data analysis is an inefficient process using “a large amount of computing resources and computer programmer time”.
11.Before construing the specification, I note the comments of Middleton J in Eli Lilly and Company Limited v Apotex Pty Ltd:[3]
[3][2013] FCA 214;100 IPR 451 at [139].
“It is well settled that the Court should, from the outset, approach the task of patent construction with a generous measure of common sense. The Court must place itself in the position of a person skilled in the relevant art, being the subject matter of the patent. From this perspective, the patent is to be read as a whole, in the context of the specification and in light of the prevailing common general knowledge and state of the relevant art at the priority date.”
12.Although there is no evidence on file as to the person skilled in the art, given the field of the invention it seems that said hypothetical person would be a computer programmer or data analyst.
The Specification
13.The application under examination consists of the specification as filed, as proposed to be amended under s104 up to and including amendment item 4, filed on 3 June 2021.
14.The body of the description has 70 numbered paragraphs. The first paragraph sets out the application’s divisional status. Paragraph [0002], as mentioned above, sets out the background of the invention in relation to inefficiencies involved in manual coding of clinical trial analysis software.
15.Paragraphs [0003]-[0011] repeat the disclosure of the claims, which I will turn to presently. Paragraphs [0011]-[0016] are a brief description of the drawings which I will also turn to presently.
16.The remainder of the description is entitled “Detailed Description of the Embodiment(s) of the Invention”.
17.This is described with reference to various “implementations” which do not appear to ever dip below a very high level of generality. Paragraph [0017] sets out that metadata-driven code generation is used to generate software for analyzing clinical data. The metadata may include “instructions that describe the data” and also describe what “artifacts” are to be output. The metadata may describe operations to be performed on the data. The artifacts are the result of the analysis which may be tables, figures or listings.
18.I would note at this point that although “Metadata” has a prominent position in the title of the application, the specification does not specifically define what is meant by this term. The common meaning of “metadata” is “data about data” such as size, date, and origin of files. However “metadata” is also commonly understood to be some form of addendum to relevant data in order to provide additional information for/about/to that data. In the present specification it appears to relate to instructions for processing data, or indeed executable code. For the purposes of construing the specification and claims, I consider that “metadata” can be construed to be anything that is not clinical trial data.
19.To use a rather crude analogy, if the clinical data constitutes the contents of a package, say a packet of instant noodles, then the metadata could be considered as the writing on the package: this could be information about the source or the nutritional value of the noodles, or it could be preparation instructions.
20.Paragraph [0018] sets out operations to be performed on clinical trial data, including validation, transformation for analysis and analysis.
21.Paragraph [0019] sets out that the metadata may include references to code in a code library (which may be self-configurable code along with parameters for configuring the code); or it may include a full copy of the program code. This paragraph sets out a specific example in which a first set of metadata may be used to generate code to analyse a first set of data and generate a first set of artifacts (a first trial) and subsequently a second set of metadata may be used to generate code to analyse a second set of data and generate a second set of artifacts (a second trial). The description states that this “may facilitate the coding and/or recoding process” e.g. by adjusting earlier program code. However I fail to see how this would be different from performing manual coding twice. I note that these embodiments correspond to the second and third definitions of the instructions in the metadata in the claims.
22.Paragraph [0020] sets out that a code generation engine may generate program code based on instructions in the metadata, and sets out a couple of different output data formats.
23.Paragraph [0021] contains statements of asserted advantages of the invention, e.g. “program code generated by the engine may be of higher quality, higher performing, and/or with fewer bugs than code written manually”. Other examples are that the code would be more consistent across instances (in response to the same metadata) and that validating metadata is easier than validating a whole new software program.
24.Paragraph [0022] describes advantages of the invention in a double programming scenario, which is said to be a conventional practice where two groups of programmers independently perform the same task to ensure it produces the same output. It is said that replacing one of the sets of programmers with procedurally generated code may “lead to the generation of higher quality code” as well as being a more “efficient and cost-effective allocation of resources”.
25.Paragraph [0023] sets out that to generate different artifacts only relatively small changes may be required to the metadata rather than “a full rewrite of the analysis program” which could provide 30-50% cost savings.
26.Paragraphs [0024]-[0027] provide a description of Figure 1, which is a self-explanatory schematic of how a code generation engine would access metadata and a code repository in order to produce code.
27.Paragraphs [0028]-[0038] provide a description of Figure 2, which is a self-explanatory flowchart setting out steps for analysis of clinical data.
28.Paragraphs [0039]-[0043] provide a description of Figure 3, which is a self-explanatory flowchart of generating program code based on metadata and executing it.
29.Paragraphs [0044]-[0046] provide a description of Figure 4, which sets out three possible choices for generating program code based on metadata: procedurally generated, retrieved from a repository, or stored inline in the metadata itself.
30.Paragraphs [0047]- [0048] and Tables 1-2 are said to list example metadata. These tables appear to show names of macro functions, variables, and some simple logical functions. There is no other context for this information and there is no evidence as to whether these functions and variables are well known in the art or instead refer to the Applicant’s own internal nomenclature.
31.Paragraphs [0049]-[0058] provide a description of Figure 5 which is a self-explanatory depiction of a computer system with input, output, storage and communication functionality.
32.Paragraphs [0059]-[0065] provide generic descriptions of execution of software on processors and display of output.
33.Paragraphs [0066]-[0070] are conventional disclaimers that are often found in Australian patent specifications with the intent of making the aforementioned disclosure as non-limiting as possible.
34.The specification ends with 12 claims, of which claims 1, 6 and 11 are independent. Claim 1 is directed to a computer-implemented method for analyzing clinical trial data. Claim 6 is directed to a [computerised] system for analyzing clinical trial data. Claim 11 is directed to computer-readable storage media storing executable instructions for causing a processor to analyze clinical trial data. The features of claims 6 and 11 correspond to the features of claim 1. I am satisfied that in substance they are identical and hence should stand and fall together, and the Applicant did not differentiate the independent claims in their written submissions.
35.The claims can be found as annex A to this decision. Notably, claim 1 is one and a half pages in length so it does not seem to be helpful to repeat it inline here. My understanding of the key features of the claim has already been summarised supra.
Manner of Manufacture
36.In Grzegorz Malewicz [2022] APO 11 (“Malewicz”) the Deputy Commissioner provided a helpful summary of the current understanding of the law regarding manner of manufacture of computer implemented inventions (pargraphs 47-51). I will not quote it in full in the interests of brevity, but the key concepts that bear repeating are:
… the High Court has stated in National Research Development Corporation v Commissioner of Patents (“NRDC”), [1959] HCA 67, (1959) 102 CLR 252 that for patentability to be found there must be “some advantage which is material, in the sense that the process belongs to a useful art as distinct from a fine art ...- that its value to the country is in the field of economic endeavour”. However, it is clear that this is not exhaustive of the considerations with the High Court in D’Arcy v Myriad Genetics Inc (“Myriad”), [2015] HCA 35, noting at [23] that a “case-by-case” methodology is required, and at [144] that the task is one to be approached as a matter of substance:
“Whatever words have been used, the matter must be looked at as one of substance and effect must be given to the true nature of the claim.”
…
To summarise the legal principles developed for computer implemented inventions it appears reasonable to suggest that a method that is otherwise unpatentable, that is implemented in a computer, will not be patentable unless some improvement in computer technology is present. To this extent, an invention may be usefully implemented in a computer and achieve a useful result, but as noted in Research Affiliates at [114], such an invention may not be patentable.
The Examiner’s Objection
37.The last examination report comes at the end of a series of six adverse examination reports spanning the present application and the parent application, during which it appears that similar issues have been traversed at length. The examiner’s position can be summarised as:
·The computer implementation steps described in the description are generic.
·The claims do not provide a technical solution to a technical problem.
·This leads to the conclusion that the substance of the invention had been determined to be a scheme or a business method.
·The fact pattern of the present application is similar to Encompass Corporation Pty Ltd v InfoTrack Pty Ltd [2019] FCAFC 161 but is not similar to eBay Inc. [2020] APO 49 or Jagwood Pty Ltd [2020] APO 38.
38.The examination report concludes as follows: “The claimed invention, as a matter of substance, is a scheme, a set of steps or instructions that a person skilled in the art can programme a generic computer to perform. Therefore, the claimed invention, including all features of all dependent claims, as a matter of substance, does not define subject matter suitable for a patent.”
Submissions
39.The Applicant’s submissions first note that the examiner has apparently conceded that the claims are novel, inventive and satisfy the requirements of section 40. As a statement of the history of this case, this is correct, although it is not unheard of for hearing officers to refer applications back to examination sections for further searching and/or reconsideration of those grounds. I would only note that this decision only considers the ground of patentable subject matter and in view of the outcome, consideration of other grounds is moot.
40.The Applicant’s submissions then point to paragraph [0019] of the description, which refers to an advantage in terms of less time and/or computing resources due to the inclusion in the metadata of inline code, references to a code repository, or procedurally generated code.
41.The Applicant then points to paragraph [0021] which states that, as a result of the invention, the generated code may be consistent in response to receiving the same metadata and therefore may be higher quality.
42.The Applicant then quotes the entirety of paragraph [0022] which discusses the double programming scenario. The Applicant emphasises that traditional [manual] double programming may be costly and may generate sub-optimal code.
43.The Applicant then asserts that addressing the problem of sub-optimal code generation is a technical problem.
44.I note that the Applicant’s submissions focus on the role of the metadata in generating the program code. This is consistent with my impression that the properties of the clinical trial data (to the extent they are defined) are not central to the substance of the invention.
45.The Applicant provided some commentary on the recent Ebay, Jagwood and ANT[4] decisions, in which the respective Delegates found that the claimed inventions provided technical problems to technical solutions in those particular cases.
[4] Advanced New Technologies Co., Ltd. [2021] APO 29
46.The Applicant then submits that it is not necessary for there to be a technical improvement in a computer for a claim to avoid being a scheme, and I concur with this. The determination is one based the substance of the invention as a whole.
47.This now brings me to the substance of the invention. The Applicant disputes, as they were invited to do, my earlier characterisation of the substance as being preparing the analysis code in advance. The Applicant provides an alternative characterisation of the substance as:
“the facilitation of the preparation of program code according to metadata such that the invention comprises a self-improving method that uses metadata from one clinical trial for another clinical trial with the generation of final program code including generated or inserted program code thereby avoiding the need to generate bespoke software”
48.I cannot agree with this characterisation. The Applicant has provided no indication from whence in the claims it could be derived, and I cannot find any reference to such a self-improving method in the claims on file. Even to the extent that reusing earlier trial code is touched on at [0019] and [0023] of the description, there is no description of “self-improvement” let alone how such self-improvement would be accomplished.
Consideration
49.Taking all of the above into consideration, I am minded to change my initial characterisation of the substance of the invention slightly.
50.In the present invention, the metadata can produce the program code in one of three ways:
·As instructions for procedurally generating code
·As an address to a code repository
·By being the code itself
51.Depending on which type of metadata is being used, I consider the substance of the invention to be either producing procedurally generated code to analyse clinical trial data, simply preparing the code in advance, or some combination of the two.
52.Now I must consider whether either of these constitute patentable subject matter. Consistent with recent[5] practice I will consider factors set out by the Commissioner’s delegate in the original Aristocrat[6] decision.
[5] See e.g. Malewicz; Accenture Global Solutions Limited [2022] APO 8; Apple Inc. [2022] APO 15; Forethought Pty Ltd [2022] APO 16
[6] Aristocrat Technologies Australia Pty Limited [2016] APO 49 - see [35]
Does the invention result in improvement in the functioning of the computer, irrespective of the data being processed? Can the invention be broadly described as an improvement in computer technology?
53.There is no indication that the invention results in an improvement in the functioning of a computer. The reference to resource savings appears to be limited to programmer rather than computing resources.
Does the invention use generic computer technology?
54.There is no indication that the invention uses anything but generic computer technology. E.g. Figure 5 and paragraphs [0049]-[0065] appear to set out implementations using conventional computer technology.
Are there any steps foreign to the normal use of computers? Is there ingenuity in the way in which the computer is utilised?
55.In computer technology important data is often accompanied by metadata, i.e. information about that important data. This is nothing new to the functioning of computers, and in this case, the metadata is merely characterised by its content. There is nothing in the claims or the specification to suggest some unusual or improved function or operation of metadata.
56.There does not appear to be any indication that the concept of metadata or its use to include instructions or information about data in a general sense is new, nor the idea of storing code inline with data (or in a repository), nor the idea of procedurally generating code. The apparently novel aspect rather appears to be the application of these steps to clinical trial data.
Is the computer merely an intermediary or tool for performing the method while adding nothing of substance to the idea?
57.This seems an accurate statement of the instant situation.
Does the invention lie in the generation, presentation, or arrangement of intellectual information?
58.The invention lies in a method for preparing intellectual information. There is no doubt that the “Artifacts” generated by the method are intellectual information. The data used by the method is characterised by its intellectual content (“clinical trial data”) and no specific technical features of clinical trial data appear to be taken into account by the method.
Does the application of the method produce a practical and useful result? Does the invention produce a technical effect? Does the invention solve a technical problem within the computer or outside the computer?
59.The crux of the Applicant’s submissions, argued with some flourish, was that
“If avoiding the generation of sub optimal computer executable code is not a technical problem requiring a technical solution, the applicant queries how else could this problem be reasonably characterised?”
60.I agree in that I do not see how the problem could be better characterised for the Applicant. I can see how this result could be “useful” in a business sense. I however do not see that it is in any way “practical” given that there is no objective measure of “code optimality”. Rather “optimality” or fitness of code for purpose would seem to be according to the taste of the person commissioning and/or reviewing the code.
61.I likewise cannot see how avoiding a sub optimal result in a field must automatically make the effect technical, particularly when the claimed solution is simply pointing to something that has been prepared earlier, or outsourcing the task to an unspecified procedurally generated algorithm.
62.By analogy, saving time by using more resource, saving resource by using more time, or increasing efficiency by providing instructions or training are conventional business trade-offs and have never been considered technical. Here, time and/or resource are arguably being saved by simply providing instructions ahead of time.
63.As an aside, the situation here appears to be very similar to the one in Malewicz where the Deputy Commissioner found (at [72]) that “While a crude and broad analogy, the saving of time by “preparing” earlier is a tenet of effective organisation of activity”.
Is the invention technological or business innovation?
64.As such the invention as disclosed and claimed can only be viewed as a business innovation for saving resources in the process of clinical trial data analysis by either preparing the analysis code ahead of time or by preparing analysis code according to the instructions in the metadata (but in an unspecified way).
65.To revisit the analogy of the packet of noodles that I prepared earlier, the claims are saying in effect “process the noodles according to the instructions on the packet” but without providing any specific information on what the instructions even are. The stated advantage is simply the ability to cook noodles more consistently and/or save the cook the time of deciding how to cook the noodles.
66.I could conceive of a situation where a specified process of procedural generation of code could be technical if it were linked to specific technical properties of the clinical trial data to be analysed. Although the specification and claims refer to properties of clinical trial data such as differentiation between study level, population level and global level, and the need to transform data into different formats, these are business and regulatory considerations, not technical considerations. In any event the specification does not provide any detail of how the preparation of analysis code would actually take these properties into account.
Conclusion
67.Each of the Aristocrat factors considered above, individually and in combination, lead to the conclusion that the substance of the claimed invention does not fall within the field of patentable subject matter. Therefore I find that the invention as claimed in claims 1, 6 and 11 is not patentable subject matter.
68.In the course of my review of the specification, as set out above, I could not identify any technical disclosure that could form the basis of a patentable claim. By extension, there cannot be any patentable subject matter in the dependent claims either and I cannot see any prospect of overcoming this by amendment.
69.I therefore refuse the application.
Felix White
Delegate of the Commissioner of Patents
Annex A – Claims as proposed to be amended
A computer-implemented method for analyzing clinical trial data, the method performed by at least one processor, the method including:
accessing, by the at least one processor, metadata that describes:
an input data set including raw data resulting from a clinical trial,
at least one artifact to be provided as output, and
at least one operation to be performed on the input data set to create the at least one artifact, wherein the metadata is arranged hierarchically to include at least two of global metadata associated with a plurality of studies, project level metadata associated with a project that includes a subset of the plurality of studies, and study level metadata associated with one of the plurality of studies, wherein the at least one operation further includes:
retrieving, from an electronic data capture (EDC) system, the input data set that includes raw data generated through the clinical trial;
transforming the raw data to a first standard format;
determining analysis data based on the transformed raw data;
transforming the analysis data to a second standard format; and
determining the at least one artifact based on the transformed analysis data;
generating, based on the metadata, program code that performs the at least one operation to create the at least one artifact, wherein the program code includes instructions to check for anomalous data in the raw data and remove data identified as anomalous, and where generating the program code includes:
accessing at least one line of the metadata and determining whether the at least one line of the metadata includes:
at least one instruction for generating program code that performs the at least one operation to create the at least one artifact,
a reference to program code in a code repository that performs the at least one operation to create the at least one artifact, or
program code included inline in the metadata that performs the at least one operation to create the at least one artifact;
if the metadata includes at least one instruction for generating program code that performs the at least one operation to create the at least one artifact, executing the at least one instruction included in the metadata to generate program code that performs the at least one operation to create the at least one artifact;
if the metadata includes a reference to program code in a code repository that performs the at least one operation to create the at least one artifact, retrieving at least a portion of the program code from the code repository based on the reference included in the metadata and inserting same into the program code;
if the metadata includes program code inline in the metadata that performs the at least one operation to create the at least one artifact, retrieving at least a portion of the program code that is included, inline, in the metadata and inserting same into the program code;
executing, by the at least one processor, final program code including the generated or inserted program code, the executing including performing the at least one operation on the input data set, wherein executing the final program code creates the at least one artifact based on the performing of the at least one operation; and
communicating, by the at least one processor, the at least one artifact for presentation through a user interface of a computing device.
A method according to claim 1, wherein the at least one artifact includes at least one of a table, a figure, or a listing of data.
A method according to either claim 1 or claim 2, wherein:
the program code is associated with an individual study performed during the clinical trial;
the input data set includes data associated with the individual study; and
the at least one artifact generated that is through execution of the program code is associated with the individual study.
A method according to any one of the preceding claims, wherein the program code is generated and executed by a same code generation engine.
A method according to claim 4, further including:
storing, by the at least one processor, at least a portion of the program code in a library that is accessible by the code generation engine for subsequent program code generation.
A system for analyzing clinical trial data, the system including:
a memory; and
at least one processor communicatively coupled to the memory, the memory storing instructions which, when executed, cause the at least one processor to perform actions including:
accessing metadata that describes:
an input data set including raw data resulting from a clinical trial,
at least one artifact to be provided as output, and
at least one operation to be performed on the input data set to create the at least one artifact wherein the metadata is arranged hierarchically to include at least two of global metadata associated with a plurality of studies, project level metadata associated with a project that includes a subset of the plurality of studies, and study level metadata associated with one of the plurality of studies, wherein the at least one operation further includes:
retrieving, from an electronic data capture (EDC) system, the input data set that includes raw data generated through the clinical trial;
transforming the raw data to a first standard format;
determining analysis data based on the transformed raw data;
transforming the analysis data to a second standard format; and
determining the at least one artifact based on the transformed analysis data;
generating, based on the metadata, program code that performs the at least one operation to create the at least one artifact, wherein the program code includes instructions to check for anomalous data in the raw data and remove data identified as anomalous, and where generating the program code includes:
accessing at least one line of the metadata and determining whether the at least one line of the metadata includes:
at least one instruction for generating the program code that performs the at least one operation to create the at least one artifact,
a reference to the program code in a code repository that performs the at least one operation to create the at least one artifact, or
the program code included inline in the metadata that performs the at least one operation to create the at least one artifact;
if the metadata includes at least one instruction for generating the program code that performs the at least one operation to create the at least one artifact, executing the at least one instruction included in the metadata to generate program code that performs the at least one operation to create the at least one artifact,
if the metadata includes a reference to the program code in a code repository that performs the at least one operation to create the at least one artifact, retrieving at least a portion of the program code from the code repository based on the reference included in the metadata and inserting same into the program code;
if the metadata includes program code inline in the metadata that performs the at least one operation to create the at least one artifact, retrieving at least a portion of the program code that is included, inline, in the metadata and inserting same into the program code;
executing final program code including the generated or inserted program code, the executing including performing the at least one operation on the input data set, wherein executing the final program code creates the at least oneartifact based on the performing of the at least one operation; and
communicating the at least one artifact for presentation through a user interface of a computing device.
A system according to claim 6, wherein the at least one artifact includes at least one of a table, a figure, or a listing of data.
A system according to either claim 6 or claim 7, wherein:
the program code is associated with an individual study performed during the clinical trial;
the input data set includes data associated with the individual study; and
the at least one artifact generated that is through execution of the program code is associated with the individual study.
A system according to any one of claims 6 to 8, wherein the program code is generated and executed by a same code generation engine.
A system according to claim 9, the actions further including:
storing at least a portion of the program code in a library that is accessible by the code generation engine for subsequent program code generation.
One or more computer-readable storage media storing instructions which, when executed, cause at least one processor to perform actions for analyzing clinical trial data, the actions including:
accessing metadata that describes:
an input data set indicating raw data resulting from a clinical trial,
at least one artifact to be provided as output, and
at least one operation to be performed on the input data set to create the at least one artifact wherein the metadata is arranged hierarchically to include at least two of global metadata associated with a plurality of studies, project level metadata associated with a project that includes a subset of the plurality of studies, and study level metadata associated with one of the plurality of studies, wherein the at least one operation further includes:
retrieving, from an electronic data capture (EDC) system, the input data set that includes raw data generated through the clinical trial;
transforming the raw data to a first standard format;
determining analysis data based on the transformed raw data;
transforming the analysis data to a second standard format; and
determining the at least one artifact based on the transformed analysis data;
generating, based on the metadata, program code that performs the at least one operation to create the at least one artifact, wherein the program code includes instructions to check for anomalous data in the raw data and remove data identified as anomalous, and where generating the program code includes:
accessing at least one line of the metadata and determining whether the at least one line of the metadata includes:
at least one instruction for generating program code that performs the at least one operation to create the at least one artifact,
a reference to program code in a code repository that performs the at least one operation to create the at least one artifact, or
program code included inline in the metadata that performs the at least one operation to create the at least one artifact;
if the metadata includes at least one instruction for generating program code that performs the at least one operation to create the at least one artifact, executing the at least one instruction included in the metadata to generate the program code that performs the at least one operation to create the at least one artifact;
if the metadata includes a reference to program code in a code repository that performs the at least one operation to create the at least one artifact, retrieving at least a portion of the program code from the code repository based on the reference included in the metadata and inserting same into the program code;
if the metadata includes program code inline in the metadata that performs the at least one operation to create the at least one artifact, retrieving at least a portion of the program code that is included, inline, in the metadata and inserting same into the program code;
executing final program code including the generated or inserted program code, the executing including performing the at least one operation on the input data set, wherein executing the final program code creates the at least one artifact based on the performing of the at least one operation; and
communicating the at least one artifact for presentation through a user interface of a computing device.
One or more computer-readable storage media according to claim 11, wherein the at least one artifact includes at least one of a table, a figure, or a listing of data.
0
3
0