Accuracy and Validations

We place a very high priority on the validation of the Model. In the end, the value of a model is directly proportional to its accuracy. The basic methodology is to compare the Model’s calculations with what happens in the real world.

Levels and types of validations:

Our Model is put through several levels of validations, which we call local, regional and global.

• Local. For every equation we check the equation against the data used to derive the equation to ensure that the fit is good, but there is no over fitting. This is a dependent local validation (see below). If there are independent sources for the same equation, we compare the equation to those data. These are independent local validations. They are called local because they test only one equation at a time.
• Regional. For each equation or closely related set of equations, we seek an empirical study that is closely related to that equation. We then simulate that study; using steps that are similar to the steps described below for global validations against clinical trials, and compare the results calculated by the Model against the results seen in the actual trial. An example of a study is a case control study of the risk of CVD as a function of various levels of fibrinogen. This type of validation is called regional because it tests collections of closely related equations, but does not draw on all the features of the Model. For example the above mentioned case control study does not test any tests, treatments, patient or provider behaviors, visits, etc. This type of validation is commonly used to validate risk factors and biological outcomes (e.g. FPG).
• Global. These validations test the integrity and accuracy of the entire Model. Typically they span every aspect of the Model, including the representation of underlying physiology, creation of simulated individuals who match those found in particular settings, development and progression of diseases, protocols, tests, treatments, behaviors, visits and other logistics, and outcomes.

The validations are of three main types:

1. Dependent: Information from the trial was used to estimate one or more equations that are directly related to the outcome that is the target of the validation. For example, results of the UKPDS trial were used to write an equation for the effect of duration and severity of diabetes and FPG on coronary artery occlusion and myocardial infarctions.
2. Semi-independent: Information from the trial was used to estimate one or more equations that affect the underlying physiology, but that do not directly affect the target outcome. For example, results of the UKPDS trial were used to write an equation for the progression of insulin resistance if untreated. The outcomes measured in the trial (e.g. heart attacks, strokes, micro-vascular outcomes) are indirectly affected by this equation, but also require more than a dozen other equations that were not estimated using any UKPDS data.
3. Independent: No information about the results of the trial was used to build the Model. The only information used was about the design of the trial, and the baseline characteristics of the trial population before the trial began, if available. Independent validations can be either prospective, if they are done before any trial results are known to us, or retrospective, if the trials results are publicly available even if they weren’t used to build the Model.

Validations against clinical trials

Our gold standard for conducting global validations is clinical trials; we use the Model to simulate real-clinical trials and compare the Model’s results with what actually occurred in the trial. The trials are chosen according to the following criteria:

• They are the landmark trials used by clinicians and experts to form their own understanding of the conditions and the effects of treatments.
• They span a broad spectrum of populations - demographics (e.g. age/sex/race-ethnicity), diseases and conditions, disease severity, prevention-treatment, and co-morbidities)
• They measure outcomes from a broad range of organ systems
• They evaluate a broad range of treatments

Trials used to validate the accuracy of the model

Initially eighteen trials were chosen by an independent committed sponsored by the American Diabetes Association. This committee also reviewed the results of the validations. The methods and results for these validations have been described in "Validation of the Archimedes Diabetes Model" by Eddy and Schlessinger, Diabetes Care Volume 26, Number 11, November 2003.

Since then the Archimedes Model is continuously tested and validated against real-world trials. The following list of trials used to validate the Model was up to date on June, 1 2008:

While each individual trial tests important parts of the Model, when taken together the trials crisscross the Model and test its validity as a whole.

These validations build confidence that the Model will be accurate for new applications. However, they do not guarantee accuracy for all problems. Our objective is to make the model as accurate as possible given the available information. There is always a chance of a surprise.

Comparison of the results calculated by the Model with the results of the actual trials for the first 74 validations. These validations involved 18 separate trials, 10 of which were not used to build the Model and provide independent validation of the Model’s accuracy.

Find out more about Archimedes Model validations »

Related:

• The Archimedes Model
• Applications & Value
• Limitations