Limitations
Because the Archimedes Model is so much more powerful than other models, it is important to understand its limitations. They relate to its functionality, customization, and validation.
Functionality
For the conditions that are currently in the Model, the Model includes the most important biomarkers and interventions needed to analyze the great majority of problems. However, there may be other conditions, biomarkers or interventions that are of interest for a particular problem that are not yet in the Model. These can be added to the Model upon request.
The level of physiological detail of the Model is determined by the level of detail of the questions it is designed to address. Typically, this is the level of detail at which clinicians think and make their decisions, the level of detail found in clinical charts, and the level of detail at which clinical trials are designed and reported. For many areas of physiology the causes of clinical phenomena are understood at a deeper level of detail by basic scientists and specialists. In the Archimedes Model there factors are represented phenomenological, not causally. For example, the Model represents causally the relationships between heart rate, stroke volume, cardiac output, peripheral resistance and blood pressure. But the Model but does not represent the effects of cardiac filling on sarcomere length and force of myocardial contraction. If there is sufficient interest, the Model can be extended to deeper levels of physiological detail.
The ability of the Model to address additional conditions, biomarkers, interventions or physiological processes depends on the available data. In general, there must be data that relate any new variables to the existing variables in the Model to which they are causally related. Our validation protocols require that there be data from at least two sources: one for building the equations, and at least one other for performing an independent validation.
Customization
The effects of interventions can depend as much on the current level of care (the "control") as on the new level of care that is the nominal target of the analysis (the "treatment"). For example, the measured effect of population-wide screening for hypertension will depend on whether it is compared to "no testing at all" versus "sporadic testing at routine office visits". The latter, in turn, can be very different in managed care settings, versus fee for service, versus no insurance. Similarly, the effect of a treatment on costs can depend as much on the cost of routine care (e.g. office visits) and downstream events (e.g. treatment of strokes) as on the cost of the hypertension test or treatment. As shown in studies of performance and practice variations, levels of care can vary widely in different settings. The default setting of the Archimedes Model is based on nationally recommended guidelines, average performance levels, and protocols and costs experienced in managed care settings. Thus it is suitable for a wide range of problems that use a "national" or "standard--of-care" perspective. However, for each new analysis it is important to determine whether a different set of assumptions would be more appropriate. To address this possibility, the Archimedes Model can be customized to represent different levels of care such as different guidelines, protocols, performance levels, and costs.
In the same way that different patterns of care and costs affect the results of an analysis, the population to which an intervention is applied will affect the results. For example, the effect of screening for hypertension will depend not only on current levels of testing, but also on the current levels of blood pressure and other risk factors in the population. Even if all else is equal, an analysis done for people in Los Angeles will be different than an analysis done for people in San Diego. The default setting for the Archimedes Model is a random sample of the US population and is suitable for a wide range of "national" or "standard-of-care" analyses. However, it may need to be changed for applications that target other populations or sub-populations. To accommodate this, the Model includes methods for constructing simulated populations that match virtually any real population using either person-specific or aggregated data.
While the mathematics and programming of the Archimedes Model are very flexible and can theoretically be customized to fit virtually any setting or population, the ability to customize the Model will depend on the available data. Archimedes scientists can work with clients to determine the need for customization and the best use of the available data.
Validations and Accuracy
We believe the Archimedes Model has been validated more rigorously than any other model in healthcare. However, it is important to understand the limitations of the validations.
- Of the first 74 exercises that were published, two of the results (3%) fell outside the 95% confidence intervals of the clinical trial. Nonetheless an independent panel did not find the discrepancies to indicate any flaws in the Model.
- The results of trials themselves can vary. Repeating a trial will inevitably deliver different results due to sampling variability and random factors (this is the reason for calculating confidence intervals for the results of trials.) Thus the Archimedes Model might match the result of a trial but still not fall within the 95% confidence interval because the trial’s result itself was misleading. This problem is greater for small populations and rare outcomes, where small numbers increase the range of uncertainty (confidence intervals) about the results.
- The ability of the Model to match a trial depends on the completeness and accuracy of the description of the trial (the two discrepancies just mentioned were attributed to the presence of an unspecified risk factor in the trial population). Similarly, when the Model is applied to a new problem, any gaps or errors in the description of the new population, treatment, outcome or setting could affect the accuracy of the Model.
- There are many gaps in trials - populations, treatments, outcomes, and settings that have never been studied with trials. By definition, if there are no trials for these questions, there is no way to validate the Model for them.
- There are always surprises - results that are unpredictable or even contrary to previously existing evidence. The Archimedes Model will be surprised by these results just as much as the experts. The Model only accurately represents medical knowledge as it is currently understood from the available evidence.
- Clinical trials are conducted in specific settings (populations, protocols, behaviors, costs). They are valuable for validating the Model’s ability to replicate specific settings and for its accuracy for predicting outcomes in the settings that are specified. However, clinical trials do not necessarily represent what occurs in other settings. The Archimedes Model includes variables that address differences in settings (e.g. different populations, protocols, and behaviors) and as described above it can be customized to fit a wide variety of settings. However, there may be no way to validate the Model for a particular setting if there is no formal empirical study of that setting.
Most of these limitations are inherent in medical evidence, and affect all types of decision making, not just the Archimedes Model. Nonetheless, they need to be understood when interpreting the results of an Archimedes analysis. The validations build confidence that the results of an Archimedes analysis will be accurate.
The Archimedes Model is not intended to replace clinical trials. Trials are the bedrock of medical practice. They are the evidence on which evidence-based medicine is based. They are also essential for building and validating models.
If it is possible to answer a question by conducting a new trial, conduct the trial. The Archimedes Model is intended for problems that cannot be practically studied empirically with formal trials or other evaluation designs. Reasons include too high a cost, too long a duration, too many options to be studied, too much disruption, and too much urgency in the decision. When factors like these make it impossible to conduct a more formal study but a decision has to be made, the Archimedes Model is the best alternative to a trial.
The Model can also be used to extend the results of an existing trial. This includes calculating outcomes over a longer follow-up period, examining outcomes in subpopulations, calculating additional outcomes such as utilization and costs, and calculating outcomes for different settings (e.g. different background protocols or behaviors).
Contact us to find out more about the Archimedes Model »