Validation & Calibration
Validation of the physiology
Because of the Model's complexity and the importance of the questions it can help answer, we conduct a rigorous suite of validations on an ongoing basis. The Archimedes Model is validated with dependent and independent validations at each of three levels:
-
Local validations: We conduct dependent and independent validations of each equation to ensure that the fit is good without over-fitting.
-
Regional validations: For each equation or related set of equations, we seek an empirical study that is closely related. We then simulate that study and compare the results against the results seen in the actual trial.
-
Global validations: To perform a complete validation of the Model, we have created an automated validation suite consisting of landmark clinical trials and epidemiological studies with multiple outcomes. We use the inclusion and exclusion criteria of the trial population and capture the distribution of patients in the real trial in order to simulate a population that matches the baseline characteristics of the trial's population. We compare the primary and secondary outcomes in our simulation with those of the real trial, and we compare biomarker profiles, utilization rates, and costs.
Global validations test the integrity and accuracy of the entire Model. Typically they span almost every aspect of the Model, including the representation of underlying physiology, creation of simulated individuals who match those found in particular settings, development and progression of diseases, protocols, tests, treatments, behaviors, visits and other logistics, and outcomes.
Calibration of care processes
The Archimedes Model is calibrated and validated by comparing the virtual outcomes of the Model to healthcare utilization and diagnosis rates observed in the real world. We build a virtual healthcare system that includes all guidelines relevant to the diseases and conditions the Model covers. We then adjust patient adherence and provider performance to calibrate the Model to the level of care desired.
Ongoing validations
New trials provide an important test bed for the Model. Often trial results are consistent with current medical knowledge, and the Model simulation replicates those trial observations. In some cases, trials provide new information that either is confirmed by Model performance or can be used to extend and enhance the Model. If appropriate, such trials are added to the suite of trials used to validate the Model.
Perhaps the best testament to the Model's predictive power is realized when the Model's recommended interventions are put to the test in real-world prospective trials and the outcomes of the real trials validate the Model’s predictions. For example, in a trial conducted for Kaiser Permanente, referred to as the A-L-L trial, the Model forecasted that the “bundled” cardioprotective medications aspirin, lisinopril, and lovastatin would reduce the risk of heart attack and stroke in a high-risk population by over 70 percent. Kaiser Permanente then implemented the regime in a three-year clinical observational study whose results closely matched those of the Model. (Dudl RJ, Wang MC, Wong M, Bellows J. Preventing Myocardial Infarction and Stroke with a Simplified Bundle of Cardioprotective Medications, Am. J. Managed Care. 2009 Oct 1;15(10):e88-94)
Click to download the full reports
