Research Protocol - Improving the evaluation, implementation and patient impact of multistate models used for clinical prediction

Prediction models are tools that are used to predict a person’s risk of medical events occurring, for example, risk of having a heart attack in the next 10 years, or the risk of cancer returning following surgery to remove a tumour. These models can be used to guide treatment decisions for patients. Normally these models predict a single outcome - like heart attack. This ignores that often healthcare needs to know about events that may lead up to this event. For example, risk of someone developing diabetes or cancer, and then how this might alter or affect their risk of heart attack. One study[1] found that the order in which diabetes, psychosis, and congestive heart failure were developed resulted in different life expectancy. Understanding this is important to guide NHS policy on how to prevent these three conditions occurring. In another example, consider individuals with chronic kidney disease. It is important to understand the speed at which the kidney function worsens in order to appropriately plan care. In all these examples, statistical techniques called ‘multistate models’ are a natural way to analyse these processes. Multistate models are designed to model health through different ‘intermediate states’. Taking the example above, diabetes, congestive heart failure, and psychosis would all be different outcome ‘states’. Multistate models allow you to estimate the likelihood of developing each condition, the order in which they will be developed, and how this changes depending on which conditions have already been developed. Currently, there is lots of research on how to build multistate models, however, there is lack of research on how to test that they work well (‘model evaluation’). Without checking that models work well, it is unethical for a model to be used to make decisions for a patient. For example, if the prediction of developing diabetes in individuals with psychosis is too high, this model should not be used to guide patient care. Errors like this are picked up when testing a model’s performance. There is a need for multistate models in a wide range of settings beyond the examples given above. However, it is not possible to assess whether these models are fit for purpose until methods for testing the performance of multistate models are developed. This project will develop statistical methods to allow other researchers to test the performance of multistate models, and ensure models used in practice are not bad quality. Specifically, the project will: 1) Identify existing methods that could be used for this goal. 2) Extend the methods where necessary, and ensure they work well by running examples on simulated data. 3) Development of software to help other researchers apply these methods. 4) Showcase the impact of these methods through a clinical example predicting diabetes, psychosis, and congestive heart failure. 5) Gain a better understanding of how patients and key stakeholders interpret the output when testing the performance of a multistate model. The impact for patients will be better outcomes by stopping poorly performing models being used to inform clinical decisions.