This is an old revision of the document!
Table of Contents
OHDSI Best Practices for Patient Level Prediction
This document is under development. Changes can be proposed and discussed via the Patient-Level Prediction Workgroup meetings.
General principles
- Transparency: others should be able to reproduce your study in every detail using the information you provide.
- Prespecify what you're going to predict and how: this will avoid fishing expeditions, p-value hacking. Run your analysis only once against the test set.
- Validation of your analysis: you should have evidence that your analysis does what you say it does (showing that statistics that are produced have nominal operating characteristics (e.g. p-value calibration), showing that specific important assumptions are met (e.g. covariate balance), using unit tests to validate pieces of code, etc.)
Best practices (generic)
Data characterisation and cleaning: Before modelling it is important to characterize the cohorts, for example by looking at the prevalence of certain covariates. Tools are being developed in the community to facilitate this.
**Dealing with missing values **: A best practice still needs to established.
Feature construction and selection: Both feature construction and selection should be completely transparent using a standardised approach to be able repeat the modelling but also to enable application of the model on unseen data.
Inclusion and exclusion criteria: All inclusion and exclusion criteria should be made explicit. It is recommended to do sensitivity analyses. Visualisation tools could help here, this will be further explored.
Validation of results: Validation of results should be done using a split-sample approach. The percentage used for training could depend on the number of cases, but as a rule of thumb 80/20 split is recommended. Hyper-parameter training should only be done on the training set. The validation set should only be used once as the final step.