Objective: This study aims to evaluate confidence interval calibration.
Rationale: Observational studies are prone to bias, but unfortunately this bias is often brushed aside with a single comment in the discussion of a paper and is never quantified. In the past we have proposed using negative controls (exposure-outcome pairs where the true relative risk is believed to be one) to produce an empirical bias distribution, and subsequently calibrate p-values. Here we propose to extend this approach to calibrated confidence intervals, which requires the use of positive controls. Since real positive controls are problematic, we will generate artificial positive controls by injection additional (simulated) outcomes on top of negative controls. To demonstrate and evaluate confidence interval calibration, we will reproduce two pairs of observational studies that have produced conflicting results. We will try to demonstrate that our calibration procedure has good internal validity by showing coverage of the confidence interval improves as measured using our negative and positive controls. We will try to show external validity by showing that after calibration our conflicting studies are no longer in disagreement.
Project Lead(s): Martijn Schuemie, Marc Suchard, George Hripcsak, Patrick Ryan, David Madigan
Coordinating Institution(s): Janssen R&D
Participating Institution(s): Janssen R&D
Full Protocol: CI Calibration protocol
Initial Proposal Date: 1/11/2016
Launch Date: 1/11/2016
Receive Results for Analysis Date: NA
Study Closure Date: 1/5/2017 (Study closed)
Results Submission: NA
CDM: V5 only
Table Accessed: CONCEPT, CONCEPT_ANCESTOR, CONDITION_ERA, CONDITION_OCCURRENCE, DRUG_ERA, DRUG_EXPOSURE, MEASUREMENT, OBSERVATION, OBSERVATION_PERIOD, PERSON, PROCEDURE_OCCURRENCE, VISIT_OCCURRENCE
Database Dialects: SQL Server, Postgres, Oracle
Software: R (>= 3.2.2, RTools), Java
Hardware: Recommended 8-core, 32GB memory, 100GB free space
How to Run: Full installation and run instructions are provided on the GitHub page below.
Results will be submitted for publication shortly.