Skip to main content

Table 1 Methods for specifying an important and/or realistic difference [5]

From: Specifying the target difference in the primary outcome for a randomised controlled trial: guidance for researchers

Name

Description

Difference specified

Anchor

The outcome of interest can be ‘anchored’ by using either a patient’s or health professional’s judgement to define an important difference. This may be achieved by comparing a patient’s health before and after treatment, and then linking this change to participants who showed improvement and/or deterioration using a more familiar outcome (for which either patients or health professionals more readily agree on what amount of change constitutes an important difference). An outcome can be anchored to another which more is known about. Contrasts between patients (such as individuals with varying severity of a disease) can also be used to determine a meaningful difference.

Important

Distribution

Approaches that determine a value based upon distributional variation. A common approach is to use a value that is larger than the inherent imprecision in the measurement and therefore likely to represent a minimal level for a noticeable difference.

Important

Health economic

Approaches that use principles of economic evaluation. These typically include both resource cost and health outcomes, and define a threshold value for the cost of a unit of health effect that a decision-maker is willing to pay, to estimate the overall net benefit of treatment. The net benefit can be analysed in a frequentist framework or take the form of a (typically Bayesian) decision-theoretic value of information analysis. Due to difficulties in implementing a value of information analysis simpler heuristic frameworks, also based on the principles of economic evaluation, have been proposed.

Important

Opinion-seeking

The target difference can be based on opinions elicited from health professionals, patients or others. Possible approaches include forming a panel of experts, surveying the membership of a professional or patient body or interviewing individuals. This elicitation process can be explicitly framed within a trial context.

Important and/or realistic

Pilot study

A pilot (or preliminary) study may be carried out where there is little evidence, or even experience, to guide expectations and determine an appropriate target difference for the trial. In a similar manner, a phase two study could be used to inform a phase three study, though this would need to take account of methodological differences (such as inclusion criteria and outcomes), which should be reflected in the target difference.

Realistic

Review of evidence base

The target difference can be derived using current evidence on the research question. Ideally, this would be from a systematic review or meta-analysis of RCTs. In the absence of randomised evidence, evidence from observational studies could be used in a similar manner. An alternative approach is to undertake a review of studies in which an important difference was determined.

Important and/or realistic

Standardised effect size

The magnitude of the effect on a standardised scale defines the value of the difference. For a continuous outcome, the standardised difference (most commonly expressed as Cohen’s d effect size) can be used. Cohen’s cutoff values of 0.2, 0.5 and 0.8 for small, medium and large effects, respectively, are often used. Thus a medium effect corresponds simply to a change in the outcome of 0.5 SDs. Binary or survival (time-to-event) outcome metrics (such as an odds, risk or hazard ratio) can be utilised in a similar manner, though no widely recognised cutoff values exist. Cohen’s cutoff values approximate odds ratios of 1.44, 2.48 and 4.27, respectively. Corresponding risk ratio values vary according to the control group event proportion.

Important