Post-trial follow-up methodology in large randomised controlled trials: a systematic review

Abstract

Background

Randomised controlled clinical trials typically have a relatively brief in-trial follow-up period which can underestimate safety signals and fail to detect long-term hazards, which may take years to appear. Extended follow-up after the scheduled closure of the trial allows detection of both persistent or enhanced beneficial effects following cessation of study treatment (i.e. a legacy effect) and the emergence of possible adverse effects (e.g. development of cancer).

Methods

A systematic review was conducted following PRISMA guidelines to qualitatively compare post-trial follow-up methods used in large randomised controlled trials. Five bibliographic databases, including Medline and the Cochrane Library, and one trial registry were searched. All large randomised controlled trials (more than 1000 adult participants) published from March 2006 to April 2017 were evaluated. Two reviewers screened and extracted data attaining > 95% concordance of papers checked. Assessment of bias in the trials was evaluated using the Cochrane Risk of Bias tool.

Results

Fifty-seven thousand three hundred and fifty-two papers were identified and 65 trials which had post-trial follow-up (PTFU) were included in the analysis. The majority of trials used more than one type of follow-up. There was no evidence of an association between the retention rates of participants in the PTFU period and the type of follow-up used. Costs of PTFU varied widely with data linkage being the most economical. It was not possible to assess associations between risk of bias during the in-trial period and proportions lost to follow-up during the PTFU period.

Discussion

Data captured during the post-trial follow-up period can add scientific value to a trial. However, there are logistical and financial barriers to overcome. Where available, data linkage via electronic registries and records is a cost-effective method which can provide data on a range of endpoints.

Systematic review registration

Not applicable for PROSPERO registration.

Background

Randomised controlled trials (RCTs) are considered to be the ‘gold standard‘ for assessing the effects of a treatment. However, these trials usually report results following a relatively brief exposure to the intervention under investigation. Longer-term follow-up of trial participants is important as persistent effects may be detected years later after treatment cessation or even enhanced benefits observed decades later – a so-called ‘legacy effect‘ [1, 2]. Furthermore, delayed hazards may only emerge several years after exposure to certain treatments. Therefore, PTFU may add significant scientific value to the evaluation of many healthcare interventions.

We define post-trial follow-up (PTFU) as extended follow-up which starts after the end of the scheduled period of the trial. Such follow-up, regardless of the primary in-trial outcome, provides important information including safety of the intervention, identification of delayed hazards and long-term beneficial effects.

Retention of participants in PTFU is important since high rates of attrition may introduce bias if reasons for withdrawal are related to the intervention [3]. There are a variety of methods for PTFU, but little research has been done to evaluate which methods for PTFU leads to the best retention rates [4]. Choice of follow-up method is often determined by the funding for the trial and the local availability of relevant data. Telephone calls, postal questionnaires and face-to face interviews are the more traditional approach to follow-up. Web-based approaches and use of routine health records and electronic registries are becoming more popular due to advancing technology and options for accessing the information inexpensively [5, 6].

This systematic review compares methods used in approaches to PTFU and aims to inform the design of PTFU for a wide range of randomised trials. The main objective was to evaluate the retention rates (or levels of attrition) of the participants followed up during PTFU and to compare this to the type of methodology used. A secondary objective was to compare the costs of post-trial methodology as funding is often limited.

Methods

The methods used in this systematic review have been described in detail previously [7] and follow PRISMA guidelines Additional file 1.

Eligibility criteria

Briefly, all large (> 1000 adult participants) RCTs which investigated a healthcare intervention (i.e. medicine, surgery or psychiatric in nature) and involved PTFU were included. Only studies published between 2006 and 2017 were included. Alternative medicines (e.g. acupuncture) or holistic interventions including physical therapy were excluded from the review. Large RCTs were only included due to the reduced risk of random error in the outcomes.

PTFU was defined as passive follow-up which had occurred either after the scheduled closure of the trial or after the primary results had been published.

Search strategy

The search was conducted in five bibliographic databases on 13 April 2017, including Embase (OvidSP) (1 March 1974 to 12 April 2017), Medline (OvidSP) (1946 to present), PubMed, Cochrane Central Register of Controlled Trials (Cochrane Library, Wiley) (issue 3 of 12, March 2017) and Cochrane Methods Register (CENTRAL) (Cochrane Library, Wiley) (issue 3 of 4, July 2017). Searches were then restricted to articles published in English since 2006. Full details of strategies are provided in Additional file 2. In addition, a database search for completed and ongoing studies was conducted at ClinicalTrials.gov (https://clinicaltrials.gov/). Studies which were not yet published ‘grey literature’ were not included in the search strategy.

Data collection

Papers identified from the ClinicalTrials.gov registry were imported into a MS Excel spreadsheet. Duplicates and studies which had less than 1000 participants were removed using a filter option. The selection of eligible papers followed a concordance strategy between two reviewers (RLB and DE) which ensured that concordance was > 95% (Fig. 1) [7].

Fig. 1

PRISMA flow diagram detailing the process of study selection and data extraction. HCI healthcare intervention, PTFU post-trial follow-up, RCT randomised controlled trial

Medical interventions were defined as an intervention that was consumed orally, inhaled, or administered by intravenous or intramuscular injections including vaccines. A surgical intervention was defined as any intervention which was invasive (apart from those mentioned above and including blood transfusions). Potential studies were checked for eligibility by two reviewers who initially reviewed abstracts and then proceeded to full paper review in a step-wise process (Fig. 1).

In addition to those described in the protocol, some additional exclusions which were not originally listed were identified during the process of performing the systematic review in keeping with our definition of PTFU. This was required due to the heterogeneity of PTFUs. These include: (1) trials that were stopped before the scheduled closure of the trial; (2) cancer trials which had an open endpoint (e.g. survival as an endpoint with no clear scheduled plan of duration); (3) trials which continued with active intervention in the PTFU period with the primary outcome of safety and (4) trials eligible for inclusion but which did not contribute novel data as they only published additional subgroup or post-hoc analyses. A table of excluded trials is provided in Additional file 3.

Full papers deemed eligible for inclusion in the systematic review were extracted using a standardised Excel spreadsheet. Data was extracted by DE and RLB and concordance was checked. Primary outcome, healthcare intervention and attrition rates were tabulated for each study. Lead trialists were contacted via email to inform them of the systematic review and to clarify information where necessary. The papers included in the review were diverse with a range of interventions and different outcome measures. Due to the high level of clinical heterogeneity a meta-analysis was not possible.

Retention rates were calculated as the proportion of participants who were lost to follow-up compared to the overall total of those who started the PTFU period. Information about the cost of the PTFU was sought from study publications or via personal communication. Two attempts were made to contact the trialist via email and if there was no response or inadequate data, the trial was excluded from the cost analysis.

Assessment of risk of bias

Risk of bias was assessed for each included RCT on their primary results using the Cochrane Risk of Bias tool. Covdence.org was used to assess the levels of bias (low risk, high risk or unclear risk) in each methodological domain (sequence generation, allocation of sequence concealment, blinding, incomplete outcome data, selective reporting bias and other bias) and decisions checked by one of the senior authors [8]. The data recorded from Covidence.org was imported into Review Manager 5 (RevMan 5) for graphical representation [9].

Results

From 57,352 papers identified, 65 studies with PTFU were included in the systematic review (Fig. 1). Fifty trials involved medical interventions and 15 involved surgical interventions. There were no eligible psychiatric trials which had (all > 1000 participants). The duration of PTFU ranged from 1 to 20 years, with a median of 4.5 years of follow-up. The number of participants followed during the post-trial period ranged from 575 to 29 862.

Five methods of follow-up were identified: postal correspondence/questionnaire (19%); clinic appointments (35%); telephone interviews (26%); electronic data linkage (52%); and review of paper medical records (26%). In addition, in individual cases, specific follow-up was performed, e.g. endoscopy follow-up only [10]. Electronic data linkage and medical records review were used exclusively together in 11% of papers; either were used in combination with other methods in 74%. Overall, 48% of trials used more than one method to follow-up participants in the post-trial period (Tables 1 and 2). On average, two methods were used for each PTFU follow-up. Where data linkage was used, it was not always feasible to follow up all participants [11]. Some trials experienced difficulty accessing national electronic data in certain countries; for example, stricter regulations are apparent in Canada and for some North American participants (Medicare and Veteran Affairs) where a specific health ID number is required to access national data (Table 3). Trials experienced difficulty in accessing routinely collected health records in 3% of included papers and PTFU was restricted to those countries with robust and accessible centrally held records and registries (e.g. Sweden and Scotland) [12, 13].

Table 1 Post-trial follow-up (PTFU) in eligible medical trials. Note retention of participants expressed as % lost to follow-up

Table 2 Post-trial follow-up (PTFU) in eligible surgical trials. Note, retention of participants is expressed as % lost to follow-up

Table 3 Registries used for data linkage during post-trial follow-up (PTFU)

Retention rates

Unfortunately, retention rates were often poorly reported in the PTFU, limiting the ability to assess the impact of methods used in relation to the proportion lost to follow-up.

All surgical trials investigated mortality as the primary outcome and, where data was available, the proportion of participants lost to follow-up in surgical trials ranged from 0.4 to 15.5%. However, data was not available for 53% of surgical trials. In medical trials, the primary outcome investigated varied more widely, although mortality as an endpoint was common and the proportion of participants lost to follow-up ranged from 0 to 22%. Data on loss to follow-up was not available in 44% of trials. Where mortality was the primary outcome, the number of participants lost to follow-up was not available in 32% of trials due to the use of mortality records where only notifications of deaths were fed back to the trialists.

Cost

Financial information was available for one third of the included trials. Consequently, it was not possible to provide a direct comparison between cost of PTFU and the different methodologies used due to the small sample size. The cost of PTFU ranged from £6000 to £14,600,000 (Table 4). Cost of PTFU per participant per year showed that IST-3 was the most economical costing £0.21 per participant per year using data linkage/medical records, closely followed by ‘Over 50s’ (£0.41) and RECORD trials (£0.46) which also used data linkage. LTPS was the most expensive PTFU per participant per year (US$531.53) using clinical appointments and telephone follow-up. ROADMAP which also followed up participants by clinic appointment only had a cost of €413.60 per participant per year.

Table 4 Comparing post-trial follow-up (PTFU) costs (where disclosed), by different follow-up methodologies

Cochrane Risk of Bias

We hypothesised that those RCTs which had poor methodology or ‘high risk of bias’ might subsequently have a PTFU which was poorly organised and, therefore, have low retention rates (or a high proportion lost to PTFU). Of the 65 papers included in the systematic review, seven were excluded from the risk of bias assessment: these were PTFUs which followed-up an amalgamation of data from more than one trial or were part of a systematic review and, therefore, not suitable to be included in the analysis (the risks of bias from individual component trials could not be combined).

Of the 58 trials considered, the risk of bias could not be fully assessed in 11 trials due to lack of information in at least one domain. Low (or unclear) risk of bias in all domains was found in 43 (74%) of those assessed. Only seven trials (12%) had at least one domain which was high risk of bias, of which one had two domains at high risk (Fig. 2). Details of the individual risk of bias domains for each included study are provided in Additional file 4.

Fig. 2

Cochrane Risk of Bias graph. Review authors’ judgements about each risk of bias item presented as percentages across all included studies

Given the small number of trials found to have a high risk of bias in at least one domain and the highly variable retention rates observed during PTFU (Table 5), it is not possible to draw any clear conclusions with respect to possible associations between risk of bias and its potential impact on the proportion of participants that were lost to follow-up in the post-trial period.

Table 5 Comparison of randomised controlled trials (RCTs) which had high risk of bias compared to the proportion lost to follow-up during post-trial follow-up (PTFU). A summary of those RCTs with no risk of bias are also detailed

Discussion

This systematic review identified that PTFU methods varied and many trials used overlapping approaches which were more costly than needed. Data was limited on retention rates and so it was difficult to draw any firm conclusions on which method was best for PTFU.

Our main findings suggest that most PTFU published in the last 11 years does not appear to be designed in a cost-effective manner. Cost of PTFU was shown to vary widely and not many trials used incentives to retain participants. Despite only a third of trialists providing complete financial information for PTFU, follow-up by clinical appointment appeared to be the most expensive method, as might be expected given the resource implications. Postal or telephone correspondence in addition to data linkage did not appear to increase the cost per participant per year considerably. However, the effect of inflation over the 11 years included in this systematic review, makes quantitative comparison of cost differences difficult. Given the limited data available we have not attempted to adjust for inflation.

Data linkage or access to medical records is likely to be the most cost-effective method of following participants due to minimal staff required. However, a number of trialists highlighted the limitations of this approach, noting it to be time-consuming and frustrating with increasing regulatory costs and country-specific restrictions. In the UK, the process of accessing data electronically has become more stringent and costly, and markedly different to the processes which were encountered 10 years ago. There is also an issue of the data lagging behind by up to 2 years in some countries, which can impact on the completeness of results for a trial. Despite this, data linkages to national registries and electronic health records have been shown to be a valid and reliable method of PTFU [12,13,14,15].

When designing this systematic review, we anticipated that papers published in the early half of the last 11 years would choose more traditional methods of PTFU, e.g. clinic- and telephone-based approaches, and more recent trials may increasingly use data linkage where available. However, this has not been the case. The majority of trials have used a variety of different methods to capture data for the same primary outcome. We were, therefore, unable to compare retention rates by each type of method used. In addition, sparsity of complete data in the review (typically poor reporting of the final number of participants at the end of the follow-up period) limited the ability to assess retention rates achieved with different PTFU methods.

We found limited evidence of high risk of bias in the methodology of the in-trial periods. A likely explanation for this is that the majority of the trials included in this review were well-designed, large RCTs in which results were published in high-impact journals. Furthermore, trials which employ poor methodology or have had negative results are more likely not to engage in PTFU due to lack of funding or interest.

Due to new guidelines (Consolidated Standards of Reporting Trials (CONSORT)) recommending increasing transparency in the reporting of RCTs, a more complete capture of data would be likely for any future study [16]. Research into appropriate methods in PTFU can only occur if there is transparency of the logistical and financial implications including number of participants lost to follow-up.

Conclusions

Post-trial follow-up of large RCTs can contribute significantly to the scientific value of a trial by determining the longer-term magnitude of the effects of an intervention. PTFU is valuable to ensure that there are no long-term hazards or beneficial effects which have been missed due to the common short in-trial periods for following up participants. However, it is not widely used as shown by the small number of eligible trials which had PTFU from the original search strategy.

Data linkage and the use of registries appear to be the most plausible and economical approach to PTFU. These methods also have the advantage of providing data for a wide range of endpoints. Improvement of electronic reporting and informatics could lead to better reporting and allow this type of method to be widely used.