Skip to main content

Advertisement

Post-trial follow-up methodology in large randomised controlled trials: a systematic review

Article metrics

Abstract

Background

Randomised controlled clinical trials typically have a relatively brief in-trial follow-up period which can underestimate safety signals and fail to detect long-term hazards, which may take years to appear. Extended follow-up after the scheduled closure of the trial allows detection of both persistent or enhanced beneficial effects following cessation of study treatment (i.e. a legacy effect) and the emergence of possible adverse effects (e.g. development of cancer).

Methods

A systematic review was conducted following PRISMA guidelines to qualitatively compare post-trial follow-up methods used in large randomised controlled trials. Five bibliographic databases, including Medline and the Cochrane Library, and one trial registry were searched. All large randomised controlled trials (more than 1000 adult participants) published from March 2006 to April 2017 were evaluated. Two reviewers screened and extracted data attaining > 95% concordance of papers checked. Assessment of bias in the trials was evaluated using the Cochrane Risk of Bias tool.

Results

Fifty-seven thousand three hundred and fifty-two papers were identified and 65 trials which had post-trial follow-up (PTFU) were included in the analysis. The majority of trials used more than one type of follow-up. There was no evidence of an association between the retention rates of participants in the PTFU period and the type of follow-up used. Costs of PTFU varied widely with data linkage being the most economical. It was not possible to assess associations between risk of bias during the in-trial period and proportions lost to follow-up during the PTFU period.

Discussion

Data captured during the post-trial follow-up period can add scientific value to a trial. However, there are logistical and financial barriers to overcome. Where available, data linkage via electronic registries and records is a cost-effective method which can provide data on a range of endpoints.

Systematic review registration

Not applicable for PROSPERO registration.

Background

Randomised controlled trials (RCTs) are considered to be the ‘gold standard‘ for assessing the effects of a treatment. However, these trials usually report results following a relatively brief exposure to the intervention under investigation. Longer-term follow-up of trial participants is important as persistent effects may be detected years later after treatment cessation or even enhanced benefits observed decades later – a so-called ‘legacy effect‘ [1, 2]. Furthermore, delayed hazards may only emerge several years after exposure to certain treatments. Therefore, PTFU may add significant scientific value to the evaluation of many healthcare interventions.

We define post-trial follow-up (PTFU) as extended follow-up which starts after the end of the scheduled period of the trial. Such follow-up, regardless of the primary in-trial outcome, provides important information including safety of the intervention, identification of delayed hazards and long-term beneficial effects.

Retention of participants in PTFU is important since high rates of attrition may introduce bias if reasons for withdrawal are related to the intervention [3]. There are a variety of methods for PTFU, but little research has been done to evaluate which methods for PTFU leads to the best retention rates [4]. Choice of follow-up method is often determined by the funding for the trial and the local availability of relevant data. Telephone calls, postal questionnaires and face-to face interviews are the more traditional approach to follow-up. Web-based approaches and use of routine health records and electronic registries are becoming more popular due to advancing technology and options for accessing the information inexpensively [5, 6].

This systematic review compares methods used in approaches to PTFU and aims to inform the design of PTFU for a wide range of randomised trials. The main objective was to evaluate the retention rates (or levels of attrition) of the participants followed up during PTFU and to compare this to the type of methodology used. A secondary objective was to compare the costs of post-trial methodology as funding is often limited.

Methods

The methods used in this systematic review have been described in detail previously [7] and follow PRISMA guidelines Additional file 1.

Eligibility criteria

Briefly, all large (> 1000 adult participants) RCTs which investigated a healthcare intervention (i.e. medicine, surgery or psychiatric in nature) and involved PTFU were included. Only studies published between 2006 and 2017 were included. Alternative medicines (e.g. acupuncture) or holistic interventions including physical therapy were excluded from the review. Large RCTs were only included due to the reduced risk of random error in the outcomes.

PTFU was defined as passive follow-up which had occurred either after the scheduled closure of the trial or after the primary results had been published.

Search strategy

The search was conducted in five bibliographic databases on 13 April 2017, including Embase (OvidSP) (1 March 1974 to 12 April 2017), Medline (OvidSP) (1946 to present), PubMed, Cochrane Central Register of Controlled Trials (Cochrane Library, Wiley) (issue 3 of 12, March 2017) and Cochrane Methods Register (CENTRAL) (Cochrane Library, Wiley) (issue 3 of 4, July 2017). Searches were then restricted to articles published in English since 2006. Full details of strategies are provided in Additional file 2. In addition, a database search for completed and ongoing studies was conducted at ClinicalTrials.gov (https://clinicaltrials.gov/). Studies which were not yet published ‘grey literature’ were not included in the search strategy.

Data collection

Papers identified from the ClinicalTrials.gov registry were imported into a MS Excel spreadsheet. Duplicates and studies which had less than 1000 participants were removed using a filter option. The selection of eligible papers followed a concordance strategy between two reviewers (RLB and DE) which ensured that concordance was > 95% (Fig. 1) [7].

Fig. 1
figure1

PRISMA flow diagram detailing the process of study selection and data extraction. HCI healthcare intervention, PTFU post-trial follow-up, RCT randomised controlled trial

Medical interventions were defined as an intervention that was consumed orally, inhaled, or administered by intravenous or intramuscular injections including vaccines. A surgical intervention was defined as any intervention which was invasive (apart from those mentioned above and including blood transfusions). Potential studies were checked for eligibility by two reviewers who initially reviewed abstracts and then proceeded to full paper review in a step-wise process (Fig. 1).

In addition to those described in the protocol, some additional exclusions which were not originally listed were identified during the process of performing the systematic review in keeping with our definition of PTFU. This was required due to the heterogeneity of PTFUs. These include: (1) trials that were stopped before the scheduled closure of the trial; (2) cancer trials which had an open endpoint (e.g. survival as an endpoint with no clear scheduled plan of duration); (3) trials which continued with active intervention in the PTFU period with the primary outcome of safety and (4) trials eligible for inclusion but which did not contribute novel data as they only published additional subgroup or post-hoc analyses. A table of excluded trials is provided in Additional file 3.

Full papers deemed eligible for inclusion in the systematic review were extracted using a standardised Excel spreadsheet. Data was extracted by DE and RLB and concordance was checked. Primary outcome, healthcare intervention and attrition rates were tabulated for each study. Lead trialists were contacted via email to inform them of the systematic review and to clarify information where necessary. The papers included in the review were diverse with a range of interventions and different outcome measures. Due to the high level of clinical heterogeneity a meta-analysis was not possible.

Retention rates were calculated as the proportion of participants who were lost to follow-up compared to the overall total of those who started the PTFU period. Information about the cost of the PTFU was sought from study publications or via personal communication. Two attempts were made to contact the trialist via email and if there was no response or inadequate data, the trial was excluded from the cost analysis.

Assessment of risk of bias

Risk of bias was assessed for each included RCT on their primary results using the Cochrane Risk of Bias tool. Covdence.org was used to assess the levels of bias (low risk, high risk or unclear risk) in each methodological domain (sequence generation, allocation of sequence concealment, blinding, incomplete outcome data, selective reporting bias and other bias) and decisions checked by one of the senior authors [8]. The data recorded from Covidence.org was imported into Review Manager 5 (RevMan 5) for graphical representation [9].

Results

From 57,352 papers identified, 65 studies with PTFU were included in the systematic review (Fig. 1). Fifty trials involved medical interventions and 15 involved surgical interventions. There were no eligible psychiatric trials which had (all > 1000 participants). The duration of PTFU ranged from 1 to 20 years, with a median of 4.5 years of follow-up. The number of participants followed during the post-trial period ranged from 575 to 29 862.

Five methods of follow-up were identified: postal correspondence/questionnaire (19%); clinic appointments (35%); telephone interviews (26%); electronic data linkage (52%); and review of paper medical records (26%). In addition, in individual cases, specific follow-up was performed, e.g. endoscopy follow-up only [10]. Electronic data linkage and medical records review were used exclusively together in 11% of papers; either were used in combination with other methods in 74%. Overall, 48% of trials used more than one method to follow-up participants in the post-trial period (Tables 1 and 2). On average, two methods were used for each PTFU follow-up. Where data linkage was used, it was not always feasible to follow up all participants [11]. Some trials experienced difficulty accessing national electronic data in certain countries; for example, stricter regulations are apparent in Canada and for some North American participants (Medicare and Veteran Affairs) where a specific health ID number is required to access national data (Table 3). Trials experienced difficulty in accessing routinely collected health records in 3% of included papers and PTFU was restricted to those countries with robust and accessible centrally held records and registries (e.g. Sweden and Scotland) [12, 13].

Table 1 Post-trial follow-up (PTFU) in eligible medical trials. Note retention of participants expressed as % lost to follow-up
Table 2 Post-trial follow-up (PTFU) in eligible surgical trials. Note, retention of participants is expressed as % lost to follow-up
Table 3 Registries used for data linkage during post-trial follow-up (PTFU)

Retention rates

Unfortunately, retention rates were often poorly reported in the PTFU, limiting the ability to assess the impact of methods used in relation to the proportion lost to follow-up.

All surgical trials investigated mortality as the primary outcome and, where data was available, the proportion of participants lost to follow-up in surgical trials ranged from 0.4 to 15.5%. However, data was not available for 53% of surgical trials. In medical trials, the primary outcome investigated varied more widely, although mortality as an endpoint was common and the proportion of participants lost to follow-up ranged from 0 to 22%. Data on loss to follow-up was not available in 44% of trials. Where mortality was the primary outcome, the number of participants lost to follow-up was not available in 32% of trials due to the use of mortality records where only notifications of deaths were fed back to the trialists.

Cost

Financial information was available for one third of the included trials. Consequently, it was not possible to provide a direct comparison between cost of PTFU and the different methodologies used due to the small sample size. The cost of PTFU ranged from £6000 to £14,600,000 (Table 4). Cost of PTFU per participant per year showed that IST-3 was the most economical costing £0.21 per participant per year using data linkage/medical records, closely followed by ‘Over 50s’ (£0.41) and RECORD trials (£0.46) which also used data linkage. LTPS was the most expensive PTFU per participant per year (US$531.53) using clinical appointments and telephone follow-up. ROADMAP which also followed up participants by clinic appointment only had a cost of €413.60 per participant per year.

Table 4 Comparing post-trial follow-up (PTFU) costs (where disclosed), by different follow-up methodologies

Cochrane Risk of Bias

We hypothesised that those RCTs which had poor methodology or ‘high risk of bias’ might subsequently have a PTFU which was poorly organised and, therefore, have low retention rates (or a high proportion lost to PTFU). Of the 65 papers included in the systematic review, seven were excluded from the risk of bias assessment: these were PTFUs which followed-up an amalgamation of data from more than one trial or were part of a systematic review and, therefore, not suitable to be included in the analysis (the risks of bias from individual component trials could not be combined).

Of the 58 trials considered, the risk of bias could not be fully assessed in 11 trials due to lack of information in at least one domain. Low (or unclear) risk of bias in all domains was found in 43 (74%) of those assessed. Only seven trials (12%) had at least one domain which was high risk of bias, of which one had two domains at high risk (Fig. 2). Details of the individual risk of bias domains for each included study are provided in Additional file 4.

Fig. 2
figure2

Cochrane Risk of Bias graph. Review authors’ judgements about each risk of bias item presented as percentages across all included studies

Given the small number of trials found to have a high risk of bias in at least one domain and the highly variable retention rates observed during PTFU (Table 5), it is not possible to draw any clear conclusions with respect to possible associations between risk of bias and its potential impact on the proportion of participants that were lost to follow-up in the post-trial period.

Table 5 Comparison of randomised controlled trials (RCTs) which had high risk of bias compared to the proportion lost to follow-up during post-trial follow-up (PTFU). A summary of those RCTs with no risk of bias are also detailed

Discussion

This systematic review identified that PTFU methods varied and many trials used overlapping approaches which were more costly than needed. Data was limited on retention rates and so it was difficult to draw any firm conclusions on which method was best for PTFU.

Our main findings suggest that most PTFU published in the last 11 years does not appear to be designed in a cost-effective manner. Cost of PTFU was shown to vary widely and not many trials used incentives to retain participants. Despite only a third of trialists providing complete financial information for PTFU, follow-up by clinical appointment appeared to be the most expensive method, as might be expected given the resource implications. Postal or telephone correspondence in addition to data linkage did not appear to increase the cost per participant per year considerably. However, the effect of inflation over the 11 years included in this systematic review, makes quantitative comparison of cost differences difficult. Given the limited data available we have not attempted to adjust for inflation.

Data linkage or access to medical records is likely to be the most cost-effective method of following participants due to minimal staff required. However, a number of trialists highlighted the limitations of this approach, noting it to be time-consuming and frustrating with increasing regulatory costs and country-specific restrictions. In the UK, the process of accessing data electronically has become more stringent and costly, and markedly different to the processes which were encountered 10 years ago. There is also an issue of the data lagging behind by up to 2 years in some countries, which can impact on the completeness of results for a trial. Despite this, data linkages to national registries and electronic health records have been shown to be a valid and reliable method of PTFU [12,13,14,15].

When designing this systematic review, we anticipated that papers published in the early half of the last 11 years would choose more traditional methods of PTFU, e.g. clinic- and telephone-based approaches, and more recent trials may increasingly use data linkage where available. However, this has not been the case. The majority of trials have used a variety of different methods to capture data for the same primary outcome. We were, therefore, unable to compare retention rates by each type of method used. In addition, sparsity of complete data in the review (typically poor reporting of the final number of participants at the end of the follow-up period) limited the ability to assess retention rates achieved with different PTFU methods.

We found limited evidence of high risk of bias in the methodology of the in-trial periods. A likely explanation for this is that the majority of the trials included in this review were well-designed, large RCTs in which results were published in high-impact journals. Furthermore, trials which employ poor methodology or have had negative results are more likely not to engage in PTFU due to lack of funding or interest.

Due to new guidelines (Consolidated Standards of Reporting Trials (CONSORT)) recommending increasing transparency in the reporting of RCTs, a more complete capture of data would be likely for any future study [16]. Research into appropriate methods in PTFU can only occur if there is transparency of the logistical and financial implications including number of participants lost to follow-up.

Conclusions

Post-trial follow-up of large RCTs can contribute significantly to the scientific value of a trial by determining the longer-term magnitude of the effects of an intervention. PTFU is valuable to ensure that there are no long-term hazards or beneficial effects which have been missed due to the common short in-trial periods for following up participants. However, it is not widely used as shown by the small number of eligible trials which had PTFU from the original search strategy.

Data linkage and the use of registries appear to be the most plausible and economical approach to PTFU. These methods also have the advantage of providing data for a wide range of endpoints. Improvement of electronic reporting and informatics could lead to better reporting and allow this type of method to be widely used.

Abbreviations

CENTRAL:

Cochrane Central Register of Controlled Trials

EMBASE:

Excerpta Medica database

PTFU:

Post-trial follow-up

RCT:

Randomised controlled trial

References

  1. 1.

    Ford I., et al., Long-term safety and efficacy of lowering low-density lipoprotein cholesterol with statin therapy: 20-year follow-up of West of Scotland Coronary Prevention Study. (1524–4539 (Electronic)).

  2. 2.

    Heart Protection Study Collaborative Group, et al. Effects on 11-year mortality and morbidity of lowering LDL cholesterol with simvastatin for about 5 years in 20,536 high-risk individuals: a randomised controlled trial. Lancet. 2011;378(9808):2013–20.

  3. 3.

    Fergusson D, et al. Post-randomisation exclusions: the intention to treat principle and excluding patients from analysis. BMJ. 2002;325(7365):652–4.

  4. 4.

    Brueton VC, et al., Strategies to improve retention in randomised trials: a Cochrane systematic review and meta-analysis. (2044–6055 (Linking)).

  5. 5.

    Barton J, Young A, Lay M., Introduction of electronic data capture method using participant-completed online web-based follow up questionnaire in mail-based study achieves expected benefits and positive participant feedback. Trials, 2015. 16(2): p44.

  6. 6.

    Scuffham P, Chaplin S, Legood R. Incidence and costs of unintentional falls in older people in the United Kingdom. J Epidemiol Community Health, 2003. 57(9): p. 740.

  7. 7.

    Llewellyn-Bennett R, Bowman L, Bulbulia R Post-trial follow-up methodology in large randomized controlled trials: a systematic review protocol. Systematic Reviews, 2016. 5: p. 214.

  8. 8.

    Covidence systematic review software, Veritas Health Innovation, Melbourne, Australia. Available at www.covidence.org.

  9. 9.

    (RevMan), R.M., [Computer program] Version 5.3. Copenhagen: The Nordic Cochrane Centre, The Cochrane Collaboration. 2014. http://community.cochrane.org/tools/review-production-tools/revman-5/about-revman-5. Accessed 30 Dec 2017.

  10. 10.

    Arber N, et al. Five-year analysis of the prevention of colorectal sporadic adenomatous polyps trial. Am J Gastroenterol. 2011;106(6):1135–46.

  11. 11.

    Margolis KL, et al. Long-term follow-up of moderately hypercholesterolemic hypertensive patients following randomization to pravastatin vs usual care: the Antihypertensive and Lipid-Lowering Treatment to Prevent Heart Attack Trial (ALLHAT-LLT). J Clin Hypertens (Greenwich). 2013;15(8):542–54. (1751–7176 (Electronic)).

  12. 12.

    Hornslien AG, et al. Effects of candesartan in acute stroke on vascular events during long-term follow-up: results from the Scandinavian Candesartan Acute Stroke Trial (SCAST). Int J Stroke. 2015;10:830–5. https://doi.org/10.1111/ijs.12477.

  13. 13.

    Lloyd SM, et al. Long-term effects of statin treatment in elderly people: extended follow-up of the PROspective Study of Pravastatin in the Elderly at Risk (PROSPER). PLoS ONE [Electronic Resource]. 2013;8(9):e72642.

  14. 14.

    Kivimaki M, et al. Validity of cardiovascular disease event ascertainment using linkage to UK hospital records. Epidemiology. 2017;28(5):735–9.

  15. 15.

    Herbert A, et al. Data Resource Profile: Hospital Episode Statistics Admitted Patient Care (HES APC). LID – https://doi.org/10.1093/ije/dyx015 [doi] FAU – Herbert, Annie. (1464–3685 (Electronic)).

  16. 16.

    CONSORT Transparent Reporting of Trials. Available from: http://www.consort-statement.org/. Accessed 30 Dec 2017.

Download references

Acknowledgements

Thank you to all corresponding trialists.

Funding

RLB has received funding from the Royal College of Surgeons of England Research Fellowship.

Availability of data and materials

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Author information

RLB designed, carried out the systematic review including screening, data capture, data analysis, interpretation of results and wrote the paper. DE screened the papers from the search strategy and identified relevant papers including checking 10% of data from the extraction stage. NR assisted in the design of the search strategy and completed the search strategy. RB, LB assisted in the search strategy and in decisions relevant to the review. LB and RB assisted in drafting the review. AH assisted in the discussions of the review and drafting. All authors read and approved the final manuscript.

Correspondence to Rebecca Llewellyn-Bennett.

Ethics declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

RLB, DE, NR, AH, LB and RB consent for publication.

Competing interests

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Additional files

Additional file 1:

PRISMA Checklist. (DOCX 30 kb)

Additional file 2:

Search strategies. Key to operators used in Medline/Ovid: where .pt. is publication type, (?) represents any single character, (*) is a group of characters, .mp.is multi-purpose search, /is Medical Subject Headings (MeSH), exp. is explode subject heading, .sh. is subject heading, (““) is phrase search. Comments: all results were downloaded with all fields displayed and in a tab delimited format. This file was then opened in Excel. Duplicates were removed. The spreadsheet sort order was changed to Enrollment A–Z and studies with fewer than 1000 enrollees will be removed. (PDF 396 kb)

Additional file 3:

Trials with long-term follow-up excluded from final analysis. *open-label study investigating safety doses of intervention. Extension study of two previous RCTs (Philipp T et al. Clin Ther 2007; 29:563–80). (PDF 147 kb)

Additional file 4:

Risk of bias shown in each domain for an individual randomised controlled trial (RCT). Red indicates high risk, yellow indicates unsure and green indicates low risk. (PDF 173 kb)

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Llewellyn-Bennett, R., Edwards, D., Roberts, N. et al. Post-trial follow-up methodology in large randomised controlled trials: a systematic review. Trials 19, 298 (2018) doi:10.1186/s13063-018-2653-0

Download citation

Keywords

  • Methodology
  • Post-trial
  • Retention
  • Randomised controlled trial
  • Cost
  • Long-term
  • Follow-up
  • Effective

Comments

By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate. Please note that comments may be removed without notice if they are flagged by another user or do not comply with our community guidelines.