Skip to main content

Table 3 Themes in responses from journals

From: COMPare: a prospective cohort study correcting and monitoring 58 misreported trials in real time

Theme and subthemes

Quote

Issue

Conflicts with CONSORT

Failure to recognise that post-commencement changes are acceptable under CONSORT but should be declared in the paper reporting the results of the trial

“On the basis of our long experience reviewing research articles, we have learned that pre-specified outcomes or analytic methods can be suboptimal or wrong” “Although pre-specification is important in science, it is not an altar at which to worship… [COMPare’s] assessments appear to be based on the premise that trials are or can be perfectly designed at the outset… and that any changes investigators make to a trial protocol or analytic procedures after the trial start date indicate bad science” (Annals Editors critique, 01/03/16).

COMPare uses CONSORT as the gold standard. CONSORT item 6b requires that trial reports declare and explain “any changes to trial outcomes after the trial commenced, with reasons” in the paper reporting the results of the trial. Changes are not forbidden; however, they should be declared in the trial report.

Stating that outcome switching doesn’t matter if the main results of the study are unlikely to be affected

“We reviewed materials associated with the articles and concluded that the information reported in the articles accurately represented the scientific and clinical intent detailed in the protocols... We found no inconsistencies between the audited articles and their related protocols that would justify changes in trial interpretation, corrections, or warnings to readers” (Trial 45, Annals, 06/04/16).

CONSORT requires all outcomes to be correctly reported; it does not distinguish between circumstances when this would, or would not, affect the overall interpretation of the intervention being trialled. It is unlikely that all outcome misreporting would change the direction or size of an overall finding; however, a culture of permissiveness around correct outcome reporting does permit misrepresentation more broadly.

Statement describing journal practices that contradict CONSORT guidance

“We view each piece individually and add the data as appropriate based on the judgment of the peer reviewers, the statistical reviewers, and the editors” (NEJM emails 1, 17/11/15).

CONSORT item 6b requires that trial reports declare and explain “any changes to trial outcomes after the trial commenced, with reasons” in the paper reporting the results of the trial.

Statement that failure to report pre-specified secondary outcomes is not of interest

“We will not ordinarily consider letters that simply... point out unpublished secondary outcomes” (JAMA emails, 09/12/15).

Denial of endorsing CONSORT, despite appearing on CONSORT’s list of endorsing journals

“The New England Journal of Medicine finds some aspects of CONSORT useful but we do not, and never have, required authors to comply with CONSORT” (NEJM emails 1, 17/11/15).

 

Timing of pre-specification

Dismissal of pre-commencement registry data as “out of date”

“The initial trial registry data… often include outdated … entries” (Annals Editors critique, 01/03/16).

“Registries... do not routinely monitor whether the data in the registry match the protocol, and may not be updated when the protocol changes. We therefore rely primarily on the protocol” (Annals Editors critique, 01/03/16).

The statement that registry data are “outdated” may reflect a broader misunderstanding about the need for outcomes to be pre-specified pre-commencement. Where the registry entry is the only accessible source of pre-specified outcomes, discrepancies should be declared as per CONSORT 6b. Even if there is a contemporaneous protocol that is not publicly accessible, the pre-specified outcomes in this protocol should match its registry entry; if not, then there are two sets of discrepant pre-specified outcomes, which requires declaration and discussion. Of note, for one trial [Trial 57, Annals, 03/05/16], we found three different sets of pre-specified outcomes in two registries (EUCTR and ClinicalTrials.gov) and one protocol from the same time period.

Stating or implying that pre-specification after trial commencement is acceptable

“We disagree with COMPare’s contention that registry data are superior to protocol information because of the timing of the former ...” (Trial 45, Annals, 06/04/16).

COMPare used pre-commencement outcomes from registry data only as a last resort when they were not available from a pre-commencement protocol. Pre-specification of outcomes should take place before trial commencement. CONSORT item 6b requires that trial reports declare and explain “any changes to trial outcomes after the trial commenced, with reasons” in the paper reporting the results of the trial.

Registries

Dismissal of registry data as unreliable

“We check the registries, but as both authors’ responses attest, registry information can be incomplete or lack sufficient detail” (Trial 45, Annals, 06/04/16).

“The initial trial registry data… often include... vague or erroneous entries” (Annals Editors critique, 01/03/16).

“Registries include only extracted information” (Annals Editors critique, 01/03/16).

Publicly accessible trial registries are a cornerstone of trial transparency. Trialists are legally required to correctly register their trials; pre-specified outcomes are a required component under WHO guidance on trial registration; and ICMJE member journals commit to ensuring that trials are appropriately registered. Where the only source of pre-commencement outcomes contains information so imprecise that correct outcome cannot be assessed, we suggest that “inadequately pre-specified outcomes” be noted in the paper reporting the trial’s results, as this presents a similar risk of bias to misreporting of clearly pre-specified outcomes.

Stating that discrepancies between outcomes pre-specified in a registry entry and those reported in the paper are the fault of the registry

“Inaccuracies in the trial registration documents are more of an issue for the individuals overseeing the trial registries” (JAMA emails, 9/12/15).

“We will not ordinarily consider letters that simply note discrepancies with the trial registration” (JAMA emails, 09/12/15).

It is the responsibility of the journal and trialist to ensure that a trial is correctly reported, with discrepancies against outcomes pre-specified prior to commencement declared as per CONSORT 6b. If there are discrepancies between the outcomes pre-specified and the outcomes reported in the paper, then the paper is discrepant, not the source of pre-specified outcomes. If the pre-specified outcomes on a registry are inconsistent with those in a contemporaneous protocol, then there are multiple sets of pre-specified outcomes and therefore the outcomes have not been correctly pre-specified: this should be noted in the results manuscript.

Rhetoric

Stating that space constraints prevent all pre-specified outcomes from being reported

“Space constraints for articles published in the Journal do not allow for all secondary and other outcomes to be reported” (NEJM emails 1, 21/11/15).

The claim that space constraints prevent all pre-specified outcomes from being reported conflicts with the finding of COMPare, and prior research on outcome misreporting, that non-pre-specified additional outcomes were routinely added, in large numbers: a mean of 5.4 novel non-pre-specified outcomes were added per trial in COMPare (range 2.9–8.3 by journal).

JAMA: “authors are not always required to report all secondary outcomes and all pre-specified exploratory or other outcomes in a single publication, as it is not always feasible given the length restrictions to include all outcomes in the primary report” (JAMA emails, 9/12/15).

General statement about supporting goals of COMPare

“Though we share COMPare’s overarching goals to assure the validity and reporting quality of biomedical studies, we do not agree with their approach” (Trial 44, Annals, 15/12/16).

All such statements were accompanied by caveats, statements that explicitly or implicitly undermined the journals’ commitment to CONSORT, or incorrect statements about specific data points.

“While the goal of the COMPare project (http://www.compare-trials.org) is noble, my colleagues and I have outlined concerns with COMPare’s approach (1)” (Trial 45, Annals, 06/04/16).

Statements about journal processes

Statement that authors are required to declare changes to outcomes

“When the review process generates requests for authors to report outcomes not specified in the protocol or the authors choose themselves to present such outcomes, we ask authors to indicate these as post hoc or exploratory analyses” (Annals Editors critique, 12/02/16).

We cannot verify whether Annals ask authors to do this; however, we can confirm that trials reported in Annals are routinely non-compliant with CONSORT, a finding which is consistent with previous research. COMPare found that, in Annals trials, only 6% of novel non-pre-specified outcomes added to trial reports were correctly indicated by the Annals manuscript as novel; a mean of 6.4 novel undeclared outcomes were added per trial; 44% of primary outcomes were correctly reported; and 31% of secondary outcomes were correctly reported.

“To be consistent with CONSORT recommendations, we ask authors to describe, either in the manuscript or in an appendix, any major differences between the trial registry and protocol, including changes to trial endpoints or procedures” (Annals Editors critique, 01/03/16).

Statement that journal has a process to ensure correct outcome reporting

“We carefully check for discrepancies between the protocol and the manuscript” (JAMA emails, 09/12/15).

We cannot verify JAMA’s internal processes; however, we can confirm that trials reported in JAMA are routinely non-compliant with CONSORT, a finding which is consistent with previous research. COMPare found that, in JAMA trials, 39% of novel outcomes added to trial reports were correctly indicated as novel; a mean of 4.1 novel undeclared outcomes were added per trial; only 82% of primary outcomes were correctly reported; and 70% of secondary outcomes were correctly reported.

“We agree that it is important for researchers to pre-specify primary and secondary outcomes before conducting a trial and to report outcomes accurately in their publications. In fact, we carefully monitor this during editorial review” (JAMA emails, 9/12/15).

Placing responsibility on others (for example, trialists or reader)

Stating that readers can see for themselves whether outcomes reported are discrepant with those pre-specified

NEJM: “Any interested reader can compare the published article, the trial registration and the protocol (which was published with the article) with the reported results to view discrepancies” (NEJM emails 1, 21/11/15).

COMPare found that accessing documents and assessing trials for correct outcome reporting took between 1 and 7 hours per trial.

Passing responsibility to trialists rather than journals or editors

The Lancet published 15 out of 20 letters, mostly with accompanying responses from trialists: the majority of author responses expressed further misunderstandings about what constitutes correct outcome reporting, as reported in the accompanying paper on trialists’ responses. The Lancet made no comment themselves [all correspondence]. We asked the journal to clarify their position in our follow-up correspondence: “Since The Lancet have a longstanding positive commitment to improving reporting standards, lead the REWARD campaign on research integrity, and endorse CONSORT, we would welcome their perspective on why undeclared outcome switching in PETIT2 (and others) was apparently not addressed prior to publication; whether they now view outcome switching as acceptable; or whether they disagree that it has happened here”. We received no reply and our letter was not published (Trial 9, Lancet, 05/02/16).

Where a journal is listed as endorsing the CONSORT guidelines on trial reporting, it is reasonable to expect that they will take responsibility for ensuring that trials are reported consistently with these guidelines.

Placing responsibility on trial registry staff

“Inaccuracies in the trial registration documents are more of an issue for the individuals overseeing the trial registries” (JAMA emails, 9/12/15).

As above, if there are discrepancies between the outcomes pre-specified and the outcomes reported in the paper, then the paper, not the source of pre-specified outcomes, is discrepant.

  1. References throughout are to the correspondence archive at COMPare-trials.org/data containing the full public correspondence on all trials, and all correspondence with editors, organised by trial ID and date, or journal name for general correspondence. Abbreviations: COMPare Centre for Evidence-Based Medicine Outcome Monitoring Project, CONSORT Consolidated Standards of Reporting Trials, EUCTR European Union Clinical Trials Register, ICMJE International Committee of Medical Journal Editors, JAMA Journal of the American Medical Association, NEJM New England Journal of Medicine, WHO World Health Organization