Hogeschool van Amsterdam

  1. Home
  2. Toolkit
  3. Finishing
  4. Outcome reporting

Outcome reporting: complete, transparent and timely

Misunderstanding statistics, dubious signals from reviewers and editors lead many of us to focus on ‘significant’ findings. This practice jeopardizes the scientific literature. Note: much of the underlying and motivating literature for this chapter is so-called research on research (or meta-research) performed on randomized clinical trials.[1] However, we propose that you apply the principles laid out in this chapter in all types of research, either quantitative or qualitative.

Why devote a chapter on outcome reporting?

It has been shown for randomized clinical trials, arguably the most strictly regulated type of research today, that quite often authors (1) do not report on the trial’s outcomes at all, (2) do not report all pre-specified outcomes, (3) switch the primary outcome (often related to statistical significance), (4) report trials with statistically non-significant (primary) outcomes much later, (5) silently add outcomes that were not pre-specified, and (6) do not report on adverse effects. [1-4] In addition, authors may highlight particular outcomes in particular subgroups, downplaying the original aims of the trial.[5] All these practices do or can distort the global body of evidence, may bias systematic reviews, and endanger patients through biased practice guidelines for healthcare professionals. It is likely that research less strictly regulated than clinical trials may be affected even more by these questionable practices of selective reporting of outcomes, endpoints or results.[6,7]

What is complete, transparent and timely outcome reporting?

Complete, transparent and timely outcome reporting may be defined as reporting, in the public domain, all pre-specified outcomes that were stipulated in the most recent version of the study protocol, reasonably soon after finishing the study’s (data) analyses, while adhering to the (data) analysis plan. Put differently, it refers to a reporting process that avoids the six harmful practices mentioned in the previous paragraph.

Research studies often have more than one objective. Trialists are usually expected to choose a single primary outcome (to guide sample size calculation), but the number of secondary outcomes often exceeds 10. In a sample of 67 clinical trials, the median number of pre-specified outcomes was 9 (interquartile range from 1 to 16; mean 13; range 1 to 84). [2] In that sample, on average, more than 5 outcomes were ‘silently added’ in the first publication containing results. That is, these outcomes were published without an explanation, although the study protocol had not pre-specified them.

How is complete, transparent and timely outcome reporting related to the chapters on study protocol, (statistical) analysis plan, and preregistration in this online manual?

Preregistration of study protocols (and analysis plans) is an element in the prevention of selective outcome reporting and non-reporting bias.[8,9] At least, proper preregistration enables readers to check if publications are in line with the original plans or any recent amendments thereof. We hope that researchers at our center of expertise will be inspired by the rationale behind our emphasis on complete, transparent and timely outcome reporting and avoid the related damage to the global scientific enterprise and healthcare.

When to report completely, transparently and timely? And what do we mean by ‘timely’?

As a general guidance, inspired by the American FDA Amendments Act (FDAAA) of 2007 and the Final Rule for Clinical Trials Registration and Results Information Submission, we recommend that you report all pre-specified outcomes within 12 months after data collection on the last participant was completed.

How and where to report completely, transparently and timely?

When the time for reporting has come, and this may be a sequence of several papers, return to (the most recent version of) your study protocol and make sure that you (eventually) report on all pre-specified outcomes including adverse effects. Report the outcomes in (a series of) open access publications and on the preregistration platform. Make sure that the information is identical in all outlets (no discrepancies between publications and online platforms). If you report outcomes that you have not pre-specified in the (most recent) study protocol, make sure that you carefully explain the reason(s) why you report these outcomes even though the study protocol does not mention them. If any such outcomes involve data collection activities not mentioned in the information to participants, you may have overlooked an important ethical requirement for which you may want to ask advice from a privacy officer or an ethics committee.

What if you have no (published) study protocol or data-analysis plan?

Use your conscience. Think about the popular definition of research integrity as ‘doing the right thing even when no one is watching’. Do not be seduced to data dredging or harking (hypothesizing after the results are known, also known as the Texan shooting range.

Discuss openly the absence of a publicly preregistered protocol or analysis plan in your limitations section. Present results of sensitivity analyses (re-doing your analyses using different, but plausible, methods or assumptions). And finally, do a better job in your next study.

What if I do participatory action research (PAR)?

Here are some recommendations on how to enhance transparent outcome reporting in PAR.

  1. Transparent Documentation: PAR projects can document initial intentions, questions, methods, and anticipated outcomes in a shared space. This doesn't restrict change but rather ensures a transparent record of it.

  1. Reflection Logs: Given that PAR is iterative, maintaining logs that capture reflections, decisions, and the rationale behind changes can prevent unintentional selection in outcome reporting. These logs provide a window into the decision-making process.

  1. Collaborative Data Analysis and triangulation: By involving multiple stakeholders in the analysis process, there can be a consensus on emergent themes and patterns. This collective wisdom can minimizing the risk of misinterpreting or selectively highlighting data.

  1. Iterative Feedback: Engaging the community continuously for feedback ensures that emerging findings are grounded in the lived realities of participants, thereby minimizing the risk of misinterpreting or selectively highlighting data.

  1. Third-party Reviews: Engaging external researchers or community members to review and challenge findings can provide an added layer of scrutiny.

  1. Open Process Sharing: Make the entire process — from planning, data collection, to analysis — transparent and available to all participants. This can serve as a method of self-checking and also allow for corrections if a participant feels their views have been misconstrued.

  1. Ongoing Engagement: Unlike traditional studies which have a defined endpoint, PAR often involves longer-term engagements with communities. This means that the validity and applicability of findings can be tested in real-world scenarios over time, allowing for adjustments based on practical outcomes.

What if editors or reviewers insist that you omit particular outcomes or those they deem uninteresting or not statistically significant?

In such cases we recommend that you resist such invitations and refer the reviewer(s) or editor to this chapter or to the literature cited in this chapter and explain why you prefer to report on all pre-specified outcomes. Feel free to ask assistance from Urban Vitality Open Science Support staff in writing a rebuttal letter via opensciencesupport@hva.nl.

References

  1. https://www.compare-trials.org/ , accessed on 9 August 2023

Published by  Urban Vitality 25 August 2023

Toolkit