Registered report

Learning outcomes

Learners ...

• understand what a registered report is
• understand the goal of the learners project
• create an informal design document

For teachers

Prior:

• How do you design your research project?
• How does an ideal program looks like?
• What is a registered report?

What is a registered report?

It is research that is peer-reviewed twice, where the first peer-review takes place before doing an experiment. The paper before the experiment (with its hypotheses, methods, ways to draw a conclusion) is made public upon publication.

Why use a registered report?

• to make science more reproducible [Munafò et al., 2017]
• to report null findings more honestly [Allen and Mehler, 2019]
• to do better science [Soderberg et al., 2021]

How are null findings reported more honestly?

Figure 3 in [Allen and Mehler, 2019] shows this difference, between 'traditional' papers and registered reports:

One can see that traditional papers report a null finding around four times less often.

How much better is the science?

Figure 3 in [Soderberg et al., 2021] shows this difference, between 'regular' papers and registered reports:

In this experiment, researches selected 'regular' papers and registered reports. The researchers then asked reviewers to score papers on the feature shown in the figure, where these reviewers were unaware of this experimental variable.

The values are in the standardized effect size metric, i.e. a value of 1.0 means that this value is 1.0 standard deviations higher.

How does a registered report blend with this course?

A registered report is a more informal form of design, that is close to what scientists are familiar with, that fits perfectly well with the phase of the course.

An example draft registered report

Introduction: industrialization may have an impact on the climate

Hypothesis: the average yearly temperature remains the same

Methods: use a Kolmogov-Smirnov test to test if the first/earliest third of the temperatures come from the same underlying distribution as the last/latest third of the temperatures, using an alpha value of 0.05.

Conclusion: if the distributions are similar, the average yearly temperature has remained the same. If the distributions are different, it means that industrialization may have changed temperatures

Discussion: this is only a correlational study, with no direct link between temperature and industrialization being measured

Exercises

Exercise 1: the ideal program

In this course, we will create a Python package for a research project. This package should create all texts, tables and figures needed for publication.

What would be your ideal use of this Python package? That is, how would you ideally document the Python code on how to (re)do the experiment?

Answer

This answer is personal.

However, I (Richel) think the following use is close to ideal:

import weather
weather.do_experiment()

I use weather as the name of the Python package, but any reasonable name will do.

I think this code is ideal, as it is both readable and short.

Do you agree with the answer?

Exercise 2: output from the ideal program

Assume an ideal program to do the experiment described at 'an example draft registered report'.

What should this program produce at least? Can you imagine other things that may be produced that can be used in the final paper?

Answer

It should at least produce:

• The p value of the statistical experiment

It may also produce:

• A figure showing the two distribution of weather temperatures (i.e. the first/earliest third and the last/latest third), with an indication if these distributions differ

Exercise 3: your draft registered report

In an earlier exercises, you've written down a hypothesis.

In this exercise, expand on this hypothesis and turn it into a draft registered report. Make the report as minimal as possible: it should serve as an informal design document, not be a full journal submission. You are allowed to make things up if needed.

Save your results in the learners project, in the learners folder.

References

• [Allen and Mehler, 2019] Allen, Christopher, and David MA Mehler. "Open science challenges, benefits and tips in early career and beyond." PLoS biology 17.5 (2019): e3000246. Paper homepage

• [Chambers, 2019] Chambers, Chris. "What’s next for registered reports?." Nature 573.7773 (2019): 187-189. Comment (i.e. it is not a paper) homepage

• [Munafò et al., 2017] Munafò, Marcus R., et al. "A manifesto for reproducible science." Nature human behaviour 1.1 (2017): 0021. Paper homepage

• [Soderberg et al., 2021] Soderberg, Courtney K., et al. "Initial evidence of research quality of registered reports compared with the standard publishing model." Nature Human Behaviour 5.8 (2021): 990-997. Paper homepage