Research Quality During the Pandemic

Leonhard Held, you’re a biostatistician, and Huldrych Günthard, you’re an infectious disease specialist. How has Covid-19 affected your working life?

Huldrych Günthard: It has thrown my day-to-day life completely off balance. As the chief of service for infectious diseases at USZ, I was kind of bowled over by the new virus. We’ve treated a large number of patients who are seriously ill with Covid. It has barely left me any time for my HIV research. But now we’ve started our own Covid research project, which is supported by the Swiss National Science Foundation. On top of that we’ve had loads of media inquiries. I’ve never experienced anything like it – the need for information is huge. I’m in something of a unique situation as my wife, Alexandra Trkola, heads up the UZH’s Institute of Medical Virology. So we’re both pretty snowed under at the moment (laughs): Her in the laboratory, me at the hospital – our lives pretty much revolve around coronavirus right now.

Have you been working on the front line treating patients?

Günthard: Yes, I decided to fully dedicate myself to clinical practice. We’ve also carried out various studies looking at how to improve the treatment of coronavirus patients. I’ve seen between 200 and 300 Covid patients and am now very familiar with this disease.

Leonhard Held, how has Covid-19 been keeping you busy?

Leohnhard Held: Unlike Huldrych Günthard, I’m not a clinician, so I have huge respect for those like him who are still carrying out research alongside their clinical practice at this time. One of my research interests is statistical methods in infectious disease epidemiology. As part of a project funded by the Swiss National Science Foundation, we are analyzing the impact of measures such as school and border closures. I’ve also compiled forecasts on hospital occupancy for the Zurich Department of Health, and evaluated scientific publications on Covid.

The pandemic has really fueled research and led to a flood of publications. What are the implications of this?

Held: Colleagues are talking about a tsunami or a second epidemic – the preprint epidemic. There are now tens of thousands of these preprints on Covid-19. Preprints are preliminary publications on research findings that are published before they have been reviewed by a scientific journal. The scientific quality of these publications therefore varies widely. There are some very good preprints out there that are useful and important, for example on vaccine development. In this context it was important that they were published very quickly and did not have to undergo a lengthy review process. Meanwhile, there have also been many poor preprints that in some cases even had to be retracted.

How do you navigate such an unmanageable amount of scientific information? Who brings order and decides what is good and useful, and what isn’t?

Held: I’m not sure there is much order (laughs). Anybody can publish a preprint online. Often they provoke little or no reaction, unless they are linked to a big name, in which case the publication will trigger a lively debate.

So what’s the point of these preprints if the quality is so variable and is often sub-standard – do they even benefit efforts to control the pandemic?

Günthard: I’m in two minds about this one: If the quality of the preprint is good, then it’s great because we can get the key information quickly and can deploy the findings in clinical settings when treating patients. But if the quality is poor, then of course they are useless. Personally, I look out for well-known and trusted names when navigating the preprint jungle. What I find tricky nowadays is that it’s common for researchers to announce a preprint with a press release. This used to be done when a paper was published in a first-class academic journal. Now, people broadcast the preprint to the world to get publicity and money for research. This is something I struggle with. Having said that, this sort of publication can sometimes make sense, if for example a new diagnostic method has been developed that makes it easier to detect SARS-CoV-2. An example of this is the broad-based saliva study in which we showed that saliva-based coronavirus tests are a suitable way of testing large groups of people for the virus. This has led to the method being rolled out for large-scale coronavirus testing in schools.

Leonhard Held, in an article you noted that many scientists publish preprints that are are sub-standard. Why is that?

Held: The issue is that many authors are either unaware of the publication standards or fail to comply with them. For example, there are established guidelines on how to present predictive models. We use such models to better assess the development of an infection, for example. But obviously many people don’t take any notice of these extremely useful guidelines and the result is sub-standard preprints.

Let’s look at a concrete example: Can we have confidence in the Covid-19 vaccines on the basis of current studies?

Günthard: I was enormously impressed with how rapidly the new vaccines were developed. This is something I wouldn’t have thought possible – first, the fact that scientists were even able to develop vaccines in such a short space of time, and second that the clinical trials on their efficacy and side effects progressed so quickly. We have some very reliable data on the efficacy of the vaccines. Obviously what’s missing are long-term studies. In terms of mRNA vaccines, I was skeptical at first as this is a new and little-studied method. But I then found out that there are long-term studies on the treatment of cancer patients using mRNA-based therapies. We therefore have more experience than I was initially aware of. So I would say that yes, we can have confidence in the available vaccines

Held: For the last few years, the US and European drug licensing authorities have had fast-track procedures in place that are linked to certain conditions, so that an active substance can be authorized quickly in situations where there is an urgent need. The authorizations are then based on one trial, but are contingent on other conditions being met. This applies to all the current authorizations for Covid-19 vaccines. In other words, the questions of whether there may be side effects, or whether the vaccines only work in certain age groups, are still being investigated. For the authorization of coronavirus vaccines, the authorities have been very flexible, which means we are now in the fortunate position of having several different vaccines at our disposal.

That’s justified – but is it tenable in terms of the reliability of the vaccines?

Held: Yes, it makes sense in situations when medical interventions are needed as urgently as they are at the moment. The evidence from the individual trials is already quite good. It’s remarkable that we have such good vaccines with 90 to 95% efficacy when you think that for other infectious diseases such as HIV, we still don’t have a vaccine.

Günthard: I think the licensing authorities have done a stellar job. They were not at all dogmatic. Authorization has been a rolling process. This was crucial in being able to grant authorization shortly after the trials were concluded. Many authorities have really pulled out all the stops during the pandemic – Swissmedic, the ethics committees, hospital management bodies. I’ve never seen anything quite like it. Processes that usually take two months suddenly only take two weeks. And yet nothing has been botched. This goes to show that if you’re well organized, these kinds of processes could also be quicker under normal circumstances.

What lessons can be learned from the pandemic in terms of research quality?

Held: As we have seen, the current system for evaluating the quality of research doesn’t work that well in an emergency situation. One example is the peer review, where academic work is assessed by independent experts. I’ve personally noticed that it’s increasingly difficult to find expert reviewers because they are all busy with other things and no longer have time to review papers seriously. The peer review model therefore needs to be closely looked at. One problem is that these days, scholars are measured based on what they publish ("publish or perish"). No one really cares whether they have been peer reviewers. We therefore need to make this activity more appealing by valuing reviewers and raising their profile so that expert reviews and therefore quality control can be carried out more swiftly in times like these.

Another suggestion to improve quality is that planned research projects should be reviewed before the research is even started. What would be the benefit of that?

Held: This is a new idea, which originally came from social sciences and is already very popular in psychology. You write a proposal on a research project and this is reviewed by an expert. More and more journals are doing this. They then commit to publishing the subsequent results, provided of course that the initial review was positive. One advantage of this model is that negative results are increasingly published. This is somewhat rare in the traditional review model. The drawback is that there is a delay because the report must first be reviewed. While this may not be particularly helpful during a pandemic, the quality of the research will probably be enhanced in the end.

Günthard: That’s true. At the same time, it’s crazy how we want to regiment everything now. Particularly in exploratory studies which do not involve drug development, we have to be careful to ensure that the processes do not become completely inefficient and that we don’t completely take away the joy of research.

Leonhard Held, you lead the Center for Reproducible Science (CRS) at UZH. How can the CRS contribute to improving quality in research?

Held: We offer regular courses in good research practice for researchers, and doctoral and post-doctoral students. We therefore teach the foundations of good research. The interdisciplinary nature of the CRS allows us to exchange different approaches and research methods to ensure quality.

Some researchers have expressed controversial views about the pandemic in the media. This sometimes gives rise to confusing contradictions. Do such disagreements erode trust in science?

I don’t believe there has been a loss of trust. Despite a few negative examples, scientists have done a great job during the pandemic all over the world. I also think the media discourse has been good. I’ve rarely seen such engaged and intensive public debate on scientific issues. And this has been at a very high level, which I think is positive. But it’s dangerous when experts get carried away and say things that we don’t know for certain. Saying that we don’t yet know something is quickly dismissed as boring by the media. But that shouldn’t incite researchers to present suppositions as fact when the data are lacking.

Held: I agree. The problem has been that the uncertainties that (always) exist in scientific research are in some cases no longer communicated. Even now, there is still a great deal we don’t know about Covid-19; researchers should make that clear in their statements.

Do the public understand that?

Held: Getting this fact across is a major problem and a major challenge. We as scientists still have some work to do here. Incidentally, a number of studies have been carried out on the topic of loss of trust. They have shown that trust in science hasn’t declined during the course of the pandemic, but has actually increased.

Günthard: I’d say it’s mainly politicians who have a trust problem at the moment. But of course scientists aren’t perfect either – at the end of the day, no-one is.

Will science come out of this crisis stronger or weaker?

Günthard: I think stronger. As I said before, the achievements of scientists and manufacturers in developing vaccines have been amazing – both in terms of quality and speed.

Held: I can only agree. And this also applies to clinical trials for Covid-19 treatments, such as the Recovery trial led by Oxford University, which has already delivered some hugely valuable insights into which Covid therapies are beneficial and which are not. This is a great example of how effective science can be in times like these.