I was recently asked to step in at two Café Scientifique talks for the London based charity Sense About Science to present the talk: Asking for Evidence (AfE) as part of their campaign of the same name. In a culture where the “questioning of authority isn’t always welcome” as Director of Sense about Science Tracey Brown remarks, how can we know that products, companies and MPs will deliver on their claims? We don’t! Unless of course we quite literally ask for evidence!
However questions posed after the talks literally kept me awake that night. Together these questions form part of a culture where actually the evidence itself is questioned and not trusted. In truth, this culture needs changing and is one that many want to change, and are starting to.
I have somewhat condensed and re-worded my favourite (and yet I think the most significant) questions to a punchy one-liner, and while my answers here are not the ones I gave at the time of being asked, they certainly prompted further discussion and hopefully will give you plenty of food for thought!
Question 1: How can we trust evidence when the peer review is corrupt?
The idea that the peer review system is corrupt began in the 19th century. In 1845, the deterioration of the system had already begun and in 1903 there was an inquiry into it (Nature). The issue of peer review was rooted within anonymity, the idea of which came from English periodicals at the time, where an anonymous person speaks for the group that they represent, and has full authority to do so reflected in the phrase “ex cathedra”. These anonymous reviewers’ bring their “envy, hatred, malice and all uncharitable-ness” (pg 308 Nature) preventing good science from being published. These days however there are calls for examinations of the system and a better awareness. For instance the pressure to publish has become a significant factor in reducing research quality (Nature and The Independent). So yes, while we recognise that the system is broken, measures can be taken and we can promote them.
Question 2: How can we tell if evidence is affected by us humans?
Scientists are humans and as the saying goes “to err is human.” Scientists can bring much into the scientific process starting with their personal agendas, be it further grant money, fame and other incentives that institutes bring. So when bad science is called out because it is being funded by corporations whose bottom line flies in the face of good evidence, the question that everyone starts asking is “who funds this?” The extreme of this is when people start assuming all corporations are evil. Tracey Brown offers up suggestions in her Guardian article on how to make these motives more transparent.
Flawed methodology and interpretation can subsequently arise from personal agendas…sometimes. In some cases, scientists may simply be ignorant. Reviewers for instance don’t always spot statistical errors (The Independent). The use of statistical significance in research papers has become such an issue that it is now described as “polluting the literature” (The Lancet). The standard of methods can differ between journals therefore better papers have more of a chance in getting into Nature than PLoS for instance. But even flawed methodology can arise from personal agendas; methods such as modelling can be fine-tuned to paint a picture that is in the best interests of the scientist. Other bias that requires mentioning is cherry-picking and even sub-conscious bias can lend a hand. This issue has become such so entrenched in evidence that Nature has brought this issue directly to the scientists. All of this combined leads to the idea that most of what is published is actually incorrect and busts the notion that science is a self-correcting mechanism. But with this awareness comes suggestions for it (The Economist and The Lancet).
Question 3: What about people that refuse evidence because of an emotional investment?
This is a hard one but it is also a classic! Pick your favourite controversial subject: Genetically modified organisms (GMO’s), anti-vaxxers? Mine is GMO’s. The evidence shows that they are not harmful if eaten (Making Sense of GM), nevertheless emotions run high as people believe the sensationalism that has emerged over many years. In terms of dealing with this, I have my own personal opinions that were formulated by reading the book The Honest Broker by Roger A. Pielke. While describing how emotional values often cloud scientific values in the policy making process, I matured to the fact that these emotions need to acknowledged and not dismissed rudely, which I have witnessed by some scientists and even by some scientifically-inclined members of the public! I think it is important to address these issues in a respectful manner and Tracey summed up my personal thoughts on this eloquently within one sentence at The Annual Sense About Science Lecture 2015 The Ugly Truth: “How can you get the public to trust you when you don’t trust them?”
Questions 4: What do we do/ who should we believe when there is conflicting evidence from two opposing sides?
I have a relatively short answer to this: get an independent expert! This is also where I start to gush about the work that Sense About Science does as they represent that neutral ground where they seek the truth. All I can say is check out their website. In some cases, conflicting evidence still exists despite the science falling on one side quite dominantly (usually because someone is still trying to enforce their own agenda) e.g. climate change, anti-vaxxers, GMO’s. The independent expert exists to dispel the inaccurate evidence and put the debate to rest. But what about the areas within science that are still in debate? Well as it was pointed by one at the Headingley Café Scientifique, the best thing to do is provide the current evidence without bias, personal agendas and flawed methodology at that specific moment in time and watch the evidence evolve.
Altogether, the process of peer review, the human element and the emotions that comes with it, and the conflicting and evolving evidence combine together showing that evidence is not a black and white picture that some may claim. I’m sure that there are other factors I have not included here that contribute to this cloudy image of evidence. However the key point I am trying to make here is that evidence is not just about correcting or presenting specific scientific issues, but that we first to need understand the complexity and richness of evidence itself.