From protoscience to proper science: The path ahead for reforming psychology
I have two confessions to make. The first is that there are times when I regret pursuing psychology. When I started out, nearly 20 years ago, it was because I found the idea of science intriguing. Here was a profession that stood apart from all others. The aim wasn’t to make a profit, or sell people things, or win arguments in a courtroom, but to add to the sum of human knowledge and help solve problems along the way. I was drawn to a number of sciences – immunology, biology, astronomy – but psychology was the most fascinating. What could be more profound than understanding the inner workings of our own minds?
As an undergraduate I would spend entire afternoons flicking through early 20th-century volumes of the Journal of Experimental Psychology in the basement of the university library. The authors of these dusty articles seemed intent on discovering new truths but they were careful and cautious in their writing. They rarely strayed beyond the data and signposted it when they did. They always highlighted limitations and made sensible suggestions for future research. They were honest brokers who embraced the Mertonian virtues of science.
Inspired by their example, I pursued it as a career. But about ten years in, I realised that the science being described in those old tomes wasn’t much like the work being done around me, and it wasn’t much like the work I was doing myself. Modern psychologists were playing a very different game.
Fresh concerns raised over academic conduct of major US nutrition and behaviour lab
Read moreOur mission wasn’t about truth seeking. How painfully naive I had been. It was about crunching through as many experiments as possible as quickly as possible, finding ways to make ambiguous data look beautiful, publishing frequently in prestigious journals, getting to work in so-and-so’s “rockstar” lab (which of course always produced beautiful results and published exclusively in said prestigious journals), winning large public grants, and basically getting as famous and powerful as possible. The longer I spent in psychology, the more I realised that the ethos of our forebears had been eroded down to a nub – the quest to scale Everest reduced to a ten-minute dash for cigarettes.
I began to question my career choices and toyed with idea of dumping academia altogether. But instead I decided to take the hulking great chip on my shoulder and turn it into a book. For better or worse, the Seven Deadly Sins of Psychology is what came of it.
Seven Deadly SinsMy aim with this book was to document the fundamental problems I see with research practices in psychology and how we can fix them. The seven sins, in turn, are bias, hidden flexibility, unreliability, data hoarding, corruptibility, internment and bean counting. They cover the full spectrum of academic practice, from the way we design and report experiments, to the way we handle fraud cases, to the bizarre ways we attempt to measure the quality of science and scientists.
Let’s take the first sin as an example. One major form of bias is publication bias: a well-known form of malpractice in which journals selectively publish results that are clear and novel, rejecting studies of equivalent quality that happened to produce negative or less conclusive findings. Because researchers must publish or perish, publication bias in turn drives researchers to engage in biased research practices to produce publishable results, regardless of whether those results are credible. One such routine practice is a form of hindsight bias in which an unexpected result (usually cherry picked out of a dataset) is written up as though the author predicted it from the beginning. Reinventing history helps authors create more compelling narratives, but such inferences are no different to randomly spraying a wall with a machine gun and then drawing a bullseye around whereever the bullets happened to land.
Sign up for Lab notes - the Guardian's weekly science update
Read moreOne of the best ways to guard against bias is study pre-registration: writing down in advance our study predictions, how we plan to acquire data, and how we plan to analyse it once we get it. In science it makes sense to treat our future self as a different person to our past self, and indeed to treat that person as a hostile entity. Past me may be genuinely interested in the answer to a question, but future me knows that I need to play the academic game to advance my career, and so will tempt me toward bias. Past me can help keep future me honest by pre-registering his intentions.
In turn, journals have the power to eliminate bias by deciding what gets published based on detailed study protocols, before results even exist. This new format of publication, called a Registered Report, breaks the cycle of bias and holds great promise for improving the reliability of published research. Even though Registered Reports began in psychology, they have now been adopted by journals in psychiatry, nutrition, computer science, political science, and many other fields. The 50th journal to launch them was BMC Biology, showing the potential for psychology to help formulate solutions in neighbouring disciplines.
Another confessionWhen I talk to psychologists about these problems, their stock response tends to be: “Hang on, none of these sins are unique to psychology, so why are you picking on us?” I understand the gripe. Many sciences indeed suffer from the deadly sins, from the social sciences right through to clinical trials. But a misery shared is little consolation for psychology. As psychologists we are supposed to be experts in human behaviour. We are supposed to be equipped with the knowledge and skills to deal with bias and malpractice, and to help other sciences overcome them. We were never supposed to be in thrall to them.
Which brings me to my second confession. As time goes by, I find myself having dwindling respect for my senior colleagues in psychology. So much has been said now about the reproducibility crisis, both in psychology and science in general, that none can honestly profess ignorance. And yet so many remain silent. I see these people much as I see my former self: experts at winning, lawyering their way through their academic careers; otherwise intelligent people cranking the handle in a broken machine. They don’t care if the system is broken because it seems to work for them. They don’t see how psychology is failing its public mission because their careers succeeded.
On the other hand I have a deep and abiding respect for senior psychologists who are facing up to the reality that we need to change the way we work, and I admire even more the growing ranks of younger scientists who are championing reform. They are chafing against an academic establishment that, far from rewarding their efforts, at times labels them as trouble-makers and terrorists. If reform succeeds it will, in large part, be a victory owed to those scientists who refused to be silenced and forced the powerful to pay attention. The overarching message of my book to them, as to all psychologists, is: stay inspired and keep shouting. Some of us, at least, are listening.
Source: https://www.theguardian.com/science/head-quarters/2017/may/09/from-protoscience-to-proper-science-the-path-ahead-for-reforming-psychology
As an undergraduate I would spend entire afternoons flicking through early 20th-century volumes of the Journal of Experimental Psychology in the basement of the university library. The authors of these dusty articles seemed intent on discovering new truths but they were careful and cautious in their writing. They rarely strayed beyond the data and signposted it when they did. They always highlighted limitations and made sensible suggestions for future research. They were honest brokers who embraced the Mertonian virtues of science.
Inspired by their example, I pursued it as a career. But about ten years in, I realised that the science being described in those old tomes wasn’t much like the work being done around me, and it wasn’t much like the work I was doing myself. Modern psychologists were playing a very different game.
Fresh concerns raised over academic conduct of major US nutrition and behaviour lab
Read moreOur mission wasn’t about truth seeking. How painfully naive I had been. It was about crunching through as many experiments as possible as quickly as possible, finding ways to make ambiguous data look beautiful, publishing frequently in prestigious journals, getting to work in so-and-so’s “rockstar” lab (which of course always produced beautiful results and published exclusively in said prestigious journals), winning large public grants, and basically getting as famous and powerful as possible. The longer I spent in psychology, the more I realised that the ethos of our forebears had been eroded down to a nub – the quest to scale Everest reduced to a ten-minute dash for cigarettes.
I began to question my career choices and toyed with idea of dumping academia altogether. But instead I decided to take the hulking great chip on my shoulder and turn it into a book. For better or worse, the Seven Deadly Sins of Psychology is what came of it.
Seven Deadly SinsMy aim with this book was to document the fundamental problems I see with research practices in psychology and how we can fix them. The seven sins, in turn, are bias, hidden flexibility, unreliability, data hoarding, corruptibility, internment and bean counting. They cover the full spectrum of academic practice, from the way we design and report experiments, to the way we handle fraud cases, to the bizarre ways we attempt to measure the quality of science and scientists.
Let’s take the first sin as an example. One major form of bias is publication bias: a well-known form of malpractice in which journals selectively publish results that are clear and novel, rejecting studies of equivalent quality that happened to produce negative or less conclusive findings. Because researchers must publish or perish, publication bias in turn drives researchers to engage in biased research practices to produce publishable results, regardless of whether those results are credible. One such routine practice is a form of hindsight bias in which an unexpected result (usually cherry picked out of a dataset) is written up as though the author predicted it from the beginning. Reinventing history helps authors create more compelling narratives, but such inferences are no different to randomly spraying a wall with a machine gun and then drawing a bullseye around whereever the bullets happened to land.
Sign up for Lab notes - the Guardian's weekly science update
Read moreOne of the best ways to guard against bias is study pre-registration: writing down in advance our study predictions, how we plan to acquire data, and how we plan to analyse it once we get it. In science it makes sense to treat our future self as a different person to our past self, and indeed to treat that person as a hostile entity. Past me may be genuinely interested in the answer to a question, but future me knows that I need to play the academic game to advance my career, and so will tempt me toward bias. Past me can help keep future me honest by pre-registering his intentions.
In turn, journals have the power to eliminate bias by deciding what gets published based on detailed study protocols, before results even exist. This new format of publication, called a Registered Report, breaks the cycle of bias and holds great promise for improving the reliability of published research. Even though Registered Reports began in psychology, they have now been adopted by journals in psychiatry, nutrition, computer science, political science, and many other fields. The 50th journal to launch them was BMC Biology, showing the potential for psychology to help formulate solutions in neighbouring disciplines.
Another confessionWhen I talk to psychologists about these problems, their stock response tends to be: “Hang on, none of these sins are unique to psychology, so why are you picking on us?” I understand the gripe. Many sciences indeed suffer from the deadly sins, from the social sciences right through to clinical trials. But a misery shared is little consolation for psychology. As psychologists we are supposed to be experts in human behaviour. We are supposed to be equipped with the knowledge and skills to deal with bias and malpractice, and to help other sciences overcome them. We were never supposed to be in thrall to them.
Which brings me to my second confession. As time goes by, I find myself having dwindling respect for my senior colleagues in psychology. So much has been said now about the reproducibility crisis, both in psychology and science in general, that none can honestly profess ignorance. And yet so many remain silent. I see these people much as I see my former self: experts at winning, lawyering their way through their academic careers; otherwise intelligent people cranking the handle in a broken machine. They don’t care if the system is broken because it seems to work for them. They don’t see how psychology is failing its public mission because their careers succeeded.
On the other hand I have a deep and abiding respect for senior psychologists who are facing up to the reality that we need to change the way we work, and I admire even more the growing ranks of younger scientists who are championing reform. They are chafing against an academic establishment that, far from rewarding their efforts, at times labels them as trouble-makers and terrorists. If reform succeeds it will, in large part, be a victory owed to those scientists who refused to be silenced and forced the powerful to pay attention. The overarching message of my book to them, as to all psychologists, is: stay inspired and keep shouting. Some of us, at least, are listening.
Source: https://www.theguardian.com/science/head-quarters/2017/may/09/from-protoscience-to-proper-science-the-path-ahead-for-reforming-psychology