Tag Archives: replication crisis

Sex, Gender, and Culture

In Sex is a Spectrum, Augustin Fuentes begins Chapter 6, No Biological Battles of the Sexes, by bringing up John Gray’s 1993 pop psychology text, Men Are from Mars, Women Are from Venus. Gray’s book stayed on the bestseller list for 121 weeks. Fuentes notes that, “more than three decades later, [it] is still selling tens of thousands of copies.” The views expressed in it even continue to be heard among some biologists to this day.

As a true believer of the liberal religion, I have a shameful sin to confess. I read that book in the mid-nineties and, at the time, found it compelling. In retelling an ancient cultural master narrative, it confirmed longstanding Western biases of gender norms and social roles. Ignorant and naive, I didn’t understand how I had been enculturated and indoctrinated. I now seek to make amends for the error of my ways.

Of course, all the way back in the ancient late 20th century, a good liberal such as myself having been drawn into that particular ideological realism simply made me a typical American. Such material was considered enlightened even, as acknowledging these perceived differences was deemed a way of respecting the supposed distinctive and unique worth of each sex, that a mutual balance was beneficial and necessary. A yin-yang kind of thing.

Back then, that was a genuine expression of core values within social liberalism and egalitarian leftism. I was raised in uber-liberal religion, the Unity Church, that already in my childhood had a majority of women ministers and did same sex marriage ceremonies. Books like Gray’s would’ve been popular among Unity congregants. It perfectly fit into New Thought theology and progressive religiosity, along with Jungian-inflected spirituality, of the divine as both female and male.

That theological and cultural brand of left-liberalism was radical for its time and still remains radical in many ways. Take, for example, the Unity minister Marianne Williamson who has twice run as a Democratic candidate with a progressive message of love, compassion, and care for all; and John Gray endorsed her. That’s particularly radical in this moment of rising fascism in the Republican Party and fascist complicity in the Democratic Party.

But the fact of the matter is scientific knowledge has advanced immensely over the past several decades. What seemed obvious with earlier understandings are now seen as, at least partly, to be the product of cultural influences. Culture doesn’t only shape how we perceive but also how we act, each feeding back into the other. Cultural outcomes get taken at face value and so are treated as objective facts, which in turn justifies the cultural assumptions and practices that reinforce those cultural outcomes.

That isn’t to say there are no real biological differences between the sexes, if they’re far fewer and far smaller than we once believed. The problem is that, because of WEIRD bias of researchers and test subjects, we’re forced to honestly and humbly admit we’re downright ignorant to an immense degree. We simply have no practical way to disentangle all the confounding factors so as to make an ‘objective’ assessment. So, we’re not sure what to make of the differences we do detect, as we don’t know what’s causing them.

As an example, in the MBTI personality test, approximately three-quarters of men are Thinking types and three-quarters of women are Feeling types.* That leaves, of course, one in four who don’t fit gender stereotypes of personality. Even if culture played no role, that would still be a significant portion. But then consider that, if we could somehow control for confounders, most likely some of the gender difference would disappear. How much? No one knows.

Speaking more broadly, none of this should be surprising. The challenging and overturning of old binary assumptions about sex and gender has been going on for a while now. I’m not sure when the topic first appeared on my radar. But obviously it’s been a hot topic for a long while. And I was writing about it at least as early as about a decade ago (What is inheritance?; & Is the Tide Starting to Turn on Genetics and Culture?).

Furentes’ book, published in 2025, hopefully will help push changes along. Maybe one day culture and science will more closely coincide. As sex is a spectrum, we should remind ourselves that culture too is a spectrum. If we are to transcend our own cultural reality tunnel, researchers will have to confront the replication crisis by expanding their focus to entirely different cultures, if that’s a challenge. Entirely non-WEIRD populations are disappearing quickly with spreading modernization and Westernization.

* * *

*Note:

Admittedly, I’m biased in my assessment of bias. I’m one of those ‘girly’ men who is a Feeling type. Not only that but specifically one with dominant Feeling and of the Introverted variety (Fi), which is even more ‘girly’. I’m one of the notorious INFPs known as the most idealistic of idealists, prone to unmanly flights of fancy and all things artsy-fartsy. Of course, it goes without question that I lack all proper macho dominance behavior and have been an utter failure within social Darwinism.

It’s probably caused by my having been culturally raised in a ‘girly’ religion with strong women as role models. Besides female ministers who demonstrated leadership, my mother was assertive, confident, college-educated, and financially independent. She had her own professional career and had internalized basic feminist values. And we can’t forget that I’ve spent most of my life in a liberal college town (diversity, political correctness, progressivism, etc). That is to say, there never was any hope for me.

* * *

Sex is a Spectrum
by Augustin Fuente
Chapter 6: No Biological Battle of the Sexes
pp. 111-

Of Minds, Means, and Behavior

The “females and males as very different” camp often points to cognition (“minds”) as the proof of their position. Everyone knows men and women think differently, right? Over the past few decades, massive studies called meta-analyses reviewed patterns in many of these cognitive variables, such as math, verbal, and spatial-ability skills, communication dynamics (verbal and nonverbal), social and personality variables such as aggression, negotiation, helping, sexuality, leadership, introversion/extroversion, general psychological well-being, some motor behaviors (throwing, balance, flexibility, etc), and a few other psychological states and behaviors (moral reasoning, cheating behavior, etc). These meta-analyses involved data form more than twenty thousand separate studies involving more than twelve million participants. The results are clear: Across most topic areas in psychological science, the difference in responses and outcomes between males and females is small or very small.

Here “small” and “very small” are measures of how far apart the means of massively overlapping variations are. That is, pretty much everything being measured in these studies overlaps almost completely between 3G categories (usually based on self-reported genders), but the means of the distributions of the measured variables, when separated by 3G-sex category or gender, can be different from one another. Think of the height example from earlier in the book: 78 percent of folks in the United States are not identifiable to 3G sex simply by height, but the means of the overlapping distributions of 3G-male and 3G-female heights are different. So, on average, one can say 3G males are taller than 3G females. But that might not tell you much at all about any specific individual, or about height as a biological characteristic, given the massive overlap between 3G categories. In the “mind” studies, most of the differences between the means are much, much smaller than in the height example. The difference between the means in these studies is assessed by the common statistical tool called the “cohens d” measure, which reflects how far apart the means in the overlapping distributions of the measurements are in standardized units. So “small” suggests that the means are very close to one another and “very small” even closer (with almost 100 percent overlap). The between-gender differences in these huge meta-analyses were small in 46.1 percent of all cases and very small in 39.4 percent of all cases. The largest and most recent metanalysis demonstrated this same massive overlap, with about 84 percent of mean differences being small and very small across most traits examined. Again, we really aren’t so different. And cognition is certainly not binary.

Failure of Nutritional Knowledge in Science and Practice

“The idea that the same experiment will always produce the same result, no matter who performs it, is one of the cornerstones of science’s claim to truth. However, more than 70% of the researchers (pdf), who took part in a recent study published in Nature have tried and failed to replicate another scientist’s experiment. Another study found that at least 50% of life science research cannot be replicated. The same holds for 51% of economics papers”
~Julian Kirchherr, Why we can’t trust academic journals to tell the scientific truth

“The case against science is straightforward: much of the scientific literature, perhaps half, may simply be untrue”
~Richard Horton, editor in chief of The Lancet, one of the leading medical journals where nutritional studies are published

“It is simply no longer possible to believe much of the clinical research that is published, or to rely on the judgment of trusted physicians or authoritative medical guidelines. I take no pleasure in this conclusion, which I reached slowly and reluctantly over my two decades as an editor”
~Dr. Marcia Angell, former editor in chief of The New England Journal of Medicine

“Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true. Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias.”
~John Ioannidis, Why Most Published Research Findings Are False

“Possibly, the large majority of produced systematic reviews and meta‐analyses are unnecessary, misleading, and/or conflicted.”
~John Ioannidis, The Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta‐analyses

“Nutritional epidemiologists valiantly work in an important, challenging frontier of science and health. However, methods used to-date (even by the best scientists with best intentions) have yielded little reliable, useful information.”
~John Ioannidis, Unreformed nutritional epidemiology: a lamp post in the dark forest

“Associations with cancer risk or benefits have been claimed for most food ingredients. Many single studies highlight implausibly large effects, even though evidence is weak. Effect sizes shrink in meta-analyses.”
~Jonathan Schoenfeld & John Ioannidis, Is everything we eat associated with cancer? A systematic cookbook review

“Some nutrition scientists and much of the public often consider epidemiologic associations of nutritional factors to represent causal effects that can inform public health policy and guidelines. However, the emerging picture of nutritional epidemiology is difficult to reconcile with good scientific principles. The field needs radical reform.”
~John Ioannidis, The Challenge of Reforming Nutritional Epidemiologic Research

“Incoming residents to a pediatric residency program appear to be deficient in basic nutritional knowledge. With the ever increasing burden of obesity and its associated co-morbidities on society, it is imperative that medical education focuses on preparing physicians to appropriately counsel all populations on proper nutrition.”
~M. Castillo, R. Feinstein, J Tsang & M. Fisher, Basic nutrition knowledge of recent medical graduates entering a pediatric residency program.

“Many US medical schools still fail to prepare future physicians for everyday nutrition challenges in clinical practice. Nutrition is a dominant contributor to most chronic diseases and a key determinant of poor treatment outcomes. It cannot be a realistic expectation for physicians to effectively address obesity, diabetes, metabolic syndrome, hospital malnutrition, and many other conditions as long as they are not taught during medical school how to recognize and treat the nutritional root causes.”
~Kelly Adams, W. Scott Butsch & Martin Kohlmeier, The State of Nutrition Education at US Medical Schools

* * *

I’ve written about this topic before. In some of those earlier posts, I used a few of the above quotes. But I also came across some new quotes that emphasize the point. I decided to gather them all together in one place without analysis commentary, as they speak for themselves. I’ll allow myself to make a single note of significance.

A lot of medical research is done by doctors. In Rigor Mortis, Richard Harris points out that doctors aren’t generally well educated and trained in research methodology or statistical analysis. My cousin who does medical research confirmed this observation. On top of that, doctors when they were back in medical school also weren’t taught much about diet and nutrition — interns right out of medical school get about half the nutritional questions wrong, which would be a failing grade.

So, combine doctors not trained in research doing research on diet and nutrition which they never learned much about. It is not surprising that nutritional studies is one of the worst areas of replication crisis. The following are the prior posts about all of this:

Flawed Scientific Research
Scientific Failure and Self Experimentation
Clearing Away the Rubbish
Most Mainstream Doctors Would Fail Nutrition

* * *

Bonus Video – Below is a speech given by Dr. Aseem Malhotra at the European Parliament last year and another speech by Dr. Michael Eades. Among other things, he covers some of the bad methodologies, deceptive or misleading practices, and conflicts of interest.

Sometimes research is intentionally bad because of the biases of funding and ideological agendas, an issue I’ve covered numerous times before. It can’t all be blamed on the insufficient education of doctors in their doing research. After the video, I’ll throw in the links to those other pieces as well.

 

 

 

Cold War Silencing of Science
Eliminating Dietary Dissent
Dietary Dictocrats of EAT-Lancet
Monsanto is Safe and Good, Says Monsanto

Another Example of the Replication Crisis

A Waste of 1,000 Research Papers
by Ed Yong

Between them, these 18 genes have been the subject of more than 1,000 research papers, on depression alone. And for what? If the new study is right, these genes have nothing to do with depression. “This should be a real cautionary tale,” Keller adds. “How on Earth could we have spent 20 years and hundreds of millions of dollars studying pure noise?”

“What bothers me isn’t just that people said [the gene] mattered and it didn’t,” wrote the psychiatrist Scott Alexander in a widely shared blog post. “It’s that we built whole imaginary edifices on top of this idea of [it] mattering.” Researchers studied how SLC6A4 affects emotion centers in the brain, how its influence varies in different countries and demographics, and how it interacts with other genes. It’s as if they’d been “describing the life cycle of unicorns, what unicorns eat, all the different subspecies of unicorn, which cuts of unicorn meat are tastiest, and a blow-by-blow account of a wrestling match between unicorns and Bigfoot,” Alexander wrote. […]

“We’re told that science self-corrects, but what the candidate gene literature demonstrates is that it often self-corrects very slowly, and very wastefully, even when the writing has been on the wall for a very long time,” Munafo adds.

Many fields of science, from psychology to cancer biology, have been dealing with similar problems: Entire lines of research may be based on faulty results. The reasons for this so-called “reproducibility crisis” are manifold. Sometimes, researchers futz with their data until they get something interesting, or retrofit their questions to match their answers. Other times, they selectively publish positive results while sweeping negative ones under the rug, creating a false impression of building evidence.

Beyond a few cases of outright misconduct, these practices are rarely done to deceive. They’re an almost inevitable product of an academic world that rewards scientists, above all else, for publishing papers in high-profile journals—journals that prefer flashy studies that make new discoveries over duller ones that check existing work. People are rewarded for being productive rather than being right, for building ever upward instead of checking the foundations. These incentives allow weak studies to be published. And once enough have amassed, they create a collective perception of strength that can be hard to pierce. […]

Similar debates have played out in other fields. When one group of psychologists started trying to reproduce classic results in much larger studies, their peers argued that any failures might simply be due to differences between the new groups of volunteers and the originals. This excuse has eroded with time, but to Border, it feels familiar. “There’s an unwillingness to part with a previous hypothesis,” he says. “It’s hard to wrap your head around the fact that maybe you were on a wild goose chase for years.”

Keller worries that these problems will be used as ammunition to distrust science as a whole. “People ask, ‘Well, if scientists are publishing crap, why should we believe global warming and evolution,’” he says. “But there’s a real difference: Some people were skeptical about candidate genes even back in the 1990s. There was never unanimity or consensus in the way there is for human-made global warming and the theory of evolution.”

(Credit to Nina Teicholz for bringing my attention to this article.)

Clearing Away the Rubbish

“The case against science is straightforward: much of the scientific literature, perhaps half, may simply be untrue”
~Richard Horton, editor in chief of The Lancet

“It is simply no longer possible to believe much of the clinical research that is published, or to rely on the judgment of trusted physicians or authoritative medical guidelines. I take no pleasure in this conclusion, which I reached slowly and reluctantly over my two decades as an editor”
~Dr. Marcia Angell, former editor in chief of NEJM

Back in September, there was a scientific paper published in Clinical Cardiology, a peer reviewed medical journal that is “an official journal of the American Society for Preventive Cardiology” (Wikipedia). It got a ton of attention from news media, social media, and the blogosphere. The reason for all the attention is that, in the conclusion, the authors claimed that low-carb diets had proven the least healthy over a year period:

“One-year lowered-carbohydrate diet significantly increases cardiovascular risks, while a low-to-moderate-fat diet significantly reduces cardiovascular risk factors. Vegan diets were intermediate. Lowered-carbohydrate dieters were least inclined to continue dieting after conclusion of the study. Reductions in coronary blood flow reversed with appropriate dietary intervention. The major dietary effect on atherosclerotic coronary artery disease is inflammation and not weight loss.”

It has recently been retracted and it has come out that the lead author, Richard M. Fleming, has a long history of fraud going back to 2002 with two FBI convictions of fraud in 2009, following his self-confession. He has also since been debarred by the U.S. Food and Drug Administration. (But his closest brush with fame or infamy was his leaking the medical records of Dr. Robert Atkins, a leak that was behind a smear campaign.) As for his co-authors: “Three of the authors work at Fleming’s medical imaging company in California, one is a deceased psychologist from Iowa, another is a pediatric nutritionist from New York and one is a Kellogg’s employee from Illinois. How this group was able to run a 12-month diet trial in 120 subjects is something of a mystery” (George Henderson). Even before the retraction, many wondered how it ever passed peer-review considering the low quality of the study: “This study has so many methodological holes in it that it has no real value.” (Low Carb Studies BLOG).

But of course, none of that has been reported as widely as the paper originally was. So, most people who read about it still assume it is valid evidence. This is related to the replication crisis, as even researchers are often unaware of retractions, that is when journals will allow retractions to be published at all, something they are reluctant to do because it delegitimizes their authority. So, a lot of low quality or in some cases deceptive research goes unchallenged and unverified, neither confirmed nor disconfirmed. It’s rare when any study falls under the scrutiny of replication. If not for the lead author’s criminal background in the Fleming case, this probably would have been another paper that could have slipped past and been forgotten or else, without replication, repeatedly cited in future research. As such, bad research builds on bad research, creating the appearance of mounting evidence, but in reality it is a house of cards (consider the takedown of Ancel Keys and gang in the work by numerous authors: Gary Taubes’ Good Calories, Bad Calories; Nina Tiecholz’s The Big Fat Surprise; Sally Fallon Morrell’s Nourishing Diets; et cetera).

This is why the systemic problem and failure is referred to as a crisis. Fairly or unfairly, the legitimacy of entire fields of science are being questioned. Even scientists no longer are certain which research is valid or not. The few attempts at determining the seriousness of the situation by replicating studies has found a surprisingly low replication rate. And this problem is worse in the medical field than in many other fields, partly because of the kind of funding involved and more importantly because of how few doctors are educated in statistics or trained in research methodology. It is even worse with nutrition, as the average doctor gets about half the questions wrong when asked about this topic, and keep in mind that so much of the nutritional research is done by doctors. An example of problematic dietary study is that of Dr. Fleming himself. We’d be better off letting physicists and geologists do nutritional research.

There is more than a half century of research that conventional medical and dietary opinions are based upon. In some major cases, re-analysis of data has shown completely opposite conclusions. For example, the most famous study by Ancel Keys blamed saturated fat for heart disease, while recent reappraisal has shown the data actually shows a stronger link to sugar as the culprit. Meanwhile, no study has ever directly linked saturated fat to heart disease. The confusion has come because, in the Standard American Diet (SAD), saturated fat and sugar have been conflated in the population under study. Yet, even in cases like that of Keys when we now know what the data shows, Keys’ original misleading conclusions are still referenced as authoritative.

The only time this crisis comes to attention is when the researcher gets attention. If Keys wasn’t famous and Fleming wasn’t criminal, no one would have bothered with their research. Lots of research gets continually cited without much thought, as the authority of research accumulates over time by being cited which encourages further citation. It’s similar to how legal precedents can get set, even when the initial precedent was intentionally misinterpreted for that very purpose.

To dig through the original data, assuming it is available and one knows where to find it, is more work than most are willing to do. There is no glory or praise to be gained in doing it, nor will it promote one’s career or profit one’s bank account. If anything, there are plenty of disincentives in place, as academic careers in science are dependent on original research. Furthermore, private researchers working in corporations, for obvious reasons, tend to be even less open about their data and that makes scrutiny even more difficult. If a company found their own research didn’t replicate, they would be the last in line to announce it to the world and instead would likely bury it where it never would be found.

There is no system put into place to guard against the flaws of the system itself. And the news media is in an almost continual state of failure when it comes to scientific reporting. The crisis has been stewing for decades, occasionally being mentioned, but mostly suppressed, until now when it has gotten so bad as to be undeniable. The internet has created alternative flows of information and so much of the scrutiny, delayed for too long, is now coming from below. If this had happened at an earlier time, Fleming might have gotten away with it. But times have changed. And in crisis, there is opportunity or at very least there is hope for open debate. So bring on the debate, just as soon as we clear away some of the rubbish.

* * *

Retracted: Long‐term health effects of the three major diets under self‐management with advice, yields high adherence and equal weight loss, but very different long‐term cardiovascular health effects as measured by myocardial perfusion imaging and specific markers of inflammatory coronary artery disease

The above article, published online on 27 September 2018 in Wiley Online Library (wileyonlinelibrary.com), has been withdrawn by agreement between the journal Editor in Chief, A. John Camm and Wiley Periodicals, Inc. The article has been withdrawn due to concerns with data integrity and an undisclosed conflict of interest by the lead author.

A convicted felon writes a paper on hotly debated diets. What could go wrong?
by Ivan Oransky, Retraction Watch

Pro-tip for journals and publishers: When you decide to publish a paper about a subject — say, diets — that you know will draw a great deal of scrutiny from vocal proponents of alternatives, make sure it’s as close to airtight as possible.

And in the event that the paper turns out not to be so airtight, write a retraction notice that’s not vague and useless.

Oh, and make sure the lead author of said study isn’t a convicted felon who pleaded guilty to healthcare fraud.

“If only we were describing a hypothetical.

On second thought: A man of many talents — with a spotty scientific record
by Adam Marcus, Boston Globe

Richard M. Fleming may be a man of many talents, but his record as a scientist has been spotty. Fleming, who bills himself on Twitter as “PhD, MD, JD AND NOW Actor-Singer!!!”, was a co-author of short-lived paper in the journal Clinical Cardiology purporting to find health benefits from a diet with low or modest amounts of fat. The paper came out in late September — just a day before the Food and Drug Administration banned Fleming from participating in any drug studies. Why? Two prior convictions for fraud in 2009.

It didn’t take long for others to begin poking holes in the new article. One researcher found multiple errors in the data and noted that the study evidently had been completed in 2002. The journal ultimately retracted the article, citing “concerns with data integrity and an undisclosed conflict of interest by the lead author.” But Fleming, who objected to the retraction, persevered. On Nov. 5, he republished the study in another journal — proving that grit, determination, and a receptive publisher are more important than a spotless resume.

Scientific Failure and Self Experimentation

In 2005, John P. A. Ioannidis wrote “Why Most Published Research Findings Are False” that was published in PloS journal. It is the most cited paper in that journal’s history and it has led to much discussion in the media. That paper was a theoretical model but has since been well supported — as Ioannidis explained in an interview with Julia Belluz:

“There are now tons of empirical studies on this. One field that probably attracted a lot of attention is preclinical research on drug targets, for example, research done in academic labs on cell cultures, trying to propose a mechanism of action for drugs that can be developed. There are papers showing that, if you look at a large number of these studies, only about 10 to 25 percent of them could be reproduced by other investigators. Animal research has also attracted a lot of attention and has had a number of empirical evaluations, many of them showing that almost everything that gets published is claimed to be “significant”. Nevertheless, there are big problems in the designs of these studies, and there’s very little reproducibility of results. Most of these studies don’t pan out when you try to move forward to human experimentation.

“Even for randomized controlled trials [considered the gold standard of evidence in medicine and beyond] we have empirical evidence about their modest replication. We have data suggesting only about half of the trials registered [on public databases so people know they were done] are published in journals. Among those published, only about half of the outcomes the researchers set out to study are actually reported. Then half — or more — of the results that are published are interpreted inappropriately, with spin favoring preconceptions of sponsors’ agendas. If you multiply these levels of loss or distortion, even for randomized trials, it’s only a modest fraction of the evidence that is going to be credible.”

This is part of the replication crisis that has been known about for decades, although rarely acknowledged or taken seriously. And it is a crisis that isn’t limited to single studies —- Ioannidis wrote that, “Possibly, the large majority of produced systematic reviews and meta-analyses are unnecessary, misleading, and/or conflicted” (from a paper reported in the Pacific Standard). The crisis cuts across numerous fields, from economics and genetics to neuroscience and psychology. But to my mind, medical research stands out. Evidence-based medicine is only as good as the available evidence — it has been “hijacked to serve agendas different from what it originally aimed for,” as stated by Ioannidis. (A great book on this topic, by the way, is Richard Harris’ Rigor Mortis.) Studies done by or funded by drug companies, for example, are more likely to come to positive results for efficacy and negative results for side effects. And because the government has severely decreased public funding since the Reagan administration, so much of research is now linked to big pharma. From a Retraction Watch interview, Ioannidis says:

“Since clinical research that can generate useful clinical evidence has fallen off the radar screen of many/most public funders, it is largely left up to the industry to support it. The sales and marketing departments in most companies are more powerful than their R&D departments. Hence, the design, conduct, reporting, and dissemination of this clinical evidence becomes an advertisement tool. As for “basic” research, as I explain in the paper, the current system favors PIs who make a primary focus of their career how to absorb more money. Success in obtaining (more) funding in a fiercely competitive world is what counts the most. Given that much “basic” research is justifiably unpredictable in terms of its yield, we are encouraging aggressive gamblers. Unfortunately, it is not gambling for getting major, high-risk discoveries (which would have been nice), it is gambling for simply getting more money.”

I’ve become familiar with this collective failure through reading on diet and nutrition. Some of the key figures in that field, specifically Ancel Keys, were either intentionally fraudulent or really bad at science. Yet the basic paradigm of dietary recommendations that was instituted by Keys remains in place. The fact that Keys was so influential demonstrates the sad state of affairs. Ioannidis has also covered this area and come to similar dire conclusions. Along with Jonathan Schoenfeld, he considered the question “Is everything we eat associated with cancer?”

“After choosing fifty common ingredients out of a cookbook, they set out to find studies linking them to cancer rates – and found 216 studies on forty different ingredients. Of course, most of the studies disagreed with each other. Most ingredients had multiple studies claiming they increased and decreased the risk of getting cancer. Most of the statistical evidence was weak, and meta-analyses usually showed much smaller effects on cancer rates than the original studies.”
(Alex Reinhart, What have we wrought?)

That is a serious and rather personal issue, not an academic exercise. There is so much bad research out there or else confused and conflicting. It’s about impossible for the average person to wade through it all and come to a certain conclusion. Researchers and doctors are as mired in it as the rest of us. Doctors, in particular, are busy people and don’t typically read anything beyond short articles and literature reviews, and even those they likely only skim in spare moments. Besides, most doctors aren’t trained in research and statistics, anyhow. Even if they were better educated and informed, the science itself is in a far from optimal state and one can find all kinds of conclusions. Take the conflict between two prestigious British journals, the Lancet and the BMJ, the former arguing for statin use and the latter more circumspect. In the context of efficacy and side effects, the disagreement is over diverse issues and confounders of cholesterol, inflammation, artherosclerosis, heart disease, etc — all overlapping.

Recently, my dad went to his doctor who said that research in respectable journals strongly supported statin use. Sure, that is true. But the opposite is equally true, in that there are also respectable journals that don’t support wide use of statins. It depends on which journals one chooses to read. My dad’s doctor didn’t have the time to discuss the issue, as that is the nature of the US medical system. So, probably in not wanting to get caught up in fruitless debate, the doctor agreed to my dad stopping statins and seeing what happens. With failure among researchers to come to consensus, it leaves the patient to be a guinea pig in his own personal experiment. Because of the lack of good data, self-experimentation has become a central practice in diet and nutrition. There are so many opinions out there that, if one cares about one’s health, one is forced to try different approaches and find out what seems to work, even as this methodology is open to many pitfalls and hardy guarantees success. But the individual person dealing with a major health concern often has no other choice, at least not until the science improves.

This isn’t necessarily a reason for despair. At least, a public debate is now happening. Ioannidis, among others, sees the solution as not difficult (psychology, despite its own failings, might end up being key in improving research standards; and also organizations are being set up to promote better standards, including The Nutrition Science Initiative started by the science journalist Gary Taubes, someone often cited by those interested in alternative health views). We simply need to require greater transparency and accountability in the scientific process. That is to say science should be democratic. The failure of science is directly related to the failure seen in politics and economics, related to powerful forces of big money and other systemic biases. It is not so much a failure as it is a success toward ulterior motives. That needs to change.

* * *

Many scientific “truths” are, in fact, false
by Olivia Goldhill

Are most published research findings false?
by Erica Seigneur

The Decline Effect – Why Most Published Research Findings are False
by Paul Crichton

Beware those scientific studies—most are wrong, researcher warns
by Ivan Couronne

The Truthiness Of Scientific Research
by Judith Rich Harris

Is most published research really wrong?
by Geoffrey P Webb

Are Scientists Doing Too Much Research?
by Peter Bruce