How False Information Becomes "Fact"

Posted by $ MikeMarotta 9 years ago to Science
39 comments | Share | Best of... | Flag

The gist of the problem is a basic application of Bayes’ rule: If true positives are rare and false positives are not uncommon, most published positives will be false positives. If in addition most published studies are positive, a high fraction of the published literature will be false.

http://retractionwatch.com/2016/10/05...


All Comments


Previous comments...   You are currently on page 2.
  • Posted by $ 9 years ago
    Why Evidence is Not Enough
    "Generally, people rated authors as experts when the views coincided with their own. Kahan and his team created three authors and their books. All three had the same high level of academic standing. (Doctorates from major schools.) In every case, two different, opposing views were written for each author and randomly shown to subjects. The topics were gun control, nuclear power plants, and global warming.

    In short, we tend to agree with and thereby validate experts who agree with us. When presented with facts opposed to our commitments, we denigrate the status of the provider." (from my blog) The original paper from Dan M. Kahan, Hank Jenkins-Smith, and Donald Braman of Yale Law School is archived here:

    http://www.motherjones.com/files/kaha...
    Reply | Permalink  
  • Posted by $ 9 years ago in reply to this comment.
    Gee... I had no idea. Thanks!
    "After many decades of promoting aspirin, the FDA now says that if you have not experienced a heart problem, you should not be taking a daily aspirin—even if you have a family history of heart disease. This represents a significant departure from FDA's prior position on aspirin for the prevention of heart attacks.

    On its website, the FDA now says:

    "FDA has concluded that the data do not support the use of aspirin as a preventive medication by people who have not had a heart attack, stroke or cardiovascular problems, a use that is called 'primary prevention.' In such people, the benefit has not been established but risks — such as dangerous bleeding into the brain or stomach — are still present."

    Their announcement was prompted by Bayer's request to change its aspirin label to indicate it can help prevent heart attacks in healthy individuals. Aspirin generated $1.27 billion in sales for Bayer last year, and from Bayer's request, it appears they want everyone to be taking their drug.

    But the FDA says "not so fast"—and rightly so. Evidence in support of using aspirin preventatively has gone from weak to weaker to nonexistent. This is why I've been advising against it for more than a decade. It looks as though aspirin, even "low-dose aspirin" (LDA), may do far more harm than good." -- from several websites including this one: http://articles.mercola.com/sites/art...
    which includes a table of seven studies showing no correlation between aspirin and heart attacks. (Some suggest worse outcomes for heart patients.)
    Reply | Permalink  
  • Posted by $ 9 years ago in reply to this comment.
    It is not that we knew before was false, but that what we learned since was more complete. We go from shovels to steam shovels, from domesticating horses to engaging engines with hundreds of horsepower. We still have shovels and horses - and a good thing, too, about those shovels, considering horses ...

    So, too, in electronics, we still use Benjamin Franklin's notion of positive charge, though we mean the absence of negative charges. It is just a way to conceptualize, and thereby design a device and solve a problem. Even thinking of "charges" as little balls with minus signs is not "right" when maybe you should be thinking of fields. But the word "field" is an analogy...

    A hippopotamus is not really a horse. But they are both real and we know quite a lot about them both that we did not when each was first seen.
    Reply | Permalink  
  • 10
    Posted by $ CBJ 9 years ago in reply to this comment.
    Re: “If this were true, then we could not know anything, right?” Wrong. I honestly don’t know how you could draw such a conclusion from my post. What I said was that governments fund much of science and determines the type of research to be funded, that most of the climate science cheating involved government-funded science, and that careful scrutiny is required for acceptance of scientific findings, especially those funded by governments. A far cry from saying “we could not know anything.” If you’re going to find my position “absurd,” at least derive that conclusion from what I actually say.
    Reply | Permalink  
  • -1
    Posted by CircuitGuy 9 years ago in reply to this comment.
    If this were true, then we could not know anything, right? If it's whether a new drug works better than the ones that are available in generic, the answer will be yes, right? You just follow the money? If it's political, like whether GMOs pose health risks, whether vaccines are safe, whether life on earth evolved on earth through natural selection, whether ESP is real, enough people have an answer that they wish were true that there's no point in making observations.

    I just find this absurd. We have to go on the evidence we have. I suspect because burning fuel powers the global economy that pressure from that might cause scientists to understate the risks of AGW, but I cannot rely on my guessing (i.e. making $hit up) based on politics. I have to go on the evidence we actually have.
    Reply | Permalink  
  • Posted by CircuitGuy 9 years ago in reply to this comment.
    Ha. It's an example of how a Google search can lead you down an interesting unrelated path. Based on the abstracts/intros, they don't relate to what I'm saying. My understanding of the posted article is that there are some false positives in research, and there is a bias to publish positive results. Ideally journals would be biased toward publishing surprising results. So eventually science would self-correct when someone tried to use the false information in the future.

    It's unfortunate that they had to add the disclaimer toward the end that their theory should not be construed as saying we don't know anything. I first learned about it in college in the 90s. I considered it part of post-modernism, but I'm not sure that's right. They said look how biases influenced the observations in craniometry. They said it shows how science just reflects the values of the practitioners. I came to reject this completely. We know the racist conclusions of craniometry are wrong because of science, not because we learned not to be racist and physical differences between the races could fuel racism. Knowing that human observation and memory are unreliable can be incorporated into science. That's why we do blinded tests.

    This same way of thinking is alive in Michael Pollan's book. He says first science told us macro nutrients were all the body needed, and then they discovered micronutrients. They can't just pick one answer and stick to it.

    I have only a casual understanding of philosophy, but this post-modernism, solipsism, we-don't-know-anything, just-pick-a-desirable-conclusion-and-stick-to-it, or whatever its correct name may be seems completely contrary to the ideas in the Rand books I've read.
    Reply | Permalink  
  • Posted by $ CBJ 9 years ago in reply to this comment.
    "Science" doesn't find anything, scientists do. And governments are the chief funders of scientists, and therefore have a big say in what research gets funded and what results they favor. Most of the cheating uncovered in climate research involved government-funded organizations. No scientific "finding" should be accepted without careful scrutiny, but government-funded science needs extra critical attention because of its agenda of increasing its power using "scientific findings" as an excuse.
    Reply | Permalink  
  • Posted by $ 9 years ago in reply to this comment.
    Allow me to also point out the ambivalence in "... well-funded groups with powerful incentives ..." as if poverty and disinterest were categorical virtues. You took the bait also, claiming that governments have a "powerful incentive" to believe in APG. I understand the narrative: they gain even more power by enacting controls and those regulations specifically reduce the populations of poor regions, being a genocide by the rich against the poor. That, in fact, was a key motivation for Margaret Sanger and other advocates of birth control, to reduce the poor and prevent inferior beings from overwhelming us with their numbers -- assuming that "us" means "we" and that "we" and "us" are not among the overabundant and unnecessary poor relative to Al Gore and George Soros. Oh...

    On the other hand, I look to the necessary fact that truths have multiple proofs and validations. Look at the sheer volume of proofs of the Pythagorean Theorem. So, too, with APG and the Green Revolution (both meanings).

    When railroads first were built, farmers claimed that the smoke and noise were destructive. The courts disagreed. That thinking was applied to all pollution. Both Cincinnati and Pittsburgh have city museums that tell of "darkness at noon" from all the coal burning at the turn of the 19th and 20th centuries. Men in offices took a clean shirt to work because their outer wear would be black from coal ash. People in poor areas (valleys) suffered from lung diseases such as asthma, while the wealthy on hilltops were loathe to go into the urban metropolis.

    But libertarian property theory says that you do not have a right to destroy your neighbor's values (or your neighbor) with your pollution. Clean up your act: don't make it someone else's negative externality.

    Myself, I see a lot of value in global warming. English wine was known during the medieval warming when the glacier we still call "Greenland" was settled. But the fact seems, indeed, to be a fact, value it as you please.
    Reply | Permalink  
  • Posted by $ 9 years ago in reply to this comment.
    Thomas Edison launched a smear campaign against alternating current. He killed an elephant by electrocution. He claimed that in the near future, electrocuting criminals would be called "westinghousing." As powerful and well-funded as he was, he could not prevent the general market acceptance of the better form of power transmission. None of that takes away from Edison's genius and hard-won success in other areas. The point is only that competing interests ("special interests") are part of the fabric of a free society.

    Similarly, whether APG is arguable or not, the thesis of the article remains important. We have a tendency to publish positive results. We get rewarded for positive results. "We tried this and it failed" does not get much support, but could nonetheless be the more important finding.
    Reply | Permalink  
  • Posted by $ WilliamShipley 9 years ago in reply to this comment.
    The skepticism people get funded to the tune of millions of dollars, the global warming people to the tune of billions. Three orders of magnitude more funding.

    Because if you can't prove there is a "problem" you can't plunder the economy with carbon taxes and mitigation funding that doesn't solve the problem but is "a step in the right direction" (i.e. worthless).
    Reply | Permalink  
  • Posted by CircuitGuy 9 years ago in reply to this comment.
    If this were true and science were completely corrupt, then we wouldn't be able know anything in science because interest groups would just "create any evidence" necessary to get the answer they want. There would be no point in testing if a new medication were safer than an existing one; the answer would always be "yes". We'd be tempted to throw up our hands and just pick whatever answers we wish were true.

    If science just gave us the answer the largest and most well-funded groups desired, we would not expect it to have discovered that burning stuff, which drives most economic activity, is damaging the environment. It's encouraging that science is able to find answers that almost everyone wishes were not true.

    The authors of this article mentions facts that people wish weren't true (human activities are affecting the climate in costly ways, vaccines do not cause autism, and smoking increases risk of cancer) and says they have nothing to do with the process they're describing in which positive "noise" gets published more often than negative "noise".
    Reply | Permalink  
  • Posted by $ CBJ 9 years ago
    He says that anthropogenic climate change has "been established despite well-funded groups with powerful incentives to expose any evidence that might give cause for skepticism." He doesn't mention the even more well-funded groups and governments with powerful incentives to create any evidence that might give cause for acceptance.
    Reply | Permalink  
  • Posted by CircuitGuy 9 years ago
    Is there a corresponding bias in favor of publishing surprising negative results, e.g. a study showing no correlation between use of aspirin and heart attack. I ask b/c it seems like this bias is similar to a bias toward positive results but works in the opposite direction.
    Reply | Permalink  

  • Comment hidden. Undo