Foundations of social and psychological studies get a brutal jolt

[swag]

http://www.slate.com/articles/health_and_science/cover_story/2016/03/ego_depletion_an_influential_theory_in_psychology_may_have_just_been_debunked.1.html

And here’s the pessimist’s counterargument: It’s easy to imagine how one bad result could lead directly to another. Ego depletion is such a bold, pervasive theory that you can test it in a thousand different ways. Instead of baking up a tray of chocolate chip cookies, you can tempt your students with an overflowing bowl of M&Ms. Instead of having subjects talk to people of another race, you can ask them to recall a time that they were victimized by racism. Different versions of the standard paradigm all produce the same effect—that’s the nature of the Big Idea. That means you can tweak the concept however you want, and however many times you need, until you’ve stumbled on a version that seems to give a positive result. But then your replication of the concept won’t always mean you have a real result. It will only show that you’ve tried a lot of different methods—that you had the willpower to stick with your hypothesis until you found an experiment that worked.

Taken at face value, the new Registered Replication Report doesn’t invalidate everything we thought we knew about willpower. A person’s self-control can lapse, of course. We just don’t know exactly when or why. It might even be the case that Baumeister has it exactly right—that people hold a reservoir of mental strength that drains each time we use it. But the two-task method that he and Tice invented 20 years ago now appears to be in doubt. As a result, an entire literature has been rendered suspect.

“At this point we have to start over and say, This is Year One,” says Inzlicht, referring not just to the sum total of ego depletion research, but to the entire field of social psychology.

All the old methods are in doubt. Even meta-analyses, which once were thought to yield a gold standard for evaluating bodies of research now seem somewhat worthless. “Meta-analyses are fucked,” Inzlicht warned me. If you analyze 200 lousy studies, you’ll get a lousy answer in the end. It’s garbage in, garbage out.

[bemildred]

1. Bullshit will only take you so far.

The "subjects" might well give up because they are smarter, for one thing.

But yeah. I think "ego depletion" is real, marketing types love it to death, but it's not at all a mechanical sort of thing. Context is everything.

[malthaussen]

2. Reading Mr Baumeister's suggestions to save his research...

... leave me shouting at the computer. The things he suggested to increase the amount of self-control needed in the first part of the depletion exercise seem to me would actually require less. More self-control to write a word than tap a key? Really?

Interesting that we were talking about statistics just a bit earlier today. I confess, I am somewhat skeptical of social "science," although it has unquestionably produced useful work: I think one needs to control for a lot more variables than is possible to secure reliable results when dealing with organisms that (may) have free will. It appears that this skepticism is borne out, to an extent, by what has happened here.

A quote from Mr Baumeister about the "olden days: "You worked with people, and got them into the right psychological state and then measured the consequences."
This was supposed to be a valid methodology?

-- Mal

[bemildred]

3. It is all subjective theorizing dressed up with simple math.

If they dealt with that honestly, I wouldn't mind what they do. It is useful work. But it is not objective science, and they will have these issues with what would be called Orientalism in literature, not allowing ones subjects to speak for and characterize themselves, until they do. The reasons the experiments don't replicate is because they are littered with the projections of the experimenters, who do not replicate with the experiment.

Nobody disrespects History or Anthropology because they don't make claims of objectivity. Feelings and opinions are not facts, and just because some guy "observes" them, that doesn't make them facts.

[malthaussen]

5. Yeah, I like Edward Said, too.:)

Spot on with what you say, I think. I am an historian by training, I know the field is all about interpretation. That sociology claims to be a "science" makes me grind my teeth, but of course that helps with the ol' funding.

One of the reasons I dislike the revered Foundation trilogy is that Asimov is so evangelical about "psychohistory." It is the wet dream of materialist philosophers everywhere to have human behavior finally reduced to a predictive science. (Of course, predictive science has a few problems, too, starting with the fact that it is only as good as its observational instruments, but that's another rant. I have a wholly emotional, illogical, and personal aversion to that particular dream.

-- Mal

[bemildred]

6. Well, reductive rationalism and materialism got blown out of the water in the early 20th.

So you needn't worry about that. Although many people still harbor an irrational affection for them.

It is actually a much more interesting universe without them too, I think.

[bemildred]

4. Which is not that far off from what you said, now I think about it. nt

[Kalidurga]

7. LOL

You can't do experiments on this kind of thing and expect a valid result for what is going to happen in the real world with thousands of variables like culture, age, and expectations. And that is the problem with most scientific studies they have to control for variables. But, in real life variables exist.

[bananas]

8. "Why Most Published Research Findings Are False" by John P. A. Ioannidis

The article mentions this is a problem in all fields and links to

http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124

Why Most Published Research Findings Are False
John P. A. Ioannidis

PLOS

Published: August 30, 2005
•DOI: 10.1371/journal.pmed.0020124

Abstract

Summary

There is increasing concern that most current published research findings are false. The probability that a research claim is true may depend on study power and bias, the number of other studies on the same question, and, importantly, the ratio of true to no relationships among the relationships probed in each scientific field. In this framework, a research finding is less likely to be true when the studies conducted in a field are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater flexibility in designs, definitions, outcomes, and analytical modes; when there is greater financial and other interest and prejudice; and when more teams are involved in a scientific field in chase of statistical significance. Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true. Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias. In this essay, I discuss the implications of these problems for the conduct and interpretation of research.