Public Misunderstanding Of Studies

[megimoo]

"Oh Joy,We've found Old Mother Numbers Little Brother."

Over at Bioephemera, Jessica Palmer agree with Language Log’s Mark Liberman in his admonition against the use of “generic plurals” in science reporting. Language Log:

This would lead us to avoid statements like “men are happier than women”, or “boys don’t respond to sounds as rapidly as do girls”, or “Asians have a more collectivist mentality than Europeans do"” — or “the brains of violent criminals are physically and functionally different from the rest of us”. At least, we should avoid this way of talking about the results of scientific investigations.

The reason? Most members of the general public don’t understand statistical-distribution talk, and instead tend to interpret such statements as expressing general (and essential) properties of the groups involved. This is especially true when the statements express the conclusions of an apparently authoritative scientific study, rather than merely someone’s personal opinion, which is easy to discount.

The problem, in case you don’t see it from what’s quoted above, is this (I’m going to make some details up, just to give an example):

Suppose some researchers do a study in which they ask people how happy they are, on a scale of 1 to 10. Suppose that they ask 50 men and 50 women, and the average happiness rating for the men is 7.3, while the average score for the women is 7.1. Now suppose that the study is reported in the news with the statement that “men are happier than women.”

Or let’s be even more straightforward: suppose the 50 men and 50 women are simply asked, “On the whole, are you happy?” 37 of the men and 36 of the women say, “Yes.” And the newspapers report that, according to a recent study, “men are happier than women.”

Of course, George reads that over his morning coffee, and says, “Hey, Martha. It says here that I’m happier than you. Ha! I always knew there was something wrong. Maybe you need some of that Prozac stuff.”

But we can’t generalize a finding based on average aspects of a group... to particular individuals in the general population. Martha may be far happier than George, and the study doesn’t say otherwise. George just doesn’t understand.

http://www.scientificblogging.com/staring_empty_pages/public_misunderstanding_studies

[pssvr]

We also can't discount the relevance of the actual question that was asked. It is asinine to say that men are happier than women as anything more than a personal opinion; no scientific study can determine that. A scientific study could determine that American men who have phones and answer surveys about happiness are more likely to rate themselves as happy than American women who have phones and answer surveys about happiness. That's all.

To emphasize this distinction, consider something I once read in the Reader's Digest (and I'm paraphrasing slightly, but you'll see what I mean): "such-and-such percent of Chinese adults say that the child is the ruler of the family."

What's wrong with this study? The fact that we can infer that Chinese adults did not say that because in general many Chinese adults do not speak English. In fact one of two things is true:

Either the survey was conducted in Chinese, or

The survey was conducted on only those Chinese people who happen to speak English.

If the latter is true, we see that the results are falsely reported because the excerpt just said 'Chinese adults'; it failed to mention that the sample was from the English-speaking portion of the population.

If the former is true, we see that the results are again falsely reported because literal translations between Chinese and English are impossible to achieve. It's plausible that 'child' was translated literally, and that 'family' was translated approximately literally (not perfectly, since different cultures have different concepts of family), but it is utterly ridiculous to think that 'ruler' was translated literally. The Chinese have a wholly different concept of ruling and leadership than we do. When we hear a person say the child is the ruler of the family, our mind conjures an image of a child giving his parents orders. But is that typical of a Chinese household? Certainly not. Thus whatever the case may be, the results are reported in a horribly false manner.

Read with a skeptical eye; 95% of statistics are made up.