Sign in Subscribe
Society

Spotting Statistical Integrity

Examine how statistics can be used to support inaccurate conclusions. Learn practical ways to evaluate claims and ask probing questions.

Kayla Staph

Kayla Staph

— 4 min read
Share
Spotting Statistical Integrity
Photo by Utsav Srestha / Unsplash

“As the sales of ice cream increase, the murder rate of the city increases. Do ice cream sales cause murders to increase?” Our professor asked.

We shook our heads no. That conclusion is ridiculous.

She continued, “As temperatures increase after the winter season, ice cream sales and the murder rate increase. Do warmer temperatures cause both to increase?”

A third factor, a confounding factor, was introduced that might impact the two occurrences.

She proceeded to explain an important principle in statistics and research methods: correlation does not prove causation. A positive correlation means that as one factor rises, so does the other. A negative correlation means that as one factor rises, the other factor falls or as one factor falls, the other factor rises. 

If a correlation exists, the correlation describes the strength of the relationship between two factors, not causal conclusions. The relationship could be coincidental or driven by unknown third factors.

The problem is not that people communicate correlations in speeches, social media posts, etc. The problem is when a correlation is presented as causation, which leads to inaccurate conclusions or oversimplification of complex problems. This may lead us to unwittingly participate in someone else’s manipulation of the numbers or support unhelpful solutions.

Example: Someone claims playing video games leads to violent behavior because these events are positively correlated. The person proposes a ban on video games to reduce violence. In reality, the correlation is weak and other factors such as social environment, mental health, and upbringing are much more important predictors. Therefore, the ban would do little to reduce violence.

Commit this to memory. Let it be your statistical analysis mantra:

Correlation does not prove causation

Absolute Frequency versus Relative Frequency

Absolute frequency are raw numbers: 100 water bottles were sold. Relative frequency are the raw numbers in relation to something else: 52% of all items sold were water bottles or 35% of people at the game bought a water bottle.

Both stats have their uses. Absolute and relative frequency stats might be accurate, but that does not mean they are communicated correctly or ethically.

Example: A social media post claims that a plant health treatment is dangerous. The post revealed 30 plants died after receiving the treatment versus only 10 untreated plants died. Looking at raw numbers, absolute frequency, the treatment appears harmful. However, if 1,000 plants were treated (3% mortality) and only 100 were untreated (10% mortality), the relative frequency shows the opposite conclusion (if all other factors were controlled). If the social media poster knew all the numbers, then the post’s original presentation and conclusion was manipulative.

Therefore, when a raw number is presented, you can ask, “What’s the relative frequency for that number?” You may uncover new insights and be able to say to someone:

"You keep using that number."

But sometimes, raw numbers illuminate a problem that would otherwise be lost or distorted within relative frequency.

Example: If there are 1,000,000 people in a city and 0.5% are infected with a life-threatening illness, the small percentage makes it seem like it’s not a big deal. But that percentage means there are 5,000 people infected and the hospitals can only accommodate 3,000 people at a time. That would make the situation far more dire. 

Another way relative frequencies can distort reality is when the sample size is too small to provide statistically significant results.

Example: You see an ad for a supplement that says 80% of people saw positive results after a month of daily intake. Then you learn there were only 10 participants. The relative frequency is accurate, but the result is statistically insignificant.

Therefore:

Examine both absolute and relative frequency

The Availability Heuristic

The availability heuristic is over reliance on top of the mind information because the information is recent, frequent, or stands out. When we are influenced by the availability heuristic, our brains will take a short cut and overestimate an event’s likelihood. This can lead to incorrect assumptions.

Example: Most people overestimate the likelihood of plane crashes and shark attacks. While events like these remain serious, problems begin when people knowingly use the availability heuristic to exploit natural fears of an occurrence for unethical purposes. 

When a common narrative is on replay, wherever we get our information, we can ask:

  1. What is the relative frequency of this event?
  2. Is this event concerning regardless of relative frequency?
  3. What is the appropriate response given the answers to questions 1 and 2?

In some instances, communicating the same type of event is necessary, such as the case with nonprofits. They have missions to help others and must communicate the need. This can lead people who receive newsletters to overestimate the event's frequency, but that does not mean they are being manipulated. Furthermore, frequently highlighting an event may expose a previously unknown pattern.

That’s why:

"Just the facts please" might be more complicated than it appears

Summary

Next time you hear or read a stat, consider thinking through some of these questions:

  • Is this concluding a causal relationship?
  • What other factors could be impacting this stat?
  • What is the relative frequency of that number?
  • What does the raw data say?
  • What can I learn from analyzing both the relative and absolute frequencies?
  • What was the sample size and the representation within it?
  • Why does this media outlet, news station, influencer, organization, etc., present data about the same type of events repeatedly? What is the context?

Need an exercise to help your team avoid cognitive bias? Consider using Starbursting or contact us to learn more about our Intelligence Analysis training.

A Case for Integrity

Though we can learn basic statistics and research methods, we can’t all be statisticians, and we do not have the time to double check every number thrown at us. But, we can cultivate a societal expectation that people who communicate the numbers do so in an ethical way. This does not mean perfection. When we discover there has been a pattern of malicious intent or willfully distorted conclusions, we can choose not to elevate those resources or individuals (or engage the content, remember the algorithm?). Likewise, when we discover there has been a pattern of good-faith intent and dedication to accurate conclusions, we can choose to elevate those resources or individuals. That choice is ours.

Read more