P-Values, Confidence Intervals and Effect Sizes: How to Read Them Together

A useful reading sequence is:

Estimate → uncertainty → compatibility with the null → practical importance → study credibility

Not:

p-value → celebration or despair → next table

That distinction matters because a statistically convincing result can be tiny, while an uncertain result can still be compatible with effects that would matter greatly.

An effect estimate tells you the magnitude and usually the direction of the observed difference or association.

Depending on the study, that might be:

• a mean difference;
• a risk difference;
• a risk ratio;
• an odds ratio;
• a rate ratio;
• a hazard ratio;
• a regression coefficient;
• a correlation coefficient;
• a standardized mean difference.

Suppose an intervention increases contraceptive uptake from 51% to 58%.

One effect estimate is the absolute difference:

58% − 51% = 7 percentage points

Now the reader knows something useful.

The intervention group had an estimated uptake seven percentage points higher than the comparison group.

The p-value cannot tell you that magnitude.

Now suppose the result is:

Risk difference = 7 percentage points
95% CI = 1 to 13 percentage points

The point estimate is seven.

The interval shows that there is still uncertainty around it.

Under the statistical model and assumptions used, the data are compatible with effects ranging from a small increase of around one percentage point to an increase of around 13 percentage points.

That range matters.

If a one-point change would be too small to justify a programme, but a 13-point change would alter national policy, the uncertainty is not a footnote.

It is part of the decision.

Now add:

p = 0.03

The p-value contributes information about how compatible the observed data are with the tested null model.

It does not tell you:

• that the intervention has a 97% chance of working;
• that the observed effect is large;
• that the result is important;
• that bias is absent;
• that the design supports a causal conclusion;
• that the result will replicate.

A small p-value does not upgrade a weak study into a strong one.

Seven percentage points higher uptake.

The interval spans roughly one to 13 percentage points.

For a risk difference, the usual null value is zero.

This interval does not include zero.

That cannot be determined from p = 0.03.

You need context.

Would a seven-point increase justify the intervention cost?

Would one point?

Would 13 points?

Would benefits be equitable across groups?

Are there adverse consequences?

Now leave the table and inspect the study.

• Was allocation appropriate?
• Was the outcome measured well?
• Was missing data substantial?
• Was the analysis prespecified?
• Was clustering handled?
• Were important confounding structures addressed, if observational?
• Does the design support the conclusion being made?

The numerical result is only as credible as the research process that produced it.

RR = 0.60
95% CI 0.48-0.75
p < 0.001

The estimate suggests a substantial reduction, and the interval is reasonably precise.

You still need to ask whether the design is credible and whether the effect is important in context.

But the p-value is not carrying the interpretation alone.

RR = 0.98
95% CI 0.97-0.99
p < 0.001

This may be highly statistically convincing.

It is still only an estimated 2% relative difference.

Whether that matters depends on:

• baseline risk;
• outcome severity;
• population size;
• intervention cost;
• potential harms;
• feasibility;
• equity.

“Highly significant” does not mean “highly important.”

RR = 0.70
95% CI 0.45-1.09
p = 0.11

The point estimate suggests a potentially meaningful reduction.

The confidence interval is wide and includes the null.

Calling this simply:

“There was no effect.”

throws away most of the information.

The appropriate conclusion is about uncertainty.

RR = 1.01
95% CI 0.99-1.03
p = 0.35

The result is not conventionally statistically significant.

But the narrow interval may tell you something important: large relative effects are not very compatible with these data under the model.

That can be scientifically useful.

Another common problem is using “effect size” as though it refers to one special statistical number.

An effect estimate should match the research question.

If your outcome is binary, a risk ratio, risk difference or odds ratio may be relevant.

If your outcome is continuous, a mean difference may be more interpretable than a standardized effect.

If you need to compare effects across different measurement scales, a standardized effect measure may be useful.

The important question is:

What quantity best communicates the magnitude of the phenomenon I am studying?

Do not calculate Cohen's d merely because someone told you every study needs “an effect size.”

Suppose disease risk falls from:

2% to 1%

The relative risk is:

0.50

That is a 50% relative reduction.

The absolute risk difference is:

1 percentage point

Both are correct.

They answer different aspects of the question.

In public-health decision-making, absolute effects can be especially important because they connect directly to the number of events potentially prevented.

Whenever possible, choose effect measures that help the intended reader understand the actual consequences.

When you encounter a quantitative result, use this sequence:

What happened, and in which direction?

How wide is the confidence interval?

Null effects? Trivial effects? Important benefit? Potential harm?

How compatible are the data with the tested null model?

Clinically? Practically? Socially? Programmatically?

Design, measurement, analysis and bias still matter.

That is interpretation.

Everything else is number watching.