Your p-value is not your finding
You ran the analysis, got p < 0.05, and wrote "significant" in your results section. Then a reviewer asked, "Significant how?" and you weren't sure what to say.
A p-value tells you whether a result is probably not due to chance alone. It doesn't tell you how big the effect is, how sure you can be about its size, or whether anyone should care.
01: Which study would you trust?
Two studies test a school tutoring programme. Both measure exam scores out of 100.
Study A
18,000 students Scored 0.8 points higher than the comparison group 95% CI: 0.5 to 1.1 p < 0.001
Study B
60 students Scored 6.0 points higher than the comparison group 95% CI: −0.7 to 12.7 p = 0.080
If you go by p-values alone, Study A wins easily. Now suppose 5 points is the smallest difference that would change a student's grade or a school's decision to adopt the programme.
Study A's effect is real but tiny. Even the top of its interval, 1.1 points, is a fraction of the 5 that would matter. With 18,000 students, the study could detect an effect too small to matter to anyone. The p-value is impressive. The finding isn't.
Study B's best estimate, 6.0 points, would matter. But its interval runs from slightly worse to almost 13 points better. That doesn't make it a hidden success. It's inconclusive.
Neither study supports the claim "this programme meaningfully improves exam scores." Study A calls for modest claims. Study B calls for a larger study.
02: The Bench Check
Before you write up any result, ask three questions.
- 1What is the estimated effect? Report the actual size, in units your reader understands. "Significant" isn't a size.
- 2How precise is the estimate? Read the confidence interval. A narrow interval means you know the size well. A wide one means you don't, whatever the p-value says.
- 3Would the smallest plausible effect still matter? Compare the whole interval with the difference that matters in practice. If even the lower bound matters, the finding is strong. If even the upper bound is too small, the effect is real but unimportant. If the interval spans both, say so.
Decide what difference would matter before you see your results, not after.
This week's action
Take the most significant result in your thesis or paper and cover the p-value. Can you still say whether the effect matters? If not, add the effect size, the confidence interval and the difference that would matter. Tell us which result you tested: hello@methodsbench.com
The guide covers the reverse case, why small effects can matter at population scale, and exactly what to write in your results section.
Two opportunities worth your time
Lacuna Fund / ACTS: Open Datasets for SDGsUp to US$10,000 per project (3 awards) · Deadline: 13 November 2026Train or scale machine-learning models, or build tools, using existing Lacuna Fund open datasets in any SDG area. Best for: Research teams headquartered in a Global South low- or middle-income country with machine learning and subject expertise. Individuals apply through an institutional sponsor. Questions are due 28 October.View opportunity →Gates Cambridge Scholarship 2027–28Full-cost scholarship · Deadline: 8 December 2026 or 6 January 2027, depending on course (11:59 pm UK time)Covers fees, maintenance, return airfare and visa costs for a full-time Master's or PhD at Cambridge. Best for: Non-UK applicants with a strong academic record. You apply through the Cambridge postgraduate application, so choose your course first.View opportunity →P.S. Not sure your test was the right one in the first place? The Statistical Analysis Decision Tool points you to an appropriate test in a few questions.
See you next week. The Methods Bench
Make research easier, one useful email at a time.
Practical methods, tools and new Methods Bench resources.