How to Choose the Right Statistical Test Without Memorising a Table

Suppose the broad topic is contraceptive use.

Four questions could be:

1. What proportion of women currently use contraception?
2. Does contraceptive use differ between urban and rural women?
3. What factors are associated with contraceptive use?
4. Did contraceptive use change among the same women after an intervention?

Same outcome topic.

Different analytical questions.

The statistical method follows the question.

Not the subject area.

The outcome often determines the family of analyses you should consider.

Common outcome types include:

Examples:

• uses contraception: yes/no;
• depressed: yes/no;
• treatment success: yes/no.

Possible analyses may involve:

• proportions;
• risk differences;
• risk ratios;
• odds ratios;
• binary-outcome regression models.

Examples:

• blood pressure;
• PHQ-9 score;
• weight;
• knowledge score.

Possible analyses may involve:

• means;
• mean differences;
• linear models;
• methods accounting for repeated or clustered measurements.

Examples:

• number of clinic visits;
• number of depressive episodes;
• number of adverse events.

Count outcomes often require models designed for counts, depending on their distribution and research question.

Examples:

• poor / fair / good / excellent;
• ordered severity levels;
• Likert-type outcome categories.

The order contains information.

Treating an ordinal outcome as an unordered category can waste it.

Treating it as continuous may or may not be defensible depending on the measure and analysis.

Examples:

• time to contraceptive discontinuation;
• time to relapse;
• time to death.

Now censoring and event timing matter.

A simple comparison of final proportions may throw away important information.

Do you want to:

• describe it?
• compare two groups?
• compare several groups?
• estimate an exposure-outcome association?
• adjust for confounding?
• model repeated change?
• predict it?
• estimate time to an event?

“Binary outcome” does not automatically mean one test.

For example:

What proportion currently uses contraception?

requires an estimate of a proportion and uncertainty.

Does use differ between urban and rural groups?

requires a comparison.

What is the adjusted association between distance and use?

requires an appropriate regression model.

Different task.

What proportion of women use contraception?

You need a proportion and confidence interval.

No group-comparison test is required to answer the primary descriptive question.

Does use differ between urban and rural women?

Now you are comparing binary outcomes between groups.

A chi-square or exact test might help assess the unadjusted association in some settings.

But report the actual group proportions and an interpretable effect estimate.

Do not finish with:

p = 0.02.

Is residence associated with contraceptive use after accounting for prespecified confounders?

Now you need a regression approach appropriate to the binary outcome and the estimand.

Depending on the question, you may prefer:

• logistic regression for odds;
• log-binomial or modified Poisson approaches for risk/prevalence ratios;
• another model appropriate to the design.

The test-name table has become a modelling question.

Did use change among the same women before and after an intervention?

The observations are paired.

That dependence changes the analysis.

Treating baseline and follow-up as independent groups ignores the within-person structure.

Same variable.

Different statistical method.

This is one of the easiest ways to choose the wrong test.

Example:

Compare mean blood pressure among different people in intervention and control groups.

Example:

Measure blood pressure before and after treatment in the same participants.

The two observations from the same person are related.

Example:

Cases individually matched to controls.

The matching creates dependence that should be reflected in analysis.

Example:

PHQ-9 measured at baseline, two weeks, six months and 12 months.

Now each participant contributes several correlated observations.

Example:

Students are sampled within schools.

People in the same school may be more similar than people from different schools.

The analysis needs to account for the study's correlation structure.

Suppose you want a causal or explanatory estimate from observational data.

Now variable roles matter.

Potential questions include:

• Which variables confound the exposure-outcome relation?
• Are there design variables?
• Is clustering present?
• Are sampling weights needed?
• Is effect modification important?

Do not choose regression simply because:

“multivariate analysis is more advanced.”

Use regression because the question requires modelling or adjustment.

And choose covariates for reasons consistent with the estimand.

Our guides on covariate selection explain why “include everything with p < 0.20” is not a universal confounder-selection strategy.

Every model makes assumptions.

The useful question is not:

“Is my data normal?”

It is:

Which assumptions matter for this estimator/model, and are they reasonable enough for this analysis?

Depending on the method, assumptions may involve:

• independence;
• distribution of residuals;
• linearity;
• variance structure;
• proportional hazards;
• correct link function;
• absence of severe sparsity;
• model specification;
• missing-data assumptions.

A normality test p-value does not make these decisions for you.

A workflow such as:

Shapiro-Wilk p < 0.05 → data are non-normal → Mann-Whitney.

can be too mechanical.

Ask:

• What quantity are you trying to compare?
• Are you interested in means, distributions or ranks?
• How large is the sample?
• What does the distribution actually look like?
• Are there outliers?
• What assumptions apply to the test statistic or model?
• Is transformation meaningful?
• Would a robust or model-based method better answer the estimand?

The alternative method should answer the scientific question.

Not merely survive a normality test.

This is the question giant statistical-test tables often forget.

Suppose a chi-square test gives:

p = 0.004

What should the reader learn?

Not just:

There was an association.

Maybe you should report:

• 63% vs 48%;
• risk difference = 15 percentage points;
• prevalence ratio;
• odds ratio;
• confidence interval.

The analysis should end in an interpretable estimate.

Use this only as an orientation.

Not as autopilot.

The phrase possible starting point is deliberate.

Real designs can require more specific approaches.

Not as much as introductory tables imply.

Many common comparisons can be expressed within a general linear-model framework.

For example:

• a two-group mean comparison can be represented by a linear regression with a binary group indicator;
• ANOVA can be represented within linear modelling;
• regression can incorporate adjustment and interactions that a simple two-group test cannot.

Understanding the model often gives you more flexibility than memorizing disconnected test names.

But more flexible is not automatically better.

Use the simplest model that answers the question appropriately.

Both concern categorical data, but the choice depends on the table structure and whether large-sample approximations are adequate.

Do not use:

“sample size below 30 = Fisher”

as a universal rule.

Look at the contingency table and expected cell counts, and consider whether an exact method or model is appropriate.

More importantly, report the association estimate.

The p-value is not the substantive result.

Do not jump automatically from “two variables” to “multiple regression.”

Ask why each predictor is in the model.

Possible purposes include:

• estimating a specific adjusted exposure effect;
• prediction;
• descriptive modelling;
• confounding control;
• effect modification.

Those goals lead to different modelling choices and interpretations.

A model with ten predictors is not automatically more rigorous than one with three.

If your study used:

• stratification;
• cluster sampling;
• unequal selection probabilities;
• survey weights;

ordinary standard errors may be wrong if the design is ignored.

Likewise, a cluster randomized trial cannot be analysed as though 500 individuals were independently randomized if randomization occurred at 20 schools.

The analysis must know how the data were generated.

Before choosing a test, answer:

Difference? Association? Prediction? Change? Time to event?

Binary? Continuous? Count? Ordinal? Time-to-event?

One group? Two groups? Several groups? Exposure-outcome model?

Independent? Paired? Repeated? Clustered?

Confounding? Design? Covariates? Weights?

Are they reasonable for these data?

Mean difference? Risk difference? Risk ratio? Odds ratio? Rate ratio? Coefficient? Hazard ratio?

Only now pick the method.