Critical Appraisal and Evidence-Based Medicine for Emergency Clinicians

The first fork is interventional versus observational: did the investigators allocate the exposure, or only watch it? Everything about what a study can claim follows from that. The second fork, within observational work, is the direction of sampling: did you sample on exposure and wait for outcomes (cohort), or sample on outcome and look back for exposure (case–control)?

The pyramid with RCTs near the top is a statement about average susceptibility to bias, not a ranking of individual papers. A small, unblinded, selectively reported RCT stopped early for benefit is weaker evidence than a large, carefully adjusted prospective cohort with complete follow-up. Judge the study, not its label.

An explanatory trial asks whether a thing can work under ideal conditions: tight eligibility, protocol-driven care, expert centres. A pragmatic trial asks whether it does work in the real world: broad eligibility, usual care as comparator, routinely collected outcomes. Pragmatic trials generalise better and tend to show smaller effects — which is usually the truth rather than a failure. When a pragmatic trial disappoints after an explanatory one impressed, the effect has not disappeared; it has been measured under conditions you actually work in.

Before any test is chosen or any p-value computed, a paper has to say what its data are. Almost every later decision — which summary to report, which test is legitimate, whether a mean is even meaningful — follows from the answer.

• Mean — the arithmetic average. Uses every value, which is its strength and its weakness: one extreme value drags it. Meaningful only for interval or ratio data.
• Median — the middle value when ordered. Unaffected by how extreme the extremes are, so it survives skew and outliers. The right choice for most emergency-department time data.
• Mode — the commonest value. Rarely useful for continuous data, but it is the only measure of the middle available for nominal data, and it is what matters when a distribution has two peaks.

In a perfectly symmetric distribution the three coincide. The gap between mean and median is itself a measure of skew — if a paper reports both and they differ substantially, the data are skewed whatever else the text says.

A standard deviation is only interpretable if the distribution is roughly symmetric, because its usefulness comes entirely from the normal distribution's fixed proportions: about 68 % of observations within one SD of the mean, 95 % within two, and 99.7 % within three (strictly, 95 % falls within 1.96 SD — which is where the 1.96 in every confidence interval comes from). Quote an SD on a heavily skewed variable and "mean ± 2 SD" will include impossible values — negative door-to-CT times, negative lactates. That is the quickest way to spot a summary that should have been a median.

See the normal distribution figure and the skewed-dataset figure, where the same 40 numbers are shown as a histogram, a box plot and a summary table.

They answer different questions and are not interchangeable.

• SD describes the patients. It says how spread out the individuals are. It does not shrink as you enrol more of them — a bigger study measures the same underlying variability more precisely, it does not make people more alike.
• SEM describes the estimate. It says how precisely you have pinned down the mean, and it does shrink with √n. It is the building block of the confidence interval: roughly mean ± 2 SEM.

A centile is the value below which that percentage of observations fall; the median is the 50th. Paediatric growth and observation charts are built from them, which is why "below the 2nd centile" is a statement about a population, not a diagnosis.

Most laboratory reference ranges are the central 95 % of a healthy reference population — which has a consequence people rarely state out loud: 1 healthy person in 20 will fall outside the range for any given test, by construction. Order a panel of 20 independent tests on a well person and the expected number of "abnormal" results is one. A reference range is not a threshold for disease, and an isolated marginal abnormality on a broad panel is usually the arithmetic working as designed.

You will rarely choose a test yourself. You will constantly need to judge whether the one a paper used was reasonable — and that judgement reduces to three questions asked in order.

1. What kind of outcome is it? Continuous, ordinal, or categorical (see chapter 03).
2. How many groups are being compared? Two, or more than two. (A single group against a known reference value is the one-sample case, and takes a one-sample t-test, or a Wilcoxon signed-rank test against that value.)
3. Are the observations paired or independent? Paired means the same patient measured twice, or patients matched one to one — before and after, left eye and right eye, two methods on the same blood sample. Analysing paired data as though it were independent throws away the pairing, which is usually the whole point of the design.

Two practical notes on that table. Fisher's exact is the safe choice whenever any expected cell count is small (the usual rule of thumb is under 5) — it is what the fragility calculator on this site uses. And ANOVA answers only "is there a difference somewhere"; it does not say which groups differ, which requires post-hoc comparisons with an explicit correction for multiplicity.

"Parametric" tests assume the data follow a particular distribution — usually the normal — and additionally, for the t-test and ANOVA, that group variances are similar and observations are independent. Non-parametric tests replace the values with their ranks and so assume much less.

• When the assumptions hold, parametric tests have more power, which is the whole reason to prefer them.
• Skew alone is less dangerous than it is usually made to sound, provided the groups are the same size. The central limit theorem applies to the mean rather than to the data, so with balanced arms a t-test on heavily skewed data holds its false-positive rate at or below the nominal 5 % even at a dozen per group. What it loses is power — on skewed data a t-test can need appreciably more patients than a rank test to detect the same real difference.
• The genuinely dangerous combination is unequal group sizes together with unequal spread, and it is far more common in published work than skew alone. Student's t-test assumes equal variances; break that and the balance at the same time and its error rate is not approximately anything — it can reject a true null far more often than 5 % of the time when the smaller group is the more variable one, and almost never when it is the less variable one. Welch's correction, which does not assume equal variances, is the default in most software for exactly this reason, and a paper comparing a small group against a much larger one should be using it.
• A non-parametric test on data that were fine either way costs a little power and almost never misleads. Reaching for one is a mild inefficiency, not an error.

Every test above returns a p-value, and a p-value is a statement about compatibility with a null hypothesis and nothing else. It does not give the size of the effect, its direction in clinically useful units, or its precision. A paper whose results section is a list of p-values has reported the least informative summary available. What you want, and should look for instead, is the estimate with its confidence interval — a difference in means, a difference in medians, a risk difference — in the units a patient would recognise. See chapter 11.

Bias is systematic error — it moves the estimate in a direction and does not shrink when you enrol more patients. That last clause is the whole reason bias matters more than imprecision: a bigger study fixes imprecision and entrenches bias. The five domains below follow a trial through its own timeline, which is also the order in which to look for them.

• Immortal time bias. If a patient must survive long enough to receive the treatment, the treated group is composed of survivors before treatment does anything. This makes ineffective treatments look protective in database studies, and it is subtle enough to reach print regularly.
• Confounding by indication. Sicker patients get the more aggressive treatment, so the aggressive treatment looks harmful. This is the reason observational comparisons of, say, intubation strategies mislead so reliably.
• Lead-time and length bias. Screening-detected disease appears to survive longer partly because the clock started earlier, and because slow-growing disease is over-represented among cases found by screening.

A confounder is associated with the exposure, is a cause of the outcome independently of the exposure, and is not simply a step on the causal path between them. That third clause matters: adjusting for a mediator removes part of the effect you are trying to measure, and adjusting for a collider — a common consequence of exposure and outcome — can manufacture an association out of nothing.

Table 2 fallacy. The other coefficients in a multivariable model are not each a valid causal estimate for their own variable; the model was specified to estimate one exposure effect, and the covariates' coefficients carry a mixture of direct and confounded signal. A paper that presents its whole adjustment table as a list of independent risk factors is over-reading its own model.

These three are routinely conflated, and they protect against different failures at different moments.

• Sequence generation decides the order of assignments. It must be genuinely random — computer-generated, often in blocks, sometimes stratified by centre or severity. Alternation, date of birth and hospital number are not random and are trivially predictable.
• Allocation concealment stops the person enrolling a patient from knowing the next assignment. This is the step that prevents selection bias, and it is possible even when blinding is impossible: you can conceal allocation for a surgical trial that nobody can blind. Sealed sequentially numbered opaque envelopes, central telephone or web randomisation, and identical numbered treatment packs all achieve it. CRASH-2 and HALT-IT both used the numbered-pack method: eight identical packs in a box, differing only by number, so the clinician could not steer a sicker patient towards the drug.
• Blinding (masking) stops people knowing the assignment afterwards, and prevents performance and detection bias. It applies separately to patients, treating clinicians, outcome assessors, data analysts and adjudication committees, and a paper should say which of those were masked rather than claiming to be "double-blind" and leaving you to guess.

Baseline tables and their misuse. The baseline table exists to let you judge whether randomisation produced comparable groups, and to see whom the trial enrolled. It should not carry p-values: in a properly randomised trial any baseline difference is by definition chance, so testing it answers a question nobody asked. What you want instead is the size of the imbalance and whether the imbalanced variable is prognostically important.

Outcome choice is where a trial decides how useful it is allowed to be, and it is decided before a single patient is enrolled.

A surrogate stands in for what you care about: lactate clearance for survival, radiographic union for function, viral load for illness. Surrogates make trials smaller and faster, and they mislead whenever the intervention affects the marker by a route that does not run through the outcome. The question to ask is not "is this surrogate correlated with the outcome" — it usually is — but "do interventions that move this surrogate reliably move the outcome?" That is a far higher bar, and most surrogates fail it.

Composites (death, MI or urgent revascularisation; death or dependency) buy statistical power by collecting more events. They are legitimate when the components are of similar importance to the patient, of similar frequency, and plausibly moved in the same direction by the intervention. They mislead when a frequent, mild, discretionary component — usually something like "unplanned re-attendance" — drives the whole result while the component anyone cares about does not move. Always look for the component-by-component breakdown; if it is absent, treat the composite as unresolved.

Mortality, function, symptom burden, and days alive and out of hospital are outcomes patients recognise. Patient-reported outcome measures (PROMs) capture them directly, at the cost of needing a validated instrument, a minimal clinically important difference (MCID) established independently of the trial, and an honest account of missing questionnaires. A statistically significant change on a scale is meaningless without knowing the MCID, and quoting one from the same trial that found the effect is circular.

Why ITT is the default. Dropping non-compliers destroys the one thing randomisation gave you, because non-compliance is not random: people stop drugs because of side effects, because they deteriorate, or because they recover. Per-protocol analysis therefore compares a filtered intervention group with an unfiltered control group, and the filter is prognostic. ITT preserves the comparison and answers the question a clinician actually faces, which is "if I prescribe this, what happens" — including the patients who will not take it.

Reading the numbers-analysed line is not pedantry. CRASH-2 randomised 20,211 patients and analysed 10,060 of 10,096 and 10,067 of 10,115 — losses of well under 1 % in both arms, which is why its ITT analysis is credible at face value. HALT-IT states the opposite arrangement plainly: its primary analysis excluded patients who received neither dose and those without outcome data, which is a modified ITT, and the paper says so in its methods rather than leaving you to discover it. A trial that randomises 500 and analyses 380 has a problem regardless of what it calls the analysis.

• Complete-case analysis assumes data are missing completely at random, which they almost never are.
• Last observation carried forward is not conservative; it can bias either way depending on the disease trajectory.
• Multiple imputation is the usual defensible approach, and is only as good as the variables predicting missingness.
• A sensitivity analysis under a plausibly unfavourable assumption is the part that actually reassures. If a conclusion survives assuming every lost patient in the treatment arm did badly, believe it.

Take a trial with a control event rate CER and an experimental event rate EER.

The relative measures answer "how much does this shift a person's risk", and they travel reasonably well between populations of different baseline risk. The absolute measures answer "how many people benefit", and they do not travel at all — they depend entirely on baseline risk. This is why a relative risk reduction quoted without a baseline risk is uninterpretable, and why it is the favourite number of anyone selling something.

Identical relative effect; fifty-fold difference in how many people you must treat. If the intervention carries a 1-in-200 risk of serious harm, it is clearly worth doing in the first row and clearly not in the second — and no statistical test will tell you that, because it is a judgement about magnitude, not significance.

An odds ratio is (a/b)/(c/d) on the 2×2 table. It is the natural output of logistic regression and of case–control studies, where risks cannot be estimated. Its trap is simple: the OR exaggerates the RR, and increasingly so as the outcome becomes more common. For an outcome affecting 2 % of controls, OR and RR are nearly identical; for one affecting 40 %, an OR of 2.0 corresponds to a risk ratio nearer 1.4. Treating a published OR as a risk ratio is one of the most common misreadings in clinical conversation.

A hazard ratio compares the instantaneous rate of the event between groups, using time-to-event data, and assumes proportional hazards — that the ratio is roughly constant over follow-up. When survival curves cross or diverge late, that assumption fails and the single HR hides the shape of what happened. Look at the curves and the numbers-at-risk row beneath them; late-separating curves with thin numbers at risk are where over-interpretation lives.

The treatment-effect calculator computes all of these with confidence intervals from raw event counts.

A p-value is the probability of observing data at least as extreme as yours if the null hypothesis were true and the model were correct. That is all it is. It is a statement about data given a hypothesis, not about a hypothesis given data, and the difference is not academic.

A 95 % confidence interval is the range of effects reasonably compatible with the data. It carries everything the p-value carries (if it excludes the null, p < 0.05) and adds the two things a clinician needs: the plausible size of the effect, and the precision of the estimate. Read both of its ends and ask what each would mean clinically:

• Both ends clinically important, same direction — a useful positive result.
• One end important, the other trivial — a positive result that has not established a worthwhile effect.
• Interval spans the null but both ends trivial — genuinely reassuring; this is what "no important difference" looks like.
• Interval spans the null and both ends important — an uninformative trial, whatever its p-value. ANDROMEDA-SHOCK's 28-day mortality hazard ratio of 0.75 (95 % CI 0.55 to 1.02, p = 0.06) is exactly this: compatible with a 45 % relative reduction in death and with a 2 % increase. Calling that "no difference" discards half the information. See the worked appraisal.

Test twenty independent true nulls at α = 0.05 and you expect one "significant" result. Trials handle this by nominating a single primary outcome, pre-specifying a small number of comparisons, and either splitting alpha (Bonferroni, hierarchical testing) or accepting that everything else is exploratory. PROPPR illustrates the honest handling: 23 pre-specified complications were examined and none differed, and the paper reports that as a safety statement rather than mining it for a finding.

The previous chapter ended on an awkward admission: the confidence interval does not mean what almost everyone takes it to mean, and the p-value answers a question nobody asked. Bayesian inference is the framework that answers the question people actually have — given these data, how likely is it that this treatment works? — at the cost of requiring you to state what you believed beforehand.

• The prior is what was believed about the effect before this study: a distribution over possible effect sizes, not a single guess.
• The likelihood is what the new data say, on their own.
• The posterior combines them, and is the answer — a full distribution over effect sizes, from which any question can be read off directly.

That last row is the practical difference, and it is the reason Bayesian re-analyses of intensive care and emergency trials have become common. A frequentist trial reporting p = 0.06 is obliged to say it failed to reject the null; a Bayesian analysis of the same patients can put a probability on benefit directly.

The honest objection to Bayesian analysis is that the prior can be chosen to get the answer you want. The honest answer is that a good Bayesian analysis makes the prior explicit and then shows how much it mattered. Look for:

• A vague or reference prior, which deliberately contributes almost nothing, so the posterior is driven by the data alone. With a vague prior the credible interval and the confidence interval are usually near-identical — which tells you the Bayesian machinery is not doing the work.
• A sceptical prior, centred on no effect and tight enough to make a large claimed benefit implausible. If a result survives a sceptical prior, it is robust.
• An enthusiastic prior, centred on the hoped-for effect, used as the other bookend.
• An evidence-based prior from previous trials or a meta-analysis — the most defensible and the least often available.

• Posterior probability of benefit — P(effect favours treatment). Beware that this is often near 90 % even for trials everyone agrees were unconvincing; "more likely than not" is a low bar.
• Probability of a clinically important benefit — P(effect exceeds some threshold you would act on). Far more useful, and it forces someone to name the threshold.
• Region of practical equivalence (ROPE) — a band around zero within which the effect, if real, would not change practice. Asking how much of the posterior falls inside it is a direct way to say "this difference does not matter", which frequentist analysis cannot do at all.
• Bayes factor — how much the data shift the odds between two hypotheses, independent of how likely you thought each was to begin with. Note that it is not free of assumptions about effect size: a Bayes factor depends on the distribution of effects assumed under the alternative, so two analyses of identical data can report different Bayes factors because they specified the alternative differently.

Bayesian analysis is most valuable exactly where the frequentist framework is least informative: trials that miss the threshold, trials that stop early (where the framework has no multiplicity penalty, because the posterior does not care how many times you looked), small trials in rare conditions where a well-justified prior adds real information, and adaptive designs. It is not a way of rescuing a negative trial — applied to a result with a tight interval around no effect, it returns a posterior tightly centred on no effect, which is the same conclusion in different words.

Both frameworks describe the same data. They differ in what they are willing to put a probability on, and the practical instruction is the same for both: read the interval and the effect size, not the verdict.

Power is the probability of detecting an effect of a specified size if it is really there. It rises with the number of events (not patients), with the size of the effect sought, and with tolerance for a larger α. Four quantities are locked together — α, power, effect size and sample size — so fixing any three fixes the fourth.

Calculating power after the fact from the observed effect adds no information — it is a deterministic function of the p-value. If a trial was inconclusive, the confidence interval already says so, and says it better. Report the interval; do not compute observed power.

Repeatedly testing accumulating data inflates the type I error, so trials use alpha-spending (O'Brien–Fleming, Pocock) with a data monitoring committee and pre-specified stopping rules for benefit, harm or futility. Two practical consequences:

• Trials stopped early for benefit systematically overestimate the effect. You stop at a random high point in the accumulating estimate, and the fewer the events at stopping, the greater the overestimate.
• Stopping for futility is usually sound, and stopping for harm is mandatory — but neither licenses the confident negative conclusion that often follows.

The fragility index is the number of patients whose outcome would have to change, in one arm, to move a significant result across p = 0.05. It is an intuition pump rather than a formal inference: a trial whose headline depends on two events is not the same evidence as one whose headline depends on eighty, even when both report p = 0.04. Its limitations are real — it depends on the test used, ignores effect size and time-to-event data, and can be gamed — so use it to calibrate confidence, not to overturn a result. The fragility calculator computes it with Fisher's exact test.

Subgroup analysis is where most over-interpretation in clinical medicine originates, and also where some of its most important findings came from. The difference between the two is knowable, and it is not whether the subgroup result is "significant".

Splitting a trial into subgroups and looking for which ones have p < 0.05 is close to guaranteed to mislead. Each subgroup is smaller and therefore less powered, so significance within subgroups tracks subgroup size as much as effect. The famous demonstration is the ISIS-2 astrological subgroup: aspirin appeared not to work in patients born under Gemini and Libra. Nobody believes that, and the statistics that produced it are the same statistics that produce the subgroup claims people do believe.

The question is not "is the effect significant in this subgroup" but "is the effect different between subgroups". That is a formal test for interaction (or, for an ordered variable, for trend), and it should be pre-specified, few in number, biologically motivated, and consistent across related outcomes.

1. Was it pre-specified, and is the protocol available to check?
2. How many subgroups were examined in total? (The denominator is rarely stated.)
3. Is there a formal interaction test, and what is its p-value?
4. Is the direction biologically plausible, and does it hold for related outcomes?
5. Is the gradient monotonic where the variable is ordered?
6. Has it replicated anywhere else?

One or two yeses is a hypothesis. Most of them, as with CRASH-2 timing, is something you can act on while still calling it what it is.

A superiority trial tries to show a difference and treats failure as failure. A non-inferiority trial tries to show the new treatment is not unacceptably worse than the old — usually because it is cheaper, safer, oral, or available at 3 a.m. Its logic is inverted, and so are several of its incentives.

Everything hangs on the pre-specified non-inferiority margin Δ: the largest loss of efficacy you are willing to accept. Set it narrow and the trial is honest and expensive. Set it wide and almost anything passes. The margin should be justified from the size of the comparator's own established effect over placebo — preserving a stated fraction of it — and not from what makes the sample size affordable. A non-inferiority trial with no justification for its margin cannot be appraised, only believed or not.

Equivalence trials are two-sided non-inferiority: the interval must sit inside ±Δ. They are rare in clinical medicine outside bioequivalence. What is common, and wrong, is calling a failed superiority trial "equivalent" — you cannot switch design after the fact, because the failed trial never specified a margin.

Sensitivity and specificity are read down the columns — they condition on disease status, and are therefore roughly stable across populations of different prevalence (roughly, not exactly — see spectrum bias below). PPV and NPV are read across the rows, so they depend on prevalence and are not properties of the test at all. A d-dimer's NPV in a low-risk outpatient population and in a critical care unit are different numbers for the same assay. Quoting a PPV without the prevalence it was measured at is meaningless.

Likelihood ratios convert a pre-test probability into a post-test probability, which is what a clinician is actually doing when ordering a test. They work multiplicatively on odds:

Everything above assumed the test gives a yes or a no. Most do not — troponin, lactate, d-dimer and every clinical score produce a number, and somebody has to choose where to cut it.

Take every possible cut-off in turn. At each one, compute sensitivity and 1 − specificity, and plot that pair as a point. Join the points and you have the receiver operating characteristic curve. Each point on it is a cut-off, so the curve is not a property the test has in the abstract — it is the complete menu of trade-offs the test offers.

• The top-left corner is perfection: everyone with disease detected, nobody without it flagged.
• The diagonal is a coin toss — a test whose curve lies on it carries no information at all.
• Moving up and right means loosening the threshold: more sensitivity, more false positives. Moving down and left means tightening it.

The ROC figure shows a curve alongside the two distributions it was built from, which is the thing worth internalising: the curve is a summary of how much those two distributions overlap. Where they separate cleanly the curve bulges towards the corner; where they overlap heavily it collapses onto the diagonal.

The area under the curve (AUC, or c-statistic) summarises discrimination across all thresholds at once, and has a clean interpretation: the probability that a randomly chosen patient with the disease scores higher than a randomly chosen patient without it. 0.5 is chance; 1.0 is perfect. The conventional, and entirely arbitrary, descriptive bands are ~0.7–0.8 acceptable, 0.8–0.9 good, above 0.9 excellent.

The Youden index is the most commonly used rule: maximise (sensitivity + specificity − 1), which geometrically is the point on the curve furthest above the diagonal. It is marked on the figure. Its hidden assumption is the one that matters: Youden weights a false negative and a false positive equally.

In emergency medicine they are almost never equal. A missed subarachnoid haemorrhage and an unnecessary CT head are not comparable harms, and no arithmetic derived only from the curve knows that. So:

• For a rule-out decision, fix the sensitivity you require first — often 98–100 % — and read off whatever specificity that costs. The cut-off is then a clinical decision that the curve merely prices.
• For a rule-in decision, fix specificity and accept the sensitivity.
• Beware a cut-off that was chosen in the same dataset used to report its performance. Optimising the threshold and then quoting the accuracy achieved at it is a form of overfitting, and the figures will not reproduce in a validation set.
• Remember that sensitivity and specificity move in opposite directions along the curve, always. Any paper reporting that a new cut-off improved both should prompt you to check whether the population also changed.

• Spectrum bias. Accuracy measured in florid cases against healthy volunteers is always flattering. It must be measured in consecutive patients in whom the test would actually be used.
• Partial verification / referral bias. If the reference standard is applied preferentially to test-positives, sensitivity is inflated and specificity deflated.
• Incorporation bias. If the test result contributes to the reference standard, the test is being compared with itself.
• Differential verification. Test-positives get CT, test-negatives get a phone call at 30 days. These are not the same reference standard.
• Indeterminate results excluded. Dropping uninterpretable scans improves the numbers and misrepresents clinical reality.

QUADAS-2 in the Checklists tab is the structured way to walk through these.

Emergency medicine runs on prediction rules, and almost every argument about one comes down to confusing the three stages of its life.

• What was the outcome, exactly? "Clinically important" outcomes are defined by the authors, and the definition determines the numbers. PECARN's ciTBI is death from TBI, neurosurgery, intubation for more than 24 h, or admission for at least two nights — a definition that deliberately excludes CT findings that change nothing.
• Which patients were eligible? A rule validated in GCS 14–15 says nothing about GCS 13. This is the single commonest misapplication of a decision rule at the bedside.
• Is it a rule-out rule or a rule-in rule? Rule-out tools are built for sensitivity and will have poor specificity by design. Criticising a rule-out tool for over-investigating is criticising it for working.
• What is the miss rate you are accepting, and does it match the clinical stakes? A 2-in-3800 miss rate is acceptable for a condition where the missed cases are non-operative; it is not automatically acceptable elsewhere.
• Was it intended to override judgement or to support it? Most are explicitly the latter, and most disputes come from using them as the former.

A prognostic study asks what happens to this patient over time, not whether an intervention helps. Its central technical problem is censoring: most patients have not had the event when the study ends, and simply excluding them would throw away nearly all the data.

• Kaplan–Meier estimates the survival curve while handling censoring, on the assumption that censoring is non-informative — that those censored had the same future risk as those who remained. This fails badly when people are lost because they deteriorated.
• The log-rank test compares whole curves, and is the most powerful test available when hazards are proportional. Its weakness is the mirror image of that strength: it accumulates the difference between observed and expected events across the whole follow-up, so when curves cross, an early disadvantage and a late advantage cancel inside the statistic. Two obviously and importantly different curves can therefore return a comfortably non-significant log-rank p. A non-significant log-rank with crossing curves is a reason to look at the plot and at survival on a fixed date, not a reason to conclude there is no difference.
• Cox proportional hazards gives an adjusted hazard ratio, and assumes proportional hazards across follow-up. Where that fails, the honest reporting is restricted mean survival time, or a hazard ratio quoted separately by period.
• Competing risks. If a patient can die of something else first, they can never have the event of interest — and Kaplan–Meier will overstate cumulative incidence. In elderly and critically ill cohorts, competing risks are the norm, and the correct tool is a cumulative incidence function, not 1 minus KM.

For appraising a prognostic model specifically, the relevant reporting standard is TRIPOD, and the risk-of-bias tool is PROBAST; for a prognostic factor study, QUIPS.

A systematic review is a study whose unit of analysis is the study: a pre-registered question, an explicit search, duplicate screening and extraction, formal risk-of-bias assessment, and a stated synthesis plan. A meta-analysis is the optional statistical pooling step. A review can be excellent without pooling, and pooling without the rest is not a systematic review.

A fixed-effect model assumes one true effect and that differences between studies are sampling noise. A random-effects model assumes the true effect varies between studies and estimates the distribution's mean. Random effects gives wider intervals and weights small studies relatively more — which matters, because small studies are the ones most susceptible to bias. Neither model repairs heterogeneity; they make different assumptions about it.

1. The studies — how many, how large, and are they the population you care about?
2. The individual intervals — do they overlap, and do any sit clearly apart?
3. The weights — is the pooled estimate one large trial with decoration, or a genuine synthesis?
4. The diamond — its centre and, more importantly, its width against the line of no effect.
5. The heterogeneity line — I², τ², and a prediction interval if offered.
6. The risk-of-bias column — and whether a sensitivity analysis restricted to low-risk studies changes the answer. If removing the high-risk studies removes the effect, the effect belongs to the bias.

Small studies with unremarkable results are less likely to be published, so meta-analyses over-represent positive small studies and overestimate effects. A funnel plot shows effect against precision; asymmetry suggests missing small negative studies. Egger's and Begg's tests formalise it. All of them are underpowered below roughly ten studies, and asymmetry has innocent explanations — genuine small-study effects, differing quality, or a real relationship between dose and population risk. Treat funnel asymmetry as a prompt, not a verdict.

• Individual patient data (IPD) meta-analysis obtains the raw data, allowing consistent outcome definitions, proper time-to-event analysis and credible subgroup work. It is the strongest form and the rarest.
• Network meta-analysis compares treatments never trialled head to head, through their common comparators. It adds one critical assumption — transitivity — and requires that the trials be similar enough for indirect comparison to mean anything. Check the network diagram for how much of the conclusion rests on indirect evidence, and look for a formal test of inconsistency.

GRADE separates two judgements that are constantly run together: how certain are we about the effect, and what should we therefore do. They can come apart in both directions, and understanding that is most of what a clinician needs from GRADE.

Certainty is rated per outcome, not per study and not per guideline. RCT bodies start high, observational bodies start low, and the rating then moves.

The four resulting levels — high, moderate, low, very low — are statements about how likely further research is to change the estimate, not about study design alone.

A strong recommendation ("we recommend") means almost all informed patients would choose this, and it can reasonably become a quality standard. A conditional or weak recommendation ("we suggest") means the right choice depends on values and circumstances, and shared decision-making is the point rather than a courtesy. Strength depends on certainty plus the balance of benefits and harms, the variability of patient values, and resource use.

The tool is AGREE II, but three questions do most of the work: who wrote it and who paid (conflicts declared and managed, chair independence); is the link from evidence to recommendation traceable (does each recommendation carry its evidence and grade, or does the reader have to take it on trust); and is it current (a named review date, with a statement of what has changed). A guideline recommendation quoted without its number and its grade cannot be checked, which is why both belong in any citation of one.

Everything above assumes the paper is an honest report of what was done. That assumption is usually safe and is not free.

Prospective registration exists so that the primary outcome cannot be chosen after the results are known. Comparing the registry entry with the publication is a five-minute check that catches the single most consequential form of reporting bias: a primary outcome demoted, a secondary promoted, or a new outcome appearing. Registration numbers are given precisely so this is possible — CRASH-2 lists ISRCTN86750102 and NCT00375258; HALT-IT lists ISRCTN11225767 and NCT01658124; PROPPR lists NCT01545232; ANDROMEDA-SHOCK lists NCT03078712.

Spin is presentation that implies more than the data support without stating anything false. Its recurring forms:

• A conclusion resting on a secondary or subgroup outcome while the primary was null.
• "A trend towards benefit" for a non-significant result — there is no such thing as a trend in a single trial; there is an estimate and an interval.
• Relative risk reductions in the abstract, absolute risks nowhere.
• "Well tolerated" in place of the harms table.
• A title that states a causal claim an observational design cannot support.
• An abstract conclusion broader than the population enrolled.

A declared commercial interest does not invalidate a study, and its absence does not sanctify one — academic, intellectual and career conflicts are real and rarely declared. What matters is whether the funder had a structural role in the parts that can be steered: design, conduct, analysis, and the right to publish a negative result. Note that CRASH-2's funding included both public (UK NIHR Health Technology Assessment programme) and commercial (Pfizer) sources, alongside charitable funders, and that it reported a modest effect on a hard outcome — the combination is worth noting rather than either dismissing or ignoring.

Early positive results are, on average, exaggerated — a consequence of publication bias, small samples, flexible analysis and the winner's curse of reporting whichever estimate crossed the threshold. The practical implication is a prior, not a cynicism: expect a first striking result to shrink on replication, and be slower to adopt on a single trial than the trial's own confidence interval suggests. In a field where a single-centre trial can change practice within a month, that prior is the most useful thing on this page.

Paper mills, fabricated datasets and undisclosed generative-AI text are now part of the landscape, most heavily in low-barrier journals. Signals worth noticing: implausibly clean baseline tables, distributions too uniform, an author list with no traceable institutional link to the work, and reference lists containing citations that do not resolve. Check that a citation exists and says what it is claimed to say — the most reliably detectable defect in an unsound paper is not its statistics but its references.

Qualitative work answers questions quantitative work cannot: why a pathway is not followed, what a diagnosis means to a patient, how a team actually makes a decision under pressure. Appraising it against RCT criteria is a category error — sample size, blinding and generalisability are the wrong questions. The right ones are:

• Is the method matched to the question? Grounded theory to build explanation, phenomenology for lived experience, ethnography for practice as it happens, framework analysis for applied policy questions.
• Sampling — purposive and justified, rather than convenient, and pursued to the point where new data stop adding categories.
• Reflexivity — has the researcher's own position and its influence on the data been examined and stated?
• Analytic rigour — independent coding, an audit trail from data to theme, and negative cases sought and reported rather than smoothed away.
• Are the data shown? Themes should be evidenced with quotations that a reader can weigh, not merely asserted.
• Transferability, not generalisability — enough contextual description for you to judge whether it applies to your department.

The reporting standards are COREQ (interviews and focus groups) and SRQR; CASP publishes a qualitative checklist, and GRADE-CERQual rates confidence in findings from qualitative evidence synthesis.

The question to ask is why the two strands were combined and how they were integrated. A paper that reports a survey and an interview study side by side, with no integration, is two papers sharing a title. Genuine mixed-methods work states its sequence (explanatory, exploratory or convergent) and shows where the strands informed each other.

An economic evaluation compares costs and consequences of alternatives. Key features to check: the perspective (health service versus societal — it changes which costs count), the time horizon and discount rate, whether the outcome is a cost-effectiveness ratio (cost per event avoided) or a cost–utility ratio (cost per QALY), the incremental cost-effectiveness ratio rather than average costs, and the sensitivity analysis — a probabilistic one with a cost-effectiveness acceptability curve, since a point ICER without uncertainty is not a result. The reporting standard is CHEERS. Be alert to models whose conclusion is driven by one poorly evidenced input; the sensitivity analysis is where that shows, which is why it is the section to read first.

This is the sequence to use on a paper someone has just sent you claiming it changes everything. It is deliberately ordered so that the cheapest disqualifying questions come first.

1. Write your PICO question (30 s). What would you change if this were true?
2. Title, journal, date, design (30 s). Do not read the abstract conclusion yet.
3. Population (1 min). Eligibility criteria and the baseline table. Is your patient in this trial? This step disqualifies more papers than every statistical consideration combined.
4. Primary outcome, as pre-specified (1 min). Is it patient-important? Is it a composite? Check the registry entry if something feels moved.
5. Allocation and blinding (1 min). Concealed? Who was masked? How subjective is the outcome?
6. The flow diagram (1 min). Randomised versus analysed, in both arms. What happened to the difference?
7. The primary result, as an absolute number with its interval (2 min). Convert it yourself. Compute the NNT. Read both ends of the interval as clinical scenarios.
8. Harms (1 min). Was the trial capable of detecting the harm that would matter, and is there a number needed to harm?
9. Applicability (1 min). Your department's baseline risk, your case mix, whether you can actually deliver the intervention in the time window the trial used.
10. Now read the abstract's conclusion (30 s), and notice any gap between it and what you just concluded. That gap is the appraisal.

• Pick papers that could change practice, not papers that are easy to criticise. Demolishing a bad paper teaches nothing anyone can use on shift.
• Circulate the paper and the PICO question in advance, and ask everyone to write down their prediction of the result. Predictions expose priors, and priors are what actually get updated.
• Assign the domains — one person on population, one on outcome definition, one on the analysis, one on harms. Structure beats enthusiasm.
• Appoint someone to defend the paper. Journal clubs drift into unanimous criticism, which is comfortable and uninstructive.
• Finish with a decision, recorded: adopt, adopt with caveats, await replication, or reject — and who will check whether local practice actually changed. A journal club without an owned outcome is a reading group.
• Keep the appraisals. A department that can find last year's appraisal of the trial now being cited at it has an institutional memory, which is rarer and more valuable than any individual's appraisal skill.

Appraisal is not a search for reasons to dismiss. The commonest failure among people who have just learned it is reflexive scepticism, which is indistinguishable in its effects from credulity: both end in practice unchanged by evidence. The aim is calibration — believing large, well-conducted, replicated findings about patients like yours, believing them proportionately less as each of those conditions weakens, and being able to say out loud which condition is doing the work in any given case.