Critical appraisal for speech and language therapists: observational study

Cohort study: People with something in common are recruited and followed up over time (remember those Roman soldiers marching off into the future) in order to observe outcomes of interest. NB Recruitment happens before the appearance of the outcome. This is the strongest observational method (next best to experimental) for providing evidence for questions about causal relationships, such as, Do children with language impairment develop poorer social networks than their peers? or, Are people who have received (a new medical or surgical intervention) less likely to need speech and language therapy follow-up than those receiving conventional treatment? However, check how well the findings support inference of causality. Case-controlled study: People with a particular outcome (cases) are identified and matched with controls in order to answer questions such as, Are young people not in employment, education or training more likely to have a history of primary language impairment? Case-controlled studies are good for investigating rare or negative outcomes, but they are more susceptible to bias so provide less robust evidence than cohort studies, and their power is limited to evidence of strength of association. The inference of causality from findings of an individual study is dodgy it should be done only in the context of a broader evidence base. Cross-sectional survey: A representative sample of patients, practitioners, carers etc. are interviewed, examined or otherwise studied to gain information on a clinical question, such as, Are speech and language therapy services accessible to children in local authority care? or, What influences speech and language therapy intervention for patients with [condition]?. The data for cross-sectional studies are collected at a single time point but may refer to retrospective information, such as what intervention people with aphasia received during the first year following a stroke. It is even dodgier to infer causality from the results of a survey! NB There is a separate tool for surveys that use questionnaires. Use the following questions to appraise a report of a case-controlled study or cross-sectional survey with respect to the study results, their validity and whether they will help with clinical practice or decision-making. There is space for you to add your own notes. The prompts are there to highlight why an issue is important but they shouldnt limit your thinking. There is quite a lot of overlap between questions.

Study reference:

Did the study address a clearly focused issue? Which population was studied? Which risk factors or outcomes were investigated? Did they try to detect a beneficial or harmful effect? Is the underlying issue one of causation? Can you formulate aims into a clinical question? Is it important for your practice?

Aims

Adapted from Public Health Resource Unit, England (2006) Critical Appraisal Skills Programme by Jennifer Reid, Speech and Language Therapy Service, NHS Fife. Cartoons by Fran commissioned by Speech and Language Therapy in Practice. Accompanies Reid, J. (2011) Journal Club 5: observational studies, Speech & Language Therapy in Practice Autumn, pp.18-21. Please attribute these sources if you use, distribute or adapt any of this material.

Method

Choice of method appropriate to answer the study question? Did method address the study question? Is an observational design an appropriate way of addressing the question under the circumstances? Is this a cohort, case-controlled or cross-sectional study? If a cohort study, did recruitment precede occurrence of outcome? If a case-controlled study, is the outcome rare or harmful? Authors cautious about inferring causality?

Adapted from Public Health Resource Unit, England (2006) Critical Appraisal Skills Programme by Jennifer Reid, Speech and Language Therapy Service, NHS Fife. Cartoons by Fran commissioned by Speech and Language Therapy in Practice. Accompanies Reid, J. (2011) Journal Club 5: observational studies, Speech & Language Therapy in Practice Autumn, pp.18-21. Please attribute these sources if you use, distribute or adapt any of this material.

Sampling and recruitment

Were the cases recruited in an acceptable way? Look for selection bias which might compromise the validity of the findings. Participants/cases representative of defined population (geographically and/or temporally)? Time frame of prospective study relevant to the development of the outcome? How has attrition (loss to follow-up) been dealt with? Caseness defined precisely? (One or more wrongly allocated individuals may seriously influence the results of case-controlled study.) Are cases incident or prevalent, and is this appropriate for addressing question? Anything special about the cases (that would hinder generalisation of findings)? Sufficient number of participants selected? Power calculation used?

Were the controls selected in an acceptable way? Look for selection bias which might compromise the extent to which the findings can be generalised. Anything special about the controls? Non-response high? Could non-respondents be different in any way? Established reliable system for selecting all the cases? Are they matched, population-based or randomly selected? Rationale okay? Systematic methods used for matching? Sufficient number of controls selected?

Adapted from Public Health Resource Unit, England (2006) Critical Appraisal Skills Programme by Jennifer Reid, Speech and Language Therapy Service, NHS Fife. Cartoons by Fran commissioned by Speech and Language Therapy in Practice. Accompanies Reid, J. (2011) Journal Club 5: observational studies, Speech & Language Therapy in Practice Autumn, pp.18-21. Please attribute these sources if you use, distribute or adapt any of this material.

Were phenomena (both cohort/caseness criteria and outcomes) measured accurately enough to minimise bias? Look for measurement, recall or classification bias. Phenomena defined clearly and measured accurately? Subjective or objective measurements? Do the measures measure what they are supposed to measure? Have they been validated? Measurement methods similar in cases and controls? Blinding (case vs. control) incorporated where feasible?

Accurate measurement

What confounding factors have the authors accounted for?

Have they missed any important ones (genetic; environmental; socio-economic)?

Confounding
The influence of unforeseen factors

Have they taken account of potential confounding factors in the design and/or analysis? Look for restrictions in design and techniques to correct, control or adjust for confounding factors e.g. modelling, stratified-, regression-, or sensitivity analysis. Youll need to dig in the numbers section (eek!) and look for phrases like, once the effect of age was controlled for

Adapted from Public Health Resource Unit, England (2006) Critical Appraisal Skills Programme by Jennifer Reid, Speech and Language Therapy Service, NHS Fife. Cartoons by Fran commissioned by Speech and Language Therapy in Practice. Accompanies Reid, J. (2011) Journal Club 5: observational studies, Speech & Language Therapy in Practice Autumn, pp.18-21. Please attribute these sources if you use, distribute or adapt any of this material.

Results

What are the results of this study? In a nutshell How are the results expressed? Analysis appropriate to the design? How strong is the association between the phenomena being measured? Is it negative or positive, and does this make intuitive sense? Are the results adjusted for confounding? Might confounding still explain the association? Has adjustment made a big difference to the overall result? Do the findings answer the study questions?

How precise are the results and the estimate of risk? Size of p-value or confidence intervals? Have they considered all the important variables? How did they evaluate the effect of individuals refusing to participate?

Do you believe the results? Consider: A big effect is hard to ignore, but only if high quality study design and methods. Could the results be due to chance, bias or confounding? Design and methods so flawed as to make the results unreliable? Can this study support causal reasoning? (NB Cohort study is usually stronger design for investigating causality.) Consider Bradford Hill criteria for inferring causality.

Findings

Adapted from Public Health Resource Unit, England (2006) Critical Appraisal Skills Programme by Jennifer Reid, Speech and Language Therapy Service, NHS Fife. Cartoons by Fran commissioned by Speech and Language Therapy in Practice. Accompanies Reid, J. (2011) Journal Club 5: observational studies, Speech & Language Therapy in Practice Autumn, pp.18-21. Please attribute these sources if you use, distribute or adapt any of this material.

 Do the results of this study fit with other available evidence? Consider evidence from other studies (all types). A well-conducted systematic review would be particularly helpful. You may need to conduct (or commission) a literature review and appraise the quantity and quality of the available evidence before you are able to assess this. Bear in mind costs and benefits - the issue in question will be particularly important (i.e. specific, relevant, timely, with resource implications) for your service before deciding to invest resources in a more comprehensive review of the evidence base in this area. Fit with other evidence

Can the results be applied to the local population? To what extent, if any, can we generalise these findings? Do you have concerns that the study participants are different from your clients in important ways? Does your local setting differ much from that of the study? Can you estimate the local benefits and harms?

Application

Does this study have implications for my practice for my colleagues or care group for the service as a whole? Is there a further question to be asked? NB One observational study rarely provides sufficiently robust evidence to recommend changes to clinical practice or decision-making. However, for certain questions observational studies provide the only evidence. Recommendations from observational studies are always stronger when supported by other evidence.

Impact Summary

Adapted from Public Health Resource Unit, England (2006) Critical Appraisal Skills Programme by Jennifer Reid, Speech and Language Therapy Service, NHS Fife. Cartoons by Fran commissioned by Speech and Language Therapy in Practice. Accompanies Reid, J. (2011) Journal Club 5: observational studies, Speech & Language Therapy in Practice Autumn, pp.18-21. Please attribute these sources if you use, distribute or adapt any of this material.