Skip to Main Content

HPPA (Physician Assistant): Systematic Reviews

What is a Systematic Review?

  • Systematic reviews have increasingly replaced traditional narrative reviews and expert commentaries as a way of summarising research evidence.
  • Systematic reviews attempt to bring the same level of rigour to reviewing research evidence as should be used in producing that research evidence in the first place.
  • Systematic reviews should be based on a peer-reviewed protocol so that they can be replicated if necessary.
  • High quality systematic reviews seek to:
    • Identify all relevant published and unpublished evidence
    • Select studies or reports for inclusion
    • Assess the quality of each study or report
    • Synthesise the findings from individual studies or reports in an unbiased way
    • Interpret the findings and present a balanced and impartial summary of the findings with due consideration of any flaws in the evidence.
  • Many high quality peer-reviewed systematic reviews are available in journals as well as from databases and other electronic sources.
  • Systematic reviews may examine quantitative or qualitative evidence;put simply, when the two or more types of evidence are examined within one review it is called a mixed-method systematic review.
  • Systematic reviewing techniques are in a period of rapid development.Many systematic reviews still look at clinical effectiveness, but methods now exist to enable reviewers to examine issues of appropriateness,feasibility and meaningfulness.
  • Not all published systematic reviews have been produced with meticulous care; therefore, the findings may sometimes mislead.Interrogating published reports by asking a series of questions can uncover deficiencies.

Credit: Hemingway, P., & Brereton, N. (2009). What is a systematic review. What is, 1-8.

Cochrane Database of Systematic Reviews

A systematic review summarises the results of available carefully designed healthcare studies (controlled trials) and provides a high level of evidence on the effectiveness of healthcare interventions. Judgments may be made about the evidence and inform recommendations for healthcare.

These reviews are complicated and depend largely on what clinical trials are available, how they were carried out (the quality of the trials) and the health outcomes that were measured. Review authors pool numerical data about effects of the treatment through a process called meta-analyses. Then authors assess the evidence for any benefits or harms from those treatments. In this way, systematic reviews are able to summarise the existing clinical research on a topic.

The review plan

Review authors set about their task very methodically following, step by step, an advance plan called a protocol. The protocol describes the steps that will be followed when preparing a review. Cochrane protocols are published in the Cochrane Library so that people can provide comments to improve them before the actual review has been conducted.  A protocol describes:

  • the way existing studies are found;
  • how the relevant studies are judged in terms of their usefulness in answering the review question;
  • how the results of the separate studies are brought together to give an overall measure of effectiveness (benefits and harms) - statistical techniques used to combine the results are called meta-analysis.

The review question

The purpose of the review is generally stated as: To assess the effects of [intervention or comparison] for [health problem] in [types of people, disease or problem], and healthcare setting if appropriate. The parts of the review question are often referred to as ‘PICO’ (Participants, Interventions, Comparisons and Outcomes). The included studies generally randomly assign participants to the intervention under investigation or the control or comparative intervention. 

The review title

Titles of Cochrane reviews also have a set layout: Intervention for problem in a disease or population, and sometimes an outcome. An example is:  Surgical excision margins for primary cutaneous melanoma. This is a statement of the types of population (participants in controlled clinical studies), types of interventions (and what they are compared to, even if it is no treatment), and the types of outcomes that are of interest.

Cochrane Handbook of Systematic Reviews  of Interventions (Version 6, September 2018)

Credit: Cochrane Database of Systematic Reviews

Searching the Cochrane Database of Systematic Reviews

Five Steps to Conducting a Systematic Review

Step 1: Framing questions for a review

The problems to be addressed by the review should bespecified in the form of clear, unambiguous and structuredquestions before beginning the review work. Once the reviewquestions have been set, modifications to the protocol shouldbe allowed only if alternative ways of defining the populations,interventions, outcomes or study designs become apparent

Step 2: Identifying relevant work

The search for studies should be extensive. Multiple resources(both computerized and printed) should be searched withoutlanguage restrictions. The study selection criteria should flowdirectly from the review questions and be specifiedapriori.Reasons for inclusion and exclusion should be recorded

Step 3: Assessing the quality of studies

Study quality assessment is relevant to every step of a review.Question formulation (Step 1) and study selection criteria (Step2) should describe the minimum acceptable level of design.Selected studies should be subjected to a more refined qualityassessment by use of general critical appraisal guides anddesign-based quality checklists (Step 3). These detailedquality assessments will be used for exploring heterogeneityand informing decisions regarding suitability of meta-analysis(Step 4). In addition they help in assessing the strength ofinferences and making recommendations for future research(Step 5)

Step 4: Summarizing the evidence

Data synthesis consists of tabulation of study characteristics,quality and effects as well as use of statistical methods forexploring differences between studies and combining theireffects (meta-analysis). Exploration of heterogeneity and itssources should be planned in advance (Step 3). If an overallmeta-analysis cannot be done, subgroup meta-analysis maybe feasible

Step 5: Interpreting the findings

The issues highlighted in each of the four steps above shouldbe met. The risk of publication bias and related biases shouldbe explored. Exploration for heterogeneity should helpdetermine whether the overall summary can be trusted, and, ifnot, the effects observed in high-quality studies should beused for generating inferences. Any recommendations shouldbe graded by reference to the strengths and weaknesses of systematic review.

Credit: Khan, K. S., Kunz, R., Kleijnen, J., & Antes, G. (2003). Five steps to conducting a systematic review. Journal of the royal society of medicine, 96(3), 118-121.

Cochrane SA Webinar: Developing a search strategy for systematic reviews

Five Levels of Systematic Reviews

Hierarchies of evidence for questions of therapy, prevention, aetiology or harm:

Level 1a Systematic review (with homogeneity) of randomised controlled trials (RCTs)

Level 1b Individual RCT (with narrow confidence interval)

Level 1c All-or-none studies

Level 2a Systematic review (with homogeneity) of cohort studies

Level 2b Individual cohort study (including low quality RCT; eg <80% follow-up)

Level 2c ‘Outcomes’ research; ecological studies

Level 3a Systematic reviews (with homogeneity) of case-control studies

Level 3b Individual case-control study

Level 4 Case series (and poor quality cohort and case-control studies)

Level 5 Expert opinion without explicit critical appraisal, or based on physiology, bench research on 'first principles'

Beyond The Search: Maximizing the Quality of Systematic Reviews