What are Systematic Reviews?

Get the big picture, with less bias
Taken from http://informedhealthonline.org, written by Hilda Bastian. 20 September 2003

How can we know whether a treatment works?

Some healthcare treatments can make you better but some are fairly useless. Sometimes the treatment can be worse than the disease. At other times it can look as though a drug or other treatment helped, while really the benefit came from something else that happened at the same time (like having a couple of days in bed). Or, the disease just ran its course and you would have got better anyway.

If you want to know if something is really worth trying, you need good evidence about the treatment’s effects – ideally from high quality trials.

A trial is an experiment. People are assigned into separate groups to receive the different forms of care that are to be compared. Sometimes, this also includes a placebo (dummy) treatment. The experiences, progress and outcomes of people in these separate groups (who should be much the same except for the difference in treatments) are then compared.

In randomised trials, people are assigned into the groups in random ways that trial to eliminate the potential for manipulation of who gets what treatment. Trials are the only way to be sure that you have eliminated, as far as possible, potential factors other than the treatment that could be responsible for improvements or adverse effects.

Not just one study at a time
Only rarely is a single trial big and strong enough to provide, on its own, a reliable answer about the effects of a health care treatment. One small trial could suggest something works – but it could be outweighed by another 10 trials showing the opposite.

Often only those trials showing benefit from treatment are made public, while the results of trials showing no benefit remain unpublished. This can make healthcare treatments look more effective than they really are.

A systematic review can combine the results of many trials and give you the state of the art in research knowledge on that healthcare treatment.

Getting the best answer – step by step

Systematic reviews involve a systematic, step by step process. People doing a systematic review need to:

1. Know what kind of research has the power to answer the question, and what criteria and standards each piece of that research has to meet.

2. Search thoroughly and as comprehensively as possible for every study, published and unpublished, that could fit the criteria.

3. Apply the criteria, objectively, to determine which research to include – the goal is not to pick and choose studies according to personal, ideological or commercial biases.Objectively assess each study, and then look comprehensively at the bigger picture – what do we know, what don’t we know and how the results can be interpreted.

4. Test the process, results and interpretations in the review with an international group of experts, including consumers (people affected by the condition or treatment).

5. Report on the steps taken so that others can follow the research path and find out if mistakes were made, if other interpretations may be just as valid, and whether the conclusions are reliable.

Staying up-to-date

Cochrane reviews go through all the steps of systematic review – and they go one step further. The reviews are updated. The Cochrane Collaboration’s goal is to check every review, at least every two years, and update them when new research or other information emerges. The Collaboration is making steady progress towards reaching this goal.

The Cochrane Collaboration process includes an extensive hunt for all trials – whether they are published or not, and whether they are in English or another language. The Collaboration has built the world’s largest database of trials, called The Cochrane Controlled Trials Register (the CCTR). By late 2003, the CCTR included nearly 380,000 trials, and it’s still growing. The CCTR is published in The Cochrane Library.

The final result, a Cochrane review, provides the most reliable answer known to the question of “Does this treatment work?”

Less bias and answers for the future
Many health care treatments will not stand up to this level of scrutiny. The final “verdict” will be that there simply is not enough strong evidence to give us an answer. While that is disappointing when you are looking for an answer, it at least stops us being mislead by over-optimistic (or over-pessimistic) beliefs, and by ideological, professional and commercial biases. It also provides a guide to where more research is needed, so that if not now, then at least in the future we can have an independent, scientifically rigorous and reliable answer about what treatments might be able to help us.

Hilda Bastian. 20 September 2003.