Randomized, Controlled Trials, Observational Studies, and the Hierarchy of Research Designs
et al., New England Journal of Medicine, doi:10.1056/NEJM200006223422507, Jun 2000
Meta-analysis comparing 99 studies (both randomized controlled trials and observational studies) across five clinical topics, showing remarkably similar results between study designs contrary to the prevailing belief that observational studies overestimate treatment effects. The analysis included studies on BCG vaccine effectiveness against tuberculosis, screening mammography for breast cancer mortality, cholesterol levels and trauma deaths, and hypertension treatment effects on stroke and coronary heart disease. Observational studies actually showed less variability in results than RCTs, with some RCTs paradoxically showing results opposite to the pooled estimates. Authors hypothesize that observational studies may include broader, more representative populations and reflect real-world clinical practice better than the restricted populations and protocols of RCTs.
Concato et al., 22 Jun 2000, peer-reviewed, 3 authors.
Contact: john.concato@yale.edu.
RANDOMIZED, CONTROLLED TRIALS, OBSERVATIONAL STUDIES, AND THE HIERARCHY OF RESEARCH DESIGNS
Background In the hierarchy of research designs, the results of randomized, controlled trials are considered to be evidence of the highest grade, whereas observational studies are viewed as having less validity because they reportedly overestimate treatment effects. We used published meta-analyses to identify randomized clinical trials and observational studies that examined the same clinical topics. We then compared the results of the original reports according to the type of research design. Methods A search of the Medline data base for ar- ticles published in five major medical journals from 1991 to 1995 identified meta-analyses of randomized, controlled trials and meta-analyses of either cohort or case-control studies that assessed the same intervention. For each of five topics, summary estimates and 95 percent confidence intervals were calculated on the basis of data from the individual randomized, controlled trials and the individual observational studies.
Results For the five clinical topics and 99 reports evaluated, the average results of the observational studies were remarkably similar to those of the randomized, controlled trials. For example, analysis of 13 randomized, controlled trials of the effectiveness of bacille Calmette-Guérin vaccine in preventing active tuberculosis yielded a relative risk of 0.49 (95 percent confidence interval, 0.34 to 0.70) among vaccinated patients, as compared with an odds ratio of 0.50 (95 percent confidence interval, 0.39 to 0.65) from 10 case-control studies. In addition, the range of the point estimates for the effect of vaccination was wider for the randomized, controlled trials (0.20 to 1.56) than for the observational studies (0.17 to 0.84).
Conclusions The results of well-designed observa- tional studies (with either a cohort or a case-control design) do not systematically overestimate the magnitude of the effects of treatment as compared with those in randomized, controlled trials on the same topic.
References
Abel, Koch, The role of randomization in clinical studies: myths and beliefs, J Clin Epidemiol
Angell, Kassirer, Alternative medicine -the risks of untested and unregulated remedies, N Engl J Med
Angus, Birmingham, Balk, E5 murine monoclonal antiendotoxin antibody in gram-negative sepsis: a randomized controlled trial, JAMA
Byar, Simon, Friedewald, Randomized clinical trials: perspectives on some recent ideas, N Engl J Med
Chalmers, Celano, Sacks, Smith, Bias in treatment assignment in controlled clinical trials, N Engl J Med
Colditz, Brewer, Berkey, Efficacy of BCG vaccine in the prevention of tuberculosis: meta-analysis of the published literature, JAMA
Collins, Peto, Macmahon, Blood pressure, stroke, and coronary heart disease. 2. Short-term reductions in blood pressure: overview of randomised drug trials in their epidemiological context, Lancet
Cummings, Psaty, The association between cholesterol and death from injury, Ann Intern Med
Demissie, Mills, Rhoads, Empirical comparison of the results of randomized controlled trials and case-control studies in evaluating the effectiveness of screening mammography, J Clin Epidemiol
Dersimonian, Laird, Meta-analysis in clinical trials, Control Clin Trials
Feinstein, Current problems and future challenges in randomized clinical trials, Circulation
Fleiss, The statistical basis of meta-analysis, Stat Methods Med Res
Guyatt, Sackett, Sinclair, Hayward, Cook et al., Users' guide to the medical literature. IX. A method for grading health care recommendations, Erratum
Horn, Evolving strategies in the treatment of sepsis and systemic inflammatory response syndrome (SIRS), QJM
Horwitz, Complexity and contradiction in clinical trial research, Am J Med
Horwitz, Feinstein, Methodologic standards and contradictory results in case-control research, Am J Med
Horwitz, Viscoli, Clemens, Sadock, Developing improved observational methods for evaluating therapeutic effectiveness, Am J Med
Jacobs, Blackburn, Higgins, Report of the Conference on Low Blood Cholesterol: mortality associations, Circulation
Juni, Witschi, Bloch, Egger, The hazards of scoring the quality of clinical trials for meta-analysis, JAMA
Kerlikowske, Grady, Rubin, Sandrock, Ernster, Efficacy of screening mammography: a meta-analysis, JAMA
Kunz, Oxman, The unpredictability paradox: review of empirical comparisons of randomised and non-randomised clinical trials, BMJ
Lelorier, Grégoire, Benhaddad, Lapierre, Derderian, Discrepancies between meta-analyses and subsequent large randomized, controlled trials, N Engl J Med
Lipsey, Wilson, The efficacy of psychological, educational, and behavioral treatment: confirmation from meta-analysis, Am Psychol
Macmahon, Peto, Cutler, Blood pressure, stroke, and coronary heart disease. 1. Prolonged differences in blood pressure: prospective observational studies corrected for the regression dilution bias, Lancet
Mckee, Britton, Black, Mcpherson, Sanderson et al., Methods in health services research: interpreting the evidence: choosing between randomised and non-randomised studies, BMJ
Medicine, Group, Evidence-based medicine: a new approach to teaching the practice of medicine, JAMA
Sacks, Chalmers, Smith, Randomized versus historical controls for clinical trials, Am J Med
Sacks, Chalmers, Smith, Sensitivity and specificity of clinical trials: randomized v historical controls, Arch Intern Med
Schulz, Chalmers, Hayes, Altman, Empirical evidence of bias: dimensions of methodological quality associated with estimates of treatment effects in controlled trials, JAMA
DOI record:
{
"DOI": "10.1056/nejm200006223422507",
"ISSN": [
"0028-4793",
"1533-4406"
],
"URL": "http://dx.doi.org/10.1056/NEJM200006223422507",
"alternative-id": [
"10.1056/NEJM200006223422507"
],
"author": [
{
"affiliation": [],
"family": "Concato",
"given": "John",
"sequence": "first"
},
{
"affiliation": [],
"family": "Shah",
"given": "Nirav",
"sequence": "additional"
},
{
"affiliation": [],
"family": "Horwitz",
"given": "Ralph I.",
"sequence": "additional"
}
],
"container-title": "New England Journal of Medicine",
"container-title-short": "N Engl J Med",
"content-domain": {
"crossmark-restriction": false,
"domain": []
},
"created": {
"date-parts": [
[
2002,
7,
27
]
],
"date-time": "2002-07-27T08:56:19Z",
"timestamp": 1027760179000
},
"deposited": {
"date-parts": [
[
2017,
12,
16
]
],
"date-time": "2017-12-16T20:12:52Z",
"timestamp": 1513455172000
},
"indexed": {
"date-parts": [
[
2025,
12,
21
]
],
"date-time": "2025-12-21T17:52:26Z",
"timestamp": 1766339546970
},
"is-referenced-by-count": 2540,
"issue": "25",
"issued": {
"date-parts": [
[
2000,
6,
22
]
]
},
"journal-issue": {
"issue": "25",
"published-print": {
"date-parts": [
[
2000,
6,
22
]
]
}
},
"language": "en",
"link": [
{
"URL": "http://www.nejm.org/doi/pdf/10.1056/NEJM200006223422507",
"content-type": "unspecified",
"content-version": "vor",
"intended-application": "similarity-checking"
}
],
"member": "150",
"original-title": [],
"page": "1887-1892",
"prefix": "10.1056",
"published": {
"date-parts": [
[
2000,
6,
22
]
]
},
"published-print": {
"date-parts": [
[
2000,
6,
22
]
]
},
"publisher": "Massachusetts Medical Society",
"reference": [
{
"DOI": "10.1136/bmj.2.4582.769",
"doi-asserted-by": "publisher",
"key": "r001"
},
{
"DOI": "10.1056/NEJM197607082950204",
"doi-asserted-by": "publisher",
"key": "r002"
},
{
"DOI": "10.1161/01.CIR.70.5.767",
"doi-asserted-by": "publisher",
"key": "r003"
},
{
"DOI": "10.1016/S0895-4356(99)00041-4",
"doi-asserted-by": "publisher",
"key": "r004"
},
{
"DOI": "10.1016/0002-9343(82)90815-4",
"doi-asserted-by": "publisher",
"key": "r005"
},
{
"DOI": "10.1001/jama.268.17.2420",
"doi-asserted-by": "publisher",
"key": "r006"
},
{
"key": "r007",
"unstructured": "Preventive Services Task Force. Guide to clinical preventive services: report of the U.S. Preventive Services Task Force. 2nd ed. Baltimore: Williams & Wilkins, 1996."
},
{
"DOI": "10.1001/jama.273.5.408",
"doi-asserted-by": "publisher",
"key": "r008"
},
{
"DOI": "10.1016/0002-9343(90)90182-D",
"doi-asserted-by": "publisher",
"key": "r009"
},
{
"DOI": "10.1016/0002-9343(79)91164-1",
"doi-asserted-by": "publisher",
"key": "r010"
},
{
"DOI": "10.1016/0197-2456(86)90046-2",
"doi-asserted-by": "publisher",
"key": "r011"
},
{
"DOI": "10.1177/096228029300200202",
"doi-asserted-by": "publisher",
"key": "r012"
},
{
"DOI": "10.1016/0140-6736(90)90878-9",
"doi-asserted-by": "publisher",
"key": "r013"
},
{
"DOI": "10.1001/jama.271.9.698",
"doi-asserted-by": "publisher",
"key": "r014"
},
{
"DOI": "10.1001/jama.273.2.149",
"doi-asserted-by": "publisher",
"key": "r015"
},
{
"DOI": "10.7326/0003-4819-120-10-199405150-00006",
"author": "Cummings P",
"doi-asserted-by": "crossref",
"first-page": "848",
"journal-title": "Ann Intern Med",
"key": "r016",
"volume": "120",
"year": "1994"
},
{
"DOI": "10.1161/01.CIR.86.3.1046",
"author": "Jacobs D",
"doi-asserted-by": "crossref",
"first-page": "1046",
"journal-title": "Circulation",
"key": "r017",
"volume": "86",
"year": "1992"
},
{
"DOI": "10.1016/0140-6736(90)90944-Z",
"doi-asserted-by": "publisher",
"key": "r018"
},
{
"DOI": "10.1136/bmj.319.7205.312",
"author": "McKee M",
"doi-asserted-by": "crossref",
"first-page": "312",
"journal-title": "BMJ",
"key": "r019",
"volume": "319",
"year": "1999"
},
{
"DOI": "10.1001/jama.247.12.1707",
"doi-asserted-by": "publisher",
"key": "r020"
},
{
"DOI": "10.1037/0003-066X.48.12.1181",
"doi-asserted-by": "publisher",
"key": "r021"
},
{
"DOI": "10.1016/0002-9343(87)90450-5",
"doi-asserted-by": "publisher",
"key": "r022"
},
{
"DOI": "10.1093/qjmed/91.4.265",
"doi-asserted-by": "publisher",
"key": "r023"
},
{
"DOI": "10.1001/jama.283.13.1723",
"doi-asserted-by": "publisher",
"key": "r024"
},
{
"DOI": "10.1056/NEJM199708213370806",
"doi-asserted-by": "publisher",
"key": "r025"
},
{
"DOI": "10.1001/jama.274.22.1800",
"doi-asserted-by": "publisher",
"key": "r026"
},
{
"DOI": "10.1056/NEJM198312013092204",
"doi-asserted-by": "publisher",
"key": "r027"
},
{
"DOI": "10.1001/archinte.143.4.753",
"doi-asserted-by": "publisher",
"key": "r028"
},
{
"DOI": "10.1136/bmj.317.7167.1185",
"author": "Kunz R",
"doi-asserted-by": "crossref",
"first-page": "1185",
"journal-title": "BMJ",
"key": "r029",
"volume": "317",
"year": "1998"
},
{
"DOI": "10.1001/jama.282.11.1054",
"doi-asserted-by": "publisher",
"key": "r030"
},
{
"DOI": "10.1016/S0895-4356(97)00243-6",
"doi-asserted-by": "publisher",
"key": "r031"
},
{
"DOI": "10.1056/NEJM199809173391210",
"doi-asserted-by": "publisher",
"key": "r032"
}
],
"reference-count": 32,
"references-count": 32,
"relation": {},
"resource": {
"primary": {
"URL": "http://www.nejm.org/doi/abs/10.1056/NEJM200006223422507"
}
},
"score": 1,
"short-title": [],
"source": "Crossref",
"subject": [],
"subtitle": [],
"title": "Randomized, Controlled Trials, Observational Studies, and the Hierarchy of Research Designs",
"type": "journal-article",
"volume": "342"
}