Close
About
FAQ
Home
Collections
Login
USC Login
Register
0
Selected
Invert selection
Deselect all
Deselect all
Click here to refresh results
Click here to refresh results
USC
/
Digital Library
/
University of Southern California Dissertations and Theses
/
The quality of reporting of systematic reviews in periodontology journals: a cross sectional survey
(USC Thesis Other)
The quality of reporting of systematic reviews in periodontology journals: a cross sectional survey
PDF
Download
Share
Open document
Flip pages
Contact Us
Contact Us
Copy asset link
Request this asset
Transcript (if available)
Content
THE
QUALITY
OF
REPORTING
OF
SYSTEMATIC
REVIEWS
IN
PERIODONTOLOGY
JOURNALS
–A
CROSS
SECTIONAL
SURVEY
BY
Husain
Mohammad
_____________________________________________________________________________________________
A Thesis Presentation to the
FACULTY OF THE USC GRADUATE SCHOOL
UNIVERSITY OF SOUTHERN CALIFORNIA
In Partial Fulfillment of the
Requirements for the Degree
MASTER OF SCIENCE
(CRANIOFACIAL BIOLOGY)
December 2015
Copyright
2015
Husain
Mohammad
ii
Dedication
This thesis is dedicated to my parents, my brother and sisters and to my wife for
their great support and love. Special thanks to my parents for their sacrifices to get me
to where I am today and to my beloved wife for her constant patience and
encouragement. None of these achievements would have been possible without their
continuous support.
iii
ACKNOWLEDGMENT
I owe my sincere gratitude to the following amazing individuals who made this project
possible and very enjoyable:
Dr. Kian Kar - for his mentorship in this project and for being the chair of my thesis
committee;
Drs. Michael Paine and Glenn Sameshima - for their support as my thesis committee
members;
Dr. Satish Kumar – for developing the idea of this project, for his great mentorship, and
valuable advises on the project, and for participating in this study as a second reviewer
and contributing to valuable discussions.
Mrs. Abrar Alawadh – for her advises with the statistical representation of the results and
the overall esthetics of such representations.
iv
TABLE OF CONTENTS
Dedication ii
Acknowledgment iii
List of Tables v
List of Figures vi
Abstract 1
Chapter 1 Introduction 5
Chapter 2 Materials and Methods 12
Chapter 3 Results 33
Chapter 4 Discussion 50
Chapter 5 Conclusion 69
Bibliography 70
Tables 82
v
LIST OF TABLES
Table 1 Included studies 82
Table 2 Excluded studies and reasons for exclusion 85
vi
LIST OF FIGURES
Figure 1 Search strategy of the project 18
Figure 2 Tested comparisons 21
Figure 3 AMSTAR items “JoP 2010” 35
Figure 4 AMSTAR items “JoP 2013” 36
Figure 5 Glenny items “JoP 2010” 38
Figure 6 Glenny items “JoP 2013” 38
Figure 7 AMSTAR items “JCP 2010” 40
Figure 8 AMSTAR items “JCP 2013” 40
Figure 9 Glenny items “JCP 2010” 41
Figure 10 Glenny items “JCP 2013” 42
Figure 11 Trends of reporting quality in JoP, 2010-2013 (AMSTAR) 45
Figure 12 Trends of reporting quality in JoP, 2010-2013 (Glenny) 45
Figure 13 Trends of reporting quality in JCP, 2010-2013 (AMSTAR) 47
Figure 14 Trends of reporting quality in JCP, 2010-2013 (Glenny) 47
Figure 15 Item “K” JoP 2010 48
Figure 16 Item “K” JoP 2013 48
Figure 17 Item “K” JCP 2010 48
Figure 18 Item “K” JCP 2013 48
Figure 19 % of adherence amongst all SR in JoP 49
Figure 20 % of adherence amongst all SR in JCP 50
1
ABSTRACT
BACKGROUND:
Systematic reviews (SR) with or without simultaneous meta-analyses intend to
synthesize the vast amount of published primary research to answer a focused research
question. Stringent research criteria must be followed to publish a high quality SR in
order to attain clinically relevant answer to the focused research question. Validated
instruments have been developed to assess the quality of SR such as the Assessment of
Multiple Systematic Reviews (AMSTAR) and a checklist by Glenny and colleagues.
Despite the best efforts of journal editors and publishers, the quality of reporting of SR
remains below par impacting the evidence-based clinical decision-making process.
PURPOSE:
To assess the quality of reporting of Systematic Reviews with and without meta-
analyses in two widely read periodontology journals, namely, Journal of Periodontology
(JoP) and Journal or Clinical Periodontology (JCP) published in the years 2010 and 2013.
MATERIAL AND METHODS:
Electronic search using National Library of Medicine’s Medline database
supplemented by manual hand search of JoP and JCP was done independently by two
2
reviewers. AMSTAR statement and Glenny’s checklist were used to evaluate the
reporting quality of the selected Systematic Reviews from both journals. Each item on
these instruments was assigned an appropriate score (1=Yes, 0=No or N/A=Not
applicable). The percentage of the number of articles fulfilling each item in each checklist
for the years of 2010 and 2013 was calculated, excluding any non-applicable items from
such calculation. Descriptive statistics were used to analyze the data. Trends in
improvement of reporting were assessed between the two years in both the journals. The
percentage of SR adhering to three different predetermined percentages for each checklist
(i.e.; ≥50%, ≥75% and ≥90%) was calculated for each journal.
RESULTS:
Evaluation of JoP 2010 SR using the AMSTAR statement revealed low scores of
(0%, 14.2% and 14.2%) in items 11, 4 and 10 respectively and high scores (100%) in
items 1, 2, 3 and 6. Evaluation of JoP 2010 SR using the Glenny checklist revealed low
scores (14.2%, 28.5% and 50%) in items D, E, and J respectively and high scores (100%)
in items A,C,G,I, and M.
Evaluation of JoP 2013 SR using the AMSTAR statement revealed low scores of (0%,
7% and 35.7%) in items 11, 4 and 10 respectively and high scores (100%) in items 1, 6
and 9. Evaluation of JoP 2013 SR using the Glenny checklist revealed low scores of (7%,
14%, 38.4%) in items D, E, and J respectively and high scores (100%) in items A, B, I, M
and O. Evaluation of JCP 2010 SR using the AMSTAR statement revealed low scores of
(0%, 14.2% and 42.8%) in items 11, 10 and 4 respectively and high scores (100%) in
3
items 1, 3 & 6. Evaluation of JCP 2010 SR using the Glenny checklist revealed low
scores of (28.5%, 40% and 42.8%) in items E, J and D respectively and high scores
(100%) in items A, B, C, M and O. Evaluation of JCP 2013 SR using the AMSTAR
statement revealed low scores of (0%, 11.1% and 35.2%) in items 11, 4 and 10
respectively and high score (100%) in item 1. Evaluation of JCP 2013 SR using the
Glenny checklist revealed low scores of (11.1%, 20% and 50%) in items D, and J and E
respectively and high scores (100%) in items A, B, I, M and O.
For both journals, AMSTAR item #4 (Was the status of publication, i.e., gray
literature, used as an inclusion criteria?), #10 (Was the likelihood of publication bias
assessed?) and #11 (was the conflict of interest included) were consistently scored low in
both the years of assessment. For both journals, Glenny item #D (search for published
and un-published literature), #E (were all languages considered) and #J (was it stated
that the quality assessment was carried out by at least 2 reviewers) were consistently
scored low in both the years of assessment.
For both journals, AMSTAR items #1 (Was an a priori design provided?) and #6
(were the characteristics of the included studies provided?) were consistently scored high
in both the years of assessment.
For both journals, Glenny item #A (Did review addressed a focused question), #B
(did author looked for appropriate papers), #I (did the reviewers include the attempt to
assess the quality of the included studies in the analysis) #M (are the results clearly
4
displayed), and #O (were results of review interpreted appropriately) were consistently
scored high in both the years of assessment.
In JoP, the overall mean AMSTAR score/ Systematic Review for all of the SR
ranged between 36-81% in 2010 and 18%-90% in 2013. The corresponding overall score
for Glenny checklist ranged between 45-92% in 2010 and 38%-92% in 2013.
In JCP, the overall mean AMSTAR score/Systematic Review for all of the SR
ranged between 36-81% in 2010 and 45-81% in 2013. The corresponding overall score
for Glenny checklist ranged between 57-92% in 2010 and 54-92% in 2013.
The higher the predetermined percentage of adherence, the lower the total
percentage of SR adhering to such percentage in each Journal.
CONCLUSION:
The quality of reporting of SR in JoP and JCP according to AMSTAR statement
and Glenny’s checklist can be improved in the areas identified through this study.
FUNDING
No research funding was obtained.
5
CHAPTER 1- INTRODUCTION
Wide varieties of studies are being published in the health science field assessing
and evaluating different techniques and material to deliver the best evidence based
services for the community. Within the dental field, about 500 journals are publishing at
least 43.000-research article annually
1
. Different study designs have been incorporated in
the literature to examine a range of regimens and techniques in order to deliver valuable
and reliable evidence, either to support or oppose these regimens. Some of these study
designs include Cohort studies, Randomized Controlled Trials as well as Systematic
Reviews and Meta-analyses
2
. Due to the large number of studies published in the health
science field and the limited time of health professionals, it is always a challenge to be up
to date with all the current literature. Hence, Systematic Reviews facilitate such process
by compiling all the evidence of a certain topic in one article. Subsequently, the number
of Systematic Reviews have been growing significantly
3
. It was reported that a minimum
of approximately 2,500 new Systematic Reviews published in English are indexed
annually in Medline database
4
. It could be speculated that the overall number of
Systematic Review indexed in Medline would be even more if all languages are
considered.
Systematic Reviews are characterized by thorough selection and assessment of the
available literature in a well-designed, unbiased, transparent systematic manner to infer
reliable results and conclusions, which can help guiding the clinical decisions for
patients’ care. Systematic Reviews differ from traditional narrative reviews, as they
6
should always be formulated with specific criteria to help with systematically appraising
the quality of the included primary research and the subsequent conclusions
5
. The data
within a Systematic Review can be pooled giving a conclusion based on combining a
range of results with a specific statistical test (Meta-analysis). However, sometimes, it is
not possible to combine the data due to significant heterogeneity among the primary
studies, which can be clinical or statistical in nature
1
.
Such lack of consistency cause the
results to be given in a narrative manner, which is not as powerful as a meta-analysis. In
such cases, despite displaying the results narratively, they would still be based on a
Systematic Review, as the methodological approach was consistent with conducting a
Systematic Review. Furthermore, Systematic Reviews themselves are secondary research
of primary research. In other words, having a focused question with a scope that is
narrow enough to be clearly answered is equivalent to formulating a research or a
hypothesis question in primary research
5
.
Formulating a Systematic Review starts by defining a focused questions. Ideally,
the focused question should not be a broad question. This is because a broad question will
increase the potential for not being able to accurately answer the question. Therefore, the
narrower the focused question, the better the chance of achieving a reliable conclusion by
answering such question. A generally accepted format of the focused question is (PICO)
where P=Population, I=Intervention, C=Comparison and O=Outcome. If the primary
research included in a Systematic Review is composed of epidemiological studies,
exposure can substitute intervention when the focused question in formulated
5
. The
second step is to define specific selection criteria (inclusion and exclusion criteria). This
7
will help in achieving homogeneity of the primary research selected, which is important
for pooling the results, hence, providing stronger and more reliable conclusions. With
regards to the selection of study designs, Randomized Controlled Trials are the most
appropriate type when evaluating the effectiveness of a therapy. When a therapy is an
emerging intervention, Case series, Case-Control trials as well as animal studies can be
considered
5
. The third step is providing a clear explanation of the search strategy
showing all the databases used for the search (electronic or non-electronic), and the
languages included. Such step will help to clarify whether or not selection bias was
present
5
. If the final results after excluding non-eligible studies are to be pooled, a
statistical assessment of heterogeneity has to be done because such factor will affect the
reliability of the results and conclusions. Finally, the conclusion have to be confined
within the protocol limitation and should not extend beyond the focused question
5
.
On the other hand, Narrative reviews are general overviews of a topic rather than
answering a specific question. They lack the systematic search of the literature and the
precise selection criteria. In addition, critical appraisal of the included studies is not
mandatory for conducting such reviews and the author can provide a personal opinion in
the conclusion, especially, when data is lacking
6
.
Combining the results of previous studies was proposed in the medical literature
since the 1970’s. The term “Meta” was giving to such combination within the same
decade. It is defined as “more comprehensive”
7
. Meta-analyses are characterized by
pooling the results of multiple primary researches using specific statistical tests to
8
provide a quantitative estimate of the weighted available evidence. Evidence based on
meta-analyses provides higher statistical power as they combine several studies with
subsequent increase in the sample size, hence, higher power and more definitive
conclusions. However, the quality of the primary research and the methodological
approach used to conduct the meta-analyses should be evaluated very carefully, this is
because the quality of the pooled results is dependent on the quality of included primary
research
7
.
Summarizing a large amount of both published and unpublished data with some
conclusions is of great value for clinicians to rely on due to their restricted available time
for reading
3
. In addition, Systematic Reviews can guide researchers by identifying areas
in which there is shortage of literature and provide suggestions on improving these
specific areas in future research
5
. Finally, Compiling all the available high quality
evidence of a certain topic in well designed Systematic Review is known to be strongest
level of evidence in the hierarchy of different scientific study designs
2
.
Regarding the field of Periodontology, research has been growing significantly in
the last two decades. Such growth have revealed new surgical techniques and
biomaterials, all of which are aiming to deliver evidence based therapy with optimum
quality and safety for patients
8
. Despite Systematic Reviews being important tools for
evidence based dentistry, several methodological inconsistencies and lack of search
transparency have been reported, rendering the quality of Systematic Reviews and their
conclusions to be questionable for evidence based decisions in Periodontology
3
.
9
It seems like many authors of Systematic Reviews lack the commitment of
conducting high quality Systematic Reviews. This could be attributed to the fact that
Systematic Reviews are considered to be secondary research, hence, whatever the
Systematic Review reports is based on the quality of the primary research that is being
reviewed. Therefore, authors may not be taking full responsibility for the quality of the
evidence as they are reviewing what have been already published. However, authors
should always keep in mind that careful selection criteria of primary research are
extremely important. Furthermore, those criteria should be strictly followed to provide
reliable conclusions. Apart from careful selection of the primary literature, other criteria
for other aspects should also be followed when conducting a Systematic Review, as it is
indeed a form of research it self, and it should be conducted with certain criteria to
qualify as a reliable research that can be translated clinically
5
.
Many biased medical Systematic Reviews have been reported in the late 1980s
with poor methodology
9
. Therefore, quality assessment of Systematic Reviews and
Meta-analysis has been reported in the biomedical field since 1987
to help provide
strategies to improve the quality of Systematic Review at that time
7
. On the other hand,
quality assessment has been reported only recently in the field of periodontology
10
.
To overcome such methodological inconsistencies in conducting Systematic
Reviews, guidelines for conducting and reporting Systematic Reviews have been
published. Quality Of Reports Of Meta-analysis (QUOROM) statement was published in
2000
11
. It was based on a group of experts who agreed upon a checklist that have
10
recommendations with regards to conducting and reporting six aspects in Systematic
Reviews, starting from the title of the article and ending with the discussion
11
. Each one
of the aspects has sub-headings with further clarifications. In addition, a flow chart was
established showing a thorough explanation of the steps involved in reaching the final
number of articles, which would indeed help improving the transparency of the search
strategy
11
. Despite having the (QUOROM) statement, the quality of Systematic Reviews
was still suboptimal
12
. Hence, an updated version of the (QUOROM) statement (i.e.
PRISMA) was published in 2009
12
. PRISMA is an acronym for Preferred Reporting
Items for Systematic reviews and Meta-Analyses. This was published not only to
improve the reporting quality of Systematic Reviews, but also to address the
advancements in the knowledge of Systematic Reviews’ conduction and provide an up to
date guidance to meet an optimum quality by additions and modifications which were
thought to be necessary based on the consensus that the involved experts agreed upon in
their meetings
12
. These guidelines provide the blue prints needed for Systematic Reviews
with good quality, together with the Cochrane handbook for Systematic Reviews of
interventions, which was published in 2008
13
.
A checklist by Glenny and colleagues was published in 2003 aiming to evaluate
the reporting quality of Systematic Reviews by answering fifteen questions which
corresponds to the design and methods used to conduct the Systematic Review
1
.
Assessment of Multiple Systematic Reviews (AMSTAR) checklist was published five
years later to assess the quality of Systematic Reviews
14
. AMSTAR includes eleven
checkpoints with very specific criteria to guide answering each point
14
. Both of these
11
checklists have recently started to gain popularity in the Periodontology literature and
have been used to evaluate the reporting quality of Systematic Reviews in such field
3, 10,
15,
16
.
In a report evaluating the reporting quality of Systematic Reviews for the
treatment of recession-type defects, only 20% of the 10 studies reviewed to assess
different treatment modalities were consistent with Glenny criteria. Many differences in
methodology between the involved Systematic Reviews were evident causing the quality
of these Systematic Reviews to vary significantly
3
. Another quality assessment of
Systematic Reviews for guided tissue regeneration (GTR) in humans showed that only 1
out of the 14 Systematic Reviews fulfilled all the AMSTAR criteria with a range of 2-11
for the individual article adherence to the items of the checklist. In addition, their mean
score based on Glenny’s checklist ranged from 2-14, indicating and confirming lack of
uniformity
10
. These findings were in corroboration with another recent publication that
reported significant variation in the Systematic Reviews evaluating short dental implants,
as
well
as another article evaluating the reporting quality of
Systematic Reviews in
alveolar ridge preservation
15,
16
.
The objective of this descriptive cross sectional survey is to assess the reporting
quality of Systematic Reviews with and without meta-analyses in two commonly used
Periodontology Journals published in 2013 and 2010. The reporting quality of the two
journals in 2010 and 2013 will be evaluated based on validated tools. A brief comparison
based on mean values between the journals will be done. In addition, the trend variation
12
will be assessed between the years of 2010 and 2013 for each journal. The goal is to help
improve the reporting quality of future Systematic Reviews if needed, and provide more
reliable clinical and scientific conclusions. Therefore, confirming that daily clinical
decisions in the field of Periodontology are made based on solid and well-conducted
evidence from the literature, which will help optimizing the quality of care for patients.
The Primary question to be answered is “ what is the reporting quality of
Systematic Reviews published in Journal of Periodontology and Journal of Clinical
Periodontology in the year of 2013 and 2010 according to Glenny et al
1
. and AMSTAR
14
checklists?
The secondary outcomes to be assessed is whether there is any difference in the
reporting quality between the two journals in those two time periods, and whether there
have been any improvements in the reporting quality of Systematic Reviews in Journal of
Periodontology and Journal of Clinical Periodontology between 2010 and 2013 according
to the selected checklists.
CHAPTER 2- MATERIALS AND METHODS:
Amongst Periodontology Journals that publish original scientific articles, the first and
second ranked Journals (Journal of Clinical Periodontology Journal of Periodontology
respectively) based on the most recent five years impact factor were selected for
13
evaluation
17
. Those two Journals were also selected because they are the most broadly
used Journals in Periodontology. Furthermore, they provide different perspectives as one
of them represent the official publication of the American Academy of Periodontology
(Journal of Periodontology) and the other one represent the official publication of the
European Federation of Periodontology (Journal of Clinical Periodontology).
Systematic Reviews published between 01/01/2013 -12/31/2013, and 01/01/2010-
12/31/2010 in Journal of Periodontology and Journal of Clinical Periodontology were
selected using PubMed electronic database and hand search. One database was used
(PubMed) because both of the selected Journals are indexed in such database. Eligibility
criteria of the included Systematic Reviews were selected based on discussion and
consensus between the authors of the project. The following inclusion and exclusion
criteria were used for articles selection:
Inclusion Criteria:
1- Systematic Reviews in basic science and clinical Periodontology with and without
meta-analyses published in 2013 and 2010, in Journal of Periodontology and Journal
of Clinical Periodontology.
2- Systematic reviews that only includes human data.
3- Systematic reviews of primary research.
14
Exclusion Criteria:
1- Position/Consensus reports from the American Academy of Periodontology and /or
the European Federation of Periodontology.
2- Reviews of secondary research.
Electronic search using PubMed database, supplemented by manual search of
those two Journals was carried out independently by two reviewers (HM and SK). The
overall results were examined by two reviewers for eligibility to be included in the
project. The reporting quality of the Systematic Reviews was assessed by one reviewer
(HM). Questionable items amongst all the selected Systematic Reviews were
independently assessed by second reviewer (SK) to achieve consensus. Disagreement
between the reviewers was resolved by discussion, if the disagreement persists, a third
reviewer (KK) was involved for final consensus.
Search strategy was carried out by typing each journal abbreviation (i.e. J.
Periodontol and J. Clin Periodontol) in the search field of PubMed database on separate
occasions. Filtering tools on the left side of the database webpage were utilized by
selecting Systematic Reviews option in the article types section, resulting in a number of
Systematic Reviews in this initial search. Advanced search was then used by selecting the
volume number of 2013 issues for each Journal and adding it to the initial search. The
same methodology was done using the volume number of 2010 issues for each Journal.
15
We found that this search methodology was better and more accurate than using the
dates’ specification in the filtering tools, which was misleading, as some Systematic
Reviews did not show on the results page. Furthermore, the results page displayed many
Systematic Reviews that were published a year later relative to the inclusion criteria, but
were electronically early published (Epub) in one of the years selected for this project.
Language restrictions were not added as both journals publish articles in English only.
For Journal of Periodontology, The initial search characterized by “J.Periodontol” in
the search field identified 8270 articles. Systematic Reviews filtering dropped the number
to 151 articles. Volume 84 (2013) identified a total of 21 publications. 7 of these
publications were duplicates in the Journal of Clinical Periodontology 2013 issues (i.e.
Supplemental publications of a workshop jointly held by the European Federation of
Periodontology and American Academy of Periodontology)
18-24
. 6 articles were excluded
after title and abstract screening and an additional one was excluded following
methodology screening. A total of 7 Systematic Reviews were excluded for the following
reasons; 1 position paper
25
, 2 reviews of secondary research
10 ,13
,1 retrospective study
26
, 1 consensus report
22
(from supplemental publication), 2 narrative reviews
23, 24
(from
supplemental publication). Finally, hand search was carried out and the results were
similar to the electronic search. After Full text screening, a total of 14 Systematic
Reviews (4 of which were from supplemental publications) were included from this
search
18-21, 27-36
.
16
Volume 81 of 2010 issues in Journal of Periodontology showed an initial result of 9
publications. Abstract and title screening resulted in excluding two articles because they
were on animals
37, 38
. Another article was excluded because the data was derived from
humans and animals, which was not consistent with the inclusion criteria
39
. Secondary
check with hand search identified an additional article that was consistent with the
inclusion criteria
40
. Full text screening resulted in a final total number of 7 Systematic
Reviews eligible for inclusion
41 ,42,
43,
44,
40, 45
46
.
For Journal of Clinical periodontology, “J.Clin Periodontol “ was typed into the
search engine of PubMed database resulting in 4787 publications. Filtering tools were
similarly utilized by selecting Systematic Reviews, reducing the number of publication to
196 articles. Volume selection was then carried out by the advanced search option.
Volume 40 for 2013 issues was added to the initial search resulting in 24 publications. 7
of which were duplicates in Journal of Periodontology 2013 issues (i.e. Supplemental
publications of a workshop jointly held by the European Federation of Periodontology
and American Academy of Periodontology)
18-24
. Therefore, they were excluded from the
total number of publications achieved in Journal of Clinical Periodontology 2013 issues.
Following title and abstract screening, 3 more articles were excluded. 1 was neither a
clinical nor a scientific article
47
, 1 was a critical appraisal of Systematic Reviews on a
specific topic in periodontology (Review of secondary research)
48
and 1 was an animal
study
49
. Hand search was carried out for confirmation and the results were similar to the
electronic search. After Full text screening, a total of 14 Systematic Reviews were
included
50-63
.
17
Although the duplicate supplemental articles fulfilling the inclusion criteria were
excluded from the final total number of Systematic Reviews selected for Journal of
Clinical Periodontology 2013 issues, those that were selected from Journal of
Periodontology 2013 issues were included in the data analysis of both Journals, as they
will have equal affect on the results
18, 19,
20,
21
. This means that the total number of
Systematic Reviews analyzed for Journal of Clinical Periodontology, 2013 was 18
Systematic Reviews.
Finally, The addition of Volume 37, which corresponds to the 2010 issues in Journal
of Clinical Periodontology, resulted initially in 8 publications. Abstract and title
screening resulted in excluding one article because it was a review of secondary research
3
. The same number of publications was verified with a secondary hand search. Full text
screening identified a total of 7 Systematic Reviews eligible for inclusion
64-70
.
Figure 1 demonstrates the search strategy of the project.
*7
articles
were
duplicates
in
the
2013
publications.
3
of
which
were
excluded
for
not
being
eligible
with
the
selection
criteria
of
the
project.
18
PubMed
J.Clin
Periodontol
J.Periodontol
Filter
Article
Type:
Systematic
Review
Advanced
Search:
Volume
Number
2013
Vol:
84
2010
Vol:
81
2013
Vol:
40
2010
Vol:
37
21
Articles
9
Articles
24
Articles
Title
&
Abstract
Screening
8
Articles
7
Articles
Excluded
*(3
not
eligible)
3
Articles
Excluded
10
Articles
Excluded
*(4
duplicates/3
not
eligible)
1
Article
Excluded
Hand
Search
No
Additional
Articles
1
Additional
Article
No
Additional
Articles
No
Additional
Articles
Full
Text
Screening
14
Articles
Included
7
Articles
Included
7
Articles
Included
14
Articles
Included
Figure
1
19
AMSTAR
14
and Glenny et al.
1
Checklists were used to evaluate the reporting quality
of the selected Systematic Reviews from both Journals. Descriptive statistics were used
to analyze the data and briefly compare the reporting quality of Systematic Reviews
between the two journals in 2013 and 2010. Each item on those checklists received a
numerical score. The scores were either a 1=yes or a 0=no or not applicable options. The
percentage of the number of articles fulfilling each item in each checklist in the years of
2010 and 2013 was calculated, excluding any non-applicable items from such calculation
to provide fair and accurate results. In addition, the mean value of adherence for each
Systematic Review in both Journals was calculated to evaluate the range of the mean
adherence amongst the selected articles. Any criterion with a score = 0 for being “not
applicable” was removed from the “number of terms” section in the denominator when
the mean values were calculated (Mean=sum of terms/number of terms). This is because
the authors of the project believed that a criterion that is not applicable should not be
tested. Hence, addressing that in the mean calculation will improve the accuracy of the
results.
Item #K from Glenny’s check list (are the results given in a narrative or pooled
statistical analysis) was not addressed in such calculations due to the intrinsic limitations
represented by significant heterogeneity of the primary research included in some
Systematic Review, resulting in narrative presentation of the data which can not be
controlled by the authors. A separate calculation was done looking at the percentage of
pooled results in contrast to narrative data for each Journal in each of the selected years.
20
In cases where the Systematic Reviews presented pooled and narrative data, they were
included in the narrative section.
The overall mean percentage of individual item consistency across the selected
Systematic Reviews was calculated for each journal in each year. Comparison was then
carried out between the two journal in 2010 and 2013 using each checklist. In addition,
Descriptive trend analysis was completed to evaluate the difference in the reporting
quality of Systematic Reviews within each Journal between 2010 and 2013 using the
percentage of adherence amongst the selected Systematic Reviews for each item in each
checklist.
Finally, the percentage of adherence (i.e.; using the mean adherence% of each
Systematic Review) amongst the total number of the selected Systematic Reviews for
both years in each journal, using each checklist, was calculated at three different levels,
namely, those with a mean adherence score of 50% or more, ≥75% and ≥90%. This was
done to assess how the total number of Systematic Reviews adhering to each checklist
relate to the increase in percentage of adherence. Those percentages were selected to
provide a perspective of the percentage of SR adhering to at least ½ of the criteria of each
checklist, three quarters of the criteria (≥75%) and close to all the criteria (≥90%).
21
Figure 2 demonstrates the tested comparisons.
AMSTAR checklist was established to provide a tool for assessing the
methodological quality of Systematic Reviews by addressing components that were not
addressed in the previous published tools
14
. The authors of AMSTAR combined the
results of older tools, namely, the Enhanced Overview Quality Assessment Questionnaire
(OQAQ) published in the early 90s
71
,and an older checklist published in the late 80s
7
.
Those two instruments were the most widely used checklists despite the availability of
over 24 tools at that time. Therefore, the authors of AMSTAR selected them.
Furthermore, the authors believed that new components have to be added in addition to
2013
2010
JCP
2013
2010
Trend
Trend
JOP
Figure
2
22
those covered by these old checklists for assessing Systematic Reviews. These new items
were considered to be of great importance judged by experts in the field of research. The
additional items were language restrictions, publication bias and publication status “i.e.
grey literature”
14
. Those additional items have been shown to significantly influence the
overall results of a Systematic Review leading to biased conclusions. For instance, not
including grey literature in a Systematic Review have been shown to overestimate
favorable results which could reflect negatively on patients’ care
72
. Language restrictions
have been shown to have an effect on the end results of Systematic Reviews. However,
such finding remains controversial and the effect of it remains unclear
73
. This is probably
why language restriction per se was not addressed in AMSTAR, instead, it was included
along with publication status item, which will not allow the solo evaluation of language
restriction. The Overview Quality Assessment Questionnaire (OQAQ) addressed
selection bias only. Publication bias was never addressed in either of the included
checklists. Therefore, it was added due to the reported significant effects of such bias on
the results
74
. Publication bias refers to increased publications of large studies with
positive result compared to smaller studies with negative or neutral results. Based on this
selection of empirical data of older evaluation tools and more updated experts opinions, a
checklist was established that is composed of 11 items with concise criteria.
There was no attempt to communicate with the authors of the selected Systematic
Reviews in this project as the assumption was made to believe that all articles included
have good methodological quality. Therefore, AMSTAR was used to assess the reporting
quality of Systematic Reviews despite the fact that the authors of AMSTAR emphasized
23
that this checklist was established to assess the methodological quality of a Systematic
Reviews and not the reporting quality. In addition, AMSTAR has already been used in
the literature for assessment of the reporting quality of Systematic Reviews
10, 15,
3, 75
.
Validity, reliability, feasibility and consistency of AMSTAR have been confirmed by
assessing several publications based on experts’ feedback. This ensures the viability of
such instrument
76
. Furthermore, the authors of AMSTAR provided specific criteria to
guide answering each one of the 11 items, making it easier on the reviewers to follow
very consistently. This can indeed improve the objectivity of the evaluation
14
. Finally,
AMSTAR checklist is being used by several agencies including the Canadian Agency for
Drugs and Technologies in Health and the Cochrane effective Practice and Organization
of Care Group (EPOC) indicating its world wide reliability
76
.
The following is an exact copy of the AMSTAR criteria with additional notes. The
additional notes below each criterion were derived from “Beverley Shea and/or Jeremy
Grimshaw (first and/or second authors of AMSTAR) in June and October 2008, and they
were updated in July and September 2010
77
:
“ 1. Was an 'a priori' design provided?
• The research question and inclusion criteria should be established before the
conduct of the review.
• Note: Need to refer to a protocol, ethics approval, or pre-determined/a priori
published research objectives to score a “Yes.”
24
2. Was there duplicate study selection and data extraction?
• There should be at least two independent data extractors and a consensus
procedure for disagreements should be in place.
• Note: 2 people do study selection, 2 people do data extraction, consensus process
or one person checks the other’s work.
3. Was a comprehensive literature search performed?
• At least two electronic sources should be searched. The report must include years
and databases used (e.g., Central, EMBASE, and MEDLINE). Key words and/or
MESH terms must be stated and where feasible the search strategy should be
provided. All searches should be supplemented by consulting current contents,
reviews, textbooks, specialized registers, or experts in the particular field of study,
and by reviewing the references in the studies found.
• Note: If at least 2 sources + one supplementary strategy used, select “Yes”
(Cochrane register/Central counts as 2 sources; a grey literature search counts as
supplementary).
4. Was the status of publication (i.e. grey literature) used as an inclusion criterion?
• The authors should state that they searched for reports regardless of their
publication type. The authors should state whether or not they excluded any
reports (from the Systematic Review), based on their publication status, language
etc.
• Note: If review indicates that there was a search for “grey literature” or
25
“unpublished literature,” indicate, “Yes.” SIGLE database, dissertations,
conference proceedings, and trial registries are all considered grey for this
purpose. If searching a source that contains both grey and non-grey, must specify
that they were searching for grey/unpublished lit.
5. Was a list of studies (included and excluded) provided?
• A list of included and excluded studies should be provided.
• Note: Acceptable if the excluded studies are referenced. If there is an electronic
link to the list but the link is dead, select “No.”
6. Were the characteristics of the included studies provided?
• In an aggregated form such as a table, data from the original studies should be
provided on the participants, interventions and outcomes. The ranges of
characteristics in all the studies analyzed e.g., age, race, sex, relevant
socioeconomic data, disease status, duration, severity, or other diseases should be
reported.
• Note: Acceptable if not in table format as long as they are described as above.
7. Was the scientific quality of the included studies assessed and documented?
• 'A priori' methods of assessment should be provided (e.g., for effectiveness
studies if the author(s) chose to include only randomized, double-blind, placebo
controlled studies, or allocation concealment as inclusion criteria). For other types
26
of studies, alternative items will be relevant.
• Note: Can include use of a quality-scoring tool or checklist, e.g., Jadad scale, risk
of bias, sensitivity analysis, etc., or a description of quality items, with some kind
of result for each study (“low” or “high” is fine, as long as it is clear which studies
scored “low” and which scored “high”; a summary score/range for all studies is
not acceptable).
8. Was the scientific quality of the included studies used appropriately in formulating
conclusions?
• The results of the methodological rigor and scientific quality should be considered
in the analysis and the conclusions of the review, and explicitly stated in
formulating recommendations.
• Note: Might say something such as “the results should be interpreted with caution
due to poor quality of included studies.” Cannot score “Yes” for this question if
scored “no” for question 7.
9. Were the methods used to combine the findings of studies appropriate?
• For the pooled results, a test should be done to ensure the studies were
combinable, to assess their homogeneity (i.e., Chi-squared test for homogeneity,
I
2
). If heterogeneity exists, a random effects model should be used and/or the
clinical appropriateness of combining should be taken into consideration (i.e., is it
sensible to combine?).
• Note: Indicate, “Yes” if they mention or describe heterogeneity, i.e., if they
27
explain that they cannot pool because of heterogeneity/variability between
interventions.
10. Was the likelihood of publication bias assessed?
• An assessment of publication bias should include a combination of graphical aids
(e.g., funnel plot, other available tests) and/or statistical tests (e.g., Egger
regression test, Hedges Olken).
• Note: If no test values or funnel plot included, score “No”. Score “Yes” if
mentions that publication bias could not be assessed because there were fewer
than 10 included studies.
11. Was the conflict of interest included?
• Potential sources of support should be clearly acknowledged in both the
systematic review and the included studies.
• Note: To get a “Yes,” must indicate source of funding or support for the
Systematic Review and for each of the included studies.”
AMSTAR was selected for this project due to the clarity of its criteria and the
lucid explanation of each question within the list. Furthermore, it is a relatively new
checklist and it has been used frequently for assessing the reporting quality of Systematic
Reviews in dentistry and Periodontology
3,
10,
15,
16
. It was chosen to be the primary tool to
evaluate the reporting quality of Systematic Reviews for this project. Since AMSTAR
28
was published in 2007, the authors of this project decided to provide a time lapse of three
years since the publication date, so that the checklist can be assessed and applied for
evaluations. Therefore, Systematic Reviews published in the selected Journals in the year
of 2010 were included. An agreement was reached amongst the authors of the project to
provide additional three years period (2010-2013) to look at the trend differences in the
reporting quality of Systematic Reviews within the selected Journals. A gap of three
years was agreed upon to signify the difference of the reporting quality between the
Systematic Reviews based on the selected checklist. Furthermore, the reporting quality of
the Systematic Reviews published in each journal in the year of 2013 was assessed
because these were the most recent published Systematic Reviews relative to the starting
time of the project, which was October ,2014.
It was believed that using one checklist for this project might propose a risk of
bias. Therefore, the decision was made to include another checklist to reduce the risk of
bias associated with using a single checklist and to help improve the reliability of the
results as well as strengthening the assessment of the reporting quality of the selected
Systematic Reviews. Hence, Glenny and colleagues
1
checklist was selected as it has been
used in the Periodontology literature
3,
10,
15,
16
.
Regarding the checklist conducted by Glenny and colleagues, the Cochrane oral
health group as well as other reviewers published it in 2003. The items of this checklist
were derived from the Quality of Reporting Of Meta-analysis criteria QUOROM
11
and
the Meta-analysis Of Observational Studies in Epidemiology assessment tool (MOOSE)
29
78
. Glenny et al. checklist is composed of fifteen checkpoints
1
. Some of which overlaps
with those proposed by AMSTAR. However, we felt that the former checklist was not as
objective as AMSTAR because the authors did not provide detailed explanation to guide
the answer of each question for most of the items. Therefore, discussions between the
authors of this project for consensus were more common when Glenny’s criteria were
used for the assessment of the involved Systematic Reviews. Validity assessment of such
checklist was evaluated when it was published. It revealed high percentage of agreement
amongst the reviewers, who were at least one clinician and one methodologist
1
. This
indicates the consistency of this checklist when used amongst experts and it’s subsequent
optimal reliability.
The checklist is composed of the following questions, the notes below each
question represent the consensus achieved by the authors of this project:
A. Did review address a focused question?
• If all the components of the focused question/questions have been addressed even if
the final conclusion is not conclusive based on the available evidence. The score is a
“Yes”
B. Did authors look for appropriate papers?
• Upon discussion, consensus was reached to answer the question with a “Yes” if all
the possible keywords/Mesh terms that are related to the focused question have been
30
used in the search methodology.
C. Do you think authors attempted to identify all relevant studies?
• If at least two electronic databases supplemented with searching the bibliographies of
the included studies, a “Yes” can be given as an answer.
D. Search for published and unpublished literature
• If the Systematic Review reports looking at any form of unpublished literature as part
of the search strategy, the answer is a “Yes”
E. Were all languages considered?
• There should be no language restrictions to score a “Yes”
F. Was any hand searching carried out?
• If authors report manual/hand search, whether it was for references of the included
articles, or additional Journals related to the question of the Systematic Review, the
score is a “Yes”
G. Was it stated that the inclusion criteria were carried out by at least two reviewers?
• “The inclusion criteria have to be applied independently by at least two reviewers
when studies are being selected for a systematic review”
1
. In other words, the
application of the criteria in selecting and analyzing the articles included in the
31
project has to be done by at least two reviewers to score a “Yes”.
H. Did reviewers attempt to assess the quality of the included studies?
• Initially, it was thought to have the answer consistent with question 8 in AMSTAR
checklist (i.e.; was the scientific quality of the included studies assessed and
documented). The answer to this question based on AMSTAR criteria requires the
quality of each study involved in the Systematic Review to be reported. In other
words, summaries or ranges are not acceptable and will score a “No”. However, as
the Systematic Reviews for this project were being analyzed, we noticed that some
quality assessments were done for a group of studies that share a certain aspect in
common (for instance; Randomized Controlled Trials Vs. Controlled Clinical Trials).
We believed that in this instance, the authors did attempt to assess the quality of the
included studies and deserve credit for that, despite the fact that the assessment was
not done in a manner consistent with AMSTAR criteria. Therefore, we decided to
score a “Yes” for any attempt using quality scoring tools or checklists even if the
results of the quality assessment are given in summery or range format.
I. If so did they include this in the analysis?
• The results of the quality should be included in the analysis and the final conclusions
to help maintaining high quality practice. If the author reports the effect the quality of
the primary studies have on the results and conclusions, a “Yes” score is given for
this question
32
J. Was it stated that the quality assessment was carried out by at least two reviewers?
• At least two reviewers should be involved in the quality assessment to score a “Yes”
K. Are the results given in a narrative or pooled statistical analysis?
• All the reported results have to be based on pooled data with out any narrative data to
be considered as pooled.
L. If the results have been combined was it reasonable to do so?
• “Prior to conducting a Meta-analysis, consistency of the of the treatment effect across
the primary studies must be tested with a statistical heterogeneity test. In addition to
that, a clear presentation of the characteristics of the primary studies have to be
included to score a “Yes”
”
1
M. Are the results clearly displayed?
• If the results can be interpreted easily without vagueness, the score should be a “Yes”
N. Was an assessment of heterogeneity made and reasons for variation discussed?
• Heterogeneity, Clinical or statistical in nature must be assessed to score a “Yes”,
Note: if authors report clinical heterogeneity without testing for it and report that the
results could not be pooled because of such heterogeneity, the score should be a “yes”
33
O. Were results of review interpreted appropriately?
• The results should be interpreted based on the quality of the primary research
involved, and the variations between the studies. Authors should address all the
limitations, which necessitates the results to be taken with caution to score a “Yes”
CHAPTER 3- RESULTS
A total of 42 Systematic Reviews were included for evaluation using both checklists,
7 Systematic Reviews published in Journal of Periodontology (JoP) and Journal of
Clinical Periodontology (JCP) in 2010, and 14 Systematic Reviews published in JoP and
JCP in 2013. However, the 14 Systematic reviews in JoP, 2013 include the supplemental
articles which were duplicates in JCP, 2013. This is why the 4 supplemental Systematic
Reviews that fulfilled the inclusion criteria
(18, 19,
20,
21)
were only included in one journal
(i.e. JoP) in the methodology section to avoid having duplicates. However, those
supplemental publications were included in the analysis of both journals. The addition of
these supplemental publications in JCP 2013 for analysis caused the final number of the
analyzed Systematic Reviews in JCP, 2013 to be 18 rather than 14.
The results revealed that the mean range of the overall adherence per Systematic
Review to AMSTAR checklist
(
14)
in Journal of Periodontology, 2010 was between 0.36-
0.81. The corresponding range for 2013 publications was between 0.18-0.9. Regarding
34
the range for Journal of Periodontology based on Glenny’s
(1)
criteria, it was between
0.45-0.92 in 2010 and 0.38-0.92 in 2013.
The mean range of the overall adherence per Systematic Review for Journal of
Clinical Periodontology was 0.36-0.81 in 2010 and 0.45-0.81 in 2013 for AMSTAR
checklist. The corresponding range based on Glenny’s criteria was 0.57-0.92 and 0.54-
0.92 in 2010 and 2013 respectively.
Regarding the adherence of the included Systematic Reviews for each item in each
checklist within 2010 and 2013, Journal of Periodontology, 2010 (Figure 3) showed a
100% adherence to items #1(was a ‘priori’ design provided), #2(was there duplicate
study selection and data extraction), #3(was a comprehensive literature search performed)
and #6(were the characteristics of the included studies provided) in AMSTAR checklist.
Items #5(was a list of studies “included and excluded” provided), #7(was the scientific
quality if the included studies assessed and documented) and #8(was the scientific quality
of the included studies used appropriately in formulating conclusions) showed 71.4% of
adherence. This was followed by Item #9(were the methods used to combine the findings
of studies appropriate) which showed 66.6%. Items #4(was the status of publication ”i.e.
grey literature” used as an inclusion criteria) and #10(was the likelihood of publication
bias assessed) revealed a significant drop in the percentage of adherence for the tested
Systematic Reviews reaching 14.2%. Finally, item #11(was the conflict of interest
included) showed the lowest adherence percentage amongst the evaluated articles, which
was 0%.
35
Furthermore, evaluation of the same Journal using the same checklist in 2013 (Figure
4) revealed a 100% adherence for items #1, #6 and #9. Item #3 revealed 78.5% followed
by items #7 and #8, which showed 71.4% of consistency amongst the 14 included
Systematic Reviews. Item #2 showed 64.2% whereas Item #5 revealed 57.1%. Item #10
revealed relatively low percentage of adherence, which was 35.7%. Similarly, Item #4
showed a very low adherence percentage (7%). Regarding Item #11, the result was
similar to the result from 2010 evaluation (0%).
Figure
3
0
20
40
60
80
100
1
2
3
4
5
6
7
8
9
10
11
100
100
100
14.2
71.4
100
71.4
71.4
66.6
14.2
0
%
Consistency
AMSTAR Items “JOP 2010”
36
Figure
4
Moving on to the percentage of adherence of Systematic Reviews from Journal of
Periodontology with items from Glenny’s checklist, Systematic Reviews published in
2010 (Figure 5) showed a 100% adherence for items #A (did review address the focused
question), #C (do you think the author attempted to identify all relevant studies), #G (was
it stated that the inclusion criteria were carried out by at least two reviewers), #I (if
quality assessment was attempted, was it included in the analysis) and #M (are the results
clearly displayed). A slightly lower percentage of adherence (85.7%) was calculated for
items# B (did the authors look for appropriate papers), #F (was any hand-searching
carried out), #H (did reviewer attempt to assess the quality of the included studies) and
#O (were the results of the review interpreted appropriately). Furthermore, item #L (if the
results have been combined, was it reasonable to do so) showed even more reduction
yielding an adherence of 80% amongst the 7 evaluated Systematic Reviews. The results
then revealed further reduction for items #N (was an assessment of heterogeneity made
0
20
40
60
80
100
1
2
3
4
5
6
7
8
9
10
11
100
64.2
78.5
7
57.1
100
71.4
71.4
100
35.7
0
%
Consistency
AMSTAR Items “JOP 2013”
37
and reasons for variation discussed) and #J (was it stated that the quality assessment was
carried out by at least two reviewers) reaching 57.1% and 50% respectively. Finally, item
#E (were all languages considered) showed very poor adherence (28.5%) followed by the
lowest adherence for item #D (search for published and unpublished literature) which
was 14.2%.
The results for Journal of Periodontology 2013 publications using the same checklist
(Figure 6) revealed 100% consistency with five items, namely, items #A, #B, #I, #M and
#O. Items #H and #N revealed 85.7% adherence amongst the 14 included articles. Item
#L showed an overall adherence of 81.8% followed by 78.5% for items #C, #F and #G. A
big reduction in the percentage of adherence amongst the included Systematic Reviews
was shown for items #J, #E and #D with 38.4%, 14% and 7% respectively. Item #D
showed the lowest adherence score amongst all other items which was also shown in the
2010 results of the same Journal.
38
Figure
5
Figure
6
Concerning Journal of Clinical Periodontology, the highest percentage of adherence
to AMSTAR items in 2010 (Figure 7) was observed for items #1, #3 and #6 which was
0
20
40
60
80
100
A
B
C
D
E
F
G
H
I
J
L
M
N
O
100
85.7
100
14.2
28.5
85.7
100
85.7
100
50
80
100
57.1
85.7
%
Consistency
Glenny Items “JOP 2010”
0
20
40
60
80
100
A
B
C
D
E
F
G
H
I
J
L
M
N
O
100
100
78.5
7
14
78.5
78.5
85.7
100
38.4
81.8
100
85.7
100
%
Consistency
Glenny Items “JOP 2013”
39
100%. This was followed by items #2, #5 and #9 with a score of 85.7%. Items #7 and #8
showed 57.1% of adherence amongst the 7 included Systematic Reviews followed by
42.8% for item #4 and 14.2% for item #10. Item #11 showed 0% of adherence, which is
similar to the findings from Journal of Periodontology.
Regarding the year of 2013 for the same Journal and the same checklist (Figure 8),
100% adherence was only found for a one item, which was item #1. Items #6 and #9,
revealed almost similar percentages of adherence, which were 94.4% and 94.1%
respectively. This was followed by item #3 with 88.8%. 83.3% of the evaluated
Systematic Reviews were consistent with item #2 followed by a small reduction in the
percentage of adherence for items #8 (82.3%) and #7 (70.5%). A relatively more
prominent drop in the percentage of adherence was calculated for items #5 with 55.5%
followed by a significant drop in such percentage for items #10 and #4 with 35.2% and
11.1% respectively. Similar to the previous year as well as the other Journal, the lowest
percentage of adherence was for item #11 (0%)
40
Figure
7
Figure
8
The reporting quality of Systematic Reviews from Journal of Clinical Periodontology
in the year of 2010 using Glenny’s checklist (Figure 9), revealed 100% adherence to
items #A, #B, #C, #M and #O. The percentage of consistency for items #G and #N was
0
20
40
60
80
100
1
2
3
4
5
6
7
8
9
10
11
100
85.7
100
42.8
85.7
100
57.1
57.1
85.7
14.2
0
%Consistency
AMSTAR Items “JCP 2010”
0
20
40
60
80
100
1
2
3
4
5
6
7
8
9
10
11
100
83.3
88.8
11.1
55.5
94.4
70.5
82.3
94.1
35.2
0
%Consistency
AMSTAR Items “JCP 2013”
41
0
20
40
60
80
100
A
B
C
D
E
F
G
H
I
J
L
M
N
O
100
100
100
42.8
28.5
57
85.7
57
80
40
83.3
100
85.7
100
%Consistency
Glenny Items “JCP 2010”
85.7% followed by item #L with 83.3% and item #I with 80%. Both, items #F and #H
revealed 57% adherence followed by a slight reduction reaching 42.8% and 40% for
items #D and #J respectively. The item that showed the least percentage of adherence
within the evaluated Systematic Reviews in the year of 2010 was item #E with 28.5%.
Regarding the evaluation for the year of 2013 in Journal of Clinical Periodontology
using Glenny’s checklist (Figure 10), The 18 Systematic Reviews including the
supplemental publications showed 100% consistency with items #A, #B, #I, #M and #O.
This was followed by 93.7% for item #L. Both, items #C and #G showed 88.8%
adherence which was almost similar to items #H and #N with 88.2%. A slight reduction
was shown for item #F with 77.7% followed by further reduction for item E with 50%
adherence. A big reduction in the percentage of adherence for items #J and #D (20%,
11.1% respectively) was evident, with item #D showing the lowest percentage of
adherence.
Figure
9
42
Figure
10
Generally, the result revealed consistent low scores of items #4 (was the status of
publication ”i.e. grey literature” used as an inclusion), #10 (was the likelihood of
publication bias assessed) and #11 (was the conflict of interest stated) amongst both
journal in both the years of assessment. Evaluation of reporting of Systematic Reviews
using Glenny’s checklist showed consistent low scores for items #D (search for published
and unpublished literature), #E (were all languages considered) and #J (was it stated that
the quality assessment was carried out by at least two reviewers) in both journals for both
the years of assessment.
The mean percentage of individual item consistency across the Systematic Reviews
was calculated for both journals in both years of evaluation, using each checklist. In
2010, AMSTAR and Glenny mean percentages for Journal of Periodontology was 64.4%
and 76.6% respectively. The corresponding values for Journal of Clinical Periodontology
0
20
40
60
80
100
A
B
C
D
E
F
G
H
I
J
L
M
N
O
100
100
88.8
11.1
50
77.7
88.8
88.2
100
20
93.7
100
88.2
100
%Consistency
Glenny Items “JCP 2013”
43
were 66.2% and 75.7% respectively. It can be seen that the overall percentage of
consistency for articles from Journal of Clinical Periodontology was higher by 1.8% for
AMSTAR items, however, it was lower by 0.9% for items from Glenny’s check list. The
difference between the two Journals was very minimal.
Regarding the corresponding values for the 2013 results, the mean percentage of
individual item consistency across the Systematic Reviews using AMSTAR and Glenny
for Journal of Periodontology was 62.3% and 74.8% respectively. The corresponding
values for Journal of Clinical Periodontology were 65% and 79% respectively. In this
year, the difference in percentage of consistency seems to be more pronounced, and
favoring Journal of Clinical Periodontology by 2.7% for AMSTAR and 4.2% for
Glenny’s items. It is important to emphasize that these values represent an overall broad
picture of the percentage of consistency for each Journal within each checklist. It is not
possible to compare the two journals thoroughly based on these values.
Concerning the trends of improvement within each one of the selected Journals,
descriptive trend analysis between 2010 and 2013 revealed improvement in the mean
percentage of the individual item consistency across the articles in Journal of Clinical
Periodontology only, which was 3.3% (75.7%-79%) according to Glenny’s checklist. The
same Journal showed reduction in the mean percentage according to AMSTAR by 1.2%
(66.2%-65%). Furthermore, Journal of Periodontology showed a reduction in such
percentage by 2.1% (64.4%-62.3%) and 1.8% (76.6%-74.8%) according to AMSTAR
and Glenny’s checklists respectively.
44
Detailed descriptive trend analysis for each item in AMSTAR checklist amongst the
selected Systematic Reviews from Journal of Periodontology (Figure 11) did not show
any difference between 2010 and 2013 publications for items #1, #6, #7, #8, and #11.
Item #9 showed improvement by 33.4% (66.6%-100%) and item #10 improved by 21.5%
(14.2%-35.7%). Regarding the remaining items (i.e. #2, #3, #4 and #5), all of them
revealed reduction in the percentage of adherence between the three years period. Item #2
showed the highest percentage of reduction by 35.8% (100%-64.2%) followed by item #3
with 21.5% (100%-78.5%), item 5 with 14.3% (71.4%-57.1%) and item #4 with 7.2%
(14.2%-7%).
The same journal (JoP) revealed no changes in the percentages of consistency
amongst the Systematic Reviews between 2010 and 2013 with items #A, #I, #H and #M
in Glenny’s checklist (Figure 12), which were 100% in both years except for item #H
(85.7%). Both, items #B and #O improved from 85.7% in 2010 to 100% in 2013 which is
a significant improvement reaching perfect reporting of those two items amongst the
Systematic Reviews in 2013. Item #L improved very slightly by 1.8% (80%-81.8%)
whereas item #N improved by 28.6% (from 57.1%-85.7%). Regarding the remaining
items which are item #C, #D, #E, #F, #G, and #J, a drop in the percentage of adherence
was noted amongst them. Items #C and #G were reduced by 21.5% (#C “100%-78.5%”,
#G ”100%-78.5%”) followed by items #E and #J with 14.5% (28.5%-14%) and 11.6%
(50%-28.4%) respectively between 2010 and 2013. In addition, items #D and #F showed
the least reduction between the three years period which was 7.2% (#D ”14.2%-7%”, #F
”85.7%-78.5%”)
45
Figure
11
Figure
12
Regarding the trends of reporting between 2010 and 2013 for Systematic Reviews in
Journal of Clinical Periodontology, according to AMSTAR items (Figure 13), items #1
and #11 showed no changes in the percentage of adherence within the selected
1
2
3
4
5
6
7
8
9
10
11
%
Consistency
2010
100
100
100
14.2
71.4
100
71.4
71.4
66.6
14.2
0
%Consistency
2013
100
64.2
78.5
7
57.1
100
71.4
71.4
100
35.7
0
0
20
40
60
80
100
Trends
of
Reporting
quality
in
JOP
2010-‐2013
(AMSTAR)
A
B
C
D
E
F
G
H
I
J
L
M
N
O
%
Consistency
2010
100
85.7
100
14.2
28.5
85.7
100
85.7
100
50
80
100
57.1
85.7
%Consistency
2013
100
100
78.5
7
14
78.5
78.5
85.7
100
38.4
81.8
100
85.7
100
0
20
40
60
80
100
Trends
of
Reporting
quality
in
JOP
2010-‐2013
(Glenny)
46
Systematic Reviews between the three years period as the percentage remained 100% for
item #1 and 0% for item #11. Such consistent results for those two items were also
evident in Journal of Periodontology with the same percentages. Items #7, #8, #9 and #10
revealed improvement between 2010 and 2013 by 13.4% (57.1%-70.5%), 25.2%(57.1%-
82.3%), 8.4%(85.7%-94.1%) and 21%(14.2%-35.2%) respectively. Items #2-#6 showed
reduction in the percentage of adherence between 2010 and 2013. Items #3 and #6 were
both reduced from perfect adherence amongst the tested Systematic Reviews (100%) by
11.2% and 5.6% respectively (item #3 ”100%-88.8%”, item #6 ”100%-94.4%”). Item #2
revealed small reduction in the percentage of adherence, which was 2.4% (85.7%-
83.3%). Finally, items #4 and #5 showed almost similar reduction by 31.7% and 30.2%
respectively (item #4 ”42.8%-11.1%”, item #5 ”85.7%-55.5%”).
Moving on to the detailed descriptive trend analysis for Systematic Reviews from
Journal of Clinical Periodontology based on items from Glenny’s checklist (Figure 14),
Items #A, #B, #M and #O revealed 100% consistency in 2010 which persisted in 2013.
Majority of the items showed improvement between 2010 and 2013 including items #E,
#F, #G, #H, #I, #L and #N. Items #E, #F and #I showed almost similar percentages of
improvement by 21.5% (28.5%-50%), 20.7% (57%-77.7%) and 20% (80%-100%)
respectively. Item #N showed slight improvement by 2.5% (85.7%-88.2%). Furthermore,
item #G was improved by 3.1% (85.7%-88.8%), followed by 9.9% for item #L (83.8%-
93.7%) and 31.2% for item #H (57%-88.2%). On the other hand, adherence percentage to
item #C reduced from 100% to 88.8% (11.2%), and both, items #D and #J dropped by
31.7% and 20% respectively (item #D ”42.8%-11.1%, item #J ”40%-20%”).
47
A
B
C
D
E
F
G
H
I
J
L
M
N
O
%
Consistency
2010
100
100
100
42.8
28.5
57
85.7
57
80
40
83.8
100
85.7
100
%Consistency
2013
100
100
88.8
11.1
50
77.7
88.8
88.2
100
20
93.7
100
88.2
100
0
20
40
60
80
100
Trends
of
Reporting
quality
in
JCP
2010-‐2013
(Glenny)
Figure
13
Figure
14
Regarding Item K (are the results given in a narrative or pooled statistical analysis)
from Glenny’s checklist, it was not included in the assessment of the reporting quality of
Systematic Reviews for the selected Journal. Furthermore, if the results of any Systematic
1
2
3
4
5
6
7
8
9
10
11
%
Consistency
2010
100
85.7
100
42.8
85.7
100
57.1
57.1
85.7
14.2
0
%Consistency
2013
100
83.3
88.8
11.1
55.5
94.4
70.5
82.3
94.1
35.2
0
0
20
40
60
80
100
Trends
of
Reporting
quality
in
JCP
2010-‐2013
(AMSTAR)
48
Review was partially pooled, and the subsequent conclusion was based on both, the
pooled data and narratively analyzed data, the answer to item K was considered as
narrative (i.e. narrative/both in the Pie Chart). The results for such question were
analyzed separately and showed that the percentage of the Systematic Reviews presenting
completely pooled data was 29% and 50% for Journal of Periodontology in 2010 (Figure
15) and 2013 (Figure 16) respectively. The corresponding values for Journal of Clinical
Periodontology was 57% and 61% for 2010 (Figure 17) and 2013 (Figure 18)
respectively.
50%
50%
Item K “JOP
2013”
Pooled
Narrative/
Both
57%
43%
Item K “JCP
2010”
Pooled
Narrative/
Both
61%
39%
Item K “JCP
2013”
Pooled
Narrative/
Both
29%
71%
Item K “JOP
2010”
Pooled
Narrative/
Both
Figure
15
Figure
16
Figure
17 Figure
18
49
Regarding the percentage of adherence amongst all Systematic Reviews in (JoP)
and (JCP), the results revealed significant reduction in the number of Systematic reviews
adhering to each checklist for each one of the selected Journals as the predetermined
percentage of adherence increased from ≥50% to ≥90%. This is represented in (figure 19)
and (figure 20) for Journal of Periodontology and Journal of Clinical Periodontology
respectively. An interesting finding was the percentage of adherence in (JCP) Systematic
Reviews when the predetermined percentage of adherence was ≥90%, which was
equivalent to 0% according to AMSTAR items.
% Of Adherence Amongst All SR in JoP
Figure
19
50
CHAPTER 4- DISCUSSION
In this descriptive cross sectional survey of the reporting quality of Systematic
Reviews in Journal of Periodontology and Journal of Clinical Periodontology, the mean
percentage of individual item consistency for Journal of Periodontology was 64.4% and
76.6% in 2010 for AMSTAR and Gelnny items respectively. The corresponding mean
percentages for 2013 were 62.3% and 74.8%. Regarding Journal of Clinical
Periodontology, the mean percentage of adherence was 66.2% and 75.7% in 2010 for
AMSTAR and Gelnny items respectively. The corresponding mean percentages for 2013
were 65% and 79%. Previous reports did not provide the mean percentage of individual
% Of Adherence Amongst All SR in
JCP
Figure
20
51
item adherence, however, it was possible to calculate such percentage based on the
provided data
3,
10,
15,
16
. The calculation of such percentage was done similar to the
methodology of this project, (i.e. by excluding any non-applicable item from the total
number of items). However, one of the options that was present in the previous reports
was “Can not Answer” which scores zero if present. Such item was also removed from
the total number when the percentage calculation was done for the previous reports to
compare their results with the results of this project. The range of the mean percentage of
individual item consistency to AMSTAR checklist for the previous reports was from 45%
to 74%. It can be seen that the range of the previous reports is much wider compared to
the overall range of the current project, which was 62.3% to 66.2% for both Journals in
the years of 2010 and 2013. This could be attributed to the specific selection of
Systematic Reviews for a specific topic amongst many Journals and many years in the
previous reports in contrast to evaluating all Systematic Reviews within two selected
journals in specific years. This means that the range of years of the included Systematic
Reviews, and the journals from which the Systematic reviews were selected in the
previous reports were much wider, which is likely to result in a wider range of
consistency with AMSTAR items. This shows that lack of consistency in reporting
essential items for good quality Systematic Reviews is likely to be even more prominent
when different Journals and wider time periods are included. Therefore, having specific
tools with specific items that should be used amongst all scientific Journals to provide
consistently reproducible results with optimum reporting quality and reliable conclusions
is extremely important.
52
Regarding Glenny’s Checklist, previous reports consistency ranged from 53% to
79.4%,
(3,
10,
15,
16)
which is much wider than the results derived from the total number
Systematic Reviews in the current project (74.8% to 79%). This is most probably due to
the same reasons discussed for the variation in the range with AMSTAR checklist.
In this project, the mean value of adherence of each Systematic Review to each
checklist in each journal within each year was calculated. This calculation was not done
in the previous reports and it was not possible to calculate it based on the available data
3,
10,
15,
16
. However, the authors of this project believed that having such mean values will
provide an additional insight to the range of adherence of the selected Systematic
Reviews with each checklist in each year. This will subsequently indicates the level of
consistency in reporting of each Journal in each year. Our results revealed that the range
of the mean values for the number of articles evaluated in Journal of Periodontology by
AMSTAR was 0.36-0.81 in 2010 and 0.18-0.9 in 2013. The corresponding range for
articles evaluated by Gelnny’s checklist was 0.45-0.92 and 0.38-0.92 respectively.
Furthermore, Journal of Clinical Periodontology revealed a mean AMSTAR range of
0.36-0.81 in 2010 and 0.45-0.81 in 2013. With regards to the values for Glenny’s
checklist, they ranged from 0.57-0.92 and 0.54-0.92 in 2010 and 2013 respectively. Such
wide range in these mean values is a representation of the significant variation and lack
of reporting consistency between the Systematic Reviews in both Journal and both years.
These results are in agreement with the lack of uniformity and the significant variation
reported in the previous reports
3,
10,
15,
16
.
53
Amongst the Systematic Reviews in both journals in the years of 2010 and 2013,
adherence to item #4 (was the status publication “i.e. grey literature” used as an inclusion
criteria) in AMSTAR checklist was consistently low. The term “grey Literature” refers to
unpublished or incompletely published literature such as conference proceeding,
abstracts, graduate thesis and company reports. This item has been shown to be poorly
reported in pervious reports as well
3, 10, 15, 16
. In fact, one of the reports evaluating the
quality of 10 Systematic Reviews on short dental implants reported 0% adherence for
item #4
15
. This lack of adherence to item #4 could deteriorate the reliability of the
conclusions derived from these Systematic Reviews. This is because not including grey
literature in Meta-analysis has been shown to significantly increase the potential of
overestimating favorable results
72
. Such overestimation had been reported when a
number of Systematic Reviews and Meta-Analyses were analyzed with and without the
inclusion of the gray literature and showed that excluding grey literature in some of the
Systematic Reviews changed the overall pooled results dramatically from being
statistically insignificant to statistically significant. In addition, larger odds ratio estimates
and larger estimate of intervention effectiveness were evident when the results were
pooled without including grey literature in contrast to the same Systematic Reviews with
the inclusion of grey literature
72
. This could indeed compromise the quality of the
conclusions derived from Systematic Reviews with subsequent wrong clinical translation.
In addition to item #4 from AMSTAR checklist, Item #D (search for published and
un-published literature) from Glenny’s checklist was consistently poorly reported. This
item overlaps with item #4 from AMSTAR, as both items address the same point, which
54
is the inclusion of unpublished data in Systematic Reviews. Despite the importance of
including unpublished data, it can be argued that it is not always applicable and easy to
apply. For instance, Access for such data may not always be easy or it might be limited,
especially, for Systematic Reviews not being conducted in academic institutions.
Furthermore, data that is unpublished might have poor reliability and accuracy to be
published in peer-reviewed Journals since some of them might have been rejected for
publication in the first place, and including such data with inferior quality could
significantly compromise the weight of the included high quality data. Subsequently,
compromising the quality of the conclusions derived from these Systematic Reviews
7
. In
other words, including grey literature can act as a double edge sword.
On the other hand, Including grey literature has been shown to be an essential
component for preventing publication bias interfering with the quality and reliability of
the final outcome of a Systematic Review, as it has been reported that studies with
positive outcome are more likely to be published in contrast to studies with negative or
even neutral outcomes
79
. In addition, previous reports have shown that only positive
results in specific topics are published in some Asian countries
80
. All of these factors
along with the previously discussed factors will certainly contribute to overestimating the
beneficial outcome of an intervention leading to inaccurate answers to the focused
questions, which will result in poor clinical translation and application.
55
Regarding publication bias, which is represented by item #10 in AMSTAR checklist,
it was also poorly reported in both, Journal of Periodontology and Journal of Clinical
Periodontology in the years of 2010 and 2013. Such finding was very consistent with
previous reports evaluating the reporting quality of Systematic Reviews in specific topics
in Periodontology, in which the percentage of adherence with such item ranged from 0%-
33%
3
,
10,
15, 16
.
Reporting such aspect in Systematic Reviews is a definite requirement for
improvement due to the great impact it has on the reporting quality and the conclusions
derived from the articles along with the previous items discussed. It was noticed
however, that some improvement in reporting publication bias was present amongst the
two journals between 2010 and 2013 by 21.5% for JoP and 21% for JCP, yet, the
percentage of adherence was still low and in need of significant improvement (14.2%-
35.7% and 14.2%-35.2% for JoP and JCP in 2010 and 2013 respectively).
Since the effect of not including grey literature and un-published data in Systematic
Reviews have been reported to increase the risk of publication bias, items #4 (was the
status of publication “i.e. grey literature” used as an inclusion criterion), item #D (search
for published and unpublished literature) and item #10 (was the likelihood of publication
bias assessed) within all the Systematic Reviews in this project were correlated based on
the percentage of adherence to each one of these items. It was found that only 11 out of
the 42 Systematic Reviews (26%) included in this project reported on publication bias
36,
31,
60, 54,
51, 62, 61, 43, 68, 27, 20
. Amongst these 11 articles, 10 did not include grey literature
and unpublished data in their search strategy
36,
31,
60, 54,
51, 62, 61, 43, 27, 20
. Within these 10
articles, 2 reported definitive presence of publication bias
27, 36
, and 3 reported possible
56
publication bias
31, 51, 54
. Publication bias was assessed with specific statistical tests
(funnel plot & Egger regression test) when possible. In some Systematic Reviews, it was
not possible to do these statistical tests due to the small number of primary research
included, these include two of the articles that reported possible publication bias
31, 51
.
Further details about one of the Systematic Reviews that reported possible publication
bias is that one the statistical tests did indicate the presence of publication bias (funnel
plot asymmetry), however another test did not show such bias (Egger regression test).
Therefore, despite reporting no significant bias in the results section, the authors of the
Systematic Review stated that an exclusion of publication bias can be assumed, such
assumption indicates lack of certainty, hence, the actual possibility of the presence of
publication bias
54
. All of the Systematic Reviews that reported the presence of
publication bias and even those which reported the possible presence of publication bias
never included grey literature and unpublished literature in their inclusion criteria. Such
finding is very consistent with what have been reported in the literature on the high risk
of the presence of publication bias as result of not including grey and unpublished
literature
79
.
The results of the project have also shown consistent low adherence to item #E (were
all languages considered) in Glenny’s checklist amongst the selected Systematic
Reviews. English language was the most common language used for inclusion criteria of
the selected Systematic Reviews. Out of the 28 Systematic Reviews that reported
language restriction, 24 articles specified that only English publications were selected for
inclusion
33,
32,
36,
30, 29, 34, 50, 60, 55, 51, 61, 62,
46, 67,
65,
28,
19, 20,
40, 27, 42, 43, 69, 18
. The remaining 4
57
Systematic Reviews required English as well as other specific languages for inclusion
criteria
41, 63, 64, 70
. These language specification has been reported to increase the potential
of publication bias
3
. Such bias could be a result of excluding a significant amount of
publications in other languages which may have had an effect on the outcome of the
Systematic Review
3
. In fact, it was reported that the inclusion of articles in languages
other than English have a significant effect on the width of Confidence Interval, making
it narrower with subsequent statistical significant difference in the results’ precision
despite not having a significant effect on the estimate difference
81
. It was also reported
that articles reporting positive results are more likely to be published in English language
Journals compared to article that publish negative results which were shown to be
published more in non English Journals
73
. Furthermore, inclusion of other languages will
increase the cumulative power of the results as the sample size will be bigger
81
.
Despite all of these important aspects of including all languages, it is sometimes not
practical to do so due to the limited access for publications in other languages, the need
of translation which is not always feasible, as well the additional time and cost associated
with these factors. Another argument is that Randomized Controlled Trials published in
English have been shown to have better methodological quality than those in languages
other than English
82
, however, this is an area of controversy as other reports showed no
difference in the methodological quality between the two
83
. In this project, Systematic
Reviews that reported definitive publication bias
(27, 36)
had English language publications
only, in their inclusion criteria. Regarding those, which reported possible publication bias
(31, 51, 54)
, only one Systematic Review excluded languages other than English in the
58
inclusion criteria
51
. It is difficult to correlate language restrictions to publication bias in
this project due to the limited number of studies reporting publication bias, however,
within such limitations, it was noted that the only two Systematic Reviews that reported
the presence of publication bias based on statistical test did have language restriction
(English only) in their inclusion criteria which is consistent with previous reports
claiming the increase in publication bias risk when such restriction is present. Further
investigations are needed looking specifically into such aspect and including a larger
number of Systematic Reviews that reports positive publication bias to have more reliable
conclusions.
Item #11 (was the conflict of interest included) in AMSTAR checklist was constantly
poorly reported. Such item showed the worst percentage of adherence amongst all other
items in both Journals as the percentage of adherence was consistently 0% for both
journals and both years. Despite all of the included Systematic Reviews in both journals
reporting conflict of interest for the actual Systematic Reviews, none of them reported
conflict of interest or funding sources of the primary research included in these reviews.
Since Shea et al. 2007, clearly stats that this item can only be fulfilled when the reviewers
report conflict of interest for both, the Systematic Review and the primary research
included
77
, consistency with item #11 in AMSTAR was not met by any of the included
Systematic Reviews. We found that our results were not consistent with previous reports
evaluating the reporting quality of Systematic Reviews in specific aspect of
Periodontology as their percentage of Systematic Reviews fulfilling item #11 ranged
from 40% to 63%
3, 10, 16
. However, the author of one of the reports stated in the
59
discussion that disclosing conflict of interest for each article included in the Systematic
Review evaluated was not addressed
10
, in other words, they were only looking at the
conflict of interest of the Systematic Review, not the primary research included, which is
not what the author of AMSTAR recommends. Furthermore, two additional articles
reported relatively higher percentages of adherence for item #11
15,16
. However, in those
two articles, the methodology used for evaluating the fulfillment of item #11 was not
clear, yet, it is possible that it was the same as the previous article discussed because all
of these articles share the same authorship, hence, sharing the same methodology and the
relatively higher percentage of adherence for items #11 is very likely. Similarly, an older
report by a different author showed relatively high adherence for item #11 with lack of
clear explanation to the fulfillment criteria
3
. Despite the adherence percentage of this
report being high relative to the results of the current project, it was generally low (40%),
indicating the lack of consistent reporting of conflict of interest in Systematic Reviews in
Periodontology, which was also reported in the medical filed
84
. Conflict of interest has
been shown to increase the potential of masking and neglecting significant findings that
would interfere with such interest
85
. Therefore, reporting conflict of interest is one of the
key components for a good quality Systematic Reviews as it closely relates to publication
bias and subsequent biased conclusions. This is true for the Systematic Review as well as
the primary research included in the Systematic Review because conflict of interest
within the primary research will give pooled results that are based on such conflict. This
will lead to an overall unreliable conclusion due to the high potential of bias associated
with it. Therefore, authors should pay close attention to such aspect and should clearly
report conflict of interest of the primary research as well as the Systematic Reviews so
60
that the reader can interpret the information based on the reliability and the clinical
integrity of what has been concluded and make up a subsequent informed clinical
decision for patients.
Assessing the quality of the primary research included in a Systematic Review is
extremely important, as poor primary research will distort the pooled outcome of a
Systematic Review resulting in poor reliability. Evidence have shown that there is
selective reporting as well as manipulation of outcomes in Randomized Trials, in
addition, it was reported that statistically significant results are twice as likely to be
reported compared to non-statistical significant results
86
. In order for quality assessment
to be addressed with high reproducibility and unbiased approach, Glenny and colleagues
(
1)
recommended having at least two reviewers assessing such aspect. This is represented
by Item #J which stats; (was it stated that the quality assessment was carried out by two
reviewers). Unfortunately, item #J was found to be consistently poorly reported in this
project. Such poor reporting was consistent with previous reports using the same
checklist for evaluating the reporting quality of Systematic Reviews of specific topics in
Periodontology
15, 16
. On the other hand, one previous article reported a very high
percentage of adherence for such item (100%)
3
. Such significant difference in the
percentage of adherence for this article compared to the current project could be
attributed to the relative small number of the Systematic Reviews included (n=10)
(3)
compared to this project (n=42), hence, a possible higher potential for variation amongst
the included Systematic Reviews with regards to the adherence to such item. This
indicates the lack of consistency in reporting such aspect amongst Systematic Reviews in
61
the field of Periodontology, which will increase the potential of compromising the
reliability and reproducibility of the quality assessment with subsequent negative effect
on the final conclusions.
Descriptive trend analysis for individual item consistency between 2010 and 2013 in
JoP revealed improvement in two items only from AMSTAR checklist. These were items
#9 (were the methods used to combine the findings of studies appropriate) and #10 (was
the likelihood of publication bias assessed). Despite improvement of the Systematic
Reviews’ consistency with item #10 from 14.2% in 2010 to 35.7% in 2013, the
percentage of the adherence is still relatively low and in need for significant
improvement. Furthermore, the remaining items which have been shown to be
consistently poorly reported in the results, namely, items #4 (was the status of publication
“i.e. grey literature” used as an inclusion criteria) and #11(was the conflict of interest
included), showed different trend between the three years period. In other words,
consistency with item #4 amongst the Systematic Reviews published in JoP 2013 was
lower than those published in 2010 by 7.2% (14.2% in 2010 and 7% in 2013). Despite
such reduction being relatively small, it is very significant as it shows that such
consistently poorly reported item seems to be reported even less in more recent
publications. On the other hand, item #11 remained the most poorly reported item with
0% adherence in 2010 and 2013 publications. This indicates the lack of effort to improve
the reporting of the items that have been shown to be consistently poorly reported, which
could lead to further deterioration of the reporting quality of Systematic Reviews
published in Journal of Periodontology.
62
Moving on to the trend of improvement in Journal of Periodontology (JoP) according
to Glenny’s checklist, four out the fourteen items improved, including items #B, #L, #N
and #O. in addition, Three items (#A, #I and #M) showed consistent perfect reporting in
both the years of assessment. On the other hand, six items (#C, #D, #E, #F, #G and #J)
showed lower percentage of adherence amongst the Systematic Reviews in 2013 in
contrast to 2010 indicating that almost half the items in Glenny’s checklist are being
addressed less in the 2013 Systematic Reviews. Amongst those six items are the
consistently poorly reported items (#D ” search for published and unpublished literature”,
#E ” were all languages considered” and #J ” was it stated that the quality assessment was
carried out by two reviewers) which showed even less adherence in 2013 in contrast to
2010. This will indeed increase the negative impact on the reporting quality of Systematic
Reviews and the subsequent conclusions derived from them.
Regarding the descriptive trend analysis for individual item consistency between
2010 and 2013 in Journal of Clinical Periodontology (JCP), improvement was noted in
four items according to AMSTAR checklist, which are items #7, #8, #9 and #10.
Amongst those four items is one of the items that have been shown to be consistently
poorly reported. (i.e. item #10). Despite the improvement in the percentage of reporting
such item in 2013 relative to 2010 (14.2% in 2010- 35.2% in 2013), it was still relatively
poorly reported necessitating the need to address it more consistently so that the reporting
quality can improve more significantly in future publications. Regarding the other items
that were shown to be consistently poorly reported in the results, item #4 revealed lower
adherence percentage in 2013 in contrast to 2010 by 31.7% (42.8% in 2010- 11.1% in
63
2013). In addition, item #11 showed consistent 0% reporting without any improvement
between the three years period. Based on these results, it seems like despite the six years
period since the publication of AMSTAR checklist
(
14)
relative to most recent
publications selected in this project (i.e. 2013 publications), addressing conflict of interest
in the primary research within Systematic Reviews (item#11) has never been considered
in both, Journal of Periodontology and Journal of Clinical Periodontology in the years of
2010 and 2013. This is a significant drawback on the reporting quality of the Systematic
Reviews from the two journals due to the impact of such item on the overall quality and
reliability of the conclusions
85
.
Trend of improvement in Journal of clinical periodontology according to
Glenny’s checklist showed that the adherence percentage to six items improved from
2010 to 2013. This shows that almost half of the items in the checklist are improving in
terms of the number of Systematic Review fulfilling these items. These items are #E, #F,
#G, #H, #I, #L, and #N. Amongst these items is item #E which has been shown to be
consistently poorly reported. Approximately third of the included Systematic Reviews in
2010 (28.5%) looked at primary research with all languages (i.e. Item E) in contrast to
half of the selected Systematic Reviews published in 2013 (50%), which is indeed a
relatively good improvement. However, the discrepancy between the numbers of
Systematic Reviews analyzed in each year (n=7 in 2010, n=18 in 2013) could have
contributed to such difference in adherence, never the less, this could indicate that the
bigger the number of the evaluated Systematic Reviews, the greater the potential for more
article addressing item #E which has been shown to have a great impact on the reliability
64
of the results and conclusions
3,73, 81
. On the other hand, items #D and #J both showed
less consistency amongst the selected articles in 2013 compared to 2010 with item #D
dropping from 42.8% in 2010 to 11.1%in 2013, and items #J dropping from 40%-20%
between the three years period. Lack of reporting of these items along with the
suboptimal effort to improve their reporting in more recent publications can have
significant negative effects on the quality of the evaluated Systematic Reviews. Hence, it
is really important improve the reporting quality of Systematic Reviews by utilizing
different validated tools to deliver the best standard of care for patients.
Item K (are the results given in narrative or pooled statistical analysis) from Glenny’s
checklist
(1)
was not included in evaluating the reporting quality of Systematic Reviews in
Journal of Periodontology and Journal of Clinical Periodontology. The results of such
item were included separately. This is because the limitations for not being able to pool
the result is not something the author of a Systematic Review can always control or
adjust. It depends on the statistical homogeneity of the primary research as well as the
clinical homogeneity of the entity being investigated. If significant heterogeneity is
present amongst the primary research of a specific topic, whether it was statistical or
clinical in nature, the results should not be pooled because the conclusion will be based
on significant variations and discrepancies leading to lack of reliability. Clinical and
scientific research should focus on both, clinical/scientific applicability and biostatistics.
Using statistics to raise the statistical power of the results is of great importance,
however, it has to be applied under the right conditions to help support and reinforce the
conclusions rather than distorting them. One of the benefits of such item is showing the
65
disciplines in which consistent scientific evidence is lacking, and provide
recommendations based upon these findings so that the future publications focus more on
such disciplines allowing them to be reliably pooled to provide stronger level of
evidence.
Assessment of Multiple Systematic Reviews (AMSTAR)
(14)
checklist and items
published by Glenny and colleagues
(1)
are both validated items for evaluating the
reporting quality of Systematic Reviews. There are minor differences between the items
in both checklists, however, both checklists share similarities amongst many items, some
of which showed similar results in the Systematic Reviews assessed in this project.
Examples include item #4 in AMSTAR (was the status of publication “i.e. grey
literature” used as an inclusion criterion?) and item #D (search for published and
unpublished literature). We noted that some important items which tend to have
significant effects on the results were present in one of the included checklists, for
instance, languages restriction was only specifically addressed in Glenny’s checklist
(item #E” were all languages considered ”) and seems to contribute significantly to the
final outcome of Systematic Reviews despite having some controversial evidence for
such aspect as discussed earlier
73,81, 82, 83
. On the other hand, publication bias and
conflict of interest (items #10 “was the likelihood of publication bias assessed” and item
#11 “was the conflict of interest included”) were only addressed in AMSTAR checklist.
Those two items have also been shown to contribute significantly to the integrity of the
final outcome as discussed earlier
79,85
. Therefore, it was believed that using two
checklists would provide great value to this project to address most of the components
66
that are likely to significantly affect the outcome of Systematic Reviews and correlate the
findings between the two checklists to evaluate the consistency of the results with what
has been previously reported in the literature.
AMSTAR checklist has very specific criteria for answering each of its items, which
improves the objectivity and the subsequent reproducibility of the tool. Such aspect was
lacking in Glenny and colleagues’ checklist. However, we were able to overcome such
limitation by discussing each item and coming up with consensus to provide consistent
and reproducible results. We noticed several overlapping between the items in the two
checklists, however, due to the fact that AMSTAR have specific criteria for answering
each question, minor differences in the adherence percentages resulted between items that
evaluated very similar aspects. For instance, item #7 in AMSTAR (was the scientific
quality of the included studies assessed and documented) is not necessarily consistent
with item #H (did reviewers attempt to assess the quality of the included studies) despite
sounding the same. Such lack of consistency, which was noted in some of the evaluated
Systematic Reviews
(19, 33, 41, 50, 58, 63),
was due to the difference in the criteria needed to
fulfill each item. The authors of AMSTAR require the quality assessment to be done for
each article included in the primary research with a specific score per article. On the other
hand, some Systematic Reviews tend to give a score summery which we believed should
be credited for the authors when answering item #H in Gelnny’s checklist as it is indeed
an attempt to assess the quality of the included studies within Systematic Reviews.
67
Furthermore, item #8 in AMSTAR (was the scientific quality of the included studies
used appropriately in formulating conclusions) and item #I in Glenny’s checklist (was the
quality assessment included in the analysis) should be consistent as both address the
effect the quality of the primary research has on the final outcome of the Systematic
Review. However, this was not always evident in our project due to the strict criteria in
AMSTAR checklist, which necessitate item #7 (was the scientific quality of the included
studies assessed and documented) to score a “yes” so that item #8 can score a “yes” as
well. As discussed earlier, item #7 requires quality assessment for each of the included
articles in the primary research to be assessed, which does not match the criteria for item
#H (did reviewers attempt to assess the quality of the studies) in Glenny’s checklist,
hence, the lack of consistency in some of the assessed Systematic Reviews between items
#8 in AMASTAR and item #I in Glenny’s check list
33,
63,41,
19
.
The fact that item #8 (was the scientific quality of the included studies used
appropriately in formulating conclusions) in AMSTAR checklist criteria is dependent on
item #7 (was the scientific quality of the included studies assessed and documented) has
been shown to be one of the limitation of such checklist as one of included Systematic
Reviews did mention that their results should be interpreted with caution due to the
limitations of the Systematic Review which address item #8. However, it was not
possible to score a “yes” as item #7 methodology for evaluating the scientific quality of
the included primary research did not fulfill the criteria of AMSTAR as it was assessed
for a group of articles rather that having individual score per article
19
.
68
It should be emphasized that this project evaluates the reporting of quality of
Systematic Reviews published in Journal of Periodontology and Journal of Clinical
Periodontology in 2010 and 2013. Poorly reported item does not necessarily means that
such items were not included in the methodology of conducting the Systematic Reviews.
Based on these findings, the recommendation for the authors is to focus more on
reporting the poorly reported items identified in both checklists in order to allow
thorough interpretation by readers including clinicians, which is an essential aspect for
effective clinical translation of the highest level of evidence available in the literature.
Some of the limitations of this project include the number of evaluated Systematic
Reviews (n=42). A bigger number will indeed enhance the power of the results.
However, it seems like the trend of the poorly reported items was very consistent, hence,
it is fair to assume that the trend will be similar even if the number of Systematic
Reviews evaluated was bigger. In addition to that, using a checklist with relative
subjectivity (i.e. Glenny et al.
1
) can increase the potential of biased opinion due to the
lack of definitive fulfillment criteria for some the it’s items. However, Maximum effort
was conducted to limit such bias by thorough discussion between the authors of the items
in order to reach consensus as needed. Furthermore, statistical testing of inter-rater and
intra-rater agreements was not done, which could have caused lack of consistency in
addressing the items on the checklists. Such agreement testing would have been more
valuable for Glenny’s items due to the relative subjectivity in contrast to AMSTAR
items. However, discussion of questionable items between reviewers revealed a generally
high percentage of agreement, which was above 90%.
69
CHAPTER 5- CONCLUSION
In conclusion, the reporting quality of Systematic Reviews in Journal of
Periodontology and Journal of Clinical Periodontology according to AMSTAR and
Glenny’s checklist can be improved in the areas identified through this project.
70
BIBLIOGRAPHY
1.
Glenny
AM,
Esposito
M,
Coulthard
P,
Worthington
HV.
The
assessment
of
systematic
reviews
in
dentistry.
European
journal
of
oral
sciences
2003;111:85-‐92.
2.
Sutherland
SE.
Evidence-‐based
dentistry:
Part
IV.
Research
design
and
levels
of
evidence.
Journal
2001;67:375-‐378.
3.
Chambrone
L,
Faggion
CM,
Jr.,
Pannuti
CM,
Chambrone
LA.
Evidence-‐based
periodontal
plastic
surgery:
an
assessment
of
quality
of
systematic
reviews
in
the
treatment
of
recession-‐type
defects.
Journal
of
clinical
periodontology
2010;37:1110-‐1118.
4.
Moher
D,
Tetzlaff
J,
Tricco
AC,
Sampson
M,
Altman
DG.
Epidemiology
and
reporting
characteristics
of
systematic
reviews.
PLoS
medicine
2007;4:e78.
5.
Needleman
IG.
A
guide
to
systematic
reviews.
Journal
of
clinical
periodontology
2002;29
6.
Suppl3:6-‐9;discussion37-‐38.
http://libguides.mssm.edu/c.php?g=168543&p=1107631.Accessed:
January,
2015.
7.
Sacks
HS,
Berrier
J,
Reitman
D,
Ancona-‐Berk
VA,
Chalmers
TC.
Meta-‐analyses
of
randomized
controlled
trials.
The
New
England
journal
of
medicine
1987;316:450-‐455.
71
8.
Avula
H,
Pandey
R,
Bolla
V,
Rao
H,
Avula
JK.
Periodontal
Research:
Basics
and
beyond
-‐
Part
I
(Defining
the
research
problem,
study
design
and
levels
of
evidence).
Journal
of
Indian
Society
of
Periodontology
2013;17:565-‐570.
9.
Mulrow
CD.
The
medical
review
article:
state
of
the
science.
Annals
of
internal
medicine
1987;106:485-‐488.
10.
Elangovan
S,
Avila-‐Ortiz
G,
Johnson
GK,
Karimbux
N,
Allareddy
V.
Quality
assessment
of
systematic
reviews
on
periodontal
regeneration
in
humans.
Journal
of
periodontology
2013;84:176-‐185.
11.
Moher
D,
Cook
DJ,
Eastwood
S,
Olkin
I,
Rennie
D,
Stroup
DF.
Improving
the
Quality
of
Reports
of
Meta-‐Analyses
of
Randomised
Controlled
Trials:
The
QUOROM
Statement.
Onkologie
2000;23:597-‐602.
12.
Liberati
A,
Altman
DG,
Tetzlaff
J,
et
al.
The
PRISMA
statement
for
reporting
systematic
reviews
and
meta-‐analyses
of
studies
that
evaluate
health
care
interventions:
explanation
and
elaboration.
Journal
of
clinical
epidemiology
2009;62:e1-‐34.
13.
Group
C.
Cochrane
Handbook
of
Systenatic
Reviews
of
interventions.
UK:
Wiley-‐Blackwell
2008.
14.
Shea
BJ,
Grimshaw
JM,
Wells
GA,
et
al.
Development
of
AMSTAR:
a
measurement
tool
to
assess
the
methodological
quality
of
systematic
reviews.
BMC
medical
research
methodology
2007;7:10.
15.
Elangovan
S,
Mawardi
HH,
Karimbux
NY.
Quality
assessment
of
systematic
reviews
on
short
dental
implants.
Journal
of
periodontology
2013;84:758-‐
767.
72
16.
Juan
G.
De
Buitrago
GA-‐O,
Satheesh
Elangovan.
quality
assessmnet
of
systematic
reviews
on
alveolar
ridge
preservation
.
JADA
2013;144:1349-‐
1357.
17.
ReutersT.http://admin
apps.webofknowledge.com/JCR/JCR?RQ=LIST_SUMMARY_JOURNAL.
2013.
Accesses:
December,
2014.
18.
Borgnakke
WS,
Ylostalo
PV,
Taylor
GW,
Genco
RJ.
Effect
of
periodontal
disease
on
diabetes:
systematic
review
of
epidemiologic
observational
evidence.
Journal
of
periodontology
2013;84:S135-‐152.
19.
D'Aiuto
F,
Orlandi
M,
Gunsolley
JC.
Evidence
that
periodontal
treatment
improves
biomarkers
and
CVD
outcomes.
Journal
of
periodontology
2013;84:S85-‐S105.
20.
Engebretson
S,
Kocher
T.
Evidence
that
periodontal
treatment
improves
diabetes
outcomes:
a
systematic
review
and
meta-‐analysis.
Journal
of
periodontology
2013;84:S153-‐169.
21.
Ide
M,
Papapanou
PN.
Epidemiology
of
association
between
maternal
periodontal
disease
and
adverse
pregnancy
outcomes-‐-‐systematic
review.
Journal
of
periodontology
2013;84:S181-‐194.
22.
Linden
GJ,
Herzberg
MC,
working
group
4
of
the
joint
EFPAAPw.
Periodontitis
and
systemic
diseases:
a
record
of
discussions
of
working
group
4
of
the
Joint
EFP/AAP
Workshop
on
Periodontitis
and
Systemic
Diseases.
Journal
of
periodontology
2013;84:S20-‐23.
73
23.
Linden
GJ,
Lyons
A,
Scannapieco
FA.
Periodontal
systemic
associations:
review
of
the
evidence.
Journal
of
periodontology
2013;84:S8-‐S19.
24.
Reyes
L,
Herrera
D,
Kozarov
E,
Rolda
S,
Progulske-‐Fox
A.
Periodontal
bacterial
invasion
and
infection:
contribution
to
atherosclerotic
pathology.
Journal
of
periodontology
2013;84:S30-‐50.
25.
American
Academy
of
Periodontology
statement
on
the
use
of
moderate
sedation
by
periodontists.
Journal
of
periodontology
2013;84:435.
26.
Soardi
E,
Cosci
F,
Checchi
V,
Pellegrino
G,
Bozzoli
P,
Felice
P.
Radiographic
analysis
of
a
transalveolar
sinus-‐lift
technique:
a
multipractice
retrospective
study
with
a
mean
follow-‐up
of
5
years.
Journal
of
periodontology
2013;84:1039-‐1047.
27.
Lin
GH,
Chan
HL,
Wang
HL.
The
significance
of
keratinized
mucosa
on
implant
health:
a
systematic
review.
Journal
of
periodontology
2013;84:1755-‐
1767.
28.
Monje
A,
Fu
JH,
Chan
HL,
et
al.
Do
implant
length
and
width
matter
for
short
dental
implants
(<10
mm)?
A
meta-‐analysis
of
prospective
studies.
Journal
of
periodontology
2013;84:1783-‐1791.
29.
Atieh
MA,
Alsabeeha
NH,
Faggion
CM,
Jr.,
Duncan
WJ.
The
frequency
of
peri-‐
implant
diseases:
a
systematic
review
and
meta-‐analysis.
Journal
of
periodontology
2013;84:1586-‐1598.
30.
Ricci
L,
Perrotti
V,
Ravera
L,
Scarano
A,
Piattelli
A,
Iezzi
G.
Rehabilitation
of
deficient
alveolar
ridges
using
titanium
grids
before
and
simultaneously
with
74
implant
placement:
a
systematic
review.
Journal
of
periodontology
2013;84:1234-‐1242.
31.
Sgolastra
F,
Severino
M,
Pietropaoli
D,
Gatto
R,
Monaco
A.
Effectiveness
of
periodontal
treatment
to
improve
metabolic
control
in
patients
with
chronic
periodontitis
and
type
2
diabetes:
a
meta-‐analysis
of
randomized
clinical
trials.
Journal
of
periodontology
2013;84:958-‐973.
32.
Monje
A,
Chan
HL,
Fu
JH,
Suarez
F,
Galindo-‐Moreno
P,
Wang
HL.
Are
short
dental
implants
(<10
mm)
effective?
a
meta-‐analysis
on
prospective
clinical
trials.
Journal
of
periodontology
2013;84:895-‐904.
33.
Darby
IB,
Morris
KH.
A
systematic
review
of
the
use
of
growth
factors
in
human
periodontal
regeneration.
Journal
of
periodontology
2013;84:465-‐476.
34.
Zandbergen
D,
Slot
DE,
Cobb
CM,
Van
der
Weijden
FA.
The
clinical
effect
of
scaling
and
root
planing
and
the
concomitant
administration
of
systemic
amoxicillin
and
metronidazole:
a
systematic
review.
Journal
of
periodontology
2013;84:332-‐351.
35.
Douglas
de
Oliveira
DW,
Oliveira-‐Ferreira
F,
Flecha
OD,
Goncalves
PF.
Is
surgical
root
coverage
effective
for
the
treatment
of
cervical
dentin
hypersensitivity?
A
systematic
review.
Journal
of
periodontology
2013;84:295-‐306.
36.
Suarez
F,
Chan
HL,
Monje
A,
Galindo-‐Moreno
P,
Wang
HL.
Effect
of
the
timing
of
restoration
on
implant
marginal
bone
loss:
a
systematic
review.
Journal
of
periodontology
2013;84:159-‐169.
75
37.
Dhingra
K,
Vandana
KL.
Prophylactic
vaccination
against
periodontal
disease:
a
systematic
review
of
preclinical
studies.
Journal
of
periodontology
2010;81:1529-‐1546.
38.
Chambrone
L,
Chambrone
LA,
Lima
LA.
Effects
of
occlusal
overload
on
peri-‐
implant
tissue
health:
a
systematic
review
of
animal-‐model
studies.
Journal
of
periodontology
2010;81:1367-‐1378.
39.
Waasdorp
JA,
Evian
CI,
Mandracchia
M.
Immediate
placement
of
implants
into
infected
sites:
a
systematic
review
of
the
literature.
Journal
of
periodontology
2010;81:801-‐808.
40.
Javed
F,
Almas
K.
Osseointegration
of
dental
implants
in
patients
undergoing
bisphosphonate
treatment:
a
literature
review.
Journal
of
periodontology
2010;81:479-‐484.
41.
Chaffee
BW,
Weston
SJ.
Association
between
chronic
periodontal
disease
and
obesity:
a
systematic
review
and
meta-‐analysis.
Journal
of
periodontology
2010;81:1708-‐1724.
42.
Angaji
M,
Gelskey
S,
Nogueira-‐Filho
G,
Brothwell
D.
A
systematic
review
of
clinical
efficacy
of
adjunctive
antibiotics
in
the
treatment
of
smokers
with
periodontitis.
Journal
of
periodontology
2010;81:1518-‐1528.
43.
Atieh
MA,
Ibrahim
HM,
Atieh
AH.
Platform
switching
for
marginal
bone
preservation
around
dental
implants:
a
systematic
review
and
meta-‐analysis.
Journal
of
periodontology
2010;81:1350-‐1366.
44.
Chambrone
L,
Sukekava
F,
Araujo
MG,
Pustiglioni
FE,
Chambrone
LA,
Lima
LA.
Root-‐coverage
procedures
for
the
treatment
of
localized
recession-‐type
76
defects:
a
Cochrane
systematic
review.
Journal
of
periodontology
2010;81:452-‐478.
45.
Azarpazhooh
A,
Shah
PS,
Tenenbaum
HC,
Goldberg
MB.
The
effect
of
photodynamic
therapy
for
periodontitis:
a
systematic
review
and
meta-‐
analysis.
Journal
of
periodontology
2010;81:4-‐14.
46.
Shiau
HJ,
Reynolds
MA.
Sex
differences
in
destructive
periodontal
disease:
a
systematic
review.
Journal
of
periodontology
2010;81:1379-‐1389.
47.
Faggion
CM,
Jr.,
Atieh
MA,
Park
S.
Search
strategies
in
systematic
reviews
in
periodontology
and
implant
dentistry.
Journal
of
clinical
periodontology
2013;40:883-‐888.
48.
Faggion
CM,
Jr.,
Giannakopoulos
NN.
Critical
appraisal
of
systematic
reviews
on
the
effect
of
a
history
of
periodontitis
on
dental
implant
loss.
Journal
of
clinical
periodontology
2013;40:542-‐552.
49.
Katsamakis
S,
Slot
DE,
Van
der
Sluis
LW,
Van
der
Weijden
F.
Histological
responses
of
the
periodontium
to
MTA:
a
systematic
review.
Journal
of
clinical
periodontology
2013;40:334-‐344.
50.
Tran
DT,
Gay
I,
Du
XL,
et
al.
Assessing
periodontitis
in
populations:
a
systematic
review
of
the
validity
of
partial-‐mouth
examination
protocols.
Journal
of
clinical
periodontology
2013;40:1064-‐1071.
51.
Li
W,
Xiao
L,
Hu
J.
Matrix
metalloproteinase-‐1
promoter
-‐1607
1G/2G
polymorphism
and
chronic
periodontitis
susceptibility:
a
meta-‐analysis
and
systematic
review.
Journal
of
clinical
periodontology
2013;40:1095-‐1103.
77
52.
Faggion
CM,
Jr.,
Tu
YK,
Giannakopoulos
NN.
Reporting
adverse
events
in
randomized
controlled
trials
in
periodontology:
a
systematic
review.
Journal
of
clinical
periodontology
2013;40:889-‐895.
53.
Chambrone
L,
Preshaw
PM,
Rosa
EF,
et
al.
Effects
of
smoking
cessation
on
the
outcomes
of
non-‐surgical
periodontal
therapy:
a
systematic
review
and
individual
patient
data
meta-‐analysis.
Journal
of
clinical
periodontology
2013;40:607-‐615.
54.
Sgolastra
F,
Petrucci
A,
Severino
M,
Graziani
F,
Gatto
R,
Monaco
A.
Adjunctive
photodynamic
therapy
to
non-‐surgical
treatment
of
chronic
periodontitis:
a
systematic
review
and
meta-‐analysis.
Journal
of
clinical
periodontology
2013;40:514-‐526.
55.
Figuero
E,
Carrillo-‐de-‐Albornoz
A,
Martin
C,
Tobias
A,
Herrera
D.
Effect
of
pregnancy
on
gingival
inflammation
in
systemically
healthy
women:
a
systematic
review.
Journal
of
clinical
periodontology
2013;40:457-‐473.
56.
Schmidt
JC,
Sahrmann
P,
Weiger
R,
Schmidlin
PR,
Walter
C.
Biologic
width
dimensions-‐-‐a
systematic
review.
Journal
of
clinical
periodontology
2013;40:493-‐504.
57.
Chambrone
L,
Foz
AM,
Guglielmetti
MR,
et
al.
Periodontitis
and
chronic
kidney
disease:
a
systematic
review
of
the
association
of
diseases
and
the
effect
of
periodontal
treatment
on
estimated
glomerular
filtration
rate.
Journal
of
clinical
periodontology
2013;40:443-‐456.
58.
de
Waal
YC,
van
Winkelhoff
AJ,
Meijer
HJ,
Raghoebar
GM,
Winkel
EG.
Differences
in
peri-‐implant
conditions
between
fully
and
partially
edentulous
78
subjects:
a
systematic
review.
Journal
of
clinical
periodontology
2013;40:266-‐
286.
59.
Buti
J,
Baccini
M,
Nieri
M,
La
Marca
M,
Pini-‐Prato
GP.
Bayesian
network
meta-‐
analysis
of
root
coverage
procedures:
ranking
efficacy
and
identification
of
best
treatment.
Journal
of
clinical
periodontology
2013;40:372-‐386.
60.
Matesanz-‐Perez
P,
Garcia-‐Gargallo
M,
Figuero
E,
Bascones-‐Martinez
A,
Sanz
M,
Herrera
D.
A
systematic
review
on
the
effects
of
local
antimicrobials
as
adjuncts
to
subgingival
debridement,
compared
with
subgingival
debridement
alone,
in
the
treatment
of
chronic
periodontitis.
Journal
of
clinical
periodontology
2013;40:227-‐241.
61.
de
Brandao
ML,
Vettore
MV,
Vidigal
Junior
GM.
Peri-‐implant
bone
loss
in
cement-‐
and
screw-‐retained
prostheses:
systematic
review
and
meta-‐
analysis.
Journal
of
clinical
periodontology
2013;40:287-‐295.
62.
Pan
Y,
Li
D,
Cai
Q,
et
al.
MMP-‐9
-‐1562C>T
contributes
to
periodontitis
susceptibility.
Journal
of
clinical
periodontology
2013;40:125-‐130.
63.
Lin
PY,
Cheng
YW,
Chu
CY,
Chien
KL,
Lin
CP,
Tu
YK.
In-‐office
treatment
for
dentin
hypersensitivity:
a
systematic
review
and
network
meta-‐analysis.
Journal
of
clinical
periodontology
2013;40:53-‐64.
64.
Kunnen
A,
van
Doormaal
JJ,
Abbas
F,
Aarnoudse
JG,
van
Pampus
MG,
Faas
MM.
Periodontal
disease
and
pre-‐eclampsia:
a
systematic
review.
Journal
of
clinical
periodontology
2010;37:1075-‐1087.
65.
Berchier
CE,
Slot
DE,
Van
der
Weijden
GA.
The
efficacy
of
0.12%
chlorhexidine
mouthrinse
compared
with
0.2%
on
plaque
accumulation
and
79
periodontal
parameters:
a
systematic
review.
Journal
of
clinical
periodontology
2010;37:829-‐839.
66.
Chambrone
L,
Chambrone
D,
Lima
LA,
Chambrone
LA.
Predictors
of
tooth
loss
during
long-‐term
periodontal
maintenance:
a
systematic
review
of
observational
studies.
Journal
of
clinical
periodontology
2010;37:675-‐684.
67.
Chao
YL,
Chen
HH,
Mei
CC,
Tu
YK,
Lu
HK.
Meta-‐regression
analysis
of
the
initial
bone
height
for
predicting
implant
survival
rates
of
two
sinus
elevation
procedures.
Journal
of
clinical
periodontology
2010;37:456-‐465.
68.
Dimou
NL,
Nikolopoulos
GK,
Hamodrakas
SJ,
Bagos
PG.
Fcgamma
receptor
polymorphisms
and
their
association
with
periodontal
disease:
a
meta-‐
analysis.
Journal
of
clinical
periodontology
2010;37:255-‐265.
69.
Slot
W,
Raghoebar
GM,
Vissink
A,
Huddleston
Slater
JJ,
Meijer
HJ.
A
systematic
review
of
implant-‐supported
maxillary
overdentures
after
a
mean
observation
period
of
at
least
1
year.
Journal
of
clinical
periodontology
2010;37:98-‐110.
70.
Tu
YK,
Woolston
A,
Faggion
CM,
Jr.
Do
bone
grafts
or
barrier
membranes
provide
additional
treatment
effects
for
infrabony
lesions
treated
with
enamel
matrix
derivatives?
A
network
meta-‐analysis
of
randomized-‐
controlled
trials.
Journal
of
clinical
periodontology
2010;37:59-‐79.
71.
Oxman
AD,
Guyatt
GH.
Validation
of
an
index
of
the
quality
of
review
articles.
Journal
of
clinical
epidemiology
1991;44:1271-‐1278.
80
72.
McAuley
L,
Pham
B,
Tugwell
P,
Moher
D.
Does
the
inclusion
of
grey
literature
influence
estimates
of
intervention
effectiveness
reported
in
meta-‐analyses?
Lancet
2000;356:1228-‐1231.
73.
Morrison
A,
Polisena
J,
Husereau
D,
et
al.
The
effect
of
English-‐language
restriction
on
systematic
review-‐based
meta-‐analyses:
a
systematic
review
of
empirical
studies.
International
journal
of
technology
assessment
in
health
care
2012;28:138-‐144.
74.
Sharif
MO,
Janjua-‐Sharif
FN,
Ali
H,
Ahmed
F.
Systematic
reviews
explained:
AMSTAR-‐how
to
tell
the
good
from
the
bad
and
the
ugly.
Oral
health
and
dental
management
2013;12:9-‐16.
75.
Tirlapur
SA,
Riordain
RN,
Khan
KS,
Collaboration
E-‐C.
Variations
in
the
reporting
of
outcomes
used
in
systematic
reviews
of
treatment
effectiveness
research
in
bladder
pain
syndrome.
European
journal
of
obstetrics,
gynecology,
and
reproductive
biology
2014;180:61-‐67.
76.
Shea
BJ,
Hamel
C,
Wells
GA,
et
al.
AMSTAR
is
a
reliable
and
valid
measurement
tool
to
assess
the
methodological
quality
of
systematic
reviews.
Journal
of
clinical
epidemiology
2009;62:1013-‐1020.
77.
Shea.
http://amstar.ca/About_Amstar.php.
BMC
medical
research
methodology
2007;7.
Accessed:
October,2014
78.
Stroup
DF,
Berlin
JA,
Morton
SC,
et
al.
Meta-‐analysis
of
observational
studies
in
epidemiology:
a
proposal
for
reporting.
Meta-‐analysis
Of
Observational
Studies
in
Epidemiology
(MOOSE)
group.
Jama
2000;283:2008-‐2012.
81
79.
Jadad
AR,
Rennie
D.
The
randomized
controlled
trial
gets
a
middle-‐aged
checkup.
Jama
1998;279:319-‐320.
80.
Andrew
Vickers
NG,
Robert
Harland,
and,
Rees
R.
Do
Certain
Countries
produce
only
positive
results?
A
systematic
review
of
controlled
Trials.
Controlled
Clnical
Trail
1998;19:159-‐166.
81.
Moher
D,
Pham
B,
Klassen
TP,
et
al.
What
contributions
do
languages
other
than
English
make
on
the
results
of
meta-‐analyses?
Journal
of
clinical
epidemiology
2000;53:964-‐972.
82.
Juni
P,
Holenstein
F,
Sterne
J,
Bartlett
C,
Egger
M.
Direction
and
impact
of
language
bias
in
meta-‐analyses
of
controlled
trials:
empirical
study.
International
journal
of
epidemiology
2002;31:115-‐123.
83.
Moher
D,
Fortin
P,
Jadad
AR,
et
al.
Completeness
of
reporting
of
trials
published
in
languages
other
than
English:
implications
for
conduct
and
reporting
of
systematic
reviews.
Lancet
1996;347:363-‐366.
84.
Papanikolaou
GN,
Baltogianni
MS,
Contopoulos-‐Ioannidis
DG,
Haidich
AB,
Giannakakis
IA,
Ioannidis
JP.
Reporting
of
conflicts
of
interest
in
guidelines
of
preventive
and
therapeutic
interventions.
BMC
medical
research
methodology
2001;1:3.
85.
Topol
EJ.
Failing
the
public
health-‐-‐rofecoxib,
Merck,
and
the
FDA.
The
New
England
journal
of
medicine
2004;351:1707-‐1709.
86.
Chan
AW,
Hrobjartsson
A,
Haahr
MT,
Gotzsche
PC,
Altman
DG.
Empirical
evidence
for
selective
reporting
of
outcomes
in
randomized
trials:
comparison
of
protocols
to
published
articles.
Jama
2004;291:2457-‐2465.
82
TABLE 1
Systematic
Reviews
included
Journal
Year
1. A
systematic
review
of
the
use
of
growth
factors
in
human
periodontal
regeneration
JoP
2013
2. Do
implant
length
and
width
matter
for
short
dental
implants
(less
than
10mm)?
A
meta
-‐analysis
of
prospective
studies
JoP
2013
3. The
significance
of
keratinized
mucosa
on
implant
health:
A
systematic
review
JoP
2013
4. Are
short
Dental
implants
(less
than
10mm)
effective?
A
meta-‐analysis
on
prospective
clinical
trials
JoP
2013
5. Effect
of
the
timing
of
restoration
on
implant
marginal
bone
loss:
A
systematic
review
JoP
2013
6. Effectiveness
of
periodontal
treatment
to
improve
metabolic
control
in
patients
with
chronic
periodontitis
and
type
II
diabetes:
a
meta-‐analysis
of
Randomized
Clinical
Trials
JoP
2013
7. Is
surgical
root
coverage
effective
for
the
treatment
of
cervical
dentine
hypersensitivity?
A
systematic
review
JoP
2013
8. Rehabilitation
of
deficient
Alveolar
ridges
using
Titanium
Grids
before
and
simultaneously
with
implant
placement:
A
systematic
review
JoP
2013
9. The
frequency
of
peri-‐implant
disease:
A
systematic
review
and
Meta-‐analysis
JoP
2013
10. The
clinical
effect
of
scaling
and
root
planing
and
the
concomitant
administration
of
systemic
amoxicillin
and
metronidazole:
a
systematic
review
JoP
2013
11. A
systematic
review
on
the
effects
of
local
antimicrobials
as
adjunct
to
sub-‐gingival
debridement,
Compared
with
sub-‐
gingival
debridement
alone,
in
the
treatment
of
chronic
periodontitis
JCP
2013
12. Adjunctive
photodynamic
therapy
to
non-‐surgical
treatment
of
chronic
periodontitis:
a
systematic
review
and
meta-‐
analysis
JCP
2013
83
13. Assessing
periodontitis
in
populations:
a
systematic
review
of
the
validity
of
partial
mouth
examination
protocols
JCP
2013
14. Biological
width
dimensions:
a
systematic
review
JCP
2013
15. Bayesian
network
meta-‐analysis
of
root
coverage
procedures:
ranking
efficacy
and
identification
of
best
treatment
JCP
2013
16. Differences
in
peri-‐implant
conditions
between
fully
and
partially
edentulous
subjects:
a
systematic
review
JCP
2013
17. Effect
of
pregnancy
on
gingival
inflammation
in
systemically
healthy
women:
a
systematic
review
JCP
2013
18. Effect
of
smoking
cessation
on
the
outcome
of
non-‐surgical
periodontal
therapy:
a
systematic
review
and
individual
patient
data
meta-‐analysis
JCP
2013
19. In
office
treatment
for
dentin
hypersensitivity:
a
systematic
review
and
network
meta-‐analysis
JCP
2013
20. Matrix
Metalloproteinase
-‐1
promoter-‐1607
1G/2G
polymorphism
and
chronic
periodontitis
susceptibility:
a
meta-‐analysis
and
systematic
review.
JCP
2013
21. MMP-‐9-‐1562C-‐T
contributes
to
periodontitis
susceptibility
JCP
2013
22. Peri-‐implants
bone
loss
in
cement
retained
and
screw
retained
prosthesis:
Systematic
review
JCP
2013
23. Periodontitis
and
chronic
kidney
disease:
a
systematic
review
of
the
association
of
disease
and
the
effect
of
periodontal
treatment
on
estimated
glomerular
filtration
rate.
JCP
2013
24. Reporting
adverse
events
in
RCT
in
periodontology:
a
systematic
review
JCP
2013
25. A
systematic
review
of
clinical
efficacy
of
adjunctive
antibiotics
in
the
treatment
of
smokers
with
periodontitis
JoP
2010
26. Association
between
chronic
Periodontal
disease
and
obesity:
A
systematic
review
and
meta-‐analysis
JoP
2010
84
27. Platform
switching
for
marginal
bone
preservation
around
dental
implants:
Systematic
Review
and
Meta-‐analysis
JoP
2010
28. Root
coverage
procedures
for
treatment
of
localized
recession-‐Type
defects:
A
Cochrane
systematic
review
JoP
2010
29. Sex
Differences
in
destructive
periodontal
disease:
A
systematic
review
JoP
2010
30. The
effect
of
photodynamic
therapy
for
periodontitis:
A
systematic
review
and
meta-‐Analysis
JoP
2010
31. Osseointegration
of
dental
implants
in
patients
undergoing
Bisphosphonate
treatment:
A
literature
review.
JoP
2010
32. A
systematic
review
of
implant
supported
maxillary
over-‐
dentures
after
mean
observation
period
of
at
least
1
year
JCP
2010
33. Do
bone
grafts
or
barrier
membranes
provide
additional
treatment
effects
for
infrabony
lesions
treated
with
enamel
matrix
derivatives?
A
network
meta-‐analysis
of
RCT
JCP
2010
34. Fc
gamma
receptor
polymorphism
and
their
association
with
periodontal
disease:
A
meta-‐analysis
JCP
2010
35. Meta-‐regression
analysis
of
the
initial
bone
height
for
predicting
implant
survival
rates
of
two
sinus
elevation
procedures.
JCP
2010
36. Periodontal
disease
and
pre-‐eclampsia:
A
systematic
review
JCP
2010
37. Predictors
of
tooth
loss
during
long-‐term
maintenance:
A
systematic
review
of
observational
studies
JCP
2010
38. The
efficacy
of
0.12%of
chlorhexidine
mouth
wash
compared
with
0.2%
on
plaque
accumulation
and
periodontal
parameters:
A
systematic
review
JCP
2010
39. Evidence
that
periodontal
treatment
improves
biomarkers
and
CVD
outcome
JoP
&
JCP
2013
40. Effect
of
periodontal
disease
on
diabetes:
Systematic
review
of
epidemiological
observational
studies
JoP
&
JCP
2013
41. Epidemiology
of
association
between
maternal
periodontal
disease
and
adverse
pregnancy
outcomes
–
Systematic
JoP
&
JCP
2013
85
review
42. Evidence
that
periodontal
treatment
improves
diabetes
outcomes:
A
systematic
review
and
meta-‐analysis
JoP
&
JCP
2013
TABLE 2
Excluded
Articles
Journal
Year
of
publication
Reason
for
exclusion
1. American Academy of
Periodontology Statement
on the use of Moderate
Sedation by Periodontitis
JoP
2013
Position
paper
from
the
American
Academy
of
Periodontology
2. Quality Assessment of
Systematic Reviews on
Periodontal Regeneration in
Humans
JoP
2103
Review
of
secondary
research
3. Quality Assessment of
Systematic Reviews on
Short Dental Implants
JoP
2013
Review
of
secondary
research
4. Radiographic Analysis of a
Transalveolar Sinus-Lift
Technique: A Multi-practice
Retrospective Study With a
Mean Follow-Up of 5 Years
JoP
2013
Retrospective
study
5. Critical appraisal of
systematic reviews on the
effect of a history of
periodontitis on dental
implant loss
JCP
2103
Review
of
secondary
research
6. Search strategies in
systematic Reviews in
periodontology and implant
JCP
2013
Neither
a
clinical
nor
a
scientific
study
86
dentistry
7. Histological responses of
the periodontium to MTA: a
systematic review
JCP
2013
Animal
study
8. Periodontal bacterial
invasion and infection:
contribution to
atherosclerotic pathology
JoP
&
JCP
2013
Narrative
review
9. Periodontal systemic
associations: review of the
evidence
JoP
&
JCP
2013
Narrative
review
10. Periodontitis and systemic
diseases: a record of
discussions of working
group 4 of the Joint
EFP/AAP Workshop on
Periodontitis and Systemic
Diseases
JoP
&
JCP
2013
Consensus
report
11. Effects of Occlusal
Overload on Peri-Implant
Tissue Health: A Systematic
Review of Animal-Model
Studies
JoP
2010
Animal
study
12. Immediate
placement
of
implants
into
infected
sites:
a
systematic
review
of
the
literature
JoP
2010
Data
From
animal
and
Human
Studies
13. Prophylactic Vaccination
Against Periodontal
Disease: A Systematic
Review of Preclinical
Studies
JoP
2010
Animal
study
14. Evidence-based periodontal
plastic surgery: an
assessment of quality of
systematic reviews in the
treatment of recession-type
JCP
2013
Review
of
secondary
research
87
defects
Abstract (if available)
Abstract
BACKGROUND: Systematic reviews (SR) with or without simultaneous meta-analyses intend to synthesize the vast amount of published primary research to answer a focused research question. Stringent research criteria must be followed to publish a high quality SR in order to attain clinically relevant answer to the focused research question. Validated instruments have been developed to assess the quality of SR such as the Assessment of Multiple Systematic Reviews (AMSTAR) and a checklist by Glenny and colleagues. Despite the best efforts of journal editors and publishers, the quality of reporting of SR remains below par impacting the evidence-based clinical decision-making process. ❧ PURPOSE: To assess the quality of reporting of Systematic Reviews with and without meta-analyses in two widely read periodontology journals, namely, Journal of Periodontology (JoP) and Journal or Clinical Periodontology (JCP) published in the years 2010 and 2013. ❧ MATERIAL AND METHODS: Electronic search using National Library of Medicine’s Medline database supplemented by manual hand search of JoP and JCP was done independently by two reviewers. AMSTAR statement and Glenny’s checklist were used to evaluate the reporting quality of the selected Systematic Reviews from both journals. Each item on these instruments was assigned an appropriate score (1=Yes, 0=No or N/A=Not applicable). The percentage of the number of articles fulfilling each item in each checklist for the years of 2010 and 2013 was calculated, excluding any non-applicable items from such calculation. Descriptive statistics were used to analyze the data. Trends in improvement of reporting were assessed between the two years in both the journals. The percentage of SR adhering to three different predetermined percentages for each checklist (i.e.
Linked assets
University of Southern California Dissertations and Theses
Conceptually similar
PDF
The quality of reporting in systematic reviews: a cross sectional survey in periodontology journals
PDF
Quality of reporting of observational studies in periodontology and implant dentistry: a cross-sectional survey
PDF
Reporting quality of randomized controlled trials of periodontal diseases in journal abstracts: a cross-sectional survey and bibliometric analysis
PDF
Assessment of the quality of reporting observational studies in periodontal literature
PDF
Reporting quality of cohort studies in periodontology before and after the STROBE publication: a trend analysis
PDF
Relationship of buccal bone plate thickness and healing of extraction sockets with or without alveolar ridge preservation: a systematic review
PDF
The effect of vertical level discrepancy of adjacent dental implants on crestal bone resorption: a retrospective radiographic analysis
PDF
Patient reported outcomes measures (PROMs) and medication count of the three coronal advancement methods for the treatment of multiple gingival recession defects
PDF
Maxillary sinus floor and alveolar crest alterations following extraction of maxillary molars: a retrospective CBCT analysis
PDF
Three-dimensional volumetric analysis of gingival augmentation for the treatment of multiple recession defects by vestibular incision subperiosteal tunnel acces (VISTA)
PDF
Efficacy of fibrin-assisted soft-tissue promotion (FASTP) in treatment of multiple gingival recession defects: a retrospective 3-D volumetric analysis
PDF
Dimensional changes in alveolar bone following extraction of maxillary molars in humans: a retrospective CBCT analysis
PDF
Healing of extraction sockets treated with anorganic bovine bone minerals: a micro-CT analysis
PDF
The performance of light emitting diode (LED) light curing units and dental radiometers
PDF
3D volumetric analysis of post extraction maxillary sinus floor changes: a retrospective CBCT analysis
PDF
Alveolar ridge dimensional changes following ridge preservation procedure: CBCT linear analysis in non-human primate model
PDF
The effect of crown dimensions & implant dimensions on peri-implant marginal bone loss: a retrospective analysis
PDF
A randomized controlled clinical trial evaluating the efficacy of grafting the facial gap at immediately placed implants in the anterior maxilla: 3D analysis of bone and soft tissue changes
PDF
Quality of life of patients with cleft lip and palate undergoing orthodontic treatment during early vs. late adolescence
PDF
The utility of bleeding on probing and 0.25% sodium hypochlorite rinse in the treatment of periodontal disease
Asset Metadata
Creator
Mohammad, Husain (author)
Core Title
The quality of reporting of systematic reviews in periodontology journals: a cross sectional survey
School
School of Dentistry
Degree
Master of Science
Degree Program
Craniofacial Biology
Publication Date
10/20/2015
Defense Date
06/17/2015
Publisher
University of Southern California
(original),
University of Southern California. Libraries
(digital)
Tag
OAI-PMH Harvest,periodontology journals,reporting quality,survey,systematic reviews
Format
application/pdf
(imt)
Language
English
Contributor
Electronically uploaded by the author
(provenance)
Advisor
Kar, Kian (
committee chair
)
Creator Email
hmmohamm@usc.edu,husain.m.h.m@gmail.com
Permanent Link (DOI)
https://doi.org/10.25549/usctheses-c40-192807
Unique identifier
UC11278950
Identifier
etd-MohammadHu-3983.pdf (filename),usctheses-c40-192807 (legacy record id)
Legacy Identifier
etd-MohammadHu-3983.pdf
Dmrecord
192807
Document Type
Thesis
Format
application/pdf (imt)
Rights
Mohammad, Husain
Type
texts
Source
University of Southern California
(contributing entity),
University of Southern California Dissertations and Theses
(collection)
Access Conditions
The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the a...
Repository Name
University of Southern California Digital Library
Repository Location
USC Digital Library, University of Southern California, University Park Campus MC 2810, 3434 South Grand Avenue, 2nd Floor, Los Angeles, California 90089-2810, USA
Tags
periodontology journals
reporting quality
systematic reviews