Search strategy

Multiple electronic database searches were undertaken by a qualified information specialist from inception to January 2016 including searches in Medline, Embase, Web of Science and Cochrane Database of Systematic Reviews with supplementary searches of other online resources. The search combined subject headings and free text words including terms for gastroenteritis and multiplex polymerase chain reaction. Full searches are available in S1 File. Reference lists of all reviews and included studies were screened and trial websites were searched for ongoing studies. Authors were contacted to seek clarification on study populations when necessary.

Study eligibility criteria

Studies of adults and children with suspected gastroenteritis comparing GPP tests with comprehensive coverage of bacteria, viruses and parasites with standard microbiology techniques reporting test performance, patient management, clinical and patient reported outcomes were included. The setting considered was clinical laboratories receiving samples from primary and secondary care. Eligible study designs followed a hierarchy of best available evidence with the most desirable being 1) test-treat trials comparing clinically relevant outcomes (e.g., morbidity, mortality, length of stay and length of isolation) for patients randomised to either conventional testing or GPP. This was followed by 2) clinical diagnostic test accuracy studies that compare the index tests (GPP) and the comparator (standard microbiology methods) to an adequate reference standard, 3) studies that compare discrepant results between the index tests (GPP) and the comparator (standard microbiology methods) using an unbiased umpire test [2], 4) studies of agreement and disagreement between the index tests (GPP) and the comparator without using an unbiased umpire test and 5) studies of head to head comparisons of different index tests (GPP) reporting agreement of tests. Only studies that reported sufficient raw data to calculate positive and negative agreement by pathogen were considered.

Studies were excluded if they considered partial tests with coverage of less than the three groups of pathogens and if no positive and negative agreement by pathogen could be determined. Additionally, reviews, biological studies, case reports, editorials and opinions, poster presentations without supporting abstracts, non-English language reports, and meeting abstracts without sufficient numerical detail on test performance per pathogen were excluded.

Study selection

Two reviewers independently screened the titles and abstracts of all records identified by the searches. Full texts of all studies deemed potentially relevant were obtained and two reviewers independently assessed these for inclusion. Discrepancies at both stages were resolved through discussion.

Data extraction

Test results for GPP and standard microbiology methods were extracted at the pathogen level into two-by-two contingency tables following the format in Table 1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Contingency table of test agreement.

https://doi.org/10.1371/journal.pone.0173196.t001

Assessment of risk of bias and applicability

Quality assessment of included studies was undertaken by two independent reviewers and used tailored QUADAS-2 [4]. Quality assessment assessed the risk of bias and applicability concerns for included studies at the pathogen level where the GPP method was the index test, conventional methods were the comparator and any efforts to verify discordant results were assessed under the reference standard domain. The main adaptation of the QUADAS-2 tool consisted of the addition of a domain for the comparator. The eligible studies compared GPP testing to a comparator which consisted broadly speaking of a range of standard microbiology tests which cannot be classed as the reference standard because GPP testing may be superior to standard microbiology methods. Therefore, the comparator was assessed in addition to the index test and the reference standard. Similar signalling questions in terms of blinding and threshold as for the index test were considered for the comparator. Furthermore, we added signalling questions to the ‘reference standard’ as well as the ‘flow and timing’ domain requiring for a low risk of bias judgment, that the verification methods used in the studies were independent and unbiased, all discordant results rather than a proportion had received verification and that all samples had received the comparator methods for all pathogens considered in the study.

Data synthesis

In the absence of a reference standard we calculated positive agreement (a/a+c) and negative agreement (d/b+d) when benchmarked against the comparator (mainly standard microbiology methods) and positive agreement (a/a+b) and negative agreement (d/c+d) when benchmarked against GPP for each pathogen using methods outlined in the Statistical Guidance on Reporting Results from Studies Evaluating Diagnostic Tests by the FDA [3]. This is equivalent to measuring the sensitivity and specificity of GPP using standard microbiology methods as the reference standard, and the sensitivity and specificity of standard microbiology methods using GPP as the reference standard. Positive and negative agreements, using the two benchmarks, were then meta-analysed by pathogen if the denominator was ≥20 using random effects meta-analysis of proportions using the metaprop command in Stata SE 14.1 [5] and reported in tables and Forest plots. Methods for bivariate analysis of diagnostic tests findings were not used because of restrictive requirements for the number of studies within each pathogen (minimum of 4 studies with complete data). Exact binomial methods were used to estimate 95% confidence intervals using the Freeman-Tukey transformation of proportions, and the I² statistic of between study heterogeneity was computed. Data verifying discordant results was tabulated to explore the option of discrepant analysis using a suitable resolving test [6].

Study characteristics

Included studies only represented study designs 4 and 5 described in the methods section. Studies were heterogeneous in terms of participants included (hospital versus community, risk, comorbidities), country of origin (developing versus developed), standard microbiology methods used and number and type of pathogens considered (see S1 Table for study characteristics). Of the 10 studies eight evaluated the xTAG GPP test, one study evaluated the FilmArray GPP test and a further study evaluated both tests.

Risk of bias and concerns regarding applicability of study findings

The risk of bias of the included studies of test accuracy was generally high (S2 Table). None of the studies used a reference standard against which the GPP tests and standard microbiology methods could be reliably evaluated. Instead, in most studies, the GPP tests were compared against the standard microbiology methods, biasing the assessment. Discrepant results between GPP test and standard microbiology methods were verified at the pathogen level in 4/10 studies, although confirmatory tests were not adequately independent of GPP and/or standard microbiology tests. In many cases, the standard microbiology methods were not performed for all pathogens covered by the GPP test. There were concerns about the applicability and relevance of standard microbiology methods and verification tests used in the majority of studies, in reference to routine clinical practice.

Test performance

Pooled estimates of positive and negative agreement by pathogen and overall for all pathogens between GPP and standard microbiology methods are given in Table 2 (standard microbiology methods provide the benchmark) and Table 3 (GPP test provides the benchmark). Contingency tables of the raw data of test agreement by pathogen informing the meta-analysis are provided in S3 Table. Overall, more studies contributed to the calculation of negative than positive agreement as only studies with sufficient numbers (denominator ≥20) were included in the analysis and the presence of pathogens was a rare event. For a number of pathogens, E. coli O157, ETEC,STEC, Vibrio cholera, Yersinia enterocolitica and Entamoeba histolytica (rare pathogens or no test requested by physician and marked as empty rows in the Tables 2 and 3) limited data were available and no positive agreement could be estimated.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Positive and negative agreement: xTAG vs. standard microbiology methods (benchmark).

https://doi.org/10.1371/journal.pone.0173196.t002

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Positive and negative agreement: Standard microbiology methods vs. xTAG (Benchmark).

https://doi.org/10.1371/journal.pone.0173196.t003

Standard microbiology methods providing the benchmark.

Meta-analysis showed that when standard microbiology methods provided the benchmark, virtually all positive cases found by xTAG were confirmed by conventional testing leading to high levels of positive agreement findings (0.93 [95% CI 0.90 to 0.96]) (Table 2). Additional positives identified by xTAG were few compared to the vast majority of specimens that are pathogen-negative, thus negative agreement remained high (0.98 [95% CI 0.98 to 0.99]). Although overall findings were indicative, they nonetheless feature an equal weighting of pathogen-level findings not reflecting the prevalence of individual pathogens.

There was generally little variation between pathogens for both positive and negative agreement. Positive agreement for adenovirus was an exception where positive agreement was considerably lower at 0.56. This is visualised in the Forest plot in Fig 2. Gu et al. 2015 [13] reported that an additional 20 samples positive for adenovirus detected by comparator were due to the use of multiplex PCR, which detected all serotypes while xTAG only detected adenovirus 40/41 resulting in the poor agreement of tests for this virus. The outlying finding for Salmonella (Pankhurst et al., 2014 [16]), caused by a high number of missed Salmonella infections by xTAG, could not be explained.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Positive agreement: xTAG vs. conventional testing (benchmark).

https://doi.org/10.1371/journal.pone.0173196.g002

Generally, both tests agreed about the absence of pathogens, masking the relatively small number of disagreements. The Forest plot shows, however, that there were a few outliers where studies report a significantly higher number of positives for certain pathogens with xTAG compared to standard microbiology methods, specifically Campylobacter and norovirus in a small study of 49 adult kidney transplant recipients [9] and Salmonella in a study where bacteria were tested by PCR as well as culture.[11] (Figure A in S1 Figs)

Used as a measure of overall heterogeneity of estimates, I² is moderate (for positive agreement) to high (for negative agreement) at the pathogen level.

In summary, using standard microbiology methods as benchmark the impression was that GPP testing and standard microbiology methods provide very similar results.

GPP testing providing the benchmark.

Levels of agreement when xTAG provided the benchmark are shown in Table 3.The positive agreement between xTAG and conventional methods was considerably reduced when xTAG provided the benchmark. The Forest plot in Fig 3 visualises the inconsistency and variation in positive agreement between studies across all pathogens. When xTAG was the benchmark, the positive cases ‘missed’ by standard microbiology methods had a considerable impact on the positive agreement findings.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Positive agreement: Conventional testing vs. xTAG (Benchmark).

https://doi.org/10.1371/journal.pone.0173196.g003

Since the overall positive agreement is 0.68 (95% CI: 0.58 to 0.77), inverting these figures means that xTAG finds about 1.5 times more positive results (95% CI: 1.3 to 1.7). Negative agreement was consistently very high across studies and pathogens (Figure B in S1 Figs) with the exception of the Gu et al. (2015) study [13] discussed previously. Heterogeneity was moderate to high when considering I² and was higher for positive agreement than for negative agreement. Using the GPP test as the benchmark, it becomes clear GPP testing detects significantly more pathogens.

Only two studies [7 13] contributed data to the meta-analytic evaluation of the FilmArray GPP test with detailed outcomes reported in a recent HTA report by the same authors. Qualitatively these findings were similar to the findings reported here for the xTAG GPP test.

Verification of discordant results

In the absence of a reference standard to verify test results, four studies [7 10 12 16] verified discordant pathogen findings by pathogen that did not agree when tested by GPP and conventional methods (S4 Table). Verification methods were PCR based. Even though verification methods differed from standard microbiology methods and used different molecular targets when compared to GPP assays, they could not be considered independent. GPP assays essentially use PCR and other PCR-based methods would be expected to resolve discordant results in their favour. A further complexity was that sometimes conventional methods included some use of PCR. Discordant analysis of GPP positive/ standard microbiology methods negative generally favoured GPP as anticipated, however analysis of GPP negative/ standard microbiology method outcomes more often favoured standard microbiology methods (S4 Table). Discordant analysis failed to resolve all discordant samples; for a considerable number of discordant results, discrepant analysis did not help to identify the underlying cause of the discrepancy. No particular pattern for any specific pathogen emerged from the discordant analyses.

Principal study findings

Our meta-analysis of ten primary studies comparing GPP testing with standard microbiology methods reports the range of possible outcomes of positive agreement when each method in turn was considered the benchmark. Positive agreement ranged from 0.93 (95% CI 0.90 to 0.96) when conventional methods provided the benchmark to 0.68 (95% CI: 0.58 to 0.77) when GPP provided the benchmark while negative agreement was consistently high. This was due to the large number of negative results agreed upon by both methods. No previous systematic review of GPP tests was identified in our review, but assuming the use of conventional testing as a reference standard would mask issues with the use of GPP tests. In particular, gastrointestinal pathogen panel tests generate significantly more additional positive results. These, however, are of uncertain clinical importance in the absence of an appropriate reference standard or a suitable resolver test for discrepant analysis. On the other hand high positive agreement when conventional methods provided the benchmark could not be consistently shown by all studies for all pathogens suggesting that GPP performance differs for different pathogens. Pankhurst et al. (2014) [16] reported a great number of additional positives with conventional laboratory methods for Salmonella spp. which were missed with molecular based tests. Poor detection of some pathogens in some studies requires further investigation and assessment of which assays should be reported within a GPP.

Strengths and weaknesses

In the absence of more robust methods of assessing test accuracy, positive and negative agreements have been produced against a benchmark, as recommended by FDA guidance. [3] The meta-analytic outcomes reported here are of exploratory nature summarising the available evidence, illustrating patterns in the data and describing heterogeneity. Test agreement is not a measure of test accuracy as it neither considers either approach as the ‘truth’ nor does it consider if both tests while agreeing are actually wrong. Rather, by varying the benchmark, different views about the level of agreement between tests are explored. Findings were typically heterogeneous, probably reflecting in part methodological and statistical heterogeneity and drawing from studies of variable quality. High levels of heterogeneity for negative agreement were partly driven by variations in large and very precise study estimates. Additionally, a high I² was possibly caused by differences in thresholds used across studies. We report I² as a statistical measure of between study heterogeneity which is not often used in test accuracy studies because it does not account for heterogeneity due to a threshold effect. As the studies included a number of different standard microbiology tests and did not report thresholds it is difficult to judge to what extent a threshold effect existed. Therefore the finding of high levels of heterogeneity is of interest and we have listed a number of reasons that could explain this heterogeneity but cannot exclude that differences in threshold caused at least some of this heterogeneity.

The presence of very small sample sizes and heterogeneity creates particular methodological problems for meta-analyses of diagnostic test performance. A textbook approach might include all studies (large and small) regardless of patient numbers and assess test accuracy using bivariate methods. However in the present analysis, inclusion of very small studies (<20) has a dramatic impact upon sparsely informed random effect models. Bivariate analyses are balanced, including studies contributing to sensitivity and specificity estimates of test accuracy. Additionally available routines require a minimum of four pathogen studies to work. Use of the bivariate approach would exclude a large number of informative negative agreement studies from the analyses presented in this paper. Pragmatically, the greater inclusivity of univariate analysis of agreement values was preferred to the theoretical correctness of bivariate analysis.

Analyses are presented at the pathogen level requiring independence assumptions both within and between pathogens, i.e. repeat samples of the same patient are not included and having one pathogen does not affect the likelihood of having another pathogen. Pooled summary estimates (across pathogens) have been included for information. However, these pooled estimates are not weighted by the prevalence of the different pathogens, and include varying multiple usage of samples where studies have tested samples with varying components of the conventional panel of tests, thus violating the independence assumption. Accepting these limitations and issues the summary findings remain qualitatively informative.

Within the clinical studies identified, many pathogens were present only at low prevalence, and the context of studies included a mixture of different patient populations (e.g. children, immunocompromised patients, community) each with their own distribution of prevalence of pathogens. This was not considered in sensitivity analyses due to the number of potential covariates (not always well quantified) and issues of multiplicity.

This review has evaluated GPP systems according to their current specification, but it is anticipated the coverage of these systems will continue to evolve in response to changing pathogen prevalence, hence the evaluation problem is a dynamic one.

Meaning of a positive test outcome: Implications for clinicians and policymakers

If conventional methods accurately identify clinically important disease then GPP testing would correctly identify the same positive cases but add further false positive patients who may receive unnecessary treatment and potentially a delayed return to normal activities. However, if GPP testing is accurate (all of its positives are clinically important) then current testing misses clinically important pathogens, potentially resulting in under-treatment and impaired infection control measures. Expertise is important when identifying parasites by standard microbiology methods, where detection is dependent upon the life-cycle of the parasite and its appearance in the inhomogeneous stool sample. These difficulties may be overcome by using GPP testing. The consequence for clinical care is complicated since most infections are self-limiting and require no pathogen specific treatment, just hygiene, hydration and watchful waiting. Only for a select few pathogens is specific treatment recommended (C. difficile, some strains of Salmonella, Shigella, E. coli (non-STEC), Campylobacter and Giardia) although not all patients are treated. By reducing reporting from three to one days, GPP testing has the potential to streamline the management of non-infectious cases and use of hospital isolation rooms, intended to reduce the spread of infection. Currently there is no robust evidence to support changes in hospital care due to GPP testing and their plausibility is uncertain. For example, length of stay in hospital and use of isolation rooms may be primarily driven by comorbidity and hygiene rather than identification of infectious agents.

Currently the clinical importance of the additional pathogens identified by GPP testing is uncertain and there is concern that at least some of the additional findings are non-viable pathogens. GPP tests target microbial DNA and RNA. This will result in challenges associated with the interpretation of GPP test results in clinical practice, a concern shared by other authors [17]. Firstly, GPP tests cannot distinguish between viable and non-viable pathogens. The detection of microbial nucleic acid does not necessarily imply the presence of viable, replicating organisms responsible for disease. Secondly, many pathogens can exist asymptomatically (e.g., norovirus and Salmonella spp) or sub-clinically (e.g., C. difficile nontoxigenic strains) in a colonization-like status [18–20] where association with disease is unlikely. It is believed that this in part explains the increased number of positive results as well as the increased findings of co-infections with GPP testing. Further understanding is needed as to how these results should be interpreted. However the inclusiveness of panel tests may provide a gain for users, e.g. Enteroaggregative E. coli (EAEC) is not detected by routine culture.

In addition to providing pathogen-level analysis, studies typically reported overall levels of detection of pathogens with a GPP test compared to a battery of standard microbiology methods. Reporting the total number of pathogens detected may mislead by confounding greater GPP ‘sensitivity’ to detect specific pathogens with different coverage of pathogens by GPP and conventional methods. Concordance of methods depends on including a common number and type of pathogens tested. Thus discrepancies between GPP tests and conventional methods may result not only from differences in accuracy but also from differences in their respective targets. For example, discrepancies reported by Gu et al. (2015) [13] for adenovirus may have been due to comparator PCR identifying all adenovirus strains while the GPP system only identified adenovirus 40 and 41. Similarly Pankhurst et al. (2014) [16] reported that xTAG has two targets for C. difficile (genes for toxins A and B) and includes two primers against norovirus GI and GII strains. However, the single quantitative PCRs used as comparators in the study targeted only the gene for toxin B or the GII strain. This misalignment problem may be exacerbated when comparing different GPP systems with their varying coverage.

An additional limitation of GPP tests is that although the presence of bacterial pathogens is identified there is no bacterial culture to support either antimicrobial susceptibility testing or subtyping to support public health surveillance. Culturing from positive samples may therefore be required to guide antimicrobial treatment or public health investigation when these are required.

Future research needs

Agreement measures are not measures of test performance in a conventional sense, and only adequately designed research will resolve uncertainties about the introduction of GPP testing. It may not be possible to design a study with an adequate independent reference standard. Molecular methods may not be the best option to address the problem, as the presence of pathogen DNA may not answer the question of the clinical importance of the identified pathogens. On the other hand there is widespread belief that conventional microbiology laboratory methodologies are going to be outperformed by new PCR based technology. A randomised controlled trial, randomising patients to conventional or GPP testing, would establish the relative clinical and cost-effectiveness outcomes of the different approaches.