Results of the search

Figure 1 summarises the studies that we identified, screened and selected for this review. Our search resulted in 6782 hits, including 6571 primary studies, 128 reviews, 46 conference abstracts, and 37 registered trials. We identified 45 additional articles by checking the reference lists of retrieved reviews and by performing a 'Related articles' search in PubMed. We removed duplicates (n = 3016), leaving 3811 records for title and abstract screening. Of these, we requested 268 full‐text articles for review, of which we excluded 216 (see Figure 1 for reasons for exclusion). Fifty‐two studies met our inclusion criteria and are included in the final review.

Figure 1

---

---

PRISMA flow diagram: results of the search for studies evaluating the diagnostic accuracy of blood CEA to detect recurrent colorectal cancer in patients following curative resection.

Included studies

Prevalence

Included studies were published between 1974 and 2014 and were conducted across 22 countries. All studies were conducted in secondary care, except one Norweigian prospective study (Johnson 1985) in which follow‐up was conducted in both primary and secondary care. In total, 9717 patients were included, and 2951 recurrences detected. The median number of participants in the studies was 139 (interquartile range (IQR): 72 to 247) and the proportion of recurrences detected ranged from 13.5% (Fezoulidis 1987) to 72.3% (Ochoa‐Figueroa 2012) (median: 29.5%, IQR: 24.3 to 36.3%).

Study Design

In 24 studies (46%) a prospective design was used, three of which were randomised controlled trials (RCTs) (McCall 1994; Ohlsson 1995; Steele 1982). One study prospectively followed up a cohort of patients of whom some were identified retrospectively (Tate 1982), while another sampled retrospectively from a prospective cohort (Korner 2007). The remaining 26 studies (50%) used a retrospective design.

Clincal features of included patients

Location of recurrence

The location of recurrence was reported in 25 studies (48%) including local, locoregional, and distant recurrence. However, the description of CRC recurrence was heterogeneous and all studies lacked 2 x 2 tables for the diagnostic accuracy of CEA to detect recurrence at each location (Characteristics of included studies).

Staging of primary colorectal cancer

Apart from the two studies (4%) which included only patients with rectal cancer (Barillari 1992; Fezoulidis 1987), the majority of studies (n = 50, 96%) included patients with both colon and rectal cancer.

Thirty‐three studies (63%) used the Dukes staging to describe the primary CRC. A further 11 studies (21%) used the TNM grading system and one study (2%) used the Astler‐Coller staging. The staging was unclear or not reported in the remaining seven studies (13%) (Carlsson 1983; Kohler 1980; Koizumi 1992; Li Destri 1998; Mittal 2011; Ochoa‐Figueroa 2012; Wood 1980).

Of those using Dukes staging, seven included Dukes A ‐ D (Banaszkiewicz 2011; Carpelan‐Holmström 2004; Jubert 1978; Mach 1978; Mariani 1980; Seregni 1992; Yu 1992); 15 included Dukes A ‐ C (Barillari 1992; Deveney 1984; Farinon 1980; Fezoulidis 1987; Fucini 1987; Graffner 1985; Hine 1984; Irvine 2007; Kato 1980; Korner 2007; Luporini 1979; Mackay 1974; McCall 1994; Ohlsson 1995; Triboulet 1983); three used Dukes B ‐ C (Beart 1981; Steele 1982; Wang 1994); two used Dukes C (Hara 2008; Tobaruela 1997); one used Dukes A ‐ C plus palliative cases (Johnson 1985); one used Dukes A ‐ C plus unknown cases (Tate 1982); and four used Dukes A ‐ D plus unknown cases (Bjerkeset 1988; Engarås 2003; Miles 1995; Minton 1985).

Of the 11 studies using the TNM grading system: five included TNM I ‐ III (Kanellos 2006a; Ohtsuka 2008; Park 2009; Tang 2009; Yakabe 2010); four used TNM I ‐ IV (Carriquiry 1999; Nishida 1988; Peng 2013; Staib 2000); and one included TNM II ‐ III (Kim 2013). Only one study reported 2 x 2 tables by stage, reporting on TNM II and TNM III (Hara 2010).

The study that used Astler‐Coller staging included A ‐ C2 (Lucha 1997).

Smokers

Three studies explicitly excluded smokers (Kanellos 2006a; Mariani 1980; Staib 2000), four studies explicitly included some smokers (but there was no way of identifying these patients in the 2 x 2 tables), and the remaining studies did not report smoking status. In the two studies which gave precise figures for smoking prevalence, it was low at 2% smokers (Fucini 1987) and 9% heavy smokers (Mach 1978).

Investigations for residual disease

In 43 studies (83%) it was not clear which (if any) perioperative investigations were done to ensure there was no residual disease before entering follow‐up. In the nine studies that reported this information, three reported using a persistent postoperative elevation of CEA as evidence of residual disease (Hara 2008; Irvine 2007; Steele 1982); one used "signs" of malignancy at the first follow‐up examination (Tate 1982); one used preoperative colonoscopy to resect any lesions outside the section of bowel planned for resection (Banaszkiewicz 2011); one reported using the intraoperative detection of gross residual disease (Lucha 1997); one specified no gross residual disease and clear resection margins (Bjerkeset 1988); one used preoperative abdominal CT and interoperative palpation to exclude liver metastases (Kanellos 2006a); and one reported using preoperative barium enema (BE), chest x‐ray (CXR), liver function tests (LFTs) and CEA, and postoperative BE and colonoscopy to ensure there was no residual disease (Ohlsson 1995).

Treatment

In 14 studies (27%) some (but not all) patients received chemotherapy, and in no studies was a subgroup analysis performed comparing the diagnostic accuracy of CEA in those receiving chemotherapy compared to those who did not (Characteristics of included studies).

Reference standard

In 38 studies (73%) a composite reference standard was used, the composition of which varied greatly between studies (see Characteristics of included studies). In 12 of these, a predefined multimodal follow‐up schedule was used for each patient (although the composition of these varied across studies) (Banaszkiewicz 2011; Carlsson 1983; Fucini 1987; Hara 2008; Irvine 2007; Jubert 1978; Kanellos 2006a; McCall 1994; Ohlsson 1995; Park 2009; Peng 2013; Steele 1982). In 26 studies (50%) a predefined composite follow‐up schedule was used to trigger further investigations for suspected recurrence.

A single investigation was used in three studies (6%) (Mittal 2011; Ochoa‐Figueroa 2012; Staib 2000), of which one reported 2 x 2 tables separately for PET and for CT (Ochoa‐Figueroa 2012).

In the remaining 11 studies (21%), it was unclear what was used as a reference standard.

CEA measurement

The use of predefined follow‐up schedules resulted in multiple CEA measurements being available for analysis.

Eight studies (15%) reported the accuracy of the CEA measurement closest to the time at which recurrence was detected by the reference standard, whilst nine studies (17%) defined CEA as positive if any CEA measurement crossed the threshold at any time within the follow‐up period. In a subset of studies, the authors stated clearly that a single 'positive' measurement would be followed up by a repeat test to confirm the result.

For the remaining 35 studies (67%), it was impossible to unpick which CEA value had been used, due to limited reporting.

Reporting units

CEA studies have used both ng/mL and µg/L in their publications. Numerically these are the same value and for consistency we have used µg/L throughout the review.

Laboratory technique

Details regarding laboratory methods for CEA analysis were inconsistently reported across the included studies. Based on the available information relating to laboratory technique, we were able to able to group the studies as follows:

1. Twenty‐two studies (42%) analysed samples before the introduction of the international reference preparation (IRP) using manual RIA and IRMA methods;

2. Seven studies (13%) used an identifiable laboratory technique following introduction of IRP;

3. Eight studies (15%) used unfamiliar laboratory techniques after the introduction of IRP;

4. Fifteen studies (29%) did not report laboratory technique.

For the seven studies reporting an identifiable laboratory technique following IRP introduction, six distinct techniques were used: Autodelfia post‐year 2000 (Carpelan‐Holmström 2004; Engarås 2003); Abbott automated instrumentation (Korner 2007); Bayer Immuno 1 (Irvine 2007); Siemens ADVIA centaur (Kim 2013); Roche elecsys (Mittal 2011); and Diasorin/byk santec liaison (Staib 2000). Across these, four thresholds were reported: 3 µg/L (Staib 2000); 5 µg/L (Carpelan‐Holmström 2004; Kim 2013; Mittal 2011); 5.6 µg/L (Engarås 2003); and 10 µg/L (Irvine 2007; Korner 2007).

Forty‐three studies (83%) did not report an estimation of CEA method reproducibility nor an indication of method accuracy. Of the remaining nine studies, three (6%) reported both an estimation of reproducibility and an indication of method accuracy (Carpelan‐Holmström 2004; Engarås 2003; Steele 1982), four (8%) clearly reported only an estimation of reproducibility (Fucini 1987; Hine 1984; Mach 1978; Mackay 1974), and the remaining two (4%) reported only the indication of method accuracy (Irvine 2007; Miles 1995).

Excluded studies

Of the 216 excluded full‐text articles (Figure 1; Characteristics of excluded studies):

• 152 studies (70%) did not report complete 2 x 2 data, and 74 (34%) reported no 2 x 2 data at all: 59 (27%) only reported recurrences; 16 (7%) only reported CEA positive cases; and three (1%) only CEA negative);

• 23 studies (11%) did not conduct a single‐point diagnostic test accuracy study (14 (6%) used alternative analyses (trend, nomogram, slope, or median CEA); five were case‐control studies (2%); three (1%) were review articles; and one was an economic analysis);

• 14 studies (6%) did not report an analysis of serum CEA measurements taken as part of a follow‐up schedule (seven (5%) reported preoperative CEA measurements; six (3%) reported the prognostic value of one postoperative CEA measurement; and one used intraoperative portal vein sampling);

• eight studies (4%) included fewer than 30 patients;

• six studies were unavailable or needed translation (five studies (2%) were not retrieved after worldwide search by the British Library, and we were not able to translate the remaining study);

• five studies (2%) did not clearly report colorectal cancer recurrence (three (1%) reported on only liver metastases; and two (1%) reported colorectal cancer recurrence together with other cancer types);

• five studies (2%) reported datasets already included in the review;

• three studies (1%) reported non‐curative surgery.

We have not included two large RCTs in the review: the FACS trial (as 2 x 2 data were not reported in the published paper (Primrose 2014)), and the CEASL trial, which was published following our search and did not report on negative CEA cases (Treasure 2014).

Methodological quality of included studies

We assessed all 52 studies using the QUADAS‐2 framework. Figure 2 shows the summary of overall risk of bias and applicability concerns, and Figure 3 presents the risk of bias and applicability concerns as overall percentages.

Figure 2

---

---

QUADAS‐2 risk of bias and applicability concerns summary including review authors' judgements about each domain for each included study

Figure 3

---

---

QUADAS‐2 risk of bias and applicability concerns graph including review authors' judgements about each domain presented as percentages across included studies

Three studies (6%), including 516 participants of whom 177 experienced recurrence, were assessed as being at low risk of bias and low concern regarding applicability across all domains (Barillari 1992; Irvine 2007; McCall 1994). Across these studies, each reported a different threshold (3, 10, and 5 µg/L respectively) using CEA test kits from three different manufacturers (with poor description of method accuracy). Each study applied a different but "appropriate" follow‐up schedule to detect recurrence. Consequently, the planned subgroup analysis of high‐quality studies (low risk of bias in all four domains) was not feasible.

Risk of bias

We judged 34 studies (65%) to be at high risk of bias in at least one of the four domains (Figure 3).

For the patient selection domain, items were weighted so that the presence of inappropriate exclusions had more influence on the overall judgement than the presence of a consecutive or random sample. Of the 27 studies judged to be at high risk of bias for patient selection (52%), inappropriate exclusions were based on:

• advanced age (Carlsson 1983; Graffner 1985; Korner 2007; Ohlsson 1995);

• previous malignancy (Ohtsuka 2008);

• poor prognosis for further surgery (Banaszkiewicz 2011; Ohlsson 1995);

• preoperative CEA values (Carpelan‐Holmström 2004; Carriquiry 1999; Farinon 1980; Miles 1995; Tang 2009; Wang 1994);

• temporary rises in CEA (Mackay 1974);

• non‐rising CEA (Mittal 2011);

• using a minimum follow‐up period (Carriquiry 1999; Kim 2013; Korner 2007; Mackay 1974);

• 'incomplete' follow‐up measurements (Carpelan‐Holmström 2004; Carriquiry 1999; Hara 2008; Kato 1980; Korner 2007; Nishida 1988);

• factors related to other follow‐up tests (Fucini 1987; Peng 2013; Tang 2009; Triboulet 1983);

• early signs of malignancy or death (Carlsson 1983; Tate 1982);

• smoking status (Kanellos 2006a; Mariani 1980);

• concomitant benign disease or recent surgery (Kanellos 2006a; Mariani 1980; Park 2009; Peng 2013; Staib 2000);

• patients not presenting a “diagnostic problem” (Staib 2000).

There were no studies deemed to be at high risk of bias based on the judgements made about the index test.

There were no studies at high risk of bias based on the judgements made about the reference standard, and in 17 (33%) the risk was unclear.

Thirteen studies (25%) were deemed to be at high risk of bias based on flow and timing. In four studies, not all patients were included in the final analysis (Beart 1981; Bjerkeset 1988; Kohler 1980; Park 2009). In the remaining nine studies, a raised CEA value triggered the reference standard which could introduce work‐up bias and result in false negative CEA results being misclassified as true negative results (Lucha 1997; Mackay 1974; Mariani 1980; Miles 1995; Tang 2009; Tobaruela 1997; Triboulet 1983; Wood 1980; Yu 1992).

Applicability concerns

We judged 37 studies (71%) to be at low risk of applicability concerns in all three domains (Figure 3). We rated only one study (Ochoa‐Figueroa 2012) at high risk of applicability concerns in relation to patient selection, as it did not include all patients undergoing postoperative follow‐up, but only those referred with suspected recurrence to the Department of Nuclear Medicine for fluoro‐deoxy‐glucose (FDG) PET‐CT. There were no studies deemed to be at high risk for applicability based on the index test or reference standard.

Unclear risk

Of the 364 domains, we deemed 85 (23%) to be at unclear risk of bias or applicability. For the vast majority of these items poor reporting accounted for the unclear rating.

Findings

Diagnostic accuracy

The forest plot in Figure 4 (Analysis 1) shows the range of sensitivity and specificity of CEA for the detection of recurrent colorectal cancer across all 52 included studies.

Figure 4

---

---

Forest plot for all 52 included studies for the threshold reported closest to 5 µg/L

TP = true positive; FP = false positive; FN = false negative; TN = true negative

The blue square depicts the sensitivity and specificity for each study and the horizontal line represents the corresponding 95% confidence interval for these estimates.

For studies reporting accuracy at more than one threshold, 2 x 2 data at the threshold closest to 5 μg/L are included in the plot (5 μg/L was the most commonly reported threshold).

.Sensitivity ranged from 41% to 97% and specificity from 52% to 100%.

Figure 5 plots each of the 52 studies in ROC space. The size of each box is proportional to the inverse standard error for sensitivity and specificity for each study (a larger box indicates greater precision).

Figure 5

---

---

Scatter plot of sensitivity versus specificity for all 52 studies, regardless of threshold.

Each box represents the 2 x 2 data extracted from each study, with the width of the boxes being proportional to the inverse standard error of the specificity and the height of the boxes proportional to the inverse standard error of the sensitivity.

Effect of CEA threshold on diagnostic accuracy

Forty‐one studies (79%) reported accuracy at just a single threshold. A wide range of thresholds were reported (2 to 40 µg/L). Four studies (8%) did not report which threshold they used (Graffner 1985; Johnson 1985; Ohlsson 1995; Seregni 1992).Seven studies (13%) reported 2 x 2 data for more than one threshold:

• Banaszkiewicz 2011: 5 and 10 µg/L

• Bjerkeset 1988: 3.5 and 5 µg/L

• Carlsson 1983: 3 and 7.5 µg/L

• Korner 2007: 4 and 10 µg/L

• Mittal 2011: 3, 5, 10, 20, and 50 µg/L

• Steele 1982: 2.5, 5, 10, and 20 µg/L

• Wood 1980: 25 and 40 µg/L

The forest plots in Figure 6 (Analysis 2) show the range of sensitivity and specificity for studies reporting the accuracy of CEA at cut‐off values of 2.5, 5 and 10 µg/L.

Figure 6

---

---

Forest plot broken down by threshold: CEA at 2.5µg/L, CEA at 5µg/L, CEA at 10µg/L.

TP = true positive; FP = false positive; FN = false negative; TN = true negative

The blue square depicts the sensitivity and specificity for each study and the horizontal line represents the corresponding 95% confidence intervals for these estimates.

The summary ROC curves and the summary estimates including confidence ellipses for the threshold values of 2.5, 5, and 10 µg/L (Analyses 3, 4 and 5) can be found in Figure 7, Figure 8 and Figure 9 respectively.

Figure 7

---

---

Summary ROC plot of accuracy at a threshold of 2.5 µg/L.

Each box represents the 2 x 2 data extracted from each study. The width of the box is proportional to the number of patients who did not experience recurrence in each study, and the height is proportional to the number of patients that did develop recurrent CRC.

The filled circle is the pooled estimate for sensitivity and specificity and the line running through it is the summary ROC curve.

The smaller dotted ellipse represents the 95% credible region around the summary estimate; the larger dashed ellipse represents the 95% prediction region.

Figure 8

---

---

Summary ROC plot of accuracy at a threshold of 5 µg/L.

Each box represents the 2 x 2 data extracted from each study.

The width of the box is proportional to the number of patients who did not experience recurrence in each study, and the height is proportional to the number of patients that did develop recurrent CRC.

The filled circle is the pooled estimate for sensitivity and specificity and the line running through it is the summary ROC curve.

The smaller dotted ellipse represents the 95% credible region around the summary estimate; the larger dashed ellipse represents the 95% prediction region.

Figure 9

---

---

Summary ROC plot of accuracy at a threshold of 10 µg/L.

Each box represents the 2 x 2 data extracted from each study.

The width of the box is proportional to the number of patients who did not experience recurrence in each study, and the height is proportional to the number of patients that did develop recurrent CRC.

The filled circle is the pooled estimate for sensitivity and specificity and the line running through it is the summary ROC curve.

The smaller dotted ellipse represents the 95% credible region around the summary estimate; the larger dashed ellipse represents the 95% prediction region.

In the seven studies reporting a threshold of 2.5 µg/L, the sensitivity ranged from 65% to 91% and specificity from 34% to 98%. The pooled sensitivity of these studies was 82% (95% CI 78% to 86%) and pooled specificity 80% (95% CI 59% to 92%). Assuming that the proportion of patients with recurrence in any single testing period is 2% (based on our observed prevalence of recurrence of 30% and national guidance to conduct 14 to 15 CEA tests during follow‐up), for every 1000 patients tested at a threshold of 2.5 µg/L, 16 cases of recurrence will be detected, four cases will be missed, and there will be 196 false alarms (people referred unnecessarily for further testing). More precise estimates of test performance using the incidence data reported by Sargent 2007 can be found in summary of findings Table 2.

In the 23 studies which reported the impact of applying a threshold of 5 µg/L, sensitivity ranged from 43% to 93% and specificity from 60% to 100%. The pooled sensitivity of these studies was 71% (95% CI 64% to 76%) and pooled specificity 88% (95% CI 84% to 92%). For every 1000 patients tested at a threshold of 5 µg/L, 14 cases of recurrence will be detected, six cases will be missed, and there will be 118 false alarms. More precise estimates of test performance using the incidence data reported by Sargent 2007 can be found in summary of findings Table 3

In the seven studies reporting the impact of applying a threshold of 10 µg/L, sensitivity ranged from 41% to 87% and specificity from 88% to 100%. The pooled sensitivity of these studies was 68% (95% CI 53% to 79%) and pooled specificity 97% (95% CI 90% to 99%). For every 1000 patients tested at a threshold of 10 µg/L, 14 cases of recurrence will be detected, seven cases will be missed, and there will be 29 false alarms. More precise estimates of test performance using the incidence data reported by Sargent 2007 can be found in summary of findings Table 4.

Effect of the timing of CEA measurement

As previously described, we used two approaches when choosing which CEA measurement to include in the 2 x 2 tables. The first was to evaluate the CEA measurement taken closest to the time point at which recurrence was detected; the second was to look across all measurements to assess whether any had crossed the threshold during the entire follow‐up period.

Including only those studies reporting accuracy at a threshold of 5 µg/L, we carried out a subgroup analysis for these two strategies.

We adopted the first strategy in eight studies, for which the pooled sensitivity and specificity were 69.0% (95% CI 57.3% to 78.7%) and 90.0% (95% CI 77.8% to 95.9%) respectively. We adopted the second strategy in nine studies, for which the pooled sensitivity and specificity were 64.5% (95% CI 55.2% to 72.9%) and 89.5% (95% CI 83.4% to 93.5%) respectively.

Effect of laboratory technique

We were unable to carry out a subgroup analysis based on specific laboratory techniques, as reporting was so limited that it is was difficult to identify groups of studies where we could be confident that they had all used consistent methods.

For those studies reporting accuracy at a threshold of 5 µg/L, we carried out a subgroup analysis comparing the variability in accuracy before and after the introduction of the international reference preparation (IRP 73/601) calibration. We excluded one study (Li Destri 1998) from this analysis, as there was insufficient information about the timing of the sample analysis and laboratory technique. There were 11 studies predating the introduction of the IRP, providing a pooled sensitivity of 73.6% (95% CI 63.2% to 81.8%) and a pooled specificity of 88.5% (95% CI 83.2% to 92.2%), and 11 studies used methods which incorporated the IRP, resulting in a pooled sensitivity of 67.9% (95% CI 58.6% to 75.9%) and a pooled specificity of 88.6% (95% CI 80.0% to 93.7%). These results indicate no significant reduction in variability, and this was confirmed when we added it as a covariate in the metaregression (P = 0.958).

Effect of patient selection on diagnostic accuracy

When restricting the analyses to the 11 studies deemed to be at low risk of bias in the patient selection domain of the QUADAS‐2 assessment, the sensitivity ranged from 43% to 93% and specificity from 61% to 99%.

We added the patient selection risk of bias item as an ordinal covariate (low risk = 6, unclear risk = 6 and high risk = 11) in the metaregression analysis for those studies reporting accuracy at 5 µg/L. The effect of this covariate was not significant (P = 0.771).

Effect of index test on diagnostic accuracy

There were no studies deemed to be at high risk of bias in the index test domain of the QUADAS‐2 assessment. When restricting the analyses to the 37 studies (71%) deemed to be at low risk of bias in the index test domain of the QUADAS‐2 assessment, the sensitivity ranged from 41% to 97% and specificity from 52% to 100%.

We added the index test risk of bias item as a covariate (low risk = 15, unclear risk = 8) in the metaregression analysis for those studies reporting accuracy at 5 µg/L. The effect of this covariate was not significant (P = 0.901).

Effect of the reference standard on diagnostic accuracy

There were also no studies deemed to be at high risk of bias in the reference standard domain of the QUADAS‐2 assessment. When restricting the analyses to the 35 studies (67%) deemed to be at low risk of bias in the reference standard domain of the QUADAS‐2 assessment, the sensitivity ranged from 41% to 97% and specificity from 52% to 100%.

We added the reference standard risk of bias item as a covariate (low risk = 17, unclear risk = 6) in the metaregression analysis for those studies reporting accuracy at 5 µg/L. The effect of this covariate was not significant (P = 0.292).

Effect of flow and timing on diagnostic accuracy

When restricting the analyses to the 25 studies (48%) deemed to be at low risk of bias in the flow and timing domain of the QUADAS‐2 assessment, the sensitivity ranged from 41% to 95% and specificity from 52% to 100%.

We added the flow and timing risk of bias item as an ordinal covariate (low risk = 12, unclear risk = 6 and high risk = 5) in the metaregression analysis for those studies reporting accuracy at 5 µg/L. The effect of this covariate was not significant (P = 0.664).