Search strategy and data sources

We re-used the search strategy employed in our prior published systematic review on ALF definitions [13]. Briefly, Ovid Embase(R) (1980 to 2012), Ovid MEDLINE(R) (1950 to 2012) and Ovid MEDLINE(R) In-Process & Other Non-Indexed Citations (1950 to 2012) for journal articles were searched based on keywords in title, abstract and MeSH terms. The following query was used: (prognosis OR prognostic OR predict*) AND (acute liver failure OR fulminant hepatic failure OR acute liver injury OR acute hepatic failure OR (acute on chronic AND liver failure)). “Liver failure” and “prognosis” were used as MeSh terms. The final search considered studies published up to 01 January 2012.

Inclusion and exclusion criteria

We included articles only when they reported original data from a clinical trial or observational study on patients with ALF and if one of their main objectives was either developing one or more new prognostic models or modifying existing ones for predicting outcome (mortality/survival or LT) for ALF patients. Validation studies (defined as studies that validated the performance of earlier published models, without modification, on new data sets) assessing the performance of the established prediction models KCC, MELD, and Clichy criteria were excluded.

All duplicate articles resulting from the query above were removed and only English articles were considered. In the first step we excluded conference abstracts, paper reviews, comments and case studies. In the next step irrelevant studies were excluded based on titles and abstracts, followed by exclusion of the remaining studies based on their full text.

Two reviewers independently screened the titles and abstracts. Discrepancies between the 2 reviewers were resolved by consensus involving a 3rd reviewer. Figure S1 displays the search flowchart.

Data collection and analysis

The studies that were included were classified as either developing a new model or modifying an existing model. A study developing a new model was termed a development study, in that it described and assessed the performance of a prediction model that had not previously been published. Development studies had to include at least one newly developed model, but may also have included other established existing models, such as the KCC, MELD, Clichy criteria, or any other more recently developed model for comparison. A study was classified as modifying an existing model if it described and assessed a modified version of a previously published model by, for example, adding or removing variables.

For each of the included studies, the general study characteristics (e.g. setting, study year, inclusion criteria, outcome and patients' characteristics) and model characteristics (e.g. the reported intended use of the prognostic model, technique(s) used for development and performance and validation of the model) were extracted. The quality of the study was reported using a structured data collection form as proposed by Medlock et al. [14]. As before, two reviewers extracted and summarized data and scored the methodological and reporting quality. Discrepancies between the reviewers were resolved by involving a 3rd reviewer.

Prediction models

We distinguished two types of prediction models: regression models (e.g. survival Cox model, linear or logistic model); or decision models (e.g. a decision tree, discriminant analysis, or a decision rule based on a score variable). For the description of the models we recorded: the timing of variable measurement (e.g. at admission or peak value during some interval), type of prediction model, strategy of model development and the final model itself in the form of a prognostic formula or a decision rule.

Variables

All input variables (potential predictors or confounders that were considered during model development), also known as covariates, were listed for each prediction model. We also report on the final input variables that remained in the final model after employing a variable elimination strategy.

Prognostic performance measures

The performance measures were divided into the following 4 categories:

1. Statistical measures of performance based on a given cut-off point that result in a binary prediction (0 or 1): sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), predictive accuracy (PA), positive likelihood ratio (PLR) and negative likelihood ratio (NLR).
2. Discrimination measures assessing the ability of the model to assign higher predicted probabilities of the outcome (e.g. death or LT) in patients actually having the outcome than those not having it. The most common measure is the AUC (Area Under the Receiver Operating Characteristic (ROC) curve).
3. Calibration measures assessing the proximity between the predicted probabilities and the actual risk of a group of similar patients. For example, the model was considered to be well calibrated when 25% of patients with a predicted mortality of 25% did indeed have the event. Common measures of (mis)calibration are the Hosmer-Lemeshow statistics.
4. Accuracy measures assessing the average proximity of a predicted probability for an individual patient to his/her actual outcome. These include elements of both discrimination and calibration [15] and are typically measured by the Brier score or Brier skill score (which is the proportion of explained variance).

We required that a study report on at least 2 of the above-mentioned performance categories in order to score positively in the quality assessment framework on model performance.

Validation pertains to assessing the statistical performance of a model. In order for prognostic models to be used in clinical practice they need to be credible, and validation is an important component for reinforcing model credibility. Apparent validation (estimating performance on the same sample from which the model was developed) may be biased due to leveraging on coincidental correlations in the development sample. For studies developing new models it is imperative to at least internally validate them. Internal validation means estimating the performance that would be obtained from a different sample but from the same population of the developmental dataset. This can be obtained by either estimating performance on a separate reserved subsample from the developmental sample or by resampling techniques from the developmental set using bootstrap or cross-validation techniques. Validation performed on a separate prospective dataset from the same setting is called temporal validation, and validation on a dataset in a different setting is called external validation [16].

Quality assessment

The framework proposed by Medlock et al. [14] which was in part based on Minne et al. [17] and Hayden et al. [18] was used for assessment of the quality of content of the studies and the assessment of the performance of the models.

The framework has two sections: a reporting quality section for assessing the quality of reporting in the study, and a methodological section consisting of a number of questions assessing the rigor of model development and validation.

The reporting score consisted of 19 items (38 possible points) including description of study population, choice of predictors/variables to test, missing data, outcome measure and model description including its intended use and performance measures.

The methodological score consisted of 6 items (12 possible points) including a sufficient number of cases to support the number of variables, a representative population and validation.

Each item in the framework was rated as no (N, with 0 points), partly (P, with 1 point), or yes (Y, with 2 points). For example, the setting and study period should be reported to score 2 points and when only one is reported it scores 1 point.

The higher the score in each part of the framework was considered as higher quality of the study in that part.

Study characteristics

Searching the online databases resulted in 1233 articles. Initial screening of titles and abstracts resulted in 125 articles for full text review, of which 20 articles met our inclusion criteria and were included in this review.

Detailed characteristics of these studies are presented in Table S1. The smallest and largest study included 23 and 588 patients, respectively. Mean (SD) and median [IQR] of sample size was 159 (164) and 99 [60–171], respectively.

In one study, gender was not reported (17% of patients), and 59% of patients were female among the remaining studies.

Five studies were conducted prospectively, 6 studies retrospectively and in 9 studies the data collection method was not reported.

Model characteristics

Variables.

Table S2 in the supplement lists 103 prognostic indicators (input variables) that were considered as potential predictors in the developed models. The table also includes the cut-off values for obtaining categorical indicators from continuous ones, which varied among studies; otherwise, the indicator was used as a continuous variable. Table 1 lists the indicators considered in more than ten studies in addition to the number of studies in which the indicator was categorized. The most commonly used indicator was bilirubin (16 times), which was selected in 8 final models. Twenty-three of 103 variables were use as categorical variables. The most commonly categorized variable was prothrombin time (PT, 10 times).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Summary of the most often studied indicators (n = number of studies).

https://doi.org/10.1371/journal.pone.0050952.t001

Principal findings

In this review we identified, summarized and assessed the quality of available models in the literature for prediction of poor outcome in adult patients with ALF. There is a marked heterogeneity in the included studies and models (shown in Table S2 in the supplement) in terms of: variability of the characteristics of included populations, inclusion criteria, mortality rates, outcome (inclusion or exclusion of transplanted patients), considered predictors, and choice of reference models for comparison. Of note, 45% of the studies did not report whether data were collected prospectively or retrospectively. Although all studies aimed at including ALF patients, the definition of this disease differs among studies [13]. Despite this heterogeneity some general remarks can be made.

Model development usually relied on regression analysis, including logistic, linear or Cox regression (survival analysis). The models were usually constructed from clinical and/or demographic data to predict mortality or survival, only four studies used the combined outcome of death and transplantation. Generally the intended use of the developed models was clear, namely supporting decisions on whether to perform transplantation. However, half of the studies included small samples (<100 patients) and performed no internal, temporal or external validation. None of the studies reported on how well the model was calibrated. This is notable, as decision makers need to know how well a predicted probability corresponds to the true risk in the population. All studies comparing newly developed models with the “standard” models like MELD and/or KCC, SOFA and SAPS reported improved performance on these established models based on sensitivity, specificity, PPV, NPV, PA, PLR, and NLR. However, only 3 studies used additionally the AUC, and there are no calibration-related comparisons. Surprisingly the commonly used Clichy criteria were not used as a reference standard in the included studies. In general the apparent arbitrariness of selecting the reference models raises concerns about reporting bias. It will be useful when future studies attempt to compare the developed models to a standard set of other published reference models, including newly developed ones, for the same population.

Strengths and limitations

To our knowledge this is the first systematic review exclusively dedicated to the assessment of the performance of newly developed prognostic models for adult patients with ALF. In our former review [13] we reported that there is a wide diversity in ALF definitions used in the literature, which hinders comparability and quantitative analysis among studies. In this review we identified and characterized newly developed prognostic models of mortality for ALF patients and assessed the quality of their respective studies. Our search has been extensive and we used an earlier published framework for quality assessment [14].

In the systematic review by Craig et al. [31] 14 studies were included which test the association of variables with poor outcome in paracetamol-induced ALF patients. The quality of these studies were assessed semi-quantitatively using a coarse grading system (poor, moderate, good or excellent) along six potential sources of bias in prognostic studies [18]. Another recent meta-analysis by McPhail et al. [10] assessed the quality of studies validating KCC and restricted to acetaminophen-induced ALF. These studies are excluded in our review. Our review included studies pertaining to ALF patients independently of aetiology; it extends the quality assessment framework in [12] and distinguishes between reporting and methodological quality; it includes only prognostic models, not merely tests of pre-selected variables; and it considers newly developed models rather than validation of existing ones.

We intentionally excluded external validation studies like [1] and [4] since these would correspond to the already established models (MELD, KCC, Clichy Criteria) and there are already existing reviews on those models such as [10], [11], [12], [32]. In our review we did include the studies on the development of KCC [27] and Clichy Criteria [20] for ALF patients. There is no journal paper describing the development of the MELD for ALF patients. MELD has been primary designed to estimate short-term post-procedural mortality of cirrhotic patients undergoing transjugular intrahepatic portosystemic shunt [33], and later in patients with end-stage chronic liver disease of diverse aetiology and severity [34]. In 2002 MELD was implemented in the USA by the United Network for Organ Sharing for organ allocation in patients with chronic liver disease awaiting LT [21]. In 2003 in a conference abstract Aydin et al. [35] suggested that MELD can be used as a complementary tool to predict prognosis in ALF patients. MELD received increasingly more attention and was applied as predictor for ALF patients [21]. We hence consider this latter paper as the first one officially dedicated to ALF patients and included in our review.

Implications and recommendations

In order to be clinically useful, predictive models need to be credible. This credibility is largely dependent on the model validity. As reported by Cook [15] evaluation of models for medical use should take the purpose of the model into account. Evaluation of prognostic models should not be confined to only ROC curve analysis, but should assess various relevant performance measure covering at least both discrimination and calibration.

Validation of a model is necessary to provide evidence of its potential to accurately predict outcomes especially at the individual patient level. As reported by Altman et al. [36] un-validated models should not be used in clinical practice. However, our review revealed that the great majority of models have not even been internally validated nor has their calibration been assessed. Surprisingly, only four studies [37], [24], [27], [25] performed some form of validation of their models. Future studies should provide calibration performance assessment (using e.g. the Hosmer-Lemeshow goodness-of-fit statistics or, even better, the Brier score) and should undergo adequate internal validation, a good choice would be the use of bootstrap techniques to this end. Moreover, in an additional search we found than none of these models has been externally validated elsewhere.

Many studies (e.g. [29], [24]) have considered the transplanted patient group and non-survivors in one group. Patients who receive transplantation may consist of the most severe cases and would have died without transplantation. Those transplanted patients are probably similar to those who do not survive. A separate analysis should be performed to compare the transplantation patient characteristics to the non-survivor group, like in the study of Dabos et al. [38]. When the groups are similar one can consider forming the group with the combined outcome “transplanted or died,” at least for sensitivity analysis.

In the literature different kinds of models are applied, like logistic regression or Cox regression. One should consider the aim of the study when developing a model. Logistic regression is appropriate when the aim of the study is prediction of an event (mortality or survival) without regard to how long it took for the event to occur. When the aim of the study is to predict the time until event occurrence then the Cox regression is an appropriate choice.

Most of the studies did not report on missing data. Only five studies [3], [27], [37], [38], [39] reported how they handled the missing values. Excluding cases with missing values reduces the sample size and can bias the results. One should compare cases with missing and non-missing values on other known variables to check for bias. In addition (multiple) imputation of the missing variables should be considered [40].

There was marked heterogeneity in the included variables. The majority of the evaluated variables where used only once (65/99). The reason for choosing the initial variables was often not clear. Only four studies [3], [20], [28], [29] stated this reason. Selection of the variables entered into the multivariate analysis was mostly based on the significant results from univariate analysis. When the number of variables is not excessive one should consider whether variable selection is required, and if so, consider using an information criterion (such as the Akaike Information Criterion) in the (e.g. stepwise) selection process instead of relying on p-values ≤0.05 in univariate analysis.

Many studies did not report why and how the continuous variables were categorized. Categorization causes information loss and has consequently less precise coefficient estimation and reduced statistical power to detect an association between the variable and patient outcome. For example, in the study of Tylor et al. [19] when creatinine was considered as a continuous variable it was significantly associated with mortality in univariate analysis, but not when it was used as a categorical variable. Categorization of continuous variables without compelling reasons should hence be avoided. If categorization is unavoidable then the choice for the cut-off points should be motivated.

Because many variables indicate categories, or are represented as such, most models will have only a small set of different predicted probabilities. For example when using 2 binary variables in the model there are at most 4 discrete different predicted probabilities and no continuous range. Continuity in the predicted probability would allow models to exhibit smooth behaviour in which small changes in the covariate values are reflected by small changes in the outcome. This leads to better distinctions among patients.

In our systematic review we extracted variables which were used for constructing the models (Table S2). These variables comprise a huge number of possible predictors. Based on clinical experience and from a theoretical point of view we would suggest to include variables involved in the pathophysiology of ALF as well as age and grade of HE. Specifically the following variables may be considered: plasma ammonia, which rises as a consequence of impaired hepatic urea synthesis and contributes to the development of HE; plasma bilirubine, which increases as a consequence of impaired biliary excretion and coagulopathy as a consequence of decreased protein synthesis, especially of clotting factors, expressed by INR. We also think that some aspect of impaired capacity of the ALF patient to maintain metabolic homeostasis should be considered. In this respect plasma lactate might be relevant. In addition the biomarkers of the inflammatory response of ALF might be considered: e.g. plasma ratio of IL6/IL10.

Due to the methodological and reporting quality limitations generally encountered in the included studies we would recommend to develop new models that consider most important relevant variables and follow the methodological and reporting recommendations presented in this study. The predictive performance (and hopefully clinical value) of such models will need to be tested in prospective large cohorts of ALF patients with different etiologies.

In our sub-analysis we concluded that there was no relevant association between any of impact factor and citations with any of the quality of methodological and reporting score but that there was a clear positive association between year of publication and reporting score.

Conclusion

This systematic review provides an overview of models for prediction of poor outcome in patients with acute liver failure. These prognostic models were developed to support clinicians' decisions, but they should be improved before being clinically useful. Future studies could be improved by paying more attention to (internal) validation, the inclusion of model calibration aspects, better consideration of the transplantation patient group, better reporting and handling of missing data, use of absolute risk measures, explicit considerations for considering and selecting predictors, the use of a more extensive set of reference models, and the inclusion of continuous variables without categorizing them, as well as clear reporting on the study design. It is hoped that the results of this review can be useful for developers of future prognostic models for ALF patients.