Virtual reality exposure therapy (VRET) for social anxiety disorder (SAD) is a new technology-based form of exposure therapy that uses computer-generated virtual social environments as a means of systematically exposing patients to feared stimuli. An advantage of VRET relative to exposure in vivo is that virtual exposure takes place in a highly controllable environment. Recently, Horigome et al. (Reference Horigome, Kurokawa, Sawada, Kudo, Shiga, Mimura and Kishimoto2020) published a meta-analysis on the efficacy of VRET for SAD and reported that 22 studies [11 randomized controlled trials (RCTs) and 11 non-randomized studies] fulfilled their inclusion criteria. We have previously systematically reviewed the literature on the efficacy of VRET for SAD (Carl et al., Reference Carl, Stein, Levihn-Coon, Pogue, Rothbaum, Emmelkamp and Powers2019; Emmelkamp, Meyerbröker, & Morina, Reference Emmelkamp, Meyerbröker and Morina2020; Kampmann, Emmelkamp, & Morina, Reference Kampmann, Emmelkamp and Morina2016b) and were surprised to read that a total 22 trials have investigated the efficacy of VRET for SAD. Therefore, we investigated with great interest potential reasons for the discrepancy in the number of RCTs in the review by Horigome et al. and our reviews.

Horigome and colleagues reported the following inclusion criteria for their meta-analyses: (1) a diagnosis of SAD, fear of public speaking, or public speaking anxiety; (2) VRET consisted of at least three sessions; and (3) study participants numbered a minimum of 10 patients. We critically evaluated the studies in the meta-analysis by Horigome et al. by applying their inclusion criteria and evaluating whether all of the treatments labeled as VRET did indeed consist of at least 50% of virtual reality components. As a result, we argue that the publication by Horigome and colleagues misleadingly implies that there is a larger body of literature on the efficacy of VRET for SAD than there actually is.

First, a clarification regarding the total number of included studies is needed. In both the abstract and the results section the authors reported that a total of 22 studies were included in their analyses. However, this number relates to the number of publications that the authors used for data extraction. A close look at Table 1 in Horigome et al. reveals that the authors counted the trial by Anderson et al. (Reference Anderson, Price, Edwards, Obasaju, Schmertz, Zimand and Calamaras2013) as well as the one by Safir, Wallach, and Bar-Zvi (Reference Safir, Wallach and Bar-Zvi2012) twice, respectively. Accordingly, they rather included a total of 20 studies described in 22 publications. Second, we argue that two of the included studies were erroneously labeled as VRET. First, the study by Lister et al. (2010Footnote ^†Footnote ¹) used virtual reality for assessment purposes only and their treatment did not comprise VRET. Second, the study by Yuen et al. (2019¹) comprised of in vivo exposure in addition to VRET. More specifically, during sessions, participants were asked to engage in exposure to a real audience using video-conferencing. For homework, they received VRET while applying techniques of acceptance and commitment therapy and were encouraged to engage in at least three in vivo exposure exercises in-between sessions. Accordingly, <50% of treatment consisted of VRET and therefore any potential treatment efficacy might have resulted from other treatment elements than VRET (Morina, Ijntema, Meyerbröker, & Emmelkamp, Reference Morina, Ijntema, Meyerbröker and Emmelkamp2015). Finally, six studies (Denizci Nazligul et al., 2019; Grillon et al., 2006; Harris et al., 2002; Kovar, 2018; North et al., 1998; Roy et al., 2003¹) included <10 participants in VRET. Horigome et al. reported that ‘study participants numbered 10 or more people’, without specifying whether this number referred to the number of participants per condition or per trial. Research, however, strongly suggests that trials with small samples (i.e. low statistical power) reduce the likelihood that a statistically significant result reflects a true effect (Ioannidis, Reference Ioannidis2005). In fact, treatment effect estimates are often significantly larger in trails with small samples (Dechartres, Trinquart, Boutron, & Ravaud, Reference Dechartres, Trinquart, Boutron and Ravaud2013; Kjaergard, Villumsen, & Gluud, Reference Kjaergard, Villumsen and Gluud2001; Turner, Bird, & Higgins, Reference Turner, Bird and Higgins2013). A minimum of 10 participants per condition was also applied in other related meta-analyses (e.g. Kampmann et al., Reference Kampmann, Emmelkamp and Morina2016b; Morina et al., Reference Morina, Ijntema, Meyerbröker and Emmelkamp2015). Consequently, we excluded the six trials with <10 patients per condition and argue that only 12 studies need to be included in the meta-analysis. In addition, Horigome et al. compared the efficacy of VRET to waitlist conditions and treatment as usual (TAU) combined. However, only one trial that used cognitive therapy as a comparison condition (Wallach, Safir, & Bar-Zvi, Reference Wallach, Safir and Bar-Zvi2011) was defined as TAU and the remaining studies comprised waitlist conditions. Although it is questionable to categorize cognitive therapy as TAU, given their definition of cognitive behavioral therapy involving exposure in vivo as the gold standard SAD treatment, we also argue that it is clinically more useful to report on the efficacy of VRET as compared to waitlist only.

Table 1. Within and between group comparison results

k, number of treatment arms; n, total number of participants in all relevant trials; PI, prediction interval.

Note: Effect sizes (ES) are reported as Hedges' g; the I ²-statistic and Q-statistic are indicators of heterogeneity in percentages, with higher percentages indicating high heterogeneity.

^a Number of treatment arms (i.e. k) is exactly the same as number of studies.

^b Only the longest available follow-up data per trial was retained, except for Anderson et al. (Reference Anderson, Price, Edwards, Obasaju, Schmertz, Zimand and Calamaras2013) were the 12 months follow-up was retained rather than the exceptionally long 72 months follow-up.

In light of these limitations, we deemed it important to reanalyze the data on the efficacy of VRET for SAD. To this end, we included 12 studies in our meta-analysis, six of which were RCTs (Anderson et al., Reference Anderson, Price, Edwards, Obasaju, Schmertz, Zimand and Calamaras2013; Bouchard et al., Reference Bouchard, Dumoulin, Robillard, Guitard, Klinger, Forget and Roucaut2017; Kampmann et al., Reference Kampmann, Emmelkamp, Hartanto, Brinkman, Zijlstra and Morina2016a; Klinger et al., Reference Klinger, Bouchard, Légeron, Roy, Lauer, Chemin and Nugues2005; Safir et al., Reference Safir, Wallach and Bar-Zvi2012; Wallach et al., Reference Wallach, Safir and Bar-Zvi2011) and the remaining six were non-controlled studies (Anderson, Zimand, Schmertz, & Ferrer, Reference Anderson, Zimand, Schmertz and Ferrer2007; Gebara, de Barros-Neto, Gertsenchtein, & Lotufo-Neto, Reference Gebara, de Barros-Neto, Gertsenchtein and Lotufo-Neto2016; Geraets et al., Reference Geraets, Veling, Witlox, Staring, Matthijssen and Cath2019; Kim et al., Reference Kim, Hong, Kim, Jung, Kyeong and Kim2017; Robillard, Bouchard, Dumoulin, Guitard, & Klinger, Reference Robillard, Bouchard, Dumoulin, Guitard and Klinger2010; Stupar-Rutenfrans, Ketelaars, & van Gisbergen, Reference Stupar-Rutenfrans, Ketelaars and van Gisbergen2017).

To calculate uncontrolled effect sizes, the posttreatment or follow-up mean was subtracted from the pretreatment mean and for controlled effect sizes the control group mean was subtracted from the treatment group mean at posttreatment or follow-up, respectively, and divided by the pooled standard deviation. Subsequently, the outcome was multiplied by a sample size correction factor J = 1–(3/(4df − 1)) to obtain the effect size Hedges' g. Analyses were completed with the metafor package (v.1.9.8) in R 3.5 using random-effects models given the heterogeneity of the studies (R Core Team, 2015; Viechtbauer, Reference Viechtbauer2010).

Detailed results of our analyses are presented in Table 1. Across all interventions, a large pre–post effect size was found. Effect sizes were further large when pre-assessment scores of social anxiety were compared to follow-up assessment scores at different time points. These findings indicate that VRET can significantly reduce symptoms of social anxiety. Note, however, that uncontrolled effect sizes do not account for the impact of time on symptoms, therefore controlled effect sizes need to be viewed as more reliable when it comes to assessing treatment efficacy. These analyses revealed that the pooled between-group effect size comparing VRET to waitlist at post-treatment was also large. Finally, the comparison of VRET to exposure in vivo yielded no significant differences between the two treatments, neither at post-assessment nor at follow-up.

When conducting our analyses on treatment efficacy at follow-up, we deviated from Horigome et al. to some extent. First, we did not conduct a meta-analysis with one trial only given the fact that a meta-analysis is per definition a technique that combines the results of multiple studies. In fact, in previous meta-analyses we have argued that a minimum of four trials is needed for a meaningful meta-analysis (Morina et al., Reference Morina, Ijntema, Meyerbröker and Emmelkamp2015; Morina, Koerssen, & Pollet, Reference Morina, Koerssen and Pollet2016). Second and related to this, we decided to conduct an additional follow-up analysis with a larger amount of trials by including only the longest available follow-up data per trial [with the exception of Anderson et al. (Reference Anderson, Price, Edwards, Obasaju, Schmertz, Zimand and Calamaras2013) were the 12 months follow-up was retained rather than the exceptionally long 72 months follow-up]. As can be seen in Table 1, change from pretreatment to follow-up resulted in a large effect size across VRET treatments. Furthermore, there was no significant difference between VRET and exposure in vivo at follow-up.

Our within-group comparison findings are mostly in line with those reported by Horigome et al. Nonetheless, our findings show a stronger treatment efficacy of VRET from pre-treatment to post-treatment and to follow-up. With respect to between-group comparisons, our results vary from those reported by Horigome et al. to some greater extent than the within-group findings. First, the comparison of VRET to wait-list is smaller in our meta-analysis, albeit still in the range of a large effect. Second, our analyses of the efficacy of VRET relative to exposure in vivo produced insignificant results only, whereas Horigome et al. report significant results between VRET and exposure in vivo at follow-up that go in both directions. Yet, we need to interpret these results with caution as these are based on a limited number of RCTs.