Skip to main content

Advertisement

SpringerLink
Log in

Cognitive interviewing of the US National Cancer Institute’s Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE)

  • Published:
Quality of Life Research Aims and scope Submit manuscript

Abstract

Purpose

The National Cancer Institute’s Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE) is a library of question items that enables patient reporting of adverse events (AEs) in clinical trials. This study contributes content validity evidence of the PRO-CTCAE by incorporating cancer patient input of the relevance and comprehensiveness of the item library.

Methods

Cognitive interviews were conducted among patients undergoing chemotherapy or radiation therapy at multiple sites to evaluate comprehension, memory retrieval, judgment, and response mapping related to AE terms (e.g., nausea), attribute terms (regarding frequency, severity, or interference), response options, and recall period. Three interview rounds were conducted with ≥20 patients completing each item per round. Items were modified and retested if ≥3 patients exhibited cognitive difficulties or if experienced by ≤25 % patients.

Results

One hundred and twenty-seven patients participated (35 % ≤high school, 28 % non-white, and 59 % female). Most AE terms (63/80) generated no cognitive difficulties. The remaining 17 were modified without further difficulties by Round 3. Terms were comprehended regardless of education level. Attribute terms and response options required no modifications. Patient adherence to recall period (7 days) was improved when the reference period was incorporated.

Conclusions

This study provides evidence confirming comprehension of the US English language versions of items in the PRO-CTCAE library for measuring symptomatic AEs from the patient perspective within the context of cancer treatment. Several minor changes were made to the items to improve item clarity, comprehension, and ease of response judgment. This study helps to establish the content validity of PRO-CTCAE items for patient reporting of AEs during cancer treatment.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  1. Trotti, A., Colevas, A. D., Setser, A., Rusch, V., Jaques, D., Budach, V., et al. (2003). CTCAE v3.0: Development of a comprehensive grading system for the adverse events of cancer treatment. Seminars in Radiation Oncology, 13, 176–181.

    Article  PubMed  Google Scholar 

  2. National Cancer Institute, National Institutes of Health, US Department of Health and Human Services. Common Terminology Criteria for Adverse Events (CTCAE) Version 4.0. Published May 28, 2009; Revised version 4.03 June 14, 2010 (Vol. Available from: http://evs.nci.nih.gov/ftp1/CTCAE/CTCAE_4.03_2010-06-14_QuickReference_5x7.pdf. Accessed 20 May 2013).

  3. Basch, E. (2010). The missing voice of patients in drug-safety reporting. New England Journal of Medicine, 362(10), 865–869. doi:10.1056/NEJMp0911494.

    Article  CAS  PubMed Central  PubMed  Google Scholar 

  4. Basch, E., Iasonos, A., McDonough, T., Barz, A., Culkin, A., Kris, M. G., et al. (2006). Patient versus clinician symptom reporting using the National Cancer Institute Common Terminology Criteria for Adverse Events: Results of a questionnaire-based study. Lancet Oncology, 7, 903–909. doi:10.1016/S1470-2045(06)70910-X.

    Article  PubMed  Google Scholar 

  5. Basch, E., Jia, X., Heller, G., Barz, A., Sit, L., Fruscione, M., et al. (2009). Adverse symptom event reporting by patients versus clinicians: Relationships with clinical outcomes. Journal of the National Cancer Institute, 101(23), 1624–1632. doi:10.1093/jnci/djp386.

    Article  PubMed  Google Scholar 

  6. Atkinson, T. M., Li, Y., Coffey, C. W., Sit, L., Shaw, M., Lavene, D., et al. (2012). Reliability of adverse symptom event reporting by clinicians. Quality of Life Research, 21(7), 1159–1164. doi:10.1007/s11136-011-0031-4.

    Article  PubMed Central  PubMed  Google Scholar 

  7. Xiao, C., Polomano, R., & Bruner, D. W. (2012). Comparison between patient-reported and clinician-observed symptoms in oncology. Cancer Nursing,. doi:10.1097/NCC.0b013e318269040f.

    Google Scholar 

  8. Greimel, E. R., Bjelic-Radisic, V., Pfisterer, J., Hilpert, F., Daghofer, F., Pujade-Lauraine, E., et al. (2011). Toxicity and quality of life outcomes in ovarian cancer patients participating in randomized controlled trials. Supportive Care in Cancer, 19(9), 1421–1427. doi:10.1007/s00520-010-0969-8.

    Article  PubMed  Google Scholar 

  9. Flores, L. T., Bennett, A. V., Law, E. B., Hajj, C., Griffith, M. P., & Goodman, K. A. (2012). Patient-reported outcomes versus clinician symptom reporting during chemoradiation for rectal cancer. Gastrointestinal Cancer Research, 5(4), 119–124.

    PubMed  Google Scholar 

  10. Basch, E., Iasonos, A., Barz, A., Culkin, A., Kris, M. G., Artz, D., et al. (2007). Long-term toxicity monitoring via electronic patient-reported outcomes in patients receiving chemotherapy. Journal of Clinical Oncology, 25(34), 5374–5380. doi:10.1200/JCO.2007.11.2243.

    Article  PubMed  Google Scholar 

  11. Tang, B., Giuliani, M., Le, L. W., Higgins, J., Bezjak, A., Brade, A., et al. (2013). Capturing acute toxicity data during lung radiotherapy by using a patient-reported assessment tool. Clinical Lung Cancer, 14(2), 108–112. doi:10.1016/j.cllc.2012.06.003.

    Article  PubMed  Google Scholar 

  12. US Department of Health and Human Services. Guidance for industry. patient-reported outcome measures: Use in medical development to support labeling claims (December 2009. Available from: http://www.fda.gov/downloads/Drugs/GuidanceComplianceRegulatoryInformation/Guidances/UCM193282.pdf. Accessed 20 May 2013).

  13. European Medicines Agency. Committee for medicinal products for human use (CHMP). Pre-authorisation evaluation of medicines for human use: Reflection paper on the regulatory guidance for the use of health-related quality of life (HRQL) measures in the evaluation of medicinal products (January 2006, Available from: http://www.ispor.org/workpaper/emea-hrql-guidance.pdf. Accessed 20 May 2013).

  14. Ashley, L., Jones, H., Thomas, J., Forman, D., Newsham, A., Morris, E., et al. (2011). Integrating cancer survivors’ experiences into UK cancer registries: Design and development of the ePOCS system (electronic Patient-reported Outcomes from Cancer Survivors). British Journal of Cancer, 105, S74–S81. doi:10.1038/Bjc.2011.424.

    Article  PubMed Central  PubMed  Google Scholar 

  15. Banerjee, A. K., & Ingate, S. (2012). Web-based patient-reported outcomes in drug safety and risk management: Challenges and opportunities? Drug Safety, 35(6), 437–446. doi:10.2165/11632390-000000000-00000.

    Article  PubMed  Google Scholar 

  16. Boers-Doets, C. B., Gelderblom, H., Lacouture, M. E., Epstein, J. B., Nortier, J. W., & Kaptein, A. A. (2013). Experiences with the FACT-EGFRI-18 instrument in EGFRI-associated mucocutaneous adverse events. Supportive Care in Cancer,. doi:10.1007/s00520-013-1752-4.

    Google Scholar 

  17. Chan, A., & Tan, E. H. (2011). How well does the MESTT correlate with CTCAE scale for the grading of dermatological toxicities associated with oral tyrosine kinase inhibitors? Supportive Care in Cancer, 19(10), 1667–1674. doi:10.1007/s00520-010-0999-2.

    Article  PubMed  Google Scholar 

  18. Farnell, D. J., Mandall, P., Anandadas, C., Routledge, J., Burns, M. P., Logue, J. P., et al. (2010). Development of a patient-reported questionnaire for collecting toxicity data following prostate brachytherapy. Radiotherapy and Oncology, 97(1), 136–142. doi:10.1016/j.radonc.2010.05.011.

    Article  PubMed  Google Scholar 

  19. Quinten, C., Maringwa, J., Gotay, C. C., Martinelli, F., Coens, C., Reeve, B. B., et al. (2011). Patient self-reports of symptoms and clinician ratings as predictors of overall cancer survival. Journal of the National Cancer Institute, 103(24), 1851–1858. doi:10.1093/Jnci/Djr485.

    Article  PubMed  Google Scholar 

  20. Stacey, D., Bakker, D., Ballantyne, B., Chapman, K., Cumminger, J., Green, E., et al. (2012). Managing symptoms during cancer treatments: evaluating the implementation of evidence-informed remote support protocols. Implementation Science, 7, 110. doi:10.1186/1748-5908-7-110.

    Article  PubMed  Google Scholar 

  21. Johansen, M. A., Henriksen, E., Horsch, A., Schuster, T., & Berntsen, G. K. R. (2012). Electronic symptom reporting between patient and provider for improved health care service quality: A systematic review of randomized controlled trials. Part 1: State of the Art. Journal of Medical Internet Research, 14(5). doi:10.2196/jmir.2214.

  22. Johansen, M. A., Henriksen, E., Berntsen, G., & Horsch, A. (2011). Electronic symptom reporting by patients: A literature review. Studies in Health Technology and Informatics, 169, 13–17.

    PubMed  Google Scholar 

  23. Seow, H., King, S., Green, E., Pereira, J., & Sawka, C. (2011). Perspectives of patients on the utility of electronic patient-reported outcomes on cancer care. Journal of Clinical Oncology, 29(31), 4213–4214. doi:10.1200/JCO.2011.37.9750.

    Article  PubMed  Google Scholar 

  24. King, A., Daniels, J., Lim, J., Cochrane, D. D., Taylor, A., & Ansermino, J. M. (2010). Time to listen: A review of methods to solicit patient reports of adverse events. Quality and Safety in Health Care, 19(2), 148–157. doi:10.1136/qshc.2008.030114.

    Article  CAS  PubMed  Google Scholar 

  25. Zhu, J., Stuver, S. O., Epstein, A. M., Schneider, E. C., Weissman, J. S., & Weingart, S. N. (2011). Can we rely on patients’ reports of adverse events? Medical Care, 49(10), 948–955. doi:10.1097/MLR.0b013e31822047a8.

    Article  PubMed  Google Scholar 

  26. Detmar, S. B., Muller, M. J., Schornagel, J. H., Wever, L. D., & Aaronson, N. K. (2002). Health-related quality-of-life assessments and patient-physician communication: A randomized controlled trial. JAMA, 288(23), 3027–3034.

    Article  PubMed  Google Scholar 

  27. Basch, E. M., Reeve, B. B., Mitchell, S. A., Clauser, S. B., Minasian, L., Sit, L., et al. (2011). Electronic toxicity monitoring and patient-reported outcomes. Cancer Journal, 17(4), 231–234. doi:10.1097/PPO.0b013e31822c28b3.

    Article  Google Scholar 

  28. Stull, D. E., Leidy, N. K., Parasuraman, B., & Chassany, O. (2009). Optimal recall periods for patient-reported outcomes: Challenges and potential solutions. Current Medical Research and Opinion, 25(4), 929–942. doi:10.1185/03007990902774765.

    Article  PubMed  Google Scholar 

  29. Kirkova, J., Davis, M. P., Walsh, D., Tiernan, E., O’Leary, N., LeGrand, S. B., et al. (2006). Cancer symptom assessment instruments: A systematic review. Journal of Clinical Oncology, 24(9), 1459–1473. doi:10.1200/JCO.2005.02.8332.

    Article  PubMed  Google Scholar 

  30. Wild, D., Eremenco, S., Mear, I., Martin, M., Houchin, C., Gawlicki, M., et al. (2009). Multinational trials-recommendations on the translations required, approaches to using the same language in different countries, and the approaches to support pooling the data: The ISPOR patient-reported outcomes translation and linguistic validation good research practices task force report. Value in Health, 12(4), 430–440. doi:10.1111/j.1524-4733.2008.00471.x.

    Article  PubMed  Google Scholar 

  31. Basch, E., Reeve, B. B., Mitchell, S. A., Clauser, S. B., Minasian, L. M., Atkinson, T. M., et al. (Under Review). Development of the National Cancer Institute’s Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE). Journal of National Cancer Institute.

  32. Willis, G. B. (2005). Cognitive interviewing: A tool for improving questionnaire design. Thousand Oaks, CA: SAGE Publications, Inc.

    Google Scholar 

  33. Patrick, D. L., Burke, L. B., Gwaltney, C. J., Leidy, N. K., Martin, M. L., Molsen, E., et al. (2011). Content validity-establishing and reporting the evidence in newly Developed patient-reported outcomes (PRO) instruments for medical product evaluation: ISPOR PRO good research practices task force report: Part 2-assessing respondent understanding. Value in Health, 14(8), 978–988. doi:10.1016/j.jval.2011.06.013.

    Article  PubMed  Google Scholar 

  34. Bowen, G. A. (2008). Naturalistic inquiry and the saturation concept: A research note. Qualitative Research, 8(1), 137–152. doi:10.1177/1468794107085301.

    Article  Google Scholar 

  35. Guest, G., Bunce, A., & Johnson, L. (2006). How many interviews are enough? An experiment with data saturation and variability. Field Methods, 18, 59–82. doi:10.1177/1525822X05279903.

    Article  Google Scholar 

  36. Jobe, J. B., & Mingay, D. J. (1989). Cognitive research improves questionnaires. American Journal of Public Health, 79(8), 1053–1055.

    Article  CAS  PubMed  Google Scholar 

  37. Jobe, J. B. (2003). Cognitive psychology and self-reports: Models and methods. Quality of Life Research, 12(3), 219–227.

    Article  PubMed  Google Scholar 

  38. Miller, K. (2003). Conducting cognitive interviews to understand question-response limitations. American Journal of Health Behavior, 27(Suppl 3), S264–S272.

    Article  PubMed  Google Scholar 

  39. Schwarz, N., Oyserman, D., & Petycheva, E. (2010). Cognition, communication, and culture: Implications for the survey response process. In J. A. Harkness, M. Braun, B. Edwards, T. P. Johnson, L. E. Lyberg, P. P. Mohler, et al. (Eds.), Survey methods in multinational, multicultural, and multiregional contexts (pp. 177–190). Hoboken, NJ: John Wiley & Sons.

    Google Scholar 

  40. Tourangeau, R. (1984). Cognitive science and survey methods. In T. Jabine, M. Straf, J. Tanur, & R. Tourangeau (Eds.), Cognitive aspects of survey design: Building a bridge between disciplines (pp. 73–100). Washington, DC: National Academy Press.

    Google Scholar 

  41. Namey, E., Guest, G., Thairy, L., & Johnson, L. (2008). Data reduction techniques for large qualitative data sets. In G. Guest & K. M. MacQueen (Eds.), Handbook for team-based qualitative research (pp. 137–161). Lanham, MD: AltaMira Press.

    Google Scholar 

  42. Saldana, J. (2009). The coding manual for qualitative researchers. Los Angeles, CA: Sage Publications.

    Google Scholar 

  43. Basch, E. (2012). Beyond the FDA PRO guidance: Steps toward integrating meaningful patient-reported outcomes into regulatory trials and US drug labels. Value in Health, 15(3), 401–403. doi:10.1016/j.jval.2012.03.1385.

    Article  PubMed  Google Scholar 

  44. Symptom Management and Quality of Life Steering Committee. Clinical trials planning meeting—building bridges: The identification of core symptoms and health-related quality of life domains for use in cancer research (September 22–23, 2011. Available from: http://transformingtrials.cancer.gov/files/SxQOLSCPROCTPMExecutive%20Summary_FINAL.pdf Accessed 7 June 2012).

  45. Mitchell, S. A., Lang, K., Nichols, C., Clauser, S. B., Federico, V., Lalla, D., et al. (2012). Validation of the NCI Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE) in women receiving treatment for metastatic breast cancer. Chicago, IL: Annual Meeting of the American Society of Clinical Oncology.

    Google Scholar 

  46. Dueck, A. C., Mendoza, T., Mitchell, S. A., Reeve, B. B., Castro, K., Denicoff, A., et al. (2012). Validity and reliability of the Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE). Chicago, IL: Annual Meeting of the American Society of Clinical Oncology.

    Google Scholar 

  47. Coons, S. J., Gwaltney, C. J., Hays, R. D., Lundy, J. J., Sloan, J. A., Revicki, D. A., et al. (2009). Recommendations on evidence needed to support measurement equivalence between electronic and paper-based patient-reported outcome (PRO) measures: ISPOR ePRO good research practices task force report. Value in Health, 12(4), 419–429. doi:10.1111/j.1524-4733.2008.00470.x.

    Article  PubMed  Google Scholar 

  48. (2012) Methodological standards and patient-centeredness in comparative effectiveness research—The PCORI perspective. JAMA, 307(15), 1636–1640. doi:10.1001/jama.2012.466.

Download references

Acknowledgments

Work described in this report was supported by contracts from the United States National Cancer Institute, HHSN261201000043C and HHSN261201000063C. Each author met all International Committee of Medical Journal Editors (ICMJE) requirements (i.e., conception/design/acquisition/analysis, drafting/revising article, and final approval) for authorship.

Author information

Authors and Affiliations

  1. Department of Psychiatry and Behavioral Sciences, Memorial Sloan-Kettering Cancer Center, 641 Lexington Ave., 7th Floor, New York, NY, 10022, USA

    Jennifer L. Hay, Thomas M. Atkinson, Lauren Rogak, Laura Sit & Ethan Basch

  2. University of North Carolina at Chapel Hill, Chapel Hill, NC, USA

    Bryce B. Reeve, Antonia V. Bennett & Ethan Basch

  3. United States National Cancer Institute, Rockville, MD, USA

    Sandra A. Mitchell, Gordon Willis, Lori M. Minasian, Steven B. Clauser, Andrea Denicoff, Ann O’Mara & Alice Chen

  4. University of Texas M.D. Anderson Cancer Center, Houston, TX, USA

    Tito R. Mendoza

  5. Dana-Farber Cancer Institute, Boston, MA, USA

    Joshua Gagne, Vish Viswanath & Deborah Schrag

  6. New York, NY, USA

    Diane B. Paul

Authors
  1. Jennifer L. Hay
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Thomas M. Atkinson
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Bryce B. Reeve
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Sandra A. Mitchell
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Tito R. Mendoza
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Gordon Willis
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Lori M. Minasian
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Steven B. Clauser
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Andrea Denicoff
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Ann O’Mara
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Alice Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Antonia V. Bennett
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Diane B. Paul
    View author publications

    You can also search for this author in PubMed Google Scholar

  14. Joshua Gagne
    View author publications

    You can also search for this author in PubMed Google Scholar

  15. Lauren Rogak
    View author publications

    You can also search for this author in PubMed Google Scholar

  16. Laura Sit
    View author publications

    You can also search for this author in PubMed Google Scholar

  17. Vish Viswanath
    View author publications

    You can also search for this author in PubMed Google Scholar

  18. Deborah Schrag
    View author publications

    You can also search for this author in PubMed Google Scholar

  19. Ethan Basch
    View author publications

    You can also search for this author in PubMed Google Scholar

Consortia

the NCI PRO-CTCAE Study Group

Corresponding author

Correspondence to Jennifer L. Hay.

Additional information

NCI PRO-CTCAE Study Group members are Cindy Geoghegan, Jeff A. Sloan (Mayo Clinic, Rochester, MN, USA), Charles S. Cleeland (University of Texas M.D. Anderson Cancer Center, Houston, TX, USA), Amy P. Abernethy (Duke University Medical Center, Durham, NC, USA), Deborah W. Bruner (University of Pennsylvania School of Nursing, Philadelphia, PA, USA), Allison Barz (The Children’s Hospital of Philadelphia, Philadelphia, PA, USA), Amylou C. Dueck (Mayo Clinic, Scottsdale, AZ, USA), and Yuelin Li (Memorial Sloan-Kettering Cancer Center, New York, NY, USA).

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (DOCX 23 kb)

Rights and permissions

Reprints and permissions

About this article

Cite this article

Hay, J.L., Atkinson, T.M., Reeve, B.B. et al. Cognitive interviewing of the US National Cancer Institute’s Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE). Qual Life Res 23, 257–269 (2014). https://doi.org/10.1007/s11136-013-0470-1

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11136-013-0470-1

Keywords

Search

Navigation