Participation in a quality of care study and consequences for generalizability of general practice research

Abstract

Background. Non-participation of General Practitioners (GPs) hampers primary care research. Using existing network structures can improve participation, but may introduce sampling effects.

Objectives. To examine the role of network-based recruitment and other determinants of participation, and to estimate effects of sampling and non-participation on generalizability.

Methods. For a study of quality of care assessment, we recruited GPs from a regionally defined population and a GP network. Effects of sampling and non-participation were analysed by comparing characteristics between participants, target samples, and reference data for all German GPs. Factors influencing study participation were assessed in multiple logistic regression.

Results. Compared to the regional sample, network GPs were more likely to complete survey questionnaires (92% versus 69%) and to participate in the study (66% versus 23%). Compared to national reference data, study participants from both populations were younger, had a higher level of professional training, and included more men. These differences were already present in the network target sample, but were largely attributable to selective participation in the regional sample. Network membership remained the strongest determinant of participation in multiple logistic regression (odds ratio 5.01; 95% confidence interval 2.53–9.91). A younger age, higher professional training, and membership in the German Society of General Practice were also predictive of participation.

Conclusions. Although network-based recruitment of GPs increases participation rates, sample effects are similar in size and direction as effects of non-participation in the regionally defined population. Careful analysis of participants based on publicly available data is therefore crucial for the assessment of generalizability.

Wetzel D, Himmel W, Heidenreich R, Hummers-Pradier E, Kochen MM, Rogausch A, Sigle J, Boeckmann H, Kuehnel S, Niebling W and Scheidt-Nave C. Participation in a quality of care study and consequences for generalizability of general practice research. Family Practice 2005; 22: 458–464.

Introduction

Research in primary care settings yields a valuable source of data for general practice research and health care planning. One of the major obstacles and potential threats to the validity of research involving GPs and other community-based doctors is non-participation.^1,2 This is particular true for quality of care studies.³ Recruitment based on existing physician networks can be a powerful tool to increase participation rates,² but may lead to sampling effects. Recent general practice research has drawn attention to the fact that improving participation rates does not necessarily improve the generalizability of the study results. Several authors recommended to study the effects of potential bias in greater detail.^1,2,4

MedViP [Medizinische Versorgung in der Praxis (Medical Care in General Practice)] is a primary health care research project funded by the German Ministry of Education and Research. The MedViP study is coordinated by the Department of General Practice, University of Goettingen Medical School. The feasibility and validity of computerized medical records for quality of care assessment in German general practices will be investigated. Doctors were recruited from a network of GPs and from a regionally defined but otherwise unselected target population of GPs. The aims of the present analysis were: (1) to examine the effects of sampling and non-participation on the representativeness of the study population; and (2) to assess the relative impact of network-based recruitment and other factors on study participation.

Methods

Study design

The design and aims of the main study have been described in detail elsewhere.^5,6 In brief, MedViP is a cluster study, where the individual practice represents the unit of analysis. Using medical data routinely entered into the computer, process measures will be compared to current clinical guideline recommendations, especially for patients with asthma or chronic obstructive pulmonary disease (COPD), heart failure, or urinary tract infections. Participating practices receive a one-time compensation of 500 Euro and Post Graduate Education Allowance Points. Ethical approval was granted by the Ethics Committees of the University of Goettingen and the University of Freiburg Medical Schools.

Target population

In January 2002, a total of 460 primary care doctors (board certified GPs or general internists; practitioners without board certification in any particular specialty) were registered in 363 general practices in the administrative district of Goettingen, Lower Saxony. Their names and addresses were obtained from the KV Goettingen (Kassenaerztliche Vereinigung, the regional Association of Statutory Health Insurance Physicians). Excluding 13 doctors from 10 practices (currently not or no longer in practice), the net Goettingen target sample consisted of 447 GPs in 353 practices. The GP quality assurance network in the vicinity of Freiburg, Baden-Wuerttemberg listed 83 doctors in 66 practices (www.qualitaetspraxen.de). The majority of these doctors have previously contributed to research projects. Two authors are network members (HB; WN) and have been involved in the design of the MedViP study.

Recruitment

A two-stage protocol was used to recruit the study population:

1. In September 2002, the coordinating center contacted eligible doctors by mail (doctors in group practices were addressed individually) and informed about the aims and contents of the MedViP study. Doctors were asked to return an enclosed standardized questionnaire on practice- and doctor-related characteristics, irrespective of their decision to participate in the research project. The standardized questionnaire (Appendix 1) consisted of questions concerning main characteristics of the study population, computer equipment and access to the internet, interest to join the MedViP study (yes; need further information; no), and, if applicable the main reason for decline (no time, not interested, other). Two reminders were mailed at intervals of 6 to 8 weeks. Doctors who completed the questionnaire were counted as survey responders.

2. GPs who were interested in joining the research project received further information illustrating the study protocol and time schedule. Between December 2002 and January 2004, technical staff were sent out to these practices to assist the computer export of electronic medical data via a standard interface (Behandlungs-Daten-Traeger, BDT) as previously described.⁷ Doctors who successfully completed the data export were counted as study participants.

Publicly available data

The KBV (Kassenaerztliche Bundesvereinigung, the national Association of Statutory Health Insurance Physicians) provided aggregate reference data about age, gender, time in office practice, vocational training status and practice type among German GPs. Respective information for the MedViP target samples was obtained from the KV Goettingen and the office of the Freiburg GP network.

GPs involved in teaching medical students were identified from faculty listings at the Universities of Goettingen and Freiburg Medical Schools. Members of the German Society of General Practice and Family Medicine (DEGAM) were identified from membership directories.

Statistical analysis

In order to estimate the effects of sampling and non-participation, main characteristics of the German GP reference population from KBV statistics were compared to the MedViP target samples, survey responders and study participants in Goettingen and Freiburg, respectively. Relative frequencies, means and 95% confidence intervals were calculated and compared between groups for overlap.

Among survey responders, descriptive data obtained from postal questionnaires as well as information regarding teaching and DEGAM membership were compared by target group (Freiburg versus Goettingen), and between study participants and non-participants within target groups.

The Statistical Analysis Software (SAS) package, Version 8.2 was used for data analysis. A P-level of <0.05 was considered statistically significant based on two-sided tests.

Chi-Square or Fisher's exact test (dichotomous data), and Wilcoxon test (continuous data) were used to test for group differences as appropriate. Correlation between variables was assessed by Spearman's rho. A multiple logistic regression model was fitted to examine the independent effects of target group and covariates on study participation (yes versus no). Only variables that showed significance at the 5% level (P < 0.05) in univariate analyses were entered into the model. Dummy variables were computed for variables with more than two response levels.

Results

Response and participation rates

Recruitment results are summarized in Figures 1a and 1b and Table 1. Overall, 92% (76/83) of Freiburg network members and 69% (308/447) of eligible GPs in the Goettingen area responded to the initial postal survey (survey responders). The proportion of doctors who initially agreed to participate in the MedViP project was about twice as high (67% versus 32%) in Freiburg (56/83) as in Goettingen (144/447) (P < 0.001).

Twenty Goettingen doctors and one doctor from Freiburg had later withdrawn their initial consent to participate in the study for of a variety of reasons (mainly ‘work overload’). Data from another 20 doctors in Goettingen could not be obtained due to technical reasons (software or hardware did not allow data export).

Eventually, 104 doctors (23%) from the Goettingen sample and 55 doctors from the QP network (66%) successfully completed the baseline data export from their practice computers (study participants, Figures 1a and 1b, Table 1).

Reasons for non-participation

Among survey responders who declined to participate in the study (n = 144 in Goettingen, n = 21 in Freiburg), 117 doctors in Goettingen and 19 in Freiburg had checked their reason for non-participation on the survey questionnaire. In both groups, ‘no time’ was by far the predominant reason (60% in Goettingen, 68% in Freiburg, data not shown). Nine percent in Goettingen had checked ‘not interested’. ‘Other reasons’ was checked by 32% in Goettingen (n = 37) and in Freiburg (n = 6), respectively.

Where ‘other reasons’ were specified, in Goettingen they included advanced age or bad health (n = 10) as well as technical or practice related obstacles (n = 13), such as lack of computerization or specialization on psychotherapy or homeopathy. One doctor wrote that he did not understand the goal of the study, and two wrote that they were too frustrated with budget restraints or guideline dictations. Another two doctors expressed concerns with data protection. In Freiburg technical and practice related reasons prevailed (n = 3), and one doctor indicated that he was already involved in a different research project.

Effects of sampling and non-participation

Study participants in Goettingen significantly differed from the German reference population of GPs with respect to nearly all characteristics available from publicly available data (Table 1). They were significantly younger, more likely to be male, to be board certified GPs and to be DEGAM-members. Most differences were entirely attributable to non-participation effects, with some exceptions: sampling effects apparently accounted for the overrepresentation of doctors in solo practices in the target sample and contributed to the overrepresentation of board-certified GPs and the underrepresentation of general internists working in primary care. To check whether data from the 20 Goettingen doctors who could not participate due to technical reasons alone would have changed results, we recalculated these analyses counting them as study participants. The results did not materially change and the estimates were generally intermediate between survey responders and ‘real’ study participants (data not shown, available upon request).

Study participants from the Freiburg network significantly differed from the reference population with respect to a lower proportion of GPs without board certification and a higher proportion of board certified internists. This difference was already evident in the target sample (Table 1). To a lesser degree (slightly overlapping 95%CI) this was also true for a lower proportion of female doctors and a longer mean time in office practice among study participants compared to all German GPs.

Determinants of participation

Among survey responders, study participants in Goettingen were significantly less likely than non-participants to be female or to be GPs without board certification, but more likely to teach medical students, to be a DEGAM member, to use a practice computer system and to run a vocational training practice (Table 2). They were also, on average, 3 years younger, a difference that was also reflected by less years in office practice.

Study participants in the Freiburg network differed from non-participants in only one aspect: they were more likely to have access to the internet at the office (P < 0.001).

In multiple logistic regression (analysis), belonging to the Freiburg network remained the strongest independent determinant of study participation (odds ratio 5.01; 95% confidence interval 2.53–9.91). A younger age, board certification and DEGAM-membership were also significant predictors (Table 3).

Discussion

In line with studies from the UK, the USA and Canada,² the use of existing network structures in our study proved to be very effective with respect to increasing the participation rate. Recruitment of GPs from an established network nearly tripled the participation rate in this quality of primary care study (66% versus 23%). The difference in participation rates could not be explained by differences in basic characteristics of the target populations.

As recently suggested by Barclay and colleagues,⁴ we used publicly available data to assess the size, direction and source of potential selection effects in our study populations compared to the population of German GPs at large. Study participants from both target groups differed from the German national GP population in a similar direction albeit for different reasons. Effects of non-participation predominated in the Goettingen regional sample as opposed to sampling effects in the Freiburg network. This means that higher participation rates do not guarantee a better representativeness of the study population, because selection effects due to non-participation may be traded for selection effects due to sampling. Non-participation among Goettingen GPs as well as preselection of the Freiburg GP network resulted in overrepresentation of male doctors, GPs with board certification and doctors holding a DEGAM membership, although the differences in the network sample were not always statistically significant, probably due to small numbers. Goettingen but not Freiburg study participants were significantly younger than the average of German GPs, an effect that was entirely attributable to selective participation as previously described by others.^8,9 Sampling effects accounted for the fact that the two target samples significantly differed in the proportion of GPs trained as internists. They were overrepresented among network GPs but underrepresented in the Goettingen regional sample.

Results from multiple logistic regression confirmed results of previous studies, as a younger age^8,9 and a higher level of professional training⁸ were also significantly associated with participation, in addition to the strong influence of network membership.

Participation rates in general practice research strongly depend on the topic of the study¹⁰ as well as on the incentives offered to participants.¹¹ While there was no difference in monetary incentives (500 Euro per participating practice), network GPs can be expected to have been particularly interested in the study topic, i.e. the quality of care for general practice patients. This was reflected by a number of differences between the Freiburg and Goettingen study populations: study participants from the Freiburg network had a higher vocational training level, a longer experience in general practice, and were more likely to have internet access at the office and to currently participate in quality circles. Nevertheless, adjustment for these variables did not explain the difference in participation.

The interpretation of our study-results is limited in several ways. First, our comparison between participants and non-participants may be biased by non-response. Although overall response rates to the postal questionnaire were reasonably high,⁴ 31% of eligible GPs in Goettingen and 8% in Freiburg did not answer the postal questionnaire. However, comparison between survey responders and the target sample for demographic variables and other publicly available characteristics revealed no significant differences; hence a severe bias at this point is rather unlikely.

Secondly, with respect to the observed effect of ‘network recruiting’ on participation rates, we cannot separate the effects of local peer support from other network issues. Therefore, ‘network membership’ may represent multiple characteristics. Further research needs to separate these different components.

Third, by using a standardized questionnaire, we inevitably missed important factors influencing participation. As previously reported by others,^2,12,13 ‘no time’ was the reason for non-participation most frequently reported in the present study. Although time restraints are an important issue in general practice, this argument may also be used as a socially desirable excuse for not getting involved in research activities. Our results support the view that female GPs, GPs who were older and less well trained obviously felt less able to cope with, or were less willing to accept additional workload resulting from study participation. Qualitative interviews will be performed in a subset of non-participants to get deeper insight into the ‘true’ reasons for non-participation.

In summary, a considerably higher participation rate among network-based GPs was largely traded for sampling effects on the study population. This is likely to bias results of a study dealing with quality of care largely in the same direction and to a comparable degree as effects due to non-participation in the regional sample. As for our study, this could result in an overestimation of the ability to construct quality of care indices from electronic patient records in general practices.¹⁴

For the sake of generalizability of primary care research, our study results are a plea to identify and, if possible, to eliminate factors that impede higher participation rates in unselected populations of GPs. Network-based recruitment nevertheless assures higher participation rates, but sampling effects should be considered. Careful analysis of the study population based on publicly available data is therefore essential to the interpretation of study results and to assess the generalizability of general practice research.

Declaration

Funding: the MedViP study is supported by a grant of the German Ministry of Education and Research (01GK0201) and an unconditional grant by Ratiopharm GmbH, Ulm.

Ethical approval: granted by the Ethics Committees of the University of Goettingen and the University of Freiburg Medical Schools.

Conflicts of interests: none.

Appendix 1 Questionnaire concerning characteristics of GPs

1. Years in practice?

2. Years of your vocational training?

3. Age?

4. Practice size?

   • <800 listed patients per quarter

   • 800–1200 listed patients per quarter

   • >1200 listed patients per quarter

5. Do you take part in a quality circle? yes no

6. Is your practice involved in vocational training of junior doctors? yes no

7. Do you use a computer in your practice?

8. Do you have access to the internet? yes, in practice yes, at home no

9. Would you like to participate in the study?

   • Yes, I would like to participate in the study

   • I am interested, but I want more information, please contact me

   • I do not want to participate in the study

10. If you do not want to participate, what is your main reason?

    • Not interested

    • No time

    • Other reason

References

Author notes

aDepartment of General Practice, University of Goettingen, bQP-Quality Practices GmbH Bad Krozingen, cCentre for Social Science Methodology, University of Goettingen, dTeaching Unit for General Practice, University of Freiburg and eRobert Koch Institute Berlin, Germany