The validity of questionnaire-based diagnoses: the third Nord-Trøndelag Health Study 2006–2008

Abstract

The Nord-Trøndelag Health Study (HUNT 3) performed in 2006–2008 is a replication of the cross-sectional survey from 1995 to 1997 (HUNT 2). The aim of the present study was to assess the sensitivity and specificity of questionnaire-based headache diagnoses using a personal interview by a neurologist as a gold standard. For the questionnaire-based status as headache sufferer, a sensitivity of 88%, a specificity of 86%, and a kappa statistic of 0.70 were found. Chronic headache, chronic tension-type headache (TTH), and medication overuse headache (MOH) were diagnosed with a specificity of ≥99%, and a kappa statistic of ≥0.73. Lower figures were found for the diagnoses of migraine and TTH. For individuals with headache ≥1 day per month, a sensitivity of 58% (migraine) and 96% (TTH), a specificity of 91 and 69%, and a kappa statistic of 0.54 and 0.44 were found, respectively. The specificity for migraine with aura was 95%. In conclusion, the HUNT 3-questionnaire is a valid tool for identifying headache sufferers, and diagnosing patients with chronic headache, including chronic TTH and MOH. The more moderate sensitivity for migraine and TTH makes the questionnaire-based diagnoses of migraine and TTH suboptimal for determining the prevalence. However, the high specificity of the questionnaire-based diagnosis of migraine, in particular for migraine with aura, makes the questionnaire a valid tool for diagnosing patients with migraine for genetic studies.

Introduction

A careful history taken by a headache specialist is the gold standard for making a valid headache diagnosis. Nevertheless, in large-scale population-based studies, a less time-consuming and costly strategy has to be chosen. A self-administrated questionnaire for the diagnosis of the most common headache disorders ensures standardization [1], but validation of the diagnostic instrument is required in epidemiological surveys to assess the precision of questionnaire diagnoses [1–3].

The headache part of the third Nord-Trøndelag Health Study (HUNT 3) performed in 2006–2008 is a replication of the cross-sectional survey from 1995 to 1997 (HUNT 2) [4]. In order to validate questionnaire-based headache diagnoses and other information, a clinical interview performed by neurologist was done in a random sample of participants in HUNT 3. The aim of the present study was to assess the sensitivity and specificity of questionnaire-based headache diagnoses using a personal interview by a neurologist as a gold standard.

Materials and methods

The third Nord-Trøndelag Health Survey (HUNT 3)

All inhabitants aged 13 years or more in Nord-Trøndelag county of Norway were invited to participate in the third Nord-Trøndelag Health Survey between October 2006 and June 2008 (“Helseundersøkelsen i Nord-Trøndelag” = HUNT). Among 94,194 invited adults aged 20 years or more, a total of 50,839 (54%) answered the first questionnaire (Q1) that was enclosed with the invitation letter. They also participated in a brief medical examination, at which they were given the second questionnaire (Q2) to be filled in and returned from home. The Q2 included a total of 14 headache questions (Table 1) that were designed to determine whether the person suffered from headache, and fulfilled the ICHD-II criteria [5] for migraine or tension-type headache (TTH). The screening question was “Have you suffered from headache during the last year?”, and only individuals who answered “yes” were asked to fill in the other headache questions. 39,701 (42%) of 94,194 invited answered the first headache question. Headache sufferers were further asked to report how their headaches usually were regarding pain intensity, attack duration, and accompanying symptoms (Table 1). To reduce the risk of misunderstanding, all questions were made as clear and simple as possible. Regarding attack duration, the participants were not instructed to report the duration of “untreated attacks”, partly because some individuals always use attack medication for their headaches. In another part of the Q2, the individuals were also asked to state the consumption of over-the-counter (OCT) drugs (painkillers) because of headache during the last month with four answer options: seldom or never, 1–3 times per week, 4–6 times per week, or daily [38,583 (41%) responded].

Table 1 Headache questions in the second questionnaire (Q2)

Study population of the validation study

The method of the present validation study has been described elsewhere [6]. The Nord-Trøndelag County was divided in 25 study areas, and the validation study was performed in two of these (Verdal and Stjørdal). The main survey in Verdal was performed from September to November 2007, and in Stjørdal from December 2007 to April 2008. The participation rates in the HUNT 3 study were 52% of the whole population in Verdal and 50% in Stjørdal. In the present study, a random sample of individuals who had participated in HUNT 3 in the two communities were invited to a face-to-face interview performed by neurologist focusing on four different topics covered in the questionnaire, namely alcohol, sleep, headache and musculoskeletal complaints [6]. The participants’ responses to the questionnaire in HUNT 3 were unknown to the interviewers, and the main objective of the study was to evaluate the validity of questionnaire-based information.

Invitation letters were sent to a random sample on the basis of a list of participants in Verdal and Stjørdal. In HUNT 2, the participation rate was strongly age dependent, with the highest participation in the age group 60–69, and lowest in the age group 20–29 [9]. To ensure acceptably balanced participation in the present study for both genders in all age groups, potential participants were selected from the list of HUNT 3 participants consecutively in the following order: man ≤50 years, man >50 years, woman ≤50 years, and woman >50 years.

Among participants in HUNT 3 living in Verdal and Stjørdal, a random sample of 563 persons got an invitation letter which included general information about the validation study, also informing that they would be contacted on telephone by our research assistants to give further information and to make an appointment for the personal interview. The list of persons who participated in Verdal and Stjørdal were received shortly before the time of the interview. Hence, due to lack of time, in case the research assistants were unable to get in contact despite of a minimum of two attempts they were instructed to call the next person on the list.

Headache diagnoses

For the validation study, a semi-structured face-to-face interview was performed by a total of eight neurologists with special training in headache. When indicated, a clinical examination was performed. Based on the information from the clinical interview, headache disorders were classified in accordance with the ICHD-II criteria [5], except for medication overuse headache (MOH) for which the revised version was used [7]. Up to three different headache types were diagnosed in each individual. Among the 220 individuals who reported headache during the last year [6], 117 subjects got two headache diagnoses, whereas 16 got three.

In the questionnaire-based HUNT 3 study, the diagnoses of migraine and TTH were based on information in Q2, and the diagnoses were mutually exclusive. The diagnosis of migraine was made according to four different sets of criteria listed in Table 2. The restrictive migraine criteria set was based on ICHD-II criteria [5], except that duration less than 4 h was accepted because it was not specifically asked for untreated headache attacks in Q2. We have previously reported that asking whether individuals had suffered from headache during the last year yielded high positive predictive value and high specificity for identifying individuals with migraine ≥1 day/month [6]. Thus, because this restrictive screening question was used, the validity of migraine ≥1 day/month was evaluated. For migraine with aura only visual disturbance was included in the criteria set, because a positive answer to the question regarding sensory symptoms prior to or during headache had a very low sensitivity. Self-reported diagnosis of migraine was also considered separately, because high specificity and positive predictive value of this statement were found in HUNT 2 [8]. In accordance with the HUNT 2 study, self-reported migraine was integrated in the liberal migraine criteria set. The HUNT 2 questionnaire had no question about pain intensity, and accordingly the criterion C was modified requiring only at least one of the following three characteristics: pulsating quality, unilateral location, or aggravation by physical activity [8]. The liberal migraine criteria used in HUNT 2 was also evaluated in the present HUNT 3 population.

Table 2 Different sets of criteria for the diagnosis of migraine based on information in the questionnaire

The questionnaire-based diagnosis of TTH was based on the ICHD-II criteria [5]. We have previously reported that very few subjects with infrequent TTH consider themselves as headache sufferers [6], and that a high positive predictive value and high specificity for identifying individuals with TTH ≥1 day/month were found among headache sufferers [6]. As a consequence of these findings, only the validity of TTH >1 day/month was evaluated.

Chronic headache was defined as headache more than 14 days per month. To fulfil the questionnaire-based diagnosis of MOH the participants had to report headache >14 days per month and use of analgesics four times per week or more during the last month.

Ethics

The present study was an integrated part of the HUNT 3 project which was approved by the Regional Committee for Ethics in Medical Research and the Norwegian Data Inspectorate.

Statistics

Demographic data were compared between participants and non-participants with independent samples t test for continuous variables and with the chi-squared test for categorical variables. Two-tailed estimations of significance were used, and the level of significance was set at P < 0.05.

Sensitivity, specificity and Cohen’s kappa statistics with 95% CI were calculated for different headache diagnoses based on information in Q2 using neurologists’ headache diagnoses as a gold standard.

Results

Participation rate in the clinical interview

Among the 563 potential participants (279 males), 171 were not reached by at least two telephone calls. A total of 297 out of the 392 persons contacted by telephone participated in the clinical interview (53% out of the total invited group) (Fig. 1).

Fig. 1

Diagram of the invited population according to type of participation

Among the 392 persons who answered on telephone, only 29 stated that they did not want to participate, whereas 66 wanted to participate, but were unable to come because they were out of town, had sick children, were busy in job, or they had forgotten the invitation.

Compared to the 266 non-participants, the 297 participants were older (mean age 52.3 vs. 48.6 years, P = 0.004) and slightly more likely to be men (51 vs. 47%, P = 0.37). No significant difference was found between participants and non-participants with regard to bodily pain lasting ≥6 months (40 vs. 36%, P = 0.30), self-reported good or very good health (75 vs. 79%, P = 0.25), or proportion of working individuals (76 vs. 82%, P = 0.12).

Response rate to the headache questionnaire

The first headache question in Q2 was answered by 543 out of 563 (96%) invited persons (256 non-participants and 287 participants). 108 (37.6%) of the participants suffered from headache. Among these, 105 (97%) answered self-reported type of headache, and 82 (76%) answered all headache questions. When considering agreement between questionnaire-based diagnoses and those made in the clinical interviews, individuals with incomplete information in the questionnaire were excluded from the analyses. These were: three with chronic headache (not reported headache frequency in Q2), two with MOH (not reported use of analgesics in Q2), 14 with migraine, and 25 with TTH.

Prevalence of headache among participants versus non-participants

The prevalence of headache sufferers did not differ significantly between non-participants and participants (overall 39.5 vs. 37.5%, P = 0.72; men 31.4 vs. 32.9%, P = 0.90; women 46.7 vs. 42.6%, P = 0.55). Similarly, no significant difference was found between participants and non-participants regarding prevalence of migraine (12.5 vs. 16.2%, P = 0.20), TTH ≥1 day per month (14.8 vs. 11.3%, P = 0.22), or chronic headache (3.4 vs. 1.9%, P = 0.28).

Reliability of the Q2 diagnoses

The mean interval between answering the Q2 and the validation interview was 50 days (95% CI 48–52 days; median 45 days range, 9–90 days).

The sensitivity, specificity, and the kappa statistic are shown in Table 3. Several diagnostic subtypes were evaluated, and the highest figures were found for headache suffering, chronic headache, and MOH.

Table 3 Sensitivity, specificity and kappa value of questionnaire-based headache diagnosis

Overall, the questionnaire-based diagnoses of migraine (MA or MO or both) had a sensitivity of 51%, a specificity of 95%, and a kappa statistics of 0.50. Considering those with migraine ≥1 day per month the figures changed to 58%, 91%, and 0.54, respectively. Correspondingly, the sensitivity, specificity, and kappa statistics of TTH ≥1 day per month were 96%, 69%, and 0.44.

Discussion

The agreement between the neurologists’ interview and the questionnaire-based diagnoses for headache suffering and chronic headache, including CTTH and MOH, was very good, whereas the agreement for migraine and TTH was more moderate.

Methodological considerations

Less than half of the invited adults (42%) responded to the headache questions in HUNT 3. It should be emphasised that our results are only valid for these responders. Furthermore, in the present study only 53% of the random sample of invited participants was interviewed. However, selective participation due to headache status seems less likely, because the prevalence of headache, migraine and TTH included were quite similar among participants and non-participants. This finding is of particular interest, because such a comparison between participants and non-participants can rarely be done.

The reported agreement between the neurologists’ interview and the questionnaire-based diagnoses may have been influenced by the number of excluded persons from the analyses because of incomplete information in the questionnaire. However, none of the participants were excluded for the analysis of headache suffering, only three for chronic headache, whereas the number of excluded persons was higher for migraine and TTH (14 and 25, respectively).

The agreement for MOH was good, although it was diagnosed differently in the interview and questionnaire. In the interview, the revised version from 2006 was used [7], accepting an intake of drugs like tripans, opioids, and combination medications ≥10 days per month on a regular basis of 3 months. In contrast, for the questionnaire-based diagnosis use of analgesics four times per week or more during the last month was accepted.

The fact that there was a mean time interval of 50 days between the questionnaire and the interview may imply that the frequency of headache attacks and the headache characteristics could have changed during this period, which may have reduced the agreement between responses in the questionnaire and clinical interview.

A main problem of studies based on self-administrated questionnaires is to correctly diagnose patients with the co-existence of two or more headaches, usually migraine and tension-type headache. Questions using the term “usually” regarding features of headache may not be ideal if one tries to make the respondent differentiate between different subtypes of headache.

Present and previous studies

The agreement between the questionnaire and clinical interview with respect to being a headache sufferer or not was better in the present study than in the validation study performed in HUNT 2 (kappa value 0.70 vs. 0.57) [8]. Similarly, the agreement was better for chronic headache (0.75 vs. 0.44). A probable reason for the better results this time is that the time span between the Q2 to the validation study was shorter (mean of 50 days) than in the HUNT 2 study (5–9 months) [8].

As in most previous population-based studies comparing questionnaire-based and interview-based diagnosis [8–13], a moderate agreement rate was found for migraine. A better agreement has been found in validation studies focusing on migraine patients recruited from specialist practice [1], or on individuals with self-reported migraine [14]. Our moderate agreement rate could in part be explained by the mean time span of 50 days between the questionnaire and interview, because it has previously been shown that key diagnostic features of migraine are not consistently reported or experienced over time. Another potential problem of the questionnaire was to identify patients with co-existence of migraine and tension-type headache, because some respondents with migraine may not keep the different subtypes clear when answering the questions. It may also be that many patients have somewhat atypical migraine (probable migraine), and in these patients the distinction between migraine and TTH will be difficult.

As only a moderate sensitivity was found for the questionnaire-based diagnosis of migraine and TTH, the questionnaire is not optimal for estimating prevalence of these headache types. In particular, the prevalence of migraine may be overestimated when diagnosing migraine by the questionnaire, because headache duration less than 4 h was accepted. However, the high specificity (>90) for the questionnaire-based diagnosis of migraine makes the questionnaire a valid tool to identify a population of individuals with migraine suitable for genetic studies.

The fact that the precision of the headache diagnoses in HUNT 3 seems to be as good as or even better than in HUNT 2 will enable comparisons of the results between the two studies, and will offer an exiting opportunity to explore possible causes and risk factors for headaches in a prospective study of large population.

Conclusion

The HUNT 3-questionnaire is a valid tool for diagnosing patients as headache sufferers, and with chronic headache, including chronic TTH and MOH. The more moderate agreement for migraine and TTH makes the questionnaire-based diagnoses of migraine and TTH suboptimal for determining the prevalence in the population. However, the high specificity of the questionnaire-based diagnosis of migraine, in particular for migraine with aura, makes the questionnaire a valid tool for diagnosing patients with migraine for genetic studies.