The FFQ is designed to measure the usual food intake of an individual over a defined period. It is easy and relatively inexpensive to administer and is therefore the preferred dietary assessment method for use in large-scale nutritional epidemiological studies(Reference Willett1). However, data from an FFQ do not represent the ‘true’ usual diet as such data suffer from random and systematic errors, which attenuate relative risk estimates in studies on diet and disease risk(Reference Kipnis, Midthune and Freedman2–Reference Thomas, Stram and Dwyer4). Since an FFQ is typically used to determine the relationship between foods or food groups and disease risk, and also to identify foods or food groups that contribute to specific nutrients of interest, it is important to evaluate the extent to which such questionnaires can measure true intakes. One approach to examining the performance of an FFQ is through a validation study in which assessment of individual diets by means of a questionnaire is compared with assessment using a more precise reference method(Reference Willett and Lenart5).
The Adventist Health Study-2 (AHS-2) is a prospective cohort study of over 96 000 adult Adventists in the USA and Canada. Over 25 000 cohort members are blacks of US and Caribbean descent; the remaining are of other races, mostly whites. The primary aim of the study is to relate usual dietary habits to cancer outcomes. The AHS-2 cohort is relatively unique for its wide range of dietary patterns compared with the general Western population. For example, 8 % are vegan, 28 % are lacto-ovo vegetarian, 16 % are semi- or pesco-vegetarian and 48 % are non-vegetarian. In addition, there exists large variation in the consumption of plant foods such as nuts, soya, other legumes and grains. Dietary exposure in AHS-2 is assessed primarily by means of an FFQ. The validation of nutrient intake estimates has been reported previously(Reference Jaceldo-Siegl, Knutsen and Sabate6). On average, energy-adjusted de-attenuated validity correlations were 0·60 in whites and 0·52 in blacks across fifty-one nutrients. In the present study, we sought to validate intake estimates of foods and food groups obtained using the AHS-2 FFQ by comparing them with intake estimates obtained using repeated 24 h dietary recalls (24HDR).
Methods
Calibration substudy design
The methods of recruitment and selection of calibration study participants have been described previously(Reference Jaceldo-Siegl, Knutsen and Sabate6). Briefly, participants in the calibration study were randomly selected from the parent cohort(Reference Akbar, Jaceldo-Siegl and Fraser7) by church and then by subject-within-church. Both calibration study and parent cohorts were similar with regard to distribution of age, gender, education and vegetarian status, although by design we included approximately equal numbers of blacks (n 461) and whites (n 550) in the calibration study. In addition, in both cohorts, over 65 % were women; whites were older (mean 60·6 (sd 13·9) years) than blacks (mean 55·6 (sd 12·7) years). Whites also had lower BMI (mean 26·4 (sd 5·7) kg/m2) compared with blacks (mean 28·8 (sd 6·4) kg/m2)(Reference Jaceldo-Siegl, Knutsen and Sabate6).
We obtained a set of three 24HDR (one Saturday, one Sunday and one weekday intake) from each participant during the first 2 months of the calibration study, which was then repeated approximately 6 months later. The aim was to obtain two sets of recalls (a total of six 24HDR) from each participant. During the 6-month interval, participants completed a self-administered FFQ, which queried respondents about their food intake during the previous year. The study was approved by the institutional review board of Loma Linda University and all participants gave written informed consent.
Dietary assessment
The AHS-2 FFQ is a quantitative and comprehensive instrument originally designed to include foods commonly consumed by US Adventists. The questionnaire was later revised to reduce the respondents’ fatigue and to accommodate foods specific to black Adventists of US and Caribbean origin(Reference Akbar, Jaceldo-Siegl and Fraser7). All versions of the FFQ consist of two major sections. Across all versions, the first section is a food list that includes 130–141 items of fruits, vegetables, legumes, grains, oils, dairy, fish, eggs and beverages, and the second consists of sixty-three to seventy-nine items of commercially prepared products, such as dietary supplements, dry cereals and vegetarian protein products that require respondents to examine food labels. Frequency categories range from never or rarely to ≥6 servings/d and vary with food type to allow respondents to define their daily intake with greater specificity. Portion sizes (e.g. cup, tablespoon, slice, patty) include a given standard, 
or less and 1
or more of the standard serving. Pictures of common foods or beverages typically served together are included with the questionnaire to assist participants in estimating portion sizes. The questionnaire was sent to each participant, completed at home and then mailed back to AHS-2. Respondents were asked to report on their intake over the previous 1 year. Upon receipt of the questionnaire, study personnel reviewed the FFQ for completeness and, when necessary, followed up by telephone to clarify any ambiguous or incomplete information.
The 24HDR was administered unannounced and information was obtained by telephone. Each participant was provided a two-dimensional food portion visual (The Nutrition Consulting Enterprises, Framingham, MA, USA) to assist with portion size estimates. Trained research dietitians used standard probes and a multiple-pass approach method to collect detailed information on all foods, beverages and supplements consumed by each participant during the previous 24 h. All recall interviews were digitally recorded for subsequent quality check. Later, an experienced research dietitian evaluated randomly selected recall interviews (∼5 %) and compared them with the recording, as a quality control measure.
Recall and FFQ data were entered using the Nutrition Data System for Research version 4·06 or 5·0 (NDS-R, Nutrition Coordinating Center, Minneapolis, MN, USA); the analytic data used in the present study were based on the NDS-R 2008 database. Information on foods not found in the NDS-R database was obtained from the US Department of Agriculture, from individual manufacturers and from the Caribbean Food and Nutrition Institute. Considerable attention was given to creating recipes for home-cooked vegetarian dishes (n > 500), homemade and commercial soya and nut milks (n > 180) and for commercial meat analogues (n 309) that were frequently consumed by our study population. For the latter we contacted manufacturers or worked with a senior food technologist with experience in this industry in order to create recipes.
Calculation of food intake
Foods from the FFQ and 24HDR were categorized into forty-seven foods or food groups. Frequency categories from the FFQ were converted to daily intake; thus, never or rarely was assigned a weight of 0; 1–3 servings/month was assigned a weight of 0·067; 1 serving/week was assigned 0·143; 2–4 servings/week was assigned 0·429; 5–6 servings/week was assigned 0·786; 1 serving/d was assigned 1; 2–3 servings/d was assigned 2·5; ≥4 servings/d was assigned 4·5 and ≥6 servings/d was assigned a weight of 6·5. Portion size categories were assigned weights of 1 for standard serving, 0·5 for 
or less and 1·5 for 1
or more. Food intake estimates (in g or kJ) from the FFQ data were calculated using the product-sum method(Reference Willett1), except where noted. FFQ with estimated total energy intake of <2093 kJ (500 kcal) or >18 833 kJ (4500 kcal) were excluded from the analyses.
The dietary habits of Adventists are often rather different on Saturdays, Sundays and typical weekdays. Thus, within each of the two sets of 24HDR, each day was weighted appropriately to produce a synthetic week (Saturday intake + Sunday intake + 5 × weekday intake) and then divided by 7 to obtain the mean daily food intake estimate. For those who completed six 24HDR (n 950), the mean daily intake was calculated by dividing the total of the two synthetic weeks by 14. Food intake estimates were calculated for each of the forty-seven foods by summing the energy content (kJ) or weight (g) of each item contributing to that food or food group.
Statistical analysis
Mean, sd and median food group intakes in grams were calculated for the FFQ and 24HDR data.
An important feature of dietary data is that some foods or food groups may contain a significant proportion of zero intakes. This leads to the need for transformations in correctional analyses and for some special issues relating to both energy adjustment and de-attenuation of validity correlation coefficients for within-person errors in the recalls. The zeros usually exacerbate skew and we found that most of these variables have a positive skew; correlations are thus usually improved after log transformation. Because of the zeros in the data, we used log(x + 1), as then zeros stay as zeros after transformation.
Energy adjustment using the residual method is attractive as the resulting values are independent of energy intake. However, the disadvantage of the usual approach is that different participants, who had zero values before energy adjustment, will often end up with different non-zero values after adjustment, a non-intuitive result. Indeed, a small number may have negative values even after adding back the mean value to the residual, as is commonly done.
Thus, we performed a partitioned energy adjustment in which data that are initially zero remain zero and energy adjustment using the residual method is performed on only non-zero data, both for the questionnaire and for each synthetic week of the recalls separately. A log(x) transformation is applied to non-zero data before energy adjustment. After the energy adjustment, if y = energy-adjusted residual + mean of log(x), we finally take log[exp(y) + 1] to be the energy-adjusted non-zero data. These values are then combined with the zero data points, all of which are now on the same logarithmic scale, and the non-zero data are thus energy adjusted.
An attempt was made to match the exact food reported using recalls with the information obtained through the FFQ, although in many instances several different but related foods from the recalls were found to be closely matched to a particular FFQ food or food group. We used two approaches to assess the validity of intake estimates from the FFQ: (i) cross-classification by quartile to measure agreement between the two dietary assessment methods; and (ii) de-attenuated correlation coefficients. First, we calculated the proportion of participants with zero intake of a specific food from both the FFQ and recalls. Where participants had a non-zero value in either method, we calculated log-transformed energy-adjusted food intake estimates as described above and then classified the intake into quartiles. We then compared the classification of quartiles of intake between the FFQ and recalls by calculating the proportion of participants who had exact agreement (EA), the proportion of participants whose intakes deviated by one or two quartiles and those whose intakes were grossly misclassified (disagreement by three quartiles). We also calculated the proportion of participants with EA by quartile in which we included both zero and non-zero values.
De-attenuated correlation coefficients between questionnaire and recall values were estimated by removing the effects of random within-person errors in the recall data. The log-transformed energy-adjusted values for each of the two synthetic weeks of recalls were used in this procedure. Again, a partitioned approach was used, arguing that where both weeks of recalls take a zero value (all six recalls are zero) the true within-person variance is zero, or close to zero, and no within-person error adjustment is required. Partitioning the correlation coefficient to zero and non-zero (nz) data (defined as at least one recall week being non-zero) and estimating the within-person variance for only non-zero data lead to the following formula correlating questionnaire (Q) and recall (R) data
where r 0 = uncorrected correlation, r 1 = corrected correlation, P nz = proportion of non-zeros and R nzw = within-person variance of non-zeros from recalls.
A bias factor was calculated as the proportional bias ((uncorrected−corrected)/corrected) that would be observed in a regression coefficient if the uncorrected FFQ estimate were used as the independent variable, where in fact the mean of a large number of recall estimates is the truth. The bias factor is related to validity, as the biased regression coefficient results from dietary measurement error. We calculated 95 % CI for all validity coefficients using bootstrap re-sampling and the BCa method(Reference Efron and Tibshirani8). Analyses were performed using the SAS statistical software package release 9·2 (SAS Institute Inc., Cary, NC, USA) and R version 2·10·1 (http://www.r-project.org/).
Results
The untransformed mean and median intake estimates of thirty-four of forty-seven foods or food groups were higher in the data obtained from the FFQ compared with those from recall data (Table 1). Food groups in which estimates obtained from recalls were higher than those from the FFQ in both races included soya milk, meat, poultry, fish and all types of cheese. However, for vegetables, fruits and beverages the FFQ yielded higher results consistently. Food groups in which intakes were higher in whites than in blacks (from both FFQ and recalls) included berries, dried fruits, avocado, tomatoes, potatoes, legumes, all vegetables, tofu, soya milk, margarine, butter, peanut butter, tree nuts, all nuts and dairy foods, as well as breads and dry cereals made from whole grains. Blacks on the other hand showed higher intakes of cruciferous vegetables, vegetarian meat products, peanuts, meat, poultry, fish, total whole grains and total refined grains. For beverages, whites consumed more soya milk, drinking water and coffee, whereas blacks had higher intakes of fruit juice, tea and soda according to both assessment methods.
Table 1 Food and food group intakes from the FFQ and 24HDRFootnote † by race in Adventist Health Study-2 calibration study participants (2003–2008)
24HDR, 24 h dietary recalls.
* Intake from the FFQ was significantly different in blacks compared with whites at P < 0·05 (test for the equality of means across race).
** Intake from the 24HDR was significantly different in blacks compared with whites at P < 0·05 (test for the equality of means across race).
† Average of two sets of three weighted 24HDR.
The extent of agreement between the FFQ and recalls according to quartiles of intake for each food group is presented in Table 2. The proportion of EA when all participants were included ranged from 29 % (white bread) to 87 % (processed meat) in whites and from 25 % (onion) to 88 % (coffee) in blacks. We identified seven uncommonly eaten food groups the intakes of which were equal to zero in >50 % of participants, according to both FFQ and recalls. These included such foods as avocado (blacks), tree nuts (blacks), unprocessed and processed meat (both races), all red meat (whites only), fish (whites) and coffee (both races). Excluding participants whose intake of any food group was zero in both FFQ and recalls, the proportion of EA among the non-zero consumers ranged from 5 % (processed meat) to 53 % (all dairy food) in whites and from 7 % (processed meat) to 42 % (all dairy food) in blacks. Among the non-zero consumers of any food group, the proportion of gross misclassification (GM) by FFQ (disagreement by three quartiles) ranged from 1 % (processed meat, total meat, poultry and fish and total dairy) to 11 % (peanuts) in whites and from 1 % (processed meat and total meat, poultry and fish) to 15 % (peanuts) in blacks. Of the forty-seven food groups, one in whites and six in blacks had >10 % GM.
Table 2 Agreement between the categorization of food intake estimated from the FFQ and 24HDR by race in Adventist Health Study-2 calibration study participants (2003–2008)Footnote †
24HDR, 24 h dietary recalls; GM, gross misclassification.
† Columns 2–6 in both ethnic groups should add to 100 % (aside from rounding errors).
‡ The proportion of participants with zero intake in both the FFQ and 24HDR was first determined, and then intakes of the remaining participants with non-zero values in either dietary assessment method were categorized into quartiles of intake.
§ The proportion of participants with exact agreement by quartile in the FFQ and 24HDR when all zero and non-zero values were included.
∥ GM is disagreement by three quartiles.
All uncorrected and de-attenuated correlation coefficients reported for each of the forty-seven foods or food groups were energy adjusted (Table 3). De-attenuated validity correlations of all foods or food groups averaged 0·59 in whites and 0·43 in blacks. Of the forty-seven foods or food groups, forty-three in whites and thirty-three in blacks had correlations >0·40. Validity correlations in whites for foods from animal sources were 0·64 (average for red meats), 0·76 (poultry), 0·53 (fish), 0·71 (average for dairy) and 0·64 (eggs). In blacks, these were 0·59 (average for red meats), 0·77 (poultry), 0·57 (fish), 0·54 (average for dairy) and 0·52 (eggs). For all fruits, all vegetables, soya foods, and all nuts and seeds, de-attenuated correlations were 0·68, 0·66, 0·64 (average of vegetarian meats, tofu and soya milk) and 0·58, respectively, in whites. These values were 0·52, 0·41, 0·49 (average of vegetarian meats, tofu and soya milk) and 0·47, respectively, in blacks. Foods with the lowest validity correlations were drinking water in whites (r = 0·14) and onion in blacks (r = 0·15). The food group ‘all meat, poultry, fish’ had the highest validity in both races: r = 0·86 (95 % CI 0·82, 0·90) in whites and r = 0·85 (95 % CI 0·79, 0·89) in blacks. On stratification by gender (results not shown), average validity correlations in whites were 0·61 in men and 0·60 in women. In blacks, these were 0·50 in men and 0·48 in women. Bias factor (Table 3) averaged −0·49 in whites and −0·60 in blacks. In thirty-nine of the forty-seven assessed foods or food groups, the absolute value of the bias factor was larger in blacks than in whites. Food groups with bias factors <−0·70 (seven in whites and thirteen in blacks) included such foods as onions, peanuts, tree nuts, white bread and drinking water.
Table 3 Energy-adjusted validity correlationsFootnote † comparing log-transformed data from an FFQ to that from repeated 24HDR: Adventist Health Study-2 calibration study (2003–2008)
† Uncorrected and de-attenuated correlations were energy adjusted.
‡ Corrected for attenuation due to within-person error in the recalls.
§ Bias factor is the proportional bias in a univariate regression if the uncorrected FFQ variable is the independent variable.
Discussion
In this report, we provide an evaluation of the performance of a comprehensive quantitative FFQ in estimating intakes of forty-seven foods and food groups in a representative sample of white and black members of the AHS-2 cohort. Cross-classification by quartiles produced proportions of EA and GM in the FFQ and 24HDR that were similar in both whites and blacks. In both races combined, estimates from the FFQ in forty-three of forty-seven foods or food groups were moderately to highly correlated with the 24HDR; however, validity correlations were higher in whites compared with blacks.
The sample size of the present food validation study (n 1011) is relatively large compared with those of most other studies. Other reports had sample sizes that ranged from 104(Reference Bohlscheid-Thomas, Hoting and Boeing9) to 197(Reference Pisani, Faggiano and Krogh10) to 246(Reference Johansson, Hallmans and Wikman11, Reference Toft, Kristoffersen and Ladelund12). Food validation studies have typically compared validity by gender(Reference Pisani, Faggiano and Krogh10–Reference Ocke, Bueno-de-Mesquita and Pols14). One study assessed the influence of other personal characteristics on the relative validity of food intake estimates(Reference Marks, Hughes and van der Pols15). Because both blacks and whites were included in the cohort, the present study also compared relative performance of the FFQ by race.
That the mean estimates of thirty-four of forty-seven foods or food groups were higher in the data obtained from the FFQ than in the 24HDR data is not surprising, given the evidence of measurement bias in FFQ assessments, since respondents may overestimate their frequency of actual intake when provided with a long list of foods to recall(Reference Kristal, Vizenor and Patterson16). When the estimates of such foods as soya milk, meat, poultry, fish and all types of cheese were in fact higher in recalls than in the FFQ, it was attributed to a possible result of the questionnaire design. For example, soya milk was not included in the food list but rather queried as an open-ended question (and it was the last item) in the FFQ that required the respondent to write the brand name of the soya milk and then estimate the frequency and portion size consumed. This may have underestimated the intake values recorded in the FFQ as respondents may have perhaps found the format burdensome and may have chosen to skip the item. On the other hand, providing a list of soya milk options may have elicited a response (rather than an omission). Our finding that estimates of meat, poultry and other foods of animal origin were lower in the FFQ than in the 24HDR is similar to those of other cohorts in Europe(Reference Bohlscheid-Thomas, Hoting and Boeing9, Reference Pisani, Faggiano and Krogh10, Reference Ocke, Bueno-de-Mesquita and Pols14), Shanghai(Reference Shu, Yang and Jin17, Reference Villegas, Yang and Liu18) and the USA(Reference Feskanich, Rimm and Giovannucci19, Reference Salvini, Hunter and Sampson20). This underestimation by the FFQ may be the consequence of a combination of factors. First is the provision of a relatively short list of such foods in the FFQ, or the possibility that the standard portion size provided in the FFQ is lower than the actual average portion consumed in this population. Either of these conditions could produce lower estimates using the FFQ compared with the 24HDR. Another likely explanation is social desirability bias(Reference Cade, Burley and Warm21). Respondents may have tended to underestimate the intakes of these foods, particularly in this population in which plant-based diets are encouraged among church members.
Assessing intake estimates according to categorization by quartiles provides information on the degree of agreement between the FFQ and the reference measure. In the present study, proportions of EA and GM between the FFQ and 24HDR were similar in both blacks and whites (range: 25–88 % EA and 1–15 % GM). Although eleven foods or food groups had an EA of <35 %, the majority had good agreement, particularly those with a focus on the Adventist lifestyle (e.g. avoidance of meats and coffee, or consumption of plant-based foods such as soya and tree nuts). For example, the proportion of EA for meat and poultry was ≥60 %, whereas in other studies EA for these foods is typically between 30 % and 40 %(Reference Marks, Hughes and van der Pols15, Reference Shu, Yang and Jin17, Reference Villegas, Yang and Liu18). Avocado, tree nuts, fish and coffee were also among the foods that had high proportions of EA (>60 %) and relatively low proportions of GM (<7 %). We note that more than 50 % of respondents reported zero intakes of these foods according to both FFQ and 24HDR; thus, zeros contributed to the relatively high proportion of EA in estimating these uncommonly eaten foods. The additional information gained from isolating ‘zero’ intakes is the identification of foods or food groups that are rarely or commonly consumed in the population. Interestingly, foods or food groups commonly consumed (proportion of zeros <5 %), or perhaps those that were consumed in many forms or included in mixed dishes, such as onions, appeared to have lower performance compared with rarely consumed foods.
One of the unique features of the AHS-2 cohort is the diversity in dietary habits among its members, ranging from vegans (who consume meat, fish and dairy foods <1 serving/month) to lacto-ovo vegetarians (who consume meat and fish <1 serving/month and dairy foods 1 serving/month to 1 serving/week) to non-vegetarians (who consume meat or fish ≥1 serving/week). We anticipate that future studies investigating diet–disease relationships in this population will use as exposure variables those foods and food groups that are related to these dietary patterns. Therefore, it is important that the questionnaire has relatively good facility in assessing such foods or food groups. FFQ estimates of red meat and poultry, as well as all meat, poultry and fish combined, were highly correlated (r > 0·7) with 24HDR estimates in both blacks and whites, although processed meat had low-to-moderate correlations. By comparison, validity correlations for red meat and poultry reported by others ranged from 0·27 to 0·65(Reference Bohlscheid-Thomas, Hoting and Boeing9, Reference Johansson, Hallmans and Wikman11, Reference Marks, Hughes and van der Pols15, Reference Salvini, Hunter and Sampson20, Reference Khani, Ye and Terry22); in addition, in one study in men, the de-attenuated correlation for processed meat was 0·83(Reference Feskanich, Rimm and Giovannucci19). Validity correlations for most dairy foods in AHS-2 were moderate to high, consistent with the results from other studies. For individual vegetables, fruits and nuts, validity correlations on average were low to moderate as was observed in other cohorts(Reference Bohlscheid-Thomas, Hoting and Boeing9–Reference Johansson, Hallmans and Wikman11, Reference Ocke, Bueno-de-Mesquita and Pols14, Reference Khani, Ye and Terry22), although validity was relatively high when grouped together. The range of reported validity correlations of soya products in men(Reference Villegas, Yang and Liu18) and women(Reference Shu, Yang and Jin17) in Shanghai was between those observed in the present study.
For foods in which the uncorrected regression coefficient was high, bias factor tended to be high. Foods with poorer validity generally were associated with severe biases. For example, biases associated with the uncorrected correlation for water (de-attenuated validity correlation of 0·16 in blacks and 0·14 in whites) had a factor of −0·77 in blacks and −0·84 in whites. On the other hand, the high uncorrected correlation for all meat, poultry and fish (de-attenuated validity of 0·85 in blacks and 0·86 in whites) biased regression coefficients downwards by only 11 % and 15 %, respectively.
When comparing FFQ validity according to race, our results from cross-classification by quartiles were similar in both races. De-attenuated energy-adjusted correlations, however, were higher in whites than in blacks. That errors were greater overall in blacks than in whites may be partly due to their unfamiliarity with research studies, as well as due to a lack of awareness of the type and amount of foods consumed, or due to the lower educational attainment among blacks on average than among whites(Reference Kristal, Feng and Coates23, Reference Mayer-Davis, Vitolins and Carmichael24).
Energy adjustment and de-attenuation of correlation coefficients will produce the best estimates of the desired quantities when there is a consistent rationale for their use. We argue that in certain situations involving zero intakes, which were common in our data, the rationale for these adjustments does not exist or is unclear. This motivated the use of partitioned methods of energy adjustment and de-attenuation of correlation coefficients that have, to our knowledge, not been used by others.
In summary, data from the AHS-2 FFQ have comparatively good validity for many foods and food groups, although not for all. For these as well as for foods that have relatively poor validity, use of biomarker-guided or traditional regression calibration(Reference Fraser and Shavlik25–Reference Spiegelman, Schneeweiss and McDermott27) to correct measurement error will allow us to interpret diet–disease analyses more clearly.
Acknowledgements
The authors gratefully acknowledge support from the National Cancer Institute (5RO1 CA 094594). None of the authors have conflicts of interest. Authors' contributions: study design, G.E.F.; coordination of data collection, K.J.-S., J.S., S.F.K., E.H., W.L.B., R.P.H., T.L.B., H.B.; data analyses, J.F., data interpretation, K.J.-S., J.S., S.F.K., G.E.F.; writing of the manuscript, K.J.-S., G.E.F.; editing of the manuscript, J.F., J.S., S.F.K.
