This post is co-authored with Joachim De Weerdt, Kathleen Beegle, and John Gibson
Global numbers on hunger prevalence are widely followed, not least because halving the proportion of hungry people from 1990 to 2015 is one of the prominent Millennium Development Goals (and achievements under this goal seem to be quite slight till date). The standard approach to these numbers – one that combines mean per capita food consumption derived from national Food Balance Sheets (FBS) with a survey based measure of the distribution of caloric intake – faces several measurement challenges; an alternative that relies on household consumption surveys for direct estimates of hunger has been increasingly promoted. Unfortunately our findings from a recent working paper raise questions about the cross-country validity of both approaches as currently practiced.
In both the standard approach (the FBS method) and the consumption survey approach that solely relies on Household Consumption and Expenditure Surveys (HCES), the use of household survey data is fundamental. The FBS method utilizes the caloric variance estimated across households by survey while the HCES method measures hunger directly as a function of the observed total household food consumption in relation to estimated household caloric need. One challenge to these approaches, however, is that the design of household consumption surveys varies widely around the world with respect to numerous dimensions. These dimensions include:
(a) the method of data capture – which is typically either diary or a recall questionnaire
(b) the reference period over which consumption is measured – which can vary from one day to one week to one month
(c) the degree of commodity detail – which can range from a handful of aggregate commodities to over 400 relatively detailed commodities.
We previously blogged about our consumption survey experiment in Tanzania that found large and significant differences in recorded total household consumption solely due to the effect of survey design. Now our new working paper studies the implications of survey design for hunger measurement. And it turns out that hunger estimates are even more sensitive to survey design than overall consumption because most of the design differences in global practice are due to differences in food consumption measurement. For example, Dupriez and co-authors draw on surveys from over 100 countries to find that, while the modal recall period is 7-days, a 7-day recall is only used in 31% of the surveys. Non-food consumption measurement varies a good bit less around the world.
Our study explores the net effect of reporting error arising from 7 different very common designs of consumption survey, including one that we see as a survey “gold standard” – an intensively supervised personal diary with detailed cross-checks across household members. Our study is a randomized within-village experiment with good covariate balance that extends over an entire calendar year, so the results should not be subject to seasonal or community level confounders. The only difference in population hunger numbers should be due to the particular survey design administered. And golly what a difference it is.
The estimated daily per capita (kilo)calorie intake ranges from 1793 to 2677 depending on the survey module. As a consequence hunger prevalence ranges from 18.8 to 68.4 percent (see Table 1). This wide range in prevalence translates to a difference of more than 23 million people in Tanzania, a country with a population of 45 million according to the 2012 census.
And it’s not just prevalence estimates - the targeting of the hungry will also be influenced by survey design. The hunger profiles suggested by each module differ as the hunger indicator interacts with household size, wealth, education, and other measures. For example, vis-à-vis the “gold standard”, recall surveys underestimate hunger as the household grows richer but overestimate hunger as the household increases in size.
So far we’ve focused on the HCES method but the global hunger estimates derived from the FBS method are also subject to the vagaries of cross-country survey design. Remember that, in this method, the variance of calorie consumption is measured through HCES even though the mean is taken from the National FBS. Now there are numerous grounds to question the accuracy of the balance sheets including the degree to which they capture national post-harvest stores and losses as well as the accuracy of their root crop yields – but that is beyond the scope of this post (and the related paper). However it is clear that survey design variation will also affect the FBS derived estimates. In Figure 1 we apply the estimated variances from the seven different modules to the mean caloric intake of our “gold standard”. The resultant hunger prevalence estimates range from 17 to 28 percent on the basis of survey design driven differences in the estimated second moment of the calorie distribution.
We believe the survey approach to hunger measurement holds promise, especially since the availability of HCES surveys around the world are increasing every year and collect a lot of important covariate information. However much more work remains to improve accuracy, most notably: (a) harmonizing survey design around the globe and (b) devising creative ways to validate self-reported consumption against a known benchmark.
Global numbers on hunger prevalence are widely followed, not least because halving the proportion of hungry people from 1990 to 2015 is one of the prominent Millennium Development Goals (and achievements under this goal seem to be quite slight till date). The standard approach to these numbers – one that combines mean per capita food consumption derived from national Food Balance Sheets (FBS) with a survey based measure of the distribution of caloric intake – faces several measurement challenges; an alternative that relies on household consumption surveys for direct estimates of hunger has been increasingly promoted. Unfortunately our findings from a recent working paper raise questions about the cross-country validity of both approaches as currently practiced.
In both the standard approach (the FBS method) and the consumption survey approach that solely relies on Household Consumption and Expenditure Surveys (HCES), the use of household survey data is fundamental. The FBS method utilizes the caloric variance estimated across households by survey while the HCES method measures hunger directly as a function of the observed total household food consumption in relation to estimated household caloric need. One challenge to these approaches, however, is that the design of household consumption surveys varies widely around the world with respect to numerous dimensions. These dimensions include:
(a) the method of data capture – which is typically either diary or a recall questionnaire
(b) the reference period over which consumption is measured – which can vary from one day to one week to one month
(c) the degree of commodity detail – which can range from a handful of aggregate commodities to over 400 relatively detailed commodities.
We previously blogged about our consumption survey experiment in Tanzania that found large and significant differences in recorded total household consumption solely due to the effect of survey design. Now our new working paper studies the implications of survey design for hunger measurement. And it turns out that hunger estimates are even more sensitive to survey design than overall consumption because most of the design differences in global practice are due to differences in food consumption measurement. For example, Dupriez and co-authors draw on surveys from over 100 countries to find that, while the modal recall period is 7-days, a 7-day recall is only used in 31% of the surveys. Non-food consumption measurement varies a good bit less around the world.
Our study explores the net effect of reporting error arising from 7 different very common designs of consumption survey, including one that we see as a survey “gold standard” – an intensively supervised personal diary with detailed cross-checks across household members. Our study is a randomized within-village experiment with good covariate balance that extends over an entire calendar year, so the results should not be subject to seasonal or community level confounders. The only difference in population hunger numbers should be due to the particular survey design administered. And golly what a difference it is.
The estimated daily per capita (kilo)calorie intake ranges from 1793 to 2677 depending on the survey module. As a consequence hunger prevalence ranges from 18.8 to 68.4 percent (see Table 1). This wide range in prevalence translates to a difference of more than 23 million people in Tanzania, a country with a population of 45 million according to the 2012 census.
And it’s not just prevalence estimates - the targeting of the hungry will also be influenced by survey design. The hunger profiles suggested by each module differ as the hunger indicator interacts with household size, wealth, education, and other measures. For example, vis-à-vis the “gold standard”, recall surveys underestimate hunger as the household grows richer but overestimate hunger as the household increases in size.
So far we’ve focused on the HCES method but the global hunger estimates derived from the FBS method are also subject to the vagaries of cross-country survey design. Remember that, in this method, the variance of calorie consumption is measured through HCES even though the mean is taken from the National FBS. Now there are numerous grounds to question the accuracy of the balance sheets including the degree to which they capture national post-harvest stores and losses as well as the accuracy of their root crop yields – but that is beyond the scope of this post (and the related paper). However it is clear that survey design variation will also affect the FBS derived estimates. In Figure 1 we apply the estimated variances from the seven different modules to the mean caloric intake of our “gold standard”. The resultant hunger prevalence estimates range from 17 to 28 percent on the basis of survey design driven differences in the estimated second moment of the calorie distribution.
We believe the survey approach to hunger measurement holds promise, especially since the availability of HCES surveys around the world are increasing every year and collect a lot of important covariate information. However much more work remains to improve accuracy, most notably: (a) harmonizing survey design around the globe and (b) devising creative ways to validate self-reported consumption against a known benchmark.
Join the Conversation