The Cover Up: Complicity in Rwanda’s Lies

Recently the World Bank published a paper on poverty in Rwanda. The aim of the paper was to deal with the debate started in 2015 by Filip Reyntjens, and which continues on roape.net, on the reliability of Rwandan poverty statistics. Despite the objectives, when properly calculated, the evidence presented by the World Bank actually strongly supports the claim that poverty has increased in Rwanda. Yet the selective and even misleading presentation of supporting empirical evidence by the World Bank is, to say the least, disturbing. Our Rwanda experts ask if the World Bank is guilty of a worrying level of leniency and incompetence, or outright complicity in the manipulation of Rwanda’s official statistics.

In September 2018, the World Bank published a paper entitled ‘Revisiting the Poverty Trend in Rwanda 2010/11-2013/14’, the stated aim of which was to resolve [i.e. shut down?] the debate initiated three years earlier by Filip Reyntjens (2015), and followed up by several roape.net blog posts, about the reliability of Rwandan official poverty statistics.

In the World Bank’s own words, the ‘unfounded claims about stagnation in poverty (…) creat[e] confusion and muddy the primary purpose of welfare estimation and poverty monitoring, limiting timely and relevant policy making’ (World Bank 2018, p.2). This candid introduction has the merit of clarifying from the outset that the chief concern of the World Bank in this matter is to shield policy-makers from the confusing interference of prying academics. Their paper , aims, not to check or question the National Institute of Statistics of Rwanda’s (NISR) results, but to provide an ex-post theoretical rationale for them, so to let policy-makers get on with their ‘primary purpose’ of making ‘timely’, if not necessarily ‘relevant’ policy. Yet relevance, of course, requires robust evidence, which in turns requires academic scrutiny.

In so doing, the World Bank has effectively managed to write an academic paper in reverse: the findings (i.e. NISR’s claimed poverty reduction) are defined at the start of the paper, and assumed, as a matter of faith, to hold true. The World Bank then proceeds to construct a theoretical framework around those findings to show that it is hypothetically possible to arrive at conclusions that are consistent with NISR’s official poverty statistics. The limited empirical evidence that is presented invariably supports the pre-defined conclusions and is presented without any robustness checks or even basic caveats, and sometimes even with serious mistakes. The considerable body of adverse empirical evidence that has been generated in various blogs and papers is simply ignored or assumed away without further justification.

By carefully reviewing each of their claims and correcting the  mistakes and omissions in the World Bank’s paper, we will show that the World Bank’s paper probably will help to settle this debate, but possibly not in the way that the World Bank intended.

The main contribution of the World Bank’s paper is to lay out in detail the various steps involved in NISR’s poverty estimation. This makes it easier to understand NISR’s decision-making process, and to identify precisely the sources of discrepancies between the various poverty estimations. In particular, the World Bank clarifies that the NISR poverty statistics were produced in two stages. This blogpost will look at each of NISR’s estimation stages in turn, before looking at the additional supporting evidence presented by the World Bank in support of NISR’s claims.

Original NISR Poverty Estimates

In the first stage of poverty estimations, the NISR produced official poverty statistics (NISR 2015), which were based on a semi-normative minimum consumption basket and concluded that 39.1% of the Rwandan population lived below the poverty line (down from 44.9% four years earlier). The key step in this poverty estimation was step five: ‘In the last step, to arrive at the “minimum” cost of obtaining the “defined” basket, higher cost per calorie items are replaced by lower cost per calorie items in their respective meta food category’ (World Bank 2018, p.7). The crucial detail left out in the World Bank’s presentation is that it is this fifth step that allowed NISR to lower the EICV4 poverty line by close to 20%, and crucially, it is this step alone that accounts for the 6 percentage points decrease in poverty between EICV3 and EICV4 in the NISR 2015 poverty report (EICV is the acronym for Enquête Intégrale sur les Conditions de Vie des ménages, or Integrated Household Living Conditions Survey in English). Without this fifth step, NISR would have concluded that poverty increased by between 5 and 7 percentage points between 2010 and 2014.

Since the poverty line used in 2010 did not include this fifth step, it is not possible (in fact it would be preposterous, even by the World Bank’s own standards) to compare the official poverty rate produced in 2010 with the official poverty rate published in 2014. The World Bank euphemistically hints at this incomparability when they state that ‘Condition (ii) – comparability of COLIs – and condition (iii) – the update of poverty lines, however, may not hold.’ In plain language, the key reason why the comparison of official poverty rates in EICV3 and 4 ‘may not hold’ is that the two surveys use different types of poverty lines (an average empirical four-step poverty line in EICV3 and a minimum semi-normative five-step poverty line in EICV4).

The lack of rigor in the World Bank’s ‘may not’ conclusion is surprising, given the apparent effort that has been put into ensuring mathematical formalization of the arguments. In fact, the vagueness of the ‘may not’ conclusion effectively defeats the purpose of opting for mathematical formalization to present this particular argument, since the main aim of formalization is to generate precise and rigorous statements that can be empirically verified as true or false. If they insist on formalization, the World Bank should go all the way in this effort to establish definitively whether EICV3’s average four-step poverty line and EICV4’s five-step minimum poverty line are comparable (and they aren’t!) Alternatively, the World Bank could abandon their formalization effort, which in this case appears aimed at excluding non-experts from the discussion rather than facilitating expert verification, and simply explain in layman’s terms why they think that an average four-step consumption basked ‘may’ in some cases be comparable with a minimum five-step consumption basket (yet they won’t be able to!)

The key question here is whether the change in the type of poverty line is, itself, driving the poverty reduction (i.e. whether the poverty reduction can be seen as an artifact of the methodological change), or whether such poverty reduction would have occurred regardless of what specific methodological choices had been made in EICV3 and EICV4. This is a question that the World Bank could, for instance, have tried to answer by carrying out robustness checks to see to what extent more or less comparable consumption baskets generate similar poverty trends in Rwanda. Alternatively, if pressed for time, they could just have referred to the evidence already presented in Filip Reyntjens 2015 article and the ROAPE blogs, which already carried out such tests.[1]

Of course, the more sensitive the poverty estimates are to such changes, the more conservative we should be about introducing changes that could artificially ‘engineer’ a poverty reduction merely through the choice of an advantageous consumption basket. Checking the empirical robustness of NISR’s results would, arguably, have been a more valuable contribution to this debate than the World Bank’s incomplete attempt to provide a theoretical justification for why fundamentally different types of poverty lines ‘may’ sometimes hypothetically be comparable.

For, ultimately, the question that needs to be answered in this debate is an empirical one and not a theoretical one. What we want to know is not whether it is possible to find a mathematical formula that is theoretically compatible with the numbers that NISR generates, but whether or not those numbers accurately represent the living conditions of the individuals whose living conditions they claim to represent. To date, the only independent information available to answer this question stems from qualitative data, which appear to quite consistently show that conditions have not improved for significant sections of the population (see Ansoms et al., 2017). The onus should therefore be on quantitative researchers to test and explain if and why their quantitative data are leading to different conclusions, not to merely show that it ‘may’ theoretically be possible to generate numbers that appear to look like the official NISR statistics.

Revised NISR Poverty Estimates

The second step in NISR’s poverty estimation was published 6 months later (NISR 2016), in response to the challenges raised by Reyntjens and others to NISR’s first set of poverty estimations. The timing here is important because it says something about the decision-making process, which in turn provides clues about the motivations and reliability of NISR’s poverty estimates. Indeed, the NISR 2016 report uses a completely different methodology to estimate the 2010-14 poverty trends and abandons earlier efforts to update the poverty line to account for changes in consumption patterns. This suggests that NISR itself recognized that their original trend estimates did not stand up to scrutiny and needed to be discarded in favor of a different methodology. Again, the World Bank euphemistically hints at this when they state that:

Both NISR (2015) and NISR (2016) produce a similar poverty trend (and more or less similar rates). But NISR’s (2016) approach is the one that produces comparable poverty estimates, using a common poverty line and a consistent COLI, one which uses the same budget weights, source data on prices, and base month (World Bank 2015, p.15).

In other words, it appears that the same (invalid) results that had been generated in the original official NISR poverty report, could also be generated using a completely different methodology, which had initially been discarded on the grounds that it failed to capture the important changes in consumption patterns that had occurred between 2001 and 2014. The fact that no mention was made of the alternative methodology in the original report, suggests that this convergence of results was nothing more than a fortunate ex-post coincidence that was discovered after the shortcomings of the original NISR 2015 report had been exposed.

Surprisingly, no explanation has been provided by NISR or the World Bank to explain (a) whether they stand by the initial methodology (and hence results) published in 2015, (b) why they decided to change the estimation method 6 months later and abandoned efforts to update the consumption basket to account for changes in consumption patterns, which had previously been claimed to be indispensable and (c) whether or not the new methodology adopted in 2016 can be trusted despite the fact that it doesn’t capture the crucial changes in consumption patterns.

Whereas the 2015 report updated the quantities of items in the consumption basket to account for changes in consumption patterns between 2001 and 2014, the 2016 report keeps quantities constant and only updates prices. This means that any change in the poverty line will now come solely from the price data. This also means that the accuracy of poverty estimates will depend on whether the chosen inflation rate accurately represents the actual price changes faced by poor people over this period. Consequently, the most important issue at this point should be to ensure that the chosen price data is robust and reliable for the purpose at hand.

A valuable contribution to this debate would therefore have been to carry out robustness checks on the NISR 2016 findings to see whether and to what extent the conclusions reached would have been affected by the choice of one of the other price data sources mentioned in the discussion (the Rwanda Ministry of Agriculture and Animal Resources, ESOKO – a communication platform aimed at smallholder farmers – and EICV). Alternatively, if pressed for time, the World Bank could simply have referred to the evidence already presented by Sam Desiere on this issue on roape.net, which showed that the choice of price data greatly affected poverty results in Rwanda. As stated in our earlier blogpost, this is not a mere technical detail that can be assumed away, but a fundamental question about what goods Rwanda’s poor were actually able to purchase with their income.

Instead, the World Bank takes as given the fact that the Consumer Price Index (CPI) data used by NISR in their 2016 report constitutes a better data source than the one used in the 2015 report and in all other poverty studies, and then proceeds to formalize (not explain or substantiate) the self-evident fact that: if you accept all of the assumptions and data choices made by NISR in their 2016 report, then all of the conclusions reached in that report do hold true. This is hardly a surprising finding and it contributes nothing to advance our understanding of whether living conditions have actually improved for Rwanda’s poor.

Crucially, no arguments have been provided by the World Bank to counter Sam Desiere’s claim that the official CPI data do not constitute an adequate basis for assessing the living conditions of the poor, nor to explain why we should prefer the official CPI data introduced in 2016 over all the alternative price datasets (MINAGRI, ESOKO, EICV) used in previous and subsequent studies.

Once this fact is recognized, the rest of the World Bank paper can essentially be disregarded, since all the subsequent findings derive from, and hinge on, the acceptance of the official CPI as the best price data to use in poverty estimates for Rwanda – a crucial assumption that the authors have omitted.

Other Supporting Evidence

Before concluding, we’ll address two other important and potentially valid points raised in the World Bank paper. First, the authors point to the S2S analysis (survey-to-survey) carried out in the NISR (2016) paper, which, in their opinion corroborates the poverty reduction findings. The S2S approach imputes poverty rates indirectly from non-consumption data on health, education, housing conditions, etc. The authors claim that the S2S ‘confirms that the official poverty trend is reliable’ (World Bank 2018, p.14). Essentially, the S2S applies, in regression form, the same line of argument used by Lee Crawfurd and Dónal Ring , which is to say that, given that non-monetary indicators of wellbeing have improved in Rwanda, it must also be the case that monetary poverty has improved. As such, the S2S is liable to the same criticism that we had raised earlier against Crawfurd and Ring, namely:

• this line of argument assumes that other NISR data are more reliable than monetary poverty data. In the context of an argument about the reliability of NISR data, this is an assumption that would need to be tested and proven before being taken as a starting point for any subsequent claim;
• There is no reason to assume that monetary poverty and non-monetary wellbeing indicators should follow parallel trends in Rwanda. The evidence produced so far suggests that there may have been a specific policy failure in the agricultural sector, which would presumably have a direct impact on individuals’ nutrition and monetary income. There has been no suggestion that this policy failure has affected access to health care, education or any other dimension of wellbeing. Therefore, it is perfectly possible, and indeed probable, that increases in monetary poverty have been accompanied by improvements in access to health care, education etc.

The second line of argument raised by the authors concerns Engel’s law (which is an observation in economics that argues that as income rises, the amount of income spent on food falls, which may accompany an absolute rise in food expenditure). The authors claim that the fact that the share of food consumption has declined between EICV3 and EICV4 ‘corroborates the poverty decline.’ This would be true, if it weren’t for the fact that the data actually show the opposite of what the World Bank claims, namely it shows that the share of food consumption increased sharply between EICV3 and EICV4. To reach their results, the World Bank have used the invented notion of ‘real food budget shares’, where the nominal budget shares have been deflated by price changes from 2010 to 2014. Since food inflation has been higher than general inflation over this period, they are deflating the nominator (food consumption) more than the denominator (total consumption) in 2014, but not in 2010! This fantastical adjustment alone accounts for the decrease in food consumption share in the World Bank paper.

Had they used the standard indicator, they would have found that from 2010 to 2014, the food consumption share rose from 64% to 67% for the population as a whole, and from 71% to 75% for the bottom two quintiles. If they were worried that auto-consumption might have been over-estimated in the 2014 survey due to erroneous price-imputations (e.g. if EICV price data are unreliable, as the NISR now seems to think), they could have focused on monetary expenditures instead, to stay true to the original formulation of Engel’s law. In that case they would have found that the food expenditure share increased from 46% to 52% for the whole population and from 51% to 58% for the bottom two quintiles.

In other words, whichever way we look at it, when properly calculated, the evidence presented by the World Bank actually strongly supports the claim that poverty increased in Rwanda between 2010 and 2014. The selective and even misleading presentation of supporting empirical evidence by the World Bank in this section, as well as their failure to carry out basic robustness checks or even to state the obvious caveats mentioned above, is, to say the least, disturbing.

Conclusion

The World Bank contribution has the merit of clarifying, although not stating explicitly, that the original EICV4 poverty estimates published by NISR in 2015 were not comparable with EICV3 poverty estimates, and that the official poverty trends published by NISR in 2015 were therefore not valid. Thanks to the World Bank’s careful description of the EICV3 and EICV4 methodologies, it should therefore now be possible to put this part of the discussion to rest.

Secondly, the World Bank confirms that it is (co-incidentally, it seems) possible to generate very similar results to those published by NISR in 2015 by using a completely different (originally discarded) methodology, so long as you use official CPI data to update the poverty line. Unfortunately, the World Bank have forgotten to prove that the CPI data constitutes an adequate basis for updating the poverty line and have failed to respond to Sam Desiere’s claim that it doesn’t. Instead, the World Bank has chosen not to justify or substantiate the self-evident fact that: if you accept all of NISR’s assumptions and data choices, then NISR’s conclusions regarding poverty do hold. This could hardly be considered as a substantial contribution to this discussion, and does nothing to advance, let alone ‘resolve’, the debate opened by Reyntjens in 2015.

Luckily, the World Bank’s analysis does provide some additional clues that may help us settle this debate, once and for all. Indeed, the only substantive piece of empirical evidence presented by the World Bank to support NISR’s claim is the food budget share, which the World Bank claims decreased between 2010 and 2014, providing strong empirical evidence for an improvement in living conditions. The only problem, is that this crucial piece of evidence is presented with a (suspiciously) serious methodological mistake. When this mistake is corrected, the data unequivocally shows that the share of food consumption/expenditures has increased significantly in Rwanda between 2010 and 2014, especially amongst the poorest households, thus providing the strongest independent empirical evidence to-date of an increase in poverty.

So, it seems that, in the end, the World Bank has, unwittingly, achieved its aim of settling the Rwandan poverty discussion after all. Indeed, when taken together, the evidence presented by the World Bank provides compelling evidence that poverty, most likely, did increase sharply in Rwanda between 2010 and 2014. The details presented in the World Bank paper on the estimation process also provides the strongest hints to-date that the ‘mistakes’ in the NISR estimates probably were intentional, or at least not entirely honest and transparent. Finally, the World Bank’s paper helps us to understand its own role in this process, pointing, at best, to a worrying level of leniency and incompetence amongst World Bank staff, and at worst, to outright complicity in the manipulation of Rwanda’s official statistics.

To conclude this response, we would like to suggest that, to redeem itself, the World Bank could use some of its considerable financial resources to help answer some of the outstanding questions on Rwandan official statistics. In particular, we would suggest focusing on explaining why household consumption growth in household survey data has been so low in Rwanda in recent years, compared to the figures used in the official GDP estimates. They may also wish to look into whether increases in total energy consumption and other production inputs are consistent with stated GDP growth figures.

We will leave it to others to comment on the broader political implications of the World Bank lending its name and expertise to an effort ostensibly aimed at providing theoretical cover for the manipulation of official statistics by a notoriously authoritarian regime, rather than contributing to promoting transparency and holding that government accountable for real results.

The authors of this article have asked for anonymity.

Featured Photograph: Food aid distribution in Rwanda, in 2008.

Notes

[1] The World Bank claimed that they were ‘unable to replicate’ Reyntjen’s results but omitted to mention that the same results had been generated in the original ROAPE blog, for which syntax files were publicly available, making ‘replication’ unnecessary.