IN this renewed debate on poverty measurement in Malaysia, one of the underlying issues is access to microdata.

Let’s be clear here. We are talking about microdata, not disaggregated data. The latter is easily obtainable from the Department of Statistics Malaysia’s (DOSM) website and the levels of disaggregation have improved in recent years.

But why is disaggregated data insufficient when it comes to poverty measurement? Why do we need microdata?

Microdata allows researchers to analyse more complex questions, beyond the averages, which may be more useful for policymaking.

For example, without microdata, we cannot even calculate the severity of poverty (beyond the headcount ratio), nor understand some of the poverty dynamics when we change the unit of analysis from households to individuals.

Statistical techniques to identify the determinants of poverty are also limited when we do not have microdata.

But why is microdata so elusive and inaccessible?

I have not been able to get a clear reply, but applying the principle of charity here, I thought these could be some of the main reasons:

1.            DOSM needs to balance confidentiality and access when it comes to microdata. Confidentiality is key to maintaining trust with respondents and helps ensure future cooperation when surveys are conducted.

2.            DOSM’s role as the guardian of national statistics (defined in the Statistics Act 1965 as ‘to collect and interpret statistics’) means that they have the right to be concerned with the misinterpretation of statistics.

3.            DOSM has to deal with additional queries from users when it releases more complex data files and may lack the resources/budget to handle them or develop supporting tools.

All the reasons above are legitimate, but the problems they pose are not insurmountable and should be weighed against the benefits of having more access to microdata.

On confidentiality, the practice of releasing anonymised microdata files either as public use files (US) or licensed files (Australia, Netherlands) is quite common.

Statistical techniques can be applied to ensure anonymity and a disclosure review board can be set up as a further safeguard.

On misinterpretation, metadata can be released alongside the microdata to explain the data structure and how statistics should and shouldn’t be used. In a post-truth world, it might be more productive for researchers to debate one another and point out potential misinterpretations of the data, instead of putting this burden entirely on a single authority. 

On resources/budget, my view is that the release of microdata provides the foundation for social science research to flourish in Malaysia.

Imagine different researchers competing to prove their theories right, challenging one another with different techniques applied to the microdata.

The government can save costs from having to commission expensive consultancy-based reports, because knowledge is generated in the public domain.

Will these benefits not outweigh the costs needed to make microdata more accessible?

The first core principle recommended by the United Nations Economic Commission for Europe (UNECE) on managing statistical confidentiality and microdata access is: “It is appropriate for microdata collected for official statistical purposes to be used for statistical analysis to support research as long as confidentiality is protected.”

The Canberra Group Handbook on Household Income Statistics, which the economic affairs minister cited, also states that, “As much as possible, without breaching the confidentiality of information collected, public use files (anonymised micro datasets) should be made available.”

Closer to home, Indonesia is well-known for its microdata being accessible.

I know of one case where a Malaysian PhD student, in a prestigious university, changed her thesis from Malaysia to Indonesia because of microdata issues.

In 2010, Singapore amended the Statistics Act to provide researchers access to anonymised microdata.

In 2013, the Philippines reformed its statistical agencies by setting up the Philippine Statistics Authority to improve statistical quality and timeliness as well as access to microdata.

Ken Simler from the World Bank said that raising the poverty line in Malaysia is an idea whose time has come.

I would add that making microdata accessible, and the procedures clearer and more transparent, is the other idea whose time has come, and a much-needed precedent for a more inclusive way we determine the poverty line. – September 17, 2019.

* Christopher Choong reads The Malaysian Insight.