A Dive into the Human Rights Measurement Initiative 2021 Survey

By Mennah Abdelwahab – May 2021

Improving data to improve lives

We are producing impactful world-leading human rights data to help improve people’s lives.

We aim to systematically measure all the rights in the International Bill of Human Rights, in every country in the world. By measuring human rights, we are shining a light on what’s really going on, and giving governments a measurement incentive to treat people better.

Each year our civil and political human rights data are gathered using our carefully designed, multilingual survey of subject matter experts. So far we are reaching about a fifth of the world’s countries, and we expand every year, at the cost of around USD 20,000 a year for each new country. This covers expenses like translation, ambassador liaison, research, survey administration, high levels of cybersecurity, dissemination of country scores on our free, online Rights Tracker, and engaging with in-country stakeholders to help them make the most of the resulting data.

From 13 countries to 39

We started running the survey in 13 countries in 2017, then 19 in 2019, and 33 in 2020. In late 2020, we launched a crowdfunding campaign to fund our civil and political rights data expansion to East and Southeast Asia with a final total of USD 44,000. With that, we added Hong Kong, Malaysia, and Taiwan, and were also able to reach more respondents in some countries we’d already started working in. We’re excited to be producing civil and political rights data for these countries for the first time in 2021 – we’re now up to 39 countries in total.

What are ‘Anchoring Vignettes’ and why are they important?

After two exciting and anxious months, the survey closes, and the data are processed. By April each year, we have responses from hundreds of human rights practitioners, telling us how common rights violations were in their country, in the previous year.

A key step in processing the data is analysing respondents’ answers to questions about a set of hypothetical countries: the ‘anchoring vignettes’. The answers people give to the questions about these anchoring vignettes tell us how to interpret their answers about their own country. This allows us to properly compare peoples’ responses, even if those people understand the question differently from one another, or interpret the scale (from, eg, ‘slightly’ to ‘extremely’) differently.

There’s also another reason these vignettes are important. The anchoring vignettes describe three fictional country situations which always represent three categories: one country that’s doing quite well in respecting human rights, one country that’s doing very badly, and one that’s in the middle. We expect every respondent to give slightly different responses in where they would put each country on the scale, but everyone should place them in the same order, with the good country scoring best, and the terrible country scoring worst. If a respondent puts them in a different order, we have to assume they either aren’t paying attention, or they don’t have a clear understanding of human rights. If they order the vignettes wrongly, we have to disqualify their answer for their own country for this section. The bottom line is that answering the vignette questions properly is vital if you want your other answers to contribute to the Rights Tracker data.

The vignettes can feel a bit weird to read when people start the survey, but they are an absolutely crucial part of the methodology, helping to ensure the validity and credibility of our data.

All survey respondents get asked all questions

One of the things that sometimes surprises people is that we ask everyone questions about all sorts of human rights, not just the ones that they are particularly ‘expert’ in.

This is very deliberate, for two reasons. The first is that if you are working daily with human rights in one area, you will have a much better idea than the average person about all other rights, too, and your insights will be valuable.

The second is that we know from a lot of research that it’s hard for people to self-select accurately – some people will modestly say they’re not experts in any area, and some people will say they’re experts in things they don’t know much about. So it actually leads to a better result to ask all the human rights people we’re in contact about all human rights, and produce scores within uncertainty bands.

It gets better each year

Our decisions to make changes to the survey are heavily informed by the feedback we receive from the respondents and Ambassadors. We know, for example, that several respondents have commented on the length of our survey. Throughout the rest of 2021 we will be testing ways to make the survey less taxing for future survey respondents. Thank you to all of you who have attempted to tackle this enormous challenge!

Survey respondents and responses are anonymous

Throughout the process, we protect the anonymity of our survey respondents. We recognise that reporting on human rights conditions can be a dangerous endeavour in some countries.

No one – not even us – can tell who each set of responses has come from, and we also have a process for stripping the most sensitive information from these responses before they are summarised into generic metrics and published on our Rights Tracker.

A tool you can use

Once we have developed the country scores and summarised the qualitative responses people give in their own words, we publish the data on the Rights Tracker. On this website, anyone can explore each country’s respect for eight civil and political rights as well as five economic and social rights (measured using a different method). It also allows users to compare countries’ performances and delve into which groups of people are at particular risk of facing violations for each right.

Human rights advocates, journalists, governments, civil society, investors, and others can use the human rights data on our Rights Tracker to improve people’s lives. The data over time can reveal trends, and provide hard numbers to pair with powerful personal stories, to strengthen advocacy and lobbying. Looking at the numbers can help shape priorities and strategy.

Find out more!

We are excited to launch our 2021 dataset on 24 June 2021. To stay up to date on our data launch, please subscribe to our newsletter and follow us on Twitter, Facebook, and LinkedIn.