Last year, the COVID-19 pandemic forced 优蜜传媒to rethink how we survey the world. Since 2005, we have conducted the majority of our World Poll surveys through face-to-face interviews -- which certainly wasn't safe or even possible to do in most places in 2020, and still isn't today.
After concluding that we could safely switch from face-to-face interviews to computer-assisted telephone interviews (CATI) and still maintain the data quality that meets our standards, that's what we did. And, in 2020, in close partnership with our regional teams around the world, we carried out surveys in 116 countries and areas.
We encountered a fair number of challenges along the way -- some expected and some not. However, our teams were extremely resourceful and always found solutions that did not compromise data quality or put people in harm's way during the pandemic.
We learned a lot that we can use in our research now -- as the world still reels from the pandemic -- and in the future when CATI may be the optimal methodology.
Infrastructural and Technology Challenges
- We spent a fair amount of time planning and implementing the right technical solutions in each region and testing the various infrastructural elements that are required for smooth data collection. However, one of the biggest challenges, particularly in developing countries, was network connectivity. Slow or spotty connectivity meant dropped calls. Interviewers often had to call people back, and this hurt productivity.
Takeaway: Having an offline option for managing the sample is crucial in these situations. Although we had already planned to have this ready in a limited number of countries, we ended up implementing this in more countries across a broader area.
- Many countries also lacked mature landline infrastructure, which we factored into our research design. As a result, roughly half of the countries were set up as mobile-only. In general, random-digit-dial (RDD) samples in these mobile-only countries performed better than expected. They yielded contact and survey cooperation rates in some regions that, though not as high as what we attain in face-to-face interviewing, were encouraging. These certainly motivated interviewers to stay engaged.
At the same time, we had a significantly harder time working with landline RDD samples in countries that were using both landline and mobile samples. Although we expected a large portion of the landline RDD sample to be nonworking or disconnected numbers, executing the call-design protocols under tight timelines placed significant pressure on the CATI system and on interviewer morale.
Takeaway: We anticipate the continued need to use dual-frame samples in some countries, but we will continue to focus on improving sample efficiency and more realistic timelines for data collection.
- The investments we've made over the years in testing online training methods paid off in 2020. All interviewer and supervisor training in 2020 was done remotely. The e-learning platform we used, in combination with other methods to conduct interviewer and supervisor training in several countries, holds tremendous potential.
Takeaway: We plan to expand the use of these tools to a wider range of countries in 2021.
People Challenges
-
Partners in most countries were able to identify, motivate and retain interviewers who were successful at conducting CATI interviews. However, the productivity lost because of technology challenges, lengthy interviews and poor landline sample performance all dampened interviewer morale, which led to higher attrition in some countries.
Takeaway: There are no easy solutions other than to fix technology issues. In the midst of a pandemic, with limited job opportunities, the attrition related to this issue may not have been as widespread. However, as the pandemic situation improves, this could prove to be a major challenge. Yet, the challenge did push teams to come up with innovations for both interviewer and supervisor incentives and use other mechanisms to boost interviewer retention.
- Although remote interviewing (where interviewers were primarily working from home) because of COVID restrictions came with its own set of challenges, it brought a lot more women into the interviewer pool. In the past, it had been harder to attract female interviewers for face-to-face data collection in certain regions.
Takeaway: The ease of working from home, with flexible hours and being able to take care of their family, was a big draw for women. The continued success of the remote interviewing strategy in attracting and retaining a more diverse interviewer pool will largely rest on how successful we are at resolving the technology challenges and what incentives are in place for interviewer retention.
Data Challenges
-
The success of efforts to increase representativeness in countries with lower telephone coverage by treating the mobile phone as a household device and selecting an adult at random within those households varied across the set of countries where this was implemented. This was largely attributable to cultural and logistical reasons.
Takeaway: We are conducting a thorough review of roadblocks and methods to overcome them at the country level to try to improve the efficacy of this method. It has the potential to improve representation of certain demographic segments.
- Demographic skews due to coverage and nonresponse remain a significant challenge. Even in countries with high overall telephone coverage, there is a greater propensity to get cooperation from respondents who are younger (aged 15 to 34), male, more highly educated, single, employed, and living in urban areas.
While some of this reflects limited access to mobile phones among certain demographic segments, even in countries where a higher share of respondents other than the one reached on the phone was selected, the skews are substantial.
Takeaway: In countries where dual-frame designs are used, altering the proportion of the landline/mobile mix might be one way to minimize these skews -- but in mobile-only countries, this remains a major challenge.
- Item nonresponse is not a significant issue. Items that typically yield higher nonresponse in face-to-face interviewing mode also tend to exhibit higher nonresponse over the telephone. This can be attributed to consistent training of interviewers and supervisors, the implementation of standard interviewing protocols, and a workforce that is largely stable.
- While trends on a majority of items remain consistent, measures related to the economy, jobs and confidence in leadership exhibit more variability, suggesting that some of this is related to the pandemic and how it has affected individuals across different countries. Data are still pouring in, and more work is needed to tease out the effects of coverage, nonresponse, social desirability bias and the pandemic.
Looking Forward
There are still a lot of data from the 2020 surveys to be analyzed to evaluate the effects of the mode switch and to fine-tune the processes for a smoother administration in 2021.
While we all hoped for a more normal 2021, one in which we can go back to collecting data using the most appropriate mode for each country, given where we are with the spread of the pandemic and the progress made with respect to vaccination globally, we do not expect any dramatic change in our approach this year.
We are cautiously optimistic that several countries with low or stable infection rates may be able to revert to face-to-face data collection -- although with more constraints, including social distancing and mask wearing. In those situations, it's possible that we could then switch to CATI if the situation on the ground changed rapidly. The remaining countries will continue another year with CATI as the primary data collection mode. Regardless, the research will continue.
For complete methodology and specific survey dates, please review .
Learn more about how the works.
