[EAI Opinion Review] Debate on the Methodology of Public Opinion Polls for the June 2nd Local Elections

EAI OPINION Review No. 201006-05

Problems and Proposed Improvements in Election Public Opinion Poll Methodologies

The Gap Between Public Opinion Polls and Public Opinion

Following the June 2nd local elections, the public was likely surprised on at least two occasions. First, they were surprised by the severity of public sentiment towards the President and the ruling party, which was dramatically revealed overnight. Second, they were surprised again by how different the 'public opinion' perceived through various opinion polls was from reality. However, a significant reason why the majority of the public misjudged the 'public sentiment' in this local election was their misunderstanding of 'public opinion' based on the poll results reported by the media before the election. Therefore, it would not be an exaggeration to say that the two major surprises experienced by the public immediately after the election were ultimately due to the pre-election opinion polls that failed to accurately reflect public sentiment. In major battleground areas, including the Seoul metropolitan area, predictive polls conducted six days before the election, or on election day, or two to three days prior, all failed to predict the winners, leading to arguments questioning the utility of opinion polls.

Certainly, some argue that while public opinion trends may have shifted dramatically in the six days before the election when poll results could not be publicly disclosed, there are perspectives that acknowledge these shifts. Others contend that it is problematic to question the methodology of opinion polls, particularly telephone surveys, because some polls conducted during the six-day disclosure ban, though not published, did predict significant shifts in public opinion (EAI Opinion Review No. 2, "Unexpected Results in the Seoul Metropolitan Area: Hidden Votes or Changed Votes?" 2010.6.24). While acknowledging the considerable persuasiveness of these arguments, the author believes it is time to seriously examine the methodological flaws in the practices of election opinion polling in Korea. This article aims to pinpoint the reasons why telephone surveys, predominantly used for election polling, have failed to accurately represent voter sentiment and to propose an alternative solution.

Two Problems in Election Opinion Polls: Inaccuracy and Instability

The problems that confused us regarding the opinion polls publicized by the media during this election can be broadly categorized into two: the problem of 'inaccuracy' and the problem of 'instability'.

[Figure 1] Types of Problems in June 2nd Local Election Opinion Polls (Unit: %)

f2e34dd1a320d063

f2e34dd1a320d063

f2e34dd1a320d063

Source: Telephone survey results (Hankook Ilbo, May 12th and June 3rd issues), ARS survey results from Company B's website

First, the most criticized aspect of this election was the problem of inaccuracy. This criticism primarily targeted survey results obtained through telephone surveys. Polls conducted up to seven days before the election, with public disclosure permitted from May 8th, consistently showed Oh Se-hoon with a significant lead over Han Myung-sook. Notably, even polls conducted just one week before the election showed Oh Se-hoon leading Han Myung-sook by 17.8 percentage points. While polls conducted during the disclosure ban period (May 29-31) showed the gap narrowing to 11.2 percentage points, and in the Incheon mayoral election poll, Ahn Sang-soo led Song Young-gil by 38.3% to 41.2%, indicating actual changes, the failure to anticipate the extremely close race in Seoul (47.4% vs. 46.8%) means the polls cannot escape criticism for inaccuracy. Particularly in Gyeonggi Province, polls conducted from May 29-31 showed a 14.7 percentage point gap between Kim Moon-soo and Yoo Si-min, making it difficult to solely blame the poll disclosure regulations.

Second, there is the problem of instability. The issue began to surface around May 15th when surveys conducted via automated response systems (ARS: Automatic Response System or IVR: Interactive Voice Response), rather than by interviewers via telephone, showed Oh Se-hoon narrowing the gap with Han Myung-sook to 7.5 percentage points, while conventional telephone polls at the time showed a difference of 15-20 percentage points. This led to speculation that telephone surveys favored the Grand National Party candidate, while ARS surveys favored the Democratic Party candidate, amplifying the controversy (Chosun Ilbo, "Major Polling Agencies Not Conducting ARS Surveys" May 18th issue). However, examining the trend of ARS results before and after that period reveals inaccuracies and instabilities that are difficult to explain by actual shifts in public opinion. In fact, a survey by the same organization on May 8th, one week prior, showed a 20.3 percentage point difference, which narrowed to 7.5 percentage points in the May 15th survey. A week later, the gap widened again to 25.8 percentage points in the May 22nd survey, showing an even larger discrepancy with the actual vote count than telephone surveys. The result of 57.1% for Oh Se-hoon on May 22nd, which was 5-6 percentage points higher than the 50.4% he received in telephone surveys around the same time (May 24-26), rendered the assessment that ARS surveys were favorable to the Democratic Party moot.

As demonstrated, the results of the June 2nd local election opinion polls, conducted using various methodologies and published unfiltered through various media outlets throughout the election period, failed to predict the extremely close race in the Seoul area. This was sufficient to foster widespread distrust in opinion polls.

The Problem of Representativeness

What, then, were the underlying causes? This discussion will focus primarily on the issues with telephone survey methodologies, rather than the ARS (IVR) method. While criticisms regarding the unscientific nature of ARS (IVR) surveys have been discussed multiple times, the problems associated with telephone survey methodologies have rarely been a subject of public discourse. Given that telephone survey results, known as the most credible method for estimating support levels in electoral competition, are now perceived as the source of confusion during this election, a more focused explanation is warranted. However, recent issues related to ARS (IVR) surveys will be briefly addressed in the appendix.

The Issue of Representativeness of the Sampling Frame

Despite various discussions on the problems of telephone opinion polls, the answer from the surveyor's perspective is surprisingly simple: the respondents (sample) did not represent the entire population (sampling population, i.e., voters in an election survey) that the survey aimed to cover. Pre-election polls must select respondents in a way that represents all eligible voters. Similarly, predictive polls released on election day must select respondents that represent all those who voted. If these textbook principles were followed, exit polls conducted with voters leaving the polling stations, as well as telephone-based trend and predictive polls, would not produce results that are wildly different from actual public opinion or sentiment. However, the current situation arose due to practical limitations in adhering to these textbook principles.

The structural limitations related to the representativeness of telephone opinion polls can be summarized into three main points.

First, the sampling frame used by polling companies to contact telephone survey respondents does not sufficiently include the entire population. Currently, telephone opinion polls are conducted using telephone numbers listed by KT that subscribers have authorized for public disclosure. As of October 2009, the number of such listed phone numbers was approximately 7.6 million. According to the 2005 Population Census, there were over 15.8 million households in Korea. Therefore, the proportion of KT-listed telephone numbers for which subscribers have authorized public disclosure is less than 50% of the total households. This means that when conducting telephone surveys, the pool of households eligible to participate is limited to less than 50% of all households.

Second, the KT telephone numbers that are subject to opinion polls do not represent the entire population of households. Even if the rate of publicly disclosed numbers among KT telephone numbers is less than 50%, if we assume that these households represent the general population, then 7.6 million phone lines could be considered a sufficient number and thus acceptable as a sampling frame. However, it is reasonable to assume that publicly disclosed numbers among KT listed numbers may not represent the entire population or its members in terms of variables such as education and income, given that individuals with higher education and income are more likely to be reluctant to disclose their phone numbers than those with lower education and income. Furthermore, as of 2010, the proportion of single-person households in Seoul reached 20.8%. Many of these individuals, primarily in their 20s and mid-30s, are young, unmarried individuals who primarily use mobile phones and do not have landlines. Increasingly, even households with two or more members, such as newly married couples, are opting not to have landlines and instead use individual mobile phones. Additionally, the number of households subscribing to services other than KT, such as internet phones and integrated phone services, is gradually increasing. Considering these factors, it can be assumed that the approximately 7.6 million phone numbers utilized by polling companies among KT listed numbers do not evenly include the entire 15.8 million households.

Third, it cannot be guaranteed that respondents participating in telephone surveys represent their respective demographic groups. To protect respondents' privacy, telephone surveys are not conducted after 9 PM. This implies that respondents participating in telephone surveys are individuals who are home by 9 PM. In this regard, it can be assumed that younger individuals or those in the workforce who participate in telephone surveys may have different political leanings than their peers in the same age group or workforce who return home after 9 PM.

Advantages and Limitations of the Random Digit Dialing (RDD) Method

Given the realities of polling described above, it is true that current telephone survey methods have been used, and their results have been utilized without significant public questioning or verification. Of course, polling companies and relevant experts in Korea are not unaware of these polling conditions. To overcome the limitations of KT household telephone numbers, attention has turned to the application of RDD (Random Digit Dialing), a method that attempts telephone surveys by randomly generating the last four digits of a phone number beyond the area code and exchange (Heo Myung-hoe et al. 2008).

However, as it is still in its early stages, there are still aspects that require improvement. Despite RDD being a potential alternative to telephone surveys using household landlines to ensure representativeness, there are practical obstacles preventing its widespread adoption. These include the fact that RDD surveys cost 1.5 to 2 times more than conventional telephone surveys and are not easily conducted within a single day. To universally apply the RDD method to telephone opinion polls, collaborative efforts among surveyors, clients, and relevant experts are necessary.

Furthermore, while the RDD method using KT household phone numbers theoretically offers a solution to the sampling frame problem in telephone surveys, it is not entirely sufficient. This is because the KT household phone RDD method cannot encompass individuals who exclusively use mobile phones, and subscribers to internet or integrated phone services are excluded from the outset. Ultimately, the only viable alternative to resolve the debate over the representativeness of telephone opinion polls is to utilize mobile phones, whose penetration rate exceeds 100%. Although there are limitations to conducting comprehensive surveys targeting mobile phone users due to legal restrictions concerning personal information protection, practical solutions must be devised.

The Problem of Respondent Selection Methods and Alternatives

The above discussion primarily examined the problems and alternatives of existing household telephone survey methods from the perspective of the sampling frame. This section will now discuss the current situation and alternatives in the respondent selection process.

Opinion polls using household telephone numbers generally involve two stages. The first stage is selecting household telephone numbers, and the second stage is selecting appropriate respondents from those households and ensuring their participation in the survey. The conditions for enhancing survey reliability differ depending on the characteristics of each stage. The first stage, selecting household telephone numbers, is closely related to the structural condition of the sampling frame, and efforts mentioned above are needed to improve survey reliability. Conversely, the second stage, selecting respondents and requesting their participation, is closely linked to the survey environment of our society.

In the case of media election opinion polls, which were the subject of much debate during the recent local elections, the survey period is typically one day, and surveys are primarily conducted on weekdays. These conditions can lead to a situation where participants do not adequately represent the entire respondent pool. As discussed earlier, telephone opinion polls using household telephone numbers are more likely to include respondents who are at home. However, many students and office workers return home after 9 PM, when telephone surveys are usually completed. Considering this, it is difficult to survey the target number of individuals in these groups within the limited one-day survey period, and those who do participate may have specific tendencies or lifestyles. Therefore, the participants may not be representative of the entire population.

Technical Alternative: Adjusting Survey Schedule and Duration

To address these issues, it is necessary to extend the survey period to 2-3 days and conduct surveys on weekends rather than weekdays. Extending the survey period to 2-3 days allows for more flexible surveying in the evenings on different days, making it easier to reach and survey individuals from groups that are difficult to sample, such as students, office/professional workers, and production/skilled/labor workers, in numbers closer to the target. Conducting surveys on weekends, rather than weekdays, is also important because these groups are more likely to be at home on weekends than on weekdays. The reason why these basic survey environment considerations are not implemented in practice is not unrelated to the urgency felt by political circles and media organizations, who are the clients commissioning the opinion polls. In other words, a shift in the perception of the survey clients is also an important prerequisite for issues related to the survey period and days.

The Problem of Survey Response Rate: Quota Sampling is an Unscientific Sampling Method

Currently, telephone opinion polls in Korea do not employ scientific sampling methods such as simple random sampling or stratified random sampling. Instead, they use quota sampling. Stratified random sampling, used when the population is composed of diverse strata, and quota sampling both involve identifying the overall composition of respondents and pre-determining the number of respondents to participate in the survey, with surveyors then working to fill these quotas (Singleton and Straits 1999). Quota sampling is not recognized as a scientific sampling method because it compromises randomness, which is crucial for scientifically ensuring sample representativeness, by allowing surveyors (or institutions) to determine the final respondent selection process at their convenience (Erikson and Tedin 2005: 26). In contrast, stratified random sampling, after allocating sample quotas, involves randomly selecting the actual households to be surveyed and making maximum efforts to obtain responses without arbitrary replacement, thereby minimizing the overrepresentation of respondents with specific tendencies (politically engaged voters) and the underrepresentation of respondents who are reluctant to participate.

When the initially selected household (respondent) is arbitrarily replaced, the issue of response rate arises. The use of quota sampling in telephone opinion polls means that if contact fails or a refusal occurs with a randomly selected household (respondent), the surveyor can immediately replace them until the target number is met. For example, if a 20-something male is sought in a household contacted by phone, and that individual is out, the surveyor can call another household to find a 20-something male instead of calling back the same household. Similarly, if a respondent hesitates to answer, immediate replacement can occur, systematically excluding individuals who must be away from home during certain hours or allowing for the overrepresentation of politically engaged respondents. While this method is efficient for polling companies that must meet target respondent numbers within a limited time, it fundamentally infringes upon the sampling and probability theories that underpin scientific opinion polling.

Of course, quota sampling, even with its low response rate, is not an inherently flawed or useless method. It is still employed in other countries, not just Korea, due to its methodological advantages, and as will be explained later, there are complementary measures. However, it is important to emphasize that these advantages should not be overly justified to the point of ignoring the limitations of quota sampling, and greater caution should be exercised in interpreting the survey results.

Strengthening Call Backs

Considering that ensuring representativeness fundamentally requires using the initially randomly selected household as the valid survey sample, it is necessary to more rigorously apply call backs, i.e., attempting to contact respondents again at different times or on different days when the initial contact was busy or the appropriate respondent was unavailable, thereby minimizing sample replacement. The table below [Table 1] confirms that there is a difference between the results of contacting only once and contacting up to twice for the same population (Cho Sung-gyeom et al. 2007). As shown in the table, when contacted twice, the response success rate for younger individuals under 30 in the evening hours increases significantly.

[Table 1] Response Rate by Age Group for First and Second Contact (Unit: %)

*Source: Cho Sung-gyeom et al. (2007)

Application of Appropriate Quota Variables: Applying Occupation or Education Quotas

Increasing the survey response rate through the joint efforts of survey clients, media, and the polling industry is a long-term solution. However, in the short term, deviating from current practices in sample allocation and weighting, and considering factors that influence voters' political attitudes when allocating samples, can be a short-term and practical measure.

Theoretically, the more quota variables used, the higher the reliability of the survey. Currently, most election surveys in Korea use gender, age, and region as quota variables. This is because the number of individuals within these categories can be clearly identified. However, as Korean society becomes increasingly complex and capitalism and democracy mature, relying solely on gender, age, and region as variables is insufficient to represent the population in opinion polls. Now, not only in marketing research but also in political research, occupation and income should be considered key variables, and education level is also a significant factor. The Gallup organization in the United States applies weights based on race, Hispanic origin, and education level, in addition to gender, age, and region. Therefore, Korea's surveys should also consider expanding quota variables. Recently, occupation has become a variable that actively influences voters' political attitudes in Korea and should be considered as a primary basis for sample allocation.

As confirmed in [Table 2], during the recent local elections, significant differences in candidate support tendencies were observed, particularly by occupation. Oh Se-hoon had higher support among self-employed individuals, full-time homemakers, and the unemployed/retired/no response categories, while Han Myung-sook had higher support among blue-collar workers and students. The support gap between the two candidates was not large among office/professional workers. Given that Oh Se-hoon's overall support was 12% higher than Han Myung-sook's, the differences in support by occupation are highly significant.

[Table 2] Support by Occupation: Seoul Mayoral Election (Unit: %)

*Source: Korea Research (2010.5)

Indeed, as shown in [Table 3], when occupation is not used as a quota variable, self-employed individuals and homemakers, who are more likely to be at home during survey times, tend to be overrepresented, while white-collar and blue-collar workers tend to be underrepresented. This tendency is more pronounced in weekday surveys than in weekend surveys. In other words, this ultimately leads to an underestimation of opposition party support and an overestimation of ruling party support. Considering these points, Korea Research has been conducting surveys based on its own occupational distribution tables since the 2007 presidential election prediction polls, and ultimately applies occupational weights (refer to Appendix 2 for the occupational distribution table used as a basis for regional weighting).

[Table 3] Occupational Distribution Differences by Survey Day in Seoul Area (Unit: %)

*Source: Company A (2010.5)

The Falsity of the Hidden Ballot Controversy

We should not attribute the inaccuracy of the recent local election telephone polls to the suppressive nature of our society, nor to the strategic choices or lack of communication skills of respondents. Doing so would, intentionally or not, lead to the negation of telephone polls themselves. There is no persuasive evidence to suggest that the Korean people are less capable of expressing their opinions than those in the United States, Japan, or Germany, particularly in areas related to politics. Furthermore, it cannot be said that exit polls, which require face-to-face confirmation of voting behavior, are less burdensome than telephone polls, where respondents only need to state their voting intentions without direct contact. The reason exit polls could predict election results more accurately than telephone polls is that they surveyed actual voters through a rigorous sampling process, not because respondents felt less burdened and provided more honest answers compared to telephone polls. The Korean people have presented a significant issue not only to the political sphere but also to the polling industry through these local elections. While we must wait to see how the political sector responds, the polling industry's response is clear: the problem lies not with telephone polls themselves, but with the methods and environment of telephone polling in our society. It is necessary for all involved in polling to work together to improve this situation.■

Kim Choon-seok, Korea Research · Jeong Han-ul, EAI Center for Public Opinion Research

References

Cho, Sung-gyeom, Ji-yeon Kim, Na Yoon-jung, and Myung-jin Lee. 2007. “Problems and Improvement Directions for Election Polling: Focusing on Telephone Polls in the 2006 Local Elections.” <Survey Research> Vol. 8, No. 1. pp.31-54.

Heo, Myung-hoe and Young-won Kim. 2008. “RDD Sample vs. Telephone Directory Sample: A Case Study of the 2007 Presidential Election Forecast.” <Survey Research> Vol. 9, No. 3. pp.55-69.

Erikson, Robert A. and Kent L. Tedin, 2005. American Public Opinion: Its Origins, Contents and Impact. New York: Pearson Longman.

Singleton, Jr. Royce A. and Bruce C. Straits, 1999. Approaches to Social Research: Third Edition. New York: Oxford University Press.