[EAI Opinion Review] 2012 Presidential Election Opinion Polls: What is the Problem?

f2e34dd1a320d063

f2e34dd1a320d063

f2e34dd1a320d063

1. The Paradox of Declining Credibility Amidst a Flood of Daily Opinion Polls

Daily opinion polls on the presidential election are being released in large numbers. The 2012 presidential election polls have one distinct characteristic compared to the past. The daily rolling average method, where the total sample is divided equally into portions released every two or three days, with new participants recruited daily to replace existing ones (and then averaged with the remaining sample), has been widely adopted. In addition to this, there are regular monthly surveys, ad-hoc surveys released as needed, and panel surveys tracking changes among the same respondents conducted by my institution. The survey methods have become more diverse than ever, and public opinion changes in response to events are being announced.

This is a stark contrast to the late 1980s, the early period of democratization, when media outlets conducted election polls but were unable to properly announce candidate approval ratings. While issues such as the prohibition of opinion poll announcements within a week of the election and restrictions on exit poll distances remain to be resolved, we can now access information related to opinion polls through the media almost daily. Assuming that the volume of opinion polls reflects public sentiment, this presidential election is being conducted in an environment that allows for a more accurate reflection of public opinion than any previous election.

Does increased volume lead to increased accuracy? As survey methods diversify and the amount of information grows, the number of voters who utilize opinion poll results when choosing a candidate is actually decreasing. Even more paradoxically, voices expressing distrust in opinion poll results are growing louder. In fact, according to the joint election panel survey results from EAI, SBS, JoongAng Ilbo, and Korea Research, the proportion of respondents who utilized opinion poll reports when choosing a candidate reached 44.6% in the 2006 local elections. This figure has since fallen to 27.5% by the 2007 presidential election, the 2008 general election, and the 2010 local elections.

[Figure 1] Percentage (%) of Respondents Who Reported Being Influenced by Opinion Poll Results When Choosing a Candidate in Recent Elections

Source: Joint Election Panel Survey by EAI, SBS, JoongAng Ilbo, and Korea Research (2006-2010)

2. Confusing Survey Results

Recent presidential election poll reports present several types of cases that cause confusion among the general public.

Differences Between Polling Agencies in Daily Surveys

Several types of cases that cause confusion among the public can be identified in recent presidential election poll reports.

First, there are cases where polls conducted during the same period show differences between agencies. The survey results released on October 11th by Korea Gallup and the Asan Institute for Policy Studies (AIPS) & Research & Research (R&R), which periodically publish daily rolling average surveys for this election, are representative examples. Let's examine the poll conducted by Korea Gallup from October 8th to 10th. In a head-to-head matchup, Saenuri Party candidate Park Geun-hye received 49%, while independent candidate Ahn Cheol-soo received 45%. Candidate Park (51%) also comfortably led against candidate Moon Jae-in (42%). In a survey of preference for a unified opposition candidate, candidate Moon led candidate Ahn significantly with 49% to 35%.

However, in the survey conducted by AIPS and R&R during the same period, both candidates Ahn and Moon led candidate Park in a head-to-head matchup. Candidate Ahn (50.6%) received approximately 9 percentage points more than candidate Park (41.7%), and candidate Moon (48.2%) also held a lead of nearly 4 percentage points over candidate Park (44.3%). In contrast, the preference for a unified opposition candidate showed a tie between Ahn Cheol-soo at 39% and Moon Jae-in at 39%.

The survey results from JTBC and Realmeter, conducted over October 9th and 10th, were similar to the AIPS results, showing candidate Park trailing candidate Ahn. The results were 43% (Park) vs. 49% (Ahn) and 45% (Park) vs. 47% (Ahn). In the preference for a unified candidate, candidate Ahn led candidate Moon by 7 percentage points with 42% to 35%, showing a difference.

[Table 1] Key Daily Survey Results and Methodological Differences Announced on October 11th

Source: Korea Gallup [Daily Political Indicators] Survey Results October 8-10, 2012, JTBC․Realmeter [2012 Election Daily Opinion Poll],

Asan Institute for Policy Studies website and YTN News [Asan Institute for Policy Studies Opinion Poll]

Note: Bold indicates a difference exceeding the margin of error; red indicates the leading candidate.

Confusing Chuseok Public Sentiment Polls

Second, the Chuseok public sentiment polls, which garnered significant attention from the political sphere, also showed discrepancies across 10 media organizations. Each media outlet conducted polls on October 1st to 3rd, immediately after Chuseok, over one or two days to track changes in public sentiment during the holiday ([Table 2]), in addition to the regular daily rolling surveys mentioned earlier.

First, in 9 out of the 10 media organization surveys, candidate Ahn led candidate Park in a head-to-head matchup. Among these, 5 surveys (MBC․Korea Research, Hankook Ilbo․Korea Research, JTBC․Realmeter, Sisa Journal․Realmeter, View & Poll․Research View) showed candidate Ahn leading candidate Park beyond the margin of error, and in 4 surveys (Chosun Ilbo․Media Research, Asan Institute for Policy Studies․Research & Research, Dong-A Ilbo․Research & Research, Herald Business․Realmeter), candidate Ahn led within the margin of error. Conversely, only the survey by Kookmin Ilbo․Global Research showed candidate Park leading candidate Ahn within the margin of error.

In contrast, in the matchup between candidate Park and candidate Moon, only 2 surveys (View & Poll․Research View, Sisa Journal․Realmeter) showed candidate Moon leading candidate Park beyond the margin of error, and in 5 surveys (MBC․Korea Research, Hankook Ilbo․Korea Research, Herald Business․Realmeter, JTBC․Realmeter, Asan Institute for Policy Studies․Research & Research), candidate Moon's lead was within the margin of error. Conversely, 3 surveys showed candidate Park leading, with Chosun Ilbo․Media Research and Dong-A Ilbo․Research & Research showing results within the margin of error, and Kookmin Ilbo․Global Research showing results beyond the margin of error. Generally, there were many surveys within the margin of error, and the rankings were mixed, making it somewhat confusing.

[Table 2] Comparison of Survey Results Conducted by Various Media Outlets Immediately After Chuseok

Note: Bold indicates a difference exceeding the margin of error; red indicates the leading candidate.

Differences Between Surveys Conducted by the Same Agency at the Same Time

Third, during the Chuseok holiday period, differences in survey results from the same agency, depending on the media outlet that published them, further exacerbated confusion.

Let's examine the survey conducted by Research & Research between October 1st and 3rd. The survey conducted on October 2nd was published by Dong-A Ilbo, and the survey conducted from October 1st to 3rd was published by the Asan Institute for Policy Studies. In the head-to-head matchup between Park and Ahn, where Ahn led, the difference was 4.5 percentage points for Dong-A Ilbo and 2.0 percentage points for AIPS, not a significant difference. However, in the head-to-head matchup between Park and Moon, Dong-A Ilbo reported Park Geun-hye at 47.4% and Moon Jae-in at 44.5%, with Park leading. In contrast, the daily rolling survey by AIPS showed Park at 42.9% and Moon at 44.5%, reversing the ranking.

Significant differences are also found between the survey released by Sisa Journal (conducted by Realmeter on October 3rd) and the daily rolling survey for JTBC (conducted on October 2nd-3rd). In the Realmeter survey, while the rankings did not change in the head-to-head matchups between Park and Ahn, and Park and Moon, the gaps were considerably large. First, in the Park vs. Ahn matchup, the JTBC survey showed Park at 44.9% and Ahn at 50.0%, a difference of only 5.1 percentage points. However, in the Sisa Journal survey, Park received 37.7%, trailing Ahn by a significant 13.8 percentage points (Ahn at 51.5%). In the hypothetical matchup between Park and Moon, the JTBC survey showed an extremely close race with Park at 47.2% and Moon at 47.7%. The Sisa Journal survey, however, showed a gap of 7 percentage points, with 38.6% for Park and 45.6% for Moon. Looking solely at Park's approval rating, there is a difference of 8.6 percentage points.

In contrast, the MBC survey conducted on the 2nd and released on the evening of the 3rd, and the Korea Research survey conducted on the 4th (handled by Korea Research), showed relative consistency. Both the Hankook Ilbo (Park 41.1%, Ahn 49.7%) and MBC (Park 40.8%, Ahn 47.7%) surveys showed candidate Ahn leading candidate Park beyond the margin of error in a head-to-head matchup. In the head-to-head matchup between Park and Moon, both Hankook Ilbo and MBC showed candidate Moon leading candidate Park within the margin of error.

Consequently, significant variations occurred even within the same polling agency for the Chuseok public sentiment polls around October 4th and the surveys around October 10th. Of course, these comparisons of survey results do not provide definitive evidence of critical flaws in specific surveys or suggest that certain polling agencies are more reliable. Furthermore, it is not desirable for general distrust in opinion poll methodologies to spread. However, voters have grounds to question the reliability of opinion polls and their reporting. When the rankings of candidates change depending on the media outlet, even for surveys from the same polling agency, voters are understandably confused.

[Figure 2] Differences in Survey Results by the Same Agency at Similar Times on October 4th

Case 1. Dong-A Ilbo Survey and Asan Institute for Policy Studies Survey Results Announced on October 4th

Note: Dong-A Ilbo survey conducted on October 2nd with 1000 respondents; Asan Institute for Policy Studies survey conducted October 1st-3rd with 1000 respondents.

Case 2. JTBC/Realmeter Daily Survey and Sisa Journal/Realmeter Same-Day Survey Announced on October 4th

Note: JTBC survey conducted October 2nd-3rd with 1500 respondents; Sisa Journal survey conducted October 3rd with 1000 respondents.

Case 3. Hankook Ilbo/Korea Research Same-Day Survey and MBC/Korea Research Same-Day Survey Announced on October 4th

Note: Both surveys were conducted on October 2nd using interviewer-based telephone surveys (50% landline + 50% mobile phone).

3. What is the Problem?

What is causing the increasing confusion in these opinion polls and their reporting? Above all, methodological issues can be pointed out.

Arbitrary Survey Methodologies

To put it somewhat hyperbolically, I would call the methodology of Korean election opinion polls a state of anarchy. There are significant problems due to the prevalence of arbitrary survey methods. It is no exaggeration to say that Korean election opinion polls are undergoing a transitional period in terms of survey methodology, especially after the 2010 local elections. The introduction of Random Digit Dialing (RDD) and the shift towards mobile phone surveys, either in conjunction with household landline surveys or exclusively, represent a fundamental re-evaluation of opinion poll methodologies following the widespread inaccuracies of polls predicting a victory for the Grand National Party in the June 2nd, 2010 local elections.

Until 2011, most domestic opinion polling agencies used a method of randomly selecting samples from KT's directory of household landline numbers. When numbers are extracted from the KT household telephone directory, numbers from non-KT providers (starting with 070) or households that do not wish to disclose their phone information are inherently excluded from the sampling frame. Furthermore, as the number of individuals, particularly those living alone, who do not have a household landline and only possess a mobile phone increases, these individuals are also inherently excluded from the opinion poll sampling frame. It is known that households and individuals excluded from the sampling frame in this manner constitute more than half of the total population. Therefore, the existing survey method inherently has the potential to violate the principle of 'representativeness,' which requires that sampled respondents accurately represent the entire electorate.

Consequently, RDD, which randomly dials numbers instead of selecting from KT's household telephone directory to include non-listed households in the sample, was introduced as an alternative. Additionally, the use of mobile phone surveys has increased.

The problem is that, without any verification or discussion on whether the RDD and mobile phone adoption methods are truly effective alternatives, each polling agency has begun to competitively introduce differentiated survey methods. Among the aforementioned polling agencies, Korea Gallup and Research View adopt a 100% mobile phone RDD method, while Korea Research and Media Research use a 50% mix of household landline RDD and mobile phone RDD. Realmeter, which conducts daily surveys, uses an 80% household landline and 20% mobile phone RDD method. The issue is that, in the current situation, none of these proportions—100%, 50%, or 20% mobile phone usage—can be theoretically validated.

The 100% mobile phone survey method, despite the increase in mobile phone ownership, still excludes voters who do not own mobile phones, particularly among low-income and elderly populations, thus creating a different kind of sampling representativeness problem. Conversely, individuals who own multiple mobile phones are likely to be overrepresented compared to those who own one or none. When combining household landline and mobile phone surveys, the proportion of this mix is determined by the arbitrary judgment of the polling agency, as the overall distribution of individuals who only own mobile phones and not landlines is unknown.

Furthermore, it is questionable how strictly the principle of excluding respondents who own landlines from mobile phone surveys is adhered to. In principle, mobile phone surveys were introduced to represent voters who exclusively own mobile phones and not landlines, but actual statistics are lacking. Therefore, the problem of over- or under-representation of certain respondents cannot be fundamentally prevented. Harshly assessed, the current survey methods have resolved the issue of landline representativeness only to create a new problem of mobile phone representativeness.

Spread of Misconceptions: "Are Mobile Phone/Non-Listed Household RDD Surveys Pro-Progressive?"

Election journalism also played a role in this hasty shift in survey methodologies. Since the Asan Institute for Policy Studies first published a comparative analysis of KT-listed household landline surveys, non-listed household landline surveys, and mobile phone surveys in January 2011, the media has adopted the shift to mobile phone and non-listed household landline surveys without sufficient verification or confirmation. In the Asan Institute for Policy Studies' survey, the presidential approval rating was 48% for KT-listed household landline surveys, while it was 42% for non-listed household landline surveys. This result was interpreted as indicating that non-listed household phone surveys are pro-opposition and progressive, suggesting the existence of hidden opposition votes. Subsequently, the assertion that "non-listed households and individuals who only own mobile phones are pro-opposition and progressive, and these hidden votes caused opinion poll predictions to be wrong in the local elections" was widely disseminated through the media.

[Table 3] Difference in National Approval Ratings Between Respondents from Listed and Non-Listed Households, and Composition of Respondents by Age Group in Each Survey (%)

Interpreting these results directly as a difference between listed and non-listed household surveys is problematic. First, with a sample size of 1000, a 6% point difference falls within the statistical margin of error. In other words, the difference in response rates between the two groups is not statistically significant, yet the media at the time over-interpreted it as a substantial difference. Second, it was necessary to examine whether the difference in approval ratings between the two groups was influenced by the difference in survey methods as an actual mediating factor. That is, it was necessary to confirm whether a difference still existed under conditions where the social characteristics of the samples from KT household phone directories and the samples from non-listed household or mobile phone surveys were consistent. Only when it is proven that, for example, 20-somethings extracted from the KT phone directory are strongly conservative, while 20-somethings extracted from mobile phones or non-listed households are progressive, can the hypothesis that mobile phone and non-listed household surveys are progressive be validated.

However, the respondents from listed household surveys tend to be older, while respondents from non-listed household surveys are overrepresented by younger individuals. Specifically, for the 30s and 40s age groups, respondents from listed households were 11.5% and 19.2% respectively, below the overall average. Conversely, for the 60 and older age group, 29.5% were from listed households, exceeding the overall average. Ultimately, the tendency for listed household phone surveys to show pro-government leanings and non-listed household/mobile phone surveys to show pro-opposition leanings is likely due to the overrepresentation of older, pro-government individuals in listed households and the overrepresentation of 'younger, pro-opposition individuals' in non-listed households, rather than an actual difference in survey methodology. Therefore, it would be more appropriate to attribute the differences in political attitudes observed in the two survey results to generational effects rather than methodological differences.

This pattern is also confirmed by the Asan Institute for Policy Studies survey results from March 2011 ([Figure 3]), which compared mobile phone, non-listed household phone, and listed household phone surveys. Before applying weights, the sample composition showed overrepresentation of the 20-30 age group in mobile phone and non-listed household phone surveys, and overrepresentation of the 50-60 age group in listed household phone surveys. After applying weights proportional to the voter age distribution, the differences in national approval ratings between the mobile phone surveys conducted by Media Research and R&R, as well as the listed household phone survey by R&R, were within the margin of error and showed no significant meaning.

[Figure 3] Differences in Presidential Approval Ratings by Mobile Phone and Household Phone Surveys (%)

Source: Asan Institute for Policy Studies March press release (mobile phone surveys conducted simultaneously by Media Research and R&R)

Despite this, the media and related industries at the time did not seriously question the prevailing notion that the cause of the 2010 opinion poll credibility crisis was the existence of "hidden opposition votes in mobile phones/non-listed households." Consequently, RDD methods and the combination of mobile phone surveys or 100% mobile phone surveys, which were not adequately verified, rapidly replaced the KT-listed household number survey method within just one year, leading up to the 2012 general elections and presidential election. Has the crisis of opinion poll credibility been resolved now that the KT-listed household number survey method has disappeared? And are the mobile phone surveys and non-listed household surveys, introduced to capture hidden opposition votes, yielding the expected results? However, based on current survey results, this does not seem to be the case. In fact, the Korea Gallup survey results, which use a 100% mobile phone method, show candidate Park Geun-hye of the ruling party being evaluated relatively more competitively than surveys from other agencies that only partially include mobile phone proportions. Conversely, another polling agency using a 100% mobile phone method predicted before the April 11th general election that the opposition party would have an advantage due to 6% of opposition votes not captured by other survey methods that combined landline surveys. However, the election ultimately ended in a victory for the Saenuri Party.

Are ARS/IVR Surveys the Method Used by Gallup in the US?

Another issue is the reliability of low-cost 'automatic response' surveys, also known as ARS (Automatic Response System) or IVR (Interactive Voice Response) surveys, which are used instead of interviewers. Survey organizations that adopt this method claim it proved its accuracy in the 2010 local elections. They also claim this method is used by organizations like Gallup. In 2010, many media outlets reported these claims without verification, leading to the widespread acceptance of this misinformation as fact.

However, caution is advised when applying ARS survey methods to election polls. First, as revealed in the case of the Progressive Justice Party's poll manipulation during the selection of the opposition's unified candidate for the April 11 general election, it is fundamentally impossible for interviewers to verify basic respondent information (e.g., gender and age by voice) or filter out prank responses. Therefore, by strict methodological standards, the ARS survey method is not recognized as a scientific survey method. Second, the low response rate, resulting from a high refusal rate or a high rate of survey abandonment, is also a factor that weakens the reliability of the survey. Third, contrary to the claims of ARS survey organizations, reputable polling organizations in developed countries, such as Gallup, do not use IVR surveys for election polls. Gallup, for instance, recommends its limited use for specific purposes such as collecting customer complaints in the marketing sector. In fact, information on IVR survey items is organized under a sub-menu for the marketing sector, and it is not used at all in major public elections in the United States, such as presidential or general elections. Upon inquiry, our institute confirmed that Gallup does not use the IVR method for its public opinion research, including presidential election polls (Gallup does not use IVR technology for its public opinion research which includes our predisential polls. Figure 4).

[Figure 4] Reply from Gallup regarding the use of ARS (IVR) survey methods for presidential elections (screenshot)

4. Overcoming Horse-Race Election Journalism

While the problems with the survey methods themselves warrant attention, the practice of election journalism, which sensationalistically reports survey results without even a minimal understanding of polling methodologies, cannot be overlooked. Survey results can significantly alleviate cognitive dissonance if interpreted cautiously, taking into account margins of error. Polls are fundamentally a process of measuring the opinion distribution of the entire electorate of 40 million people using a sample of 1,000 to 1,500 individuals. Various errors can occur in this process.

First, there is sampling error. This is the error that occurs during the process of collecting a sample of 1,000 to 1,500 individuals. The error referred to as 'sampling error of ±X% at a 95% confidence level' is precisely this sampling error. Typically, assuming random sampling, the margin of error for a sample of 1,000 is set at ±3.1%. However, in addition to the margin of error determined by sample size, many factors can expand this range. Considering the low response rates and short survey periods in domestic polls, polling organizations use quota sampling, where respondents are substituted according to the convenience of the survey organization, instead of scientific random sampling. Overseas survey methodology textbooks teach that in such cases, the margin of error should be set wider than with random sampling.

For regional analysis, not national analysis, the sample size decreases, and the sampling error increases further. For example, in regional analysis, Busan and Gyeongnam, with relatively large populations, account for 15% of the total electorate. Based on a sample of 1,000, this would correspond to about 150 individuals. That is, there are limitations and errors in inferring the distribution of the 6 million voters in Busan and Gyeongnam through 150 individuals. The recent interpretation that the gap between candidates Ahn Cheol-soo and Moon Jae-in in Honam has narrowed is difficult to ascertain accurately from a single survey result.

In addition to sampling error, various non-sampling errors also occur during the process of collecting, accumulating, organizing, and converting data, depending on the interviewer's proficiency and the survey organization's know-how and management system. Therefore, it is desirable to evaluate the figures derived from polls by considering the sufficient possibility and scope of error.

However, the current media reporting practice is accustomed to evaluating even changes within the margin of error as actual changes in public opinion, rather than analyzing them cautiously by broadly considering the margin of error. 'Margin of error' means that 'differences in figures within that range cannot be considered statistically significant.' In other words, ranking battles within the margin of error are statistically meaningless, and expressions like 'leading within the margin of error' or 'trailing within the margin of error' are not strictly applicable. However, our media goes a step further, actively interpreting 1-2 percentage point changes or differences over a day or two and focusing on horse-race rankings.

For instance, consider the poll results released by Research & Research after the recent Chuseok holiday, reported by the Dong-A Ilbo and the Asan Policy Institute. In a head-to-head contest between candidate Park Geun-hye and candidate Moon Jae-in, the Dong-A Ilbo reported Park's lead (47.4% vs. 44.5%), while the Asan Policy Institute reported Moon's lead (44.5% vs. 42.9%). The difference between the two survey results falls within the permissible margin of error. In the current media reporting climate, which exaggerates differences within the margin of error, these conflicting rankings are jarring and confusing. Observing the media's attitude towards the current confusion in poll reporting, I am concerned that we may witness a repeat of 2010. When problems arise, the media is often more concerned with finding immediate targets for blame than with thoroughly verifying and confirming what is problematic and what is not.

Dramatic changes in public opinion attract readers' attention and generate interest. It is understandable to acknowledge the problem-awareness and survival instincts of survey organizations and media outlets that seek to leverage this. However, the crisis of trust cannot be resolved in such a manner. The priority should be to honestly reveal what interpretations are possible through polls and what are not. And the surest path to regaining trust is to read the situation accurately, step by step, even if it takes a roundabout route. Public opinion is variable, but it does not fluctuate without reason. Distrust is overcome when consistent efforts are accumulated over time. Above all, the biggest question is whether our understanding of voter opinion has deepened commensurately with the introduction of various survey methods and the increase in the volume of surveys. Not only politics but also political polling is in crisis.■