Testing the Representativeness of a Multimode Survey in South Korea: Results from KAMOS
Sung Kyum Cho, Sarah Prusoff LoCascio, Kay-O Lee, Deok-Hyun Jang, Jong Min Lee
Asian Journal for Public Opinion Research. 2017. February, 4(2): 73-87
The Korean Academic Multimode Open Survey (KAMOS) is a national survey first conducted in 2016. Stratified cluster random sampling was used in an initial face-to-face survey during which panel members were recruited. The second survey allowed invited panel members to answer online or by phone. KAMOS includes both longitudinal items and omnibus items, i.e., researchers can propose questions to include on KAMOS. This paper seeks to establish that KAMOS is representative of the South Korean adult population. The demographic variables from the first survey were comparable to demographic variables from two well-respected surveys in South Korea: the KOSTAT Social Survey and the Gallup Korea Omnibus Survey. To ensure that there was no substantial difference between those who answered the first survey and those who answered the second survey, we compared the results of 22 items from the first survey. The 2,000 panel members who were invited to participate in the second survey had similar responses to the 1,008 of those who responded to the second survey. Based on our findings, KAMOS can be considered a representative sample.
KAMOS; online panel survey; multimode survey; South Korea; Korean Academic Multimode Open Survey
- The Need for a Representative Survey in South KoreaPublic opinion surveys are an important tool, especially in a democracy. They can be used to help form public policy and by public campaigns to understand current attitudes and help plan for future campaigns. Scholars, especially in the social sciences, often use public opinion surveys in their research. To be useful, a survey should be as accurate as possible. Although probability sampling is considered the most scientific and accurate sampling method, it is also cost prohibitive for many scholars. Some have been tempted to use online panels, but most Internet panels in Korea are problematic, because not everyone uses the Internet. The Internet penetration rate is high in South Korea, with 88.3% of those aged 3 or older having used the Internet within the past month, according to a survey conducted in July 2016 (Ministry of Science, ICT, and Future Planning, 2016). Therefore, if a survey is only conducted online, a portion of the population (just over 11.7% in the case of South Korea) will not be included, making non-coverage error a serious concern. In addition, many online survey panels are voluntary panels, which is not the gold standard in sampling. It became clear that scholars needed access to a panel that is inexpensive or free to use, but representative of the South Korean public. Based on this need, the National Research Foundation (NRF) supported the Korean Academic Multimode Open Survey (KAMOS), with funding from the Ministry of Education. This paper seeks to evaluate the results from the first year of the KAMOS survey to determine whether it can be considered representative.
- Literature Review: Attempts from Other CountriesOther countries have also felt the need for a nationally representative, probability-based, online survey panel. The first four such panels in Europe that attempted to include non-Internet using households were: the LISS (Longitudinal Internet Studies for the Social sciences) Panel in the Netherlands, the GIP and GESIS Panels in Germany, and the ELIPSS Panel in France (Norway and Iceland also made probability-based Internet panels, but did not attempt to include the non-Internet using population) ( Blom et al., 2016 ). All four of these panels used probability-sampling offline to recruit panel members. To include non-Internet using households, two of the panels, LISS and GESIS, provided Internet and equipment to those households. The ELIPSS Panel provided a tablet and 3G Internet access to all their panel members. The GIP Panel chose to use a multimode approach instead: paper questionnaires are mailed to their panel members without Internet access or those who prefer not to answer online ( Blom et al., 2016 ). In the USA, probability-based Internet panels include Knowledge Networks (GFK Knowledge Panel®), the American Life Panel (RAND), the Understanding America Study, the Gallup Panel, and AmeriSpeak (NORC). Like their counterparts in Europe, panel members for these panels were recruited offline, using addresses, phone numbers, or a combination of the two ( Hays, Liu, & Kapteyn, 2015 ; AmeriSpeak, 2015 ; Gallup Panel, 2017 ). The recruitment rates for these panels are not always as high as ideal. The highest was the LISS Panel, with 48.3%, i.e., of their sample, 48.3% agreed to join the panel ( Blom et al., 2016 ). In comparison, the KnowledgePanel® has an approximate signup rate of about 10% ( Hays et al., 2015 ). It has been suggested that, while non-coverage error and error due to self-selection that come with non-probability based Internet panels is a concern, probability-sampling based panels may also have problems if too many people refuse to participate or drop out ( Hays et al., 2015 ).
- Creation of KAMOSKAMOS, like the other panels mentioned, uses two-stage stratified cluster sample design. The primary sampling unit (PSU) is the geographical units created by Statistics Korea (KOSTAT) and the secondary sampling unit (SSU) is the household. The PSUs are selected with probability proportional to the size of the sampling unit. Out of 82,666 statistical blocks made by KOSTAT, 106 statistical blocks were selected. The SSUs are sampled with equal probability. We attempted to interview all members of selected households aged 18 or older and invited them to join the KAMOS panel. KOSTAT also interviews all members of selected households for many of their surveys, including the KOSTAT Social Survey. Respondents who do not have Internet access may participate in KAMOS via face-to-face interviews or telephone interviews, making KAMOS a multimode survey. In the second survey, which was mostly conducted online, 9.6% of respondents used another mode. The initial face-to-face KAMOS survey took place February-May 2016. The sample size was 4,770 people. The response rate was 63% (3,004 people) and participants were asked to participate in the panel. For subsequent surveys, a random sample of 2,000 of the panel members is invited to respond. The second survey took place June 23-August 9, 2016, and the response rate was 50.4% (1,008 people).
MethodologyThe results of the KAMOS survey are weighted based on age, sex, and geographical region of Korea and the weighted results were then tested for representativeness. We analyzed the representativeness of the panel in two ways. First, we wanted to confirm that the demographics of the KAMOS respondents seemed similar to other reputable surveys in Korea. Therefore, we compared the demographics of our respondents to two major surveys in South Korea. The first survey is the KOSTAT Social Survey, which is conducted annually by KOSTAT, the government organization that is responsible for official statistics. The demographic variables from the KOSTAT Social Survey can be considered representative of the country. It should be noted that there are some expected differences, because the KOSTAT Social Survey interviews households instead of individuals. We used the results from the 2015 KOSTAT Social Survey. We also compared our results to the Gallup Korea Omnibus Survey, which is conducted every two months by a private company in South Korea. We looked at the average of the Gallup Korea Omnibus Survey results from 2015. Not all of our demographic variables were collected by both surveys. To ensure that the results from our panel continued to be valid, we compared the second survey (primarily online, with 9.6% offline) to our initial respondents. Rather than continuing to look only at demographic variables, we checked whether there were actual differences between the answers of those who responded to the first survey and those who responded to the second survey. We took the responses from the first survey of the 2,000 respondents (randomly selected from the 3,004 respondents to the first survey) who were invited to participate in the second survey and compared these responses to the first survey responses of the 1,008 of these 2,000 who responded to the second survey. If there were no statistically significant differences between these two sets of responses, then it would suggest that the KAMOS panel is representative.
- Demographic Variables from the Initial SurveyMost of the differences found in the demographic variables between KAMOS and the KOSTAT Social Survey and Gallup Korea Omnibus Survey were relatively small and suggest that, based on demographic variables; KAMOS is representative of the Korean public. As shown in Table 1 , KAMOS respondents may have been slightly more educated than the respondents to other surveys, with fewer having left school before high school and slightly more having attended a university, but again, these were small differences, fewer than 5 percentage points.
Education Level (%)
Job Type (%)
Marital Status (%)
Self-Reported Social Class (%)
Approximate Monthly Household Income in US Dollars1
Number of Household Members (%)
Type of Dwelling (%)
- Comparison of the First and Second Survey ResultsOf the 72 non-demographic items included on the first KAMOS survey, 22 items could be grouped into positive and negative responses (on a 4-point Likert-type scale). By comparing the results of the first survey respondents who were invited to participate in the second survey to the first survey responses of these who responded to the second survey (50.4%), we hoped to determine if there was any substantial difference between those who answered the first survey only and those who answered both surveys. The differences between the two sets of data were not statistically significant (p<0.05). Of the 22 questions we looked at, only 2 had a difference larger than 4 percentage points. The first of these two questions, Q12 was,
- First Survey Results – Invited/Accepted Results from 1st survey
ConclusionOur analysis suggests that KAMOS can be considered representative of the South Korean adult population. The demographic variables of KAMOS respondents/panel recruits matched the demographic variables of respondents to other respected surveys. A comparison between the answers to selected questions from the first survey by those invited to respond and those who actually responded to the second survey showed only small differences, suggesting that the panel remained substantially the same. Continued monitoring of the KAMOS results will of course be necessary to ensure that it continues to be representative. Like other panels, such as the LISS panel ( Blom et al., 2016 ), we anticipate the need to refresh the KAMOS panel at regular intervals. The current proposed plan is to refresh the panel annually, i.e, every year there will be a face-to-face survey during which new panelists will be recruited to participate in the multimode survey; there will be two additional surveys per year. The schedule may change after the analysis of the third survey or other future surveys. In the meantime, KAMOS is available as a representative survey for scholars interested in the public opinion of South Koreans.
AmeriSpeak 2015 The AmeriSpeak advantage. http://www.norc.org/PDFs/AmeriSpeak%20Advantage%20updated%20%202015%2001%2026-FINAL%20WEB%20FILE.PDF
Blom A. G. , Bosnjak M. , Cornilleau A. , Cousteaux A. S. , Das M. , Douhou S. , Krieger U. 2016 A comparison of four probability-based online and mixed-mode panels in Europe. Social Science Computer Review 34 (1) 8 - 25
Gallup 2017 Reach the Right Audience: Gallup Panel. http://www.gallup.com/services/172364/gallup-panel.aspx
Hays R.D. , Liu H. , Kapteyn A. 2015 Use of Internet panels to conduct surveys. Behav Res Methods 47 (3) 685 - 690 DOI : 10.3758/s13428-015-0617-9
Ministry of Science, ICT, and Future Planning (MSIFT) 2017 2016 Internet iyong siltae josa [2016 Survey on the Internet usage]. http://msip.go.kr/SYNAP/skin/doc.html?fn=974286241e56e1b5eeaa66c586891402&rs=/SYNAP/sn3hcv/result/201702/
Statistics Korea (KOSTAT) 2010 Yundobyuljosakyulkwa [Census Result by Year]. http://www.census.go.kr/dat/ysr/ysrList.do?q_menu=5&q_sub=7