ISSN 2288-6168 (Online)
Article Information

Title

Evaluation of Survey Data Quality Based on Interviewers’ Assessments: An Example from Taiwan’s Election and Democratization Study


Author's

Chi-lin Tsai(, Taiwan)

Tsung-Wei Liu(, Taiwan)

Yi-ju Chen(mannertw@gmail.com., Taiwan)


Abstract

Researchers usually examine the quality of survey data by several conventional measures of reliability and validity. However, those measures are mainly designed to examine the quality of an individual measurement, rather than the quality of a data set as a whole. There is a relative lack of methods for evaluation of the overall data quality. This paper attempts to fill this gap. We propose using interviewers’ assessments as one of criteria for evaluating the overall data quality. Interviewers are the ones who literally conduct and thus directly observe interviews. Taiwan’s Election and Democratization Studies (TEDS) have required interviewers to assess how trustworthy the responses of each of their interviewees are, and to provide several descriptions about the process and environment of the interviews. We use this information to evaluate the data quality of TEDS surveys and compare it with the results from the conventional test-retest method. The findings are that the interviewer assessment is a fair indicator of the overall reliability of attitudinal questions but not a good indicator when factual questions are examined. Regarding the evaluation of data validity, more data is required to see whether or not interviewers’ assessment is informative in terms of data quality.


Keywords

data quality, interviewers’ assessment, reliability,, validity,, Taiwan’s Election and Democratization Study


Full Text

View Full Text


References

Achen, C. H. (1975). Mass political attitudes and the survey response. American Political Science Review, 69(4), 1218-1231. https://doi.org/10.2307/1955282

Alvarez, R. M., & Brehm J. (2002). Hard choice, easy answer. Princeton, NJ: Princeton University Press.

Carmines, E. G., & Zeller, R. A. (1979). Reliability and validity assessment. London, England: Sage Publications.

Chu, Y. H. (2004). Tai Wan Xuan Ju Yu Min Zhu Hua Diao Zha, 2003 [Taiwan's Election and Democratization Study, 2003] (TEDS2003). (NSC92-2420-H-001-004). Taipei, Taiwan: Guo Ke Hui Zhuan Ti Yan Jiu Ji Hua Bao Gao Shu [National Science Council Research Project].

Converse, P. E. (1964). The nature of belief systems in mass publics. In Ideology and discontent, Apter, D. (ed). New York, NY: Free Press.

de Vaus, D. (2001). Research design in social research. London: Sage Publications.

Huang, C. (2002). Tai Wan Xuan Ju Yu Min Zhu Hua Diao Zha, 2001 [Taiwan's Election and Democratization Study, 2001] (TEDS2001). (NSC90-2420-H-194-001). Taipei, Taiwan: Guo Ke Hui Zhuan Ti Yan Jiu Ji Hua Bao Gao Shu [National Science Council Research Project].

Huang, C. (2003). Tai Wan Xuan Ju Yu Min Zhu Hua Diao Zha, 2002: Bei Gao Liang Shi Xuan Ju Fang Wen An [Taiwan's Election and Democratization Study, 2002: the Survey of Taipei and Kaohsiung Cities Mayoral Elections] (TEDS2002). (NSC91-2420-H-194-001-SSS). Taipei, Taiwan: Guo Ke Hui Zhuan Ti Yan Jiu Ji Hua Bao Gao Shu [National Science Council Research Project].

Johnson, J. B., Joslyn, R. A., & Reynolds, H. T. (2001). Political science research methods. Washington, DC: CQ Press.

Lavrakas, P. J. (2013). Presidential address: Applying a total error perspective for improving research quality in the social, behavioral, and marketing sciences. Public Opinion Quarterly, 77(3), 831-850. https://doi.org/10.1093/poq/nft033

Liu, T. W., & Chen, K. H. (2004, September). Data quality of the Taiwan’s election and democratization study: Examination of retest and interviewer assessment. Presented at the International Conference of the 2003 Taiwan’s Election and Democratization, Taipei, Taiwan.

Maxim, P. S. (1999). Quantitative research methods in the social sciences. Oxford, England: Oxford University Press.

Olson, K. (2006). Survey participation, nonresponse bias, measurement error bias, and total bias. Public Opinion Quarterly, 70(5), 737-758. https://doi.org/10.1093/poq/nfl038

Peytchev, A., Peytcheva, E., & Groves, R. M. (2010). Measurement error, unit nonresponse, and self-reports of abortion experiences. Public Opinion Quarterly, 74(4), 319-327. https://doi.org/10.1093/poq/nfq002

Shiao, Y. C. 2006. “Tai Wan Xuan Ju Yu Min Zhu Hua Diao Zha” Zai Ce Xin Du Zhi fen Xi [Taiwan Analysis of test-retest reliability in Taiwan's election and democratization study]. Xuan Ju Yan Jiu [Journal of Electoral Studies], 13(2), 117-144.

Sniderman, P. M., Brody, R. A., & Tetlock, P. E. (1991). Reasoning and choice. Cambridge, England: Cambridge University Press.

Tourangeau, R., Groves, R. M., & Redline, C. D. (2010). Sensitive topics and reluctant respondents: Demonstrating a link between nonresponse bias and measurement error. Public Opinion Quarterly, 74(3), 413-432. https://doi.org/10.1093/poq/nfq004

Tudd, C. M., Smith, E. R., & Kidder, L. H. (1991). Research methods in social relations. Orlando, FL: Harcourt Brace Jovanovich.

Zaller, J. (1992). The nature and origins of mass opinion. Cambridge, England: Cambridge University Press.