Handbook of Web Surveys. Jelke Bethlehem
K. & Vehovar, V. (2008), Internet Surveys. In: de Leeuw, E., Hox, J. J., & Dillman, D. A. (eds.), International Handbook of Survey Methodology. Lawrence Erlbaum Associates, New York.
13 Singer, E. & Ye, C. (2013), The Use of Incentives in Surveys. Annals of the American Academy of Political and Social Sciences, 645 (1), pp. 112–141.
14 Steinmetz, S., Bianchi, A., Tijdens, K., & Biffignandi, S. (2014), Improving Web Survey Quality—Potentials and Constraints of Propensity Score Weighting. In: Callegaro, M., Baker, R., Bethlehem, J., Göritz, A., Krosnick, J.A., & Lavrakas, P. J. (eds.), Online Panel Research: A Data Quality Perspective. Wiley, Chichester. pp. 273–298.
Chapter Three A Framework for Steps and Errors in Web Surveys
3.1 Introduction
The framework proposed in this chapter is a collection of interrelated ideas and concepts that provide guidance for researchers undertaking a web survey. To carry out a survey, one must conduct a process that involves many steps and decisions. The process‐oriented approach divides the survey into steps that define the flow of the survey process, and this approach must be applied to web surveys as well. Steps differ in some respects from the traditional modes. Thus, the flow of the web survey process must be explicitly defined. Bethlehem and Biffignandi (2012) and Tourangeau, Conrad, and Couper (2013) pointed out the specifics of the web procedures without focusing on the whole process. Thorsdottir and Biffignandi1 presented a flowchart for the steps of a probability‐based web survey. This chapter follows the flowchart approach to illustrate the process, the decisions involved at the various steps of the process, and the related risks of errors. The flowchart will help the surveyor2 in properly planning a web survey and will help researchers highlight the steps of the survey process for their respective research. Starting from the flowchart presented by Thorsdottir and Biffignandi (Webdatanet Conference, 2015), the flowchart presented in this chapter is slightly different since mobile extension of the web survey is also considered (i.e., mobile web survey). Considering the wide diffusion of mobile devices (especially smartphones), mobile web survey is becoming a very frequent, say, mostly a standard, situation. When a web survey is administered if no disallow is activated, the survey may be received and completed not only using a PC but even from a mobile device. This situation is called mobile web survey, i.e., no prevention toward mobile devices in a web survey (for a discussion of the mobile surveys, see Chapter 2). For a detailed description of the choices for smartphone participation in web surveys, see Peterson et al. (2017). Mobile web survey implies several methodological challenges; nevertheless the survey process requires only a few differences with respect to the web‐only survey.
The flowchart divides the mobile web (or the web‐only survey) process into the following six major steps:
1 Determine the survey objective.
2 Metadata definition.
3 Designing the mobile web or only web survey (deciding on the mode, the sampling frame, and the sampling approach, designing the questionnaire, designing the paradata methodology, and selecting the sample, software, or programming language).
4 Collecting the data.
5 Creating the database.
6 Processing data.
The use of the flowchart brings a total survey error (TSE) perspective, which is an effective approach for understanding error sources in a comprehensive way (Weisberg, 2005). Biemer TSE approach aims at helping researchers make design decisions that maximize data quality within the constraints of a limited budget. TSE perspective disaggregates overall error into component, mainly distinguishing between sampling and non‐sampling errors. The TSE paradigm (see, for example, Platek and Särndal, 2001 and the ensuing discussions) refers to the concept of optimally allocating the available survey resources to minimize TSE for key estimates. In principle, to apply the TSE paradigm, the major sources of error should be identified so that the survey resources can be allocated to reduce their errors, within specified constraints on costs. An overview of the TSE history and recent research results on the relationships between different types of errors is given in Biemer et al. (2017). The idea of the TSE is of optimally balancing the dimensions of survey quality within the survey budget and schedule. It is possible to distinguish between TSE and total survey quality. This considers the fitness for use of an estimate, that is, quality from both the producer and user perspectives. The “fitness for use” concept implies not only accuracy is considered; for instance, attributes such as the timeliness, accessibility, and comparability of the data have to be granted in the survey process, as well as various types of errors (see Chapter 5), such as measurement, nonresponse, and so forth (Groves, 1989).
The message in this chapter is that both TSE and total quality survey perspective embody the need to consider different error sources and that errors occur at every step of the survey process and, sometimes, they are interrelated.
In this chapter, a framework of the web survey process is proposed. The framework's main purpose is to provide a clear overview of the necessary decisions when organizing a mobile web survey or web‐only survey and to create a shared overview of the survey process steps. Practitioners, as well as researchers, can refer to the flowchart and obtain a clear understanding of the procedure to follow, the choices to do, and the type of errors that might occur. Thus, there is a more complete, integrated perspective in studying and interpreting errors. Understanding them becomes more easily.
3.2 Theory
The mobile web survey or the web‐only survey process differs in some respects from a survey based on traditional modes (paper and pencil, telephone, and fax). Errors arising in mobile web surveys and only mobile surveys have some specific characteristics due mainly to the coverage aspects of the target population and to various other possibilities (the availability of auxiliary variables, the time required to deliver the questionnaire and to receive the completed questionnaire, technical skills, the equipment needed, and many other situations). Therefore, a well‐defined outline of the survey research steps and of the errors that might occur at each step is vital for the surveyor. Furthermore, in many cases, the errors at different steps have some relationships, even if the relationship is not clearly defined and formalized. Thus, decisions pertaining to the reduction of an error at a specific step could increase the errors occurring at other steps of the survey process. For instance, increasing response rate by stressing the interviewee with a high number of solicitations might decrease the quality of the data (more item nonresponse, less accuracy, and so forth). Therefore, the overall quality of the survey process could be either improved or deteriorated depending on the decision made.
The flowchart mentioned in the previous section is in Figure 3.1. It shows the main steps and sub‐steps for a web‐only or mobile web survey based on probability sample.
Figure 3.1 Flowchart of a probability‐based web‐only or mobile web survey
The six main steps (gray highlighted) and their related sub‐steps (hell gray and white highlighted) are listed in the previous section. Hereunder, detailed comments of the flowchart steps are reported.
The first step is Determine the survey objectives, and it is a preliminary point of every survey, independent from the mode. Objectives need to be clearly stated and the questions to be identified. Since this flowchart considers web mode (either web only or mobile web), it should be critically evaluated if the objectives