Handbook of Web Surveys. Jelke Bethlehem
expression goes back to rural America. Farmers would throw a handful of straws into the air to see which way the wind was blowing.
It took until the 1920s before more attention was paid to sampling aspects. Lienhard (2003) describes how George Gallup worked out new ways to measure interest in newspaper articles. Gallup used quota sampling. The idea was to investigate a group of people that could be considered representative for the population. Hundreds of interviewers across the country visited people. Interviewers were given quota for different groups of respondents. They had to interview so many middle‐class urban women, so many lower‐class rural men, etc. In total, approximately 3,000 interviews were conducted out for a survey.
Gallup's approach was in great contrast with that of the Literary Digest magazine, which was at that time the leading polling organization. This magazine conducted regular “America Speaks” polls. It based its predictions on returned questionnaire forms that were sent to addresses taken from telephone directories books and automobile registration lists. The sample size for these polls was on the order of two million people. So the sample size was much larger than that of Gallup's polls.
The presidential election of 1936 turned out to be decisive for both methods. This is described by Utts (1999). Gallup correctly predicted Franklin Roosevelt to be the new president, whereas Literary Digest predicted that Alf Landon would beat Franklin Roosevelt. The prediction based on the very large sample size turned out to be wrong. The explanation was that the sampling technique of Literary Digest did not produce representative samples. In the 1930s, cars and telephones were typically owned by middle‐ and upper‐class people. These people tended to vote Republican, whereas lower‐class people were more inclined to vote Democrat. Consequently, Republicans were overrepresented in the Literary Digest sample.
As a result of this historic mistake, opinion researchers learned that they should rely on more scientific ways of sample selection. They also learned that the way a sample is selected is more important than the size of the sample.
The classical theory of survey sampling was more or less completed in 1952. Horvitz and Thompson (1952) developed a general theory for constructing unbiased estimates. Whatever the selection probabilities are, as long as they are known and positive, it is always possible to construct a useful estimate. Horvitz and Thompson completed the classical theory, and the random sampling approach was almost unanimously accepted. Most of the classical books about sampling were also published by then (Cochran, 1953; Deming, 1950; Hansen, Hurvitz, and Madow, 1953; Yates, 1949).
1.2.2 TRADITIONAL DATA COLLECTION
There were three modes of data collection in the early days of survey research: face‐to‐face interviewing, mail interviewing, and telephone interviewing. Each mode had its advantages and disadvantages.
Face‐to‐face interviewing was already used for the first censuses. Thus, it is not a surprise it was also used for surveys. Face‐to‐face interviewing means that interviewers visit the persons selected in the sample. Well‐trained interviewers will be successful in persuading reluctant persons to participate in the survey. Therefore, response rates of face‐to‐face surveys are usually higher than surveys not involving interviewers (for example, mail surveys). Interviewers can also assist respondents in giving the right answers to the questions. This often results in better data. However, the presence of interviewers can also be a drawback. Research suggests that respondents are more inclined to answer sensitive questions properly if there are no interviewers present.
Survey agencies often send a letter announcing the visit of the interviewer. Such a letter can also give additional information about the survey, explain why it is important to participate, and assure that the collected information is treated confidentially. As a result, the respondents are not taken by surprise by the interviewers.
The response rate of a face‐to‐face survey is usually high and so is quality of the collected data. But a price has to be paid literally: face‐to‐face interviewing is much more expensive. A team of interviewers has to be trained and paid. They also have to travel, which costs time and money.
Mail interviewing is much less expensive than face‐to‐face interviewing. Paper questionnaires are sent by mail to persons selected in the sample. They are invited to answer the questions and to return the completed questionnaire to the survey agency. A mail survey is not interviewers based. Therefore, it is a cheaper mode of data collection than face‐to‐face survey. Data collection involve mailing costs (letters, postage, envelopes) both for sending the questionnaire and for delivering the questionnaire back; similar costs have to be considered for each reminder. Therefore, costs for stamps and questionnaire printing could be not completely irrelevant. Another advantage of mail survey is that the absence of interviewers can be experienced as less threatening for potential respondents. Therefore, respondents are more inclined to answer sensitive questions properly.
The absence of interviewers also has a number of disadvantages. There are no interviewers to explain questions or assist respondents in answering them. This may cause respondents to misinterpret questions, which has a negative impact on the quality of the collected data. Furthermore, it is not possible to use show cards. A show card is typically used for answering closed questions. Such a card contains the list of all possible answers to a question. Respondents can read through the list at their own pace and select the answer corresponding to their situation or opinion. Mail surveys put high demands on the design of the paper questionnaire. For example, it should be clear to all respondents how to navigate through the questionnaire and how to answer questions.
Since the persuasive power of the interviewers is absent, response rates of mail surveys tend to be low. Of course, reminder letters can be sent, but this is often not very successful to let the response rate become very high. More often survey questionnaire forms end up in the pile of old newspapers.
In summary, the costs of a mail survey are relatively low, but often a price has to be paid in terms of data quality: response rates tend to be low, and also the quality of the collected data is also often not very good. Dillman (2007) believes, however, that good results can be obtained by applying his Tailored Design Method. This is a set of guidelines for designing and formatting mail survey questionnaires. They pay attention to all aspects of the survey process that may affect response rates or data quality.
Face‐to‐face interviewing was preferred in the early days of survey interviewing in the Netherlands. The idea was in the 1940s that poor people had poor writing skills and they were not interested in the topics of the surveys. Therefore, they had a lower probability to complete mail questionnaires. People completing and returning questionnaire forms were assumed to be more interested in the survey topics because their intelligence and socioeconomic position was above average.
A third mode of data collection is telephone interviewing. Interviewers are needed to conduct a telephone survey, but not as many as for a face‐to‐face survey, since they do not have to travel from one respondent to the next. They can remain in the call center of the survey agency and conduct more interviews in the same amount of time. Therefore, interviewer costs are less high. An advantage of telephone interviewing over face‐to‐face interviewing is that respondents may be more inclined to answer sensitive questions, because the interviewer is not present in the room. A drawback in the first days of telephone surveys may be that telephone coverage in the population was low. Not every respondent could be contacted by telephone.
Telephone interviewing has some limitations. Interviews cannot last too long, and question answer may not be written. Obviously, no show cards can be used; lists can be presented by reading them out loud (by the interviewers).
This implies a possible recency effect in the answers. Another problem may be the lack of a proper sampling frame. Telephone directories may suffer from severe under‐coverage because many people do not want their phone number to be listed in the directory. Another new development is that increasingly people replace their landline phone by a mobile phone. This fact increases under‐coverage in the telephone directories. For example, according to Cobben and Bethlehem (2005), only between 60% and 70% of the Dutch population