SO 300 Research Methods Paper

Introduction

This assignment examines and applies concepts in the Survey Methods chapter and gives you the opportunity to evaluate surveys.

Instructions

  1. Using the principles in Chapter 7 of surveys, locate 2 surveys (can be related to your area of interest). One of these should be a survey/ questionnaire that you will justify as high quality. The second survey should be one that you will justify as poor or deficient.
  2. In an APA Style paper, present your responses to the following prompts:
    1. Describe each survey applying key concepts from the textbook [be sure to describe the:
    2. Purpose or goal of the surveys
    3. IV’s and DV’s  ****the items (IVs) that are presented and the ways participants can respond (DVs),
    4. Critique of the surveys along with a justification of whether you judged it as high quality or poor quality.
    5. A statement about improving upon the surveys.
  3. Be sure to include internal citations and references for both surveys.

Your paper must be presented in proper APA 7e format.

Your paper should have a cover page, text body section, and a references. An abstract is not necessary – do not put an abstract into your paper.

Book: Making Sense of the Social World Methods of Investigation Sixth Edition Daniel F. Chambliss

Survey Research

Chapter 7

 

 

1

Chambliss, Making Sense of the Social World, 6e. © SAGE Publishing, 2020

Learning Objectives

Explain the strengths and weaknesses of omnibus surveys.

Explain the problem of sampling on the dependent variable.

Discuss the advantages and disadvantages of including “don’t know” and neutral responses among response choices and of using open-ended questions.

2

 

 

2

Chambliss, Making Sense of the Social World, 6e. © SAGE Publishing, 2020

Learning Objectives

List the different methods for improving survey questions.

Outline a cover letter for a survey that contains each of the required elements.

List the strengths and weaknesses of each mode of survey design, giving particular attention to response rates.

Discuss the key ethical issues in survey research.

3

 

 

3

Chambliss, Making Sense of the Social World, 6e. © SAGE Publishing, 2020

Why Is Survey Research So Popular?

Survey research

Popularity due to three advantages:

Versatility

Efficiency

Generalizability

Omnibus survey

 

4

 

Learning Objective: 7-1: Explain the strengths and weaknesses of omnibus surveys.

 

Survey research: Research in which information is collected from a sample of individuals through their responses to a set of standardized questions. Omnibus survey: A survey that covers a range of topics of interest to different social scientists.

4

Chambliss, Making Sense of the Social World, 6e. © SAGE Publishing, 2020

How Should We Write Survey Questions?

Be Clear; Avoid Confusing Phrasing

Simple, direct approach

Avoid negative phrases or words

Especially double negatives

Avoid double-barreled questions

Identify what information each question is to obtain

5

 

Learning Objective: 7-1: Explain the strengths and weaknesses of omnibus surveys.

Learning Objective: 7-4: List the different methods for improving survey questions.

 

Double negatives: A question or statement that contains two negatives, which can muddy the meaning of the question.

Double-barreled question: A single survey question that actually asks two questions but allows only one answer.

5

Chambliss, Making Sense of the Social World, 6e. © SAGE Publishing, 2020

How Should We Write Survey Questions?

Minimize Bias

Words should not trigger biases (unless this is the intent)

Must test reactions to phrasing of a question.

6

 

Learning Objective: 7-1: Explain the strengths and weaknesses of omnibus surveys.

Learning Objective: 7-4: List the different methods for improving survey questions.

 

6

Chambliss, Making Sense of the Social World, 6e. © SAGE Publishing, 2020

How Should We Write Survey Questions?

Allow for Disagreement

Some respondents “agree” with a statement just to avoid disagreeing.

Present both sides of attitude scales in question itself.

Phrase all response choices in socially approved ways.

7

 

Learning Objective: 7-1: Explain the strengths and weaknesses of omnibus surveys.

Learning Objective: 7-4: List the different methods for improving survey questions.

 

7

Chambliss, Making Sense of the Social World, 6e. © SAGE Publishing, 2020

How Should We Write Survey Questions?

Don’t Ask Questions They Can’t Answer

Respondents should be competent to answer questions.

Kinds of questions to mitigate this:

Filter questions

Skip pattern

Contingent question

8

 

Learning Objective: 7-1: Explain the strengths and weaknesses of omnibus surveys.

Learning Objective: 7-4: List the different methods for improving survey questions.

 

Filter questions: A survey question used to identify a subset of respondents who then are asked other questions.

Skip pattern: The unique combination of questions created in a survey by filter questions and contingent questions.

Contingent question: A question that is asked of only a subset of survey respondents.

8

Chambliss, Making Sense of the Social World, 6e. © SAGE Publishing, 2020

How Should We Write Survey Questions?

Allow for Uncertainty

Different types of respondents:

Floaters

Fence-sitters

Can be managed by . . .

Including “no opinion” category

Including neutral category

Adding open-ended questions

 

9

 

Learning Objective: 7-3: Discuss the advantages and disadvantages of including “don’t know” and neutral responses among response choices and of using open-ended questions.

Learning Objective: 7-4: List the different methods for improving survey questions.

 

Floaters: Survey respondents who provide an opinion on a topic in response to a closed-ended question that does not include a “Don’t know” option but who will choose “Don’t know” if it is available.

Fence-sitters: Survey respondents who see themselves as being neutral on an issue and choose a middle (neutral) response that is offered.

9

Chambliss, Making Sense of the Social World, 6e. © SAGE Publishing, 2020

How Should We Write Survey Questions?

Make Response Categories Exhaustive and Mutually Exclusive

Must be one and only one possible response for everyone who is asked the question.

Two exceptions:

Filter questions

“Check all that apply” questions

10

 

Learning Objective: 7-3: Discuss the advantages and disadvantages of including “don’t know” and neutral responses among response choices and of using open-ended questions.

Learning Objective: 7-4: List the different methods for improving survey questions.

 

10

Chambliss, Making Sense of the Social World, 6e. © SAGE Publishing, 2020

How Should Questionnaires Be Designed?

Questionnaire

 

Interview schedule

11

 

Learning Objective: 7-6: List the strengths and weaknesses of each mode of survey design, giving particular attention to response rates.

 

Questionnaire: A survey instrument containing the questions in a self-administered survey.

Interview schedule: A survey instrument containing the questions asked by the interviewer in an in-person or phone survey.

11

Chambliss, Making Sense of the Social World, 6e. © SAGE Publishing, 2020

How Should Questionnaires Be Designed?

Build on Existing Instruments

Use an existing instrument if it measures the concept of concern.

Consider appropriateness for your survey population.

12

 

Learning Objective: 7-6: List the strengths and weaknesses of each mode of survey design, giving particular attention to response rates.

 

12

Chambliss, Making Sense of the Social World, 6e. © SAGE Publishing, 2020

How Should Questionnaires Be Designed?

Refine and Test Questions

Methods for testing questions:

Cognitive interview

Conduct a pilot study

Interpretive questions

13

 

Learning Objective: 7-4: List the different methods for improving survey questions.

Learning Objective: 7-6: List the strengths and weaknesses of each mode of survey design, giving particular attention to response rates.

 

Cognitive interview: A technique for evaluating questions in which researchers ask people test questions, and then probe with follow-up questions to learn how they understood the question and what their answers mean.

Interpretive questions: Questions included in a questionnaire or interview schedule to help explain answers to other important questions.

13

Chambliss, Making Sense of the Social World, 6e. © SAGE Publishing, 2020

How Should Questionnaires Be Designed?

Maintain Consistent Focus

Have measures of all independent and dependent variables you plan to use.

Long lists of redundant or unimportant questions bore respondents.

Omnibus surveys are an exception.

14

 

Learning Objective: 7-6: List the strengths and weaknesses of each mode of survey design, giving particular attention to response rates.

14

Chambliss, Making Sense of the Social World, 6e. © SAGE Publishing, 2020

How Should Questionnaires Be Designed?

Order the Questions

Sort questions into thematic categories.

First question should . . .

Interest the respondent

Be easy to answer

Apply to everyone

NOT be sensitive

Context effects

15

 

Learning Objective: 7-6: List the strengths and weaknesses of each mode of survey design, giving particular attention to response rates.

 

Context effects: In survey research, refers to the influence that earlier questions may have on how subsequent questions are answered.

15

Chambliss, Making Sense of the Social World, 6e. © SAGE Publishing, 2020

How Should Questionnaires Be Designed?

Make the Questionnaire Attractive

Plenty of white space between questions.

List response choices vertically.

Distinguish response choices clearly.

Indicate skip patterns with graphics.

Use distinctive formatting for instructions.

Booklets are better than multi-pagers.

16

 

Learning Objective: 7-6: List the strengths and weaknesses of each mode of survey design, giving particular attention to response rates.

 

16

Chambliss, Making Sense of the Social World, 6e. © SAGE Publishing, 2020

What Are the Alternatives for Administering Surveys?

Surveys can be administered in five ways

Each approach differs in one or more way:

Manner of administration

Questionnaire structure

Setting

Cost

17

 

Learning Objective: 7-6: List the strengths and weaknesses of each mode of survey design, giving particular attention to response rates.

Learning Objective: 7-5: Outline a cover letter for a survey that contains each of the required elements.

 

17

Chambliss, Making Sense of the Social World, 6e. © SAGE Publishing, 2020

What Are the Alternatives for Administering Surveys?

Mailed, Self-Administered Surveys

Low response rate hurts representativeness.

Response rates can be improved with a cover letter.

18

 

Learning Objective: 7-6: List the strengths and weaknesses of each mode of survey design, giving particular attention to response rates.

Learning Objective: 7-5: Outline a cover letter for a survey that contains each of the required elements.

 

Mailed (self-administered) survey: A survey involving a mailed questionnaire to be completed by the respondent.

Cover letter: The letter sent with a mailed questionnaire that explains the survey’s purpose and auspices and encourages the respondent to participate.

18

Chambliss, Making Sense of the Social World, 6e. © SAGE Publishing, 2020

What Are the Alternatives for Administering Surveys?

19

 

Learning Objective: 7-6: List the strengths and weaknesses of each mode of survey design, giving particular attention to response rates.

Learning Objective: 7-5: Outline a cover letter for a survey that contains each of the required elements.

 

19

Chambliss, Making Sense of the Social World, 6e. © SAGE Publishing, 2020

What Are the Alternatives for Administering Surveys?

Group-Administered Surveys

Purpose of survey should be explained before distribution.

Use a cover letter.

Distribute envelopes to ensure confidentiality.

If respondents feel coerced, may not answer honestly.

20

 

Learning Objective: 7-6: List the strengths and weaknesses of each mode of survey design, giving particular attention to response rates.

 

Group-administered survey: A survey that is completed by individual respondents who are assembled in a group.

20

Chambliss, Making Sense of the Social World, 6e. © SAGE Publishing, 2020

What Are the Alternatives for Administering Surveys?

Telephone Surveys

Reaching Sampling Units

Most use RDD to contact households.

With cell phones, accurate coverage of random samples is almost impossible.

Use multiple callbacks to maximize response.

Interactive voice response

21

 

Learning Objective: 7-6: List the strengths and weaknesses of each mode of survey design, giving particular attention to response rates.

 

Phone survey: A survey in which interviewers question respondents over the phone and record their answers.

Interactive voice response: A survey in which respondents receive automated calls and answer questions by pressing numbers on their touch-tone phones or speaking numbers that are interpreted by computerized voice recognition software.

21

Chambliss, Making Sense of the Social World, 6e. © SAGE Publishing, 2020

What Are the Alternatives for Administering Surveys?

In-Person Interviews

Advantages:

High response rates

Questionnaires can be longer, more complex

Interview can control order of questions

Can monitor setting of the interview

Interpretations of questions can be clarified

 

22

 

Learning Objective: 7-6: List the strengths and weaknesses of each mode of survey design, giving particular attention to response rates.

 

In-person interview: A survey in which an interviewer questions respondents face-to-face and records their answers.

22

Chambliss, Making Sense of the Social World, 6e. © SAGE Publishing, 2020

What Are the Alternatives for Administering Surveys?

In-Person Interviews

Disadvantages:

Must carefully train and supervise interviewers.

Computer-assisted personal interview

Harder for respondents to give honest answers about sensitive topics.

 

23

 

Learning Objective: 7-6: List the strengths and weaknesses of each mode of survey design, giving particular attention to response rates.

 

Computer-assisted personal interview (CAPI): A personal interview in which the laptop computer is used to display interview questions and to process responses that the interviewer types in, as well as to check that these responses fall within allowed ranges.

23

Chambliss, Making Sense of the Social World, 6e. © SAGE Publishing, 2020

What Are the Alternatives for Administering Surveys?

In-Person Interviews

Maximizing Response to Interviews

Contact rates tend to be lower in central cities.

Less education associated with higher rates of “Don’t know” responses.

Simply asking certain questions can depress response rates.

 

24

 

Learning Objective: 7-6: List the strengths and weaknesses of each mode of survey design, giving particular attention to response rates.

 

24

Chambliss, Making Sense of the Social World, 6e. © SAGE Publishing, 2020

What Are the Alternatives for Administering Surveys?

Electronic Surveys

E-mail survey

Easy to develop and use.

Cumbersome for longer surveys.

 

 

25

 

Learning Objective: 7-6: List the strengths and weaknesses of each mode of survey design, giving particular attention to response rates.

 

Electronic survey: A survey that is sent and answered by computer, either through e-mail or on the web.

E-mail survey: A survey that is sent and answered through e-mail.

25

Chambliss, Making Sense of the Social World, 6e. © SAGE Publishing, 2020

What Are the Alternatives for Administering Surveys?

Electronic Surveys

Web or online survey

Advantages:

Questions can be tailored for respondents

Can elicit more honest reports about socially undesirable behavior.

Disadvantages:

Coverage bias

Low rates of completion

26

 

Learning Objective: 7-6: List the strengths and weaknesses of each mode of survey design, giving particular attention to response rates.

 

Web/online survey: A survey that is accessed and responded to on the world wide web.

26

Chambliss, Making Sense of the Social World, 6e. © SAGE Publishing, 2020

A Comparison of Survey Designs

Mailed surveys

Weakest from sampling standpoint

Cheap and useful for sensitive questions

Phone surveys

Sampling and response problems

Must be of limited length and complexity

 

 

27

 

Learning Objective: 7-6: List the strengths and weaknesses of each mode of survey design, giving particular attention to response rates.

 

27

Chambliss, Making Sense of the Social World, 6e. © SAGE Publishing, 2020

A Comparison of Survey Designs

In-person surveys

Can be long and complex

Require training

Electronic surveys

Advantages depend on population

Can be effective for those with access and ability.

28

 

Learning Objective: 7-6: List the strengths and weaknesses of each mode of survey design, giving particular attention to response rates.

 

28

Chambliss, Making Sense of the Social World, 6e. © SAGE Publishing, 2020

Ethical Issues in Survey Research

Survey research poses fewer ethical dilemmas than experimental designs.

Confidentiality is primary focus of ethical concern.

Preserve anonymity

Possible only if no follow-ups needed

 

 

29

 

Learning Objective: 7-7: Discuss the key ethical issues in survey research.

 

Anonymity: Provided by research in which no identifying information is recorded that could be used to link respondents to their responses.

29

Chambliss, Making Sense of the Social World, 6e. © SAGE Publishing, 2020

Conclusion

Surveys are efficient and productive

Most popular research method in sociology

Due to widespread use, some survey results are worthless

Careful planning and construction are necessary

30