H. Wesley Perkins, Ph.D. National Social Norms Resource Center, August 2005.

See also: The Social Norms Review (all issues).

Download the original PDF (6 pages, 81 KB)

Contents

Welcome to the premier issue of The Social Norms Review!

This new, electronic publication of the National Social Norms Resource Center is intended to provide a forum for the in-depth examination of topics and issues of relevance to the social norms approach. Issued quarterly, the Review will feature timely and important articles by researchers, theorists, and practitioners of the social norms approach.

For our premier issue we are pleased to offer “Critical Concerns for Evaluating Social Norms Interventions with Survey Data,” by Dr. H. Wesley Perkins. Based on a presentation given by Dr. Perkins in early 2005 to a select meeting of universities using the social norms approach to address alcohol-related issues, the concerns it raises have clear implications for all those involved in the evaluation of outcome data. Needless to say, each of these concerns is applicable not only to alcohol-related projects, but to the full range of issues that the social norms approach is used to address, such as seat-belt usage, tax compliance, the reduction of tobacco use, etc.

We hope that you find both this and future issues of the Review to be informative and helpful, and we welcome your comments, suggestions, and submissions.

Rich Rice [email protected]
Editor, The Social Norms Review

Critical Concerns for Evaluating Social Norms Interventions with Survey Data

By H. Wesley Perkins, Ph.D.
Hobart and William Smith Colleges

The social norms approach is an intensely data-driven process. For example, data are gathered in social norms projects in order to establish baseline measures, to provide verifiable information for normative messages, to identify effective media channels and credible messages (sometimes called market research), to perform process evaluation and, as part of outcome evaluation, to assess the effectiveness of the intervention. Each of these areas is important, of course, and has its own particular challenges. In this article I would like to focus on a number of key issues affecting outcome evaluation; specifically, I want to examine in detail seven concerns that are critically important for the proper evaluation of survey data gathered in a social norms project.

I. Establishing Clean, Reliable Data

A. Consistent Coding

The importance of an accurate listing of the codes assigned to the response options of each survey item cannot be over-stressed. As a general rule, one should maintain consistent coding across multiple survey administrations. For example, if gender is coded as “1” for “female” and “2” for “male” at baseline (also called Time 1), then this coding should be maintained for subsequent survey administrations. Simple coding errors are more likely to occur when different people edit the survey or manage the data from year to year.

While the above suggestion may seem obvious, simple coding errors often mask positive findings or suggest an effect where one did not occur. It is good idea to perform “spot checks” of your data. If your surveys are scanned using an optical scanner, check with the scanner operator to ensure that he or she is using the correct codes. If the data is hand entered, randomly select several bundles to check the consistency of the data entry team. It is also a good idea to scan the frequency data for findings that are inconsistent with your expectations as these may suggest coding errors. For example, in looking at data on alcohol use one might be understandably surprised to discover that women were 5 times more likely than men to get arrested. A check on the coding may reveal that the gender values had been reversed from previous iterations of the survey.

B. Decision Rules for Malicious Data and Outliers

Extremely aberrant responses (outliers), respondents’ accidental reporting errors, intentionally false responses, and logical impossibilities must be dealt with and documented in a consistent fashion. Such odd data (often referred to as “noise”) can distort the data analysis and potentially mask the impact of an intervention and/or lead to inappropriate conclusions. One must establish rules to filter aberrant response data from the data set before analysis. Examples of aberrant data might include: 1) a respondent who indicates a weight of 800 lbs. or the consumption of 99 alcoholic drinks at a party or bar, or 2) a respondent who indicates having no alcoholic drinks during the past 30 days on one survey item and having driven a car after drinking on four occasions during the last month on a separate survey item.

If a single inconsistency appears amidst an otherwise cogent set of responses provided by a survey respondent it may be appropriate to simply treat the aberrant response as missing data (assuming an unintentional error by the respondent). If, however, the respondent provides several inconsistencies or preposterous answers, then it is likely that the entire case record should be omitted from further analyses with the data set.

C. Decision Rules for Items Left Blank

Blank responses must be addressed for each survey item and documented. For example, respondents might be presented with a list of survey items about negative consequences of drinking or a list of intervention messages they may have seen and asked to indicate by checking a corresponding box which items they had experienced during the past school year. A non-response to survey items such as these is usually inferred to mean “I did not experience this consequence” or “I did not see this message.”

Respondents will occasionally skip over survey items, however, and if none of the boxes is checked in these examples, one should attempt to determine if the lack of response to these items really means “no” or if the data should be coded as “missing” for that respondent. One might examine items preceding or following such a list to see if the rest of the page was completed as one indication. Ideally, a box indicating “no” for each item or a box at the end of the list indicating “none of the above” should be included to make the response clear.

II. Demographic Variation across Samples

It is important to identify any demographic variability in your samples when comparing the results from different survey administrations. Ideally, the sample demographics should be representative of the populations from which they were drawn and similar across survey administrations if the population has not changed. Because this is not always the case, demographic variation from sample to sample must be addressed. For example, a sample drawn at baseline (Time 1) is 45% male and 55% female. A sample drawn post-intervention (Time 2) is 35% male and 65% female. If the data from both genders are combined, it is possible that any change (or lack of expected change) seen in the outcome variables at Time 2 (e.g., drinking behavior) are simply the result of the higher percentage of females at Time 2 and not the intervention.

Although sample weighting is sometimes used to minimize the impact of demographic variability, it is safer and easier to analyze the demographic categories separately if sample sizes are large and variation has occurred in only one or two demographic variables. By comparing the Time 1 and Time 2 males and females separately, for example, the effect of this demographic variability is eliminated. This also allows you to assess the impact of an intervention on different subsets of your population. If there is demographic variation on several demographic dimensions from sample to sample, then it is best to simultaneously control for all of these differences by using a multivariate analytic procedure such as regression analysis where each of the demographic variables is allowed to predict the variable under investigation (e.g. amount of drinking) simultaneously with the introduction of a variable representing the Time1/Time 2 periods.

III. Who is the Intended Target for the Evaluation?

In social norms projects, data is often collected for a variety of purposes. Some information is collected explicitly to develop messages and media content. Other information is collected for outcome evaluation. Still other information is used in both media development and outcome evaluation. If a project uses only one data collection method (i.e., a survey), it is likely that the data collected will be used for both purposes (two “masters”). Sometimes these two masters may demand different answers from the data. It is easy to confuse the two. One should consider if a portion of the data is more appropriate for evaluating the impact of an intervention rather than automatically including the entire database in an assessment.

For example, in some social norms projects drinking behavior data are collected from graduate students as well as undergraduates. Graduate students typically drink fewer drinks per occasion than undergraduates, thereby generating a more appealing media message (i.e., lower) and more realistic picture of the drinks-per-occasion norm for the entire university student community. However, when evaluating an intervention, the inclusion of the graduate students in the data analysis will likely diminish the significance of any reductions in drinks-per-occasion among the undergraduates (i.e., the likely target of your intervention) because very few graduate students may be changing their behavior.

Similarly, a cogent argument can be made that the greatest impact of a social norms alcohol intervention may occur first or most notably among the heaviest drinkers (for example, men drinking 8+ and women drinking 6+). These heavy drinkers, although relatively few in the population, account for the most negative consequences. Although an effective social norms campaign should address the entire student body, a well-conceived outcome evaluation should isolate and target specific portions of the sample to determine the effectiveness of an intervention. By eliminating the “dead weight” of the relatively unchanging low-impact portions of the population within the data set (non-drinkers, light drinkers, and moderate drinkers), one may be able to more easily identify any real effect in a Time1-Time2 comparison.

IV. Turnover in Student Populations Each Year/Term

Correcting misperceived social norms using various marketing efforts is a process that requires a population to be exposed to true norm messages. When comparing baseline (Time 1) data to post-intervention (Time 2) data for evaluation purposes, it is important to identify the portion of the post-intervention sample that was never exposed to the social norms messages. Incoming first-year or transfer students who were not on campus during the initial year of the social norm effort might have been present at the institution for only a small portion of the exposure period at the time of the post sample. To effectively assess the impact of an intervention, it is appropriate to consider only those students at the post-intervention who were likely to have been available for the full intervention period. Students who could not have been influenced by the intervention because they were physically not present at the institution represent another form of “dead weight.” Their presence in the data set makes it more difficult to observe any real change. (This selection issue concerning whom to analyze becomes more complex and even more care must be taken in assessment research when the social norms intervention extends over a multi-year timeframe including measurements at Time 3, Time 4, etc.)

V. Measures of Exposure to Intervention

As previously stated, correcting misperceived social norms using various marketing efforts is a process that requires a population to be exposed to true norm messages. That said, there should be some pre-post measure of exposure to the social norms messages of an intervention. The measure should explicitly describe the project’s messages in such a way that it is unlikely that the respondent would mistakenly include background messages from other sources in his or her response.

Here is an example of such a survey item: “About how many times, if ever, during this school year have you seen or heard information about what the majority or most students think and do regarding alcohol use based on data from your school?”

VI. Assessing Perceptions of Norms

Social norms theory states that behavior is strongly influenced by perception of group norms. Consequently, any program claiming to use a social norms approach for population behavior change must have pre and post measures of the perceptions of social norms. Remember, it may be that perceptions have changed in only a subgroup of the population. Therefore it is important to assess the perception status of the various subgroups making up the sample as well as the sample as a whole.

VII. Assessing Outcome Impact on Personal Behavior and Experience

Generally speaking, a median is the best measure when identifying population norms for measures such as Number of drinks per occasion, Occasions of drinking per week, etc. because this type of data can be highly skewed in its distribution. Means (averages) give undo weight to outliers within the data and may not be a good reflection of the majority of the target population. When it comes to assessment of impact, however, comparing the medians at pre- and post-intervention times may not show any change as the majority may not have changed their behavior and thus the median might not have shifted. Likewise the mean may change only slightly overall as the more problematic individuals begin to respond to changes in the perceived norm. One way to protect against masking impact is to use cut-points. Looking at the percent of college students who consume more than 7 drinks per occasion, for example, may provide a different and more meaningful evaluative perspective than simply looking at the median or mean drinks per occasion. (This is true for perception data as well.)

Tracking negative consequences is another good evaluative measure. Indeed, most projects are funded to improve the health and safety of the target population. When assessing alcohol-related negative consequences within the sample, it is often wise to differentiate items and analyses by gender. Many alcohol-related negative consequences show a strong gender bias. Specifically, men are more likely to report anti-social negative behaviors affecting others as well as themselves. Women tend to report mostly self-destructive consequences.

(The author thanks Michael Haines, Greg Barker, and Richard Rice for their comments and suggestions in preparing this summary.)

© 2005 H. Wesley Perkins

News Notes

  • The 2006 National Conference on the Social Norms Approach is currently scheduled to take place in Denver, Colorado next July. Specific dates, as well as a call for programs, will be posted on the NSNRC web site as soon as they are available.
  • An important new article appears in the July 2005 issue of the Journal of Studies on Alcohol: Perkins, H. W., Haines, M. P., and Rice, R. "Misperceiving the College Drinking Norm and Related Problems: A Nationwide Study of Exposure to Prevention Information, Perceived Norms and Student Alcohol Misuse." Journal of Studies on Alcohol, 2005, 66: July issue.
  • A revised and expanded version of the Guide to Marketing Social Norms for Health Promotion in Schools and Communities has just been issued. Additional concerns related to social norms data collection and analysis are discussed extensively in this guidebook. You can freely download a copy from the NSNRC web site: http://www.socialnorms.org/Guidebook/guidebook.php

About this publication

The Social Norms Review is a quarterly electronic publication of the National Social Norms Resource Center, with issues appearing in January, April, August, and October. Copyright reverts to the author on publication. National Social Norms Resource Center, 148 N. 3rd Street, DeKalb, IL 60115.

© 2005 H. Wesley Perkins

Download the original PDF (6 pages, 81 KB)