Stephen A. Banning
[WJMCR 4:3 June 2001]
Sections:
Abstract|Introduction|Method|Results|Conclusion
Abstract
This study suggests federal and state program assessment evaluations use a differential impact index to alert evaluators of the presence of a first-person or third-person effect that may result in major miscalculations. While the third-person effect hypothesis in communication has undergone considerable testing since it’s inception nearly three decades ago, this research is the first project to investigate an applied use for the concept. This study introduces a formula for calculating a differential impact index and demonstrates with an experiment using middle-school age students in a Drug Abuse Resistance Education (DARE) program that a third-person effect can produce a confounding variable into traditional program evaluations.
Introduction
The DARE program, standing for Drug Abuse Resistance Education, is a widely implemented program in the United States for fifth grade students, and receives funds of $750 million dollars annually. However, critics say DARE has not been proven to be effective.1
The most recent basis for the criticism is a study conducted by the Research Triangle Institute. R.T.I. findings indicate that the DARE program has no effect in preventing drug use, although R.T.I. evaluation methods have been called into question.2
At the same time that the research into the DARE program is being challenged, research suggesting an improved method of program evaluation has emerged. This new method of program evaluation would attempt to eliminate threats to internal validity with the use of an index that would provide a basis for evaluating self-assessment data.
The index described above was suggested in earlier research into a concept known as the third-person effect.3 The third-person effect occurs when a person believes others are more influenced by a message than they themselves are influenced.4Studies indicate this denial of influence increases when the person believes it is socially undesirable to be influenced.
This phenomenon is problematic for evaluations of programs such as the DARE program, because the young people in the DARE program may be likely to see being influenced by a drug awareness program as socially undesirable. Therefore, by the very nature of the influence, young people in a DARE program are more likely to deny that the program had influenced them (and actually believe that the program had not influenced them), regardless of whether or not it had influenced them. Literature Review
The concept of the third-person effect first emerged in a 1983 study by Davison, in which he stated that the third-person effect occurs when people believe others are more affected by media messages than they themselves are, and that this assessment is an overestimation. Subsequent research into the third-person effect has found that, contrary to the traditional belief that communication has a direct persuasive impact on the audience, the audience’s perception of the form of the communication is vitally important.5Mutz noted: “[In regard to the third-person effect] the effect that the communication achieves is not due to any direct persuasive influence of the message itself, but rather to the behavior of those persons who anticipate, or think they perceive, some reaction on the part of others, and behave differently as a result.”6 According to Perloff7 and Cohen and Davis8, people react to communication depending on how they think other people understand the communication. In other words, peer pressure can be a powerful determinant in whether a person is likely to deny that a communication has had a persuasive impact on them. Thus, traditional self-assessment program evaluations may be less accurate when peer pressure is high.
Further studies have indicated that the level of third-person effect (level of the likelihood that a person will deny that a communication has had a persuasive impact on them) increases when a person believes it is socially undesirable to be affected by the communication.9
Banning, 1997 notes:
Students involved in such a [DARE] program would probably not report that the program had affected them. This is because, among young people, there may be a negative stigma attached to persuasion by a police officer. Thus, if a greater third-person effect was created by the negative stigma of the message . . . the young people in a DARE program would be unlikely to report the program had affected them, regardless of whether or not it had affected them. Program evaluators should adjust their scores for the third-person effect. For example, DARE participants could be asked to rate how the program affected themselves and how it affected others. If a t-test indicated a significant difference between the two groups, evidence of a third-person effect, it could be concluded that the responses were not an accurate assessment of the DARE program’s effectiveness. Thus, checking for a third-person effect is a way of checking the validity of the evaluation. The presence of a third-person effect in evaluations indicates a lack of validity. Constructing a formula to adjust for the third-person could result in more valid program evaluations and a way of circumventing some of the traditional problems that self-reporting measures incur.10
Actually, a third-person effect is not the only evidence of an internal validity problem. Even if a person assumes a message effects him or her more than others the internal validity suffers. This is known as the first-person effect.
A first-person effect occurs when a person believes a message has more of an effect on others than it has on him or her self. It is the reverse of the third-person effect and is therefore sometimes called the “reverse third-person effect”. Researchers say a differential impact occurs when there is an effect that is either a third-person effect or first-person effect. In other words a differential impact means an effect is present, but the direction is not specified.
Significant evidence of a differential impact, either a first-person effect or a third-person effect indicates an internal validity problem. For example, if a subject were asked whether the media has strong effects, the reply might be in reference to whether the subject was thinking in terms of others or themselves and whether the question elicited a differential impact. The traditional way of asking about the impact of messages (be they media or in regard to an intervention such as the DARE program) does not eliminate the possibility of confounding by the differential impact variable. Thus, instead of asking whether the media have strong effects, a researcher could ask a subject how much the subject believes the media have strong effects in relation to themselves and how much they believe the media have strong effects in relation to others. This use of specific questions would help identify the presence of a differential impact.
Introduction
The purpose of this research is to demonstrate that adjusting for differential impacts may result in more reliable measurement instruments and more accurate measurements of a program’s effectiveness, in this experiment, for the DARE program. Accountability is becoming a common theme among federal and state governmental agencies as administrators attempt to maximize scarce resources. This accountability often takes the form of assessment of a program’s effectiveness. Where the product is an intangible, the assessment often depends upon self-assessment instruments. Thus, the funding of an effective program may rest on whether or not the instrument has accurately reflected the success of the program.
Use of assessment instruments that are not sensitive enough to measure a program’s success may result in the elimination of federal or state funding for a successful program. Accuracy, a high level of validity, is crucial. Use of assessment instruments with a high level of sensitivity allows administrators to make better decisions, and can give taxpayers and funding agencies more confidence that resources are being allocated properly.
Research indicates the third-person effect may result in a lack of accuracy in assessment instruments. With the use of self-assessment tests that have questions built in and allow for researchers to measure and adjust for the level of the third-person effect, however, the accuracy of the tests could be greatly improved. This is especially likely in situations involving young people, where the third-person effect is likely to be high and self-assessments tests are likely to underestimate the influence of a program intervention. In brief, this research has the potential to raise the sensitivity of current program evaluation instruments.
The breadth of applications to which this research applies is expansive. There are immediate potential benefits to DARE programs and similar drug abuse prevention programs that use self-assessment evaluations to gauge success or failure of the intervention.
Specifically, determining whether DARE is effective is key in regard to decisions to keep DARE in place, or to retool and implement a new program. However, this application is just one of many potential uses of this research. Other potential benefits include the possible improvement of the gamut of university extension, state and federal program evaluation instruments that involve self-assessment instruments.
The primary goal of this research was not to evaluate the DARE program per se but to evaluate DARE program assessment tests. More specifically, the goal of the test was to develop and implement an instrument that can be used to evaluate whether the third-person effect influences traditional program evaluations. This, in turn, may allow for specific adjustments to be made based on the denial of program effects among respondents.
This leads to the formation of the study hypotheses:H1: The respondents will show a statistically significant differential impact between the set of questions which asks how much the students thought DARE had helped them stay off drugs and the set of questions that asks how much they believed DARE had helped other students stay off drugs.
This first set of questions pertains to how students perceive the DARE message’s effect on themselves and is referred to hereafter as the questions pertaining to “self.” The second set of questions pertains to how students perceive the DARE message’s effect on others and is referred to hereafter as the questions pertaining to “others.”H2: The respondents will show a third-person effect toward DARE officers.
The finding of a third-person effect has become the norm in studies measuring for the third-person effect. One researcher found that the third-person effect was found in 13 out of 14 studies reviewed .11 A more recent study of third-person effect research found that the third-person effect was present in 15 out of 16 studies reviewed.12 Furthermore, a 1995 study with adolescents found evidence of a third-person effect regarding drunk driving public service announcements because the subjects considered the subject “nerdy.”13 Therefore, it is consistent with previous research to predict that the differential impact predicted in H1 will be a third-person effect and not a first person effect.
Method
As discussed earlier, the basic definition for the “third-person effect” is the phenomenon created by a person believing messages have a greater effect on others than on him or her self. Therefore, the operational method of defining the third-person effect in this study will be the same method used in many other third-person effect studies and is not unique to this research. In this study, a measure of the third-person effect is created by a number of “couplets,” paired questions in the questionnaire.The concept of using pairs of questions to come up with a third-person effect score seems to have originated with Davison and refined in a study by Cohen, Mutz, Price, and Gunther.14 Two other studies which used paired questions to come up with a third-person effect score specifically cite Cohen et al. as their source for this system.15 Previously cited third-person effect studies that have used paired questions to come up with a third-person effect score include Mutz , Perloff and Rojas et al. Two other methods for calculating the third-person effect have also been used. Tiedge, Silverblatt, Havice, and Rosenfeld16 used a system of three questions instead of two questions, and Lasorsa17 and Lometti, Ashby, and Welch18 coded open ended questions to arrive at third-person effect scores.The example given below illustrates the procedure that was used in this research.
Question #1. How much do you think DARE has helped you stay off drugs?1 A LOT
2 SOME
3 NOT MUCH
4 NOT AT ALL
Question #2. How much do you think DARE has helped others in the United States stay off drugs?1 A LOT
2 SOME
3 NOT MUCH
4 NOT AT ALLThis is a format based on the previously cited third-person effect studies by Cohen and Davis, and Gunther and Thorson. The respondent who checked a lower rank on the second question than on the first question of the couplet, has indicated a third-person effect, because he or she has indicated a belief that the program has had a greater effect on others than on him or her self. In the above example, the responses would first be reverse coded.As applied to this experiment, the presence of a third-person effect will be interpreted as problematic for the traditional interpretation of the data because of the danger that the resulting variance is being suppressed by the third-person effect. This is a threat to internal validity. In other words, the presence of the third-person effect is a red-flag that the data may not reflect, and may underestimate, the full effect of a program. In summary, the finding of a high level of the third-person effect score for the DARE evaluations would indicate the possibility that the third-person effect may be a suppresser variable, resulting in an underestimation of the actual effect of the program. Understanding the presence of the third-person effect as a suppresser variable in an evaluation allows the program administrators to better understand the results of their measuring instruments, and can allow them to better explain the results of the data and evaluate a program’s level of success.While demonstrating that a differential impact can call into question the validity of a program evaluation helps illuminate the problem, the solution requires additional conceptualization. The solution proposed in this research is: 1) the use of a standardized system of operationally defining a differential impact for comparison purposes, and 2) the reduction of differential impact levels to an index that can be used for cross study comparisons. The use of an index has not been used in previous third-person effect research and is presented here for the first time.The first step in the process is to run a t-test to determine if there is a significant difference between the two groups of scores. The proposed formula for determining whether the effect is a third-person effect or a first-person effect is as follows: How this would work with raw scores can be seen in the following: if the sum of the scores of the questions pertaining to “self” is 75, and the sum of the scores of the questions pertaining to “others” is 50, the differential impact level would be -25. The integer 25 gives some understanding of the magnitude of the difference, while the negative sign reveals that the differential impact is a first-person effect.Describing this example illustrates a minor problem with the use of this formula. While use of the formula reveals information about the score sets, the information is not easily compared to other findings. However, this can be accomplished with the use of the differential impact level index.The proposed differential impact index is as follows: (100 * |e|)/ The manner in which this would work with raw scores can be seen in the following: if the sum of the scores of the questions pertaining to “self” is 75, the sum of the scores of the questions pertaining to “others” is 50, and the differential impact level is -25, the index would be 33.333. Because the sign on the differential impact level was negative, we would have already determined that the differential impact is a first-person effect.In context with a differential impact index score of 33.333, we can say that the scores on the questions pertaining to “self” were 33.333% higher than the scores on the questions pertaining to “others.” If the sign on the differential impact level had been positive, we could say that there was a third-person effect in operation and that the scores on the questions pertaining to “others” were 33.333% higher than the scores on the questions pertaining to “self.” In summary, the differential impact index is the difference between the scores from the questions in the “self” and “others” groups. This is valuable tool for researchers in that it creates a shorthand method for explaining the magnitude of the differential impact findings in a study. A researcher could express the above index by saying a 33.333% first-person effect index had been revealed.
Results
The instrument was administered to 119 fifth grade students at a medium sized Midwestern grade school. The group consisted of five classes which had recently completed a DARE program. The gender of the 119 subjects was about evenly divided, with 59 males, 59 females, and one respondent not reporting. Approximately 79% were white, 11% African-American and the remaining 10% a mixture of Mexican-American, Asian, Native-American, Puerto-Rican and others. Eighty-two percent of the respondents were eleven-years-old, 12% were ten-years-old and the remaining six percent were twelve-years-old.A t-test was run on the two groups of questions described previously. A significant difference was found at the .0001 alpha level, supporting H1, which stated: “The respondents will show a statistically significant differential impact between the set of questions which asks how much the students thought DARE had helped them stay off drugs and the set of questions which asks how much tstated that the differential impact would be a third-person effect.The differential impact index formula described earlier (100 * |e|)/sDiscussionThe fact that H1 was confirmed has implications for program evaluations in that it reveals how question wording can over or under inflate the apparent effect of an intervention. On the positive side, hope can be found in the use of the differential impact index introduced herein in that the level of differential impact on respondents can be measured. This is a valuable tool in understanding the psychology behind the statistics in program evaluations and interventions and in assessing whether the responses have internal validity.The fact that H2 was confirmed is interesting in that while the third-person effect is commonly found among adults 20, only one other third-person effect study 21 has included middle-school age subjects as part of the sample pool. As applied to the DARE program, the results show fifth graders in the study were more likely to see DARE affecting others and less likely to see it affecting them.If a program evaluator had surveyed the students, the results might have seemed point two different ways. If the survey asked the students whether they felt DARE was effective, more of the students would have been likely to say it was. However, if the students were asked whether DARE had affected them, the students would have less likely to say it had. This discrepancy in results could cause an under representation of the effectiveness of the DARE program. However, it is a discrepancy that can be explained by the third-person effect hypothesis, which says that people are more likely to think others will be more affected by a media message than they themselves are.
Conclusion
The third-person and first-person effects do seem to have an effect on how people assess a program, and this effect could skew results in an evaluation. Had this experiment not used the differential impact index our understanding of the numbers would have been much different. We might have concluded that the DARE program is very strong if we only looked at responses to the question which asked respondents how the DARE program affected others. We might have concluded that the DARE program is not as effective if we only looked at responses to the questions which asked respondents how the DARE program affected themselves.The question which now arises is what a high level of differential impact shows. On the one hand, it shows the presence of a confounding or suppressing variable that means a reduced level of internal validity for either set of responses mentioned the preceding paragraph. In the case of a real world program evaluation, such a high level of differential impact would necessitate either further testing to increase internal validity or very cautious treatment of the program evaluation results.On the other hand, if the differential impact level were found to be low, one could assume confounding variables pertaining to the third-person effect were not likely to be at work, internal validity could be assumed to be higher, and the program evaluation results taken more seriously. For example, a differential impact index of 0 would indicate high internal validity.Program evaluators should seriously consider using the third-person effect index as a way of gaining additional information about program evaluations and their internal validity. Calculation of the differential impact index requires only the sums of the scores from individual groups and not use of the complete database. Thus, in cases where the sums of the scores from individual groups is available, it would be possible to compute the differential impact index on past studies without access to their databases.On the other end of the spectrum, a minimalist approach requires little effort on the part of the program evaluator. A questionnaire with a set of two questions as described earlier and a repeated measures t-test will indicate if there is a significant differential impact between the two groups of questions. A significant difference would be a warning sign of the presence of internal validity problems related to a differential impact.Future research into what causes the third-person and first-person effects could also benefit program and intervention evaluators. When more is known about what variables cause the third-person effect and how those variables interact, the specific confounding and suppressing variables may be able to be identified, further increasing the internal validity of program evaluation instruments.In the past, researchers have looked at macro applications of the third-person effect. This research presents a very applied micro level application of the third-person effect and answers the question of why researchers should research the third-person effect. The answer is that the third-person effect research has real world applications and consequences in 51.034 = i
About the Author: Stephen A. Banning is an assistant professor of journalism at the Manship School of Communication at Louisiana State University.