[WJMCR 8:2 March 2005]
The battle is as old as academia. One side is filled with the certainty of numbers; the other with the confidence of language. The battle is not at all confided to mass communication, but has ranged far and wide, from research into management1 to family therapy2 to tourism3 to electrostatics.4 The debate can turn brutal. A recent correspondence with an anonymous reviewer for a major communication journal evoked the declaration that �qualitative research is not really a research method. Perhaps it is a form of analysis.� The reviewer went on to suggest that interpretive essays and other forms of qualitative research were not of the status of �real research� as exemplified by quantitative methods. Researchers fond of qualitative methods have equally confrontational opinions to offer, albeit, more defensive in nature.5
This debate has consequences for researchers wishing to be published, an especially important issue for tenure-track professors. In 1994, Cooper, Potter, and Dupagne suggested, based on a study of mass communication research in eight journals, that authors using qualitative methods seemed to be losing the publishing war to those who preferred quantitative techniques.6 The authors suggested that rather than qualitative methods being used more often, as was believed to be the case at the time, the findings suggested just the opposite�that quantitative research was still the favored methodology. Kamhawi and Weaver supported these findings in 2003.7 They found that 73.7% of research articles found in 10 major journals between 1980 and 1999 were quantitative. As noted by Cooper, et al., �the apparent rise in qualitative methods heightens the continued debate over quantitative and qualitative research and has important ramifications for students and mass media education.�8
As noted by Smeyers in his plea for tolerance of diversity in research designs:
The tension between the generality of approach in causally-driven quantitative educational research and the individuality of particular cases is exemplified in the types of reasoning employed. Unlike the scientific search for antecedents, still popular in some forms of educational research, investigating particular persons and policies necessarily requires a form of practical reasoning.9
Both the Cooper and Kamhawi studies touch on a debate that has raged in all fields of social science, medical and other schools of research. At the center is the unresolved issue: which approach, qualitative or quantitative, generates the most valid results? The debate is more than a matter of semantics over whether �to count or not to count.� This is a deep-seated struggle and the sides are far from complacent. �Quals� are accused of being data-soft and unscientific. �Quantoids� are labeled as unrealistic and over-reaching. Qualitative methods are seen as too interpretive and lacking structure. Quantitative research is accused of presenting misleading precision and tending to suggest causation when there is only correlation. This is not a war fought only in mass communication, but indeed in the social sciences in general.10
As more and more mass communication journals either appear strictly online (Web Journal of Mass Communication Research) or maintain both print and online versions of their journals (Critical Studies in Media Communication, Human Communication Research, Quarterly Journal of Speech, and others), the question seems natural: are the conclusions of Cooper, et al.,11 and Kawhawi and Weaver12holding true for research into online mass communication? That is, has the apparent stilt toward more quantitative research publication held true for research into online mass communication?
Why Examine Trends in Online Mass Communication Research?
Mass communication research dealing with the Internet and World Wide Web offers an excellent opportunity to track the changes of focus within a discipline. In just the past decade, online communication has captured the attention of researchers in all “channels” of mass communication. And, as print, broadcast, advertising, and public relations rush into a fused medium, the web offers researchers a new channel rich with possibilities.13 One sign of acceptance of the Internet, and more specifically the World Wide Web, as a routine research topic is that authors seem to take less time in recent articles explaining the history or functions of either medium. Consider that Fredin and David in 1998, while musing on the debate over the Internet as �the medium of the future,� took time to define terms such as �browsing.�14 On the other hand, just five years later, Singer jumped directly into a discussion about online coverage of elections without explaining what the web is, how it works, or little more than a cursory definition of the subject of her research�online newspapers.15
In addition, decisions by communication journals to present special issues devoted solely to the Internet or the web may also be on the decline, which may indicate a more general acceptance of the topic as mainstream. The Journal of Communication in 1996 devoted one issue to the Internet. The journal did not do that again in the following 30 issues. The journal did not abandon the Internet as a suitable research topic. It simply blended in articles addressing online issues with those dealing with other research areas. Five of 11 articles in the autumn 2003 issue of Journalism and Mass Communication Quarterly addressed the Internet and web. The issue, significantly, was not titled as especially online-oriented.
As Couper notes,
Clearly, we stand on the threshold of a new era for survey research. Whatever one�s views about the likely future of survey research. Whatever one�s stand on the threshold of a new era for survey research … Whatever one�s views about the likely future for survey research, the current impact of the Web on survey data collection is worthy of serious research attention.16
He goes on to argue that, in order to refine the methodology of web surveying and to eliminate the downsides such as poor response, researchers must be involved in the technical aspects of online survey construction. He points to the creation of online panels recruited by companies and universities as an example of more involvement by researchers in cultivating the advantages of the Internet. Notably these are tools could be used by quantitative and qualitative researchers alike.
Newhagen and Rafaeli stressed the need for mass communication researchers to address online communication.
This problem of having the critical dimensions of a new technology hidden from view also can be seen in the current trend among publishers to dump text into a computer network and call it an �electronic newspaper.� If journalism does not come to grips with the impact those architectural differences have on the way people use information, it may have trouble finding a home on the Net.17
As noted by Cooper, et al.,18 and others, the perceived rise in the use of qualitative methods in 1970s and 1980s research, as noted by many researchers19, was followed by an increase in the inclusion of qualitative methods in mass media courses. However, the previously cited research by Cooper, et al., and Kamhawi and Weaver suggest the conclusion that the emphasis in qualitative research courses was increasing was flawed: quantitative research remained the dominant form of research in the period, they posited.
Schools face substantial pressures as more students enroll; state support, and endowments also are declining.20 Given the state of limited resources, schools of mass communication face the fiscal reality that some courses will go untaught. Determining which can be spared may result in the elimination of courses seen as less useful to a student in the long term, and, in the long run may result in fewer qualitative research courses being offered. The likelihood of publication may, therefore, play a key role in determining how universities should educate future researchers. Why teach qualitative method if the chances of publication are slimmer than a more favored quantitative approach?
Finally, the maturity of a medium also may be a factor. Given its relative newness, the Internet and web, hereinafter �online communication,� provide an excellent opportunity to trace the changes in preferred research methods over a short period of time within a quickly evolving medium. Is there a favored style of research at a particular period in the evolution of a new medium? Are research methods, such as interpretive essays, more favored at the emergence of a new media, while surveys and experiments are more favored as the new media matures?
Does the theoretical basis of the research have some bearing on the type of methodology used? This may be a two-step model in which the structure and workings of online communication favors one theoretical method that then favors a particular method.
Ultimately, however, the nature of online communication research itself may play a part in favoring one methodological style over another. Elements of online communication may be economic in nature, making some styles of research easier, faster, and/or less expensive.
This study attempts to test two hypotheses:
H1. Among 33 mass communication and communication journals, the pace and percentage of publication of research into online mass communication will have increased over the period 1993-2003.
H2. The proportion of published articles dealing with research into online mass communication that utilize qualitative research methods (versus quantitative research methods) will have fallen over the period of the study.
This article updates research presented by the author to the 1999 conference of the Association for Education in Journalism and Mass Communication.
Journal Selection Criteria
One of the features of the web is the perception that all sites are created equal if not in impact, at least in screen size. It is the very egalitarian quality of online communication that has sparked much of the discussion among futurists, such as Lessig,21 Negroponte,22 and Shapiro.23 As Shapiro noted, the advent of the web is the advent of publishing anarchy�the uncontrolled, unorganized ability of anyone to present a site roughly the same size and quality as any other site, including those of the �mainstream� media. This not only occurred �out there,� but in mass communication with online publications like Web Journal of Mass Communication Research.
Given the relative ease with which online journals may be created and published, the question of how to measure which journals are the �leading publications� is not easy. Research such as this requires the line be drawn somewhere. Cooper, et al., chose journals based on circulation numbers (higher than 2,000) and a blind review process that rejected four of every five submissions.24 Kamhawi and Weaver�s standard only slightly differed, requiring a circulation of at least 1,500.25 Measures of acceptance rates and circulation, according to Garner and Dyer, presume that readership models acceptance and that rejections model quality.26 Tomasello used basically the same approach, narrowing the selection of journals based on Social Science Abstracts.27 In all cases, however, the underlying economic issues libraries now face were overlooked. It is common practice in the publishing industry among such companies as Elsevier Publishing to �bundle� less popular journals with more popular ones, thus forcing libraries, a leading subscriber of most academic journals, to purchase a package. In addition, the ongoing migration and creation of journals online calls into further question the value of the circulation measure.
Rather than adopting a standard that is likely to face serious challenges in the future, this research focuses on a wider collection of mass communication and communication journals. Colleagues were polled at three major universities about which journals they felt were significant. The complete list is presented in Table 1 with proportion of online communication research articles published in the period of study. Yes, it is always possible to include a few more journals. After all, most researchers have a �favored� publication. But it is unlikely that adding one or two to the 33 already examined would have had any impact on the results, given the extreme nature of the results. The lines between certain areas of research, such as advertising and marketing, and public relations and interpersonal speech, are regrettably gray. An effort was made to focus the work on mass communication journals, not business or speech journals, though, no doubt, pertinent research can be found in journals associated with those fields.
|TABLE 1. Distribution of Mass Communication Research Articles Dealing with Online Communication by Publication in Thirty-Three Journals, 1993-2003, including the Percentage These Online Articles Represent When Compared to All Articles Published by Each Journal. N=476|
|Communication Law & Policy||31||18.6|
|Communications and the Law||15||6.6|
|Critical Studies in Media Communication||13||6.1|
|Human Communication Research||4||1.6|
|International Journal of Advertising||2||0.7|
|International Journal of Communication Law & Policy||23||37.8|
|Internet Research: Electronic Networking Applications and Policy||31||8.9|
|Journal of Advertising||17||6.2|
|Journal of Advertising Research||59||15.4|
|Journal of Broadcasting & Electronic Media||26||7.6|
|Journal of Communication||16||3.9|
|Journal of Communication Inquiry||10||5.0|
|Journal of Consumer Affairs||4||1.4|
|Journal of Consumer Marketing||17||4.9|
|Journal of Consumer Psychology||12||4.1|
|Journal of Consumer Research||4||1.1|
|Journal of Current Issues and Research in Advertising||10||8.1|
|Journal of Public Relations Research||1||0.7|
|Journalism & Communication Monographs||3||6.3|
|Journalism & Mass Communication Quarterly||49||9.5|
|Mass Communication & Society||3||2.3|
|Media Studies Journal||27||5.0|
|Media, Culture and Society||5||1.4|
|Newspaper Research Journal||40||10.8|
|Public Relations Review||19||6.2|
|Public Opinion Quarterly||1||0.3|
|Visual Communication Quarterly||3||3.9|
|Web Journal of Mass Communication Research||1||4.0|
Article selection criteria and measures
This study uses a fused definition of Internet-based mass communication based in large part on the previously cited writings of Lessig,28 Negroponte,29 and Shapiro.30 An article is considered to be focused on online communication research if it deals with communication carried through a many-to-many network with the applications necessary to handle the information processing located at the ends or edges of the network. This definition includes such forms as the web, electronic mailing lists, bulletin boards, newsgroups, chat rooms, online virtual reality games, and video/audio teleconferencing. It does not include online telephone technologies nor one-to-one e-mail. The effort here is to distinguish speech communication research from mass communication research though the line is hardly bright. In addition, the intended audience of the article is taken into account. If the article targets online mass communication, it is included; if it addresses mass communication pedagogy, it is not. Notably, however, a dozen articles that targeted online pedagogy were identified, all published in Journalism and Mass Communication Educator, beginning in 1996.
This research intends to provide �scholars with an overall sense of an area�s progress or current status,� as was the case for Tomasello.31 The �progress or current status� of an area of research is operationalized into four measures: total articles by year, theoretical approach, research method and journal.
Unit of Analysis
Included in this research are articles that focus on online communication. Not included are editorials, book reviews, product or service updates, or outlines. Coders read titles, abstracts, and, in some cases, specific sections of articles, such as �methods,� to determine the frequency of publication, the method of research.
The nature of the data in this research is a full population. Given that this study is intended to update existing research, a pilot was deemed unnecessary. Articles were identified by using a keyword search within several electronic databases, such as Article First, Pers Abstract, Expanded Academic ASAP and others, based on the database that presented the most comprehensive access to a particular journal. The effort was intended to be exhaustive.
Selecting the search terms for this project posed no simple task. A variety of terms might have been proposed, including Internet, as well as online, new technology, World Wide Web, e-mail, global village, and interactive, to name a few. Adding to the complexity of the task was the worry that use of an overly broad term might draw in articles related only peripherally to online communications. However, direct examination of a fixed set of articles aided in determining the efficacy of the terms.
Journalism Abstracts, a compilation of dissertations and theses, uses an indexing scheme that shed some light on the relative overlap of search terms. Examining the indexes for the journal between 1993 and 1996 revealed that �new technology� was indeed the catch-all term used in 1993, with one thesis cited. In 1994, eighteen research papers addressed �new technologies,� with no index heading for �Internet� or �web.� Of these, a cursory examination of the titles suggested that four deal with the Internet or World Wide Web. Two deal with online newspapers, one with access to online technologies, and one with organizational culture online. An examination of the titles in 1994 also provided some insight into the topics that the editors of the journal considered �new technology.� These articles deal with cable, advances in film, digital photography, and videotex.
By 1995, �Internet� was indexed, but not �web.� Sixteen papers were indexed under �Internet,� while �new technology� had eighteen citations. Interestingly, of these 18 indexed under �new technology,� six were also indexed under � Internet,� while four that might have been indexed in that category were not. These deal with obscenity online, online marketing, news groups, and e-mail. In 1996, �World Wide Web� was used as an index term, with six papers appearing. �Internet� had nine, and �new technology� had 27. Of these under �new technology,� only seven might have been indexed under �Internet� or �World Wide Web,� but were not. These deal with online news (three), online information sources, online communities, newsgroups, and online television sites.
This data suggests that the terms �Internet� and �web� were synonymous with �new technology” in 1994 and earlier, but that both grew to be indexed in their own right after 1995. In addition, the term �new technology� included many subjects not directly related to the Internet or web, such as cable and digital imaging software. Thus, using a term like �new technology,� while more inclusive, resulted in more articles being captured by the search that are not related to the subject of online communications. It was consistently found in 1996 that the words �Internet� and �web� were not synonymous, with very few papers indexed under both terms. It was also found that �online� and �on-line� generate results outside of the specific terms �Internet� and �web�
Finally, given the focus on the �many-to-many� definition of online communication used for this study, several articles in publications such as Journal of Communication are not included in this study. For instance, the 1996 issue of Journal of Communication that Tomasello referenced as a special �Net� issue,32 contains only two articles that meet the definition of �mass� communication. The remainder focuses on one-to-one communication. Rather than engage in a debate between what is �mass� and what is �communication,� a hotly argued subject in itself, the author chose instead to focus on the more traditional definition previously cited. Thus the search string �(Internet or web or online or on-line) and communication� was used to search the 33 selected journals.
Two points of data were gathered for each article: total articles by year and research method, qualitative (such as: interpretive-essay including history, and interview/case study), policy analysis (such as law analysis), or quantitative (such as content analysis, survey, experiment, and model building).
Two coders were used. Agreement regarding which articles qualified as online communication research was 99.1%. Percentage of agreement for identification of journal, 99.9%; and research method, 97.4%.
The results are presented proportionally over time and for each method by year in Table 2.
|TABLE 2. Distribution of Mass Communication Research Articles Dealing with Online Communication by Research Method in Thirty-three Journals, 1993-2003, including the Percentage These Online Articles Represent When Compared to All Other Online Communication Articles Published Each Period. N=476|
|Qualitative Method (N=226)||67.1%||48.6%||40.6%|
|Survey – Interview/Case Study (N=133)||19.7%||24.3%||33.8%|
|Interpretive Essay (N=35)||47.4%||24.3%||6.8%|
|Policy Method (N=105)||18.4%||28.2%||18.3%|
|Quantitative Method (N=145)||14.4%||23.3%||41.1%|
|Meta-Analysis/ Model Building (N=19)||3.9%||2.8%||5.0%|
|Survey – Content Analysis (N=75)||7.9%||16.6%||16.9%|
Proportion Over Time
The results confirm H1: the proportion of online communication research publication over time presented in Table 2 consistently increases each year, with the exception of a very slight drop from 2000 to 2001.
This might be related to a significant increase in mass communication law and policy research. Federal legislation passed in 199633 and 1998,34 as well as, laws under consideration35 attracted considerable research interest. Also, more general policy analyses of national36 and international37 regulatory trends added to the increase in 2000.
Finally, the much sharper increase in 2003 may be a momentary spike in publishing or reflect a more substantial turn to online communication research. Revisiting this research area in a few years may reveal whether this shift in interest sustains the 2002 to 2003 increase of 33.3%.
Examining the results in Tables 1, it is notable that almost half of the research articles appeared in six journals: Communication Law & Policy, International Journal of Communication Law & Policy, Internet Research, Journal of Advertising Research, Journalism & Mass Communication Quarterly, and Newspaper Research Journal. On the other hand, a third of the journals published less than 8% of the research in question: Communication Theory, Human Communication Research, International Journal of Advertising, Journal of Consumer Affairs, Journal of Consumer Research, Journal of Public Relations Research, Mass Communication & Society, Media Psychology, Media, Culture & Society, Public Opinion Quarterly, Visual Communication Quarterly, and Web Journal of Mass Communication Research.
In the first case, the six journals cited are all considered leaders in their fields of study. The second case is more complex and will be discussed later.
Several journals presented a clear bias for one style versus another. Not surprisingly, the law journals were solidly in the qualitative area, and, in fact, represent 46.5% of all qualitative research identified in this study. Equally unsurprising were those that favored quantitative methods almost exclusively: Journal of Advertising Research, Journalism and Mass Communication Quarterly, Newspaper Research Journal, and Public Relations Review. In the case of Law journals, had they been deleted from this study, as could be argued, the proportion of online communication articles related to qualitative methods would have fallen by almost half. If anything, the inclusion of the law articles may have skewed the results away from an even more severe shift toward quantitative methods.
Method (Table 2)
The pattern of results shows a significant shift toward quantitative methods, thus supporting H2. Overall, research methods fall into the two larger areas under study here, and within one specialized area, law. Quantitative research includes surveys and content analysis, model building, and experiment. Qualitative research includes interview/case studies, interpretive essay, and interpretive policy analysis. Meta analysis may fall into either category, but, in this study, was a quantitative method.
|TABLE 3. Distribution of 502 Mass Communication Research Articles Dealing with Online Communication by Theory, Method, and Year in Thirty-three Journals, 1993-2003. N=502|
|Sources: See Table 1.|
The gross numbers for qualitative verses quantitative are somewhat misleading. While qualitative research (47.5%, 226) presents overall as higher than quantitative research (30.5%, 145), the distribution over time reveal a significant shift toward the latter. While qualitative methods were preferred in the period 1998 to 2000 (48.6% to 23.3%), the trend in the last three years is persuasively toward quantitative methods. In fact, in the last year of the study, qualitative methods fell to 12% of all articles published.
Also of note was the decline, as a percentage and in real numbers, of the use of the interpretive essays method, with a high of 17 (44.7%) in 1997 and only five (5.4%) in 2003. Meanwhile, experimental methodology jumped from just a few in the early 1990s to 15 articles (16.3%) in 2003.
This research presents a more comprehensive and complex view of online communication research trends than Cooper, et al., found in 2001.38 In a sense, online communication research mirrors a pattern of how any new communication channel might be explored. The frequency of articles reflects an adoption pattern related directly to the types of research methods used. It is a classic step-wise approach to new phenomena described by many practitioners that demonstrates an evolution from discussion, to information gathering, to testing, to model building, to re-testing, to experiments, to model refinement.39
The slow increasing trend in published online communication research seen in Table Two ends in a dramatic spike of 33% in published research between 2002 and 2003. As researchers become more familiar with the medium and as the medium itself settles into a sustained pattern of use, interest in the topic would predictably increase. And given the converging nature of online communications combined with the web being considered a channel unto itself, it is likely overall research of online communications will continue to increase.
Finally, we are presented a clear shift in methodology that suggests more than a simple agreement with the ultimate conclusions of previously cited researchers that quantitative methods are favored over qualitative. The results do show an ultimate preference in 2003 for quantitative methods. But the real picture is far more interesting. Over time a significant shift (p<.001) in research methods occurred in just 11 years. Whereas previous research by Cooper, et al.,40 and Kamhawi and Weaver41 reported no significant shift in research over the decades their studies covered, this research shows a powerful and inextricable shift toward quantitative research over a single decade. The early preference toward qualitative research was all but wiped out by a preference for quantitative research. In fact, were it not for the numbers of articles dealing with law and policy research, quantitative would hold an almost exclusive preference in the last year of this study.
Add to this the loss during the period of this study of Media Studies Journal, which was exclusively qualitative and ceased publication in the summer of 2003,42 and the future trend of research in online communications bodes even more powerfully quantitative.
The data suggest that online communications may attract more than expected numbers of qualitative research over the period, certainly more than reported by Kamhawi and Weaver43 for offline research topics. However, the trend was consistently down over the period of study, suggesting a strong shift toward quantitative methods, such as experiments and surveys.
It may be the newness of online communications that draws more early application of qualitative methodologies. No doubt some more interpretative techniques would have been used in the early days of the printing press, had the opportunity presented itself. In fact, discussion of the impact of the printing press did proceed in essay form for some centuries after Gutenberg:
He who first shortened the labor of copyists by device of movable types was disbanding hired armies, and cashiering most kings and senates, and creating a whole new democratic world: he had invented the art of printing.44
Clearly, the value of qualitative research in defining a new communication form in the moments after invention is well founded. At these times, debate swirls around possible impacts and implications of new media, whether in the form of a press or a kilobit. The question is whether interpretive essays and other form of qualitative research can offer further illumination after those provided in the initial stages of mass communication transformations.
And why is quantitative research so rare in early years of invention? It may be the expense required or the lack of general population subjects that might have suppressed timely quantitative research in the early 1990s. Unlike the more simplistic content analysis cases involving advertising or news stories, online communication requires computers and other resources, as well as the knowledge to use these tools to be effectively completed. On the other hand, e-mail actually reduces the expense of conducting surveys, while, admittedly, injecting other, new, issues of reliability and validity.
Finally, it may be that researchers are lured by the belief that �hard results��those with numbers and significance�are valued in academia over what appear to some to be not �real research.� It may be as Curtin and Maier suggest:
A limitation of this study is that the results cannot be generalized to the larger population. Quantitative studies are needed to confirm or deny the universality of the constructs that emerged from this research. But the emergent constructs are highly representative of the two divergent viewpoints-those who are math anxious and those who are not. As such, they present themselves as worthy of further examination and lead to the following suggestions for newsroom managers and journalism educators.45
Perhaps qualitative research is useful only in exploring, not in analysis. This would seem short-sighted and needlessly biased, placing �numbers� above �words.� Online communication researchers (and journals) need not succumb to the siren call of quantitative analysis as the only true and worthwhile path. What we seek is insight and understanding, not just the numerically-driven validity of �significant� variance.
However, this research suggests that we as researchers and publishers are confusing complexity with accuracy, in a vain and short-sighted race to the provable at the expense of the knowable. The balance of these two families of research should be re-established, for both have powerful roles in helping to describe, clarify, and explain.
Finally, the little attention or declining attention between 1993 and 2003 in some methods, such as interpretive essays, should give mass communication department administrators cause to pause. Given the falling numbers of research publications accepting interpretive essays in the area of online communication, the future of this research method is marginal, at best. While there is little doubt that this method, as with case studies�also a weak area in this study�will continue to attract the attention of some researcher, the emphasis for educational programs may shift to serve the more �successful� publishing styles. The potential impact on educational programs is clear and serious. It makes little sense to be teaching a method to graduate students, for instance, when such low acceptance rates among preferred journals seems to put issues of future employment and eventual tenure into serious question.
And yet, does the temptation to focus on the �now� in this rapidly changing environment trap us as researchers into a constant �rear-view mirror� perspective, constantly watching what has already happened in a landscape that increasingly renders the present distant from the very recent past. Our research becomes more rapidly useless, or at least of questionable values, when other events overtake us. Consider the value of studying the impact of telemarketing, FTP sites, and other modes of mass communication displaced within years of their apices by new models.
Interpretive essays by �futurists,� such as Lessig, Negroponte, and Shapiro, are forceful in their nature and impact. They provide valuable guideposts, navigation researchers may use to model their work within the bounds of what is useful and appropriate. Simply being able to find significant variance is not enough. Our research should illuminate and inspire.
This study examines the trends in mass communication research in an area that promises to be important for decades to come. It represents one of the broadest efforts in terms of journals. Its findings, especially in the preference for some areas of research methods, are worth following and reassessing in the coming years.