3. Research Quality
Understanding Definitions of and Challenges to Quality in the Knowledge Production Process1
© 2023 Jesse C. Starkey et al., CC BY-NC 4.0 https://doi.org/10.11647/OBP.0364.04
Quality is a characteristic of thought and statement that is recognized by a nonthinking process. Because definitions are a product of rigid, formal thinking, quality cannot be defined.
Robert M. Pirsig, Zen and the Art of Motorcycle Maintenance (1974: 200)
Quality is integral to scholarly research and publishing, but we have struggled to agree on how to define it. This is due, in part, to the complexity and dynamic nature of research and publishing. For example, a graduate student may come out of a high-quality program (based on published rankings), produce low-quality research (based on a professor’s or other peer assessment), improve the manuscript’s quality through peer review, then publish it in a low-quality journal (based on impact factor), but then get picked up by the media and distributed to a wide audience, thus resulting in high readership and citation numbers (an indicator of quality).
By addressing this topic through empirical research and gathering insights from diverse stakeholders in the scholarly publishing world, this chapter provides an overview of perspectives on quality in all stages of scholarly knowledge production, situated in a world where we face numerous threats to the global publishing enterprise. From the training of graduate students, to the production of research, to the vetting process for manuscripts, and finally into the publishing and dispersal of peer-reviewed scholarly publications, this chapter provides a glimpse into how people in the business of academic publishing — both leaders and emerging talent — view quality in terms of scholarly knowledge production and distribution, all while navigating a wide array of ethical pitfalls.
But why does understanding what constitutes quality in scholarly knowledge production even matter? One might question whether such an exploration is simply a matter of egocentric navel-gazing, an opportunity for those of us in the ivory tower to justify research simply for research’s sake. It matters because institutions of higher education are the training ground for the individuals who go on to make great changes in society. So how can we safeguard those hallowed grounds so that they do not succumb to the continued assault (‘In Defense of Knowledge’ 2020) on knowledge and facts — especially in an environment where predatory journals are eroding the credibility of academic publishing?
Our findings echoed what scholars in diverse fields have been grappling with for decades — there is a systemic flaw in global higher education that focuses on quantity over quality (e.g., Pyne 2017; Skolnik 2000), giving rise to an environment where journals that tout fast turn-around times or guaranteed manuscript acceptance are viewed as career-savers, rather than threats to academic integrity. Yet our study adds some important nuances to the conversation — we share the combined voices of stakeholders across the publishing world, and from around the globe. This chapter presents their perspectives, coupled with previous empirical research, to attempt a holistic understanding of the difficult task of defining and understanding quality in the context of scholarly research and publication in an environment fraught with potential ethical challenges.
Predatory Journals are a Marker of Low Quality
An important backdrop to the discussion of quality in scholarly knowledge production is the growing threat of predatory publishers, journals, and conferences. The term ‘predatory’ and how it came to be associated with digital publications claiming to contribute to the scholarly knowledge production environment is explored in depth in other chapters of this book, so here we position predatory journals and their publishing practices as the opposite of ‘good quality’ open access, due to the low-quality science they publish, the editorial team (often fake), the peer review (also often fake), and culminated by their focus on profits over services. One of our study participants was particularly critical of the role predatory journals played in ‘preying on people’s ignorance’ (P02),2 especially outside the sphere of Euro-American3 institutional affluence, where scholars have fewer resources and often less training about publication ethics, and work within institutional cultures that value quantity over quality.
Background
A quick scan of the literature on ‘quality in higher education’ shows scholars from numerous fields addressing the topic in a range of ways. Since 2020 alone, researchers have examined service quality in higher education from a marketing perspective (Alfy and Abukari 2020), the economic perspective of quality graduate education (Bairagya and Joy 2021), sustainability in higher education (Vykydal and others 2020), the business of higher education (Cavallone and others 2021), teaching quality (Giraleas 2021), and a range of other topics including labor and learning outcomes, student satisfaction, and civic engagement (Bloch and others 2021). But one area that has surprisingly been neglected in the literature of quality in higher education is quality in the knowledge production process.
One component of that process is peer review. Rigorous peer review has long been the hallmark of successful, high-quality scientific publications (Roll 2019). The review from impartial peers is supposed to ensure the credibility of the scientific process, offer authors ways to improve their manuscripts, and increase the quality of publications that ultimately lead the way to scientific advancements (Roll 2019). Yet in recent decades doubt has been cast on the peer-review process, partly spurred by the rise in predatory journals (although some mainstream, well-known journals are now under fire for their peer-review quality — see Vazire 2020 for examples). Articles and blogposts in mainstream media with academic and general audiences have tackled the issues with peer review in recent years (e.g., Carroll 2018; Humphries 2021; Michael 2019; Vazire 2020) and concluded that despite its flaws and inadequacies, peer review is still the best option for vetting scholarly knowledge production. Therefore, despite attempts to revamp the peer-review process, it remains the primary way of vetting the quality of manuscripts that end up being published in scholarly journals — and thus an important element when attempting to understand quality in the knowledge production process. Lack of peer review is also a common trait of predatory journals, making their potential impact on the global body of knowledge even more damaging.
When examining quality in the production and distribution of scholarly research, much of the existing literature (Lindsey 1989; McGrail and others 2006; Zerem 2017) points to the quality of the journal as a primary indicator of scientific research quality, with special emphasis placed on impact factors and more recently, H-Index rankings. Yet the impact factor was not even designed for this purpose (McKiernan and others 2019). To further complicate the issue, some indexing and impact ranking organizations are fraudulent, requiring readers and potential authors to be more critical in their assessment of journals, rather than simply relying on statements of index inclusion or impact metrics listed on a journal or publisher’s website.
Now, with the rise of predatory publishing venues (Shrestha and others 2019) targeting desperate scholars who need lines on their CV (Pond and others 2019), individuals and organizations have been trying to develop ways to vet the quality of scholarly publication outlets. More recently organizations — both for- and non-profit — have joined this endeavor. Forrester and others (2017) compared the established services typically used by librarians to find information about journals with new services aimed at directly helping authors select journals. They found that even when using identical search terms, the different services returned vastly different results, suggesting that an inherent bias pervades many of the tools (many are owned by specific publishing companies). Yet Forrester and colleagues (2017) remained hopeful in their conclusion, noting that ‘as these tools increase the transparency of journal information and their editorial processes, this could lead to an overall improvement in academic quality control’ (p. 286). Koerber and others (2020) compared two approaches to classifying journals and publishers — safelists and watchlists — and found challenges with either approach, suggesting that the ‘list’ approach to classifying safe or predatory publishing outlets may not be the answer to the paradox of predatory publishing.
While there are a few published frameworks that try to delineate specific markers of quality in academic research (e.g., Frambach and others 2013; Tracy 2010; Welch and Piekkari 2017), succinct markers of quality remain elusive. However, when looking at quality in academia, many scholars (e.g., Biggs 2001, 2011; Bowden and Marton 2003; Brennan and Shah 2000; Lagrosen and others 2004; Suleman 2018; Teeroovengadum and others 2019) rely on the landmark works of Lee Harvey and Diana Green (1993) who argue for a multi-dimensional conceptualization of quality. Therefore, embracing epistemic diversity and acknowledging multiple definitions of quality will be our road map to understand this complex issue: ‘What does quality in scholarly knowledge production mean to you?’
Challenges to Quality
If we can show that a world without quality functions abnormally, then we have shown that Quality exists, whether it is defined or not.
Robert M. Pirsig, Zen and the Art of Motorcycle Maintenance (1974: 210)
We begin our journey of understanding quality in scholarly knowledge production where Phaedrus began envisioning a world without quality in Zen — where the absence or low levels of quality causes a breakdown in functionality. It is often easier to say what is wrong with something, and to note the absence of quality, rather than defining markers of quality in concrete terms. When our participants were asked what quality in scholarly publication meant to them, they often answered as if they were sitting with Phaedrus as he began his intellectual journey. They shared examples of where quality was missing, and confirmed many common notions about the challenges of being researchers (knowledge producers) in higher education: the Euro-American perspective on research remains dominant, evaluation methods around the world differ drastically, the increasing pressures to publish lead to a cycle of constant rejection and pressure to sensationalize findings, peer review is not as unbiased as we want to think, and strong mentorship for graduate students and junior scholars is sometimes lacking.
Pressure to Publish
The ‘publish or perish’ paradigm in higher education places the academic researcher as the ‘product’ that needs to deliver the ‘service’ of producing scholarly publications, creating a system where the institution feels obligated to evaluate whether their ‘product’ is producing the ‘service’ at the most cost-effective rate, in order to benefit the institution’s bottom line. From an institutional perspective, higher numbers of publications equal higher quality, but from an individual perspective, the pressure to continually ‘fling [manuscripts] to a journal […] rather than taking the time and the effort to really prove yourself wrong’ (P45) has led to a degradation in the quality of scholarly knowledge production. This pressure to publish was especially salient to some PhD students, where their national systems require them to publish before their degree can be conferred (e.g., a country in Southeast Asia, P15; and a country in Africa, P03), placing publications as ‘kind of the currency of the field’ (P08). An interesting nuance in the statements about pressure to publish was the notion that this soaring need for scholars to find outlets for their work was being met by ‘creating new journals’ (P32, P35), many of them falling into the ‘predatory’ category. A scholar from Southeast Asia agreed with the sentiment that the pressure to publish led to the rise in predatory journals, stating, ‘There is a demand for it because, [for] I and other academics here, there is a pressure to publish’ (P41). In large part, this continually mounting pressure to publish has stemmed from a rise in institutional valuation of the quantity of publications, rather than the quality.
Valuing Quantity Over Quality
The complicated interaction between personal ethical values and institutional pressures was especially salient as participants discussed the trend in academia to value the number of publications racked up by an individual, rather than the impact, practical application, or other less concrete markers of quality in those publications. This concept was addressed from a range of angles, including scholars noting cultures of publishing where, during the tenure process, the types of journals a manuscript appears in does not matter (P02), or commenting that ‘in some areas, unfortunately, they’re mainly counting publications not thinking about quality and so you can just publish a lot and you’re considered productive’ (P08) and ‘in many parts of the world, there is no discernible difference between the quality and the quantity’ (P24).
Others spoke about the focus on quantity over quality from a human resources perspective, with one participant from North America noting ‘there are really perverse incentives in academia […] quality has become […] a question of numbers […] the number of publications per academic’ (P22). A European participant agreed, noting that ‘if one criterion is only the number of publications, then this incentivizes to publish in such predatory journals or [be] published in the lower journals’ (P38). This fracture between institutional and individual perceptions of quality in terms of the numbers of publications produced exemplifies one of the challenges of understanding quality in scholarly knowledge production.
Having to Adhere to Euro-American Publishing Standards
Given the global flow of information facilitated by technology, it was no surprise that scholars felt compelled to address the challenges associated with navigating the differences between their disciplinary expectations on national, regional, and global scales and spoke to the ‘different publishing cultures across all these countries’ (P10). They noted both the pressure to publish in English and in high-ranking journals, along with a desire to produce work that was meaningful and useful in their native countries. Another scholar added ‘quality is associated with the Western world’ noting that while they were not necessarily opposed to this global view, there were other regions that were producing high quality work too but were not being recognized because ‘they don’t meet the kind of processes that the Western publication industry is familiar with’ (P41). This speaks to the complexity of evaluating quality in a global context when considering scholars who produce work in accordance with their local or regional environments, who may then face a different set of standards or expectations at the national or international scale, creating a scenario where the purpose or functionality of their research products changes depending on the context in which they are operating. Scholars from outside the Euro-American regions may also find themselves struggling to produce knowledge that is both beneficial and useful for their home environment and acceptable in the global sphere of knowledge production.
When they talked about global inequalities, participants emphasized resources as an important difference among disciplines and scholars from different nations or even different institutions within a specific country. One European researcher hypothesized that researchers working outside the Euro-American sphere of affluence ‘[…] just didn’t have the resources or the knowledge of how to do things properly’ (P18). This notion of inequities in resource availability between scholars working inside or outside of affluent Euro-American regions was also addressed in terms of how a supposedly global organization like the International Communication Association (ICA), the flagship organization of the Communication discipline, perpetuates biases (P34). They further suggested that international organizations can, in some cases, contribute to an environment where scholars working in resource-poor countries or institutions are expected to maintain the same quality standards as resource-rich scholars — and when they fail to do so, it perpetuates biases against non-Euro-American scholars in Euro-American journals and professional organizations.
Values related to international standards of methodology were also questioned by one participant who stated, ‘international publishing means Western publishing’, and then went on to give the example of Latin America, a region this participant suggested has:
[…] their own research cultures, and they are very good in building their own databases and own publication networks […] in Latin America, they developed a very good and working network. They have prestigious journals with high impact factor. They cite each other, extremely, so they are good at that. We can see that their research is, we can speak about Latin American research integrity because these articles meet their standards. When you are not Latin American, it will be different for you. (P35)
Thus, despite the obvious benefits of having a global flow of information, it became clear that the new global environment for knowledge production also presents new challenges to understanding what constitutes quality across the world.
Getting Rejected
While rejection is a constant occurrence in the life of an academic, the realization by one scholar that ‘there’s an 80% rejection rate […] for the good journals in the US, that means, on average, you have to submit at least four times and go through peer review four times probably just to get published’ (P15) led them to wonder how that affects the knowledge production process. Another scholar suggested that constant rejection may lead to desperate scholars turning away from traditional, reputable publishers, and instead, to ‘just rely on predatory journal[s]’ (P03).
While rejection of manuscripts due to poor methodology, bad journal fit, or sub-par writing was accepted as a marker of quality assurance, getting rejected unfairly was a particularly salient issue for scholars working outside the Euro-American regions. The anecdotal evidence shared by our participants pointed to perceptions that the researchers’ country affected their likelihood of acceptance, such as the story shared by one scholar who argued that ‘when you submit your paper from Nigeria, and the same paper is submitted claiming that it is from Harvard, the referees will extremely overvalue the paper [from Harvard]’ (P35). This suggests that the objectives for some journals do not necessarily coincide with how knowledge producers across the globe believe their work should be received. The perception that unfair rejections happen to scholars working outside the Euro-American region highlights the issue with conceptualizing quality from a singular standpoint and has led to centuries of erasure of epistemic diversity in scholarly knowledge production.
Needing to Sensationalize Findings
Due to some of the institutional and disciplinary pressures mentioned in the sections above, several of our participants mentioned the temptation to over-sensationalize their work as a challenge to producing quality science (P15, P38), suggesting a ‘preponderance of people just wanting cool findings that you just throw at whatever high-impact journal’ (P45).
Another senior scholar spoke of the importance of instilling a sense of personal responsibility in junior scholars not to succumb to ‘temptations to maybe make the story a bit more streamlined […], or to make it a bit more simple, a bit easier to understand’ (P38). Another scholar added nuance to this discussion, stating that they were:
[…] always skeptical for research that support author hypothesis. Because if everything is supported then of course it may show that you are smart enough to predict everything but also it would say that probably you picked the result that just support your hypothesis. (P31)
But the trend of only publishing research that supports hypotheses or that is statistically significant is not just an issue for authors to address — it is also relevant to the journal editors and reviewers that function as gatekeepers. Many scholars spoke of the additional pressure from journals to only publish work that supported their hypotheses, rather than understanding that ‘sometimes negative results can be very important as well’ (P26), and that ‘I think it’s really important to have some of these venues that publish the negative things as well, because a lot of times an advance comes off something negative not off something positive’ (P08). One participant summed up the issue of feeling pressured to sensationalize their work and not publicize negative results as ‘the most harmful thing probably for scientific progress […] Because a null finding can be much more revealing than a finding’ (P48).
Flawed Peer Review System
Despite peer review’s continuing status as the gold standard for gatekeeping quality in scholarly knowledge production, it is not without its flaws and issues. Although only a couple of our participants offered a concrete definition of what a rigorous or stringent peer-review process might look like; as in the case of the fictional students in Zen, our participants could identify when a peer review had not been good, noting markers such as minimal reviewer comments (P04, P05, P10) or being asked to review manuscripts that are not within their expertise (P08). Through examining the flaws of peer review, and the challenges those flaws present in terms of quality in the knowledge production process, participants identified several major challenges to quality in the peer-review system: issues with reviewers; actions of deliberate fraud; the length of time peer review often takes; disbelief in the reality of double-blind reviews; and authors fearing the review process, rather than embracing it as part of the knowledge production process.
Issues with Reviewers
When considering the role of peer reviewers, scholars were especially sensitive to the difficulty of getting people to conduct constructive, critical reviews, leading one participant to share their fears of not wanting ‘to be the peer reviewer who says no to an article that costs someone their job’ (P06). Participants also noted that often reviewers are pressed for time, resulting in lower quality reviews, with comments like, ‘I think a lot of times, you know, the reviews are rushed in and they’re not as careful’ (P08), and ‘we’re all overworked and there’s no way we can put that much scrutiny into every empirical claim’ (P12).
Another participant hypothesized that some of the systemic issues of peer review revolved around the fact that,
[…] most of the reviewers are junior people […] Senior [scholars] don’t want to review […] the mentality is ‘I did all my reviews in the past, I’ve done my dues’ […] But the cost to the whole process because the junior people are good at finding faults, they’re really good, they’ll find a methodological flaw […] but we need a certain maturity or time to be able to see the bigger thing. And so I think that’s one of the things which is ecologically terribly missing in terms of production of good research is to get attention of senior reviewers. (P33)
Actions of Deliberate Fraud
Other participants noted that while peer review is important to vet scholarly work, ‘there’s a lot of evidence that the peer-review process is imperfect’ (P45) and ‘there’s fraud in peer review too […] but let’s just assume that’s all there is right now’ (P14) and ‘there are sometimes systemic failures to maintain a high level of peer review’ (P22). In considering the potential for fraud or unethical practices in publishing, one participant shared a story of a colleague’s review process where ‘the editor had asked him to include a bunch of citations from that same journal’ (P16), indicating there was a definitive effort on the part of the editorial staff to try to increase the impact factor of the journal through citation milling.
The concept of systemic failure was echoed by another participant who agreed peer review was the best way to ensure quality in vetting manuscripts ‘when it’s not used in a predatory way, you know what I mean? Like, “Oh, I hate that person”’ (P16). Other participants noted instances of people having to find their own reviewers, which could lead to them selecting people who would review their work favorably, rather than with an honest and critical eye (P25, P26).
The Peer Review Process Is Too Lengthy
Length of time from submission to publication was also suggested as a challenge to the process of getting new knowledge out to a wider audience, with one participant stating that many researchers ‘don’t want to wait for the peer review to happen in like more than six months’, and continuing that many scholars ‘are looking for the journals that can publish very swiftly’ (P19). At least four other participants (P13, P25, P26, P27) pointed to the speed from submission to publication as a reason for selecting journals with less stringent peer-review practices, as summarized by this comment:
They also are concerned about the timeframe, for them it’s too long, they cannot wait. They want a quick publishing and quick response, they want to publish immediately. Some of them they don’t want to go through the reviewing process, so that is very, very sad. (P39)
Double-Blind Is a Myth
Others doubted the double-blind process with comments such as, ‘I think double blind is a myth’ (P33), or questioned the validity of publishers claiming a double-blind process: ‘we hold double-blind peer review as our tried-and-true standard, but nobody really knows what that is. You don’t really know because it is double blind and it’s all behind the scenes’ (P21).
The idea that double-blind peer reviewing is a myth was especially salient to participants working outside of the Euro-American regions who often felt their work was unfairly rejected, simply because of their country of origin or the ethnic identities associated with their names, ‘I would say editors and reviewers judge based on your name, based on your country, or based on your institution, and that should not be the case […] because that’s not fair’ (P26). Along similar lines, another scholar claimed that non-Euro-American scholars faced additional scrutiny of their data:
[…] the authors in the other developing countries and sometimes the reviewers, they don’t know too much about the context. So they always do want lots of details. But that is not the case for the US journal or for the submissions that are using the US data. (P46)
The notion of bias against international scholars was expanded upon by another scholar who shared experiences of being an editor for a journal and having difficulty securing reviewers for submissions from outside the Euro-American regions: ‘Many scholars for example refused or declined to review articles outside their country. For example, there are a lot of American scholars who won’t do article reviews from developing countries’ (P31).
Authors Afraid of the Peer Review Process
Another important nuance in authors’ perceptions of the review process was addressed by some of our participants who felt the review process was too critical and unfair, or who saw ‘the review as punishment’ (P42) or wanted ‘no hassle’ (P15). Often, comments about the need for ensuring manuscripts were properly vetted before publication were also followed by doubts about the equity of the system, suggesting some scholars ‘publish in predatory journals because there is no very stringent peer-review system or language is not a barrier in that because even a low-quality writings are also accepted in predatory journals’ (P17).
Inconsistencies in Evaluation Methods
When we move from the macro-context of the global flow of information down to a more micro-level of quality assessment at the institutional level, defining quality becomes even more problematic. Academia, by its very nature, encompasses all fields, meaning an approach that works for a certain field may not work for another, as one participant noted: ‘the disciplines are so, you know, different’ (P08), a point emphasized by another participant who pointed out the differing expectations of research output between disciplines, ‘Like in the field of Computer Science […] they don’t value the journal publication too much. They value the conference proceeding publication’ (P46).
A bigger problem identified by several scholars from various parts of the world was summarized by one European participant who noted that ‘there is some assessment system, but nobody cares’ (P35), and then went on to share that quality research was not valued as much as being connected to the right people in the country where they work. A scholar from South Asia noted their national system of ranking publications did not ‘look at the readership’ (P19). This scholar went on to say, ‘They don’t look at the quality of the journal. They only look at the ratings’ (P19) before continuing to describe how this has lowered the production of quality research because so much focus has shifted on playing the ratings game.
At the institutional level, defining quality remains problematic due to the inconsistencies in evaluation methods between institutions and fields, such as varying methods for evaluating research production or placing higher importance on different forms of production. These inconsistencies make it difficult to assess quality in knowledge production between institutions or fields. Furthermore, the vast differences in evaluation of the knowledge production process (e.g., impact, citations, tenure and promotion) position evaluation methods more as a challenge to quality than a marker of its existence due to the necessity for researchers to consider how their work will be evaluated rather than focusing on the extension of knowledge and scientific applications.
Lack of Mentorship
There was some evidence, through the anecdotal narratives shared by our participants, that a lack of mentorship related to publishing ethics was allowing junior scholars to make mistakes in how they chose publication outlets for their scholarship, suggesting a flaw in the systematic role of mentors in preparing junior scholars to enter the academic publishing field. A North American university librarian summarized the problem of effective mentoring for junior scholars, stating:
But I’m just flabbergasted by how many students we have, who will go through the whole process and get their PhD, and then contact me a year later or more and not know how to approach these issues of how to avoid publishing in something that’s not good for them […] Because their advisors didn’t talk to them about it or it never came up. I don’t know how to fix that. (P06)
There are challenges at every step of the process when we consider how scholars view themselves and their work within the system of global scholarly knowledge production. These challenges suggest that much work is still needed to understand how to bridge the gap between institutional perspectives on quality and individual actions that produce the work in question.
Defining Quality
He singled out aspects of Quality such as unity, vividness, authority, economy, sensitivity, clarity, emphasis, flow, suspense, brilliance, precision, proportion, depth and so on; kept each of these as poorly defined as Quality itself…
Robert M. Pirsig, Zen and the Art of Motorcycle Maintenance (1974: 202)
Just as Phaedrus, the alter-ego of the narrator in Pirsig’s Zen and the Art of Motorcycle Maintenance, attempted — unsuccessfully — to draw a definition of quality out of his students, so did we with the interview participants, asking them to define ‘quality’ in relation to scholarly publishing in any way they saw fit. And just like the fictional students in Zen, our participants struggled to concisely define quality, instead turning to ways that quality can be demonstrated in the various steps of scholarly publishing, just as the students did in Zen. Some participants aligned quality with equally hazy concepts, such as ‘science [that] has been done properly’ (P18), whereas others attempted to define markers of quality in a more concrete way. Here, we can see the applicability of a multi-dimensional conceptualization of quality as our participants attempted to describe quality throughout the scholarly knowledge production process — from the training of graduate students in research ethics, to conducting the research, evaluating it, and eventually publishing and disseminating it. In the following sections, we show how participants described quality in each step of the knowledge production process.
Importance of Institutional Culture
For some of our participants, quality in terms of graduate student training is embodied at the institutional level, where the emphasis on ethical publication practices needed to come from the top down. One participant noted, ‘It needs to be first in the administration because when the administration care about predatory journal I think the student will just follow and the scientists also we just follow’ (P03). Another participant pointed toward the human resources graduate students have in their departments as a marker of quality: ‘The first thing I always tell them is that they need to talk with people in their department, and not just their advisor, the more the merrier in a lot of ways’ (P06). The point about the importance of the departmental culture was emphasized further by a junior scholar as he reflected on his faculty mentors during his graduate training: ‘I got to participate kind of hands on really early in research studies and projects that allowed me to kind of sharpen my skill set and fully determine if an academic route, again is where I wanted to go’ (P37).
Modeling Ethical Research Behavior
There was a unanimous sense from our participants that modeling ethical research behavior is the most powerful way to ensure graduate students emerge from their training with the intention of maintaining ethical research practices throughout their career. Senior faculty typically spoke about their role in training graduate students in terms of ‘individual responsibility’ (P34) and ‘being guards of [graduate student training]’ (P04), noting that faculty need ‘to become more vigilant I suppose as educators in that regard’ (P34).
Other faculty pointed toward the necessity of being open about the process of producing quality research, saying, ‘I try to give them a realistic picture’ (P34), and emphasizing the importance of sharing the ways their own careers and success in academia unfolded as he continued by saying, ‘You don’t see the trajectories. You don’t see how much help they got, or maybe the compromises they had to make in their lives with regards to family, all these kinds of things’ (P34). Another senior scholar agreed that openness is key, adding nuance related to the two-way flow of information needed between senior faculty and students when she stated, ‘Another way is make them be open […]. Make them feel confident that we can speak about their research project freely’ (P03).
When reflecting on the type of training they received about publishing ethics during their graduate training, one participant stated, ‘[My advisor] did sit us down and she gave us the ethics talk, and made sure what we were doing was correct and good to publish, but besides that, there wasn’t too much guidance’ (P30). Yet despite that lack of mentoring related to publishing ethics, somewhere along the way the participant picked up the importance of following ethical practices, and now tries to instill those qualities in their mentees.
This notion of modeling behavior for their mentees was echoed by several other scholars with comments such as, ‘The students are there. They’re looking [at] those things, those behaviors, so we are teaching them. So then we need to be an example, a good example to them’ (P26), and ‘So it is far beyond supervising […] It is kind of mentoring them […] all that type of skills that you can help them to acquire’ (P03). One young researcher exemplified how the culture of ethical research is handed down from mentor to mentee as he said ‘I’ve been trained by people […] who have the highest standards of research […] I try to pass that on now to the students I’m teaching, and I’m advising’ (P34). Conversely, one participant noted that bad ethical behaviors can also be modeled when she said, ‘the student know that the supervisor don’t care about what type of journal he sees’ (P03), emphasizing the crucial role mentors play in protecting quality in graduate student training.
Throughout the interviews, our participants repeatedly spoke about the role of faculty mentors, increasing awareness and efficacy of research integrity, and addressing cultural differences in international academies as ways to increase the quality of graduate student training related to scholarly knowledge production. The importance and centrality of mentors who model ethical research practices embodies the conceptualizations of quality as both a process and an end result, so that when graduate students move into the job market they are prepared to carry out research in an ethical manner. (See Chapter 5 for more on this topic.)
Committing to Ethical Behavior
Committing to ethical behavior when no one was looking was discussed in a range of ways, from statements relating to values such as, ‘Just behaving, being a good citizen in the scholarly world and not doing bad things with regard to your own research’ (P05) and ’the moral value of honesty is essential for the scientific process to progress’ (P08) to more concrete suggestions related to research practices, ‘not tampering with data’ (P19) and ‘don’t go and steal someone else’s work’ (P30). Participants touched on the need for ethical behavior in every part of the research process, from going through the ‘IRB process’ (P02), to ‘respecting your subjects’ (P10) to ‘present your results as it is’ (P02). Ultimately, committing to ethical behavior was best summed up by this participant who said, ‘you have to be dedicated to the truth, period, no matter what’ (P08).
Following Scientific Protocol
Following scientific protocol was the method of producing quality research that our participants mentioned most frequently. Some participants directly connected quality to following scientific protocol with statements like, ‘quality research is research that follows the scientific method’ (P02) and ‘Following scientific good practice’ (P24). While some participants broadly associated quality with scientific protocol, others delved more deeply into specific aspects that must be present to demonstrate that protocol was followed. Rigor was mentioned explicitly by numerous participants (P09, P14, P16, P31, P41, P48), along with replicable methods (P02, P08, P12, P19, P25, P31, P38, P46, P48) as ways to ensure scientific protocol was being followed, as summarized by this participant: ‘I realize quality is a judgment call, but to me it is very key about […] very clear methods, very clear controls, very clear descriptions of all your methods, ultimately in science, it’s supposed to be repeated’ (P08).
Dedication to Transparency
After committing to ethical behavior and following scientific protocol, a dedication to transparency was the next most common attribute of quality in scientific knowledge production addressed by our participants. Some framed transparency as ‘being willing to show your work’ (P04), whereas others positioned it as ‘giving appropriate credit to the people who were involved’ (P09) or mentioned specific mechanisms to increase transparency such as ‘preprints are one step in the right direction to giving a little bit more transparency because you can see how the work evolved’ (P12). An interesting nuance emerged between the qualitative and quantitative scholars in terms of inductive versus deductive reasoning and how information emerges from data differently from those perspectives, but one qualitative scholar still noted the importance of transparency even in the inductive process:
If you can be transparent and collect as much data along the way about what you’ve done, how you’ve done it, why you’ve done it, who you’ve done it with, what your results [are]. The more open you can be about that, the more likely you are to pick up what went wrong, what went right, share it, build on it. (P21)
Know Your Field
Because reproducibility is considered essential to science, our participants were adamant that a scholar who knows their field will be better equipped to produce quality research, stating ‘Academic research, especially a rigorous research journal, would ask you to have both knowledge of the past but also build new things for the future’ (P31). In this sense, quality was couched as ‘fitting your study in with the rest of the field’ (P08) while still ‘accepting the results that you are getting even though sometimes it could be that they don’t agree with the literature’ (P26). There was also an insistence that ‘a good quality article should be able to build on some sense of some people in the past and have their own new ideas that help us to understand the phenomenon or explain the phenomenon’ (P31).
Research that Has Been Deeply Thought About
This theme emerged primarily from the researchers we interviewed, showing a dedication to embracing the internalized process of knowledge creation, where ‘the researcher has to be curious on something. It’s the research that the researcher would do even if they had no gain, nothing to gain from doing that’ (P27) and ‘Quality for me has therefore also a lot to do with a kind of passion’ (34). Additionally, there was a sense that researchers needed to take the time to produce research that ‘has been thought over and discussed’ (P06), and that ‘Ultimately I can still stand behind what I would have wrote then ten years ago’ (P34) so that the research ‘can create a dialogue with the existing literature’ (P46) and continue to build upon the existing knowledge base.
Peer Review as a Marker of Quality in Journals
In general, when our participants were asked about how manuscripts and scholarly journals are judged for quality, they agreed it was through ‘robust’ (P11) or ‘rigorous peer review’ (P14, P22), or through a ‘stringent peer review process’ (P04), calling the peer-review process ‘essential’ (P05) to producing quality scholarly publications, and thus ensuring the quality of the journals themselves. Our participants broadly credited the watchful eyes of the editorial team and reviewers with the quality of a journal, as summarized by a researcher from South Asia:
You should look at a journal where the editorial board looks at your paper, critically comments on your paper, puts in for reviewers, and reviewers look at your paper again, and review the comments and try to seek your answers on the questions raised, and help you to improve upon the paper. (P19)
Others were more specific in their emphasis that peer review is what elevates a journal to a place of quality with comments such as ‘Science makes a hell of a difference […] The quality of peer review is so much greater […] and so the Nature is the same. They’re top journals’ (P18) and ‘the more prestigious the journal is, the more prestigious the reviewer […] That means that the review process is also […] higher quality’ (P26).
One participant whose job entailed assisting new journals to get started also noted the importance of transparency in the peer-review process, and his continual work to get journals to provide ‘evidence of their peer-review process and trying to get them to improve their processes there so they are more transparent as well’ (P29). Another participant with editorial experience also noted that the quality of the reviewers directly affected the quality of the journal, but placed the responsibility on the editor(s) to:
[…] choose, or at least get the attention of the right reviewers, then the right set of reviewers for different papers. And it requires an understanding of the paper, it requires the understanding of the reviewer, it requires that you command enough attention of the potential reviewer, that he or she will be investing that much time in your journal because people invest times in journals because they believe in it, they like the work and so on. So there are a lot of art elements to this. (P33)
A further nuance emerged in the discussion of rigorous peer review and journal quality when one participant noted, ‘The Lancet got in trouble for a COVID article. It has an extremely rigorous peer-review process so there are no foolproof methods of assessing quality that work 100% of the time’ (P22). This notion that even well-respected journals might have questionable peer-review practices was corroborated by another participant who shared that a top-tier journal in their field ‘only requires one reviewer’ (P30). The participant questioned the quality of a single-reviewer process by saying, ‘So you only have to please one reviewer, which at first I was very leery […] because if you get someone who really doesn’t like your work, it’s done. If you get someone who likes your work, but doesn’t want to put in the time to improve it, then is that really a good paper that’s coming out?’ (P30).
Peer Review as Quality Assessment for Individual Manuscripts
Many participants put their full faith in the peer-review process when it was undertaken rigorously and by peers with the correct expertise with comments such as, ‘For me, quality research is peer reviewed and from a journal where I can look up what their peer-review policies are’ (P07) and ‘I wouldn’t trust the peer review, the quality of the process unless the work were being judged by true peers, people with a true understanding of that material’ (P5). This notion of peer reviews serving as a filter for manuscripts before they are published was summarized by a participant who stated:
Well, I think having the peer review is a good filter because the manuscript is supposed to be reviewed by at least three people, which is the editor and two reviewers. In theory, those reviewers should not have any direct contact with the main author or the author, the coauthors, at least in the three previous years or something like this. Those reviewers are supposed to be experts on the field. I think this is a good filter. It’s not the best, for sure. But at least I think if the editor did a good job selecting a good reviewer, that can be a really nice filter. (P26)
One participant with decades of experience as a leading scholar positioned the peer-review process as the ultimate way ‘to assure quality and integrity in academic publishing’ (P32), with another participant simply stating that ‘good academic publishing would have independent, unbiased reviewers’ (P38). The closest description of what a quality peer review should look like came from one participant who stated:
So a reviewer’s job is to evaluate the theoretical soundness of a paper, whether the arguments made on theoretical grounds are sound, which means they have accurate premises with references that are true, plus a logical conclusion. And whether the methodology is sound, and for that, we have methodological criteria. (P48)
One final nuance to the importance of peer review in protecting the quality of scientific knowledge was an exploration of the potential positive impacts a more open peer review process could have on the quality of reviews that are conducted, which would ultimately improve the quality of the individual manuscripts, and thus the journals, and ultimately the entire field of study:
Frontiers, for example. They have created a highly technological version of the peer review process […] one feature that I think tremendously improves their quality […is] the option for open review […] you submit, you get an anonymous review, the reviewers anonymously review, but then after they submitted their first review, they have an option to make their reviews open. So then the author can directly engage with the reviewer either anonymously or openly […] There’s not just accountability on the side of the author. Of course, you have to make sure that at an early stage of this process, the reviewer is protected. So the reviewer is anonymous and can actually reject freely and then the name will not be revealed. But once it’s through this stage, then the reviewer can work together with the author to then ultimately publish the paper. And I have also no problem then to put the reviewers on the paper as someone who contributed to the paper. I would say, if I’m a reviewer, I wrote sometimes twenty, thirty pages, single-space as a review and I saw many things implemented in the paper. If the authors put me in a note, in an acknowledgement […] I think that’s great. And in review processes and merit cases, you can bring this on and say, ‘Look, my reviews actually helped to make the paper better. What’s wrong with that? (P48)
While it would be wonderfully satisfying to end this section with some concise definition of quality in scholarly knowledge production, the reality is that quality is just as diverse, nuanced, complex, and ever-changing as the fields themselves. Thus, rather than a concise, unified definition of quality to conclude this section, we instead offer a roadmap for celebrating the epistemic diversity offered by a global system of knowledge production. Using a multi-faceted conceptualization of quality, we surmise that quality is both a process and an end goal. Thus, we can now turn a more critical lens on who is qualified to judge whether quality exists at each stage of the knowledge production cycle.
Judging Quality in Graduate Student Training
Quite simply, the faculty at an institution are the judges of graduate students, and thus tasked with ensuring quality in the students they mentor and teach. Faculty often serve in multiple roles for graduate students — as teachers in coursework, as supervisors in research and teaching capacities, and as guides or collaborators in research endeavors. In attempting to measure or assess graduate student education in terms of excellence, challenges emerge. How can one measure the quality of graduate student training based on the faculty in the department or the resources available to the student? Is it conceivable that a student with average faculty and few material resources might discover the next great scientific advance? In short, the answer is yes, which leads to the necessity of examining other dimensions of quality in terms of graduate student training and the development of ethical (and quality?) research practices.
Aside from the faculty who oversee a graduate student’s education, there are also some external sources of quality checking at the graduate student level. For example, manuscripts that are submitted for peer review in academic journals can serve as another point where the work of the graduate student is assessed for quality. Meaningful critiques of submitted work can often serve as encouraging turning points in a young academic’s career, whereas critical evaluations without suggestions for improvement can have the opposite effect — they may discourage young scholars from seeking rigorous peer review out of fear of being cruelly rejected. Finally, the penultimate assessment of quality would seem to be whether or not a graduate student can be employed in their job or career of choice soon after graduation — thus assigning the gatekeeping role of quality assessment to the faculty hiring search committees. When looking at quality in graduate student training with the backdrop of predatory publishing, it becomes even more complex to judge graduate student quality. If graduate students are not properly trained in how to vet academic journals, can they be held responsible for publishing their work in a predatory journal? The role of mentors, advisors, and senior faculty in ensuring graduate students understand the potential pitfalls of predatory publishing is discussed in detail in Chapter 5.
Judging Quality in Scholarly Research Production
Judging quality in the production of scholarly knowledge is a tricky process during the actual knowledge production process itself. The most salient judgement of quality typically comes after the research has been conducted — in the form of peer review. Yet when participants shared their perspectives of what constitutes quality in terms of the knowledge production process, they often listed attributes such as transparency, rigor, and other elements of ethical research practices that may be difficult for a peer reviewer to assess. For researchers working in collaborative teams, or in environments where there are opportunities for informal peer reviews during the research process, this issue of judging quality before submission to a journal seems to resolve itself. But what about researchers who work independently, or who are not housed in a department or institution where pre-submission reviews are possible? How can researchers in those positions ensure they are producing quality work? Possible solutions are discussed in the section below.
Judging Quality in the Vetting Process for Scholarly Publications
While there is a large consensus in both the study we conducted and the scholarly and mainstream literature that peer review is the gold standard for vetting scholarly publications and ensuring the quality of published academic manuscripts, there is almost no discussion in those same venues of who ‘judges the judges’ so to speak. While there are agreed-upon elements of a quality peer review (e.g., clarity, constructive criticism, attention to detail, etc.), there is really only one gatekeeper that sees the peer reviews and has the authority to determine their quality — the editor(s) of the journal (Michael 2019). Yet editors may or may not be providing guidance on their reviewers’ performance — and in some cases they may not be expert enough in the content area to judge whether or not the review was accurate and fair. So what can be done to ensure the quality of the primary component responsible for quality scholarly knowledge being disseminated to the public and communities of interest? Some possible suggestions are offered in the concluding chapter of this volume.
Judging Quality in the Production and Distribution for Scholarly Publications
Judging the quality of publication and distribution of scholarly publications is a complicated mix between individual and institutional evaluations, coupled with third-party matrixes aimed at categorizing or ranking journals. At the individual level, as noted by our participants, journals are often judged by the articles they publish, as well as how they move authors through the process, and how their peer-review process is perceived. These individual perspectives are often at odds with institutional perspectives. A case in point is the participant who noted that they did not like (or respect) the top-ranked journal in their field because they rely on a single peer reviewer. Other participants also shared narratives of receiving lackluster peer reviews from top-ranked journals, indicating a rift between personal experiences and the perceptions of quality at the institutional/departmental/field levels.
In terms of institutional judgements of quality, numerous examples of field- or department- or institutional-level ‘lists’ of credible or preferred journals were given by participants. In addition to those instances, there are third party organizations, such as Directory of Open Access Journals (DOAJ) and the Committee on Publication Ethics (COPE) that gatekeep membership as a way of safeguarding (or encouraging) quality journals. Other indices, such as Scimago Journal & Country Rank, or Clarivate’s Journal Citation Reports attempt to rank journals based on citations, acceptance rates, or other identified markers in an effort to delineate quality journals from those that are sub-par or even predatory.
An added complexity related to judging quality in the production and distribution of scholarly publications is the issue of citations, and how citation numbers are used to evaluate whether an individual is creating impactful (quality?) research. The i10-index and citation counts embedded in Google Scholar are a perfect example of this complexity. Some institutions have begun looking at citation counts when evaluating their faculty, leading to a cycle where researchers might be inclined to over-cite themselves as a way to bump up their citation counts and i10-index (a ranking that indicates how many publications an author has with more than ten citations). Additionally, given that Google Scholar does not discern between known predatory and legitimate publications, authors who are looking to game the system and get higher citation rankings may be able to self-cite and submit sub-par publications to predatory journals.
Conclusion
When discussing the quality of the production of scholarly research, participants addressed the defining characteristics of quality research, challenges in producing quality work, and mechanisms they knew about to safeguard or increase research quality in their environments. When examining these elements in the context of increasing numbers of predatory publications vying for author submissions, the individual, ethical components of ensuring quality in the knowledge production process become even more important to safeguard academia.
One of our interview participants, a young North American publishing consultant, described the ethics of quality, touching on many of the separate strands mentioned by participants above:
To me quality is multifactorial […] Are we ethically reporting the data? And what’s the meaning, the purpose behind the data, behind the research? Is this to advance science, advance patient care? Or advance somebody’s academic ego? And also quality means, I guess, does it properly answer the research question? And I think ethics falls into quality. If you follow the ethical guidelines, then […] it’s quality research, and quality research should be reported and should be made available to the public. (P49)
From these definitions of quality we can already begin to see the disparate ways quality is understood in relation to producing knowledge — from the transformation of the researcher in their pursuit of new ways of seeing and interpreting the world, to how well the researchers carried out those activities, to how well the journals or publishers carried out their stated purpose of vetting and distributing scholarly knowledge — and begin to grasp how impossible of a task it may be to come to a unified explanation of quality in terms of scholarly knowledge production. Although our participants did not put forth a definition of quality that could be operationalized, their comments did reveal four primary areas where quality can be addressed in the production of scientific knowledge: training graduate students, the actual production of the research, how the research is vetted, and finally, how it is published. Through paying close attention to quality at each of these stages of knowledge production, it may be possible to successfully navigate the pitfalls of predatory publishing and to create impactful, meaningful knowledge in all areas of research and scholarship.
Key Takeaways
- Quality is integral to scholarly publishing, but concrete definitions remain elusive.
- Predatory journals are widely considered to be a threat to scholarly publishing.
- Challenges to research quality identified by our work include the following: the constant pressure to publish; valuing the quantity of publications over their quality; the dominance of Euro-American publishing standards; having manuscripts repeatedly rejected for publication; pressure to sensationalize research findings; a flawed peer-review system; issues with the quality of peer reviewers; actions of deliberate fraud; length of the peer-review process; the myth of double-blind reviews; fear of the peer-review process; inconsistencies in institutional evaluation methods; and lack of mentorship for junior researchers.
- Definitions of research quality identified by our research include the following: a strong institutional culture; modeling ethical research behavior by senior researchers; committing to ethical behavior; following scientific protocol; a dedication to transparency; knowledge of your field; taking the time to think deeply about your research; and peer review.
- At each stage of the knowledge production process there are gatekeepers tasked with judging the quality of the research.
- Faculty are the primary judges of quality during graduate student training.
- Judging quality during research production is a combination of informal peer checks and balances (e.g., team collaborations) and individual adherence to ethical principles of research (e.g., transparency and rigorous data analysis).
- Peer review is the primary way to judge quality as manuscripts move from production to publication.
- Journal rankings and citation numbers are often the primary indicator of quality after the publication of scholarly knowledge.
Discussion Questions
- Why is it important to understand quality in all the stages of the knowledge production process?
- Are there other challenges to quality that you can think of that are not included in this chapter?
- Of the challenges to quality identified in this chapter, ask individual participants to list them in order of most importance to least. Then, as a group, discuss similarities and differences in the ranking and why the group felt certain challenges were most salient.
- What are some ways to overcome the pain of having a manuscript rejected?
- Why is peer review discussed as both a challenge to quality and a way to help define it?
Activity
Important note: Begin this activity BEFORE assigning the chapter as reading material. This activity could be included during a larger thematic discussion of research ethics or even a research methods class.
Steps:
- Read the short quotes about quality from Zen in class (or post them on a discussion board for an asynchronous course). Facilitate a short verbal discussion (10–15 min) about how the group defines quality in terms of research production, then capture class thoughts for later re-examination (e.g., digital document, poster, audio/video capture). [Quotes are at the beginning of chapter sections]
- Present class with a definition for ‘predatory publishing’ [see Chapter One for the definition used in this book, or use a definition of your choice from existing literature].
- Ask the class to connect the attributes of quality they developed in Step 1 to the topic of predatory publishing they discussed in Step 2. Capture the themes they develop related to quality and how it relates to the challenges to quality presented by predatory publishing and the definitions of quality that are the markers of ethical research practices (To make the activity more concrete for lower-level students, the facilitator might want to discuss a specific journal or individual article that either contains markers of quality or which shows a clear absence of quality).
- Read the chapter and compare how the class discussion of quality in terms of predatory publishing and research publication ethics relates to what is presented in the chapter. Some follow-up discussion questions for after the class has read the chapter could include:
- Did the class consider the different global environments and pressures that affect researchers and scholars in different national settings?
- For settings outside the Euro-American sphere: How does the pressure to publish in English and in Euro-American journals affect your confidence in conducting ethical research?
- Faculty could share stories of getting their manuscripts rejected to help the students understand that it is part of the writing process and can serve as a way to improve manuscripts.
- Discuss peer review and how, despite its flaws, it still serves as an important gatekeeping mechanism for scholarly knowledge production.
- Introduce the importance of mentorship and encourage students to have frank discussions with their mentors and advisors about research ethics.
- Discuss the importance of institutional culture and ask students to reflect on the research behaviors they have seen from their faculty.
- Ask students to reflect on what they know about their field. The class could collaborate to develop a list of leading scholars in the field, and to research which journals are respected and credible in the field.
- Show students examples of peer review and discuss whether it was ‘quality’ or not. For example, the facilitator could show the class some examples of peer review they have received from one of their publications, or the class could attempt to find examples of open peer review on Publons.
References
Alfy, Shahira El and Abukari, Abdulai. 2020. ‘Revisiting Perceived Service Quality in Higher Education: Uncovering Service Quality Dimensions for Postgraduate Students’, Journal of Marketing for Higher Education, 30.1: 1–25, https://doi.org/10.1080/08841241.2019.1648360
Bairagya, Indrajit and Joy, Bino. 2022. ‘What Determines the Quality of Higher Education? A Study of Commerce Graduates in Kerala (India)’, Journal of the Asia Pacific Economy, 27.1: 1–25, https://doi.org/10.1080/13547860.2020.1870067
Biggs, John. 2001. ‘The Reflective Institution: Assuring and Enhancing the Quality of Teaching and Learning’, Higher Education, 41: 221–38, https://doi.org/10.1023/A:1004181331049
——. 2011. Teaching for Quality Learning at University: What the Student Does (New York: McGraw-Hill Education)
Bloch, Carter., Degn, Lise., Nygaard, Signe and Haase, Sanne. 2021. ‘Does Quality Work Work? A Systematic Review of Academic Literature on Quality Initiatives in Higher Education’, Assessment & Evaluation in Higher Education, 46.5: 701–18, https://doi.org/10.1080/02602938.2020.1813250
Bowden, J., and Marton, F. 2003. The University of Learning: Beyond Quality and Competence (London: Routledge)
Brennan, J., and Shah, T. 2000. Managing Quality in Higher Education: An International Perspective on Institutional Assessment and Change (Maidenhead, UK: Open University Press)
Carroll, Aaron E. 2018. ‘Peer Review: The Worst Way to Judge Research, except for All the Others’, The New York Times, https://www.nytimes.com/2018/11/05/upshot/peer-review-the-worst-way-to-judge-research-except-for-all-the-others.html
Cavallone, Mauro., Ciasullo, Maria Vincenza., Douglas, Jacqueline and Palumbo, Rocco. 2021. ‘Framing Higher Education Quality from a Business Perspective: Setting the Conditions for Value Co-Creation’, Studies in Higher Education, 46.6: 1099–111, https://doi.org/10.1080/03075079.2019.1672644
Forrester, Amy., Björk, Bo-Christer and Tenopir, Carol. 2017. ‘New Web Services That Help Authors Choose Journals’, Learned Publishing, 30.4: 281–87, https://doi.org/10.1002/leap.1112
Frambach, Janneke M., van der Vleuten, Cees P.M. and Durning, Steven J. 2013. ‘AM Last Page: Quality Criteria in Qualitative and Quantitative Research.’, Academic Medicine, 88.4: 552, https://journals.lww.com/academicmedicine/Citation/2013/04000/AM_Last_Page__Quality_Criteria_in_Qualitative_and.29.aspx
Giraleas, Dimitris. 2021. ‘Can We Assess Teaching Quality on the Basis of Student Outcomes? A Stochastic Frontier Application’, Studies in Higher Education, 46.7: 1325–39, https://doi.org/10.1080/03075079.2019.1679762
Harvey, Lee, and Green, Diana. 1993. ‘Defining Quality’, Assessment & Evaluation in Higher Education, 18.1: 9–34, https://doi.org/10.1080/0260293930180102
Humphries, Mark. 2021. ‘The Absurdity of Peer Review. What the Pandemic Revealed about Scientific Publishing’, Elemental/Medium, https://elemental.medium.com/the-absurdity-of-peer-review-1d58e5d9e661
‘In Defense of Knowledge and Higher Education’. 2020. American Association of University Professors, https://www.aaup.org/report/defense-knowledge-and-higher-education
Koerber, Amy., Starkey, Jesse C., Ardon-Dryer, Karin., Cummins, R. Glenn., Eko, Lyombe and others. 2020. ‘A Qualitative Content Analysis of Watchlists vs Safelists: How Do They Address the Issue of Predatory Publishing?’, Journal of Academic Librarianship, 46.6, https://doi.org/10.1016/j.acalib.2020.102236
Lagrosen, Stefan., Seyyed-Hashemi, Roxana and Leitner, Markus. 2004. ‘Examination of the Dimensions of Quality in Higher Education’, Quality Assurance in Education, 12.2: 61–69, https://doi.org/10.1108/09684880410536431
Lindsey, Duncan. 1989. ‘Using Citation Counts as a Measure of Quality in Science Measuring What’s Measurable Rather than What’s Valid’, Scientometrics, 15.3–4: 189–203, https://doi.org/10.1007/BF02017198
McGrail, Matthew R., Rickard, Claire M., and Jones, Rebecca. 2006. ‘Publish or Perish: A Systematic Review of Interventions to Increase Academic Publication Rates’, Higher Education Research and Development, 25.1: 19–35, https://doi.org/10.1080/07294360500453053
McKiernan, Erin., Schimanski, Lesley., Nieves, Carol Muñoz., Matthias, Lisa., Niles, Meredith, and others. 2019. ‘Use of the Journal Impact Factor in Academic Review, Promotion, and Tenure Evaluations’, PeerJ Preprints, 8: 1–12, https://doi.org/10.7287/peerj.preprints.27638
Michael, Ann. 2019. ‘Ask The Chefs: Peer Review Quality’, The Scholarly Kitchen, https://scholarlykitchen.sspnet.org/2019/09/12/ask-the-chefs-peer-review-quality/
Pirsig, Robert M. 1974. Zen and the Art of Motorcycle Maintenance (New York City: Bantam Books)
Pond, Brooks B., Stacy D. Brown, David W. Stewart, David S. Roane, and Sam Harirforoosh. 2019. ‘Faculty Applicants’ Attempt to Inflate CVs Using Predatory Journals’, American Journal of Pharmaceutical Education, 83.1: 12–14, https://doi.org/10.5688/ajpe7210
Pyne, Derek. 2017. ‘The Rewards of Predatory Publications at a Small Business School’, Journal of Scholarly Publishing, 48.3: 137–60, https://doi.org/10.3138/jsp.48.3.137
Roll, Shawn C. 2019. ‘The Value and Process of High-Quality Peer Review in Scientific Professional Journals’, Journal of Diagnostic Medical Sonography, 35.5: 359–62, https://doi.org/10.1177/8756479319853800
Shrestha, Jiban., Subedi, Subash., Shokati, Behzad and Chaudhary, Amit. 2018. ‘Predatory Journals: A Threat to Scholarly Publishing’, Journal of Education and Research, 8.1: 89–101, https://doi.org/10.3126/jer.v8i1.25482
Skolnik, Michael. 2000. ‘Does Counting Publications Provide Any Useful Information about Academic Performance?’, Teacher Education Quarterly, 27.2: 15–25
Suleman, Fátima. 2017. ‘The Employability Skills of Higher Education Graduates: Insights into Conceptual Frameworks and Methodological Options’, Higher Education, 76: 263–78, https://doi.org/10.1007/s10734-017-0207-0
Teeroovengadum, Viraiyan., Nunkoo, Robin., Gronroos, Christian., Kamalanabhan, T. J. and Seebaluck, Ashley Keshwar. 2019. ‘Higher Education Service Quality, Student Satisfaction and Loyalty: Validating the HESQUAL Scale and Testing an Improved Structural Model’, Quality Assurance in Education, 27.4: 427–45, https://doi.org/10.1108/QAE-01-2019-0003
Tracy, Sarah J. 2010. ‘Qualitative Quality: Eight “Big-Tent” Criteria for Excellent Qualitative Research’, Qualitative Inquiry, 16.10: 837–51, https://doi.org/10.1177/1077800410383121
Vazire, Simine. 2020. ‘Peer-Reviewed Scientific Journals Don’t Really Do Their Job’, Wired, https://www.wired.com/story/peer-reviewed-scientific-journals-dont-really-do-their-job/
Vykydal, David., Folta, Martin and Nenadál, Jaroslav. 2020. ‘A Study of Quality Assessment in Higher Education within the Context of Sustainable Development: A Case Study from Czech Republic’, Sustainability, 12.11: 4769, https://doi.org/10.3390/SU12114769
Welch, Catherine, and Piekkari, Rebecca. 2017. ‘How Should We (Not) Judge the “quality” of Qualitative Research? A Re-Assessment of Current Evaluative Criteria in International Business’, Journal of World Business, 52.5: 714–25, https://doi.org/10.1016/j.jwb.2017.05.007
Zerem, Enver. 2017. ‘The Ranking of Scientists Based on Scientific Publications Assessment’, Journal of Biomedical Informatics, 75: 107–9, https://doi.org/10.1016/j.jbi.2017.10.007
1 Portions of this chapter were previously published in Discover Education. The original manuscript is available online at https://doi.org/10.1007/s44217-022-00007-w. The dataset (NVivo file) that accompanies the published article is also relevant to this chapter and is available at https://doi.org/10.18738/T8/LD7SSX.
2 Our Texas Data Repository Dataverse includes a table showing participant demographic information. See (“Participant Occupation and Regional Demographics Table”). All quotations from interviews are reported without correction of grammatical errors or other irregularities. Some quotes were abbreviated using […] to achieve clarity of the original message.
3 The terms ‘Euro-American’ and ‘non-Euro-American’ are used in this chapter to describe sociocultural and demographic populations in place of ‘Western and Eastern’ or ‘Global North or South’ to reflect the geographic contexts we are trying to describe.