Analysis of the Language Used in the Reports of Peer-Review Journals

Introduction

Communicative genres or events are one of the methods by which people can communicate with each other. These events could be written, spoken, or conventional and often share certain characteristics and functions (Paltridge, 2017). Genres may be introduced to a particular audience in certain contexts. Genres are changed over time due to the change in values of that event or genre. Language is used in different ways according to the purpose, the content of the genre, and the relationship between people and the audience we write for, or the person we speak to (Paltridge, 2017). The use of language in a genre relies heavily on whether the text is spoken or written, and the cultural or social contexts it occurs.

Nygaard (2015) indicates that academic institutions around the world pay much attention to research production in highly prestigious journals since the academic reputation of an institution is closely related to the research production of individuals working in that institution. In the process of academic publication, the peer review system plays a vital role in identifying the rigor and quality of potentially published articles (Fletcher & Fletcher 2003). This process begins when a researcher submits a research paper to a journal. The editor of that journal checks the suitability of that paper to be published and if it was not suitable, the editor will reject it and this process is called “desk rejection”. If the editor finds the paper suitable to publish, the paper moves to the “peer reviewing” process, in which two or more (blind) reviewers are assigned to check the quality and rigor of the submitted paper. The reviewers will then recommend publishing the article as is, accepting with minor revision, accepting with major revision, revising, and resubmitting for further review, or rejecting it (Fletcher & Fletcher 2003).  

Many of the researchers encountered both positive and negative peer reviews. The received peer reviews contain a recommendation to accept or reject research manuscripts. However, the final decision on whether to accept or reject the manuscript is ultimately editorial (Delgado, Garretson, & Delgado, 2019). Journals request reviewers to assess the quality of research papers according to certain rubrics such as their novelty, contribution, comprehensiveness of literature review, and soundness of methodology.

The genre of the peer review process has received much attention over the last few decades (Belcher, 2007; Tahririan & Sadri, 2013).  Researchers tend to study and analyze the genre of peer review to understand the social practices and norms, which could give more insights into this opaque and controversial genre (Belcher, 2007). There is no doubt that previous literature has provided new ideas and thoughts in understanding the difficulties and problems of the peer-review process (Tahririan & Sadri, 2013). However, most of the previous research works have been carried out in medical journals. To the best of the researcher’s knowledge, there is a scarcity of research conducted on peer review analysis of papers taken from different disciplines. The present study aims at analyzing the genre moves and language used (negative and positive expressions) in peer review reports from a corpus of multidisciplinary research papers submitted by Jordanian researchers from different disciplines.

Literature Review

The Genre of Reviewers’ Reports

Language is used in different ways due to the purpose and content of the genre, and the relationship between the writer and the reader (Paltridge, 2017). Paltridge (2017) also mentioned that genres are not exemplary; they differ from each other, but a text may be a good example of a certain genre. Martin and Rose (2007) define a genre as “a staged, goal-oriented, purposeful activity in which speakers engage as members of our culture” (p. 8). Swales (1998) uses ‘metaphor’ instead of ‘definition’ when discussing genres. He claims that using the word definition could “prevent us from seeing newly explored or newly emerging genres for what they really are” (p. 61).

Genres are recognized through the represented texts, and they are often dynamic in that they evolve and improve to respond to the social changes and emerging ideologies and views which stand beyond the use of that genre (Swales, 2004). From this point, it can be said that genre users and their communities play a significant role in the reshaping of genres (Devitt, 2015; Schryer, 1994). The notion of genre is not defined based on the discourse form but based on the action it aims to achieve (Miller, 1984). This action is realized by people as a method of doing something, thus the genre is said to be a ‘social agreement’ (Miller & Bazerman, 2011). They added that genre is an agreement about the different ways of doing things in language, in particular, and in cultural and social contexts in general. Genres might take typical forms or content in which people realize and draw on through their engagement with the use of genres.

All the previously mentioned points are viable to reviewers’ reports, dwell in a certain setting, have a specific goal, and are provided for a specific community. The reports of reviewers normally discuss certain content, use typical language choices, and are often organized in typical ways. The presentation of peer review reports has changed due to technology changes. Many journals use online submission and review system that allows reviewers to address the points that should be considered in the review, and how to submit their reviews. Reviewers’ reports reflect the reviewer’s views of the research paper submitted and they also provide their opinions about that research (Paltridge, 2017). The reviewers would also be able to decide whether the submitted research is desired for publication or not. The peer-review process goes through different steps which should appear in a logical sequence. Figure 1 represents a simplified illustration of the peer review process in most academic journals (Miller, 1984).

Figure 1. The Sequence of the Peer-Review Process

The Discourse Structure of Peer Review Reports

Many researchers (Fortanet, 2008; Samraj, 2016; Swales, 2004;) have identified the moves of peer review discourse. Fortanet (2008) analyzed the peer review reports in two different areas (applied linguistics and business). Based on this analysis, four stages or moves emerged as follows:

Move 1: Summarizing judgment regarding suitability for publication

Move 2: Outlining the article

Move 3: Points of criticism

Move 4: Conclusion and recommendation (Fortanet, 2008, p. 35)

Samraj (2016) analyzed the discourse structure of 25 major revisions, in addition to 25 rejected reviews. She found similar moves of review discourse to those suggested by Fortanet (2008). Samraj (2016) also analyzed the commentary sections in the major revisions and rejected reviews concerning lexico-grammatical aspects and negative evaluations. She concluded that both reviews (major revisions and rejected reviews) were different from an organizational point of view.

In fact, the topic of move analysis was first introduced by Swales (1998). This introduction is represented by analyzing the moves of discourse structures of introduction sections in research articles. Swale’s approach in analyzing discourse turned out to be a reference for much research focusing on the written genre, especially in the Field of ESP (Paltridge, 2015). Move structure analyzes written texts in terms of the rhetorical movement of each move that describes the text. It also analyzes what each move is accomplishing, its function within the text, as well as the similarities and differences in academic writing. The typical rhetorical patterns of genres are useful for novice authors to reflect their positions, views, and attitudes within a context of a certain genre (Giltrow, Gooding, Burgoyne, & Sawatshy, 2014).

Text analysis examines the sequence of text moves which includes some obligatory moves and some optional moves (Paltridge, 2017). It also examines the variance of reviewers’ suggestions or feedback according to the status of the submitted manuscript (i.e., accepted, minor revision, major revision, or rejected reviews). The analysis of texts determines the compulsory and optional moves in the mentioned types of reviews (Paltridge, 2017). The analysis of both obligatory and optional moves helps authors to expect what points to be reviewed regarding the submitted manuscripts. These points may include literature review, methodology, results, and discussion (Samraj, 2016).

Previous Studies

Several studies (Delgado, et al., 2019; Dobele, 2015; Falkenberg & Soranno, 2018; Paltridge, 2015) have been conducted to analyze the language of peer reviews comments received by refereed journals in different disciplines. Delgado, et al., (2019) studied the negative and positive expressions and words of peer review comments of the published manuscripts in the BMJ. The study also examined the sentiment differences of manuscripts subject to additional review rounds, review reports of initially rejected manuscripts, and published manuscripts after revision. The data source of this study was all peer review reports of published manuscripts in the BMJ from September 2014 to the end of 2017. The data includes 1716 peer review reports containing 908,932 words and expressions. The findings of this study revealed that words like “bias”, “confounding”, and “risk were among the most frequent negative words, while words like “important”, “well”, and “clear”, were among the most frequent positive words. Concerning sentiment differences, the findings showed that manuscripts published after the appeal had lower positive sentiment and joy, and higher scores of fears, sadness, and anger compared to those which were initially accepted. The authors of this study concluded that peer review reports of initially accepted papers were less negative and less related to high bias and risk.

Falkenberg and Soranno (2018) analyzed the peer review comments of 26 manuscripts submitted to Limnology and Oceanography: Letters. The study aimed at determining whether the editor’s decision on a manuscript was related to the content or to the agreement of the manuscript decision. The authors found that the quality of review reports was based on the content of the review rather than the agreement of the manuscript decision. In other words, the comments given by the reviewers were more specific to the objectives of the research, the nature of the analysis, results and discussion, and the importance of the research.

Paltridge (2015) studied the ways that reviewers ask for revisions to be made on submitted articles to peer review journals. The dataset of this study contained 59 review reports. Forty-five reviewers were requested to complete a questionnaire asking about their experience in peer reviews, how they learned to carry out reviews, and the problems they encounter in reviewing research articles. The study revealed that revisions take different forms such as suggestions, directions, requests, clarification, and recommendations. The study also showed that a considerable number of revision requests were direct requests, while most of them were indirect. For novel authors, it was very difficult to decipher the indirect requests to change. These indirect requests were directions to make very specific amendments, so the findings of this study are useful for novel researchers because they provide more insights on how to respond to reviewers’ comments, therefore, increasing the publication opportunities.   

Dobele (2015) analyzed the consistency of publication decisions and the effect of feedback on authors, in terms of the comments tone and the number of written comments. The data source of the study included feedback from six journals and one edited book in two conferences in Australia. The findings revealed that the publication decision whether to accept or reject the research paper was generally consistent. It was clear that the provided feedback was constructive feedback aimed to increase the publication chances. On the other hand, the number of written comments was limited, and, in some cases, the reviewers’ comments were inconsistent. This inconsistency led to further work for editors, so the findings suggested that the peer review process needs managerial changes to reach distinguished peer review comments.

Method

Dataset

The dataset of this study includes a corpus of peer review reports sent to Jordanian researchers (Ph.D. candidates) who have submitted their manuscripts to international refereed journals. The corpus includes 43 reports related to three different disciplines (Applied Linguistics, Sociology, and Accounting). The selected papers should have been published from 2016 to the end of 2020. There were 15 reports selected from the Applied Linguistics discipline, 14 from Sociology, and 14 from Accounting. The reports include 17 with major revisions, 19 with minor revisions, and 7 rejected submissions (Table 1).

Table 1. Description of the Received Peer-Review Reports

Data Analysis

The researcher conducted a content analysis on the compiled corpus to determine the moves, negative, and positive expressions used in peer review comments. This method of analysis is normally used in corpus linguistics and discourse analysis studies. It is extremely important to sort out similar basic patterns in the corpus of data (Gorsky, et al., 2012). Based on the content analysis, the researcher identified the discourse moves, negative and positive expressions of rejected reports, minor revisions, and major revisions of the manuscripts submitted by Jordanian Ph.D. candidates.

Results and Discussion

This section presents the analysis of the peer review reports in the three categories (i.e., papers with minor revisions, papers with major revisions, and rejected papers). The four moves of peer reviews discourse for each category are also analyzed and presented.

Papers with Minor Revisions

This section presents the moves of peer review discourse found in the papers with minor revisions (Table 2).

Table 2. Peer Reviews in the Papers with Minor Revisions (n= 19)

As shown in Table 2, the first move, that is, summarizing judgment occurred in all peer review reports, could be considered compulsory for reports with minor revisions recommendations. An example of Move 1 is shown below:

In my respect, this is an interesting paper and would be useful to teachers and researchers as it approaches research from a qualitative aspect.

(Applied Linguistics focused journal)

The second most frequent move was Move 4, conclusion and recommendation, which occurred in 15 of the reports that recommended minor revisions. The following instance shows one occurrence of this move:

It would be better if the findings are not only presented in terms of the percentage of the response for each questionnaire’s item but also how this means to students learning engagement.

(Accounting focused journal)

Move 3, points of criticism occurred in 4 out of 19 reports with minor revisions decisions. This is illustrated in the following example:

Few grammatical errors and inappropriate use of vocabularies [sic.] in the text need to be corrected. Therefore, language editing will be helpful.

(Sociology focused Journal)

The least frequent move was Move 2, outlining the article, with only one instance. The scarcity of Move 2 occurrence does not, of course, mean that reviewers did not outline the article, but they might have provided their comments on the manuscript itself. An example of this move is as follows:

The paper is written in accordance with the research convention. The focus and the findings are of a good significance [sic.], especially for those who are expecting some research on students’ engagement and the use of ICT in the teaching of English. However, the way the data is interpreted need [sic.] more elaborate endeavor to enlighten new ways of dealing with similar contexts.

(Applied Linguistics focused Journal)

Papers with Major Revisions

The corpus of this study contained 17 papers with major revisions. The moves of peer review discourse in these reports are illustrated in Table 3.

Table 3. Peer Reviews in the Papers with Major Revisions (n= 17)

According to Table 3, Move 1, summarizing judgment occurred in 14 out of 17 major revisions peer review reports. An example of Move 1 is as follows:

The topic of the paper is original. It has some theoretical rationale. However, in doing so, it does not consider other major works in the debate surrounding native and non-native.

(Applied Linguistics focused Journal)

The second most frequent move that occurred in papers with major revisions was Move 4, conclusion and recommendation with 13 instances. An example of Move 4 is as follows:

The paper has followed the required guidelines and provides an inspiring idea, but I do not think the paper could be published in its present form.

(Accounting focused Journal)

Move 3, points of criticism occurred more frequently in major revision reviews (11 out of 17). This is normal since reviews with major revisions expect authors to go thoroughly in their research and fix the major research elements. An example of this move is as follows:

There are too many basic grammatical mistakes that need to be fixed throughout the manuscript.

(Sociology focused Journal)

The least occurred move was Move 2, outlining the article, with only 3 instancesout of 17 major revisions reviews. The following is one example of this move:

The topic of the paper is original. It has some theoretical rationale. However, in doing so, it does not consider other major works in the debate surrounding native and non-native. The author(s) may wish to refer to the works of those scholars before discussing their findings and making bold claims from the findings.

 (Applied Linguistics focused Journal)

Rejected Papers

Table 4 shows the moves of peer review discourses in the rejected papers.

Table 4. Peer Reviews in the Rejected Papers (n= 7)

It seems that the only compulsory move in the rejected reviews was Move 3, points of criticism, which existed in all reviews. Reviewers explained why the article should not be accepted for publication. The points of criticism were many and so long, in one case four pages of criticism. The following is just a short excerpt of a review that contains points of criticism:

Abstract needs improvement. The significance and objectives of the study are missing. It also needs to clarify the theoretical framework. Research design is not mentioned clearly, i.e., qualitative, or quantitative. There is no clarification of the target population with age, gender, and other perspectives. Which SPSS version is used and what types of tests applied [sic.]? There should be more detail about the result of the study.

(Accounting focused Journal)

Both Move 1, summarizing judgment, and Move 4, conclusion and recommendation, occurred in 6 out of 7 rejected reviews. An example of Move 1 is as follows:

I do not feel this manuscript is suitable for publication in this journal.

 (Sociology focused Journal)

The following is an example of Move 4:

Based on the previously mentioned comments, I recommend rejecting this manuscript.

 (Accounting focused Journal)

The least occurred move was Move 2, outlining the article, with only two instances. For example:

The assurance of the quality of the research conducts [sic.]is not sufficiently reported. The writing of the manuscript has also serious methodological flaws.

 (Applied Linguistics focused Journal)

It can be seen from the above analysis that each category of manuscripts had certain moves that could be considered compulsory for the category of recommendation. For instance, the most frequent move that occurred in minor revisions, major revisions, and rejected review reports was Move 1 in which the reviewers provided their judgment on the submitted manuscripts. This is not surprising given the fact that in each of the recommendation categories, the reviewers explained the reason for not accepting the manuscript to be published as it is. The findings of this analysis are somehow similar to Fortanet’s (2008) analysis of peer-review reports. In Fortanet’s (2008) study, Move 3 was among the most frequent moves which was not the case in the present study. She also did not explain the category of recommendation reviews.

The move analysis presented in this study is broader than the analyses in the previous studies. Most of the previous studies did not include the categories of recommendation or included limited categories such as accepted and minor revisions reviews. The analysis of peer-review reports presented in this study supports the view of Paltridge (2017) in that reviews with major revisions seem to have a directive function, while rejected reviews appear to have largely negative comments and expressions compared to minor and major revisions reviews.

Sentiment Analysis

The corpus of this study consisted of 43 peer review reports containing 39,388 words. The peer-review reports have been selected from three different academic disciplines (i.e. Applied Linguistics, Sociology, and Accounting). Figure 2 presents all positive words obtained from the language analysis of peer-review reports.

Figure 2. The Positive Words Occurred in Peer-Review Reports

The most frequent positive words obtained from the peer-review reports included “well” (62), “good” (42), “original” (38), “Significant” (32), and “clear” (29). The positive word “well” appears in comments like “well-prepared paper”, “well-written study”, and “well done” manuscript. The word “good” occurs in phrases like “this is a good study”, “the paper studies a good topic”, and “this study addresses a good point”. The word “original” occurs in contexts like “this paper is original” and “this study provides original contribution”. The word “significant” appears in contexts like “this is a significant study”, “the researcher provided significant questions”, and “significant research area has been raised here”, while the word “clear” occurs in “the manuscript shows a clear structure” and “the paper is clear with a focused theme”. Figure 3 shows all negative words that occurred in all peer-review reports.

Figure 3. The Negative Words Occurred in Peer-Review Reports

The most frequent negative words that occurred in the peer-reviews reports included “unclear” (67), “inconsistent” (46), “poor” (43), “lack” (38), and “weakness” (31). The negative word “unclear” was mainly related to the methodology used in the study and it appears in comments like “the research design was not clear” and “there was no clear explanation of the targeted population”. The word “inconsistent” was associated with the use of referencing style, “the use of in-text citations was inconsistent with the APA 6th edition referencing style”. The word “poor” occurs in contexts related to the language used in the manuscript “the grammar and language used in this paper were poor”. The word “lack/s” was associated with the literature review and problem statement “this study lacks critical reflection from…”, while the word “weakness” was mainly related to the discussion part of the manuscript “a lot of weaknesses are found concerning the discussion part”.

Conclusion and Recommendations

The present study investigated the discourse structure, negative, and positive expressions used in the peer-review of minor revisions, major revisions, and rejected reports. ‘Summarizing judgment’ and ‘conclusion and recommendation’ discourse moves were found in all peer-review reports and were the most frequent moves. Words like “well”, “good”, and “original” were among the most frequent positive words used in the reports; while the words “unclear”, “inconsistent”, and “poor” were the most frequent negative words. The areas that reviewers most commented on were the research objectives, research questions, and research methodology. The areas of research methodology include research design and research instrument/s. Very few comments were made on the results or findings of the study. This implies that the research objectives, questions, and methodology were more significant than the results to many of the reviewers.

The findings of the present analysis suggest that research manuscripts will unlikely be accepted in academic journals unless the review reports are connected with positive words and/or expressions such as well-structured, the research methodology is solid and clear, and the content is suitable for a wide readership. This study helps researchers to demystify the peer-review process and draws their attention to the wording and comments of the reviewers' reports.

The findings of this study could not be generalized to all reports outside or even inside the fields included in this analysis. This is because of the relatively low number of peer-review reports, or the nature of academic fields included in this study. Further research is recommended to include a larger number of peer-review reports to provide more insights into the discourse structure of peer-review to both readers and authors. Finally, it would be of interest to study the variance of peer-review moves and the language used according to the blinding of peer reviewers in future analyses.

Ethics Committee Approval

The author confirms that the study does not need ethics committee approval according to the research integrity rules in their country (Date of Confirmation: 24.02.2021).

Acknowledgment

The authors extend their appreciation to the Deanship of Scientific Research at King Khalid University for funding this work through Big Research Groups under grant number (RGP.2 /53/43).