A Comparative Study of High-quality Academic Journals in Southeast Asia.

ABSTRACT This study compared the characteristics of the top-ten, multidisciplinary, academic journals, as ranked by the SCImago Journal & Country Rank database in 2014, in Southeast Asia (we used ASEAN, ex-Singapore, as a proxy for Southeast Asia) – Songklanakarin Journal of Science and Technology (TH), Sains Malaysiana (MY), Walailak Journal of Science and Technology (TH), Chiang Mai Journal of Science (TH), Maejo International Journal of Science and Technology (TH), ScienceAsia (TH), Chiang Mai University Journal of Natural Sciences (TH), Journal of Mathematical and Fundamental Sciences (ID), Journal of Science and Technology in the Tropics (MY), Malaysia Journal of Science (MY) – to analyze the factors for success and provide lessons for regional journals. We analyzed the ten selected journals using the SCImago’s journal metrics derived from its Scopus database, and compared four metrics: 1) SJR indicator, 2) number of documents published, 3) country affiliation of published authors, and 4) authors most frequently published. We also interviewed Editors-in-Chief and journal officers to gather both ‘demographic’ data about each journal and insights into how they are managed and issues they perceive with trying to improve the quality of their publications. Thailand is heavily represented in the top-ten regional, multidisciplinary journals, with six of the top-ten, and four of the top-five. However, the region as a whole is relatively weak, with its top-ranked regional journal only in the second quartile of all multidisciplinary science journals worldwide, and its seventh ranked journal already falling into the fourth quartile – Southeast Asia as a region is clearly an emerging, rather than established region for internationally recognized academic journals. As the Editors pointed out, these emerging regional journals face several challenges – limited visibility, tight budgets, and volunteer staff, among other obstacles. But, as they also suggested, a continued focus on quality content, finding quality peer reviewers, seeking out current and cutting-edge research, among others, can help lift these emerging journals to further impact and prominence.

Keywords: Academic journal, Scholarly journal, Journal indexing, Journal performance metrics, Southeast Asia

INTRODUCTION

The ‘marketplace’ for journals is increasingly crowded (30,132 academic journals listed in Scopus in 2014), and includes all types, from open source to subscription-based, online to print, free to fee-based submissions, university-based to private, among other variants. Smaller regional journals trying to establish themselves face a competitive field, but still play an important role for both universities and researchers. For an academic institution, its own journal can be a source of pride and enhance the university’s visibility, as well as offering a closer-to-home outlet for its own researchers to publish. But whatever the rationale for establishing the journal, the quality of its papers/publications remains the most important factor to its reputation, visibility, and even its continued existence.

In Thailand, a quasi-public organization, TCI, that bridges government and the academic sector oversees the development and quality standards of Thailand-based journals.

TCI manages the Thai Journal Citation Index (also referred to as TCI). Much like international indices, TCI has established a series of standards that Thai journals must meet before inclusion in its index, with two levels of quality. Meeting the standards of TCI is similar to or a first step to- wards inclusion in the international databases, depending on criteria and quality. In fact, TCI, in an agreement with Elsevier, is the ‘gatekeeper’ for Scopus in Thailand.

In 2014, 696 journals were published in Thailand, of which 275 are included in Tier 1 of TCI, 145 of which are science and technology journals, the type studied here. These, which include the six Thai journals studied here, are the most likely to achieve the quality standards for inclusion or eventual inclusion in one or more of the international databases. We have chosen to analyze 10 regional, multi-disciplinary journals of high quality, but still more regional than global in visibility and reputation, and look for commonalities or lessons as to how these journals have achieved what they have, and what they might focus on to further improve quality.

METHODOLOGY

Sample selection

We focused on emerging, regional, non-commercial journals, selecting multidisciplinary science and techno- logy journals in Southeast Asia for our analysis. The journals were selected from those included in the SCImago Journal & Country Rank database as of 2014 (http://www.scimagojr.com). To select our sample, we chose ‘Multidisciplinary’ as the subject category and Asiatic as the Region/Country. 

We then manually removed all but the ASEAN countries, which we used as a proxy for the Southeast Asia region. We also removed from consideration all journals that were commercially managed; this, in effect, excluded the top Singapore journals from our sample. We then selected the top-ten journals in 2014 that remained based on our search criteria – non-commercial, multidisciplinary science and technology journals published in Southeast Asia (defined as ASEAN, ex-Singapore). Figure 1 shows a screenshot of our selection criteria in the SCImago Journal & Country Rank database.

The final sample and their affiliations are shown in Table 1.

Table 1. The selected sample of top-ten, non-commercial, multidisciplinary, Southeast Asian journals.

Figure 1. A screen shot of the search criteria for selecting the top multidisci- plinary journals in the Asiatic Region in the SCImago Journal & Country Rank database (source: http://www.scimagojr.com).

While Walailak Journal of Science and Technology was founded in 2004, it was first included in SCImago’s database in 2012, with data first appearing in 2013. Of note, the Journal of Science and Technology in the Tropics and the Malaysia Journal of Science appear to have ceased publication since conducting our analysis, although they remain in the SCIma- go’s database.

Analysis of the top journals

The journals were analyzed and compared by a variety of metrics, both quantitative and qualitative.

Quantitative analysis. The journals were analyzed quantitatively based on data found in the SCImago Journal & Country Rank database (http://www.scimagojr.com). Data was sampled from a five-year period – 2010-14. Journals were ranked based on their SJR indicator, and overall quality further assessed using quartile rankings and impact factors. The number of papers published per year was determined from SCIma- go’s Scopus database by selecting the journal name and ‘source documents by year’. International collaboration was derived from SCImago’s Scopus database by selecting the journal name and ‘country-territory affiliation of all authors’, and deselecting the authors from a journal’s home country. The H-indices of the journals and authors were derived from SCImago’s Scopus database. For authors, we selected the ten most frequently published authors in each journal, and examined their H-indices, individually and collective- ly, to provide some idea of the impact of each journal’s ‘top’ authors (http:// www.scopus.com).

Trends and five-year averages of all data were analyzed.

Qualitative analysis. We first compared the journal management of the 10 studied journals drawing from journal ‘demographic’ data as reported by SCImago Journal & Country Rank, and each journal’s official website.

We then developed questionnaires with 26 open-ended questions. Nine questions asked about the journal’s characteristics or ‘demographics’ – for example, the number of articles and issues published per year and the type of peer review system used (single/ double blind, number of reviewers). Seventeen questions asked about more analytical-type data, such as the problems and obstacles of publishing the journals, including recommendations for improving the journals.

These were then completed by interview, either in person or by phone, in September 2015. We were able to conduct interviews and gather complete responses from all six Thai-based journals (five Editor-in-Chiefs and one Assistant Editor-in-Chief), but none of the others. These six responses were included in our analysis.

After collecting the questionnaire data, the 26 items were reduced to 10 key themes for analysis:

• Number of papers per issue
• Frequency of issues per year
• Scope of journal
• Rejection rate
• Submission method(s)
• Special issues
• Remuneration method
• Best practice of each journal
• Problems and obstacles
• Recommendations

RESULTS

Sampled journal characteristics

The general characteristics of the sampled journals are shown in Table 2. Seven of the journals are published by universities (or their faculties) and three by foundations, institutions, or associations. Six of the top-ten are Thai, three are Malaysian, and one is Indonesian; having excluded Singapore, none of the other Southeast Asian countries are represented in the top-ten regional journals. All but two of the journals been published for at least 10 years, with six for nearly 40 years or more. All publish issues at least twice a year, with six publishing 2-3 issues per year and another four publishing 6 or 12 per year. All but one charge no fees to authors for either submission or publication. Surprisingly, offering an online sub- mission option does not appear to be critical to success, as two of the top-ten still only provide ‘offline’ submission systems.

While two of the oldest journals are the only two in the second quartile, the oldest is only in the fourth quartile, and two of the youngest are in the third quartile; perhaps counterintuitive, the age of the journal appears to be unrelated to its ranking. In talking with the editors, quality, at least at the level of these less-visible regional journals that rely on mostly volunteer appointments from among whom are often otherwise overworked academics, appears to have much to do with the Editor and editorial team. Anecdotally, well-recognized editors and/or permanent editorial teams appear to lift a journal’s rankings. Some journals achieved or maintained high rankings under a particular editor, but fell back with changes in management. 

Only four (Sains Malaysiana, Chiang Mai Journal of Science, Maejo International Journal of Science and Technology, and ScienceAsia) of our top-ten journals were also included in the Web of Science (ISI) database, a more selective index. Journal management reported that authors were most interested in submitting to journals that were listed in both indices (Scopus and ISI).

The journal H-index rankings of our sample ranged from 15 to 2 (higher is better). The journal H-in- dex values tend to follow the SJR indicator; the higher the SJR indicator, the higher the journal’s H-index, with two exceptions. The third ranked journal in our sample, Walailak Journal of Science and Technology, had the lowest journal H-index value (2); although this likely reflects its only recent promotion into the Scopus database. The sixth ranked journal in our sample, ScienceAsia, has the highest journal H-Index (15); the Editor reported recently publishing several high-quality papers that received many citations.

SJR Indicator

Figure 2 shows the trends in the sampled journals’ SJR indicator. Five of the journals (Songklanakarin Journal of Science and Technology, Sains Malaysiana, Maejo International Journal of Science and Technology, Chiang Mai Journal of Science, and ScienceAsia) stand out as a group with higher SJRs (between 0.15-0.25 for most or all of the 5-year period) than the others, although three of these (Maejo International Journal of Science and Technology, Chiang Mai Journal of Science, and ScienceAsia) show troubling declines in their rankings over the last 2-3 years. The Chiang Mai Journal of Science showed the largest decline, perhaps because it started printing more articles in 2013, and even more in 2014 – more than doubling its output in one year (see Figure 3). ScienceAsia showed a similar increase in articles published 2013, but returned to its norm in 2014 (see Figure 3) – as the SJR indicator has a lag component, the increased selectivity again in 2014 would not yet be reflected in the indicator. Of note, the Mae- jo International Journal of Science and Technology’s SJR indicator fell along with the number of published articles. Of the other five journals, the Walailak Journal of Science and Technology, a recent entrant into the SJR database, improved substantially in just one year – jumping from sixth in its first year of inclusion to third in its second year. The remaining four journals are in a lower band of 0.10-0.15 (Chiang Mai University Journal of Natural Sciences, Journal of Mathematical and Fundamental Sciences, Journal of Science and Technology in the Tropics, and Malaysia Journal of Science).

These SJRs place these top-ten Southeast Asian journals within the second to fourth quartiles of all multidisciplinary journals in the SCIma- go database. With the seventh top regional journal already falling into the fourth quartile, it shows the relative weakness of Southeast Asia as a region – clearly an emerging, rather than established region for internationally recognized academic journals. The highest ranked Southeast Asian journal, Songklanakarin Journal of Science and Technology, is only the 20^th among all (n=51) multidisciplinary journals in the entire Asiatic region; and 909^th among all journals/ all subject categories (n= 2,231) in the Asiatic region.

Based on the 5-year SJR indicator average, Thailand has four of the top-five journals (Chiang Mai Journal of Science, Songklanakarin Journal of Science and Technology, ScienceAsia, and Maejo International Journal of Science and Technology). While the composition and order has changed somewhat, in the most recent year of analysis, 2014, Thailand still had four of the top-five journals (Songklanakarin Journal of Science and Technology, Walailak Journal of Science and Technology, Chiang Mai Journal of Science, and Maejo International Journal of Science and Technology).

Table 2. General characteristics of the sampled journals – the top-ten, non-commercial, multidisciplinary, Southeast Asian journals.

Figure 2. SJR indicator of the top-ten, non-commercial, multidisciplinary, Southeast Asian journals from 2010 to 2014.

Number of articles published (total documents)

Sains Malaysiana published the most articles (213) per year; it publishes monthly, more often than any of the other journals in our sample. Songklanakarin Journal of Science and Technology’s was a distant second (89), with three others in the 70-80s. The remaining five journals published less than 40 articles per year on average. While there appears to be no direct relationship between the number or papers or journals published per year and SJR indicator, as an indicator of quality, journals that are able to publish frequently and on schedule reflect a level of sustained submissions as well as streamlined management of the editorial and publication process. All else being equal, with more submissions, the journals were able to be more selective, improving quality in a virtuous circle. However, quick jumps in the publication rate unaccompanied by matching submission rates can lead to lower selectivity. Particularly for university-based journals as sampled here, which tend to publish journals both for its own prestige as well as to provide a publication outlet for its professors, balancing selectivity with providing more opportunities to publish can be conflicting mandates, and difficult to manage. Some of the Editors reported these tensions.

One possible outlet for increasing the publication rate while maintaining quality in advance of a corresponding rise in submissions is through special issues, which can gather together pre-selected papers from conferences or other proceedings on timely/current research topics.

Figure 3. The number of articles published annually by each of the top-ten, non-commercial, multidisciplinary, Southeast Asian journals from 2010 to 2014.

International collaboration

The ability to attract foreign authors tends to indicate increasing visibility of a journal, usually because of increased quality. However, this does not appear to be the case with our sample, as no obvious trend appeared between international collaboration and journal rank. For international collaboration (at least one co-author of a paper from a country other than the journal’s country of publication) over the five-year period 2010-14 (Figure 4), as measured by papers with a foreign author divided by total papers published, the Journal of Science and Technology in the Tropics had the highest average (33%), but was among the lowest ranked over the period, while the highest ranked journal, Songklanakarin Journal of Science and Technology, had one of the lowest rates (11%) of international collaboration. The highest ranked regional journals sampled here may be attracting high quality authors from their own countries, who would be the most familiar with the journals. To move from a respected regional journal to a more global journal, however, these journals will have to increase their international visibility.

Figure 4. International collaboration: the percentage of articles published annually that include at least one foreign co-author of the top-ten, non-commercial, multidisciplinary, Southeast Asian journals from 2010 to 2014.

Author H-index

The H-indices (from Scopus) of the ten lead authors most frequently published in each journal were analyzed (Table 3). While those journals with a higher collective H-index for its ten most frequently published lead authors tended to be grouped near the top in terms of the SJR indicator, no clear trend emerged. The two journals with the highest average H-index for its ten most frequently published lead authors – Chiang Mai Journal of Science (20.8) and Walailak Journal of Science and Technology (18.6) – were the fourth and third ranked journals, respectively, in 2014. In addition, the top ranked journal – Songklanakarin Journal of Science and Technology – had only the seventh best collective H-index for its top-ten authors (9.2). From the range within these top-ten (the H-index of the first and tenth authors), it would appear that many of those journals with a particularly high average have been able to do so by recruiting a few high-impact authors. The most frequently published author in the Chiang Mai Journal of Science has an H-index of 56, while its tenth most published author has an H-index less than 1. This is similar with Walailak Journal of Science and Technology, with a range of 55 to 1. In contrast, the Songklanakarin Journal of Science and Technology has a considerably lower top-ten author H-index average, yet the author its published tenth most often, had an H-index of 3, indicating greater depth across its top-ten.

Table 3. H-Index of the top-ten most published lead authors in each journal from 2010-14.

Scope of journals

A journal’s scope is determined by the owner, board, Editor, and/ or editorial team of each journal, and maintained in careful selection of submissions to consider for publication. The editors and staff of the sampled journals reported that careful selection of the scope was one of the main factors affecting a journal’s quality or impact factor; even in the multidisciplinary category, the clearer and more specific or narrow the scope, the more likely the journal is to be included in the ISI and/or Scopus databases. When the Chiang Mai University Journal of Natural Sciences, published since 2002, narrowed its scope to exclude the social sciences in 2007, indexing in Scopus followed in 2009. Defining the specific fields to be considered within a multidisciplinary scope provided clarity to authors – for instance, Walailak Journal of Science and Technology limits submissions to the Agricultural and Biological Sciences, Health Sciences, Physical Sciences, Engineering, Information and Communication Technology; although multidisciplinary and including seemingly disparate fields, the Editor reported that this specificity has helped it succeed.

Number of rejected articles

The self-reported rejection rates of the journals in our sample appeared to have little relationship to SJR indicator values, as eight of the journals, despite varied rankings, all shared self-reported rejection rates of roughly 80%. The other two reported roughly 50%, yet one was ranked first (Songklanakarin Journal of Science and Technology) and the other seventh (Chiang Mai University Journal of Natural Science).

From the interviews, the Editors reported the most common reasons for rejecting articles, in order of frequency, were: 1) content out of scope, 2) poor research content, either of low quality or inaccurate, and 3) plagiarism. Other reasons mentioned included not following the journal guidelines and poor language/writing.

Article submission methods

While most journals offered an online submission method, common today, the Maejo International Journal of Science and Technology succeeds while only offering paper or email submissions.

Special issue support

Special issues collect papers from academic conferences or other forums with a central theme. All of the journals in our sample publish special issues from time to time. Special issues, by drawing from conferences, provide an opportunity for journals to promote themselves and increase their visibility to researchers beyond their normal channels. According to the Editor of the Wailailak Journal of Science and Technology, by collecting similar topics together, special issues increased citations for the articles compared to publishing them in a regular issue of broadly multidisciplinary journal. When the Chiang Mai University Journal of Natural Sciences published a special issue on Agricultural and Natural Resources in 2012 that gathered outstanding papers from various institutions, its citation rate increased and the journal moved into the third quartile in 2012. 

Editorial remuneration policies

The editorial staffs of most of the Thai journals work on a volunteer basis. They are responsible for the journals as a sideline to their regular academic appointment, which usually includes research and teaching requirements, so it is difficult to devote the time necessary to fully manage the journals. In addition, as pointed out by the Editor of the Chiang Mai Journal of Science, overseeing a journal is not necessarily well-regarded within the academic institution, and does not count toward academic workload requirements, so editors do not prioritize their journal work.

The Maejo International Journal of Science and Technology is an exception to this, employing a full-time Editor solely responsible for the journal. Launched in 2007, it has developed quickly, and is now included in both Scopus and ISI.

CONCLUSION

Best practices

The Editors were asked for their views on best practices for journals. These included:

• Focusing on both the quality of content and writing/language in papers.
• Rejecting papers if even one of the reviewers provides sufficient and substantive rationale for doing so.
• Publishing on schedule.
• Selecting the most recent or cutting-edge research, as it tends to be cited more. This can include publishing special issues, which can serve to gather conference papers of a similar theme together, often representing the most recent research in the field.
• Having a stable of willing and technically strong reviewers, including recruiting reviewers from the international databases, such as Scopus and ISI.
• Managing the journal in a manner that is efficient for both journal staff and submitting authors.
• Having strong editorial staff.
• Using strategies to select outstanding articles for special issues that alternate with regular issues.

Problems and obstacles

The Editors were also asked to discuss problems and obstacles for producing quality journals. These included:

• Limited staff, with many of them volunteers.
• Limited time for journal work because of the demands, including teaching, of their primary academic appointment.
• Lack of status for journal work.
• Slow peer review process.
• Limited budgets for journals.
• Bureaucratic academic systems.

Suggestions

The Editors provided the following suggestions for publishing academic journals:

• Focus first and foremost on the quality of the content.
• Devote full-time staff to journals.
• Design website interfaces that are clear and easy for submitting authors.
• Increase budgets devoted to journals.
• Create a centralized database (in Thailand) for journals to share quality reviewers.
• Follow the standards and methods set by each journal’s institution, along with TCI.
• Publish papers online as quickly as possible to make them available to other researchers, increasing the chances for timely citations; publish the full-text, formatted journal articles later.

REFERENCES

http://www.scimagoir.com. Retrieved January 11, 2017.

http://www.scopus.com. Retrieved January 11, 2017.

BIBLIOGRAPHY

Thai Journal Citation Index. (2005). What is H-Index? Retrieved 2 March, 2015 from http://www. kmutt.ac.th/jif/public_html/ FAQ.html (in Thai)

Chaimin, C., & Plaichan, P. (2008). Factors affecting the success of scientific. International Journals of Academic Staff of the Faculty of Agriculture, Chiang Mai University, During 2004-2006. Chiang Mai: Faculty of Agriculture. (in Thai). Apairoj, T. (2012). A study of core competency for human resources development: case study of office of the president, Mahidol University, Salaya Campus, Thailand. Academic Services Journal, 22(1) January – April, 59-72. (in Thai)

Erawan, P. (2002). Qualitative re- search. Retrieved August 10, 2015 from http://www.udru.ac.th/kmudru/document/research2. pdf (in Thai)

Wittayawuttikul, R. (2012). Index of journal performance metrics. Bangkok, Mahidol University. (in Thai)

Wittayawuttikul, R. (2013). Index of academic journals / scholarly journal which information scientist should know? Bangkok. National Research Council of Thailand. (in Thai)

Wittayawuttikul, R. (2013). The Investigation about Index of journal performance metrics. Bangkok. Mahidol University.

Lincharoen, U. (2012). Qualitative data analysis techniques. Educational Measurement Mahasarakham University Journal, 17(1), 17-29. (in Thai)

Falagas, E. M., Vasilios. D.K., Arencibia-Jorge, R., & Karageorgopoulos, E. D. (2008). Comparison of SCImago journal rank indicator with journal impact factor. The FASEB Journal-Life Science Forum, 22, 2623-2628. doi: 10.1096/fj.08-107938.

Gonzalez-Pereira, B., Guerrero-Bote, V.P., & Moya-Anegon, F. (2010). A new approach to the metric of journals’ scientific prestige: The SJR indicator. Journal of Informatics, 4, 379-391. 

Guerrero-Bote, P. V., & Mya-Anegon,F. (2012). A further step forward in measuring journal’s scientific prestige: The SJR2 indicator. Journal of Informatics, 6, 674-688. doi: http://dx.doi.org/10.1016/j.joi.2012.07.001

Seglen, P.O. (1997). Citation and journal impact factor: questionable indicators of research quality. Allergy, 52, 1050-1056.

Van Servellen, A. (2015). Research metrics workshop. Presented at Chiang Mai University, Thailand. Weale, A.R., Bailey, M., & Lear, P.A. (2004). The level of non-citation of articles within a journal as a measure of quality: a comparison to the impact factor. BMC: Medical   Research   Methodology, 4, 14. doi:10.1186/1471-2288-4-14.