This article is the English translation of “Géographes, que valent vos articles ? Les bases de données commerciales en question”. La Lettre de l’InSHS, INstitut des Sciences Humaines et Sociales – CNRS, Issue 55, septembre 2018.
The question of validity and use of bibliometrics databases to characterize the productions of Social Sciences and Humanities (SSH) researchers are constantly revived in France, not only by the Haut Comité à l’Evaluation de la Recherche et de l’Enseignement Supérieur (HCERES), which is currently establishing its evaluation criteria Conversation suggested by : Michèle Dassa CNRS-InSHS and Christine Kosmopoulos CNRS-UMR Géographie-cités, but also by the Observatoire des Sciences et Techniques (OST), which organized a seminar in May 2018 bringing together specialists from different non-English-speaking countries. Based on the objective publication data produced by RIBAC and the JournalBase comparative database, it seems useful to us to make a new contribution to this reflection here.
Indeed, we have already published (Dassa, Kosmopoulos, Pumain, 2010) on the evaluation of the main tools for the identification of journals in the social sciences and humanities, a study which led to the development of JournalBase. For the record, JournalBase, updated annually since 2010, is a bilingual web platform for querying humanities and social sciences journals and comparing their referencing in the WOS, Scopus and HCERES journal lists and the former ERIH lists. JournalBase lists a total of 16,312 journals for 2017 in SSH, which is twice as many as Scopus (8,280 journals in SSH), almost four times as many as WOS (about 4,300 in SSH), and almost twice as many as HCERES (8,777 in SSH). The vocation of JournalBase is not bibliometrics, as no indicator or impact factor is associated with it, it mainly allows comparisons of the census of journals between private and public bibliometrics tools in order to bring reliable elements to the debate on the possible use of bibliometrics in SSH. The data on journals are enriched with the languages used, the countries of publication, the ISSNs, the URLs of the journal sites, etc. JournalBase, in open access, receives each month more than 1000 visitors, including 600 different ones, and more than 14 000 pages are consulted, half of them by American Internet users.
In addition, the exhaustive census in RIBAC since 2011 of the production of researchers at the SSH Institute of the CNRS, now provides us with new data on publications. This is why, based on the referencing of articles in scientific journals entered by researchers in the RIBAC observatory, we propose to examine here in which the journals researchers publications are best represented. On the basis of RIBAC, we will first look comparatively at the data from Scopus (Elsevier), WOS (Clarivate analytics) and the HCERES lists, as well as at the comparative tool JournalBase, which combines all these sources (after eliminating duplicates). Then, in a second step, we will study more specifically the results of Scopus/SCImago compared to the results of RIBAC. SCImago is a Spanish open access application based exclusively on Scopus data and whose use is currently debated. To illustrate our study, we take the example of the community of geographers of the CNRS attached to the section 39 of the national committee (Spaces, territories and societies), which gathers on average 150 researchers. We will examine the data captured over 3 years, from 2014 to 2016.
For the record, RIBAC is a secure web platform that allows CNRS-SSH researchers to register their scientific output of various kinds (articles, chapters, websites, media, databases, etc.) every year. Since 2011, RIBAC includes 100% of the scientific productions of CNRS researchers at the SSH Institute. To our knowledge, there is no other database that lists the diversity of SSH productions described in RIBAC. We are particularly interested here in scientific articles, the only data comparable to the “bibliometric” tools Scopus/SCImago, WOS, HCERES.
Between 2014 and 2016, geographers published, according to RIBAC, 629 scientific articles in 354 different journals. Among these 354 journals, 17 concentrate 23% of the publications (i.e. 147 publications, see Table 1). For the remaining 337 journals, we note a strong scattering since 243 journals (70%) record only one article over the 3 years studied, the others being distributed between 2 to 4 articles over 3 years. Concerning the referencing of the 354 journals where geographers published during this period, 190 are listed in JournalBase (54%) on the one hand, and 154 in the HCERES lists (43%), 102 in Scopus (29%) and 64 in the WOS (18%) on the other hand. Thus, by connecting the different sources HCERES, Scopus, WOS, we can see that JournalBase displays a more representative coverage of journals where geographers publish according to the data extracted from RIBAC. Comparing the different sources individually, the HCERES journal lists (mostly former AERES) are ahead of Scopus and WOS in terms of the number of SSH journals referenced in relation to RIBAC data.
Among the 17 journals mostly used for publications in geography between 2014 and 2016, Cybergeo: European Journal of Geography, ranks first quite far ahead of the other journals in terms of the number of articles published by CNRS researchers (Table 1).
Table 1: Top 17 journals for section 39 by number of articles published from 2014 to 2016 (source RIBAC).
|Journals||number of articles|
|Cybergeo: European Journal of Geography||23|
|Revue Internationale de Géomatique||11|
|Annales de Géographie||9|
|Revue archéologique du Centre de la France||8|
|Ecology and Society: a journal of integrative science for resilience and sustainability||8|
|Lecture Notes in Computer Science||8|
|Environmental Modelling and Software||7|
|Les cahiers d’outre-mer||6|
|VertigO – La Revue électronique en Sciences de l’Environnement||6|
|Revue francophone sur la santé et les territoires||5|
Of these 17 journals (table 1), 12 are listed in JournalBase (i.e. 70%), 12 are also listed in the HCERES lists, 4 are listed in Scopus-SSH(24%) and only 1 is listed in WOS-SSH (SSCI and AHCI) (6%) (table 2). We note that 12 of the 17 journals are published in France, 9 of which are actually listed in JournalBase and only 5 in Scopus. We also note that at least 5 French journals (in italics in Table 2) are not listed in any bibliometrics source, even though they are highly frequented by the geography community.
Table 2: Presence of the 17 most published journals (2014-2016) in the different sources and list of journals listed in JournalBase (RIBAC and JournalBase sources)
|Journals||articles published between 2014 and 2016||JournalBase||HCERES lists||Scopus-SSH||WOS-SSH||Country|
|Cybergeo: European Journal of Geography||23||yes||yes||yes||France|
|Revue Internationale de Géomatique||11||yes||yes||France|
|Annales de Géographie||9||yes||yes||yes||France|
|Revue archéologique du Centre de la France||8||yes||yes||France|
|Urban Studies||8||yes||yes||yes||yes||United Kingdom|
|Ecology and society: a journal of integrative science for resilience and sustainability||8||Canada|
|Lecture Notes in Computer Science||8||Switzerland|
|Environmental Modelling and Software||7||yes||yes||United Kingdom|
|Les Cahiers d’Outre-mer||6||yes||yes||France|
|VertigO – La Revue électronique en Sciences de l’Environnement||6||yes||yes||France|
|Revue francophone sur la santé et les territoires||5||France|
It should be noted that some journals, in which the majority of the researchers in section 39 publish, are not part of the stricto sensu field of the discipline (e.g. Revue archéologique du Centre de la France, Lecture Notes in Computer Science, Plos One) (table 2)
We present below a study of the data obtained from SCImago on geography journals published in France, in order to compare them with the information actually reported by researchers in RIBAC. SCImago is an open access online application developed by Spanish academics specialized in scientific information analysis and associated with the Spanish Higher Council for Scientific Research (CSIC). The tool is based exclusively on journals indexed in the Scopus database. It should be noted that SCImago ranks journals according to the Scopus indicator, the Science Journal Rank (SJR). It is therefore through these rankings that we will see the visibility of the publications of geographers registered in RIBAC.
To access the geography journals on the site, you have to select the journals (of the “journals” type) by major field, then by subfield. Once the broad field “Social Sciences” was selected, we studied the 3 sub-fields that seemed to cover the discipline “geography”: “urban studies”, “geography, planning and development” and “transportation”. The queries were performed in July 2018.
143 journals are listed in the “urban studies” subfield. The ranking by SCImago SJR indicator shows that the majority of journals are published in the USA, the UK or the Netherlands (headquarters of Elsevier, producer of Scopus). The first journal not published in these 3 countries is Czech (Critical Housing Analysis). It was added in 2018 and is ranked 39th. The first French journal is Urban Morphology, ranked 64th . It is also listed in JournalBase but no CNRS researcher has published in this journal over the 3 years studied (source RIBAC). The 2nd French journal is Articulo-Journal of urban resarch, which is ranked 88th . It is listed in JournalBase but only one CNRS researcher has published 2 articles in this journal over the 3 years studied (source RIBAC).
The other French journals are Méditerranée, Projections, Sapiens, Territoire en Mouvement and Histoire urbaine, ranked 91st , 92nd , 98th , 116th and 138th respectively. Among these journals, there are only 2 articles in the journal Territoire en Mouvement and therefore only 1 journal among the 12 most published French journals (source RIBAC). In total, 6 journals published in France are referenced in this subfield.
There are 634 journals in this field, so it is by far the largest field in Scopus/SCImago for geography (for information, JournalBase lists 946 journals in geography). What is the coverage of the journals in this field compared to the RIBAC data? The ranking by decreasing RLS shows that the majority of journals are published in the USA, UK or the Netherlands. The first journal not published in these 3 countries is Swedish (Ambio), ranked 29th. From these results, it can be seen that CNRS research geographers published a total of 38 articles from 2014 to 2016, in 11 French journals among the 22 French journals listed in SCImago in the field of “geography, planning and development” (Table 3) which are distributed in the following journals: Géographie Economie Société (3 articles), Flux (1 article), Annales de Géographie (9 articles), Revue de Géographie Alpine (3 articles), L’Espace Géographique (12), Herodote (3 articles), Etudes Rurales (1), Bulletin de L’Association de Géographes Français (1 article), Territoire en Mouvement (2 articles), Sud-Ouest Européen (1 article), Espace-Populations-Sociétés (2 articles). We also note that in this large field, only 2 journals (in italics in Table 3) are among the 12 most published French journals listed in RIBAC. We also note that there is no trace of Cybergeo in these results, we will come back to this subject in the discussion.
Table 3: List of the 22 French journals listed in the field of “geography, planning and development” and their ranking according to the descending SJR (source SCImago, request 20 July 2018)
|Rank SCImago||Titre of the journal||articles published from 2014 to 2016 by CNRS researchers|
|386||Géographie Economie Société||3|
|444||Annales de Géographie||9|
|462||Revue de Géographie Alpine||3|
|553||Cahiers d’Etudes Africaines||0|
|575||Field Actions Science Report||0|
|583||Bulletin de L’Association de Géographes Français||1|
|590||Géographie et Cultures||0|
|601||Territoire en Mouvement||2|
|635||Revue du Marche Commun et de L’Union Européenne||0|
|662||Histoire et Sociétés Rurales||0|
|669||Bulletin de liaison des membres de la Société de géographie||0|
91 journals are listed in the “Transportation” subfield. The ranking by descending Scopus SJR indicator shows that 70% of the journals in this category are published in the USA, the UK or the Netherlands. The first journal not published in these 3 countries is German, but again in English (Public Transport), ranked 42. There are only two French journals in this category, Recherche – Transports – Sécurité and Cahiers Scientifiques du Transport, ranked 79th and 85th respectively. These 2 journals are barely used by researchers since there is only 1 article published from 2014 to 2016 in Recherche-Transport-Sécurité.
These 3 subfields together therefore include 42 articles published between 2014 and 2016 in 13 different journals and list a total of 30 geography journals published in France.
JournalBase does not classify journals, unlike Scopus/SCImago, and does not provide any indicators, but it does provide precise information (language, url, publisher, country of publisher, language, sources in which the journal is referenced, etc.) on journals by discipline, and RIBAC exhaustively records all publications by CNRS researchers.
If we stick to articles issued in journals published in France between 2014 and 2016 in geography, we can see that CNRS researchers published 129 articles (Table 4) in 35 journals (source RIBAC) out of the 65 French journals listed in JournalBase in the discipline, while SCImago lists only 42 articles in 13 journals out of 30 French journals listed in total in SCImago/Scopus in the 3 subfields studied above.
Table 4: List of 35 French journals listed in the field of “geography” in JournalBase in which CNRS researchers published from 2014 to 2016 (sources RIBAC and JournalBase)
|Journals||Number of articles|
|Cybergeo: European Journal of Geography||23|
|Revue Internationale de Géomatique||11|
|Annales de Géographie||9|
|AutrePart. Revue de Sciences Sociales au sud||4|
|Natures Sciences Sociétés||4|
|Revue d’Histoire des Sciences Humaines||4|
|Géographie Economie Société||3|
|Revue de Géographie Alpine / Journal of Alpine Research||3|
|NETCOM – Networks and Communication Studies||3|
|Espaces et Sociétés||2|
|Espace, Populations, Sociétés||2|
|Norois – Revue géographique de l’ouest et des pays de l’Atlantique nord||2|
|Revue Européenne des Migrations Internationales||2|
|Territoire en mouvement||2|
|Bulletin de l’Association de géographes français||1|
|Développement Durable & Territoires||1|
|Géomorphologie : relief, processus, environnement||1|
|Problèmes d’Amérique latine||1|
|Sud Ouest Européen||1|
|Projets de paysage||1|
According to the sample taken by RIBAC over three years of publication, the bibliographic databases give uneven partial representations of the domain of journals used by researchers. Among the 354 journals in which Section 39 researchers published between 2014 and 2016 (source RIBAC), 190 are listed in JournalBase (54%) including: 154 in the HCERES Geography list (43%), 102 in Scopus/SCImago (29%) and 64 in WOS (18%). If we focus on journals published in France, we find 35 journals in which geographers have published 129 articles (source: RIBAC), whereas Scopus/SCImago only lists one third of them (cf. part 3). If we zoom in even more precisely on the 12 most published French journals in the discipline according to RIBAC (table 1) for the period studied, we find an even greater deficit in terms of representativeness in the commercial sources, since only 3 French journals out of the 12 are listed in the geography subfields of Scopus/SCImago, whereas 10 are listed in JournalBase under the heading geography. JournalBase therefore shows a coverage of journals much closer to the reality observed in RIBAC, whereas the SCImago data is very far from the reality of the publication practices of CNRS researchers.
As we have seen above, RIBAC data indicate that Cybergeo is the most used journal in geography to disseminate the research work of CNRS researchers in the discipline. It is an open access journal, available on the OpenEdition website. However, Cybergeo, although referenced in Scopus, is invisible when searched in the SCImago/Scopus geography journals. The reason is that Cybergeo is not identified as a SCImago/Scopus geography journal, but as a mixed journal with a physical geography bias, which is the opposite of the reality. This is probably not the only case since only 3 journals out of the 17 most published in RIBAC (table 2), were found in the SCImago geography sections. For example, the journal Géomorphologie: relief, processus, environnement, published by the Institut de Géographie de Paris, is not classified in geography, but in the field Earth and Planetary Sciences/Earth-Surface Processes. Similarly, the journal L’Information géographique is listed in the field of medicine.
We already pointed out in 2010 the deplorable quality of the Scopus data and the recurrent referencing problems for the humanities and social sciences which led Scopus to remove at the time all the keyword rankings of their 20,000 or so journals in order to adopt a more generalist referencing (as described above in SCImago ). Nevertheless, in its haste to compete with the WOS by integrating many more journals than the latter, Scopus has multiplied the number of SSH journals in its database to reach today nearly 8,300 journals in all disciplines, which may appear to be an advantage over the WOS, which counts about 4,300, but the problem of data quality remains.
In concrete terms, on the basis of our comparative study, the poor informational quality of the Scopus database has the effect of producing not only rankings of French journals that do not correspond, far from it, to what we observe in RIBAC, but also erroneous disciplinary distributions, non-exhaustive analyses of botched content and, in the end, indicators whose scientific value cannot be demonstrated. This under-representation has already been highlighted in many studies and in other fields of the humanities and social sciences, as these commercial tools identify a majority of Anglo-American journals (Mongeon and Paul-Hus, 2016).
The WOS, because of its long history in the census practice, is advancing more cautiously in the integration of non-English and non-Anglo-Saxon journals with the result of having a few European journals, for the moment very little representative of the scientific production in our disciplines, if we rely on the RIBAC data. It remains to recommend as sources of references the selective lists of the HCERES, or the journals listed in JournalBase which cover almost 2/3 of the scientific production officially listed in RIBAC. They have the advantage of being evolutive, and their information is controlled by the scientific community. These lists of journals representing a perimeter of scientificity had moreover been preferred to journal rankings, a few years ago in France, when the AERES lists were set up. It is therefore possible to define a set of journals selected by the scientific community without using rankings, which are more complex to establish and less consensual.
On the other hand, RIBAC was specifically designed to take into account the great diversity of publication activities and media in SSH. This tool has the advantage of identifying the variety of researcher’s productions according to the disciplines and therefore generating real data on research activities. This data thus provides reliable, verifiable arguments, which now allow us to analyze the validity of the results obtained with commercial databases such as Scopus and WOS. Sometimes wrongly accused of being an evaluation tool, RIBAC was developed as close as possible to the practices of researchers, with their participation and in their interest, in the face of the hegemony of commercial bibliometrics companies. Its exhaustive, unprecedented inventory makes it possible to display both the specificities and the variety and complexity of scientific practices within the various disciplines, which cannot be reduced to uniform models of practice, nor to quantitative indicators generated by databases over which the scientific community has no control and which, for journals, do not correspond to reality, as we show in this article. In the same spirit, moreover, the Leiden Manifesto for Research Measurement advocates in its 6th principle (Hicks, Wouters, Waltman, de Rijcke, Rafols, 2015) to take into account the differences between disciplines in terms of publication.
The Scopus/SCImago and WOS databases offer a partial and truncated representation of the publications of scientific articles by CNRS researchers in geography compared to the data extracted from RIBAC for this discipline. Aware of the limitations and the low number of European journals in their SSH databases, the producers of these tools (Elsevier and Clarivate Analytics) have sought to improve their coverage. Scopus regularly adds new journals, Thomson Reuters acquired by Clarivate Analytics launched in 2015 a new database, “Emerging Sources Citation Index” (ESCI). Although there have been improvements, the problem of predominantly Anglo-American coverage persists, for the WOS SSCI and AHCI databases and for Scopus. In 2017, there were 60% Anglo-American journals in Scopus-SSH, 61% in the WOS Humanities database (AHCI) and 76% in the WOS Social Sciences database (SSCI) compared to only 48% in JournalBase.
In addition, there is the question of the quality of the data contained in these databases, with the effects of erroneous rankings. The use of bibliometrics indicators such as the SJR indicator (Scopus/SCImago) or the impact factor (WOS) to rank journals and evaluate the scientific production of researchers and units is therefore de facto biased if we relate them to the dissemination of scientific work observed in RIBAC. Moreover, the CNRS indicated on its twitter account on July 14, 2018, that it is now a signatory of the San Francisco Declaration (DORA) which aims to improve the evaluation of research. “This text challenges the misuse of the Journal Impact Factor  indicator of reputation.”
In conclusion, the examination of geography journals shows that the indicators generated by Scopus/SCImago are not representative of the reality they are supposed to measure, nor do they adapt to the specificities and practices of each discipline. It is therefore essential to be aware of the risks involved in using these indicators and commercial databases, if we were to rely on the results they produce, both in the referencing of journals and in their impact, as scientific evaluation criteria. The development of JournalBase and its availability to the SSH scientific community in open access provides concrete data to compare the different bibliometrics sources and contextualize the scientific productions of researchers. Without being exhaustive, JournalBase remains to date the most representative documentary source of journals commonly used in SSH, without any bibliometrics ranking or indicator.
Dassa M. Kosmopoulos C. , Pumain D. , 2010, “JournalBase. Comparing international scientific databases in the social sciences and humanities (SSH)”, Cybergeo : European Journal of Geography. http://cybergeo.revues.org/22864
Hicks D. , Wouters P. , Waltman L. , de Rijcke S. , Rafols I. , 2015, “Bibliometrics: The Leiden Manifesto for research metrics”, Nature, Vol.520, N°7548, 429-431.
Mongeon P. and Paul-Hus A. 2016, “The journal coverage of Web of Science and Scopus: a comparative analysis, Scientometrics, N°106, 213-228
 National Center of Scientific Research, France
 “The portal includes the journals and country scientific indicators developed from the information contained in the Scopus® database (Elsevier B.V.)” (https://www.scimagojr.com/aboutus.php)
 source RIBAC 2014, 2015 and 2016
 Lists originally produced by the AERES
“SCImago is a research group from the Consejo Superior de Investigaciones Científicas (CSIC), University of Granada, Extremadura, Carlos III (Madrid) and Alcalá de Henares, dedicated to information analysis, representation and retrieval by means of visualisation techniques”
 It should be noted, however, that 164 journals in which geographers publish registered in RIBAC are not present in any of the three sources (HCERES, Scopus/ SCImago, WOS) and therefore do not appear in JournalBase.
 Social Sciences (miscellaneous)– Scope: Online publications grouped under the aegis “Geoscience e-Journals” (a former web ring converted into a genuine portal) incorporate those peer-reviewed electronic journals dedicated to open access publishing in the field of Earth Sciences (geochemistry, geophysics, paleontology, regional geology, stratigraphy, geography, etc.), that is those e-journals that permit free online access to all or at least a major part of their current issue and/or archives (SCImago page accessed on 23/7/2018).