Start Submission Become a Reviewer

Reading: The Relationship Between Institutional Factors, Citation and Altmetric Counts of Publication...

Download

A- A+
Alt. Display

Research

The Relationship Between Institutional Factors, Citation and Altmetric Counts of Publications from Finnish Universities

Authors:

Kim Holmberg ,

University of Turku, FI
X close

Timothy Bowman,

Wayne State University, US
X close

Fereshteh Didegah,

Simon Fraser University, CA
X close

Jonne Lehtimäki

University of Turku, FI
X close

Abstract

The goals of this research were two-fold. First, this research set out to investigate possible institutional characteristics that may or may not have an influence on online attention or, in other words, the number of altmetric events surrounding the scientific articles from that institution. The results suggest that international connections are important in the accumulation of altmetric events, possibly due to the creation of weak ties between researchers and their institutions. Second, it was studied whether the institutional research profile, i.e. in what fields the institution published, matched with the distribution of altmetric events across the same fields of science. Our analysis shows that the universities’ research profiles are not always reflected in the online events. Overall, the results of both goals of this research reflect a complex system where the received online attention can be attributed to many different factors.

How to Cite: Holmberg, K., Bowman, T., Didegah, F. and Lehtimäki, J., 2019. The Relationship Between Institutional Factors, Citation and Altmetric Counts of Publications from Finnish Universities. Journal of Altmetrics, 2(1), p.5. DOI: http://doi.org/10.29024/joa.20
71
Views
17
Downloads
  Published on 28 Aug 2019
 Accepted on 05 Aug 2019            Submitted on 07 Jun 2019

Introduction

The idea for altmetrics originates from researchers’ need to find new ways to locate relevant and interesting scientific articles (i.e., filtering) from the ever-increasing amount of scientific publications (Priem et al. 2010). Yet altmetric events have primarily been researched from a research evaluation perspective, with some qualitative approaches to analyze the online mentions of research products recently having been introduced. Earlier research on altmetrics has focused on investigating how different altmetric counts are connected to citation counts (e.g., Thelwall, Haustein, Larivière & Sugimoto, 2013; Haustein, Lariviére, Thelwall, Amyot, & Peters, 2014; Haustein, Peters, Sugimoto, Thelwall & Larivière, 2014; Bornmann, 2015), in some cases finding evidence of a connection between the two metrics. Other studies have analysed the potential influence of various document characteristics (e.g., discipline, title length, number of references and level of collaboration) on future altmetric events that research outputs attract (e.g., Haustein, Costas & Larivière, 2015; Didegah, Bowman & Holmberg, 2016) or how citation and altmetric counts differ for different disciplines (e.g., Costas, Zahedi & Wouters 2015). The goal of many earlier studies has been to explain the meaning of altmetrics and to understand what the online attention some research receives could reveal about the research at an article level. This article continues this line of research and investigates effects at an institutional level by studying altmetric events of publications from Finnish universities.

The first goal of this research is to identify if some institutional properties such as size of staff, amount of external funding, and number of international research visits have a connection to the level of online visibility the research publications receive on different online platforms. This line of investigation could reveal some new information about the mechanisms behind the creation of altmetrics and their possible connection to institutional properties of the organizations producing scientific outputs. We analyze the events aggregated by Altmetric.com and Mendeley associated with research articles from 10 universities in Finland between the years 2012 to 2014 from Wikipedia, Twitter, Facebook, mainstream news, blogs, and CiteULike, in combination with Mendeley readership counts retrieved from the Mendeley API. The second goal of this research is to investigate how the research profiles of the institutions (as measured by the distribution of Web of Science (WoS) indexed publications across different disciplines) correspond to the distribution of online attention (i.e. altmetric events) the same publications have received on different platforms. In other words, the second goal of this investigation can reveal some new insights into how well altmetrics can reflect the research profiles of universities. This article proceeds as follows: in Section 2, the literature on the subject is summarised; in Section 3, the data collection and methodology of the study are described. In Section 4, we present the results of the study (section 4.1 addresses the first research question and 4.2 addresses the second research question) and in Section 5 we discuss the results and conclude with our findings.

Background

Altmetrics (short for alternative metrics) has emerged as a potential complementary data source for metrics connected to research performance. Indicators derived from scientific publications and citations are frequently used to measure scientific impact, but they do not take the complexity of scientific activities into account. Citations, for example, only reflect how often other researchers have used a specific scientific article, thus only reflecting the scientific impact of research, while research can and often is expected to have much wider impact on the society. As altmetrics are aggregated from online platforms open to the general public (as well as researchers), they have the potential to reflect both new forms of scholarly communication and the attention received from a wider audience outside of academia. However, there are still many unanswered questions about the applicability and reliability of altmetrics. Altmetrics are not without challenges. Earlier research has shown how only a fraction of scientific outputs receive online attention that generates altmetrics (e.g., Costas, Zahedi & Wouters, 2015). Altmetrics can be manipulated unintentionally or intentionally by automated accounts or so-called bots on various platforms (Haustein, et al. 2016). Data quality issues and the dependency on the availability of both APIs for data collection and DOIs for identification place great challenges for altmetrics research (Haustein, 2016). Furthermore, the heterogeneity of altmetrics makes it important to view altmetric events identified on different platforms separately (Haustein, 2016). For instance, earlier research into the reasons for engaging with research outputs online has shown how the motivations vary between platforms and how the reasons for engagement vary even within the platforms (Holmberg & Vainio, 2018).

Many studies have approached altmetrics by studying correlations between traditional bibliometric measures and different altmetrics. Thelwall, Haustein, Larivière and Sugimoto (2013) identify that there is a relationship between blog mentions, Facebook wall posts, forum posts, mainstream media mentions, research highlights and tweets on article citation counts. However, they find that the coverage and overall number of mentions was very low for all studied platforms, with the possible exception of Twitter. More recently, some other studies have also found a connection between different altmetrics and later number of citations (e.g., Wang et al. 2017; Finch, O’Hanlon, & Dudley, 2018; Costas, Zahedi & Wouters, 2015), while other studies have not found any connection (e.g., Hassan et al. 2017; Delli et al. 2017; O’Connor et al. 2017; Ruano et al. 2018). These findings may be a result of several factors including differences in methodology, data samples, or possibly changes in the usage of specific platforms over time. While more research is clearly needed to understand how altmetrics are generated and what aspects of scholarly communication the accumulated online attention research receives can reflect, some aspects about the usefulness of specific data sources are emerging. Mendeley readership, for instance, has been suggested to be an important source for altmetric data due to its scholarly user base and similarity to citations. Thelwall (2017), for example, suggests that Mendeley reader counts could be used for early citation impact evidence (if used with caution), as they tend to have strong correlations with citation counts across almost all scientific fields. Twitter, however, may not be as suitable as a data source for altmetrics. Robinson-Garcia et al. (2017) find that only a small portion of the tweets mentioning scientific articles included some commentary about the article or some other evidence of engagement with the article. The majority of tweets was “almost entirely mechanical and devoid of original thought” and some were generated automatically by bots.

Most of earlier altmetrics research has focused on the possibilities of using altmetrics as article level metrics, while research on the applicability of institutional or country level altmetrics is almost non-existent. Alhoori et al. (2014) studied country level altmetrics and suggested that altmetrics could support research evaluation at that level. Alhoori et al. (2014) discovered a weak connection between aggregated country level altmetrics and more traditional impact measures, such as number of publications and citations. In more traditional scientometrics research aggregations of measurable events to various levels are more common. The much criticized (see e.g., Lariviére & Sugimoto, 2018) Journal Impact Factor (JIF), for instance, is an aggregation of the number of publications and citations a specific journal receives in a specific time frame. One of the criticisms surrounding the JIF is that it can be heavily influenced by a few articles that receive an exceptional amount of citations; Seglen (1997) writes “the most cited half of the articles are cited, on average, 10 times as often as the least cited half”. More recently it has been discovered that up to 75% of articles have fewer citations than the JIF of the journals would predict (Lariviére et al. 2016). It appears that the complexity of scientific activities is lost when aggregating bibliometric data to higher levels. This research investigates whether this also holds for altmetrics and whether aggregating altmetrics to an institutional level is useful in revealing some new aspects of altmetrics and the outside influence potentially influencing the creation of altmetrics. The goals of this research can be summarized into the following two research questions:

  1. How do specific institutional properties influence the level of attention research outputs from a specific institution receive?
  2. How well are the institutional research profiles reflected in the altmetrics events?

Data and methodology

The data about Finnish research publications was retrieved from the national Juuli research publications database. Juuli is maintained by the National Library of Finland in collaboration with the Finnish Ministry of Education and Culture and CDC—IT Centre for Science. The data for the database is collected annually from Finnish research organisations. For this article, a total of 114,496 publications were collected from 14 Finnish universities ranging from the years 2012 through 2014. CrossRef was queried through their API in an effort to add any missing digital object identifiers (DOI) to the data, after which a DOI was identified for 38,819 publications. These DOIs were used to search the altmetric data provided by Altmetric.com. This data showed that a total of 12,438 Finnish research publications from 2012–2014 had at least one recorded altmetric event. For some publications it was discovered that researchers from more than one Finnish university had collaborated, and these publications were counted for each participating university in this analysis. After these steps, the final data compiled for the study contained a total of 13,031 Finnish research publications from 2012 through 2014 with at least one altmetric event captured by Altmetric.com. A summary of the amount of articles included in the study from different universities can be seen in Table 1.

Table 1

Finnish research publications from 2012 to 2014 with at least one altmetric event, by university.

University Publications With assigned DOIs With at least one altmetrics event Percentage of publications with identified altmetrics events

University of Helsinki 30,296 11,066 3,966 13.1
University of Turku 12,196 4,203 1,706 14.0
Aalto University 12,091 3,377 1,177 9.7
University of Eastern Finland 7,680 3,231 1,149 15.0
University of Jyväskylä 8,060 2,924 1,072 13.3
University of Oulu 6,618 3,156 1,058 16.0
University of Tampere 6,568 2,346 939 14.3
Tampere University of Technology 5,158 2,876 675 13.1
Åbo Akademi University 3,878 1,429 501 12.9
Lappeenranta University of Technology 2,557 1,162 314 12.3
University of Lapland 2,414 637 230 9.5
University of Vaasa 1,703 640 166 9.7
Hanken School of Economics 785 332 101 12.9
University of Arts 421 33 12 2.9
Total 100,425 37,412 13,066 13.0

The analysis consisted of both bibliometric and altmetric data about the research publications and descriptive data about the universities. The altmetric data contains mentions of the research publications in blog posts, news articles, Facebook posts, Twitter posts, CiteULike, Mendeley and Wikipedia articles.

The descriptive data from universities consisted of the number of publications and citations in the same time period (retrieved from Web of Science), and the logarithmised amount of external funding the universities received, the proportion of foreign researchers, number of research visits, and the amount of research personnel. The latter data was retrieved from the Finnish Vipunen database, which is a national education statistics database maintained by the Ministry of Education and Culture and the Finnish National Agency for Education. In order to answer the first research question these factors were examined in relation to the altmetric events and citations. The approach of this article is similar to that of Alhoori et al. (2014), Didegah and Thelwall (2013), Thelwall, Haustein, Larivière and Sugimoto (2013), and Torres-Salinas, Robinson-Garcia and Jiménez-Contreras (2016), but examines different factors and university-level data with linear regressions instead of country-level data with correlations. This article attempts to approach the question by employing regression models. It should be noted that the data might contain some of the problems listed by Haustein (2016) as challenges of altmetrics. For example, the amount of level 4 staff (professors) or external funding in universities might very well be driving factors for total other research staff, i.e. influencing the number of other research staff. Table 2 lists some descriptive statistics for the universities in the sample. In order to answer the second research question, the universities’ research profiles—as measured by the normalized attention received from different altmetric events by the Organisation for Economic Co-operation and Development (OECD) main categories—were compared with the universities research profiles based on their research outputs as measured by Web of Science classification of the fields of publications. Due to a low number of publications in some areas, OECD categories were merged, which resulted in four main categories used for this study:

  1. Agricultural Sciences, Engineering and Technology
  2. Medical and Health Sciences
  3. Natural Sciences
  4. Social Sciences and Humanities.

Table 2

Summary statistics of Finnish universities.

Median Mean Standard deviation (n)

Level 4 staff 179.8 184.4 140.8
Total research staff 1121.9 1285.9 1065.4
International visits (from Finland) 209.5 292.2 188.5
International visits (to Finland) 158.8 211.9 160.6
Publications 1554.2 1750.2 1673.8
Outside funding (mn) 45.8 48.2 45.9

Results

University-level factors’ influence on altmetric events

The university level factors were chosen to represent different aspects of the universities’ activities: the size of the university, their internationality (to some degree), publishing activity and their level of success in securing external research funding. The following factors were tested:

  1. Level 4 staff (total working hours of professors per year)
  2. Other research staff (total working hours per year)
  3. International research visits from Finland (number of visits with a duration of at least two weeks)
  4. International research visits to Finland (number of visits with a duration of at least two weeks)
  5. Peer-reviewed published journal articles
  6. Amount of external research funding accrued from outside the university (in millions of euros)

First, the effect of each potential factor was examined separately in order to avoid multicollinearity issues. The levels of significance were omitted from the tables with the ordinary least squares estimates as all variables were found to be individually statistically significant at the 1% level. It should be noted that the coefficients of the effect of the total share of foreign researchers should not be directly compared to other estimates, which are based on absolute value variables, because they define the effect of a change of 1% as opposed to a change in absolute values.

The first test is defined as:

(1)
altmetricmeasurei,t=β1factori,t+ϵi,t

where altmetric measure is the activity in altmetric events, factori,t is the tested variable from the list above, for the universities i in year (t), and ∈i,t is the error term which defines the difference between the estimated values against a linear effect.

Table 3 presents the ordinary least squares estimates (standard errors in parentheses) for the chosen factors on different altmetric measures. The levels of significance were omitted from the table as all the factors were found to be strongly statistically significant as individual explanatory variables at the 1% level. The estimates explain how the change of a single unit affects the altmetric measures. For example, an increase of one full-time professor in a university increases Web of Science citations by 1.513, Wikipedia citations by 0.107 and tweets by 12.145.

Table 3

The relationship between university factors and different types of social media events.

Web of Science CiteULike Wikipedia Blogs News Facebook Twitter

Level 4 staff 1.51
(0.07)
0.76
(0.04)
0.11
(0.02)
0.41
(0.02)
0.77
(0.08)
1.11
(0.07)
12.15
(0.86)
Other research staff 0.26
(0.01)
0.13
(0.01)
0.02
(0.00)
0.07
(0.00)
0.13
(0.01)
0.19
(0.01)
2.08
(0.16)
International visits from Finland 1.02
(0.15)
0.51
(0.08)
0.09
(0.02)
0.28
(0.05)
0.50
(0.12)
0.78
(0.12)
8.34
(1.41)
International visits to Finland 1.23
(0.17)
0.62
(0.09)
0.12
(0.02)
0.33
(0.06)
0.57
(0.14)
0.91
(0.14)
9.49
(1.69)
A-type publications 0.15
(0.00)
0.07
(0.00)
0.01
(0.00)
0.04
(0.00)
0.08
(0.01)
0.11
(0.01)
1.18
(0.08)
Outside funding 5.22
(0.36)
2.68
(0.21)
0.33
(0.08)
1.47
(0.13)
2.57
(0.43)
3.58
(0.42)
39.74
(4.84)

Some conclusions that can be construed from the estimates in Table 3:

  1. Level 4 staff members are, on average, notably more efficient than other research staff in publishing research that is shared through the studied platforms. Whether this is due to these staff members publishing more or being more active in sharing their research through the studied channels is an open question.
  2. Foreign academic visitors to Finland have an influence on how much attention Finnish research receives online. They are, on average, somewhat more active in publishing research that is shared through the studied channels. Similarly academic visits from Finland also have a positive influence, so visiting a foreign university increases the altmetric visibility of research.
  3. Outside funding is a substantial factor in published research, although it can be argued that investing the same amount into permanent research staff could yield higher returns in research in the long run as investing a million euros for just a single year earns three to four times the effect of a single level 4 staff member, or about twenty times the same for a single other research staff member.

Many of the studied factors are strongly connected to each other. For example, the amount of level 4 staff members has a strong effect on the amount of total staff. In statistical analysis this problem of two factors defining each other is called endogeneity. In order to further study the effects of the chosen factors on the altmetric measures with multiple regression models, this inherent problem has to be addressed. This can be attempted with instrumental variable methods. Based on the results of OLS-estimates in Table 3, both the level 4 staff members and outside funding appear to be strong, unrelated driving factors for all the altmetric measures. Other research staff and the amount of users for the studied altmetric source are used as instruments to eliminate some of the effects of differing amounts of users for an altmetric channel and variances in total amount of research staff in universities.

The results in Table 3 give some evidence of the relationship between all the factor variables and the altmetric measures to be somewhat linear, thus the following two-stage least squares estimation is used for estimating.1

(2)
altmetricmeasurei,t=α+β1level 4staffi,t+β2outside fundingi,t+ϵi,t

where altmetric measure is the activity in altmetric events, level 4 staff and outside funding are the variables presented earlier from each university on an annual basis and

(3)
ϵi,t=μ research staffi,t+δ altmetric measure usersi,t+γi,t

where research staffi,t is the variable presented earlier, altmetric measure users is the number of users/sharers in the dataset and γi,t is the error term.

Table 4 presents the two-stage least squares estimates (standard errors in parentheses) for level 4 staff and outside funding when instrumented with users of the altmetric channel in question and non-professor research staff. The amount of professors is statistically significant at the 1% level for Web of Science, CiteULike, and Wikipedia citations, statistically significant at the 5% level for Facebook posts and Twitter posts, and at the 10% level for blog posts and news mentions.2 The amount of outside funding has a weak statistically significant, slightly negative effect on CiteULike readers, Wikipedia citations, and Facebook posts and a positive effect on Twitter posts.

Table 4

The relationships between different types of social media posts, level 4 staff and outside funding. Instruments: Users of altmetric channel and non-professor research staff.

Web of Science CiteULike Wikipedia Blogs News Facebook Twitter

Level 4 staff 3.412***
(1.010)      
1.859***
(0.547)      
0.417***
(0.149)      
2.745*    
(1.598)      
6.119*    
(3.157)      
6.608**  
(2.400)      
86.2**  
(40.4)      
Outside funding –4.410      
(3.101)      
–3.093*    
(1.680)      
–0.967**  
(0.456)      
–6.639      
(4.871)      
–14.73      
(9.63)      
–15.80**  
(7.31)      
0.001*    
(>0.001)      
(Constant) –143.4***
(51.8)      
–78.7***
(28.1)      
–17.5      
(7.3)      
120.4      
(70.8)      
–251.8*    
(142.6)      
–256.2*    
(118.6)      
–3334.2*    
(1864.7)      
Adjusted R2 0.85       0.79       0.51       0.36       0.39       0.45       0.36      
N 10,757       5,171       760       2,730       5,333       7,846       86,163      

*** = Statistically significant at 1% level.

** = Statistically significant at 5% level.

* = Statistically significant at 10% level.

Using the coefficients and adjusted R2 to directly compare the two-step least squares regressions to each other should not be done as the set of instruments change in each estimation. These measures can, however, provide some evidence for the fits of regressions. The amount of level 4 staff especially explains a portion of the changes in Web of Science citations counts CiteULike readers, and Wikipedia citations. For blogs, news posts, Facebook posts, and tweets, the model explains a smaller portion of the changes in altmetric activity when controlled with user/sharer counts and other research staff.

University-level research profiles and altmetric profiles

The research profiles of the universities (as measured by the distribution of published research articles across different research areas) were also examined to determine if the distribution of altmetric events across research areas would reflect the distribution of research outputs across the same research areas. Table 5 presents the distribution of published scientific articles by university and the distribution of the different altmetric events across the OECD categories. For instance, 47.4 percent of the WoS indexed publications from Aalto University are in Natural Sciences, 37.2 percent in Agricultural Sciences, Engineering, and Technology, 10.2 percent in Social Science and Humanities, and 5.2 percent in Medical and Health Sciences. Yet, the majority of publications from Aalto University that have received some attention on Twitter and Facebook (48.1% and 59.6% respectively) are in Medical and Health Sciences.

Table 5

University research profiles based on the distribution of events and publications (%) across four major research areas (OECD) by universities.

WoS profile Wikipedia Twitter Facebook News Blog CiteULike Mendeley Citations (WoS)

Aalto University Medical and health sciences 5.2 10.9 48.1 59.6 30.2 24.0 37.8 31.5 39.0
Natural sciences 47.4 50.9 27.0 22.1 28.1 51.4 34.6 36.1 38.4
Social sciences and humanities 10.2 5.5 8.8 5.2 3.7 1.9 6.5 12.0 3.5
Agricultural sciences, engineering and technology 37.2 32.7 16.1 13.1 38.1 22.6 21.1 20.4 19.2
University of Helsinki Medical and health sciences 36.8 24.4 47.7 44.8 33.7 28.9 39.9 31.0 34.0
Natural sciences 36.9 48.2 24.7 23.1 27.8 38.8 38.2 39.7 51.2
Social sciences and humanities 12.9 7.3 6.9 4.6 3.8 4.2 3.7 10.4 2.7
Agricultural sciences, engineering and technology 13.3 20.2 20.8 27.5 34.8 28.1 18.3 18.9 12.2
University of Eastern Finland Medical and health sciences 44.2 53.5 60.7 61.4 57.6 44.1 40.6 32.1 42.7
Natural sciences 30.3 27.9 19.0 16.4 17.1 33.9 40.4 44.2 42.3
Social sciences and humanities 7.6 0.0 5.8 4.1 3.5 3.8 5.3 7.5 2.3
Agricultural sciences, engineering and technology 17.8 18.6 14.4 18.2 21.9 18.3 13.7 16.2 12.8
University of Jyväskylä Medical and health sciences 17.3 16.1 34.6 28.6 23.7 17.5 31.3 24.8 20.9
Natural sciences 47.6 54.8 20.7 19.0 21.4 35.8 29.6 38.0 50.1
Social sciences and humanities 21.5 3.2 7.9 6.5 9.6 3.3 6.8 10.1 2.6
Agricultural sciences, engineering and technology 13.6 25.8 36.8 46.0 45.3 43.4 32.3 27.1 26.5
Lappeenranta University of Technology Medical and health sciences 0.7 19.0 56.2 78.8 70.7 44.0 42.7 30.9 40.7
Natural sciences 44.2 71.4 31.5 13.6 27.6 50.0 41.8 54.5 53.1
Social sciences and humanities 10.1 4.8 4.4 2.0 1.7 4.0 3.6 3.8 1.0
Agricultural sciences, engineering and technology 45.0 4.8 7.9 5.6 0.0 2.0 11.8 10.7 5.2
University of Oulu Medical and health sciences 27.9 27.7 51.3 44.6 33.6 21.5 40.0 29.7 35.7
Natural sciences 42.6 44.7 22.7 13.1 26.3 37.1 34.7 38.1 43.4
Social sciences and humanities 7.0 4.3 6.7 24.5 0.0 3.1 2.3 6.9 2.4
Agricultural sciences, engineering and technology 22.6 23.4 19.3 17.8 40.1 38.3 23.0 25.4 18.5
Tampere University of Technology Medical and health sciences 5.0 22.7 49.3 31.0 26.7 26.1 44.0 30.6 43.5
Natural sciences 47.3 50.0 25.3 24.1 41.9 46.7 31.1 40.5 40.8
Social sciences and humanities 5.1 4.5 10.5 6.0 4.3 6.5 4.9 12.2 2.7
Agricultural sciences, engineering and technology 42.6 22.7 14.8 38.8 27.1 20.7 20.0 16.7 13.0
University of Tampere Medical and health sciences 62.9 34.8 66.1 69.1 42.5 41.6 40.9 35.5 38.5
Natural sciences 16.6 54.3 18.1 16.2 40.7 37.1 39.4 35.9 44.8
Social sciences and humanities 13.7 6.5 4.8 5.3 4.7 2.0 5.8 12.0 3.9
Agricultural sciences, engineering and technology 6.7 4.3 11.0 9.3 12.1 19.3 13.9 16.6 12.8
University of Turku Medical and health sciences 40.4 15.5 52.9 62.9 42.4 24.0 37.6 30.1 23.1
Natural sciences 35.7 76.5 17.4 13.7 18.5 35.7 41.5 40.1 65.6
Social sciences and humanities 12.6 2.7 8.5 6.6 2.4 8.1 4.7 11.8 2.7
Agricultural sciences, engineering and technology 11.2 5.3 21.2 16.7 36.6 32.1 16.3 18.0 8.6
Åbo Akademi University Medical and health sciences 12.6 40.0 67.1 54.8 60.4 38.2 39.7 29.5 41.7
Natural sciences 44.7 40.0 17.1 22.1 21.1 34.0 35.0 40.5 38.8
Social sciences and humanities 13.0 0.0 3.1 4.3 0.3 4.9 2.3 8.9 3.5
Agricultural sciences, engineering and technology 29.7 20.0 12.8 18.7 18.2 22.9 22.9 21.1 16.0

In addition, the results show how the altmetric events from different sources are divided across the OECD categories for each university. While some universities are doing particularly well in Medical and Health Sciences on all altmetric sources (even when they do not necessarily have a medical school like Aalto University), other universities are doing especially well in Social Sciences on Facebook or Engineering and Technology in news sources. The results paint a picture of universities receiving online attention that may be different from their primary research profiles. Future research could focus on qualitative analysis of these reasons.

The distributions of the events were compared between the OECD categories for all universities. The average distributions of events across different platforms (including WoS publications and citations) are presented in Figure 1. The results reflect the overall popularity of Medical and Health Sciences articles on platforms such as Twitter and Facebook, while articles in Natural Sciences receive much lower attention on Twitter and Facebook than the publishing activity in the field would suggest. The results also demonstrate an overall low attention across all platforms received by articles in Social Sciences and Humanities. The results reflect how different types of research receives more attention on different platforms. The platforms showing distributions closest to that of citations may be further evidence of the platforms closer connection or more important role in scholarly communication, however, further research is needed to confirm this.

Figure 1 

Average distribution of events across different platforms by merged OECD categories.

The results from the Spearman Rank correlation between the distributions (as shown in Table 6) indicate that on average the distribution of altmetric events for University of Helsinki and University of Eastern Finland across different research areas correspond very well with the distribution of their research output (0.641 and 0.742 respectively), while the altmetric events for Tampere University of Technology, University of Jyväskylä, and Aalto University do not correspond that well with their research output (–0.042, 0.001, and 0.121 respectively). The implications of these results are further discussed in the next section.

Table 6

Correlations between distributions of events across fields of research and universities research profiles based on the distributions of their publications in Web of Science.

Variables Aalto University University of Helsinki University of Eastern Finland University of Jyväskylä Lappeenranta University of Technology University of Oulu Tampere University of Technology University of Tampere University of Turku Åbo Akademi University

Wikipedia: posts 0.80 1.00 1.00 0.20 –0.21 1.00 0.40 0.80 0.60 –0.20
Twitter: posts –0.20 0.80 1.00 –0.80 –0.40 0.80 –0.20 0.80 0.40 –0.20
Facebook: posts –0.20 0.40 0.80 –0.80 –0.40 –0.40 0.00 0.80 0.40 –0.20
News: posts 0.00 0.20 0.80 –0.80 –0.80 0.20 0.80 0.80 0.40 –0.20
Blogs: posts 0.40 1.00 1.00 –0.40 –0.40 0.40 0.40 0.80 0.00 –0.20
CiteULike: reader –0.20 0.80 1.00 –0.80 –0.40 0.80 –0.20 0.80 0.60 –0.20
Mendeley: readers 0.40 1.00 0.80 0.20 0.00 1.00 0.40 0.60 0.60 0.40
WoS: citations –0.20 1.00 1.00 0.20 0.00 1.00 –0.20 0.60 0.60 –0.20

Discussion

The goals of this research were two-fold. First, this research set out to investigate possible institutional characteristics that may or may not have a connection to or influence on the online attention of the research outputs from that institution, in other words, the number of altmetric events surrounding the scientific articles from that institution. Second, it was studied whether the institutional research profile, i.e. in what fields the institution published, matched with the distribution of altmetric events across the same fields of science. This research is, however, not without limitations. One clear limitation of this research is that it focused only on scientific articles and specifically scientific articles that had received at least one altmetric event. As altmetrics data are now being collected also for scientific books (Williams, 2017), although the data still has some challenges (Torres-Salinas, Gorraiz, & Robinson-Garcia, 2018), future research could include scientific books in similar studies. Furthermore, the coverage of Web of Science may also have had some influence on the results (in particular on the second part) of this research (Mongeon & Paul-Hus, 2016). A second limitation of this study is that DOIs were used to match research articles with identified altmetrics events, thus neglecting many research articles that do not have DOIs assigned to them and that do not have any identified altmetrics events attached to them. As the use of DOIs becomes more comprehensive, and as other methods to identify altmetrics connected to research articles are developed, we can expect that future research can work with more comprehensive datasets. Furthermore, the impact that possible self-promotional activities of the authors of the investigated publications may have had on the altmetrics is unclear. Investigating possible intentional manipulation of altmetrics was, however, beyond the scope of this research.

The results of this study suggest that international connections are important in the accumulation of altmetric events. International visits both from and to Finland have a clear statistical influence on the altmetric events, especially on the number of tweets (and to some degree to the number of news articles). One possible explanation might be the influence of foreign academic visitors to Finland and visitors from Finland on a wider international network of people who become aware of published research. For example, Granovetter (1973) discusses the benefits of weak network ties in the diffusion of influence and information, mobility opportunity, and community organization. From this point of view, visiting scholars might create a weak tie between Finnish research (and researchers) and a wider international audience. Similarly, the effect that foreign university staff has on the altmetric events might be attributed to their existing networks and the diffusion of information through their established networks. Therefore, the effect might be due to an extension of the overall network availability of the published research rather than an individual researcher being more effective. A possible direction for future research would be to apply more advanced econometric methods to study the data. For example, one might use maximum likelihood estimates or random effect models to diminish the multicollinearity present in the data. It could also be interesting to study whether the factors used in this study affect altmetric measures in different ways across different institutions.

As for the investigated institutional research profiles and their similarity or dissimilarity with the distribution of altmetric events across different research areas, the results reflect 1) the popularity of articles in Medical and Health Sciences on some platforms, and 2) how the research profile of an institution is not necessarily reflected in the online attention the published work from that institution receives. Other studies have also found that articles from medical sciences receive more attention than articles from other fields (e.g., Cho, 2017), which can be possibly explained by the audiences’ general interest in medical matters, as many people are influenced directly by some medical findings. Another explanation could be that some of the articles gaining significant attention have received the attention due to their curious or humorous titles (Haustein et al. 2014). The second finding might be explained by some particularly popular articles that receive significant online attention, thus skewing the attention in benefit of that particular research area. Future research using more qualitative methods, such as content analysis, could confirm this hypothesis. It is, nevertheless, clear that the altmetric events do not necessarily reflect the institutional research profiles.

Overall, the results reflect a complex system where the received online attention can be attributed to many different factors. This may suggest that aggregating altmetrics to an institutional level may inherit similar problems as the aggregation of citations to calculate Journal Impact Factors have, i.e. how some popular articles can skew the end result and should thus be avoided.

Notes

1For example Wooldridge (2015) defines two-step least squares as a potential econometric method for studying an endogenous sample. 

2In statistics, the significance, or P-value, of a regression estimate defines how reliable the effect is. A lower P-value means a stronger reliability for the effect described by the resulting estimate. 

Acknowledgements

The authors wish to thank the reviewers for their constructive comments.

Funding Information

This research was financed by The Finnish Ministry of Education and Culture’s Open Science and Research Initiative 2014–2017 (funding number: OKM/33/524/2015).

Competing Interests

The authors have no competing interests to declare.

References

  1. Alhoori, H., Furuta, R., Tabet, M., Samaka, M., & Fox, E. A. (2014). Altmetrics for country-level research assessment. In International Conference on Asian Digital Libraries (pp. 59–64). Springer International Publishing. DOI: https://doi.org/10.1007/978-3-319-12823-8_7 

  2. Bornmann, L. (2015). Alternative metrics in scientometrics: A meta-analysis of research into three altmetrics. Scientometrics, 103(3), 1123–1144. DOI: https://doi.org/10.1007/s11192-015-1565-y 

  3. Cho, J. (2017). A comparative study of the impact of Korean research articles in four academic fields using altmetrics. Perofrmance Measurement and Metrics, 18(1), 38–51. DOI: https://doi.org/10.1108/PMM-02-2016-0005 

  4. Costas, R., Zahedi, Z., & Wouters, P. (2015). Do “altmetrics” correlate with citations? Extensive comparison of altmetric indicators with citations from a multidisciplinary perspective. Journal of the Association for Information Science and Technology, 66(10), 2003–2019. DOI: https://doi.org/10.1002/asi.23309 

  5. Delli, K., Livas, C., Spijkervet, F. K. L., & Vissink, A. (2017). Measuring the social impact of dental research: An insight into the most influential articles on the web. Oral Diseases, 23, 1155–1161. DOI: https://doi.org/10.1111/odi.12714 

  6. Didegah, F., Bowman, T. D., & Holmberg, K. (2016). On the differences between citations and altmetrics: An investigation of factors driving altmetrics vs. citations for Finnish articles. Journal of the Association for Information Science and Technology. 

  7. Didegah, F., & Thelwall, M. (2013). Which factors help authors produce the highest impact research? Collaboration, journal and document properties. Journal of Informetrics, 7(4), 861–873. DOI: https://doi.org/10.1016/j.joi.2013.08.006 

  8. Finch, T., O’Hanlon, N., & Dudley, S. P. (2018). Tweeting birds: Online mentions predict future citations in ornithology. Royal Society Open Science, 4, 171371. DOI: https://doi.org/10.1098/rsos.171371 

  9. Granovetter, M. S. (1973). The strength of weak ties. American Journal of Sociology, 78(6), 1360–1380. DOI: https://doi.org/10.1086/225469 

  10. Hassan, S.-U., Imran, M., Gillani, U., Aljohani, N. R., Bowman, T. D., & Didegah, F. (2017). Measuring social media activity of scientific literature: An exhaustive comparison of scopus and novel altmetrics big data. Scientometrics, 113(2), 1037–1057. DOI: https://doi.org/10.1007/s11192-017-2512-x 

  11. Haustein, S. (2016). Grand challenges in altmetrics: Heterogeneity, data quality and dependencies. Scientometrics, 108(1), 413–423. DOI: https://doi.org/10.1007/s11192-016-1910-9 

  12. Haustein, S., Bowman, T. D., Holmberg, K., Tsou, A., Sugimoto, C. R., & Lariviére, V. (2016). Tweets as impact indicators: Examining the implications of automated “bot” accounts on Twitter. Journal of the Association for Information Science and Technology, 67(1), 232–238. DOI: https://doi.org/10.1002/asi.23456 

  13. Haustein, S., Costas, R., & Larivière, V. (2015). Characterizing social media metrics of scholarly papers: The effect of document properties and collaboration patterns. PLoS ONE, 10(3). DOI: https://doi.org/10.1371/journal.pone.0120495 

  14. Haustein, S., Lariviére, V., Thelwall, M., Amyot, D., & Peters, I. (2014). Tweets vs. Mendeley readers: How do these two social media metrics differ? IT—Information Technology, 56(5), 207–215. DOI: https://doi.org/10.1515/itit-2014-1048 

  15. Haustein, S., Peters, I., Sugimoto, C. R., Thelwall, M., & Larivière, V. (2014). Tweeting biomedicine: An analysis of tweets and citations in the biomedical literature. Journal of the Association for Information Science and Technology, 65(4), 656–669. DOI: https://doi.org/10.1002/asi.23101 

  16. Holmberg, K., & Vainio, J. (2018). Why do some research articles receive more online attention and higher altmetrics? Reasons for online success according to the authors. Scientometrics, 116(1), 435–447. DOI: https://doi.org/10.1007/s11192-018-2710-1 

  17. Lariviére, V., Kiermer, V., MacCallum, C. J., McNutt, M., Patterson, M., Pulverer, B., et al. (2016). A simple proposal for the publication of journal citation distributions. bioRxiv, July 5, 2016. DOI: https://doi.org/10.1101/062109 

  18. Lariviére, V., & Sugimoto, C. R. (2018). The journal impact factor: A brief history, critique, and discussion of adverse effects. In W. Glänzel, H. F. Moed, U. Schmoch, & M. Thelwall (Eds.), Springer Handbook of Science and Technology Indicators. Cham (Switzerland): Springer International Publishing. 

  19. Mongeon, P., & Paul-Hus, A. (2016). The journal coverage of web of science and scopus: A comparative analysis. Scientometrics, 106(1), 213–228. DOI: https://doi.org/10.1007/s11192-015-1765-5 

  20. O’Connor, E. M., Nason, G. J., O’Kelly, F., Manecksha, R. P., & Loeb, S. (2017). Newsworthiness vs. scientific impact: Are the most highly cited urology papers the most widely disseminated in the media? BJU International, 120, 441–454. DOI: https://doi.org/10.1111/bju.13881 

  21. Priem, J., Taraborelli, D., Groth, P., & Neylon, C. (2010). Altmetrics: A manifesto. Retrieved from http://altmetrics.org/manifesto/. 

  22. Robinson-Garcia, N., Costas, R., Isett, K., Melkers, J., & Hicks, D. (2017). The unbeareable emptiness of tweeting—about journal articles. PLoS ONE, 12(8), e0183551. DOI: https://doi.org/10.1371/journal.pone.0183551 

  23. Ruano, J., Aguilar-Luque, M., Gómez-Garcia, F., Alcalde Mellado, P., Gay-Mimbrera, J., Carmona-Fernandez, P. J., et al. (2018). The differential impact of scientific quality, bibliometric factors, and social media activity on the influence of systematic reviews and meta-analyses about psoriasis. PLoS ONE, 13(1), e0191124. https://doi.org/10.1371/journal.pone.0191124 

  24. Seglen, P. O. (1997). Why the impact factor of journals should not be used for evaluating research. British Medical Journal, 314, 498–502. DOI: https://doi.org/10.1136/bmj.314.7079.497 

  25. Thelwall, M. (2017). Are mendeley reader counts useful impact indicators in all fields? Scientometrics, 113, 1721–1731. DOI: https://doi.org/10.1007/s11192-017-2557-x 

  26. Thelwall, M., Haustein, S., Larivière, V., & Sugimoto, C. R. (2013). Do altmetrics work? Twitters and ten other social web services. PLoS ONE, 8(5), e64841. DOI: https://doi.org/10.1371/journal.pone.0064841 

  27. Torres-Salinas, D., Gorraiz, J., & Robinson-Garcia, N. (2018). The insoluble problems of books: What does Altmetric.com have to offer? ASLIB Journal of Information Management, 70(6), 691–707. DOI: https://doi.org/10.1108/AJIM-06-2018-0152 

  28. Torres-Salinas, D., Robinson-Garcia, N., & Jiménez-Contreras, E. (2016). Can we use altmetrics at the institutional level? A case study analysing the coverage by research areas of four Spanish universities. arXiv preprint arXiv:1606.00232. 

  29. Wang, J., Alotaibi, N. M., Ibrahim, G. M., Kulkarni, A. V., & Lozano, A. M. (2017). The spectrum of altmetrics in neurosurgery: The top 100 “trending” articles in neurosurgical journals. World Neurosurgery, 103, 883–895. DOI: https://doi.org/10.1016/j.wneu.2017.04.157 

  30. Williams, C. (2017). Altmetrics for books: A guide for editors. Altmetric.com Blog. Retrieved on 29 April, 2019, from https://www.altmetric.com/blog/altmetrics-for-books-a-guide-for-editors/ 

  31. Wooldridge, J. M. (2015). Introductory econometrics: A modern approach. Nelson Education. 

comments powered by Disqus