Background

Traditionally, the impact of research has been measured through citations (Cronin 1984; Garfield 1964; Halevi & Moed 2015; Moed 2006, 2015; Seglen 1997). The number of citations a paper receives over time has been directly correlated to the perceived impact of the article. The more citations a paper receives, the higher the impact of the research. However, there are a few fundamental issues with this assumption and approach. First, citations only capture a fragment of the overall usefulness or the impact of a paper overall. They are selective in nature and ignore many different aspects of impact, particularly readership rates (Bazrafshan, Haghdoost & Zare 2015; Davis 2011; Davis et al. 2008; Garfield 1979; Maflahi & Thelwall 2016). This is especially true for canonized works that are no longer cited despite the fact that they are continuously studied and form the foundations of current research. Citations are also limited to the scientific community and therefore only capture the scholarly impact of research papers. Laypeople do not cite papers in traditional way; therefore, the societal impact of a research paper is completely ignored when it comes to where lay engagement with scientific content is concerned. A recent study shows that even the scientific community has varying perceptions of impact that is not necessarily citations-driven (Borchardt et al. 2018).

The Rise of Altmetrics

In the past decade, a new set of metrics aiming to measure the impact of research in diverse, typically non-traditional settings has been developed. These metrics are labelled “altmetrics” which stands for “alternative metrics” (Priem et al. 2010). The assumption with altmetrics is that research can demonstrate impact well beyond the narrow lens of citations as well as include measures of overall usage and social impact. Such metrics were designed to capture the reach of scholarly and research output beyond the scientific community. The ability to do so was mostly driven by the ability to track individual articles via identifiers such as DOIs, PubMed IDs and others by using advanced crawlers to identify these articles on publishers’ platforms once they are downloaded or clicked on or once they appear anywhere on the web and social media platforms. Over a decade later, we now see that these metrics can indivertibly capture both public perception and overall understanding of scientific reporting.

In the medical and biomedical arenas, the ability of these metrics to capture the public opinion, understanding, and perception of scientific findings becomes even more important. As can be seen and heard on the news daily, medical and biomedical research is now quickly translated into public health recommendations and is disseminated to a vast audience of non-specialists (Campion 2004; George, Rovniak & Kraschnewski 2013; Woloshin & Schwartz 2002). In addition, with the advent of open access research and availability of scientific articles over the internet, a wider audience is now able to share, comment on, and learn from articles found obtained on the web at no cost. These phenomena have their merits but also present some serious challenges. The obvious merit of open research is the ability of laypeople to inform themselves about the latest findings and breakthroughs in medical research, even and especially concerning their own health. However, the same availability and access might result in misinterpretations, incorrect conclusions, and oversimplification of results can lead to dangerous actions regarding health and treatment (George et al. 2013; Sumner et al. 2016; Taylor et al. 2015).

Sentiment Analysis and Opinion Mining

Sentiment analysis and opinion mining on social media are methods used to gauge public opinion captured via the posts shared on social media platforms in order to track positive, negative, or neutral sentiments towards a given concept. The literature shows that sentiment analysis and opinion mining on social media channels have been used in several arenas including brand management, politics, and finance and stock market predications (Abacı 2017; Oliveira, Bermejo & dos Santos 2017; Nguyen & Shirai 2015; Anjali Devi, Sapkota & Obulesh 2018; Ali et al. 2017).

Sentiment analysis and opinion mining have also been used in library and information science. These methods have been applied to study the context of citations and determine their overall sentiments (Athar & Teufel 2012; Small 2011; Xu et al. 2015; Yousif et al. 2017; Yu 2013). The main purpose of these bibliometric methods is to gauge the nature of citations in scientific papers and to be able to determine whether they are positive or negative. While the overall number of citations is an important measure of a paper’s impact, there also needs to be a way to understand whether citations are positive or negative. If most citations cite a paper in a negative way, for example as a misleading or inaccurate example of research, sentiment analysis could in theory discover the trend and alert the scientific community, the journal, and the publishing institution that the paper and its author/s are potentially having a negative impact on science. A content analysis to discover sentiments towards scientific articles on Twitter was conducted by Thelwall et al. (2013). In this study, the researchers analyzed a random sample of 270 tweets of scientific articles from various publishers. Sentiment analysis was conducted manually by three investigators. The results showed that the majority of tweets relating to scientific articles were objective, typically mentioning the title of the article or some major findings. This study was relatively small, manually analyzed, and focused on twitter feeds that were limited in length and scope. In 2015, Shema, Bar-Ilan and Thelwall conducted similar content analysis on blog mentions of scientific articles. This research analyzed 391 blog posts relating to scientific articles. Using a methodology similar to the previously mentioned article, this study found that the most prevalent sentiment was objective and posts mainly focused on discussion or explanation of the findings.

In the medical and health arenas, sentiment analysis and opinion mining methods are not used as often and are mostly applied to track public opinions on issues such as drug addiction (Raja et al. 2016) or very general public health issues (Palomino et al. 2016; Yang, Lee & Kuo 2016), all using keywords as trackers.

Sentiment analysis of social and news media of medical and biomedical research does not exist yet. Platforms that offer altmetric indicators tracking for articles do not yet offer sentiment analysis or opinion mining of the content posted by people when they discuss research articles. Misuse and misinterpretation of medical research can be potentially dangerous to the public and lead to misdiagnosis, misuse of medications, and overall confusion about the results of medical studies. Therefore, there is a need to create a systematic process to track public and news media’s understanding, use, and sharing of medical research to ensure accuracy and proper use of such studies. Such a platform will also enable scientists to communicate their findings effectively by understanding how the public conceives their research. Finally, such a platform will enable community health workers and institutions to be aware of any misconceptions of the public regarding medical scientific research and enable them to address such misconceptions with the communities with which they work.

An Initiative to Track Sentiment in Altmetrics

In the past three years, the Levy Library at The Icahn School of Medicine at Mount Sinai has been working closely with the scientists and faculty members of the Icahn School of Medicine at Mount Sinai and the Mount Sinai health system to build a comprehensive platform to track the research produced by the institution, which collects a wide variety of metrics related to it. The system was customized to the institution and implemented by the Mount Sinai Health System (MSHS) librarians with high attention to accuracy.

The PlumX platform allowed the librarians to implement individual profiles for 5,200 researchers and clinicians at Mount Sinai by using individual Scopus and ORCID IDs as well as ensure complete coverage of their output by adding DOIs, PubMed IDs and PMCIDs to any additional research output that was missing from either Scopus or ORCID. Each article and artifact is individually tracked for citations but is also measured by metrics related to the social impact of research. These include overall usage (downloads, for example) as well as mentions by social media, news media, blogs, etc.

At present, the MSHS PlumX platform hosts a comprehensive dataset that includes articles, book chapters, conference proceedings, and other forms of scholarly output produced by the faculty. The platform also aggregates various metrics including citations, usage, and altmetric indicators such as public mentions on social media platforms such as Facebook, Twitter, LinkedIn, etc., as well as discussions, blogs, Wikipedia, news media coverage, and more. However, the system does not include the ability to analyze the content on a textual level to examine sentiments and opinions. Unless each individual article is examined for their metrics and the actual text that appears in each mention or discussion, there is no way to discover who is participating in a specific discussion or to determine what exactly is being discussed. There are “analytics” embedded into the current system that are limited to the overall number of citations, usage, mentions, etc., and at this stage these analytics do not offer the means to make any topical or textual analysis.

Example

So that we might demonstrate the benefits of developing sentiment analysis of altmetric data, we manually collected the textual content of social media and news media articles selected from PlumX. The following example used freely available tools for automatic sentiment analysis that demonstrate the potential of such systems. That said, these freely available solutions cannot support articles’ analytics beyond the identification of simple keywords.

In our example we selected a recent article co-written by Icahn School of Medicine at Mount Sinai researcher Alison M. Groate, which reported the results of a meta-analysis that proved that there is no genetic influence on stress and depression. This meta-analysis disproved a decade-long assumption that some people are prone to depression and stress because of their genetic makeup. The article was widely discussed by the news and social media since it disputed decades of research (see Figure 1).

Figure 1 

Article dashboard tracking social and news media metrics for an article. https://plu.mx/mtsinai/a/0VVKxOFYIG_FTKzHV43FQJ-W_fzhZh2QnQiUpoTYY0Q/?display-tab=summary-content.

According to PlumX, this article received over 1200 social media mentions. These include 900 tweets and retweets, 338 shares on Facebook and coverage from 10 news media articles (see Figure 2). The sheer number of social media mentions is too high to be read individually, and there is no obvious way to understand whether the comments are positive, negative, or neutral. The only manner by which one can understand the overall sentiments of the participants. whether positive, negative, or otherwise, is to read the actual tweets, news articles, etc. Since the article disputes decades of research that tied depression and stress to genetic makeup, it would have been useful and enlightening to understand the overall sentiments expressed in the social and news media coverage.

Figure 2 

Examples of social media and news media mentions on article dashboard.

In order to demonstrate the potential of our proposed sentiment analysis, we manually copied and pasted the Twitter comments into a free sentiment analysis platform (“Python NLTK Sentiment Analysis with Text Classification Demo” n.d.) (see Figure 3).

Figure 3 

Sentiment analysis of the Twitter comments (manually copied and pasted).

As can be seen from the above label, the system coded the comments as a negative sentiment.

We found the same negative sentiment when we copied and pasted the news media articles into this system (see Figure 4).

Figure 4 

Sentiment analysis of news media (manually copied and pasted).

However, when we plugged some keywords related to the topic, i.e. “5-HTLPR genotype depression” in a sentiment analysis web-based tool “SocialMention” (“Real Time Search – Social Mention” n.d.) it became clear that the negative sentiment was actually directed towards the previously held scientific stance that genetics are a major cause of stress and depression, while the sentiment towards the article itself was positive (see Figure 5).

Figure 5 

Sentiment analysis of the topic.

As can be seen, there is a discrepancy between the sentiments expressed in comments related to the article itself and the overall perception of the topic. It is difficult to know precisely the issues and concerns raised by the participants and journalists unless we build a system that can analyze these sentiments in an accurate manner and also distinguish between scientific and laypeople’s comments on the topic. We feel that developing this system will enable scientists, laypeople, and public health workers to understand the public sentiments towards a topic and address any misconceptions or misinterpretations.

Discussion

Sentiment analysis could be a promising new way to gain insight into the societal impact of scientific output. Altmetric indicators, with their ability to include social and news media mentions of scientific output, have enabled us to see the nature of comments and discussions of scientific articles. However, since there are many articles that gain massive attention from both social and news media, reading all the news articles or the entire comments section, let alone organizing that information in an insightful way, is an unrealistic methodology. Therefore, a sentiment analysis tool that can track the overall “mood” of the conversation could be beneficial for anyone looking for information regarding a new study or research findings.

Challenges

Sentiment analysis tools for publications does not exist today. Such a system will have to be developed specifically for such a purpose. To do this requires both financial and scientific investments.

Scientific investment in such development will have to examine the validity of these indicators. For example, sentiment analysis for publications might prove to be more complicated than analysis of brands or individual people. The process relies on recurring words that capture positive or negative sentiments. However, the social and news media discussion of an article might include negative or positive sentiments that are directed more at the topic rather than the article itself. In our example, the negative sentiment tracked for the article targets the topic rather than the article. In the various discussions around the article, participants criticize the fact that for decades, genetics was directly linked to depression and stress. When reading the comments and articles, it becomes clear that the negativity is directed towards past practices and not the current discovery. This presents a significant challenge for future sentiment analysis, mainly the challenge of enabling a machine to decide whether the negative or positive sentiment is directly related to the content or findings of the article.

An additional challenge lies in language processing. While English might be the prevalent language of scientific publications, it is not necessarily the language used in social and news media around the world. Conversations on scientific findings take place on a global scale. Therefore, any sentiment analysis of such data will have to tackle data processing of various languages, and while technical solutions might exist to accomplish this, there is also the issue of interpretation. Especially with shares and comments on social media, people might use local dialects, specific words that express sentiments in a region or country that might be interpreted differently even by someone who speaks that same language.

We must also consider distinguishing between professionals and laypeople as they share their opinions on social media. While analyzing sentiments towards scientific articles, there is merit in understanding who is participating in the conversation. Scientists might have different opinions than laypeople towards the same content. It will be important to demonstrate similarities and differences between professional and lay opinions in order for sentiment indicators to be useful for the public. Such a capability will also be able to provide an indication of a gap, if it exists, in understanding or interpreting scientific papers.

Benefits to the public

Despite the challenges listed above, we believe that sentiment and opinion mining of scientific articles has potential to benefit the public. These methods have been used commercially and politically for several years with the purpose of increasing sales and predicting voters’ actions. However, these methods were never used to analyze research-related content. We believe that providing the public with the open system we describe can enable people to see the overall opinions of others regarding scientific developments, and this would result in a better-informed public. We also believe that sentiment analysis would assist in educating readers as well as cautioning them when necessary. If, for example, pervasive negative sentiment was found in social and news media towards a new drug or medical device, it would be beneficial for the public to be aware of it. In the same way, public health workers could use the system to better understand the impact of research findings or recommendations on the public and to gain insight into how the public perceives certain research discoveries. In turn, these professionals respond accordingly, perhaps with new educational initiatives. Finally, this proposed dashboard could also be available in public libraries, where librarians can facilitate scientific discussion among patrons.