Cross-metric compatability and inconsistencies of altmetrics
et al., Scientometrics, doi:10.1007/s11192-018-2674-1, Feb 2018
Analysis of altmetric providers PlumX and Altmetric.
Meschede et al., 17 Feb 2018, peer-reviewed, 2 authors.
Contact: christine.meschede@hhu.de.
Abstract: ## Cross-metric compatability and inconsistencies of altmetrics
Christine Meschede 1 • Tobias Siebenlist 1
Received: 26 July 2017 / Published online: 17 February 2018 Ó Akade ´miai Kiado ´, Budapest, Hungary 2018
Abstract Metrics like the number of tweets or Mendeley readers are currently discussed as an alternative to evaluate research. These alternative metrics (altmetrics) still need to be evaluated in order to fully understand their meaning, their benefits and limitations. While several preceding studies concentrate on correlations of altmetrics with classical measures like citations, this study aims at investigating metric-compatibility within altmetrics. For this purpose, 5000 journal articles from six disciplines have been analyzed regarding their metrics with the help of the aggregators PlumX and Altmetric.com. For this set, the highest numbers of events have been recognized regarding Mendeley readers, followed by Twitter and Facebook mentions. Thereby variations considering the aggregators could be observed. Intra-correlations between the metrics across one aggregator have been calculated, as well as inter-correlations for the corresponding metrics across the aggregators. For both aggregators, low to medium intra-correlations could be calculated which shows the diversity of the different metrics. Regarding inter-correlations, PlumX and Altmetric.com are highly consistent concerning Mendeley readers ( r ¼ 0 : 97) and Wikipedia mentions ( r ¼ 0 : 82), whereas the consistency concerning Twitter ( r ¼ 0 : 49), blogs ( r ¼ 0 : 46) and Reddit ( r ¼ 0 : 41) is on a moderate level. The sources Facebook ( r ¼ 0 : 29), Google ? ( r ¼ 0 : 28) and News ( r ¼ 0 : 11) show only low correlations.
Keywords Altmetrics Aggregators Correlations Cross-metric compatibility Inconsistencies Mendeley
& Christine Meschede christine.meschede@hhu.de
1 Department of Information Science, Heinrich Heine University Du ¨sseldorf, Du ¨sseldorf, Germany
DOI record:
{
"DOI": "10.1007/s11192-018-2674-1",
"ISSN": [
"0138-9130",
"1588-2861"
],
"URL": "http://dx.doi.org/10.1007/s11192-018-2674-1",
"alternative-id": [
"2674"
],
"assertion": [
{
"group": {
"label": "Article History",
"name": "ArticleHistory"
},
"label": "Received",
"name": "received",
"order": 1,
"value": "26 July 2017"
},
{
"group": {
"label": "Article History",
"name": "ArticleHistory"
},
"label": "First Online",
"name": "first_online",
"order": 2,
"value": "17 February 2018"
}
],
"author": [
{
"ORCID": "https://orcid.org/0000-0002-1282-2517",
"affiliation": [],
"authenticated-orcid": false,
"family": "Meschede",
"given": "Christine",
"role": [
{
"role": "author",
"vocabulary": "crossref"
}
],
"sequence": "first"
},
{
"affiliation": [],
"family": "Siebenlist",
"given": "Tobias",
"role": [
{
"role": "author",
"vocabulary": "crossref"
}
],
"sequence": "additional"
}
],
"container-title": "Scientometrics",
"container-title-short": "Scientometrics",
"content-domain": {
"crossmark-restriction": false,
"domain": [
"link.springer.com"
]
},
"created": {
"date-parts": [
[
2018,
2,
17
]
],
"date-time": "2018-02-17T03:10:27Z",
"timestamp": 1518837027000
},
"deposited": {
"date-parts": [
[
2025,
7,
1
]
],
"date-time": "2025-07-01T18:06:30Z",
"timestamp": 1751393190000
},
"indexed": {
"date-parts": [
[
2026,
8,
17
]
],
"date-time": "2026-08-17T15:49:10Z",
"timestamp": 1786981750166,
"version": "build-2736575974"
},
"is-referenced-by-count": 42,
"issue": "1",
"issued": {
"date-parts": [
[
2018,
2,
17
]
]
},
"journal-issue": {
"issue": "1",
"published-print": {
"date-parts": [
[
2018,
4
]
]
}
},
"language": "en",
"license": [
{
"URL": "http://www.springer.com/tdm",
"content-version": "tdm",
"delay-in-days": 0,
"start": {
"date-parts": [
[
2018,
2,
17
]
],
"date-time": "2018-02-17T00:00:00Z",
"timestamp": 1518825600000
}
}
],
"link": [
{
"URL": "http://link.springer.com/article/10.1007/s11192-018-2674-1/fulltext.html",
"content-type": "text/html",
"content-version": "vor",
"intended-application": "text-mining"
},
{
"URL": "http://link.springer.com/content/pdf/10.1007/s11192-018-2674-1.pdf",
"content-type": "application/pdf",
"content-version": "vor",
"intended-application": "text-mining"
},
{
"URL": "http://link.springer.com/content/pdf/10.1007/s11192-018-2674-1.pdf",
"content-type": "application/pdf",
"content-version": "vor",
"intended-application": "similarity-checking"
}
],
"member": "297",
"original-title": [],
"page": "283-297",
"prefix": "10.1007",
"published": {
"date-parts": [
[
2018,
2,
17
]
]
},
"published-online": {
"date-parts": [
[
2018,
2,
17
]
]
},
"published-print": {
"date-parts": [
[
2018,
4
]
]
},
"publisher": "Springer Science and Business Media LLC",
"reference": [
{
"key": "2674_CR1",
"unstructured": "Allen, P. (2017). World-wide news coverage in PlumX. https://plumanalytics.com/world-wide-news-coverage-plumx/ . Accessed 2 Dec 2017."
},
{
"DOI": "10.1108/EUM0000000007205",
"author": "TC Almind",
"doi-asserted-by": "publisher",
"first-page": "404",
"issue": "4",
"journal-title": "Journal of Documentation",
"key": "2674_CR2",
"unstructured": "Almind, T. C., & Ingwersen, P. (1997). Informetric analyses on the world wide web: Methodological approaches to ‘webometrics’. Journal of Documentation, 53(4), 404–426. https://doi.org/10.1108/EUM0000000007205 .",
"volume": "53",
"year": "1997"
},
{
"key": "2674_CR3",
"unstructured": "Altmetric. (2015). Sources of attention. https://www.altmetric.com/about-our-data/our-sources/ . Accessed 30 Jan 2018."
},
{
"key": "2674_CR4",
"unstructured": "Altmetric. (2016a). How is the altmetric attention score calculated? https://help.altmetric.com/support/solutions/articles/6000060969-how-is-the-altmetric-score-calculated- . Accessed 30 Jan 2018."
},
{
"key": "2674_CR5",
"unstructured": "Altmetric. (2016b). Mainstream media outlets. https://www.altmetric.com/about-our-data/our-sources/news/ . Accessed 2 Dec 2017."
},
{
"key": "2674_CR6",
"unstructured": "Altmetric. (2017). When did altmetric start tracking attention to each attention source? https://help.altmetric.com/support/solutions/articles/6000136884-when-did-altmetric-start-tracking-attention-to-each-attention-source- . Accessed 30 Jan 2018."
},
{
"DOI": "10.1007/s11192-015-1565-y",
"author": "L Bornmann",
"doi-asserted-by": "publisher",
"first-page": "1123",
"issue": "3",
"journal-title": "Scientometrics",
"key": "2674_CR7",
"unstructured": "Bornmann, L. (2015). Alternative metrics in scientometrics: A meta-analysis of research into three altmetrics. Scientometrics, 103(3), 1123–1144. https://doi.org/10.1007/s11192-015-1565-y .",
"volume": "103",
"year": "2015"
},
{
"DOI": "10.1007/s11192-016-2077-0",
"author": "M Erdt",
"doi-asserted-by": "publisher",
"first-page": "1117",
"issue": "2",
"journal-title": "Scientometrics",
"key": "2674_CR8",
"unstructured": "Erdt, M., Nagarajan, A., Sin, S.-C. J., & Theng, Y.-L. (2016). Altmetrics: An analysis of the state-of-the-art in measuring research impact on social media. Scientometrics, 109(2), 1117–1166. https://doi.org/10.1007/s11192-016-2077-0 .",
"volume": "109",
"year": "2016"
},
{
"DOI": "10.1126/science.122.3159.108",
"author": "E Garfield",
"doi-asserted-by": "publisher",
"first-page": "108",
"issue": "3159",
"journal-title": "Science",
"key": "2674_CR9",
"unstructured": "Garfield, E. (1955). Citation indexes for science. Science, 122(3159), 108–111. https://doi.org/10.1126/science.123.3185.61-a .",
"volume": "122",
"year": "1955"
},
{
"DOI": "10.1007/s11192-016-1910-9",
"author": "S Haustein",
"doi-asserted-by": "publisher",
"first-page": "413",
"issue": "1",
"journal-title": "Scientometrics",
"key": "2674_CR10",
"unstructured": "Haustein, S. (2016). Grand challenges in altmetrics: Heterogeneity, data quality and dependencies. Scientometrics, 108(1), 413–423. https://doi.org/10.1007/s11192-016-1910-9 .",
"volume": "108",
"year": "2016"
},
{
"author": "S Haustein",
"first-page": "372",
"key": "2674_CR11",
"unstructured": "Haustein, S., Bowman, T. D., & Costas, R. (2015). Interpreting “altmetrics”: Viewing acts on social media through the lens of citation and social theories. In C. R. Sugimoto (Ed.), Theories of informetrics and scholarly communication (pp. 372–405). Berlin/Boston: De Gruyter.",
"volume-title": "Theories of informetrics and scholarly communication",
"year": "2015"
},
{
"DOI": "10.1002/asi.23456",
"author": "S Haustein",
"doi-asserted-by": "publisher",
"first-page": "232",
"issue": "1",
"journal-title": "Journal of the Association for Information Science and Technology",
"key": "2674_CR12",
"unstructured": "Haustein, S., Bowman, T. D., Holmberg, K., Tsou, A., Sugimoto, C. R., & Larivière, V. (2016). Tweets as impact indicators: Examining the implications of automated “bot” accounts on Twitter. Journal of the Association for Information Science and Technology, 67(1), 232–238. https://doi.org/10.1002/asi.23456 .",
"volume": "67",
"year": "2016"
},
{
"DOI": "10.1007/s11192-013-1221-3",
"author": "S Haustein",
"doi-asserted-by": "publisher",
"first-page": "1145",
"issue": "2",
"journal-title": "Scientometrics",
"key": "2674_CR13",
"unstructured": "Haustein, S., Peters, I., Bar-Ilan, J., Priem, J., Shema, H., & Terliesner, J. (2014). Coverage and adoption of altmetrics sources in the bibliometric community. Scientometrics, 101(2), 1145–1163. https://doi.org/10.1007/s11192-013-1221-3 .",
"volume": "101",
"year": "2014"
},
{
"DOI": "10.1016/j.joi.2011.04.002",
"author": "S Haustein",
"doi-asserted-by": "publisher",
"first-page": "446",
"issue": "3",
"journal-title": "Journal of Informetrics",
"key": "2674_CR14",
"unstructured": "Haustein, S., & Siebenlist, T. (2011). Applying social bookmarking data to evaluate journal usage. Journal of Informetrics, 5(3), 446–457. https://doi.org/10.1016/j.joi.2011.04.002 .",
"volume": "5",
"year": "2011"
},
{
"DOI": "10.1038/520429a",
"author": "D Hicks",
"doi-asserted-by": "publisher",
"first-page": "429",
"issue": "7548",
"journal-title": "Nature",
"key": "2674_CR15",
"unstructured": "Hicks, D., Wouters, P., Waltman, L., de Rijcke, S., & Rafols, I. (2015). Bibliometrics: The Leiden Manifesto for research metrics. Nature, 520(7548), 429–431. https://doi.org/10.1038/520429a .",
"volume": "520",
"year": "2015"
},
{
"DOI": "10.1073/pnas.0507655102",
"author": "JE Hirsch",
"doi-asserted-by": "publisher",
"first-page": "16569",
"issue": "46",
"journal-title": "Proceedings of the National Academy of Sciences of the United States of America",
"key": "2674_CR16",
"unstructured": "Hirsch, J. E. (2005). An index to quantify an individual’s scientific research output. Proceedings of the National Academy of Sciences of the United States of America, 102(46), 16569–16572.",
"volume": "102",
"year": "2005"
},
{
"DOI": "10.1007/s11192-014-1229-3",
"author": "K Holmberg",
"doi-asserted-by": "publisher",
"first-page": "1027",
"issue": "2",
"journal-title": "Scientometrics",
"key": "2674_CR17",
"unstructured": "Holmberg, K., & Thelwall, M. (2014). Disciplinary differences in Twitter scholarly communication. Scientometrics, 101(2), 1027–1042. https://doi.org/10.1007/s11192-014-1229-3 .",
"volume": "101",
"year": "2014"
},
{
"author": "A Jobmann",
"first-page": "1",
"issue": "1",
"journal-title": "Bibliometrie-Praxis und Forschung",
"key": "2674_CR18",
"unstructured": "Jobmann, A., Hoffmann, C. P., Künne, S., Peters, I., Schmitz, J., & Wollnik-Korn, G. (2014). Altmetrics for large, multidisciplinary research groups: Comparison of current tools. Bibliometrie-Praxis und Forschung, 3(1), 1–19.",
"volume": "3",
"year": "2014"
},
{
"key": "2674_CR19",
"unstructured": "Lohr, C. (2017). New in captures: CiteULike readership. https://plumanalytics.com/new-in-captures-citeulike-readership/ . Accessed 12 Dec 2017."
},
{
"DOI": "10.13140/RG.2.1.4929.1363",
"doi-asserted-by": "publisher",
"key": "2674_CR20",
"unstructured": "Martin, J. (2015). Responsible metrics. In J. Wilsdon (Ed.), The metric tide. Report of the independent review of the role of metrics in research assessment and management (pp. 134–147). HEFCE. https://doi.org/10.13140/RG.2.1.4929.1363 ."
},
{
"DOI": "10.1086/354848",
"author": "RK Merton",
"doi-asserted-by": "publisher",
"first-page": "606",
"issue": "4",
"journal-title": "ISIS",
"key": "2674_CR21",
"unstructured": "Merton, R. K. (1988). The Matthew effect in science, II: Cumulative advantage and the symbolism of intellectual property. ISIS, 79(4), 606–623. https://doi.org/10.1086/354848 .",
"volume": "79",
"year": "1988"
},
{
"DOI": "10.1007/s00005-009-0001-5",
"author": "HF Moed",
"doi-asserted-by": "publisher",
"first-page": "13",
"issue": "1",
"journal-title": "Archivum Immunologiae et Therapiae Experimentalis",
"key": "2674_CR22",
"unstructured": "Moed, H. F. (2009). New developments in the use of citation analysis in research evaluation. Archivum Immunologiae et Therapiae Experimentalis, 57(1), 13–18. https://doi.org/10.1007/s00005-009-0001-5 .",
"volume": "57",
"year": "2009"
},
{
"DOI": "10.1007/s11192-017-2559-8",
"author": "B Mukherjee",
"doi-asserted-by": "publisher",
"first-page": "253",
"issue": "1",
"journal-title": "Scientometrics",
"key": "2674_CR23",
"unstructured": "Mukherjee, B., Subotić, S., & Chaubey, A. K. (2018). And now for something completely different: The congruence of the altmetric attention score’s structure between different article groups. Scientometrics, 114(1), 253–275. https://doi.org/10.1007/s11192-017-2559-8 .",
"volume": "114",
"year": "2018"
},
{
"DOI": "10.3233/ISU-2011-0623",
"author": "D Nicholas",
"doi-asserted-by": "publisher",
"first-page": "61",
"issue": "1–2",
"journal-title": "Information Services and Use",
"key": "2674_CR24",
"unstructured": "Nicholas, D., & Rowlands, I. (2011). Social media use in the research workflow. Information Services and Use, 31(1–2), 61–83. https://doi.org/10.3233/ISU-2011-0623 .",
"volume": "31",
"year": "2011"
},
{
"key": "2674_CR25",
"unstructured": "Plum Analytics. (2016). Mention metrics. https://plumanalytics.com/learn/about-metrics/mention-metrics/ . Accessed 2 Dec 2017."
},
{
"key": "2674_CR26",
"unstructured": "Plum Analytics. (2017). Leadership. http://plumanalytics.com/about/leadership/ . Accessed 30 Jan 2018."
},
{
"key": "2674_CR27",
"unstructured": "Priem, J. (2010). I like the term #articlelevelmetrics, but it fails to imply *diversity* of measures. Lately, I’m liking #altmetrics. https://twitter.com/jasonpriem/status/25844968813 . Accessed 30 Jan 2018."
},
{
"key": "2674_CR28",
"unstructured": "Priem, J., Taraborelli, D., Groth, P., & Neylon, C. (2010). Altmetrics: A manifesto. http://altmetrics.org/manifesto . Accessed 30 Jan 2018."
},
{
"DOI": "10.1371/journal.pone.0035869",
"author": "H Shema",
"doi-asserted-by": "publisher",
"first-page": "1",
"issue": "5",
"journal-title": "PLoS ONE",
"key": "2674_CR29",
"unstructured": "Shema, H., Bar-Ilan, J., & Thelwall, M. (2012). Research blogs and the discussion of scholarly information. PLoS ONE, 7(5), 1–8. https://doi.org/10.1371/journal.pone.0035869 .",
"volume": "7",
"year": "2012"
},
{
"DOI": "10.2307/1412159",
"author": "C Spearman",
"doi-asserted-by": "publisher",
"first-page": "72",
"issue": "1",
"journal-title": "The American Journal of Psychology",
"key": "2674_CR30",
"unstructured": "Spearman, C. (1904). The proof and measurement of association between two things. The American Journal of Psychology, 15(1), 72–101. https://doi.org/10.2307/1412159 .",
"volume": "15",
"year": "1904"
},
{
"DOI": "10.1007/s11192-013-1117-2",
"author": "P Sud",
"doi-asserted-by": "publisher",
"first-page": "1131",
"issue": "2",
"journal-title": "Scientometrics",
"key": "2674_CR31",
"unstructured": "Sud, P., & Thelwall, M. (2014). Evaluating altmetrics. Scientometrics, 98(2), 1131–1143. https://doi.org/10.1007/s11192-013-1117-2 .",
"volume": "98",
"year": "2014"
},
{
"DOI": "10.1515/9783110308464-019",
"author": "M Thelwall",
"doi-asserted-by": "crossref",
"first-page": "337",
"key": "2674_CR32",
"unstructured": "Thelwall, M. (2016). Webometrics and altmetrics: Home birth vs. hospital birth. In C. R. Sugimoto (Ed.), Theories of informetrics and scholarly communication (pp. 337–346). Berlin/Boston: De Gruyter.",
"volume-title": "Theories of informetrics and scholarly communication",
"year": "2016"
},
{
"key": "2674_CR33",
"unstructured": "Torres-Salinas, D., Robinson-Garcia, N., & Jiménez-Contreras, E. (2016). Can we use altmetrics at the institutional level? A case study analysing the coverage by research areas of four Spanish Universities. In Proceedings of the 21st international conference on science and technology indicators, València, Spain. http://dx.doi.org/10.4995/STI2016.2016.xxxx ."
},
{
"DOI": "10.1007/978-3-319-09785-5_16",
"doi-asserted-by": "publisher",
"key": "2674_CR34",
"unstructured": "Weller, K. (2015). Social media and altmetrics: An overview of current alternative approaches to measuring scholarly impact. In I. M. Welpe, J. Wollersheim, S. Ringelhan & M. Osterloh (Eds.), Incentives and performance: governance of research organizations (pp. 261–276). Cham, Switzerland: Springer International Publishing. https://doi.org/10.1007/978-3-319-09785-5_16 ."
},
{
"DOI": "10.1007/s11192-014-1264-0",
"author": "Z Zahedi",
"doi-asserted-by": "publisher",
"first-page": "1491",
"issue": "2",
"journal-title": "Scientometrics",
"key": "2674_CR35",
"unstructured": "Zahedi, Z., Costas, R., & Wouters, P. (2014). How well developed are altmetrics? A cross-disciplinary analysis of the presence of ‘alternative metrics’ in scientific publications. Scientometrics, 101(2), 1491–1513. https://doi.org/10.1007/s11192-014-1264-0 .",
"volume": "101",
"year": "2014"
},
{
"DOI": "10.6084/m9.figshare.1041821",
"doi-asserted-by": "publisher",
"key": "2674_CR36",
"unstructured": "Zahedi, Z., Fenner, M., & Costas, R. (2014). How consistent are altmetrics providers? Study of 1000 PLOS ONE publications using the PLOS ALM, Mendeley and Altmetric. com APIs. altmetrics 14. In Workshop at the Web Science Conference Bloomington, USA, pp. 4–6. https://doi.org/10.6084/m9.figshare.1041821 ."
}
],
"reference-count": 36,
"references-count": 36,
"relation": {},
"resource": {
"primary": {
"URL": "http://link.springer.com/10.1007/s11192-018-2674-1"
}
},
"score": 1,
"short-title": [],
"source": "Crossref",
"subject": [],
"subtitle": [],
"title": "Cross-metric compatability and inconsistencies of altmetrics",
"type": "journal-article",
"update-policy": "https://doi.org/10.1007/springer_crossmark_policy",
"volume": "115"
}