ACRL STS Hot Topics Discussion

act what all stem librarians need to know this is the first of what I hope to be many webinars from the hot topics discussion group and we’re saying stem librarians on campuses across the country are at the forefront of much of the debate around research metrics and impact assessment but practical information on how these metrics are being implemented into library services can be surprisingly hard to come by we hope to remedy that for you we have three speakers today and I’m going to do all the bios now then we will have three three presentations and then questions at the end and I’m going to ask you if you have a question during the course of the speaker’s presentations if you’d want to start a private chat with me and just shoot me the question I’ll add it to my list of questions and then you can we’ll do those all at the end all right so Daniella Solomon received a BS in electrical engineering from Yassi Technical University in Romania and an MS and library science and information from the University of Arizona she joined Kelvin Smith library in 2012 as a research services liaison to biomedical engineering civil engineering Electrical Engineering and computer science material science and engineering and mechanical and aerospace engineering departments her passion is to connect campus community with the best engineering resources available at Kelvin Smith library Daniela is an active member of American Society of engineering education engineering libraries division she’s interested in scholarship metrics open access and instruction our second speaker rachael Borchardt is a associate librarian at american university and works as a library liaison to the science departments she holds an MLIS and medical librarianship from the university of pittsburgh as well as dual ba degrees in psychology and neuroscience from Oberlin College her professional research focuses on the intersection of metrics and libraries and she has co-authored several on the topic and a recent book publication titled meaningful metrics a 21st century librarians guide to bibliometrics altmetrics and research impact published by the American Library Association and available from the ala store and Amazon our third speaker Stacey conceal is the outreach and engagement manager altmetric a data science company that uncovers the attention that research receives online she studies incentive systems in academia info metrics and has written and presented widely about altmetrics open science and library services previously Stacey worked with teams that impact story Indiana University and applause all right our first speaker is Daniella Daniella still setting up her microphone from the looks of it she’s having some audio difficulties hello can you hear me yes hello can you hear me now hear you yeah okay thank you okay thank you so much I’m sorry for the trouble I’ve been running up the stairs to fix it so hi everybody thank you for participating so my presentation is going to be academic impact metrics perceptions me in fact but since the main topic of this webinar is research intact I think it is important to start by clarifying what is understood by impact nowadays until recent years the common understanding of impact was academic impact however a more comprehensive definition that reflects the current research environment is the definition developed by the Research Council in UK the monsterbowl contribution that excellent research makes to society and the economy so what do you think that there are two sets of keywords within the definition that I would like to bring to your attention the first set is society and economy the emphasis on research influence on society and economy beyond the academic merit is the result of the need to provide evidence of benefits

from funds invested in research and there are various there are various ways to account for such influence from policy making technical innovations and generating wealth through solving environmental issues the second set is contribution and excellence and I would like to bring in discussion what constitutes as contribution and excellence and houses can be demonstrated contribution can be demonstrated through both quantitative and qualitative means for the academic impact quantitative assessment could include numbers such as publication number citations usage grants rate of acceptance etc for the society in fact it could be the number of people impacted by changes in policies or procedures or for the impact on economy it could be the numbers demonstrating contribution to economic growth through technological innovations demonstrating contribution by quality qualitative means is a little more challenging to do but it is equally important with the quantitative part the encounter issues are caused mainly by the difficulties in collecting this kind of information and I think Stacy and Rachel will have more to say about this part later in this webinar so what is excellent research defining what excellent research is posed to be more challenging than defining contribution the criteria defining excellent research are worth the topic but how does one define a worthy topic next is rich regal but how process which figure when there has been observed an increase in number of retracted articles and sincerity how to assess sincerity when there are tragedies meant published take results it is therefore important that all these elements are taken in consideration it is also necessary to use a wide variety of indicators to account for impact in order to insure a comprehensive assessment of the quality of research and its broader impact since it has become important to demonstrate impact to funding agencies as I think the research impact is in the interest of researchers and institutions and the obvious place to start is tracking the impact of research outwards or the academic impact current practices for assessing academic impact are limited quantitative metrics Demong the most common metrics in use are number of publications citation counts journal impact factor and h-index this metrics have created strong controversies related to what is really being measured immaturity of metrics and tools and limitation of academic freedom and creativity as well as the effect of publishing culture when considering this matrix it is important to consider what exactly is being measured and the strength of the evidence it provides for example number of publications although we live in a publish or perish academic environment the current scholarly communication system makes the publishing process difficult without necessarily ensuring the desired level of quality as such number of publication is a Productivity measure and does not demonstrate quality of research additionally this metric did not keep up with the recent changes in scholarly communication and has been slow to include new forms of scholarships for example datasets or multi medium petition counts citation counts are used to measure the influence of a paper on a subsequent literature and it is based on the assumption that citations represent the positive recognition of such influence however as it has been proven many times there are various reason for citing like disagreement or coercion besides the influence issue citation counts available are incomplete or they depend on the venue of indexing as well as there is a large number of errors citations take a long time to accumulate and think there are large variations in citation behavior among different disciplines citation counts are not useful for comparison across fields other characteristics of citation count magic our sensitivity to popular trends in

science and under us under estimation of the contribution of applied scientists journal impact factor is a metric derived from citation counts that tentatively measure the impact for journals published annually by Thomson Reuters as journal citation reports the impact factor is the average number of citations received by the papers published by a journal in the preceding two years when calculating impact factor all citations are given equal weight and there is no differentiation based on the citing journal prestige co-authorship or type of publication this type is prevalent the impact factor has a number of serious problems such as a short time period is considered when cup for calculating when calculating our impact factor unclear methodology or the fact that impact factor is web of science data bond and citations from elsewhere are not counted more critically is it improper use as a metric for individual researchers when it is clear that it does not reflect the value of individual papers and finally h-index well that is another citation based magic agent a attempts to measure both the productivity and quality of the published work of a scholar and is readily available in some sources or can be easily calculated what is important to understand about H index is that it represents a single measure and cannot be used in isolation among the recognized agent s challenges is the fact that it cannot be used to compare researchers at different stages in their careers nor can be used to compare scientists from different disciplines it is also difficult to determine it for scientists and enter the interdisciplinary subjects regardless of whether they are good impact factors or not age index and impact factors have become the golden standards in assessing the in fact the results of a survey published in Nature news in 2010 showed that this metrics are used at many institutions for tenure and promotion and that there is a strong disagreement toward their use as tools for evaluation one of the reasons is that metrics are uncomfortable topic for scholars well intentioned not always well-informed often ill applied it is also believed that metrics distort scientific priorities and undermine research integrity by encouraging quantity and not quality publishing topics of popular interest or called inclination to fabricate results and also the belief is not everything that counts can be counted and not everything that can be counted counts in conclusion magics constitutes a proxy measure for academic impact each metric measures only one aspect of the impact and all our venue dependent metrics are more difficult to gather for interdisciplinary subject not useful in comparing scholars and prone to gaining and as we all know the metrics are toe to toe STEM disciplines so what do librarians need to know it is important to have knowledge of what each metric measures there are issues and challenges as well as the tools to be used even more importantly is to have knowledge on how to maximize the impact as measured by these metrics because this is what faculty and administrators are most interested in awareness of the variety of metrics available in order to correct some of the issues of serving the traditional metrics many other matrix have have been defined such as variations of h-index correcting for the age of the Arthur or Carter ship or additional journal matrix that extent that time period used for calculations offer the means to compare across disparate fields of research or offer free alternatives to impact factor these metrics can be used stand alone or in combination with other matrix new developments books and data citation indexes as matrix all these are new ways in which we can

demonstrate impact get new data and also matrix represent a new opportunity for library services as librarians are well positioned to contribute to impact assessment efforts due to knowledge of and experience with the tools providing the magics in conclusion as no single metric can encompass all of the multidimensional complexity of impact it is important to use a wide variety of indicators that go beyond the bibliometric analysis and academic output libraries could be at the forefront of assessment efforts by developing expertise in offering new services as needed we just have to be open to learning new skills and explore new horizons and with this thank you for your attention so I think I can take over from here for the second part so I’m gonna kind of take a takeover where Daniela left off in shifting from more traditional bibliometrics and starting to talk about altmetrics with particularly emphasis on how libraries and stem and stem libraries are starting to talk about and implement these metrics so a quick overview is going to be first Stacey’s going to talk more in depth about altmetrics themselves but then we’ll look at some ways that libraries are supporting metrics and some recent developments that are happening within the altmetrics community so first a brief introduction I think probably a lot of today’s participants have heard or are relatively familiar with altmetrics but for those of you who it’s new the more or less official definition for right now is the creation and study of new metrics based on the social web for analyzing and informing scholarship my personal definition that I prefer is anything that’s really non citation based metrics which I think kind of creates a broader category and is more easily understood as separate from bibliometrics you can see here all metrics are drawn from a wide variety of sources so this is really just a small sample of different places where we can gather online metrics that really tell us something about how scholarship and scholarly works are being shared and disseminated and discussed as they move through an online environment and so as they come from a bunch of different sources gathering all of these metrics is also in some ways rather tricky because they live in so many places and this is something that I always stress to my faculty because it’s not always a completely intuitive journal article usage is probably one of the trickier ones and that some publishers currently provide usage metrics for example web of science actually just started within the past month to put usage counts within the web of science database so you can see the the first image here with The Times cited in the last 180 days those are actually live usage counts that are coming from web of science that are measuring every time someone downloads a publication within that database alternately publishers like PLAs and Taylor & Francis and Elsevier are also making their own usage data available separately of databases like web of science when a paper is shared on a scholarly network like Mendeley or ResearchGate we can also track the views and downloads that result within that environment so down at the bottom the five publications 111 views that is statistics that are coming from ResearchGate additionally the individual oh I don’t know if anyone else can see my okay sorry something happen we can also get metrics directly from many individual sites so in the bottom right corner is a demonstration of the number of views of my SlideShare presentations over time and finally we do have tools that harvest a whole bunch of metrics and make them together in one tool one example of being altmetric and in the upper right corner is an example of displaying different types of metrics that are available for one journal article its Stacy from altmetric we’ll

be talking a bit more about that so now that we’ve gathered all metrics what are some of the ways in which researchers and scholars are starting to use these altmetrics these definitions are actually hot off the presses I’ll be talking a little bit more about where these are coming from a bit later in the talk but to kind of gather and put them into three broad categories of use for altmetrics as we’re seeing them now the first thing probably most obvious is kind of as a natural complement to bibliometrics as a way to assess research with that in mind I think one of the key messages when it comes to all metrics is that what we’re looking at is not necessarily impact itself and impact is a difficult word to define and really as Danielle alluded to really track down in terms of quantitative and qualitative evaluations and I think what we’re seeing happen when we discuss all metrics is more of the online orient attention or engagement that publications and other scholarly works are receiving that said there is some discussion within the altmetrics community and some publications that are trying to look at are there certain altmetrics that better predict later citation counts and I’ll talk about about that a bit more later as well the second major category would be to showcase your achievements so not just assessing but saying look at this article look how well it did and those are used in situations like tenure and promotion when applying for grants or at the end of a grant when you’re to demonstrate the impact that your research had based on the grant this is a good time to mention that there is increasing calls for being able to demonstrate your impact including the so called broader impact demonstration that’s one that NIH specifically asked for which tends to align really well without metrics because we find that altmetrics sometimes are doing a much better job of measuring that broader impact than existing metrics do finally the third category about metrics that people think about wes is the discovery element but of actually using alt metrics to find research find collaborators and to decide where to publish do I want to publish with the journal that is making those metrics and usage statistics available to me do I want one that’s more actively involved in the altmetrics community that sort of discussion but again with the web of science you should counts as an example we can actually sort within web of science by the usage in the past six months to see what articles are people actively downloading and looking at now or within the last six months even if they’re citation counts may be low so it’s kind of a new way to discover research that we haven’t had before so moving on to some types of library support so I’ll talk about five basic types of ways that we’re seeing library supporting metrics and altmetrics starting with the good ol research guide there are many examples out there if you just do a search for site edu and altmetrics or research impact or metrics you’ll find many many good examples but a good one has some basic information maybe walks you through step by step how to sign up or start using some of these tools unc health sciences is the one that’s displayed here and i think it’s a good example for that reason what I see as a good future direction for these guides is to start customizing for your specific disciplinary audience so this is for the Health Sciences but we could make one specific for say biological sciences or for biology that customizes the tools that make most sense for that discipline next we have workshops and I personally of the types of ways that we can support this is the one that I’ve kind of found the most traction with you think research guides a very passive level of support and that it relies someone to physically go to that guide whereas a workshop is kind of a an active participation of whoever is there during that workshop so some tips that I have in along the right-hand side we can kind of see how my own presentations have evolved in the past three years so these are some of my takeaways based on workshops that I’ve done here at American University that targeting the information to the relevant audience will have a much greater impact and that includes using the terminology that’s really going to speak to them so you’ll notice in fall 2012 I called it bibliometrics an impact factor and this past fall I’ve kind of changed it to focusing on your research impact and I think that’s really spoken to faculty another big tip that I have

is that a workshop just focusing on metrics may not be as interesting as one that covers some wider issues and really contextualizes metrics so in my the presentation that I did in fall 2015 I was invited to speak during the orientation for new tenure-track faculty so I knew that you know where they are in their research process and what information is going to be most important to them at that point so we talked about maximizing research impact in three stages which are listed along the left-hand side strategically choosing a good publication venue using things like impact factor but also those other considerations that we talked about the discovery layer of altmetrics maximizing your exposure and this can include things like scholarly networks open access and some of those other scholarly communication issues and then finally measuring impact and engagements and that’s when we start to talk more about both traditional tricks and altmetrics that are available mentioning tools like impact story that they can use to gather all those metrics together so if you want to see the slides that I gave during that presentation the tiny URL is up on the screen but my recommendation for the future is to continue packaging and even offering a menu of topics for possible presentations to specific groups of people one of the nice things about metrics is that it does pair really well with some of these other larger issues in academia including open access institution repositories scholarly communication etc etc so the more that you can make it more appealing to those individual faculty members I think the better success these types of workshops will have just a side note as you’re starting to think about delivering presentations there is a wonderful online toolkit that actually has some pre packaged content for delivering presentations and if you see in the screenshot of their homepage they actually have already tailored materials for geography and computer science and I’m hoping that more disciplines will be added in the future because that’s another great way to take a more general topic and really narrow it to a science specific focus next consultations this is probably something that many of us are already doing but it’s kind of the most basic support we can provide is really sitting one-on-one with a faculty member with some sort of metric need or interest and talking them through their research and what tools would be best for them so I put up this handout because I think it’s a really does a good job of outlying all of the services that their library provides and what exactly we can help with and what we can provide for them but again in the customizing theme I could easily see having handouts that are customized for each discipline or sub discipline something like that so that they’re really aware of the tools that are available that are specific to their department or discipline and symposia are another fairly popular at gaining popularity a way of supporting we’ve kind of brought our conversations through the library a lot of them bring outside speakers to talk about topics that are related to metrics and often that comes under the broader theme of scholarly communications so I know at AU we’ve had a series for a little over a year Georgetown also has one this is Justin in the DC area Catholic University also has a scholarly communication symposia but I do know that some libraries instead of bringing outside speakers will actually pull together faculty from their University to give talks which I think in some ways has a different impact in that I think faculty respond to other faculty very well so it’s something I’m interested in looking at the future is trying to get faculty to talk about some of these scholarly communication and metrics issues but one of the nice things about it is that we can then use these symposia as a way to highlight what the library is able to do and how that we can support these issues and finally there are institutional level altmetrics products I’m not going to talk about them in great detail but just to note plummet cannot metric both offer these products that are really designed to do that research assessment and showcase achievement goals of altmetrics at a very high level and there’s a URL if you want to see some more about University of Pittsburgh’s implementation and some of the barriers and pitfalls that they encountered along the way so finally because I know I’m running out of time what’s happening right now with altmetrics so we’ll talk a bit about the work that my so is doing some current research that’s happening and the ways

that some two specific STEM disciplines are implementing metrics so first of a nice so working groups there are currently three working groups that are meeting right now I can attest it because I’m part of the definitions in case uses group so when I made the reference to the three broad classifications of altmetrics and their uses that actually is a working document that is coming from that definitions in case uses group so it may change before these are finalized but hot off the presses as of a week ago or so but in addition to that working group we also have two other working groups that are working on the data behind out metrics as well as the methodology of how we calculate those altmetrics so what I see is kind of the end result it’s not going to solve all of our questions or answer all of our questions that we have without metrics but I think will help to improve the reliability and help to standardize some of the fuzziness if not all of it but ultimately what I hope that it will do is help increase others confidence and altmetrics and that includes people like evaluators and people like grant funders people who might be using these like researchers so look out for nice oh we should be publishing our results within I would say the next six months or so but there will be opportunities for feedback so keep your eye on this research as it happens some other current research that’s happening information scientists have a lot to say about all metrics and they’re continuously publishing on this topic and keeping up to date with that research is a really great way to start to figure out how can I translate this out metrics to my specific community so as you can see along the right-hand side are three examples of articles of taking out metrics and applying it to specific disciplines or areas of academia to try and figure out what does this look like so that we don’t have a one-size-fits-all approach some of the other research that’s being done is to try as I said to correlate different types of metrics and you can see in the lower left corner this is one of the first studies that was done by Jason priam and Heather Pugh avoir called alt metrics in the wild that was just looking at different types of metrics and what correlates with each other this particular study found that there’s about a point six percent correlation between PDF downloads and later citations so those kinds of studies are being done and it’s a good way to start to package different types of metrics that are available for different purposes so for my faculty I like to tell them for very recent articles pulling those usage metrics can be at least a decent predictor of future citations as long as they’re explaining it in a file for say tenure that it’s definitely a type of metric that we can use now that we couldn’t use before and finally a couple of examples of ways that specific disciplines are dealing with metrics this one is actually a little old but a lot of people aren’t aware of this but the American Mathematical Society in large part due to the issues that Daniela was talking about with impact factor produced a report essentially a statement in 2009 and the the link to the PDF is available if you’d like to look at it but it’s a say essentially a condemnation of impact factor for their fields saying please do not use it as a way to evaluate our research I think that this is a really really good example of ways in which specific disciplines can take charge of their own metrics to start to say this is how we want to be measured and this is how we feel we are most accurately evaluated and finally say she’s going to talk a bit more about the becker model but i just wanted to mention it as another example of how we’re starting to see this discipline specific metrics evolving so the link is available for you but they’ve essentially taken a ton of different types of activities within biomedicine and listed different indicators of how we can measure each of those types of activities for example on this page conferences data databases funding and gray literature materials so with that in mind I’m going to save questions for later and turn things over so much Rachel eloise would you mind putting up the slides for mice wonderful thank you so while those are loading my name is Stacy tank ale and the outreach and engagement manager at altmetric we are one of several companies that gather up some of the types of metrics that Daniella and

Rachel covered in their sections so I’m going to talk very briefly about what types of metrics you might be able to find depending upon different disciplines that you’re serving many of you are liaison librarians across a great variety of STEM disciplines so we’re going to talk about specific examples that might be relevant to you and your faculty of these traces that are left online these altmetrics that can be really really relevant to showcasing impacts beyond traditional impacts for your faculty so let’s go ahead and get started Rachel touched upon one of my favorite definitions of impact and that is the Becker model and this short list right here just pulls out a few key points that the Becker model makes in terms of the types the different types of impact you might have particularly in the biomedical sciences you might as a researcher make a contribution to the knowledge base we measured that with citations right we were pretty good at knowing and tracking how that sort of impact happens there are other newer types of impact that before we weren’t necessarily able to track very easily anyway things like implementation of policy or legislation changes to clinical and research practice and so on and all of these different types of impact they have associated metrics that serve as indicators for that impact so what do I mean when I say indicators very jargony right let’s break that down so citations are indicators for your contribution to the knowledge base they’re indicators for change and understanding and they’re only indicators they don’t directly correlate to a change in understanding because we’ve found the researchers cite each other’s work for many different reasons researchers they may cite you or they may cite your researchers to say dr Bronner he’s made a great change in our understanding of breast cancer they might also cite him to say dr Bronner introduced this method in this paper that he wrote in 1976 that we used for our own data analysis or collecting of stem cells or what have you they might cite dr. Bronner to say dr Bronner was wrong we and we’re gonna prove how and why so citations can merely serve as indicators and when you look at the raw counts of citations let’s say if dr. Bronner’s paper has 15 citations whereas you know some of his other earlier work only has one or two citations that is an indicator for you or an indicator for dr. Bronner that something’s happening there that’s something you want to go look and dig in and see okay this might be making an impact let’s find out how and similarly we have these other metrics that exist online now that research is Network researches web native more researchers are conducting their scholarly activities online they’re discussing other people’s scholarship online so we have a lot of traces that we can gather like we can text mine mentions that scholarship receives in policy documents or legal code we can look and see the discussions that are happening on social media and mainstream media we can text mined patents to find out if research is having an economic benefit so we’ve got a lot of different indicators that as an umbrella they fall into this group that we tend to call altmetrics they’re distinct from citations right because we’re looking like Rachel defined at the social web these things that we weren’t able to measure before using tools like web of science or Scopus in order to understand other impacts among other audience is not just a scholarly audience but among members of the public policymakers and so on so we’ve got this treasure trove of data right that now exists thanks to the ubiquity of the Internet where can you find this data how do you pull it all together there are a number of platforms all across the web wikipedia figshare Dryad Twitter without mainstream media sites you could go to each of those to find out metrics for scholarship but you can also use an aggregator a service like we provide an alt metric or like impact story provides which pull together a lot of these metrics for you into a single report so you can just go to one place to find them and so I’ll just tell you very briefly about two of the services that we offer I used to work for impact story so I hope they don’t mind if I speak for them and showcasing some of the different metrics you can get but I’ll start out without metric so we are a commercial entity we provide a lot of different services some of you may already be familiar with our Explorer which we track over 4 million articles and we look at basically all sharing and discussion of scholarship that happens across the web but that’s really an enterprise solution and what’s

relevant today is our bookmarklet so this is a browser bookmarklet that you can install if you go to altmetric comm slash bookmarklet and anytime you’re on an article on the web you can click the altmetric it button the bookmarklet and you’ll get this summary report which we see here in the upper right that gives you a brief overview of all the metrics for that particular article if you click through you see a report like this which not only provides the metrics but also provides things like data visualizations and crucially the full text or snippets anyway and links to the full text of these mentions that are happening in the mainstream media what research bloggers are saying about a piece of research it links out to citations in policy documents in Wikipedia and so on and beyond so we provided a number of services and we’d connect to a number of services I should say one of which is faculty of a thousand Prime which some of you might be familiar with for those who aren’t this is a service where experts in biomedical sciences and life sciences they’re invited to review a paper that’s already been published and review it for its relevancy to the field and also rate it in terms of quality so we see here this particular paper it’s gotten three stars it’s been recognized by experts as being exceptional work and in particular that it changes clinical practice so this would be a really good example for researchers and the biomedical sciences to include in tenure and promotion package or a grant application or even an end-of-year review if they wanted to say hey I’m making high-quality and impactful work in the biomedical sciences for those who are working in economics Public Health any field that might have relevancy to public policy we also collect mentions for public policy we scrape a number of sources including UNESCO the World Bank and you at UK government in order to find where research has been cited in public policy documents and then there’s a question of quality and impact in medicine and translational science one of my favorite examples of that is Wikipedia citations many are skeptical which I get I understand Wikipedia is relatively new right it’s only been around for 10 or 20 years now but it is one of the highest and most widely read websites on the Internet period and as it turns out a great majority of edits to biomedical Wikipedia are made by researchers and doctors and practitioners find Wikipedia to be their number one source of health care information and if they notice that there are errors in Wikipedia articles they will edit them as well so we’ve got experts in medicine and translational sciences actively contributing to and maintaining the quality of Wikipedia articles so showcasing your your research and/or your researchers your your Faculty’s research finding where they’ve been cited in Wikipedia can really be a good indicator for impact in medicine and translational science so we’ve got a number of data sources for our bookmarklet and I mentioned before that we’ve got about metric Explorer 2 which if you’d like the free library in addition you can email me after this presentation to get access to that we’re happy to provide that like I said it provides that forty thousand foot view of all articles that have been published that we we track but now let’s talk about impact stories so at altmetric we’re really good at tracking articles we started out as a service for publishers we’re excellent attracting research articles we’ve recently moved into ebooks we also do a bit of tracking for scholarly products that have do eyes and other persistent identifiers but there are a number of products that exist out there on the web that don’t have these persistent identifiers it might just live on a researchers website or a site like github or Dryad and those are the types of products that impact story is really good at tracking impacts for so impact story if you sign up for a free account so it’s a free trial anyway they’ve since moved to a subscription model in the past year so what you do is you connect your third-party services like orchid or your SlideShare account and what this does is it automatically imports all of your research products that these other services know about or that they might store and also goes out across the web and finds Associated metrics for those so it imports a bunch of stuff automatically it’s super easy to set up we see here there are a number of different types of research outputs that impact story profiles track let’s

talk about and specific in particular software products so if you wanted to showcase the influence of your work if you’re developing code in bioinformatics chem informatics any other informatics perhaps digital humanities although that’s less relevant for this group right also if you’re working in the Computer Sciences and developing code as a researcher you might put your research on github which for those who aren’t familiar with it is a collaborative coding website and when you’re on github other people can star your software so file a sift which we see here is an example of Halle BIC software that she’s uploaded to github and other people have started which is akin to if you’re familiar with Twitter if you use Twitter you might favorite a tweet on Twitter that’s kind of a hat tip to say hey I’m reading this or I want to remember this for later it’s also for those of us who are more analog it’s like underlining a really compelling passage in a book it’s your way of saying this is good where this is interesting I want to return to it later so we’ve got 44 people which is 90 it’s more than 99% of all other software products and impact story currently tracks but who is actually starring this work I think that’s in many maze ways more relevant so we’ve got researchers and coders from across the world who are really interested in phylla sift and that can may be in and of itself applying for grant funding that might show a broader impact particularly if you’ve got a very very well-known bioinformatics software developer who has shown interest in your work so that’s just one way that impact story showcases the impacts of non-article outputs they also do articles but they’re really really good at showcasing non-article outputs like software like data and so on so I just want to emphasize I’m coming to a finish here but metrics really only tell a very small part of the story anytime you see numbers it’s really just an indicator for potential impacts that might exist if you look at the qualitative data so who is saying what about research and in what context I just want to give some very quick examples of how these metrics are being used from both impact story and altmetric we’re seeing faculty from across the sciences and also the humanities use altmetrics when going up for tenure so they might supplement their CV that they submit with their tenure dossier they might put alt metrics in their dossier librarian so I want to give some specific examples of how librarians are supporting altmetrics so Rachel talked about these classes that are happening Rebecca resin excel and at UMass Medical School library is giving a lot of these classes as of late she’s also doing a lot of reports upon request for committees and department chairs and faculty and these reports are being used in many ways but what I think is really really important about the work that Rebecca is doing is that they’re very very careful to define the metrics and their sources and their limitations so she’ll include all sorts of citation metrics and also sum up metrics in these reports and she’ll be very careful to say for example Google Scholar this is a big number but it includes non peer-reviewed work in the citation count so take this with a grain of salt and so on grant applications support at Gault or Health Sciences Library at Northwestern Karin gutsman and her team they’re doing really really good work they’ve got a great online guide that gives specific examples for researchers to use metrics in their nih biosketch she’s also doing one on one researcher consultations and just yesterday we did a webinar that where she and her colleague pamela shaw they broke down exactly the sorts of services that they’re providing so I’ll share these slides after this webinar and you can go in and check out the recording there and then creating live guides Rachel also touched upon this hundreds of these guides now exist if you want to roll your own I when I was an impact story created to altmetrics that our cease are to live guides that are CC by license so you’re free to use them with attribution and a guide for researchers and a guide for librarians so that’s it for me I think we can now turn it over to questions all right thank you very much daniela rachel and stacy also thank you to all attendees at the moment it’s great to see so many people show up we actually seem to have broken ala connect and people were turned away at the door so i’m really gratified to see there’s so much interest in the topic and these

webinars real quickly i posted in the chat window on assessment form so if people would please fill that out before before leaving and then under the questions again if you have other questions if you would open a private chat with me and just shoot me the question otherwise i guess i will read them out of the regular chat if they come up but our current question is can the speakers comment on any experience with normalized citation metrics which take field and age and type of article into account yes so this is stacy i’d be happy to answer that I think normalized citation metrics are the only citation metrics that should really be used especially in terms of evaluation purposes and so I think it’s our duty as librarians to learn these normalized field normalized citation metrics many of which exist I’d encourage we can send around some links after the webinar with specific examples but when you’re talking to faculty when they’re coming to you for advice when you’re compiling metrics on their behalf which I know many of you do explain this is what a field normalized citation metric is and it’s much more relevant than citation count and it’s going to help those who aren’t in your field who might be reviewing your tenure dossier or your grant application to understand the actual impact of the work that you do because as a physicist you might get you know hundreds of citations for a paper that might be pretty normal and if you’re in mathematics you know you’re going to get maybe 10 or 20 tops the citation differences are very very vast from field to field so I encourage you to encourage your faculty to use these sorts of metrics okay another quick procedural question there were we are intending on posting the recording of this so if you want to come back and review anything you can and we also hope to find a way to put the slides up if people want to be able to look at them apart from the presentation we’ll see if we can do that we can’t guarantee it but we hope so the next question is do you have any recommendations for measuring the impacts and use of datasets so this is Stacy again impact story they provide some very basic metrics for datasets that are hosted on figshare and Dryad which is a small portion of the datasets that exist out in the world but it’s still very useful for many researchers so they take things like downloads and citation counts in addition to social media metrics that we can gather so that’s a good place to start I’d also encourage you to check out the data level metrics initiative that I think it’s data sight and Martin Fenner are putting together that along with the work that was done by PLoS and California digital library they recently wrapped up a big survey where they went out and they interviewed data managers researchers a bunch of people as to what are the most relevant metrics that they’re using in terms of understanding the impacts of their data that was part of a grant called making data count and it’s a really good starting point for understanding okay what’s the lay of the land citations are big but data citation is not something that is a widely practiced yet so that would be the best case scenario but short of that many people are using things like downloads and pageviews to understand the relevance of and the interest in their data as well but then you’ve got other types of metrics that you might be able to capture like if you were to share numeric data on github for example you could look at things like stars or Forks that’s how many times people have cloned your data set so they can work with it on their own and maybe add to it or do analysis on it locally there are a number of many metrics so I’d encourage you to check out the making data count and the data level metrics initiative add to that in case you aren’t aware web of science also has a data Citation Index that has been around for maybe two years or so so there are definitely a lot of different groups that are trying to tackle this problem of appropriate citations for data sets so I think we’re getting there but you know maybe a while before this is all standardized thanks to the help of people like Martin Fenner alright another question I have gotten many questions particularly by math faculty whether so I this is Rachel I guess I’ll go first my personal take is I don’t see any reason why they wouldn’t count especially if we are talking about

citations usually people tend to get a little weirder when it comes to things like article views or PDF downloads but if it’s a citation that’s resulting and say it’s a white paper or their own preprint or a journal article it’s a pretty strong statement of impact even if it is a preprint I think the only thing that we would want to do is make sure that if there is a final publication that those citations get merged together but the information is the information whether it’s gone through a final editing process or not so I’m okay with it but I know a lot of citation ultimately comes down to culture within disciplines culture within departments culture within universities so if nothing else you can always direct it back at them and say maybe this is a discussion to take up at a departmental meeting so Stacy I’d say that a big maybe the the preprint versus post-print distinction is the most important thing right so encouraging your faculty to make it clear that let’s say when they’re compiling their CV for whether for personal user end of year annual review to have separate sections for work that has undergone peer review and has subsequent citations versus work that hasn’t yet been peer reviewed versus work that maybe it’s peer reviewed it’s in print but it’s also you know up on a place like archive or another preprint server and therefore accumulating citations as just a matter of course so I think that’s when many people is kind of that peer review versus not peer reviewed version and so I may be couch it in those terms as well know if the preprint are deposited in our institutional repositories this can offer other kind of information about the preprints like usage downloads and view so maybe you can separate citations and views because we we know that these numbers come from different sources and as matt says in his comments collapsing this in one number might affect the h-index so I I would suggest to have them listed but not collapse not put them together all right thank you very much I I think we actually need to end now at two o’clock we’ve used up our time thank you very much everybody for attending thank you speakers thank you alloy sharp for all your hard work behind the scenes keeping this running we hope top is gonna do more of these presentations please fill out the the form the assessment form so we can get some ideas of of kind of how interested you are again I’ll I’ll reach at that question and we’ll include the the URL in the in the follow-up email when we tell you when we’ve gotten the thing posted so recording and everything so thank you very much