Download to read offline
![How can we measure
societal impact? UK REF
Wilsdon et al., 2015
“[…] for the impact component
of the REF, it is not currently
feasible to use quantitative
indicators in place of narrative
impact case studies, or the
impact template”](https://image.slidesharecdn.com/presentaciondts-socialimpact-161112180948/85/Presentacion-SocietalImpact-Daniel-Torres-21-320.jpg)
![Altmetrics and the ‘broader’
impacts of science
Wilsdon et al., 2015
“[…] wider use of quantitative
indicators, and the emergence of
alternative metrics for societal
impact, could support the
transition to a more open,
accountable and outward-facing
research system”](https://image.slidesharecdn.com/presentaciondts-socialimpact-161112180948/85/Presentacion-SocietalImpact-Daniel-Torres-22-320.jpg)
![Case: societal impact
indicators at UGR
Multidimensional evaluation using biplot.
Research at the University of Granada
Source: Milanes-Guisado, Y. (2015). Multidimensional evaluation of research. Micro analysis
at the university of Granada during the period 2009-2013 [doctoral thesis]. Universidad de
Granada. Supervisors: Daniel Torres-Salinas, José Navarrete, Francisco Solis
Research questions:
• ¿What is the metric
profile of research
groups taking into
account bibliometric
and non-bibliometric
indicators?
Methodology:
• Time frame: 2009-2013
• Analysis unit:
161 research groups = 2216 res.
• Methods:
Correlations
biplot analysis](https://image.slidesharecdn.com/presentaciondts-socialimpact-161112180948/85/Presentacion-SocietalImpact-Daniel-Torres-24-320.jpg)
![European Summer School | September 8, 2016
Especial thanks to Yusnelkis Milanés
for providing data from her PhD thesis
www.ingenio.upv.es
Source: Milanes-Guisado, Y. (2015). Multidimensional evaluation of research. Micro analysis
at the university of Granada during the period 2009-2013 [doctoral thesis]. Universidad de
Granada. Supervisors: Daniel Torres-Salinas, José Navarrete, Francisco Solis](https://image.slidesharecdn.com/presentaciondts-socialimpact-161112180948/85/Presentacion-SocietalImpact-Daniel-Torres-43-320.jpg)
Uploaded byEC3metrics Spin-Off
Presentación SocietalImpact Daniel Torres
The document discusses measuring societal impact of research. It defines societal impact as social, environmental, cultural or economic benefits from academic activities. Measuring societal impact is challenging due to attribution problems. The UK REF attempted to allocate funding based on non-academic impact using impact case studies. Altmetrics were discussed as a potential way to measure broader impacts, though they may be more related to scientific impact. A case study at the University of Granada used bibliometric and non-bibliometric indicators across nine dimensions, finding some research groups with clear societal impact orientation. Potential applications of altmetrics include analyzing social networks and identifying research communities. Both qualitative and quantitative assessments have limitations in measuring societal impact.
More Related Content
Similar to Presentación SocietalImpact Daniel Torres
![[DSC Europe 25] Milos Belcevic - Product Professional's Journey to Full-Stack...](https://cdn.slidesharecdn.com/ss_thumbnails/1zovd6fgsycdg4wvgvls-milos-belcevic-product-professionals-journey-to-full-stack-product-developer-260123083019-d993120d-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Paula Garcia Esteban -Building the Future: The Role of Data S...](https://cdn.slidesharecdn.com/ss_thumbnails/9ld1r1bsqpwve8qfvphy-paula-garcia-esteban-building-the-future-260122103838-4171f5cb-thumbnail.jpg?width=640&height=640&fit=bounds)
Presentación SocietalImpact Daniel Torres
- 1. European Summer School|September 8, 2016 Societal Impact Nicolas Robinson-Garcia Daniel Torres-Salinas INGENIO (CSIC-UPV) Univ Navarra and Univ Granada (ec3metrics & Medialab ugr) www.ingenio.upv.es
- 2. Objectives Understand the motivationsthat lead for demands of societal impact indicators 1 Know the limitations and current problems existing to meet such demands 2 Learn about potential uses of altmetrics to showcase ‘aspects’ of societal impact 3
- 3. Index Brief Intro What issocietal impact? How can we measure societal impact? Altmetrics and the ‘broader impacts’ of research Potential uses of altmetrics Food for thought
- 4. “The last 30years have witnessed a slow, but continuous process of reorganization of European universities” Geuna & Muscia, 2009 Brief intro https://static.pexels.com/photos/30824/pexels-photo.jpg
- 5. Brief intro Global competitionand budget cuts Accountability and transparency Evaluation and funding
- 6. Brief intro • Bibliometricsis seen as an answer to provide impact measures within and even outside the scientific sphere citation analysis – patent analysis – univ-industry collab • In the recent years there has been a critical revision on the limitations of these metrics scientific impact ≠ societal impact http://www.logic-canvas.com/wp-content/uploads/2013/11/Transparency-Glass.jpg
- 7. Brief intro DORA Declaration Againstthe use of journal based indicators Leiden Manifesto Best practices on the use of bibliometric indicators The Metric Tide UK Report on the role of bibliometric indicators in research evaluation
- 8. What is societalimpact? Societal impact is usually referred as the social, environmental, cultural or economic benefits derived from academic activities Teaching Research Third Mission SOCIETAL IMPACT
- 9. What is societalimpact? Type of impact Output Activities Scientific- scholarly Knowledge Growth Papers, books, reports, cases, magazines Increase and organise the stock of knowledgePublication outlets Research networks Workshops, collaboration, conferences, committees Build and convene networks Supply of human capital Societal Cultural Archives, museums, festivals, exhibition and performance, events Social Policy changes, social benefits Problem solving Socioeconomic Patents, spin-offs, start-ups, contracts, consultancy Support entrepreneurialismTechnological Source: Robinson-Garcia, van Leeuwen & Rafols, in progress
- 10. How can wemeasure societal impact? Socioeconomic & Technological Impact • Largest body of literature • Synergies between industry and university • Tracking of knowledge transfer Social Impact • Translational research • Environmental research Cultural Impact • Field-specific • Largely unexplored • Affects mainly the Social Sciences & Humanities
- 11. How can wemeasure societal impact? The attribution problem Researcher publishes a paper Researcher gives lectures? Media coverage? Patent protection? Advocacy? Local council implements new policy Increase on tourism rates Employment rates increase in industry Creates a spin-off?
- 12. How can wemeasure societal impact? The attribution problem Researcher publishes a paper undertakes a research project? Local council implements new policy Researcher gives lectures? Media coverage? Patent protection? Advocacy? Increase on tourism rates Employment rates increase in industry Creates a spin-off?
- 13. How can wemeasure societal impact? An example. El Cabanyal The district •Old village, now a district •Architectonic value The sting •Access to the beach •Salvem el Cabanyal The link? •Few papers (National journals) •Not all researchers involved are authors The attribution problem
- 14. GOALS How can wemeasure societal impact? UK REF Selective allocation of research funding to Higher Education Institutions Produce evidence of benefits of public investment in research Provide benchmarking information and reputational yardsticks King’s College London & Digital Science, 2015
- 15. First attempt toallocate funding based on non- academic impact How can we measure societal impact? UK REF King’s College London & Digital Science, 2015 “Impact is defined as ‘any effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia’ (REF, 2011)”
- 16. How can wemeasure societal impact? UK REF PANEL A Biomedical & Health Sciences 6 UoA – 1586 case studies PANEL D Humanities & Arts 10 UoA – 1617 case studies PANEL C Social Sciences 10 UoA – 1965 case studies PANEL B Life & Earth Sciences, Engineering, Physics, Maths 9 UoA – 1469 case studies
- 17. Assessing societal impactthrough impact case studies How can we measure societal impact? UK REF IMPACT CASE STUDY TEMPLATE 1. Summary of the impact 2. Description of the underpinning research 3. References of the research 4. Details of the impact 5. Sources to corroborate the impact
- 18. How can wemeasure societal impact? UK REF Case studies by type of impact
- 19. How can wemeasure societal impact? UK REF Case studies by type of impact
- 20. How can wemeasure societal impact? UK REF Ratings by type of impact
- 21. How can wemeasure societal impact? UK REF Wilsdon et al., 2015 “[…] for the impact component of the REF, it is not currently feasible to use quantitative indicators in place of narrative impact case studies, or the impact template”
- 22. Altmetrics and the‘broader’ impacts of science Wilsdon et al., 2015 “[…] wider use of quantitative indicators, and the emergence of alternative metrics for societal impact, could support the transition to a more open, accountable and outward-facing research system”
- 23. Altmetrics and the‘broader’ impacts of science Classic perspective Author perspective Two perspectives
- 24. Case: societal impact indicatorsat UGR Multidimensional evaluation using biplot. Research at the University of Granada Source: Milanes-Guisado, Y. (2015). Multidimensional evaluation of research. Micro analysis at the university of Granada during the period 2009-2013 [doctoral thesis]. Universidad de Granada. Supervisors: Daniel Torres-Salinas, José Navarrete, Francisco Solis Research questions: • ¿What is the metric profile of research groups taking into account bibliometric and non-bibliometric indicators? Methodology: • Time frame: 2009-2013 • Analysis unit: 161 research groups = 2216 res. • Methods: Correlations biplot analysis
- 25. Case: societal impact indicatorsat UGR We calculated 46 indicators and classified them into 9 different dimensions We reduced and make a biplot analysis Multidimensional evaluation using biplot. Research at the University of Granada
- 26. Case: societal impact indicatorsat UGR Altmetrics Funding and projects Scientific output Knowledge transfer Media and news impact Scientific impact – Citations Teaching dissemination 9 Dimensions of academic activity and impact
- 27. Case: societal impact indicatorsat UGR Altmetrics Funding and projects Scientific output Knowledge transfer Media and news impact Scientific impact – Citations Teaching dissemination Societal impact related dimensions
- 28. Case: societal impact indicatorsat UGR Altmetrics Google + Mentions Facebook Likes Twitter Indicators Knowledge Transfer Patents Contracts Spin offs Media and News Impact Newspapers contributions Public exposure Mentions in Newspapers Teaching Dissemination Teaching Material Innovation Projects Some examples of “societal impact” indicators
- 29. Case: societal impact indicatorsat UGR Some examples of the information sources
- 30. Case: societal impact indicatorsat UGR Method remarks 1) Calculation of 46 indicators for research groups in four areas 2) Normalization and standarization of indicators values using z-scores 3) Representation of dimensions and groups into two dimensions using biplot analysis 4) Reducing the biplot with clockwise representation Now we are going to see an example in Exact Sciences
- 31. Potential applications of altmetrics Theseare the dimensions (blue) These are the research groups (green) BIPLOT: “EXACT SCIENCES 52” Research groups
- 32. Potential applications of altmetrics TeachingDissemination Media and News Impact Knowledge Transfers Altmetrics These are cluster and dimensions used to identify SOCIETAL IMPACT
- 33. Potential applications of altmetrics Altmetrics Whatwe see is that most of the research groups are in dimensions related to traditional publications and the impact system (ie: Web of Science) Scientific Impact Scientific output
- 34. Potential applications of altmetrics Altmetrics DOESALTMETRICS MEASURE SOCIETAL IMPACT? We see that altmetrics is more related to scientific impact and scientific output (at the UGR) Scientific Impact Scientific output
- 35. Potential applications of altmetrics TeachingDissemination Media and News Impact Knowledge Transfers So, for Exact Sciences at the University of Granada we have a really clear quadrant related to SOCIETAL IMPACT
- 36. Potential applications of altmetrics Wehave identified four groups with a clear Societal Impact orientation FQM325 FQM171 FQM338 FQM325 FQM324
- 37. Teaching dissemination Media andnews impact Funding and projects National scientific output Knowledge transfer Scientific impact - Citations International scientific output Altmetrics and webmetrics Finally this is the general and multidimensional overview of the exact sciences at the UGR
- 38. Potential applications of altmetrics Social engagement throughTwitter 1.Select a researcher 2.Identify its follower/followee network 3.Identify different communities of contacts 4.Identify geographical/cognitive/… proximities
- 39. Potential applications of altmetrics Social engagement throughTwitter L Waltman’s Twitter network
- 40. Potential applications of altmetrics LWaltman’s Twitter network Social engagement through Twitter
- 41. • Threats derivedfrom a purely qualitative assessment Peripheries and prestige • Threats derived from purely quantitative assessment Biases by type and scope of impacts Visualizations as a balanced solution Food for thought
- 42. European Summer School| September 8, 2016 Questions? Nicolas Robinson-Garcia Daniel Torres-Salinas INGENIO (CSIC-UPV) EC3Metrics SL www.ingenio.upv.es
- 43. European Summer School| September 8, 2016 Especial thanks to Yusnelkis Milanés for providing data from her PhD thesis www.ingenio.upv.es Source: Milanes-Guisado, Y. (2015). Multidimensional evaluation of research. Micro analysis at the university of Granada during the period 2009-2013 [doctoral thesis]. Universidad de Granada. Supervisors: Daniel Torres-Salinas, José Navarrete, Francisco Solis