Aim
The aim of this sandpit is to develop research projects that develop, validate, and critique novel scientometric indicators for use in research assessment and in metascience research.
The increasing availability of scientific output data and metadata, combined with advances in artificial intelligence and machine learning abilities in analysing semantic content, is creating the conditions for a paradigm shift in scientometrics. These developments offer the opportunity to develop new measures of research excellence and impact that go beyond traditional citation-based metrics.
This is reflected in a growing range of innovations in the measurement and understanding of research quality. These include indicators of novelty, disruption, interdisciplinarity, collaboration, quality, replicability, integrity, semantic relevance, research papers to to patents and policy and more sophisticated methods for classifying types of citations. In addition, a recent independent report REF-AI argues that there is a growing imperative to consider the potential for using large language models for assessment in the UK’s Research Excellence Framework 2029.
However, many of these indicators and methods lack validation, theoretical reflection, and deep engagement with end users and the research community. This sandpit seeks to address these challenges.
This sandpit will focus on the following three key areas where improved scientometric indicators could have significant impact:
- evaluation of research portfolios, for instance, in national research assessment exercises, internal analyses by research institutions, or for use by research funders
- individual programme or project-level evaluation within a public or private research institution or research funding organisation
- the field of metascience where no standard suite of agreed measures exists to enable consistent comparison of research and policy interventions
This sandpit will fund 12-month research projects to deliver actionable research findings, including the development of new open access indicators and measures. The intention is not to support exploratory, open-ended work, but rather to develop measures and indicators to a point where they can be readily implemented by research funders, policymakers and other potential users. Throughout these projects, the funded cohort of grant holders will form an expert challenge group, acting as both as critical friend and peer support for each other.
Sandpit process
The sandpit will be an intensive, interactive workshop bringing together scientometric experts from a range of institutions and perspectives to work together over four days.
The sandpit will be led by the UK Metascience Unit, who will be supported by a team of mentors. The mentors and a small number of stakeholders will attend the sandpit but will not be eligible to receive research funding. Instead, their role will be to assist participants in defining and exploring challenges in this area. The mentors will act as independent reviewers, making a funding recommendation on the emergent projects.
The sandpit process can be broken down into several stages as follows:
- defining the scope, namely, inclusion and exclusion criteria for indicator development
- drawing upon perspectives from relevant stakeholders and the expertise brought by the participants to share understandings of the challenges
- taking part in sessions focused on the challenges and using creative thinking techniques to identify approaches to help tackle these challenges
- capturing the outputs of the process in the form of highly innovative research project proposals
- reaching a funding decision on projects developed at the sandpit using ‘real-time’ expert review
Participants are not required to develop specific plans for research activities prior to the sandpit. Ideas for activities will be co-developed collaboratively between participants during the process.
Projects developed through the process will pitch for funding on the final day of the sandpit.
Attendance at the sandpit does not guarantee UKRI funding. It is our intention that this sandpit will be a valuable experience for all attendees irrespective of whether funding is secured.
Relationship to Metascience research grants round 2
The Metascience Unit is committed to experimenting in the delivery of research funding. We believe that a successful research portfolio balances both targeted and curiosity-driven approaches. Separate to this sandpit opportunity, we have launched Metascience research grants round 2 which is a curiosity-driven funding opportunity for applicant-led metascience research projects. The opportunity welcomes applications in the field of scientometrics as well as other areas of metascience.
In contrast, this sandpit opportunity is a targeted event to co-create a set of interrelated projects as a group, all centred around the specific objective of developing scientometrics for research assessment.
Applicants may apply to both the Metascience grants round 2 and the Scientometrics for research assessment metascience sandpit.
Trusted Research and Innovation (TR&I)
UKRI is committed in ensuring that effective international collaboration in research and innovation takes place with integrity and within strong ethical frameworks. TR&I is a UKRI work programme designed to help protect all those working in our thriving and collaborative international sector by enabling partnerships to be as open as possible, and as secure as necessary. Our TR&I Principles set out UKRI’s expectations of organisations funded by UKRI in relation to due diligence for international collaboration.
As such, applicants for UKRI funding may be asked to demonstrate how their proposed projects will comply with our approach and expectation towards TR&I, identifying potential risks and the relevant controls you will put in place to help proportionately reduce these risks.
See further guidance and information about TR&I, including where applicants can find additional support.
