Guidance for peer reviewers using Je-S - ESRC

Training for peer reviewers

We have an online training tool available for peer reviewers. The training consists of 3 modules and takes approximately 45 minutes to complete.

The training is hosted on the Science and Technology Facilities Council (STFC) learning management system. See our user access guidance to set up an account and access the training.

Checklist for preparing a review

Before starting your review:

  • ensure you have no conflict of interest to declare with the application
  • ensure you read the entire application thoroughly
  • familiarise yourself with ESRC standards of service and code of conduct
  • ensure you read the specification of the funding opportunity for the application you are assessing
  • familiarise yourself with grades used at ESRC and their descriptors

Please refer to the guidance below before and during your review:

If anything is unclear, contact ESRC at email: esrcpeerreview@esrc.ukri.org.

When conducting a review

As a reviewer you should:

  • be realistic about your own confidence and expertise. You should provide clear evidence of your own expertise in the subject area and state if you are unsure about something
  • base your review on the quality of the social science on its own terms, accordingly to the case made by the applicants
  • review the ideas, concepts and approaches of the research, not the specific format or presentation of the document itself. Elegance of presentation is not in itself an assessment criterion for an ESRC grant, but the clarity of presentation may help or hinder your ability to review an application, so a comment to this effect would be appropriate
  • identify the strengths and weaknesses of the application and where possible provide a clear view on which should be accorded the greater significance and why. This may include:
    • raising issues or concerns with the application in the form of explicit questions for the applicants, as this assists the panel to assess how complete and convincing the principal investigator’s response is
    • providing an evaluation of the risks associated with the project
    • contextualising the application within current work in the field, and comment on its relative importance and significance
    • identifying any inconsistencies and contradictions in the application
    • identifying issues needing clarification by the applicant in their response
  • make a reasoned judgement against each assessment criteria, based on the evidence provided (or lack of it) and your wider expertise in the area
  • always provide evidence to support your observations
  • in the case of interdisciplinary applications, check whether the different disciplines link up in a coherent way
  • be receptive to new ideas and approaches to thinking within the discipline as well as methodology
  • provide enough information to enable a judgement on the relative quality of this application compared to others

The application to be reviewed may include a link to a website containing information on the proposed research. Reviewers are not required to consider this additional information when providing comments on an application. If reviewers do choose to look at this information, note that it is possible that their anonymity to the applicant may be compromised.

Reviewers’ scores and comments will be fed back to the applicant. It is therefore important that reviewers ensure their scores reflect their textual comments. In addition, comments should clarify their assessment (score) of the different elements of the application.

Checklist for the reviewer form

Before submitting a reviewer form, you should check if you have:

  • provided an impartial, objective, fair and analytical assessment of the application you are reviewing
  • not included any personal comments
  • made constructive criticism wherever possible, identifying how any issues could be realistically addressed by the principal investigator
  • avoided overly negative wording and phrases
  • provided an evaluation, not a description of the work proposed
  • ensured that the language used is clear and jargon-free. It should be understood by a non-expert
  • justified your final grade by, and consistent with, their comments

 

Value for money assessment

Academic reviewers are asked to assess the value for money of the proposed research. This section provides further advice on completing the value for money section of the assessment form, and should be read in conjunction with the existing guidance for applying assessment criteria.

In the value for money section you are asked two connected questions (below), which should help you decide whether the resources requested are justified and therefore whether the project offers value for money:

  1. Will the proposed project be undertaken in an efficient and effective way, and is the applicant requesting only resources that are essential to the completion of the project?
  2. Do the importance and the quality of the proposed research (and associated outcomes) justify the amount of resource required?

Response to question 1

For the first question you should consider whether the proposal has demonstrated an understanding of the amount of work to be done. Has the applicant identified the level of staffing (both the amount in full time equivalent and the experience and skills), travel and subsistence, and other costs that will be needed to achieve the aims of the project?

You might want to consider, for example in terms of the work planned for the research assistant, whether the amount of work is achievable within the timescale for someone with that level of expertise, and with the level of support and resource described.

You should also consider whether the applicant has clearly linked how each resource (for instance, research assistant, visit to archive, use of materials) will be used in undertaking the project.

In some cases, an application may be to fund part of a much larger project, where funding is only being sought for a subset of the research and will be a contribution to a more substantial output. In that case, you will need to consider whether the applicants have been clear and transparent in terms of what exactly the requested funding is for (for example, how the proposed project relates to any funding received from other sources).

Response to question 2

In the second question, if you have established that the project looks as though it will be run efficiently and effectively, you are asked to consider whether the importance and significance of the research is appropriate for the amount of resource requested. For example, if the proposed project requires a large number of research assistants but the output is going to be limited (in either scope or potential audience), then this might not provide good value for money.

Below, using some illustrative examples, we try to further clarify how reviewers should be approaching this part of a review.

We are not expecting reviewers to comment on whether £200 is the correct price for a flight. We would like them to consider whether all the trips that have been requested on a proposal are justified as being needed to conduct the research.

We are not expecting reviewers to comment on whether the researcher should receive a certain salary. We would instead like them to assess whether there is sufficient work, of the appropriate level, to warrant a researcher being employed for the period which the proposal requests.

We are not expecting reviewers to comment on whether a project should cost £300,000 rather than £350,000. We would like them to consider whether the total amount of resources (staff, trips or equipment) is warranted by the amount of project outputs and if the project is going to be completed in an effective way.

We are not expecting reviewers to comment on whether a particular piece of equipment (such as a scanner) should cost £1,000. We would ask them to consider whether a piece of equipment offers the most efficient way to do that piece of work (for example, whether scanning is the best method).

Frequently asked questions for peer reviewers

1. Can I download a copy of the peer review form?

ESRC uses the Joint Electronic Submission (Je-S) system to support its peer review process. You will need to log in to your Je-S account to review a proposal.

2. This isn’t really my area of expertise, are you sure you’ve got the right person?

When we select peer reviewers, we try to find individuals who can comment on a range of issues relevant to the proposal, including scientific merit, research design, the particular methodology proposed and the proposed data analysis. It may be that your interests match a very specific aspect of the proposal you have been sent. If you are in any doubt, email ESRC on esrcpeerreview@esrc.ukri.org.

3. I’ve never done this before. How long do my comments have to be?

There is no set length for peer review comments – you may be able to summarise your views concisely, or you may wish to comment on a number of specific aspects of the proposal or final report in detail. Relatively short comments that are clearly stated, well justified and constructive are more useful to any committee or panel (as well as applicants) than rambling or unfocused ones. The average review is between one and two pages in length.

4. I cannot provide a response by the deadline. Can I have an extension?

In the first instance you should email ESRC on esrcpeerreview@esrc.ukri.org. If a review is received after the due date and you have not sought prior approval to submit it late, ESRC may not be able to use it.

We do realise that everyone we approach is already very busy, and we will try to accommodate the wishes of reviewers and issue extensions to deadlines wherever possible. However, a  timely response from all peer reviewers is crucial to obtain a sufficient number of reviews by the panel cut-off date.

5. Will my review remain confidential?

Our peer review forms have two sections to them – a confidential section and a section for comments which will be fed back to the applicant. Usually, the whole form will be forwarded to the panel who will make a recommendation on the proposal.

Only the comments for the feedback section will be sent back, anonymously, to the applicant.

6. I think that there is a conflict of interest. Should I really comment on this proposal?

If you feel you may have a potential conflict of interest, you should note this in the confidential section of the reply form or email ESRC on esrcpeerreview@esrc.ukri.org for advice. For example, if you are reviewing an application and you plan to submit your own application to the same grants scheme in the near future, you should state this.

We accept that it is likely that academics who work in the same field may know each other, and this does not necessarily bar you from commenting on a proposal. Most of our peer review forms contain a confidential section on ‘knowledge of applicant’ where this information can be declared.

If your knowledge of or relationship with the applicant is such that you feel it would be difficult to be objective when commenting on the proposal, email ESRC on esrcpeerreview@esrc.ukri.org. You should state that you would rather not comment on this occasion, and explain why. We can make a note of this for future reference if necessary.

7. How did you come to select me as a peer reviewer?

ESRC has a Peer Review College from which our peer reviewers are selected. However, we also seek peer reviews from outside the college membership where appropriate. Reviewers are selected by ESRC case officers who have a background in a wide variety of social science areas.

8. I cannot provide a review but my colleague might be able to. Can I pass the application or report on to them?

This type of suggestion is very useful. Since applications and reports must be treated confidentially, do not pass these documents on to colleagues. Instead, you should decline to review, then enter the names and contact details of any potential reviewers using the Je-S system.

9. Will I be paid to carry out this review?

Unfortunately, except in very exceptional circumstances, we are unable to pay reviewers.

Frequently asked questions for Je-S electronic peer review

Find details of how to complete a review using the Joint Electronic Submissions (Je-S) system in the Peer Review (Reviewers’ Functionality).

1. What is the Joint Electronic Submissions (Je-S) system?

Je-S is a website currently used for completing and submitting research proposals to all research councils, including ESRC, and peer review of proposals.

2. Why is ESRC using the Je-S system for peer review?

We are committed to full participation in the Je-S system. The system provides a common interface to all research councils for applicants, research organisations and reviewers. ESRC currently provides proposal submission and peer review via the Je-S system.

There are specific advantages for both applicants and reviewers, including:

  • applications are no longer posted which means that confidential proposals are more securely handled since the chosen reviewer receives these direct in their personal Je-S account
  • there are no time delays particularly for non-UK based reviewers
  • the reviewer can view the full proposal, together with any associated documents, before deciding whether they wish to undertake the review.

3. I’ve never used Je-S before and do not have an account. How do I get one?

If you are an ESRC Peer Review College member, we will have added your details to the Je-S database in order to create an account. When you are notified of your first review request, the email you are sent will include a link that will take you to where you can activate your account. In order to do this, you will need to enter a username and password and some prompts and responses in case you forget your login details.

If you are not an ESRC Peer Review College member, an account will be created for you before a review is requested. You will be able to activate this account by following the link in the email sent notifying you of the review request.

When your account has been activated, you will be taken to the assigned document summary screen in the system where you should select the ‘peer review’ option.

4. When I first tried to access the peer review documents I’ve been sent, I got to a screen called ‘reviewer protocols’ and couldn’t get any further. What is this?

The reviewer protocols outline the research councils’ expectations regarding issues such as confidentiality of the material contained in the review documents and conflicts of interest. Reviewers must agree to abide by these protocols before being presented with any review material. Once you have indicated your agreement to the protocols, you will not be asked to do it again. You will, however, be asked to reaffirm your agreement once a year.

5. I’d like to see the proposal before I decide whether to do the review or not. How can I do this?

You may view the full proposal before making a decision about whether to complete the review.

You will need to agree to the reviewer protocols before accessing the review materials. Once you have done this, you will be presented with the document menu screen and you should select view documents to review from the document data menu.

6. Can I get a printed copy of the review form, so that I can see all the questions I need to answer before I begin making any responses in the system?

It is possible to print a blank copy of the review form.

In the document menu screen, you should select ‘create document for printing’ from the document import and export menu. If you then choose to include blank sections under the print options, you can create a printout of the form that includes all questions.

7. Do I need to answer all the questions on the form?

You need to answer questions in all ‘edit’ sections of the form if you are completing the review, or ‘decline to review’ only if you are declining the review request.

There are maximum character limits (including spacing and punctuation) for each question but no minimum limits. If you are unsure whether you have missed a question, you can select validate document at any time. You will then be provided with a list of questions that still need to be answered.

Once you have completed the review you must select the ‘submit document’ option in order for ESRC to receive your response.

8. What information should I be giving in response to each question on the form? Are any guidance notes available?

Within Je-S there is context-sensitive help text into which we have inserted guidance notes for completing the form.

If you select ‘help’ at the top-right corner of the screen while in any section of the review form, the appropriate guidance will appear. It is important that you take account of the scheme under which the proposal has been submitted, as there will be scheme-specific guidance in many cases. You can find the scheme in the banner at the top of each screen.

9. I’m not sure what to do next. Who can I contact for help?

If you need any assistance or advice in your use of the system or experience any problems, you should contact the Je-S helpdesk. They can be contacted by telephone on 01793 444164 or by email on jeshelp@je-s.ukri.org and are available from 8:30am to 5:00pm Monday to Thursday and 8:30am to 4:30pm Friday, UK time (excluding public holidays).

10. I’m not sure that I’m eligible to complete the review. Who can I contact for advice?

If you need advice on your eligibility to carry out the review or have any questions about information contained in the proposal or on the review process, you should contact the ESRC officer whose details appear in the instructions to reviewer section of the form.

11. I do not want to complete the review that I have been asked to. What should I do?

It would be very helpful if you could tell us that you are unable or unwilling to complete the review as soon as possible so that we may approach an alternative reviewer.

Ideally, you should do this by selecting ‘decline to review’ under the document data menu in the Je-S review form.

It is important, once you have provided the information requested in this section, that you select the ‘decline to review’ button. This is necessary for your response to be forwarded to ESRC.

12. I have elected to decline the review request and have been asked to suggest an alternative reviewer. Do I need to do this?

This is optional. However, it would be helpful if you could suggest an alternative reviewer.

Last updated: 29 August 2023

This is the website for UKRI: our seven research councils, Research England and Innovate UK. Let us know if you have feedback or would like to help improve our online products and services.