Open

Standardise or not to Standardise?

Posted on Tue 14 April 2015
Written by Keith Turkington

Written by Keith, MD of Fluent Technology, grant management guru, process engineer, frequent flyer, father of 3, ufology geek and keen cook.

I have been fortunate to sit in on a number of review panels over the years and have observed many different ways in which review panels and committees reach their decisions.   What I find most interesting is how different individuals approach the job of scoring applications.  Even though there can be guidelines and a common scoring framework some reviewers like to be generous with their scores and take off points if they feel applications are missing essential details.  Others reviewers are “less generous” with their scores (I was tempted to say meaner but that would be cruel!) and only award points when they feel an application merits it.

There is no right answer.  Whether you start high and work down or start low and build from there the work of a reviewer scoring applications is vital to many organisations and reviewers should be allowed some freedom to determine how they apportion scores.   However, how do we deal with the possibility that one application could be reviewed by a number of low-scoring reviewers and another could be scored by high-scoring reviewers?  If this were to happen it is possible that 2 very similar applications could end up with markedly different (average)  scores.   I can hear the “but’s…” starting i.e. “but we discuss scores at our panel meetings!”, “but we check the spread between reviewers scores!” etc.  I think these are all good things to do but, just to be a little controversial, this does not address the problem I am raising – what happens if you get 3 generous or 3 less than generous reviewers?

So what is the answer?  Applying a statistical method i.e. a standardisation technique (e.g. z-scores) to validate the scores applied to any and all applications by all reviewers.  Using a standardisation technique can help ensure that those applications that need discussed are clearly identified.  The benefit of using a standardisation technique is that it takes into account a reviewer’s score for an application in the context of their previous scores for other applications and compares this to the other reviewers’ scores for the same application in the context of their previous scores for other applications.  If that sounds complicated, don’t worry!    The point is that by using a statistical method you are highlighting any disparity between scores and removing the uncertainty that might exist by relying only on a manual review of scores.

In short if you want to have confidence in your scoring approach then use a standardisation technique!   And remember there is no truth to the allegation that statisticians are mean. They are just your standard normal deviates…

 

 

Recommended Posts.

 

Accepting Creepy Projects.

Written by Keith.

It was all going so well.  The project was running smoothly, every team-member knew their responsibilities. We were on timeand to budget.  But then it happened…someone suggested a little change…

https://www.flexigrant.com/accepting-creepy-projects/

 

Infographic: 5 Core Questions To Help You Assess Your Grant Management Processes.

Written by Claire.

So you’ve made the decision as an organisation to upgrade your grant management system…but you need something that’s tailored to you. Your grant process is different from others, you have more review stages, your grants are larger, you only have a limited amount of resources in terms of finance and employees. Whatever the reason…you can’t get a new system until you assess where the current one is working for you and where it’s not.

https://www.flexigrant.com/infographic-5-core-questions-to-help-you-assess-your-grant-management-processes/

 

5 Sites With The Best Free Stock Photography.

Written by Claire.

We don’t know about you, but we love images! Why? Because a picture speaks a thousand words.

We’re in the middle of the ‘image marketing era’ and the direction that social media is taking is the testament to how true this is. Platforms like Instagram are getting triple the amount of engagement than that of its counterpart, Facebook (source:The Marketing Eye, Sept 2015). This is all great in theory…BUT stock photography is expensive (especially if you’re a small organisation with an even smaller marketing budget!)…

https://www.flexigrant.com/5-sites-with-the-best-free-stock-photography/