Various academics in STFC areas will be spending chunks of the Easter holiday weekend filling in entries on something called ResearchFish. The intent of this is to provide information to STFC about the ‘outputs’ of the research grants that they fund. The possible outputs are highly varied, ranging from the concrete and easy to measure, such as scientific papers, to the much harder to measure, such as influencing the public and politicians.
While the intent of this effort is something I can agree with, to allow STFC to concretely demonstrate to BIS and HMG that we are achieving something, the implementation remains as ham fisted and tone deaf as it was in the previous incarnation, e-VAL.
The problem is that the system is far from light touch. A series of questions are asked for each individual activity so, for example, if you organise a series of outreach events, you have to provide details for each, even if those details are broadly similar. The same goes for many of the other areas that it asks about. It is essentially repeating many of the problems that the older e-VAL system had, though wrapped up in a more flashy Web 2.0 manner and now fed through an external private company (of which more later).
It also repeats one of the key issues with e-VAL which is that it does not allow searching of ADS, the primary astrophysics repository of publication information. This is utterly braindead, and is one of the core criticisms I had of e-VAL. It can’t be too difficult to scrape information from ADS so why it is not included is beyond me! There are also some curious lacunae in the proscriptive lists of options. There is nothing that allows you to describe an event as an exhibition, for example, despite the fact that two of the highest impact events in the UK science outreach calendar, the Big Bang Fair and the Royal Society Summer Exhibition are, in fact, exhibitions. Also, if you specify an event as aimed at school children you’re then asked to say which age group – primary, secondary, sixth form etc. – without the opportunity of choosing more than one. Quite how we’re meant to classify something like the RS SUmmer Exhibition that is aimed at schools, adults, press, and policy makers is left as a (frustrating) exercise to the reader.
The problem with all this is that the process penalises both success, in the sense that if you do more you have to waste more time reporting it at the level of forensic minutiae it’s after, and it also penalises diligence, in the sense that those who want to produce a fully accurate report with all the details that STFC are after, will end up spending a lot of time doing it.
There is an interesting cultural issue here in that the kind of person who works for the civil service will probably not think twice about ploughing through page after page of detailed questioning about a series of activities, while the kind of person who is an academic will know that there are far better and more productive ways of spending their time. And given the fact that pretty much nothing has been done to correct the problems reported last year, I doubt many of us will waste out time complaining either.
There is also the issue of this now being run by a private company (at least as far as I can tell). I presume that they are going to curate the data and will likely end up using it for their own purposes as well as the ones STFC intend it for. I don’t like that. I also don’t like the fact that I am required to agree to various terms and conditions as part of this process, that, as a grant holder, I am required to do.
So, STFC has largely failed to address the criticism of e-VAL, continues to go for maximal data collection, in the sense of asking many more questions about things than they’re likely to need, just to be sure, and then foisting this off on hard pressed researchers who would do better spending more time creating impact than reporting on it. I also have the suspicion that much of this information is already in STFC’s files from grant reports etc., or could be collected through that route rather than adding extra reporting burdens.
The end result will be something that is more fulfilled in the breach by those reporting, and an incomplete and inaccurate result. Better to go for coarser resolution and more complete data than the fine grain reporting they say they want and the inevitable incompleteness this will generate. I, for example, will not be providing details on each of the 90 science panels and talks I organised at the 2009 Worldcon, or the 19 such items I organised at last year’s Eastercon. For one thing I don’t know how many people attended each one.