Disturbing the Universe

David L Clements, science and science fiction

A Stinking Fish

4 Comments

Various academics in STFC areas will be spending chunks of the Easter holiday weekend filling in entries on something called ResearchFish. The intent of this is to provide information to STFC about the ‘outputs’ of the research grants that they fund. The possible outputs are highly varied, ranging from the concrete and easy to measure, such as scientific papers, to the much harder to measure, such as influencing the public and politicians.

While the intent of this effort is something I can agree with, to allow STFC to concretely demonstrate to BIS and HMG that we are achieving something, the implementation remains as ham fisted and tone deaf as it was in the previous incarnation, e-VAL.

The problem is that the system is far from light touch. A series of questions are asked for each individual activity so, for example, if you organise a series of outreach events, you have to provide details for each, even if those details are broadly similar. The same goes for many of the other areas that it asks about. It is essentially repeating many of the problems that the older e-VAL system had, though wrapped up in a more flashy Web 2.0 manner and now fed through an external private company (of which more later).

It also repeats one of the key issues with e-VAL which is that it does not allow searching of ADS, the primary astrophysics repository of publication information. This is utterly braindead, and is one of the core criticisms I had of e-VAL. It can’t be too difficult to scrape information from ADS so why it is not included is beyond me! There are also some curious lacunae in the proscriptive lists of options. There is nothing that allows you to describe an event as an exhibition, for example, despite the fact that two of the highest impact events in the UK science outreach calendar, the Big Bang Fair and the Royal Society Summer Exhibition are, in fact, exhibitions. Also, if you specify an event as aimed at school children you’re then asked to say which age group – primary, secondary, sixth form etc. – without the opportunity of choosing more than one. Quite how we’re meant to classify something like the RS SUmmer Exhibition that is aimed at schools, adults, press, and policy makers is left as a (frustrating) exercise to the reader.

The problem with all this is that the process penalises both success, in the sense that if you do more you have to waste more time reporting it at the level of forensic minutiae it’s after, and it also penalises diligence, in the sense that those who want to produce a fully accurate report with all the details that STFC are after, will end up spending a lot of time doing it.

There is an interesting cultural issue here in that the kind of person who works for the civil service will probably not think twice about ploughing through page after page of detailed questioning about a series of activities, while the kind of person who is an academic will know that there are far better and more productive ways of spending their time. And given the fact that pretty much nothing has been done to correct the problems reported last year, I doubt many of us will waste out time complaining either.

There is also the issue of this now being run by a private company (at least as far as I can tell). I presume that they are going to curate the data and will likely end up using it for their own purposes as well as the ones STFC intend it for. I don’t like that. I also don’t like the fact that I am required to agree to various terms and conditions as part of this process, that, as a grant holder, I am required to do.

So, STFC has largely failed to address the criticism of e-VAL, continues to go for maximal data collection, in the sense of asking many more questions about things than they’re likely to need, just to be sure, and then foisting this off on hard pressed researchers who would do better spending more time creating impact than reporting on it. I also have the suspicion that much of this information is already in STFC’s files from grant reports etc., or could be collected through that route rather than adding extra reporting burdens.

The end result will be something that is more fulfilled in the breach by those reporting, and an incomplete and inaccurate result. Better to go for coarser resolution and more complete data than the fine grain reporting they say they want and the inevitable incompleteness this will generate. I, for example, will not be providing details on each of the 90 science panels and talks I organised at the 2009 Worldcon, or the 19 such items I organised at last year’s Eastercon. For one thing I don’t know how many people attended each one.

Advertisements

Author: davecl

Astronomy, science, science fiction

4 thoughts on “A Stinking Fish

  1. Maybe I missed some earlier announcement, but I got an email about this a few months ago but didn’t really know what it was all about. I started the process but, as you mention, it just required far too much effort and after a few minutes I gave up. If I had a good sense of what it was for, I might have persevered. What made it worse was that, because of REF, I had just had to input all of this kind of stuff on my universities database and didn’t really have the time or the inclination to do it all again.

  2. This is one of the many problems. We’re told to do it, and just expected to grit our teeth and get on with it, no matter how poorly designed and poorly executed it might be.

    The impression I’ve got is that the people who worked on its design, and the design of e-VAL, are people who can give the job of filling it in to their secretaries, and so they don’t care about the needless duplication and over-gathering of overly-specific information.

    • Are we obliged to complete it? I saw an email recently which I thought suggested that we did not need to complete this, but I’ve just reread it and am not sure that my first interpretation was correct.

      • My understanding is that we’re meant to complete it, but they’re not going to track accuracy or completeness. It was suggested to me that I should waste no more than 30 minutes on it. I spent more like an hour or two in the end, much of which was very frustrating.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s