Confidence in Simulations for Science

Last week I attended the excellent SummerSim conference in Toronto, which focuses on all domains of simulation application. An item high on my research agenda in recent years, and that of other researchers at York, has been establishing confidence that results of complex system simulations are adequate representations of those systems. Simulation is used to investigate biological, sociological, political, financial systems, and more. We typically simulate these systems to aid in understanding them, however if they are not well understood to begin with, how do you create representative simulations? It is this bootstrapping problem that techniques such as the CoSMoS process and argumentation structures seek to address. I have looked at this problem from domain modelling, statistical, and calibration perspectives.

Myself and others at York were curious as to whether similar issues were being identified by simulation practitioners in other disciplines. As such, we organised a workshop examining “confidence in simulations for science” workshop at SummerSim, hoping to draw representatives from the eclectic¬†mix of disciplines represented. Our format was perhaps unusual; we did not wish to run a tutorial on techniques being developed at York, not did we want full paper presentations requiring peer review. This was a very preliminary examination of how far the issues we have been facing permeate through simulation endeavour in general. We put out a call for abstracts, hoping mostly to catch people who would already be attending the conference; I did not believe that many would find the finances to visit a conference when there was no publication to be gained (austere times that we live in). We received a number of very encouraging emails from people in the artificial life and synthetic biology community who sadly could not attend for our workshop clashed with other conferences they were already committed to. We received one abstract submission which was later retracted as the author could no longer attend the conference. We needed a plan B. We had prepared introductory material to the problem, and slides posing questions to the audience that we hoped would fuel a discussion. Fearing that we may be faced with a wall of blank stares, we prepared a lot of slides on activities at York addressing these questions, and hoped that we would not need to use them.

We were correct. The workshop was run in two 90 minute sessions. In the first session we had the three organisers: Paul Andrews, Kieran Alden and myself (Jon Timmis was sadly unable to attend the conference) and three participants. The size worked well, it was a very comfortable sized group for discussions. In the second half we were joined by others and had 7 participants. The feedback was very positive: we did generate a lot of discussion; the problems we have identified are pertinent to other simulation domains, and have not been solved elsewhere; the articulation of the problems was praised; the format was also well received, and we were given useful suggestions for how to improve it in the future; and we were informed that a sole outlet dedicated to these issues (rather than addressing them solely by-discipline) manner was useful. We shall have to think carefully about what the outputs of the workshop will be, there seems to be scope for a continuation. This shall be decided in the coming week, when Kieran (who is currently on holiday) returns with the minutes from the workshop. At the very least we have a mailing list of interested parties who can stay in contact and work together on this.

This was my first experience of chairing a workshop, and there are some lessons to take away. First, you will rarely struggle to generate discussions at a conference, though it helps to have a plan on how to lead in (I believe that Paul knew this already, having run numerous CoSMoS workshops in the past, but its one thing to be told and another to witness it firsthand). Second, try to ensure that the workshop’s details appear next to the main conference timetable in the programme. All conference attendees were given a staple bound programme that fell open in the centre at some coloured pages detailing where and when sessions were taking place. Alas the workshops were listed on page 3 or so, and I suspect many never saw them. Third, our initial suspicions that short un-refereed abstracts will not attract attendees to a conference was largely correct (though we did generate interest ¬†in a few parties who could not attend). As such, a conference like SummerSim is appropriate, as it captures such a wide range of simulation practitioners. I thoroughly enjoyed running the workshop, and will seek to do it again in the future. I encourage all aspiring academics to give it a shot!

… and Toronto is well worth visiting …