SocSimFest 2021 webpage banner

Quantifying the uncertainty in agent-based models

(aka what the ♠€⅋@ is going on in my model and why?)


Alison Heppenstall and Nick Malleson

University of Leeds and The Alan Turing Institute, UK
A.J.Heppenstall@leeds.ac.uk


Slides available at:
https://urban-analytics.github.io/dust/presentations.html

Acknowledgements

Josie McCulloch, Patricia Ternes, Robert Clay,

Jiaqi Ge, Jonathan Ward, Minh Kieu

Greetings from Yorkshire (Northern England)

Pictures from Yorkshire

What have we been up to?

Pictures of activities that Alison and Nick have been doing

A year of home-schooling...

Happy home schooling
Photo by Natasha Hall on Unsplash
Alison's children not enjoying home schooling

Talk overview

Why is ABM and calibration difficult?

Why should we bother?

Framework from UQ

Adding time element

Towards live simulations

Diagram of the sims

Agent-Based Modelling

Systems are driven by individuals

(cars, people, ants, trees, whatever)

Individual-level modelling

Rather than controlling from the top, try to represent the individuals

Autonomous, interacting agents

Represent individuals or groups

Situated in a virtual environment

Crooks et al. (2019), Agent-based Modelling and Geographical Information Systems: A Practical Primer, Sage, London, UK.
Example ABM papers

Calibration

Finding the optima not a problem, finding the global is…

Model discrepancy – error between optimised error and an observation

Uncertainties – between model and real system

Computational cost: potentially lots of parameters = lots of runs

And added delights with ABM...

Nature of the system

Variable dimensionality

Distance metric based on pattern measurement

Emulation of model structure with many different rules

Lots of ABMs used to fight covid spread

ABM and Uncertainty

Different types of uncertainty in a model

Parameter uncertainty:

Which parameters? Which values? Equifinality.

Observation uncertainty

Imprecise, noisy data, natural variability

Model uncertainty / model discrepancy:

Model is a simplification (relies on assumptions and imperfections)

Ensemble variance:

Stochastic variance

Uncertainty Quantification

Model runs with large and small uncertainty. Other areas have well developed approaches to this

History Matching (HM)

HM is a procedure used to reduce the size of the parameter space (Craig et al. 1997)

HM discards areas found to be implausible, leaving a (usually) much smaller non-implausible region of inputs

Approximate Bayesian Computation (ABC)

ABC estimates a posterior distribution that quantifies the probability of specific parameter values given the observed data

Combined ...

Steps in using history matching and ABC

Sugarscape

Model results from sugarscape

Here we are looking at two parameters:

Maximum possible metabolism of an agent

Maximum possible vision of an agent

Outcome measure is size of population that the model can sustain (66).

Framework followed

Define the parameter space to be explored

Quantify all uncertainties in the model and observation

Run HM on the parameter space

Run ABC, using the HM results as a prior

Sugarscape: HM

history matcing results: plausible parameter combinations

Sugarscape - small parameter space. Measure the implausiblity of each parameter pair (metabolism, vision)

10 waves performed – use plausible space from wave 1 as input to wave 2 and so forth

Sugarscape: ABC

history matcing results: most likely parameter combinations

Results show that the parameters with the highest probability of matching the observation (i.e. sustaining a population of 66 agents) are where {metabolism, vision} are {4, 7}.

Rapid Assistance in Modelling the Pandemic (RAMP): Urban Analytics

Microsimulation of disease spread

Includes shopping, schooling, working

Calibrated on hospital admissions data

Screenshot of the RAMP GUI Screenshot of the RAMP GUI

RAMP: Urban Analytics

ABC Parameter posteriors

Posterior distributions of the model parameters

RAMP: Urban Analytics

ABC results uncertainty

Posterior distributions of the model parameters

Reducing Uncertainty in Real Time

Example: Urban Mobility

People are the drivers of processes in cities

We need to understand mobility patterns:

Pollution – who is being exposed? Where are the hotspots?

Economy – can we attract more people to our city centres?

Disease - which times / places have large numbers of interactions

Google Mobility Data: people spending much more time at home

Problem: Models will Diverge

Lots of uncertainties

Inputs (measurement noise)

Parameter values

Model structure

Natural stochasticity

... and that's assuming that the system hasn't changed fundamentally

Possible Solution: Data Assimilation

Used in meteorology and hydrology to update running models with new data.

Try to improve estimates of the true system state by combining:

Noisy, real-world observations

Model estimates of the system state

Diagram of data assimilation and an ABM

Example: Real Time Crowd Modelling

Simulating a city in real-time is too hard!! (for now). Let's start with a crowd

Can we reduce the uncertainty in an agent-based crowd simulation in real time?

Pedestrian trace data (Grand Central Terminal, New York City)

B. Zhou, X. Wang and X. Tang. (2012) Understanding Collective Crowd Behaviors: Learning a Mixture Model of Dynamic Pedestrian-Agents. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2012

http://www.ee.cuhk.edu.hk/~xgwang/grandcentral.html

Data assimilation with a Particle Filter

Flowchart of the experimental design. Lots of models ('particles') are run simultaneously.

Particle Filter Results

Predictions of the PF with 10 agents and 10 particles - most particles
                              predict the locations of the agents well.

Particle Filter Results

Predictions of the PF with 40 agents and 10 particles - now the model is
                              too complex for only 10 particles to reliably estimate its behaviour.

Preliminary Particle Filter Results

Box Environment: More particles = lower error

median absolute error change with number of agents and particles: greater
                              complexity caused by larger numbers of agents can be mitigated by increasing
                              the numbers of particles.
Malleson, Nick, Kevin Minors, Le-Minh Kieu, Jonathan A. Ward, Andrew West, and Alison Heppenstall. (2020) Simulating Crowds in Real Time with Agent-Based Modelling and a Particle Filter. Journal of Artificial Societies and Social Simulation 23(3) DOI: 10.18564/jasss.4266.

Difficulties (I)

Exponential increase in complexity

Number of collisions increases exponentially with number of agents
N. Malleson, K. Minors, Le-Minh Kieu, J. A. Ward, A. West and Alison Heppenstall (2020). Simulating Crowds in Real Time with Agent-Based Modelling and a Particle Filter. Journal of Artificial Societies and Social Simulation (JASSS) 23(3), 3.

Solutions (I)

A big server

More particles!

Will need high performance computing, but we have lots of that

Better PF algorithms (we used the most basic)

Alternative data assimilation methods

More advanced particle filtering is available

Unscented / Ensemble Kalman Filters

Solutions (I)

Unscented Kalman Filter

Shows that the UKF works best when model and observation error are similar
Clay, Robert, Le-Minh Kieu, Jonathan A. Ward, Alison Heppenstall, and Nick Malleson (2020) Towards Real-Time Crowd Simulation Under Uncertainty Using an Agent-Based Model and an Unscented Kalman Filter’. In Advances in Practical Applications of Agents, Multi-Agent Systems, and Trustworthiness. The PAAMS Collection 12092:68–79. Lecture Notes in Computer Science. DOI:10.1007/978-3-030-49778-1_6.
Shows that the UKF works best when model and observation error are similar
Patricia Ternes, Jonathan A. Ward, Alison Heppenstall, Vijay Kumar, Le-MinhKieu, and Nick Malleson. Using data assimilation to reduce uncertainty in anagent-based pedestrian simulation in real time. In review

Difficulties (II)

Model Discrepancy

Some behaviours are more predictable than others

No obvious 'solution' to this

But shows how important data assimilation could be

Other Methods (ongoing)

Ensemble Kalman Filter

Ward, Jonathan A., Andrew J. Evans, and Nicolas S. Malleson. (2016) Dynamic Calibration of Agent-Based Models Using Data Assimilation. Royal Society Open Science 3(4). DOI: 10.1098/rsos.150703.

Unscented Kalman Filter

Clay, Robert, Le-Minh Kieu, Jonathan A. Ward, Alison Heppenstall, and Nick Malleson (2020) Towards Real-Time Crowd Simulation Under Uncertainty Using an Agent-Based Model and an Unscented Kalman Filter’. In Advances in Practical Applications of Agents, Multi-Agent Systems, and Trustworthiness. The PAAMS Collection 12092:68–79. Lecture Notes in Computer Science. DOI:10.1007/978-3-030-49778-1_6.

Quantum Field Theory - Creation and Annihilation Operators

Tang, Daniel. (2019) Data Assimilation in Agent-Based Models Using Creation and Annihilation Operators. ArXiv:1910.09442 [Cs]. arxiv.org/abs/1910.09442.

Ethical Implications

Data Bias

Need to be very careful: biased data -> biased models

The digital divide

Tracking People

Advantage with these methods is we don't need to track people

Models work with counts of flows

Although may need work on mapping from the data to the model domain (e.g. Lueck at al. (2019))

Summary - Towards Live* Simulations

Potential for large-scale, agent-based models to transform policy making

Pollution, economy, disease spread, ...

But need to better understand the uncertainty in our models

What do we need?

Uncertainty quantification for ABMs

Efficient data assimilation methods for ABMs

High-resolution data (the identifiability problem)

Better models ?

*Swarup, S., and H. S. Mortveit (2020) Live Simulations. In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems, 1721–25.

SocSimFest 2021 webpage banner

Quantifying the uncertainty in agent-based models

(aka what the ♠€⅋@ is going on in my model and why?)


Alison Heppenstall and Nick Malleson

University of Leeds and The Alan Turing Institute, UK
A.J.Heppenstall@leeds.ac.uk


Slides available at:
https://urban-analytics.github.io/dust/presentations.html