However, reading RL generalisation research can be challenging, as there is confusion about the exact problem being tackled and the use of terminology. This is because generalisation refers to a class of problems within RL, rather than a specific problem. Claiming to improve “generalisation” without being more specific about the type of generalisation that is being improved is underspecified. In fact, it seems unlikely we could improve all types of generalisation with a single method, as an analogy of the No Free Lunch theorem may apply.

To address this confusion, we’ve written a survey and critical review of the field of generalisation in RL. We formally describe the class of generalisation problems and use this formalism to discuss benchmarks for generalisation as well as methods. Given the field is so young, there’s also a lot of future directions to explore, and we highlight ones we think are important.

Figure 1: A visualisation of three types of environment (columns) with respect to their graphical model, training and testing distribution and example benchmarks (rows). Classical RL has focused on environments where training and testing are identical (singleton environments, first column) but in the real world training and testing environments will be different, either from the same distribution (IID Generalisation Environments, second column) or from different distributions (OOD Generalisation Environments, third column).

Formalising Generalisation

So what exactly do we mean when we say generalisation in RL? All research in this area works in a setting where we have a collection of tasks, levels or environments, and generalisation is measured by training and testing on different subsets of this collection. In the paper, we describe a formal object, the Contextual Markov Decision Process (CMDP), which can capture all of these different settings. This formalism builds on and combines previous works formalising generalisation. Each task, level or environment is determined by a context, which for example could be a random seed for a procedural content generation process (as in OpenAI Procgen), a task id (as in Meta-World) or a parameter vector for a controllable simulation and rendering process (as in Distracting Control Suite).

To test for generalisation, we then split the set of contexts into training and testing sets. The policy trains on the training set, and is then evaluated on the testing set, which generally contains unseen contexts, and hence unseen levels, tasks or environments. A generalisation problem is hence specified by a CMDP and a choice of training and testing context sets. As generalisation can be a vague term, we call this problem zero-shot policy transfer, again building on terminology from previous works.

Figure2: Visualising a CMDP. The state is split into an underlying state s’ and a context c, and this context stays the same within episodes but varies between them. Often the policy only observes the underlying state and not the context.

The set of assumptions we make here is very general. This makes it impossible to guarantee generalisation performance given only these assumptions, and also makes it unlikely that a method can improve generalisation across the whole space of problems. This reinforces the point that generalisation is a class of problems, and researchers in this field should make clear the additional assumptions they’re making when developing methods and benchmarks. We describe a collection of possible additional assumptions that can be made in the paper.

Benchmarking Generalisation

Now we have a formal way of specifying a generalisation problem, we can talk about the range of benchmarks for generalisation. Thinking hard about the benchmarks in the field is important, as they define what progress looks like and which problems get solved, especially in RL where we normally can’t test our algorithms on real-world problems. We describe benchmarks as a combination of an environment (a CMDP) and an evaluation protocol (a choice of training and testing context sets).

There are a lot of different environments which enable testing generalisation in RL, and we categorise them all in the paper according to style, context set, and variation. See Table 1 in the paper for more details.

The key attribute of interest in environments is the structure of the context set. Environments with a purely black-box procedural content generation (PCG) approach to creating new levels (such as OpenAI Procgen or NLE) have just the set of all seeds as the context set. This set has no additional structure which we can use, which limits the range of possible evaluation protocols. In fact, the only choice between protocols is the size of the training set (see Figure 3).

Figure 3: Visualisation of Evaluation Protocols for PCG Environments. A uses a single training context, and the entire set for testing. B uses a small collection of random training contexts, and the entire set for testing. C reverses this, using the entire set for training apart from random held out contexts that are used for testing.

In contrast to purely PCG approaches, many environments have a structured, controllable context set (such as Distracting Control Suite or Meta-World). These context sets are generally a product of multiple factors of variation such as gravity or friction coefficients, visual distractions or object colour and quantity. In this setting, we have a lot more options when it comes to choosing an evaluation protocol, which enables us to test for a variety of different kinds of generalisation, from fully interpolating within the context set, through combinatorially interpolating to unseen combinations of seen factors, to extrapolating out of distribution on single or multiple factors (see Figure 4).

Figure 4: Visualisation of Evaluation Protocols for Controllable Environments. Each diagram visualises one training set and multiple testing sets. A chooses the range for each factor independently in training, resulting in a convex shape. Testing distributions ca be interpolation (red), extrapolation along a single factor (either green square) or extrapolation along both factors (blue). In B and C, the ranges for each factor are linked, resulting in a non-convex shape for the training distribution. This enables testing combinatorial interpolation (yellow), where factors take values seen during training independently but not in combination. The difference from B to C is width of the training distribution on the axes we expect the agent to generalise along. In C the policy can’t learn that the two factors can vary independently, making generalisation harder. In realistic settings settings this space will normally be higher than two dimensions and contain non-continuous and non-ordinal axes.

One of the key takeaways from the survey is that purely PCG environments aren’t useful for making progress in generalisation research, as they only test a “general” kind of generalisation without disentangling why methods succeed and fail. We recommend a combination of controllable and PCG context sets: using controllable factors of variation at a high level to enable precise scientific experimentation to understand how methods perform, and using PCG at a lower level to generate the sufficient diversity required for policies to generalise without hand-crafting each level separately.

Methods for Generalisation

Of course, as well as thinking about the problems, we want to understand how we can solve them. This leads us to our categorisation of methods for improving generalisation in RL. A generalisation problem occurs when the training and testing context sets are different, and the policy then learns to rely on features of the training environments which may change at test time. Conceptualising generalisation this way leads us to classify generalisation methods into three categories: Those that try and increase the similarity between the training and testing data and objectives, those that explicitly handle the difference in features between training and testing, and those that handle RL-specific generalisation problems or optimisation improvements (see Figure 5).

Figure 5: Categorisation of methods for tackling generalisation in reinforcement learning

You can see the paper for the full breakdown of these categories and the methods within them, but for now we’ll highlight a few key areas where we believe exciting future work could be done.

The first is environment generation. This research direction aims to produce methods for actively sampling from context sets so to enable efficient training in this space and generalisation across the full range of levels in the context set. Environment generation methods aiming to maximize the regret of the student agent hold promise in generating curricula that cover the full space of challenges in the context space. When this context space is unbounded, such methods can lead to an open-ended learning process. Early examples include POET and PAIRED, and there is exciting work happening in our lab improving on these methods by understanding how PAIRED and curriculum methods like PLR can be understood as two sides of the same coin, as well as improvements to the generation process, such as adding an editing capability to the environment generator and preventing potentially harmful selective-sampling bias due to the curriculum.

The next area we want to highlight is fast online adaptation. While there’s lots of work on Meta-RL, much of it assumes access to multiple training episodes before evaluation, which isn’t the setting we’re interested in. However, several methods do adapt online within a single episode, using either a hard-coded or learned update rule. This work will be especially useful in more challenging settings where adaptation will be necessary to get good performance (for example in reward-function or dynamics variation settings).

Finally, there’s very little work tackling RL-specific issues for generalisation. Recent work showed that the non-stationarity of the data distribution for RL negatively affects generalisation performance, but more investigation into problems of generalisation which arise in the RL setting specifically will be beneficial. For example, the use of bootstrapping and TD learning, the exploration-exploitation tradeoff and the high variance of data collection and learning could all interact with generalisation performance in ways we don’t yet understand.

A table capturing all the methods we discuss in the paper. Looking at the primary adjustment made by each method, green normal-text methods adjust the training environment, red monospace-text methods adjust the architecture, and blue italic-text methods adjust the loss function.

What’s Next for Generalisation?

So that’s where the field is at right now. But what next? There are many different research paths to be explored, and we highlight a few crucial avenues here.

Firstly, offline RL seems particularly useful in applying RL to the real world, but almost all work in this area tackles tasks where the offline data is collected from the same environment that’s used for evaluation. Jointly tackling generalisation and offline RL will be important in applying these methods to a diverse set of real-world scenarios.

Thinking further about realistic problem settings and deployment scenarios, continual RL is another extension of the generalisation problem which is particularly relevant. Policies will be deployed in a setting where they continue to see more data and where the tasks and environments change, and developing methods that can thrive in this setting is needed.

Finally, more research on handling generalisation to novel rewards or goals is necessary to have systems that can do a variety of tasks in a variety of environments, rather than just a single task in a variety of environments. This will involve understanding the best way to specify goals to an agent, trading off between approaches that make RL optimisation easier and those that are usable for humans to specify goals.

This was just a quick peek of everything we cover in the survey, but there’s lots more useful information in the paper such as a discussion of compositional generalisation, context-efficient learning, inductive biases, and further structural assumptions for improved generalisation. Check it out for all the gory details of generalisation in RL!

Thanks to Amy Zhang, Minqi Jiang and Tim Rocktäschel for comments on drafts of this post. This post is based on A Survey Of Generalisation In Deep Reinforcement Learning, which was joint work with Amy Zhang, Edward Grefenstette, and Tim Rocktäschel.

Introduction

We’ve previously written about what interpretability research might be. In this post we think about how different kinds of interpretability research (even loosely formulated) can help AI alignment research agendas and proposals. It seems that there are meaningful differences in the kind of tools and research different agendas would benefit from, and we aim to make these differences clearer. This is useful in helping prioritise what kinds of interpretability research are likely worth doing.

Framing

• In solving the problem of AI Alignment, we’ll need to both answer research questions (e.g Is it optimal to tell the truth in a debate game?) and work out how to complete tasks reliably and quickly (e.g. Given an AI, tell me whether it will allow us to turn it off).

  • The research questions we only need to answer once (and hence can spend significant research effort on), whereas completing one of the tasks can’t be too intensive to make it feasible to complete it regularly, but will require strong tools or methods.

• Interpretability research will be useful for alignment by enabling and enhancing other research proposals and agendas, and so one of the best ways of thinking about what kind of interpretability research to do is to think about other proposals and how specifically interpretability can help them. We see three broad ways interpretability could enable other proposals:

  • Developing a more formal theory of interpretability and explainability could help with avenues such as open source game-theory and mechanistic transparency, where some sense of Interpretation or Understanding is required in an algorithmic sense.

  • Theoretical exploration of components or tools, such as exploring viable general interpretation methods and their desiderata, future strong versions, promises & limits. This will help in understanding how we may be able to use interpretability in the future in other proposals, in scenarios we don’t yet encounter or haven’t considered. This side of interpretability is likely valuable long-term and neglected by the current mainstream ML research. In a sense this post is the result of trying to do this kind of research.

  • Enabling or amplifying the insights gained from experiments performed by researchers (both alignment and mainstream) today, helping them develop answers to their proposals’ research questions.

Different types of interpretability can help different areas of AI alignment

As expressed in the previous section, interpretability can help with a wide range of different research agendas and proposals in different ways. We want to give a few examples examining this idea, and then draw some general conclusions from the patterns expressed in these examples.

Iterated Distillation and Amplification (IDA)

There are a variety of research questions that this proposal will need to tackle to succeed:

• How do we build a Distillation process that preserves alignment?

  • Does Distillation of form X preserve alignment?

  • What properties does distillation of form X preserve?

• How do we build an amplification process that increases the capabilities of the AI while not diminishing the alignment?

  • Is the Factored Cognition hypothesis correct?

• etc. etc etc.

A key problem which we need to be able to solve for IDA to be safe is to consistently be able to validate whether the distilled agent’s behaviour is faithful to the overseer’s behaviour (or the behaviour the overseer aims to incentivise). We think interpretability research is a key way in which we could perform this validation. Methods which work with the form of agent produced by IDA would be necessary; this might be RL agents or another paradigm not yet encountered. An interesting feature of these methods would be that they only need to work comparatively: they might not need to say something about the distilled agent’s behaviour in isolation. We can use our method to validate the changes at each iteration, and by assumption we have faith in the initial agent, so by similar reasoning to IDA working in the first place, we’d be happy with the final agent even if we can’t completely understand it in isolation.

In terms of the research questions which interpretability might be able to help with, If we start doing experiments with amplification and distillation of AIs, then using interpretability methods on the produced agents might help us generate good intuitions or even proofs of how different properties are changed in the amplification and distillation stages. This is quite generic, and It’s plausible there’s limited work to be done in the true case, as a key part of the process is the human (or collection of humans) being amplified. Human interpretability is a whole different subject from AI interpretability, and while there might be some overlap (systems biology & neuroscience and deep learning have learned from each other before), it’s uncertain whether that kind of work is necessary.

AI Safety Via Debate

For an explanation of this agenda, see here and here. For debate, there’s a broad range of research questions which would need to be answered, including questions (often related to the ones in IDA) about whether Debate works and produces safe AI, such as Does debate incentivise truth-seeking behaviour or just maximally persuasive behaviour?. We currently think it’s less likely that AI interpretability can help with these kinds of questions; game theory and the social science seem best posed to answer them.

It also includes tasks we’ll need to be able to perform repeatedly and reliably. In the current technical research on debate, the debate game is often set up with one debater arguing for a truthful statement, and the other arguing against said statement. In this setting, there are questions such as

• Is this debater telling the truth or lying?

• Is this debater honestly presenting the evidence it’s presenting or not?

I believe interpretability methods can help the judges in the debate games answer these questions more effectively, by enabling them to understand what is influencing a debater’s action at a specific time. Work such as Influence Functions could enable us to see which training data is most impacting the current AI’s decision, which could give us insight into what the agent’s current motivation is (i.e. if the most influential training data was a time when the debater lied and got away with it, winning the debate, it seems more likely that the debater is currently lying). In the general case, the debaters might not know the truth value of the statement they’re arguing for. Even in this case if our interpretability methods can spot when an agent is trying to be deceitful we can punish this, to try to encourage truth-seeking behaviour. (NOTE: This depends a lot on the operationalisation of “trying to be deceitful”, and it’s unclear whether using interpretability in this way would encourage truth-seeking behaviour or just cleverer deceitful behaviour). In both the case of knowing the truth value of the debate and not, if the judge is another AI, then our interpretability methods would only need to enable that AI to gain insight, not humans.

Other methods may also be able to help with these kinds of problems; If we can understand the internal structure of the debater in some way, then see which parts of the internals are activated when it acts in a certain way (effectively an MRI for a neural network), then we might be able to understand when it’s lying or telling the truth.

In general these methods need only be local (applied to a debater’s actions at specific time steps) to still enhance and empower the judge.

Recursive Reward Modelling

See here for an explanation of this alignment proposal. The main idea is that we learn a reward model from humans which we can then use to teach AI. We recurse on this setup by using the AI we’ve trained so far to help assess the behaviour of the new AI, to try and help us understand whether it’s behaviour is good or bad, so the reward model we learn is as good as possible.

In terms of research questions that this method has, they’re in a similar vein to the IDA proposal, but with more of a focus on AIs in the recursion. This might make it more amenable to interpretability methods helping us to find out whether reward modelling is a solution which works well for creating strong aligned AI. If we’re able to build reward models and examine their internals, we can see how well reward modelling tends to work in different scenarios. On the recursive side, having a consistent insight into the differences between the successive reward models we create would be useful. As in IDA, we may only need to look at the difference between reward models and AIs rather than understanding them on their own.

In terms of tasks interpretability could help with, one possibility is interpretability methods to be used by the helper AI in inspecting the behaviour of the current AI, to induce a loss function not just on it’s behaviour but also it’s internal reasoning. If we had interpretability methods which are strong enough to give this kind of insight then this could be a strong increase in the effectiveness of this method. We would need transparency techniques to allow the helper AI to get a good enough insight on the current AI’s reasoning to induce some kind of reward signal, and methods which actually generate this insight. In this case humans might not ever see the output of the interpretability method, as it’s just used by the reward model to judge the AI’s reasoning.

Mesa Optimisation

For an explanation of mesa optimisation, see here. This is one of the most convincing (I believe) examples of how interpretability can be useful, and is concerned with the problem of inner alignment. The classic question here is something like Does this agent contain an unaligned mesa-optimiser?. More generally, Does this training process produce agents which contain unaligned mesa-optimisers?.

I think the research questions around mesa optimisation are ones that interpretability methods will definitely help us solve; this problem is about how the training procedure of an AI unfolds, and how the internals of the AI reason. Insight could be gained on these kinds of problems through a broad range of methods: Influence Functions, Loss Change Allocation and other methods which introspect on the training procedure seem particularly useful. We think transparency approaches (such as those motivated in An analytical perspective on AI alignment) also have a good chance of enabling the insights we want in a way which doesn’t require running the agent (in simulation or not) as it doesn’t aim to produce insight from behavioural examination.

Principally, sufficiently strong and farsighted optimizers should not produce unaligned mesa-optimisers, as doing so would usually not be optimal in the long term. Insofar as interpretability helps us (or the advanced optimisation loops we build) with predicting model behaviour in novel situations, it could help us to filter out some unaligned mesa-optimizers. There are two caveats: First, this assumes that our optimizers are already quite powerful without the interpretability method, as even unaligned mesa-optimizers might improve generalization compared to weaker models that do not utilize any optimization. Second, filtering some unaligned mesa-optimizers could turn out net-negative, if the remaining ones are especially malicious, since they are relatively more likely to be selected after filtering.

Proposal-agnostic ideas

There are some tasks interpretability might help us with which are proposal-agnostic. We won’t go into a lot of detail on these kinds of ideas here, as I’m focusing on differences between proposals, but we thought it was worth mentioning two ideas.

An often proposed idea for how interpretability could help with alignment it through auditing final models produced by any kind of method. In this setting, once we have produced a model, before deploying it to the real world, we first inspect it with our interpretability tools to make sure it’s not misaligned. This capability probably won’t solve AI alignment on it’s own, as if we discover the AI is misaligned, it doesn’t provide us with a next step apart from training another AI and hoping this one works out. It’s possible that it might be enough, if misaligned models are a small part of the model space, and retraining can quickly get us to an aligned model, but this seems unlikely.

A second idea, put forward by Chris Olah, is that interpretability can improve the field of ML, making it a more rigorous, scientific discipline which tries to truly understand the models we’re creating, rather than just pushing for the best results. This is a long shot, but if we achieved this kind of fundamental reorientation, it seems like it would be very beneficial for AI safety and alignment, as we’d understand how our models worked a lot more (and research which built understanding would be rewarded), rather than just creating really good optimisers with little insight into how they work and what behaviour they’ll have.

Conclusions and Patterns

• Recursive approaches could benefit from methods which compare models, which might be easier to create than methods which look at a single model in isolation

  • This applies to IDA and recursive reward modelling most obviously

• Interpretability seems like it may be able to create a kind of regularisation or optimisation pressure on a trained model to be aligned, and this would be useful in a variety of proposals. In some of these settings a human never has to see the result of the interpretability method.

  • This is different from our previous thoughts, where we said interpretability methods should often have a human in the loop. This is a blurry area, but we think research in this direction would be useful, whether you label it interpretability or not.

• For research questions, proposals which can produce experiments instantiated with ML tools can benefit more from the insights gained from applying interpretability methods to these experiments.

• Most agendas want methods which give a global view of the AI’s reasoning and internal representations. These kinds of methods will probably be harder to create, but more likely to be helpful. Debate seems like it could still benefit from just local explanations or interpretations.

• In general, most agendas focus on creating or understanding agents. The current paradigm for this kind of research is reinforcement learning, and so more research focusing on interpreting RL agents and training procedures could benefit alignment in general.

  • RL might not be the prevailing paradigm when we actually build AGI - but it’s likely to be closer to the prevailing paradigm that supervised learning, as it has certain characteristics, such as path-dependence, and significant non-learned components (e.g. MCTS), that we will need to keep in mind when building interpretability tools.

These differences can help you decide which kinds of interpretability research to work on

What does all this mean? What’s the use in thinking about all of this? We believe this kind of thinking is useful to help prioritise which kinds of research we might want to pursue, both within interpretability and when considering between interpretability and other related fields. If we believe a particular proposal is more or less likely than others to produce aligned AI, then we would preferentially work on interpretability research which we believe will help this proposal over research which wouldn’t, as it wouldn’t be as useful. These thoughts would of course also be influenced by what AI timelines you find most plausible, as this influences which AI Alignment proposal seems most likely to succeed, or which proposal most needs to succeed.

An important consideration is whether the interpretability research which seems useful for alignment is research which we expect the mainstream ML research community to work on and solve suitably. If this would happen, then it seems comparatively better to work on other alignment research which is just as necessary but isn’t being worked on by the mainstream. Currently however, we don’t think this is the case. There’s little research which focuses on interpreting reinforcement learning agents (which seems very relevant to AI Alignment), or with an explicit focus on producing methods which scale to stronger AIs and are aimed at solving the tasks we want to focus on in AI Alignment.

Overall, we think interpretability research, if targeted towards methods and insights which will help with AI alignment proposals or agendas, is a useful set of research directions to pursue. I’ve talked about a few of these directions in this post, but no doubt there are more.

In this post we lay out some ideas around framing interpretability research which we have found quite useful. Our framing is goal-oriented, which we believe is important for making sure interpretability research is meaningful. We also go over a variety of dimensions which we think are useful to consider when thinking about interpretability research. We wanted to have a shared vocabulary when talking about this kind of research, and found that these ideas helped us communicate effectively.

One of our motivations for having these thoughts and discussions is so we can understand the relevance of interpretability to alignment, and to help us think about which categories or dimensions of interpretability research are important for alignment of strong AI. In a coming post we discuss interpretability and alignment, using the ideas from this post and other previous writing on the subject.

Interpretability, transparency, explainability?

A quick note on terminology. Many terms are used to mean many similar things in the interpretability research literature. We think of Interpretability as the broadest possible category, which is why it can be hard to define and operationalise. Two specific terms are used with frequency apart from interpretability - explainability and transparency - and following the definitions in The Mythos Of Model Interpretability (summarised in more detail in the appendix), we’ll use them as follows:

• Explainability is usually a post-hoc action, where we want to understand and explain why a model made a specific prediction (or enacted certain behaviour) after the fact.
• Transparency is normally seen as a more intrinsic property of a model, and is about understanding it’s internal workings and mechanisms. This is often a harder property to achieve. This can often mean a human would be able to “Run an approximation of the algorithm in their head”.

Dimensions for interpretability

We found two dimensions most useful to consider when thinking about interpretability. The first dimension (Goals) is concerned with which goals interpretability research aims to fulfil. The second (Enabling Humans) refers to the idea that interpretability is human-centred, and we feel that interpretability methods normally must have a human involved in interpreting the model for them to be meaningful. We could also consider agents acting on behalf of humans (i.e. in amplification or recursive reward modelling). These aren’t the only axes on which to consider interpretability research (see Other Dimensions), but they are the two we felt were the most useful.

Goals

Possibly the most important dimension to consider is the Goals of interpretability methods. We describe five generic goals (not specific to certain domains). We found that this helps us the most when thinking about what kinds of interpretability methods we should pursue developing and what a specific interpretability method is hoping to achieve. This is descriptive, in that these are (generic) goals researchers have put forward for using interpretability methods, or that interpretability methods have been used to achieve. There’s overlap between these goals, and especially the first three are interrelated, as are the last two.

The goals are:

• Predicting behaviour - This is about being able to predict a model’s behaviour in novel scenarios. This normally will be without having a full formal definition of the scenario (otherwise we should just run it through the model and see what it does). We might care about predicting large classes of behaviour (“The model never acts against human interest”) which is overlapping with the Assurance of properties goal below; we might also care about more specific classes of behaviour (“Will the agent smash the red vase in this gridworld?”).
  • Some current methods which might help with this are Counterfactual States for Atari Agents via Generative Deep Learning and Finding and Visualizing Weaknesses of Deep Reinforcement Learning Agents, as they both visualise what states would produce certain behaviours, and hence enable us to predict behaviours in these states.
  • One way in which this goal can be validated is to test whether it enables users to construct inputs which produce a certain behaviour, rather than predicting the input given the behaviour. This would have to be done without access to the model (otherwise you could just optimise for that behaviour). The two methods above are examples of this kind of approach: Having seen several visualisations of counterfactual states, you could be able to produce counterfactual states on your own without the method, which demonstrates you’ve gained some understanding of the model enough to predict it’s behaviour.
• Assurance of properties - This is related to the Predicting Behaviour goal, but where the goal is to give broader assurances that are better described as properties rather than specific behavioural predictions. In the strongest form, this might take the form of formal verification, but in weaker forms we might be satisfied with visualisations or explanations of behaviour which imply that the model has a certain property.
  • Understanding Black-box Predictions via Influence Functions and Estimating Training Data Influence by Tracking Gradient Descent are both methods designed to find training data which is influential for specific model decisions. If a model’s influential training points for a specific action are unrelated to this action, we might suppose that the model has learned spurious correlations rather than a robust approach. These methods can hence give us a small amount of assurance that the model is reasoning correctly about it’s inputs.
  • Visualising adversarial examples might give us some assurance of the robustness (or not) of the network to adversarial attacks.
  • We could also consider value learning as an interpretability method. In the case of perfect IRL, we can find the revealed preferences of an agent, and having these preferences would allow you to examine some properties of the agent. This would also help with Predicting Behaviour.
• Persuasion of properties - This is similar to the Assurance of properties goal, but with the difference that the aim is purely persuasion of a person, rather than a truthful representation of the model’s properties. Often interpretability methods might be used to convince a non-technical person (perhaps an auditor or a companies machine learning models) that the model has certain properties. This goal can often be problematic, as the incentives come apart from truthfully presenting the behaviour or properties of the model. That is, we could create methods that produce pretty interesting visualisations which seem to show the model is doing the right thing, when the visualisations aren’t actually dependent on the model’s parameters.
  • Sanity Checks for Saliency Maps and The (Un)reliability of Saliency methods both seem to point at the possibility that saliency maps are doing this implicitly. Obviously no researcher is explicitly thinking “Let’s find the prettiest picture to convince people or some interesting property”, but without better forms of validation this seems like it could happen by accident for some methods. This was one of our motivations for coming up with a goal-oriented framing in the first place
  • Note that we’re not saying this a goal one should aim for, and it’s rarely if ever explicitly mentioned as a goal in literature; however, it likely to have been (at least implicitly) optimised for in certain methods, as the above bullet point alludes to.
• Improving model performance - Many methods are designed to give us understanding which we can use to improve the performance of the model. We might find weaknesses in how the model currently performs, or unbalanced training data, which we can use to improve the model’s performance.
  • For example, saliency maps (Visualizing and Understanding Atari Agents, Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization, many others) tell us what the model is attending to in the input, which can tell us whether it’s learning the correct signals and information to attend to. This information might help us improve the model behaviour.
  • When trying to improve the robustness of our models, especially by adversarial training, we might use interpretability methods to find uninterpretable or seemingly un-robust behaviours, and bias our adversarial training towards these behaviours. If the behaviours are reliant on spurious correlations, then we can improve the robustness of our model by training on examples where these correlations break down, and if they in fact are robust, then we may have learned something new about the environment or task which we didn’t realise beforehand.
• Debugging model - Similar to the Improving model performance, and especially in deep learning, this is about finding incorrect model behaviour and fixing it.
  • To clarify the difference from Improving model performance, we might say a model is implemented and executing correctly if it’s implementation is fully faithful to the conception in your head, or to some pseudocode, mathematical formula or design document. In this case, we might use interpretability to Improve model performance by giving us insights into how it behaves.
  • However, we might say a model is implemented (and hence executing) incorrectly if it’s implementation is unfaithful to the pseudocode, mathematical formulation or design document, most likely due to an error in implementation, or a possibly hidden assumption being violated in the training process. For example, we might assume that all the training data is labelled correctly, but it might not be, and we may be able to find this using interpretability methods. This would fall under Debugging models rather than Improving model performance.
  • This might take the form of an interactive debugger or some logging view such as tensorboard, or visualisations of the flow of gradients through the network, to make sure they’re flowing the way you expect.

A key caveat is that many methods might not target a specific goal in the above list, but instead aim to produce in the end-user of the interpretability method some Generic Understanding or a Gearful Model of the object being interpreted. This understanding could then serve in the fulfilment of a variety of goals, such as aiding the user in predicting behaviour, or understanding why the model fails and hence improving its performance. A key example of these kinds of Generic Understanding methods are the series of methods based on Feature Visualisation, especially the Activation Atlas. The Activation Atlas paper is a good example of a method giving Generic Understanding which is then cashed out in the ability to produce adversarial examples aimed at specific things using only overlapping natural images, which is a kind of Predicting Behaviour.

We want to stress, however, that validating whether a interpretability method produces genuine understanding seems very difficult (or perhaps impossible) to do without seeing whether the understanding can be cashed out to achieve one of the goals above. The aim when trying to produce understanding should not be for the user to just feel like they understand the method, but rather for the user’s understanding to enable them to better achieve one of the goals above. If a method produces some fuzzy feeling of understanding but doesn’t seem useful for any downstream task, then it’s worth considering what it’s useful for.

Enabling Humans

The second dimension, Enabling Humans, is about whether the method enables a human to achieve one of the goals above, as opposed to achieving the goal directly. That is, there are many methods or algorithms in machine learning which Improve model performance, such as the Adam optimiser or adding more layers to your neural network. These techniques are obviously not interpretability techniques, but it’s useful to think about the difference between them and a technique we would call interpretability which does enable an improvement in model performance.

We think the key difference is the existence of a human in a chain between interpretability method and in this case Improvement of model performance. An interpretability method will Enable Humans to achieve the goals above. For a method such as automatic verification (which could be considered interpretability) the verifier is doing the vast majority of the work, but the human still needs to decide what to verify, i.e. they need to ask the right questions. On the other hand, methods such as feature visualisation and saliency maps only provide some visualisation of the model behaviour or internals, and hence still require human effort to achieve one of the goals above.

For some of the goals above this is basically guaranteed: When trying to gain Assurance of Properties, we’re the human in the loop, and the method could give us these assurances or enable us to find them for ourselves. However as mentioned above, for Improving model performance the distinction becomes more necessary.

This dimension is more difficult to reason about than the Goals dimension, and is hence less fleshed out. However, we think it’s just as important to consider what the role of the human is in an interpretability method’s use.

A Map of Methods

These dimensions aren’t an operationalisation of interpretability, and don’t present necessary and sufficient conditions, and we’re uncertain whether it’d be useful to have a strict dividing line. We do think that the framing is useful even if it’s not a formal definition of what interpretability is, as it helped us to discuss what goals interpretability methods are trying to achieve, and whether they’re enabling humans to achieve those goals or not. To test out our ideas with a variety of current methods, we produced a map of current interpretability methods on these two dimensions. This helped us in earlier formulations to spot problems with the description of the framing, relations between various parts of it, and similar patterns of goal behaviour from different methods. This is just an illustration of applying some of our ideas to current methods. Bold lines indicate that method helping more with that goal. We don’t want to stake a lot on exactly where the bold lines are drawn.

Other dimensions

The two dimensions described above were the ones we felt were the most useful at a higher level when thinking about generating ideas for interpretability methods. However, there’s a few other dimensions which we think are also important to think about. This can also be useful in a generative sense. If we draw out the matrix of all dimensions, we can consider whether methods in certain points of the matrix would be useful, and how they might look. These other important dimensions are:

• Interpreting the training process vs finished model (vs both). A method might aim to interpret how training happens (such as LCA: Loss Change Allocation for Neural Network Training or Understanding Black-box Predictions via Influence Functions), or it might be interpreting the finished model (such as Feature Visualization).
  • We think this dimension can still be orthogonal to the Goals dimension, as we might want Assurances of properties or Predicting Behaviour of the training process as opposed to the finished model. The distinction is more blurry when it comes to Improving model performance or Debugging models, but we can still say a method looks at the training process of the finished model (or both) to aid in achieving this goal. In an online/meta-learning setup this also not be a crisp definition, as these paradigms have training as part of the “finished model”.
• Level of access to components of training and model. In some settings, we might have access to all the training data, the whole training process (i.e. the mini batches and loss and model checkpoints at every iteration), the final model (i.e. it’s architecture and weights), and a copy of the environment/simulation (especially relevant in reinforcement learning). In this full information setting many more interpretability methods are possible than in a setting where we only have access to the final model (architecture and weights), or even just examples of it’s behaviour.
  • We find it useful to think about what resources the interpretability method requires to produce its results. This is especially pertinent in reinforcement learning, as some methods such as in Understanding DQNs require running the model for many time steps in the environment to gain insight from a behavioural standpoint, but this can be problematic depending on the type and simulatability of the environment.
• Ground Truth Access. Some interpretability methods may rely on access to the ground truth, either from a human’s evaluation or otherwise. In some settings (such as natural image classification) this is the case, but in reinforcement learning we don’t know what the optimal policy in most settings is. This will become more pertinent as we get stronger AIs and they start to overtake humans on domains other than board games.
  • Methods like NetDisect require dense pixel level labellings of images, an even stronger notion of ground truth than just class labels for inputs, and gradient-based saliency methods require the ground truth label to take the gradient with, whereas perturbation based saliency methods and methods such as feature visualisation have no such requirement.

What’s next?

While the ideas in this post helped us deconfuse and discuss interpretability research in general, we think an ideal use-case for the framing (which also motivates us) is talking about interpretability in AI alignment. This problem setting is different from those tackled in the standard machine learning literature: We’ll care about a subset of the goals described above, and the way in which a method enables humans will also deserve more scrutiny. A second post coming soon will focus on this issue, using the ideas in this post and other previous writing on the subject. To give an (obvious) sneak preview, Persuasion of Properties is a goal we would want to avoid explicitly optimising for when thinking about aligning strong AI.

Appendix: Previous Literature

There have been previous works addressing the problem of defining and discussing interpretability research. We here summarise two works which seemed to be the most discussed.

The Mythos of Model Interpretability tackles the problem that interpretability is an under-defined term. The author first presents five desiderata which cover most interpretability researchers’ motivations for doing interpretability research: Trust, Causality, Informativeness, Transferability, and Fair and ethical decision making. In many problems, the problem specification used to train the model (i.e. the loss function, which normally just optimises accuracy) does not capture all of the desired properties of the model (i.e. not discriminating different races, having an understanding of causality, being transferable to other similar problems). Hence, optimising to solve this problem specification will not produce these properties. In these cases, Interpretability can be used to validate that these extra properties hold. This point is made both in this paper and following one.

The paper describes a taxonomy of interpretability methods, with the top level topics of transparency and post-hoc interpretability:

• Transparency is about understanding what procedure the algorithm is implementing, and in its strongest form about being to simulate the algorithm yourself. This definition has problems when we consider superhuman AI, as a human presumably won’t be able to simulate the AI’s algorithm. (See One Way to Think About ML Transparency for an attempt to solve this specific problem by extending the definition from human-simulatable in practice to human-simulatable in theory without constant access to the model being simulated).
• Post-hoc Explanations refers to methods which aim to understand or explain why a machine learning model has made a certain decision, either through examples, visualisation or explicit textual explanation produced by the model itself.

Towards A Rigorous Science of Interpretable Machine Learning echoes the point made in the above paper that interpretability research is aimed at addressing the case where the loss function a model is optimised for doesn’t capture all of the desired properties of the trained model. To the list of possible use-cases or problem underspecification cases it adds having a Multi-objective trade-off or only having trained with a Proxy reward function. It provides a semi-formal description of the variety of ways in which interpretability methods or models can be validated, and the links between these methods and the desiderata or methods described above.

Some papers we haven’t looked into but also seem to tackle a similar topic:

• The Challenge of Crafting Intelligible Intelligence
• Techniques for Interpretable Machine Learning
• Explicability? Legibility? Predictability? Transparency? Privacy? Security? The Emerging Landscape of Interpretable Agent Behavior
• The What, the Why, and the How of Artificial Explanations in Automated Decision-Making
• Explaining Explanations: An Overview of Interpretability of Machine Learning

This work was done as part of the AISRP 2019 https://aisrp.org/