A computational model of second-order social reasoning

A Computational Model of Second-Order Social Reasoning Leendert van Maanen ([email protected]) & Rineke Verbrugge ([email protected]) Department of Artificial Intelligence, University of Groningen P.O. Box 407, 9700 AK Groningen, The Netherlands Abstract Osborne & Rubinstein, 1994). A more complex example is This paper presents the first computational cognitive model of Cluedo (Van Ditmarsch, 2002) in which not all information second-order social reasoning. The model uses a decision tree is known to each player, and players also have to reason strategy to reason about the opponent’s behavior. We about what information they will provide to their opponents hypothesize that a decision tree strategy requires (1) by making a move, in addition to reasoning about optimal declarative memory, and (2) working memory. Declarative moves, for example, “I don't want Alice to know that I know memory is required to retrieve successive reasoning steps, that she has the ace of hearts”. In this paper, we will focus while working memory is required to temporarily store these on a simpler game called Marble Drop in which all reasoning steps while the next step is retrieved from memory. The model fit on data from a social reasoning game supports information about the current game state is known. Marble the validity of the model. This initial result leads to an explicit Drop is equivalent to the well-known centipede game prediction for an experiment in which the reasoning game is (Rosenthal, 1981) and will be discussed in detail in later combined with another task that requires the same cognitive sections. resources as hypothesized by the model. This work is a first step towards understanding higher-order social reasoning What are important questions in social reasoning? from a cognitive modeling perspective. Two issues stand out in studying social reasoning. The first Keywords: reasoning; theory of mind; cognitive models; relates to human performance on games such as Marble ACT-R Drop. Up to this point we have described behavior as Introduction “optimal” or “rational”, but it turns out that humans perform significantly suboptimally on these games as the complexity What is social reasoning? increases (Flobbe, Verbrugge, Hendriks, & Krämer, 2008; The ability to successfully interact with others requires Hedden & Zhang, 2002). Flobbe et al. for example found knowledge on how your actions are going be interpreted by that participants in a centipede game only correctly perform others. Additionally, successful interaction requires the 75.5% of second-order games, whereas they are near-perfect ability to reason about the actions that other people might on the first-order games (97%). take to respond to, or even to anticipate, your own actions. The second issue relates to the role of memory in (Verbrugge, 2009). A term that is often used in connection reasoning tasks. Taking the perspective of others about your with this ability is theory of mind (Premack & Woodruff, own mental states and then incorporating that knowledge in 1978). In this paper we will present a computational your own reasoning must require some form of working cognitive model of second-order theory of mind, calling the memory. In this paper, we will present the first process second-order social reasoning. computational model that explicitly addresses both issues. Contrary to the case of first-order mental state attributions After a brief overview of other models of social such as "she plans to move her queen", second-order social reasoning, we will introduce our model. Then we will reasoning requires the ability to attribute mental states about present the model fit on relevant data and we will discuss mental states to others, as in "she believes that I intend to how this model can contribute new insights in the sacrifice my horse" (Perner & Wimmer, 1985). In higher- understanding of social reasoning. order social reasoning, this ability is recursively applied for successful behavior. The cognitive model presented in this Formal models of social cognition paper will be the first that explicitly addresses higher-order Social reasoning has been formally studied from a number social reasoning. We will present a theory on how people of perspectives. These perspectives differ in the amount of reason in second-order social reasoning games, as well as cognitive validity that is considered. One perspective is to explicit predictions on how behavior changes if the task is study social cognition as an interactive game (Camerer, made more complex. 2003). This game-theoretic perspective assumes that people Second-order social reasoning has often been studied by are rational agents, optimizing their gain by applying use of simple strategic games in which success is only strategic reasoning. However, many experiments have warranted if the players successfully anticipate each other’s shown that people are not completely rational in this sense. moves. A very simple example of such a game is tic-tac-toe For example, McKelvey and Palfrey (1992) have shown that (also known as noughts-and-crosses), in which each player in a traditional centipede game participants do not behave has all information available on the playing board, and rationally. In this version of the game, the payoffs are players have to take into account what the optimal move is distributed in such a way that the optimal strategy is to for the opponent (that is, games of perfect information, always end the game at the first move (i.e., Nash 259 equilibrium, Nash, 1951). However, in McKelvey and architecture to make explicit predictions for a particular Palfrey’s experiment participants continued the game for social reasoning task. some rounds before ending it. One interpretation of this Two modules of ACT-R deserve extra attention in the result is that the game-theoretic perspective fails to take into light of our model of second-order social reasoning: the account the reasoning abilities of participants. That is, due declarative memory module and the problem state module. to cognitive constraints such as working memory capacity, The declarative memory module retrieves information from participants may be unable to perform optimal strategic long-term memory, called chunks. Each chunk in memory is reasoning, even if in principle they are willing to do so. represented by an activation value that represents the A different perspective, that focuses on cognitive validity likelihood that that item can be retrieved. If the activation in developing formal models, is that of a cognitive value drops below a certain minimal value (the retrieval architecture (Anderson, 2007; Newell, 1990). Cognitive threshold), the related information is no longer accessible. models developed within this framework aim to explain In that case, the system will report a retrieval failure after a certain aspects of cognition by assuming only general constant time factor. If the activation value is above the cognitive principles. However, the current cognitive models retrieval threshold, the information is accessible. However, that describe social interactions do not take second-order the time needed to retrieve it from memory depends on how reasoning into account. For example, cognitive models of active the item actually is. The more active, the faster the simple games exist in which it is important to know the retrieval will be. Connected to the declarative memory opponent’s behavior (e.g., Lebiere & West, 1999; West, module is a retrieval buffer, which may contain one Lebiere, & Bothell, 2006). These cognitive models (retrieved) item at a time. If another item is retrieved, it is demonstrate that declarative memory is important in playing stored in the retrieval buffer, with the previous item being strategically. In the current work however, we are less pushed back to long-term memory. interested in how people adapt their strategy to an opposing The problem state module (sometimes referred to as the strategy, but rather we are studying the cognitive limitations imaginal module) contains a buffer in which information of explicit second-order reasoning. Related to this, Hendriks can be temporarily stored. Typically, this information and colleagues (e.g., Hendriks, Van Rijn, & Valkenier, contains a subsolution to the problem at hand. In the case of 2007; Van Rij, Van Rijn, & Hendriks, in press) have studied a social reasoning task, this may be the outcome of a the development of first-order theory-of-mind in language reasoning step that will be relevant in subsequent reasoning. using computational cognitive modeling. Storing information in the problem state buffer is associated with a time cost (typically 200ms). The model that we An ACT-R model of social reasoning present in this paper relies on the combination of the To provide a full model of second-order social reasoning, declarative module and the problem state buffer. That is, the we implemented our model in the cognitive architecture model retrieves relevant information from memory and ACT-R (Anderson, 2007). ACT-R aspires to explain all of moves that information to the problem state buffer if new cognition using one theoretical framework. To achieve this, information is retrieved from memory that needs to be the heart of ACT-R consists of a procedural memory stored in the retrieval buffer. system, which contains condition-action pairs known as production rules. Besides the procedural module, ACT-R Marble Drop game has designated modules for specific types of information. To study the reasoning processes that are involved in social For example, the visual module processes visual reasoning, we developed a cognitive model of a reasoning information, whereas the declarative memory module game in which in order to play optimally the players have to processes declarative or factual information. Each module anticipate each other’s moves. The particular game that was has a buffer that may contain one unit of information (a analyzed and modeled is a variant of the centipede game chunk). If the current contents of all buffers in the system called Marble Drop (Meijering, Van Maanen, Van Rijn, & matches the conditions of a particular production rule, that Verbrugge, 2010). rule fires and its actions are executed. Each action may refer Marble Drop is a marble run game containing trapdoors to an operation in one of the modules. (Figure 1). Players take turns in deciding whether to open This general layout of the cognitive system enables the one trapdoor or the other. In each turn, opening one trapdoor development of models in which different kinds of leads to the end of the game, whereas opening the other information can be processed at the same time, while each trapdoor means that the game continues to the next bin on module can only process one unit of information at a time. the right and the opponent may choose which trapdoor to Based on this feature, ACT-R predicts specific interference open. If a player decides to end the game, both players effects if different aspects of a task require the same receive the credits that are associated with that stage of the cognitive resource at the same time (e.g., Borst, Taatgen, & game. If a player decides to continue the game, the players Van Rijn, 2010; Van Maanen & Van Rijn, 2010; Van traverse to a new stage with which new credits are Maanen, Van Rijn, & Borst, 2009). In the discussion section associated. Because all credits are known in advance, both of the current paper we will use this feature of the players can reason about their opponent’s possible moves further on in the game. The players can do this by applying 260  Finally, chunks representing ordinal information are stored in declarative memory. This means that the model contains knowledge on the relative magnitudes of each combination of payoff values. A model run starts with the initial comparison of two payoff values (Figure 2). For second-order games, that initial comparison is always a comparison between the player’s own payoffs in Bins C and D. First, it retrieves from declarative memory where the first payoff is located on the screen (Bin D in Figure 1). If it retrieves that knowledge, the model attends Bin D and tries to retrieve the magnitude of the observed payoff. At the same time, the model stores the current comparison in the problem state Figure 1. The interface of a second-order Marble Drop buffer, to free the retrieval buffer for the upcoming payoff game. Color shades of the marbles in the experiment are information. represented by numbers. Because in the experiment the payoffs are represented by backward induction (Van der Hoek & Verbrugge, 2002; shaded marbles, the model has to retrieve the value Verbrugge & Mol, 2008). For example, a player can reason corresponding to the observed shade. Next, the model that his opponent wants the highest payoff in bins C and D. retrieves the location information for the other payoff value As a result the player knows the maximal payoff that he can that is part of the current comparison. Again to free the get from bins C and D, and can then compare that retrieval buffer, the payoff value of the first payoff is stored information to his own payoff in bin B. If it is possible in a in the problem state buffer. The payoff is attended and the particular game for a player to behave optimally by directly corresponding value is retrieved from memory. Finally, the predicting its opponent’s actions, we refer to this game as two values are compared by trying to retrieve a chunk with being first-order. In a second-order game it is necessary to ordinal information from memory. Based on the outcome of predict the opponent’s predictions of ones own actions in this retrieval the model now retrieves a new payoff order to behave optimally. In principle, Marble Drop games comparison. For example (Figure 1), if the value in bin D could be developed for third-order or even higher-order was smaller than the value in bin B, the model attends the games. payoff in bin B, and compares that with the payoff in bin A. If the value in bin D was larger than the value in bin B, then The Model the model attends the opponent’s payoff in bin D, and The model follows a backward induction strategy to predict compares that with the opponent’s payoff in bin C. The the opponent’s moves further on in the game. Hedden and model continues to compare payoffs following the decision Zhang (2002) provide a decision tree analysis of this tree (Hedden & Zhang, 2002) until it reaches the bottom of process for their matrix version of the game.1 The model has the tree. There, it decides its action based on the final knowledge on how to solve Marble Drop games for all comparison. possible distributions of payoffs over the bins of the marble run game. That is, the model stores chunks containing Model fit The model was tested against data from a Marble information on which payoffs to compare at each step. In Drop task (Meijering et al., 2010). In the experiment the addition, chunks representing the magnitudes of the payoff participants were asked to solve zero-order, first-order, and shades are stored in declarative memory, as well as chunks second-order Marble Drop problems. In all these conditions, representing the location of the payoffs on the screen. participants were instructed to indicate the optimal first Figure 2. Flow chart of the model activity in ACT-R modules. The width of each box denotes the duration of each stage. Arrows indicate possible next actions. 1 an analysis that shows the logical equivalence of these games can be found at http://www.ai.rug.nl/~leendert/Equivalence.pdf 261  Figure 3. Model fit to data from Meijering et al. (2010). Left: Response time, Right: Accuracy. move as quickly as possible. That is, even in second-order second-order social reasoning. The model can be considered games participants had to make only one choice. However, as a cognitively plausible implementation of that analysis. because the opponent always played rationally (and the participants were informed of this), there was always only Model predictions one optimal choice. Our model can be used to provide explicit predictions Figure 3 presents the model fit on both response times regarding the use of memory in second-order social and accuracy of the first moves. The fit on the response cognition (Verbrugge, 2009). In particular, the model relies times is very good (R2 = 1.0; RMSE = 0.42 s). The fit on the on various declarative memory retrieval steps, in accuracy data is slightly less (RMSE = 0.067, R2 = 0.2), but combination with storage of information in a problem state this may be attributed to lack of data, making the estimated buffer. An explicit prediction would be that second-order means less reliable.2 theory of mind reasoning would be affected by performing As the order of the Marble Drop reasoning problems another task at the same time that would require the same increases, the model requires more time to respond. This is resources (Borst et al., 2010). To our knowledge, such an because more comparisons have to be made, and therefore experiment has not been done yet. Therefore, in the more information has to be retrieved from declarative remainder of this paper we would like to propose such an memory and stored in the problem state buffer. These steps experiment, combined with explicit, quantitative predictions take time, increasing the response time for higher-order provided by the model. By providing the predictions of our reasoning problems. Because of the similar behavioral model before actually doing the experiment, we counter the patterns between model and data, this study supports the criticism that insufficiently constrained cognitive models view that participants in this task follow the same reasoning can be made to fit any dataset (Roberts & Pashler, 2000). steps as the model does. That is, participants in a social A task that would require the same resources as reasoning game follow a decision tree to make the correct hypothesized for social reasoning is a tone counting task. decision. Participants are presented with tones of two different pitches and are requested to count the number of tones for each Discussion & Predictions pitch. This task would tap into the same cognitive resources as hypothesized for the Marble Drop reasoning task, as First model of second-order social reasoning maintaining two counters at the same time can be The ACT-R model of second-order social reasoning considered a heavy working memory load. A control described in this paper is the first cognitive model to condition in this task would be one in which participants account for second-order social reasoning. Other cognitive would not need to maintain a counter, but rather just say models in the field of social reasoning have either not “high” or “low” every time they heard a tone of a particular explicitly addressed orders of reasoning (e.g., Lebiere & (higher or lower) pitch. Because the control task does not West, 1999; West et al., 2006), or have focused on first- require maintaining a counter (a problem state), concurrent order reasoning only (e.g., Hendriks et al., 2007; Van Rij et execution of this task and the social reasoning task does not al., in press). pose a conflict, and the different stages of the tasks could be Because the model is based on Hedden and Zhang’s interleaved without much loss of time (Anderson, Taatgen, (2002) decision tree analysis of behavior in 2x2 matrix & Byrne, 2005). games, the model provides support for their theory of A dual-task model of social reasoning A simple model of this task would involve maintaining the current counter in a 2 problem state buffer. In addition, the model would – upon As the data presented here are actually the practice block of the hearing a tone – check whether the pitch of the tone is the experiment performed by Meijering et al. (2010), the number of same as the pitch of the previous tone. Specifically, the observations per participant was 4 for zero-order games, and 8 for first and second-order games. model compares the pitch of the tone with the pitch 262  Figure 4. Model predictions for the dual-task social reasoning task. Left: Response time, Right: Accuracy. associated with the counter in the problem state buffer. If the model predicts an increase in the response time, and no this is the case, the model then retrieves the subsequent change in accuracy. This is because the single response task number of the stored counter from memory. If this is not the used as secondary task in the Control condition does not case, the model retrieves the other counter from memory, share any resources with the Marble Drop task. Thus, and based on that retrieves the subsequent number. responding to the tones only adds time to the Marble Drop Such a model would require both the problem state buffer response, but does not change the difficulty of the task. In and the retrieval buffer, resulting in interference with contrast, the tone counting task that the model performs in performance on the Marble Drop game. For the control task, the Interference condition adds considerable time to the both the retrieval and the problem state resources are not response. In addition, the accuracy of the model decreases required. The model of the control task consists of a simple as well. Moreover, the mean response time in the stimulus response mechanism: When a tone of a particular Interference condition increases dramatically to 27s (Figure pitch is heard, the model responds with a vocal response 5), whereas the mean response time in the control condition (either “high” or “low”). is 8.3s, which is only slightly above the mean response time We adapted our model to also perform the tone counting of the single response task (7.7s). Our interpretation of these task. The model was extended with a control mechanism results is that the tone counting task and the Marble Drop that maintained which task was currently given preference task share a cognitive resource. In particular, both tasks (Salvucci & Taatgen, 2008). The model performs the require a problem state buffer for maintaining intermediate Marble Drop task until a tone is presented. At that point a results. Swapping these problem states takes extra time and switch is made to the counting task. If necessary, the model is prone to errors, explaining the increased reaction times tries to retrieve the current count and restore the problem and the decreased accuracy. state of the counting task. Then, it retrieves the subsequent number from declarative memory followed by a vocal Conclusion response saying the number. After that, the model tries to This paper presents the first computational cognitive model restore the problem state of the Marble Drop task by of second-order social reasoning. The model uses a decision retrieving a comparison from memory. tree strategy to reason about the opponent’s behavior in a social reasoning game. We hypothesize that a decision tree Model predictions We ran the second-order reasoning strategy requires (1) declarative memory, and (2) working model in three conditions for a sufficient number of trials to memory. Declarative memory is required to retrieve obtain a stable estimate of the predicted response. In the successive reasoning steps, while working memory is first condition (Single) the model only performed the required to temporarily store these reasoning steps while the Marble Drop task. In the Control condition, the model performed the Marble Drop task in combination with the simple response task. The tones were presented with stimulus onset asynchronies (SOAs) of 2s, 5s, 8s, 11s, 14s, 17s, 20s, and 23s. Only those tones were presented that preceded the model response on the reasoning task. In the Interference condition, the model performed the Marble Drop task in combination with the tone counting task. The tones were presented similarly as in the Control condition. Figure 4 presents the predicted reaction time and accuracy of the dual-task model as a function of the number of tones presented. The left-most data point in each graph (where the number of tones is zero) represents the behavior of the model under single-task conditions. This is the same as the model fit presented in Figure 3. For the Control condition Figure 5. Predicted mean response time for the dual-task model. Left: Response time, Right: Accuracy. 263 next step is retrieved from memory. We implemented Cognitive Science Society. Austin, TX: Cognitive Science working memory as a problem state buffer using the ACT-R Society. cognitive architecture (Borst et al., 2010). The model fit on Nash, J. (1951). Non-cooperative games. The Annals of data from a social reasoning game called Marble Drop Mathematics, 54(2), 286-295. (Meijering et al., 2010) supports the validity of the model. Newell, A. (1990). Unified theories of cognition. This initial result leads to an explicit prediction for an Cambridge, MA: Harvard UP. experiment in which the reasoning game is combined with Osborne, M., & Rubinstein, A. (1994). A course in game another task that requires the same cognitive resources as theory. Cambridge, MA: MIT Press. hypothesized by the model. In particular, if the other task Perner, J. & Wimmer, H. (1985). "John thinks that Mary also requires the problem state resource, the interference of thinks that...": Attribution of second-order beliefs by 5- to that task is substantial. On the other hand, a secondary task 10-year old children. Journal of Experimental Child that is equivalent but does not require the problem state Psychology, 5, 125-137. resource exhibits minimal interference. This work is a first Premack, D., & Woodruff, G. (1978). Does the chimpanzee step towards understanding higher-order social reasoning have a theory of mind? Behavioral and Brain Sciences, 4, from a cognitive modeling perspective. 515-526. Roberts, S., & Pashler, H. (2000). How persuasive is a good Acknowledgements fit? A comment on theory testing. Psychological Review, This research was supported by NWO Vici grant NWO- 107(2), 358-367. 277-80-001 awarded to Rineke Verbrugge. Rosenthal, R. (1981). Games of perfect information, predatory pricing, and the chain store. Journal of References Economic Theory, 25, 92-100. Salvucci, D. D., & Taatgen, N. A. (2008). Threaded Anderson, J. R. (2007). How can the human mind occur in cognition: An integrated theory of concurrent the physical universe? New York: Oxford UP. multitasking. Psychological Review, 115(1), 101-130. Anderson, J.R., Taatgen, N.A. & Byrne, M.D. (2005). Van der Hoek, W., & Verbrugge, R. (2002). Epistemic Learning to Achieve Perfect Time Sharing: Architectural logic: A survey. In L. A. Petrosjan & V. V. Mazalov Implications of Hazeltine, Teague, & Ivry (2002). Journal (Eds.), Game theory and applications (Vol. 8, pp. 53-94). of Experimental Psychology: Human Perception and New York: Nova Science. Performance, 31(4), 749-761. Van Ditmarsch, H. P. (2002). The description of game Borst, J. P., Taatgen, N. A., & Van Rijn, H. (2010). The actions in cluedo. In L. A. Petrosian & V. V. Mazalov problem state: A cognitive bottleneck in multitasking. (Eds.), Game theory and applications (Vol. 8, pp. 1-28). Journal of Experimental Psychology: Learning, Memory, Hauppage, NY: Nova Science Publishers. and Cognition, 36(2), 363-382. Van Maanen, L., & Van Rijn, H. (2010). The locus of the Camerer, C. F. (2003). Behavioral game theory: Gratton effect in picture-word interference. Topics in Experiments in strategic interaction. Princeton: Princeton Cognitive Science, 2(1), 168-180. UP. Van Maanen, L., Van Rijn, H., & Borst, J. P. (2009). Stroop Flobbe, L., Verbrugge, R., Hendriks, P., & Krämer, I. and picture-word interference are two sides of the same (2008). Children's application of theory of mind in coin. Psychonomic Bulletin & Review, 16(6), 987-999. reasoning and language. Journal of Logic, Language and Van Rij, J., Van Rijn, H., & Hendriks, P. (in press). Information, 17(4), 417-442. Cognitive architectures and language acquisition: A case Hedden, T., & Zhang, J. (2002). What do you think I think study in pronoun comprehension. Journal of Child you think?: Strategic reasoning in matrix games. Language. Cognition, 85(1), 1-36. Verbrugge, R. (2009). Logic and social cognition: The facts Hendriks, P., Van Rijn, H., & Valkenier, B. (2007). matter, and so do computational models. Journal of Learning to reason about speaker's alternatives in Philosophical Logic, 38(6), 649-680. sentence comprehension: A computational account. Verbrugge, R., & Mol, L. (2008). Learning to apply theory Lingua, 117(11), 1879-1896. of mind. Journal of Logic, Language and Information, Lebiere, C., & West, R. L. (1999). A dynamic ACT-R 17(4), 489-511. model of simple games, Proceedings of the Twenty-First West, R. L., Lebiere, C., & Bothell, D. (2006). Cognitive Annual Conference of the Cognitive Science Society (pp. architectures, game playing, and human evolution. In R. 296-301): Erlbaum. Sun (Ed.), Cognition and multi-agent interaction: From McKelvey, R. D., & Palfrey, T. R. (1992). An experimental cognitive modeling to social simulation (pp. 103-123). study of the centipede game. Econometrica, 60(4), 803- New York, NY: Cambridge UP. 836. Wickelgren, W. A. (1977). Speed-accuracy tradeoff and Meijering, B., Van Maanen, L., Van Rijn, H., & Verbrugge, information-processing dynamics. Acta Psychologica, R. (2010). The facilitative effect of context on second- 41(1), 67-85. order social reasoning. In R. Catrambone & S. Ohlsson (Eds.), Proceedings of the 32nd Annual Conference of the 264