Colloquium

Competition among cooperators: Altruism and reciprocity

Peter Danielson*
Centre for Applied Ethics, University of British Columbia, Vancouver, BC, Canada V6T 1Z2

Levine argues that neither self-interest nor altruism explains experimental results in bargaining and public goods games. Subjects preferences appear also to be sensitive to their opponents perceived altruism. Sethi and Somanathan provide a general account of reciprocal preferences that survive under evolutionary pressure. Although a wide variety of reciprocal strategies pass this evolutionary test, Sethi and Somanthan conjecture that fewer are likely to survive when reciprocal strategies compete with each other. This paper develops evolutionary agent-based models to test their conjecture in cases where reciprocal preferences can differ in a variety of games. We conrm that reciprocity is necessary but not sufcient for optimal cooperation. We explore the theme of competition among reciprocal cooperators and display three interesting emergent organizations: racing to the moral high ground, unstable cycles of preference change, and, when we implement reciprocal mechanisms, hierarchies resulting from exploiting fellow cooperators. If reciprocity is a basic mechanism facilitating cooperation, we can expect interaction that evolves around it to be complex, non-optimal, and resistant to change.

he topic for this colloquium is competition and cooperation as factors in emergent human organization. Sober and Wilson (1) note that the behavior labeled cooperation by evolutionary game theorists is the same as that discussed in the evolution of altruism literature. So our topic can be taken very generally; indeed, it is the central theoretical problem of sociobiology (1). Naturally, the main question is how is altruismcooperation possible among agents selected by competitive evolutionary processes. But this possibility question can be misleading. For example, both Axelrod (2) and Gauthier (3) advanced the discussion of these issues by insisting that there are situations in which cooperative agents responsive to the behavior or disposition of their opponents do as well or better than straightforwardly competitive agents. Yet each defended a single cooperative strategyTit for Tat and Constrained Maximization, respectivelythat is not a unique equilibrium. [Of course, in Gauthiers case, critics point out that Constrained Maximization doesnt even claim to be an equilibrium strategy (4). The extended preference concepts discussed below can be seen as a way to avoid this criticism.] Neither attended to the competition between cooperative strategies and both neglected less nice alternative strategies. In earlier work (5), I extended Gauthiers model to address competition among cooperators, finding that what I called reciprocal cooperators, who demand discriminatory responsiveness from cooperative partners, exploit some more tolerant cooperators, and thereby supplant nicer Constrained Maximizers. More generally, the combination of evolution, altruism, and reciprocity need not result in populations of equal, optimal, and tolerant cooperators. Most generally, because were inclined to favor cooperation, we need methods that challenge our intuitive biases. Evolutionary simulation can be a good test because evolution builds in strong competitive pressure. But simulations can easily confirm biases, unless we allow the generator to range
www.pnas.orgcgidoi10.1073pnas.082079899

quite widely, to include sometimes counterintuitive possibilities. [This is the critical justification for Binmores (6) harsh criticism of Axelrod (2, 7).] In particular, when we specify the mechanisms of reciprocity, we shall see that they have surprises in store for us (8). The present paper runs a similar line of inquiry with a new starting point. We begin with Levines account of human cooperative behavior in experiments using bargaining and public goods games. Although it is commonly agreed that self-interest will not account for these experimental results, Levine goes further and argues that simple altruism is also inadequate. A better fit is obtained from preferences that are sensitive to ones opponents perceived altruism. Thus, Levine argues that a kind of reciprocity of altruism is required to explain human cooperative behavior. Sethi and Somanathan (9) modify the reciprocal component of Levines construction to achieve evolutionary stability under a variety of selective regimes. They raise the question of what would happen were various reciprocal cooperators placed in competition with each other. To attempt to answer this question, we constructed two kinds of models. In the first set of models, agents are capable of quite complex preference interactions. They can be altruistic toward others and react to other agents altruism and reaction functions. The agents in the second set of models have simpler interactions, as they implement working reciprocal strategy mechanisms rather than preferences. We test these agents in a variety of situations. (We use eight commonly discussed two-by-two games.) Although we confirm the basic result that reciprocity allows agents to cooperate where neither self-interested nor simple altruistic agents would, we also discover that some forms of reciprocity lead to unexpectedemergentsocial structures. Some reciprocally altruistic agents can race to the moral high ground, and, ironically, treat less altruistic agents exploitatively. Others are trapped in cycles of preference change. Finally, although reciprocity is necessary to stable cooperation, it need not be optimal. When we implement one family of simple reciprocal mechanisms, exploitative hierarchies emerge as well as equal cooperative outcomes. Previous Work In this section, we discuss only the immediate ancestors of the present paper; see ref. 10 for a broader survey of the literature
This paper results from the Arthur M. Sackler Colloquium of the National Academy of Sciences, Adaptive Agents, Intelligence, and Emergent Human Organization: Capturing Complexity through Agent-Based Modeling, held October 4 6, 2001, at the Arnold and Mabel Beckman Center of the National Academies of Science and Engineering in Irvine, CA. *E-mail: pad@ethics.ubc.ca.
Although it is not surprising that, as agents, we encourage others to cooperate, it must be

stressed that the values cooperation (pareto) optimizes are local (evidently, typically within range of our encouragements). As social scientists, we seek to understand cooperations causes and constituent mechanisms. As applied social scientists (applied ethicists), we will sometimes seek to encourage cooperation and sometimes to undermine it. (Think of the original story of the prisoners dilemma.) So we can account for a bias toward cooperation, but it is a bandwagon we should be cautious about riding. We need good tools to disengage our theoretical ethical intuitions from our moralistic responses and then to isolate, test, and improve new moral mechanisms.

PNAS May 14, 2002 vol. 99 suppl. 3 72377242

on rationality and evolution. This paper builds on two recent streams of research. First, Levine (11) asks to what extent can a simple model of players who are not selfish explain the data from a variety of experiments, including both ultimatum and public goods games? Following Levine, we construct interacting utility functions that have parameters for both pure altruism and a reciprocity factor, . When 0, the model can be regarded as incorporating an element of fairness, . . . One of our major conclusions is that 0 [pure altruism] is not consistent with data from the ultimatum game (11); therefore, according to Levine, experiments reveal that human players are spiteful as well as altruistic and this can be accounted for by preferences that reciprocate something. (That is, preferences that are not exogenous but are instead functions of some feature of the opposing player. Throughout this paper, opposition denotes pairing in games without prejudice to the players interests or preferences; opponents need not be competitors.) The question arises, what do they reciprocate? Second, Sethi and Somanathan (9) introduce an additional test: survival under evolutionary pressure. Their work takes Guth and Yaaris (12) evolutionary approach to preferences: instead of assuming that individual preferences are exogenously given, we think of an evolutionary process where preferences are determined as evolutionarily stable strategies. In this approach, reproductive success is purely a function of the resources earned via strategic interaction, whereas preferences only determine agents moves. I use this approach in two papers (13, ) to evolve both rational preferences that track their interests, and deviant preferences that do not, using only modest assumptions about information. But if we provide more information, agents may evolve preferences that only need not track their objective interests, their preferences may also respond to other agents preferences. Casting these models in terms of preferences instead of strategies resolves some procedural problems (see, for example, ref. 8). So my agent-based model can be used to extend Sethi and Somanathans argument. Sethi and Somanathan argue that the evolutionary demands of the situation further structure the preference function. This function must allow spite toward nonaltruistic materialists, and hence focus on differences in altruism. This paper extends their line of argument, confirming their conjecture: while a wide range of parameter values is consistent with survival against materialists, a much narrower range may be expected to survive when several members of this class of preferences are in competition with each other (9). We show, first, that attending only to differences in altruism leads first to invidious races to the moral high ground. Suggesting further modifications of the function leads to unstable patterns of preference change. Finally, although reciprocity is necessary to stable cooperation, it need not be optimal. Indeed, when we specify a family of reciprocal mechanisms, exploitative hierarchies emerge as well as fair optimal outcomes. Assumptions Here we highlight the assumptions we make that are not standard in the economics literature. (i) We assume that preferences are not exogenous but instead vary under evolutionary pressure. Players choices are fixed by their preferences in the short run of a set of games but over the longer run of the simulation, the distribution of preferences represented in the population are driven by evolutionary pressures. (ii) Altruistic and reciprocal parameters assume that preferences can be compared (11). (iii) These models assume strong
Danielson,

Table 1. Eight sequential game outcomes

OR CO KS AG PD UPD BS CK

3,3 3,3 0,3 2,0 2,2 3,2 1,3 4,4

1,2 0,0 1,2 1,1 0,3 0,3 0,0 1,5

2,1 0,0 2,1 3,3 3,0 4,0 0,0 5,1

0,0 2,2 3,0 0,2 1,1 1,1 2,1 0,0

common knowledge. Players know each others preferences, including their altruistic and reciprocal components. (iv) All players have the same formal preference function, which includes arguments for others altruistic and reciprocal parameters. Players differ only in the value of their altruistic and reciprocal parameters. This assumption may seem inconsistent with our evolutionary approach. Of course, it would be better to have a more constructive model, where the altruistic and reciprocal apparatus evolves. We move in this direction below. Obviously iiiv are strong and unrealistic assumptions. We make them to connect our argument with Levines and Sethi and Somanathans interesting work and to make our own simulations tractable. We are, of course, assuming and not recommending comparability, transparency, and homogeneity. Assumption i is weaker than the norm, and we weaken assumption iv toward the end of this article. Modeling Altruism and Reciprocity Levines model can be viewed as a particular parameterization of a class of models in which preferences depend on payoffs to an individual player and to his rivals, as well as depending on his own type and the type of his rivals (11). Following Sethi and Somanathan (9), where is the material payoff function of a n-player game x, player is utility function ui has the direct component and weight for the opponents material payoff shown in Eq. 1. Because we will use only 2-person games, we can simplify and focus on one opponent, j. u i x x
j

ij j x

[1]

has two components, pure altruistic and reciprocal . Levine relates them by Eq. 2. ij i i i 1 i
[2]

Sethi and Somanathan modify this to allow to go negative, which they call spite, in case playerj is less altruistic than playeri; see Eq. 3.

ij

i i j i 1 i

[3]

P., a paper presented to the Society for Machines and Mentality, Eastern Division of the American Philosophical Association, December 2730, 2000, New York.

Agent-Based Models All of the models below are based on a common coevolutionary framework. Each generation is a round-robin tournament in which 60 agents plays a round of 8 sequential games, ranging from trivial coordination and constant sum to prisoners dilemma (PD), battle of the sexes, and chicken (see Table 1). Each
Danielson

7238 www.pnas.orgcgidoi10.1073pnas.082079899

player plays both roles P1 and P2 (see game tree above Table 1) in each game with each of the other players in each generation. The equilibrium scores (italicized) for one round total 32 and the joint optimal scores (bolded) total 40. Each players fitness is its score in 8 games 2 roles 59 opponents. About half the population is retained (for continuity), and the other half is composed of new genetically programmed offspring (14). In particular, the top 2 players are cloned, 24 players are selected with a chance proportional to their fitness, 30 new players are created by crossover from 15 parents, and 4 new players are single-point mutations of their parents. Parents are selected with a chance proportional to their fitness. The players are lisp functions that output preferences in the first two models and moves in the third. (We used a variant of SCHEME that runs under JAVA on most common computing platforms. Thanks to Ken Anderson, Tim Hickey, and Peter Norvig for this tool. Program available at http:jscheme.sourceforge.net.) Preference Models In models 1 and 2, all players interact rationally based on their preferences. Differences between players are caused by different preferences. Although choice is determined by the players preferences, selection is driven by the material payoffs in the game matrices. Sethi and Somanathan argue that agents with reciprocal preferences have a strategic advantage over altruists or materially selfish agents, a given player with the former preference obtains a greater payoff than an otherwise identical player with the latter preference at any equilibrium (9). Agents with reciprocal preferences will therefore be selected under a wide range of evolutionary regimes. The question remains: what happens when agents differ in altruism or reciprocity? We turn to this question of competition between cooperators now. Racing to the Top First, we implemented Eq. 3. Because Sethi and Somanathan specify that 0 1, we needed to limit the range of . However, to make cooperation possible, our games payoff ranges required 0 2. In contrast, was not subject to an upper limit, but was required to be 0. Each agent is characterized as a pair of values (, ), which, together with the opponents , yield a general preference function which can be applied to the payoffs of each game. For example, consider two reciprocal altruists each with 2 and 2, and the prisoners dilemma in row 5 of Table 1. The material payoffs for P1 and P2 are, in the order (ll, lr, rl, rr), (2, 0, 3, 1) and (2, 3, 0, 1), respectively . The value for each is 0.67. Applying to the outcomes, we get: (3.33, 2, 3, 1.67) and (3.33, 3, 2, 1.67). Notice that this high makes P2 an unconditional cooperator, preferring the left branch in each case (3.33 3 and 2 1.67). It is only the reciprocal element, which requires that P1 also be similarly altruistic, preferring to choose left as well, that leads to a cooperative, rather than an exploitative, outcome. Given their differences, we should calibrate our implementation with Sethi and Somanathans claims. First, a pair of materialists [in our implementation: (0, 0)] will never cooperate and will achieve an equilibrium score of 32. Second, a pair of reciprocal altruistic (2, 2) agents will cooperate in the four social dilemmas, and achieve an optimal score of 40. Finally, the reciprocal (2, 2) agents are spiteful enough to threaten materialists into yielding in many games, with an outcome of 21 to the materialist and 34 to the reciprocators. These outcomes guarantee that reciprocators will invade materialists under individual selection. And this is what happens in our evolutionary simulation.
Danielson

Fig. 1.

Rising reciprocity.

We use a genetic algorithm to drive our evolutionary simulation. To apply a genetic programmer to these simple agents, we only need a set of (closed arithmetic) functions that will raise and lower their parameters. We initialize the population with various pairs of values, and let the function set work on these values [e.g., here is a typical player: (INCA (INCA (INCL BOTH))); programming code (scheme functions and variables) is written in all capitals]. Starting with BOTH (1, 1), using an incremental value of 0.2, yields this pair: (1.4, 1.2). If we seed a population with all materialists, reciprocators immediately invade it. So far, this finding agrees with Sethi and Somanathan. But if we start with a random population of various reciprocators and materialists, we get a surprising result. Typically, will climb to the limit 2, and will continue to increase; see the left scale on Fig. 1. This increase happens because just as a higher allows reciprocators to be spiteful and invade materialists, it also allows higher reciprocators to spitefully invade lower. This process is self-limiting, as once gets too high, cooperation between similar reciprocators falls off. Thus the average score per game of the population rises until jumps. The equilibrium score is 2.0 and the optimum is 2.5; see the scale on the right side of Fig. 1. I write higher and lower in quotes above to alert us to the insidious aspect of this process. Given the way Eq. 3 links and , higher values allow one to exploit ones less altruistic opponents. Of course, according to the high values in the prevalent reciprocal preference function, one would indeed be more altruistic, were ones opponent worthy, but in fact higher values for these parameters allow one to be less altruistic; indeed, spiteful, to lesser fellow cooperators. This observation should warn us about interpreting this model of reciprocal altruism. The ease with which one can misinterpret this type of reciprocity may give us pause. It is not unreasonable for well-intentioned agents to give their betters more leeway out of respect for their higher values. But, unfortunately, evolution (or learning) will find this out, leading to exploitation of these cooperators and a race to the (formally) higher moral ground. Finally, please note that this implementation is no criticism of Sethi and Somanathan. Their model can be protected from this process by fixing a limit on . However, given evolutionary pressures to increase , this fix amounts to fixing at that limit, and we will have thereby lost the ability to explore the interaction of a variety of reciprocal altruists. Reciprocal Preferences Can Be Unstable To continue our investigation of various reciprocal agents, we fix the parameters in a different way. First, we note that the trick of
Admittedly,

this is overkill when we only need to vary two parameters. Actually, we use genetic programming (15), which is even more overpowered. The reason is to use a uniform evolutionary device for these simple models as well as the more complex ones that follow and still others explored in refs. 8 and 16.

PNAS May 14, 2002 vol. 99 suppl. 3 7239

irrelevant to the social dilemmas and cooperation. Then I would have missed this interesting unstable ecology, driven by KS at one turn. Indeed, the explicit arbitrariness of our game and function sets is a welcome reminder that this paper only attempts to sample some interesting possibilities of what can happen when different sorts of reciprocal cooperators interact. Were we attempting to demonstrate something stronger we would need to account for the generator of agent mechanisms and social situations. We would also, we should add, need to run the simulations longer than our sample runs of 40 generations or explain why these short runs suffice.
Fig. 2. Unstable preference change.

increasing only works because Eq. 3 is not fully reciprocal. Eq. 3 has respond to differences between ones own and ones opponents , but not to differences in . To correct this deficiency, we introduce our variant reciprocity factor, made explicit by using in place of , sensitive to itself. Secondly, the bias favoring higher seems to be a mistake. As we have seen, given a reciprocal linking of , differences matter, whichever of the opponents has higher ; this is suggested by our deconstruction of higher altruism in the previous section. Unfortunately, introducing differences complicates the model. The problem with differences is that they need a reference point, else they will not generate the range 1 1 needed for the spiteful survival trait. Eq. 4 is one example of a way to fix a reference point. Here is implicitly altruistic subject to the absolute difference between the two players compared with a given threshold, T.

ij i T i j

[4]

In a particularly interesting run, we set the threshold T 2 and got the results shown in Fig. 2, which plots the value of the best scoring player in each generation on the left-hand scale. In this case, High invades Low, and Medium invades High. High and Low interact as we expect; High cooperate together and Low do not. High invade Low by virtue of spite; High threatens Low in the game of chicken. The key to the complex dynamics is the ability of Medium to invade High. Medium can invade High because of the constant sum game included in the set. The altruistic equilibrium will typically be different from the selfinterest equilibrium, even with constant sums. Medium exploits Highs choice of this equilibrium. Finally, unsurprisingly, Low exploits Medium in the social dilemmas. Only HighHigh and MediumMedium populations cooperate, so this model spends about one-third of its cycle in a noncooperative period. The right-hand scale in Fig. 2 plots the populations mean score. Notice that the initial drop as the maladapted initial population is exploited, followed by the quick rise to near optimal (2.5) levels. Each time drops, it is followed by a drop in population mean score, with low points at Generation 17 and 28, where many exploited players score below the equilibrium (2.0) level. A limitation of this testing method is the arbitrariness of the reciprocal functions and their parameters, many of which (not discussed here) lead to stable reciprocal cooperation. We try to remove some of this arbitrariness in the next section. However, one source of arbitrariness is to be applauded. The set of eight games was not chosen as especially relevant to the current task; it was simply adopted from earlier, unrelated work, where they were thought to represent the interesting possibilities of social interaction. The present results turn on the variety of games used in the test. Had I created a game suite for the task at hand, I likely would have omitted the constant sum game (KS in Table 1) as
7240 www.pnas.orgcgidoi10.1073pnas.082079899

A Strategy-Based Model A second way to test reciprocity is to model more explicitly the way the agents reason, allowing reciprocity to enter their strategies. The previous model was built on a common rationality mechanism, which allowed P1 to inform its choices by looking ahead in the game tree and using P2s preferences to predict the outcomes of P2s choices. On top of this (explicitly modeled) rationality component, it permitted (without modeling the mechanisms) access to ones own and ones opponents parameters , , and as needed. The game apparatus of the present model is much simpler; it omits all of these components. Agents have instead access to functions to compare various material outcomes from various points of view. For example (P1 NODE1 NODE2 NODE3) chooses NODE1 or NODE2 according to which is better for P1; NODE3 is selected on equality. Other functions compare nodes from P2s perspective, choose the smaller, or compare from the joint perspective of both players. Thus altruism is possible, but, not surprisingly, lacking any means to discriminate between opponents, it does not evolve in this basic model. In ref. 10 I demonstrate the evolution of rationality with this model. Here I elaborate it to try to evolve reciprocity. Our primitive functions provide no basis for reciprocity, so we add one. We use a matching function to avoid the computational problem of strategies computing other strategies reciprocal properties. Our function (MATCH YES NO) compares two players programs, selecting the YES component if they are identical and the NO component otherwise. But reciprocity is only adaptive if the behavior reciprocated is group adaptive, and the alternative is individually adaptive, and match requires exact coordination of these strategies. These are high demands at the crude levels of these agents code. For example, here is a simple (designed) rational player: (CXM (P1 (P2 (LL) (LR)) (P2 (RL) (RR))) (P2 (LL) (LR)) (P2 (RL) (RR))). Therefore, the evolutionary accessibility of stable matching strategies depends on us making more compact representations available in the function set, for example (IMAX) for the above code and (UMAX) for its jointly maximizing alternative. Given these additional functions, MATCH based reciprocators [e.g.,

Fig. 3.

A hierarchy.

Danielson

Table 2. A prisoners dilemma

C C D 3.5 0 D 4.5 3

(MATCH (UMAX) (IMAX))] which fully cooperates with and only with those that match it, and plays rationally with all others, readily evolve. Limits of Matching However, match-based reciprocators need not be full cooperators. When we actually implement reciprocity as matching, we see that it can reinforce a variety of behaviors, not merely full cooperation. Matching will lock in arbitrary features of the first match-based reciprocal cooperators and often lead to suboptimal semicooperation. The very first run of the model illustrates this. In generation 10 this incomprehensible (to me) program evolved and subsequently took over the population: (BOTH (P1 (RL) (MATCH (RR) (P2L)) (CXM (P2L) (LL) (P1 (LL) (LL) (P2R)))) (P1 (RL) (P2L) (RL))). It uses low-level components to reach the high, but not quite optimal score of 39 of a fully cooperative 40 in the eight games, managing to cooperate in the prisoners dilemma, chicken, and battle of the sexes. This reciprocator has three features worth our attention: (i) it is highly stable (because MATCH blocks further evolution), (ii) it is non-optimal, and (iii) it is complex. Finally, the tendency to lock into an arbitrary convention allows the possibility that reciprocal cooperators exploit one another. First, notice that demanding identity for a match is unnecessarily strong. All that is required is that players match in the first YES branch; the second NO branch is irrelevant to insuring that matchers reciprocate (this finding means that spite toward nonmatchers is not required). We introduce the new function, YMATCH, which incorporates this weaker test. YMATCH-based reciprocal cooperators proliferate more readily than MATCH-based agents do, because it is easier for YMATCH-based agents to get a successful matching pair to seed the process. YMATCH will allow some reciprocal cooperators to exploit others. All that is needed is for the functions they coordinate around to generate different behavior based on some (otherwise) irrelevant feature, such as length. Here is an example (YMATCH (BIGGER (IMAX) (UMAX) (UMAX)) (IMAX)), which cooperates when a similar opponent is the same length or longer, but plays rationally when it is the longer of the two and thereby exploits shorter cooperators. Strategies like this will lead to unequal hierarchies with scores correlated with length; see Fig. 3. Justifying Exploitation This exploitation and resulting inequality among cooperators may seem a pointless artifact of my implementation or a mistake on the part of the would-be reciprocal cooperators. But do not dismiss this result so quickly. First, YMATCH takes advantage of the evolutionary dynamics to spread. Thus, given that reciprocity must be implemented by some set of mechanisms, something like our BIGGER function may infect any real reciprocal system subject to evolution or learning. Second, although the unequal scores YMATCH and BIGGER allow serve no useful social purpose in the present model (which fact supports talk of infection), this need not be the case. To illustrate, consider a simple model of local interaction, where agents play the prisoners dilemma in Table 2 with their immediate neighbors on a line (16, 17). The dynamics are driven by imitation; each player copies the strategy of his immediate neighbor who does best. Let the players consist of Defectors and Reciprocators of two types: R1 cooperates when
Danielson

Fig. 4.

Spatial PD. Solid bars, cooperating; hatched bars, defecting.

at least one of its immediate neighbors is the same type; R2 cooperates only when both are the same type. Fig. 4 shows the scores of players in the middle of a longer line, with three R1 then three D and then three R2. Dark bars are cooperating; light bars are defecting. Notice that the R1 exposed to a D neighbor cooperates (because he has one matching neighbor) and the R2 similarly exposed defects. This allows the edge R2 to ex ploit his R2 neighbor, who c ooperates. Thus again reciprocity allows exploitation between reciprocal cooperators. Consider now the dynamics of the system, which depend on the outcomes at the edge between the types. The rightmost R1 copies his D neighbor, so D moves left. The rightmost D player copies his R2 neighbor, so R2 moves left. Generally, R1 will lose players to D although R2 will gain them. So in this model, the spread of cooperation depends on cooperators counting those that exploit them in this edge case as cooperators (the PD payoffs were chosen to generate these dynamics). So we see that that our length-sensitive matchers are not making an obvious mistake, nor is the outcome a pointless artifact. Some learning or evolutionary regimes might require exploitation (which might better be termed sacrifice in this case) for the proliferation of cooperation. However, the justification of exploitation and its tendency to evolve are quite distinct. Exploitation is justified only in the spatial case but evolves in both. Conclusions Our results support the conclusions of Levine and Sethi and Somanathan. Reciprocity is necessary to stabilizing cooperation. It allows altruistic cooperators to do better than self-interested rational agents. However, when we move toward implementing reciprocal mechanisms, the issue of competition between cooperators becomes central. At the most abstract level, reciprocal cooperators can compete by having more altruistic preferences and they can find themselves with dynamically unstable preferences. When we implement reciprocal mechanisms, cooperators can be exploited by those with whom they have coordinated on reciprocity. Reciprocity, although stable, need not be optimal and can be opaque in its complexity. In the wider context of recent work on game theory and ethics, these results are not surprising. Binmore notes, while the emphasis is usually on the fact that such reciprocity mechanisms can be used to sustain Pareto-efficient equilibria in indefinitely repeated games, it is important to recognize that the same mechanism can also be used to stabilize inefficient equilibria (18). Boyd and Richersons apt title, Punishment allows the evolution of cooperation (or anything else) in sizable groups, warns of similar properties in reciprocal social systems (19).
I thank Rik Blok and Josh Epstein for very helpful comments on earlier versions, and Bob Axtell, Rajiv Sethi, and Brian Skyrms for helpful feedback on this research project.
PNAS May 14, 2002 vol. 99 suppl. 3 7241

1. Sober, E. & Wilson, D. S. (1998) Unto Others: The Evolution and Psychology of Unselfish Behavior (Harvard Univ. Press, Cambridge, MA). 2. Axelrod, R. M. (1984) The Evolution of Cooperation (Basic Books, New York). 3. Gauthier, D. (1986) Morals by Agreement (Oxford Univ. Press, Oxford). 4. Binmore, K. (1994) Game Theory and the Social Contract: Playing Fair (MIT Press, Cambridge, MA). 5. Danielson, P. (1992) Artificial Morality: Virtuous Robots for Virtual Games (Routledge, London). 6. Binmore, K. (1998) J. Art. Soc. Social Sim. 1. 7. Axelrod, R. M. (1997) The Complexity of Cooperation (Princeton Univ. Press, Princeton). 8. Danielson, P. (1998) in Modeling Rationality, Morality, and Evolution, ed. Danielson, P. (Oxford Univ. Press, New York), Vol. 7, pp. 423 441. 9. Sethi, R. & Somanathan, E. (2001) J. Econ. Theor. 97, 273297. 10. Danielson, P. (forthcoming) in The Handbook of Rationality, eds. Mele, A. & Rawling, P. (Oxford Univ. Press, Oxford), in press.

11. Levine, D. K. (1998) Review of Economic Dynamics 1, 593 622. 12. Guth, W. & Yaari, M. (1992) in Explaining Process and Change Approaches to Evolutionary Economics, ed. Witt, U. (Univ. of Michigan Press, Ann Arbor). 13. Danielson, P. (2002) J. Interest Group Formal Appl. Logic, in press. 14. Koza, J. R. (1992) Genetic Programming: On the Programming of Computers by Means of Natural Selection (MIT Press, Cambridge, MA). 15. Danielson, P. (2001) in Practical Rationality and Preference: Essays for David Gauthier, eds. Morris, C. & Ripstein, A. (Cambridge Univ. Press, New York), pp. 173188. 16. Danielson, P. (1998) Can. J. Philos. 28, 627 652. 17. Eshel, I., Samuelson, L. & Shaked, A. (1998) Am. Econ. Rev. 88, 157179. 18. Binmore, K. (1998) Game Theory and the Social Contract: Just Playing (MIT Press, Cambridge, MA). 19. Boyd, R. & Richerson, P. J. (1992) Ethol. Sociobiol. 13, 171195.

7242 www.pnas.orgcgidoi10.1073pnas.082079899

Danielson