Evolutionary Stability for Interactions among Kin under Quantitative Inheritance.

(1)

Copyright 0 1989 by the Genetics Society of America

Evolutionary Stability for Interactions Among Kin

Under Quantitative Inheritance

Alan B.

Harper’

Department of Zoology, University of Washington, Seattle, Washington 98195

Manuscript received August 1 , 1987 Accepted for publication December 17, 1988

ABSTRACT

The theory of evolutionarily stable strategies (ESS) predicts the long-term evolutionary outcome of frequency-dependent selection by making a number of simplifying assumptions about the genetic basis of inheritance. I use a symmetrized multilocus model of quantitative inheritance without mutation to analyze the results of interactions between pairs of related individuals and compare the equilibria to those found by ESS analysis. It is assumed that the fitness changes due to interactions can be approximated by the exponential of a quadratic surface. The major results are the following. (1) The evolutionarily stable phenotypes found by ESS analysis are always equilibria of the model studied here. (2) When relatives interact, one of the two conditions for stability of equilibria differs between the two models; this can be accounted for by positing that the inclusive fitness function for quantitative characters is slightly different from the inclusive fitness function for characters determined by a single locus. (3) The inclusion of environmental variance will in general change the equilibrium phenotype, but the equilibria of ESS analysis are changed to the same extent by environmental variance. (4) A class of genetically polymorphic equilibria occur, which in the present model are always unstable. These results expand the range of conditions under which one can validly predict the evolution of pairwise interactions using ESS analysis.

T

HE theory of evolutionarily stable strategies

(ESSs) has been proposed as a method of determining the evolutionary outcome of frequency-dependent selection (MAYNARD SMITH and PRICE 1973; MAYNARD SMITH 1982). An ESS of a game is a state in which no rare strategy can do better than the common strategy in a population. One particular form of frequency-dependent selection is the evolution of interactions among relatives (MICHOD 1982; KARLIN and MATESSI 1983). In this paper I will examine a genetic model of pairwise interaction between relatives involving a single phenotypic character. Models of such interactions have been analyzed using ESS theory (MAYNARD SMITH 1982), but such analyses leave in doubt the actual dynamic evolution of the phenotype. In this paper I will show that the stable equilibria of a particular model of polygenic inheritance are similar, and often identical, to ESSs and will offer a graphical interpretation of the nature of the two sorts of equilibria.

Let w ( x , y) be the viability of an individual of phenotype x after it interacts with an individual of phenotype y; x and y are unidimensional real variables. If individuals interact at random, an ESS (MAYNARD SMITH and PRICE 1973) is defined as a phenotype x * for which

w ( x * , x * ) 2 w ( x , x*); ( 1 4

98101.

Genetics 121: 877-889 (April, 1989)

’

Present address: Box 654, 1916 Pike Place #12, Seattle, Washington

and in the case of equality in (la), then

w ( x * , x)

>

w ( x , x) (1b)

for all x # x * . When conditions (la, b) hold, then no rare mutant affecting just this phenotype can invade an infinite panmictic population with haploid inheritance that is fixed for phenotype x * , nor can a nonre- cessive mutation invade a diploid population. Because of the close correspondence of the conditions for an ESS and the uninvadability condition for panmictic populations, I will use the terms “ESS” and “uninvadable phenotype” interchangeably.

If a single genotype results in a probability distribution of phenotypes, then one needs to replace w in (la, b) with

6,

where

6

is defined as the expected fitness of genotype x when it interacts with an individual of genotype y, averaged over the joint distribution of the phenotypes produced by x and y. This is the method used to analyze the evolution of the probabilities of adopting one of two or more strategies in an interaction (see MAYNARD SMITH 1982, Appendix D; THOMAS 1984). When the phenotype is a continuous character, such as a morphological trait, one might expect the phenotype to be determined by both genetic and environmental effects (FALCONER 198 l). By integrating over the distribution of environmental deviations one can convert the phenotypic fitness function w to the genotypic function 6 and find ESSs for such characters.

(2)

878 A. B. Harper

phenotypes can vary continuously over a range of values. When the phenotypes can vary continuously, determining the validity of criterion (1) requires knowledge of the global properties of w . It is often easier to find “ESSs” that are determined by only the local properties of w in the neighborhood of x

*

(MAY- N A R D SMITH 1982). If

d w ( x , y)/dx

Ix=y=x*

= 0 ( 2 4

d2w(x, y)/dx‘

I

<

0, (2b)

and

then rare mutations of small effect cannot invade a population fixed for x * . Conditions (2a, b) are not exactly the same as conditions ( l ) , but conditions (2a, b) guarantee that

w ( x * , x * )

>

W ( X *

+

t , x * )

for t small and nonzero.

When the phenotypes can vary continuously, it is not necessarily true that the invasion of rare mutants of small effect will take the population toward the ESS (ESHEL and MOTRO 1981; ESHEL 1983). In a sense, this means that not all ESSs are attainable; for some ESSs if the population starts near the ESS the contin- ued invasion of favored mutations will cause the population to evolve away from the ESS, even though if the population were exactly at the ESS, no mutation could invade. ESHEL and MOTRO have shown that when the populations is fixed for a gene with phenotype x *

+

t then mutations with phenotypic effect of the order t2 that vary in the direction of the ESS can

invade j u st when

d‘w(x, y)/dx2

+

d’w(x, y)/dxdyI x=y=x*

<

0. (2c)

ESHEL ( 1 983) has called phenotypes that meet conditions (2a-c) “continuously stable strategies,” but here 1 will incorporate condition (2c) into the definition of an ESS to avoid complexity of terminology.

The foregoing methods are used when individuals interact in randomly formed pairs. When individuals are more likely to interact with relatives, the invasion conditions must be modified to reflect this. If an individual with a rare genotype which produces phenotype x is in a population in which the common phenotype is y then the fitness of this individual will be

W ( r , ( X , y) = (1

-

r)w(x, y)

+

T W ( X , x), (3)

where r is the probability that the rare genotype interacts with another of the same genotype. In a diploid population without inbreeding, T is j us t the

average relatedness of interactants. T h e conditions for an ESS for interactions among relatives are (1) with w(?) substituted for w. Following the same reason- ing as above, it can be seen that the conditions for uninvadability by mutations of small effect and for continuous stability can be obtained by substituting

q r ) for w in (1) and (2) (see also MAYNARD SMITH

1982, Appendix F; GRAFEN 1979). T h e conditions for an ESS among relatives are then

dw(,)(X, y)/dx = wx

+

my = 0, ( 4 4

d2w(,)(x, r ) / d x 2 = w X x

+

2mXy

+

mYy

<

0, (4b)

and

d2W(?)(X, y)/dx’

+

d2W& y)/dxdy

( 4 4 = wxx

+

(1

+

T)WV

+

myy

<

0.

T h e subscripts x and y on w indicate partial derivatives of w with respect to the subscript variable, evaluated a t x = y = x * .

A number of investigations have considered the question: when can the equilibria of other genetic models be identified with the ESS phenotypes found by invasion analysis? T h e models used are generally of two types, it is assumed either that there are a number of discrete phenotypes possible (called here “n-strategy games”) or that there is a continuous, one- dimensional, range of phenotypes (“continuous-strategy games”). Most of these investigations have looked at the case where individuals interact at random.

In the n-strategy random-encounter game it is clear that at any ESS the fitness of individuals adopting each of the various strategies must be equal, otherwise a phenotype that adopted only the most fit strategy could invade. SLATKIN ( 1 978, see also LESSARD 1984) has shown in the n-strategy random-encounter game that if there are a sufficient variety of mutations in a diploid population, evolution will tend to equilibrate the fitnesses of the phenotypes, leading to the same result as predicted by ESS analysis. MAYNARD SMITH (1 98 1) has shown that when there are just two phenotypes, diploid inheritance, and two alleles affecting the probability of having each phenotype, then “it is almost true to say that the population will evolve to an ESS if it can, and if it cannot it will approach the ESS as closely as the genetic system will allow”

(p.

(3)

Quantitative Evolutionary Stability 879

SLATKIN (1979) considered three models for the evolution of a normally-distributed character subject to frequency- and density-dependent selection. Two of the models concern us here. In a continuous-genotype model, similar to the model used below, t w o sorts of equilibria can occur: either all heritable variation is lost from the population, or heritable variation remains and certain average fitness effects exactly cancel. T h e second equilibrium is characterized by the two conditions

and

where p ( z It,

.I)

is a normal density function with specified mean and variance, and w ( z , i , u:) is the fitness of phenotype z when the population is distributed according to

p .

SLATKIN shows that conditions (5a, b) also characterize phenotypes that are uninvadable by mutations that change

t

and :.a Conditions (5a, b) are equivalent to the ESS condition (2a) for the two characters the mean value of z and the variance in expression of z.

T h e extensive literature on kin selection (reviewed in KARLIN and MATESSI 1983 and MICHOD 1982) can be seen, in part, as an examination of whether w ( ~ ) can be used as a summary measure of selection involving interactions among relatives. Many models of kin selection assume that there are two strategies that an individual can assume in an interaction, altruistic or non-altruistic, and that an individual’s genotype affects its probability of being altruistic. This is equivalent to assuming that w ( ~ ) is linear. Let x and y be the probabilities of being altruistic for one individual and its relative. Then the fitness of an individual after an interaction is

w ( x , y) = 1

+

yP

-

XY, (6)

where y represents the amount that the individual’s fitness is decreased by being altruistic, and

P

represents the amount an altruistic relative increases the individual’s fitness. If w ( ~ ) were an adequate fitness measure, then under (6) one would expect larger values of the phenotype to have higher fitness just when rP

>

y.

When w is linear, it is found that the effects of dominance (in the probability of being altruistic) among alleles at a single locus (CAVALLI-SFORZA and FELDMAN 1978; UYENOYAMA and FELDMAN 1981; UYENOYAMA, FELDMAN and MUELLER 198 l), and the effect of recombination between two loci (MUELLER and FELDMAN 1985) can lead to a class of interior equilibria (“structural equilibria”) that cannot be predicted by the naive use of w ( r ) as a measure of inclusive

fitness. In many single-locus models, the interior equilibria can be simply described by considering the average effect of a substitution of alleles (UYENOYAMA and FELDMAN 1981; UYENOYAMA 1984). T h e structural equilibria only occur when heterozygotes are not approximately intermediate in phenotype between the homozygotes with which they share alleles; for nearly additive genes, w ( ~ ) is an adequate measure of the direction of selection when w is linear (TORO et al.

1982). Quantitative genetic models with additive alleles in which it is assumed that either w is linear or log w is linear also have behavior consistent with the predictions of wb) (YOKOYAMA and FELSENSTEIN 1978; AOKI 1982; CROW and AOKI 1982; ENCELS 1983). However, when w is linear, there can be no stable phenotypic equilibria, either “altruism” or “selfish- ness” is increased to the limits of heritable variation. These results indicate that w ( ~ ) is an adequate measure of inclusive fitness when genes are nearly additive and the fitness function is linear, i.e., interactions among relatives with fitness function w evolve in the same direction as interactions among nonrelatives with fitness function w(+ [BOYD and RICHERSON’S (1980) analysis of a nonlinear fitness function using a quantitative genetic model is discussed later.]

(4)

880 A . B. Harper

T H E MODEL

In this paper I use FELSENSTEIN’S (1979) symmetrized version of LANDE’S (1975) model of the inheritance of a one-dimensional quantitative character. T h e assumptions and limitations of this model have been discussed in detail elsewhere (FELSENSTEIN 1977; TURELLI 1984) and are only briefly reviewed here. LANDE’S original unsymmetrized model assumes that the inheritance of the character is determined by a finite number, n , of loci in an infinite population. T h e contribution of the effect of one allele is a real number, which contributes additively to the phenotype, so that the phenotype of an individual is the sum of the effects of all 2 n alleles, plus any environmental deviation. It is assumed that the distribution of allelic effects is multivariate normal, so that the population genetic structure can be specified by the mean allelic effects at each locus, and the variances of allelic effects within and the covariances between loci. This last assumption requires that a number of approximations be made. If the fitness function is not normal, the distribution of allelic effects cannot be normal both before and after selection. Recombination adds the problem that the population after recombination consists of a mixture of recombinant and nonrecombinant classes. Each class may be multivariate normal, but in general a mixture of normal distributions is not itself normal. Mutation leads to the same sort of approximation as recombination, since the population consists of a mixture of mutant and nonmutant classes. All these approximations are achieved by just keeping track of the first and second moments of allelic effects as the population proceeds through the steps of mating, selection, recombination, and mutation, and treating the population at each step as multivariate normal with the proper moments. T h e import of TURELLI’S (1984) suggestion that mutational rates may be low enough, and mutational effects large enough, that the distribution of allelic effects will be non-normal is left to the discussion below. Simulations reported in FEL-

SENSTEIN (1 979) suggest that the approximations for

the effects of recombination and selection are generally appropriate, even in the case of wildly non-normal fitness functions.

FELSENSTEIN’S symmetrized version of LANDE’S model assumes that the recombination rate T is the

same between all pairs of loci, that the mean and variance at every locus is the same, and that the covariance between every pair of loci is the same. T h e symmetric recombination rates cannot be realized by any biological linkage map unless r = 0 or T =

?h.

T h e

equilibrium results that I find depend only on the presence of recombination between loci, not on the magnitude of it, suggesting that differing recombination rates among pairs of loci would not change the results. Perhaps more troubling is the assumption of symmetric variance and covariances. T h e equilibria

that I find below depend only on the sum of the variance within and covariance among loci, suggesting that a nonsymmetric apportionment of variance will not affect the position of the equilibria. However, the only forms of instability of equilibria that can be found with this model are when an equilibrium is unstable to the same perturbation in all variances or all covariances. This model cannot identify instabilities in which, say, at some loci the variance grows and at others it shrinks. As far as I know, this type of insta- bility has not been reported in a quantitative genetic model.

RECURSION EQUATIONS

FELSENSTEIN (1 979) has derived the recursion equations for the symmetrized LANDE model with nor- optimizing selection. Before selection, the variance of allelic effects at each locus is V , / ( 2 n ) at generation t ,

the covariance between any two loci is C l / ( 2 n ( n

-

l)), and the average effect at any locus is M t / ( 2 n ) . T h e breeding value of an individual is then the sum of 2n random values normally distributed with these moments, implying that the breeding value is normally distributed with mean MI and variance C , + V,. The nor-optimal fitness function for breeding value ( is usually written as

(I will use

i

for a genotypic fitness function, and w for a phenotypic fitness function, ( to represent a breeding value, and z a phenotype.) T h e curvature of the fitness function is S and the optimal phenotype (or pessimal phenotype if

S

<

0) is P. When Equation 7a changes from optimizing to disruptive selection

S”

passes through m, so I will reparameterize (7a) as

i(0

cz exp(YtAS-2

+

B o , (7b)

where S = -1/A and P = -B/A. It turns out that Vl and Ct enter into the recursion equations only as a sum, so it will be convenient to define G , = V ,

+

C , as the total genetic variance in generation t. Reparame- terizing FELSENSTEIN’S (1 979) recursion equations for V,, C , , and M,:

1 AG:

Vt+l = vt

+

-

2n 1

-

AG;

c1+,

= C,(1

-

T )

+

-

n - _(8b)

1 AG: 2 n 1

-

AG, ’

and

PAIRWISE INTERACTIONS

(5)

Quantitative Evolutionary Stability 88 1

This will allow us to find the equilibria of Equations 8 and compare these equilibria to conditions (4). Con- ditions (4) result from the assumption that individuals form pairs, interact, and have viability fitness w as a result of the interaction. I will allow the possibility of a correlation of phenotypes in interacting pairs either through a correlation of breeding values or through a correlation of environmental effects. T h e possibility of phenotypic assortative formation of pairs is not considered here, as such assortment could affect both the genetic and phenotypic correlations together in a complicated way.

If the population is panmictic, the covariance among loci, C f , and the variance within loci, V,, con- tribute proportionally to the correlation among the breeding values of two individuals. This can be seen by the following argument. T h e only way for the genotypes of two individuals to covary in a panmictic population is for the individuals to share one or more ancestors. If, say, individuals a and b share an ancestor, the covariance between a and b due to any one locus is the probability of sharing an allele from this ancestor times the variance at that locus in the ancestor’s generation. T h e covariance between locus 1 in a and locus 2 in b due to this common ancestor is the probability of a and b both having alleles from this ancestor times the covariance between the loci in the ancestor. T h e recombination between the t w o loci does not enter into this calculation. This means that if the average proportion of alleles a and b which are shared due to common descent from generation t

-

i is P C , then the covariance between a and b due to

these ancestors is pc(Vf-;

+

Cf-i) = p&-i. T h e value of PC for any type of relation can be determined by standard methods (FALCONER 198 1).

Unless the population has reached an equilibrium, the values of V,-i and Cf-i will differ for each generation. However, I will make the approximation that the covariance between the breeding values of two individuals can be represented by pc(VI

+

C,) = pGGf ignoring the variation among generations. This will be exactly true at any equilibrium. If the correlation among interacting individuals is due to the encounters of kin in a panmictic population, this approximation will not affect the equilibria1 results reported below. This is because kin are related through sharing common ancestors a fixed number of generations before the present generation ( 1 generation back in the case of siblings, 2 generations back in the case of cousins, etc.), and violations of the approximation will only add second-order effects to the perturbation analyses. However, if the population is inbreeding, then individuals can be related through genes shared by a common ancestor any number of generations before, and this approximation would fail. This approximation will also fail far away from an equilibrium.

T h e correlation among the phenotypes of breeding

individuals may also have an environmental compo- nent. I assume that the phenotype of an individual is the sum of its breeding value and an independent normally distributed environmental deviation: z, =

5;

+

ex, where = 0, and Var{e,) = E . T h e environmental deviations of a pair of individuals may covary: Cov( e,,

cy) = PEE.

The covariance between the phenotypes of t w o interacting individuals is then the sum of the phenotypic and genotypic covariances: Cov(zx, zy) = P G G ,

+

pEE. In order to derive the genotypic fitness function, we will need the covariance between the phenotypes of two interacting individuals conditioned upon the breeding value of one of the pair: Cov(zx, zy

I

S,).

Given an individual’s breeding value {,, the breeding value of the individual it interacts with,

rY,

is normally distributed with moments

ry

= (1

-

m ) M f

+

P C L and

-

Var(S;l = ( 1

-

pZp)G, (9)

for relatives of any fixed value of PC.

T h e distribution of the phenotypes of an interacting pair, conditioned upon a certain breeding value for one member of the pair, is a bivariate normal distribution (MORRISON 1976). Let

p ( .

.) be a generic probability density. Standard theory tells us that

p(zx,

zy

I

5;) has the following moments:

-

.zx = rx, z y - = (1

-

PC)M

+

Pcs;, ( 1 0 )

Varlz,) = E, Var(zy) = ( 1

-

pZ)C,

+

E , and

Cov(z,, zy) = PcGt

+

PEE.

(6)

882 A. B. Harper

nation has, which itself has some support from simulations (FELSENSTEIN 1979).

We now have, from (lo), the joint distribution of phenotypes of interacting relatives given the breeding value of one of the relatives. If we then assume that the phenotypic fitness function describing the fitness of an individual after an interaction can be approximated by an exponential of a quadratic function of the two phenotypes, ie., that the fitness function has the shape of a bivariate normal surface, then we can obtain the genotypic fitness function, in the form of (7b). T h e genotypic fitness function and the recursion Equations 8 completely specify the evolution of the population.

The derivation of the genotypic fitness function is shown in APPENDIX A, the results of which are sketched here. As in (l), let w(z,, zy) be the fitness of an individual of phenotype z , which interacts with an individual of phenotype zy. T h e fitness of breeding value 3; is then the average fitness resulting from the distribution of phenotypes produced by 5;:

It is easier to understand (1 1) if we decompose the phenotypes z, and zy into a breeding value and an independent environmental deviation:

* p ( r y

I

r x k q y

=

J

w x 9fy”y

I

3;)dr.Y.

Here $({,,

lY)

is the expected fitness of an individual with breeding value

rx

when interacting with an individual of breeding value

S;

averaged over the distribution of phenotypes that can be produced by these two breeding values.

Let us approximate the logarithm of the phenotypic fitness function w with a second order Taylor series expanded about a certain phenotype, L O . Let

let v?, v,, vxy, and vD be the other first- and second- order partials of log w evaluated at (zo, zo), and let voo

= log w(z0, zo), resulting in

4 x 7

Y)

= exp(Ww(x9

Y)))

=

exp(vo0

+

v,(x

-

LO)

(1 3)

+

vyo)

-

20)

+

1/2vxy(x

-

zo)2

+

V,(X

-

z0)o)

-

I o )

+

%tlDo)

-

zoy).

T h e approximation in (13) allows us to complete the integral in (1 2), and this gives us the genotypic fitness function needed to complete the recursion equations of the model. The value of (1 2 ) is shown in (A6) of APPENDIX A.

EQUILIBRIA

We can now find the equilibria and stability conditions of the recursion equations. It turns out that the recursion equations for the genetic variance and covariance, V , and

Cf,

are independent of the mean phenotype, M t . Following FELSENSTEIN (1 979), take

(n

-

1) times (Sa) minus (Sb), and substitute t = t

+

1

= w, and find that C, = 0. Since then G, = V,, we see from (Sa) that the only equilibria are A = 0 or G = 0. From (8c) we see that M can be at an equilibrium either when G = 0, or when A = 0 and B = 0.

This analysis implies that there are two classes of equilibria for Equations 8: either there is no heritable variation, G = 0, or the genotypic fitness function is flat and there is no selection, A = B = 0. This is the same behavior as described by SLATKIN (1 979). T h e stability of these two classes of equilibria is analyzed in APPENDIX B. I t turns out that there are no stable equilibria with nonzero genetic variance in this model. Of course, when the approximations made in this model are violated, especially the assumptions that the phenotype distribution is normal and that the fitness function can be approximated as in (1 3), kin selection could maintain genetic variation.

COMPARISON OF T H E G = 0 EQUILIBRIUM W I T H ESS ANALYSIS

Remember that the ESS method finds an ESS by requiring that the partial derivatives of the fitness surface satisfy certain conditions when the partials are evaluated at the ESS. T h e equivalent solution for the present model is the conditions on the partial derivatives of (13) necessary for a phenotype to be a stable equilibrium of the model when the equilibrium is the point zo around which w was expanded in (1 3). These conditions are derived in APPENDIX B, and are

3,

+

PC;, = 0, ( 1 4 4

Vxx

+

2pcsxy

+

p:v, 0, ( 14b)

3,

+

(1

+

P C ) k Y

+

PCV, 0. (1 4 4 and

Here the parameters, G,, etc. are the partial derivatives of the logarithm of the genotypic fitness function seen in (1 2 ) , ie.,

-

d log w x , Cy)

I

vx =

ai-,

I =<= y 20

(7)

the distribution of environmental deviations, and will differ from the phenotypic fitness function in (1 2) if there is any environmental variance in the expression of the phenotype. T h e values of Gx etc. are shown in Equations A2 of APPENDIX A. The conditions (1 4) can be compared to the conditions for an ESS by applying the ESS conditions (4a-c) to the present model. T h e ESS conditions are obtained by applying conditions (4) to the genotypic fitness function

i ( r X ,

lY),

see AP- PENDIX B for the details. It turns out that that (14a) and (1 4c) are the same as the conditions (4a) and (4c), respectively, but that (4b) becomes

s x x

+

2PC3Xy

+

pcsyy

+

PC( 1

-

PC)s:

<

0, (15)

which will generally differ from (14b) unless PC is 0 or 1.

To recapitulate, the present model finds two classes of equilibria when relatives interact. At equilibria of one class, there is positive genetic variance maintained by selection. Such equilibria are always unstable. At the second class, genetic variance is eliminated by selection, and these equilibria are identical to the equilibria found by ESS analysis (4a). But the stability conditions for the second class of equilibria are generally different than the ESS conditions (4b, c).

T h e difference in the stability conditions can be understood by consideration of the assumptions that have been made about how pairs of interacting relatives are formed. In ESS (or invasion) analysis it is assumed that only two genotypes are segregating in the population, so that three types of interacting pairs can be formed using the wild type and the rare mutant. Under this assumption, Equation 3 describes the inclusive fitness of a rare genotype. Under the assumptions of polygenic inheritance, it is assumed that the breeding value of an individual is determined by a large number of loci and that random sampling will cause relatives to share on average PC of the alleles at these loci. This suggests that the equation that should describe the inclusive fitness of individuals when relatives interact is

W ( P ) ( X , y) = w ( x , (1

-

Pc)y

+

PC%). (1 6)

If we then apply the ESS conditions (2a-c) to w(+ we get the following conditions:

h ( , , ~ ( x , y)/dx = wx

+

pcwr = 0 , (1 7a)

d2w(p)(x, y)/ax‘ = wXx

+

2

pcwxY

+

p h ,

<

0, (17b)

a2w(P)(x? r)/d3cey

+

a2w ( P ) ( X , r)/ax’ (1 7 4 = w x x

+

(1

+

PC)WXy

+

pcw,

<

0.

Conditions (1 7) are the same as ( 1 4) as can be seen by substituting

i

for w in (16). It is interesting to note that the conditions for a stable equilibrium under polygenic inheritance have the same algebraic form whether the partials of the fitness function or of the

logarithm of the fitness function are used. This is not true of the ESS conditions (4a-c).

There is a simple graphical interpretation of the difference between the standard ESS conditions (4a-c) and the conditions (1 7a-c). In Figure l a the function i, the fitness of breeding value x when it interacts with breeding value y, is shown as a surface; in Figure 1 b

i

is drawn using level curves. Obviously, any monomorphic equilibrium must be a point on the line $(x, x), shown crossing the surface. T h e conditions (4a) and (1 7a) require that a line, labeled “A” in Figure 1, which goes through the equilibrium, is parallel to the x-y plane, and has slope pc in the x-y plane, be tangent to the surface. In this figure PC; is chosen to be Yz. T h e conditions (4c) and (17c) require that the surface increase along both the arrows “B” which are parallel to “A.” The third condition for stability under polygenic inheritance is that the surface have negative (downward) curvature along the direction of the arrow “A.” T h e equivalent condition for uninvadability by single mutations is that the average of the curvatures of the surface along the arrows “C” and “C”’, weighted by PC and 1

-

P C , respectively, must be negative. This interpretation helps us understand when the ESS conditions and the polygenic stability conditions might differ: just when the surface is saddle shaped so that the curvature of w changes sign between the directions of the arrows “C,” and “C’.” T h e stability conditions of the two models are always the same when pc = 0 or pC = 1.

T H E EFFECTS OF ENVIRONMENTAL VARIANCE

Often in the study of frequency-dependent evolution, models ignore the fact that environmental variance affects the distribution of phenotypes produced by a particular breeding value. Equations A2 allow us to examine the effect of including environmental variance and covariance among the interacting pairs. If there were no environmental variance then the phenotypic fitness function could be substituted for the genotypic fitness function. Similarly, if the logarithm of the fitness function had no curvature, the genotypic and phenotypic fitness would,be the same. YOKOYAMA and FELSENSTEIN (1 978) have examined the case when the log fitness function is flat, i.e., when w is an exponential surface, and, consistent with these results, found that environmental covariance does not change the predictions of kin selection theory.

BOYD and RICHERSON (1980) have also considered the effect of phenotypic variation on kin selection. They conclude that the heritability of a character will influence the amount of altruism present in the pop- ulation, and the equilibria found under a model of quantitative genetics differ from those found by ESS analysis. It is instructive to examine how they reached these conclusions.

(8)

884 A. B. Harper

a

A

ri

3 Y v)

fn a

C

ii

L

Breeding value x

b

Breeding value x

FIGURE 1 .-A graphical interpretation o f the differences between the stability conditions under invasion analysis and under polygenic inharit;lnce for a trait that influences interactions among relatives. T h e surface G(x, y) is shown in a perspective view in a, and using contour lincs i n b. T h e conditions for a stable equilibrium are discussed in the text.

individual is a function of both its own phenotype and of the average phenotype of a group of kin. By just considering groups of 2 kin, their results can be compared directly with mine. They use a model of quan- titative inheritance introduced by CAVALLI-SFORZA and FELDMAN (1 976), but it seems that the important difference between our assumptions is that they consider the effects of mutational variance, and I do not. They assume that at a mutation-selection balance the distribution of breeding values is approximately normal, and this allows them to examine equilibria at which there is non-zero heritability. They also assume that there is no correlation in the environmental deviations among interacting kin.

For two individuals BOYD and RICHERSON’S phenotype fitness function is

from which they find the following equilibrium mean phenotype:

~ ~ ( 4 7

+

(1

-

PE)G

+

(1

-

PC)E)

Here E is the environmental variance, and G is the genetic variance at equilibrium. T h e equilibrium can also be found using the methods above: obtain the derivatives v,, etc., of (1 8), calculate the derivatives of the genetic fitness function 3, using (A2), and obtain the equilibrium from (1 4a). T h e equilibrium phenotype obtained in this way is

~ ~ ( 4 7

+

(1

+

P E ) ( ~

-

pc)E)

x* =

+

2 P A 1

+

P C P

+

( P C

-

P E F )

4 7

+

(1

+

p c ) ( E

-

P E E

+

2 s ) (20)

Where the assumptions of the model overlap, pE = 0 and G = 0, the results agree.

BOYD and RICHERSON characterize the equilibria of their model as having quite different properties than the ESSs of (18). This is because they compared the equilibria of the quantitative genetic model in the presence of phenotypic and genetic variance to an ESS analysis in which phenotypic variance was assumed to be absent. It is possible to find the ESS when there is environmental and genetic variance. Assume that the phenotype of an individual is determined by a single locus and a random deviation. This random deviation has variance G

+

E , being due to both the effects of other loci and environmental effects; both the environmental and genetic variation are assumed to be uncorrelated with the alleles at the locus used for invasion analysis. Since BOYD and RICHERSON assume that the environmental deviations of interacting relatives are uncorrelated, the phenotypic covariance of individuals will be pcG. From these assumptions we can derive the genotypic fitness function 5. (This is easily done by substituting E

+

G for E in (A2b-f) and pcG/(G

+ E )

for pn-note that the phenotypic variance is the sum of the genotypic and environmental variances). T h e ESS is a t (14a), which evaluates to (19). T h u s we see that if genetic and environmental variance are taken into account in an ESS analysis, the resulting equilibria for the mean phenotype are identical to those found in a polygenic model with the same genetic and environmental variance. This equiv- alence demonstrates the effect of phenotypic variance: since the genotypic fitness function is determined by the phenotypic fitness function and the relationship between genotype and phenotype, phenotypic variance will change the shape of the genotypic fitness function and therefore the uninvadable phenotype found by ESS analysis.

DISCUSSION

(9)

when the trait is inherited as a quantitative character. I assume that the (viability) fitness of an individual is a function of that individual’s phenotype and of the phenotype of another individual with which it interacts. If x and y are the phenotypes of two individuals, the fitness of the x individual after an interaction with y is w ( x , y). This assumption induces frequency-dependent selection on the phenotypic character. I find the equilibria of the model assuming that log w can be accurately approximated by its first and second derivatives.

T h e most important result is that ESS analysis and the present model of quantitative inheritance find the same equilibria1 phenotypes, but the stability conditions for the equilibria of the two models differ. A continuously stable ESS (ESHEL 1983) is determined by three conditions; (4a) which defines an equilibrium, (4b) which determines the uninvadability of an equilibrium, and (4c) which determines whether the invasion of dominant mutations of small effect will tend to take the population towards an equilibrium. There are three conditions for a stable equilibrium in the present model, two of the conditions, (1 7a) and (1 7c), are identical to the conditions (4a) and (4c), respectively, while the third condition, (1 7b), is similar to, but not identical to, condition (4b). T h e conditions (4b) and (17b) are identical when unrelated individuals interact, when identical twins interact, or when the the genotypic fitness function (pictured in Figure 1) is not saddle-shaped.

The differences between the stability conditions can be seen as due to the fact that invasion analysis assumes that there are only two genotypes segregating in a population, while models of quantitative inheritance allow a continuous distribution of genotypes. This difference results from the assumption of this paper that the distribution of the breeding values of pairs of related individuals can be approximated by a bivariate normal distribution. This approximation will be valid only when a large number of loci of roughly equal effect determine the inheritance of the phenotype. There seem to be no similar conditions for the evolution of a trait influenced by genes of small effect at a few loci (but see MUELLER and FELDMAN 1985), but it is to be hoped that the above conditions for one locus and many loci will bracket intermediate cases.

ESS theory has been used to investigate situations in which interactions might be expected to maintain phenotypic variance in a character, e.g., the “war of attrition” model in which the ESS is a distribution of waiting times (MAYNARD SMITH 1982). There are at least three reasons why it does not seem possible to naively apply the results obtained here to models of such interactions. I assume here that the fitness after an interaction can be adequately described by an exponential of a quadratic function of the phenotypes of the interactants-this cannot be an adequate ap-

proximation when selection is expected to maintain genetic variance as we have shown that no stable genetically polymorphic equilibria exist when we make this assumption. T h e equilibrium distribution of phenotypes may not be normal as is assumed here- ESS analysis of the war of attrition suggests that interactions may select for an exponential distribution of phenotypes. Thirdly, if the distribution of phenotypes in the population is not normal, the distribution of interactants cannot be bivariate normal as is assumed here. However, the results obtained here could be used to analyze such models if it is possible, and biologically reasonable, to assume that the phenotypic distribution of any one individual can be described as a parametric function and to assume that the genotype (plus any environmental deviation) determines the value of the parameters of the phenotypic distribution. T h e phenotypic fitness function, w can then be obtained as a function of these parameters, and then the conditions (4) and (17) can be used to find the stable equilibria expected for the parameters.

T h e model used here ignores the effects of mutations, and it must be asked how robust the conclusions are. Mutational variance will have two effects, it will cause the equilibrium genetic variance to be non-zero and, depending on the distribution of mutational effects, it may cause the allelic effects at each locus to be highly non-normal (TURELLI 1984, 1988). T h e effect of positive genetic variance on the equilibrium mean phenotype can be taken into account by incorporating the genetic variance and covariance into the “environmental” variance and covariance in calculat- ing the coefficients of the genotypic fitness function of (1 2). This was the method used above to find the evolutionarily stable phenotype in the presence of genetic variance, and such a method finds the same equilibria as a quantitative genetic model in which mutational effects are explicitly considered (BOYD and RICHERSON 1980). This method works because as long as each locus has a small contribution towards the total genetic variance, and genetic correlations among the loci are negligible, then the effects of other loci on the variation of phenotypes can be accounted for in the calculation of the genotypic fitness function. T h e effects of each gene on its own fitness through its effects on relatives is accounted for by the inclusive fitness function (1 6). Incorporating genetic variance in this way can be used to find the equilibrium mean phenotype defined by (17a), but the effect of mutations on the stability conditions (1 7b, c) is more subtle.

(10)

886 A. B. Harper

little can be said about the dynamics of evolution without detailed knowledge of the genes affecting the trait, if the number of loci is large enough then under directional selection the mean should tend to evolve to higher fitnesses, and under optimizing selection the mean should evolve toward the optimum and the variance tend to contract to some mutation-selection balance. The conditions (1 7a) and (1 7b) are the conditions that the fitness function induced by pairwise interactions should be optimizing with optimal phenotype x* just when the population has mean phenotype x*, and (17a) and (17c) are the conditions that the induced fitness function will increase in the direction of x* when the population mean is displaced from x * . This gives us confidence that our results are ap- plicable when genetic effects are not normally distributed, as long as the induced fitness function is the same under different distributions of allelic effects. T h e induced fitness function is the expectation of w over the distribution of interactants that it meets. If the number of loci is large enough, then we might expect that the distribution of breeding values of relatives will be binormally distributed even if the effects at each locus are non-normal. This observation, together with BARTON and TURELLI’S (1 987) analysis, suggest that conditions (1 7a-c) will describe evolutionarily stable phenotypes even under a non-normal distribution of genes at each locus, as long as there are enough loci, of approximately equal effect, so that breeding values are approximately normally distributed. T o apply these results, however, one must fix the genetic variance at a particular value and calculate the terms in (A2) as if the genetic variance and covariance were incorporated into the environmental variance. If (1 7b, c) are then satisfied, this genetic variance and the mean phenotype predicted by (1 7a) should be a stable equilibrium under an appropriate, but un- known, mutation rate. Even under frequency independent selection, predicting the genetic variance at equilibrium from the mutation rate is difficult: it would seem that adding frequency dependence could only compound the difficulty.

Comparing the results of ESS analysis, of the present model, and of the exact models of kin selection illuminates how the genetic basis of inheritance affects the evolution of a character that influences interactions among kin. T h e exact models of kin selection (CAVALLI-SFORZA and FELDMAN 1978; UYENOYAMA and FELDMAN 198 1 , UYENOYAMA, FELDMAN and MUELLER 1981) treat the case when the fitness function w is linear, and examine the population genetics of alleles of an arbitrary degree of dominance. In these models it is found that unless the alleles determine the phenotype nearly additively, the existence and stability of polymorphic equilibria cannot be predicted using inclusive fitness arguments. For instance, it is possible for a partially recessive allele to be able

to invade when rare, and then rise to a stable intermediate frequency without being fixed. Such unex- pected behavior is found only for alleles of large effect on the phenotype; for alleles of small enough effect, any allele that can invade a population will always be fixed unless the allele is overdominant in its effects on the phenotype (TORO et al. 1982).

Comparison of the results of ESS analysis, the present model, and the exact models of kin selection suggest the following generalizations. When a character is determined by alleles of substantial effect on the phenotype at one or a few loci, and especially when the alleles determine the phenotype nonaddi- tively, one cannot predict the existence or the stability of genetic equilibria by the use of an inclusive fitness arguments. But when differences in the phenotype are caused by alleles of small effect with primarily additive effects, then the analysis of an inclusive fitness function can be substituted for formal analysis of a genetic model. T h e appropriate inclusive fitness function arising from invasion analysis is (3), giving stability conditions (4a-c); the inclusive fitness function in the present model is (16), giving stability conditions (1 7a-c). In most cases, (4a-c) and (1 7a-c) will predict the same stable equilibria; but under some parameter values the stability conditions can differ and in such cases both sets of conditions should be used to determine the “evolutionarily stable” phenotype.

J . FEUENSTEIN and M . SLATKIN contributed in uncountable ways

to this project. M . FELDMAN and M . TURELLI kindly helped me understand the implications o f their work for the model presented here. Three anonymous reviewers and the corresponding editor pointed out a number of errors and imprecisions in earlier drafts, and have markedly improved the paper. Some o f the results were obtained using Macsyma beta release 308.2 running on a Sun computer at the University of California at Berkeley. Parts o f this work were supported by a National Science Foundation Predoctoral Fellowship.

L I T E R A T U R E CITED

AOKI, K., 1982 Additive polygenic formulation o f Hamilton’s model o f kin selection. Heredity 49: 163-169.

BARTON, N. H . , and M . TURELLI, 1987 Adaptive landscapes, genetic distance and the evolution o f quantitative characters. Genet. Res. 49: 157-173.

BOYD, R., and P. J. RICHERSON, 1980 Effect o f phenotypic varia- tion on kin selection. Proc. Natl. Acad. Sci. USA 77: 7506- 7509.

BROWN, J. S., and T. L. VINCENT, 1987 A theory for the evolu-

BULMER, M. G., 1980 The Mathematical Theory of Population Ge-

CAVALLI-SFORZA, L. L., and M . W. FELDMAN, 1976 Evolution of

continuous variation: direct approach through joint distribu- tion o f genotypes and phenotypes. Proc. Natl. Acad. Sci. USA

CAVALLI-SFORZA, L. L., and M . W. FELDMAN, 1978 Darwinian selection and “altruism.” Theor. Popul. Biol. 14: 268-280.

CROW, J . F . , and K. AOKI, 1982 Group selection for a polygenic behavioral trait: a differential proliferation model. Proc. Natl. Acad. Sci. USA 79: 2628-263 1.

DICKENS, D. W., and R . A. CLARK, 1987 Games theory and tionary game. Theor. Popul. Biol. 31: 140-166.

netics. Clarendon Press, Oxford.

(11)

Quantitative Evolutionary Stability 887

siblicide in the kittiwake gull, Rissa tridactyla. J. Theor. Biol.

125: 301-305.

ENGELS, W. R., 1983 Evolution of altruistic behavior by kin selec-

tion: An alternative approach. Proc. Natl. Acad. Sci. USA 80:

ESHEL, l., 1982 Evolutionarily stable strategies and viability selec-

tion in Mendelian populations. Theor. Popul. Biol. 22: 204-

217.

ESHEL, I . , 1983 Evolutionary and continuous stability. J. Theor. Biol. 103: 99- 1 1 1.

ESHEL, I., and M. W. FELDMAN, 1984 Initial increase of new

mutants and some continuity properties of ESS in two-locus

systems. Am. Nat. 124: 631-640.

ESHEL, I., and U. MOTRO, 1981 Kin selection and strong evolu-

tionary stability of mutual help. Theor. Popul. Biol. 1 9 420-

433.

FALCONER, D. S., 1981 Introduction to Quantitative Genetics, Ed. 2.

Longman, London.

FEUENSTEIN, J., 1977 Multivariate normal genetic models with a

finite number of loci, pp. 227-246 in Proceedings of the Inter-

national Conference on Quantitative Genetics, edited by E. POL-

LAK, O. KEMPTHORNE and T. B. BAILEY, JR. Iowa State Uni- versity Press, Ames.

FELSENSTEIN, J., 1979 Excursions along the interface between

disruptive and stabilizing selection. Genetics 93: 773-795.

GODFRAY, H . C. J., 1987 The evolution of clutch size in parasitic

wasps. Am. Nat. 129: 221-233.

GRAFEN, A., 1979 The hawk-dove game played between relatives.

Anim. Behav. 27: 905-907.

HARPER, A. B., 1986 The evolution of begging: sibling competi-

tion and parent-offspring conflict. Am. Nat. 128: 99-1 14.

HINES, W. G. S., 1987a Can and will a sexual population evolve

to an ESS: The multi-locus linkage equilibrium case. J. Theor.

Biol. 126: 1-5.

HINES, W. G. S., 1987b Evolutionarily stable strategies: a review

of basic theory. Theor. Popul. Biol. 31: 195-272.

KARLIN, S., and C. MATESSI, 1983 Kin selection and altruism.

Proc. R. SOC. Lond. Ser. B 2 1 9 327-353.

LANDE, R., 1975 The maintenance of genetic variability by mu-

tation in a polygenic character with linked loci. Genet. Res. 26:

LESSARD, S. 1984 Evolutionary dynamics in frequency-dependent

two-phenotype models. Theor. Popul. Biol. 25: 210-234.

MAYNARD SMITH, J., 1981 Will a sexual population evolve to an

MAYNARD SMITH, J., 1982 Evolution and the Theory of Games.

MAYNARD SMITH, J., and G. R. PRICE, 1973 The logic of animal

MICHOD, R. E., 1982 The theory of kin selection. Annu. Rev.

MORRISON, D. F., 1976 Multivariate Statistical Methods. McGraw-

515-518.

221-235.

ESS? AIII. Nat. 117: 1015-1018.

Cambridge University Press, Cambridge.

conflict. Nature 2 4 6 15- 18.

EcoI. Syst. 13: 23-55.

Hill, New York.

MUELLER, L. D., and M. W. FELDMAN, 1985 Population genetic

theory of kin selection: a two-locus model. Am. Nat. 125: 535-

549

PARKER, G. A., and M. R. MACNAIR, 1979 Models of parent-

offspring conflict. IV. Suppression: evolutionary retaliation by

the parent. Anim. Behav. 27: 1210-1235.

SCHUSTER, P., K. SIGMUND, J. HOFBAUER and R. WOLFF,

1981 Selfregulation of behaviour in animal societies. I. Sym-

metric contests. Biol. Cybernet. 4 0 1-8.

selection. Am. Nat. 112: 845-859.

SLATKIN, M., 1978 On the equilibration of fitnesses by natural

SLATKIN, M., 1979 Frequency- and density-dependent selection

SLATKIN, M., 1980 Ecological character displacement. Ecology

on a quantitative character. Genetics 93: 755-77 1.

61: 163-177.

THOMAS, B., 1984 Evolutionary stability: states and strategies.

Theor. Popul. Biol. 2 6 49-67.

Toro, M., R. Abugov, B. Charlesworth and R. E. MICHOD,

1982 Exact versus heuristic models of kin selection. J. Theor.

Biol. 97: 669-7 13.

TURELLI, M., 1984 Hereditable genetic variation via mutation-

selection balance: Lerch's Zeta meets the abdominal bristle.

Theor. Popul. Biol. 2 5 138-193.

TURELLI, M., 1988 Population genetic models for polygenic var-

iation and evolution, pp. 601-618 in Proceedings ofthe Second

International Conference on Quantitative Genetics, edited by B. S.

WEIR, E. J. EISEN, M. M. GooDMANand G. NAMKOONG. Sinauer

Associates. Sunderland, Mass.

UYENOYAMA, M. K., 1984 Inbreeding and the evolution of altru-

ism under kin selection: effects on relatedness and group struc-

ture. Evolution 38: 778-795.

UYENOYAMA, M. K., and M. W. FELDMAN, 1981 On relatedness

and adaptive topography in kin selection. Theor. Popul. Biol.

UYENOYAMA, M. K., M. W. FELDMAN and L. D. MUELLER,

1981 Population genetic theory of kin selection: multiple

alleles at one locus. Proc. Natl. Acad. Sci. USA 78: 5036-5040.

YOKOYAMA, S . , and J. FEUENSTEIN, 1978 A model of kin selection

for an altruistic trait considered as a quantitative character.

Proc. Natl. Acad. Sci. USA 75: 420-422.

19: 87-123.

Communicating editor: M. TURELLI

APPENDIX A

The fitness function arising from painvise interactions

We want to obtain the genotypic fitness function given the phenotypic fitness function w and the phenotypic distribution associated with a genotype. We use G(lX,

cy)

for the expected fitness of an individual with breeding value <x

when interacting with an individual with breeding value lY, and GI({.) for the expectation of G(cx,

cy)

over

cy.

Equation

11 shows G ( Q as a function of the phenotypic fitness function and the phenotypic distribution; (1 2) gives us the definition ofG(S;,

cy).

Equation 13 gives us the phenotypic fitness as a second-order approximation of the fitness function using a Taylor series expansion about a specific phenotype,

ZO, letting uz, v,, v,, vxy, and v, be the partial derivatives of the logarithm of the phenotypic fitness function with respect to the subscript values. It should be noted that unless v,,

and v, are negative, and vxy suitably bounded, some fitnesses will be indefinitely large, giving absurd results when there is large phenotypic variation. From ( 1 2) we have:

G(cx,

cy)

is then the value within the braces, and can be obtained in closed form:

(12)

888 A . B. Harper

and

c = 1 - E(v,,

+

2pt:v,

+

vyy)

( A 2 4

+

E‘(1

-

pE)(v,,v,

-

?I:,).

In Equations A2 the u, etc. are the partial derivatives of the logarithm of the phenotypic fitness function w(z,, z,). As can be seen from (A2b); the S, etc. are the partial derivatives of the genotypic fitness function

zi,(cx,

{,) evaluated at = {, =

zO. T h e phenotypic and genotypic fitness functions will usually differ due to the effects of environmental variance. We now want to obtain (Al) in closed form. T h e distribution of the breeding values is known from (9):

Let {x,

cy,

and MI be measured as deviations from the

phenotype about which we have expanded the fitness function:

{x = zo

+

6 ~ , {, = zo

+

d,, M, = zo

+

6u, (A4)

and substitute into (A3)

Substituting (A5) and (A2) into (Al), we can find the fitness of an individual whose breeding value differs by 6, from Zo:

G(6,) =

s

W ( z 0

+

a,,

Z O

+

6y)p(6,

I

Q d 6 ,

J

exp(S,6,

+

Cy6,

+

5,,62/2

+

C,6x6y

+ ~,,6;/2)P(6Y

I

6 W Y (A6)

where

= exp(A6f/2

+

B6,),

a, = C ,

-

pc( 1

-

pc)&,/G ’

a, = 5,

-

PC( 1

-

pc)GM/G ’

ax, = Sxy

+

pc/G ’

a, = C,

-

l/G’ (A7) axr = G,, - p?;/G ‘ G

‘

= (1

-

p%)G,

if a, < 0. (When a,, > 0 the integral diverges). Equation A6 is in the form of (7b); (A6) and (8) are the recursion equations for this model. We can see that the value of A in (A6) is independent of 6” and therefore the recursion Equations 8a and 8b are independent of 8c.

APPENDIX B

Stability criteria

There are two types of equilibria for Equations 8, either there is no selection on the character, A = B = 0, and G has some value G , or there is no heritable variation and G = 0. T o find the stability of these classes of equilibria, we must look at the first derivatives of (8). Because (8a, b) are independent of M , we can consider the stability of these two equations together, and then consider the stability of (8c). T h e Jacobian matrix for [V, C]‘ at G = G is

i:

2aA/av

6 ‘aA/av

(n

-

1)6?aA/av (n

-

ly?aA/av

I + -

2n 2n

, ( B l 4

2n 1 - r + 2 n

and the Jacobian at G = 0 is

Examining the G =

6

equilibrium first, (Bla) implies that

it is stable to perturbations in V and C just when dA/dV < 0. T h e stability conditions for M are determined by the sign of aM,+,/aM,. Since A = 0 at this equilibrium, (Sc) gives us

aM,+l/aM, = 1

+

daB/aM. (B2)

Equations Bla and B2 determine the following cqnditions for the existence of a stable equilibrium with G = G :

T h e first two conditions define the existence of the equilibrium, the third is the requirement that the genetic variance be non-negative, the fourth comes from APPENDIX A as a condition for (A6) to converge, and the last two are the conditions for the stability. If the condition a, < 0 fails, then the average fitness of the population,

IJ

{,).

p,(S;,

{y)d{xd{y, is infinite, suggesting that G is so large that

zi, is not adequately approximated byAa quadratic function. Using

(46)

apd (A7), the value G that gives A = 0 can be found: G = G ’/( 1 - pz),

where

or

T h e first equilikrium is an artifact of the order of integration in (AI), as at G‘ = 0 we are not integrating over a well- defined density. At the second, w e have

T h e following argument shows that the equilibrium (B4b) is always unstable when it exists. From (A7), a, =

(G’G, - l)/G’, which is <O by assumption, implying G’S, < 1. In order for aA/aV < 0 we would need the numerator of (B5) to be positive, or S..Sw - Sf > 0. Now note that using (B4b)

Since the denominator of (B6) must be positive, we have an

> 0 and we find that conditions (B3) cannot all be true. This sho-ws, in effect, that if there were a stable equilibrium at G

= G ’, then the average fitness of the population would be infinite, which suggests that such equilibria could exist if either w were a more complicated function, not approxi- mated by a second-order Taylor series (SLATKIN 1980), or if the distribution of phenotypes were limited to a finite set of values. It is difficult to deal with such cases analytically.

We can now examine the stability conditions of the G =

0 equilibrium. We see that the Jacobian matrix (B1 b) implies stability in C if r > 0 but neutral stability in V. To determine the stability in the V direction, we must look at a ‘ V , + ~ / a V ~ at V = C = 0, if this second derivative is <0 then the equilibrium is stable to all perturbations in V and C. This second derivative is A/n, and we see that the G = 0 equilibrium is stable if

A < 0. 037)

(13)

Quantitative Evclutionary Stability 889

straightforward, since (8c) implies that any value of M is a neutrally stable equilibrium for G = 0. As suggested by

FELSENSTEIN (1979) we can expand the equations out, not at G = 0, but at some small value for G , G = 6 ~ . This would be appropriate if there were a constant flux of mutations that increase the variance of the trait. If M changes when G is small but non-zero, then it seems unlikely that this value of M would be maintained in a population. It should be noted that in the recursion equations V , and C, enter only as G, = V,

+

C,, so that we do not have to distinguish between the mutational effects at one locus and effects between loci. Substituting G = 6C into (8c) we see that the equilibrium value mean phenotype is

M = - B / A , (B8) and that dM,+,/dM, < 1 when

A

+

dB/dM < 0 . (B9) In order to compare the above to the ESS conditions, we need to evaluate (B7-9) when the point, zO, around which

we have approximated the phenotypic fitness function, w , is the equilibrium, M . This will give us conditions which the partial derivatives of w , evaluated at a certain phenotype, must satisfy in order for that phenotype to be a stable

equilibrium of the model. I t is also convenient to choose a scale of measurement such that the equilibrium phenotype

M is 0. Evaluating the conditions above (in the order B8, B7, B9) at zO = &, = M = 0 , using (A6) and (A7), gives us

the conditions listed in (1 4a-c).

T o compare the conditions (14) with the ESS conditions (4), we need to find the ESSs for the model developed above. We do this by retaining the effects of environmental variation, but replacing the assumption of quantitative genetic variation with the ESS assumption that the population is genetically monomorphic except for the invasion of a single rare allele. When there is environmental variation in the expression of the phenotype with variance E and covariance between the interactants of p,&, then fitness of individuals with breeding values x and y is ;(x, y), where zi, is defined in (A2b). To find the ESS, we just apply the ESS conditions (4) to (A2b) at x = y = ZO, giving:

5,

+

?+y = 0, (B 1 Oa)

S,

+

27-5,

+

.;YY

+

r(1

-

r)S; < 0, (Blob) and

Cxx

+

(1

+

r)C,

+

ri& < 0. (B 1 Oc)

In Equations B10, r is the probability of a rare genotype interacting with another rare genotype; this is the same as the genetic correlation between interactants, p C , so that conditions (B10) can be compared to (14) by substituting p C