An Application of Yield Management for Internet Service Providers

Internet Service Providers

1_{Department of Operations and Information Management, School of Business} Administration, U41-IM, University of Connecticut, Storrs, Connecticut 06269-0241

2_{College of Business Administration, 214 Hayden Hall, Northeastern University,} Boston, Massachusetts 02115

Received March 1999; revised January 2001; accepted 31 January 2001

Abstract: In this paper we study strategies for better utilizing the network capacity of Internet Service Providers (ISPs) when they are faced with stochastic and dynamic arrivals and depar-tures of customers attempting to log-on or log-off, respectively. We propose a method in which, depending on the number of modems available, and the arrival and departure rates of different classes of customers, a decision is made whether to accept or reject a log-on request. The problem is formulated as a continuous time Markov Decision Process for which optimal policies can be readily derived using techniques such as value iteration. This decision maximizes the discounted value to ISPs while improving service levels for higher class customers. The methodology is sim-ilar to yield management techniques successfully used in airlines, hotels, etc. However, there are sufficient differences, such as no predefined time horizon or reservations, that make this model interesting to pursue and challenging. This work was completed in collaboration with one of the largest ISPs in Connecticut. The problem is topical, and approaches such as those proposed here are sought by users.c 2001 John Wiley & Sons, Inc. Naval Research Logistics 48: 348–362, 2001 Keywords: yield management; internet service providers; continuous time MDP

1. INTRODUCTION AND MOTIVATION

Internet Service Providers (ISPs), companies that are engaged in providing direct online ac-cess to the Internet to individuals and corporations, are increasingly challenged to keep up with competition from other ISPs and the so-called ‘‘commercial services’’ like America Online and Prodigy. These commercial services do provide access to the Internet, but their real draw is their proprietary content that reflects their tie-ups with particular merchandisers, television channels, and business associates. ISPs have to deal with rapid technological advancements and growing demand from an expanding customer base while at the same time maintaining a desirable cus-tomer service level. It is not uncommon to hear of frustrated users who have been ‘‘bumped,’’ or simply disconnected from the network arbitrarily, or of others who struggle to get in at peak hours—typically lunch hours on business days, evenings, or on days of inclement weather. Recall

Correspondence to:S.K. Nair c

the problems that America Online (AOL) faced due to capacity limitations. AOL had to spend about $350 million to add capacity to overcome these problems.

In this paper we study optimal strategies for utilizing the network capacity of ISPs when they are faced with stochastic arrivals and departures of customers attempting to log-on or -off, respectively. The optimal policy maximizes discounted net profits over an infinite plan-ing horizon for the ISPs while improvplan-ing service levels for higher class customers. This research was conducted in collaboration with one of the largest ISPs in Connecticut.

Our approach is based on the general yield management framework that has been successfully applied to a broad spectrum of applications in the service sector. Pioneering efforts of American Airlines [2] were successfully replicated by the other airlines and by companies like Hertz for car rentals (Carroll and Grimes [5]), the leading hotel chains for rooms, and United Artists for managing the sales of movie rights. American Airlines estimated a quantifiable benefit of $1.4 billion over the period 1989–1992 and expects an annual revenue contribution of over $500 million to continue into the future (Smith, Leimkuhler, and Darrow [17]). Recently the term perishable-asset revenue management or PARM has been coined (Weatherford and Bodily [20]) to cover the separate but related problems of yield management, overbooking, and pricing. We shall stick with the term yield management (YM) which in essence is an integrated demand-management, overbooking and capacity utilization system that focuses on improving a strategic objective, such as revenue or service level, by carefully setting differential treatments for various market segments and dynamically reallocating fixed but perishable capacity between segments. The perishability of the resource, that is, the existence of a date or time after which it is either unavailable or it ages at a significant cost—distinguishes this class of problems from inventory control problems (Weatherford and Bodily [20]). Airline seats, theater seats, hotel rooms, fashionable clothing, and traffic on communication channels are a few examples of such perishable assets. Airlines, for instance, deny advance bookings to price-sensitive customers for peak travel periods because they anticipate enough demand from the higher paying customers. We are not aware of any work in yield management for ISPs. Based upon our discussions with our collaborators, whom we will call Deciles, Inc., the Internet services division of a regional telephone company and a recent entrant in the field of ISPs; and after a close examination of their operational topology, we identified the presence of three common characteristics as articulated by Weatherford and Bodily [20] with situations where yield management is currently practiced.

Perishable assets.Although there exist many stages that a user dialing-in from his or her home or office needs to cross before reaching the Internet backbone, as exhibited in Figure 1, the bottleneck points were identified by the managers of Deciles as the ISP’s modem racks. A modem rack is a set of 24 modems that is typically connected to a T1 type transmission channel that serves as a gateway for users trying to log-on. The number of modem racks a given access point has is proportional to the population of the area the access point covers. A modems to users ratio (MUR) such as 1:10 (all numbers are camouflaged to protect proprietary data) is planned, and this significantly influences service. Obviously, a 1:8 MUR will provide better service and a higher chance of access to the network than an MUR of 1:10. Access points are located across Connecticut such that any user could dial into the system using a local number. Thus the perishable asset here is the modem capacity available at any instant of time, at any access point, for users to log onto. This capacity perishes and is regenerated with time and is dependent on the rate of users logging on and logging off over time.

A fixed number of units.Based on the 1:10 MUR, a total of 1200 modems or 50 modem racks are distributed across the state of Connecticut. This allows Deciles to service a maximum of 1200 customers at any one time on the network and cover a population of 12,000(1200×10). Thus the

Figure 1. Steps involved in connecting a user to the Internet.

assumption is that 1 in 10 customers would log-in at any time. In case this assumption is violated, then the service quality would deteriorate and customers would either be denied access, bumped, or asked to retry. Thus the total number of modems in the system is the maximum capacity available. While the above-mentioned scenario best represents the current capacity constraints of ISPs, future trends in Internet access technology indicate the likelihood of packet-switching approaches that may not require a dedicated modem. In such systems the densities of information exchanges is more significant than the number of modems. We defer the analysis of such systems to future research. Deciles needs to manage its fixed perishable capacity in the face of uncertain demand. At a strategic level, the number of modems used, and its location around the state is another interesting optimization problem. This compares directly with the well-known news vendor problem where the vendor must decide how many papers to order in the face of uncertain demand. However, in that problem the order quantity may be varied without much expense and the decision is tactical in nature, whereas in the ISP situation the problem is strategic with major capital expenses involved.

The possibility of segmenting price-sensitive customers. Unlike airlines, ISPs do not at present differentiate between their customers. Instead they offer a variety of services based on the number of access hours. Typically a base fee is charged which allows up to certain fixed number of Internet access hours. Additional access hours are charged on an hourly basis. Another approach adopted by providers is to provide unlimited access for higher fees. We do not consider the case where providers, such as Netzero.com, provide free dial-up service and rely solely on advertising for their revenues. Based on our discussion with the managers at Deciles, Inc., we proposed to segment their customers into two classes which we callPlatinumandGold. The basis for this segmentation is quality of service. Platinum customers would be guaranteed a higher quality of service which we shall define as the ‘‘probability of getting access to the Internet when they

attempt to dial-in.’’ For this they would pay higher fees compared to gold customers who would correspondingly receive a lower quality of service. To demonstrate how an ISP can operationalize such a segmentation is one of the goals of this research. Customer segmentation gives us the opportunity of introducing incentive mechanisms similar to those pursued by airlines. One can imagine free upgrades to higher classes conditioned upon certain numbers of logged hours like the airlines do with air miles. Also due to the recent telecommunications deregulation bill, competition has heated up for both the long distance and local service, forcing Deciles to think of innovative services to provide to customers.

The remainder of the paper is organized as follows. Section 2 discusses the differences between YM for ISPs and traditional YM applications, and linkages of this work to the admission control in queueing literature. Section 3 presents the problem formulation and the continuous time Markov decision process model, and Section 4 discusses the solution methodology. Finally, Section 5 concludes by summarizing the research effort and discusses directions for future research.

2. LITERATURE REVIEW

2. 1. Differences between YM for ISPs and Traditional YM Applications

As one hotel industry expert put it ‘‘yield management is charging a differentrate for the sameservice to adifferentindividual’’ (Nykiel [15]). In effect all yield management applications attempt to arrive at an optimal tradeoff between average price paid and capacity utilization. This is achieved by making decisions at two levels. First, at thetacticallevel, decisions have to made that determine:

1. Aggregate capacity. For the airlines it could be the fleet size and mix or, for hotels, the number of rooms. In our case it is the number of modems racks and their location. The location problem here is not different from the typical set covering or the maximal population covering models, on which extensive work has been done (Fisher and Kedia [7]).

2. Market segmentation policy. The key decisions here would be to arrive at the number of segments and thevalueof each segment. Care should be taken that arbitrage opportunities—the possibility that customers belonging to the low-price segments acquire goods or services expressly for resale to others in higher price segments—do not exist. Airlines, for instance, have restrictions such as mandatory Saturday night stayover and making tickets nonrefundable in event of cancellations. It is here that yield management interacts with marketing and uses theories ofquantity discountingortime of purchaseto segment customers. 3. Price setting. Based upon the market segmentation decisions and using knowl-edge of demand patterns, decision-makers have to decide what price is appropri-ate for a particular segment. For ISPs, this may be a flat fee or a pay-as-you-go approach.

In close relation to the tactical level decisions there are theoperationallevel decisions. Here the primary concern is to make optimal day-to-day decisions that ensure the attainment of specified objectives. Hotels, for instance, have to decide whether or not to accept a customer’s request for a reservation based on the number of available rooms belonging to that customer’s segment and the expected mix of customers that the hotel expects in the remainder of the time horizon, which is usually 6 p.m. each day. Airlines make similar decisions on requests for seat reservations

on particular flights. There are three issues that make YM in ISPs different from airline/hotel applications from a modeling standpoint:

1. The ISP problem is inherently continuous both in state and time. True capacity is ‘‘modem-hours’’ rather than number of modems, and customers draw upon this capacity in a continuous fashion. There is no natural cutoff time which could be used as a time horizon to solve the problem. Airlines use flight takeoff time and hotels use 6 p.m. For ISPs, customers log-on and -off all day long. 2. Service is determined by the time it takes to get on the network. Thus the request

and the service happen simultaneously. This is not the case in airlines and hotels where the request is made at one time (making the reservation) and the capacity is used up at another (the flight taking off or the hotel room gets occupied). 3. For the above reason, overbooking is not an issue in YM for ISPs.

We address the first problem by formulating our problem as an infinite horizon, homogeneous, continuous-time Markov decision process (CTMDP) for which optimal policies are readily derived using standard techniques such as value iteration.

The second issue creates an operational problem. How does one give priority to platinum customers over gold customers without knowing beforehand which type of customer is making the call. Once the customer seizes the modem, she is on the network and the service is already initiated. Deciles wanted to avoid bumping customers once they were on the network. We worked around this problem by proposing to supplement the customer-authentication process that is activated every time a customer tries to log-on. Here our algorithm could be used to decide, based on the class of the customer (eitherPlatinumorGold), whether to let the customer log-on or not. This decision would require knowledge of the number of modems available at the time, the customer profile, and the expected mix of customers that may attempt to utilize the service in the near future. An alternative approach would be to utilize dynamic pricing that would, based on the system load, indicate to customers the spot premium they could pay to obtain a higher class of service. In this paper we assume that customer segmentation is undertakena priori.

The existing research has focused primarily on the operational level (Bitran and Mondschein [3] and Lee and Hersh [11]). Most of the yield management literature assumes simplified dynamic relationships about customer behavior during the planning horizon. Alstrup et al. [1] assume that all customers arrivesequentiallyduring the target date while studying booking policies for a single flight leg with two types of customers. Ladany [9] makes a similar assumption in his formulation for managing reservations in the motel industry.

Lee and Hersh [11] relax this assumption for the airline reservation problem and do not require any advance knowledge about the arrival pattern for the various booking classes. They generate critical valueswhich represent the optimal policy for making accept/reject decisions provided that the systems parameters remain constant, an approach followed by us.

Bodily and Weatherford [4] and Weatherford and Bodily [20] present a comprehensive taxon-omy for general PARM problems. Table 1 uses their taxontaxon-omy to place our problem in comparison with the airline and hotel problems as formulated by Lee and Hersh [11] and Bitran and Mond-schein [3], respectively.

Lautenbacher and Stidham [10] describe the underlying Markov decision process in the single-leg airline yield management problem. They introduce the termsstaticanddynamicfor approaches that allow customers of different classes to book concomitantly, and those that assume the de-mand for different fare classes arrive in some predetermined order, respectively. We model the ISP problemdynamically. Subramanian, Stidham, and Lautenbacher [19], present a dynamic pro-gramming approach to the airline yield management problem with overbooking, cancellations,

Table 1. Taxonomical comparison of selectY Mmodels, based on Bodily and Weatherford [4].

Elements Airlines Hotels ISP

Resource Discrete Discrete Continuous

Capacity Fixed Fixed Fixed

Cutoff time Yes Yes No

Prices Predetermined Predetermined Predetermined

Willingness to pay Buildup Buildup Not applicable

Discount price classes k k k

Arrival pattern Stochastic Stochastic Stochastic

Departure pattern No No Stochastic log-offs

Show-up of discount reservation Certain Certain Stochastic

Show-up of full price Certain Certain Stochastic

Group reservations Yes Yes Not applicable

Overbooking Yes Yes Not applicable

Diversion No No Possible

Displacement No Downgrading Not applicable

Bumping procedure None None None

Asset control mechanism Nested Nested Nested

Decision rule Dynamic Dynamic Dynamic

and no shows. Bitran and Mondschein [3] develop renting policies for hotels making no assump-tions concerning the particular order between the arrival of different types of customers which is treated as a stochastic process. They propose heuristics for problems that involve multiple night stays based upon results obtained from the single-night case. This issue is particularly relevant for our problem, because, in contrast to business travelers who visit airport hotels and motels for single nights (Ladany [9]), virtually no user shall log-on to the Internet for just a single decision period. Typically people log onto the Internet to visit some web sites or browse through contents of newsgroups, both of which consume time. We have addressed this issue by not just considering the number of users attempting to log-on but also the number logging-off during a decision period. Observed distributions of these two phenomena are closely related as evident in Figure 2. One of the conditions in our model is that customers are not bumped once they have been connected, which is an undesirable business practice in the first place.

2. 2. Linkages to the Admissions Control in Queueing Literature

Another perspective of the YM for ISPs problem can be obtained from examining the vast literature on admission control in queueing systems. Stidham [18] reviews this branch of the literature for static and dynamic systems, in the case of single and multiple servers. This research stream (Naor [14], Yechiali [22], and Mendelson and Yechaili [13]) brings to light the effect of negative network externalities imposed by individually optimizing decision rules. Such rules neglect the impact of a given job on the performance of the system as a whole. These negative externalities translate into deteriorated performance for existing users as a result of admission of additional jobs and are typically remedied by congestion tolls. Similarly, in our setting, a short-sighted individually optimizing approach would accept any class of customer provided there was any available capacity and a nonnegative revenue contribution. This, however, would decrease the probability of higher-valued customers getting access in the successive epochs and would not lead to maximal long-term expected revenue, as shown in Section 4. In light of the above, we

Figure 2. Log-in/off distribution by quarter hour for Deciles, Inc.

borrow from this literature the overall approach of considering the impact of a log-on request on the long-term benefit from the system as a whole.

In the analysis of many traditional queueing systems (Stidham [18]), jobs are buffered if no server is immediately available, and servers are assumed to have a certain processing rate. Both of these factors may ultimately influence important parameters like the average response time and the long-run average number of jobs in the system. Contrast this with the YM for ISPs case where the average response time and the average number of jobs in the system are not of concern and the jobs have random durations that are dependent on the web-browsing behavior of the consumers, who cannot be buffered. We assume that the ISP desires to maximize long-term expected discounted benefit given a modem capacity constraint and multiple customer segments. In queueing, other criteria have been used, such as the power criteria proposed by Kleinrock [8] which takes the ratio of the throughput and the average response time to achieve admission control.

Xu and Shantikumar [21] introduce a dual system to address optimal admission control for a first-come first-served ordered-entryM/M/mqueueing system that maximizes expected dis-counted profit. The dual approach constructs an isomorphic preemptive last-come, first-served ordered-entry queueing system that is subject to expulsion control. Their unique approach deter-mines the solution from the behavior of individual customers and could be used in the YM for ISPs problem. Applying their approach to our setting with multiple service classes would provide an alternative to the conventional dynamic programming approach based on value iteration, and is an exciting direction for future research.

Additional related streams of work can be found in providing admission control in ATM net-works that allow for multiple service classes (e.g., audio, video, text) that could have different quality service requirements. Elwalid and Mitra [6] provide analytic approximations, using which they calculate the admissible set but fail to provide an economic objective function which is a must in our setting.

In summary, there is a close association with the various aspects of the admission control literature with the YM problem for ISPs. By formulating our problem as a CTMDP, as done by

Puterman [16], and using value iteration to solve for the optimal control policy, we do draw upon the findings of this interesting research stream.

3. PROBLEM FORMULATION

We consider the case where ISPs have segmented their markets into two classes,Platinum denoted by 1 andGolddenoted by 2. Letλ1andλ2represent the Poisson arrival rates of platinum and gold class customers, respectively. Also, assuming exponential stays letµ1andµ2represent the service rates of platinum and gold class customers, respectively.

The problem is formulated as a queuing admission control model in the form of a continuous-time Markov decision process (CTMDP), a special case of the general semi-Markov decision process. In such systems, intertransition times are exponentially distributed, and actions are cho-sen at every decision epoch. In what follows, we use notation and terminology consistent with Puterman [16]. Our results are an extension of his discussion of the single server single customer class queue admission problem to amultiserversituation withmultiple customer classes.

We assume that the ISP’s objective is to maximize theexpected total discounted rewardwhere α >0is the continuous time discount rate,α= ln(λ), whereλis the discrete time discount rate, λ= 1/(1 +i), whereiis the interest rate.

Suppose the state is defined ashi, j, biwhen there areiplatinum andjgold customers in the system, and eventboccurs. We denoteb = 1as a platinum arrival, andb = −1as a platinum departure. Similarly, we denoteb= 2as a gold arrival andb=−2as a gold departure. Suppose that there are a total ofMmodems available at the ISP; then clearlyi+j≤M at all times.

In this situation, the decision epochs are when a customer arrives or departs. When a customer arrives, the possible actions,a, would be to admit the customer, denoted bya= 1, or to refuse service to the customer, denoted bya= 0. When there are no arrivals, or when there is a departure, the only action possible would be to continue, also denoted asa= 0.

Letβijbabe the rate of transition out of statehi, j, bigiven actionais taken. Let the transition

probabilities be denoted byq(k|hi, j, bi), which is the probability of transition from statehi, j, bi to statek.

In statehi, j,1i, wherei+j < M, that is, when a platinum customer arrives and there are still modems available; when the action is to admit, that is,a= 1, the transition probabilities can be written as q(k|hi, j,1i) =          (i+ 1)µ1/βij11, k=hi, j,−1i, jµ2/βij11, k=hi+ 1, j−1,−2i, j ≥1, λ1/βij11, k=hi+ 1, j,1i, λ2/βij11, k=hi+ 1, j,2i, (1)

where the rate of transition out of the state,βij11 = (i+ 1)µ1+jµ2+λ1+λ2. These can be explained in the following manner: We use the convention of noting the state just after a departure and just before an arrival. In statehi, j,1i, when the decision is to accept the platinum customer that arrives,a= 1, the number of platinum customers in the system gets incremented toi+ 1. The next event could be a platinum departure,b =−1, a gold departure,b = −2, a platinum arrival,b = 1, or a gold arrival,b = 2. That is, when there is a platinum departure the system would transition to statehi, j,−1iat a rate of(i+ 1)µ1since there arei+ 1customers in the system, when there is a gold departure, the system will transition to statehi+ 1, j−1,−2iat a rate ofjµ2, when there is a platinum arrival, the system will transition to statehi+ 1, j,1iat a rate ofλ1, and when there is a gold arrival, the system will transition to statehi+ 1, j,2iat a rate

ofλ2. Since any of these four events could occur next, the next decision epoch occurs at the rate ofβij11= (i+ 1)µ1+jµ2+λ1+λ2. Therefore, the transition probability of going from state hi, j,1ito statehi, j,−1iis(i+ 1)µ1/βij11, the transition probability to statehi+ 1, j−1,−2iis jµ2/βij11, the transition probability to statehi+1, j,1iisλ1/βij11, and the transition probability to statehi+ 1, j,2iisλ2/βij11.

In statehi, j,2i, wherei+j < M, that is, when a gold customer arrives and there are modems available; when the action is to admit, that is,a= 1, the transition probabilities can be written as

q(k|hi, j,2i) =          iµ1/βij21, k=hi−1, j+ 1,−1i, i≥1, (j+ 1)µ2/βij21, k=hi, j,−2i, λ1/βij21, k=hi, j+ 1,1i, λ2/βij21, k=hi, j+ 1,2i, (2)

where the rate of transition out of the state,βij21=iµ1+(j+1)µ2+λ1+λ2. The explanation is similar to that given above noting that when the decision is to accept a gold customer, the number of gold customers in the system get incremented toj+ 1.

In all stateshi, j, bi, wherei+j ≤M andb ∈ {−2,−1,1,2}, when the action is to refuse service or continue,a= 0, the transition probabilities can be written as

q(k|hi, j, bi) =          iµ1/βijb0, k=hi−1, j,−1i, i >1, jµ2/βijb0, k=hi, j−1,−2i, j >1, λ1/βijb0, k=hi, j,1i λ2/βijb0, k=hi, j,2i, (3)

where the rate of transition out of the state,βijb0=iµ1+jµ2+λ1+λ2. Again the explanation is similar to the one above, noting that since the action is to refuse entry to the customer, the number of platinum and gold customers is not immediately incremented. Note that

βij11=β(i+1)jb0, (4)

βij21=βi(j+1)b0. (5)

Suppose every arriving platinum customer contributes a revenue ofK1, and every arriving gold customer a revenue ofK2, as modeled in Puterman [16]. Let the system holding cost due to the use of network infrastructure be at a rate ofc1(i)when there areiplatinum customers in the system, andc2(j)the corresponding rate for gold customers when there arej gold customers in the system. We assume thatc1(i)andc2(j)are nondecreasing and convex,K1 ≥ K2 and c1(i) ≤ c2(i)for alli. As a first approximation, the revenue could be estimated by the ratio of total membership fee and other revenue from a class of customers to the average number of log-ons in a month by that class of customer. Similarly, the holding cost rate could be estimated by apportioning the network and overhead costs based on usage by each class of customer. This approach to estimating costs would be akin to estimating ordering and holding costs in inventory models.

The expected discounted rewardr(hi, j, bi, a), between decision epochs, given the system is in statehi, j, biand actionais taken, could be written as (from Puterman [16, 11.5.3])

r(hi, j,1i,1) =K1−[c1(i_α+ 1) +_+β c2(j)]

r(hi, j,2i,1) =K2−[c1(i) +_α+c_β2(j+ 1)]

ij21 , (7)

r(hi, j, bi,0) =−[c1(_αi₊) +_βc2(j)]

ijb0 (8)

forb∈ {−2,−1,1,2}.

We analyze the model by using uniformization (Lippman [12]). The idea of uniformization is to convert a continuous time model with state-dependent exponential transition rates (theβijba

stated above) to a model with a state-independent transition rate,C. The trick is to ensure that the probabilistic behavior of the uniformized model be same as that of the original model. This can be done by ensuring that theinfinitesimal generatorof the two models is identical. We pick a constant transition rate that would be larger than all possible transition ratesβijba. A look at

equations forβij11, βij21, andβijb0will show that the maximum value that these could achieve would be

C=Mmax(µ1, µ2) +λ1+λ2.

Since the constantC is chosen to be at least as large as the largest rate seen in the original system, uniformization results in more frequent transitions than in the original system. One could think of these as ‘‘fictitious’’ transitions from a state to itself. Because of this, we need to adjust the transition probabilities and rewards under the uniformized system (designated with a tilde) as follows:

˜

q(k|hi, j, bi) =

1−[1−q(khi, j, bi)]βijba/C, k=hi, j, bi,

q(k|hi, j, bi)βijba/C, k6=hi, j, bi, (9)

˜

r(hi, j, bi, a) =r(hi, j, bi, a)α+βijba α+C . The optimality equation has the form

V(hi, j, bi) = max a ( ˜ r(hi, j, bi, a) +_α+C_C X k ˜ q(k|hi, j, bi)V(k) ) ,

and from Puterman [16, Theorem 11.5.3, p. 566], if a maximum is attained for eachV(·)above, then a stationary deterministic optimal policy exists, since the action space is finite and compact, and the rewards, transition rates, and transition probabilities are continuous inafor each state.

We can also write these value functions in expanded form as

Vhi, j,1i= max          (α+βij11)K1−[c1(i+1)+c2(j)] α+C +α+1C [(i+ 1)µ1V(hi, j,−1i) +jµ2V(hi+ 1, j−1,−2i) +λ1V(hi+ 1, j,1i) +λ2V(hi+ 1, j,2i) + (C−βij11)V(hi, j,1i)] V(hi, j,−1i), Vhi, j,2i= max          (α+βij21)K2−[c1(i)+c2(j+1)] α+C +α+1C [iµ1V(hi−1, j+ 1,−1i) + (j+ 1)µ2V(hi, j,−2i) +λ1V(hi, j+ 1,1i) +λ2V(hi, j+ 1,2i) + (C−βij21)V(hi, j,2i)] V(hi, j,−1i),

where

V(hi, j,−1i) = −c1(i_α) +_+Cc2(j)+_α+1_C[iµ1V(hi−1, j,−1i) +jµ2V(hi, j−1,−2i) +λ1V(hi, j,1i) +λ2V(hi, j,2i) + (C−βijb0)V(hi, j,−1i)]. By taking the last term to the left-hand side and simplifying, we get

V(hi, j,−1i) =−c1(i) +c2(j) α+βijb0 +

1

α+βijb0[iµ1V(hi−1, j,−1i)

+jµ2V(hi, j−1,−2i) +λ1V(hi, j,1i) +λ2V(hi, j,2i)] (10) andV(hi, j,−2i) =V(hi, j,−1i).

From the above equations and from (4) and (5) we can also see that Vhi, j,1i= max    (α+βij11)K1 α+C +V(hi+ 1, j,−1i) − C α+C [V(hi+ 1, j,−1i)−V(hi, j,1i)] V(hi, j,−1i), which simplifies to Vhi, j,1i= max K1+ V(hi+ 1, j,−1i) V(hi, j,−1i). (11) Similarly, Vhi, j,2i= max K2+V(hi, j+ 1,−1i) V(hi, j,−1i). (12) 4. SOLUTION METHODOLOGY Let ∆ij1=K1+ [V(hi+ 1, j,−1i)−V(hi, j,−1i)], (13) ∆ij2=K2+ [V(hi, j+ 1,−1i)−V(hi, j,−1i)], (14) which is the difference between the accept and reject rows in (11) and (12). It is clear then from (13) and (14) that, in stateVhi, j,1i, the optimal decision would be to accept the platinum customer if∆ij1 >0, and in stateVhi, j,2ithe optimal decision would be to accept the gold customer, ∆ij2>0. We will show next that∆ij1and∆ij2are nonincreasing iniandj.

PROOF: We prove these by induction using a policy iteration approach. SetV0 _{= 0; then} from (10) we have V1_{(hi+ 1, j,−1i)−V}1_{(hi, j,−1i) =−}c1(i+ 1)−c1(i) α+βijb0 , V1_{(hi, j+ 1,−1i)−V}1_{(hi, j,−1i) =−}c2(j+ 1)−c2(j) α+βijb0 .

Since the costs are increasing convex, andβijb0increases linearly iniandj, the above equa-tions are nonincreasing in i andj. Suppose the result were true for Vn−1_{(hi+ 1, j,−1i)−} Vn−1_{(hi, j,−1i)andV}n−1_{(hi, j+ 1,−1i)−V}n−1_{(hi, j,−1i). We can then show from (10) that} Vn_{(hi+ 1, j,−1i)−V}n_{(hi, j,−1i)is nonincreasing iniandj. The result follows by induction}

and the fact thatVn_{(·)converges monotonically to the optimalV(·)(see Puterman [16], Section}

6.11).

We now present a control limit result for this model.

THEOREM 1: If the optimal decision in stateV(hi, j∗_{,2i)is to reject the gold customer, then}

the optimal decision in stateV(hi, j,2i), j > j∗_{, is also to reject the gold customer.}

PROOF: The proof follows directly from Lemma 1 and the paragraph before Lemma 1. EXAMPLE: Suppose the lump sum rewards areK1= 20andK2= 10for platinum and gold customers respectively. Let the cost rates bec1(I) = 1.3i_{andc2(j) = 1.8}j_{. Let the maximum}

number of modems, M = 9. In this case one could confirm that there would be 220 states hi, j, bi. Suppose the arrival rates for platinum and gold customers areλ1= 0.05andλ2= 0.2, respectively, and their service rates areµ1= 0.5andµ2= 0.25, respectively. Suppose the discrete time discount rateλ = 0.9, then the continuous time discount rate,α = ln(λ) = 0.105. The uniformization constantC=Mmax(µ1, µ2) +λ1+λ2= 4.75. Tables 2 and 3 gives values of ∆ij1and∆ij2under various conditions, and Figure 3 shows these graphically.

Figure 3 shows that when there are already three gold customers in the system, then no more gold customers are admitted. Therefore, gold customers may be rejected even if there are available modems in the system. For this example, this threshold value remains the same for varying values of number of platinum customers in the system, as can be seen from Figure 4 top. Figure 4 bottom shows that, as the number of gold customers increase, the incremental value from platinum customers also reduces, albeit by a very small amount.

Table 2. Values of∆i,j,bfori=1 and varyingj.

j i=1,b=1 i=1,b=2 0 17.90 5.87 1 17.90 3.52 2 17.90 1.49 3 17.90 −0.08 4 17.90 −0.85 5 17.90 −1.79 6 17.90 −2.90 7 17.59 −4.37

Table 3. Values of∆i,j,2for varyingiandj. j i=1 i=2 i=3 i=4 i j=1 j=2 j=3 j=4 0 5.87 5.87 5.87 5.87 0 3.52 1.49 −0.06 −0.65 1 3.52 3.52 3.52 3.52 1 3.52 1.49 −0.08 −0.85 2 1.49 1.49 1.49 1.49 2 3.52 1.49 −0.10 −1.05 3 −0.08 −0.10 −0.12 −0.15 3 3.52 1.49 −0.12 −1.25 4 −0.85 −1.05 −1.25 −1.64 4 3.52 1.49 −0.15 −1.64 5 −1.79 −2.17 −2.75 5 3.52 1.49 −0.35 6 −2.90 −3.66 6 3.51 1.28 7 −4.37 7 3.25

5. CONCLUSION AND EXTENSIONS

This paper studies optimal policies for allocating modems capacity among segments of cus-tomers using a continuous time Markov decision process model formulation considering stochastic arrivals and departures of customers attempting to use the Internet. As with other yield manage-ment applications, a limited capacity plays a critical role in the determination of such optimal strategies. However, there are sufficient differences, such as no prespecified time horizon and no prebooking or reservations, to make this model interesting to pursue. Finally, it also seems to be a tool that is needed by ISPs.

Real time implementation is made possible by relying on a database of critical values generated by the model, which serves as input to the customer authentication module which gets activated every time a customer dials up the ISP to get connected to the Internet. This database would have to be updated periodically when there is a significant change in the system or customer parameters. More research needs to be done on modeling customer segment specific, and time of day/week specific customer behavior of using the Internet. Another approach may be to assess a price for each service in real time, and give better service to customers who are willing to pay more at that instant in time. However, these will make the model more complicated.

Figure 3. Incremental values,∆ij1and∆ij2, of accepting platinum and gold customers, respectively, as a function of gold customers in the system,j.

Figure 4. Values of∆i,j,2for varying number of gold and platinum customers.

Future research also needs to be done to incorporate real-time learning capabilities into our model so that it can dynamically detect patterns in log-in/off behavior using neural networks. Such a tool would automatically alter the behavior of the system in response to environmental changes like lunch-hour traffic jams on the information superhighway and inclement weather.

ACKNOWLEDGMENTS

We thank Bob D., Dave C., and Tony D. of ‘‘Deciles, Inc.,’’ for taking the time to explain the intricacies of the Internet to us beyond what we knew before, which was not much. We thank Professor Marty Puterman for his advice. We also express our gratitude to the Associate Editor and a Referee for a very careful reading of our manuscript and for making many constructive suggestions.

REFERENCES

[1] J. Alstrup, S. Boas, O. Madsen, and R. Vidal, Booking policy for flights with two types of passengers, Eur J Oper Res 27(3) (1986), 274–288.

[2] American Airlines Annual Report, The art of yield management, 1987, pp. 22–25.

[3] G.R. Bitran and S.V. Mondschein, An application of yield management to the hotel industry considering multiple day stays, Oper Res 43(3) (1995), 427–443.

[4] S.E. Bodily and L.R. Weatherford, Perishable-asset revenue management: Generic and multiple-price yield management with diversion, Omega 23(2) (1995), 173–185.

[5] W.J. Carroll and R.C. Grimes, Evolutionary change in product management: Experiences in the car rental industry, Interfaces 25(5) (1995), 84–104.

[6] A.I. Elwalid and D. Mitra, Statistical multiplexing with loss priorities in rate-based congestion control of high-speed networks, IEEE Trans Commun 42 (1994), 2989–3002.

[7] M. Fisher and P. Kedia, Optimal solution of set covering/partitioning problems using dual heuristics, Manage Sci 36 (1990), 674–688, 43.1.1–43.10.

[8] L. Kleinrock, Power and deterministic rules of thumb for probabilistic problems in computer commu-nications, Proc Int Conf Commun, June 1979.

[9] S. Ladany, Dynamic operating rules for motel reservations, Decision Sci 7 (1976), 829–840. [10] C.J. Lautenbacher and S. Stidham, The underlying Markov decision process in the single-leg airline

yield management problem, Transport Sci 33(2) (1999), 136–146.

[11] T.C. Lee and M. Hersh, A model for dynamic airline seat inventory control with multiple seat bookings, Transport Sci 27(3) (1993), 252–265.

[12] S.A. Lippman, Applying a new device in the optimization of exponential queueing systems, Oper Res 23 (1975), 687–710.

[13] H. Mendelson and U. Yechiali, Controlling the GI/M/1 queue by conditional acceptance of customers, Eur J Oper Res 7 (1981), 11–85.

[14] P. Naor, On the regulation of queue size by levying tolls, Econometrica 37 (1969), 15–24.

[15] R. Nykiel, 1990’s: Decade of yield management, presented at 1989 Yield Management Multi-Industry Conference, Charlotte, NC, 1989.

[16] M.L. Puterman, Markov decision processes: Discrete stochastic dynamic programming, Wiley, New York, 1994.

[17] B.C. Smith, J.F. Leimkuhler, and R.M. Darrow, Yield management at American Airlines, Interfaces 22(1) (1992), 8–31.

[18] S. Stidham, Jr., Optimal control of admission to a queueing system, IEEE Trans Automat Control AC-30(8) (1985), 705–713.

[19] J. Subramanian, S. Stidham, and C.J. Lautenbacher, Airline yield management problem with over-booking, cancellations, and no shows, Transport Sci 33(2) (1999), 147–167.

[20] L.R. Weatherford and S.E. Bodily, A taxonomy and research overview of perishable-asset revenue management: Yield management, overbooking, and pricing, Oper Res 40(5) (1992), 831–844. [21] S.H. Xu and J.G. Shanthikumar, Optimal expulsion control—a dual approach to admission control,

Oper Res 41(6) (1993), 1137–1152.

[22] U. Yechiali, On optimal balking rules and toll charges in a GI/M/1 queueing process, Oper Res 19 (1971), 349–370.