• No results found

Long-Term Resource Fairness

N/A
N/A
Protected

Academic year: 2021

Share "Long-Term Resource Fairness"

Copied!
31
0
0

Loading.... (view fulltext now)

Full text

(1)

Shanjiang Tang, Bu-Sung Lee, Bingsheng He, Haikun Liu School of Computer Engineering

Nanyang Technological University

Long-Term Resource Fairness

Towards

Economic

Fairness

on

(2)

Pay-As-You-Use is Pervasive

Charge users based on the amount of resources

used over time (e.g., Hourly).

Advantages

– Elasticity

– Flexibility

– Cost efficiency

Pay-as-you-use is becoming common and popular.

(3)

Twitter’s Cluster

Resource Utilization =

User resource demands are

heterogeneous.

– Users have different demands.

– A user’s demand is changing over time.

Static provisioning/partitioning causes

underutilization.

Resource utilization is a

critical

problem in such

pay-as-you-use environments.

– Providers waste resources

( waste investment and lose profit).

(4)

Resource Sharing

can improve resource utilization.

– Allow underloaded users to release resources to other users.

– Allow overloaded users to temporarily use more resources (from others).

 Reduce the idle resources at runtime.

 Resolve resource contention across users.

What about fairness

?

– If the fairness is not solved, resource sharing is unlikely to achieve in pay-as-you-use environments.

(5)

Pay-as-you-use Fairness:

Resource-as-you-pay

The total resources a user gained should be

proportional to her payment.

This is a Service-Level Agreement (SLA).

60 $ 40 $ A: B: 60% 40% Resource Service A B

(6)

Fair Policy in Existing Systems

State-of-the-art: Max-min fairness

– Select the user with the minimum allocation/share ratio every time.

– Consider the present requirement only (memoryless).

• Memoryless fairness has severe problems in pay-as-you-use environments, violating the following properties:

– Resource-as-you-pay fairness guarantee.

– Non-Trivial workload incentive and sharing incentive.

(7)

Problems with MemoryLess

Fairness

Resource-as-you-pay Fairness

Problem

– E.g., A, B equally pay for total resource of 100 units.

Time New Demand A B

t1 20 100 A B Accumulate Resource Usage: 20 80 20 80 Unsatisfied Demand A B 0 20 Current Allocation at t1:

(8)

Problems with MemoryLess

Fairness

Resource-as-you-pay Fairness

Problem

– E.g., A, B equally pay for total resource of 100 units.

Time New Demand A B

t1 20 100 t2 40 60 A B Accumulate Resource Usage: 40 60 60 140 A B Unsatisfied Demand 0 20 Current Allocation at t2:

(9)

Problems with MemoryLess

Fairness

Resource-as-you-pay Fairness

Problem

– E.g., A, B equally pay for total resource of 100 units.

Time New Demand A B

t1 20 100 t2 40 60 t3 80 50 A B Accumulated resource usage: 50 50 110 190 A B Unsatisfied Demand 30 20 Current Allocation at t3:

(10)

Problems with MemoryLess

Fairness

Resource-as-you-pay Fairness

Problem

– E.g., A, B equally pay for total resource of 100 units.

Time New Demand A B

t1 20 100 t2 40 60 t3 80 50 t4 60 50 A B Accumulated resource usage: 50 50 160 240 A B Unsatisfied Demand 40 20

Existing Fair Policy fails to satisfy Resource-as-you-pay fairness!!!

Current Allocation at t4:

(11)

MemoryLess Fairness Violates

Sharing Incentives

Non-trivial workload and sharing incentive

Problem

– Yielding resources to others have no benefits.

– Suppose A, B, and C equally pay for total resource of 100 units. A has 13 idle resource units. In that case, A can be selfish, either idle or running trivial workloads.

CPU 20 33 33 A: B: C: A’s idle resource 13

(12)

Cheating User Benefits on

MemoryLess Fairness

Truthfulness

Problem

– Suppose A, B, C equally pay for a cluster of 100 units, with true demand to be 33, 21 and 80, respectively.

– Case 1: all are honest.

– Case 2: User A cheats and claims the demand to be 40.

33 21 33 A: B: C: 6 6 A’s cheating gets benefits 33 21 33 A: B: C: 12 Case 1: A is honest Case 2: A is cheating

(13)

Our Work

Challenges:

can we find a fair sharing policy that

satisfies the following properties?

Resource-as-you-pay fairness

Non-trivial workload and sharing incentives

Truthfulness

Our Solution:

Long-Term Resource Fairness

– Ensure resource fairness over a period of time.

(14)

Long-Term Resource Fairness

Basic Concept:

Loan agreement

(Lending

w/o

interests)

– When resources are not needed, users can lend the resources to others.

– When more resources are needed, others should give back.

(15)

Long-Term Resource Fairness

Satisfy

Pay-as-you-use Fairness

Time New Demand A B

t1 20 100 A B Accumulated resource usage: 20 80 20 80 Unsatisfied Demand A B 0 20 Current Allocation at t1: A B Lend Resources: 30 -30

(16)

Long-Term Resource Fairness

Satisfy

Pay-as-you-use Fairness

Time New Demand A B

t1 20 100 t2 40 60 A B Accumulated resource usage: 40 60 60 140 A B Unsatisfied Demand 0 20 Current Allocation at t2: A B Lend Resources: 40 -40

(17)

Long-Term Resource Fairness

Satisfy

Pay-as-you-use Fairness

Time New Demand A B

t1 20 100 t2 40 60 A B Accumulated resource usage: 40 60 60 140 A B Unsatisfied Demand 0 20 Current Allocation at t2: A B Lend Resources: 40 -40 t3 80 50

(18)

Long-Term Resource Fairness

Satisfy

Pay-as-you-use Fairness

Time New Demand A B

t1 20 100 t2 40 60 t3 80 50 A B Accumulated resource usage: 80 20 140 160 A B Unsatisfied Demand 0 50 Current Allocation at t3: A B Lend Resources: 10 -10

(19)

Long-Term Resource Fairness

Satisfy

Pay-as-you-use Fairness

Time New Demand A B

t1 20 100 t2 40 60 t3 80 50 A B Accumulated resource usage: 80 20 140 160 A B Unsatisfied Demand 0 50 Current Allocation at t3: A B Lend Resources: 10 -10 t4 60 50

(20)

Long-Term Resource Fairness

Satisfy

Pay-as-you-use Fairness

Time New Demand A B

t1 20 100 t2 40 60 t3 80 50 t4 60 50 A B Accumulated resource usage: 60 40 200 200 A B Unsatisfied Demand 0 60

Long-Term Resource Fairness satisfy Resource-as-you-pay fairness.

Current Allocation at t4: A B Lend Resources: 0 0

(21)

Other Properties of Long-Term

Resource Fairness

Satisfy

non-trivial workload and sharing incentives

– Running trivial workload can waste money.

– Not sharing idle resource can waste money.

Users cannot get benefits by

lying

(strategy proof).

(22)

LTYARN

Implement Long-Term Resource Fairness in YARN

– Extend memoryless min fairness to long-term max-min fairness.

– Add a few components into resource manager

Support

full

long-term and time window-based

requirements.

Currently support a single resource type (main

memory).

(23)

LTYARN Design

Quantum Updater (QU)

– Estimates task execution time.

– Updates the resource usage history periodically.

Resource Controller (RC)

– Manages and updates resource for each queue.

Resource Allocator (RA)

– Performs long-term resource allocation.

(24)

Evaluation

A Hadoop Cluster

– 10 nodes, each with two Intel X5675 CPUs (6 cores per CPU with 3.07 GHz), 24GB DDR3 memory, 56GB hard disks.

– YARN-2.2.0, configured with 24GB memory per node.

Macro-benchmarks

– Synthetic Facebook Workload

– Purdue Workload

– HIVE/TPC-H

– Spark

(25)

Metrics

Evaluation metrics

Fairness degree for each user (>1 for sharing

benefits; <1 for sharing loss)

Resource-as-you-pay fairness

Application performance

Benchmark scenario

The four macro benchmarks equally share the

cluster.

Each benchmark runs in a separate queue.

(26)

Sharing Benefit/Loss

LTYARN enables sharing benefits for all applications.

(b). LTYARN (a). YARN

(27)

Resource-as-you-pay Fairness

Results

(28)

Performance Results

• Sharing always achieves a better performance.

• Long-term fairness is comparable to memory-less fairness (max-min).

(29)

Conclusions

Max-min resource fairness is memoryless and

unsuitable for pay-as-you-use computing.

We define long-term resource fairness that can satisfy

the desirable properties.

We develop LTYARN by integrating long-term

resource fairness into YARN

(30)

We are Hosting IEEE CloudCom 2014 in

Singapore

Deadline for paper submissions: July 31, 2014

Notification of Paper acceptance: September 2,

2014

(31)

Thanks!

Question?

References

Related documents

Based in Hong Kong since 2007, Chia Ching was involved in a number of notable initial public offerings and listings on the Main Board of the Hong Kong Stock Exchange acting for

His research focuses on the effects of physical activity on mental health, specifically conducting intervention research examining the efficacy of exercise as a treatment for

For the multi-class classification problem, a new multi-class classifier, termed as multiple birth support vector machine (MBSVM), has been proposed[20].. In this classifier, it seeks

One of the country’s leading marketing personalities, she is the founder of Altman Dedicated Direct, a company that epitomizes creative marketing with expertise in e-commerce,

STRUCTURE OF DHBW PROGRAMMES Theory (general university entrance exam) Approv al (selected by partner companies) Practice Final exams Job.. Success

NASA Herschel Science Center - page 20 PACS NHSC OT2 DP WORKSHOP September 2012 Allows you to perform a cone Search around your Favorite target: A link allows you to

Notice that in this model linearity in human capital accumulation implies that growth rates are constant in the long run, even with increasing returns in the aggregate