Analyzing Cloud Costs in a
Standard IT Cost Model
Analyzing Cloud Costs
in a Standard IT Cost Model
Executive Summary
Enterprise use of public cloud services has made managing IT costs more complicated than ever. CIOs and CFOs need to track, analyze and manage not only traditional infrastructure and applications but also the company’s usage and cost of public cloud-based services such as Amazon Web Services (AWS). Adding to the complexity of hybrid IT cost management, different parts of an organization require different views of cloud cost and usage data. CFOs and Corporate Finance organizations typically look at costs from the corporate reporting perspective of Cost Centers while CIOs, IT Finance, Infrastructure, and Application leaders need context and granularity to accelerate better decisions that include cost by project, applications, services, and resource types such as compute, storage, and network.
AWS provides as many as 28,000 product offerings generating up to millions of billing line items each month, creating complexity managing IT costs. To solve these challenges IT needs a standard solution that incorporates costs based on cloud usage with on-premises IT costs for an integrated view of today’s hybrid IT business.
An Explosion of Cloud Offerings
AWS packaging and pricing is very flexible, and as a result, very complex. It includes 28,000+ possible services; the number and composition of which are constantly changing. AWS organizes its service portfolio by categories, and within each category are several product groups (fig. 1.0). Within each AWS product group are as many as 8,000 possible service offerings based on the options and configurations the
customer chooses. For example, within the EC2 (Elastic Cloud Compute) group in the Compute category, a customer will select an option from
each of following categories to configure their instance (fig. 2.0).
AWS Product Offering
Category Product Group
Compute
EC2 (Elastic Compute Cloud) Elastic Load Balancing Auto Scaling
Database
Amazon RDS Dynamo DB Amazon Redshift Amazon ElastiCache
Application Services
Amazon AppStream Amazon CloudSearch Amazon Elastic Transcoder Amazon FPS
Amazon SES Amazon SNS Amazon SQS Amazon SWF
Analytics
Amazon EMR Amazon Kinesis AWS Data Pipeline
Applications
Amazon WorkSpaces Amazon Zocalo
AWS Marketplace
3rd Party Business Software 3rd Party Developer Tools 3rd Party Infra Software
Generally, pricing is by instance and can include an up front charge (Reserved Instances only) combined with “per instance per hour” pricing
(typically under $1.00 and measured to a tenth of a cent), billed monthly, excluding tax. Both up front and usage charges are determined by the options chosen above, as well as volume of usage and length of term (1 year or 3 years for Reserved Instances). In addition to instance-based
services there can be charges for data transfer in and out of AWS. An overview of Amazon’s pricing model can be found at http://aws.amazon. com/ec2/pricing/ and billing information athttp://aws.amazon.com/ec2/faqs/.
Fig 1.0
Fig 2.0
EC2 Configuration Options
Operating Systems Linux (RHEL, SLES, etc), Windows…
Regions US East, US West (Oregon), US West (Northern California), EU, Asia Pacific…
Instance Types On-Demand, Reserved, General Purpose, Compute Optimized, GPU Instance, Memory Optimized, Storage Optimized…
Performance Level
# of “ECUs”, an Amazon-specific measure of integer processing performance (needed since underlying hardware can change)
Integrating Public Cloud into a Standard IT Cost Model
Product configuration flexibility is great for users, but creates new challenges for IT management. When AWS generates monthly bills, each resource yields dozens or hundreds of distinct billing line items. All of this fidelity is bundled under a single invoice that the finance department
pays each month, but the granularity and meaning is typically lost in cost reporting to IT and business leaders. There is no management view of how much cloud spend is going to storage vs. compute vs. network, how much of the enterprise’s total compute spend is driven by public cloud vs. traditional models, or how public cloud is impacting the costs of projects, applications, and services that the business consumes.
To understand cloud spend in context of the enterprise technology business, IT needs both a unified model for categorizing cloud and non-cloud
costs together, and automation to map millions of billing line items into the IT cost model each month.
To automate the mapping process, 2nd Watch and Apptio have worked together on a mapping table that specifies where each Amazon product fits within a standard cost model at the “IT Sub-Tower” level (see illustration below). This mapping is now embedded in the Apptio Cost
Transparency application as part of an end-to-end solution for integrating AWS usage into the costing process, including direct integration with AWS billing, cost categorization, modeling of total costs of cloud usage including internal labor, and self-service analytics on how cloud impacts the broader IT cost model.
Apptio maintains this AWS product mapping table and regularly updates it when AWS makes changes. During each scheduled retrieval of AWS billing records, the Apptio DataLink AWS connector automatically checks for updates to the Apptio mapping table, loading them into a dedicated dataset within Apptio Cost Transparency. This alleviates the need to maintain or manually refresh these mappings.
Apptio TBM Unified Model™ (ATUM™)
Cost Categories
Comprised of Other Other Telecom Telecom Depreciation Lease Expense Maintenance & Support
Facilities & Power
Consulting Managed Service Provider
Cloud Service Provider
Outside Services
Depreciation Lease Expense Maintenance & Support
Software Internal Labor Internal Labor Depreciation Lease Expense Maintenance & Support
Hardware External Labor External Labor Workspace Mobile Devices Service Desk Field Support Cloud Desktop End User App Dev App Support &
Ops LoBS oftware Cloud Apps Application Ops Center Project Mgmt. Client Mgmt. Cloud Ops Delivery
IT Mgmt. & Strategic Planning
Enterprise Architecture IT Finance IT Vendor Mgmt.
IT Management Security Compliance Disaster Recovery Cloud DR Security & Compliance Central Print Post Processing Output Circuits Usage Communication LAN WAN Voice Other Network Cloud Network Network Database Middleware Mainframe Database Mainframe Middleware Cloud Platform Windows Linux Unix Mainframe Converged Infrastructure Cloud Windows Cloud Linux Tier 1 Tier 2 Tier 3 Tier 4 Cloud Storage Cloud Archive Storage Compute Enterprise Data Center Other Facilities Data Center
IT Towers & Sub-Towers
Cost Pools & Sub-Pools
Using only data directly from AWS together with actuals from a company’s general ledger, Apptio Cost Transparency can categorize AWS costs into IT Sub-Tower categories such as Cloud Windows in Compute, or Cloud Archive in Storage. Going beyond the IT Tower level of the cost model up to Applications and Business Units requires additional data. (For more information on Apptio Cost Transparency cost model and solution, please visit http://www.apptio.com/applications/ cost-transparency.)
Adding Business Context Through Billing & Tagging
A key strategy for IT cost transparency is to help both IT and its business partners accelerate better decisions by putting cost into the context where key decisions are made, including applications, projects, and business unit consumption. Apptio Cost Transparency does this by
correlating financial and IT data directly from management systems through an automated cost model. For example, Apptio can use a list of
which applications are supported by which servers and storage, from a spreadsheet or an application dependency mapping tool, as one of the most accurate methods for determining the run costs of an application. However, many organizations have not yet adapted their management processes and tools to maintain application relationship data for cloud infrastructure. For these organizations, there are two AWS mechanisms which can be leveraged to provide needed business context: linked accounts and tagging.
Linked Accounts
AWS provides access to its products and services through individual accounts, each with its own credentials and the ability to purchase and manage AWS services. Many enterprises have many AWS accounts at the team and departmental levels (whether they know it or not). These can be spread throughout application development, projects, IT infrastructure, and business units. This decentralization fosters agility, but left unmanaged, this fragmentation of unlinked accounts creates blind spots in cost and operational management, as well as increasing costs by missing out on discounts for aggregate consumption volumes.
To gain the benefits of central visibility while preserving the benefits of decentralized agility, enterprises can link individual accounts
into one “master account” paid through an IT cost center. The linkage between individual accounts enables Apptio to pull in all usage and costs for the enterprise, while using the owner of each individual account (e.g. an application team or business unit) as a way to analyze cost “by customer” in self-service analytics. And, it can qualify enterprises for deeper volume discounts. Using a linked account paid by IT brings cloud spend out of the shadows while still enabling local departmental, team, or business unit agility.
Apptio Cost
Transparency provides analysis of public cloud costs broken out by product (per the service mapping), by customer (linked account owners), by provider (AWS, Microsoft AZURE™…) and more.
For workloads with infrastructures that do not need to share resources, individual AWS accounts can be used to group usage by workload or project. This can be accomplished by establishing individual AWS accounts for each workload and mapping them directly to your enterprise organizational structure. In the example below, AWS accounts are created for each project or workload then rolled up to provide consolidated usage information by department and business unit.
Tagging
In many organizations there is a need to share key resources, such as databases, across multiple workloads. In these cases it is best to use AWS tags to group usage and expenses. This method requires careful provisioning of resources and the creation of a schema to allocate shared resources that cannot be tagged across the enterprise.
For organizations that lack a coordinated approach to tagging today, here is a five phase approach to get started:
Phase 1: Scope
Answer some basic questions to decide where to focus first:
• When and how do your business, application, project, and operations teams request and provision cloud services? • Which of them have a tool, process, or policy for request and/or provisioning that tagging could snap into?
• If no such process exists, where is cloud usage big enough, or expected to grow enough, to focus on implementing a process? • Which team would be easiest to move along quickly?
Phase 2: Tagging strategy
A tagging strategy will help ensure that operational tools, as well as cost analysis, will have the data needed to answer critical questions accurately. Components of a Tagging Strategy include:
• An executive summary of the objectives, customer stakeholder groups, and SMEs. This summary will serve as the project’s north star and keep you on course in the later phases.
• “Key” and “value” definitions documented as customer’s standards.
• Key image map in Visio for ease of editing. This visual aid is especially helpful when AWS resources require multiple tags.
Phase 3: Determine which resources are in scope
In order to execute a tagging strategy, you will need to know details about the AWS resources that are in-scope: • AWS resource type (EC2, RDS, Redshift, DynamoDB, S3 Bucket, etc.)
• AWS instance ID and corresponding account number • Host name
• Availability zone / location
Tagging allows enterprises to assign its own metadata to each tag-able resource. Tags do not have any semantic meaning to AWS resources and are interpreted strictly as a string of characters. Tags
are made up of both a “key” and a “value”. AWS allows up to 10 keys
for each resource and each key can have unlimited values enabling very detailed grouping of resources. Tagging should be set up based on the needs of the organization and the AWS architecture design. The image below illustrates how to establish a tagging scheme for a 2-Tier auto-scalable web application.
As the project moves from Web Sandbox to Web Staging to Web Production, you can also use tags to track usage. When the application is in the sandbox all resources are tagged with the key “Web Sandbox” and the appropriate value (Environment, Owner, App, and/or IT Tower). When the project moves to “Web Staging” you simply replace the original key and values with the ones associated with the next step in development.
Even basic tags can make it easier for Apptio Cost Transparency to associate AWS resources to applications and services:
• How much of a particular application, or an entire application portfolio, is comprised of AWS services?
• How much of which projects are using cloud resources? • Which cloud costs are for development vs. test vs. production?
Phase 4: Make regular adjustments for optimization
AWS releases several new products and features each month. In order to keep up with the growing demands of a new AWS deployment, there will be ongoing adjustments and optimizations that can take several man hours each month.
• Maintenance of tags, standards, and best practices • AWS resource tag automation
• AWS tag integration with cloud formation templates • Onsite or virtual training on cost management
Limitations
As an enterprise expands it AWS usage, it will eventually reach limitations to a manual approach for managing individual accounts and tagging. Managing numerous logins and passwords, as well as having to go through the AWS setup process for each account can be onerous. Also, AWS tags are applied only to individual accounts.
When using AWS tags beyond application and project names, a tagging schema needs to be created and controlled. Mistakes in tagging can cause a resource to be unassociated to applications and projects in the cost model. In addition, some AWS resources cannot be tagged, requiring a different cost distribution scheme to be created and maintained to allocate those costs across the enterprise.
With limitations to account-based and tag-based grouping, the tracking methodology needs be aligned to the structure of the environment(s).
For environments where the resources are 100% dedicated to a specific workload, grouping by AWS account works best. Using AWS tagging is
ideal for environments where you need to share resources across multiple workloads.
Companies that want a more scalable approach for managing AWS accounts and tagging should consider solutions from third parties like 2nd Watch. 2nd Watch 2W Insight enables grouping of tags across AWS accounts, and provides tools to track and analyze cloud costs by cost center, business unit department etc. 2W Insight can be purchased by contacting us at www.2ndwatch.com.
About Apptio’s Cost Transparency Application
Apptio Cost Transparency is a SaaS application for managing the cost of IT by combining financial data from a company’s general ledger(s)
with IT asset and operations data from any management system. This results in the creation of an integrated understanding of what is driving operating expenses and capital investments and how they are aligned for value. Cost information is made actionable with role-based views into
portfolios of projects, applications, and services; the cost and consumption of these portfolios by each business unit; the foundational costs
of vendors, labor, and infrastructure; and the quality of data from your management systems. Visit
http://www.apptio.com/applications/cost-transparency to learn more.
Apptio is the leading provider of cloud-based Technology Business Management (TBM) software that helps CIOs manage the business of IT. For more information, visit the Apptio website or the Apptio blog at www.apptio.com.
2nd Watch is an enterprise workload management provider that helps companies accelerate data center capacity growth through adoption of the public cloud. The company’s public cloud-native services and tools implement and automate critical workload management processes including migration, procurement, provisioning, operations, financial management, and governance. 2nd Watch has helped hundreds of customers increase agility and lower operation costs by shifting workloads into more than 75,000 instances in the public cloud. The venture-backed company is headquartered in Seattle, Washington. To learn more about 2nd Watch, visit www.2ndwatch.com or call 1-888-317-7920.
