White Paper
The Importance of Automating
the End to End Pipeline
for Continuous Delivery
Executive Summary
Continuous Delivery (CD) can transform businesses by enabling innovation, improving product quality, and dramatically decreasing time to market, risks and costs. To implement an effective CD methodology, enterprises must integrate, orchestrate, automate, and manage potentially hundreds of disparate point tools and environments used across their development, operations, and IT organizations.
Some believe that they have implemented an automated CD methodology just because their individual groups (silos) are using automated tools. But, if one examines the flow between these groups, what is more likely found is a scenario riddled with manual handoffs. Besides being error prone, this manual interaction is slow, does not provide a ‘single pane of glass’ visibility and control of the entire end-to-end pipeline, and doesn’t efficiently implement CD. ElectricFlow is an enterprise-ready CD automation platform that automates and integrates the product pipeline across development, operations, and IT. ElectricFlow provides unique value to an organization by enabling orchestration, visibility, and control of all phases of the end-to-end software delivery process as well as providing a platform to integrate a disparate set of tools and processes. ElectricFlow enables CD agility, predictability and security across build-test-deploy pipelines.
Avoiding Common Mistakes
At first glance, automating CD in an enterprise environment that uses hundreds (or thousands) of different tools, processes, environments, and workflows may seem a daunting task. The first step in doing this is to avoid the common mistakes that are often made:
• Investing resources on custom scripts and ‘glue’ to integrate and automate the product pipeline. Not only
is this not scalable, but these scripts are difficult to maintain but they consume resources that would be better spent on adding business value.
• Creating an infrastructure that isn’t flexible to support changing business needs. Managing the end-to-end pipeline is as much about creating business value is it is about releasing software product. As such, flexibility is needed to support changing business needs and objectives.
• Creating an infrastructure or IT requirements that impose limitations on the various teams that are part of the end-to-end pipeline. This stifles innovation and productivity.
• Trying to utilize tools for tasks that they were not designed to do. Examples of this include using continuous integration tools for deployment, configuration management tools for end-to-end orchestration, or using point tools designed for single project usage to manage an entire organization. Implementing Continuous Delivery
For the operations and deployment side of the pipeline, IT often manages hundreds or thousands of computing resources in various combinations of physical hardware, private cloud, and public cloud machines. When combined with the many variations of environments, versions of application and middleware, this becomes an almost impossible task to manage and track.
When a business decides to automate their enterprise CD initiative, these problems will likely surface: • Tool duplication: tools from different vendors are
used to perform similar tasks, or the same tool is used in different ways by different groups.
• Tool misuse: trying to use tools to do more than their intended purpose such as attempting to use a configuration management tool for automation and orchestration.
• Tool scalability: managing multiple instances or versions of the same tool chains is difficult because Introduction
those tools were not designed for enterprise usage. • Process flexibility: business needs often dictate
that different processes are used in different parts of the organization; therefore flexibility in automation is required.
• Control and visibility of the entire pipeline: CD covers the entire pipeline from development through delivery. Being able to orchestrate and monitor this entire pipeline is crucial. Integration with a wide variety of tools is essential.
• Governance: Ensuring that internal and external standards are adhered to and that appropriate data is available for audit purposes is mandatory (even if that data comes from many different tools).
• Large Scale Deployment: Home-grown scripts and point tools not designed for enterprises do not scale well. Achieving CD, controlling costs, and managing hundreds or thousands of computing resources becomes problematic with these alternative solutions. This is especially an issue when the computing resources are deployed globally and consists of a combination of physical machines, private cloud, and public cloud virtual machines.
The elusive last mile of implementing an effective enterprise continuous delivery initiative requires integrating, automating, managing, and orchestrating the many different tools and processes that are used across the enterprise. Unfortunately, this is not a simple problem to solve and one that should not be undertaken by in-house IT resources writing homegrown glue. Instead, businesses must consider a purpose-built, end-to-end, highly-available, enterprise-ready automation platform that:
• Provides flexibility – utilizes open standards and integrates with many different tools; easily implements and automates different types of business logic supporting any development/release/deployment cycles including manual gates and dynamic processes.
• Provides scalability & high availability – the platform must be purpose built for enterprises and offer redundancy, predictability, and high availability. It must support a global scale on the magnitude of 1000s of people, servers, and teams located globally. It must offer the ability to control and share resources equitably globally, helping to maximize the existing infrastructure.
• Provides extensibility – the platform must support open standards and have a robust API, support
two-way communication between tools effectively enabling a disparate set of tools to be chained together
• Provides visibility – at any given moment, the platform must be able to provide the high-level status of the entire application pipeline, available environments, and log information for hundreds or thousands of builds. In addition, the platform must immediately notify when an error occurs and provide adequate data to fix the problem. The platform must be able to report on the exact configuration of all components that are used in any particular build. • Provides governance – at all times, compliance with
internal and/or external regulations is required. The platform must also ensure security and provide data needed for any audits.
ElectricFlow
Electric Cloud’s ElectricFlow is an enterprise-ready DevOps process automation platform. ElectricFlow provides distributed teams shared control and visibility into infrastructure, tool chains and processes. It accelerates and automates the software delivery process to enable agility, predictability and security across build-test-deploy pipelines.
ElectricFlow with Continuous Integration Tools One of the key features of ElectricFlow is its ability to orchestrate and manage a wide variety of existing infrastructure and tools. As an example of this, consider the continuous integration tool that many development organizations use called Jenkins. Jenkins was purpose-built for continuous integration (CI) on a small project basis. CI is only one part of continuous delivery. A common problem occurs when Jenkins is used on multiple projects and that is some organizations discover that the low cost and simplicity of setting up a Jenkins server leads to a proliferation of one-off Jenkins instances. These “islands of innovation” are the result of a “get it done/ shadow IT” mindset among quick-moving development teams. This decentralization of Jenkins CI activities is sometimes referred to as “Jenkins sprawl,” and presents organizations pursuing a Continuous Delivery model with challenges. What is needed is an orchestration and management tool, such as
ElectricFlow, to provide the necessary orchestration and automation layer of many Jenkins instances to enable truly automated, end-to-end continuous delivery flow at an enterprise level.
By adopting ElectricFlow, organizations can:
• Extend the continuous integration infrastructure that Jenkins provides with an end-to-end automated continuous delivery infrastructure
• Gain a centralized view of Jenkins instances in the enterprise
• Prioritize and migrate projects to ElectricFlow to increase collaboration and sharing of artifacts across the enterprise
• Accelerate organization-wide CD with a build-test-deploy Center of Excellence to establish best practices and leverage pooled enterprise resources ElectricFlow with Configuration Management Systems
Configuration management systems such as Chef, Puppet, and Ansible enable infrastructure to be represented as code which then is versionable, buildable, and repeatable. While these tools are very good at configuration management, this represents only one part of end-to-end CD. ElectricFlow integrates these tools into an end-to-end continuous delivery process. With ElectricFlow, one can orchestrate the configuration of virtual machine instances as part of the environment set-up portion of the CD workflow. All of this can easily be packaged into a reusable procedure called from the main automation pipeline for either build/test or application deploy needs.
Managing Virtualization Platforms
The promise of a non-proprietary and extensible private cloud (e.g. OpenStack), as well as the use of public and hybrid clouds, has revolutionized IT operations. This ties in well with another trend in the industry, continuous application delivery, which automates steps between code-check-in and
production deployment to provide a more predictable path to application deployment.
ElectricFlow allows infrastructure and applications to be integrated in a scalable and flexible way that maximizes compute resources. For example with
a click of a button, ElectricFlow can dynamically provision a set of machines with OpenStack, configure them, deploy an application to them, run integration tests, and ask a group for manual approval.
360 Degree View of your Product Pipeline The above section mentions just a few of the tools that are used to move software from development to delivery. In reality, many tools are used throughout the various parts of an organization. Each of these tools have their own UI, their own logins, and may require a different skillset in the user. ElectricFlow allows all of these tools to be orchestrated into a reusable high-level pipeline process, alleviating manual handoff from process to process.
Besides the problems associated with orchestrating a disparate set of tools, there is a problem of data visibility. When trying to collect data for internal audits or for regulatory purposes, the data resides in many different tools and is extracted out in a variety of ways. However, using ElectricFlow alleviates this problem by providing visibility in one single place - in ElectricFlow. By doing so, from a single place, one has access to all of the details about what changes went into a build, which unit tests passed, what the code quality was, the version of the artifacts used in the build, the environments and applications deployed, how many tests passed/failed, etc. ElectricFlow also captures manual approvals that occur along the way in the pipeline. Because all data is available in a single system, one has a 360 control of the process and visibility into the process.
Governance
ElectricFlow addresses governance issues by providing:
• Reusable processes that can be shared between teams to eliminate process duplication, enabling agility across the organization
• Access control can be used to restrict who has access to what
• Full access to information in a single pane of glass which allows users to add automated actions based on custom logic
• Process auditing and the ability to instantly roll back to earlier versions
ElectricFlow Automates Continuous Delivery for the Enterprise
• Flexible execution: ElectricFlow provides a flexible workflow automation engine easily allowing third party products to be integrated, orchestrated, and managed.
• Visibility: Since all CD activities occur within ElectricFlow, activity can be viewed at various levels of granularity, from executive overviews to process debugging.
• Infrastructure as code: Because all applications, environments, workflows are modeled and are represented by code, the scripts and structure that make up the CD process can live in source code control.
• Integrations: ElectricFlow integrates with any tool that exposes an API via REST, SOAP, HTTP, CLI, etc. For example: SCM tools like Git, Subversion, or Perforce; CI tools such as Jenkins; configuration management tools such as Chef and Puppet; virtual machine infrastructure such as OpenStack, VMware, Amazon, Azure. ElectricFlow provides a rich API that can be used to pull all the information needed to provide visibility to the end users.
• Deployment automation: Applications, components, environments, and automated deployment processes are modeled in ElectricFlow. By linking application tiers to environment tiers with tier maps, processes are reused across environments of different sizes and configurations (development, integration, staging, production, etc.) ElectricFlow provides easy access to determine which application versions are running on each environment. • Resource management: Run processes on an explicit resource, pick a resource from a pool, or broadcast
it across an environment. Dynamically spin up and tear down as resources are needed. Resources/pools/ environments can be managed as an elastic cloud to maximize hardware utilization.
• Artifact management: ElectricFlow provides a built-in mechanism to publish and retrieve artifacts. Artifact publishers and retrievers are tracked by the system to ensure visibility and auditability. Artifacts dependencies are a very powerful mechanism used to combine and test several individually developed components into applications or products. ElectricFlow also integrates with major artifact repositories like Artifactory and Nexus. • Preflight: ElectricFlow provides a preflight mechanism whereby an end-user can run a production build/test/
deploy process to test their code changes before committing them. ElectricFlow simulates the commit by using a clean source code copy and overlaying the user’s changes (which may include product source code, configuration files, infrastructure as code, etc.) The preflight mechanism eliminates the chance of an invalid change because code is not committed if the preflight fails.
• Access & security: ElectricFlow provides both coarse and fine-grained access control to manage which users can read, modify, and execute the various objects. LDAP/ActiveDirectory groups and access control can be used to set up role-based access control (RBAC). ElectricFlow provides a mechanism for securely storing credentials and controlling access to processes to which they are attached.
• Enterprise scalability: Runs processes at an enterprise scale – supports any number of applications, resources, environments, and teams on ElectricFlow’s multi-tiered platform. The horizontally scalable ElectricFlow server allows you to add multiple servers behind a load balancer to increase throughput.
Conclusions
ElectricFlow is purpose-built to support the complete product lifecycle from development, to testing, to deployment at an enterprise scale. With the core goal of providing new capabilities to customers quickly, every aspect of the full end to end software delivery needs to be optimized rather than living in isolated silos. ElectricFlow allows organizations to optimize the entire process from the time developer has committed code until the delivery of new features.
ElectricFlow provides the ability to automate and orchestrate a disparate set of processes and tools that support continuous integration, continuous delivery, automated release automation, and deployment. By leveraging the following capabilities, ElectricFlow enables faster time to market, high quality, increased productivity and resource utilization, and reduced costs.
Corporate Headquarters
Electric Cloud, Inc.
35 S. Market St, Ste 100, San Jose, CA 95113 T: 408.419.4300 F: 408.419.4399
info@electric-cloud.com www.electric-cloud.com
Electric Cloud Europe
1650 Arlington Business Park Theale, Reading
Berkshire RG7 4SA United Kingdom T: +44 (0) 0207.872.5500
europe.info@electric-cloud.com
Electric Cloud Japan KK
22F Shibuya Mark City West 1-12-1 Dogenzaka, Shibuya-ku Tokyo 150-0043 Japan T: +81.3.4360.5375
japan-info@electric-cloud.com
Electric Cloud China
New City Center Plaza, No.70, Room 908, Tong Chuan Road, Shanghai, 200333, China
T: +86 13601825314 / +86 13761649476
china.info@electric-cloud.com
©Electric Cloud, Inc. All rights reserved. Electric Cloud, ElectricAccelerator, and ElectricFlow are trademarks of Electric Cloud, Inc. All other names are used for identification purposes only and are trademarks of their respective companies.
About Electric Cloud
Electric Cloud powers Continuous Delivery. We help organizations developing web, mobile, and embedded systems deliver better software faster by automating and accelerating build, test and release processes at scale. Industry leaders like Cisco, E*TRADE, Gap, GE, Huawei and Qualcomm use Electric Cloud solutions and services to boost DevOps productivity and Agile throughput.
