IBM Research Day: Containers Changing the Game in Cloud. Gosia Steinder Distinguished Research Staff Member

(1)

IBM Research Day: Containers—

Changing the Game in Cloud

Gosia Steinder (

[email protected]

)

(2)

Please Note:

1

• IBM’s statements regarding its plans, directions, and intent are subject to change or withdrawal without notice at IBM’s sole

discretion.

• Information regarding potential future products is intended to outline our general product direction and it should not be relied on in

making a purchasing decision.

• The information mentioned regarding potential future products is not a commitment, promise, or legal obligation to deliver any

material, code or functionality. Information about potential future products may not be incorporated into any contract.

• The development, release, and timing of any future features or functionality described for our products remains at our sole discretion.

• Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual

throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the

amount of multiprogramming in the user’s job stream, the I/O configuration, the storage configuration, and the workload processed.

Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.

(3)

Containers are not just for microservices

Server Host OS Docker Engine

Ap

pl

_B

Ap

pl

_A

Ap

pl

_A

Ap

pl

_A

Ap

pl

_A

Ap

pl

_A

Ap

pl

_B

Ap

pl

_A

Ap

pl

_A

Guest OS Docker Engine

Ap

pl

_A

Ap

pl

_A

Ap

pl

_A

Ap

pl

_B

Ap

pl

_B

Ap

pl

_A

Ap

pl

_A

Ap

pl

_A

Server

Host OS

Hypervisor

Server

Host OS

Hypervisor

Multiple-tenants

Improved visibility

and control

Bare metal performance

Improved density

Visibility and control

•  Container has direct access to shared libraries on host

•  Container has direct access to network, storage, accelerators

•  Provider can inspect installed/running packages, configuration

•  Provider can view memory usage stats

•  Provider can monitor and control kernel calls

Container-native cloud

Containers on IaaS cloud

Managed by

customer

Managed by

provider

Single-tenant

Limited visibility

and control

Simplified but flexible environment for customer

•  Customer can deploy applications with flexible topologies,

arbitrary runtimes

•  Customers only worry about containerized applications

•  Provider manages the operating system – deployment,

patching, monitoring, health

Container-native cloud enables new differentiating capabilities in the areas of security, compliance,

and performance management: with better guarantees and simpler to use.

(4)

Deep Visibility

!

Operational Analytics

! Insights !

Customer Value

-‐ OS Info

-‐ Processes

-‐ Disk Info

-‐ Metrics

-‐ Network Info

-‐ Packages

-‐ Files

-‐ Conﬁg Info

From Container

-‐ Docker metadata

(docker inspect)

-‐ CPU metrics

(/cgroup/cpuacct/)

-‐ Memory metrics

(/cgroup/memory)

-‐ Docker history

Docker Run1me

Conﬁg

Annotator

Vulnerability

Annotator

Compliance

Annotator

Password

Annotator

SW

Annotator

Licence

Annotator

-‐ Audit Subsystem

-‐ Syscall Tracing

-‐ System Integrity

PlaAorm

Index (Data)

Vuln. & Compl. Analysis Secure Conﬁg Analysis Forensic Security & Compl. Pipeline Service Remedia1on Service

(5)

(6)

Vulnerability Advisor enables acKve client control over container security

0

50

100 allow

warn

block

Policy change Percentage

Policy relaxed Policy strengthened

Policy change New threats

(7)

Changing the game in cloud with containers

Container-‐naKve cloud allows customers to incrementally deliver, scale, and modify high value container-‐based workloads with

unprecedented visibility, insight, and control, and enterprise-‐grade security, compliance, stability, and performance.

Our point of view

Already in IBM Containers on Bluemix

Always on, tamper-‐proof, built in monitoring of containers and images

Automated, built-‐in

vulnerability analy1cs of container images

Vulnerability tesKng built-‐ in into container devops pipeline

Innova1on pipeline

Leading container plaAorm with advanced orchestraKon, open architecture, increased security, operaKonal excellence, and agility

Customizable and detailed visibility of event-‐based and snapshot-‐based rich data for images and containers

AnalyKcs based insight into security and compliance

posture of images and containers based on customizable policies

Control over security and compliance posture via devops-‐level checks and automated remediaKon Industry-‐ﬁrst container

Container-‐na1ve plaAorm

for micro-‐services with

isola1on expected for public cloud

(8)

Container orchestration – open-source landscape

• Mesos (Mesosphere) – resource management platform enabling partitioning of compute

resources across multiple workloads

• Marathon (Mesosphere) – platform as a service enabling deployment of 12-factor

applications on top of Mesos

• Swarm and Compose (Docker) – multi-host Docker container management system

• Kubernetes (Google) – platform for the management of microservices enabling fine-grained

composition of multi-container instances

Today, these communities build all-encompassing stacks with significant functional overlap, divergent APIs, virtually no

collaboration, and with significant gaps for large-scale production usage.

Our goal:

• Evolve these opensource projects to become mature production-ready platforms based on our own production usage

of these technologies

• Develop common container management APIs for cloud-native workloads and common open architecture

• Leverage Cloud Native Computing Foundation as a community to bring together these efforts and drive towards

(9)

Why these choices?

8

Compose

• Native Docker experience

• Rich life-cycle management

• Pattern deployment via Compose

• Light-weight

kubectl CAdvisor CAdvisor API server Scheduler Controller mger K8s master K8s minion K8s minion Kubelet Proxy Kubelet Proxy Etcd service state

• POD – single-host pattern, fine-grained application composition

• Desired state management

• Replication groups with autorecovery, rolling-update, autoscaling

• Richly-featured microservice platform

Mesos Slave

Allocation

module

Mesos Master

Framework

Resource / task status

Start task Offers

Start task

Mesos allows Swarm and

Kubernetes to share compute

resources

(10)

Container-native Cloud Architecture

9

Container Host

C

Orchestration

API

Cloud

Services

Operational Visibility

C C C C C

Orchestration

(single container and container group)

Image Registry

Infrastructure (VMs & bare metal)

Private IP

Network

Private NFS

Storage

(11)

Swarm – multi-tenancy and performance enhancements

• What we are working on:

–

 

Full API support on top of Mesos

–

 

Private registry integration

–

 

Authentication and authorization with pluggable auth/authz

mechanisms

–

 

Performance analysis and improvement

10

Time to create a network grows linearly with the number of

networks already present in swarm. Scalability challenge.

Time to create a network grows linearly with the number of

networks already present in Docker engine. Scalability challenge.

Our improvement

Working on it …

Swarm auth plugin

Swarm docker calls

Pass auth token

Swarm

Regular Docker engine API

Auth

Backen

d

(12)

Kubernetes – performance enhancements

• What we are working on

–

  Improved modularity and configurability

–

  Network plugins

–

  Performance and scalability

11

Default configuration: POD deployment times

dramatically increase with system occupancy

Tuned configuration: significant performance

improvement (2 orders of magnitude)

(13)

Container networking in Swarm and Kubernetes

12 Docker Engine

Kuryr

libnetwork Docker Engine

Kuryr

libnetwork

Neutron

Swarm Master

Docker Engine

Kuryr

libnetwork

CNI

plugin

Kubernetes

Kubernetes Minion Agent Docker Engine

Kuryr

libnetwork

CNI

plugin

Kubernetes Minion Agent

Neutron OVS/OVN network

•  L2 overlays

•  Subnets

•  Security groups

•  Firewalls

•  Load balancers

Rich management APIs

Containers deployed via Swarm and Kubernetes can join the same L2/L3

network – seamless private communication between Kubernetes and

Swarm parts of an application.

(14)

Using Swarm & Kubernetes with Mesos

• Mesos manages the actual resources on the cluster

• Incoming API/CLI are stored in a queue, waiting for offers from Mesos

• The framework’s scheduler is used to choose the target host from the Mesos offers

• The framework sends a “task” to Mesos slave to create the container

Swarm

Scheduler

Mesos

Master

Docker

CLI/API

framework

Kubernetes

Scheduler Mesos Agents Offers framework Offers

Tasks to Mesos

(15)

Looking forward: introducing Optimistic Offers in Mesos

• Simpler, however:

• Under Utilization

• Starving Big Tasks

• Non-optimized schedule decision

• SLA Enforcement

•  IBM is driving

Mesos-1607

(

https://issues.apache.org/jira/browse/MESOS-1607

) with Mesos community to support Optimistic

Offers

Pessimistic Offer*

**_{Optimistic Offer*}**

Framework

scheduling logic

(16)

Hardened Container Platform with Isolation and Runtime Integrity

• What: hardening of the underlying compute platform to:

–

 

Prevent breach of isolation through container privilege

escalation attacks

–

 

Detect, prevent and mitigate resource exhaustion (DoS)

attacks

–

 

Efficiently manage and audit of network isolation across the

cloud infrastructure

–

 

Continuous runtime and enforcement of platform integrity to

protect against installation of unknown software

• Why:

–

 

Strong level of assurance of isolation from other cloud

workloads, without additional management complexity and

overhead of hypervisors and VMs

–

 

Increased visibility afforded by shared platform Kernel (files,

processes, system calls) allows earlier detection of

anomalies

–

 

Ability to continuously verify (attest) workload integrity

IBM led Docker community to enhance the engine to meet reasonable security expecta:ons for a cloud service . We are pursuing further innova:ons in

isola:on and run:me integrity both in the community and in IBM cloud pla?orm.

(17)

Limit the set of Linux capabilities each container is started with. Docker, by default drops most capabilities.

Ensure that changes is capabilities are properly authorized. Capability limitation

Isolation from other containers

Kernel isolation Resource isolation

Use kernel namespaces for isolating from other containers: pid, net, ipc, mnt, utc, uts.. Leverage cgroups for resource isolation.

Network traffic shaping is an issue with default networking. No ability to isolate process id resources. Not all control knobs exposed in CLI..

All Docker containers share host kernel, but not all syscalls and capabilities are exposed to them. Inherent issue with containers.

Coloring:

Docker supports this out of box.

Docker or Linux gap – we are working on it

Inherent issue

Restrict Docker API Calls Docker API allows users to create privileged containers or change capabilities without authorization. Provider must restrict access to certain APIs and ensure access to API is authorized.

Docker Registry Use V2 registry – it has signatures for images and layers.

Securing the container platform

Follow best practice for securing a host (e.g., STIG firewall, auditd) AppArmor

Host root isolation

Hardware Assisted Verification and Isolation Use Trusted computing and TPM for host integrity verification and VT-d for better isolation

User namespaces: container root is de-priivileged on host. Docker: in v1.10 root in all containers mapped to

same unprivileged id on host

Work in progress:: Enable configurable mappings (requires Linux kernel improvement)

Docker Engine Configuration User AppArmor profiles for containers (customizable) Provider must ensure proper AppArmor configuration.

Host Security

Use AppArmor for daemon confinement (customizable). Provider must ensure proper AppArmor configuration.

(18)

Security Configuration Analytics: Detection of Misconfiguration & Breach of Isolation

Event & Log Repositories (e.g. logstash)

Ironic Nova Neutron

Monitoring Backend Database Security Configuration Analytics Forward notifications on orchestration events, API calls, …

Generate alerts if any misconfiguration is

detected Assess relative risk of

configuration anomaly

Enhanced Visibility

Tenant

1 Tenant 2 Tenant 3 Docker Engine Docker Compose template Application Developer builds & deploys complex app with (network) security

policies – FW, IPS/IDS, …

Cloud User modifies running application settings

to improve performance (e.g. change IPS policies)

Security Configuration Analytics Detects “Configuration Drift”

Generate Alerts & Remediation Actions

based on Risk and Asset Value (see next)

(19)

Recommended for you

Demos of technology in discussed today:

CCI-7280 : IBM Research Day Demo: Running Containers on Swarm and Kubernetes at Data Center Scale

YPS-7294: IBM Research Day Demo: Docker @Insane Scale on IBM Power Systems, coming next

SAD-7288: IBM Research Day Demo: Vulnerability Remediation Service, happened this morning

LABs and Education on IBM Containers:

TCD-1506: Hands-On Lab Demonstrating the Enterprise-Grade Capabilities of IBM Containers

CCD-6713: Meet the Experts Who Are Leveraging Docker Containers and Microservices to Run IBM Containers

CCD-3865: Leveraging IBM Containers for Enterprise-Scale Software Development

CDL-9409: Learn IBM Conatiners in 15 minutes

InnerCircle presentations:

DEV-6859: IBM and Docker Container Offerings, Strategy and Roadmap

Core curriculum:

CCD-2715: Building an Enterprise PaaS with Bluemix, Docker Container Services and Watson on IBM Power Systems

COC-3243: IBM Containers and Open Technologies: A Container Service Designed for the Enterprise

CCD-3518: The Bluemix Triple Threat: Cloud Foundry, Containers and Virtual Machines

(20)

Notices and Disclaimers

19

U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM.

Information in these presentations (including information relating to products that have not yet been announced by IBM) has been reviewed for accuracy as of the date of initial publication and could include unintentional technical or typographical errors. IBM shall have no responsibility to update this information. THIS DOCUMENT IS DISTRIBUTED "AS IS" WITHOUT ANY WARRANTY, EITHER EXPRESS OR IMPLIED. IN NO EVENT SHALL IBM BE LIABLE FOR ANY DAMAGE ARISING FROM THE USE OF THIS INFORMATION, INCLUDING BUT NOT LIMITED TO, LOSS OF DATA, BUSINESS INTERRUPTION, LOSS OF PROFIT OR LOSS OF OPPORTUNITY. IBM products and services are warranted according to the terms and conditions of the agreements under which they are provided.

Any statements regarding IBM's future direction, intent or product plans are subject to change or withdrawal without notice.

Performance data contained herein was generally obtained in a controlled, isolated environments. Customer examples are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual performance, cost, savings or other results in other operating environments may vary.

References in this document to IBM products, programs, or services does not imply that IBM intends to make such products, programs or services available in all countries in which IBM operates or does business.

Workshops, sessions and associated materials may have been prepared by independent session speakers, and do not necessarily reflect the views of IBM. All materials and discussions are provided for informational purposes only, and are neither intended to, nor shall constitute legal or other guidance or advice to any individual participant or their specific situation.

It is the customer’s responsibility to insure its own compliance with legal requirements and to obtain advice of competent legal counsel as to the identification and

interpretation of any relevant laws and regulatory requirements that may affect the customer’s business and any actions the customer may need to take to comply with such laws. IBM does not provide legal advice or represent or warrant that its services or products will ensure that the customer is in compliance with any law

(21)

Notices and Disclaimers Con

’

_t.

20

Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products in connection with this publication and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. IBM does not warrant the quality of any third-party products, or the ability of any such third-party products to interoperate with IBM’s products. IBM EXPRESSLY DISCLAIMS ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.

The provision of the information contained h erein is not intended to, and does not, grant any right or license under any IBM patents, copyrights, trademarks or other intellectual property right.

IBM, the IBM logo, ibm.com, Aspera®, Bluemix, Blueworks Live, CICS, Clearcase, Cognos®, DOORS®, Emptoris®, Enterprise Document Management System™, FASP®, FileNet®, Global Business Services ®, Global Technology Services ®, IBM ExperienceOne™, IBM SmartCloud®, IBM Social Business®, Information on Demand, ILOG, Maximo®, MQIntegrator®, MQSeries®, Netcool®, OMEGAMON, OpenPower, PureAnalytics™, PureApplication®, pureCluster™, PureCoverage®, PureData®,

PureExperience®, PureFlex®, pureQuery®, pureScale®, PureSystems®, QRadar®, Rational®, Rhapsody®, Smarter Commerce®, SoDA, SPSS, Sterling Commerce®, StoredIQ, Tealeaf®, Tivoli®, Trusteer®, Unica®, urban{code}®, Watson, WebSphere®, Worklight®, X-Force® and System z® Z/OS, are trademarks of International Business Machines Corporation, registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at: www.ibm.com/legal/copytrade.shtml.