Comprehensive Guide on Upgrading PKS

(1)

Comprehensive Guide on Upgrading PKS

Based on PKS 1.3, April 2019 Contact: [email protected]

(2)

Introduction

This document is intended to bring together a concise view of the VMware Enterprise PKS upgrade process, best practices for Kubernetes (K8s) clusters configuration and achieving high availability of workload, pre- and post- upgrade checklists, and tips to debug and troubleshoot upgrade process.

The document will be useful for both PKS Operation teams as well as DevOps teams.

Readers familiar with PKS and BOSH can skip the details and use the checklist sections as quick reference for their upgrades.

(4)

Summary of Upgrade Process

• As of PKS 1.3, K8s cluster upgrade is done in batch:

All K8s clusters are upgraded in a single unique operation.

• User decides whether to launch the upgrade process through Ops Manager User Interface or BOSH CLI:

PKS tile => Errands section: Upgrade all clusters errand (Optional) BOSH command: bosh run-errand …

• K8s cluster binaries are defined by the PKS tile:

Every time a new PKS tile is imported into Ops Manager (which includes a new release of K8s/Docker/NCP/etcd) - K8s cluster upgrade process must be initiated.

• PKS driven K8s cluster upgrade will update the following components (based on PKS Tile content coming from specific CFCR release)

▪ K8s, Docker, etcd, NCP (NSXT Container Plugin) as needed and when new releases are available ...

▪ SSL Certificates.

▪ WaveFront and Telemetry Pods (optional)

• Notes:

▪ As of today, the way to update the SSL certificates on the K8s cluster nodes is by upgrading the cluster.

▪ Every time new cluster is provisioned, all components will be

upgraded to versions specified in the upgraded PKS Tile by BOSH (K8s, Docker, OVS, NCP, anything). These package versions are coming from specific CFCR release

▪ In case of upgrade of both PKS Tile and NSX-T, PKS Tile should always be upgraded first to ensure that NCP versions are updated

(5)

PKS Control Plane Upgrade

Upgrading OpsManager

Following are steps for upgrading OpsManager:

• Verify that the PKS control plane remains functional by performing the following steps:

▪ Monitor the PKS Control plane by clicking the Pivotal Container Service tile, selecting Status tab, and reviewing the Pivotal Container Service VM’s data points.

▪ If any data points indicate that resources are at capacity, scale deployments accordingly.

• Download Ops Manager VM template from Pivotal Network site. • Record the FQDN address of existing Ops Manager VM.

• To avoid conflicts, power off existing Ops Manager VM.

• Deploy new Ops Manager VM by following the steps in one of these topics: ▪ AWS: Upgrading BOSH Director on AWS

▪ Azure: Upgrading BOSH Director on MS Azure ▪ GCP: Upgrading BOSH Director on GCP

▪ OpenStack: Provisioning OpenStack Infrastructure ▪ vSphere: Deploying Operations Manager to vSphere When redirected to the Welcome to Ops Manager page, select Import Existing Installation.

(6)

When prompted, enter the Decryption Passphrase for this Ops Manager installation.

Note: This passphrase is set during the initial installation of Ops Manager. If lost, the Decryption Passphrase cannot be recovered.

Click Choose File and browse to the installation ZIP file exported in the Installation step in the upgrade preparation.

Click Import

Note: Some browsers do not provide feedback on the status of the import process, and it might appear to hang.

(7)

Import new PKS and product tiles

After upgrading the Ops Manager, upgrade product versions:

• Import the product file to Ops Manager Installation Dashboard. • Hover over the product name in Available Products and click Add.

• Click the newly-added tile to review and set any configurable options.

Perform Upgrade

• Navigate to the Ops Manager Installation Dashboard. Click Review Pending Changes, then Apply Changes. This immediately initiates upgrades to all selected tiles in a single transaction.

• Click each product Tile, select the Status tab, and confirm that all VMs appear running and are in good health.

(8)

• After confirming that new Ops Manager VM functions correctly, remove old (powered off) VM.

Monitor Upgrade

Use some or all the options below to monitor the upgrade process: • BOSH-reported VM metrics

• Runtime component metrics

• Key Performance Indicators (KPI) and Key Scaling Indicators (KSI) • Run BOSH CLI status checks:

bosh -e ALIAS task TASK_NUMBER

bosh -e ALIAS vms --vitals, bosh -e ALIAS instances –ps

• Check app availability

• Check availability of Ops Manager GUI

• Check vSphere performance (if deployed on vSphere)

Collect Logs

If any issues are encountered during the upgrade, collect the following information:

• All job logs

• Task debug logs for VM upgrade tasks • Installation log from Ops Manager After Upgrade

After upgrade, perform the following to prepare new environment for use, check its health status and clean up:

• Re-Create BOSH alias

(9)

• Check the health of the deployment:

Run BOSH CLI commands to check system health status: ▪ bosh -e ALIAS -d DEPLOYMENT_NAME instances –ps

▪ bosh -e ALIAS vms –vitals

▪ Check resource settings:

▪ If custom VM Type or Persistent Disk Type options were added, ensure that resource setting values are correctly set and were not overwritten.

Note/Reason: Verify that Ops Manager Resource Config pane still lists the customized options.

An example of PKS tile resource configuration is below:

▪ Perform BOSH clean-up:

To clean up old stemcells, releases, orphaned disks, and other unused resources, execute the following command: ▪ bosh -e ALIAS clean-up --all

(10)

Upgrading PKS Tile

Follow the steps below to upgrade PKS Tile:

Download and import tiles and stemcells

Source https://network.pivotal.io/products/pivotal-container-service

Configuration settings are migrated to new version automatically. Follow the steps below to perform an upgrade.

• Download the desired version of the product from Pivotal Network. • Navigate to the Ops Manager Installation Dashboard and click Import

a Product to upload the product file.

• Under the Import a Product button, click + next to Pivotal Container Service. This adds the tile to the staging area.

• If Ops Manager does not have the Xenial stemcell required for PKS, the PKS tile displays the message Missing stemcell.

• Download and Import the Stemcell

Note:

• If stemcell library in Ops Manager already has a compatible Xenial stemcell, “missing stemcell” link does not appear. There is no need to download or import a new stemcell and this step can be skipped.

(11)

• If stemcell is updated, then master and worker K8s cluster nodes are re-created.

• if stemcell is not updated, BOSH will only update binaries on the worker and master nodes (and stop/start K8s processes)

• If Ubuntu Linux vulnerability is detected, operator will need to update the stemcell (this is true for K8s nodes and PKS control plane VM)

• For PKS tile upgrade which impacts only PKS control plane VM there is no need to enable the “Upgrade all clusters” errand

• For PKS tile upgrade which impact PKS control plane + K8s clusters (because there are new versions of K8s/Docker etc.) , need to enable the “Upgrade all clusters” errand

NOTE: BOSH errands are extra tasks designed to perform one specific function in the PKS environment. Current BOSH errands are:

• Upgrade K8s clusters, • Harbor smoke test, • NSX-T validation errand,

• Create pre-defined Wavefront alerts

Verify Errand Configuration

To verify that errands are configured correctly in the PKS tile, perform the following steps:

• Click the newly-added Pivotal Container Service tile. • Click Errands.

• Under Post-Deploy Errands, verify that the Upgrade all clusters errand is set to Default (On). The errand upgrades a single

Kubernetes cluster at a time. Upgrading PKS K8 clusters can temporarily interrupt the service, as described in Service Interruptions.

(12)

• Under Post-Deploy Errands, set the Run smoke tests errand to On. The errand uses the PKS Command Line Interface (PKS CLI) to create a Kubernetes cluster and then delete it. If the

creation or deletion fails, errand fails as well and installation of the PKS tile is aborted.

• Review other configuration panes. Click Save on any panes where the changes were made.

Warning: If Upgrading both PKS Control plane and clusters, the Upgrade All Clusters errand must be enabled.

Note: While upgrading PKS, place singleton jobs in the Availability Zone (AZ) selected when the PKS tile was first installed. Singleton jobs cannot be moved to another AZ.

Applying Changes to PKS tile

• Click Review Pending Changes. For more information about this Ops Manager page, see Reviewing Pending Product Changes. • Make sure that Pivotal Container Service selected as target

product

(13)

Monitor Upgrade

• Log in to the BOSH Director by running command:

bosh -e MY-ENVIRONMENT login

from a VM that can access the PKS deployment. For more information,

see Managing PKS Deployments with BOSH.

• Run command bosh -e MY-ENVIRONMENT tasks.

• Locate the task number for the errand in the # column of the BOSH output. • Run bosh task TASK-NUMBER, replacing TASK-NUMBER with the task

number that was located in the previous step.

(14)

Note: If upgrade fails, BOSH will stop the upgrade process and a manual intervention will be required to troubleshoot the issue.

(15)

Kubernetes Cluster Upgrade

What happens during a cluster upgrade ?

STEPS:

1. K8s cluster upgrade errand is initiated.

2. PKS Control Plane uses ODB (On Demand Broker) to interact with BOSH to get states and releases of K8s clusters. The upgrade “parent” BOSH task is triggered.

3. PKS Control Plane generates new deployment manifest for K8s cluster (based on components versions from the new Tile).

E.g. Sample Manifest

4. PKS Control Plane asks BOSH to upgrade K8s cluster using the deployment manifests generated in step 3.

(16)

5. BOSH will perform K8s cluster upgrade using ”child” BOSH tasks (executed sequentially for each cluster)

Note: Steps 3, 4 and 5 are repeated for each K8s cluster managed by PKS Control Plane.

6. Upgrades of either PKS tile and all PKS-provisioned K8s clusters or PKS tile only are available.

7. During an upgrade of PKS tile, configuration settings are automatically migrated to the new tile version.

8. BOSH-deployed products can set a number of canary instances (single worker node / single master) to upgrade first before the rest of deployment VMs. BOSH continues the upgrade only if canary instance upgrade

succeeds. If canary instance encounters an error, upgrade process stops running and other VMs are not affected.

9. PKS tile uses one canary instance when deploying or upgrading PKS. 10. The PKS API server is recreated

11. Each of deployed K8s clusters is recreated, one at a time in sequence:

▪ Master nodes are recreated

▪ Worker nodes are recreated

12. To upgrade the PKS tile only, set Upgrade all clusters

errand to Off before starting the upgrade

NOTES:

• Upgrading from PKS v1.2.5+ to PKS v1.3.x causes all certificates to be automatically regenerated. The old certificate authority is still trusted and has a validity of one year. But the new certificates are signed with a new certificate authority, which is valid for four years.

• When PKS is set to upgrade only PKS tile and not managed clusters, the K8s components version falls behind the PKS tile version. If K8s cluster components fall more than one version behind Tile, PKS cannot upgrade

(17)

the clusters. The clusters must be upgraded manually to match the PKS tile version before the next tile upgrade

• When PKS is set to upgrade both the PKS tile and managed clusters, updating any stemcell in the deployment rolls every VM in each K8s

cluster. This ensures that all the VMs are patched. With the recommended resource configuration, no workload downtime is expected.

K8 Cluster Upgrade Tasks

When a K8s cluster upgrade is performed, master nodes are upgraded first (using canary deployment by upgrading a single master node first) and then worker nodes are upgraded consequently (using canary deployment also use a single worker node as well).

BOSH operations on master node:

• Install new binary release (K8s, Docker or etcd…) • Kills current processes

• Start new processes

BOSH operations on worker node: • kubectl cordon node

• Unmount docker volumes (docker unmount) • kubectl drain node

• kubectl delete node

• docker unmount disk attached to the VM (Persistent and Ephemeral VM disks)

NOTE: All steps listed in Github: https://github.com/cloudfoundry-incubator/kubo-release/blob/master/jobs/kubelet/templates/bin/drain.erb

Following diagrams show a step-by-step flow of the upgrade tasks:

(18)

STEP 1: UPGRADE FIRST K8S MASTER NODE

Canary upgrade mode is used:

• If upgrade of first master node succeeds, then upgrade of all other master nodes is performed.

(19)

STEP 2: UPGRADE ALL OTHER K8S MASTER NODES

STEP 3: UPGRADE FIRST K8S WORKER NODE

Canary upgrade mode is used:

• If upgrade of first worker node succeeds, then upgrade of all other worker nodes is performed.

(20)

STEP 4: UPGRADE ALL OTHER K8S WORKER NODES

(21)

STEP 5: FINAL STATE OF K8S CLUSTER AFTER UPGRADE

Upon completion of Cluster upgrade process, all nodes are upgraded to K8s component versions “packaged” in PKS Tile

(22)

Monitor Upgrade

Result of the “parent” BOSH task (K8 cluster upgrade initiated by BOSH)

Running errand Upgrade all clusters errand for Pivotal Container Service

===== 2019-02-19 21:57:17 UTC Running "/usr/local/bin/bosh no-color non-interactive tty environment=10.0.0.3 --deployment=pivotal-container-service-406516468ba642b12bbb run-errand upgrade-all-service-instances"

Using environment '10.0.0.3' as client 'ops_manager'

Using deployment 'pivotal-container-service-406516468ba642b12bbb' Task 278447

Task 278447 | 21:57:18 | Preparing deployment: Preparing deployment (00:00:01)

Task 278447 | 21:57:19 | Running errand: pivotal-container-service/c1261790-a409-4306-b783-5e2f064e15c6 (0) (00:25:09) Task 278447 | 22:22:28 | Fetching logs for pivotal-container-service/c1261790-a409-4306-b783-5e2f064e15c6 (0): Finding and packing log files (00:00:01)

Task 278447 Started Tue Feb 19 21:57:18 UTC 2019 Task 278447 Finished Tue Feb 19 22:22:29 UTC 2019 Task 278447 Duration 00:25:11

Task 278447 done

Instance pivotal-container-service/c1261790-a409-4306-b783-5e2f064e15c6 Exit Code 0

Stdout [upgrade-all-service-instances] 2019/02/19 21:57:19.566155 [upgrade-all] STARTING OPERATION with 1 concurrent workers

[upgrade-all-service-instances] 2019/02/19 21:57:19.714335 [upgrade-all] Service Instances: 17eb7507-cb60-43e2-9e55-3da5118b289e d9b662d0-23e1-4239-b641-ed20ee62e692

[upgrade-all-service-instances] 2019/02/19 21:57:19.714356 [upgrade-all] Total Service Instances found: 2

[upgrade-all-service-instances] 2019/02/19 21:57:19.714363 [upgrade-all] Processing all instances. Attempt 1/5 [upgrade-all-service-instances] 2019/02/19 21:57:19.714428 [upgrade-all] [17eb7507-cb60-43e2-9e55-3da5118b289e] Starting to process service instance 1 of 2

[upgrade-all-service-instances] 2019/02/19 21:57:21.418066 [upgrade-all] [17eb7507-cb60-43e2-9e55-3da5118b289e] Result: operation accepted

[upgrade-all-service-instances] 2019/02/19 21:57:21.418137 [upgrade-all] [17eb7507-cb60-43e2-9e55-3da5118b289e] Waiting for operation to complete: bosh task id 278448

[upgrade-all-service-instances] 2019/02/19 22:10:23.881053 [upgrade-all] [17eb7507-cb60-43e2-9e55-3da5118b289e] Result: Service Instance operation success

[upgrade-all-service-instances] 2019/02/19 22:10:23.881090 [upgrade-all] [d9b662d0-23e1-4239-b641-ed20ee62e692] Starting to process service instance 2 of 2

[upgrade-all-service-instances] 2019/02/19 22:10:25.376640 [upgrade-all] [d9b662d0-23e1-4239-b641-ed20ee62e692] Result: operation accepted

[upgrade-all-service-instances] 2019/02/19 22:10:25.376663 [upgrade-all] [d9b662d0-23e1-4239-b641-ed20ee62e692] Waiting for operation to complete: bosh task id 278453

[upgrade-all-service-instances] 2019/02/19 22:22:27.946035 [upgrade-all] [d9b662d0-23e1-4239-b641-ed20ee62e692] Result: Service Instance operation success

[upgrade-all-service-instances] 2019/02/19 22:22:27.946066 [upgrade-all] Progress summary: Sleep interval until next attempt: 1m0s; Number of successful operation so far: 2; Number of service instance orphans detected so far: 0; Number of deleted instances before operation could happen: 0; Number of operations in progress (to retry) so far: 0

[upgrade-all-service-instances] 2019/02/19 22:22:27.946082 [upgrade-all] FINISHED PROCESSING Status: SUCCESS; Summary: Number of successful operations: 2; Number of service instance orphans detected: 0; Number of deleted instances before operation could happen: 0; Number of busy instances which could not be processed: 0; Number of service instances

“Parent” BOSH task for K8s cluster upgrade errand 2 K8s clusters found here Upgrade1st K8s cluster (using “child“ BOSH task) Upgrade 2nd K8s cluster (using “child“ BOSH task)

(23)

Using BOSH CLI

root@PKS-client-VM-WA:~# bosh run-errand upgrade-all-service-instances -d pivotal-container-service-c5b443119e4eae9be08a

Using environment '10.0.0.3' as client 'ops_manager'

Using deployment 'pivotal-container-service-c5b443119e4eae9be08a'

Task 15059

Task 15059 | 19:23:54 | Preparing deployment: Preparing deployment (00:00:01)

Task 15059 | 19:23:55 | Running errand: pivotal-container-service/e09bd1ef-6954-4ebd-a3d9-043dab7f1e5d (0) (00:12:03) Task 15059 | 19:35:58 | Fetching logs for pivotal-container-service/e09bd1ef-6954-4ebd-a3d9-043dab7f1e5d (0): Finding and packing log files (00:00:04)

Task 15059 Started Mon Feb 25 19:23:54 UTC 2019 Task 15059 Finished Mon Feb 25 19:36:02 UTC 2019 Task 15059 Duration 00:12:08

Task 15059 done

Result of the “parent” BOSH task

Stdout

[upgrade-all-service-instances] 2019/02/25 19:24:02.394947 [upgrade-all] STARTING OPERATION with 1 concurrent workers

[upgrade-all-service-instances] 2019/02/25 19:24:02.654120 [upgrade-all] Service Instances: 0eae1d08-00a7-4d50-a0f1-af9ddd5de64a

[upgrade-all-service-instances] 2019/02/25 19:24:02.654136 [upgrade-all] Total Service Instances found: 1

service-instances] 2019/02/25 19:24:02.654140 [upgrade-all] Processing all instances. Attempt 1 [upgrade-all-service-instances] 2019/02/25 19:24:02.654147 [upgrade-all] [0eae1d08-00a7-4d50-a0f1-af9ddd5de64a] Starting to process service instance 1 of 1

[upgrade-all-service-instances] 2019/02/25 19:24:03.855099 [upgrade-all] [0eae1d08-00a7-4d50-a0f1-af9ddd5de64a] Result: operation accepted

[upgrade-all-service-instances] 2019/02/25 19:24:03.855178 [upgrade-all] [0eae1d08-00a7-4d50-a0f1-af9ddd5de64a] Waiting for operation to complete: bosh task id 15060

[upgrade-all-service-instances] 2019/02/25 19:36:05.144914

[upgrade-all] [0eae1d08-00a7-4d50-a0f1-af9ddd5de64a] Result: Service Instance operation success

[upgrade-all-service-instances] 2019/02/25 19:36:05.144936 [upgrade-all] Progress summary: Sleep interval until next attempt: 1m0s; Number of successful operation so far: 1; Number of service instance orphans detected so far: 0; Number of deleted instances before operation could happen: 0; Number of operations in progress (to retry) so far: 0 [upgrade-all-service-instances] 2019/02/25 19:36:05.144943 [upgrade-all] FINISHED PROCESSING Status: SUCCESS; Summary: Number of successful operations: 1; Number of service instance orphans detected: 0; Number of deleted instances before operation could happen: 0; Number of busy instances which could not be processed: 0; Number of service instances that failed to process: 0

Stderr -

1 errand(s)

NOTE: If PKS upgrade fails, run the following CLI command:

“Parent” BOSH task for K8s cluster upgrade errand

(24)

bosh job <upgrade job> --result

The above commands will tell if upgrade for a specific cluster failed. If canary deployment fails, BOSH should recreate the node with previous working state (in order to maintain a K8s cluster up and running).

(25)

Application and System Level Considerations

Restrictions while upgrade is in progress

PKS API Server

• Cannot interact with PKS control plane

• Cannot interact with Kubernetes clusters

• Cannot Login though PKS CLI

• Cannot Retrieve information about clusters

• Cannot Create and delete clusters

• Cannot Resize clusters

Note: Recreating PKS API server does not affect deployed K8s clusters and their workloads. The user can still interact with them through the Kubernetes Command Line Interface, kubectl.

Master Nodes

• For single master node clusters cannot use kubectl or deploy new workloads

Note: To avoid this loss of functionality, use multi-master node K8s clusters. Access to K8s master node is available via NSX-T Virtual Server on the NSX-T Load Balancer instance allocated to the K8s cluster.

Worker Nodes

When PKS recreates worker nodes, the upgrade runs on one VM at a time.

• Node running on VM stops running containers.

• If application workloads run on a single worker node VM, apps will experience downtime. With multi worker node configuration,

(26)

workloads can be moved to other worker nodes not being drained & upgraded

Note: To avoid downtime, run at least one worker node per Availability Zone (AZ) for stateless workloads, and two worker nodes per AZ for stateful workloads (HA considerations).

(27)

Upgrade Checklist

• Determine upgrade path, check Pivotal/VMware documentation for validity • Review the Release Notes for version(s) that PKS is being upgraded to. • View workload resource usage in K8s Dashboard or other monitoring tools. • To avoid downtime, run at least one worker node per AZ for stateless

workloads and two worker nodes per AZ for stateful workloads (HA considerations).

• To avoid this loss of functionality, use multi-master clusters

• If clusters are near capacity for existing infrastructure, it is recommended to scale up the clusters before upgrade. Scale up clusters by running pks resize or create a new cluster using a larger deployment plan.

• Verify that the PKS control plane remains functional by performing the following steps:

▪ Add more workloads and create an additional cluster.

▪ Monitor the PKS control plane VM by clicking the Pivotal Container Service tile, selecting Status tab, and reviewing the Pivotal

Container Service VM’s data points. If any data points indicate

resources at capacity, scale the deployment accordingly. • Backup the PKS deployment using the following methods:

▪ Backup PKS

▪ Backup NSX-T

▪ Backup options in vCenter Server

• Verify that the Kubernetes environment is healthy:

▪ Run kubectl get nodes for all Kubernetes context and verify that all nodes are in a ready state.

▪ Run kubectl get pods –all-namespaces for all Kubernetes context and verify that all pods are running.

▪ Run bosh -d service-instance <UUID> instances –ps for each BOSH Kubernetes deployment and verify that all the processes are in a RUNNING state.

▪ Make sure there are no issues at the infrastructure layer. If using vSphere, verify that datastores have enough space, hosts have enough memory, there are no alarms, hosts are in a good state, etc. • Clean up failed Kubernetes Clusters

▪ View deployed clusters by running the following command: pks clusters

(28)

▪ If the Status of any cluster is in FAILED state, perform the

procedures described in the “Cannot Recreate a Cluster that failed to

Deploy” documentation to clean up the failed BOSH deployment.

▪ Run pks clusters command to view the deployed clusters again. If any clusters remains in the FAILED state, contact PKS Support. • PKS upgrade involves the following components:

▪ PKS CLI ▪ kubectl

▪ Pivotal Container Service ▪ Stemcell

▪ Ops Manager ▪ BOSH

• Verify NSX-T Configuration (vSphere with NSX-T Only)

If upgrading PKS for environments using vSphere with NSX-T, perform the following steps:

▪ Verify that datastores have enough space. ▪ Verify that hosts have enough memory. ▪ Verify that there are no alarms.

▪ Verify that hosts are in operational states.

▪ Verify that NSX Edge is configured for high availability using Active/Standby mode.

Note: Workloads in the Kubernetes cluster are unavailable while the NSX Edge nodes run the upgrade unless NSX Edge is configured for high availability. More information can be found here.

• Verify that Kubernetes clusters have unique external hostnames by checking for multiple clusters with the same external hostname.

Perform the following steps to verify uniqueness:

▪ Log in to the PKS CLI. For more information, see Log in to PKS CLI ▪ Log in with an account that has the UAA scope

of pks.clusters.admin. For more information about UAA scopes,

see Managing Users in PKS with UAA.

▪ View deployed PKS clusters by running the following command:

pks clusters

▪ For each deployed cluster, run pks cluster CLUSTER-NAME command to view details of that cluster.

(29)

▪ Examine the output to verify that the Kubernetes Master Host is unique for each cluster.

• Verify PKS Proxy configuration

Perform the following steps to verify PKS proxy configuration:

▪ Check whether an existing proxy is enabled: Log in to Ops Manager.

Click the Pivotal Container Service tile. Click Networking.

If HTTP/HTTPS Proxy is Disabled, no action is required. If HTTP/HTTPS Proxy is Enabled

If existing HTTP Proxy field contains any of the following values (or may have any of the following values), contact PKS Support:

▪ localhost

▪ Hostnames containing dashes, such as my-host.mydomain.com.

• Check PodDisruptionBudget Value

PKS upgrade will run endlessly and never complete if any Kubernetes application, for any reason, has had a PodDisruptionBudget created for it with maxUnavailable set to 0.

(See K8s Reference)

Perform the following steps to check PodDistributionBudget:

▪ Use the Kubernetes CLI, kubectl, to verify

the PodDisruptionBudget as the cluster administrator. Run the following command:

kubectl get poddistruptionbudgets –all-namespaces

▪ Examine the output. Any app listed should not show 0 in the MAX UNAVAILABLE column.

Note: For more information about kubectl, see Overview of kubectl in the Kubernetes documentation.

(30)

Post Upgrade

Download and re-install the PKS and Kubernetes CLI distributions.

Verify the Upgrade

After applying changes to PKS tile and upgrade is complete, perform the following steps:

▪ Verify that Kubernetes cluster environment is healthy.

▪ Verify that PKS control plane remains functional by performing the following steps:

- Add more workloads and create an additional cluster. - Monitor the PKS control plane VM by clicking the Pivotal

Container Service tile, selecting Status tab, and reviewing the Pivotal Container Service VM’s data points. If any data

points indicate resources are at capacity, scale the deployment accordingly.

Post Upgrade Checklist

• Verify the health of the current environment:

▪ Run kubectl get nodes for all Kubernetes context and verify that all nodes are in a ready state.

(31)

Example of output showing details of K8s nodes and components is shown below:

NAME STATUS ROLES AGE VERSION 38b1c0a5-f4af-4ad8-8cc8-7bd2c9ff643b Ready <none> 5h22m v1.12.7

4ad9e43a-9560-4e64-9635-975b96897910 Ready <none> 5h16m v1.12.7

e01381cd-e910-4da7-bb8f-b55c8b941741 Ready <none> 5h29m v1.12.7

▪ Run kubectl get pods –all-namespaces for all Kubernetes context and verify that all pods are running.

kubectl get pods --all-namespaces

▪ Run bosh -d MY-DEPLOYMENT instances --ps for each BOSH Kubernetes deployment and verify that all processes are in a running

state.

bosh -d MY-DEPLOYMENT instances –ps $ vice/a1b2c333d444e5f66a77

Example shown below:

bosh -d pivotal-container-service /a1b2c333d444e5f66a77 instances -ps ▪ To make sure etcd entries are synchronized, check K8s Component

status:

kubectl get cs

NAME STATUS MESSAGE ERROR scheduler Healthy ok

controller-manager Healthy ok

etcd-0 Healthy {"health":"true"}

(32)

kubectl get nodes -o wide

NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME

5c6bf0cc-bfba-4165-8972-cfb151691ed5 Ready <none> 13d

v1.12.4 172.15.0.3 172.15.0.3 Ubuntu 16.04.5 LTS 4.15.0-43-generic docker://18.6.2

63d8c40c-f1bd-476a-81c8-64e18df71a10 Ready <none> 13d

v1.12.4 172.15.0.4 172.15.0.4 Ubuntu 16.04.5 LTS 4.15.0-43-generic docker://18.6.2

▪ To make sure nothing is broken or missing after upgrade, verify that other K8s objects are in expected states

kubectl get pod, svc, ingress

…

NSX-T Container Plugin (NCP) Health (NSX-T only)

NCP Changes

NCP will be running as a bosh host process starting with PKS 1.1.5. Each master VM will have one NCP process running. One NCP process will be active and the others will be in standby. (daemon process)

PKS tile embeds a new version of NCP. Every time while creating a new K8 cluster or upgrading an existing all master nodes will run a new version of NCP

• Verify the health of NCP

▪ From the OpsManager VM Run the following

bosh -e MY-ENV login

Example:

(33)

▪ To locate the Kubernetes master node VM name and ID, run the following command

bosh -e MY-ENV -d MY-DEPLOYMENT vms

Example:

bosh -e pks -d pivotal-container-service/a1b2c333d444e5f66a77 vms NOTE: PKS API VM name begins with

pivotal-container-service and includes a BOSH-generated hash. This value is different

from the deployment hash.

▪ To establish SSH session with Kubernetes master node VM, run the following command:

bosh -e MY-ENV -d MY-DEPLOYMENT ssh VM-NAME/ID

Example:

bosh -e pks -d pivotal-container-service/a1b2c333d444e5f66a77 ssh

pivotal-container-service/000a1111-222b-3333-4cc5-de66f7a8899b

▪ From the master node VM, run the following command (as super user that can be activated by running: sudo su)

monit summary

▪ Verify that the Process: 'ncp' is running.

Use BOSH to connect to the master node via SSH and run monit summary

The Monit daemon x.x.x uptime: 8d 21h 33m

Process 'kube-apiserver' running Process 'kube-controller-manager' running Process 'kube-scheduler' running

(34)

Process 'etcd' running Process 'blackbox' running

Process 'ncp’ running  NCP process

Process 'bosh-dns' running Process 'pks-helpers-bosh-dns-resolvconf' running System 'system_localhost’ running

▪ To check if the NCP process is active or on standby, run the following command:

/var/vcap/jobs/ncp/bin/nsxcli -c get ncp-master status This instance is the NCP master

Current NCP Master id is <xxxxxx> Current NCP Instance id is <xxxxxx> Last master update at <Date>

▪ To restart the NCP process, run the following command: monit restart ncp

▪ To verify that the NCP process restarts successfully, run the following command:

monit summary

▪ Monitor the restart status with monit summary

The Monit daemon xxxxx uptime: 8d 21h 36m Process 'kube-apiserver' running Process 'kube-controller-manager' running Process 'kube-scheduler' running Process 'etcd' running Process 'blackbox' running

Process 'ncp' not monitored - restart pending  NCP is restarting

Process 'bosh-dns' running Process 'pks-helpers-bosh-dns-resolvconf' running System 'system_localhost' running

(35)

Note: Restarting the NCP process will also trigger cache rebuild. ▪ Restart NCP service

Use BOSH to connect to the master node via SSH and run command:

(36)

Best Practices for Workload High Availability with K8s

• Upgrades run on a single VM at a time. When a worker node VM goes

through the upgrade, workload could not be scheduled to run on it.

• Additional worker node VMs continue to run replicas of the workload, maintaining its uptime.

• Ensure that the pods are bound to a ReplicaSet or Deployment K8s constructs to reschedule in the event of a node failure.

• “Naked” pods (that is, Pods not bound to a ReplicaSet or Deployment are not rescheduled in the event of a node failure.

• Run workload on at least three worker node VMs using multiple replicas of the workloads distributed across those VMs for HA and LB.

• Edit the K8s manifest to define the ReplicaSet and configure an anti-affinity rule to ensure that the replicas run on separate worker nodes (see

examples below).

Set Workload Replicas

Set the number of workload replicas to handle traffic during rolling upgrades kind: Deployment metadata: # ... spec: replicas: 3 template: metadata: labels: app: APP-NAME spec: replicas: 3

Set this value to at least 3 to have at least three instances of the workload running at any time. app: APP-NAME Use this app name when defining the anti-affinity

(37)

Define Anti-Affinity Rules

To distribute the workload across multiple worker node VMs, anti-affinity rules must be used. If an anti-affinity rule is not defined, the replicated pods can be scheduled to the same worker node.

kind: Deployment metadata: # ... spec: replicas: 3 template: metadata: labels: app: APP-NAME spec: containers: - name: MY-APP image: MY-IMAGE ports: - containerPort: 12345 affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: "app" operator: In values: - APP-NAME topologyKey: "kubernetes.io/hostname" Note: ▪ If podAntiAffinity is set to

the requiredDuringSchedulingIgnoredDuringExecution value, the pod is eligible to be scheduled only on worker nodes that are not running a replica of this pod. If the requirement cannot be met, scheduling fails.

▪ If podAntiAffinity is set to

(38)

the scheduler tries to schedule pod replicas on different worker

nodes. If that is not possible, the scheduler assigns more than one

pod to the same worker node.

(K8s Documentation Reference)

Use Multi-AZ Worker Nodes

Using multiple Availability Zones (AZ) from distributed infrastructure clusters protects deployed workloads from zone failures.

Kubernetes evenly distributes pods via Replication Controller over multiple Availability Zones (AZs). For more granular control over scheduling pods, add an Anti-Affinity Rule to the K8s deployment (yaml) spec by

replacing "kubernetes.io/hostname" with "failure-domain.beta.kubernetes.io/zone".

Persistent Volumes

If AZ goes down, PersistentVolumes (PVs) and their data become

detached and cannot be automatically re-attached. To preserve PV data in the event of fallen AZ, persistent workload needs to have a failover

mechanism in place.

Depending on the underlying storage type, PVs are either completely free of zonal information or can have multiple AZ labels attached. Both options enable a PV to travel between AZs.

To ensure the uptime of the PVs during a cluster upgrade, Pivotal recommends having at least two nodes per AZ. By configuring the

workload as suggested, Kubernetes reschedules pods to the other node of the same AZ while BOSH is performing the upgrade.

(39)

Configuration Best Practices for K8s Clusters

• When defining configurations, specify the latest stable API version.

• Configuration files should be stored in version control before being pushed to the cluster. This allows to quickly roll back a configuration change if necessary.

• Write the configuration files using YAML rather than JSON format. YAML tends to be more user-friendly.

• Group related K8s objects into a single file whenever it makes sense.

• Don’t specify default values unnecessarily: simple, minimal configuration

will make errors less likely.

• Put object descriptions in annotations, to allow better introspection.

• Don’t use “naked” Pods (that is, Pods not bound to

a ReplicaSet or Deployment) if it can be avoided. “Naked” Pods will not be rescheduled in the event of a node failure.

• Create a Service before its corresponding backend workloads

(Deployments or ReplicaSets), and before any workloads that need to access it. When K8s starts a container, it provides environment variables pointing to all the Services which were running when the container was started. E.G. if a service named foo exists, all containers will get the following variables in their initial environment:

FOO_SERVICE_HOST = <host the service is running on> FOO_SERVICE_PORT = <port the service is running on>

• If you are writing code that talks to a Service, don’t use these environment variables; use the DNS name of Service instead. Service environment variables are provided only for older software which can’t be modified to use DNS lookups, and are a much less flexible way of accessing Services.

(40)

• Define and use labels that identify semantic attributes of your application or Deployment, such as { app: myapp, tier: frontend, phase: test,

deployment: v3 }. These labels can also be used to select the appropriate Pods for other resources; for example, a Service that selects all tier:

frontend Pods, or all phase: test components of app: myapp

• Use kubectl apply -f <directory> or kubectl create -f <directory>. This looks for Kubernetes configuration in all .yaml, .yml, and .json files

in <directory> and passes it to apply or create.

• Use label selectors for ‘get’ and ‘delete’ operations instead of specific object names.

• Use kubectl run and kubectl expose to quickly create single-container Deployments and Services. An example to use a Service to access an application can be found here.

(41)

Debugging and Troubleshooting Tools

Verify PKS CLI Version

The Pivotal Container Service (PKS) CLI interacts with the PKS deployment through the PKS API endpoint. Create, manage, and delete Kubernetes clusters on the PKS deployment by entering commands in the PKS CLI. The PKS CLI is under active development and commands may change between versions.

▪ To determine the version of PKS CLI installed locally, run the following command:

pks --version

For example:

pks –version

PKS CLI version: 1.0.0-build.3

SSH into the PKS VM

▪ To SSH into the PKS VM using BOSH, follow the steps below: Gather credentials and IP address information for the BOSH

Director, SSH into the Ops Manager VM, and use BOSH CLI to log in to the BOSH Director from the Ops Manager VM.

▪ To identify the PKS deployment’s name, run the following command:

bosh -e ENVIRONMENT deployments

where ENVIRONMENT is the BOSH environment alias set in Set a BOSH Environment Alias.

The PKS deployment name begins with pivotal-container-service and includes a BOSH-generated hash.

(42)

▪ To identify the PKS VM’s name, run the following command:

bosh -e ENVIRONMENT -d DEPLOYMENT vms

Where:

ENVIRONMENT is the BOSH environment alias. DEPLOYMENT is the PKS deployment name.

For example:

bosh -e pks -d pivotal- container-service/a1b2c333d444e5f66a77 vms

PKS VM name begins with pivotal-container-service and includes a BOSH-generated hash.

Note: PKS VM hash value is different from the hash in PKS deployment name.

▪ To SSH into the PKS VM, run the following command:

bosh -e ENVIRONMENT -d DEPLOYMENT ssh PKS-VM

Where:

ENVIRONMENT is the BOSH environment alias. DEPLOYMENT is the PKS deployment name. PKS-VM is the PKS VM name.

For example:

bosh -e pks \ -d pivotal-container-service/a1b2c333d444e5f66a77 \ ssh pivotal-container-service/000a1111-222b-3333-4cc5-

de66f7a8899b

View Log Files

Log files contain error messages and other information and can be used to diagnose issues with the PKS deployment. SSH into the PKS VM then follow the steps below to access PKS log files.

(43)

▪ To navigate to the PKS VM’s /var/vcap/sys/log log file directory, run the following command:

cd /var/vcap/sys/log

▪ Examine the following files:

On Master VMs, examine the kube-apiserver log file. On Worker VMs, examine the kubelet log file.

FAQs

Common FAQs

Service Interruptions Troubleshooting

(44)

Authors, Editors and Contributors

Rag Ramanathan Author/Editor

Riaz Mohamed Author /Editor

Daniel Zilberman Author /Editor

Francis Guillier Contributor

Lorenzo Paris Contributor

Contact: [email protected] References & Credits

https://confluence.eng.vmware.com/display/CNA/K1+- +K8s+cluster+upgrade+%281.3.1+-%3E+1.3.2%29+-+K8s+master+and+workers+upgrade+-+BOSH+level https://docs.pivotal.io/runtimes/pks/1-2/upgrade-pks.html https://docs.vmware.com/en/VMware-Enterprise-PKS/1.3/vmware-enterprise- pks-13/GUID-upgrading.html?hWord=N4IghgNiBcIA4GsDOACArnA5gJzAEwFMQBfIA https://kubernetes.io/docs/concepts/configuration/overview/ https://kubernetes.io https://docs.pivotal.io/pivotalcf/2-1/stemcells/

(45)

Comprehensive Guide on Upgrading PKS