Virtual Server Farms with Mainframe Linux

(1)

Sine Nomine Associates

Virtual Server Farms

with Mainframe Linux

Scott D. Courtney, Senior Engineer http://www.sinenomine.net/

(2)

Contents



Mainframe 101



Overview of Linux on zSeries



Selected customer experiences

(3)

(4)

Mainframe 101 -- History in brief



Mainframe Hardware

– IBM 360 introduced c. 1963

– IBM 370 introduced c. 1971, virtual memory added 1973

– IBM 3090 introduced c. 1985

– In about 1990, IBM renamed the ES/9000 as "S/390" (31-bit*)

– IBM zSeries (current generation) introduced in 2000 (64-bit)



Software Lineage

– VM (virtual machine) created in 1960s, officially released for 370

– VM is still in widespread use today as o.s. and as hypervisor

– First S/390 Linux port was "Bigfoot", done in 1998

– Port by IBM (contribs by others) created in "skunkworks" in 1999

– IBM's first official S/390 Linux announcement: May 17, 2000 * Older S/390 architecture was a hybrid 31/32-bit system.

(5)

Mainframe 101 -- Not your father's dinosaur!



IBM's top-of-the-line z990 Model D32 ("T-Rex") has:

– 16GB ~ 256GB main memory

– Up to 16 dedicated crypto co-processors (for SSL, etc.)

– Up to 48 GigEthernet channels, up to 12 10-Gig Ethernets

– Up to 30 LPARs (plus thousands of VMs)

– Full 64-bit architecture, but can support 31-bit VM guests

– Processors are 12-way multi-chip modules (MCMs), each with

– 8 main processors for general use

– 2 supervisory processors

– 2 spares that hot-swap if any other processor faults

– Up to four MCMs per z990, so up to 32 general processors



In addition, modern mainframes are not like in the old days:

(6)

Mainframe 101 -- Not just a big PC



Superior Industrial-spec or MIL-SPEC components, boards



ECC memory, ECC backplanes



Hipersockets are a concept simply

not found in PCs



Extremely high levels of redundancy,

fault tolerance

– MTBF measured in decades, not years

– Self-diagnosis, extensive hot-swap

capability



No such thing as "dumb" I/O --- ever!



Very high-level (i.e., "very" CISC) instruction set

– Performance not directly comparable against x86 or RISC clock

– No timer-tick interrupts (delays handled in hardware, µcode)

(7)

Mainframe 101 -- Networking



Hundreds or even thousands of "guest LANs", or "gLANs"

– Usually implemented with HiperSockets connectivity

– "Guest" used rather than "virtual" to distinguish from IEEE

802.1q VLAN (new zSeries also support that standard, though)

– Operate at backplane speeds, measured in gigabytes/second

– Separate, dedicated backplane channels, not just RAM bus

– Under VM operating system, the HiperSockets themselves can

be virtualized

– gLANs are fully isolated with security enforced by µcode



External network adapters support either L2 or L3 or both

– L3 offloads much of the processing for IP-only networks

– L2 offers bridge-like flexibility for IPX, DECnet, OSI MAP....etc.

(8)

Virtualizing the Virtual:

It’s Turtles All the Way Down!

VM can virtualize virtual hardware, n levels deep, and can run thousands of images at once.

 Test Plan Charlie: 41,400 Linux images

in an LPAR on a G5

 Test Plan Omega: 97,943 Linux images

on a ZZ7, 12-way@160 MIPS each, 16G RAM

 Thornton: Linux/390 under Hercules under Linux/390 under VM …

whoda thunk it?

These specific demonstrations are “lab queens” but the practical value of this capability is very, very real!

(9)

(10)

Limitations (real & perceived) of traditional mainframes



New hires don’t know how to run it



It’s….(gasp)…old!



Expensive per unit of processor power



Expensive software licensing, closed source



EBCDIC character set



Batch-oriented, non-interactive

– Mainframe I/O optimized for throughput, not latency

– Some of this is just perception, not reality



Slow pace of innovation



Old-fashioned development environment



Difficulty "gluing" to intranet, internet, distributed applications

(11)

Mainframe Linux Becomes Reality



Linas Vepstas: Bigfoot



IBM Boeblingen builds “official/unofficial” port



Marist College distribution



May, 2000: “It’s official!”



LinuxWorldExpo 2001: That’s “B” as in “Billion”



Features of S/390 Linux:

– Native port, not emulation

– ASCII character set just like other platforms

– Runs in LPAR, bare metal, or under VM

– S/390 architecture is in the stock kernel

(12)

Advantages Linux brings to S/390 or zSeries



Server consolidation on an unprecedented scale



Leveraging Open Source in large enterprises



Leveraging existing infrastructure, disaster planning,

hardware support, staffing, while deploying new apps



Tremendous integrating “glue” for tying mainframe to

intranet or internet



Licensing costs significantly lower



Compliant with open standards for API, languages,

and data formats



New hires now, in effect, trained on Big Iron



From IBM's perspective, new marketing direction

–

Mainframe MIPS sales per year way up since

(13)

Key Advantages of VM for Linux



Instrumented for performance and reliability monitoring



Horizontal scalability



Root access can be given without compromising the

mainframe’s native operating environment



Run parallel with existing mainframe applications



Use existing, proven, well-understood management

and backup tools



Risk-free and rapidly deployed test and pilot

environments



Distributed application development, centralized

security and infrastructure

(14)

“Best of Both Worlds”



Reliability, availability, scalability

as expected from a mainframe



Common operating system across

all architectures, from mainframe to embedded



Rapid innovation from the Linux

and Open Source community



Synergy of personnel from

(15)

Disadvantages of zSeries Linux

 Compute-bound tasks may not perform well

– Mainframe optimized for throughput, I/O, not burst latency

– Simple fact of sharing finite resources among a lot of

processes

 Optimized for high throughput, not low latency

– Bad choice for real-time embedded platform (duh!)

– May not be a good choice for streaming media (?)

 Still has financial barrier to entry relative to commodity Intel

(16)

Sine Nomine Associates’

Selected Customer Experiences



Customer 1: Email Infrastructure

(17)

Customer 1: Distributed Mail



9672-R26 mainframe as host platform



Providing DNS, SMTP, IMAP4, POP3 as primary services



Limited FTP and HTTP access as secondary services

(18)

Customer 1: Configuration



Single mainframe with many VM Linux instances for

horizontal scalability and functional partitioning



SMTP domain servers identically configured, horizontally

scaled



IMAP/POP servers accessed mail directories using NFS on

common file servers



Careful gLAN configuration to partition the bandwidth

(19)

Customer 1:

Block Diagram

nfs-home /home absolute share 1% nfs-mail /var/spool/mail absolute share 1% auth relative share 5000

WWW ftp/login IMAP POP incoming SMTP

gLAN 3 gLAN 2 gLAN 1 Router QUICKDSP absolute share 2% Legacy services IP forwarding G ig a b it O SA Point-to-point to auth VM TCP/IP stack Point-to-point Service machines: 64 MB 128 MB swap 32 MB 64 MB swap NIS master WWW server NFS client automounter NIS client WWW server NFS client automounter NIS client ftp login unique shell NFS client automounter NIS client NFS client IMAP 2000 automounter NIS client NFS client qpopper NIS client NFS client sendmail sendmail routing port-forwarded point-to-point ssh to NFS servers All machines support ssh for remote administration Each service (except DNS) may contain multiple machines behind round-robin DNS records Point-to-point from router Point-to-point from router Ports 20, 21, 23 forwarded to login Port 25 from external source forwarded to incoming Port 25 from internal source forwarded to SMTP Port 80 forwarded to WWW Port 110 forwarded to POP DNS2 DNS1 BIND load balancer BIND load balancer gLAN 4 to D N S1 , Au th , N F S se rve rs, SM T P, H e rme s

(20)

Customer 1: Lessons



Some tasks turned out to be more compute-bound than

originally anticipated

– SSH/SSL encryption

– Better with hardware acceleration on newer

mainframes

– Dynamic web content



More memory in the virtual environment does not always

equate to faster performance

– Ideal situation is actually to starve Linux from using

disk cache, let mainframe smart I/O handle this

– Linux-level swapping is extremely cheap under VM



NFS is not all that good as a distributed filesystem

– This has mproved in newer versions of NFS available

(21)

Customer 2: Financial Services



Feasibility study / test implementation



Existing application newly in production



Current platform is UNIX and NT/IIS



Web-based 4-tier implementation in Java



BEA: WebLogic Server

(22)

Customer 2: Application Port



Three people, six hours, one working app!



Zero source code changes (config files only)



Virtual network setup for app-level clustering



BEA: WebLogic Server is supported on zSeries Linux



One z/VM instance, two zSeries Linux instances, four IFL

processors – all added to an existing mainframe with zero downtime

(23)

Customer 2: Performance Testing



Gold standard: Match their existing production system’s

response time



zSeries with 1 CPU exceeded performance target by

approximately 3X



zSeries with 2 CPUs was too fast for their test environment

to saturate, but ran at least 6.5X their normal production load

(24)

Customer 2: Test Notes



Over 30 hours of intense high-load testing, zero failures of

zSeries hardware, z/VM, or Linux



Even at saturation load on one CPU, no software failures

or crashes



No application or BEA tuning for Linux platform (same

parameters as on UNIX)



Added and removed processors dynamically without

(25)

Lessons Learned



VM is critical to large scale Linux for System/390 scalability

– Limited LPAR count does not offer sufficient cost/benefit to

make the case for Linux on S/390 iron

– Loss of VM resource management and error recovery

substantially complicates system management



Lack of VM on other platforms is a major differentiator in

(26)

Challenges for zSeries Linux Deployment



Political Challenges



Technical Challenges

(27)

Political Challenges



Challenge: How to sell Linux and Open Source idea to

senior management?



zSeries Linux answers:

– Deploy alongside existing mainframe software, without

interruption to production

– Small project first, often infrastructural in nature

(28)

Political Challenges (cont’d)



zSeries Linux answers (continued):

– Personal case history: Pairing a VM guru with a Linux wiz

– Empowerment of open systems managers via access to

larger-scale resources, new data sources

– New levels of integration between legacy data and new

(29)

Technical Challenges



Backup/Recovery Solutions and DR

– Can be done within Linux, but not always best way



Performance Monitoring Instrumentation



Configuration Management



Security Management



Software Replication

– Shared read-only filesystem is one option, but more flexible

(30)

Project Planning Challenges



What type of project first?



Implementation planning

– Who is involved, and at what point in time?

– Need collaboration between mainframe and UNIX/Linux

personnel

– System automation tools from z/VM environment applied to

Linux instances

(31)

Lessons Learned



Mainframe Linux is now a proven

technology, not just an interesting

experiment



Performance issues can surprise you --

understand the workload, and benchmark



Don’t assume “conventional wisdom” is

always right



Consider non-technical factors in project

planning, especially for first Linux

deployment



VM/Linux may be the wrong answer, but

(32)

Daddy! I Wanna Play, Too!!!



Linux Community Development System

– Free root-level S/390 Linux account for 30, 60, or 90 days

– Any reasonable purpose (no spam, pr0n, game servers, but

"just learning" is fine)

– S/390 G5 or G6 system...not blazing, but okay for learning

– http://www-03.ibm.com/servers/eserver/zseries/os/linux/lcds/



Hercules

– Hardware emulation of S/390 or zSeries

– Possible, but not legal to run VM in this setting

– http://www.conmicro.cx/hercules/



Flex

– PC-based emulation, but licensed by IBM for z/VM

(33)

Resources on the Web

 This Presentation (downloadable)

http://sinenomine.net/node/520

 Linux/390 Project Home Page http://www.linuxvm.org

 IBM Linux zSeries Home

http://www.ibm.com/servers/eserver/zseries/os/linux/

 IBM VM Linux Resources http://www.vm.ibm.com/linux/

 Mainframe Historical Timeline

http://www-03.ibm.com/servers/eserver/zseries/timeline/

 IBM Linux Community Development System

http://www-1.ibm.com/servers/eserver/zseries/os/linux/lcds/

 “Dream Machine” Article Online

http://www.linuxplanet.com/linuxplanet/reports/1532/

 Sine Nomine Associates z/VM FAQ http://sinenomine.net/node/10