+ Cellular Attacks

WATCH:

ISSUE OVERVIEW

V

Since 1994: The Original Magazine of the Linux Community

™

FEBRUARY 2017 | ISSUE 274 http://www.linuxjournal.com

Detect Man-in-the-Middle

Cellular Attacks

Manage Docker Images and Containers with Puppet

BEST PRACTICES

for SysAdmin Alerts

POSTMORTEM:

What to Do After an Attack

EOF:

+

Secure Your Accounts

GEEK GUIDES

Practical books

for the most technical people on the planet.

Download books for free with a simple one-time registration.

http://geekguide.linuxjournal.com

Tame the Docker Life Cycle with SUSE

Author: John S. Tonello Sponsor: SUSE

SUSE Enterprise Storage 4

Author: Ted Schmidt Sponsor: SUSE

BotFactory: Automating the End of Cloud Sprawl

Author: John S. Tonello Sponsor: BotFactory.io

Containers 101

Author: Sol Lederman Sponsor: Puppet

An API Marketplace Primer for Mobile, Web and IoT

Author: Ted Schmidt Sponsor: IBM

Public Cloud Scalability for Enterprise Applications

Author:

Petros Koutoupis Sponsor: SUSE

Drupal 8 Migration Guide

Author: Drupalize.me Sponsor: Symantec

Beyond Cron, Part II: Deploying a Modern Scheduling Alternative

Author: Mike Diehl

Sponsor: Skybot

NEW!

GEEK GUIDES

Practical books

for the most technical people on the planet.

Download books for free with a simple one-time registration.

http://geekguide.linuxjournal.com

Tame the Docker Life Cycle with SUSE

Author:

John S. Tonello Sponsor:

SUSE

SUSE Enterprise Storage 4

Author:

Ted Schmidt Sponsor:

SUSE

BotFactory:

Automating the End of Cloud Sprawl

Author:

John S. Tonello Sponsor:

BotFactory.io

Containers 101

Author:

Sol Lederman Sponsor: Puppet

An API Marketplace Primer for Mobile, Web and IoT

Author:

Ted Schmidt Sponsor:

IBM

Public Cloud Scalability for Enterprise Applications

Author:

Petros Koutoupis Sponsor:

SUSE

Drupal 8 Migration Guide

Author:

Drupalize.me Sponsor:

Symantec

Beyond Cron, Part II:

Deploying a Modern Scheduling Alternative

Author:

Mike Diehl

Sponsor: Skybot

NEW!

CONTENTS FEBRUARY 2017 ISSUE 274

FEATURES

74 Cellular Man- in-the-Middle Detection

with SITCH

Build your own coordinated GSM anomaly detection system, using inexpensive, easy-to-source parts and open-source software.

Ash Wilson

92 Managing

Docker Instances with Puppet

Leverage Puppet roles and profiles, and discover how to target specific Docker configurations on hundreds or even thousands of

systems using simple hostname patterns.

Todd A. Jacobs

Cover Image © Can Stock Photo / woodoo

LINUX JOURNAL (ISSN 1075-3583) is published monthly by Belltown Media, Inc., PO Box 980985, Houston, TX 77098 USA.

Subscription rate is $29.50/year. Subscriptions start with the next issue.

CONTENTS

74

ON THE COVER

UÊ>˜>}iÊœVŽiÀʜ˜Ì>ˆ˜iÀÃÊ܈̅Ê*Õ««iÌ]Ê«°Ê™Ó UÊiÌiVÌÊ>˜‡ˆ˜‡Ì…i‡ˆ``iÊiÕ>ÀÊ
ÌÌ>VŽÃ]Ê«°ÊÇ{

UÊ-iVÕÀiÊ9œÕÀÊ
VVœÕ˜ÌÃÊ܈̅Ê/ܜ‡>V̜ÀÊ
Õ̅i˜ÌˆV>̈œ˜]Ê«°Ê{n UÊ iÃÌÊ*À>V̈ViÃÊvœÀÊ-ÞÃ
`“ˆ˜Ê
iÀÌÃ]Ê«°ÊÎn

UÊ*œÃ̓œÀÌi“\Ê7…>ÌÊ̜ÊœÊ
vÌiÀÊ>˜Ê
ÌÌ>VŽ]Ê«°Êxn UÊ "\ʈVÀœÃœvÌʳʈ˜Õݶ]Ê«°Ê££{

COLUMNS

32 Dave Taylor’s Work the Shell

Scissors, Paper or Rock?

38 Kyle Rankin’s Hack and /

Sysadmin 101: Alerting

48 Shawn Powers’

The Open-Source Classroom

All Your Accounts Are Belong to Us

58 Susan Sons’

Under the Sink

Postmortem

114 Doc Searls’ EOF

From vs. to + for Microsoft and Linux

IN EVERY ISSUE

8 Current_Issue.tar.gz 10 UPFRONT

30 Editors’ Choice 66 New Products 122 Advertisers Index

48 30

Executive Editor Senior Editor Associate Editor Art Director Products Editor Editor Emeritus Technical Editor Senior Columnist Security Editor Hack Editor Virtual Editor

Jill Franklin [email protected] Doc Searls

[email protected] Shawn Powers [email protected] Garrick Antikajian [email protected] James Gray

[email protected] Don Marti

[email protected] Michael Baxter [email protected] Reuven Lerner [email protected] Mick Bauer [email protected] Kyle Rankin [email protected] Bill Childers

[email protected]

President

Publisher

Associate Publisher

Director of Digital Experience

Accountant

Carlie Fairchild

[email protected] Mark Irgang

[email protected] John Grogan

[email protected] Katherine Druckman [email protected] Candy Beauchamp

[email protected] Contributing Editors

)BRAHIM (ADDAD s 2OBERT ,OVE s :ACK "ROWN s $AVE 0HILLIPS s -ARCO &IORETTI s ,UDOVIC -ARCOTTE 0AUL "ARRY s 0AUL -C+ENNEY s $AVE 4AYLOR s $IRK %LMENDORF s *USTIN 2YAN s !DAM -ONSEN

Linux Journal is published by, and is a registered trade name of, Belltown Media, Inc.

0/ "OX  (OUSTON 48  53!

Editorial Advisory Panel Nick Baronian Kalyana Krishna Chadalavada

"RIAN #ONNER s +EIR $AVIS -ICHAEL %AGER s 6ICTOR 'REGORIO

$AVID ! ,ANE s 3TEVE -ARQUEZ

$AVE -C!LLISTER s 4HOMAS 1UINLAN

#HRIS $ 3TARK s 0ATRICK 3WARTZ Advertising

% -!),: [email protected] 52,: www.linuxjournal.com/advertising

0(/.%     EXT  Subscriptions

% -!),: [email protected] 52,: www.linuxjournal.com/subscribe -!), 0/ "OX  (OUSTON 48  53!

LINUX IS A REGISTERED TRADEMARK OF ,INUS 4ORVALDS

Manage data expansion with SUSE Enterprise Storage.

SUSE Enterprise Storage, the leading open source storage solution, is highly scalable and resilient, enabling high-end functionality at a fraction of

the cost.

suse.com/storage

You cannot keep up with data explosion.

Data

Current_Issue.tar.gz

Everything Is Data, Data Is Everything

I

t doesn’t take more than a glance at the current HEADLINES TO SEE DATA SECURITY IS A VITAL PART OF ALMOST everything we do. Whether it’s concern over election HACKING OR USER ACCOUNTS BEING PUBLICIZED AFTER A WEBSITE compromise, our data integrity is more important than ever. Although there’s little we can do individually to STOP HACKERS FROM ATTACKING WEBSITES WE DONT PERSONALLY CONTROL WE ALWAYS CAN BE MORE CONSCIOUS OF HOW WE MANAGE OUR DATA AND CREDENTIALS FOR OUR OWN ACCOUNTS

As is becoming more and more common, this month, WE LOOK AT A LOT OF SECURITY ISSUES

!LTHOUGH NOT EXACTLY SECURITY RELATED $AVE 4AYLOR STARTS OFF ON ANOTHER SCRIPTING QUEST 7EVE BEEN

learning how to land on Mars, but this month, we look at how to play rock scissors paper with the command line.

It sounds like a simple endeavor, but the programmatic SIDE CAN BECOME COMPLICATED QUICKLY !S IS ALWAYS THE CASE WITH $AVES COLUMN THE OBJECTIVE IS FUN BUT THE learning experience along the way is priceless.

4HIS MONTH +YLE 2ANKIN HELPS US ALL SLEEP A LITTLE BETTER at night—not due to better security measures, but rather BY HELPING US CONFIGURE ON CALL ALERTS "EING WOKEN UP AT

AM BECAUSE A BIRD FLEW INTO THE SERVER ROOM WINDOW IS

V

VIDEO:

Shawn Powers runs through the latest issue.

SHAWN POWERS

Shawn Powers is the Associate Editor for Linux Journal. He’s also the Gadget Guy for LinuxJournal.com, and he has an

interesting collection of vintage Garfield coffee mugs. Don’t let his silly hairdo fool you, he’s a pretty ordinary guy and can be reached via email at

[email protected].

Or, swing by the

#linuxjournal IRC channel on Freenode.net.

NOT A GREAT WAY TO CATCH  WINKS +YLE SHOWS HOW TO AVOID FALSE POSITIVES but also how to make more intelligent alerts in general. Because servers seldom misbehave during regular business hours, his column is invaluable.

A while back I wrote an article on how to pick smart passwords. I think IT WAS ONLY LAST YEAR BUT IN )4 TIME THAT WAS EONS AGO 4HANKS TO A RECENT ATTEMPT AT COMPROMISING MY CELL PHONE SECURITY HAS BEEN ON THE TOP OF MY LIST RECENTLY +YLE 2ANKIN HELPED ME IDENTIFY SOME WAYS TO SECURE MY IDENTITY AND ) FIGURED IT WAS A GOOD TIME TO ELABORATE ON SOME GENERAL TIPS ON HOW TO KEEP YOUR CREDENTIALS AND ACCOUNTS SAFE !LSO FOR THE RECORD ITS INCREDIBLY AWESOME TO HAVE +YLE AS A PERSONAL FRIENDˆJUST SAYING

3USAN 3ONS TEACHES US TO LEARN FROM OUR MISTAKES AND AVOID REPEATING UNPLEASANT HISTORY 3PECIFICALLY SHE EXPLAINS HOW TO GO ABOUT DOING A POSTMORTEM ON A SECURITY ISSUE 7HETHER ITS A PRACTICE RUN A SERVER LEVEL COMPROMISE OR EVEN LEAKED ACCOUNT CREDENTIALS THE LESSONS WE LEARN FROM past problems are only as good as how detailed our postmortem procedures ARE 4HANKFULLY 3USAN IS WILLING TO SHARE HER EXPERTISE AND WE CAN ALL BENEFIT

7E GO INTO A FAIRLY SCARY WORLD WITH !SH 7ILSON THIS MONTH )T WASNT VERY LONG AGO THAT CELLULAR DATA SERVICES WERE RATHER DIFFICULT TO ATTACK 7EVE ALL BEEN CONDITIONED NOT TO TRUST OPEN 7I &I NETWORKS BUT THE CELLULAR CONNECTION ON OUR MOBILE DEVICES ISNT SOMETHING MOST OF US THINK ABOUT 4HOSE TIMES ARE CHANGING AND !SH HELPS US LEARN TO DETECT MAN IN THE MIDDLE ATTACKS ON CELLULAR NETWORKS )F YOU USE A MOBILE DEVICE AND IF YOURE READING Linux Journal, we ALL KNOW YOU ARE  THIS ARTICLE WILL BOTH INFORM AND SCARE YOU ) KNOW IT DID ME

!ND FINALLY 4ODD ! *ACOBS PROVIDES A GREAT LOOK INTO THE CURRENT

DevOps world with his article on managing Docker instances with Puppet.

)N ONE OF THOSE PEANUT BUTTER IN MY CHOCOLATE SITUATIONS COMBINING multiple DevOps tools tends to make something better than the sum OF ITS PARTS 4HIS ARTICLE BUILDS ON 4ODDS $ECEMBER  ARTICLE ABOUT provisioning Docker with Puppet, and here he describes how to manage Docker images and containers.

4HIS ISSUE CERTAINLY HAS A LOT OF SECURITY RELATED CONTENT WHICH IS GREAT IF YOU LIVE IN THE CURRENT DATA CENTRIC WORLD 4HANKFULLY IT ALSO CONTAINS other tech tips, product announcements and insight on our current TECHNOLOGY RICH WORLD 7HETHER YOURE LOOKING FOR A WAY TO DEPLOY A

more secure application or just want to learn about the latest cool mobile GAME THIS ISSUE SHOULD DO THE TRICK %NJOY
^Q

Current_Issue.tar.gz

UPFRONT

UPFRONT

NEWS + FUN

NEXT

Editors’ Choice VPREVIOUS

Current_Issue.tar.gz

V

diff -u

7…>̽ÃÊ iÜʈ˜ÊÊ

iÀ˜iÊiÛiœ«“i˜Ì

John Stultz wanted to allow specially privileged processes to migrate other processes between cgroup namespaces—essentially migrating PROCESSES FROM ONE VIRTUAL MACHINE TO ANOTHER 4HIS IS RISKY BECAUSE ONE OF THE WHOLE POINTS OF CGROUPS IS TO ISOLATE A VIRTUAL SYSTEM AND PREVENT ANY POTENTIALLY HOSTILE PROCESSES WITHIN IT FROM ESCAPING

*OHNS PATCH BASED ON IDEAS FROM Michael Kerrisk, would allow THIS PROCESS MIGRATION IF THE CONTROLLING PROCESS HAD BEEN GRANTED CAP_SYS_RESOURCE capabilities.

John explained that this originally had been an Android FEATURE CREATED so that people wouldn’t have to run their activity manager process with ROOT PRIVILEGES *OHN FELT HIS APPROACH WAS CLEANER AND MORE GENERIC

Kees Cook liked the patch, but Andy Lutomirski saw trouble up ahead. He explained:

$EVELOPMENTS ARE AFOOT TO MAKE CGROUPS DO MORE THAN RESOURCE CONTROL

&OR EXAMPLE THERES ,ANDLOCK AND THERES $ANIELS INGRESSEGRESS FILTER thing. Current cgroup controllers can mostly just DoS their controlled PROCESSES 4HESE NEW CONTROLLERS OR CONTROLLER LIKE THINGS CAN EXFILTRATE data and change semantics.

UPFRONT

Alexei Starovoitov asked if Andy knew a better approach, but Andy said he did not. He was only able to identify the problem, but had no solution to offer. He did, however, identify some constraints that any potential solution would need to adhere to. He said:

1. An insufficiently privileged process should not be able to move a victim into a dangerous cgroup.

2. An insufficiently privileged process should not be able to move itself into a dangerous cgroup and then use execve to gain privilege such that the execve’d program can be compromised.

3. An insufficiently privileged process should not be able to make an existing cgroup dangerous in a way that could compromise a victim in that cgroup.

4. An insufficiently privileged process should not be able to make a cgroup dangerous in a way that bypasses protections that would otherwise protect execve() as used by itself or some other process in that cgroup.

John didn’t know where to go with those

admonitions, and the project seemed to stall for a few weeks. Finally Andy suggested:

The cgroupfs interface is a bit unfortunate in that it doesn’t really express the constraints. To safely migrate a task, ISTM you ought to have some form of privilege over the task and some form of privilege over the cgroup. cgroupfs only handles the latter.

At Your Service

SUBSCRIPTIONS: Linux Journal is available in a variety of digital formats, including PDF, .epub, .mobi and an online digital edition, as well as apps for iOS and Android devices.

Renewing your subscription, changing your email address for issue delivery, paying your invoice, viewing your account details or other subscription inquiries can be done instantly online: http://www.linuxjournal.com/subs.

Email us at [email protected] or reach us via postal mail at Linux Journal, PO Box 980985, Houston, TX 77098 USA. Please remember to include your complete name and address when contacting us.

ACCESSING THE DIGITAL ARCHIVE:

Your monthly download notifications will have links to the various formats and to the digital archive. To access the digital archive at any time, log in at http://www.linuxjournal.com/digital.

LETTERS TO THE EDITOR: We welcome your letters and encourage you to submit them at http://www.linuxjournal.com/contact or mail them to Linux Journal, PO Box 980985, Houston, TX 77098 USA. Letters may be edited for space and clarity.

WRITING FOR US: We always are looking for contributed articles, tutorials and real-world stories for the magazine.

An author’s guide, a list of topics and due dates can be found online:

http://www.linuxjournal.com/author.

FREE e-NEWSLETTERS: Linux Journal editors publish newsletters on both a weekly and monthly basis. Receive late-breaking news, technical tips and tricks, an inside look at upcoming issues and links to in-depth stories featured on http://www.linuxjournal.com. Subscribe for free today: http://www.linuxjournal.com/

enewsletters.

ADVERTISING: Linux Journal is a great resource for readers and advertisers alike.

Request a media kit, view our current editorial calendar and advertising due dates, or learn more about other advertising and marketing opportunities by visiting us on-line: http://www.linuxjournal.com/

advertising. Contact us directly for further information: [email protected] or +1 713-344-1956 ext. 2.

UPFRONT

#!0?#'2/50?-)'2!4% OUGHT TO BE OKAY /R MAYBE CGROUPFS NEEDS TO GAIN A CONCEPT OF hDANGEROUSv CGROUPS AND FURTHER RESTRICT THEM AND

#!0?393?2%3/52#% SHOULD BE FINE FOR NON DANGEROUS CGROUPS

But, Tejun Heo OBJECTED THAT IF #!0?393?2%3/52#% WAS DISQUALIFIED DUE TO OVERLAPPING USERS IT WOULD BE BETTER TO USE A DIFFERENT CAPABILITY altogether. He suggested:

7E CANT DO IT PROPERLY ON ;CGROUPS= V BECAUSE SOME CONTROLLERS ARENT PROPERLY HIERARCHICAL AND DELEGATION MODEL ISNT WELL DEFINED &OR EXAMPLE NOTHING PREVENTS A PROCESS FROM BEING PULLED ACROSS DIFFERENT SUBTREES WITH THE SAME DELEGATION BUT V CAN DO IT PROPERLY !LL THATS NECESSARY IS TO

MAKE THE #!0 TEST /2D TO OTHER PERM CHECKS INSTEAD OF !.$ING SO THAT THE CAP just allows overriding restrictions expressed through delegation but it’s normally possible to move processes around in one’s own delegated subtree.

4EJUN WENT ON TO EXPLAIN

$ELEGATION IS AN EXPLICIT OPERATION AND REFLECTED IN THE OWNERSHIP OF THE SUBDIRECTORIES AND CGROUP INTERFACE FILES IN THEM 4HE SUBHIERARCHY CONTAINMENT IS ACHIEVED BY REQUIRING THE USER WHOS TRYING TO MIGRATE A PROCESS TO HAVE WRITE PERM ON CGROUPPROCS ON THE COMMON ANCESTOR OF the source and target in addition to the target.

)N OTHER WORDS ITS A COMPLETELY DIFFERENT APPROACH FROM THE ONE initially proposed by John.

4HE DISCUSSION ENDED INCONCLUSIVELY WITH THE MAIN QUESTION REMAINING whether to use an existing capability or write a new one.

4YPICALLY CGROUP FEATURES ARE INSANE 4HERE ARE OFTEN SECURITY ISSUES AFFECTING VIRTUAL SYSTEMS THAT WOULDNT AFFECT THE OUTER RUNNING SYSTEM FORCING ,INUX TO OFFER ONLY A WEIRD SPECIAL CASED SUBSET OF NORMAL

FEATURES !ND THERE ARE ALSO BIZARRE USE CASES SURROUNDING VARIOUS FEATURE ENHANCEMENTS IN WHICH DEVELOPERS WANT TO ADD FUNCTIONALITY TO CGROUPS THAT WOULD NOT BE DESIRABLE IN REGULAR ,INUX )TS ALL VERY HERE BE DRAGONS AND FULL OF MAGIC -IGRATING PROCESSES BETWEEN VIRTUAL SYSTEMS WILL

UPFRONT

probably be a lot like that.

Serge E. Hallyn pointed out a security issue with cgroups. He said:

Root in a user [namespace] cannot be trusted to write a traditional

SECURITYCAPABILITY XATTR )F IT WERE ALLOWED TO DO SO THEN ANY UNPRIVILEGED user on the host could map his own uid to root in a namespace, write the XATTR AND EXECUTE THE FILE WITH PRIVILEGE ON THE HOST

4HE PROBLEM WAS THAT IN THE OUTER SYSTEM A USER MIGHT LEGITIMATELY DO something like that, while on a virtualized system, it was a security hole.

Serge posted a patch to do crazy madness in order to simulate proper BEHAVIOR ON THE VIRTUAL MACHINE 4HE PATCH HE SAID hALLOWS A SIMPLE setxattr to work, allows tar/untar to work, and allows us to tar in one namespace and untar in another while preserving the capability, without RISKING LEAKING PRIVILEGE INTO A PARENT NAMESPACEv

He explained:

7HEN A TASK IN A USER NS WHICH IS PRIVILEGED WITH #!0?3%4&#!0 TOWARD THAT USER?NS ASKS TO WRITE V SECURITYCAPABILITY THE KERNEL WILL TRANSPARENTLY REWRITE THE XATTR AS A V WITH THE APPROPRIATE ROOTID

3UBSEQUENTLY ANY TASK EXECUTING THE FILE THAT HAS THE NOTED KUID AS ITS ROOT UID OR WHICH IS IN A DESCENDANT USER?NS OF SUCH A USER?NS WILL RUN THE FILE WITH CAPABILITIES

)F A TASK WRITES A V SECURITYCAPABILITY THEN IT CAN PROVIDE A UID VALID WITHIN ITS OWN USER NAMESPACE OVER WHICH IT HAS #!0?3%4&#!0 FOR THE XATTR

4HE KERNEL WILL TRANSLATE THAT TO THE ABSOLUTE UID AND WRITE THAT TO DISK

!FTER THIS A TASK IN THE WRITERS NAMESPACE WILL NOT BE ABLE TO USE THOSE capabilities, but a task in a namespace where the given uid is root will.

Eric W. Biederman GAVE A QUICK LOOK AND SAID THE PATCH SEEMED strange but correct. He said he’d go over it thoroughly and report back.

-EANWHILE -ICHAEL +ERRISK ASKED FOR SOME DOCUMENTATION PERHAPS IN THE MAN PAGES FOR user_namespaces(7) or capabilities(7), and Serge wrote some up.^—Zack Brown

UPFRONT

Non-Linux FOSS:

a Clippy That Never Forgets

I hate it when I paste something into a window, only to realize I’d copied something new into the clipboard. I usually end up with EIGHT PARAGRAPHS PASTED INTO A LOGIN BOX 4O QUOTE MY COLLEGE AGED daughter, the struggle is real.

4HANKFULLY ITS EASY TO INTEGRATE A CLIPBOARD MANAGER INTO /3 8

3EVERAL OPTIONS ARE AVAILABLE BUT MY FAVORITE HAPPENS TO BE OPEN SOURCE )F YOU HEAD OVER TO HTTPSGITHUBCOM4ERMI4&LYCUT, you’ll FIND &LYCUT WHICH IS A CLIPBOARD MANAGER THAT QUIETLY RECORDS ALL YOUR clippings and allows you to paste whichever one you want at any GIVEN TIME "Y DEFAULT IF YOU WANT TO USE &LYCUT INSTEAD OF THE SYSTEM CLIPBOARD YOU PRESS #OMMAND 3HIFT 6 INSTEAD OF JUST #OMMAND 6

A screen overlay lets you scroll through previous clippings, and you DOUBLE CLICK ON THE ONE YOU WANT TO PASTE

&LYCUT IS A VERY SIMPLE TOOL BUT ALL THE BEST ONES USUALLY ARE )F you’ve ever accidentally overwritten your clipboard, you owe it to YOURSELF TO DOWNLOAD &LYCUT EITHER FROM THE 'IT(UB PAGE OR THE -AC App store.—Shawn Powers

UPFRONT

Getting Sticky with It

!LTHOUGH THEY MIGHT NOT BE SO GOOD FOR CREDIT CARDS OR FLOPPY DISKS MAGNETS ARE ONE OF THOSE THINGS THAT ALWAYS HAVE FASCINATED ME

&OR THE PAST FEW YEARS )VE WANTED TO GET A SET OF THE ROUND :EN -AGNETS TO PLAY WITHˆTHEYRE SORT OF LIKE AN EXTRA SCIENCE Y VERSION OF ,%'/S 5NFORTUNATELY BEFORE ) WAS ABLE TO PURCHASE ANY THE 53 GOVERNMENT BANNED THEIR SALE

2ECENTLY THE FOLKS AT :EN -AGNETS WON THEIR LONG LEGAL BATTLE

These are what I made last night with my new micromagnets. I can hardly wait for the full-size ones!

UPFRONT

AND ARE ABLE TO SELL TINY STRONG MAGNETS AGAIN 4HE REGULAR SIZE :EN -AGNETS ARENT AVAILABLE YET BUT THANKFULLY PRODUCTION ONCE AGAIN CAN BEGIN )N THE MEANTIME ) WAS ABLE TO ORDER hMICROMAGNETSv FROM THE SAME COMPANY 4HEY WORK JUST LIKE :EN -AGNETS BUT ARE TINIER

I decided to order a couple sets, because I’m impatient and also to SUPPORT THE COMPANY WHO FOUGHT THE BATTLE ALLOWING MAGNETS TO BE SOLD IN THE 53 ONCE AGAIN

4O READ ABOUT THE LEGAL BATTLE CHECK OUT THE BLOG HERE

HTTPZENMAGNETSCOMMAGNET BAN CLEARED GAME ON. And while YOURE THERE FEEL FREE TO PRE ORDER SOME :EN -AGNETS ) SURE DID

—Shawn Powers

Archive 1994–2016

NOW AVAILABLE!

SAVE $10.00 by using discount code 2017ARCH at checkout.

Coupon code expires 3/28/2017

www.linuxjournal.com/archive

UPFRONT UPFRONT

Get a Haircut, Get a Real Job

)M OFTEN ASKED ABOUT WHAT THE LATEST TRENDS IN )4 WILL MEAN FOR JOB HUNTERS )TS INTERESTING FOR ME BECAUSE ALTHOUGH ) HAVENT ACTIVELY LOOKED FOR A JOB IN YEARS ) DO CREATE TRAINING THAT HELPS PEOPLE GET HIRED EVERY DAY 3O ) FIGURED A FEW TIPS FOR THE CURRENT JOB MARKET WOULD BE A GREAT WAY FOR ME TO ANSWER LOTS OF EMAILS IN ONE FELL swoop. Here it goes.

1) DevOps is no longer magic. &OR THE PAST TWO YEARS IF YOU COULD PUT h$EV/PSv ON YOUR RÏSUMÏ YOUD PRETTY MUCH GET HIRED ON PRINCIPLE ALONE ,ATELY $EV/PS HAS BECOME A UBIQUITOUS PART OF )4 AND IT ISNT THE SPECIAL SNOWFLAKE IT USED TO BE $ONT GET ME WRONG

YOU STILL NEED TO HAVE $EV/PS SKILLS ON YOUR RÏSUMψJUST KNOW THAT IT WONT GET YOU HIRED ON ITS OWN )NSTEAD MENTION WHAT SORTS OF THINGS you have done or can do utilizing DevOps.

2) Security is vital. )F YOU LOVE SECURITY THE FUTURE LOOKS BRIGHT FOR YOU "UT EVEN IF SPECIALIZING IN SECURITY ISNT WHAT YOU WANT TO DO AS A CAREER ITS IMPORTANT TO APPROACH EVERY ASPECT OF TECHNOLOGY WITH A SECURITY MINDSET 4WENTY YEARS AGO WE WORRIED ABOUT FIREWALLS BUT RARELY CONSIDERED ATTACKS COMING FROM INSIDE OUR OWN NETWORKS 4HAT WAS A POOR ATTITUDE  YEARS AGO AND NOW ITS TECHNOLOGY SUICIDE

Security isn’t something you add, it’s a way you plan.

3) Developers, developers, developers. Steve Ballmer may have SEEMED LIKE A CRAZY MAN WHEN HE SHOUTED IT ON STAGE BACK IN 

BUT NOW THAT $EV/PS IS A PART OF EVERYTHING WE DO DEVELOPER SKILLS ARE AS IMPORTANT AS EVER %VEN THE TRADITIONAL SYSTEM ADMINISTRATOR or operations person will need to have at least rudimentary

PROGRAMMING SKILLS IN ORDER TO FUNCTION IN OUR $EV/PS WORLD 0LUS HERES A SECRET PROGRAMMING IS ACTUALLY KIND OF FUN ESPECIALLY WHEN IT CAN SAVE YOU TIME ON THE JOB

4) Don’t forget your roots. In the Pixar movie WALL-E, civilization has advanced to the point that everything is automated. It means

UPFRONT

LIFE FOR PEOPLE IS EXTREMELY EASY BUT IT ALSO MEANS THEY DONT KNOW HOW TO DO ANYTHING FOR themselves. With everything in the data center and the cloud being automated, it’s easy to hire an entire team that knows nothing about the ACTUAL PROCESSES THEYRE AUTOMATING 4HAT WORKS great—until it doesn’t. Make sure you’re well VERSED IN THE UNDERLYING SYSTEMS ALMOST ALWAYS Linux), so when something goes wrong, you KNOW HOW TO FIX IT

5) Be a softy! 3OFT SKILLS COMMUNICATION skills, cooperation skills and so on) are SOMETHING WE ALL TOO OFTEN OVERLOOK IN )4

"UT NOT ONLY DO SOFT SKILLS HELP YOU IN THE interviewing process, they also help you in THE CURRENT )4 LANDSCAPE WHERE VARIOUS

disciplines are working closer than ever. Again,

$EV/PS IS MUCH TO BLAME FOR THIS BLURRING OF department lines. Any employee who is able TO COMMUNICATE CROSS DISCIPLINE ESPECIALLY ONE WHO IS ABLE TO COMMUNICATE WITH NON )4 FOLKS IS GOING TO BE INVALUABLE TO ANY

ORGANIZATION 4AKE SOME COMMUNICATION classes. You might be the only nerd in the room, but you’ll also likely have the best JOB OPPORTUNITIES
—Shawn Powers

THEY SAID IT

Remember that nobody will ever get ahead of you as long as he is kicking you in the seat of the pants.

—Walter Winchell

The great thing about a computer notebook is that no matter how much you stuff into it, it doesn’t get bigger or heavier.

—Bill Gates

Security is a kind of death.

—Tennessee Williams

Above all things, never be afraid.

The enemy who forces you to retreat is himself afraid of you at that very moment.

—Andre Maurois

There’s only one thing I hate more than lying: skim milk. Which is water that’s lying about being milk.

—Ron Swanson

UPFRONT

Gabedit: the Portal to Chemistry

-ANY CHEMISTRY SOFTWARE APPLICATIONS ARE AVAILABLE FOR DOING SCIENTIFIC WORK ON ,INUX )VE COVERED SEVERAL HERE IN PREVIOUS ISSUES OF THE

MAGAZINE AND OF THEM HAVE THEIR OWN PECULIAR SPECIALTIESˆAREAS WHERE one may work better than another. So, depending on what your research ENTAILS YOU MAY NEED TO USE MULTIPLE SOFTWARE PACKAGES TO HANDLE ALL OF THE WORK 4HIS IS WHERE 'ABEDIT WILL STEP IN TO HELP YOU OUT

'ABEDIT PROVIDES A SINGLE UNIFIED INTERFACE TO A MULTITUDE OF CHEMISTRY packages available on your system. It should be available within the PACKAGE MANAGEMENT SYSTEMS FOR MOST DISTRIBUTIONS &OR EXAMPLE ON

$EBIAN BASED SYSTEMS YOU CAN INSTALL IT WITH THE COMMAND

sudo  apt-­get  install  gabedit

Figure 1. When you first start Gabedit, you’ll get an empty project where you can begin your work.

UPFRONT

Once it’s installed, start it with the gabedit COMMAND 4HE VERY FIRST TIME YOU START 'ABEDIT YOULL SEE A SERIES OF WINDOWS DESCRIBING ALL THE DATA DIRECTORIES THAT NEED TO BE CREATED IN ORDER FOR 'ABEDIT TO RUN 4HE PANE ON THE LEFT HAND SIDE SHOWS A LISTING OF ALL THE CHEMISTRY PROGRAMS YOU COULD USE FOR YOUR WORK 4HE CENTRAL PANE PROVIDES TWO TABS ONE FOR INPUT AND ONE FOR RESULTS

4O START WORKING WITH 'ABEDIT YOU NEED TO CREATE A NEW INPUT FILE FOR THE SOFTWARE PACKAGE YOU WANT TO WORK WITH 4HE ICON BAR ACROSS THE TOP OF THE WINDOW PROVIDE BUTTONS FOR THE VARIOUS TYPES OF INPUT FILES THAT 'ABEDIT CAN USE #LICKING ON ONE OF THEM WILL POP UP A NEW WINDOW WHERE YOU CAN ENTER PARAMETERS RELEVANT FOR THAT TYPE OF INPUT FILE &OR EXAMPLE CLICKING ON THE FIRST BUTTON POPS UP A WINDOW WHERE

Figure 2. When you create a new input file, a new window pops up where you can enter the initial parameters.

UPFRONT

YOU CAN CREATE A NEW INPUT FILE FOR '!-%33

)F YOU TRY TO DO THIS AT THE BEGINNING OF YOUR WORK YOULL ACTUALLY GET AN ERROR !LL OF THESE PROGRAMS DEPEND ON SOME SET OF ATOMS DEFINED AS A GEOMETRY IN ORDER TO DO THEIR CALCULATIONS WHICH

MEANS YOU NEED TO CREATE THIS GEOMETRY FIRST #LICKING THE 'EOMETRY MENU ENTRY WILL PROVIDE A LIST OF DIFFERENT OPTIONS FOR CREATING A NEW GEOMETRY 4HE FIRST TWO ARE SPECIALIZED OPTIONS FOR 'AUSSIAN AND -OLPRO &OR THIS EXAMPLE LETS USE THE TWO OPTIONS AT THE BOTTOM OF THE LIST 4HE FIRST OPTION POPS UP A NEW WINDOW WHERE YOU CAN SELECT THE TYPE OF GEOMETRY 89: FOR EXAMPLE AND THEN CREATE A TABLE OF atoms used within your geometry.

2IGHT CLICKING INSIDE THE TABLE OF THE GEOMETRY EDITOR PROVIDES A POP UP MENU WHERE YOU CAN ADD A NEW ENTRY TO THE TABLE 4HIS ALLOWS YOU TO SELECT THE ELEMENT LOCATION AND CHARGE FOR THE NEW POINT IN THE GEOMETRY 4HIS GEOMETRY EXISTS WITHIN THE MEMORY SPACE OF THE CURRENT PROJECT WHICH MEANS IT WILL BE AVAILABLE FOR OTHER FUNCTIONS within Gabedit.

4HE OTHER AVAILABLE GEOMETRY FUNCTION IS THE DRAW FUNCTION 9OU

Figure 3. You need to create a new geometry that will be used in the calculations.

UPFRONT

can access it via the GeometryA$RAW MENU ITEM 4HIS POPS UP A NEW window where you can visualize your molecule and manipulate it BEFORE DOING ANY CALCULATIONS

Figure 4. You can add individual elements, setting their chemical properties, to your geometry.

UPFRONT

Here, you can edit the existing geometry and move elements around, or you can add or remove elements to the molecule. You EVEN CAN ADD ENTIRE FUNCTIONAL UNITS SUCH AS BENZENE RINGS OR alcohol groups.

/NCE YOU HAVE AN INPUT FILE YOU NEED TO RUN IT THROUGH THE

Figure 5. You can use the draw functionality to visualize the geometry of your collection of atoms.

UPFRONT

APPROPRIATE SOFTWARE PACKAGE IN ORDER TO GET RESULTS )F THE

programs you wish to use are installed on your local machine and ARE IN YOUR SEARCH PATH IT SHOULD JUST WORK OUT OF THE BOX )F THEY were installed in some other location, you need to tell Gabedit where they are. Clicking the SettingsA0REFERENCES MENU ITEM WILL BRING UP a new window where you can set the commands needed to run the relevant programs.

You then can run the program either by clicking the run button in the icon bar or clicking the RunARun a Computation Chemistry PROGRAM MENU ITEM 4HIS WILL PRESENT A NEW WINDOW WHERE YOU CAN SET THE PARAMETERS FOR THIS RUN

Figure 6. You can set the specific commands for each of the available chemistry packages.

UPFRONT

For a local run, you can set parameters including which program to USE WHAT FOLDER TO RUN IN AND THE FILENAMES AND COMMANDS TO EXECUTE

)F YOU SELECT h2EMOTE HOSTv INSTEAD YOU CAN CHOOSE THE PROTOCOL TO

communicate over and which host to communicate with. You also can set

Figure 7. You can set the parameters for either a local run or a remote run within the same window.

UPFRONT

the user name and password to use, along with the working directory on THE REMOTE MACHINE )F YOU FIND THAT YOUR INITIAL CHOICE OF PROGRAM ISNT OPTIMAL YOU CAN TRY ANOTHER "Y CLICKING THE 4OOLSAOpen Babel menu ITEM YOU GET A WINDOW THAT ALLOWS YOU TO DO A TRANSLATION OF INPUT FILE FROM ONE FILE FORMAT TO ANOTHER 4HIS WAY YOU CAN REUSE YOUR PREVIOUS WORK WITHIN A DIFFERENT SOFTWARE PACKAGE

'ABEDIT IS NOT ONLY USEFUL IN SETTING UP A COMPUTATIONAL CHEMISTRY PROBLEM AND RUNNING IT BUT ITS ALSO USEFUL IN ANALYZING THE RESULTS

AFTERWARD 4HE ANALYSIS FUNCTIONS ARE AVAILABLE UNDER THE 4OOLS MENU ITEM

9OU CAN SELECT TO LOAD A FILE FOR A BASIC 89 PLOT AND YOU CAN SELECT THE 4OOLSA89 PLOTTER MENU ITEM TO BRING UP THE PLOT WINDOW 2IGHT CLICKING THE PLOT WINDOW BRINGS UP A MENU WHERE YOU CAN CHANGE THE OPTIONS OF THE PLOT AS WELL AS LOAD DATA FILES TO BE PLOTTED 4HERE ALSO IS AN OPTION TO DO CONTOUR PLOTS BY CLICKING THE 4OOLSAContours plotter menu item.

!DDITIONALLY THERE IS A WHOLE SERIES OF SPECTRUM ANALYSES THAT YOU CAN APPLY AS WELL 9OU CAN DO )2 2AHMAN 56 AND %#$ SPECTRAL ANALYSIS &OR EACH OF THESE OPTIONS IN THE 4OOLS MENU YOU CAN LOAD AN OUTPUT FILE FROM A NUMBER OF DIFFERENT FILE FORMATS INCLUDING A SPECIAL 'ABEDIT FILE FORMAT

Figure 8. You can do contour plots of the results from a computation.

UPFRONT

5NDER THE .-2 SPECTRUM ENTRY OF THE 4OOLS MENU YOU CAN SELECT TO LOAD EITHER A PREVIOUSLY CALCULATED RESULTS FILE OR THE .-2 3PIN 3PIN Splitting Simulation.

Here you can set several options, such as the lineshape and the scaling.

)F YOU RIGHT CLICK THE PLOT WINDOW YOU HAVE THE SAME OPTIONS AS IN THE other plot windows. You also can add more data sets, change the plot details or the overall color theme.

7ITH 'ABEDIT YOU CAN USE QUITE A FEW OF THE AVAILABLE CHEMISTRY PACKAGES FROM A UNIFIED USER INTERFACE 7HEN DOING MORE COMPLICATED research, or doing discovery work, being able to use multiple

PACKAGES DEFINITELY WILL MAKE EVERYTHING EASIER TO HANDLE 9OU ALSO CAN EXPAND THE OPTIONS WITHIN 'ABEDIT BY ADDING YOUR OWN FUNCTIONAL units or altering the molecular mechanics parameters to be used IN YOUR WORK (OPEFULLY 'ABEDIT CAN HELP MOVE YOUR RESEARCH INTO new areas.—Joey Bernard

Figure 9. You can do NMR spectrum simulations for your molecule of choice.

T T h h e e F F i i f f t t e e e e n n t t h h A A n n n n u u a a l l

S S o o u u t t h h e e r r n n C C a a l l i i f f o o r r n n i i a a L L i i n n u u x x E E x x p p o o

M M a a r r c c h h 2 2 - - 5 5 , , 2 2 0 0 1 1 7 7

P P a a s s a a d d e e n n a a C C o o n n v v e e n n t t i i o o n n C C e e n n t t e e r r P P a a s s a a d d e e n n a a , , C C A A

h h tt t t p: p :/ // /w ww w w. w .s so o c c al a l l l in i nu ux xe ex xp po o .o . o rg r g

Us U se e P Pr ro o mo m o C Co o de d e L L J J 15 1 5X X f fo or r a a 3 30 0% %

Android Candy:

Exploding Kittens!

) DONT VERY OFTEN PLAY GAMES ) KNOW THAT SEEMS ODD BECAUSE ) DO OFTEN WRITE ABOUT GAMING (ONESTLY THOUGH ) VERY RARELY ACTUALLY TAKE the time to play video games. Recently, however, there has been an

EDITORS’

CHOICE

★

™

EDITORS’ CHOICE

NEXT

Dave Taylor’s Work the Shell VPREVIOUS

UpFront

V

exception to that rule.

/NE OF MY FAVORITE ONLINE COMICS IS The Oatmeal 4HE CREATOR COLLABORATED WITH ANOTHER GUY AND CAME UP WITH AN INCREDIBLY FUN card game called Exploding Kittens. I love the game. My teenage daughters love the game. Heck, I’ve even purchased another box so MY COLLEGE AGED DAUGHTER COULD PLAY IT WITH HER ROOMMATES .OT ONLY IS THE CARD GAME FUN BUT THEY ALSO MADE A VIDEO GAME VERSION THAT WAS ON I/3 ONLY FOR A LONG TIME

Well, no more. Now you can get Exploding Kittens FOR  AT THE Google Play store. It supports playing with random weirdos on the INTERNET ) COULD BE ONE OF THOSE WEIRDOS AND PLAYING WITH A GROUP OF FRIENDS ) WONT DESCRIBE THE GAME ITSELF OTHER THAN TO SAY ITS SILLY HILARIOUS AND FUN 0LUS THERE ARE LOTS OF AWESOME GRAPHICS DRAWN BY The Oatmeal )N FACT THIS GAME IS SO MUCH FUN FOR SUCH A REASONABLE PRICE )M GIVING IT THIS MONTHS %DITORS #HOICE AWARD EVEN THOUGH ITS NOT OPEN SOURCE "ECAUSE TRULY ITS AN INCREDIBLY FUN GAME THAT YOU CAN PLAY IN FIVE MINUTES WHILE YOURE DOING WHATEVER YOU MIGHT BE DOING THAT WOULD FACILITATE FIVE MINUTES OF QUIET TIME ON YOUR CELL PHONE

3EARCH FOR Exploding Kittens at the Google Play store, and start PLAYING NOW
—Shawn Powers

EDITORS' CHOICE

RETURN TO CONTENTS

DAVE TAYLOR

Dave Taylor has been hacking shell scripts on UNIX and Linux systems for a really long time. He’s the author of Learning Unix for Mac OS X and Wicked Cool Shell Scripts. You can find him on Twitter as @DaveTaylor, or reach him through his tech Q&A site: http://

www.AskDaveTaylor.com.

WORK THE SHELL

Scissors, Paper or Rock?

I’ve spent a lot of time in this column looking at the sky—whether it was a Martian lander or a phase of the moon program, lots of math, lots of interesting code. Now let’s land back on Earth and tackle a simple, straightforward challenge that has nothing to do with asteroids, gravitational anomalies or wormholes—well, hopefully not.

IN THIS ARTICLE, I’m going to tackle a children’s game that’s extraordinarily complicated, with many variations, and the programming task is going to BE QUITE TRICKY *UST KIDDING
2OCK 0APER 3CISSORS

OR 203 AS ITS KNOWN IS PRETTY DARN EASY TO

simulate because there aren’t really many variants or possible outcomes.

)F YOUVE NEVER PLAYED IT BEFORE ITS A ONE VS ONE GAME WHERE EACH PERSON SECRETLY CHOOSES ONE OF THREE POSSIBLE OPTIONS ROCK PAPER OR YOU GUESSED IT SCISSORS  4HE PLAYERS REVEAL THEIR CHOICES

NEXT

Kyle Rankin’s Hack and / VPREVIOUS

Editors’ Choice

V

WORK THE SHELL

simultaneously, and then there are rules about what beats what. For EXAMPLE SCISSORS BEATS PAPER BECAUSE hSCISSORS CUT PAPERv AND ROCK BEATS SCISSORS BECAUSE hROCK BEATS SCISSORSv )F BOTH PLAYERS PICK THE same option, it’s a tie and the game proceeds.

!LTHOUGH YOU CAN PLAY IT AS A ONE OFF ITS ALSO GENERALLY PLAYED AS A BEST OF THREE TO EVEN THINGS OUT SLIGHTLY ALTHOUGH IF EVERYTHINGS COMPLETELY RANDOM YOULL WIN  OF THE TIME &OR ANY GIVEN CHOICE THERES A  CHANCE THAT YOULL HAVE A TIE WHERE BOTH PLAYERS PICK THE SAME THING A  CHANCE THAT YOULL WIN AND A  CHANCE that you’ll lose.

The World Rock Paper Scissors Society

%XCEPT IN THE REAL GAME IT TURNS OUT THAT THERES PSYCHOLOGY INVOLVED TOO )N FACT ACCORDING TO THE 7ORLD 2OCK 0APER 3CISSORS 3OCIETY

http://worldrps.com  ROCK IS CHOSEN  PAPER  OF THE TIME AND SCISSORS ONLY  OF THE TIME 'OT IT

&OR THE FIRST VERSION OF THE PROGRAM HOWEVER LETS STICK WITH A

COMPLETELY RANDOM CHOICE 4HE EASY WAY TO CHOOSE A RANDOM NUMBER BETWEEN  AND  IN A ,INUX SHELL SCRIPT IS TO USE THE VARIABLE $RANDOM like this:

compchoice=$((  ($RANDOM  %  3)  +  1  ))

4HE  IS A MODULUS FUNCTION AND CAUSES THE RANDOM INTEGER TO BE DIVIDED BY  RESULTING IN A  VALUE !DD ONE AND YOUVE GOT THE

 VALUE %ASY ENOUGH

7ITH A SIMPLE SHELL ARRAY YOU CAN ADD THE NAME OF THE CHOICE

REMEMBER ARRAYS START AT INDEX  

declare  -­a  RPS;;  RPS=(nothing  rock  paper  scissors) 4HEN THE CHOICE NAME IS SPECIFIED SIMPLY AS

choicename=${RPS[$compchoice]}

4HOSE THREE LINES ARE GOOD ENOUGH FOR A TINY SCRIPT WHERE THE COMPUTER

WORK THE SHELL

can choose randomly between rock, paper and scissors:

declare  -­a  RPS;;  RPS=(nothing  rock  paper  scissors)   compchoice=$((  ($RANDOM  %  3)  +  1  ))  

echo  "The  computer  chose  ${RPS[$compchoice]}"  

exit  0

%ASY BUT NOT VERY GLAMOROUS

$  sh  rps.sh  

The  computer  chose  rock  

$

)TS CONSIDERABLY MORE FUN TO HAVE THE COMPUTER PROMPT USERS FOR THEIR SELECTION THEN hCHOOSEv ITS OWN AND DECIDE WHO WON

Making It into a Game

Interactivity is easily added by prompting users to choose whether they WANT ROCK PAPER OR SCISSORS USING A NUMERIC VALUE %VEN BETTER YOU CAN prompt them using the same numeric values you’re using internally:

echo  -­n  "Please  choose  (1  =  rock  /  2  =  paper  /  3  =  scissors):  "  

read  choice

)TS NOT A PARTICULARLY ONEROUS TASK TO ADD INTERACTIVITY EH

Now you need to compare answers and generate a result message.

4HIS IS BEST DONE IN A FUNCTION EITHER STANDALONE OR BY INCLUDING AN OUTPUT STRING AND TRACKING WINLOSS )LL GO FOR OVERKILL OF COURSE  SO HERES MY FUNCTION

results()  {  

     #  output  results  of  the  game,  increment  wins  if  appropriate        echo  ""  

     if  [  $choice  =  $compchoice  ]  ;;  then  

       echo  "You  both  chose  $choicename.  TIED!"  

 

WORK THE SHELL

     #  rock  beats  scissors.  paper  beats  rock.  scissors  beat  paper.  

     #    OR:  1  beats  3,  2  beats  1,  and  3  beats  2.  

 

     elif  [  $choice  -­eq  1  -­a  $compchoice  -­eq  3  ]  ;;  then  

     echo  "Your  rock  beats  the  computer's  scissors!  Huzzah!!"  

     wins=$((  $wins  +  1  ))  

     elif  [  $choice  -­eq  2  -­a  $compchoice  -­eq  1  ]  ;;  then        echo  "Your  paper  beats  the  computer's  rock!  Hurray!"  

     wins=$((  $wins  +  1  ))  

     elif  [  $choice  -­eq  3  -­a  $compchoice  -­eq  2  ]  ;;  then  

     echo  -­n  "Your  scissors  cut  -­  and  beat  -­  the  computer's  "  

     echo  "paper!  YAY!"  

     wins=$((  $wins  +  1  ))  

     elif  [  $choice  -­eq  3  -­a  $compchoice  -­eq  1  ]  ;;  then        echo  "The  computer's  rock  beats  your  scissors!  Boo."  

     elif  [  $choice  -­eq  1  -­a  $compchoice  -­eq  2  ]  ;;  then        echo  "The  computer's  paper  beats  your  rock!  Ptoi!"  

     elif  [  $choice  -­eq  2  -­a  $compchoice  -­eq  3  ]  ;;  then        echo  -­n  "The  computer's  scissors  cut  -­  and  beat  -­  "  

     echo  "your  paper!  Bummer."  

     else  

     echo  "Huh?  choice=$choice  and  compchoice=$computer"  

     fi   }

)TS STRAIGHTFORWARD JUST A LOT OF TYPING "UT REALLY THATS  OF THE PROGRAM !LL YOU NEED IS A LOOPING MECHANISM SO THAT YOURE hSTUCKv IN THE PROGRAM UNTIL YOU GET SICK OF THE GAMEˆ) MEAN READY TO WRAP things up.

.OTICE THAT THE ABOVE CODE TRACKS WINS BUT NOT TOTAL GAMES PLAYED

THATLL HAVE TO BE DONE IN THE MAIN CODE WHICH OF COURSE IS PRETTY STRAIGHTFORWARD BECAUSE OF HOW MUCH OF THE CODE IS PUSHED INTO THE results() FUNCTION

echo  "Rock,  paper,  scissors..."  

echo  "(quit  by  entering  'q'  to  see  your  results)"  

WORK THE SHELL

while  [  true  ]  ;;  do      echo  ""  

   echo  -­n  "Choose  (1  =  rock  /  2  =  paper  /  3  =  scissors):  "  

   read  choice  

   if  [  "$choice"  =  "q"  -­o  "$choice"  =  "quit"  -­o  -­z  "$choice"  ]      then  

       echo  ""  

       echo  "Done.  You  played  $games  games,  and  won  $wins  of  'em."  

       exit  0      fi  

   compchoice=$((  ($RANDOM  %  3)  +  1  ))      choicename=${RPS[$compchoice]}  

   games=$((  $games  +  1  ))      results  

done

! QUICK RUN REVEALS THAT SCISSORS ISNT A BAD STRATEGY WHEN THE GAME IS picking completely randomly:

$  sh  rps.sh  

Choose  (1  =  rock  /  2  =  paper  /  3  =  scissors):  3  

Your  scissors  cut  -­  and  beat  -­  the  computer's  paper!  YAY!  

$

7HEN ) TRIED IT ) HAD A SURPRISINGLY LONGER TERM RESULT AN ALL SCISSORS STRATEGY PRODUCED A  WIN RATE SIX GAMES OUT OF   3TATISTICALLY THATS UNLIKELY IF THE COMPUTER REALLY IS PICKING RANDOMLY BUT SOMETIMES random is not so random.

Let’s look at choosing paper:

$  sh  rps.sh  

Choose  (1  =  rock  /  2  =  paper  /  3  =  scissors):  2  

The  computer's  scissors  cut  -­  and  beat  -­  your  paper!  Bummer.  

$

)N FACT PLAYING ALL PAPER WON ONLY FOUR OF  GAMES ON A TRIAL AND ROCK

WORK THE SHELL

THE MOST POPULAR CHOICE 4HAT PRODUCES A WIN RATE OF THREE OUT OF ˆ

WORSE THAN PAPER

Matching Probabilities

4HE BIGGEST CHANGE YOU COULD MAKE TO THIS PROGRAM TO MATCH THE hREALv CHOICE STATISTICS IS TO STOP PICKING RANDOMLY AND INSTEAD REFLECT the percentages that the Rock Paper Scissors Society publishes: rock is CHOSEN  PAPER  AND SCISSORS ONLY  OF THE TIME

4HE EASIEST WAY TO MODEL THAT IS TO CHOOSE A RANDOM NUMBER

BETWEEN n AND THEN SAY THAT n IS ROCK n IS PAPER AND n IS SCISSORS )NSTEAD OF A SINGLE LINE WHERE THE NUMBER IS BEING CHOSEN A FUNCTION WOULD BE WELL WRITTEN AND ITS PRETTY darn easy.

4HE OTHER AREA YOU CAN EXPAND THIS IS TO ADD A FEW MORE POSSIBILITIES AND ) BET MOST EVERYONE READING THIS KNOWS HOW TO ADD hLIZARDv AND h3POCKv TO THE MIX .OT SURE (ERES HOW A FIVE OBJECT 203 GAME works: http://www.samkass.com/theories/RPSSL.html.

3O THERE YOU HAVE IT 3CIENTIFIC .OT REALLY "UT UH ROCK PAPER SCISSORSˆCOME ON
^Q

Send comments or feedback via http://www.linuxjournal.com/contact or to [email protected].

RETURN TO CONTENTS

Sysadmin 101:

Alerting

Learn from my mistakes in this article covering on-call alert best practices.

THIS IS THE FIRST IN A SERIES OF ARTICLES ON SYSTEM ADMINISTRATOR FUNDAMENTALS.

4HESE DAYS $EV/PS HAS MADE EVEN THE JOB TITLE

hSYSTEM ADMINISTRATORv SEEM A BIT ARCHAIC MUCH LIKE THE hSYSTEMS ANALYSTv TITLE IT REPLACED 4HESE $EV/PS POSITIONS ARE RATHER DIFFERENT FROM TYPICAL SYSADMIN JOBS in the past in that they have a much larger emphasis on SOFTWARE DEVELOPMENT FAR BEYOND BASIC SHELL SCRIPTING

!S A RESULT THEY OFTEN ARE FILLED WITH PEOPLE WITH

SOFTWARE DEVELOPMENT BACKGROUNDS WITHOUT MUCH PRIOR sysadmin experience. In the past, sysadmins would enter the role at a junior level and be mentored by a senior sysadmin on the team, but in many cases currently, COMPANIES GO QUITE A WHILE WITH CLOUD OUTSOURCING BEFORE THEIR FIRST $EV/PS HIRE !S A RESULT $EV/PS

engineers might be thrust into the role at a junior level

KYLE RANKIN

Kyle Rankin is a Sr.

Systems Administrator in the San Francisco Bay Area and the author of a number of books, including The Official Ubuntu Server Book, Knoppix Hacks and Ubuntu Hacks. He is currently the president of the North Bay Linux Users’ Group.

HACK AND /

NEXT

Shawn Powers’

The Open-Source Classroom

VPREVIOUS Dave Taylor’s Work the Shell

V

HACK AND /

WITH NO MENTOR AROUND APART FROM SEARCH ENGINES AND 3TACK /VERFLOW

POSTS )N THIS SERIES OF ARTICLES )M GOING TO EXPOUND ON SOME OF THE LESSONS I’ve learned through the years that might be obvious to longtime sysadmins but may be news to someone just coming into this position.

)N THIS FIRST ARTICLE ) COVER ON CALL ALERTING ,IKE WITH ANY JOB TITLE THE RESPONSIBILITIES GIVEN TO SYSADMINS $EV/PS AND 3ITE 2ELIABILITY %NGINEERS MAY DIFFER AND IN SOME CASES THEY MAY NOT INVOLVE ANY KIND OF X

ON CALL DUTIES IF YOURE LUCKY &OR EVERYONE ELSE THOUGH THERE ARE MANY WAYS TO ORGANIZE ON CALL ALERTING AND THERE ALSO ARE MANY WAYS TO SHOOT YOURSELF IN THE FOOT

4HE MAIN ENEMIES OF ON CALL ALERTING ARE FALSE POSITIVES WITH THE MAIN RISKS BEING IGNORING ALERTS OR BURNOUT FOR MEMBERS OF YOUR TEAM 4HIS ARTICLE talks about some best practices you can apply to your alerting policies that HOPEFULLY WILL REDUCE BURNOUT AND MAKE SURE ALERTS ARENT IGNORED

Alert Thresholds

! COMMON PITFALL SYSADMINS RUN INTO WHEN SETTING UP MONITORING

SYSTEMS IS TO ALERT ON TOO MANY THINGS 4HESE DAYS ITS SIMPLE TO MONITOR JUST ABOUT ANY ASPECT OF A SERVERS HEALTH SO ITS TEMPTING TO OVERLOAD YOUR MONITORING SYSTEM WITH ALL KINDS OF SYSTEM CHECKS /NE OF THE MAIN ONGOING MAINTENANCE TASKS FOR ANY MONITORING SYSTEM IS SETTING APPROPRIATE ALERT THRESHOLDS TO REDUCE FALSE POSITIVES 4HIS MEANS THE more checks you have in place, the higher the maintenance burden. As a RESULT ) HAVE A FEW DIFFERENT RULES ) APPLY TO MY MONITORING CHECKS WHEN DETERMINING THRESHOLDS FOR NOTIFICATIONS

Critical alerts must be something I want to be woken up about at 3am. ! COMMON CAUSE OF SYSADMIN BURNOUT IS BEING WOKEN UP WITH ALERTS FOR SYSTEMS THAT DONT MATTER )F YOU DONT HAVE A X INTERNATIONAL DEVELOPMENT TEAM YOU PROBABLY DONT CARE IF THE BUILD SERVER HAS A

PROBLEM AT AM OR EVEN IF YOU DO YOU PROBABLY ARE GOING TO WAIT UNTIL THE MORNING TO FIX IT "Y RESTRICTING CRITICAL ALERTS TO JUST THOSE SYSTEMS THAT MUST BE ONLINE X YOU HELP REDUCE FALSE POSITIVES AND MAKE SURE THAT REAL PROBLEMS ARE ADDRESSED QUICKLY

Critical alerts must be actionable. Some organizations send alerts WHEN JUST ABOUT ANYTHING HAPPENS ON A SYSTEM )F )M BEING WOKEN UP AT

AM ) WANT TO HAVE A SPECIFIC ACTION PLAN ASSOCIATED WITH THAT ALERT SO )

HACK AND /

CAN FIX IT !GAIN TOO MANY FALSE POSITIVES WILL BURN OUT A SYSADMIN THATS ON CALL AND NOTHING IS MORE FRUSTRATING THAN GETTING WOKEN UP WITH AN ALERT THAT YOU CANT DO ANYTHING ABOUT %VERY CRITICAL ALERT SHOULD HAVE AN OBVIOUS ACTION PLAN THE SYSADMIN CAN FOLLOW TO FIX IT

Warning alerts tell me about problems that will be critical if I don’t fix them. 4HERE ARE MANY PROBLEMS ON A SYSTEM THAT ) MAY WANT TO know about and may want to investigate, but they aren’t worth getting OUT OF BED AT AM 7ARNING ALERTS DONT TRIGGER A PAGER BUT THEY STILL SEND ME A QUIETER NOTIFICATION &OR INSTANCE IF LOAD USED DISK SPACE OR 2!- GROWS TO A CERTAIN POINT WHERE THE SYSTEM IS STILL HEALTHY BUT IF LEFT unchecked may not be, I get a warning alert so I can investigate when ) GET A CHANCE /N THE OTHER HAND IF ) GOT ONLY A WARNING ALERT BUT THE system was no longer responding, that’s an indication I may need to change my alert thresholds.

Repeat warning alerts periodically. ) THINK OF WARNING ALERTS LIKE THIS THING NAGGING AT YOU TO LOOK AT IT AND FIX IT DURING THE WORK DAY )F YOU SEND WARNING ALERTS TOO FREQUENTLY THEY JUST SPAM YOUR INBOX AND ARE IGNORED SO )VE FOUND THAT SPACING THEM OUT TO ALERT EVERY HOUR OR SO IS ENOUGH TO REMIND ME OF THE PROBLEM BUT NOT SO FREQUENT THAT ) IGNORE IT COMPLETELY

Everything else is monitored, but doesn’t send an alert. 4HERE are many things in my monitoring system that help provide overall context when I’m investigating a problem, but by themselves, they aren’t actionable and aren’t anything I want to get alerts about. In OTHER CASES ) WANT TO COLLECT METRICS FROM MY SYSTEMS TO BUILD TRENDING GRAPHS LATER ) DISABLE ALERTS ALTOGETHER ON THOSE KINDS OF CHECKS 4HEY still show up in my monitoring system and provide a good audit trail when I’m investigating a problem, but they don’t page me with useless NOTIFICATIONS

Kyle’s rule. /NE FINAL NOTE ABOUT ALERT THRESHOLDS )VE DEVELOPED A PRACTICE IN MY YEARS AS A SYSADMIN THAT )VE FOUND IS IMPORTANT ENOUGH AS a way to reduce burnout that I take it with me to every team I’m on. My rule is this:

)F SYSADMINS WERE KEPT UP DURING THE NIGHT BECAUSE OF FALSE ALARMS THEY CAN CLEAR THEIR PROJECTS FOR THE NEXT DAY AND SPEND TIME TUNING ALERT thresholds so it doesn’t happen again.

HACK AND /

4HERE IS NOTHING WORSE THAN BEING KEPT UP ALL NIGHT BECAUSE OF FALSE positive alerts and knowing that the next night will be the same and THAT THERES NOTHING YOU CAN DO ABOUT IT )F THAT KIND OF THING CONTINUES it inevitably will lead either to burnout or to sysadmins silencing their PAGERS 3ETTING ASIDE TIME FOR SYSADMINS TO FIX FALSE ALARMS HELPS BECAUSE they get a chance to improve their night’s sleep the next night. As a team lead or manager, sometimes this has meant that I’ve taken on a SYSADMINS TICKETS FOR THEM DURING THE DAY SO THEY CAN FIX ALERTS

Paging

3ENDING AN ALERT OFTEN IS REFERRED TO AS PAGING OR BEING PAGED BECAUSE IN THE PAST SYSADMINS LIKE DOCTORS CARRIED PAGERS ON THEM 4HEIR

monitoring systems were set to send a basic numerical alert to the pager when there was a problem, so that sysadmins could be alerted even when they weren’t at a computer or when they were asleep. Although we still REFER TO IT AS PAGING AND SOME OLDER SCHOOL TEAMS STILL PASS AROUND AN ACTUAL PAGER THESE DAYS NOTIFICATIONS MORE OFTEN ARE HANDLED BY ALERTS TO mobile phones.

4HE FIRST QUESTION YOU NEED TO ANSWER WHEN YOU SET UP ALERTING IS WHAT METHOD YOU WILL USE FOR NOTIFICATIONS 7HEN YOU ARE DECIDING HOW TO SET UP PAGER NOTIFICATIONS LOOK FOR A FEW SPECIFIC QUALITIES

Something that will alert you wherever you are geographically.

! NUMBER OF COOL OFFICE PROJECTS ON THE WEB EXIST WHERE A BROKEN SOFTWARE BUILD TRIGGERS A BIG RED FLASHING LIGHT IN THE OFFICE 4HAT KIND OF NOTIFICATION IS FINE FOR OFFICE HOUR ALERTS FOR NON CRITICAL SYSTEMS BUT IT ISNT APPROPRIATE AS A PAGER NOTIFICATION EVEN DURING THE DAY BECAUSE A SYSADMIN WHO IS IN A MEETING ROOM OR AT LUNCH WOULD NOT BE NOTIFIED

4HESE DAYS THIS GENERALLY MEANS SOME KIND OF NOTIFICATION NEEDS TO BE sent to your phone.

An alert should stand out from other notifications. False alarms can be a big problem with paging systems, as sysadmins naturally will START IGNORING ALERTS ,IKEWISE IF YOU USE THE SAME RINGTONE FOR ALERTS THAT YOU USE FOR ANY OTHER EMAIL YOUR BRAIN WILL START TO TUNE ALERTS OUT )F YOU USE EMAIL FOR ALERTS USE FILTERING RULES SO THAT ON CALL ALERTS GENERATE A COMPLETELY DIFFERENT AND LOUDER RINGTONE FROM REGULAR EMAILS AND VIBRATE THE PHONE AS WELL SO YOU CAN BE NOTIFIED EVEN IF YOU SILENCE YOUR PHONE

HACK AND /

or are in a loud room. In the past, when BlackBerries were popular, you COULD SET RULES SUCH THAT CERTAIN EMAILS GENERATED A h,EVEL /NEv ALERT THAT WAS DIFFERENT FROM REGULAR EMAIL NOTIFICATIONS

4HE "LACK"ERRY DAYS ARE GONE NOW AND CURRENTLY MANY ORGANIZATIONS

IN PARTICULAR STARTUPS USE 'OOGLE !PPS FOR THEIR CORPORATE EMAIL

4HE 'MAIL !NDROID APPLICATION LETS YOU SET PER FOLDER CALLED LABELS

NOTIFICATION RULES SO YOU CAN CREATE A FILTER THAT MOVES ALL ON CALL ALERTS TO A PARTICULAR FOLDER AND THEN SET THAT FOLDER SO THAT IT GENERATES A UNIQUE ALERT VIBRATES AND DOES SO FOR EVERY NEW EMAIL TO THAT FOLDER )F YOU DONT HAVE THAT OPTION MOST EMAIL SOFTWARE THAT SUPPORTS MULTIPLE ACCOUNTS WILL LET YOU SET DIFFERENT NOTIFICATIONS FOR EACH ACCOUNT SO YOU MAY NEED TO RESORT TO A SEPARATE EMAIL ACCOUNT JUST FOR ALERTS

Something that will wake you up all hours of the night. Some SYSADMINS ARE DEEP SLEEPERS AND WHATEVER NOTIFICATION SYSTEM YOU CHOOSE NEEDS TO BE SOMETHING THAT WILL WAKE THEM UP IN THE MIDDLE OF THE NIGHT !FTER ALL SERVERS ALWAYS SEEM TO MISBEHAVE AT AROUND AM

0ICK A RINGTONE THAT IS LOUD POSSIBLY OBNOXIOUS IF NECESSARY AND ALSO MAKE SURE TO ENABLE PHONE VIBRATIONS !LSO CONFIGURE YOUR ALERT SYSTEM TO RE SEND NOTIFICATIONS IF AN ALERT ISNT ACKNOWLEDGED WITHIN A COUPLE MINUTES 3OMETIMES THE FIRST ALERT ISNT ENOUGH TO WAKE PEOPLE UP COMPLETELY BUT IT MIGHT MOVE THEM FROM DEEP SLEEP TO A LIGHTER SLEEP SO THE FOLLOW UP ALERT WILL WAKE THEM UP

7HILE #HAT/PS USING CHAT AS A METHOD OF GETTING NOTIFICATIONS AND PERFORMING ADMINISTRATION TASKS MIGHT BE OKAY FOR GENERAL NON CRITICAL DAYTIME NOTIFICATIONS THEY ARE NOT APPROPRIATE FOR PAGER ALERTS %VEN IF YOU HAVE AN APPLICATION ON YOUR PHONE SET TO NOTIFY YOU ABOUT UNREAD MESSAGES IN CHAT MANY CHAT APPLICATIONS DEFAULT TO A hQUIET TIMEv IN THE MIDDLE OF THE NIGHT )F YOU DISABLE THAT YOU RISK BEING PAGED IN THE MIDDLE OF THE NIGHT JUST BECAUSE SOMEONE SENT YOU A MESSAGE !LSO MANY THIRD PARTY #HAT/PS SYSTEMS ARENT NECESSARILY KNOWN FOR THEIR

After all, servers always seem to misbehave

at around 3am.

HACK AND /

MISSION CRITICAL RELIABILITY AND HAVE HAD OUTAGES THAT HAVE SPANNED MANY hours. You don’t want your critical alerts to rely on an unreliable system.

Something that is fast and reliable. 9OUR NOTIFICATION SYSTEM NEEDS TO BE RELIABLE AND ABLE TO ALERT YOU QUICKLY AT ALL TIMES 4O ME THIS MEANS ALERTING IS DONE IN HOUSE BUT MANY ORGANIZATIONS OPT FOR THIRD PARTIES TO RECEIVE AND ESCALATE THEIR NOTIFICATIONS %VERY ADDITIONAL LAYER YOU CAN ADD TO YOUR ALERTING IS ANOTHER LAYER OF LATENCY AND ANOTHER PLACE WHERE A NOTIFICATION MAY BE DROPPED *UST MAKE SURE WHATEVER METHOD YOU CHOOSE IS RELIABLE AND THAT YOU HAVE SOME WAY OF DISCOVERING WHEN YOUR MONITORING SYSTEM ITSELF IS OFFLINE

In the next section, I cover how to set up escalations—meaning, HOW YOU ALERT OTHER MEMBERS OF THE TEAM IF THE PERSON ON CALL ISNT RESPONDING 0ART OF SETTING UP ESCALATIONS IS PICKING A SECONDARY BACKUP METHOD OF NOTIFICATION THAT RELIES ON A DIFFERENT INFRASTRUCTURE FROM YOUR PRIMARY ONE 3O IF YOU USE YOUR CORPORATE %XCHANGE SERVER FOR PRIMARY NOTIFICATIONS YOU MIGHT SELECT A PERSONAL 'MAIL ACCOUNT AS A SECONDARY

)F YOU HAVE A 'OOGLE !PPS ACCOUNT AS YOUR PRIMARY NOTIFICATION YOU MAY pick SMS as your secondary alert.

%MAIL SERVERS HAVE OUTAGES LIKE ANYTHING ELSE AND THE GOAL HERE IS TO MAKE SURE THAT EVEN IF YOUR PRIMARY METHOD OF NOTIFICATIONS HAS AN OUTAGE YOU HAVE SOME ALTERNATE WAY OF FINDING OUT ABOUT IT )VE HAD A NUMBER OF OCCASIONS WHERE MY 3-3 SECONDARY ALERT CAME IN BEFORE MY primary just due to latency with email syncing to my phone.

Create some means of alerting the whole team. In addition to having individual alerting rules that will page someone who is on call, it’s USEFUL TO HAVE SOME WAY OF PAGING AN ENTIRE TEAM IN THE EVENT OF AN hALL HANDS ON DECKv CRISIS 4HIS MAY BE A PARTICULAR EMAIL ALIAS OR A PARTICULAR key word in an email subject. However you set it up, it’s important that EVERYONE KNOWS THAT THIS IS A hPULL IN CASE OF FIREv NOTIFICATION AND SHOULDNT BE ABUSED WITH NON CRITICAL MESSAGES

Alert Escalations

/NCE YOU HAVE ALERTS SET UP THE NEXT STEP IS TO CONFIGURE ALERT

ESCALATIONS %VEN THE BEST DESIGNED NOTIFICATION SYSTEM ALERTING THE MOST WELL INTENTIONED SYSADMIN WILL FAIL FROM TIME TO TIME EITHER BECAUSE A SYSADMINS PHONE CRASHED HAD NO CELL SIGNAL OR FOR WHATEVER REASON THE

HACK AND /

sysadmin didn’t notice the alert. When that happens, you want to make SURE THAT OTHERS ON THE TEAM AND THE ON CALL PERSONS SECOND NOTIFICATION is alerted so someone can address the alert.

!LERT ESCALATIONS ARE ONE OF THOSE AREAS THAT SOME MONITORING SYSTEMS DO BETTER THAN OTHERS !LTHOUGH THE CONFIGURATION CAN BE CHALLENGING COMPARED TO OTHER SYSTEMS )VE FOUND .AGIOS TO PROVIDE A RICH SET OF ESCALATION SCHEDULES /THER ORGANIZATIONS MAY OPT TO USE A THIRD PARTY NOTIFICATION SYSTEM SPECIFICALLY BECAUSE THEIR CHOSEN MONITORING SOLUTION DOESNT HAVE THE ABILITY TO DEFINE STRONG ESCALATION PATHS ! SIMPLE

ESCALATION SYSTEM MIGHT LOOK LIKE THE FOLLOWING

Q )NITIAL ALERT GOES TO THE ON CALL SYSADMIN AND REPEATS EVERY FIVE MINUTES

Q )F THE ON CALL SYSADMIN DOESNT ACKNOWLEDGE OR FIX THE ALERT WITHIN

 MINUTES IT ESCALATES TO THE SECONDARY ALERT AND ALSO TO THE REST OF the team.

Q 4HESE ALERTS REPEAT EVERY FIVE MINUTES UNTIL THEY ARE ACKNOWLEDGED OR FIXED

4HE IDEA HERE IS TO GIVE THE ON CALL SYSADMIN TIME TO ADDRESS THE ALERT SO YOU ARENT WAKING EVERYONE UP AT AM YET ALSO PROVIDE THE REST OF THE TEAM WITH A WAY TO FIND OUT ABOUT THE ALERT IF THE FIRST SYSADMIN CANT FIX it in time or is unavailable. Depending on your particular SLAs, you may want to shorten or lengthen these time periods between escalations or MAKE THEM MORE SOPHISTICATED WITH THE ADDITION OF AN ON CALL BACKUP WHO IS ALERTED BEFORE THE FULL TEAM )N GENERAL ORGANIZE YOUR ESCALATIONS SO THEY STRIKE THE RIGHT BALANCE BETWEEN GIVING THE ON CALL PERSON A CHANCE TO RESPOND BEFORE PAGING THE ENTIRE TEAM YET NOT LETTING TOO MUCH TIME PASS IN THE EVENT OF AN OUTAGE IN CASE THE PERSON ON CALL CANT RESPOND

Alert escalations are one of those areas that

some monitoring systems do better than others.