!
Avoiding Pain Running MySQL in the
Cloud
©Continuent 2013 2
•
DBA Oracle/Mainframes/MySQL (25 Years)•
Deployment Engineer @ Continuent•
1 or 2 Customer Deployments/Week•
On Premise or Cloud deployments•
Developer for Cloud Operations and Deployments @ ContinuentQuick Continuent Facts
•
Largest Tungsten installation processes over 700 million transactions daily on 225terabytes of data
•
Tungsten Replicator was application of the year at the 2011 MySQL User Conference•
Wide variety of topologies including MySQL, Oracle, Vertica, and MongoDB are inproduction now
•
MySQL to Hadoop deployments are now in progress with multiple customers©Continuent 2013
Aims
•
Make you thinkAims
•
Make you think©Continuent 2013
Aims
•
Make you think•
Make you question•
Make you discussAims
•
Make you think•
Make you question•
Make you discuss©Continuent 2013
Why Cloud
•
Everyone wants servers now.•
Waiting months to provision a database environment is no longer acceptable.•
The business would have failed before new physical servers could be provisioned.•
Initial capital outlay is low - long term cost can be higher.•
Less internal skills are required to maintain and deploy servers.Before Cloud Came Along
(a year or so ago)
©Continuent 2013 10
On-premise/Colo deployments
•
Raise a project•
Get Approval•
Order hardware•
Rack and cable hardware•
Connect to network•
Install Operating System©Continuent 2013
Cloud deployments
•
Enter credit card number•
call APInova boot --image centos6-x86_64 --flavor m1.large db1 nova boot --image centos6-x86_64 --flavor m1.large db2 nova boot --image centos6-x86_64 --flavor m1.large db3
!
ec2-run-instances ami-xxxxx -k ${EC2_KEYPAIR} -t m1.large ec2-run-instances ami-xxxxx -k ${EC2_KEYPAIR} -t m1.large ec2-run-instances ami-xxxxx -k ${EC2_KEYPAIR} -t m1.large
!
©Continuent 2013 14
Cloud Providers
•
Public Clouds•
Amazon AWS•
Google Cloud•
RackSpace•
HP Cloud•
Many others•
On Premise Clouds•
VMware•
Openstack©Continuent 2013
Where is a cloud server located ?
16 Ireland (3 AZ) Sao Paulo (2 AZ) Northern Virginia (5 AZ) Oregon (3 AZ) California (3 AZ) Singapore (2 AZ) Tokyo (3 AZ) Sydney (2 AZ)
AWS Availability Zones
Region
Availability Zone Availability Zone
Availability Zone
Region
©Continuent 2013 18
Why is working in the
Cloud Different?
Where is my server?
db1 db2 db3
©Continuent 2013
Where is my server?
20 db1 db2 EC2 Server A db3 EC2 Server BWhere is my server?
db3
EC2 Server B
©Continuent 2013
Where is your data?
Where is your data?
©Continuent 2013
Where is your data?
24
X
Where is your data?
X
X
X
©Continuent 2013
Networking in the Cloud
•
Can be unreliable•
Where it is?•
What it is?•
Security?•
IP’s can change©Continuent 2013
High Availability - On Premise
•
Ensure servers in different Racks•
Redundant Power•
Redundant Network•
Redundant Disk•
Backup Server in DR location•
Redundant network between Datacenters©Continuent 2013
Cloud Environments Fail
©Continuent 2013
Cloud Environments Fail
Cloud Deployment Options - Simple
Region
Availability Zone Availability Zone
US-EAST-1 1B 1C S3 Backups Master Slave
©Continuent 2013
Cloud Deployment Options - Better
34
Region
Availability Zone Availability Zone
Region Availability Zone US-EAST-1 US-WEST-1 1B 1C 1C S3 Backups S3 Backups Master Slave Slave
Cloud Deployment Options - Even Better
Region
Availability Zone Availability Zone
Region Availability Zone US-EAST-1 US-WEST-1 1B 1C 1C S3 Backups S3 Backups Master Slave Slave Slave Rackspace Cloud Backups
©Continuent 2013
What to think about
•
Need to consider locations•
Have more nodes than you would with on premise•
Nodes can just disappear or stop functioning“Spain cluster has had an issue this afternoon - the current master DB has dropped off the
network/out of AWS”
What to think about
•
Plan for and expect node failures•
Don’t assume that you will be able to quickly provision new nodes.•
Take plenty of backups (S3 costs only $100/ month to store 1TB)•
Expect a few ‘bad’ nodes•
poor cpu performance•
poor disk performance•
throw them away - shoot them©Continuent 2013
Running Databases in the Cloud
AWS Relational Database Service (RDS)
•
MySQL, Oracle, SQL Service•
Read Slaves•
Multi-AZ deployments©Continuent 2013
AWS Relational Database Service (RDS)
•
AWS Only•
Limited options for replicating data out•
Very limited options for replicating in•
Slow failover with AZ’s (10 mins +)Trove
•
MySQL Only•
Single Node•
No Replication©Continuent 2013
Google Cloud
•
Google only•
Synchronous and asynchronous replication•
Across DC’s?How we deploy clusters
(in the cloud)
©Continuent 2013
Deploying in the Cloud
•
Automate everything•
Puppet allows us to work across clouds•
No need to maintain AMI’s/Images just use a standard o/s images•
‘masterless’ puppet to ensure we don’t have a SPOF•
Our install tools work across multi-nodes in parallel•
Heavy use of vagrant with AWS plugin for demos and POC installsNeed to setup secure connections
•
OpenVPN•
Need multiple servers to avoid a SPOF•
Software•
We now support secure connections•
Between DB’s©Continuent 2013
Best Practices
•
RAID EBS Volumes (RAID1)•
Backups•
xtrabackup (backed up into S3)•
EBS Snapshot•
ec2-‐consistent-‐snapshot \ --mysql --freeze-filesystem /vol \ --region eu-west-1 \--description "$(hostanme) RAID snapshot $(date +'%Y-%m-%d %H:%M:%S')" \
vol-1f9a6446 vol-649a643d
•
Use VPC’s and fixed networksBest Practices
•
Monitoring•
Nagios / Opsview•
Zabbix•
Email Alerts•
New Relic©Continuent 2013
Lesson Learnt
•
EC2 Instances fail•
One of anything is never enough•
Don’t assume you can spin up more resources instantly•
Think multi-cloud, public/private•
Resources are disposable - throw away and rebuild if neededLesson Learnt
•
Monitor everything•
Automate as much as possible©Continuent 2013
