1
F
ILE
C
LOUD
H
IGH
A
VAILABILITY
ARCHITECTURE
VERSION 1.0
FileCloud HA Architecture ... 2
Load Balancers ... 3
FileCloud Component: App server node ... 3
FileCloud Component: Mongo DB Replica set ... 3
Instructions for HA deployment in Ubuntu 14.04 ... 5
Setting up Ha-Proxy: ... 6
Setting up FileCloud Appserver nodes: ... 8
MongoDB Replica set setup ... 9
Verification of service ... 10
2
FileCloud HA Architecture
FileCloud servers can be configured for HA environment to improve service reliability and reduce downtime in your IT environment.
This deployment uses the classic 3-tier approach with Load balancer handling the client traffic, Application server nodes to serve requests and redundant database servers to store application data.
Filecloud components
App server Nodes
Mongo DB Replica set
Load
Balancer(s)
File Servers
Firewal
l
DM Z Corporate NetworkLDAP/AD
Client
s
External Network3
Load Balancers
The Load balancer will route the traffic to the FileCloud Application nodes. Load balancers (LB) provide many advantages to serving requests from your FC servers because they allow you to better control how the traffic is handled in order to provide the best performance.. If one or more App server nodes fail, the load balancer will automatically reroute traffic to other App server nodes.
Typically there is no need to scale the number of load balancers because these servers can handle a very large amount of traffic. However, more than one load balancers can be used to provide additional reliability in the event of one of the load balancer failure.
In order to protect against load balancer hardware failure, multiple A records for the load balancer host name in the DNS service can be used.
The idea here is that different clients will get different ordered lists of IP addresses corresponding to your domain name. This has the effect of distributing requests across the group of IPs in a specific manner. If an IP address is does not respond in an appropriate amount of time, the client will time out on that request and move on to the next IP address until the list is exhausted or it finds a connection that's valid.
FileCloud Component: App server node
The FileCloud App server node consists of the Apache webserver as well as the FileCloud Application code to server the client requests. The FileCloud app server nodes do not contain any application specific data. The data is retrieved from the MondoDB replica sets. Because of this, the FileCloud Appserver nodes can be added or removed without disrupting the service.
FileCloud Component: Mongo DB Replica set
The MongoDB database replica sets provide high availability with automatic failover support. Failover allows a secondary member to become primary in the event of failure to the primary DB node. The minimum number of DB nodes needed for MongoDB is three. All app server nodes will connect to primary and in the event of primary node failure, a new primary is elected and all the app server nodes will switch to the new primary.
4
Primary
Secondary
Secondary
5
Instructions for HA deployment in Ubuntu 14.04
The following instructions apply for deploying FileCloud HA in Ubuntu 14.04 environment. This can easily be adapted to other Linux flavors as well. This example is showing HTTP and can be expanded easily to use HTTPS as well. Some of the instructions will have to be adapted to your specific environment. Load Balancer: The load balancer of choice is HaProxy (http://www.haproxy.org/). HaProxy is a high performance and battle tested load balancer and allows you to scale your FileCloud deployment quickly as well.
For this set of instructions, we will use the following names.
NOTE: Before starting the install, ensure the servers are already available and their IP addresses are known.
Component Type Name
Load balancer Ha-LB
App server node 1 Ha-WS1
App server node 2 Ha-WS2
Ha-DB1
Ha-DB2 Ha-DB3
Heartbeat
Ha-WS1 Ha-WS2
6
Mongo DB server 1 Ha-DB1
Mongo DB server 2 Ha-DB2
Mongo DB server 3 Ha-DB3
Setting up Ha-Proxy:
Use apt-get command to install HAProxy apt-get install haproxy
Enable HAProxy to be started by the init script vi /etc/default/haproxy
set the ENABLED option to 1 ENABLED=1
Move the default config file to create a new default configuration file
mv /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg.save vi /etc/haproxy/haproxy.cfg
Add the following in the empty haproxy.cfg file global
log 127.0.0.1 local0 notice maxconn 2000
user haproxy group haproxy
Notes:
The log directive mentions a syslog server to which log messages will be sent. On Ubuntu rsyslog is already installed and running but it doesn't listen on any IP address. We'll modify the config files of rsyslog later.
The maxconn directive specifies the number of concurrent connections on the frontend. The default value is 2000 and should be tuned according to your VPS' configuration.
The user and group directives changes the HAProxy process to the specified user/group. These shouldn't be changed.
defaults
log global mode http
7 option httplog option dontlognull retries 3 option redispatch timeout connect 5000 timeout client 10000 timeout server 10000 Notes:
We're specifying default values in this section. The values to be modified are the various timeout directives. The connect option specifies the maximum time to wait for a connection attempt to a VPS to succeed.
The client and server timeouts apply when the client or server is expected to acknowledge or send data during the TCP process. HAProxy recommends setting the client and server timeouts to the same value. The retries directive sets the number of retries to perform on a VPS after a connection failure.
The option redispatch enables session redistribution in case of connection failures. So session stickness is overriden if a VPS goes down.
listen filecloud 0.0.0.0:80 mode http
stats enable
stats uri /haproxy?stats stats realm Strictly\ Private
stats auth proxyuser:proxypassword balance roundrobin
option http-server-close timeout http-keep-alive 3000 option forwardfor
server Ha-WS1 xx.xx.xx.xx:80 check server Ha-WS2 xx.xx.xx.xx:80 check Notes:
This contains configuration for both the frontend and backend. We are configuring HAProxy to listen on port 80 for filecloud which is just a name for identifying an application.
The stats directives enable the connection statistics page and protect it with HTTP Basic authentication using the credentials specified by the stats auth directive.
8
This page can be viewed with the URL mentioned in stats uri so in this case, it is http://Ha-LB/haproxy?stats;
The balance directive specifies the load balancing algorithm to use. Options available are Round Robin (roundrobin), Static Round Robin (static-rr), Least Connections (leastconn), Source (source), URI (uri) and URL parameter (url_param).
Information about each algorithm can be obtained from the official documentation. The server directive declares a backend server, the syntax is:
server <name> <address>[:port] [param*]
In the directive server Ha-WS1 xx.xx.xx.xx:80 , replace xx.xx.xx.xx with the actual IP address of the app server nodes.
Start HAProxy service
service haproxy start
Setting up FileCloud Appserver nodes:
The following steps have to be done for each of the FileCloud app server nodes (Ha-WS1 and Ha-WS2) Download and install FileCloud debian package (The actual URL may not be the same as shown below) cd /tmp wget http://patch.codelathe.com/tonidocloud/live/installer/file_cloud_deb.t gz tar –xzvf file_cloud_deb.tgz chmod +x install.sh ./install.sh
During the install, you can continue the install as if it is a standalone installation and provide a dummy cloud storage (something like /tmp). This will be changed later.
Stop apache and mongodb service apache2 stop service mongodb stop
9
vi /var/www/config/cloudconfig.php Add/replace the following keys
// ... Cloud Database
define("TONIDOCLOUD_DBSERVER", "mongodb://ip of ws-db1,ip of ws-db2,ip of ws-db3/?replicaSet=rs0");
// ... Audit Database
define("TONIDOCLOUD_AUDIT_DBSERVER", "mongodb:// ip of ws-db1,ip of ws-db2,ip of ws-db3/?replicaSet=rs0");
// … Settings Database
define("TONIDOCLOUD_SETTINGS_DBSERVER", "mongodb:// ip of ws-db1,ip of ws-db2,ip of ws-db3/?replicaSet=rs0");
Edit Cloud config and add/replace the following keys
vi /var/www/config/localstorageconfig.php
define("TONIDO_LOCALSTORAGE_DBSERVER", "mongodb:// ip of ws-db1,ip of ws-db2,ip of ws-db3/?replicaSet=rs0");
Start web server
service apache2 start
MongoDB Replica set setup
The following should be done on each of the database servers (HA-DB1, HA-DB2 and HA-DB3) Download and install FileCloud debian package
cd /tmp wget http://patch.codelathe.com/tonidocloud/live/installer/file_cloud_deb.t gz tar –xzvf file_cloud_deb.tgz chmod +x install.sh ./install.sh
10
During the install, you can continue the install as if it is a standalone installation and provide a dummy cloud storage (something like /tmp). This will be changed later.
Stop apache and mongodb service apache2 stop service mongodb stop
update-rc.d mongodb defaults #Start mongodb automatically update-rc.d apache2 disable #Do not start apache on DB nodes In each of the DB server (HA-DB1, HA-DB2 and HA-DB3) Open mongo configuration file vi /etc/mongodb.conf
Uncomment replSet and set it like the following replSet = rs0
Save the config file and start Mongo DB server service mongodb start
Open mongo shell mongo
Initiate the replica set rs.initiate()
Add the IP address of the other Database server (For example, when setting up WS-DB1, add the IPs of WS-DB2 and WS-DB3)
rs.add(“ip address of other WS-DB2”) rs.add(“ip address of other WS-DB3”)
Verification of service
Once all the service is set, Connect to the Haproxy web page ( http:// <loadbalancer IP>/haproxy?stats ) to ensure that Haproxy is able to see all the FileCloud app server nodes.
In each of the three database server nodes, connect to mongo shell and enter rs.status() to see the actual value
11
It should show something like rs.status() { "set" : "rs0", "date" : ISODate("2014-09-03T20:52:14Z"), "myState" : 2, "members" : [ { "_id" : 0,
"name" : "<ip of other DB>:27017", "health" : 0,
"state" : 8,
"stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : Timestamp(1409777412, 1), "optimeDate" : ISODate("2014-09-03T20:50:12Z"), "lastHeartbeat" : ISODate("2014-09-03T20:52:13Z"), "lastHeartbeatRecv" : ISODate("2014-09-03T20:51:54Z"), "pingMs" : 0 }, { "_id" : 1,
"name" : "<Ip of other DB>:27017", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 749, "optime" : Timestamp(1409777412, 1), "optimeDate" : ISODate("2014-09-03T20:50:12Z"),
"errmsg" : "syncing to: ha-db1.codelathe.com:27017",
"self" : true },
Setting Managed Storage
Since the FileCloud Appserver nodes do not store any of the application data, the managed storage must be an external location (A NAS, ISCSI , SAN , Amazon S3 or Open stack)
12
IN THIS EXAMPLE, WE ASSUME THAT EITHER NAS,NFS MOUNT IS ALREADY AVAILABLE AND MOUNTED ON EACH OF THE WEBSERVER NODES.
Open the FileCloud Admin portal by opening http:// <load balancer IP/ui/admin/index.html and log into the Administration portal.
Navigate to Settings->Storage Tab and set the mounted path and click “save”
Once the setup is complete, create user accounts by connecting to the admin portal and then log into the user account using the load balancer IP (which will route the traffic to one of the app server node). To test app server HA, Turn off one of App server by logging into the app server (say Ha-WS1) and stop Apache (using service apache2 stop) . The service will be accessible because HaProxy will reroute traffic to Ha-WS2.
To test database HA, Connect to the primary database server (Check the role by typing “mongo” in the database server and the prompt will indicate if it is a “PRIMARY” or “SECONDARY”) and turn off the mongodb server (service mongodb stop). This will cause a failover and one of the secondary will become the primary.
13
The service will be disrupted during the failover (It can take up to a minute) and the service will resume automatically.
