Introduction
This reference describes an ideal Linux infrastructure designed to cover the majority of application deployments. It is infuenced heavily by our experience in working with both large and small customers, and represents what we believe to be current best practice.
We primarily use open-source and free tools, believing them to be the best choice in most cases, however some are also available under commercial support from their vendor if that is required.
The approaches described here are applicable whether hosting on co-located servers, dedicated servers from a managed service provider, or using cloud servers.
We believe that good infrastructure is:
• Scalable – able to grow horizontally or vertically as required by the operational demands put on it by the business.
• Supplier agnostic – not tied to products or services of any organisation (including The Scale Factory), allowing portability between hosting suppliers or service organisations.
• Cost efective – no more expensive than it needs to be, either in capital or operational expenditure.
• Current – using modern tools and techniques, though without being too experimental about it.
• Repeatable – easily repeated, so new server builds or secondary datacentre rollouts are straightforward.
• Available – meets availability requirements for the business.
• Performant – delivers resources quickly and efficiently.
• Responsive to change – can deal with changing business requirements rapidly.
• Manageable – easy to operate centrally, and without the requirement of heavy staffing levels.
other tools so that it's clear what the platform is doing.
• Secure – protected from external and internal attackers.
• Auditable – transparent in process, and easy to see who has changed what and when
Basic Design
The standard Scale Factory architecture is based around one or more hosting sites. Each site contains a set of supporting infrastructure for central control and observation purposes. This supporting infrastructure enables the operation of service hosts (either physical or virtual) arranged into clusters. These clusters run applications which are then arranged into platforms.
In this fgure, there are three sites – dc1, dc2 and of1. Each of these contains support services. The site dc1 houses live and
staging platforms, which contain web and database services.
Location
The Scale Factory Reference Architecture is designed to be agnostic of where it is hosted, and can be built out in co-location suites, on managed servers or in fully cloud-hosted environments.
Depending on your application's constraints, one or more of these options may not suit.
If building out an infrastructure on your own hardware in a colocation suite, you get the highest level of control over how things are put together, and this may be desirable in the case of specialist application requirements, or for reasons of compliance. With this level of control comes an increased level of responsibility. If a RAID array throws a disc in the middle of the night, you'll need to be monitoring for and responding to that eventuality in some way.
not uncommon.
Lead times on hardware orders can sometimes take you by surprise too – it's not completely unknown for a server order to take six weeks to be fulflled, particularly when adverse weather conditions cause RAM or hard disc manufacturing difficulties.
An alternative to colocation is to take servers from a managed service provider. These organisations have hardware and network specialists on staf, and will take responsibility for keeping servers up, replacing failed discs, etc. In exchange for this, you may lose an element of control – things like network layout and server specifcation are likely to be constrained to some extent.
Managed server hosting costs vary widely across the industry, and most are open to negotiation. Some hosts charge higher setup fees in exchange for a lower monthly rate. Some ofer discounts if you pay for a whole year up-front.
Fully cloud-hosted solutions (such as Amazon EC2 or Linode) are currently very popular. They are best suited to applications whose components all scale horizontally, and/or have extremely peaky levels of demand.
Applications which require predictable performance can be problematic on cloud servers, as sharing resources with other customers means you get no guarantee of resource availability. The cheaper, smaller instances can be heavily CPU contended at times, and larger instances are more expensive month-on-month than equivalent dedicated servers with a managed hosting supplier.
Supporting Infrastructure
In a Scale Factory architecture, we install supporting infrastructure at each site. These provide services to the rest of the network, including confguration management, monitoring and backups, as well as central identity, time and naming services. This simplifes server management and ensures consistency across the network.
Core servers are provided in a high-availability pair, with some services provided in an active/active model, and some in active/passive using a foating IP address:
DHCP and TFTP services are only required in co-located sites where network confguration and automated system install is not performed by the supplier.
Naming
Sensible naming of hosts and services is more important than it appears. A good naming scheme represents the location and purpose of a device, as well as alluding to its membership of a platform, cluster or service tier.
Poor or badly-implemented naming leads to administrative complexity, and increases the likelihood of operator error.
We defne a consistent naming standard for all infrastructure components, ensure all hosts have forward and reverse DNS mappings and confgure them to report their hostname correctly.
Identity
Identity management is crucial to the operation of even a small systems infrastructure. Identity management systems are used to authenticate and authorise users.
Authentication is used to establish that a user is who they claim to be. The most obvious example of this is the use of passwords.
Authorisation is the system of checks that ensure your authenticated user can only access the resources and privileges that they need to do their jobs.
We use 389 Directory Server1 to provide central identity
Centralised identity management also means that when new employees start, or old employees leave, there is just a single place where their credentials need to be registered or removed, keeping complexity to a minimum.
Administrative users have standard user accounts like everyone else. Their superuser access is then limited to the minimum set of privileged activities required to do their job. This has the advantage that there is no need to share account passwords. The root password can then be locked away in case of emergencies. Activities performed by these privileged users can then by logged against their identity for audit purposes.
Batch processes and automated jobs are also assigned their own users, and have minimal access to superuser resources just as a human user would.
Configuration Management
When you install an operating system on a host, only the very basics of that environment are confgured. Before that host is usable in any real sense, other software needs installing, confguration fles need to be updated, and services need to be started.
The traditional, naïve approach to this problem is to make these confguration changes by hand. This approach doesn't scale well to an infrastructure of more than a couple of servers, and isn't advisable even then.
We use Puppet2 to automate this type of confguration. Given
a description of the desired server state, Puppet installs packages, updates confguration fles and starts services in order to reach that state.
Throughout the lifetime of a server, other administrative changes are occasionally required. Puppet can make these too, and will only make the minimum set of changes required to reach the new desired state.
benefts this lends to collaborative working, this also means all infrastructure changes beneft from an automated audit trail, showing which individual made changes and when.
Confguration management minimises diferences between diferent systems and environments by using a standard set of tools for building all environments. It is an important part of ensuring parity between development and production environments.
Disaster recovery and host rebuild becomes extremely straightforward when no manual confguration activity is required.
Availability
For most infrastructures, availability is important. In the most straightforward view of the world, this simply means that in order to cope with failure you need at least two of everything.
For services which maintain no local state, such as web or content servers, it is generally possible to place two or more of these behind a network load balancer.
In the case of services which hold their own state data, most provide their own approaches to high availability. For example, MySQL or SOLR database hosts can be confgured to replicate to one or more slave hosts. Memcache servers can be run as a cluster, as long as your application has a list of every node in the cluster.
Where a service or resource can only exist on a single host at a time, cluster software exists to allow multiple hosts to negotiate with each other for the right to own that service or resource, and we regularly do this to allow fail-over of master IP addresses for services.
High availability can come with associated cost, and so we look at each system's availability requirements in turn. For example in a standard Scale Factory infrastructure, we ensure that confguration management, identity, time and naming services are highly available as their unavailability can afect live service. Observational services such as monitoring and logging can run without failover to keep costs down, and largely without afecting live system availability.
We use tools from the Linux Virtual Server Project3 to provide
Backups and Disaster Recovery
Every infrastructure has important data which needs to be reliably backed up. For most web applications, this tends to be application code, database content and media assets.
We design infrastructure so that the data you want to back up is held in a small number of well-defned places.
In our standard design, we provide at least one host with plenty of (RAID mirrored) storage at each site. We run Bacula4
on this host, confgured to back up important data on a nightly schedule. This backup can then be replicated of-site, with encryption where required for security.
We run backups in a way which minimises service disruption. For example, with MySQL databases, this usually means taking snapshot backups from an otherwise unused replication slave.
A backup is only as good as your ability to restore it, and so we provide methods by which restoration tests can be performed.
Monitoring
A hosting infrastructure is a large and often complicated beast, and in an environment that can, thanks to fail-over, cope with individual node failure it isn't possible to tell how healthy the platform is by simply testing whether a website is available to the outside world.
We solve this problem by installing a monitoring system. A good monitoring system can monitor the health and performance of individual applications; the host operating system; the network; and hardware devices. The data collected is then usable for capacity planning, availability monitoring, performance analysis and troubleshooting.
We install Zabbix5 for this purpose, and provide a set of
Log Aggregation
Systems often generate a lot of log data, and attempting to troubleshoot problems in a distributed system without centralised log aggregation is a painful exercise.
Individual application servers shouldn't need huge amounts of space for log storage, and should ideally keep log data for only a limited amount of time.
We use rsyslog6 to ship live logging data from individual hosts
to a single log server, where administrators can log in and view the logging output of the whole platform in a single place.
Centralised logs make for easier troubleshooting, and software such as Splunk7 (not free) or Graylog8 can make this
troubleshooting easier by ofering an easy to use interface for querying log data. Automated fltering and querying can also drive data collection or monitoring events, to help improve operational visibility.
We're currently working on adding Graylog to our standard platform design.
Architecture
As an infrastructure grows, it pays to have thought about the architecture of the systems and software in advance. Certain architectural decisions can limit the potential for growth by making scalability difficult.
Diferent parts of a system scale at diferent rates – for example, on a photo sharing site such as Flickr, the infrastructure responsible for storage of photographs will be the fastest-growing part of the system; storage for the comments will scale at a diferent rate (probably more slowly); and the system which stores a username/password pair for each user will scale more slowly still.
Security
Diferent infrastructures require diferent levels of security, but we always work with a sensible minimum set of requirements in mind.
Using either physical frewalls or per-host iptables rules (depending on what the hosting provider can ofer), we ensure that administration is only possible from local core servers, or any trusted networks connected to the infrastructure by VPN. This includes access to SSH as well as any internal web-based admin or monitoring dashboards.
When services provide methods by which to control access, we ensure that this is granted to consumers of this service on a “least privilege” basis, granting only as much access is required, and no more. Human operators are treated in the same way, with superuser access being limited to the minimum set of permissions required by that user to do their job.
Use of confguration management tools and least-privilege superuser access, along with centralised logging and monitoring provides an audit trail for all system activities.
Development / launch process
As a code base grows, rigorous manual testing becomes impossible. When testing is seen as a large time overhead, it is often skipped in order to meet deadlines, and a code base without tests becomes a place where developers fear to tread, slowing future progress.
Automated testing should be considered mandatory for any modern software project. Continuous integration – where tests are run every time code changes – can be tricky to set up, but pays dividends over time. In an ideal world, if you have a broad suite of tests in which you trust, code launches can be automated, and happen continuously whenever changes are made. In a modern web application, change should be considered the norm, not the exception and we attempt to engineer our systems to cope with this view.
We make use of the Jenkins9 CI tool to automate test and
deployment, and provide auditable visibility of the process. The testing approach itself will depend on language environment, but the following broad categories of test will no doubt be relevant:
• BDD style scenario testing
• Integration testing
• Performance testing
• Penetration testing
As well as running tests, Jenkins can be used to trigger automated deployment of application software using tools such as Capistrano10 or Fabric11 which removes the margin for error
that software deployment can otherwise entail.
We follow many of the best practise guidelines described in Continuous Delivery12 by Jez Humble and David Farley.
