Deployment
91Running a microservice system
Turing disease
Configuration formats tend to be extended with programming constructs over time, mostly as conveniences to avoid repetition1. Now you’ve a new unasked-for
programming language in your system with no formal grammar, undefined behav- ior, and no debugging tools. Good luck!
5.7.7 Security
The microservice architecture doesn’t offer any inherent security benefits, and intro- duces new attack vectors if care isn’t taken. A common temptation is to share microservice messages with the outside world. This is dangerous, as it exposes every microservice as an attack surface.
There must be an absolute separation. You need a "Demilitarized Zone" (DMZ) between the internal world of microservice messages, and the outside world of third party clients. The DMZ must translate between the two. In practice this means that a microservice system should expose traditional integration points, such as REST APIs, and then convert requests to these APIs into messages. This allows for strict sanitiza- tion of input.
Internally, you can’t ignore that microservices communicate over a network, and networks represent an opportunity for attack. Your microservice should live within their own private networks with well-defined ingress and egress routes. The rest of the system uses these routes to interact with the microservice system. The specific microservices to which messages are routed aren’t exposed.
These precautions may still be insufficient, and you need to consider the case where an attacker has some level of access to the microservice network. You can apply the security principle of "defense in depth" to strengthen your security in layers. It’s always a trade-off between stronger security and operational impact.
Let’s build up a few layers. Microservices can be given a right of refusal. They can be made more pedantic in the messages they accept. Highly paranoid services lead to higher error rates, but can delay attackers and make attacks more expensive. For example, you can limit the amount of data that a service returns for any one request. This approach means custom work for each service.
Communication between services can require shared secrets, and as a further layer, signed messages. This protects against messages injected into the network by an attacker. The distribution and cycling of the secrets introduces operational complex- ity. The signing of messages requires key distribution and introduces latency.
If your data is sensitive, you can encrypt all communication between microservices. This also introduces latency and management overhead, and isn’t to be undertaken lightly. Consider using the Merge pattern for extremely sensitive data flows to avoid the network as much as possible.
The need for secure storage and management of secrets and encryption keys is necessary if these layers are to be effective. It’s pointless to encrypt messages if the keys are easily accessible within the network. And yet, microservices need to be able to access the secrets and keys. To solve this problem, you need to introduce another net- work element—a key management service that provides secure storage, access con- trol, and audit capabilities1.
5.7.8 Staging
The staging system is the control mechanism for the continuous delivery pipeline. It encompasses traditional elements, such as a build server for continuous integration. It can can also consist of multiple systems that test various aspects of the system, such as performance.
The staging system can also be used to provide manual gates to the delivery pipe- line. These are often unavoidable, either politically, or legally. Over time, the effective- ness of continuous delivery in managing risk and delivering business value quickly can be used to create sufficient organizational confidence to relax overly ceremonial man- ual sign-offs.
The staging system provides a self-service mechanism for development teams to push updates all the way to production. The empowerment of developer teams to do this is a critical component in the success of continuous delivery. We’ll discuss this human factor in Chapter 7.
Staging should collect statistics to measure the velocity and quality of code delivery over time. It’s important to know how long it takes, on average, to take code from con- cept to production, for a given risk level, as this tells you how efficient your continu- ous delivery pipeline is.
The staging system has the most variance between projects and organizations. The level of testing, the number of staging steps, and the mechanism of artifact genera- tion, are all highly context specific. As you grow the use of microservices and continu- ous delivery in your organization, you should avoid being too prescriptive in your definition of the staging function. You must allow teams to adapt to their own circum- stances.
5.7.9 Development
The development environment needed for microservices should enable the devel- oper to focus on a small set of services at a time, often a single service. The message abstraction layer comes into its own here as it makes it easy to mock the behavior of other services2. Instead of having to implement a complex object hierarchy, microser-
vice mocking only requires implementing sample message flows. This makes it possi- ble to unit test microservices in complete isolation from the rest of the system.
93 Summary
Microservices can be specified as a relation between inbound and outbound mes- sages. This allows the developer to focus on a small part of the system. It also enables more efficient parallel work, as messages from other microservices (which may not even be written) can easily be mocked.
Isolation isn’t always possible or appropriate. Developers often need to run small subsets of the system locally. Tooling is needed to make this practical. It isn’t advis- able for development to become dependent on running a full replica of the produc- tion system. As production grows to hundreds of different services, and beyond, it becomes extremely resource intensive to run services locally, and ultimately it becomes impossible.
If the developer is running only a subset of the system, how do you ensure that appropriate messages are provided for the other parts of the system? One common anti-pattern is to use the build or staging systems to do this. Developers end up work- ing against shared resources that have extremely non-deterministic state. This is the same anti-pattern as having a shared development database.
Each developer should provide a set of mock messages for their service. Where do these mock messages live? At one extreme you can place all mock message flows in a common mocking service. All developers commit code to this service, but conflicts are still rare, as work isn’t likely to overlap. At the other extreme you can provide a mock service along with each service implementation. The mock service is an extremely sim- ple service that returns hard-coded responses.
The practical solution for most teams is somewhere in the middle. Start with a sin- gle universal mocking service, and apply the Split pattern whenever it becomes too unwieldy. Sets of services with a common focus tend to get their own mocking service. The development environment is typically a small set of actual services, along with one or two mocking services. This keeps the number of service processes needed on a developer machine to a minimum.
The mock messages are defined by the developers building a given microservice. This has the unfortunate side-effect that the developer focuses on expected behavior. Others use their service in unexpected ways, and the mocking is incomplete. If you allow other developers to add mock messages to services they don’t own, then the mocks quickly diverge from reality. The solution is to add captured messages to the list of sample messages. Capture sample message flows from the production or staging logs, and add them to the mock service. This can be done manually for even medium- sized systems.
5.8
Summary
This chapter examined the nature of failure in complex systems. Failure is inevitable, and must be accepted. Starting from that perspective, you can work to distribute fail- ure more evenly over time, and avoid high impact catastrophes.
give the correct answer 100% of the time. The closer you want to get to 100%, the more expensive it gets.
The risk of failure is much higher than generally believed. Simple mathematical modeling of risk probabilities in component based systems (such as enterprise software) brings this reality starkly into focus.
Microservices provide an opportunity to measure risk more accurately. This enables you to define acceptable error rates that your system must meet.
By packaging microservices into immutable units of deployment, you can define a set of deployment patterns that mitigate risk and can be automated for efficient management of your production systems.
The accurate measurement of risk enables the construction of a continuous delivery pipeline that enables developers to push changes to microservices to production a high velocity and high frequency, whilst maintaining acceptable risk levels.
This chapter refrained from prescribing specific techniques and tooling. As a software architect you need the freedom to make these decisions yourself. Rather, the focus on fundamentals, and basic patterns gives you a framework to help make these decisions. You’ll need to adapt the principles in this chapter to your own context.
95 Summary
Microservices are small, but they offer big value. A microservice is a very small piece of a larger system that can be coded by one developer within one iteration. Microservices can be added and removed individually, new developers can be immediately productive, and leg- acy code is easily replaced. Developers are no longer hampered by the communication and coordination overhead caused by monolithic systems. Savvy businesses are discovering that software development productivity can be greatly enhanced with the right engineering approach that enables even junior developers to double their productivity, while reducing delivery risk.
The Tao of Microservices teaches you the path to understanding how to apply microservices architecture with your own real-world projects. This high-level book offers you a conceptual view of microservice architectures, along with core concepts and their application. You'll also find a detailed case study for the nodezoo.com sys- tem, including all source code and documentation. By the end of the book, you'll have explored in depth the key ideas of the microservice architecture and will be able to design, analyze and implement systems based on this architecture.
What's inside:
Key principles of the microservice architecture
Applying these principles to real-world projects
Implementing large-scale systems
Detailed case study
This book is for developers, architects, or managers who want to deliver faster, meet changing business requirements, and build scalable and robust systems.