MWD Advisors is a specialist IT advisory firm which focuses exclusively on issues concerning IT-business alignment. We use our significant industry experience, acknowledged expertise, and a flexible approach to advise businesses on IT architecture, integration, management, organisation and culture.
www.mwdadvisors.com
Integrating people, processes
and systems: tools and best
practices for IT teams
Neil Ward-Dutton
January 2011
Today‟s business requirements for IT capabilities really pull in the opposite direction to the way that business software applications have typically been conceived and designed. Agile techniques and SOA principles are necessary elements to successfully delivering business software in this environment – particularly with respect to integration, information exchange and time-to-value concerns – but by themselves they‟re not sufficient. The good news is that there are tools you can use that „fill the gap‟, and that also fit with the expectations of teams ready to use Agile and SOA techniques. This report highlights key features of tools that you should consider if you‟re interested in delivering IT
capabilities that integrate people, processes and systems in flexible ways. The report also explains the most important practices you should look to implement to get the best out of your tool investments.
Summary
The scope, shape and stability of software applications have changed radically in recent years
In the past five years, widespread adoption of Service-Oriented Architecture (SOA) in industry has presented a situation where the assumptions of the 1970s, 1980s and even 1990s have been turned on their heads. Now, delivery of new business software capabilities often involves at least as much integration work as development of net new
functionality – and business expectations are for rapid delivery of new capabilities and changes within weeks or months.
Of course, the speed of delivery of new business software capabilities and the speed of capability change that are now expected aren‟t just architecture-enabled improvements within the scope of the IT organisation: they‟re forcing factors for IT shops because business pressures demand them. Particularly since the start of the economic slowdown that preceded global recession in 2008, businesses needing to make IT investments have placed ever more stringent demands on project payback periods.
Although SOA and Agile development practices are
necessary, they’re not sufficient to meet today’s needs
When we put all these factors together, we see that today‟s business requirements for IT capabilities really pull in the opposite direction to the way that business software applications have typically been conceived and designed. There are five main areas where moving architecture goalposts are working against the status quo of business software application development: scope, information exchange, integration, rate of change and time-to-value. Agile techniques and SOA principles are necessary elements to successfully delivering business software in this
environment – particularly with respect to integration, information exchange and time-to-value concerns – but by themselves they‟re not sufficient. This is principally because by themselves they‟re primarily focused on design and development work, and don‟t pull the other big stakeholders in the IT capability lifecycle (project sponsors, admin and operations staff) into the centre of activity.
Look for model-driven, SOA-enabled BPM tools and platforms that can generate business-meaningful metrics from applications in operation
The good news is that there are tools that „fill the gap‟, and that also fit with the expectations of teams ready to use Agile and SOA techniques. These SOA-enabled Business Process
Management (BPM) tools provide many features that amplify
the benefits SOA and Agile development efforts. The best tools combine model-driven design and development, focused on business process concepts, with model-driven monitoring that provides business-meaningful operational metrics, support for event-driven behaviour, and a clear separation of design concerns.
With the right combination of tools and practices, you can achieve very significant results: ROI payback periods of 6-9 months are far from uncommon, and ROI payback in as little as 3 months is possible.
The changing requirements environment
A network of enterprise change pressure points
It seems that every IT industry presentation or white paper these days starts out with a discussion of how the pressure to change is becoming ever more persistent. Still, though, it‟s true: change is more of a forcing factor in application delivery than it used to be. What‟s not so often called out, though, is that a lot of the business-initiated change pressure swirling around has a common consequence, and that‟s the importance of putting the concept of business process optimisation more front-and-centre in business and IT change efforts. In an environment where competition is ever fiercer, customer expectations are more informed and more stringent, supply chains are more complex and business and technology are becoming ever more tightly intertwined, a focus on end-to-end business process improvement is absolutely crucial.
We see large organisations engaging in change programmes where the same key themes come up time and time again: more efficient and effective interaction with external parties; more business model flexibility; better governance and regulatory compliance and better visibility of performance and risk; faster and more agile product/service delivery; and more effective revenue assurance. None of these change programmes can be addressed with individual, standalone business software applications: all of these are business process issues that need to be explored across team, department and potentially organisational boundaries.
The shape and stability of business software applications have
changed significantly
Since the application of software first started becoming a mainstream concern in industry, when mainframe computers were the only game in town, the shape and stability of enterprise software have changed hugely. Initially, applications stood by themselves: the only interfaces that applications had were used by computer operators. Application development projects typically took years, and applications were changed at a glacial pace. The arrival of the client-server computing era in the 1990s didn‟t change things much; but as the Internet started to gain a foothold within enterprises in the late 1990s and as e-commerce drove huge volumes of application development work, application
integration became a much more obvious concern for most. What‟s more, business expectations regarding the time it should take to develop an application (or change an application) started to shift palpably.
In the past five years, widespread adoption of Service-Oriented Architecture (SOA) in industry has presented a situation where the assumptions of the 1970s, 1980s and even 1990s have been turned on their heads. Now, delivery of new business software capabilities often involves at least as much integration work as development of net new functionality – and business expectations are for rapid delivery of new capabilities and changes within weeks or months.
Coping with the “new normal” of IT investment
Of course, the speed of delivery of new business software capabilities and the speed of capability change that are now expected aren‟t just architecture-enabled improvements within the scope of the IT organisation: they‟re forcing factors for IT shops because business pressures demand them. Particularly since the start of the economic slowdown that preceded global recession in 2008, businesses needing to make IT investments have placed ever more stringent demands on project payback periods. The kinds of „big bang‟ IT investments that used to take 9-12 months and consume multiple millions of dollars are now very rare; most organisations – even the largest – are looking for ways to manage IT investment programs so that value can be demonstrated at three-month (or at most six-month) intervals. The people with significant IT investment requirements want clear, step-by-step roadmaps with clear payback on each step-by-step of the journey.
Beyond agile development and SOA: integrating
people, processes and systems
Surveying the field of IT-business operations
When we put all the factors highlighted in the previous section together, we see that today‟s business requirements for IT capabilities really pull in the opposite direction to the way that business software applications have typically been conceived and designed. Figure 1 provides an overview of some of the ways in which solution architecture goalposts have moved over the past 10 years.
Figure 1: Today‟s IT capability requirements can‟t be fulfilled by „standalone‟ applications
The past 10 years have seen the architectural context for the delivery of IT capabilities turned on its head.
As the figure shows, there are five main areas where moving architecture goalposts are working against the status quo of business software application development. Three of these concern the shape of software applications:
Scope. 10 years ago the core functionality of an IT capability focused on how it could co-ordinate a set of database transactions (some data entry, some reporting). Now, driven by the current focuses of business improvement and transformation that cross team, department and event organisational boundaries, the scope for IT capabilities has changed dramatically. Now the most interesting issue is not how software can co-ordinate sets of transactions; but how software can help co-ordinate and manage the flow and execution and work of groups of people.
Information. 10 years ago the ways in which IT capabilities provided information to businesses was through individual users pulling information from them, either through reports or on-screen forms. Now, increased change pressures and the increased levels of interconnection between applications mean that IT capabilities need to be designed from the start to make information available to external systems as well as people – based more around a push model than a pull model.
Integration. 10 years ago any integration between discrete IT capabilities was likely point-to-point, and could be designed within the context of an application development project. Now, realisations about the brittleness of point-to-point capabilities in the face of change means that integration has to be considered as a dynamic, service-based capability that exists independently of, and is invested in separately from, individual applications.
Scope
Information
Rate of change
Time to value Integration
Transactions Departments, organisations
Pulled by individuals
Static, point-to-point
Years
Years
Event-based push to people, systems
Dynamic, service based
Weeks
The phrase „throwing software over the wall‟ has been in use for decades as shorthand for the organisational silos that often exist within IT shops. It‟s not as if cultural, technology and skills barriers between those who specify requirements, those who develop solutions and those who operate and administer those solutions are anything new. Figure 2 provides a graphical view of the two walls that still frequently exist: one between „the business‟ and developers, and another between developers and operations and administration staff.
Figure 2: Many organisations still operate silos that partition the software delivery lifecycle
‘Traditional’ use of documentation, development and administration tools creates not just one ‘wall’ but two – and there are multiple important ramifications.
As the figure shows, it‟s important to look at both these walls from multiple perspectives. Most people focus on what happens when developers fail to communicate effectively with operations and admin staff – and that‟s clearly important. But there are other important areas to consider:
Miscommunication between business sponsors and analysts, and developers. Opportunities for delay and confusion in communication between those with a need and those providing the „answer‟ abound – particularly when communication is based primarily on static requirements specification documents containing hundreds of pages. The issue of
miscommunication is of course important to consider in terms of tracing requirements from the business intent to eventual delivery of capability features. But given the rate of change in requirements for IT capabilities already discusses, it‟s at least as important to consider this in relation to the way that changed requirements are translated into changed capability features. Lack of effective information passing from operations to business sponsors, analysts.
When operations and administration staff provide information about their contribution to the value of capabilities it‟s most often in the form of statistics about infrastructure health and performance. This can be useful, but such information rarely provides any insights that have value in a business context. If the infrastructure underpinning a key IT capability supporting sales and marketing is working well, does that mean that orders are flowing freely through the sales process? What does information about the uptime of a set of servers supporting a call-centre capability say about customer service levels?
Explore,
measure
Define
Execute,
monitor
Business
analysts Developers Operations Business stakeholders
Sponsor,
drive
usage
X
Difficult to engage stakeholders
X
Difficult to trace requirementsX
Difficult to have confidence in changeWhat‟s common in all of these areas of potential miscommunication is a fundamental mismatch between the information needed to track and manage IT capability value and what‟s actually available. These mismatches create significant risks to organisations trying to deliver IT capabilities that really meet today‟s challenging and fast-changing business needs.
The role of Agile development and SOA
Agile development: tackling the change agenda... in part
In the handful of years since Agile techniques began to be investigated and adopted by mainstream business software development shops, it‟s become clear that Agile development approaches can have significant impacts on development timescales and payback periods. Well-trained teams using Agile development approaches can ship working systems on 14-, 21- or 28-day cycles and get copious user feedback as they build out and refine application features. In the business world we‟re in, where requirements are challenging and fast-changing, Agile development techniques can be a truly great fit. But we‟re also seeing that Agile development techniques can only take you so far in meeting the needs thrown up by today‟s field of IT and business operations. There are two particular areas of
shortcoming. The first is that although Agile techniques have been shown to help development teams work more collaboratively with business analysts and users, the same is not so true of systems administration and operations teams. There are many examples of Agile projects where the interface with systems administration and operations teams has proven to be a major bottleneck: managers in charge of live operational platforms aren‟t prepared to deploy code that‟s been developed using Agile techniques because they don‟t trust its quality. To get around this they commission further testing – meaning that in many cases testing cycles are replicated. This is hardly in the spirit of Agile.
The second area of shortcoming relates to the purpose of Agile techniques (and more specifically what Agile isn‟t focused on). Although Agile techniques can be used to drive delivery and change faster, Agile techniques have nothing to add when it comes to making the link between the
operational environment and the business expectations of project sponsors. Without the right tools and frameworks as a foundation, you won‟t get any help from Agile techniques in this area.
SOA: addressing integration challenges... but the wider issues?
SOA has an important role to play in meeting the requirements of today‟s IT and business operations landscape, and not only because it helps developers and architects build shared, flexible integration capabilities that can be managed separately from individual applications. SOA work is also valuable because over time, catalogues of reusable services can accelerate the pace of application delivery, and also – if done right – SOA can improve the quality of IT capabilities by improving knowledge about the structures and dependencies that exist within complex systems.
However just as is the case with Agile development techniques, SOA – though very valuable – isn‟t enough by itself to break down the walls that so many organisations struggle with (see figure 3). Specifically, as is the case with Agile techniques, SOA concepts and tools don‟t necessarily deliver value in administration and operations teams. In theory SOA can, but the practice falls some way behind the theory in most cases. And as is also the case with Agile techniques, SOA techniques and tools don‟t by themselves provide a natural foundation for communication with non-technical
stakeholders. Again, in theory SOA practice can deliver a unified, transferable context for managing IT capabilities across technical and non-technical stakeholder groups: but because SOA is in most cases pursued as a technical integration-focused effort and doesn‟t encompass business architecture or service management thinking, the practice doesn‟t tend to match the theory.
As is the case with Agile techniques, SOA concepts and tools don’t necessarily deliver value in administration and operations teams.
Explore,
measure
Define
Execute,
monitor
Business
analysts Developers Operations Business stakeholders
Sponsor,
drive
usage
SOA
Tools and practices for dealing with today’s field
of IT-business operations
Let‟s quickly recap where we are. Today‟s field of IT and business operations is characterised by pressures and complications in the scope, information requirements, integration requirements, rate of change and time-to-value requirements of IT capabilities. Both SOA techniques and Agile development techniques have roles to play – particularly in addressing issues relating to information, integration and time-to-value. But even a combination of SOA and Agile isn‟t quite enough to really deliver on today‟s business requirements – particularly over the long haul.
The good news is that there are tools you can use that „fill the gap‟, and that also fit with the expectations of teams ready to use Agile and SOA techniques. This section explores the main characteristics of these tools, which we‟ll call SOA-enabled Business Process Management (BPM) tools. We also highlight some of the important practices that you‟ll need to employ when using these tools in order to get the best out of them.
Key features tools you should look for
There are five key tool platform features that play crucial roles in addressing today‟s IT-business field of operations:
Workflow support. With the scope of IT capabilities now concerned primarily with the co-ordination of work across teams and organisations, tools that provide frameworks for explicitly orchestrating tasks carried out by people – as well as orchestrating functionality delivered by software services – are essential.
Model-driven. Tools should place graphical „business process‟ models, which specify high-level service composition as well as human task orchestration, as the core artefact at the heart of both high-level and low-level architecture and design work – and they should compile application code directly from those graphical models. Having one graphical language that both illustrates IT capability structure and logic to stakeholders in the context of business processes, and also acts itself as code, helps to sweep away the information sharing challenges that are so often at the heart of today‟s IT delivery lifecycle challenges. These tools should be simple to use, and the resulting models need to be easy to show and share with others.
Transparent. Having business-meaningful models as a core part of both specifications and code is hugely beneficial, but it‟s even better if those models can also be used as the context for business-meaningful operational metrics once applications are in production. As we‟ve previously noted, operational metrics that focus on infrastructure health and performance are of little value if you‟re trying to demonstrate the business value of an IT capability investment that‟s been made. You should look for tools that are able to monitor health and performance in terms of instances of processes and work rates, in the context of business process models.
A clear separation of concerns. Business process models are an essential core, but “finished applications” need more than just process logic – and it‟s unreasonable to expect that any tool will enable complete software applications to be delivered without someone having to do some technical work. You should look for tools that help you draw clear boundaries around technical work (such as integrating processes with back-end resources and systems) – clearly separating model-based configuration work that can be done by non-technical people (possibly business stakeholders, in the case of business rules and policies) from work that has to be done by technical specialists. SOA-enabled BPM technology has a great advantage here.
Event-driven infrastructure. With the increased change pressures and the increased levels of interconnection between applications that we highlighted earlier, the ability for applications to make information available to external systems as well as people – based more around a push model than a pull model – is increasingly important. You should look for tools that enable events
„trickle-feed‟ information to analytics tools or other applications, or alternatively to co-ordinate state and activity across multiple application and process instances, supporting highly dynamic and flexible processing scenarios.
The right tools must be complemented with the right practices
Choosing and using tools that exhibit the features above is a great start if you want to accelerate your IT capability delivery efforts and meet the needs of today‟s IT and business field of operations; but tools by themselves are never enough. You need to make sure your teams and procedures are set up to get the most out of those tools. Our case study research has identified three particularly important practices that teams of architects and developers should look to embrace: Agile iterations, continuous integration. Although we‟ve said that Agile development techniques aren‟t by themselves sufficient to meet today‟s needs for many applications, that‟s not to say they aren‟t necessary. The right SOA-enabled BPM tools support Agile development and delivery techniques, and you should look to employ a method like Scrum if you can. Your SOA-enabled BPM tool should support continuous integration practice, and provide functional testing features too.
Collaborative requirements refinement. Particularly at the start of each iteration, you should find ways to bring developers, architects, business analysts and appropriate front-line users together to collaboratively identify and refine requirements. Ask business managers to identify front-line users who show an interest in improving things, and work to cultivate them as „champions of change‟ by involving them as much as possible. Using graphical process models provides a helpful canvas for this work, but tools don‟t by themselves ensure collaboration: you will have to work proactively and regularly with projects sponsors and champions to ensure that all the necessary people prioritise their involvement and turn up to workshops. Making these sessions focused, structured and as short as possible – which requires up-front „homework‟ to prepare the ground – will help convince people to continue to invest their time in the process. Proactive business performance measurement. If you‟re really serious about making an
impact, then work proactively to prepare the ground for measuring the business value of the investment that‟s being made in the applications you‟re delivering. Where most business value assessments fall short is in measuring the „as is‟ state of a working environment or system. Sometimes, business cases for investments are based on such measurements: but in practice, often business cases are more often based on gut feel. Work to quickly collect as many relevant measurements as you can, before you start to implement any new IT capabilities, and use these as a benchmark. Try to use your SOA-enabled BPM tool to instrument your process application(s) so that you can measure the same metrics once your application(s) are in operation, and track performance over time as applications are refined and as users become more familiar with them.
What does world class implementation look like?
As figure 4 shows, combining an investment in the kind of tools we‟ve signposted in this report with the right set of practices can delivery very significant business results. It‟s not uncommon for successful projects to be able to demonstrate working systems with sophisticated features in the space of weeks, and 100% ROI payback periods of 6-9 months are quite easy to find in the field (some projects can actually deliver positive ROI in as little as 3 months).
Figure 4: What does world class look like?
When the right tools are used in the right way, the results can be very compelling
