Five Best Practices
for Utilizing
A network change and configuration
management (NCCM) initiative has
helped Duke Energy set a solid
founda-tion for future growth and innovafounda-tion.
By enhancing and automating NCCM, the
company has improved IT efficiencies,
enforced compliance, and reduced the
risk of change on the business. Here are
five best practices from this experience.
By PAul EDMuNDs
Network Change and
Se r v ic e A U to m A tio n
A
t Duke Energy, we’ve looked into our crystal ball to foresee the utility of the future. One of the exciting things we envision is the smart meter. Initially, these meters will gather usage statistics from customer homes and businesses and send them to our data center for billing purposes. Longer term, they will allow interaction with “smart” appliances and electrical devices at the customer site.Does that mean you’ll be able to turn on your oven and start cooking dinner while caught in traffic? Perhaps. More likely, the smart meter will monitor energy consumption and notify you if utilization is unusually high, or send cost-saving tips based on your usage patterns. For example, you might get an e-mail in the summer months advising you to use the clothes dryer at 9 p.m. rather than at 1 p.m., as energy rates are lower in the evening.
What does this mean for Duke Energy’s IT infrastructure? It means that our network environment will grow, and we’ll need to manage the communications between these devices and our corporate systems. As fun as it is to prognosticate, we must stay grounded in reality. And the reality is that to get to the utility of the future, we need to lay a solid foundation today and demonstrate effective control over our current IT infra-structure. We are laying that foundation at Duke Energy with a variety of initiatives aimed at network automation.
One area of particular importance is network change and configuration management
(NCCM). In 2007, we launched an NCCM initiative that focused on best practices and new technology. Our goal was to enhance and automate change control so we could do a better job of meeting the network demands across our internal network.
We are making great strides, and in the process, we are gaining valuable experiences we would like to share with other enter-prises that are also laying the foundation for their futures.
1. FoCus FIrsT oN THE BIggEsT
PAIN PoINTs
Simply put, NCCM is about automating and simplifying configuration change across the network infrastructure — across all the switches, routers, firewalls, load balancers, wireless access points, and other network devices that keep the distributed infra-structure humming. But NCCM is not simple. NCCM products and best practices encompass many areas, including consistent configuration backup and recovery processes, adherence to best practices to enforce and demonstrate compliance, and change reporting and drift monitoring.
At the beginning of our project, we knew we could not tackle all these areas at once. We needed to focus our attention on the area that was creating the biggest problem for us. For your organization, the biggest problem might be keeping devices updated with the latest operating system to keep
NCCM is about automating and
simplifying configuration change
across the network infrastructure.
up to date with vendor security advisories, rolling out mass configuration updates, demonstrating compliance for audit purposes, or something else.
For us, the biggest issue was making changes quickly and reliably to the devices we have in our network. Our infrastructure encompasses nearly 900 routers and 2,500 switches supporting close to 18,000 IP addressable nodes. Like most companies, we have a global infrastructure, and our network connects corporate offices, power plants, and operations centers throughout the Southern and Midwest portions of the United States, as well as South America.
There was a time when most of our switches and routers came from a single vendor. As in many companies, however, growth, mergers and acquisitions, and emerging technologies have resulted in a network infrastructure that contains a mix of device types from many vendors. Each device has different management capabilities and required manual interaction to make changes. The need to align with business requirements and improve service delivery also added complexity. We are already utilizing concepts associated with Business Service Manage-ment (BSM) to help us improve on this front. Our network is segmented to support each of our different business units; today we have ten business units, but growth is always on the horizon. NCCM helps us to configure the unique router and switch settings (e.g., IDs, passwords, and community strings)
for each business unit. It is a complex job to ensure that every device is configured to meet the needs of the business unit to which it is assigned. Likewise, it is difficult to keep these devices up to date with new software versions and configuration changes required by the business.
Our NCCM solution is delivering tremendous benefits in this area by helping us uncover discrepancies. In the days when community strings were entered manually, typographical errors were common. Maybe the letter “O” was typed instead of a zero. Our solution is helping us find those errors and correct them quickly.
2. sTArT AuToMATINg
Previously, we could manage network change manually. We would telnet into each device, go into the configuration file, and modify the settings. Over time, we developed automated scripts that touched each device, made the required changes, and reported back to us. This level of automation was a substantial improvement over the manual process. However, as the network grew, it sometimes took several weeks to execute all the scripts and update all the devices. It was a tedious effort to ensure changes were made reliably.
Taking several weeks to implement a change is no longer acceptable. In addition, security has become a critical concern. As a result, we now must comply with stringent corporate security policies.
For instance, we have to keep tight control over who can access a device on the network.
Taking several weeks to implement
a change is no longer acceptable.
Se r v ic e A U to m A tio n
Otherwise, we run the risk of an unauthor-ized person making changes and impairing the network’s ability to function, thus affecting business operations at large. Not only could disruptions to internal business operations occur, but changes by unauthor-ized individuals also could interfere with the dedicated Web sites used by some of our large commercial customers to manage power deliveries. The cost of that disruption and interference could be significant. Because our security policies are very strict, if a member of a data network group leaves the company, we must immediately change all the passwords and community strings on the devices that person could access. To handle the need for speed in situations like this, we have taken a quantum leap in our level of automation. Previously, this process took a day or longer, but now it takes only minutes. Automation does not just allow companies to get things done faster. It also reduces the risk of human error, ultimately reducing the potential negative impact of change.
Recently, our firewall team determined they had improperly set a parameter on a number of firewalls that would have opened up our network to intrusions. The manager was able to run a report that searched for the improperly set parameter, and then the manager could quickly and efficiently roll out the change in a matter of hours.
3. lEvErAgE AuToMATIoN To
DrIvE CoMPlIANCE
Automation is just one benefit of imple-menting an NCCM solution. A byproduct of automation is change visibility and improved
change control. Because most NCCM tools offer reporting, role-based access control, and policy-based change templates, organiza-tions now have a way to demonstrate that effective change controls are in place.
To drive compliance, we are building a set of rules for each business unit specifying what the community string should be and how various parameters should be set. We can then apply these rules to the devices. Through reporting, we can identify devices that don’t comply with the rules, and we can fix the discrepancies. Self-healing policies enable us to set up compliance rules that automatically remediate any compliance violation on a specific device, device group, and so forth.
The compliance we are achieving for internal corporate policies also positions us to demonstrate compliance with external mandates in the United States, such as the Sarbanes-Oxley Act and the Federal Information Management Security Act. If your company is in another industry, such as health care or retailing, establishing rules will help you comply with relevant government mandates and industry standards, such as the Health Insurance Portability and Accountability Act (HIPAA) and Payment Card Industry security standards.
We are also finding that reporting enhances visibility into the network. One of our managers in the data network support group
Automation does not just allow companies to
get things done faster. It also reduces the risk
of human error, ultimately reducing the
potential negative impact of change.
regularly runs asset checks to determine how many of a particular type of device we have, which software version those devices are running, and other information he needs to track. Our NCCM tool provides a level of detail we did not have before. Consequently, this manager can get the information quickly and easily.
4. DEDICATE THE rEsourCEs
IT budgets are always limited, so obtaining resources for an NCCM initiative is never easy. But if you are going to the expense of buying an NCCM tool, it’s imperative to secure senior-level sponsorship to acquire the resources needed for a successful project.When you look into funding, don’t forget the importance of providing quality training. Although you’ll find some solutions easier to use than others, none will provide value unless the network team consistently uses the tools. So, plan and secure funding for the
necessary training, because every software package has its nuances and requires a bit of learning for effective use.
One of the major benefits from training is learning how to leverage standard and ad hoc rule sets to automate complex network changes, thereby eliminating manual inter-vention. Like most things, taking time at the beginning for training improves initial deployment success rates and in some cases can lead to near immediate ROI.
5. INTEgrATE wITH oTHEr
IT ProCEssEs
As beneficial as NCCM is on its own, its power is multiplied when integrated with other IT service management disciplines, such as trouble ticketing and change manage-ment. By getting a handle on the diverse network devices in our infrastructure and ensuring they are configured correctly and operating the way we expect, we achieved major strides in improving our service delivery. We also laid a foundation for future growth and expansion without adding to the inherent complexity of a larger network.
It’s imperative to secure senior-level
sponsorship to acquire the resources
needed for a successful project.
Address your biggest pain points first.
>
Look for ways to automate as many manual tasks as possible.
>
Leverage automation to drive compliance.
>
Dedicate enough resources to make your NCCM initiative a success.
>
Integrate your NCCM environment with trouble ticketing, change management,
>
and other disciplines.
5
tiPS
Se r v ic e A U to m A tio n
We are just beginning the next major phase of our NCCM initiative, which involves integrat-ing network change and configuration efforts with our help desk and change management systems. In the next phase of our implemen-tation, we want to automatically generate help desk tickets in response to compliance violations, incidents, and so forth. This will eliminate the manual submission of tickets and speed incident resolution. We can do the same type of integration with change manage-ment systems by issuing change tickets automatically and tracking the progress and completion of changes for auditing purposes. We also want to gather information from our devices and incorporate them into a configur ation management database (CMDB). The CMDB will capture and main-tain comprehensive information describing our IT environment. Consequently, it will provide a unified architecture for increasing operational efficiency and achieving and dem-onstrating compliance with internal policies and external standards and mandates.
ProvIDINg A solID FouNDATIoN
For THE FuTurE
Through automation, we have been able to improve internal IT efficiencies; ensure compliance with operational, regulatory, and security best practices; and reduce the risk of change on our business. Gone are the days when someone had to go to each device one by one to implement a change. Automation has eliminated human errors that often occur in manual environments. The result is better reliability and perform-ance and the ability to manage a growing infrastructure with our current staff.
Best of all, NCCM is enabling us to provide a solid foundation for future growth and innovation. By providing this foundation, the network management team is enabling Duke Energy to focus on future services that will drive customer satisfaction.
The CMDB will provide a unified architecture for
increasing operational efficiency and achieving
and demonstrating compliance with internal
policies and external standards and mandates.
ABouT THE AuTHor
Paul Edmunds is a senior systems programmer at Duke Energy in Charlotte,
N.C. He works in a system management tools group in IT Operations and is the product line manager for Duke’s network management and monitoring systems.
