openNMS reporting generation tool
Juan Pedro Escalona
DevOps Southampton, UK - 2014
Juan Pedro Escalona
DevOps / Systems Administrator with over 6 years experience administering different OS, network systems and anything that is thrown to my desk.
At the present, I work in QA Automation & Performance group, providing tools to automate processes.
Monitoring lover, background in several Open Source monitoring solutions. Six months ago, these words changed me:
- “Hi Juan Pedro. Have you met openNMS? Let me introduce you...”
Agenda
● Introduction
● Load Test Processes
● Automating processes progress
● Reporting Tool in Load test processes ● Python openNMS Library
● Demo ● Liveview ● Results ● Q & A
Agenda
● Introduction
● Load Test Processes
● Automating processes progress
● Reporting Tool in Load test processes ● Python openNMS Library
● Demo ● Liveview ● Results ● Q & A
Introduction
● Over 1000 servers in production
● Exact copy of production environment for test
● Releases every 2 weeks
● Load Test is mandatory
Agenda
● Introduction
● Load Test Processes
● Automating processes progress
● Reporting Tool in Load test processes ● Python openNMS Library
● Demo ● Liveview ● Results ● Q & A
Launch Load Test
Prepare Load Test Report Load Test
Load Test Processes
● Prepare Load Test:
Load Test Processes
● Prepare Load Test:
Set initial configuration in order to replicate production status (ie. Refresh DBs)
● Launch Load Test:
Most of load generators and monitors are compatible only with Windows OS Webload & Neoload + 25 Load Generators (Windows / Linux)
Load Test Processes
● Prepare Load Test:
Set initial configuration in order to replicate production status (ie. Refresh DBs)
● Launch Load Test:
Most of load generators and monitors are compatible only with Windows OS Webload & Neoload + 25 Load Generators (Windows / Linux)
● Report Load Test:
openNMS: Main key metrics of servers and graphs
Agenda
● Introduction
● Load Test Processes
● Automating processes progress ● Reporting Tool in Load test processes
● Python openNMS Library ● Demo
● Liveview ● Results ● Q & A
Automating Processes Progress
● Prepare Load Test:
It's complex but for the time being we use scripts to do it Orchestration: check dependencies + call scripts
Automating Processes Progress
● Prepare Load Test:
It's complex but for the time being we use scripts to do it Orchestration: check dependencies + call scripts
● Launch Load Test:
Generate load templates based on load profiles
Automating Processes Progress
● Prepare Load Test:
It's complex but for the time being we use scripts to do it Orchestration: check dependencies + call scripts
● Launch Load Test:
Generate load templates based on load profiles
Run/stop and control status of Load Generator Consoles
● Report Load Test:
Schedule report generation or trigger when load test has finished
Aggregate different servers with different type of graphs in same report Provide a single view to check status of all nodes we're testing
Automating Processes Progress > Report
●
At the beginning:
Time: 10-20 minutes
No health key metrics of server
Site Catalyst
Automating Processes Progress > Report
●
Spreadsheet reports:
Time: From 1 hour to days
OpenNMS: health statistics
Screenshot + Cut + Paste (1 node)
Automating Processes Progress > Report
●
KSC Reports:
Time: 30 minutes to several hours
Join different servers + Graphs
Save as HTML
Automating Processes Progress > Report
●
Report generation is not automatic
●
Not possible join several KSC Reports in once
●
KSC Report configuration is not easy
Agenda
● Introduction
● Load Test Processes
● Automating processes progress
● Reporting Tool in Load test processes ● Python openNMS Library
● Demo ● Liveview
● Improvements & Results ● Q & A
Reporting Tool in Load Test Processes
●
Key features
Support multiple environments (a.k.a. multiple OpenNMS instances)
Easy configuration building new Server Reports ( hosts + graphs)
Easy report generation: few clicks + few minutes
Liveview: lets you aggregate different Server Reports in a same view
just on click
Reporting Tool in Load Test Processes
●
Architecture and components:
Python OpenNMS Client module: Retrieve info from openNMS
Contributed by Jason Viloria
Memcache:
Cache all info obtained from openNMS
Django Application:
Provide UI interface
Agenda
● Introduction
● Load Test Processes
● Automating processes progress
● Reporting Tool in Load test processes ● Python openNMS Library
● Demo ● Liveview
● Improvements & Results ● Q & A
Python openNMS library
Retrieve the list of available nodes in OpenNMS instances:
●Assets: Node label, node id, categories
●
Node Resources: JMX, SNMP
●Graphs
●
Node Level: CPU, Memory
●
Interface Level: Net Interface Bandwidth, Disk IO
●Caching
Python openNMS library
Python openNMS library
Python openNMS library
Python openNMS library
Agenda
● Introduction
● Load Test Processes
● Automating processes progress
● Reporting Tool in Load test processes ● Python openNMS Library
● Demo ● Liveview
● Improvements & Results ● Q & A
Agenda
● Introduction
● Load Test Processes
● Automating processes progress
● Reporting Tool in Load test processes ● Python openNMS Library
● Demo ● Liveview
● Improvements & Results ● Q & A
Agenda
● Introduction
● Load Test Processes
● Automating processes progress
● Reporting Tool in Load test processes ● Python openNMS Library
● Demo ● Liveview
● Improvements & Results ● Q & A
Report Generation OpenNMS graphs Analysis 0 1 2 3 4 5 6 7 2 6 1 1 0.1 2
Time Spent per Stage
2012 2013 0 2 4 6 8 10 9 3
Overall Reporting Time
Jan Feb Mar Apr May Jun Jul Aug Sep Oct 0 20 40 60 80 100 120 30 24 17 32 24 39 60 105 94 65
Load Test Executions
Load Tests Capacity
Executions x 3.5 since January:
• January 2013 (1st Big Event): 30 executions • August 2013 (2nd Big Event): 105 executions
Results Analysis:
2 X analysis time
1/3 overall report time
Improvements & Results
At the beginning Spreadsheets KSC Reports Reporting Tool
● No server key
metrics
● Key metrics (1 node) ● Manual process
● Much time
● Key metrics (all) ● Save time
● Still manually
● Key metrics (all) ● Save huge time ● Automatic