VMware vRealize Automation 6.1/6.2
Logging Overview and Configuration for Log
Insight Content Pack
Distributed Architecture
2
SSO
vRA VAs
vRO
Infrastructure
Web / Manager LBs
Infrastructure
Web / Manager Servers
Infrastructure
Agents/DEMs
Clustered SQL
Server DB
Infrastructure
Fabric
NOTE: see the vRA Reference Architecture guide for detailed layouts.
Host Roles
•
SSO: Authentication from vCenter 5.5 SSO, SSO Identity Appliance or the
SSO standalone Windows host
•
vRA Virtual Appliance(s): host the CAFÉ & Code Stream services, embedded
Postgres DB instance and embedded vRO Instance; CAFÉ services can be
configured in distributed manner on multiple instances of the VA
•
vRO (External): for non-POC environments, an external vRO configuration can
be standalone or load-balanced with external DB
•
App Services: VA for application services components
•
Infrastructure Web Server: hosts web server UI, WAPI interface and Model
Manager
•
Manager Service: responsible for moving Infrastructure components through
their defined lifecycle
•
DEMs: Orchestrator & Workers – interacts with Fabric sources
Remote Logging
•
Operating Systems
–
Windows does not natively support syslog
–
Virtual Appliances (VAs) and Linux do support syslog
•
Log Insight Agent
–
Available for both Windows and Linux
–
Easy to deploy and configure; very lightweight
–
Ability to handle multiline messages and tag events
–
Properly handles log spikes and log rotation
–
Offers capabilities beyond those provided by syslog
•
Use of the Log Insight agent is
recommended
for all vRealize
Automation components (Windows and Linux)
•
The Log Insight agent configurations include custom Tags which are
leveraged in the vRA content pack. If not properly configured, some
queries may not work as expected.
Remote Logging, continued…
•
Log Insight Windows agent installation instructions:
http://pubs.vmware.com/log-insight-25/index.jsp?topic=
%2Fcom.vmware.log-insight.administration.doc
%2FGUID-455106F4-4C3D-47C1-8EF6-84992BCCEB05.html
•
Log Insight Linux agent installation instructions:
http://pubs.vmware.com/log-insight-25/index.jsp?topic=
%2Fcom.vmware.log-insight.administration.doc%2FGUID-DB4A27CF-BDA7-443F-94FB-AB9097AD8008.html
vCenter SSO
•
Identity VA (SSO VA & vCenter SSO)
–
/var/log/vmware/sso/*
•
Catalina.out (primary)
•
ssoAdminServer.log – user log in info here
•
vmware-identity-sts-perf.log
•
vmware-identity-sts.log
•
vmware-sts-idmd-perf.log
•
vmware-sts-idmd.err
•
vmware-sts-idmd.log
–
/var/log/messages – Active Directory connection info
•
Windows VIM on vCenter SSO
–
C:\ProgramData\VMware\CIS\logs\vmware-sso\
vRealize Automation – Virtual Appliances
•
vRealize Automation (vRA) & Code Stream (vRCS)
–
/var/log/vmware/vcac/catalina.out
–
/var/log/apache2/access_log
–
/var/log/apache2/ssl_request_log
–
/var/log/apache2/error_log
•
vRealize Orchestrator (embedded & external same location):
–
/var/log/vco/app-server/catalina.out
–
/var/log/vco/app-server/server.log
–
/var/log/vco/app-server/scripting.log
–
Individual plugins need to be configured for logging and may have different log locations
•
Application Services
–
/home/darwin/tcserver/darwin/logs/catalina.out
•
Artifactory (for Code Stream)
–
/storage/artifactory/home/logs/artifactory.log
–
/storage/artifactory/home/logs/access.log
–
/storage/artifactory/home/logs/request.log
–
/storage/artifactory/home/logs/import.export.log
vRealize Automation – Infrastructure
•
Exact logs & locations will depend on deployment type and
configuration; these are basic places to start!
•
Infrastructure Server (Web, Manager)
–
C:\Program Files (x86)\VMware\vCAC\Server\Logs\All
–
C:\Program Files (x86)\VMware\vCAC\Server\Config
Tool\Log\vCACConfiguration-<date>
–
C:\Program Files (x86)\VMware\vCAC\Server\Model Manager Data\Logs\
–
C:\Program Files (x86)\VMware\vCAC\Server\Model Manager Web\Logs\Repository
–
C:\Program Files (x86)\VMware\vCAC\Website\Logs\Web_Admin_All
vRealize Automation – Infrastructure, continued…
•
Some log directories and filenames are set during installation and will
depend on entered information. Information like
<THIS>
needs to be
replaced with entered information.
•
Agents
–
C:\Program Files (x86)\VMware\vCAC\Agents\<PLUGIN>\logs\<FILE>
•
<PLUGIN>
Examples: vSphereAgent, nsx, VC55Agent, VDIAgent
•
<FILE>
Examples: vSphereAgent, EpiPowerShellAgent, VdiPowerShellAgent
IMPORTANT:
The Agent name specified during installation dictates the value of
<PLUGIN>
•
DEMs
–
C:\Program Files (x86)\VMware\vCAC\Distributed Execution Manager\<DEM_NAME>
–
C:\Program Files (x86)\VMware\vCAC\Distributed Execution Manager\<DEO_NAME>
–
IMPORTANT:
The DEM/DEO name specified during installation dictates the value
vRealize Business
•
vRB Data Collector
–
/var/log/itbm-data-collector/catalina.out
–
/var/log/itbm-data-collector/itfm-vc-dc.log
–
/var/log/itbm-data-collector/localhost_access_log.*
–
/var/log/itbm-data-collector/vf.tc-events.txt
•
vRB Server
–
/var/log/itbm-server/audit.log
–
/var/log/itbm-server/catalina.out
–
/var/log/itbm-server/itfm-external-api.log
–
/var/log/itbm-server/itfm-reflib-update.log
–
/var/log/itbm-server/itfm.log
–
/var/log/itbm-server/localhost_access_log.*
–
/var/log/itbm-server/vcac.log
–
/var/log/itbm-server/vf.tc-events.txt
Log Insight Server-Side Agent Configuration
•
Log Insight agent configuration can be set client-side or server-side.
Server-side consists of three steps outlined below. The slide following
have client-side configurations.
1.
Enable vRO logging – see the vRO slide for configuration information
2.
Static configuration (copy and paste):
;;; vCenter SSO VCSA [filelog|vmw-sso] directory=/var/log/vmware/sso exclude=vmware-* event_marker=^(\[\d{4}-\d{2}-\d{2}|\d{2}-\w+-\d{4}) tags={"vmw_product":"sso”} [filelog|vmw-sso-sts-idmd-perf] directory=/var/log/vmware/sso include=vmware-sts-idmd-perf* event_marker=^\d{4}-\d{2}-\d{2}\s\S+\s\w+\s+\w+ tags={"vmw_product":"sso”} [filelog|vmw-sso-sts-perf] directory=/var/log/vmware/sso include=vmware-identity-sts-perf* event_marker=^\[\d{4}-\d{2}-\d{2}\s\S+\s\S+\s\S+\]\s+\w+ tags={"vmw_product":"sso”} [filelog|vmw-sso-sts-other] directory=/var/log/vmware/sso
Log Insight Server-Side Agent Configuration
;;; vCenter SSO Windows [filelog|vcenter-sso] directory=C:\ProgramData\VMware\CIS\logs\vmware-sso event_marker=^\[\d{1,2}/\d{1,2}/\d{4} \d{1,2}:\d{1,2}:\d{1,2} tags={"vmw_product":"sso"} [filelog|vcenter-sso-sts] directory=C:\ProgramData\VMware\CIS\runtime\VMwareSTS\logs event_marker=^\[\d{1,2}/\d{1,2}/\d{4} \d{1,2}:\d{1,2}:\d{1,2} tags={"vmw_product":"sso”} ;;; vRA [filelog|vra] directory=/var/log/vmware/vcac event_marker=^[^\d] tags={"vmw_product”:”vra”,"vmw_product_component”:”cafe”} [filelog|apache] directory=/var/log/apache2 event_marker=^[^\s] tags={“asf_product”:”http”} ;;; vRCS [filelog|vrcs] directory=/storage/artifactory/home/logs event_marker=^[^\d] tags={"vmw_product”:”vrcs”,"vmw_product_component”:”artifactory”} ;;; vRA APPD [filelog|vra-appd] directory=/home/darwin/tcserver/darwin/logs event_marker=^\w+\s\d{2}\s\d{4}\s\S+\s\w+\s+[\S+] tags={"vmw_product”:”vra”,"vmw_product_component":”appd"}
14
Log Insight Server-Side Agent Configuration
;;; Static vRA
[filelog|vra-agent-vsphere]
directory=C:\Program Files (x86)\VMware\vCAC\Agents\vSphereAgent\logs\ event_marker=^\[\d{1,2}/\d{1,2}/\d{4} \d{1,2}:\d{1,2}:\d{1,2}
tags={"vmw_product":”vra”,"vmw_product_component":"agent"} [filelog|vra-server]
directory=C:\Program Files (x86)\VMware\vCAC\Server\Logs\ include=*All.log;Repository.log
event_marker=^\[\w\w\w:\d{4}-\d{2}-\d{2}
tags={"vmw_product":”vra",”vmw_product_component":"server"} [filelog|vra-mm]
directory=C:\Program Files (x86)\VMware\vCAC\Server\Model Manager Web\Logs\ include=*All.log;Repository.log
event_marker=^\[\w\w\w:\d{4}-\d{2}-\d{2}
tags={"vmw_product":”vra",”vmw_product_component":"mm"} [filelog|vra-web]
directory=C:\Program Files (x86)\VMware\vCAC\Server\Website\Logs\ include=*All.log;Repository.log
event_marker=^\[\w\w\w:\d{4}-\d{2}-\d{2}
tags={"vmw_product":”vra",”vmw_product_component":"web"} [filelog|vra-install]
directory=C:\Program Files (x86)\VMware\vCAC\InstallLogs\ include=*All.log;Repository.log
event_marker=^\[\w\w\w:\d{4}-\d{2}-\d{2}
Log Insight Server-Side Agent Configuration
;;; vRB [filelog|vra-vrb-server] directory=/var/log/itbm-server event_marker=^[^\s] tags={"vmw_product":”vrb","vmw_product_component":"server”} [filelog|vra-vrb-data-collector] directory=/var/log/itbm-data-collector event_marker=^[^\s] tags={"vmw_product":”vrb","vmw_product_component":"data-collector"}3.
Dynamic configuration (modify everything like
<THIS>
):
;;; Dynamic vRA agent configuration;;; MANUAL CONFIGURATION CHANGES REQUIRED ;;; DO NOT JUST COPY AND PASTE THIS SECTION
;;; For every agent installed a new agent configuration section is required
;;; The name of the agent given during installation dictates the log directory name
[filelog|vra-agent-<AGENT_NAME>]
directory=C:\Program Files (x86)\VMware\vCAC\Agents\<AGENT_NAME>\logs\
event_marker=^\[\d{1,2}/\d{1,2}/\d{4} \d{1,2}:\d{1,2}:\d{1,2} tags={"vmw_product":”vra”,"vmw_product_component":"agent”}
Log Insight Server-Side Agent Configuration
;;; A DEM name can be specified during installation
;;; The name of the DEM given during installation dictates the log directory name ;;; If no name is given the DEM name is: DEM
[filelog|vra-dem]
directory=C:\Program Files (x86)\VMware\vCAC\Distributed Execution Manager\<DEM_NAME>\Logs\
include=*All.log;Repository.log
event_marker=^\[\w\w\w:\d{4}-\d{2}-\d{2}
tags={"vmw_product":”vra”,"vmw_product_component":"dem"} ;;; A DEO name can be specified during installation
;;; The name of the DEO given during installation dictates the log directory name ;;; If no name is given the DEO name is: DEO
[filelog|vra-deo]
directory=C:\Program Files (x86)\VMware\vCAC\Distributed Execution Manager\<DEO_NAME>\Logs\
include=*All.log;Repository.log
event_marker=^\[\w\w\w:\d{4}-\d{2}-\d{2}
vCenter SSO on VCSA
•
Log Insight agent configuration (recommended – copy and paste):
;;; vCenter SSO VCSA [filelog|vmw-sso] directory=/var/log/vmware/sso exclude=vmware-* event_marker=^(\[\d{4}-\d{2}-\d{2}|\d{2}-\w+-\d{4}) tags={"vmw_product":"sso”} [filelog|vmw-sso-sts-idmd-perf] directory=/var/log/vmware/sso include=vmware-sts-idmd-perf* event_marker=^\d{4}-\d{2}-\d{2}\s\S+\s\w+\s+\w+ tags={"vmw_product":"sso”} [filelog|vmw-sso-sts-perf] directory=/var/log/vmware/sso include=vmware-identity-sts-perf* event_marker=^\[\d{4}-\d{2}-\d{2}\s\S+\s\S+\s\S+\]\s+\w+ tags={"vmw_product":"sso”} [filelog|vmw-sso-sts-other] directory=/var/log/vmware/sso include=vmware-sts-idmd.*;vmware-identity-sts.* event_marker=^\[\d{4}-\d{2}-\d{2}\s\S+\s\S+\s\S+ tags={"vmw_product":"sso”}
•
Syslog configuration (restart syslog after changes):
–
/etc/syslog-ng/syslog-ng.conf
–
Set “destination logserver” to syslog host or Log Insight
vCenter SSO on Windows
•
Log Insight agent configuration (copy and paste):
;;; vCenter SSO Windows [filelog|vcenter-sso] directory=C:\ProgramData\VMware\CIS\logs\vmware-sso event_marker=^\[\d{1,2}/\d{1,2}/\d{4} \d{1,2}:\d{1,2}:\d{1,2} tags={"vmw_product":"sso"} [filelog|vcenter-sso-sts] directory=C:\ProgramData\VMware\CIS\runtime\VMwareSTS\logs event_marker=^\[\d{1,2}/\d{1,2}/\d{4} \d{1,2}:\d{1,2}:\d{1,2} tags={"vmw_product":"sso"}
vRealize Orchestrator (vRO)
•
Syslog configuration: Edit /etc/vco/app-server/log4j.xml
–
Edit section (remove comments and substitute
<HOST>
with Syslog or Log
Insight host):
<appender name="SYSLOG" class="org.apache.log4j.net.SyslogAppender"> <param name="Threshold" value="INFO"/>
<param name="Facility" value="LOCAL1"/>
<param name="SyslogHost" value=”<HOST>"/>
<param name="FacilityPrinting" value="false"/> <layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="vco: prio:%-5p thread:%t token:%X{token} wf:%X{workflowName} wfid:%X{workflow} user: %X{username} cat: %c{1} msg:%m%n"/>
</layout> </appender>
•
At end of config xml (/etc/vco/app-server/log4j.xml)
–
Edit section (remove comments for SYSLOG appender):
<root> <priority value="INFO" /> <appender-ref ref="CONSOLE" /> <appender-ref ref="FILE" /> <appender-ref ref="SYSLOG" /> <!-- <appender-ref ref="EVENT_LOG" /> --> </root>
20
vRealize Automation (vRA) & Code Stream
(vRCS)
•
Log Insight agent configuration (recommended – copy and paste):
;;; vRA
[filelog|vra]
directory=/var/log/vmware/vcac
event_marker=^[^\d]
tags={"vmw_product”:”vra”,"vmw_product_component”:”cafe”}
[filelog|apache]
directory=/var/log/apache2
event_marker=^[^\s]
tags={“asf_product”:”http”}
;;; vRCS
[filelog|vrcs]
directory=/storage/artifactory/home/logs
event_marker=^[^\d]
tags={"vmw_product”:”vrcs”,"vmw_product_component”:”artifactory”}
•
Syslog configuration (restart syslog after changes):
–
/etc/rsyslog.d/remote.conf
–
Add details for each log file (substitute
<HOST>
with Syslog or Log Insight
vRealize Automation, continued…
#
# vRA + vRCS log files
# Add to: /etc/rsyslog.d/remote.conf # Replace with Log Insight FQDN # Run: /etc/init.d/syslog restart # $ModLoad imfile # vRA $InputFileName /var/log/vmware/vcac/catalina.out $InputFileTag vcac: $InputFileStateFile stat-vcac-catalina1 $InputFileSeverity info $InputFileFacility local7 $InputRunFileMonitor $InputFileName /var/log/vco/app-server/catalina.out $InputFileTag vco: $InputFileStateFile stat-vco-catalina1 $InputFileSeverity info $InputFileFacility local7 $InputRunFileMonitor $InputFileName /var/log/apache2/access_log $InputFileTag apache: $InputFileStateFile stat-apache2-access1 $InputFileSeverity info $InputFileFacility local7 $InputRunFileMonitor $InputFileName /var/log/apache2/error_log $InputFileTag apache: $InputFileStateFile stat-apache2-error1 $InputFileSeverity error $InputFileFacility local7 $InputRunFileMonitor $InputFileName /var/log/apache2/ssl_request_log $InputFileTag apache: $InputFileStateFile stat-apache2-ssl1 $InputFileSeverity info $InputFileFacility local7 $InputRunFileMonitor
22
vRealize Automation, continued…
# vRCS $InputFileName /storage/artifactory/home/logs/artifactory.log $InputFileTag vrcs: $InputFileStateFile stat-vrcs-artifactory $InputFileSeverity info $InputFileFacility local7 $InputRunFileMonitor $InputFileName /storage/artifactory/home/logs/import.export.log $InputFileTag vrcs: $InputFileStateFile stat-vrcs-import-export $InputFileSeverity info $InputFileFacility local7 $InputRunFileMonitor $InputFileName /storage/artifactory/home/logs/access_log $InputFileTag vrcs: $InputFileStateFile stat-vrcs-access1 $InputFileSeverity info $InputFileFacility local7 $InputRunFileMonitor $InputFileName /storage/artifactory/home/logs/error_log $InputFileTag vrcs: $InputFileStateFile stat-vrcs-error1 $InputFileSeverity error $InputFileFacility local7 $InputRunFileMonitor# check for new lines every 10 seconds $InputFilePollInterval 10
Application Services
•
Log Insight agent configuration (recommended – copy and paste):
;;; vRA APPD
[filelog|vra-appd]
directory=/home/darwin/tcserver/darwin/logs
event_marker=^\w+\s\d{2}\s\d{4}\s\S+\s\w+\s+[\S+]
tags={"vmw_product”:”vra”,"vmw_product_component":”appd"}
•
Syslog configuration (restart syslog after changes):
–
/etc/syslog-ng/syslog-ng.conf
–
Add following details (substitute
<HOST>
with Syslog or Log Insight host at
end):
#
# APPD log files
# Add to: /etc/syslog-ng/syslog-ng.conf
# Replace with Log Insight FQDN
# Run: /etc/init.d/syslog restart
#
source appd {
file("/home/darwin/tcserver/darwin/logs/catalina.out" follow_freq(1) flags(no-parse) log_prefix("appd: "));
};
destination logserver { tcp("
<HOST>
" port (514)); };
log { source(appd); destination(logserver); };
log { source(src); destination(logserver); };
vRealize Automation Infrastructure
•
Log Insight agent configuration (copy and paste the static section, but
be sure to make changes to the dynamic section on next page):
;;; Static vRA agent configuration
;;; Just copy and paste the below configuration
[filelog|vra-agent-vsphere]
directory=C:\Program Files (x86)\VMware\vCAC\Agents\vSphereAgent\logs\ event_marker=^\[\d{1,2}/\d{1,2}/\d{4} \d{1,2}:\d{1,2}:\d{1,2}
tags={"vmw_product":”vra”,"vmw_product_component":"agent"} [filelog|vra-server]
directory=C:\Program Files (x86)\VMware\vCAC\Server\Logs\ include=*All.log;Repository.log
event_marker=^\[\w\w\w:\d{4}-\d{2}-\d{2}
tags={"vmw_product":”vra",”vmw_product_component":"server"} [filelog|vra-mm]
directory=C:\Program Files (x86)\VMware\vCAC\Server\Model Manager Web\Logs\ include=*All.log;Repository.log
event_marker=^\[\w\w\w:\d{4}-\d{2}-\d{2}
tags={"vmw_product":”vra",”vmw_product_component":"mm"} [filelog|vra-web]
directory=C:\Program Files (x86)\VMware\vCAC\Server\Website\Logs\ include=*All.log;Repository.log
event_marker=^\[\w\w\w:\d{4}-\d{2}-\d{2}
tags={"vmw_product":”vra",”vmw_product_component":"web"} [filelog|vra-install]
vRealize Automation Infrastructure, continued…
•
Log Insight agent configuration, continued…
;;; Dynamic vRA agent configuration;;; MANUAL CONFIGURATION CHANGES REQUIRED ;;; DO NOT JUST COPY AND PASTE THIS SECTION
;;; For every agent installed a new agent configuration section is required
;;; The name of the agent given during installation dictates the log directory name
[filelog|vra-agent-<AGENT_NAME>]
directory=C:\Program Files (x86)\VMware\vCAC\Agents\<AGENT_NAME>\logs\
event_marker=^\[\d{1,2}/\d{1,2}/\d{4} \d{1,2}:\d{1,2}:\d{1,2} tags={"vmw_product":”vra”,"vmw_product_component":"agent"} ;;; A DEM name can be specified during installation
;;; The name of the DEM given during installation dictates the log directory name ;;; If no name is given the DEM name is: DEM
[filelog|vra-dem]
directory=C:\Program Files (x86)\VMware\vCAC\Distributed Execution Manager\<DEM_NAME>\Logs\
include=*All.log;Repository.log
event_marker=^\[\w\w\w:\d{4}-\d{2}-\d{2}
tags={"vmw_product":”vra”,"vmw_product_component":"dem"} ;;; A DEO name can be specified during installation
;;; The name of the DEO given during installation dictates the log directory name ;;; If no name is given the DEO name is: DEO
[filelog|vra-deo]
directory=C:\Program Files (x86)\VMware\vCAC\Distributed Execution Manager\<DEO_NAME>\Logs\
include=*All.log;Repository.log
event_marker=^\[\w\w\w:\d{4}-\d{2}-\d{2}
tags={"vmw_product":”vra”,"vmw_product_component":"deo"}
vRealize Business Standard
•
Log Insight Agent configuration (copy and paste):
;;; vRB [filelog|vra-vrb-server] directory=/var/log/itbm-server event_marker=^[^\s] tags={"vmw_product":”vrb","vmw_product_component":"server”} [filelog|vra-vrb-data-collector] directory=/var/log/itbm-data-collector event_marker=^[^\s] tags={"vmw_product":”vrb","vmw_product_component":"data-collector"}
Aggregated Logs in Log Insight
Content Pack for vCAC 6.0 and vRA 6.1 or newer available on VMware
