TIBCO LogLogic® Unity ̶
Quick Reference Guide
Concepts
Overview
Taxonomy
TIBCO LogLogic® Unity is a sleek, modern and
scalable platform enabling technical teams to resolve
open issues, which require advanced
troubleshooting techniques, complex root cause
analysis or deep forensics. LogLogic Unity is a Log
processing Search & Alerting tool that takes data
from any source and structures that data. This
allows for intuitive, fast, and complete interaction
with data resulting in faster turn-around from open
to close in issue resolution.
Highlights:
Modular search queries – Use all or part of saved search filters to build new search queries using new building block technology
Multiple search queries – Run multiple searches at the same time
Working data sets – Work with multiple search results without losing what you are working on. Walk away and come back without losing your search results. Data lookups – Enrich your experience with lookup
tables enhancing search and alerting capabilities Data at rest correlation – Perform advanced
correlation against historical data to identify trends Data in motion correlation – Maintain advanced
correlation in memory to identify key patterns for alerting
Comprehensive APIs – Leverage core functionality using intuitive APIs built on REST
Scalable clustering technology – Scale horizontally as needed to maintain performance and storage
LogLogic leverages a defined taxonomy for proper field
normalization. This enables users to quickly map fields together
from different sources into a common schema.
Deployment
LogLogic Unity architectural view is shown in the following
illustration:
Parsing
Events
A parsing rule relates to the ability to structure data
from its original state. By providing structure a user
can then perform other functions such as grouping,
aggregations or visuals. Parsing also assists in
finding information when performing searching. In
LogLogic 6 parsing can be performed ad-hoc
producing a flexible design to work with. LogLogic 6
also comes pre-loaded with 100’s of rules. It also
provides KVP and CSV style parsers to easily extract
columns for semi-structured data.
An event in the system represents a single entry of data. Events may
be small single line entries such as syslog or they may consume
multiple lines such as JSON outputs or XML outputs.
System Tags
System tags are predefined with “sys_”. These tags are auto defined
for each log message and are indexed by the system. This enables
faster search queries and access to the data.
Field Enrichment
Field enrichment provides a way for a user to edit any field in the
system to produce outputs that are informative and intuitive. We
support lookups, math computations and standard functions to help
users define data the way they need to see it.
Filter
Bloks
Common Search Commands
Operations and functions Definition
USE Defines event sources including parsing configuration. COLUMNS Defines which columns should appear in the results.
GROUP BY Grouping can be used to group values by given column or columns. Grouping requires list of grouping columns and the list of aggregation columns.
SORT BY Sorts search results based on the expression. LIMIT Limits the size of search results to be displayed.
Common Filter Commands/Statements/Operations
Operations and functions Definition
AND Narrows your search results by only returning those events where each one of the AND conditions evaluates to true.
OR Expands your search results by returning events where either of the OR conditions evaluates to true. Equals (=) A comparison condition compares two expressions using the operator specified in the comparison, which may
be one of seven possible comparison operators with well-known meanings. The comparison condition evaluates to true only if the comparison condition is satisfied. This may be used to narrow search results. Not equals (<>), (!=)
Lower than(<) Lower or equal (,=) Greater than (>) Greater or equal (>=)
Plus (+) The arithmetic (+,-,*,/) and string concatenation (||) operators can be used to create parts of other conditions. Minus (-)
Multiply (asterisk (*)) Divide (forward slash (/)) String concatenation (||)
Function Any set of predefined functions.
IS NULL, IS NOT NULL Narrows your search results by accepting or rejecting the event based on whether the evaluated expression is null or not null. An expression most frequently becomes null if a column named in the expression has no value for the current event.
LIKE, NOT LIKE Returns true if it matched the supplied pattern (% matches any character, _ matches exactly one character). BETWEEN Supports Timestamps, Long and Integers.
IN Checks if value matches anyone of the values in a set. REGEXP, NOT REGEXP Returns true if it matches the supplied pattern.
Search Examples
Filter Blok Expression
Definition
sys_eventTime in TUE:WED Displays results from the Default data parser profile within defined time range. Use MyParserProfile | sys_eventTime in -1h Displays results from a defined data Parser Profile for a specified time range. "bob" Displays results from the default log source with Bob in the text.
Use system | sys_eventTime in 1d:NOW Displays results from the System Data Parser profile within a certain time range. "bob" and devType in
('Windows', 'Cisco PIX')
Displays results from a data source with 'bob' in the text, and for a certain set of log sources. sort by sys_eventTime DESC |
LIMIT 100
Displays sorted first page of results for events ordered by timestamp in descending order. "Bob" | sys_eventTime BETWEEN
'2012-02-14 '2012-02-14:34:34' and '2012-03-'2012-02-14 12:00:00'
Display results from the System log source with Bob in the text sys_eventTime in -1h | sort by
sys_eventTime DESC
Displays sorted events by timestamp in descending order. sys_eventTime in -1h | group
by transactionID columns COUNT()
Displays grouped results based on transaction.
sys_eventTime in -1h | group by transactionID columns max(sys_eventTime) - min(sys_eventTime) as duration
Displays grouped results based on the transaction durations. Provides an alias for the expression that will be used as column name.
Aggregation Functions
Function
Definition
(*) This will apply the function for any event with no additional constraints. All This will apply the function on all values that are not null.
Distinct This will apply the function once per distinct values. Sum This is the total value.
Avg This is the average value. Max This is the minimum value. Var This is the variance.
Correlation Bloks
Functions
Arguments
Returns
String functions
lenchar_lengthcharacter_length String Length of string argument 1
lower String Lower case of string 1
upper String Upper case of string 1
trim String Trimmed string 1 (without leading and trailing spaces) substitute String 1, String 2, String 3 Substitute string 2 by string 3 in string 1
left String, Int <int> left characters of string 1 right String, Int <int> right characters of string 1
midsubstrsubstring String, Int 1, Int 2 Characters from string 1 starting at offset <int1> for a length of <int2>
findposition String, String Index of the first occurrence of string 2 within string 1, -1 if no occurrence is found concatenate String, String, … Concatenation of all strings passed as arguments.
List functions
size List Size of list
Conditional functions
IIF Condition, then, else Returns then value if condition else, then returns else valve. Smart List functions
lookup String 1, String 2 The value associated with String2 in the smart list named String1. isInList String 1, String 2 True if the value String2 is defined in smart list named String1. Conversion functions
ToTimestamp expression, formatString or expression, formatString defaultValue
The expression, which should evaluate to a string, is interpreted as a time according to the supplied formatString. If the conversion fails, null is retured, unless a default string is provided, which is interpreted as a time and returned.
ToIP expression or expression,
defaultValue Same as ToTimestamp, except the conversion is to an IP address (Java InetAddress). ToTimestampString expression, formatString or
expression,
formatString,defaultValue
Same as ToTimestamp, except the conversion is in the opposite direction to get a printable timestamp.
ToInt expression or expression,
defaultValue The obvious conversion to integer with default value taken if not convertible. ToLong expression or expression,
defaultValue The obvious conversion to Long with default value taken if not convertible. ToString expression or expression,
defaultValue The obvious conversion to String with default value taken if not convertible. ToFloat expression or expression,
defaultValue The obvious conversion to Float with default value taken if not convertible. ToBool expression or expression,
defaultValue The obvious conversion to Boolean with default value taken if not convertible. ToDouble expression or expression,
defaultValue The obvious conversion to Double with default value taken if not convertible.
System Columns
The system (event metadata) columns are indexed so searching is faster on the system columns. All system columns are displayed with the prefix sys_ and all built-in parsers are displayed with the prefix ll_ in the column list. The following list describes all system columns in the LogLogic Unity event.
Name
Type
Column
sys_eventTime Timestamp UTC time of the event in Epoch milliseconds. sys_body String Text of the message.
sys_bodySize Integer Size in number of bytes of the body.
sys_sourceType Integer TIBCO LogLogic® Log Management Intelligence (LMI) type ID. sys_collectIP InetAddress IP from where the event originated.
sys_collectTime TimeStamp UTC time of the event when it was ingested into the LogLogic Unit event storage. sys_filename String File name for event collected from a file.
sys_fileLineNumber Integer Line number in file. sys_tenant String Customer identifier. sys_domain String Customer sub-identifier.
sys_partition Long Identifier of the portion of the data on the data node. sys_offset Long Location of the LogLogic Unity event store.
sys_eventKey String Unique key that refers to an event in the LogLogic Unity store. sys_lmiEventKey String Unique key that refers to an event in the LogLogic LMI event store. sys_ApplianceId String Identifier for the LMI Appliance.
sys_lmiDomain Integer, String LMI Domain is a component of the LMI device (source) identifier. sys_sourceDns String DNS name for the event_source_ip.
Rule Structure
Statement
Definition
Rule Structure USE <source identifier> (, <source identifier>)* Within <integer> [ d |h | m | s ] [ Fixed | Sliding ]
<event group 1> <event group 2> … [ Correlation <correlation criteria 1> <correlation criteria 2> … ] [ Autofill ]
( Set <expression> As <identifier> )*
[ Inject Correlation Event ]
[ LIMIT <integer> CORRELATION EVENTS ]
Event group structure Event Group <identifier> [ Is ( Required | Optional | Excluded ) ]
[ With Delayed Evaluation ] [ At Least <integer> Events ] [ At Most <integer> Events ] [ <identifier environment> ] [ Where <expression> ]
[ With The Same <expression> [ As <identifier> ] ( , <expression> [ As <identifier> ] )* ]
( Having <having clause> )*
[ Limits <integer> Groups And <integer> Events ]
Having Clause At (Least | Most) <integer> Distinct <expression> As <identifier> Limit <integer> Count Of <expression> Being <expression> (Greater | Lower) Than <integer>
Percentage Of <expression> Being <expression> (Greater | Lower) Than <integer>%<condition>
Correlation Criteria <event_group_identifier1>><field_identifier1> == <event_group_identifier2>><field_identifier2>
<event_group_identifier1> (Begins | Ends) [At Least <integer> [ d | h | m | s ]] [Up To <integer> [ d |h | m | s ]] (Before | After) <event_group_identifier2> (Begins|Ends)
Search Examples
Correlation Blok Example
Definition
Use systemWithin 30m
Event Group [My Events]
This rule will trigger a new alert at the first event and will accumulate all events during 30 minutes time period.
Use my source Within 30m
Event Group [My Events] Having at least 1 distinct Operation limit 1000
Having at least 1 distinct Direction limit 1000
This rule will do the same as rule 1 but the alerts generated will then give information about the number of distinct operation/direction and their values.
Use MySource Within 30m
Event Group [Builts]
where Operation = "Built" and Direction = "outbound"
Having at least 1 distinct [Source IP] limit 1000
This rule will filter events which have Operation equals to “Built” and Direction equals to “outbound”, and give count of distinct source IP and values up to 1000.
Use MySource Within 30m
Event Group [Builts] At least 41 events
where Operation = "Built" and Direction = "outbound"
With the same [Source IP] Having at least 1 distinct [Destination IP] limit 1000
This rule is looking for at least 41 events with the same criteria as the previous one, coming from the same Source IP, and giving information about the number of distinct Destination IPs and their value up to 1000.
Use MySource Within 30m
Event Group [DenyAndBuilt] where Operation = "Deny" OR Operation = "Built"
With the same [Source IP] Having at least 2 distinct [Operation] limit 1000
This rule looks for a Source IP which has events from at least 2 distinct operations.
