Juniper Secure Analytics

(1)

Juniper Secure Analytics

Big Data Management Guide

Release

2014.1

Published: 2014-03-17

(2)

Juniper Networks, Junos, Steel-Belted Radius, NetScreen, and ScreenOS are registered trademarks of Juniper Networks, Inc. in the United States and other countries. The Juniper Networks Logo, the Junos logo, and JunosE are trademarks of Juniper Networks, Inc. All other trademarks, service marks, registered trademarks, or registered service marks are the property of their respective owners.

Juniper Networks assumes no responsibility for any inaccuracies in this document. Juniper Networks reserves the right to change, modify, transfer, or otherwise revise this publication without notice.

The information in this document is current as of the date on the title page.

YEAR 2000 NOTICE

Juniper Networks hardware and software products are Year 2000 compliant. Junos OS has no known time-related limitations through the year 2038. However, the NTP application is known to have some difficulty in the year 2036.

END USER LICENSE AGREEMENT

The Juniper Networks product that is the subject of this technical documentation consists of (or is intended for use with) Juniper Networks software. Use of such software is subject to the terms and conditions of the End User License Agreement (“EULA”) posted at

http://www.juniper.net/support/eula.html. By downloading, installing or using such software, you agree to the terms and conditions of that EULA.

(3)

List of Tables

About the Documentation . . . vii

Table 1: Notice Icons . . . viii

Table 2: Text and Syntax Conventions . . . viii

Part 1 Juniper Secure Analytics Big Data Management

Chapter 1 JSA and Big Data Integration . . . 3

Table 3: Integration Components . . . 4

Table 4: Forwarding Destinations Parameters . . . 5

Table 5: Routing Rules Window Parameters . . . 8

(6)

(7)

About the Documentation

• Documentation and Release Notes on page vii

• Documentation Conventions on page vii

• Documentation Feedback on page ix

• Requesting Technical Support on page x

Documentation and Release Notes

To obtain the most current version of all Juniper Networks®technical documentation, see the product documentation page on the Juniper Networks website at

http://www.juniper.net/techpubs/.

If the information in the latest release notes differs from the information in the documentation, follow the product Release Notes.

Juniper Networks Books publishes books by Juniper Networks engineers and subject matter experts. These books go beyond the technical documentation to explore the nuances of network architecture, deployment, and administration. The current list can be viewed athttp://www.juniper.net/books.

Documentation Conventions

Table 1 on page viiidefines notice icons used in this guide.

(8)

Table 1: Notice Icons

Description Meaning

Icon

Indicates important features or instructions. Informational note

Indicates a situation that might result in loss of data or hardware damage. Caution

Alerts you to the risk of personal injury or death. Warning

Alerts you to the risk of personal injury from a laser. Laser warning

Table 2 on page viiidefines the text and syntax conventions used in this guide.

Table 2: Text and Syntax Conventions

Examples Description

Convention

To enter configuration mode, type the configurecommand:

user@host>configure Represents text that you type.

Bold text like this

user@host> show chassis alarms No alarms currently active Represents output that appears on the

terminal screen. Fixed-width text like this

• A policytermis a named structure that defines match conditions and actions.

• Junos OS CLI User Guide

• RFC 1997,BGP Communities Attribute

• Introduces or emphasizes important new terms.

• Identifies guide names.

• Identifies RFC and Internet draft titles. Italic text like this

Configure the machine’s domain name: [edit]

root@#set system domain-name

domain-name Represents variables (options for which

you substitute a value) in commands or configuration statements.

Italic text like this

• To configure a stub area, include the

stubstatement at the[edit protocols ospf area area-id]hierarchy level.

• The console port is labeledCONSOLE. Represents names of configuration

statements, commands, files, and directories; configuration hierarchy levels; or labels on routing platform

components.

Text like this

stub <default-metricmetric>;

Encloses optional keywords or variables. < > (angle brackets)

(9)

Table 2: Text and Syntax Conventions

(continued)

Examples Description

Convention

broadcast | multicast (string1|string2|string3)

Indicates a choice between the mutually exclusive keywords or variables on either side of the symbol. The set of choices is often enclosed in parentheses for clarity. | (pipe symbol)

rsvp { # Required for dynamic MPLS only

Indicates a comment specified on the same line as the configuration statement to which it applies.

# (pound sign)

community name members [

community-ids]

Encloses a variable for which you can substitute one or more values. [ ] (square brackets)

[edit]

routing-options { static {

route default { nexthop address; retain;

} } } Identifies a level in the configuration

hierarchy. Indention and braces ( { } )

Identifies a leaf statement at a configuration hierarchy level. ; (semicolon)

GUI Conventions

• In the Logical Interfaces box, select

All Interfaces.

• To cancel the configuration, click

Cancel. Represents graphical user interface (GUI)

items you click or select. Bold text like this

In the configuration editor hierarchy, selectProtocols>Ospf.

Separates levels in a hierarchy of menu selections.

>(bold right angle bracket)

Documentation Feedback

We encourage you to provide feedback, comments, and suggestions so that we can improve the documentation. You can provide feedback by using either of the following methods:

• Online feedback rating system—On any page at the Juniper Networks Technical Documentation site athttp://www.juniper.net/techpubs/index.html, simply click the stars to rate the content, and use the pop-up form to provide us with information about your experience. Alternately, you can use the online feedback form at

https://www.juniper.net/cgi-bin/docbugreport/.

• E-mail—Send your comments totechpubs-comments@juniper.net. Include the document or topic name, URL or page number, and software version (if applicable).

(10)

Requesting Technical Support

Technical product support is available through the Juniper Networks Technical Assistance Center (JTAC). If you are a customer with an active J-Care or JNASC support contract, or are covered under warranty, and need post-sales technical support, you can access our tools and resources online or open a case with JTAC.

• JTAC policies—For a complete understanding of our JTAC procedures and policies, review theJTAC User Guidelocated at

http://www.juniper.net/us/en/local/pdf/resource-guides/7100059-en.pdf.

• Product warranties—For product warranty information, visit

http://www.juniper.net/support/warranty/.

• JTAC hours of operation—The JTAC centers have resources available 24 hours a day, 7 days a week, 365 days a year.

Self-Help Online Tools and Resources

For quick and easy problem resolution, Juniper Networks has designed an online self-service portal called the Customer Support Center (CSC) that provides you with the following features:

• Find CSC offerings:http://www.juniper.net/customers/support/

• Search for known bugs:http://www2.juniper.net/kb/

• Find product documentation:http://www.juniper.net/techpubs/

• Find solutions and answer questions using our Knowledge Base:http://kb.juniper.net/

• Download the latest versions of software and review release notes:

http://www.juniper.net/customers/csc/software/

• Search technical bulletins for relevant hardware and software notifications:

https://www.juniper.net/alerts/

• Join and participate in the Juniper Networks Community Forum:

http://www.juniper.net/company/communities/

• Open a case online in the CSC Case Management tool:http://www.juniper.net/cm/

To verify service entitlement by product serial number, use our Serial Number Entitlement (SNE) Tool:https://tools.juniper.net/SerialNumberEntitlementSearch/

Opening a Case with JTAC

You can open a case with JTAC on the Web or by telephone.

• Use the Case Management tool in the CSC athttp://www.juniper.net/cm/.

• Call 1-888-314-JTAC (1-888-314-5822 toll-free in the USA, Canada, and Mexico). For international or direct-dial options in countries without toll-free numbers, see

http://www.juniper.net/support/requesting-support.html.

(11)

PART 1

Juniper Secure Analytics Big Data

Management

• JSA and Big Data Integration on page 3

(12)

(13)

CHAPTER 1

JSA and Big Data Integration

This chapter describes about the following sections:

• Configuring JSA and Big Data Integration on page 3

• Installing the Data Management Application Server Plug-In and Reference Data API on page 4

• Adding Forwarding Destinations on page 5

• Customizing the Forwarding Profile on page 6

• Configuring Routing Rules for Bulk Forwarding on page 7

Configuring JSA and Big Data Integration

To build insights from broader data sets, integrate Juniper Secure Analytics (JSA) with vendor data management applications, such as IBM InfoSphere BigInsights.

Integration process involves the following steps:

1. Install and configure your data management application on a separate server from JSA. For more information about how to install and configure your application, see your vendor documentation.

2. Configure JSA to forward raw data to the server.

3. Use the data management application to transform and enrich the data for analytic information.

4. Configure the API to send the analytic insights back to JSA.

Integration Components

Table 3 on page 4lists the components that are used to integrate the data management application with JSA.

(14)

Table 3: Integration Components

Description Components

A software application that processes and analyzes the volume, variety, and velocity of data. You must purchase your data management application separately. Although JSA can integrate with any data management application that supports Apache Hadoop, supports IBM InfoSphere BigInsights.

Data management application

A utility that JSA uses to retrieve data from a data management application.

Reference data API

A JSA component that specifies the server that hosts the data management application.

Forwarding destination

A JSA file that is automatically generated when you create a forwarding destination that uses the JSON format.

Forwarding profile

A JSA component that you use to create filter-based routing rules to forward large quantities of data. Use routing rules to specify what data is forwarded to the forwarding destination.

Routing rule

Example: Using Big Data Technologies for Advanced Data Threats

Use JSA and big data management application integration to periodically identify suspicious external domains that your systems or users are connecting to. This suspicious activity is based on inconsistencies in registration info and the correlation between actual data that is associated with the domain and actual data that is associated with some external malware domains. Your data management application receives and analyzes data from JSA and external data sources. The results are then sent back to your JSA system.

Installing the Data Management Application Server Plug-In and Reference Data API

To install the data management application server Plug-In and reference data API: 1. Log in to your Juniper Secure Analytics (JSA) console.

2. On the Admin tab, perform the following steps:

a. ClickUser Rolesand create a dedicated user role that includes theReference Data APIrole permission.

b. ClickAuthorized Servicesand create add an authorized service and assign the authorized service to the dedicated user role.

(15)

3. Download the data management application plug-in and Reference Data API RPM files fromhttp:// www.juniper.net/customers/support.

4. Install the plug-in and API on your JSA console.

•

Adding Forwarding Destinations

Before you can configure bulk or selective data forwarding, you must add forwarding destinations.

To add the forwarding destinations: 1. Click the Admin tab.

2. In the navigation pane, clickSystem Configuration. 3. Click the Forwarding Destinations icon.

4. On the toolbar, clickAdd.

5. In the Forwarding Destinations window, enter values for the parameters. Table 4 on page 5describes the forwarding destinations parameter.

Table 4: Forwarding Destinations Parameters

Description Parameter

• Payloadis the data in the format that the log source or flow source sent.

• Payloadis the data in the format that the log source sent.

• Normalizedis raw data that is parsed and prepared as readable information for the user interface.

• JSON(Javascript Object Notation) is a data-interchange format.

NOTE: JSON data can only be transmitted using the TCP protocol. Event Format

The IP address or host name of the vendor system that you want to forward data to. Destination Address

• TCP

Use theTCPprotocol to send normalized data by using the TCP protocol, you must create an off-site source at the destination address on port 32004.

• UDP

NOTE: You cannot transmit normalized and JSON data by using the UDP protocol. If you select theNormalized EventorJSONoptions, theUDPoption in the Protocol list is disabled.

Protocol

(16)

Table 4: Forwarding Destinations Parameters

(continued)

If a valid syslog header is not detected on the original syslog message, select this check box. The prefixed syslog header includes the JSA appliance host IP address in the Hostname field of the syslog header. If this check box is not selected, the data is sent unmodified.

When JSA forwards syslog messages, the outbound message is verified to ensure that it has a valid syslog header.

Prefix a syslog header if it is missing or invalid

6. ClickSave.

•

Customizing the Forwarding Profile

Customize the forwarding profile that is automatically created when you create a forwarding destination that sends data in the JSON format.

When you create a forwarding destination that sends data in JSON format, a forwarding profile is automatically created and stored in the/opt/qradar/conffolder on your system. The file name for each profile is appended with a number that relates to the forwarding destination ID. For example, forwardingprofile_jason.xml.1. You can edit the forwarding profile to transform and enrich the data to provide more information, which supports the advanced analytical processes.

The forwarding profile controls what data is forwarded, how default settings are applied, and whether to include custom properties. You can customize the name and version of the profile, and edit the pre-amble element. The preamble contains information that is required for the forwarding profile to interact with multiple receivers.

For each attribute listed in the forwarding profile, you can also customize the properties. The following is an example of a flow-specific attribute:

<attribute tag="firstPacketTimeISO" enabled="true" name="firstPacketTimeISO"

defaultValue="" enableDefaultValue="false"></attribute>

• Use theenabledproperty to specify that you want to send the field to the forwarding destination.

• Update thenameproperty from the automatically generated name.

(17)

• Specify thedefault Valueproperty for the field.

• Use theenable Default Valueproperty to enable or disable thedefault Valueproperty. The JSON forwarding mechanism sends only default values for a field if the enable Default Value is set to true.

To customize and forward the profile: 1. Log in to your system.

2. Change directories to/opt/qradar/conf.

3. Locate and open the forwarding profile file that you want to customize. 4. Customize the forwarding profile by using the following options:

• Edit the profile level options, such as the name, version, or pre-amble.

• To forward custom properties, ensure that theinclude Custom Propertiesproperty isTruein the Header section.

• To configure the time format, edit the isoTime Formatattribute in the Header section.

• In each of the attribute sections, edit the attributes that you want to enhance. 5. Close the file.

•

Configuring Routing Rules for Bulk Forwarding

After you added one or more forwarding destinations, you can create filter-based routing rules to forward large quantities of data.

You can configure routing rules to forward data in either online or offline mode:

• In Online mode, your data remains current because forwarding is performed in real time. If the forwarding destination becomes unreachable, data can potentially be lost.

• In Offline mode, all data is stored in the database and then sent to the forwarding destination. This assures that no data is lost, however, there might be delays in data forwarding.

Table 5 on page 8describes theRouting Rulesparameter.

(18)

Table 5: Routing Rules Window Parameters

This option is displayed when you select the Online option.

Specifies the Event Collector that you want this routing rule process data from. Forwarding Event

Collector

This option is displayed when you select the Offline option.

Specifies the Event Processor that you want this routing rule process data from.

NOTE: This option is not available ifDropis selected from the Routing Options pane. Forwarding Event

Processor

• TheForwardoption specifies that data is forwarded to the specified forwarding destination. Data is also stored in the database and processed by the Custom Rules Engine (CRE).

• TheDropoption specifies that data is not stored in the database and is not processed by the CRE. The data is not forwarded to a forwarding destination, but it is processed by the CRE. This option is not available if you select the Offline option.

• TheBypass Correlationoption specifies that data is not processed by the CRE, but it is stored in the database. This option is not available if you select the Offline option.

You can combine two options:

• ForwardandDrop

Data is forwarded to the specified forwarding destination. Data is not stored in the database and is processed by the CRE.

• ForwardandBypass Correlation

Data is forwarded to the specified forwarding destination. Data is also stored in the database, but it is not processed by the CRE. The CRE at the forwarded destination processes the data.

If data matches multiple rules, the safest routing option is applied. For example, if data that matches a rule that is configured to drop and a rule to bypass CRE processing, the data is not dropped. Instead, the data bypasses the CRE and is stored in the database.

All events are counted against the EPS license. Routing Options

To Configure the routing rules for bulk forwarding: 1. Click the Admin tab.

2. In the navigation pane, clickSystem Configuration. 3. Click the Routing Rules icon.

4. On the toolbar, clickAdd.

5. In the Routing Rules window, enter values for the parameters. a. Type a name and description for your routing rule.

b. From the Mode field, select one of the following options:OnlineorOffline. c. From the Forwarding Event Collector or Forwarding Event Processor list, select the

event collector from which you want to forward data.

d. From the Data Source field in the Event Filters section, select which data source you want to route:EventsorFlows.

(19)

If you select the Flow Filters option, the section title changes to Flow Filters and the Match All Incoming Events check box changes to Match All Flows.

e. To forward all incoming data, select theMatch All Incoming EventsorMatch All Incoming Flowscheck box.

NOTE: If you select this check box, you cannot add a filter.

f. To forward all incoming data, select theMatch All Incoming Eventscheck box.

NOTE: If you select this check box, you cannot add a filter.

g. To add a filter, in the Event Filters or Flow Filters section, select a filter from the first list and an operand from the second list.

h. To add a filter, in the Event Filters section, select a filter from the first list and an operand from the second list.

i. In the text box, type the value that you want to filter for, and then clickAdd Filter. j. Repeat the previous two steps for each filter that you want to add.

k. To forward log data that matches the current filters, select the Forward check box, and then select the check box for each preferred forwarding destination.

NOTE: If you select the Forward check box, you can also select either the Drop or Bypass Correlation check boxes, but not both of them.

If you want to edit, add, or delete a forwarding destination, click theManage Destinations link.

6. ClickSave.

(20)

(21)

PART 2

Index

• Index on page 13

(22)

(23)

Index

Symbols

#, comments in configuration statements...ix

( ), in syntax descriptions...ix

< >, in syntax descriptions...viii

[ ], in configuration statements...ix

{ }, in configuration statements...ix

| (pipe), in syntax descriptions...ix

B

braces, in configuration statements...ix

brackets angle, in syntax descriptions...viii

square, in configuration statements...ix

C

comments, in configuration statements...ix

conventions text and syntax...viii

curly braces, in configuration statements...ix

customer support...x

contacting JTAC...x

D

documentation comments on...ix

F

font conventions...viii

M

manuals comments on...ix

P

parentheses, in syntax descriptions...ix

S

support, technicalSeetechnical support syntax conventions...viii

T

technical support contacting JTAC...x

Juniper Secure Analytics

Juniper Secure Analytics

Big Data Management Guide

2014.1

Table of Contents

Part 1

Juniper Secure Analytics Big Data Management

Part 2

Index

List of Tables

Part 1

Juniper Secure Analytics Big Data Management

About the Documentation

Documentation and Release Notes

Documentation Conventions

Table 1: Notice Icons

Table 2: Text and Syntax Conventions

Table 2: Text and Syntax Conventions

(continued)

Documentation Feedback

Requesting Technical Support

Self-Help Online Tools and Resources

Opening a Case with JTAC

PART 1

Juniper Secure Analytics Big Data

Management

CHAPTER 1

JSA and Big Data Integration

Configuring JSA and Big Data Integration

Integration Components

Table 3: Integration Components

Example: Using Big Data Technologies for Advanced Data Threats

Installing the Data Management Application Server Plug-In and Reference Data API

Adding Forwarding Destinations

Table 4: Forwarding Destinations Parameters

Table 4: Forwarding Destinations Parameters

(continued)

Customizing the Forwarding Profile

Configuring Routing Rules for Bulk Forwarding

Table 5: Routing Rules Window Parameters

PART 2

Index

Index

Symbols

B

C

D

F

M

P

S

T