L1-Threat Intelligence - Indicators and Warnings

Overview

This warnings and indicators package populates, displays and monitors the Threat Model . The Threat Model is used to detect and contextualize potential malicious activiity based on intelligence derived from a site-specific mix of threat intelligence sources. This package uses the open source Collective Intelligence Framework (CIF) to collect and normalize threat data from open source, proprietary and internal sources. A FlexConnector then sends this data to the ESM, where rules populate the model. The model itself consists of three Active Lists. This package also provides Global Variables enabling other content to contextualize events based on the Threat Model, by accessing metadata associated with specific threat indicators. Finally, the package contains a dashboard showing the current status and historical information on the Theat Model.

Latest release 1.2.0.0 contains prepoplulated suspicious filehash in the /All Active Lists/ArcSight Activate/Solutions/Threat Intelligence/Indicators and Warnings/User Defined Reputation Data/Additional - Entity

Definition of Terms

Before detailing the package, it is important to understand the usage of the following terms in the context:

Term Description
CIF Collective Intelligence Framework, an open source product delivered, enriched and customized as part of this package. CIF is a flexible mechanism to collect and normalize Threat Intelligence from a wide variety of sources
Indicator Used to identify potentially malicious activity. In this package, may be an IPv4 address, IPv6 address or entity. Entity is a string type that is used for hashs, email addresses, URLs, user names and host names
Open Source Intelligence Threat Intelligence available to the public for free. CIF comes pre-configured to collect a variety of these, and may be configured to collect others.
Proprietary Intelligence Threat Intelligence about malicious indicators purchased by the client. Examples include ThreatCentral and ThreatStream. CIF comes pre-configured to collect some of these, and can be configured to collect others
Internal Intelligence Threat Intelligence about malicious addresses, URLs and emails derived from site-specific activity. CIF comes with sample configuration that may be customized to collect this type of intelligence
Confidence Score A number range 0 - 100 associated with each indicator that shows the likelihood that this intelligence is reliable and accurate. In general, open source intelligence has a lower score than proprietary intelligence, which in turn has a lower score than Internal Intelligence
Threat Model A set of Active Lists in ArcSight populated with suspicious indicators collected from threat intelligence sources, along with any metadata associated with each indicator.

Data Flow

Based on these definitions, This L1 package works in the following way:
  1. CIF periodically harvests open source, proprietary and or internal sources of threat intelligence, normalizes the data and stores it in its dataset
  2. A CIF feed command periodically generates a csv encapsulating the latest threat intelligence snapshop
  3. The CIF FlexConnector parses this csv and sends events to one or more ESMs
  4. Rules on the ESM populate the Threat Model, which consists of three lists:
    1. 1 IPv4 Indicators and metadata
    2. 1 IPv6 indicators and metadata
    3. 1 Entity indicators and metadata, applicable to hashes, email addresses, URLs, user names and host names.
  5. Global Variables enable other content to extract metadata associated with an indicator from the Threat Model, thus contextualizing and enriching events
  6. A dashboard reports on Threat Model activity, including breakdowns of Threat Indicators by type, origin, confidence, etc.

The L2 Situational Awareness Threat Intelligence Package builds upon this L1 package to detect and report upon malicious activity using this model. Any other ArcSight ESM content may also leverage this model to detect and contextualize potential malicious activity.

Main Use Cases

Below are the principal use cases implemented in this package:

Use Case 1: Populate Threat Model from a variety of heterogeneous intelligence feeds

This is the primary use case. The Threat Model consists of three active lists. Each active list is keyed by a specific type of indicator: IPv4 address, IPv6 Address and Entity. Entity in this case may be a URL, host name, hash, user name or email. This use case populates this list periodically with threat intelligence data collected from a variety of customer specific sources, including open source threat data, proprietary intelligence and intelligence created by the customer.

In the v1.2, suspicious hashfiles were pre-popuated in the list Additional - Entity

Use Case 2: Enrich events with Threat Model data

Global variables delivered in this package enable any other ESM content to access metadata stored in the Theat Model, as keyed by an indicator.

Use Case 3: Display and report upon Threat Model activity

A dashboard shows the current status of the Threat Model, including number of current indicators, indicator types (botnet, malware, etc), observable type (ipv4, ipv6, fqdn, etc.) and other useful information on how the Threat Model is populated.

Threat_Model_Activity.GIF

The pie charts in the dashboard allow the Customer to drill down to the specific threat data for a particular source:

Threat_Model_Activity_DrillDown.GIF

Use Case 4: Detect HIDDEN COBRA Threat

HiddenCobraThreat

The US CERT provided IOC’s that can be used to detect North Korean Malicious Cyber Activity referred as HIDDEN COBRA.

Supported Log Sources

Here are the log source types supported by this package as delivered:

Vendor Product Version(s) Comments
      IOCs to identify Hidden Cobra Threat
  CIF   Events sent from the CIF FlexConnector to populate the Threat Model
  STIX TAXII   IOCs are passed through csv file to FlexConnector to populate the Threat Model

Installation

Installation requires building a CIF server, either hosted on Amazon Web Services or in-house. Next, you will install a tar file of Activate content for this server, and customize it to collect the feeds you require. You then install a folder follower FlexConnector to send the CIF information to your ESM. Finally, you can enable/disable/customize ESM content to your site's specifications.

Prerequisites

  • This package requires the Activate Base package
  • If harvesting open source intelligence, this server will need some connectivity to the internet, preferably through a Proxy.
  • This server runs the CIF FlexConnector, which will require port 443 connectivity to the ESM.
  • The content in this package is verified for ESM release 6.8c and above.
  • You do not require any proprietary or internal threat intelligence sources to install and use this package. If you do have such sources, you may have to configure rules for them in CIF as described below.
  • It helps to have a very basic understanding of Unix/Linux line commands to install CIF. However, the actual install is very simple and only requires a single command.

We strongly recommend browsing the CIF documentation before beginning the install. The install is straightforward and easy and should take approximately 10 minutes, but it is important to understand the basic concepts behind CIF before proceeding.

ESM Configuration

  • Add the following lines to the server.properties file:
#Increase the active list maximum capacity
activelist.max_capacity=1500000

#Increase the archive/package maximum capacity
webservice.archive.maxlength=5120000

#Increase the agent request limit
agents.request.size.limit=5242880000
  • Restart the Manager for the new settings to take effect.

Console Configuration

  • Add the following lines to the console.properties file:
#Turn off URL encoding from the console.
ui.integration.url.encode=false

#Increase the archive/package maximum capacity
agent.archive.maxlength=814400

Step 1: Install L1 Threat Intelligence Package

Before getting CIF running, first install the ESM package so that you can view and monitor the incoming threat intelligence event flow.
  • Download and extract L1-Threat Intelligence - Indicator and Warning.zip into your ArcSight Console's current directory.
  • Open a command prompt and navigate to ARCSIGHT_HOME
  • Execute the L1ThreatIntelligence.bat file to install. (You will be prompted to enter the manager hostname, username, and password. The password is displayed in clear text, please be aware of your environment)
  • After install, your packages in ArcSight ESM will look like the screenshot.

To verify installation, select the Resources tab, and then in the Dashboards tree, and expand the /All Dashboards/ArcSight Activate/Solutions/Threat Intelligence/Indicators and Warnings/. You should see the Dashboards as shown below.

Step 2: Select Your Threat Intelligence Source:

a) STIX/TAXII

Install and Configure STIX/TAXII Server

Describes installation and configuration of TAXII- Trusted Automated eXchange of Indicator Information protocol for communication of cyber threat information in the STIX - Structured Threat Information eXpression language. STIX_TAXII.events for testing purpose attached.

b) CIF:

Install CIF Server

L1ThreatIntelligenceStep2 describes installation of the standard Collective Intelligence Framework (CIF) server.
Configure CIF Server

L1ThreatIntelligenceStep3. In this section, you will configure the CIF server to retrieve the threat intelligence feeds you need. You will also set up a simple job to create input for the CIF FlexConnector and another job to prune the CIF database.
Install a FlexConnector to integrate CIF server feed to ArcSight ESM

L1ThreatIntelligenceStep4. The Activate CIF FlexConnector normalizes the output generated from the CIF server's elastic database and sends it over to ArSight ESM.

c) Special Steps to collect Ransomware feeds:

Import .csv file

L1ThreatIntelligenceStep5. In this section, you use a script to get ransomware feeds directly if you do not want to use CIF.

Test Plan

L1ThreatIntelligenceTestPlan provides methodology for testing this package.

Extensibility

This package is extended by and supports the L2-Threat Intelligence - Situational Awareness package.

Resources

The link below contains a table of all resources included in this package:
Topic attachments
I Attachment Action Size Date Who Comment
Import_Select.GIFGIF Import_Select.GIF manage 18.7 K 26 Apr 2016 - 18:53 GeorgeBoitano Select Import button for package import screenshot
Packages_Select.GIFGIF Packages_Select.GIF manage 23.3 K 26 Apr 2016 - 18:52 GeorgeBoitano Select Packages in navigation pane screenshot
STIX-TAXII_Install_and_Configure_for_ArcSight.pdfpdf STIX-TAXII_Install_and_Configure_for_ArcSight.pdf manage 1128.1 K 22 Jul 2019 - 22:20 YunPeng Guide
STIX_TAXII.eventsevents STIX_TAXII.events manage 12.1 K 05 Jan 2018 - 21:25 SeemaKhan STIX TAXII sample events
Threat_Indicators_AC.GIFGIF Threat_Indicators_AC.GIF manage 21.1 K 26 Apr 2016 - 18:53 GeorgeBoitano Verify Package Install screenshot
Threat_Model_Activity.GIFGIF Threat_Model_Activity.GIF manage 438.5 K 26 Apr 2016 - 18:34 GeorgeBoitano Screen Shot of Threat Model Activity Dashboard
Threat_Model_Activity_DrillDown.GIFGIF Threat_Model_Activity_DrillDown.GIF manage 116.9 K 21 Apr 2016 - 23:13 GeorgeBoitano Drill down screen image for a particular threat intel source
arcsight_stix_taxii.zipzip arcsight_stix_taxii.zip manage 62.9 K 12 Sep 2019 - 20:53 YunPeng STIX/TAXII ArcSight Client
Topic revision: r26 - 12 Sep 2019, YunPeng


 


Activate Wiki 2.1.0.0

This site is powered by FoswikiCopyright &© by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback