Difference: RuleBestPractices (r4 vs. r3)

Rule Best Practices

There are four critical areas of the rules editor: the rule type (under the Attributes tab), Conditions, Aggregation, and Actions.

Rule Types

As of ESM v6.5, there are three types of rules: Standard, Lightweight and Pre-Persistence.

Standard rules are what pretty much everyone has seen. You can do anything a rule can do with a standard rule.


Lightweight rules are only for maintaining active and session lists. They do not send out a correlation event whenever they are triggered. If you have rules that do nothing but add or remove data from a list, convert it to a lightweight rule, unless you are also using the correlation event for consumption by another rule or a data monitor.

Pre-persistence rules are very similar to Lightweight rules, but can only set event fields. Be very careful setting event fields, you do not necessarily want to overwrite the data that is already in that field. Any pre-persistence rule that affects events from a given device should be re-evaluated every time the device or its connector or parser is updated.


Note: Pre-persistence rules are evaluated first. Neither pre-persistence nor lightweight rules send correlation or action events. This makes it difficult to test them, but there are often side-effects and workarounds.

The first workaround for testing lightweight and pre-persistence rules is to start out with them as standard rules. Just remember to set the On Every Event trigger, and that only list actions will work for lightweight rules, and only set event field actions will work for pre-persistence rules.

Best practices for the rule development:

Each rule's correlation event should be categorized (in the Actions tab, use the setEventField action).

The rules engine examines conditions in the sequence order. The more lines the rule engine examines, the more impact the system performance will suffer.
The more rules the manager has, the more impact it will have, which is common sense

? Put the more specific conditions at the top, such as "name StartsWith Spyware", because few events can meet it, as a result, the rule engine won't spend processing time examining the rest of the conditions.

? Same reason as above, put the more general conditions to the bottom, such as "Type IN (Aggregated,Base)"
? Put the more expensive conditions at the bottom, such as checking if the address is in an active list.

Don?t use ?Type = Base,? use ?Type IN (Aggregated,Base),? instead.

The UI sometimes loses your condition if you drop it in the wrong place. I have gotten into the habit of saving the rule after every condition I move. Sometimes, using ctrl+z will put it back where it was and you can try it again.

Case sensitive string conditions are faster than case insensitive conditions.
The InActiveList condition is better than a join rule with a long time window (long > 10 minutes)
!HasVulnerability and use of asset group variable functions, are very costly.
The MatchesFilter condition can be expensive, depending on the filter, especially if that filter uses any of the heavier variable functions or is very complex.

Performance Cost of operators:

1. Most expensive: Asset Variables
2. HasVulnerability
3. InActiveList
4. Least expensive: String Operators
Note that MatchesFilter does have a performance cost (not sure why, given the rules are compiled), but the use of other more expensive conditions will definitely affect the cost of using that filter. If you have an expensive filter, and you want to do a MatchesFilter on another expensive filter, your rule will have significant performance issues

The rules do what is called ? Short-Circuit Evaluation.? This means that in a simple AND condition, if the first element is false, it doesn?t matter what the second element is, both must be true for the AND condition to be true.
A similar circumstance applies to OR condition, if the first element is true, it doesn?t matter what the second element is, because only one element needs to be true for the OR condition to be true. This means you can put the simplest, or cheapest, condition to evaluate higher up.

Rule Actions

Set the rule correlation event?s Device Event Category (DEC) to /Rule/Fire/something. See the section on Device Event Category in the Activate Indicators and Warnings page.

Any data you want to show up in the correlation event should be explicitly added to the rule?s aggregation.

There are two types of aggregation conditions. Aggregate only if these fields are identical, which is the most commonly used aggregation, and Aggregate only if these fields are unique.

If you try to put the unique information into an active list, the rules engine will give you a list of attackers (set the value to @sourceUserName, instead of $sourceUserName, for example, but you'll need to put it in a device custom string).

Rule Performance

Partial Matches

Partial matches come in two flavors. The most obvious one is with join rules, where one of the rule aliases match one or more events. These events are kept in memory.
The second partial match is the aggregation threshold (also applies for non-join rules), e.g., 5 events in 2 minutes.

Make sure that:
? Individual aliases (event 1, event 2, etc.) do not match a lot of events
? Minimize the number of aliases (joining two events is better than joining three?)
? Whenever possible, use the Consume After Match option on each alias ?
o Uses the event only once to trigger the rule

The event does not need to stay in the Rule Engine for the whole time window after it produces a match
o This will reduce the number of correlation event, as a single match will not use a consumed event to match with another base event

Minimize the time windows; shorter is better. If you need more than two minutes, especially for a join rule, then consider using an active list.

If you are looking for a login to a system, followed by a connection from that system to another sensitive system, you could write a join rule, but you would need a long time window to match the two events.
If, instead, you used a session list to monitor logins, with a couple of lightweight rules to maintain its entries, you could then use a standard rule to look for connections to the sensitive system from open session entries in the SL.

If you are looking for something like file accesses on a sensitive system, and you?re looking for 10 or more such accesses over a long period of time, like say, an hour, you?re better off creating an active list to store the access information, added by a lightweight rule, with the active list's (AL?s) TTL set to your time period (there can be issues with that, too?), and have your standard rule look for the AL?s entry updated audit event and check the count to see if it is greater than or equal to you threshold (in this case, 10).

Grouping is the selection of aggregation fields. When selecting fields to aggregate, remember that you should only select the fields that you need (you need the customer resource field). Also, some fields have a much wider range of values than others.

You might get away with aggregating on the Request URL Host field, but the full URL is most likely going to be different. Use such fields only if you are looking for single events, using the On Every Event action.

Rule References and Conditions

Rules should always reference the "Generator" of the filter for their matching conditions. The Generator field is a dynamic link and will preserve through resource name changes.

Place true OR conditions or false AND conditions at the top of your filters. Type = Correlation should be placed at the bottom of the filter.

Variables and Active List Field/Column names should all be done in camelCase, starting with lowercase, and not contain any non-alpha-numeric characters.

All rules should aggregate on the customer resource, unless you are specifically looking for cross-customer events. All active and session lists and trends should include customer resource columns. All queries should also select on the customer resource; reports and query viewers do not need to show customer information, even if the query they are using has it (i.e., you can edit the report or QV to not display the column).

Rule Aggregation Settings

Always use "Resource" in aggregation; Customer Resouce, Resource, Source Zone Resource, Destination Zone Resource, and Device Destination Zone Resource.

Below is comprehensive list of the minimum fields to be aggregated; not all may apply in all cases.

CustomerSource Root Destination Source Device Destination Misc Device (optional) Misc (optional)Category
Zone Resource Address Originator Address Vendor Address Process Vendor Name Process Name Behavior
Hostname Hostname Host Name Product Host Name File Product Path File Path Custom Format
Zone Resource Zone Resource Address Zone Resource File Name Event Category File Name Device Group
Port Port Hostname Port Host Host (Request URL Host) Device Type
User Name User Name Zone Resource User Name Method Method (Request Method) Object
User ID User ID User ID Outcome
NT Domain NT Domain NT Domain Significance
Technique (optional)

When to Aggregate on Device Fields

In general, Activate solution rules should not aggregate on any of the device fields. If information is needed in the event that is directly related to any device information, map the data to one or more of the custom fields. Don't forget to use the label fields!

The exception to this is for rules in product packages. In many cases, rules in product packages are a proxy for events that are incomplete, or situations where the necessary data are spread across multiple base events. This also applies to product rules that detect product specific patterns. Because L1 or higher level solution rules may need to access the device information from the base events, product package rules should aggregate device host and version information whenever possible.

Global Variables

Global variables for the following fields exist and should be used anytime aggregation in a rule is used to preserve base event data in the rule fire.

These varilables variables must also be set in rule actions to override the default field or no value will be pulled. pulled, except in product package rules.

ArcSight Schema Field ArcSight Global Variable
User Name
sourceUserName ToUpper srcUserName
destinationUserName ToUpper dstUserName
NT Domain
sourceNtDomain ToUpper srcNtDomain
destinationNtDomain ToUpper dstNtDomain
deviceNtDomain ToUpper dvcNtDomain
Host Name
sourceHostName ToLower srcHostName
destinationHostName ToLower dstHostName
deviceHostName ToLower dvcHostName
DNS Domain
sourceDnsDomain ToLower srcDnsDomain
destinationDnsDomain ToLower dstDnsDomain
deviceDomain
deviceDnsDomain ToLower dvcDnsDomain
Fully Qualified Domain Name
sourceFqdn ToLower srcFqdn
destinationFqdn ToLower dstFqdn

Rule Set Event Field Actions

In addition to the above global variables, event categorization for the below fields should be manually overridden for all rules following the below schema.

Device Event Category Category Custom Format Field Category Significance Category Outcome Agent Severity Event Annotation Stage
/Rule/Fire/Activate/< DMiD layer>// /Attack Life Cycle/Stage/Sub-stage /Compromise /Attempt Unknown System Monitored
/Attack Life Cycle/Delivery /Hostile /Success Low Triage
/Attack Life Cycle/Exploit /Informational /Failure Medium
/Attack Life Cycle/Activities/C2 /Informationl/Alert High
/Attack Life Cycle/Activities/Lateral Recon /Informational/Error Very-High
/Attack Life Cycle/Activities/Expand Access /Informational/Warning
/Attack Life Cycle/Activities/Lateral Movement /Normal
/Attack Life Cycle/Activities/Establish Persistence /Recon
/Attack Life Cycle/Activities/Concealment /Suspicious
/Attack Life Cycle/Objectives/Confidentiality
/Attack Life Cycle/Objectives/Integrity
/Attack Life Cycle/Objectives/Availability
L1 - Indicators and Warnings Device Event Category Category Custom Format Field Category Significance Category Outcome Agent Severity Event Annotation Stage
/Rule/Fire/Activate/Application Monitoring/Indicators and Warnings/ Choose one from list Choose one from list Choose one from list Choose one from list Choose one from list
/Rule/Fire/Activate/Data Monitoring/Indicators and Warnings/
/Rule/Fire/Activate/Host Monitoring/Indicators and Warnings/
/Rule/Fire/Activate/Malware Monitoring/Indicators and Warnings/
/Rule/Fire/Activate/Network Monitoring/Indicators and Warnings/
/Rule/Fire/Activate/Perimeter Monitoring/Indicators and Warnings/
/Rule/Fire/Activate/Threat Intelligence/Indicators and Warnings/
/Rule/Fire/Activate/User Monitoring/Indicators and Warnings/
L2 - Situational Awareness Category Custom Format Field Category Significance Category Outcome Agent Severity Event Annotation Stage
/Rule/Fire/Activate/Application Monitoring/Situational Awareness/ Choose one from list Choose one from list Choose one from list Choose one from list Choose one from list
/Rule/Fire/Activate/Data Monitoring/Situational Awareness/
/Rule/Fire/Activate/Host Monitoring/Situational Awareness/
/Rule/Fire/Activate/Malware Monitoring/Situational Awareness/
/Rule/Fire/Activate/Network Monitoring/Situational Awareness/
/Rule/Fire/Activate/Perimeter Monitoring/Situational Awareness/
/Rule/Fire/Activate/Threat Intelligence/Situational Awareness/
/Rule/Fire/Activate/User Monitoring/Situational Awareness/
L3 - Threat and Impact Assessment and Threat Analysis Category Custom Format Field Category Significance Category Outcome Agent Severity Event Annotation Stage
/Rule/Fire/Activate/Application Monitoring/Threat Monitoring/Impact and Impact Assessment/ Threat Analysis/ case> Choose one from list Choose one from list Choose one from list Choose one from list Choose one from list
/Rule/Fire/Activate/Data Monitoring/Threat Monitoring/Impact and Impact Assessment/ Threat Analysis/ case>
/Rule/Fire/Activate/Host Monitoring/Threat Monitoring/Impact and Impact Assessment/ Threat Analysis/ case>
/Rule/Fire/Activate/Malware Monitoring/Threat Monitoring/Impact and Impact Assessment/ Threat Analysis/ case>
/Rule/Fire/Activate/Network Monitoring/Threat Monitoring/Impact and Impact Assessment/ Threat Analysis/ case>
/Rule/Fire/Activate/Perimeter Monitoring/Threat Monitoring/Impact and Impact Assessment/ Threat Analysis/ case>
/Rule/Fire/Activate/Threat Intelligence/Threat Intelligence/Impact and Impact Assessment/ Threat Analysis/ case>
/Rule/Fire/Activate/User Monitoring/Threat Monitoring/Impact and Impact Assessment/ Threat Analysis/ case>



-- PrenticeHayes - 14 Apr 2017

View topic | View difference side by side | History: r6 < r5 < r4 < r3 | More topic actions
 
This site is powered by FoswikiCopyright &© by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback