Rule Best Practices

There are four critical areas of the rules editor: the rule type (under the Attributes tab), Conditions, Aggregation, and Actions.

Rule Types

As of ESM v6.5, there are three types of rules: Standard, Lightweight and Pre-Persistence.

Standard rules are what pretty much everyone has seen. You can do anything a rule can do with a standard rule.


Lightweight rules are only for maintaining active and session lists. They do not send out a correlation event whenever they are triggered. If you have rules that do nothing but add or remove data from a list, convert it to a lightweight rule, unless you are also using the correlation event for consumption by another rule or a data monitor.

Pre-persistence rules are very similar to Lightweight rules, but can only set event fields. Be very careful setting event fields, you do not necessarily want to overwrite the data that is already in that field. Any pre-persistence rule that affects events from a given device should be re-evaluated every time the device or its connector or parser is updated.


Note: Pre-persistence rules are evaluated first. Neither pre-persistence nor lightweight rules send correlation or action events. This makes it difficult to test them, but there are often side-effects and workarounds.

The first workaround for testing lightweight and pre-persistence rules is to start out with them as standard rules. Just remember to set the On Every Event trigger, and that only list actions will work for lightweight rules, and only set event field actions will work for pre-persistence rules.

Best practices for the rule development:

Each rule's correlation event should be categorized (in the Actions tab, use the setEventField action).

The rules engine examines conditions in the sequence order. The more lines the rule engine examines, the more impact the system performance will suffer.
The more rules the manager has, the more impact it will have, which is common sense

• Put the more specific conditions at the top, such as "name StartsWith Spyware", because few events can meet it, as a result, the rule engine won't spend processing time examining the rest of the conditions.

• Same reason as above, put the more general conditions to the bottom, such as "Type IN (Aggregated,Base)"
• Put the more expensive conditions at the bottom, such as checking if the address is in an active list.

Don’t use “Type = Base,” use “Type IN (Aggregated,Base),” instead.

The UI sometimes loses your condition if you drop it in the wrong place. I have gotten into the habit of saving the rule after every condition I move. Sometimes, using ctrl+z will put it back where it was and you can try it again.

Case sensitive string conditions are faster than case insensitive conditions.
The InActiveList condition is better than a join rule with a long time window (long > 10 minutes)
!HasVulnerability and use of asset group variable functions, are very costly.
The MatchesFilter condition can be expensive, depending on the filter, especially if that filter uses any of the heavier variable functions or is very complex.

Performance Cost of operators:

1. Most expensive: Asset Variables
2. HasVulnerability
3. InActiveList
4. Least expensive: String Operators
Note that MatchesFilter does have a performance cost (not sure why, given the rules are compiled), but the use of other more expensive conditions will definitely affect the cost of using that filter. If you have an expensive filter, and you want to do a MatchesFilter on another expensive filter, your rule will have significant performance issues

The rules do what is called “ Short-Circuit Evaluation.” This means that in a simple AND condition, if the first element is false, it doesn’t matter what the second element is, both must be true for the AND condition to be true.
A similar circumstance applies to OR condition, if the first element is true, it doesn’t matter what the second element is, because only one element needs to be true for the OR condition to be true. This means you can put the simplest, or cheapest, condition to evaluate higher up.

Rule Actions

Set the rule correlation event’s Device Event Category (DEC) to /Rule/Fire/something. See the section on Device Event Category in the Activate Indicators and Warnings page.

Any data you want to show up in the correlation event should be explicitly added to the rule’s aggregation.

There are two types of aggregation conditions. Aggregate only if these fields are identical, which is the most commonly used aggregation, and Aggregate only if these fields are unique.

If you try to put the unique information into an active list, the rules engine will give you a list of attackers (set the value to @sourceUserName, instead of $sourceUserName, for example, but you'll need to put it in a device custom string).

MITRE ATT&CK Framework Tagging: If in case your use case matches with the MITRE framework techniques, then you can add the MITRE Tagging in the below mentioned fields:
  • deviceCustomString6 = Technique ID
  • deviceCustomeString6Label = Technique Name

Rule Performance

Partial Matches

Partial matches come in two flavors. The most obvious one is with join rules, where one of the rule aliases match one or more events. These events are kept in memory.
The second partial match is the aggregation threshold (also applies for non-join rules), e.g., 5 events in 2 minutes.

Make sure that:
• Individual aliases (event 1, event 2, etc.) do not match a lot of events
• Minimize the number of aliases (joining two events is better than joining three…)
• Whenever possible, use the Consume After Match option on each alias –
o Uses the event only once to trigger the rule

The event does not need to stay in the Rule Engine for the whole time window after it produces a match
o This will reduce the number of correlation event, as a single match will not use a consumed event to match with another base event

Minimize the time windows; shorter is better. If you need more than two minutes, especially for a join rule, then consider using an active list.

If you are looking for a login to a system, followed by a connection from that system to another sensitive system, you could write a join rule, but you would need a long time window to match the two events.
If, instead, you used a session list to monitor logins, with a couple of lightweight rules to maintain its entries, you could then use a standard rule to look for connections to the sensitive system from open session entries in the SL.

If you are looking for something like file accesses on a sensitive system, and you’re looking for 10 or more such accesses over a long period of time, like say, an hour, you’re better off creating an active list to store the access information, added by a lightweight rule, with the active list's (AL’s) TTL set to your time period (there can be issues with that, too…), and have your standard rule look for the AL’s entry updated audit event and check the count to see if it is greater than or equal to you threshold (in this case, 10).

Grouping is the selection of aggregation fields. When selecting fields to aggregate, remember that you should only select the fields that you need (you need the customer resource field). Also, some fields have a much wider range of values than others.

You might get away with aggregating on the Request URL Host field, but the full URL is most likely going to be different. Use such fields only if you are looking for single events, using the On Every Event action.

Rule References and Conditions

Rules should always reference the "Generator" of the filter for their matching conditions. The Generator field is a dynamic link and will preserve through resource name changes.

Place true OR conditions or false AND conditions at the top of your filters. Type = Correlation should be placed at the bottom of the filter.

Variables and Active List Field/Column names should all be done in camelCase, starting with lowercase, and not contain any non-alpha-numeric characters.

All rules should aggregate on the customer resource, unless you are specifically looking for cross-customer events. All active and session lists and trends should include customer resource columns. All queries should also select on the customer resource; reports and query viewers do not need to show customer information, even if the query they are using has it (i.e., you can edit the report or QV to not display the column).

Rule Aggregation Settings

Always use "Resource" in aggregation; Customer Resource, Source Zone Resource, and Destination Zone Resource.

Below is comprehensive list of the minimum fields to be aggregated; not all may apply in all cases.

Customer Root Source Destination Device Misc (optional) Category
Zone Resource Originator Address Address Vendor Process Name Behavior
    Host Name Host Name Product File Path Custom Format
    Zone Resource Zone Resource Event Category File Name Device Group
    Port Port   Host (Request URL Host) Device Type
    User Name User Name   Method (Request Method) Object
    User ID User ID     Outcome
    NT Domain NT Domain     Significance
            Technique (optional)

When to Aggregate on Device Fields

In general, Activate solution rules should not aggregate on any of the device fields. If information is needed in the event that is directly related to any device information, map the data to one or more of the custom fields. Don't forget to use the label fields!

The exception to this is for rules in product packages. In many cases, rules in product packages are a proxy for events that are incomplete, or situations where the necessary data are spread across multiple base events. This also applies to product rules that detect product specific patterns. Because L1 or higher level solution rules may need to access the device information from the base events, product package rules should aggregate device host and version information whenever possible.

Global Variables

Global variables for the following fields exist and should be used anytime aggregation in a rule is used to preserve base event data in the rule fire.

These variables must also be set in rule actions to override the default field or no value will be pulled, except in product package rules.

  ArcSight Schema Field   ArcSight Global Variable
User Name      
  sourceUserName ToUpper srcUserName
  destinationUserName ToUpper dstUserName
       
NT Domain      
  sourceNtDomain ToUpper srcNtDomain
  destinationNtDomain ToUpper dstNtDomain
  deviceNtDomain ToUpper dvcNtDomain
       
Host Name      
  sourceHostName ToLower srcHostName
  destinationHostName ToLower dstHostName
  deviceHostName ToLower dvcHostName
       
DNS Domain      
  sourceDnsDomain ToLower srcDnsDomain
  destinationDnsDomain ToLower dstDnsDomain
  deviceDomain    
  deviceDnsDomain ToLower dvcDnsDomain
       
Fully Qualified Domain Name      
  sourceFqdn ToLower srcFqdn
  destinationFqdn ToLower dstFqdn

Rule Set Event Field Actions

In addition to the above global variables, event categorization for the below fields should be manually overridden for all rules following the below schema.

Device Event Category   Category Custom Format Field   Category Significance   Category Outcome   Agent Severity   Event Annotation Stage
/Rule/Fire/Activate/< DMiD layer>/<package level>/<use case>   /Attack Life Cycle/Stage/Sub-stage   /Compromise   /Attempt   Unknown   System Monitored
    /Attack Life Cycle/Delivery   /Hostile   /Success   Low   Triage
    /Attack Life Cycle/Exploit   /Informational   /Failure   Medium    
    /Attack Life Cycle/Activities/C2   /Informationl/Alert       High    
    /Attack Life Cycle/Activities/Lateral Recon   /Informational/Error       Very-High    
    /Attack Life Cycle/Activities/Expand Access   /Informational/Warning            
    /Attack Life Cycle/Activities/Lateral Movement   /Normal            
    /Attack Life Cycle/Activities/Establish Persistence   /Recon            
    /Attack Life Cycle/Activities/Concealment   /Suspicious            
    /Attack Life Cycle/Objectives/Confidentiality                
    /Attack Life Cycle/Objectives/Integrity                
    /Attack Life Cycle/Objectives/Availability                
                     
                     
L1 - Indicators and Warnings Device Event Category   Category Custom Format Field   Category Significance   Category Outcome   Agent Severity   Event Annotation Stage
/Rule/Fire/Activate/Application Monitoring/Indicators and Warnings/<use case>   Choose one from list   Choose one from list   Choose one from list   Choose one from list   Choose one from list
/Rule/Fire/Activate/Data Monitoring/Indicators and Warnings/<use case>                    
/Rule/Fire/Activate/Host Monitoring/Indicators and Warnings/<use case>                    
/Rule/Fire/Activate/Malware Monitoring/Indicators and Warnings/<use case>                    
/Rule/Fire/Activate/Network Monitoring/Indicators and Warnings/<use case>                    
/Rule/Fire/Activate/Perimeter Monitoring/Indicators and Warnings/<use case>                    
/Rule/Fire/Activate/Threat Intelligence/Indicators and Warnings/<use case>                    
/Rule/Fire/Activate/User Monitoring/Indicators and Warnings/<use case>                    
                     
                     
L2 - Situational Awareness   Category Custom Format Field   Category Significance   Category Outcome   Agent Severity   Event Annotation Stage
/Rule/Fire/Activate/Application Monitoring/Situational Awareness/<use case>   Choose one from list   Choose one from list   Choose one from list   Choose one from list   Choose one from list
/Rule/Fire/Activate/Data Monitoring/Situational Awareness/<use case>                    
/Rule/Fire/Activate/Host Monitoring/Situational Awareness/<use case>                    
/Rule/Fire/Activate/Malware Monitoring/Situational Awareness/<use case>                    
/Rule/Fire/Activate/Network Monitoring/Situational Awareness/<use case>                    
/Rule/Fire/Activate/Perimeter Monitoring/Situational Awareness/<use case>                    
/Rule/Fire/Activate/Threat Intelligence/Situational Awareness/<use case>                    
/Rule/Fire/Activate/User Monitoring/Situational Awareness/<use case>                    
                     
                     
                     
L3 - Impact and Threat Analysis   Category Custom Format Field   Category Significance   Category Outcome   Agent Severity   Event Annotation Stage
/Rule/Fire/Activate/Application Monitoring/Impact and Threat Analysis/<use case>   Choose one from list   Choose one from list   Choose one from list   Choose one from list   Choose one from list
/Rule/Fire/Activate/Data Monitoring/Impact and Threat Analysis/<use case>                    
/Rule/Fire/Activate/Host Monitoring/Impact and Threat Analysis/<use case>                    
/Rule/Fire/Activate/Malware Monitoring/Impact and Threat Analysis/<use case>                    
/Rule/Fire/Activate/Network Monitoring/Impact and Threat Analysis/<use case>                    
/Rule/Fire/Activate/Perimeter Monitoring/Impact and Threat Analysis/<use case>                    
/Rule/Fire/Activate/Threat Intelligence/Impact and Threat Analysis/<use case>                    
/Rule/Fire/Activate/User Monitoring/Impact and Threat Analysis/<use case>                    
                     
                     


-- PrenticeHayes - 14 Apr 2017
Topic revision: r6 - 17 Oct 2019, DatNguyen


 


Activate Wiki 2.1.0.0

This site is powered by FoswikiCopyright &© by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback