Difference: HowActivateBestPractices (1 vs. 20)

Revision 20
04 Apr 2018 - Main.PrenticeHayes
Line: 1 to 1
 
META TOPICPARENT name="HowActivateDevelopment"

Activate Content Development Best Practices

Line: 188 to 194
 

Clean Packages (no extra resources included unnecessarily)

Changed:
<
<
By default, new packages will exclude most of the /All <resource_type>/ArcSight System/… level content. The current exception to this is /All Fields/ArcSight System/Event Fields. You can safely add this to the resource exclusions of the package.
>
>
By default, new packages will exclude most of the /All <resource_type>/ArcSight System/… level content. The current exception to this is /All Fields/ArcSight System/Event Fields. You can safely add this to the resource exclusions of the package. NOTE: You do not need to worry about this if you use an Activate Template package.
 

Steps to follow for removing the event fields from a package:

Added:
>
>
NOTE: You do not need to worry about this if you use an Activate Template package.
  1. In the package editor, select the Resources tab.

2. Sort the Removed Resource Column.
Line: 204 to 212
 

Cross Package Contamination

Changed:
<
<
For long-term maintenance of packages, make sure that resources only show up in one package. If you have a resource in two packages, and modify that resource and export one of them, then the old version of that resource is in the second package. If you install them in the wrong order, your resources won’t be what you expect or need.
>
>
For long-term maintenance of packages, make sure that resources only show up in one, and only one, package. If you have a resource in two packages, and modify that resource and export one of them, then the old version of that resource is in the second package. If you install them in the wrong order, your resources won’t be what you expect or need.
 
Changed:
<
<
Make sure that you're not including additional resources you don't need to export or import
>
>
Make sure that you're not including additional resources you don't need to export or import. Do not allow resources from Activate Base, or any other Activate packages, into your packages. NOTE: You do not need to worry about this if you use an Activate Template package. If you aren't using one of the Activate package templates, you can update your resource exclusions to match the templates. Also, make sure that you do not explicitly include resources from other packages, and avoid linking resources from other packages into your resource tree structure.
 

Content and Package Testing

Revision 19
03 Apr 2018 - Main.PrenticeHayes
Line: 1 to 1
 
META TOPICPARENT name="HowActivateDevelopment"

Activate Content Development Best Practices

Added:
>
>

What Packages Are Currently Being Developed?

See the Activate Development page for the current list of projects, who may be working on them, and wishlist items.
 

Resource Naming Conventions

Resource Naming Conventions Best Practices
Revision 18
09 Mar 2018 - Main.PrenticeHayes
Line: 1 to 1
 
META TOPICPARENT name="HowActivateDevelopment"

Activate Content Development Best Practices

Line: 156 to 162
  This will make the XML bits of the package sort the resources by their resource ID, rather than their resource URI. This makes it easier for us to do diffs, etc., when merging versions for release.

Once you update the server.properties file, you will need to restart your manager.
Added:
>
>

Activate Package Development Best Practice

Always start your package by using the appropriate Activate Template package. Also, if you are building an L1 package with shared filters, you should build a customizations package to preserve product package configurations when your L1 package is updated.

The Activate package templates will set up the initial package so that unintended resources cannot accidentally be included in your package.
 

Package Format and Other Useful Information

When archiving resources (except users, use the “exportuser” format), use the "export" format. Also, always check the "exclude reference IDs" checkbox.
Line: 194 to 209
  For long-term maintenance of packages, make sure that resources only show up in one package. If you have a resource in two packages, and modify that resource and export one of them, then the old version of that resource is in the second package. If you install them in the wrong order, your resources won’t be what you expect or need.

Make sure that you're not including additional resources you don't need to export or import
Deleted:
<
<

Activate Package Development Best Practice

 
Deleted:
<
<
Always start your package by using the appropriate Activate Template package.
 

Content and Package Testing

Before you submit your package for others to use, you should test your content packaging for some very basic conditions. We have provided two testing tools to help you with testing your content:
Revision 17
08 Feb 2018 - Main.PrenticeHayes
Line: 1 to 1
 
META TOPICPARENT name="HowActivateDevelopment"

Activate Content Development Best Practices

Line: 18 to 18
 
  1. Correlation

Base Events

Changed:
<
<
These are what pretty much everyone thinks of as events. These events come from SmartConnectors, and are the basic event type.
>
>
These are what pretty much everyone thinks of as events. These events come from connectors, and are the basic event type.
 

Aggregated Events

These events are really a type of base event. These events come from SmartConnectors, and are the result of the aggregation settings of the connector. An aggregated event represents a collection of more than one base event. The field, aggregatedEventCount, tells you how many base events are represented by an aggregated event. The aggregation settings of a connector will determine how often and under what conditions base events will be combined into aggregated events. An optimal aggregation setting will not result in the loss of any important data.
Line: 46 to 46
  This is better, but allows action events (although this is more of a theoretical problem, it could still happen). Action events are generally not all that useful for security use cases. They are mostly useful for rules engine/system health monitoring use cases.

Optimally, this is a good condition:
Deleted:
<
<
  Type IN (Aggregated,Base)
Changed:
<
<
You might hear some consultants state that the IN operator is better than the equivalent OR operator:
>
>
This is the equivalent OR operator:
  Type = Aggregated OR Type = Base
Deleted:
<
<
However, there is no actual proof of this, it is subjective. In reality, both the ArcSight ESM correlation (rules and data monitor engine) and the MySQL query optimizer converts the IN operator into a series of OR statements. This is what the back-end code does, we cannot (or will not!) change it. Any preference for the IN operator or the OR operator is really a matter of style, readability, and maintainability. In some cases, the IN operator statements are easier to read, while in others, OR statements can be easier to read. It all depends on the number of values, the ordering, and developer's preference.
 

Derived Event Fields

There are many fields that are not actually stored in a database, but are derived from one or more objects in the database. Take zones, for example. There are several zone field collections, e.g., source, destination, agent, device, etc. The fields available for any zone resource in the ArcSight CCE are (leaving out the group name):
Line: 85 to 81
 

Filters can be used in active channels, rules, queries, data monitors, etc. In general, when in doubt about how to write a filter that may be used in multiple resource types, optimize it for rules
Changed:
<
<
The filter Condition: InActiveList – filters that use this condition cannot be used in Active Channels
>
>
The operator: InActiveList – filters that use this condition cannot be used in Active Channels.
 
Changed:
<
<
Avoid using active lists (InActiveList) in your filters. Doing this makes them unusable in an active channel. If you keep it in the rule, and out of the filter, it becomes easier to create active channels for verification of your rule. If you really need to reference an active list and want to use the same filter in an active channel, you can use a function, GetActiveListValue, instead.
>
>
Try to avoid using the active lists operator (InActiveList) in your filters. Doing this makes them unusable in an active channel. If you keep it in the rule, and out of the filter, it becomes easier to create active channels for verification of your rule. If you really need to reference an active list and want to use the same filter in an active channel, you can create a variable, using the function GetActiveListValue, instead.
 

The function: Arithmetic | JavaMathematicalExpression (JME) can be used in rules, but not in queries.
Added:
>
>

Operators: IN vs OR

Optimally, this is a good condition:

Type IN (Aggregated,Base)

You might hear some consultants state that the IN operator is better than the equivalent OR operator:

Type = Aggregated OR Type = Base

However, there is no actual proof of this, it is subjective. In reality, both the ArcSight ESM correlation (rules and data monitor engine) and the MySQL query optimizer converts the IN operator into a series of OR statements. This is what the back-end code does, we cannot (or will not!) change it. Any preference for the IN operator or the OR operator is really a matter of style, readability, and maintainability. In some cases, the IN operator statements are easier to read, while in others, OR statements can be easier to read. It all depends on the number of values, the ordering, and developer's preference.

Short-Circuit Evaluation

Filters and conditions do what is called “ Short-Circuit Evaluation.” This means that in a simple AND condition, if the first element is false, it doesn’t matter what the second element is, both must be true for the AND condition to be true.

A similar circumstance applies to OR condition, if the first element is true, it doesn’t matter what the second element is, because only one element needs to be true for the OR condition to be true. This means you can put the simplest, or cheapest, condition to evaluate higher up.

This means that you should balance your conditions between two concepts: the cost of evaluating a given operator, and the percentage of events that can be eliminated by a given operator.

For example, the Type field is an enumerated field. This means that the data type is actually numeric, but the application displays it for you as a string (action, aggregated, base and correlation). This, of course, does not apply to events in CEF or in Logger, as the data is stored as a string. Evaluating the Type field is cheaper than evaluating the Device Product field, a string. However, if you are looking for Base events, or more correctly, non-Correlation events, 60-90 percent of events in most installations are base events, whereas for a given product, a much smaller percentage events are generated by that product. If your second condition can eliminate a larger percentage of potential events than your first condition, swap them.

The theory goes like this:
  • Condition A may take 2 time units to evaluate
  • Condition B may take 3 time units to evaluate
  • Condition A may match 90% of the potential events
  • Condition B may match 50% of the potential events
Given a thousand events, the timings for the ordering of Condition A AND Condition B are:

Evaluation of Condition A first results in a minimum of 2,000 time units per thousand events, regardless of how many events actually match. Evaluation of Condition B first results in a minimum of 3,000 time units per thousand events, regardless of how many events actually match.

Evaluating Condition A first means that Condition B will be evaluated 900 times, adding an additional 2,700 time units in processing the thousand events, for a total of 4,700 time units to process this rule per thousand events.

Evaluating Condition B first means that Condition A will be evaluated 500 times, adding an additional 1,000 time units in processing the thousand events, for a total of 4,000 time units to process this rule per thousand events, a potential savings of 700 time units per thousand evaluated events.

Therefore, put “Device Product = ArcSight ” before you put “Type = Correlation” in the general case. “Target Address in the Hostile List” should go further down the AND or OR list, since active list lookups are more computationally expensive than simple field checks. [1]

This applies to filters and conditions in rules, data monitors, and queries.
 

Fields and Variables

There are two classes of variables, global and local. Global variables are a user-definable field that can be used by other resources. Local variables are like fields, but not exactly, and are only useful in the resource in which they are defined.
Revision 16
04 Jan 2018 - Main.PrenticeHayes
Line: 1 to 1
 
META TOPICPARENT name="HowActivateDevelopment"

Activate Content Development Best Practices

Line: 106 to 111
 

Packaging

Added:
>
>
Before you start building any packages for Activate, please do the following:

Update ESM

Update your ESM’s server.properties to export packages in Resource ID order. This will make the XML bits of the package sort the resources by their resource ID, rather than their resource URI.

#Change package archive sort order (server.properties):
export.archive.reference.sort.order=id

This will make the XML bits of the package sort the resources by their resource ID, rather than their resource URI. This makes it easier for us to do diffs, etc., when merging versions for release.

Once you update the server.properties file, you will need to restart your manager.

Package Format and Other Useful Information

  When archiving resources (except users, use the “exportuser” format), use the "export" format. Also, always check the "exclude reference IDs" checkbox.

In addition to the users exception, you might find yourself needing to use a pre-populated active list, where you put specific values in a list for lookup or checking in various conditions. The export format means that no active or session list data will be included in the package. How do we get around this?
Line: 156 to 173
 

We have also provided a tool to automatically generate installation scripts for installing your packages. See the Activate Installation Script Generator Tool documentation for details.
Deleted:
<
<

Package Submission

If you are going to submit packages, please update your ESM’s server.properties to include this:

#Change package archive sort order (server.properties):
export.archive.reference.sort.order=id

This will make the XML bits of the package sort the resources by their resource ID, rather than their resource URI. This makes it easier for us to do diffs, etc., when merging versions for release.

Once you update the server.properties file, you will need to restart your manager.

Update ESM:

Update your ESM’s server.properties to export packages in Resource ID order. This will make the XML bits of the package sort the resources by their resource ID, rather than their resource URI.

Procedure:

1. Edit the /opt/arcsight/manager/config/server.properties file

2. Add or update the following lines:


#Change package archive sort order (server.properties):

export.archive.reference.sort.order=id

3. Once you update the server.properties file, you will need to restart your manager.
  -- GeorgeBoitano - 21 Jan 2016
Revision 15
31 Dec 2017 - Main.PrenticeHayes
Line: 1 to 1
 
META TOPICPARENT name="HowActivateDevelopment"

Activate Content Development Best Practices

Line: 43 to 43
 

This is better, but allows action events (although this is more of a theoretical problem, it could still happen). Action events are generally not all that useful for security use cases. They are mostly useful for rules engine/system health monitoring use cases.
Changed:
<
<
Optimally, this is the best condition:
>
>
Optimally, this is a good condition:
  Type IN (Aggregated,Base)
Changed:
<
<
There are many reasons for this. First, the Type field is an enumerated field, with a range of only four possible values (0 = Action, 1 = Aggregated, 2 = Base, 3 = Correlation). This makes it roughly equivalent in processing time to:
>
>
You might hear some consultants state that the IN operator is better than the equivalent OR operator:
  Type = Aggregated OR Type = Base
Added:
>
>

However, there is no actual proof of this, it is subjective. In reality, both the ArcSight ESM correlation (rules and data monitor engine) and the MySQL query optimizer converts the IN operator into a series of OR statements. This is what the back-end code does, we cannot (or will not!) change it. Any preference for the IN operator or the OR operator is really a matter of style, readability, and maintainability. In some cases, the IN operator statements are easier to read, while in others, OR statements can be easier to read. It all depends on the number of values, the ordering, and developer's preference.
 

Derived Event Fields

There are many fields that are not actually stored in a database, but are derived from one or more objects in the database. Take zones, for example. There are several zone field collections, e.g., source, destination, agent, device, etc. The fields available for any zone resource in the ArcSight CCE are (leaving out the group name):
Line: 74 to 76
  Type IN (Aggregated,Base)

By putting this in the base filter of a product package, any rule within that product package that uses a filter derived from that base filter does not need to use any of the anti-looping rule techniques. This requires a bit of developer discipline, because the ideal product package does not have any rules (just filters...), but there are cases where a product package might require rules. An example of this is auditd user events for Linux. The auditd user events tend to use the user ID, and not the user name. For example, the root user ID is 0, so most auditd events that reference the root user populate the user ID field with 0, and the word 'root' is not found within the event. The P-Linux product package uses a strategy to map the user IDs to the user names, and therefore, the rules in L1-User Monitoring - Indicators and Warnings package will need to consider correlation events, in addition to base and aggregated events.
Added:
>
>

Filters Best Practices

Filters can be used in active channels, rules, queries, data monitors, etc. In general, when in doubt about how to write a filter that may be used in multiple resource types, optimize it for rules

The filter Condition: InActiveList – filters that use this condition cannot be used in Active Channels

Avoid using active lists (InActiveList) in your filters. Doing this makes them unusable in an active channel. If you keep it in the rule, and out of the filter, it becomes easier to create active channels for verification of your rule. If you really need to reference an active list and want to use the same filter in an active channel, you can use a function, GetActiveListValue, instead.
 
Added:
>
>
The function: Arithmetic | JavaMathematicalExpression (JME) can be used in rules, but not in queries.

Fields and Variables

There are two classes of variables, global and local. Global variables are a user-definable field that can be used by other resources. Local variables are like fields, but not exactly, and are only useful in the resource in which they are defined.
 

Lists Best Practices

See the Active List Best Practices page for details
Line: 167 to 180
 

3. Once you update the server.properties file, you will need to restart your manager.
Deleted:
<
<

Filters

filter Condition: !InActiveList – filters that use this condition cannot be used in Active Channels

Function: Arithmetic | JavaMathematicalExpression (JME) can be used in rules, but not in queries

Filters can be used in active channels, rules, queries, data monitors, etc. In general, when in doubt about how to write a filter that may be used in multiple resource types, optimize it for rules

Avoid using active lists (InActiveList) in your filters. Doing this makes them unusable in an active channel.

If you keep it in the rule, and out of the filter, it becomes easier to create active channels for verification of your rule.

If you really need to reference an active list and want to use the same filter in an active channel, you can use a function, GetActiveListValue, instead

Fields and Variables

There are two classes of variables, global and local. Global variables are a user-definable field that can be used by other resources. Local variables are like fields, but not exactly, and are only useful in the resource in which they are defined.
  -- GeorgeBoitano - 21 Jan 2016
Revision 14
30 Dec 2017 - Main.PrenticeHayes
Line: 1 to 1
 
META TOPICPARENT name="HowActivateDevelopment"

Activate Content Development Best Practices

Added:
>
>

Resource Naming Conventions

Resource Naming Conventions Best Practices
 

Events

Event Types

Revision 13
07 Jul 2017 - Main.PrenticeHayes
Line: 1 to 1
 
META TOPICPARENT name="HowActivateDevelopment"

Activate Content Development Best Practices

Line: 137 to 142
 

Package Installation Scripts

Changed:
<
<
We have also provided a tool to automatically generate installation scripts for installing your packages. See the ctivate Installation Script Generator Tool documentation for details.
>
>
We have also provided a tool to automatically generate installation scripts for installing your packages. See the Activate Installation Script Generator Tool documentation for details.
 

Package Submission

Revision 12
06 Jun 2017 - Main.PrenticeHayes
Line: 1 to 1
 
META TOPICPARENT name="HowActivateDevelopment"

Activate Content Development Best Practices

Line: 103 to 103
 

Explicitly include individual active lists that are to have pre-populated data entries in the second package.
Changed:
<
<
When you export the first package, be sure to include the second package in the .arb file.
The pre-populated active lists should have their TTLs set to 0.
1.10.3 Clean Packages (no extra resources included unnecessarily)
By default, new packages will exclude most of the /All <resource_type>/ArcSight System/… level content. The current exception to this is /All Fields/ArcSight System/Event Fields. You can safely add this to the resource exclusions of the package.
1.10.3.1 Steps to follow for removing the event fields from a package:
1. In the package editor, select the Resources tab.
2. Sort the Removed Resource Column.
3. Add /All Fields/ArcSight System/Event Fields.
4. Check the If Not Included checkbox
>
>
When you export the first package, be sure to include the second package in the .arb file.
 
Changed:
<
<
This will avoid errors when the package is uninstalled due to the fields being in a locked group.
1.10.4 Cross Package Contamination
For long-term maintenance of packages, make sure that resources only show up in one package. If you have a resource in two packages, and modify that resource and export one of them, then the old version of that resource is in the second package. If you install them in the wrong order, your resources won’t be what you expect or need.
>
>
The pre-populated active lists should have their TTLs set to 0.
 
Changed:
<
<
make sure that you're not including additional resources you don't need to export or import
>
>

Clean Packages (no extra resources included unnecessarily)

 
Added:
>
>
By default, new packages will exclude most of the /All <resource_type>/ArcSight System/… level content. The current exception to this is /All Fields/ArcSight System/Event Fields. You can safely add this to the resource exclusions of the package.

Steps to follow for removing the event fields from a package:

1. In the package editor, select the Resources tab. 2. Sort the Removed Resource Column. 3. Add /All Fields/ArcSight System/Event Fields. 4. Check the If Not Included checkbox

This will avoid errors when the package is uninstalled due to the fields being in a locked group.

Cross Package Contamination

For long-term maintenance of packages, make sure that resources only show up in one package. If you have a resource in two packages, and modify that resource and export one of them, then the old version of that resource is in the second package. If you install them in the wrong order, your resources won’t be what you expect or need.

Make sure that you're not including additional resources you don't need to export or import

Activate Package Development Best Practice

Always start your package by using the appropriate Activate Template package.
 

Content and Package Testing

Before you submit your package for others to use, you should test your content packaging for some very basic conditions. We have provided two testing tools to help you with testing your content:
Revision 11
24 May 2017 - Main.PrenticeHayes
Line: 1 to 1
 
META TOPICPARENT name="HowActivateDevelopment"

Activate Content Development Best Practices

Line: 113 to 108
  This will avoid errors when the package is uninstalled due to the fields being in a locked group.
1.10.4 Cross Package Contamination
For long-term maintenance of packages, make sure that resources only show up in one package. If you have a resource in two packages, and modify that resource and export one of them, then the old version of that resource is in the second package. If you install them in the wrong order, your resources won’t be what you expect or need.

make sure that you're not including additional resources you don't need to export or import
Added:
>
>

Content and Package Testing

Before you submit your package for others to use, you should test your content packaging for some very basic conditions. We have provided two testing tools to help you with testing your content: These tools make sure that your content can be easily installed and uninstalled without affecting other content on the system, as well as help you identify resources that may be in a bad location (e.g., a user's personal group).

Package Installation Scripts

We have also provided a tool to automatically generate installation scripts for installing your packages. See the ctivate Installation Script Generator Tool documentation for details.
 

Package Submission

If you are going to submit packages, please update your ESM’s server.properties to include this:
Line: 136 to 144
 

3. Once you update the server.properties file, you will need to restart your manager.
Deleted:
<
<
things to embed from the pdf guide
 

Filters

filter Condition: !InActiveList – filters that use this condition cannot be used in Active Channels
Revision 10
14 Apr 2017 - Main.PrenticeHayes
Line: 1 to 1
 
META TOPICPARENT name="HowActivateDevelopment"

Activate Content Development Best Practices

Deleted:
<
<
Document a set of best practices as collected from the original documentation. Tie each back to the Methodology and the Philosophy. If it gets too long, create subpages for each best practice, but be sure to create them from this page and include a link to each on this page.
 

Events

Event Types

Line: 77 to 75
  Type IN (Aggregated,Base)

By putting this in the base filter of a product package, any rule within that product package that uses a filter derived from that base filter does not need to use any of the anti-looping rule techniques. This requires a bit of developer discipline, because the ideal product package does not have any rules (just filters...), but there are cases where a product package might require rules. An example of this is auditd user events for Linux. The auditd user events tend to use the user ID, and not the user name. For example, the root user ID is 0, so most auditd events that reference the root user populate the user ID field with 0, and the word 'root' is not found within the event. The P-Linux product package uses a strategy to map the user IDs to the user names, and therefore, the rules in L1-User Monitoring - Indicators and Warnings package will need to consider correlation events, in addition to base and aggregated events.
Deleted:
<
<

Rules

 
Changed:
<
<
There are four critical areas of the rules editor: the rule type (under the Attributes tab), Conditions, Aggregation, and Actions.

Rule Types

>
>

Lists Best Practices

See the Active List Best Practices page for details
 
Changed:
<
<
As of ESM v6.5, there are three types of rules: Standard, Lightweight and Pre-Persistence.
>
>
See the Session List Best Practices page for details.
 
Changed:
<
<
Standard rules are what pretty much everyone has seen. You can do anything a rule can do with a standard rule.
>
>

Rule Best Practices

See the Rule Best Practices page for details.
 
Deleted:
<
<

Lightweight rules are only for maintaining active and session lists. They do not send out a correlation event whenever they are triggered. If you have rules that do nothing but add or remove data from a list, convert it to a lightweight rule, unless you are also using the correlation event for consumption by another rule or a data monitor.

Pre-persistence rules are very similar to Lightweight rules, but can only set event fields. Be very careful setting event fields, you do not necessarily want to overwrite the data that is already in that field. Any pre-persistence rule that affects events from a given device should be re-evaluated every time the device or its connector or parser is updated.
Note: Pre-persistence rules are evaluated first. Neither pre-persistence nor lightweight rules send correlation or action events. This makes it difficult to test them, but there are often side-effects and workarounds.

The first workaround for testing lightweight and pre-persistence rules is to start out with them as standard rules. Just remember to set the On Every Event trigger, and that only list actions will work for lightweight rules, and only set event field actions will work for pre-persistence rules.

Best practices for the rule development:

Each rule's correlation event should be categorized (in the Actions tab, use the setEventField action).

The rules engine examines conditions in the sequence order. The more lines the rule engine examines, the more impact the system performance will suffer.
The more rules the manager has, the more impact it will have, which is common sense

• Put the more specific conditions at the top, such as "name StartsWith Spyware", because few events can meet it, as a result, the rule engine won't spend processing time examining the rest of the conditions.

• Same reason as above, put the more general conditions to the bottom, such as "Type IN (Aggregated,Base)"
• Put the more expensive conditions at the bottom, such as checking if the address is in an active list.

Don’t use “Type = Base,” use “Type IN (Aggregated,Base),” instead.

The UI sometimes loses your condition if you drop it in the wrong place. I have gotten into the habit of saving the rule after every condition I move. Sometimes, using ctrl+z will put it back where it was and you can try it again.

Case sensitive string conditions are faster than case insensitive conditions.
The InActiveList condition is better than a join rule with a long time window (long > 10 minutes)
!HasVulnerability and use of asset group variable functions, are very costly.
The MatchesFilter condition can be expensive, depending on the filter, especially if that filter uses any of the heavier variable functions or is very complex.

Performance Cost of operators:

1. Most expensive: Asset Variables
2. HasVulnerability
3. InActiveList
4. Least expensive: String Operators
Note that MatchesFilter does have a performance cost (not sure why, given the rules are compiled), but the use of other more expensive conditions will definitely affect the cost of using that filter. If you have an expensive filter, and you want to do a MatchesFilter on another expensive filter, your rule will have significant performance issues

The rules do what is called “ Short-Circuit Evaluation.” This means that in a simple AND condition, if the first element is false, it doesn’t matter what the second element is, both must be true for the AND condition to be true.
A similar circumstance applies to OR condition, if the first element is true, it doesn’t matter what the second element is, because only one element needs to be true for the OR condition to be true. This means you can put the simplest, or cheapest, condition to evaluate higher up.

Rule Actions

Set the rule correlation event’s Device Event Category (DEC) to /Rule/Fire/something. See the section on Device Event Category in the Activate Indicators and Warnings page.

Any data you want to show up in the correlation event should be explicitly added to the rule’s aggregation.

There are two types of aggregation conditions. Aggregate only if these fields are identical, which is the most commonly used aggregation, and Aggregate only if these fields are unique.

If you try to put the unique information into an active list, the rules engine will give you a list of attackers (set the value to @sourceUserName, instead of $sourceUserName, for example, but you'll need to put it in a device custom string).

Rule Performance

Partial Matches

Partial matches come in two flavors. The most obvious one is with join rules, where one of the rule aliases match one or more events. These events are kept in memory.
The second partial match is the aggregation threshold (also applies for non-join rules), e.g., 5 events in 2 minutes.

Make sure that:
• Individual aliases (event 1, event 2, etc.) do not match a lot of events
• Minimize the number of aliases (joining two events is better than joining three…)
• Whenever possible, use the Consume After Match option on each alias –
o Uses the event only once to trigger the rule

The event does not need to stay in the Rule Engine for the whole time window after it produces a match
o This will reduce the number of correlation event, as a single match will not use a consumed event to match with another base event

Minimize the time windows; shorter is better. If you need more than two minutes, especially for a join rule, then consider using an active list.

If you are looking for a login to a system, followed by a connection from that system to another sensitive system, you could write a join rule, but you would need a long time window to match the two events.
If, instead, you used a session list to monitor logins, with a couple of lightweight rules to maintain its entries, you could then use a standard rule to look for connections to the sensitive system from open session entries in the SL.

If you are looking for something like file accesses on a sensitive system, and you’re looking for 10 or more such accesses over a long period of time, like say, an hour, you’re better off creating an active list to store the access information, added by a lightweight rule, with the active list's (AL’s) TTL set to your time period (there can be issues with that, too…), and have your standard rule look for the AL’s entry updated audit event and check the count to see if it is greater than or equal to you threshold (in this case, 10).

Grouping is the selection of aggregation fields. When selecting fields to aggregate, remember that you should only select the fields that you need (you need the customer resource field). Also, some fields have a much wider range of values than others.

You might get away with aggregating on the Request URL Host field, but the full URL is most likely going to be different. Use such fields only if you are looking for single events, using the On Every Event action.

Rule References and Conditions

Rules should always reference the "Generator" of the filter for their matching conditions. The Generator field is a dynamic link and will preserve through resource name changes.

Place true OR conditions or false AND conditions at the top of your filters. Type = Correlation should be placed at the bottom of the filter.

Variables and Active List Field/Column names should all be done in camelCase, starting with lowercase, and not contain any non-alpha-numeric characters.

All rules should aggregate on the customer resource, unless you are specifically looking for cross-customer events. All active and session lists and trends should include customer resource columns. All queries should also select on the customer resource; reports and query viewers do not need to show customer information, even if the query they are using has it (i.e., you can edit the report or QV to not display the column).

Rule Aggregation Settings

Always use "Resource" in aggregation; Customer Resouce, Source Zone Resource, Destination Zone Resource, and Device Zone Resource.

Below is comprehensive list of the minimum fields to be aggregated; not all may apply in all cases.

Customer Source Destination Device Misc (optional)
Zone Resource Address Address Vendor Process Name
  Hostname Hostname Product File Path
  Zone Resource Zone Resource Address File Name
  Port Port Hostname Host
  User Name User Name Zone Resource Method
  User ID User ID    
  NT Domain NT Domain    
         
<-- /editTable -->
Global variables for the following fields exist and should be used anytime aggregation in a rule is used to preserve base event data in the rule fire.

These varilables must also be set in rule actions to override the default field or no value will be pulled.

  ArcSight Schema Field   ArcSight Global Variable
User Name      
  sourceUserName ToUpper srcUserName
  destinationUserName ToUpper dstUserName
       
NT Domain      
  sourceNtDomain ToUpper srcNtDomain
  destinationNtDomain ToUpper dstNtDomain
  deviceNtDomain ToUpper dvcNtDomain
       
Host Name      
  sourceHostName ToLower srcHostName
  destinationHostName ToLower dstHostName
  deviceHostName ToLower dvcHostName
       
DNS Domain      
  sourceDnsDomain ToLower srcDnsDomain
  destinationDnsDomain ToLower dstDnsDomain
  deviceDomain    
  deviceDnsDomain ToLower dvcDnsDomain
       
Fully Qualified Domain Name      
  sourceFqdn ToLower srcFqdn
  destinationFqdn ToLower dstFqdn
<-- /editTable -->

Rule Set Event Field Actions

In addition to the above global variables, event categorization for the below fields should be manually overridden for all rules following the below schema.

Device Event Category   Category Custom Format Field   Category Significance   Category Outcome   Agent Severity   Event Annotation Stage
/Rule/Fire/Activate/<DMiD layer>/<package level>/<use case>   /Attack Life Cycle/Stage/Sub-stage   /Compromise   /Attempt   Unknown   System Monitored
    /Attack Life Cycle/Delivery   /Hostile   /Success   Low   Triage
    /Attack Life Cycle/Exploit   /Informational   /Failure   Medium    
    /Attack Life Cycle/Activities/C2   /Informationl/Alert       High    
    /Attack Life Cycle/Activities/Lateral Recon   /Informational/Error       Very-High    
    /Attack Life Cycle/Activities/Expand Access   /Informational/Warning            
    /Attack Life Cycle/Activities/Lateral Movement   /Normal            
    /Attack Life Cycle/Activities/Establish Persistence   /Recon            
    /Attack Life Cycle/Activities/Concealment   /Suspicious            
    /Attack Life Cycle/Objectives/Confidentiality                
    /Attack Life Cycle/Objectives/Integrity                
    /Attack Life Cycle/Objectives/Availability                
                     
                     
L1 - Indicators and Warnings Device Event Category   Category Custom Format Field   Category Significance   Category Outcome   Agent Severity   Event Annotation Stage
/Rule/Fire/Activate/Application Monitoring/Indicators and Warnings/<use case>   Choose one from list   Choose one from list   Choose one from list   Choose one from list   Choose one from list
/Rule/Fire/Activate/Data Monitoring/Indicators and Warnings/<use case>                    
/Rule/Fire/Activate/Host Monitoring/Indicators and Warnings/<use case>                    
/Rule/Fire/Activate/Malware Monitoring/Indicators and Warnings/<use case>                    
/Rule/Fire/Activate/Network Monitoring/Indicators and Warnings/<use case>                    
/Rule/Fire/Activate/Perimeter Monitoring/Indicators and Warnings/<use case>                    
/Rule/Fire/Activate/Threat Intelligence/Indicators and Warnings/<use case>                    
/Rule/Fire/Activate/User Monitoring/Indicators and Warnings/<use case>                    
                     
                     
L2 - Situational Awareness   Category Custom Format Field   Category Significance   Category Outcome   Agent Severity   Event Annotation Stage
/Rule/Fire/Activate/Application Monitoring/Situational Awareness/<use case>   Choose one from list   Choose one from list   Choose one from list   Choose one from list   Choose one from list
/Rule/Fire/Activate/Data Monitoring/Situational Awareness/<use case>                    
/Rule/Fire/Activate/Host Monitoring/Situational Awareness/<use case>                    
/Rule/Fire/Activate/Malware Monitoring/Situational Awareness/<use case>                    
/Rule/Fire/Activate/Network Monitoring/Situational Awareness/<use case>                    
/Rule/Fire/Activate/Perimeter Monitoring/Situational Awareness/<use case>                    
/Rule/Fire/Activate/Threat Intelligence/Situational Awareness/<use case>                    
/Rule/Fire/Activate/User Monitoring/Situational Awareness/<use case>                    
                     
                     
                     
L3 - Threat and Impact Assessment   Category Custom Format Field   Category Significance   Category Outcome   Agent Severity   Event Annotation Stage
/Rule/Fire/Activate/Application Monitoring/Threat and Impact Assessment/<use case>   Choose one from list   Choose one from list   Choose one from list   Choose one from list   Choose one from list
/Rule/Fire/Activate/Data Monitoring/Threat and Impact Assessment/<use case>                    
/Rule/Fire/Activate/Host Monitoring/Threat and Impact Assessment/<use case>                    
/Rule/Fire/Activate/Malware Monitoring/Threat and Impact Assessment/<use case>                    
/Rule/Fire/Activate/Network Monitoring/Threat and Impact Assessment/<use case>                    
/Rule/Fire/Activate/Perimeter Monitoring/Threat and Impact Assessment/<use case>                    
/Rule/Fire/Activate/Threat Intelligence/Threat and Impact Assessment/<use case>                    
/Rule/Fire/Activate/User Monitoring/Threat and Impact Assessment/<use case>                    
                     
                     
<-- /editTable -->
 

Model Categories

network asset system default application physical
Line: 299 to 138
 

things to embed from the pdf guide
Deleted:
<
<

Active Lists

When creating an active list, you should probably include the customer resource column. This makes your AL friendly for MSSPs, or for when you need to separate out data among your business units. Not every AL will need to segregate customer data, especially for a product package or an L1 solution package.

If you are going to do anything other than check an entry to see if it is in an AL, you will need to access the data through the GetActiveListValue function, which means you need to define key fields. Key fields are also required when you want multi-mapped entries, case insensitive columns, etc. Choose your key fields wisely.

Key field selection should satisfy the following conditions:
1. Use the minimum number of key fields possible, but don’t be afraid to have multiple key fields.
2. The key field(s) selected should uniquely identify an entry.
3. The event fields from which the data is taken should never be null. (Note: this is ideal, but not necessarily always possible.)
4. When using IP address fields as a key field, always add the zone field as a key field, as well.
5. Don’t forget to use the customer field as a key field.

example for tracking versions
• Customer (key)
• SystemZone (key)
• SystemAddress (key)
• SystemHostName (key)
• SystemVersion

If you’ve done your network modeling well, then you’ll want to use the asset information, instead (column set 4):
• Customer (key)
• AssetID (key)
• AssetZone
• AssetAddress
• AssetHostName
• AssetURI
• AssetVersion

Please note that most of the asset information listed is comprised of derived fields. You can get away with something more along these lines:
  • Customer (key)
  • Asset (key)
  • additional information (data you would track, or a dummy field, if necessary)
If you find the need to check a list to see whether an entry is in it, but need to use an additional AL feature, such as case insensitive columns, use a dummy string field to be the non-key data field, since you cannot have only key fields defined in a list. Your rule to add entries to the list can put whatever string is convenient, a good example would be device custom string1.

Column Ordering

We highly recommend using a consistent column ordering for active lists. This is because many use case implementations depend on monitoring active list audit events, and the entry information in these audit events is often contained within Device Custom String4. If you are tracking information regarding a host or a user, we recommend following these key field ordering examples:
Use Column1 Column2 Column3 Column4 Column5 Column6 Column7
Users Customer NT Domain User Name        
Hosts Customer Zone IP Address Host Name Asset    
Users by Hosts Customer Zone IP Address Host Name Asset NT Domain User Name
Services by Hosts Customer Zone IP Address Host Name Asset Service Name  
Tracking local Linux user accounts might be instantiated using these columns:
  • Customer (key)
  • Zone (key)
  • IP Address (key)
  • Host Name (key)
  • Asset (key)
  • Domain (key) [for Windows, this would be the NT Domain, may not be relevant for Linux hosts]
  • User Name (key)
  • User ID
Using consistent ordering will allow the use of global variables for dealing with list data in audit events for multiple lists easier to understand and maintain.
 

Filters

filter Condition: ! InActiveList – filters that use this condition cannot be used in Active Channels
Line: 351 to 157
  There are two classes of variables, global and local. Global variables are a user-definable field that can be used by other resources. Local variables are like fields, but not exactly, and are only useful in the resource in which they are defined.

-- GeorgeBoitano - 21 Jan 2016
Changed:
<
<
>
>
 

Revision 9
14 Apr 2017 - Main.PrenticeHayes
Line: 1 to 1
 
META TOPICPARENT name="HowActivateDevelopment"
Changed:
<
<

Activate Best Practices

>
>

Activate Content Development Best Practices

 

Document a set of best practices as collected from the original documentation. Tie each back to the Methodology and the Philosophy. If it gets too long, create subpages for each best practice, but be sure to create them from this page and include a link to each on this page.
Deleted:
<
<

Connector Settings

 
Deleted:
<
<
The following standard connector settings have been tabled for discussion:
  • Port/Service Mapping
  • Uppercase Username
  • Lowercase host/domain
  • Username splitting
  • Filename/path splitting
 

Events

Event Types

Revision 8
20 Mar 2017 - Main.PrenticeHayes
Line: 1 to 1
 
META TOPICPARENT name="HowActivateDevelopment"

Activate Best Practices

Line: 140 to 135
 

If you are looking for a login to a system, followed by a connection from that system to another sensitive system, you could write a join rule, but you would need a long time window to match the two events.
If, instead, you used a session list to monitor logins, with a couple of lightweight rules to maintain its entries, you could then use a standard rule to look for connections to the sensitive system from open session entries in the SL.
Changed:
<
<
If you are looking for something like file accesses on a sensitive system, and you’re looking for 10 or more such accesses over a long period of time, like say, an hour, you’re better off creating an active list to store the access information, added by a lightweight rule, with the AL’s TTL set to your time period (there can be issues with that, too…), and have your standard rule look for the AL’s entry updated audit event and check the count to see if it is greater than or equal to you threshold (in this case, 10).
>
>
If you are looking for something like file accesses on a sensitive system, and you’re looking for 10 or more such accesses over a long period of time, like say, an hour, you’re better off creating an active list to store the access information, added by a lightweight rule, with the active list's (AL’s) TTL set to your time period (there can be issues with that, too…), and have your standard rule look for the AL’s entry updated audit event and check the count to see if it is greater than or equal to you threshold (in this case, 10).
 

Grouping is the selection of aggregation fields. When selecting fields to aggregate, remember that you should only select the fields that you need (you need the customer resource field). Also, some fields have a much wider range of values than others.
Line: 308 to 304
 

Active Lists

Changed:
<
<
when creating an active list, other than to include the customer resource column

If you are going to do anything other than check an entry to see if it is in an AL, you will need to access the data through the GetActiveListValue function, which means you need to define key fields. Key fields
>
>
When creating an active list, you should probably include the customer resource column. This makes your AL friendly for MSSPs, or for when you need to separate out data among your business units. Not every AL will need to segregate customer data, especially for a product package or an L1 solution package.
 
Changed:
<
<
are also required when you want multi-mapped entries, case insensitive columns, etc. Choose your key fields wisely.
>
>
If you are going to do anything other than check an entry to see if it is in an AL, you will need to access the data through the GetActiveListValue function, which means you need to define key fields. Key fields are also required when you want multi-mapped entries, case insensitive columns, etc. Choose your key fields wisely.
 
Changed:
<
<
Key field selection should satisfy the following conditions:
1. Use the minimum number of key fields possible, but don’t be afraid to have multiple key fields.
2. The key field(s) selected should uniquely identify an entry.
3. The event fields from which the data is taken should never be null.
4. When using IP address fields as a key field, always add the zone field as a key field, as well.
5. Don’t forget to use the customer field as a key field.
>
>
Key field selection should satisfy the following conditions:
1. Use the minimum number of key fields possible, but don’t be afraid to have multiple key fields.
2. The key field(s) selected should uniquely identify an entry.
3. The event fields from which the data is taken should never be null. (Note: this is ideal, but not necessarily always possible.)
4. When using IP address fields as a key field, always add the zone field as a key field, as well.
5. Don’t forget to use the customer field as a key field.
 
Changed:
<
<
example for tracking versions
• Customer (key)
SystemZone (key)
SystemAddress (key)
SystemHostName (key)
SystemVersion

If you’ve done your network modeling well, then you’ll want to use the asset information, instead (column set 4):
• Customer (key)
AssetID (key)
AssetZone
AssetAddress
AssetHostName
AssetURI
AssetVersion
>
>
example for tracking versions
• Customer (key)
• SystemZone (key)
• SystemAddress (key)
• SystemHostName (key)
• SystemVersion

If you’ve done your network modeling well, then you’ll want to use the asset information, instead (column set 4):
• Customer (key)
• AssetID (key)
• AssetZone
• AssetAddress
• AssetHostName
• AssetURI
• AssetVersion
 
Changed:
<
<
If you find the need to check a list to see whether an entry is in it, but need to use an additional AL feature, such as case insensitive columns, use a dummy string field to be the non-key data field, since you cannot have only key fields defined in a list. Your rule to add entries to the list can put whatever string is convenient, a good example would be device custom string1.
>
>
Please note that most of the asset information listed is comprised of derived fields. You can get away with something more along these lines:
  • Customer (key)
  • Asset (key)
  • additional information (data you would track, or a dummy field, if necessary)
If you find the need to check a list to see whether an entry is in it, but need to use an additional AL feature, such as case insensitive columns, use a dummy string field to be the non-key data field, since you cannot have only key fields defined in a list. Your rule to add entries to the list can put whatever string is convenient, a good example would be device custom string1.

Column Ordering

We highly recommend using a consistent column ordering for active lists. This is because many use case implementations depend on monitoring active list audit events, and the entry information in these audit events is often contained within Device Custom String4. If you are tracking information regarding a host or a user, we recommend following these key field ordering examples:
Use Column1 Column2 Column3 Column4 Column5 Column6 Column7
Users Customer NT Domain User Name        
Hosts Customer Zone IP Address Host Name Asset    
Users by Hosts Customer Zone IP Address Host Name Asset NT Domain User Name
Services by Hosts Customer Zone IP Address Host Name Asset Service Name  
Tracking local Linux user accounts might be instantiated using these columns:
  • Customer (key)
  • Zone (key)
  • IP Address (key)
  • Host Name (key)
  • Asset (key)
  • Domain (key) [for Windows, this would be the NT Domain, may not be relevant for Linux hosts]
  • User Name (key)
  • User ID
Using consistent ordering will allow the use of global variables for dealing with list data in audit events for multiple lists easier to understand and maintain.
 

Filters

Changed:
<
<
filter Condition: InActiveList – filters that use this condition cannot be used in Active Channels
>
>
filter Condition: ! InActiveList – filters that use this condition cannot be used in Active Channels
 
Changed:
<
<
Function: Arithmetic | JavaMathematicalExpression (JME) can be used in rules, but not in queries
>
>
Function: Arithmetic | JavaMathematicalExpression (JME) can be used in rules, but not in queries
 

Filters can be used in active channels, rules, queries, data monitors, etc. In general, when in doubt about how to write a filter that may be used in multiple resource types, optimize it for rules
Changed:
<
<
Avoid using active lists (InActiveList) in your filters. Doing this makes them unusable in an active channel.
>
>
Avoid using active lists (InActiveList) in your filters. Doing this makes them unusable in an active channel.
 

If you keep it in the rule, and out of the filter, it becomes easier to create active channels for verification of your rule.
Changed:
<
<
If you really need to reference an active list and want to use the same filter in an active channel, you can use a function, GetActiveListValue, instead
>
>
If you really need to reference an active list and want to use the same filter in an active channel, you can use a function, GetActiveListValue, instead
 

Fields and Variables

Revision 7
14 Feb 2017 - Main.PrenticeHayes
Line: 1 to 1
 
META TOPICPARENT name="HowActivateDevelopment"

Activate Best Practices

Line: 52 to 55
  Type IN (Aggregated,Base)

There are many reasons for this. First, the Type field is an enumerated field, with a range of only four possible values (0 = Action, 1 = Aggregated, 2 = Base, 3 = Correlation). This makes it roughly equivalent in processing time to:
Added:
>
>
  Type = Aggregated OR Type = Base
Added:
>
>

Derived Event Fields

There are many fields that are not actually stored in a database, but are derived from one or more objects in the database. Take zones, for example. There are several zone field collections, e.g., source, destination, agent, device, etc. The fields available for any zone resource in the ArcSight CCE are (leaving out the group name):
  • zone (resource)
  • zone external ID
  • zone ID
  • zone name
  • zone URI
The zone (resource) is actually a resource reference pointing to the zone object (more or less). There is some confusing information implied by the CCE. For example, in a resource's CCE you see <group> zone, but in the rule editor's aggregation tab, you see both the <group> zone and the <group> zone resource. The resource reference is stored in the DB, or in the event, and the UI automatically breaks it up for you when you view it in the event inspector or when viewing it in a data display (trends, active|session lists, etc.). So, the external ID, ID, name, and URI fields are derived fields. This means that when you are using them in a resource's conditions, you are adding extra processing for each item (event, list entry, etc.) for each condition check. This can be especially true of the request URL group of fields (not all the request URL fields are derived), which can be significantly large strings to parse out.

An easy way to determine whether a field is derived is by looking at it in a rule editor's aggregation settings tab. If you choose the "Add..." button, the fields that are italicized are derived. This is mostly accurate, but there are a few notable exceptions. The most notable exceptions are the attacker and target fields.

Attacker and Target

These fields are completely derived from the source and destination field groups, based on the root event field, Originator. The originator field has two possible values, "Source" and "Destination", which tells the system how to derive the attacker and target fields. If the originator field is "Source", then the attacker fields are a copy of the source fields and the target fields are a copy of the destination fields. If the originator field is "Destination", then the attacker fields are a copy of destination fields and the target fields are a copy of the source fields. The latter case is for events where a device reports the source as the originator of a communication, but the destination is considered malicious. For example, if you click on a link to a malicious web server, your system is the source of the communication, but the destination web server is the attacker, and therefore, your source system is the target.

There has been a long-standing "tradition" of using the attacker and target fields for everything. This has always been controversial, and we consider it to be incorrect for a couple of reasons:
  • Use of attacker/target implies intent, and honestly, most events should not be related to any attack
  • The attacker and target fields are derived, therefore use of them, especially in rule and data monitor conditions, introduces processing overhead

ArcSight's research of events, for categorization purposes, reveals that in a very small fraction of the events we have seen, less than 1%, the originator field is set to "Destination". For this reason, in most cases, use of the source and destination fields is recommended, unless you are writing content for a use case that specifically addresses attacks. Most Activate content does this, the noticeable exception being for the L3 Impact and Threat Analysis packages.
 

Event Types and Product Packages

We have discovered that including this condition in the basic filter for a given product can eliminate some problems and provide some potential efficiencies:
Revision 6
13 Feb 2017 - Main.PrenticeHayes
Line: 1 to 1
 
META TOPICPARENT name="HowActivateDevelopment"

Activate Best Practices

Line: 106 to 101
 

There are two types of aggregation conditions. Aggregate only if these fields are identical, which is the most commonly used aggregation, and Aggregate only if these fields are unique.
Changed:
<
<
If you try to put the unique information into an active list, you will not have it, since the rules engine will not know which one to give you from multiple matches, and it won’t give you a list of attackers.
>
>
If you try to put the unique information into an active list, the rules engine will give you a list of attackers (set the value to @sourceUserName, instead of $sourceUserName, for example, but you'll need to put it in a device custom string).
 

Rule Performance

Partial Matches

Revision 5
30 Jan 2017 - Main.PrenticeHayes
Line: 1 to 1
Changed:
<
<
META TOPICPARENT name="HowActivateApproach"
>
>
META TOPICPARENT name="HowActivateDevelopment"
 

Activate Best Practices

Revision 4
28 Oct 2016 - Main.PrenticeHayes
Line: 1 to 1
 
META TOPICPARENT name="HowActivateApproach"

Activate Best Practices

Line: 62 to 67
 

Rules

There are four critical areas of the rules editor: the rule type (under the Attributes tab), Conditions, Aggregation, and Actions.
Changed:
<
<

Correlation events are events produced by correlation, i.e., they are produced by a rule correlation event is the event that is sent out when a rule or data monitor triggers

Correlated events are events of any type (Action, Aggregated, Base, or Correlation) that have been consumed by a rule.
>
>

Rule Types

 

As of ESM v6.5, there are three types of rules: Standard, Lightweight and Pre-Persistence.
Changed:
<
<
Standard rules are what everyone is pretty much everyone has seen. You can do anything a rule can do with a standard rule.
Lightweight rules are only for maintaining active and session lists. They do not send out a correlation event whenever they are triggered. If you have rules that do nothing but add or remove data from a list, convert it to a lightweight rule, unless you are also using the correlation event for consumption by another rule or a data monitor.
>
>
Standard rules are what pretty much everyone has seen. You can do anything a rule can do with a standard rule.


Lightweight rules are only for maintaining active and session lists. They do not send out a correlation event whenever they are triggered. If you have rules that do nothing but add or remove data from a list, convert it to a lightweight rule, unless you are also using the correlation event for consumption by another rule or a data monitor.
 
Changed:
<
<
Pre-persistence rules are very similar to Lightweight rules, but can only set event fields. Be very careful setting event fields, you do not necessarily want to overwrite the data that is already in that field. Any pre-persistence rule that affects events from a given device should be re-evaluated every time the device or its connector or parser is updated.
Notes: Pre-persistence rules are evaluated first. Neither pre-persistence nor lightweight rules send correlation events. This makes it difficult to test them, but there are often side-effects and workarounds.
>
>
Pre-persistence rules are very similar to Lightweight rules, but can only set event fields. Be very careful setting event fields, you do not necessarily want to overwrite the data that is already in that field. Any pre-persistence rule that affects events from a given device should be re-evaluated every time the device or its connector or parser is updated.
Note: Pre-persistence rules are evaluated first. Neither pre-persistence nor lightweight rules send correlation or action events. This makes it difficult to test them, but there are often side-effects and workarounds.
 

The first workaround for testing lightweight and pre-persistence rules is to start out with them as standard rules. Just remember to set the On Every Event trigger, and that only list actions will work for lightweight rules, and only set event field actions will work for pre-persistence rules.
Added:
>
>

Best practices for the rule development:

Each rule's correlation event should be categorized (in the Actions tab, use the setEventField action).
 
Changed:
<
<
Each rule's correlation event should be categorized (in Actions).
>
>
The rules engine examines conditions in the sequence order. The more lines the rule engine examines, the more impact the system performance will suffer.
The more rules the manager has, the more impact it will have, which is common sense
 
Changed:
<
<
It proves our assumption - the rule engine examines the condition in the sequence order. The more lines the rule engine examines, the more impact the system performance will suffer.
2. More rules the manager has, more impact it will have, which is common sense
>
>
• Put the more specific conditions at the top, such as "name StartsWith Spyware", because few events can meet it, as a result, the rule engine won't spend processing time examining the rest of the conditions.
 
Changed:
<
<
Best practice for the rule development:
• Put the more specific conditions at the top, such as "name StartsWith Spyware", because few events can meet it, as a result, the rule engine won't examine the rest of the conditions.
• Same reason as above, put the more general conditions to the bottom, such as "Type = Correlation"
• Put the more expensive condition at the bottom, such as checking if the address is in an active list.
>
>
• Same reason as above, put the more general conditions to the bottom, such as "Type IN (Aggregated,Base)"
• Put the more expensive conditions at the bottom, such as checking if the address is in an active list.
 
Changed:
<
<
Don’t use “Type = Base,” use “Type = Correlation,” instead.
>
>
Don’t use “Type = Base,” use “Type IN (Aggregated,Base),” instead.
 
Changed:
<
<
the UI sometimes loses your condition if you drop it in the wrong place. I have gotten into the habit of saving the rule after every condition I move
>
>
The UI sometimes loses your condition if you drop it in the wrong place. I have gotten into the habit of saving the rule after every condition I move. Sometimes, using ctrl+z will put it back where it was and you can try it again.
 
Changed:
<
<
Case sensitive string conditions are faster than case insensitive conditions.
The InActiveList condition is better than a join rule with a long time window (long > 2 minutes)
HasVulnerability and use of asset group variable functions are very costly.
The MatchesFilter condition can be expensive, depending on the filter, especially if that filter uses any of the heavier variable functions or is very complex.
1.14.6.1 Performance Cost of operators:
1. Most expensive: Asset Variables
2. HasVulnerability
3. InActiveList
4. Least expensive: String Operators
Note that MatchesFilter does have a performance cost (not sure why, given the rules are compiled), but the use of other more expensive conditions will definitely affect the cost of using that filter. If you have an expensive filter, and you want to do a MatchesFilter on another expensive filter, your rule will have significant performance issues
>
>
Case sensitive string conditions are faster than case insensitive conditions.
The InActiveList condition is better than a join rule with a long time window (long > 10 minutes)
!HasVulnerability and use of asset group variable functions, are very costly.
The MatchesFilter condition can be expensive, depending on the filter, especially if that filter uses any of the heavier variable functions or is very complex.

Performance Cost of operators:

 
Changed:
<
<
The rules do what is called “34TShort-Circuit Evaluation34T.” This means that in a simple AND condition, if the first element is false, it doesn’t matter what the second element is, both must be true for the AND condition to be true.
A similar circumstance applies to OR condition, if the first element is true, it doesn’t matter what the second element is, because only one element needs to be true for the OR condition to be true. This means you can put the simplest, or cheapest, condition to evaluate higher up. in the Hostile List” should go further down the AND or OR list, since active list lookups are more computationally expensive than simple field checks
>
>
1. Most expensive: Asset Variables
2. HasVulnerability
3. InActiveList
4. Least expensive: String Operators
Note that MatchesFilter does have a performance cost (not sure why, given the rules are compiled), but the use of other more expensive conditions will definitely affect the cost of using that filter. If you have an expensive filter, and you want to do a MatchesFilter on another expensive filter, your rule will have significant performance issues
 
Changed:
<
<
set the rule correlation event’s Device Event Category (DEC) to /Rule/Fire/something
>
>
The rules do what is called “ Short-Circuit Evaluation.” This means that in a simple AND condition, if the first element is false, it doesn’t matter what the second element is, both must be true for the AND condition to be true.
A similar circumstance applies to OR condition, if the first element is true, it doesn’t matter what the second element is, because only one element needs to be true for the OR condition to be true. This means you can put the simplest, or cheapest, condition to evaluate higher up.

Rule Actions

 
Changed:
<
<
Any data you want to show up in the correlation event should be explicitly added to the rule’s aggregation
>
>
Set the rule correlation event’s Device Event Category (DEC) to /Rule/Fire/something. See the section on Device Event Category in the Activate Indicators and Warnings page.
 
Changed:
<
<
two types of aggregation conditions. Aggregate only if these fields are identical, which is the most commonly used aggregation, and Aggregate only if these fields are unique.
>
>
Any data you want to show up in the correlation event should be explicitly added to the rule’s aggregation.
 
Changed:
<
<
if you try to put the unique information into an active list, you will not have it, since the rules engine will not know which one to give you from multiple matches, and it won’t give you a list of attackers.
>
>
There are two types of aggregation conditions. Aggregate only if these fields are identical, which is the most commonly used aggregation, and Aggregate only if these fields are unique.

If you try to put the unique information into an active list, you will not have it, since the rules engine will not know which one to give you from multiple matches, and it won’t give you a list of attackers.

Rule Performance

Partial Matches

 

Partial matches come in two flavors. The most obvious one is with join rules, where one of the rule aliases match one or more events. These events are kept in memory.
The second partial match is the aggregation threshold (also applies for non-join rules), e.g., 5 events in 2 minutes.
Line: 109 to 122
 

If you are looking for something like file accesses on a sensitive system, and you’re looking for 10 or more such accesses over a long period of time, like say, an hour, you’re better off creating an active list to store the access information, added by a lightweight rule, with the AL’s TTL set to your time period (there can be issues with that, too…), and have your standard rule look for the AL’s entry updated audit event and check the count to see if it is greater than or equal to you threshold (in this case, 10).
Changed:
<
<
Grouping is the selection of aggregation fields. When selecting fields to aggregate, remember that you should only select the fields that you need (you probably need the customer resource field, you just don’t know it, yet). Also, some fields have a much wider range of values than others
>
>
Grouping is the selection of aggregation fields. When selecting fields to aggregate, remember that you should only select the fields that you need (you need the customer resource field). Also, some fields have a much wider range of values than others.
 

You might get away with aggregating on the Request URL Host field, but the full URL is most likely going to be different. Use such fields only if you are looking for single events, using the On Every Event action.
Deleted:
<
<

Rule Authoring

 

Rule References and Conditions

Rules should always reference the "Generator" of the filter for their matching conditions. The Generator field is a dynamic link and will preserve through resource name changes.
Revision 3
27 Oct 2016 - Main.PrenticeHayes
Line: 1 to 1
 
META TOPICPARENT name="HowActivateApproach"

Activate Best Practices

Document a set of best practices as collected from the original documentation. Tie each back to the Methodology and the Philosophy. If it gets too long, create subpages for each best practice, but be sure to create them from this page and include a link to each on this page.
Changed:
<
<

Connector Settings

>
>

Connector Settings

 

The following standard connector settings have been tabled for discussion:
  • Port/Service Mapping
Line: 14 to 13
 
  • Lowercase host/domain
  • Username splitting
  • Filename/path splitting
Changed:
<
<

Rule Authoring

Rule References and Conditions

>
>

Events

Event Types

There are four types events recognized by ArcSight ESM:
  1. Action
  2. Aggregated
  3. Base
  4. Correlation

Base Events

These are what pretty much everyone thinks of as events. These events come from SmartConnectors, and are the basic event type.

Aggregated Events

These events are really a type of base event. These events come from SmartConnectors, and are the result of the aggregation settings of the connector. An aggregated event represents a collection of more than one base event. The field, aggregatedEventCount, tells you how many base events are represented by an aggregated event. The aggregation settings of a connector will determine how often and under what conditions base events will be combined into aggregated events. An optimal aggregation setting will not result in the loss of any important data.

Correlation Events

There are three sources for correlation events. The best known is the event resulting from a rule triggering. The second source is from data monitors. For example, a moving average data monitor event stating that a threshold has been traversed is a correlation event (an active list event that says an entry was modified is an audit event, which is actually a base event). The third source of correlation events would be other SIMs that are sending information into ArcSight ESM.

Correlated Events

Correlated events are NOT an event type. This is a property of any type of event. Correlated events are events that were used by a rule to trigger that rule, resulting in a correlation event. For example, if a rule looked for 5 events within 1 minute for the same source attempting to connect to the same destination on port 139, and the rule triggered, the result of the rule triggering is the correlation event, and the five (base, aggregated, action, or correlation) events that caused the rule to trigger are marked as correlated. Think of it this way: A correlated event was used to create a correlation event. Correlation events can be correlated, that is, a rule can use correlation events from other rules to trigger.

Action Events

These events are generated by the rules engine when a rule action is completed. For example, a rule that modifies an active list can generate an action event.

Event Types in Conditions

This applies to any resource that has conditions, so it is in this section, rather than be repeated in sections focusing on rules, filters, queries, etc.

Rules can often "accidentally" trigger off of their own correlation events, or correlation events from other rules, that can also consume another rule's correlation events. This is called looping, and is bad. For this reason, many people insert the following conditions in their rules: Type = Base

This is bad, because it explicitly ignores aggregated events. Type != Correlation

This is better, but allows action events (although this is more of a theoretical problem, it could still happen). Action events are generally not all that useful for security use cases. They are mostly useful for rules engine/system health monitoring use cases.

Optimally, this is the best condition: Type IN (Aggregated,Base)

There are many reasons for this. First, the Type field is an enumerated field, with a range of only four possible values (0 = Action, 1 = Aggregated, 2 = Base, 3 = Correlation). This makes it roughly equivalent in processing time to: Type = Aggregated OR Type = Base

Event Types and Product Packages

We have discovered that including this condition in the basic filter for a given product can eliminate some problems and provide some potential efficiencies: Type IN (Aggregated,Base)

By putting this in the base filter of a product package, any rule within that product package that uses a filter derived from that base filter does not need to use any of the anti-looping rule techniques. This requires a bit of developer discipline, because the ideal product package does not have any rules (just filters...), but there are cases where a product package might require rules. An example of this is auditd user events for Linux. The auditd user events tend to use the user ID, and not the user name. For example, the root user ID is 0, so most auditd events that reference the root user populate the user ID field with 0, and the word 'root' is not found within the event. The P-Linux product package uses a strategy to map the user IDs to the user names, and therefore, the rules in L1-User Monitoring - Indicators and Warnings package will need to consider correlation events, in addition to base and aggregated events.

Rules

There are four critical areas of the rules editor: the rule type (under the Attributes tab), Conditions, Aggregation, and Actions.

Correlation events are events produced by correlation, i.e., they are produced by a rule correlation event is the event that is sent out when a rule or data monitor triggers

Correlated events are events of any type (Action, Aggregated, Base, or Correlation) that have been consumed by a rule.

As of ESM v6.5, there are three types of rules: Standard, Lightweight and Pre-Persistence.

Standard rules are what everyone is pretty much everyone has seen. You can do anything a rule can do with a standard rule.
Lightweight rules are only for maintaining active and session lists. They do not send out a correlation event whenever they are triggered. If you have rules that do nothing but add or remove data from a list, convert it to a lightweight rule, unless you are also using the correlation event for consumption by another rule or a data monitor.
 
Changed:
<
<
Rules should always reference the "Generator" of the filter for thier matching conditions. The Generator field is a dynamic link and will preserve through resource name changes.
>
>
Pre-persistence rules are very similar to Lightweight rules, but can only set event fields. Be very careful setting event fields, you do not necessarily want to overwrite the data that is already in that field. Any pre-persistence rule that affects events from a given device should be re-evaluated every time the device or its connector or parser is updated.
Notes: Pre-persistence rules are evaluated first. Neither pre-persistence nor lightweight rules send correlation events. This makes it difficult to test them, but there are often side-effects and workarounds.

The first workaround for testing lightweight and pre-persistence rules is to start out with them as standard rules. Just remember to set the On Every Event trigger, and that only list actions will work for lightweight rules, and only set event field actions will work for pre-persistence rules.

Each rule's correlation event should be categorized (in Actions).

It proves our assumption - the rule engine examines the condition in the sequence order. The more lines the rule engine examines, the more impact the system performance will suffer.
2. More rules the manager has, more impact it will have, which is common sense

Best practice for the rule development:
• Put the more specific conditions at the top, such as "name StartsWith Spyware", because few events can meet it, as a result, the rule engine won't examine the rest of the conditions.
• Same reason as above, put the more general conditions to the bottom, such as "Type = Correlation"
• Put the more expensive condition at the bottom, such as checking if the address is in an active list.

Don’t use “Type = Base,” use “Type = Correlation,” instead.

the UI sometimes loses your condition if you drop it in the wrong place. I have gotten into the habit of saving the rule after every condition I move

Case sensitive string conditions are faster than case insensitive conditions.
The InActiveList condition is better than a join rule with a long time window (long > 2 minutes)
HasVulnerability and use of asset group variable functions are very costly.
The MatchesFilter condition can be expensive, depending on the filter, especially if that filter uses any of the heavier variable functions or is very complex.
1.14.6.1 Performance Cost of operators:
1. Most expensive: Asset Variables
2. HasVulnerability
3. InActiveList
4. Least expensive: String Operators
Note that MatchesFilter does have a performance cost (not sure why, given the rules are compiled), but the use of other more expensive conditions will definitely affect the cost of using that filter. If you have an expensive filter, and you want to do a MatchesFilter on another expensive filter, your rule will have significant performance issues

The rules do what is called “34TShort-Circuit Evaluation34T.” This means that in a simple AND condition, if the first element is false, it doesn’t matter what the second element is, both must be true for the AND condition to be true.
A similar circumstance applies to OR condition, if the first element is true, it doesn’t matter what the second element is, because only one element needs to be true for the OR condition to be true. This means you can put the simplest, or cheapest, condition to evaluate higher up. in the Hostile List” should go further down the AND or OR list, since active list lookups are more computationally expensive than simple field checks

set the rule correlation event’s Device Event Category (DEC) to /Rule/Fire/something

Any data you want to show up in the correlation event should be explicitly added to the rule’s aggregation

two types of aggregation conditions. Aggregate only if these fields are identical, which is the most commonly used aggregation, and Aggregate only if these fields are unique.

if you try to put the unique information into an active list, you will not have it, since the rules engine will not know which one to give you from multiple matches, and it won’t give you a list of attackers.

Partial matches come in two flavors. The most obvious one is with join rules, where one of the rule aliases match one or more events. These events are kept in memory.
The second partial match is the aggregation threshold (also applies for non-join rules), e.g., 5 events in 2 minutes.

Make sure that:
• Individual aliases (event 1, event 2, etc.) do not match a lot of events
• Minimize the number of aliases (joining two events is better than joining three…)
• Whenever possible, use the Consume After Match option on each alias –
o Uses the event only once to trigger the rule

The event does not need to stay in the Rule Engine for the whole time window after it produces a match
o This will reduce the number of correlation event, as a single match will not use a consumed event to match with another base event

Minimize the time windows; shorter is better. If you need more than two minutes, especially for a join rule, then consider using an active list.

If you are looking for a login to a system, followed by a connection from that system to another sensitive system, you could write a join rule, but you would need a long time window to match the two events.
If, instead, you used a session list to monitor logins, with a couple of lightweight rules to maintain its entries, you could then use a standard rule to look for connections to the sensitive system from open session entries in the SL.

If you are looking for something like file accesses on a sensitive system, and you’re looking for 10 or more such accesses over a long period of time, like say, an hour, you’re better off creating an active list to store the access information, added by a lightweight rule, with the AL’s TTL set to your time period (there can be issues with that, too…), and have your standard rule look for the AL’s entry updated audit event and check the count to see if it is greater than or equal to you threshold (in this case, 10).

Grouping is the selection of aggregation fields. When selecting fields to aggregate, remember that you should only select the fields that you need (you probably need the customer resource field, you just don’t know it, yet). Also, some fields have a much wider range of values than others

You might get away with aggregating on the Request URL Host field, but the full URL is most likely going to be different. Use such fields only if you are looking for single events, using the On Every Event action.

Rule Authoring

Rule References and Conditions

Rules should always reference the "Generator" of the filter for their matching conditions. The Generator field is a dynamic link and will preserve through resource name changes.
 

Place true OR conditions or false AND conditions at the top of your filters. Type = Correlation should be placed at the bottom of the filter.

Variables and Active List Field/Column names should all be done in camelCase, starting with lowercase, and not contain any non-alpha-numeric characters.
Changed:
<
<

Rule Aggregation Settings

>
>

All rules should aggregate on the customer resource, unless you are specifically looking for cross-customer events. All active and session lists and trends should include customer resource columns. All queries should also select on the customer resource; reports and query viewers do not need to show customer information, even if the query they are using has it (i.e., you can edit the report or QV to not display the column).

Rule Aggregation Settings

 

Always use "Resource" in aggregation; Customer Resouce, Source Zone Resource, Destination Zone Resource, and Device Zone Resource.
Line: 48 to 148
 
  sourceUserName ToUpper srcUserName
  destinationUserName ToUpper dstUserName
       
Deleted:
<
<
       
 
NT Domain      
  sourceNtDomain ToUpper srcNtDomain
  destinationNtDomain ToUpper dstNtDomain
  deviceNtDomain ToUpper dvcNtDomain
       
Deleted:
<
<
       
 
Host Name      
  sourceHostName ToLower srcHostName
  destinationHostName ToLower dstHostName
  deviceHostName ToLower dvcHostName
       
Deleted:
<
<
       
 
DNS Domain      
  sourceDnsDomain ToLower srcDnsDomain
  destinationDnsDomain ToLower dstDnsDomain
  deviceDomain    
  deviceDnsDomain ToLower dvcDnsDomain
       
Deleted:
<
<
       
 
Fully Qualified Domain Name      
  sourceFqdn ToLower srcFqdn
  destinationFqdn ToLower dstFqdn
Deleted:
<
<
       
       
 

Changed:
<
<

Rule Set Event Field Actions

>
>

Rule Set Event Field Actions

 

In addition to the above global variables, event categorization for the below fields should be manually overridden for all rules following the below schema.

Device Event Category   Category Custom Format Field   Category Significance   Category Outcome   Agent Severity   Event Annotation Stage
<-- /editTable -->
Changed:
<
<
/Rule/Fire/Activate/<DiD layer>/<package level>/<use case>   /Attack Life Cycle/Stage/Sub-stage   /Compromise   /Attempt   Unknown   System Monitored
>
>
/Rule/Fire/Activate/<DMiD layer>/<package level>/<use case>   /Attack Life Cycle/Stage/Sub-stage   /Compromise   /Attempt   Unknown   System Monitored
 
    /Attack Life Cycle/Delivery   /Hostile   /Success   Low   Triage
    /Attack Life Cycle/Exploit   /Informational   /Failure   Medium    
    /Attack Life Cycle/Activities/C2   /Informationl/Alert       High    
Line: 128 to 222
 
/Rule/Fire/Activate/User Monitoring/Threat and Impact Assessment/<use case>                    
                     
                     
Changed:
<
<

Model Categories

>
>

Model Categories

 

network asset system default application physical

models.png
Changed:
<
<

Package Submission

>
>

Packaging

When archiving resources (except users, use the “exportuser” format), use the "export" format. Also, always check the "exclude reference IDs" checkbox.

In addition to the users exception, you might find yourself needing to use a pre-populated active list, where you put specific values in a list for lookup or checking in various conditions. The export format means that no active or session list data will be included in the package. How do we get around this?

The first package uses the export format (no list data included).

The second package uses the default format (list data included).

The first package requires the second package.

Exclude all active lists that are to have pre-populated (static) data entries from the first package.

Explicitly include individual active lists that are to have pre-populated data entries in the second package.

When you export the first package, be sure to include the second package in the .arb file.
The pre-populated active lists should have their TTLs set to 0.
1.10.3 Clean Packages (no extra resources included unnecessarily)
By default, new packages will exclude most of the /All <resource_type>/ArcSight System/… level content. The current exception to this is /All Fields/ArcSight System/Event Fields. You can safely add this to the resource exclusions of the package.
1.10.3.1 Steps to follow for removing the event fields from a package:
1. In the package editor, select the Resources tab.
2. Sort the Removed Resource Column.
3. Add /All Fields/ArcSight System/Event Fields.
4. Check the If Not Included checkbox

This will avoid errors when the package is uninstalled due to the fields being in a locked group.
1.10.4 Cross Package Contamination
For long-term maintenance of packages, make sure that resources only show up in one package. If you have a resource in two packages, and modify that resource and export one of them, then the old version of that resource is in the second package. If you install them in the wrong order, your resources won’t be what you expect or need.

make sure that you're not including additional resources you don't need to export or import

Package Submission

 

If you are going to submit packages, please update your ESM’s server.properties to include this:
Added:
>
>
 
#Change package archive sort order (server.properties):
export.archive.reference.sort.order=id

This will make the XML bits of the package sort the resources by their resource ID, rather than their resource URI. This makes it easier for us to do diffs, etc., when merging versions for release.
Line: 142 to 258
  This will make the XML bits of the package sort the resources by their resource ID, rather than their resource URI. This makes it easier for us to do diffs, etc., when merging versions for release.

Once you update the server.properties file, you will need to restart your manager.
Changed:
<
<

Update ESM:

>
>

Update ESM:

 

Update your ESM’s server.properties to export packages in Resource ID order. This will make the XML bits of the package sort the resources by their resource ID, rather than their resource URI.
Changed:
<
<

Procedure:

>
>

Procedure:

 

1. Edit the /opt/arcsight/manager/config/server.properties file
Line: 158 to 273
 

3. Once you update the server.properties file, you will need to restart your manager.
Changed:
<
<

things to embed from the pdf guide

aggregation

>
>
things to embed from the pdf guide
 
Changed:
<
<
All rules should aggregate on the customer resource, unless you are specifically looking for cross-customer events. All active and session lists and trends should include customer resource columns. All queries should also select on the customer resource; reports and query viewers do not need to show customer information, even if the query they are using has it (i.e., you can edit the report or QV to not display the column).

active list

>
>

Active Lists

 

when creating an active list, other than to include the customer resource column
Line: 175 to 288
  example for tracking versions
• Customer (key)
SystemZone (key)
SystemAddress (key)
SystemHostName (key)
SystemVersion

If you’ve done your network modeling well, then you’ll want to use the asset information, instead (column set 4):
• Customer (key)
AssetID (key)
AssetZone
AssetAddress
AssetHostName
AssetURI
AssetVersion

If you find the need to check a list to see whether an entry is in it, but need to use an additional AL feature, such as case insensitive columns, use a dummy string field to be the non-key data field, since you cannot have only key fields defined in a list. Your rule to add entries to the list can put whatever string is convenient, a good example would be device custom string1.
Changed:
<
<

filter

>
>

Filters

 

filter Condition: InActiveList – filters that use this condition cannot be used in Active Channels
Line: 188 to 301
  If you keep it in the rule, and out of the filter, it becomes easier to create active channels for verification of your rule.

If you really need to reference an active list and want to use the same filter in an active channel, you can use a function, GetActiveListValue, instead
Deleted:
<
<

packaging

When archiving resources (except users, use the “exportuser” format), use the "export" format. Also, always check the "exclude reference IDs" checkbox.

In addition to the users exception, you might find yourself needing to use a pre-populated active list, where you put specific values in a list for lookup or checking in various conditions. The export format means that no active or session list data will be included in the package. How do we get around this?

The first package uses the export format (no list data included).

The second package uses the default format (list data included).

The first package requires the second package.

Exclude all active lists that are to have pre-populated (static) data entries from the first package.

Explicitly include individual active lists that are to have pre-populated data entries in the second package.

When you export the first package, be sure to include the second package in the .arb file.
The pre-populated active lists should have their TTLs set to 0.
1.10.3 Clean Packages (no extra resources included unnecessarily)
By default, new packages will exclude most of the /All <resource_type>/ArcSight System/… level content. The current exception to this is /All Fields/ArcSight System/Event Fields. You can safely add this to the resource exclusions of the package.
1.10.3.1 Steps to follow for removing the event fields from a package:
1. In the package editor, select the Resources tab.
2. Sort the Removed Resource Column.
3. Add /All Fields/ArcSight System/Event Fields.
4. Check the If Not Included checkbox

This will avoid errors when the package is uninstalled due to the fields being in a locked group.
1.10.4 Cross Package Contamination
For long-term maintenance of packages, make sure that resources only show up in one package. If you have a resource in two packages, and modify that resource and export one of them, then the old version of that resource is in the second package. If you install them in the wrong order, your resources won’t be what you expect or need.

make sure that you're not including additional resources you don't need to export or import

rules

here are four critical areas of rules: Type, Conditions, Aggregation and Actions

Correlation events are events produced by correlation, i.e., they are produced by a rule correlation event is the event that is sent out when a rule or data monitor triggers

Correlated events are events of any type (Action, Aggregated, Base, or Correlation) that have been consumed by a rule.

As of ESM v6.5, there are three types of rules: Standard, Lightweight and Pre-Persistence.

Standard rules are what everyone is pretty much everyone has seen. You can do anything a rule can do with a standard rule.
Lightweight rules are only for maintaining active and session lists. They do not send out a correlation event whenever they are triggered. If you have rules that do nothing but add or remove data from a list, convert it to a lightweight rule, unless you are also using the correlation event for consumption by another rule or a data monitor.
 
Changed:
<
<
Pre-persistence rules are very similar to Lightweight rules, but can only set event fields. Be very careful setting event fields, you do not necessarily want to overwrite the data that is already in that field. Any pre-persistence rule that affects events from a given device should be re-evaluated every time the device or its connector or parser is updated.
Notes: Pre-persistence rules are evaluated first. Neither pre-persistence nor lightweight rules send correlation events. This makes it difficult to test them, but there are often side-effects and workarounds.

The first workaround for testing lightweight and pre-persistence rules is to start out with them as standard rules. Just remember to set the On Every Event trigger, and that only list actions will work for lightweight rules, and only set event field actions will work for pre-persistence rules.

Each rule's correlation event should be categorized (in Actions).

It proves our assumption - the rule engine examines the condition in the sequence order. The more lines the rule engine examines, the more impact the system performance will suffer.
2. More rules the manager has, more impact it will have, which is common sense

Best practice for the rule development:
• Put the more specific conditions at the top, such as "name StartsWith Spyware", because few events can meet it, as a result, the rule engine won't examine the rest of the conditions.
• Same reason as above, put the more general conditions to the bottom, such as "Type = Correlation"
• Put the more expensive condition at the bottom, such as checking if the address is in an active list.

Don’t use “Type = Base,” use “Type = Correlation,” instead.

the UI sometimes loses your condition if you drop it in the wrong place. I have gotten into the habit of saving the rule after every condition I move

Case sensitive string conditions are faster than case insensitive conditions.
The InActiveList condition is better than a join rule with a long time window (long > 2 minutes)
HasVulnerability and use of asset group variable functions are very costly.
The MatchesFilter condition can be expensive, depending on the filter, especially if that filter uses any of the heavier variable functions or is very complex.
1.14.6.1 Performance Cost of operators:
1. Most expensive: Asset Variables
2. HasVulnerability
3. InActiveList
4. Least expensive: String Operators
Note that MatchesFilter does have a performance cost (not sure why, given the rules are compiled), but the use of other more expensive conditions will definitely affect the cost of using that filter. If you have an expensive filter, and you want to do a MatchesFilter on another expensive filter, your rule will have significant performance issues

The rules do what is called “34TShort-Circuit Evaluation34T.” This means that in a simple AND condition, if the first element is false, it doesn’t matter what the second element is, both must be true for the AND condition to be true.
A similar circumstance applies to OR condition, if the first element is true, it doesn’t matter what the second element is, because only one element needs to be true for the OR condition to be true. This means you can put the simplest, or cheapest, condition to evaluate higher up. in the Hostile List” should go further down the AND or OR list, since active list lookups are more computationally expensive than simple field checks

set the rule correlation event’s Device Event Category (DEC) to /Rule/Fire/something

Any data you want to show up in the correlation event should be explicitly added to the rule’s aggregation

two types of aggregation conditions. Aggregate only if these fields are identical, which is the most commonly used aggregation, and Aggregate only if these fields are unique.

if you try to put the unique information into an active list, you will not have it, since the rules engine will not know which one to give you from multiple matches, and it won’t give you a list of attackers.

Partial matches come in two flavors. The most obvious one is with join rules, where one of the rule aliases match one or more events. These events are kept in memory.
The second partial match is the aggregation threshold (also applies for non-join rules), e.g., 5 events in 2 minutes.

Make sure that:
• Individual aliases (event 1, event 2, etc.) do not match a lot of events
• Minimize the number of aliases (joining two events is better than joining three…)
• Whenever possible, use the Consume After Match option on each alias –
o Uses the event only once to trigger the rule

The event does not need to stay in the Rule Engine for the whole time window after it produces a match
o This will reduce the number of correlation event, as a single match will not use a consumed event to match with another base event

Minimize the time windows; shorter is better. If you need more than two minutes, especially for a join rule, then consider using an active list.

If you are looking for a login to a system, followed by a connection from that system to another sensitive system, you could write a join rule, but you would need a long time window to match the two events.
If, instead, you used a session list to monitor logins, with a couple of lightweight rules to maintain its entries, you could then use a standard rule to look for connections to the sensitive system from open session entries in the SL.

If you are looking for something like file accesses on a sensitive system, and you’re looking for 10 or more such accesses over a long period of time, like say, an hour, you’re better off creating an active list to store the access information, added by a lightweight rule, with the AL’s TTL set to your time period (there can be issues with that, too…), and have your standard rule look for the AL’s entry updated audit event and check the count to see if it is greater than or equal to you threshold (in this case, 10).

Grouping is the selection of aggregation fields. When selecting fields to aggregate, remember that you should only select the fields that you need (you probably need the customer resource field, you just don’t know it, yet). Also, some fields have a much wider range of values than others

You might get away with aggregating on the Request URL Host field, but the full URL is most likely going to be different. Use such fields only if you are looking for single events, using the On Every Event action.

variables

>
>

Fields and Variables

 

There are two classes of variables, global and local. Global variables are a user-definable field that can be used by other resources. Local variables are like fields, but not exactly, and are only useful in the resource in which they are defined.
Revision 2
27 Oct 2016 - Main.PrenticeHayes
Line: 1 to 1
Deleted:
<
<
 
META TOPICPARENT name="HowActivateApproach"

Activate Best Practices

Line: 25 to 24
  Variables and Active List Field/Column names should all be done in camelCase, starting with lowercase, and not contain any non-alpha-numeric characters.

Rule Aggregation Settings

Changed:
<
<
Always use "Resource" in aggregation; Customer Resouce, Attacker Zone Resource, Target Zone Resouce, and Device Zone Resource.
>
>
Always use "Resource" in aggregation; Customer Resouce, Source Zone Resource, Destination Zone Resource, and Device Zone Resource.
 

Below is comprehensive list of the minimum fields to be aggregated; not all may apply in all cases.

<-- /editTable -->
Changed:
<
<
Customer Attacker Target Device Misc (optional)
>
>
Customer Source Destination Device Misc (optional)
 
Zone Resource Address Address Vendor Process Name
  Hostname Hostname Product File Path
  Zone Resource Zone Resource Address File Name
Line: 46 to 45
 
  ArcSight Schema Field   ArcSight Global Variable
User Name      
<-- /editTable -->
Changed:
<
<
  attackerUserName ToUpper atkUserName
  targetUserName ToUpper tgtUserName
>
>
  sourceUserName ToUpper srcUserName
  destinationUserName ToUpper dstUserName
 
       
       
NT Domain      
Changed:
<
<
  attackerNtDomain ToUpper atkNtDomain
  targetNtDomain ToUpper tgtNtDomain
>
>
  sourceNtDomain ToUpper srcNtDomain
  destinationNtDomain ToUpper dstNtDomain
 
  deviceNtDomain ToUpper dvcNtDomain
       
       
Host Name      
Changed:
<
<
  attackerHostName ToLower atkHostName
  targetHostName ToLower tgtHostName
>
>
  sourceHostName ToLower srcHostName
  destinationHostName ToLower dstHostName
 
  deviceHostName ToLower dvcHostName
       
       
DNS Domain      
Changed:
<
<
  attackerDnsDomain ToLower atkDnsDomain
  targetDnsDomain ToLower tgtDnsDomain
>
>
  sourceDnsDomain ToLower srcDnsDomain
  destinationDnsDomain ToLower dstDnsDomain
 
  deviceDomain    
  deviceDnsDomain ToLower dvcDnsDomain
       
       
Fully Qualified Domain Name      
Changed:
<
<
  attackerFqdn ToLower atkFqdn
  targetFqdn ToLower tgtFqdn
>
>
  sourceFqdn ToLower srcFqdn
  destinationFqdn ToLower dstFqdn
 
       
       

 
This site is powered by FoswikiCopyright &© by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback