Merge pull request #168 from Security-Onion-Solutions/idstools-refactor

Initial docs update for idstools refactor

Author: defensivedepth

detections.rst

@@ -30,7 +30,8 @@ Here is the list of possible status messages and what they mean:
 - **Migration Failed**: A failure occurred during the migration. The migration will stop on the first error and will not attempt to migrate to newer versions until the issue is resolved.
 - **Synchronizing**: A rule synchronization is in progress. This occurs daily, to ensure the Security Onion grid has the latest rules. 
 - **Sync Failed**: A failure occurred during the synchronization procedure. The next sync will retry within a few minutes.
-- **Rule Mismatch**: An integrity check process detected a mismatch between the deployed rules and the enabled rules. The SOC log will note the specific mismatched rules. One possible reason for this is if you had previously added custom rules to /opt/so/saltstack/local/salt/idstools/rules/local.rules and, if this is the case, then you can remove the rules from that file and re-add them using the Detections interface. Another possible reason is that :ref:`elasticsearch` has reached its disk watermark setting and is no longer allowing updates to the Detections indices.
+- **Sync Blocked**: Rule synchronization is currently blocked.
+- **Rule Mismatch**: An integrity check process detected a mismatch between the deployed rules and the enabled rules. The SOC log will note the specific mismatched rules. One possible reason is that :ref:`elasticsearch` has reached its disk watermark setting and is no longer allowing updates to the Detections indices.
 - **OK**: No known issues with the rule engine.
 
 .. tip::
@@ -147,8 +148,8 @@ Elastalert/Sigma
   - Immediate change in the UI and on disk
 
 Suricata/NIDS
-  - UI Bulk and Individual: Immediate change in the UI, disk change once the `idstools` state runs again
-  - Regex: UI and disk change once the `soc` state runs again and the :ref:`suricata` engine syncs
+  - UI Bulk and Individual: Immediate change in the UI and on disk
+  - Regex: UI and disk change once the :ref:`suricata` engine syncs
 
 Strelka/YARA
   - Immediate change in the UI, disk change once the `strelka` state runs again
@@ -160,7 +161,7 @@ Elastalert/Sigma
   - Immediate change in the UI and on disk
 
 Suricata/NIDS
-  - Immediate change in the UI, disk change once the `idstools` state runs again
+  - Immediate change in the UI and on disk
 
 Strelka/YARA
   - N/A
@@ -173,9 +174,7 @@ Elastalert/Sigma
   - Git repo (https or disk): UI and disk change once the `soc` state runs again and the :ref:`elastalert` engine syncs
 
 Suricata/NIDS
-  - ETOPEN/ETPRO: UI and disk change once the `soc` and  `idstools` states run again and the :ref:`suricata` engine syncs
-  - Custom URL: UI and disk change once the `soc` and `idstools` states run again and the :ref:`suricata` engine syncs
-  - Custom Local File: UI and disk change once the `soc` and `idstools` states run again and the :ref:`suricata` engine syncs
+  - All ruleset sources (ETOPEN, ETPRO, custom URL, local directory): UI and disk change once the :ref:`suricata` engine syncs
 
 Strelka/YARA
   - Git repo (https or disk): UI and disk change once the `soc` state runs again and the :ref:`strelka` engine syncs

images/config-item-idstools.png

@@ -87,106 +87,82 @@ You can enable external access to NIDS rules managed by :ref:`detections`. This
 - You can wait for the next grid update or click the ``SYNCHRONIZE GRID`` button under Options.
 - Once the grid is fully synchronized, the manager should listen on port 7789 for https connections from hosts defined in the ``external_suricata`` host group.
 
-Changing to a Different Ruleset
--------------------------------
+Configuring Rulesets
+--------------------
 
-Security Onion includes the Emerging Threats Open (ETOPEN) ruleset by default. If you would like to change to a different ruleset, you can do this via :ref:`administration` --> Configuration --> idstools --> config --> ruleset.
+Security Onion allows you to configure multiple NIDS rulesets. You can manage these rulesets by navigating to :ref:`administration` --> Configuration --> soc --> config --> server --> modules --> suricataengine --> rulesetSources.
 
-.. image:: images/config-item-idstools.png
-  :target: _images/config-item-idstools.png
+By default, Security Onion includes the Emerging Threats Open (ETOPEN) ruleset. You can enable additional rulesets, add custom rulesets, or disable existing ones.
 
-Security Onion offers the following choices for NIDS rulesets. The main options are ETOPEN (free) and ETPRO (commercial) but advanced users may choose a Snort ruleset if they understand the caveats as shown below.
+Ruleset Configuration Options
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-ETOPEN
-~~~~~~
+Each ruleset source has the following configuration options:
 
--  default ruleset included in Security Onion
--  optimized for :ref:`suricata`
--  **free**
+- **Ruleset Name**: Required. The unique name for this ruleset (e.g., "Emerging-Threats", "ABUSECH-SSLBL", "local-rules"). This is the name displayed in the UI.
+- **Description**: Optional description of the ruleset.
+- **Enabled**: Required. If set to false, existing rules and overrides from this ruleset will be removed.
+- **License Key**: Optional. Required for commercial rulesets like ET Pro.
+- **Source Type**: Required. Either ``url`` (downloads rules from HTTPS) or ``directory`` (reads .rules files from local filesystem).
+- **Source Path**: Required. The full URL or directory path depending on Source Type.
+- **Exclude Files**: Optional. List of file names to exclude, separated by commas (e.g., ``*deleted*, *retired*``).
+- **Ruleset License**: Required. The license type for this ruleset (e.g., "BSD", "Commercial", "CC0-1.0").
+- **Read Only**: Optional. Prevents changes to the rule itself - rules can still be enabled/disabled/tuned.
+- **Delete Unreferenced**: Optional. Deletes rules that are no longer referenced by the ruleset source.
 
-| For more information, see:
-| https://rules.emergingthreats.net/open/
+URL Source Options
+~~~~~~~~~~~~~~~~~~
 
-ETPRO
-~~~~~
+When using ``url`` as the Source Type, additional options are available:
 
--  includes ETOPEN and additional rules
--  optimized for :ref:`suricata`
--  rules retrievable as released
--  license fee per sensor (you are responsible for purchasing enough licenses for your entire deployment)
+- **urlHash**: URL to a hash file (.md5 or .sha256) for verifying the downloaded ruleset.
+- **proxyURL**: HTTP/HTTPS/SOCKS5 proxy URL for downloading the ruleset.
+- **proxyUsername**: Proxy authentication username.
+- **proxyPassword**: Proxy authentication password.
+- **proxyCACert**: Path to CA certificate file for MITM proxy verification.
+- **insecureSkipVerify**: Set to true to skip TLS certificate validation (not recommended for production).
 
-| For more information, see:
-| https://www.proofpoint.com/us/threat-insight/et-pro-ruleset  
 
-Snort Community
-~~~~~~~~~~~~~~~
-
--  NOT optimized for :ref:`suricata`
--  community-contributed rules
--  **free**
-
-| For more information, see:
-| https://www.snort.org/downloads/#rule-downloads
-| https://www.snort.org/faq/what-are-community-rules
-
-Snort Registered
+Default Rulesets
 ~~~~~~~~~~~~~~~~
 
--  NOT optimized for :ref:`suricata`
--  Snort SO (Shared Object) rules do NOT work with :ref:`suricata`
--  same rules as Snort Subscriber ruleset, except rules only retrievable after 30 days past release
--  **free**
-
-Since Shared Object rules won't work with :ref:`suricata`, you may want to disable them using a regex like ``'re:soid [0-9]+'``.
-  
-| For more information, see:
-| https://www.snort.org/downloads/#rule-downloads
-| https://snort.org/documents/registered-vs-subscriber
-
-Snort Subscriber (Talos)
-~~~~~~~~~~~~~~~~~~~~~~~~
+Emerging Threats (ETOPEN/ETPRO)
+  Security Onion includes the Emerging Threats ruleset by default. To enable ET Pro (commercial), enter your license key in the License Key field. Leave empty for ET Open (free) rules. The ruleset will be downloaded and imported within 15 minutes.
 
--  NOT optimized for :ref:`suricata`
--  Snort SO (Shared Object) rules do NOT work with :ref:`suricata`
--  rules retrievable as released
--  license fee per sensor (you are responsible for purchasing enough licenses for your entire deployment)
+  - Optimized for :ref:`suricata`
+  - ET Open is **free**, ET Pro requires a license fee per sensor
 
-Since Shared Object rules won't work with :ref:`suricata`, you may want to disable them using a regex like ``'re:soid [0-9]+'``.
+  | For more information, see:
+  | https://rules.emergingthreats.net/open/
+  | https://www.proofpoint.com/us/threat-insight/et-pro-ruleset
 
-| For more information, see:
-| https://www.snort.org/downloads/#rule-downloads
-| https://snort.org/documents/registered-vs-subscriber
+Abuse.ch SSL Blacklist (ABUSECH-SSLBL)
+  SSL certificate blacklist from Abuse.ch. Only available in non-Airgap, disabled by default.
 
-Other
-~~~~~
+  | For more information, see:
+  | https://sslbl.abuse.ch/
 
-- not officially managed/supported by Security Onion
-- license fee may or may not apply
+Local Rules
+  A directory-based ruleset source for custom local rules. Rules are read from ``/nsm/rules/custom-local-repos/local-suricata``. This ruleset is enabled by default with Read Only set to false, allowing you to edit rules directly once they are imported.
 
-If you would like to add custom rulesets, then you can do this with a configuration setting. In :ref:`soc`, navigate to :ref:`administration` --> Configuration. At the top of the page, click the ``Options`` menu and then enable the ``Show advanced settings`` option. Then filter for ``customRulesets`` and drilldown on the left side.
+Suricata Metadata Rulesets
+~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-Custom rulesets can be added either via URL or a local file placed on the Manager.
+When Suricata is configured as the metadata engine (instead of :ref:`zeek`), two additional rulesets become available:
 
-For URLs, the format is:
+SO_EXTRACTIONS
+  Extraction rules that control which file types Suricata extracts from network traffic for analysis by :ref:`strelka`. This ruleset is imported and **enabled by default** when Suricata is the metadata engine.
 
-::
-
-        {"community":true,"license":"GPLv2","ruleset":"snort-community","target-file":"community.rules","url":"https://www.snort.org/downloads/community/community-rules.tar.gz"}
-
-Here's what each option means:
+SO_FILTERS
+  Filter rules that control which metadata Suricata logs. Use these to reduce unnecessary metadata logging. This ruleset is imported but **disabled by default** when Suricata is the metadata engine.
 
-- community: Required, true or false. This disables some management options for the imported rules - they can't be deleted or edited, just tuned, duplicated, and Enabled | Disabled.
-- license: Required, the license this ruleset falls under.
-- ruleset: Required, the ruleset name or identifier.
-- target-file: Required, the name of the file that contains the rules, once it is downloaded. The file extension must be ``.rules``.
-- url: Required, the URL that the rules should be downloaded from.
+Adding Additional Rulesets
+~~~~~~~~~~~~~~~~~~~~~~
 
-For local files, the format is:
-
-::
+You can add additional rulesets via the web interface. Navigate to :ref:`administration` --> Configuration --> soc --> config --> server --> modules --> suricataengine --> rulesetSources and click the add button to create a new ruleset entry. Fill in the required fields (Ruleset Name, Source Type, Source Path, Ruleset License, and Enabled) and any optional fields as needed.
 
-        {"community":true,"license":"DRL1.1","file":"/nsm/rules/detect-suricata/custom_file/SOS-Custom_suricata.rules","ruleset":"SOS-Custom"}
+.. note::
 
-file: This is the path for the local rules file, which must be in the ``/nsm/rules/detect-suricata/custom_file/`` directory.
+        Each ruleset must have a unique name. Duplicate names will cause sync failures.
 
-The new settings will be applied within 15 minutes. At that point, you will need to wait for the scheduled rule update to take place (by default, every 24 hours), or you can force the update by navigating to :ref:`detections` --> Options dropdown menu --> Suricata --> Full Update.
+After adding or updating ruleset configuration, changes will be applied within 15 minutes.

nids.rst

@@ -93,7 +93,7 @@ By default, Security Onion uses :ref:`zeek` to record protocol metadata. If you
 -  HTTP
 -  SSL
 
-If you later find that some of that metadata is unnecessary, you can filter out the unnecessary metadata by writing rules. We have included some examples at https://raw.githubusercontent.com/Security-Onion-Solutions/securityonion/2.4/main/salt/idstools/rules/filters.rules.
+If you later find that some of that metadata is unnecessary, you can enable the SO_FILTERS ruleset to filter out unnecessary metadata. Navigate to :ref:`administration` --> Configuration --> soc --> config --> server --> modules --> suricataengine --> rulesetSources and enable the SO_FILTERS ruleset.
 
 To change your grid's metadata engine from :ref:`zeek` to Suricata, go to :ref:`administration` --> Configuration --> global --> mdengine and change the value from ``ZEEK`` to ``SURICATA``:
 
@@ -103,7 +103,7 @@ To change your grid's metadata engine from :ref:`zeek` to Suricata, go to :ref:`
 File Extraction
 ---------------
 
-If you choose Suricata for metadata, it will extract files from network traffic and :ref:`strelka` will then analyze those extracted files. If you would like to extract additional file types, then you can add file types as shown at https://raw.githubusercontent.com/Security-Onion-Solutions/securityonion/2.4/main/salt/idstools/rules/extraction.rules.
+If you choose Suricata for metadata, it will extract files from network traffic and :ref:`strelka` will then analyze those extracted files. The SO_EXTRACTIONS ruleset controls which file types are extracted and is enabled by default when Suricata is the metadata engine. You can customize file extraction by modifying rules in this ruleset.
 
 PCAP
 ----