The netnea-CRS-Upgrading-Plugin: Technical Implementation Details   Recently updated !

This entry was posted in Security and tagged CRS modsecurity security on by

In the first part of this three-part blog post series, I introduced the seamless upgrade process from CRS v3 to CRS v4 and our netnea-crs-upgrading-plugin. In this second part, we’ll dive deep into the technical implementation details of the plugin and examine how it manages the parallel execution of two CRS versions.

Prerequisites: Parallel Installation of CRS v4 Alongside CRS v3

After downloading CRS v4 and placing it next to your existing CRS v3 installation, the first critical step involves renumbering all CRS v4 rules. This prevents rule ID collisions between the two versions.

Renumbering CRS v4 Rules

We shift all CRS v4 rule IDs from the 9xxxxx range to the 89xxxxx range using a simple sed command:

sed -i '' 's/id:\(9.....\)/id:8\1/' rules/RE*.conf

This transformation is straightforward but essential. By moving v4 rules into the 89xxxxx range, we create a clean separation that also simplifies log filtering and analysis later in the process.

Handling Variable Name Changes

One of the subtle but important changes between CRS v3 and v4 is the renaming of the inbound anomaly score variable:

  • CRS v3: tx.anomaly_score_pl1
  • CRS v4: tx.inbound_anomaly_score_pl1

This change prevents interference between the two versions’ inbound scoring mechanisms. However, the outbound scoring variable name remained unchanged in both versions: tx.outbound_anomaly_score_pl1. To avoid collisions in the outbound anomaly scoring, we must rename the CRS v4 outbound variables:

sed -i '' 's/\(tx.outbound_anomaly_score_pl.\)/\1_crs4/g' rules/RES*.conf

This renames variables like tx.outbound_anomaly_score_pl1 to tx.outbound_anomaly_score_pl1_crs4, ensuring complete isolation between the two rulesets.

Configuration Compatibility

An important benefit of this parallel setup is that most configuration variables remain compatible between versions. Your crs-setup.conf file works for both CRS v3 and v4 because the vast majority of variable names didn’t change between versions.

However, there are a few settings that require attention:

The variable tx.paranoia_level was renamed to tx.blocking_paranoia_level in CRS v4. To support both versions simultaneously, you must set both variables in rule 900000:

SecAction "id:900000,phase:1,pass,nolog,\
  setvar:tx.paranoia_level=1,\
  setvar:tx.blocking_paranoia_level=1"

Similarly, if you’re using tx.executing_paranoia_level (which allows rules to run at a higher paranoia level for detection without blocking), you must also set the new CRS v4 variable tx.detection_paranoia_level:

SecAction "id:900000,phase:1,pass,nolog,\
  setvar:tx.paranoia_level=1,\
  setvar:tx.blocking_paranoia_level=1,\
  setvar:tx.executing_paranoia_level=4,\
  setvar:tx.detection_paranoia_level=4"

The format for allowed charset values in tx.allowed_request_content_type_charset changed between versions. CRS v3 used pipe-separated values:
tx.allowed_request_content_type_charset=utf-8|iso-8859-1|iso-8859
CRS v4 requires values to be prefixed and suffixed with pipe characters:
tx.allowed_request_content_type_charset=|utf-8| |iso-8859-1| |iso-8859|
Make sure to update this variable if you’re using charset restrictions in your configuration.

One significant architectural change in CRS v4 is the removal of application-specific exclusions from crs-setup.conf. In CRS v3, you could enable exclusions for specific applications using rule 900130:

SecAction \
 "id:900130,\
  phase:1,\
  nolog,\
  pass,\
  t:none,\
  setvar:tx.crs_exclusions_cpanel=1,\
  setvar:tx.crs_exclusions_drupal=1,\
  setvar:tx.crs_exclusions_wordpress=1"

In CRS v4, this mechanism no longer exists. Application-specific exclusions are now handled through dedicated plugins that must be installed separately. See the official CRS plugin documentation for details on available application plugins and how to install them.

Loading Order Matters

CRS v4 must be loaded into your web server configuration before CRS v3. This sequence is crucial because we want CRS v4 to evaluate requests first while running in log-only mode, followed by CRS v3 which will perform the actual blocking:

# === ModSecurity Core Rule Set Inclusion

Include    /opt/apache/conf/crs4/rules/*.conf
Include    /opt/apache/conf/crs3/rules/*.conf

This loading order ensures that even if CRS v4 identifies an attack, it won’t block the request (because its blocking rules have been removed by the plugin), and CRS v3 maintains control over blocking decisions during the transition phase.

Step 1: Installing the Plugin – Parallel Mode

Understanding Parallel Mode

The installation follows the standard CRS plugin process. Once installed, the plugin defaults to parallel mode, which is enabled by rule 9527100 in the netnea-crs-upgrading-config.conf file:

# Rule 9527100 enables the parallel mode of CRS3 and CRS4
# by removing the blocking rules of CRS4.
# It's step 1 of the upgrading process as described in the README.
#
SecAction \
  "id:9527100,\
   phase:1,\
   pass,\
   nolog,\
   noauditlog,\
   setvar:'tx.reporting_upgrading=upgrading-plugin running in parallel mode',\
   tag:'netnea-crs-upgrading-plugin',\
   ver:'netnea-crs-upgrading-plugin/1.0.0',\
   ctl:ruleRemoveById=8949110,ctl:ruleRemoveById=8959100,\
   ctl:ruleRemoveById=9527101-9527799"

Let’s break down what this rule accomplishes:

  1. Removes CRS v4 blocking rules: The directives ctl:ruleRemoveById=8949110 and ctl:ruleRemoveById=8959100 disable the inbound and outbound blocking rules in CRS v4, effectively putting it into log-only mode.
  2. Disables path-based and sampling configurations: The directive ctl:ruleRemoveById=9527101-9527799 removes the subsequent plugin rules that handle path-based routing and sampling mode, since we’re not using those features yet.
  3. Preserves logging: Importantly, rule 9527800 (the reporting rule) is not removed, allowing us to track which mode the plugin is operating in.

Observing Parallel Execution

To see parallel mode in action, let’s examine the log output from a simple attack simulation:

curl 'localhost/?arg=/etc/passwd'

This request triggers multiple rules across both CRS versions. Here’s the resulting log sequence (simplified for clarity):

[2025-12-10 10:15:45] ModSecurity: Warning. [id "8930120"] [msg "OS File Access Attempt"] [ver "OWASP_CRS/4.17.1"] [data "Matched Data: etc/passwd found within ARGS:arg: /etc/passwd"]

[2025-12-10 10:15:45] ModSecurity: Warning. [id "8932160"] [msg "Remote Command Execution: Unix Shell Code Found"] [ver "OWASP_CRS/4.17.1"]

[2025-12-10 10:15:45] ModSecurity: Warning. [id "930120"] [msg "OS File Access Attempt"] [ver "OWASP_CRS/3.3.7"]

[2025-12-10 10:15:45] ModSecurity: Warning. [id "932160"] [msg "Remote Command Execution: Unix Shell Code Found"] [ver "OWASP_CRS/3.3.7"]

[2025-12-10 10:15:45] ModSecurity: Access denied with code 403 (phase 2). [id "949110"] [msg "Inbound Anomaly Score Exceeded (Total Score: 10)"] [ver "OWASP_CRS/3.3.7"]

[2025-12-10 10:15:45] ModSecurity: Warning. [id "8980170"] [msg "Anomaly Scores: (Inbound Scores: blocking=10, detection=10...)"] [ver "OWASP_CRS/4.17.1"]

[2025-12-10 10:15:45] ModSecurity: Warning. [id "980130"] [msg "Inbound Anomaly Score Exceeded (Total Inbound Score: 10...)"] [ver "OWASP_CRS/3.3.7"]

[2025-12-10 10:15:45] ModSecurity: Warning. [id "9525800"] [msg "Reporting upgrading-plugin: upgrading-plugin running in parallel mode"]

Notice the execution sequence:

  1. CRS v4 detection rules match first (IDs 8930120, 8932160)
  2. CRS v3 detection rules follow (IDs 930120, 932160)
  3. CRS v3 blocks the request (ID 949110)
  4. Both correlation/reporting rules execute (IDs 8980170, 980130)
  5. Plugin reports its operational mode (ID 9525800)

This log clearly demonstrates that both rulesets are evaluating the request, but only CRS v3 is performing blocking actions. During this parallel mode phase, you can safely tune CRS v4 by analyzing its log output without any risk to production traffic.

The Tuning Window

Parallel mode is where the bulk of your migration work happens. You remain in this phase until you’ve:

  • Identified and resolved false positives in CRS v4
  • Gained confidence in CRS v4’s behavior with your specific application
  • Decided on your tuning strategy (migrate existing exclusions vs. start fresh)

Step 2: CRS v4 Begins Blocking

Once you’ve completed your tuning work and feel confident about CRS v4’s behavior, you can begin enabling blocking selectively. This brings us to step 2 of the migration process.

Transitioning from Parallel Mode

To enable the path-based and sampling features, you must first disable parallel mode by commenting out rule 9527100 in netnea-crs-upgrading-config.conf:

# SecAction \
#   "id:9527100,\
#   ...

With parallel mode disabled, the CRS v4 blocking rules are restored, and the rest of the plugin’s rules become active.

Step 2a: Path-Based Rollout

Path-based rollout allows you to specify exactly which parts of your application should be protected by each CRS version. This is particularly valuable when different sections of your application have different tuning requirements.

Configuring CRS v3 Paths

Rule 9527300 handles requests that should continue running through CRS v3:

# Enable CRS3 based on paths configured in paths_crs3.data
SecRule REQUEST_FILENAME "@pmFromFile paths_crs3.data" \
       "id:9527300,\
        phase:1,\
        pass,\
        nolog,\
        noauditlog,\
        setvar:'tx.reporting_upgrading=Enable CRS3 based on path %{MATCHED_VAR}',\
        tag:'netnea-crs-upgrading-plugin',\
        ver:'netnea-crs-upgrading-plugin/1.0.0',\
        ctl:ruleRemoveById=8900000-8999999,\
        ctl:ruleRemoveById=9527100-9527799"

The rule uses the @pmFromFile operator to efficiently match against a list of paths defined in paths_crs3.data. When a match occurs:

  1. CRS v4 rules (8900000-8999999) are removed for this request
  2. The remaining plugin rules are disabled
  3. The request proceeds through CRS v3 only

Create paths_crs3.data in your plugin folder with paths that need CRS v3 protection:

# Paths that must still run through CRS v3
/crs3
/another_crs3_path
/legacy-admin
/old-api

Configuring CRS v4 Paths

Rule 9527310 handles requests ready for CRS v4 blocking:

# Enable CRS4 based on paths configured in paths_crs4.data
SecRule REQUEST_FILENAME "@pmFromFile paths_crs4.data" \
       "id:9527310,\
        phase:1,\
        pass,\
        nolog,\
        noauditlog,\
        setvar:'tx.reporting_upgrading=Enable CRS4 based on path %{MATCHED_VAR}',\
        tag:'netnea-crs-upgrading-plugin',\
        ver:'netnea-crs-upgrading-plugin/1.0.0',\
        ctl:ruleRemoveById=901000-999999,\
        ctl:ruleRemoveById=9527100-9527799"

This rule is the inverse of 9527300. When a path matches:

  1. CRS v3 rules (901000-999999) are removed
  2. Plugin sampling rules are disabled
  3. The request proceeds through CRS v4 in blocking mode

Create paths_crs4.data with your well-tuned paths:

# Paths that can already run through CRS v4
/crs4
/another_crs4_path
/api/v2
/modern-frontend

Strategic Path Selection

When deciding which paths to assign to CRS v4 first, consider:

  • Well-tuned endpoints: APIs or paths where you’ve already eliminated all false positives
  • Low-risk areas: Static content or read-only endpoints
  • High-value targets: Critical authentication or transaction paths where you want the most up-to-date protection

Step 2b: Sampling Mode

For paths not explicitly assigned to either CRS version, the plugin provides a sampling mechanism. This allows you to gradually expose production traffic to CRS v4 in a controlled, measurable way.

Configuring the Sampling Percentage

Rule 9527200 in netnea-crs-upgrading-config.conf controls what percentage of unassigned requests should be evaluated by CRS v4:

SecAction \
  "id:9527200,\
   phase:1,\
   pass,\
   nolog,\
   noauditlog,\
   tag:'netnea-crs-upgrading-plugin',\
   ver:'netnea-crs-upgrading-plugin/1.0.0',\
   setvar:tx.sampling_percentage_crs-upgrading=10"

In this example, 10% of unassigned requests will be processed by CRS v4 in blocking mode, while 90% continue through CRS v3. This is just the configuration. The actual sampling logic is implemented in subsequent rules.

The Sampling Mechanism

The sampling implementation in netnea-crs-upgrading-before.conf uses a technique adapted from the CRS sampling feature. Rule 9527410 (not shown here, but mirrored from CRS rule 901410) generates a pseudo-random number between 0-99 based on the SHA1 hash of the request’s unique ID.

Rule 9527450 then compares this random number against your configured percentage:

SecRule TX:sampling_rnd100_crs-upgrading "!@lt %{tx.sampling_percentage_crs-upgrading}" \
    "id:9527450,\
    phase:1,\
    pass,\
    nolog,\
    noauditlog,\
    skip:1,\
    tag:'netnea-crs-upgrading-plugin',\
    ver:'netnea-crs-upgrading-plugin/1.0.0',\
    setvar:'tx.reporting_upgrading=No explicit configured path found. Enabling sampling: Running CRS3 because chosen percentage %{TX.sampling_percentage_crs-upgrading} is lower than random number %{TX.sampling_rnd100_crs-upgrading}',\
    ctl:ruleRemoveById=8900000-8999999"

If the random number is greater than or equal to your configured percentage, this rule:

  1. Disables CRS v4 (ctl:ruleRemoveById=8900000-8999999)
  2. Sets a reporting variable explaining the decision
  3. Skips the next rule using skip:1

The request then proceeds through CRS v3. If the random number is less than the configured percentage, rule 9527450 doesn’t match, and rule 9527460 executes instead:

SecRule TX:sampling_rnd100_crs-upgrading "@unconditionalMatch" \
    "id:9527460,\
    phase:1,\
    pass,\
    nolog,\
    noauditlog,\
    tag:'netnea-crs-upgrading-plugin',\
    ver:'netnea-crs-upgrading-plugin/1.0.0',\
    setvar:'tx.reporting_upgrading=No explicit configured path found. Enabling sampling: Running CRS4 because chosen percentage %{TX.sampling_percentage_crs-upgrading} is higher than random number %{TX.sampling_rnd100_crs-upgrading}',\
    ctl:ruleRemoveById=901000-999999"

This rule:

  1. Disables CRS v3 (ctl:ruleRemoveById=901000-999999)
  2. Sets a reporting variable
  3. Allows the request to proceed through CRS v4

Note that rule 9527460 uses @unconditionalMatch, which always succeeds. However, it only executes if the previous rule didn’t trigger the skip:1 directive.

Gradual Sampling Increase

The beauty of sampling mode is its gradualism. You might start with:

setvar:tx.sampling_percentage_crs-upgrading=1

After observing logs and confirming no unexpected issues, gradually increase:

setvar:tx.sampling_percentage_crs-upgrading=5
setvar:tx.sampling_percentage_crs-upgrading=10
setvar:tx.sampling_percentage_crs-upgrading=25
setvar:tx.sampling_percentage_crs-upgrading=50
setvar:tx.sampling_percentage_crs-upgrading=75
setvar:tx.sampling_percentage_crs-upgrading=100

When you reach 100%, all unassigned traffic is running through CRS v4. Combined with gradually moving paths from paths_crs3.data to paths_crs4.data, you gain complete control over the migration pace.

Logging and Observability

Throughout the migration process, visibility into which ruleset is handling each request is crucial. Rule 9527800 in netnea-crs-upgrading-after.conf provides this observability:

# Rule 9527800 can be uncommented if logging of the netnea-crs-upgrading-plugin is needed
SecAction \
    "id:9527800,\
    phase:5,\
    pass,\
    t:none,\
    noauditlog,\
    msg:'Reporting upgrading-plugin: %{tx.reporting_upgrading}',\
    tag:'reporting',\
    tag:'netnea-crs-upgrading-plugin',\
    ver:'netnea-crs-upgrading-plugin/1.0.0'"

This reporting rule executes in phase 5 (logging phase) and outputs the value of tx.reporting_upgrading, which is set by whichever plugin rule made the routing decision.

If the logging becomes too verbose or you’re confident in your configuration, you can comment out this rule to reduce log noise.

End of the Upgrading Process

The migration is complete when one of these conditions is met:

  1. All paths have been explicitly moved to paths_crs4.data
  2. The paths_crs3.data file is empty and sampling is set to 100%

At this point, CRS v3 is no longer evaluating any production traffic. You can now:

  1. Remove CRS v3: Delete the CRS v3 directory and its configuration includes
  2. Remove the plugin: Delete the netnea-crs-upgrading-plugin directory and its configuration includes
  3. Clean up CRS v4:
    • Download a fresh copy of CRS v4 OR
    • Restore the standard 9xxxxx rule ID range (no more 89xxxxx)
    • Restore the standard variable names (tx.outbound_anomaly_score_pl1 instead of tx.outbound_anomaly_score_pl1_crs4)
  4. Remove CRS v3 exclusions: Clean up any rule exclusions that were specific to CRS v3
  5. Renumber the exclusion rules from 89xxxxx back to the standard 9xxxxx range,

While you could renumber the existing CRS v4 installation, downloading a fresh copy is simpler and ensures you have a clean, standard configuration.

Conclusion

The netnea-crs-upgrading-plugin provides a practical approach to CRS migration through three key mechanisms:

  • Parallel mode for safe initial tuning
  • Path-based routing for selective, controlled rollout
  • Sampling mode for gradual traffic migration

Together, these features give you fine-grained control over the migration process, allowing you to move at a pace that suits your organization. The plugin’s architecture ensures that security is never compromised during the transition. Your production traffic remains protected by a blocking ruleset at every stage.

In the third and final part of this series, I’ll walk through a real-world migration at one of our customer sites, showing how we used these features in practice and sharing lessons learned.

The netnea-crs-upgrading-plugin is available at: https://github.com/netnea/netnea-crs-upgrading-plugin.

Fränzi at the top of the Milan Cathedral during the CRS developer retreat in 2022

Franziska Bühler