{"id":829,"date":"2016-04-26T13:39:26","date_gmt":"2016-04-26T11:39:26","guid":{"rendered":"http:\/\/www.netnea.com\/cms\/?p=829"},"modified":"2016-07-11T14:56:48","modified_gmt":"2016-07-11T12:56:48","slug":"easing-in-conditional-modsecurity-rule-execution-based-on-pseudo-random-numbers","status":"publish","type":"post","link":"https:\/\/www.netnea.com\/cms\/2016\/04\/26\/easing-in-conditional-modsecurity-rule-execution-based-on-pseudo-random-numbers\/","title":{"rendered":"Easing-in \/ conditional ModSecurity rule execution based on pseudo random numbers"},"content":{"rendered":"

Truth be told, ModSecurity does not have a random function. If we need random numbers, we need to get them
\nourselves. Josh Zlatin has shown<\/a> how you can do this with lua. If we do not want to use lua, we are left on your own devices.<\/p>\n

I am not in need of cryptographically secure random numbers. What I need is a way to do conditional
\nbranching based on random numbers. Say you start to deploy the Core Rules on a heavy traffic site. Enabling
\nthem will likely drown you in false positives. How about enabling them for 1% of the requests and tune
\nthe basics first? Then you slowly raise the percentage of requests you funnel into the core rules…<\/p>\n

Or you have a rule, which you do not trust entirely, but you need production traffic to test it. It would
\nbe a pity to break the site, but maybe you can start with enabling the rule for every 10th request?<\/p>\n

So my need for randomness is mostly sampling used to do conditional rule execution.<\/p>\n

But how can we do this? What sources of entropy can we use? And how do we transform the entropy into a
\nvalue which can be used in a SecRule statement?<\/p>\n

I see two sources of entropy. One is the unique-id of the
\nrequest, which has some randomness included. The second one is the age of the request in microseconds. If you
\ntake but the last digit of that number, that is quite random. Take the second digit from the right and you
\nstart to see a clustering (at least I did in my tests).<\/p>\n

The problem with the unique-id is, that the entropy is distributed across the whole id, mixed with static
\nparts. Most of the data is not numeric, which complicates things further. But we can transform the id
\nwith sha1 or md5 to distribute the entropy evenly. If we do this, we can – for example – take the first
\ndigit and use that as a random number.
\nWith the duration we can use the same technique. The last digit of the duration is okay, but the rest of
\nthe entropy is best extracted via a hashing function and then the first digit.<\/p>\n

Unfortunately, a new problem arises with this hashing: there is no guarantee that sha and md5
\nhashes contain digits. So we could fall in the gap with some of our requests.<\/p>\n

As kind of a fallback, I use the final digit of the epoch, that is the seconds since the start of the
\nunix time. This combined with the final digit of the duration gives me a guarantee, that I have at least
\ntwo digits of randomness.<\/p>\n

Here is the list of digits, I am using:<\/p>\n

* First digit in the SHA1-hashed unique id (data not guaranteed)
\n* First digit in the SHA1-hashed duration (microseconds; data not guaranteed)
\n* First digit in the md5-hashed unique id (data not guaranteed)
\n* First digit in the md5-hashed duration (microseconds; data not guaranteed)
\n* Last digit in the duration (microseconds; data guaranteed)
\n* Last digit in the epoch (seconds; data guaranteed)<\/p>\n

Ideally, I am getting a number with six digits. In the worst case, I am getting only two digits
\nof a bit weaker quality. My approach is then to take the first digit of this number and fill
\nit into an environment variable RND10 and to take the first two digits and fill them into RND100.<\/p>\n

Here is the recipe, which also removes the leading “0” from RND100.<\/p>\n

# === Calculating Random Numbers\r\n#\r\n# ATTENTION: These are pseudo-random numbers. There is no security value in these numbers.\r\n#\r\n# Env variable RND10: Range from 0-9\r\n# Env variable RND100: Range from 0-99\r\n#\r\n# Entropy taken from unique id, duration of request in microseconds and unix epoch.\r\n#\r\nSecRule UNIQUE_ID  \"([0-9])\"       \"id:80000,phase:1,pass,nolog,capture,t:none,t:sha1,setvar:TX.r1=%{TX.1}\"\r\nSecRule DURATION   \"([0-9])\"       \"id:80001,phase:1,pass,nolog,capture,t:none,t:sha1,setvar:TX.r2=%{TX.1}\"\r\nSecRule UNIQUE_ID  \"([0-9])\"       \"id:80002,phase:1,pass,nolog,capture,t:none,t:md5,setvar:TX.r3=%{TX.1}\"\r\nSecRule DURATION   \"([0-9])\"       \"id:80003,phase:1,pass,nolog,capture,t:none,t:md5,setvar:TX.r4=%{TX.1}\"\r\nSecRule DURATION   \"([0-9])$\"      \"id:80004,phase:1,pass,nolog,capture,t:none,setvar:TX.r5=%{TX.1}\"\r\nSecRule TIME_EPOCH \"([0-9])$\"      \"id:80005,phase:1,pass,nolog,capture,t:none,\\\r\n                                       setvar:TX.rndtmp=%{TX.r1}%{TX.r2}%{TX.r3}%{TX.r4}%{TX.r5}%{TX.1}\"\r\n\r\nSecRule TX:rndtmp  \"^([0-9])\"      \"id:80006,phase:1,pass,log,capture,t:none,\\\r\n                                       msg:'Random number RND10: %{TX.1}',setenv:RND10=%{TX.1}\"\r\nSecRule TX:rndtmp  \"^0\"            \"id:80007,phase:1,pass,nolog,skip:1\"\r\nSecRule TX:rndtmp  \"^([0-9][0-9])\" \"id:80008,phase:1,pass,log,capture,t:none,\\\r\n                                       msg:'Random number RND100: %{TX.1}',setenv:RND100=%{TX.1},skip:1\"\r\nSecRule TX:rndtmp  \"^.([0-9])\"     \"id:80009,phase:1,pass,log,capture,t:none,\\\r\n                                       msg:'Random number RND100: %{TX.1}',setenv:RND100=%{TX.1}\"\r\n<\/pre>\n
I have tested these numbers and got fairly decent stats. Let’s say good enough for my use:<\/p>\n
Distribution of RND10 (based on a sample of 100.000 requests):<\/p>\n
0 9.95%\r\n1 9.78%\r\n2 10.06%\r\n3 10.00%\r\n4 10.14%\r\n5 10.09%\r\n6 9.94%\r\n7 10.17%\r\n8 9.93%\r\n9 9.95%\r\n<\/pre>\n
Distribution of RND100 (based on a sample of 100.000 requests):<\/p>\n
0 0.97%\r\n1 0.97%\r\n2 0.99%\r\n3 0.98%\r\n4 0.98%\r\n5 0.99%\r\n6 1.01%\r\n7 1.04%\r\n8 1.02%\r\n9 1.00%\r\n10 0.99%\r\n11 0.97%\r\n12 0.97%\r\n13 0.98%\r\n14 0.94%\r\n15 0.95%\r\n16 0.98%\r\n17 0.98%\r\n18 1.00%\r\n19 1.00%\r\n20 1.02%\r\n21 0.99%\r\n22 0.98%\r\n23 0.94%\r\n24 1.07%\r\n25 0.99%\r\n26 1.02%\r\n27 1.01%\r\n28 1.03%\r\n29 1.02%\r\n30 1.02%\r\n31 0.96%\r\n32 1.00%\r\n33 1.02%\r\n34 1.03%\r\n35 1.01%\r\n36 0.97%\r\n37 0.98%\r\n38 1.01%\r\n39 1.00%\r\n40 0.97%\r\n41 0.97%\r\n42 1.05%\r\n43 1.01%\r\n44 1.02%\r\n45 1.04%\r\n46 1.00%\r\n47 1.03%\r\n48 0.99%\r\n49 1.05%\r\n50 1.02%\r\n51 0.97%\r\n52 0.97%\r\n53 1.04%\r\n54 1.03%\r\n55 0.99%\r\n56 1.03%\r\n57 0.99%\r\n58 1.00%\r\n59 1.03%\r\n60 0.98%\r\n61 1.02%\r\n62 0.99%\r\n63 0.96%\r\n64 1.00%\r\n65 1.03%\r\n66 0.99%\r\n67 0.97%\r\n68 1.00%\r\n69 1.00%\r\n70 0.96%\r\n71 1.01%\r\n72 1.07%\r\n73 1.01%\r\n74 1.02%\r\n75 1.05%\r\n76 1.02%\r\n77 1.03%\r\n78 0.99%\r\n79 1.01%\r\n80 1.01%\r\n81 0.96%\r\n82 1.00%\r\n83 0.98%\r\n84 1.00%\r\n85 0.98%\r\n86 0.97%\r\n87 1.02%\r\n88 1.00%\r\n89 1.00%\r\n90 1.04%\r\n91 0.99%\r\n92 0.99%\r\n93 0.99%\r\n94 0.98%\r\n95 1.01%\r\n96 0.93%\r\n97 1.01%\r\n98 1.01%\r\n99 1.01%\r\n<\/pre>\n
Here is how I use this in practice. I can adjust the percentage of requests filtered by the
\ncore rules via a variable named tx.percentage_reqs_crs. In this example, the core rules are
\nexecuted for 5% of the requests.<\/p>\n
# === Calculating Random Numbers\r\n#\r\n# ATTENTION: These are pseudo-random numbers. There is no security value in these numbers.\r\n#\r\n# Env variable RND10: Range from 0-9\r\n# Env variable RND100: Range from 0-99\r\n#\r\n# Entropy taken from unique id, duration of request in microseconds and unix epoch.\r\n#\r\n\r\nSecRule UNIQUE_ID  \"([0-9])\"       \"id:80000,phase:1,pass,nolog,capture,t:none,t:sha1,setvar:TX.r1=%{TX.1}\"\r\nSecRule DURATION   \"([0-9])\"       \"id:80001,phase:1,pass,nolog,capture,t:none,t:sha1,setvar:TX.r2=%{TX.1}\"\r\nSecRule UNIQUE_ID  \"([0-9])\"       \"id:80002,phase:1,pass,nolog,capture,t:none,t:md5,setvar:TX.r3=%{TX.1}\"\r\nSecRule DURATION   \"([0-9])\"       \"id:80003,phase:1,pass,nolog,capture,t:none,t:md5,setvar:TX.r4=%{TX.1}\"\r\nSecRule DURATION   \"([0-9])$\"      \"id:80004,phase:1,pass,nolog,capture,t:none,setvar:TX.r5=%{TX.1}\"\r\nSecRule TIME_EPOCH \"([0-9])$\"      \"id:80005,phase:1,pass,nolog,capture,t:none,\\\r\n                                       setvar:TX.rndtmp=%{TX.r1}%{TX.r2}%{TX.r3}%{TX.r4}%{TX.r5}%{TX.1}\"\r\n\r\nSecRule TX:rndtmp  \"^([0-9])\"      \"id:80006,phase:1,pass,log,capture,t:none,\\\r\n                                       msg:'Random number RND10: %{TX.1}',setenv:RND10=%{TX.1}\"\r\nSecRule TX:rndtmp  \"^0\"            \"id:80007,phase:1,pass,nolog,skip:1\"\r\nSecRule TX:rndtmp  \"^([0-9][0-9])\" \"id:80008,phase:1,pass,log,capture,t:none,\\\r\n                                       msg:'Random number RND100: %{TX.1}',setenv:RND100=%{TX.1},skip:1\"\r\nSecRule TX:rndtmp  \"^.([0-9])\"     \"id:80009,phase:1,pass,log,capture,t:none,\\\r\n                                       msg:'Random number RND100: %{TX.1}',setenv:RND100=%{TX.1}\"\r\n\r\n# === ModSecurity Core Rules Inclusion\r\n\r\nSecAction \"id:'10000',phase:1,t:none,setvar:tx.percentage_reqs_crs=5\"\r\n\r\nSecRule ENV:RND100 \"!@lt %{tx.percentage_reqs_crs}\" \"id:10001,phase:1,pass,nolog,t:none,skipAfter:END_CRS_INCLUDE\"\r\nSecRule ENV:RND100 \"!@lt %{tx.percentage_reqs_crs}\" \"id:10002,phase:2,pass,nolog,t:none,skipAfter:END_CRS_INCLUDE\"\r\nSecRule ENV:RND100 \"!@lt %{tx.percentage_reqs_crs}\" \"id:10003,phase:3,pass,nolog,t:none,skipAfter:END_CRS_INCLUDE\"\r\nSecRule ENV:RND100 \"!@lt %{tx.percentage_reqs_crs}\" \"id:10004,phase:4,pass,nolog,t:none,skipAfter:END_CRS_INCLUDE\"\r\nSecRule ENV:RND100 \"!@lt %{tx.percentage_reqs_crs}\" \"id:10005,phase:5,pass,nolog,t:none,skipAfter:END_CRS_INCLUDE\"\r\n\r\nInclude \/core-rules\/*.conf\r\n\r\nSecMarker END_CRS_INCLUDE\r\n<\/pre>\n
This works nicely and should help people who want to ease into the core rules or a similar set of rules.<\/p>\n
If you think there is a misconception somewhere or if you see a better source of entropy, then please let me know.
\nOf course, you can also spoil the whole fun by providing a patch for a ModSecurity random function.<\/p>\n
 <\/p>\n
 Christian Folini Follow @ChrFolini<\/a> Tweet<\/a>
\n