Here at netnea, our job is to help customers run and manage their ModSecurity / OWASP CRS based WAFs. Right now, the migration of existing installations to CRS 4 is a major concern for many of them. And it rarely comes alone: There are business constraints, tight schedules, a lack of resources (and in-house know-how), and very often the whole thing is entangled with the phasing out of legacy systems and a move into the cloud.
If you have a strong but outdated setup, it is hard to migrate to CRS 4 without lowering your security posture during the transition. The reason is the big delta between the CRS 3.3 and the CRS 4.x rule sets – and the new false positives that come with that delta. If you are still on CRS 3.2 or 3.1, extrapolate accordingly.
For the easy cases, where you stay on the same WAF, we have developed the netnea CRS upgrading plugin. It is still a process that is likely to take several months, but at least the path is clear.
In all the other situations – think platform change – things get hairy.
So here is a list of techniques that can help you ease the transition.
Migrate existing rule exclusions to CRS 4
The more rule exclusions you have, the more work this is going to be. On top of that, CRS 4 brings a lot of rules that undewent big changes as well as new rules, and new rules mean new false positives, especially in the higher paranoia levels.
Very often, the rule exclusions we encounter at our customers are heterogeneous and hard to understand. Some of them cover branches of an application that was retired years ago. Nobody dares to touch them.
That is why our advice is frequently to start over from scratch. The older the rule set, the larger the number of rule exclusions and the more complex they are, the more a fresh start is the better option.
Sampling
This is the technique the upgrading plugin is built on. It lets you test the water by funneling a small percentage of the traffic to the new rule set. The plugin gives you a great deal of flexibility, but it only works when you stay on the same WAF. If you are replacing the WAF, you may still be able to work with sampling – but then it has to happen on the load balancer or the ingress.
Platform first, rule set afterwards
This means you migrate to the new platform with the old rule set and you upgrade the rule set only afterwards.
This lowers the complexity, it lowers the risk and it gives you more time for the migration.
But if you change the platform and the WAF, you need to make sure the new WAF is actually able to run the old rule set.
The notorious case is a migration from ModSecurity to Coraza, and the culprit is the regular expression engine. CRS 3.x was written for PCRE, but Coraza is built on RE2. So effectively, Coraza can not run CRS 3.x – or only if you switch off a lot of rules. CRS 4 has overhauled the rule set to work on the more limited RE2, so this will not be a problem in the future. But if you are looking at ModSecurity → Coraza and CRS 3 → CRS 4 at the same time, it hurts.
Inline the new engine on the old platform
If you can somehow run the new WAF as a container on the old platform, you can funnel the traffic through the new WAF in monitoring mode (before the old WAF in blocking mode) and weed out the false positives that way. Obviously, this depends on your setup and it may cost you some performance. But it gives you a playground to test your new configuration on production traffic before you make the jump to the new platform.
Mirror client traffic to the new platform
If inlining is not an option, then mirroring may be. The old WAF stays in charge, but you duplicate the client traffic and send a copy to the new WAF on the new platform. Again, it depends on whether your setup supports this. If it does, it solves the problem for the request side. Not so much for the response side. But given that most rules are request rules and the response rules do not cause too many false positives anyway (performance problems are the more frequent issue with responses, to be honest), you can probably live with that.
Dump the HTTP traffic and use it as a test suite
It is quite simple to write a rule that saves every request into the audit log. Then you parse the audit log and turn it into a test suite. If you have too much traffic, sampling can help to keep the audit log afloat.
Now you can run this production traffic against the new platform and weed out the new false positives.
If you are in a privacy-sensitive setup, compliance will stop you from pulling this off – and they are totally right to do so. A dump of the traffic may contain passwords and all sorts of personal data. So this technique is better suited to websites where no personal data is involved.
Prioritize!
Most of our customers run a variety of services. Some of them are security-sensitive and run at higher paranoia levels, others are not. Some have been tuned down to a low anomaly score threshold, others have never seen any tuning at all.
You want to be sure you know what you are doing, so starting with one or two pilot services is good practice. But afterwards, you should prioritize the sensitive, high-security services. Their higher paranoia levels mean more false positives, and more false positives mean more time to tune and to migrate.
A particular customer had a fixed date when the old platform had to be switched off. We recommended to concentrate on the well-tuned services and to migrate the ones running in monitoring mode before the lights went out on the old platform, since false positives did not matter for them anyway. That way, the resources are fully spent where it matters. Cleaning and tuning the services in monitoring mode can come afterwards.
Raise the anomaly score threshold and tune down anew
The standard technique to tune a service is to start with a high CRS anomaly score threshold and then to tune down to the default value of 5 in several iterations. If you are experienced, if you have plenty of traffic and if change management lets you deploy and redeploy on production during the migration, this can give you decent results. But be aware that this is like a hermit crab changing its shell: The technique lowers your defenses in order to manage the false positives. You can pull this off, but be quick about it.
Go to paranoia level 1 during the migration, raise it step by step afterwards
Most false positives only show up in production. If you are at paranoia level 3 or even 4 and you did not have a chance to tune CRS 4 on production traffic before the transition (see the techniques above), then this is going to kill you on day one.
You can avoid this by raising the anomaly score threshold as explained above, but that comes with a big security risk.
A compromise is to drop to PL 1 for minimal false positives and then weed out the higher-PL false positives before you raise the PL again.
CRS supports this very well: You can block at PL 1 while running PL 2 or PL 3 in the background, slowly weeding out the false positives. When you think you are ready, you raise the PL.
Complement CRS with positive security
Say you have a business-to-business site where the only public-facing endpoint is the login page. Every attacker has to pass that step first.
In this situation, you can choose to lock down the login page with a custom positive rule set, while you run CRS on the rest of the service and tune it down again at your own pace.
Divide and rule
Maybe there are parts of your service where security is paramount and other parts where it does not matter quite as much. This lets you concentrate on the important endpoints first and raise the anomaly score threshold and / or lower the paranoia level for the other URIs.
Be creative and use your resources wisely.
Wrapping things up
A large CRS migration is never an easy task. It is also not hard science, but an exercise in applied security. Make sure you know what you are doing, bring all the various teams into the project (system engineers, application developers, business, security, compliance, customers!) and combine the techniques above into a migration plan that is tailored to your situation. We also suggest you allocate a generous reserve for this endeavor. Most customers need it.
And finally, if you think you would rather have an expert by your side: We are here for you, just drop us a message.
Christian Folini