{"id":1523,"date":"2020-01-13T10:09:40","date_gmt":"2020-01-13T09:09:40","guid":{"rendered":"http:\/\/www.netnea.com\/cms\/?p=1523"},"modified":"2020-02-05T05:22:06","modified_gmt":"2020-02-05T04:22:06","slug":"adding-empty-http-headers-via-libcurl-pycurl","status":"publish","type":"post","link":"https:\/\/www.netnea.com\/cms\/2020\/01\/13\/adding-empty-http-headers-via-libcurl-pycurl\/","title":{"rendered":"Adding empty HTTP headers via libcurl (pycurl)"},"content":{"rendered":"\n<p>When testing for the correct behavior of the OWASP ModSecurity Core Rule Set, a popular Web Application Firewall rule set, I needed to send empty Acccept- and User-Agent headers. This is relatively simple on the command line with curl:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">$&gt; curl --header \"User-Agent;\" https:\/\/example.com<\/pre>\n\n\n\n<p>Pulling this off with libcurl (pycurl in my case), was way more difficult though. It is also not really documented properly, hence this little blog post.<br><br>The tricky bit is that the Accept- and the User-Agent header are already set by default. So when you add an instruction to create an empty header, it will just add a 2nd User-Agent header (which will then be concatenated with the first one by the webserver. At least on the one I tested.)<br><br>So what you need to do is removing the existing header (suffux &#8220;:&#8221;) and then adding it anew as an empty header (suffix &#8220;;&#8221;). The following worked for me on pycurl 7.43:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">import pycurl\n\nc = pycurl.Curl()\nc.setopt(pycurl.URL, \"https:\/\/example.com\")\nc.setopt(pycurl.HTTPHEADER, [\"User-Agent:\"] + [\"User-Agent;\"])\n\nc.perform()\n<\/pre>\n\n\n\n<p>Here is a dump of the network traffic documenting the request (against localhost in cleartext). This proves that this is indeed what we wanted to get.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>    0x0030:  b7a9 3516 4745 5420 2f20 4854 5450 2f31  ..5.GET.\/.HTTP\/1\n    0x0040:  2e31 0d0a 486f 7374 3a20 6c6f 6361 6c68  .1..Host:.localh\n    0x0050:  6f73 740d 0a41 6363 6570 743a 202a 2f2a  ost..Accept:.*\/*\n    0x0060:  0d0a 5573 6572 2d41 6765 6e74 3a0d 0a0d  ..User-Agent:...\n    0x0070:  0a <\/code><\/pre>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"alignleft is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"\/cms\/wp-content\/uploads\/2016\/07\/portrait-round-300x300.png\" alt=\"\" width=\"79\" height=\"79\"\/><\/figure><\/div>\n\n\n\n<p><br><br>Christian Folini \/ <a href=\"https:\/\/twitter.com\/ChrFolini\">@ChrFolini<\/a> on Twitter<br><\/p>\n\n\n\n<p><br><\/p>\n","protected":false},"excerpt":{"rendered":"<p>When testing for the correct behavior of the OWASP ModSecurity Core Rule Set, a popular Web Application Firewall rule set, I needed to send empty Acccept- and User-Agent headers. This is relatively simple on the command line with curl: $&gt; curl &#8211;header &#8220;User-Agent;&#8221; https:\/\/example.com Pulling this off with libcurl (pycurl in my case), was way [&hellip;]<\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[5],"tags":[],"class_list":{"0":"post-1523","1":"post","2":"type-post","3":"status-publish","4":"format-standard","6":"category-linux","7":"czr-hentry"},"_links":{"self":[{"href":"https:\/\/www.netnea.com\/cms\/wp-json\/wp\/v2\/posts\/1523","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.netnea.com\/cms\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.netnea.com\/cms\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.netnea.com\/cms\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/www.netnea.com\/cms\/wp-json\/wp\/v2\/comments?post=1523"}],"version-history":[{"count":11,"href":"https:\/\/www.netnea.com\/cms\/wp-json\/wp\/v2\/posts\/1523\/revisions"}],"predecessor-version":[{"id":1546,"href":"https:\/\/www.netnea.com\/cms\/wp-json\/wp\/v2\/posts\/1523\/revisions\/1546"}],"wp:attachment":[{"href":"https:\/\/www.netnea.com\/cms\/wp-json\/wp\/v2\/media?parent=1523"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.netnea.com\/cms\/wp-json\/wp\/v2\/categories?post=1523"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.netnea.com\/cms\/wp-json\/wp\/v2\/tags?post=1523"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}