{"id":1824,"date":"2022-01-20T15:46:41","date_gmt":"2022-01-20T14:46:41","guid":{"rendered":"https:\/\/www.netnea.com\/cms\/?p=1824"},"modified":"2022-01-20T15:50:01","modified_gmt":"2022-01-20T14:50:01","slug":"decrypt-tls-encrypted-http-traffic-for-debugging","status":"publish","type":"post","link":"https:\/\/www.netnea.com\/cms\/2022\/01\/20\/decrypt-tls-encrypted-http-traffic-for-debugging\/","title":{"rendered":"Decrypt TLS encrypted HTTP traffic for debugging"},"content":{"rendered":"\n

To debug HTTP requests, it may be useful to capture traffic and look at the packets that are sent back and forth between the client and the server.

This is trivial when HTTP requests are sent over an unencrypted channel. In this case it is easy enough to use a tool like tcpdump<\/em> to capture the packets and inspect them with a tool like Wireshark<\/em>.

For the demonstrations below, tshark<\/em> is used instead of Wireshark. Tshark is the CLI-based version of Wireshark and provides more or less the same capabilities for dissecting network packets. The debugging shown here can of course be done using the GUI-based Wireshark as well.

To start off, let us look at an example, of how we can debug HTTP traffic with tshark. First, we run a simple Apache httpd<\/em> server that accepts plain-text connections on port 80 and TLS encrypted connections on port 443. We will run Apache httpd inside a Docker container for convenience but it would work just the same for non-containerized Apache httpd installations, whether they are installed from the distro’s software repository or self-compiled<\/a>.<\/p>\n\n\n\n

To run the Apache httpd container, create a file called Dockerfile<\/em> with the following contents. This sets up an Apache httpd webserver that accepts plain-text connections on port 80 by default. Additionally, we enable TLS encrypted connections on port 443 and for that we use the snake oil certificates provided by the ssl-cert<\/em> package. These are self-signed certificates that can be used for testing. Since they are self-signed, they will not be recognized as trusted certificates by most user-agents but that is not a problem here.<\/p>\n\n\n\n

FROM httpd:2.4

# install snakeoil certificates
RUN apt update && apt install -y ssl-cert

# enable https on port 443 with snakeoil certificates
RUN sed -i \\
   -e 's\/^#\\(Include .*httpd-ssl.conf\\)\/\\1\/' \\
   -e 's\/^#\\(LoadModule .*mod_ssl.so\\)\/\\1\/' \\
   -e 's\/^#\\(LoadModule .*mod_socache_shmcb.so\\)\/\\1\/' \\
   conf\/httpd.conf
RUN sed -i \\
   -e 's|^\\(SSLCertificateFile\\).*|\\1 \/etc\/ssl\/certs\/ssl-cert-snakeoil.pem|' \\
   -e 's|^\\(SSLCertificateKeyFile\\).*|\\1 \/etc\/ssl\/private\/ssl-cert-snakeoil.key|' \\
   conf\/extra\/httpd-ssl.conf

# additionally expose port 443
EXPOSE 443<\/pre>\n\n\n\n
Then build it with this command:<\/p>\n\n\n\n
docker build -t simple-apache-httpd .\n<\/pre>\n\n\n\n
Now, run the Apache httpd server as a container:<\/p>\n\n\n\n
# -i - Keep STDIN open even if not attached.
# -t - Allocate a pseudo-TTY.
# --rm - Automatically remove the container when it exits
# --net=\"host\" - Connect a container to a network
# --name simple-apache-httpd - Assign a name to the container
# simple-apache-httpd - The IMAGE which starts the process
docker run -it --rm --net=\"host\" --name simple-apache-httpd simple-apache-httpd<\/pre>\n\n\n\n
In a separate terminal, run tcpdump to capture traffic:<\/p>\n\n\n\n
# -v - When  parsing and printing, produce (slightly more) verbose output.
# -i any - Listen on interface.
# \"host localhost\" - Filter for all packets related to localhost
# -w dump.pcap - Write the raw packets to file rather than parsing and printing them out.
sudo tcpdump -v -i any \"src localhost || dst localhost\" -w dump.pcap<\/pre>\n\n\n\n
In yet another terminal, run the curl command to perform a plaintext request to the container:<\/p>\n\n\n\n
$ curl -v http:\/\/localhost
*   Trying 127.0.0.1:80...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 80 (#0)
> GET \/ HTTP\/1.1
> Host: localhost
> User-Agent: curl\/7.68.0
> Accept: *\/*
> 
* Mark bundle as not supporting multiuse
< HTTP\/1.1 200 OK
< Date: Thu, 02 Dec 2021 12:55:40 GMT
< Server: Apache\/2.4.51 (Unix) OpenSSL\/1.1.1k
< Last-Modified: Mon, 11 Jun 2007 18:53:14 GMT
< ETag: \"2d-432a5e4a73a80\"
< Accept-Ranges: bytes
< Content-Length: 45
< Content-Type: text\/html
< 
<html><body><h1>It works!<\/h1><\/body><\/html>
* Connection #0 to host localhost left intact<\/pre>\n\n\n\n
Stop tcpdump and look at the packet capture with tshark:<\/p>\n\n\n\n
# -r dump.pcap - Read packet data from infile\n# -Y http - Cause the specified filter to be applied before printing a decoded form of packets\n$ tshark -r dump.pcap -Y http\n   48  11.912607    127.0.0.1 \u2192 127.0.0.1    HTTP 141 GET \/ HTTP\/1.1 \n   50  11.919104    127.0.0.1 \u2192 127.0.0.1    HTTP 353 HTTP\/1.1 200 OK  (text\/html)<\/pre>\n\n\n\n
We can clearly see the HTTP request we just performed with curl.

With TLS encrypted connections this is not as simple anymore. We would not see any HTTP traffic at all, instead we only see a TLS handshake and encrypted data packets. Note that we now need to provide the -k<\/code> flag for curl since the snake oil certificates are self-signed and are therefore not trusted:<\/p>\n\n\n\n
$ curl -v -k https:\/\/localhost\n*   Trying 127.0.0.1:443...\n* TCP_NODELAY set\n* Connected to localhost (127.0.0.1) port 443 (#0)\n* ALPN, offering h2\n* ALPN, offering http\/1.1\n* successfully set certificate verify locations:\n*   CAfile: \/etc\/ssl\/certs\/ca-certificates.crt\n  CApath: \/etc\/ssl\/certs\n* TLSv1.3 (OUT), TLS handshake, Client hello (1):\n* TLSv1.3 (IN), TLS handshake, Server hello (2):\n* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):\n* TLSv1.3 (IN), TLS handshake, Certificate (11):\n* TLSv1.3 (IN), TLS handshake, CERT verify (15):\n* TLSv1.3 (IN), TLS handshake, Finished (20):\n* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):\n* TLSv1.3 (OUT), TLS handshake, Finished (20):\n* SSL connection using TLSv1.3 \/ TLS_AES_256_GCM_SHA384\n* ALPN, server accepted to use http\/1.1\n* Server certificate:\n*  subject: CN=f31d50e8d088\n*  start date: Dec  2 10:39:24 2021 GMT\n*  expire date: Nov 30 10:39:24 2031 GMT\n*  issuer: CN=f31d50e8d088\n*  SSL certificate verify result: self signed certificate (18), continuing anyway.\n> GET \/ HTTP\/1.1\n> Host: localhost\n> User-Agent: curl\/7.68.0\n> Accept: *\/*\n> \n* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):\n* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):\n* old SSL session ID is stale, removing\n* Mark bundle as not supporting multiuse\n< HTTP\/1.1 200 OK\n< Date: Thu, 02 Dec 2021 13:00:23 GMT\n< Server: Apache\/2.4.51 (Unix) OpenSSL\/1.1.1k\n< Last-Modified: Mon, 11 Jun 2007 18:53:14 GMT\n< ETag: \"2d-432a5e4a73a80\"\n< Accept-Ranges: bytes\n< Content-Length: 45\n< Content-Type: text\/html\n< \n<html><body><h1>It works!<\/h1><\/body><\/html>\n* Connection #0 to host localhost left intact<\/pre>\n\n\n\n
We cannot see the HTTP request anymore:<\/p>\n\n\n\n
$ tshark -r dump.pcap -Y http<\/pre>\n\n\n\n
We can only see the encrypted TLS packets:<\/p>\n\n\n\n
$ tshark -r dump.pcap -Y tls\n   13   1.492489    127.0.0.1 \u2192 127.0.0.1    TLSv1 585 Client Hello\n   15   1.496532    127.0.0.1 \u2192 127.0.0.1    TLSv1.3 1381 Server Hello, Change Cipher Spec, ...\n   17   1.497260    127.0.0.1 \u2192 127.0.0.1    TLSv1.3 148 Change Cipher Spec, Application Data\n   19   1.497486    127.0.0.1 \u2192 127.0.0.1    TLSv1.3 163 Application Data\n   21   1.497556    127.0.0.1 \u2192 127.0.0.1    TLSv1.3 355 Application Data\n   23   1.497709    127.0.0.1 \u2192 127.0.0.1    TLSv1.3 355 Application Data\n   25   1.498220    127.0.0.1 \u2192 127.0.0.1    TLSv1.3 375 Application Data\n   27   1.498655    127.0.0.1 \u2192 127.0.0.1    TLSv1.3 92 Application Data\n   29   1.498830    127.0.0.1 \u2192 127.0.0.1    TLSv1.3 92 Application Data<\/pre>\n\n\n\n
The traffic is now encrypted and we would need to decrypt the captured packets in order to inspect the HTTP traffic.<\/p>\n\n\n\n
For the purpose of inspecting the traffic, TLS connections can be grouped into two groups. Connections that leverage Perfect Forward Secrecy (PFS)<\/em> and connections that do not.<\/p>\n\n\n\n
The main difference between the two types of connections is that connections without PFS can be decrypted after the fact using the long-lived private key of the server. For connections with PFS, this is not possible because short-lived, ephemeral keys are negotiated between the client and server. This is usually achieved by a Diffie-Hellman key exchange or a derivative thereof. The ephemeral keys generated during the key exchange are never persisted or reused for other connections. This prevents the traffic from being decrypted without being a party to the initial handshake of the connection.<\/p>\n\n\n\n
Let us first look at connections without PFS. Here, it suffices to pass the private key of the server to Wireshark to decrypt traffic. For this example, we have to ensure, that we use TLS parameters that do not leverage PFS. To achieve that, we explicitly tell curl to use the protocol TLSv1.2<\/code> and the cipher suite CAMELLIA128-SHA<\/code>.<\/p>\n\n\n\n
We repeat the example above, capturing the traffic:<\/p>\n\n\n\n
$ curl -v -k --tlsv1.2 --tls-max 1.2 --ciphers CAMELLIA128-SHA https:\/\/localhost
*   Trying 127.0.0.1:443...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http\/1.1
* Cipher selection: CAMELLIA128-SHA
* successfully set certificate verify locations:
*   CAfile: \/etc\/ssl\/certs\/ca-certificates.crt
  CApath: \/etc\/ssl\/certs
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 \/ CAMELLIA128-SHA
* ALPN, server accepted to use http\/1.1
* Server certificate:
*  subject: CN=f31d50e8d088
*  start date: Dec  2 10:39:24 2021 GMT
*  expire date: Nov 30 10:39:24 2031 GMT
*  issuer: CN=f31d50e8d088
*  SSL certificate verify result: self signed certificate (18), continuing anyway.
> GET \/ HTTP\/1.1
> Host: localhost
> User-Agent: curl\/7.68.0
> Accept: *\/*
> 
* Mark bundle as not supporting multiuse
< HTTP\/1.1 200 OK
< Date: Thu, 02 Dec 2021 13:34:01 GMT
< Server: Apache\/2.4.51 (Unix) OpenSSL\/1.1.1k
< Last-Modified: Mon, 11 Jun 2007 18:53:14 GMT
< ETag: \"2d-432a5e4a73a80\"
< Accept-Ranges: bytes
< Content-Length: 45
< Content-Type: text\/html
< 
<html><body><h1>It works!<\/h1><\/body><\/html>
* Connection #0 to host localhost left intact<\/pre>\n\n\n\n
$ tshark -r dump.pcap -Y tls\n    6   0.420061    127.0.0.1 \u2192 127.0.0.1    TLSv1 216 Client Hello\n    8   0.420403    127.0.0.1 \u2192 127.0.0.1    TLSv1.2 943 Server Hello, Certificate, Server Hello Done\n   10   0.420756    127.0.0.1 \u2192 127.0.0.1    TLSv1.2 414 Client Key Exchange, Change Cipher Spec, ...\n   12   0.421699    127.0.0.1 \u2192 127.0.0.1    TLSv1.2 147 Change Cipher Spec, Encrypted Handshake Message\n   14   0.421923    127.0.0.1 \u2192 127.0.0.1    TLSv1.2 189 Application Data\n   16   0.422142    127.0.0.1 \u2192 127.0.0.1    TLSv1.2 397 Application Data\n   18   0.422378    127.0.0.1 \u2192 127.0.0.1    TLSv1.2 125 Encrypted Alert\n   20   0.422497    127.0.0.1 \u2192 127.0.0.1    TLSv1.2 125 Encrypted Alert<\/pre>\n\n\n\n
Now we extract the TLS private key of the server:<\/p>\n\n\n\n
docker exec -it simple-apache-httpd cat \/etc\/ssl\/private\/ssl-cert-snakeoil.key > ssl-cert-snakeoil.key<\/pre>\n\n\n\n
This allow us to use the private key in tshark by specifying it using the -o<\/code> flag:<\/p>\n\n\n\n
# -r dump.pcap - Read packet data from infile\n# -Y http - Cause the specified filter to be applied before printing a decoded form of packets\n# -o tls.keys_list:,,,ssl-cert-snakeoil.key - override values from the preferences files\n$ tshark -o tls.keys_list:,,,ssl-cert-snakeoil.key -r dump.pcap -Y tls\n    6   0.420061    127.0.0.1 \u2192 127.0.0.1    TLSv1 216 Client Hello\n    8   0.420403    127.0.0.1 \u2192 127.0.0.1    TLSv1.2 943 Server Hello, Certificate, Server Hello Done\n   10   0.420756    127.0.0.1 \u2192 127.0.0.1    TLSv1.2 414 Client Key Exchange, Change Cipher Spec, Finished\n   12   0.421699    127.0.0.1 \u2192 127.0.0.1    TLSv1.2 147 Change Cipher Spec, Finished\n   14   0.421923    127.0.0.1 \u2192 127.0.0.1    HTTP 189 GET \/ HTTP\/1.1 \n   16   0.422142    127.0.0.1 \u2192 127.0.0.1    HTTP 397 HTTP\/1.1 200 OK  (text\/html)\n   18   0.422378    127.0.0.1 \u2192 127.0.0.1    TLSv1.2 125 Alert (Level: Warning, Description: Close Notify)\n   20   0.422497    127.0.0.1 \u2192 127.0.0.1    TLSv1.2 125 Alert (Level: Warning, Description: Close Notify)<\/pre>\n\n\n\n
As stated above, this approach will not work with PFS. This means that if only TLS parameters with PFS are available, it will not be possible to use the approach described above to decrypt traffic for debugging. For TLSv1.3<\/code> this is always the case because TLSv1.3 mandates PFS. We do not have to specify this explicitly as curl will automatically use TLSv1.3<\/code>.<\/p>\n\n\n\n
To illustrate this, we show an example just as before.<\/p>\n\n\n\n
$ curl -v -k https:\/\/localhost
*   Trying 127.0.0.1:443...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http\/1.1
* successfully set certificate verify locations:
*   CAfile: \/etc\/ssl\/certs\/ca-certificates.crt
  CApath: \/etc\/ssl\/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 \/ TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http\/1.1
* Server certificate:
*  subject: CN=f31d50e8d088
*  start date: Dec  2 10:39:24 2021 GMT
*  expire date: Nov 30 10:39:24 2031 GMT
*  issuer: CN=f31d50e8d088
*  SSL certificate verify result: self signed certificate (18), continuing anyway.
> GET \/ HTTP\/1.1
> Host: localhost
> User-Agent: curl\/7.68.0
> Accept: *\/*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Mark bundle as not supporting multiuse
< HTTP\/1.1 200 OK
< Date: Thu, 02 Dec 2021 14:06:14 GMT
< Server: Apache\/2.4.51 (Unix) OpenSSL\/1.1.1k
< Last-Modified: Mon, 11 Jun 2007 18:53:14 GMT
< ETag: \"2d-432a5e4a73a80\"
< Accept-Ranges: bytes
< Content-Length: 45
< Content-Type: text\/html
< 
<html><body><h1>It works!<\/h1><\/body><\/html>
* Connection #0 to host localhost left intact<\/pre>\n\n\n\n
With PFS, we again see only the TLS handshake. Despite providing the servers private key as before, the packets remain encrypted.<\/p>\n\n\n\n
$ tshark -o tls.keys_list:,,,ssl-cert-snakeoil.key -r dump.pcap -Y tls\n   20   1.660863    127.0.0.1 \u2192 127.0.0.1    TLSv1 585 Client Hello\n   22   1.676658    127.0.0.1 \u2192 127.0.0.1    TLSv1.3 1381 Server Hello, Change Cipher Spec, ...\n   24   1.677228    127.0.0.1 \u2192 127.0.0.1    TLSv1.3 148 Change Cipher Spec, Application Data\n   26   1.677406    127.0.0.1 \u2192 127.0.0.1    TLSv1.3 355 Application Data\n   27   1.677415    127.0.0.1 \u2192 127.0.0.1    TLSv1.3 163 Application Data\n   30   1.677504    127.0.0.1 \u2192 127.0.0.1    TLSv1.3 355 Application Data\n   32   1.680826    127.0.0.1 \u2192 127.0.0.1    TLSv1.3 375 Application Data\n   34   1.681131    127.0.0.1 \u2192 127.0.0.1    TLSv1.3 92 Application Data\n   36   1.681262    127.0.0.1 \u2192 127.0.0.1    TLSv1.3 92 Application Data<\/pre>\n\n\n\n
In order to decrypt this traffic we will have to log the ephemeral keys on either the client or the server side.
More and more technologies adopt the SSLKEYLOGFILE environment variable as a way to specify a log file where the ephemeral keys can be logged. These logs can then be passed to tshark to decrypt the traffic, similarly to how we used the servers private key before. Note that the ephemeral keys are not logged at all if this environment variable is not set.<\/p>\n\n\n\n
In order to inspect traffic the SSLKEYLOGFILE environment variable must be set on the server or client side while<\/em> the traffic is generated. With this log being generated, capturing and decrypting traffic works very similarly to how it worked before.<\/p>\n\n\n\n
Let us first run an example where we set the SSLKEYLOGFILE environment variable on the client side with curl.
Make sure you use curl 7.58<\/em> or newer for this to work.<\/p>\n\n\n\n
$ SSLKEYLOGFILE=sslkeylogfile.log curl -v -k https:\/\/localhost
*   Trying 127.0.0.1:443...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http\/1.1
* successfully set certificate verify locations:
*   CAfile: \/etc\/ssl\/certs\/ca-certificates.crt
  CApath: \/etc\/ssl\/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 \/ TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http\/1.1
* Server certificate:
*  subject: CN=f31d50e8d088
*  start date: Dec  2 10:39:24 2021 GMT
*  expire date: Nov 30 10:39:24 2031 GMT
*  issuer: CN=f31d50e8d088
*  SSL certificate verify result: self signed certificate (18), continuing anyway.
> GET \/ HTTP\/1.1
> Host: localhost
> User-Agent: curl\/7.68.0
> Accept: *\/*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Mark bundle as not supporting multiuse
< HTTP\/1.1 200 OK
< Date: Thu, 02 Dec 2021 14:17:04 GMT
< Server: Apache\/2.4.51 (Unix) OpenSSL\/1.1.1k
< Last-Modified: Mon, 11 Jun 2007 18:53:14 GMT
< ETag: \"2d-432a5e4a73a80\"
< Accept-Ranges: bytes
< Content-Length: 45
< Content-Type: text\/html
< 
<html><body><h1>It works!<\/h1><\/body><\/html>
* Connection #0 to host localhost left intact<\/pre>\n\n\n\n
With the key log written by curl, we can decrypt the traffic again. Note that the server’s private key is not required anymore, the key log file is all we need.<\/p>\n\n\n\n
# -r dump.pcap - Read packet data from infile\n# -Y http - Cause the specified filter to be applied before printing a decoded form of packets\n# -o tls.keylog_file:sslkeylogfile.log - override values from the preferences files\n$ tshark -o tls.keylog_file:sslkeylogfile.log -r dump.pcap -Y tls\n    4   0.007706    127.0.0.1 \u2192 127.0.0.1    TLSv1 585 Client Hello\n    6   0.009024    127.0.0.1 \u2192 127.0.0.1    TLSv1.3 1381 Server Hello, Change Cipher Spec, ...\n    8   0.009576    127.0.0.1 \u2192 127.0.0.1    TLSv1.3 148 Change Cipher Spec, Finished\n   10   0.009790    127.0.0.1 \u2192 127.0.0.1    HTTP 163 GET \/ HTTP\/1.1 \n   11   0.009796    127.0.0.1 \u2192 127.0.0.1    TLSv1.3 355 New Session Ticket\n   14   0.009941    127.0.0.1 \u2192 127.0.0.1    TLSv1.3 355 New Session Ticket\n   16   0.010132    127.0.0.1 \u2192 127.0.0.1    HTTP 375 HTTP\/1.1 200 OK  (text\/html)\n   18   0.010355    127.0.0.1 \u2192 127.0.0.1    TLSv1.3 92 Alert (Level: Warning, Description: Close Notify)\n   20   0.010458    127.0.0.1 \u2192 127.0.0.1    TLSv1.3 92 Alert (Level: Warning, Description: Close Notify)<\/pre>\n\n\n\n
Similarly, we can set the SSLKEYLOGFILE environment variable on the server side.
For this to work, Apache httpd 2.4.49<\/em> or later is required.<\/p>\n\n\n\n
We restart the server container and pass the SSLKEYLOGFILE environment variable.<\/p>\n\n\n\n
docker run -it --rm --net=\"host\" --name simple-apache-httpd -e SSLKEYLOGFILE=\"\/usr\/local\/apache2\/logs\/sslkeylogfile.log\" simple-apache-httpd<\/pre>\n\n\n\n
The following log entry signals that the key log file will be written:<\/p>\n\n\n\n
[Tue Dec 21 20:03:20.441733 2021] [ssl:notice] [pid 1:tid 139780428569920] AH10227: Init: Logging SSL private key material to \/usr\/local\/apache2\/logs\/sslkeylogfile.log<\/pre>\n\n\n\n
We are ready to repeat the request:<\/p>\n\n\n\n
$ curl -v -k https:\/\/localhost
*   Trying 127.0.0.1:443...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http\/1.1
* successfully set certificate verify locations:
*   CAfile: \/etc\/ssl\/certs\/ca-certificates.crt
  CApath: \/etc\/ssl\/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 \/ TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http\/1.1
* Server certificate:
*  subject: CN=f31d50e8d088
*  start date: Dec  2 10:39:24 2021 GMT
*  expire date: Nov 30 10:39:24 2031 GMT
*  issuer: CN=f31d50e8d088
*  SSL certificate verify result: self signed certificate (18), continuing anyway.
> GET \/ HTTP\/1.1
> Host: localhost
> User-Agent: curl\/7.68.0
> Accept: *\/*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Mark bundle as not supporting multiuse
< HTTP\/1.1 200 OK
< Date: Thu, 02 Dec 2021 14:23:13 GMT
< Server: Apache\/2.4.51 (Unix) OpenSSL\/1.1.1k
< Last-Modified: Mon, 11 Jun 2007 18:53:14 GMT
< ETag: \"2d-432a5e4a73a80\"
< Accept-Ranges: bytes
< Content-Length: 45
< Content-Type: text\/html
< 
<html><body><h1>It works!<\/h1><\/body><\/html>
* Connection #0 to host localhost left intact<\/pre>\n\n\n\n
After we performed and captured the request, we need to get the log file from inside the container to use it with tshark.<\/p>\n\n\n\n
docker exec -it simple-apache-httpd cat \/usr\/local\/apache2\/logs\/sslkeylogfile.log > sslkeylogfile.log<\/pre>\n\n\n\n
Now we can decrypt the traffic as before by again passing the key log file to tshark:<\/p>\n\n\n\n
$ tshark -o tls.keylog_file:sslkeylogfile.log -r dump.pcap -Y tls\n    6   0.033156    127.0.0.1 \u2192 127.0.0.1    TLSv1 585 Client Hello\n    8   0.035385    127.0.0.1 \u2192 127.0.0.1    TLSv1.3 1381 Server Hello, Change Cipher Spec, ...\n   10   0.035890    127.0.0.1 \u2192 127.0.0.1    TLSv1.3 148 Change Cipher Spec, Finished\n   12   0.036026    127.0.0.1 \u2192 127.0.0.1    HTTP 163 GET \/ HTTP\/1.1 \n   14   0.036078    127.0.0.1 \u2192 127.0.0.1    TLSv1.3 355 New Session Ticket\n   16   0.036194    127.0.0.1 \u2192 127.0.0.1    TLSv1.3 355 New Session Ticket\n   18   0.036511    127.0.0.1 \u2192 127.0.0.1    HTTP 375 HTTP\/1.1 200 OK  (text\/html)\n   20   0.036814    127.0.0.1 \u2192 127.0.0.1    TLSv1.3 92 Alert (Level: Warning, Description: Close Notify)\n   22   0.036932    127.0.0.1 \u2192 127.0.0.1    TLSv1.3 92 Alert (Level: Warning, Description: Close Notify)<\/pre>\n\n\n\n
This approach, using the SSLKEYLOGFILE environment variable works with some browsers as well. However, it might be disabled through a compile time variable because of the risk it poses. If an attacker manages to set this environment variable for either the client or server and has access to the key log, the attacker could easily break TLS encryption as shown above.<\/p>\n\n\n\n
For this reason it might be a good idea to monitor the logs for the entry indicating the use of the SSLKEYLOGFILE environment variable as shown above or even add a separate log, where the SSLKEYLOGFILE variable is logged if present. This allows you to determine after the fact which requests were affected, if the environment variable was set.<\/p>\n\n\n\n
Further, it might be a good idea to monitor the environment variables of all processes to determine if this variable is set anywhere. An easy way to do this manually is the following command. It shows that the SSLKEYLOGFILE environment variable is in use and where the key log file is written:<\/p>\n\n\n\n
$ sudo cat \/proc\/*\/environ | tr '\\0' '\\n' | grep SSLKEYLOGFILE
SSLKEYLOGFILE=\/usr\/local\/apache2\/logs\/sslkeylogfile.log
SSLKEYLOGFILE=\/usr\/local\/apache2\/logs\/sslkeylogfile.log
SSLKEYLOGFILE=\/usr\/local\/apache2\/logs\/sslkeylogfile.log
SSLKEYLOGFILE=\/usr\/local\/apache2\/logs\/sslkeylogfile.log<\/pre>\n\n\n\n
To summarize, we see that the introduction of the SSLKEYLOGFILE environment variable can be of great benefit when debugging TLS encrypted connections. However, it also introduces a new attack vector to break TLS encryption that needs to be addressed by disabling or at least by monitoring its use.<\/p>\n","protected":false},"excerpt":{"rendered":"
To debug HTTP requests, it may be useful to capture traffic and look at the packets that are sent back and forth between the client and the server. This is trivial when HTTP requests are sent over an unencrypted channel. In this case it is easy enough to use a tool like tcpdump to capture […]<\/p>\n","protected":false},"author":7,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[4],"tags":[9,15,54,56,58,57,55],"class_list":{"0":"post-1824","1":"post","2":"type-post","3":"status-publish","4":"format-standard","6":"category-security","7":"tag-apache","8":"tag-security-2","9":"tag-ssl","10":"tag-tcpdump","11":"tag-tls","12":"tag-tshark","13":"tag-wireshark","14":"czr-hentry"},"_links":{"self":[{"href":"https:\/\/www.netnea.com\/cms\/wp-json\/wp\/v2\/posts\/1824","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.netnea.com\/cms\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.netnea.com\/cms\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.netnea.com\/cms\/wp-json\/wp\/v2\/users\/7"}],"replies":[{"embeddable":true,"href":"https:\/\/www.netnea.com\/cms\/wp-json\/wp\/v2\/comments?post=1824"}],"version-history":[{"count":16,"href":"https:\/\/www.netnea.com\/cms\/wp-json\/wp\/v2\/posts\/1824\/revisions"}],"predecessor-version":[{"id":1849,"href":"https:\/\/www.netnea.com\/cms\/wp-json\/wp\/v2\/posts\/1824\/revisions\/1849"}],"wp:attachment":[{"href":"https:\/\/www.netnea.com\/cms\/wp-json\/wp\/v2\/media?parent=1824"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.netnea.com\/cms\/wp-json\/wp\/v2\/categories?post=1824"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.netnea.com\/cms\/wp-json\/wp\/v2\/tags?post=1824"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}