From f7b659631714f47ba9ad1ea266796376a8fe6520 Mon Sep 17 00:00:00 2001 From: Ben Burwell Date: Thu, 6 May 2021 19:24:31 -0400 Subject: Add mitmproxy debugging post --- ...05-06-debugging-http-services-with-mitmproxy.md | 90 ++++++++++++++++++++++ 1 file changed, 90 insertions(+) create mode 100644 _posts/2021-05-06-debugging-http-services-with-mitmproxy.md diff --git a/_posts/2021-05-06-debugging-http-services-with-mitmproxy.md b/_posts/2021-05-06-debugging-http-services-with-mitmproxy.md new file mode 100644 index 0000000..8c5d7c9 --- /dev/null +++ b/_posts/2021-05-06-debugging-http-services-with-mitmproxy.md @@ -0,0 +1,90 @@ +--- +title: Debugging HTTP services with mitmproxy +--- + +I spend a lot of my time at work writing Go services that talk to other Go +services over HTTP. Much of the time, everything works as expected, but every +now and then a situation arises where I'm struggling to understand why my +program is receiving a specific value. Is my request not being built correctly? +Am I not properly deserializing the response? Logging can be helpful, but +sometimes I really just want to look at the HTTP traffic between services. + +One tool that I really love in these situations is [mitmproxy][]: "a free and +open source interactive HTTPS proxy," according to its website. + + + +There are no shortage of features and options for mitmproxy, and when I was +first exploring it they were a little daunting. I'm sure there are still tons of +things that it can do that I don't know about, but the main thing I tend to use +it for is reverse proxying. + +Reverse proxying is a pretty simple concept; basically it means that you send +your HTTP requests to a specific proxy endpoint, and the proxy repeats your +request to some specific origin server. Then the same thing just happens +backwards: the origin server sends its reply to the reverse proxy which passes +it along back to you. This means that the proxy can see (and log!) the actual +HTTP request you send, and the response sent by the origin server. + +Earlier today, I was working on a program that sent requests to a HTTP server +and my program's output didn't make sense. I wasn't sure if my requests were +being sent incorrectly, or maybe there was a bug in the server I was talking to. +So I fired up mitmproxy to take a look. In my shell, I ran: + +``` +mitmproxy --mode reverse:https://service.dev.example.com --listen-port 8080 +``` + +This opens up a log window where any requests handled by the reverse proxy will +be displayed. I quickly updated my program to make its requests to +`http://localhost:8080` instead of `https://service.dev.example.com` and re-ran +it. The request and response were logged in the terminal window, and I was +quickly able to identify that a particular dependency needed to be updated. + +Of course, you could also use Wireshark or tcpdump to inspect network traffic, +and these are great options that I also use frequently! But the main reason I +tend to turn first to mitmproxy is because it handles TLS like a [honey +badger][] -- it just doesn't give a shit. Basically, you can throw whatever you +want at it and it'll just do the right thing: + +``` +Client --TLS--> mitmproxy --TLS--> Origin +Client --HTTP--> mitmproxy --TLS--> Origin +Client --TLS--> mitmproxy --HTTP--> Origin +Client --HTTP--> mitmproxy --HTTP--> Origin +``` + +How does this work? When you first run mitmproxy, it generates a certificate +authority that it uses to generate certificates on-the-fly. All you need to do +is add the CA certificate to your OS trust store (see their [docs about +certificates here][certs]). For example, if I run `mitmproxy --mode +reverse:https://www.benburwell.com --listen-port 8080`, and then connect over +SSL, I can see the certificate that mitmproxy generated: + +``` +$ openssl s_client -connect localhost:8080 +CONNECTED(00000005) +depth=1 CN = mitmproxy, O = mitmproxy +verify error:num=19:self signed certificate in certificate chain +verify return:0 +--- +Certificate chain + 0 s:/CN=www.benburwell.com + i:/CN=mitmproxy/O=mitmproxy + 1 s:/CN=mitmproxy/O=mitmproxy + i:/CN=mitmproxy/O=mitmproxy +``` + +Here, `mitmproxy` CA has generated a certificate with `CN=www.benburwell.com` to +match the hostname I'm reverse proxying to! + +Now, there are ways to snoop on TLS encrypted traffic with Wireshark as well +using a TLS key log file, but this usually involves making somewhat non-trivial +modifications to the program you're working with. It's not _very_ complicated or +difficult, and it's a technique I've used a few times, but mitmproxy is usually +quicker and easier for me. I plan to write a post about this topic in the +future, so stay tuned! + +[mitmproxy]: https://mitmproxy.org/ +[honey badger]: https://youtu.be/4r7wHMg5Yjg +[certs]: https://docs.mitmproxy.org/stable/concepts-certificates/ -- cgit v1.2.3