<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Dominic Scheirlinck on Medium]]></title>
        <description><![CDATA[Stories by Dominic Scheirlinck on Medium]]></description>
        <link>https://medium.com/@nzdominic?source=rss-b752bed5a705------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*Upgnzc_HtuhA8k6z.jpg</url>
            <title>Stories by Dominic Scheirlinck on Medium</title>
            <link>https://medium.com/@nzdominic?source=rss-b752bed5a705------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 21:04:49 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@nzdominic/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com" rel="hub"/>
        <item>
            <title><![CDATA[How “The Internet’s Biggest Blind Spot” lead to a 15 year old security vulnerability]]></title>
            <link>https://medium.com/we-build-lightspeed/how-the-internets-biggest-blind-spot-lead-to-a-15-year-old-security-vulnerability-a2a6f6218a71?source=rss-b752bed5a705------2</link>
            <guid isPermaLink="false">https://medium.com/p/a2a6f6218a71</guid>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[security]]></category>
            <dc:creator><![CDATA[Dominic Scheirlinck]]></dc:creator>
            <pubDate>Mon, 18 Jul 2016 14:07:07 GMT</pubDate>
            <atom:updated>2016-07-19T07:41:23.876Z</atom:updated>
            <cc:license>https://creativecommons.org/licenses/by-sa/4.0/</cc:license>
            <content:encoded><![CDATA[<h4>Discovering and Disclosing httpoxy</h4><p>Over the past two weeks, I’ve been coordinating the disclosure of a pretty big and very old security vulnerability. If you’re looking for the technical details, you can head to <a href="https://httpoxy.org">httpoxy.org</a>, and if you’re looking for a non-technical explanation, you might prefer to read <a href="https://medium.com/@nzdominic/what-is-httpoxy-65a33a8a1f4d">my other Medium story about the issue</a>.</p><p>Instead, this is the story of how we discovered it, and my experience with the disclosure process.</p><h3>Background</h3><p>Vend is a retail POS, inventory management, ecommerce and customer loyalty system we run as a service. I work on the platform team, helping to ensure our retailers receive a fast, error-free and secure experience; they rely on Vend to keep them selling stuff worry-free, and 24/7.</p><p>A pretty big part of the platform team’s remit is to maintain our monitoring and logging systems. Our system for text-based logs, as opposed to events (Kafka) or metrics (statsd), is based on an “ELK” stack; the popular combination of Elasticsearch, Logstash and Kibana.</p><h3>Discovery</h3><p>Just over two weeks ago, on an otherwise ordinary Thursday, Scott Geary, my colleague, was hunting through Kibana dashboards, looking for a clue related to a support ticket we received.</p><p>Instead, Scott found an “interesting” error, and Slacked it to the rest of the team. Here’s what he saw:</p><blockquote>GuzzleHttp\Exception\ConnectException: cURL error 7: Failed to connect to proxy: Connection refused (see <a href="http://curl.haxx.se/libcurl/c/libcurl-errors.html">http://curl.haxx.se/libcurl/c/libcurl-errors.html</a>)</blockquote><p>“Hmm, that’s <em>weird</em>”, we thought. We certainly don’t have any microservices called ‘proxy’; and that’s a bare domain name? Huh.</p><p>Personally, my mind raced to the Docker Compose environment I had set up up a few days earlier. It uses the excellent <a href="https://github.com/jwilder/nginx-proxy">jwilder/nginx-proxy</a> container to provide an automatic reverse proxy. Some pretty neat magic. “Oh, <em>crap</em>,” I thought, “I’ve somehow managed to deploy this experimental development environment all the way to production!”</p><p>A quick grep of the codebase put that notion to bed. Given we had the stack-trace, we were able to quickly find a <a href="https://github.com/guzzle/guzzle/blob/502b40014c5f6c5b17894ce7ea7258b550b6b018/src/Client.php#L167">reference to HTTP_PROXY in Guzzle</a>, and it looked like this:</p><pre>// Use the standard Linux HTTP_PROXY and HTTPS_PROXY if set        if ($proxy = <strong>getenv(&#39;HTTP_PROXY&#39;)</strong>) {<br>    $defaults[&#39;proxy&#39;][&#39;http&#39;] = $proxy;        <br>}</pre><p>Cue heads asploding.</p><h3>Disclosure</h3><p>At first, we didn’t think this would be anything but a bug in Guzzle. It was serious, it was remotely exploitable, and Guzzle is <em>popular .</em></p><p>For developers who aren’t familiar with PHP (no gloating please), Guzzle is something of a best-of-breed HTTP client for PHP. It has a nice, “modern PHP” codebase and a solid functional middleware system. This means it’s an attractive dependency to use if you’re writing an SDK or API wrapper. A remotely exploitable vulnerability in Guzzle is Bad Enough that we felt it <em>couldn’t possibly</em> be widespread.</p><p>But as we continued to look at the situation, it became apparent that this bug would affect a <em>lot</em> more than a single HTTP library; nobody seemed to mention the issue regarding HTTP_PROXY specifically in any PHP security advice we could find. So, we broadened our thinking a bit, and contacted the PHP security mailing list.</p><h4>PHP’s getenv</h4><p>I’ll take a brief diversion here to discuss the issue in PHP. Obviously, there is a temptation with a vulnerability like this to blame the library authors who trusted the HTTP_PROXY “environment variable”. I don’t think that’s fair.</p><p>A big part of why I think that comes down to the function getenv, as implemented in PHP, and in particular how it was documented. There are (or were) a number of problems with the documentation:</p><ul><li>The function summary is just ‘gets the value of an environment variable’, with no caveats mentioned at all.</li><li>There’s no mention of CGI by name, just the RFC by number, in the function documentation body (which you should remember is not shown in summarised views, like the one your IDE will probably show you by default)</li><li>There’s a mention of section 4.1 in the RFC, but no deep link, because the RFC link goes to faqs.org rather than tools.ietf.org (which allows linking to specific sections)</li><li>The documentation <em>implies</em> (in a code comment in an example!) getenv is roughly the same as $_SERVER or other superglobals, but it never explicitly <em>says so</em>. The semantics of $_SERVER are very well understood; PHP programmers <em>know</em> that it can return client-provided values. So, why doesn’t the documentation make the link between the two clearer?</li></ul><p>What I would expect is that the getenv function documentation should make an explicit mention of “CGI”, <em>and</em> mention that values returned by getenv are not always “environment variables” set by the programmer, but are often under the control of the end-user via their request headers. You can achieve the latter <em>just</em> by explicitly defining the relationship between getenv and superglobals.</p><h4>Things get worse</h4><p>Alright, so, at this point, we’ve disclosed to two parties, and we’re feeling a bit bad about the state of security in PHP. It was around this point that we started to find historical mentions of the bug.</p><p>We knew all the ingredients of the vulnerability had been around for a long time. As far as we can tell, both CGI and the HTTP_PROXY header reach back twenty-odd years into the 90s. And so we naturally headed for the next question: “is this in <em>everything</em>?”. Github code search was invaluable for providing the answer: <strong>no, not everything.</strong></p><p>Specifically, that’s when we ran into the Curl mention of the vulnerability in 2001, the Ruby documentation of find_proxy (which mentions the issue), and from there, the original libwww-perl fix. That brought our list of ‘projects that fixed this years ago’ to at least three.</p><p>So, it was time to start proving the concept in other places. First up was Go, mainly just because it’s of personal interest to me, but we also use it a lot a Vend. The method for all the other places we found the issue was very similar:</p><ul><li>See if the most popular HTTP client library around for that language/platform supports HTTP_PROXY, and if it doesn’t check for CGI. (Tick for Go’s net/http)</li><li>See if there’s a way to deploy the language under CGI (Tick for Go’s net/http/cgi)</li></ul><p>And that was really it. No <em>CGI implementation</em> we could find treated HTTP_PROXY in any “special” way; if the above conditions were true, the proof of concept would succeed.</p><p>So, pretty quickly, I had Docker-based proof of concepts for Go’s net/http and Python’s requests, before I had heard back from the PHP security team. This was quickly becoming a more complex issue to disclose. Over the next couple of hours, two more disclosure emails went out to two more security groups (Python and Go), and then another couple to the common CGI (or FastCGI) servers’ security teams: Apache and Nginx.</p><h4>And then things got better</h4><p>One of my next points of disclosure was the Red Hat Product Security team. They were extremely helpful: walking me through choosing a suitable timeframe for the embargo, helping with getting CVEs assigned, and even coming to a provisional CVSS score, so I could easily tell people how bad the vulnerability was before giving them full details.</p><p>These are <em>vital</em> processes that help a disclosure process run smoothly. They also involve, for good reason, complicated decisions that need justification. And so, there’s a certain level of process involved.</p><p>By the end of the embargo period, I had sent about 60 full-length emails (in the open-source mailing list “tone” that requires careful preemptive defence against pedantry). The major CDNs were notified. Microsoft had some clarifications and some nice mitigation information to share. And on it went for days.</p><p>I’m in NZST — that’s +12 for those who don’t deal with us Kiwis regularly. That, in turn, meant conversations would often get started in the early morning; as I wanted to contribute to the discussion (and try to steer it away from pitfalls), that meant a lot of early morning starts.</p><h3>Who You Gonna Call?</h3><p>Naturally, perhaps, as the process continued I found myself wondering if I was doing a good job. From looking at recent large disclosures, there are plenty of examples of what <strong>not</strong> to do, but also plenty of distinct traps you can fall into.</p><p>Where is the <strong>incident response manual</strong> for a vulnerability like this? The simple answer is <em>there isn’t one</em> and the less satisfying answer is <em>there probably couldn’t be one.</em></p><h4>Security vulnerability disclosure is hard</h4><p>httpoxy was a hard case to disclose because of how much affected software we found. But we also had one really good thing going for us: it’s easy to mitigate, even without patched software from upstream.</p><p>Had that not been the case, things would have been even more complicated. In such situations, you’re much more at the mercy of the developers responsible for the problem, before you can responsibly disclose to other affected applications and the like.</p><p>So, there are plenty of <em>very</em> hard calls to make. I had it described to me as a triage process, and that metaphor makes a lot of sense. First, do no harm, sure. But, you have to figure out who you can help in a timely manner; you can’t just disclose publicly and go home.</p><p>For example, should I report this to CERT myself? Which closed/private mailing lists are still active, trusted, and worth disclosing to? Who has the responsibility of getting CVEs assigned once seven different security teams are coordinating a fix? Do I need to come up with the text of the CVEs myself? What do I do when a vendor asks me for a two week extension of the embargo period? Are people going to yell at me and call it grandstanding if I pick a name for my vulnerability?</p><p>I think the complexity of those questions means if we’re looking for a cookie-cutter process to follow, we won’t find it. Doing security disclosure <em>well</em> requires a close attention to the details of the issue, and experience to know what can go wrong. Nothing beats the careful eye of someone who has done this before (and especially someone who has done it a dozen times this week.) So, thanks again specifically to Kurt Seifried.</p><h4>For many, it’s just not something they’re familiar with</h4><p>I have reported a few security issues before, and I had a team of security specialists at Vend helping me out with the answers to a lot of those hard questions. But open source developers are often unpaid, contributing to a community in their spare time. Asking or, worse, expecting a volunteer maintainer to become well-versed in security disclosure norms overnight is begging for trouble. It’s not just that it’s unfair and slightly rude, but it’s <em>not going to work well</em>.</p><p>I’m well aware of catch-phrases like “security is a process not a feature”, and I’m not advocating that developers abdicate responsibility for releasing secure code and keeping it secure. But the security disclosure process is <em>something else</em>. The number of acronyms alone makes even web development look tame. And we shouldn’t pretend an interest in, say, <em>helping people make HTTP requests</em> translates to anything like an interest in <em>disclosing security bugs</em>.</p><p>Security disclosure is also a terrible thing to unleash upon a developer who might be just starting out. “Don’t be a maintainer if you don’t know how to handle security issues” is short-sighted; there’s a categorical difference between knowing how to prevent XSS or SQL injection, and knowing what to do when you’ve found an open-source security vulnerability. One is a basic set of skills I apply every time I’m working on a web app. And the other is something I only do a couple of times a year (and that I <em>don’t</em> expect most people to be doing.)</p><h4>The open source infrastructure funding issue</h4><p>There ought to “just” be <em>someone you can call. </em>Until there is, disclosures like the one for httpoxy will remain ad hoc and inconsistent. Some will be handled well, and others won’t.</p><p>It’s a hard problem. I’m heartened to see the “open source infrastructure funding” issue being discussed recently (e.g. a great discussion kicked off by <a href="https://medium.com/u/38ef5f0ee2b8">Nadia Eghbal</a>). Such discussions often explicitly consider broad cross-project vulnerabilities like httpoxy as the natural outcome of a lack of such funding. I think that’s an interesting idea. You can see how it’s directly applicable when you consider how <em>long</em> this vulnerability was around for, and the way it was fixed in a piece-meal fashion, across two decades, and across a bunch of different open source software.</p><p>Why wasn’t it picked up sooner? Why isn’t there someone just watching out for fixes in one language that might affect another? It’s the same issue as “disclosure is hard” behind a different guise: without treating open source infrastructure as an end in itself (and getting someone to fund it well), we’ll continue to be disappointed.</p><p>This <em>isn’t</em> quite a tragedy of the commons situation. The commons itself is doing fine; more people than ever are using it, and contributing to it. It’s the lack of institutions that’s holding it back. It’s like we should have funded a fire service by now, but we’re all too busy grazing our flocks on the communal feed to deal with the brush fire in the next field over. And who wants to deal with that anyway? That <em>should</em> be left to the experts. (Although, if we’re going with this metaphor, <em>please</em>, can we skip right past the age of private fire brigades and <a href="https://en.wikipedia.org/wiki/Fire_insurance_mark">fire insurance marks</a>?)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a2a6f6218a71" width="1" height="1" alt=""><hr><p><a href="https://medium.com/we-build-lightspeed/how-the-internets-biggest-blind-spot-lead-to-a-15-year-old-security-vulnerability-a2a6f6218a71">How “The Internet’s Biggest Blind Spot” lead to a 15 year old security vulnerability</a> was originally published in <a href="https://medium.com/we-build-lightspeed">WE ARE LIGHTSPEED</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What is httpoxy?]]></title>
            <link>https://medium.com/we-build-lightspeed/what-is-httpoxy-65a33a8a1f4d?source=rss-b752bed5a705------2</link>
            <guid isPermaLink="false">https://medium.com/p/65a33a8a1f4d</guid>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[httpoxy]]></category>
            <category><![CDATA[vulnerability]]></category>
            <dc:creator><![CDATA[Dominic Scheirlinck]]></dc:creator>
            <pubDate>Mon, 18 Jul 2016 14:06:00 GMT</pubDate>
            <atom:updated>2018-12-06T21:02:41.182Z</atom:updated>
            <cc:license>https://creativecommons.org/licenses/by-sa/4.0/</cc:license>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WsIOBvCHVVfkkmSzPP5vNw.jpeg" /></figure><h4>An explanation for non-technical audiences</h4><p>Hi! I’ve been telling the open source community about a security vulnerability that was recently rediscovered lurking in a bunch of software. This is an attempt at a simple explanation of the problem, for people who don’t write or deploy web applications.</p><p>If you’re looking for technical details, and mitigation instructions, you can head to <a href="https://httpoxy.org/">httpoxy.org</a>. And if you’re looking for a story about the discovery and disclosure process, you can check out <a href="https://medium.com/@nzdominic/how-the-internets-biggest-blind-spot-lead-to-a-15-year-old-security-vulnerability-a2a6f6218a71">my other Medium story on httpoxy</a>.</p><p>Also, I’d like to point out that, because we found this, we were able to prevent it from ever affecting Vend or the retailers who use us (perhaps the best advantage of being the original reporter!) Phew, right? OK, on with the story.</p><h3>Web applications like to make web requests</h3><p>The first thing I should probably explain is why a web application would make a web <em>request</em>.</p><p>Normally, you send your own request to a web application, it processes the data you give it, and it sends you back a <em>response</em> containing the page you asked for. (When all goes well!)</p><p>Sometimes, a web application will want to ask other software what to do. It’ll do so using an “API”; a piece of software that tells the application on the server how to talk to <em>other</em> applications. In the past few decades, we’ve seen the rise of APIs that are based on the same technology as the web itself; this is really nice for web application programmers, because it means servers only need to deal with responding in a single way (i.e. via HTTP).</p><p>So, these days, when you make your request to an application, the server might decide to go send its own request off to somewhere else. In a “microservice architecture”, a fashionable way to structure modern apps, this can happen quite often; sometimes even multiple times on every request.</p><h3>Web applications like environment variables</h3><p>The next thing to cover is what “environment variables” are (you’ll often see these called “env vars”).</p><p>Environment variables are a facility for storing simple fragments of text alongside the application. They’re mostly used for configuration because:</p><ul><li>Environment variables are really easy to use, and almost everything supports them, from software that will help test your application to complete deployment solutions that will ship your code to production</li><li>They let you keep important configuration data separate from the actual code of your application (for various Good Reasons, web developers try to keep these two separate; a big one is it lets them be much more flexible with how they deploy their app.)</li></ul><p>So, the important thing for our purposes is that env vars are usually set by the person deploying or developing the application.</p><h3>There’s an env var that configures a proxy</h3><p>One environment variable in particular affects the way outgoing web requests are made; that’s HTTP_PROXY (and sometimes http_proxy; a lower-case version of the same thing).</p><p>A proxy is a server that receives web requests from users, and passes them on to the actual application they’re trying to reach. You’ll often find proxies in corporate environments, where the IT team would like to be able to monitor the use of the web in their company. You’ll also find them used for performance reasons.</p><p>HTTP_PROXY is an environment variable that usually has a internet hostname in it; web clients see that, and auto-configure it as the proxy to use.</p><h3>And, there are requests that can set environment variables</h3><p>In the 1990s, a common way of deploying a web application was known as CGI: the common gateway interface. This is just a set of rules for developers to follow. By doing so, you could deploy a web application written in any language behind the same web server. Useful!</p><p>Part of CGI specifies that a whole heap of useful information should be passed to the application in environment variables.</p><p>So, specifically, when you send a request to a web app, your browser includes some metadata in addition to the URL you’re hoping to visit — these are called <em>request headers</em>. They include things like the (human) language you’re looking for, and the types of content your browser can handle. All these things are ultimately passed to the application using environment variables.</p><p>When they are, CGI prefixes the name of each header with HTTP_, to show it came from an HTTP request.</p><h3>httpoxy is <em>just</em> a combination of all of the above</h3><p>So, to describe a typical attack:</p><ol><li>The attacker makes a specially crafted request, that includes a special request header, called Proxy</li><li>CGI picks this that header, and turns it into an environment variable called HTTP_PROXY</li><li>The web application notices the HTTP_PROXY env var, and configures its web client to use it as a proxy</li><li>The web application makes a request using its client, and that request is sent to wherever the attacker wants, instead of the API the programmer wanted</li></ol><p>I hope that was a lot easier to follow than the jargon-filled technical disclosure documents we’ve published for web application developers today.</p><h3>But what’s the damage?</h3><p>That’s the bit that depends on the specific application that’s affected.</p><p>For example, the web application might make a request to a weather API; you give it a postcode, and it gives you back a five day forecast. In this case, the attacker could tweak the output of that API, and make it seem like it’s going to rain for the next few days straight.</p><p>So far, pretty innocuous right?</p><p>But that request to the weather API will likely include credentials that the application is using to identify itself to the weather forecasting service. So, that’s now something the attacker has.</p><p>And, thinking of more serious scenarios, the exact same thing <em>could</em> happen with something like a billing backend. The web application might ask “has user X paid this month?”, and then the attacker might be able to make it say “yes, that user paid for a premium account” instead of “nope, that user is in arrears”!</p><h4>Mitigating factors — the good news</h4><p>There are a few things that make httpoxy much less of an issue than it might otherwise be:</p><ul><li>A <em>lot</em> of APIs require an encrypted (SSL/TLS/HTTPS) connection these days, and those requests cannot be attacked in the same way — httpoxy only affects <em>unencrypted</em> requests made by the server</li><li>httpoxy is very easy to mitigate against; you just stop the Proxy header from reaching the app altogether, and the problem is solved.</li><li>CGI is much less widely deployed than it used to be; we have faster and better ways to serve web applications these days, and those tend to be used instead.</li></ul><p>So, httpoxy is embarrassingly simple to exploit, and you can do so remotely, and it can have fairly bad consequences. <strong>But, </strong>it’s also very easy to fix. We expect anyone affected should be able to fix their web application within a very short timeframe.</p><h3>What do I do?</h3><p>If you’re a web application developer, or you look after a deployed site, you should check if you’re affected at <a href="https://httpoxy.org">httpoxy.org</a>, and apply the fast, easy mitigation as soon as you can.</p><p>But if you’re an end-user of a web application, or just reading along for fun, then there’s not really anything for you to do other than enjoy the schadenfreude.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=65a33a8a1f4d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/we-build-lightspeed/what-is-httpoxy-65a33a8a1f4d">What is httpoxy?</a> was originally published in <a href="https://medium.com/we-build-lightspeed">WE ARE LIGHTSPEED</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>