Software Supply Chain Attack

Joel Wallenberg interviewed me on 14^th February for his article in the 28^th February edition of Grant's Interest Rate Observer entitled Memo to the bitcoiners. Alas, it is paywalled, but among the many quotes from me Wallenberg used was that blockchain-based systems "are very vulnerable to supply-chain attacks".

Exactly a week after the interview and a week before the article went to press, we got an example, the biggest cryptocurrency heist in history. Below the fold I discuss the details.

Software Liability: US vs. EU

I have written before about the double-edged sword of software vendors' ability to disclaim liability for the performance of their products. Six years ago I wrote The Internet of Torts about software embedded in the physical objects of the Internet of Things. Four years ago I wrote about Liability In The Software Supply Chain.

Source

Last October, Tom Uren wrote The EU Throws a Hand Grenade on Software Liability:

The EU and U.S. are taking very different approaches to the introduction of liability for software products. While the U.S. kicks the can down the road, the EU is rolling a hand grenade down it to see what happens.

It is past time to catch up on this issue, so follow me below the fold.

Microsoft Keys

Back in 2021 I gave a Talk At Berkeley's Information Access Seminar that summarized two long posts from two years before that:

• Certificate Transparency
• Securing The Software Supply Chain

On Friday 25^th Dan Goodin had two posts documenting that even the biggest software companies haven't fixed the problems I was talking about:

• Renegade certificate removed from Windows. Then it returns. Microsoft stays silent.
• Microsoft signing keys keep getting hijacked, to the delight of Chinese threat actors

Below the fold I update this sorry state of affairs, which I first started cataloging a decade ago.

The Philosopher of Palo Alto

I just finished reading John Tinnell's The Philosopher of Palo Alto. Based on Stanford Library's extensive archive of Mark Weiser's papers, and interviews with many participants, it is an impressively detailed and, as far as I can tell, accurate account of the "ubiquitous computing" work he drove at Xerox PARC. I strongly recommend reading it. Tinnell covers Weiser's life story to his death at age 46 in 1999, the contrast between his work and that at Nick Negroponte's MIT Media Lab, and the ultimate failure of his vision.

Tinnell quotes Lucy Suchman's critique of Weiser's approach to innovation:

Under this approach, Suchman claimed, a lab "[provided] distance from practicalities that must eventually be faced" — but facing up to those practicalities was left up to staff in some other department.

To be fair, I would say the same criticism applied to much of the Media Labs work too.

As I was at the time a member of "staff in some other department" at Sun Microsystems and then Nvidia, below the fold I discuss some of the "practicalities" that should have been faced earlier rather than later or not at all.

Blatant Self-Promotion

Liam Proven's NixOS and the changing face of Linux operating systems is a very interesting discussion of Linux distros and package management. He starts by discussing radical restructuring of Linux distros, focusing on NixOS and GoboLinux. Then he looks at less radical alternatives:

So, instead of re-architecting the way distros are built, vendors are reimplementing similar functionality using simpler tools inherited from the server world: containers, squashfs filesystems inside single files, and, for distros that have them, copy-on-write filesystems to provide rollback functionality.

The goal is to build operating systems as robust as mobile OSes: periodically, the vendor ships a thoroughly tested and integrated image which end users can't change and don't need to. In normal use, the root filesystem is mounted read-only, and there's no package manager.

Proven goes on to discuss efforts along these lines at Red Hat, openSUSE, Canonical and EndlessOS.

If you don't blow your own horn, who will do it for you? So it falls to me to point out that this is a great, but rather tricky, idea that I have implemented versions of not once, but twice. The first time more than 30 years ago for SunOS 4.1 in prototype form at Sun Microsystems, and the second time nearly twenty years ago for OpenBSD in production for the LOCKSS Program at Stanford.

Below the fold is more detailed self-promotion.

Hidden Certificate Authorities

The security of encrypted Web traffic depends upon a set of Certificate Authorities (CAs). Browsers and operating systems are configured with a list of CAs that they trust. The system is brittle, in the sense that if any of the multitude of CAs that your browser trusts is incompetent or malign, the security of all your traffic is imperiled. I've written several times on the topic of misbehaving CAs; there is a list of links at the end of the post.

In Web trust dies in darkness: Hidden Certificate Authorities undermine public crypto infrastructure, Thomas Claiburn reports on an important paper, Rusted Anchors: A National Client-Side View of Hidden Root CAs in the Web PKI Ecosystem by Yiming Zhang et al. This paper looks at what happens when, by fair means or foul, unofficial entries are added to or replace the CAs in the official list that your browser trusts. Below the fold I discuss their findings.

A Modest Proposal About Ransomware

On the evening of July 2^nd the REvil ransomware gang exploited a 0-day vulnerability to launch a supply chain attack on customers of Kaseya's Virtual System Administrator (VSA) product. The timing was perfect, with most system administrators off for the July 4^th long weekend. By the 6^th Alex Marquardt reported that Kaseya says up to 1,500 businesses compromised in massive ransomware attack. REvil, which had previously extorted $11M from meat giant JBS, announced that for the low, low price of only $70M they would provide everyone with a decryptor.

The US government's pathetic response is to tell the intelligence agencies to investigate and to beg Putin to crack down on the ransomware gangs. Good luck with that! It isn't his problem, because the gangs write their software to avoid encrypting systems that have default languages from the former USSR.

I've writtten before (here, here, here) about the importance of disrupting the cryptocurrency payment channel that enables ransomware, but it looks like the ransomware crisis has to get a great deal worse before effective action is taken. Below the fold I lay out a modest proposal that could motivate actions that would greatly reduce the risk.

Cryptocurrency's Carbon Footprint

China’s bitcoin mines could derail carbon neutrality goals, study says and Bitcoin mining emissions in China will hit 130 million tonnes by 2024, the headlines say it all. Excusing this climate-destroying externality of Proof-of-Work blockchains requires a continuous flow of new misleading arguments. Below the fold I discuss one of the more recent novelties.

Blast Radius

Last December Simon Sharwood reported on an "Infrastructure Keynote" by Amazon's Peter DeSantis in AWS is fed up with tech that wasn’t built for clouds because it has a big 'blast radius' when things go awry:

Among the nuggets he revealed was that AWS has designed its own uninterruptible power supplies (UPS) and that there’s now one in each of its racks. AWS decided on that approach because the UPS systems it needed were so big they required a dedicated room to handle the sheer quantity of lead-acid batteries required to keep its kit alive. The need to maintain that facility created more risk and made for a larger “blast radius” - the extent of an incident's impact - in the event of failure or disaster.

AWS is all about small blast radii, DeSantis explained, and in the past the company therefore wrote its own UPS firmware for third-party products.

“Software you don’t own in your infrastructure is a risk,” DeSantis said, outlining a scenario in which notifying a vendor of a firmware problem in a device commences a process of attempting to replicate the issue, followed by developing a fix and then deployment.

“It can take a year to fix an issue,” he said. And that’s many months too slow for AWS given a bug can mean downtime for customers.

This is a remarkable argument for infrastructure based on open source software, but that isn't what this post is about. Below the fold is a meditation on the concept of "blast radius", the architectural dilemma it poses, and its relevance to recent outages and compromises.

Talk At Berkeley's Information Access Seminar

Once again Cliff Lynch invited me to give a talk to the Information Access Seminar at UC Berkeley's iSchool. Preparation time was limited because these days I'm a full-time grandparent so the talk, entitled Securing The Digital Supply Chain summarizes and updates two long posts from two years ago:

• Certificate Transparency
• Securing The Software Supply Chain

The abstract was:

The Internet is suffering an epidemic of supply chain attacks, in which a trusted supplier of content is compromised and delivers malware to some or all of their clients. The recent SolarWinds compromise is just one glaring example. This talk reviews efforts to defend digital supply chains.

Below the fold, the text of the talk with links to the sources.

Effort Balancing And Rate Limits

Catalin Cimpanu reports on yet another crime wave using Bitcoin in As Bitcoin price surges, DDoS extortion gangs return in force:

In a security alert sent to its customers and shared with ZDNet this week, Radware said that during the last week of 2020 and the first week of 2021, its customers received a new wave of DDoS extortion emails.

Extortionists threatened companies with crippling DDoS attacks unless they got paid between 5 and 10 bitcoins ($150,000 to $300,000)
...
The security firm believes that the rise in the Bitcoin-to-USD price has led to some groups returning to or re-prioritizing DDoS extortion schemes.

And Dan Goodin reports on the latest technique the DDOS-ers are using in DDoSers are abusing Microsoft RDP to make attacks more powerful:

As is typical with many authenticated systems, RDP responds to login requests with a much longer sequence of bits that establish a connection between the two parties. So-called booter/stresser services, which for a fee will bombard Internet addresses with enough data to take them offline, have recently embraced RDP as a means to amplify their attacks, security firm Netscout said.

The amplification allows attackers with only modest resources to strengthen the size of the data they direct at targets. The technique works by bouncing a relatively small amount of data at the amplifying service, which in turn reflects a much larger amount of data at the final target. With an amplification factor of 85.9 to 1, 10 gigabytes-per-second of requests directed at an RDP server will deliver roughly 860Gbps to the target.

I don't know why it took me so long to figure it out, but reading Goodin's post I suddenly realized that techniques we described in Impeding attrition attacks in p2p systems, a 2004 follow-up to our award-winning 2003 SOSP paper on the architecture of the LOCKSS system, can be applied to preventing systems from being abused by DDOS-ers. Below the fold, brief details.

Liability In The Software Supply Chain

Atlantic Council Report On Software Supply Chains was already rather long when I got to the last of the report's recommendations that I wanted to discuss, the one entitled Bring Lawyers, Guns and Money. It proposes imposing liability on actors in the software supply chain, and I wrote:

The fact that software vendors use licensing to disclaim liability for the functioning of their products is at the root of the lack of security in systems. These proposals are plausible but I believe they would either be ineffective or, more likely, actively harmful. There is so much to write about them that they deserve an entire post to themselves.

Below the fold is the post they deserve.

Atlantic Council Report On Software Supply Chains

Eighteen months ago I posted a four-part series called Trust In Digital Content. The second part was Securing The Software Supply Chain, about how we know we're running the software we intended to. Now, Bruce Schneier's Survey of Supply Chain Attacks starts:

The Atlantic Council has released a report that looks at the history of computer supply chain attacks.

The Atlantic Council also has a summary of the report entitled Breaking trust: Shades of crisis across an insecure software supply chain:

Software supply chain security remains an under-appreciated domain of national security policymaking. Working to improve the security of software supporting private sector enterprise as well as sensitive Defense and Intelligence organizations requires more coherent policy response together industry and open source communities. This report profiles 115 attacks and disclosures against the software supply chain from the past decade to highlight the need for action and presents recommendations to both raise the cost of these attacks and limit their harm.

Below the fold, some commentary on the report and more recent attacks.

Twitter Fails Security 101 Again

Source

On July 15 the New York Times reported on the day's events at Twitter:

It was about 4 in the afternoon on Wednesday on the East Coast when chaos struck online. Dozens of the biggest names in America — including Joseph R. Biden Jr., Barack Obama, Kanye West, Bill Gates and Elon Musk — posted similar messages on Twitter: Send Bitcoin and the famous people would send back double your money.

Two days later Nathaniel Popper and Kate Conger's Hackers Tell the Story of the Twitter Attack From the Inside was based on interviews with some of the perpetrators:

Mr. O'Connor said other hackers had informed him that Kirk got access to the Twitter credentials when he found a way into Twitter’s internal Slack messaging channel and saw them posted there, along with a service that gave him access to the company’s servers. People investigating the case said that was consistent with what they had learned so far. A Twitter spokesman declined to comment, citing the active investigation.

Below the fold, some commentary on this and other stories of the fiasco.

Deanonymizing Ethereum Users

In last January's Bitcoin's Lightning Network I discussed A Cryptoeconomic Traffic Analysis of Bitcoin’s Lightning Network by the Hungarian team of Ferenc Béres, István A. Seres, and András A. Benczúr. They demolished the economics of the Lightning Network, writing:

Our findings on the estimated revenue from transaction fees are in line with the widespread opinion that participation is economically irrational for the majority of the large routing nodes who currently hold the network together. Either traffic or transaction fees must increase by orders of magnitude to make payment routing economically viable.

Below the fold I comment on their latest work.

Economic Limits Of Proof-of-Stake Blockchains

In 2018's Cryptocurrencies Have Limits I discussed Eric Budish's The Economic Limits Of Bitcoin And The Blockchain, an important analysis of the economics of two kinds of "51% attack" on Bitcoin and other cryptocurrencies based on "Proof-of-Work" (PoW) blockchains. Among other things, Budish shows that, for safety, the value of transactions in a block must be low relative to the fees in the block plus the reward for mining the block. In last year's The Economics Of Bitcoin Transactions I discussed Raphael Auer's Beyond the doomsday economics of “proof-of-work” in cryptocurrencies, in which Auer shows that:

proof-of-work can only achieve payment security if mining income is high, but the transaction market cannot generate an adequate level of income. ... the economic design of the transaction market fails to generate high enough fees.

Follow me below the fold for a discussion of a fascinating recent paper that extends Budish's analysis.

Bunnie Huang's Betrusted Project

The awesome Bunnie Huang asks Can We Build Trustable Hardware? It is a fascinating approach to the problem I discussed in Securing The Hardware Supply Chain:

how we can know that the hardware the software we secured is running on is doing what we expect it to?

Bunnie's experience has made him very skeptical of the integrity of the hardware supply chain:

In the process of making chips, I’ve also edited masks for chips; chips are surprisingly malleable, even post tape-out. I’ve also spent a decade wrangling supply chains, dealing with fakes, shoddy workmanship, undisclosed part substitutions – there are so many opportunities and motivations to swap out “good” chips for “bad” ones. Even if a factory could push out a perfectly vetted computer, you’ve got couriers, customs officials, and warehouse workers who can tamper the machine before it reaches the user.

Below the fold, some discussion of Bunnie's current project.

Auditing The Integrity Of Multiple Replicas

The fundamental problem in the design of the LOCKSS system was to audit the integrity of multiple replicas of content stored in unreliable, mutually untrusting systems without downloading the entire content:

• Multiple replicas, in our case lots of them, resulted from our way of dealing with the fact that the academic journals the system was designed to preserve were copyright, and the copyright was owned by rich, litigious members of the academic publishing oligopoly. We defused this issue by insisting that each library keep its own copy of the content to which it subscribed.
• Unreliable, mutually untrusting systems was a consequence. Each library's system had to be as cheap to own, administer and operate as possible, to keep the aggregate cost of the system manageable, and to keep the individual cost to a library below the level that would attract management attention. So neither the hardware nor the system administration would be especially reliable.
• Without downloading was another consequence, for two reasons. Downloading the content from lots of nodes on every audit would be both slow and expensive. But worse, it would likely have been a copyright violation and subjected us to criminal liability under the DMCA.

Our approach, published now more than 16 years ago, was to have each node in the network compare its content with that of the consensus among a randomized subset of the other nodes holding the same content. They did so using a peer-to-peer protocol using proof-of-work, in some respects one of the many precursors of Satoshi Nakamoto's Bitcoin protocol.

Lots of replicas are essential to the working of the LOCKSS protocol, but more normal systems don't have that many for obvious economic reasons. Back then there were integrity audit systems developed that didn't need an excess of replicas, including work by Mehul Shah et al, and Jaja and Song. But, primarily because the implicit threat models of most archival systems in production assumed trustworthy infrastructure, these systems were not widely used. Outside the archival space, there wasn't a requirement for them.

A decade and a half later the rise of, and risks of, cloud storage have sparked renewed interest in this problem. Yangfei Lin et al's Multiple‐replica integrity auditing schemes for cloud data storage provides a useful review of the current state-of-the-art. Below the fold, a discussion of their, and some related work.

Interesting Articles From Usenix

Unless you're a member of Usenix (why aren't you?) you'll have to wait a year to read two of three interesting preservation-related articles in the Fall 2019 issue of ;login:. Below the fold is a little taste of each of them, with links to the full papers if you don't want to wait a year:

Trust In Digital Content

This is the fourth and I hope final part of a series about trust in digital content that might be called:

Is this the real  life?
Is this just fantasy

  The series so far moved down the stack:

• The first part was Certificate Transparency, about how we know we are getting content from the Web site we intended to.
• The second part was Securing The Software Supply Chain, about how we know we're running the software we intended to, such as the browser that got the content whose certificate was transparent.
• The third part was Securing The Hardware Supply Chain, about how we can know that the hardware the software we secured is running on is doing what we expect it to.

Below the fold this part asks whether, even if the certificate, software and hardware were all perfectly secure, we could trust what we were seeing.