Library of Congress interview

An interview with me inaugurates an interview series called Insights on the Library of Congress' digital preservation blog The Signal.

Solid State Memory for Archival Use

In last year's JCDL keynote I pointed to work at Carnegie-Mellon on FAWN, the Fast Array of Wimpy Nodes and suggested that the cost savings FAWN realizes by distributing computation across a very large number of very low-power nodes might also apply to storage. Now, Ian Adams and Ethan Miller of UC Santa Cruz's Storage Systems Research Center and I have looked at this possibility more closely in a Technical Report entitled Using Storage Class Memory for Archives with DAWN, a Durable Array of Wimpy Nodes. We show that it is indeed plausible that, even at current flash memory prices, the total cost of ownership over the long term of a storage system built from very low-power system-on-chip technology and flash memory would be competitive with disk. More on this below the fold.

Amazon's Outage

I've been looking at the problems of specifying, measuring and auditing (PDF) the reliability of storage technologies since 2006. When I heard that Amazon's recent outage had lost customer's data I hoped that I could use this example as an Awful Warning that my Cassandra-like prophecies were coming true.

After some research, I can't claim that this is an example of the doom that awaits unsuspecting customers. But the outage and data loss does illustrate a number of interesting aspects of cloud storage. Details below the fold.

Disk Drive Mergers vs. Kryder's Law

A few weeks ago there were five disk drive manufacturers. With the announcements of Western Digital's acquisition of Hitachi's disk drive business, and Seagate's acquisition of Samsung's disk drive business, there will shortly be only three. Western Digital will have about 48% of the market,
Seagate will have about 40% of the market, and Toshiba will be in a marginally viable place with the remaining 12% or less.

Part of the reason for this abrupt consolidation must surely be the much higher than expected costs of the transition to the next recording technology, which I have been discussing for more than a year. This consolidation will allow the two market leaders to amortize the costs of the transition across more of the market. These costs will be much harder for Toshiba to bear, so they are likely not to survive. Even if Toshiba survives, its impact on the ability of the market leaders to increase margins will be negligible. Paying for the transition and increasing margins will both act to slow the Kryder's Law cost per byte decrease, as I have been predicting.

Update: here is the view from industry commentator Tom Coughlin.

Update on Open Access and NLM's Policy Change

We now have an interesting illustration of the effects of the change of policy at the National Library of Medicine which I discussed last November. I've been told that questions to NLM about the policy change have been met with claims that few if any e-only journals will choose to deposit content in Portico in order to be indexed in MedLine without their content being in PubMed Central. Below the fold I describe a counter-example.

Technologies Don't Die

Kevin Kelly finds the same reaction of incredulity when he pointed out that physical technologies do not die as I did when I pointed out that digital formats are not becoming obsolete. Robert Krulwich of NPR challenged Kelly, but had to retire defeated when he and the NPR listeners failed to find any but trivial examples of dead technology.

And, in related news, The Register has two articles on a working 28-year-old Seagate ST-412 disk drive from an IBM 5156 PC expansion box. They point out, as I have, that disk drives are not getting faster as fast as they are getting bigger:

The 3TB Barracuda still has one read/write head per platter surface and each head now has 300,000MB to look after, whereas the old ST-412 heads each have just 5MB to look after.

The Barracuda will take longer today to read or write an entire platter surface's capacity than the 28-year-old ST-412 will. We have increased capacity markedly but disk I/O has become a bottleneck at the platter surface level, and is set to remain that way. The Register

Revised 4/12/11 to make clear that the disk drive still works.

The New York Times Paywall

In my 2010 JCDL keynote I reported Marc Andreesen's talk at the Stanford Business School describing how legacy thinking prevented the New York Times from turning off the presses and making much more profit from a smaller company. Part of the problem Marc described was how the legacy cost base led the NYT to want to extract more money from the website than it was really capable of generating. In other words, the problem was to get costs in line with income rather than to increase income to match existing costs. This is essentially the same insight that the authors self-publishing on Amazon have made; the ability to price against a very low cost base creates demand that generates income. The NYT's paywall demonstrates that Marc was right that they are hamstrung by legacy thinking.

First, in order not to suffer a crippling loss in traffic (Murdoch's London Times lost 90% of its readership) and thus advertising revenue, the paywall cannot extract revenue from almost everyone who reads the NYT website:

• Subscribers to the print edition, who get free access.
• People who read the website a lot, who get free access sponsored by advertisers.
• People who don't read the website a lot, who can read 20 articles a month for free.
• People who get to articles free via links from search engines, blogs, etc.
• Technically sophisticated readers, who can get free access by evading the paywall

From whom can the paywall generate income? Technically unsophisticated non-subscribers who read a moderate amount but not via links from elsewhere. That's a big market. Legacy thinking requires generating lots more income but the ways to do it are all self-defeating.

Second, because the paywall that would be unnecessary if the cost base were addressed needs all these holes, it is unnecessarily complex. Therefore, as Philip Greenspun reveals, it is unnecessarily expensive. How did an organization one would think was tech-savvy end up paying $40-50M to implement a paywall, even if it is a complex one? The answer appears to be that some time ago, presumably to reduce costs on the digital side of the business, the NYT outsourced their website to Atypon. Apparently, they have repented and have taken the website back in-house, so the $40-50M is not just implementing the paywall but also insourcing.

These costs for undoing a decision to outsource, together with Boeing's 787 outsourcing fiasco, should be warnings to libraries currently being seduced to outsource their collections and functions.