The beets blog.

a guided walkthrough

2022-08-20T00:00:00+00:00

Overview

This is a step by step walkthrough for getting started with the beets CLI using a sample music library. The sample music library includes public domain recordings of Debussy, Chopin, and Virginia Liston compiled from Open Music Archive, IMSLP, and Internet Archive.

We will use this library in its messy/unaltered state to showcase the power of the beets CLI by correcting obscure metadata and organizing the music library with the auto-tagger (and a little manual input). Lastly, we will host the library locally using the beets web plugin to stream the audio once finished and admire our results. Let’s get started!

If desired, there are resources included below to help find additional recordings that are in the public domain if you’d like to grow your sample music library. Note: be sure the release you’re looking to include has metadata in the MusicBrainz database for best results.

Installing and configuring beets

Following are brief instructions for downloading and installing the beets CLI. For detailed instructions, head over to the Getting Started guide in the official documentation.

Download beets with pip install beets

Write your config in ~/.config/beets/config.yaml specifying a directory for your copied music and a library for a database file to store an index of your music.

My config.yaml looks like this:

directory: ~/beets/music
library: ~/beets/data/musiclibrary.db
plugins: web

All I’ve done to support the above config is create the beets directory inside my home folder, with empty child directories for music and data inside it. You can place your music library and data store wherever you like, and you don’t need to create the musiclibrary.db file; it will be created for you.

Importing the sample library

First, download the .zip archive of the sample music library from this public google drive file, extract the contents and take note of the path to the sample library for later use.

Take a second to scrub through the library’s organization and inconsistent file naming. On a small scale this is a prime example of how messy a music library can be in its raw form, with no standard for metadata or organization.

Enter beets! We’re going to clean up this mess and make our library beautiful.

Run beet import {path to sample library} to import the messy sample library and start using the beets auto-tagger.

In my case the path to the sample library is also in my home folder, so I will run the command beet import ~/sample-library

The autotagger starts work on Album 1, and the closest match is selection 1. Go ahead and enter that in the prompt.

Next, enter a to apply the changes found to Album 1.

Album 2 doesn’t have a matching release, so we will enter one manually by entering i and then the following MusicBrainz id 96581fca-a7b1-4c94-bb23-7b0df9f00507

Accept the changes for our manual match by selecting a

Album 3 doesn’t have a matching release either, so we will enter one manually with i and then the following MusicBrainz id 95440be5-cff3-4222-8ddc-5b7ecb99a351

We’ve located the correct album; however, we only have 11/48 tracks across a compilation of 3 CDs, hence the large output warning us of missing tracks.

The autotagger has done a decent job, but it has incorrectly matched tracks 7, 10, and 11 due to similar titles in other tracks throughout the compilation. We’ll use this opportunity to re-tag the offending tracks later in the next section.

For now, accept the changes for our manual match by selecting a

Album 4 is a series of singletons with a number of matches found by the autotagger.

Accept these changes and import the tracks with the t flag for individual tracks

The output should look like the following:

Finally, let’s check out our library in its current state with beet list

We’ve made significant progress! Now let’s correct some discrepencies that came up between the autotagger and our library.

Correcting a mistake in the sample library

We can use the command beet fields to see all the metadata that can be updated for each track in the library, in our case we’re interested in updating the title and mb_trackid fields to correct a series of mistakes in a couple of our albums.

Mistake: Album 2 - Swapped track titles and metadata

There are two tracks in our download of Moravec’s Debussy album where the titles and metadata are swapped. “Children’s Corner: Jimbo’s Lullaby” and “Clair de lune” incorrectly refer to the opposite recording. Let’s change that with beets!

Rename “Clair de lune” with the following command beet modify Clair de lune title="Children's Corner Suite"

Why not name this “Children’s Corner: Jimbo’s Lullaby”? Well, there’s another mistake in our album: this track is actually a 17 minute long recording of all movements from the “Children’s Corner” Suite, which are typically broken up track by track in a traditional release. We are renaming this way to more accurately reflect the recording we have in our library.

When running the command above, it will attempt to modify more than one track (since another track has the term “clair de lune” within its title), so use the select flag s to systematically deny or approve the title update for multiple tracks one by one. This ensures that we only update the title for the track “Clair de lune”.

Next, update the MusicBrainz track id with the following command beet modify Children\'s Corner Suite mb_trackid=9c7f4fc2-0d2b-42b6-99fd-6ac2cdf32123

This way our metadata for this track is correct. We will see a demonstration for why this is important later when we view our tracks with the web plugin.

Rename “Children’s Corner: Jimbo’s Lullaby” with the following command beet modify lullaby title="Clair de lune"

Be sure to update the MusicBrainz track id for this track as well with beet modify "Clair de lune" mb_trackid=9d6006ed-cb42-4227-945f-9e7f6766cc2c and the s tag to select and update the correct track.

Exercise - Mistake: Album 3 - Incorrect titles and metadata

There are three tracks we need to correct in Album 3. Use your knowledge of beets’ modify command from the previous mistake we corrected to update the title and mb_trackid fields for each of the tracks below.

first title	current title (incorrect)	new title (correct)	mb_trackid (correct)
Op. 62, no. 1, B major	Nocturne in B major, op. 32 no. 1	Nocturne in B major, op. 62 no. 1	dda32c0b-bd61-45f3-bb98-682bc4dbf065
Op. 62, no. 2, E major: posth.	12 Études, op. 10 no. 7 in C major “Toccata”	Nocturne in B major, op. 62 no. 2	4cec9444-351d-49c8-a42e-51734c6f27fe
Op. 72, no. 1, E minor	12 Études, op. 10 no. 10 in A-flat major	Nocturne in E minor, op. 72 no. 1	1dc71083-bb07-46c4-9765-094cc3a3469c

Check-in

At this point we have cleaned up an otherwise messy music library into a clean and updated public domain library of music. If we run beet list we can see it in all its glory. Next, let’s host it on the web!

Setting up a server with the web plugin

Following are brief instructions for setting up the beets web plugin. For detailed instructions, head over to the official documentation here: Web Plugin.
To use the beets web plugin we’ll need to download flask with pip install flask
Update the beets config.yaml with beet config -e and add the line plugins: web to the end (if you haven’t already)
Start up the server for the library with beet web and navigate to http://localhost:8337 to query, view, and listen to the music
If you search “Debussy” and play “Clair de lune” (which is a track who’s title and mb_trackid we modified), then click “view” next to “MusicBrainz entry”, you’ll notice it correctly navigates us to the page for the updated track in the MusicBrainz database.

Conclusion

Thanks for coming along on this journey! You can now enjoy an organized and well-documented sample music library of recordings found in the public domain, and take your newfound knowledge of beets to import music, correct metadata, and maintain your collection of incomplete/messy downloads. Happy listening!

Sample Music Library Sources

Debussy - Jacopo Salvatori

Debussy - Ivan Moravec

Frederic Chopin - Guiomar Novaes

Virginia Liston

we’re pretty happy with SQLite & not urgently interested in a fancier DBMS

2016-06-19T00:00:00+00:00

Every once in a while, someone suggests that beets should use a “real database.” I think this means storing music metadata in PostgreSQL or MySQL as an alternative to our current SQLite database. The idea is that a more complicated DBMS should be faster, especially for huge music libraries.

The pseudo-official position of the beets project is that supporting a new DBMS is probably not worth your time. If you’re interested in performance, please consider helping to optimize our database queries instead.

There are three reasons I’m unenthusiastic about alternative DBMSes: I’m skeptical that they will actually help performance; it’s a clear case of premature optimization; and SQLite is unbeatably convenient for desktop applications like beets.

Workload

Many people assume that Postgres or MySQL must be faster than SQLite. But performance depends on the workload, and SQLite is a great match for the beets workload.

Specifically, serious client–server DBMSes are great for web applications, where writes are frequent and concurrency is paramount. In beets, we have the opposite workload: we’re read-heavy and write-light, and we rarely need concurrent updates. The main case when beets writes to its database is on import, and import performance is dominated by I/O to the network and to music files. This mostly-read workload is exactly what SQLite was made for.

Low-Hanging Fruit

As Prof. Knuth tells us, optimization before measurement is the root of all evil. Before we embark on any big change for performance’s sake, we should have some scrap of empirical evidence to suggest that it might pay off.

There’s a better way to spend our limited developer attention. We haven’t invested at all in making our database queries efficient—we have almost no indices, no principled denormalization, and certainly no detailed profiling data. Carefully measuring our query efficiency is likely to yield asymptotically better database behavior. And since we’ve spent so little time on it so far, there’s probably a vast orchard of low-hanging fruit there.

Hassle

Even if everything else were equal, connecting to a new database backend would incur inconvenience for users and developers. SQLite is built into Python; any other DBMS would mean a new dependency. The “real databases” people usually envision are client–server systems; these are many times more frustrating to use than an embedded library that uses a flat file. SQLite isn’t perfect, of course, but it’s pretty damn good for being hassle-free.

If you’re convinced that a real database would be good for beets, I’d love to be proven wrong. But you need to prove it: measure the bottlenecks in beets first, think carefully about whether they might be query problems rather than DBMS problems, and be confident that the return in performance is worth the cost in hassle.

our solution for the hell that is filename encoding, such as it is

2016-06-04T00:00:00+00:00

By far, the worst part of working on beets is dealing with filenames. And since our job is to keep track of your files in the database, we have to deal with them all the time. This post describes the filename problems we discovered in the project’s early days, how we address them now, and some alternatives for the future.

What is a Path?

What would you say a path is? In Python terms, what should the type of the argument to open or os.listdir be?

Let’s say you think it should be text. The OS should tell us what encoding it’s using, and we get to treat its paths as human-readable strings. So the correct type is unicode on Python 2 or str on Python 3.

Here’s the thing, though: on Unixes, paths are fundamentally bytes. The arguments and return types of the standard Posix OS interfaces open(2) and opendir(2) use C char* strings (because we still live in 1969).

This means that your operating system can, and does, lie about its filesystem encoding. As we discovered in the early days of beets, Linuxes everywhere often say their filesystem encoding is one thing and then give you bytes in a completely different encoding. The OS makes no attempt to avoid handing arbitrary bytes to applications. If you just call fn.decode(sys.getfilesystemencoding()) in attempt to make turn your paths into Unicode text, Python will crash sometimes.

So, we must conclude that paths are bytes. But here’s the other thing: on Windows, paths are fundamentally text. The equivalent interfaces on Windows accept and return wide character strings—and on Python, that means unicode objects. So our grand plan to use bytes as the one true path representation is foiled.

It gets worse: to use full-length paths on Windows, you need to prefix them with the four magical characters \\?\. Every time. I know.

This contradiction is the root of all evil. It’s the cause of a huge amount of fiddly boilerplate code in beets, a good number of bug reports, and a lot of sadness during our current port to Python 3.

Our Conventions

In beets, we adhere to a set of coding conventions that, when ruthlessly enforced, avoid potential problems.

First, we need one consistent type to store in our database. We picked bytes. This way, we can consistently represent Unix filesystems, and it requires only a bit of hacking to support Windows too. In particular, beets encodes all filenames using a fixed encoding (UTF-8) on Windows, just so that the path type is always bytes on all platforms.

To make this all work, we use three pervasive little utility functions:

We use bytestring_path to force all paths to our consistent representation. If you don’t know where a path came from, you can just pass it through bytestring_path to rectify it before proceeding.
The opposite function, displayable_path, must be used to format error messages and log output. It does its best to decode the path to human-readable Unicode text, and it’s not allowed to fail—but it’s lossy. The result is only good for human consumption, not for returning back to the OS. Hence the name, which is intentionally not unicode_path.
Every argument to an OS function like open or listdir must pass through the third utility: syspath. Think of this as converting from beets’s internal representation to the OS’s own representation. On Unix, this is a no-op: the representations are the same. On Windows, this converts a bytestring path back to Unicode and then adds the ridiculous \\?\ prefix, which avoids problems with long names.

It’s not fun to force everybody to use these utilities everywhere, but it does work. Since we instated this policy, Unicode-related bugs still happen, but they’re not nearly as pervasive as they were in the project’s early days.

Must It Be This Way?

Although our solution works, I won’t pretend to love it. Here are a few alternatives we might consider for the future.

Python 3’s Surrogate Escapes

Python 3 chose the opposite answer to the root-of-all-evil contradiction: paths are always Unicode strings, not bytes. It invented surrogate escapes to represent bytes that didn’t fit the platform’s purported filesystem encoding. This way, Python 3’s Unicode str can represent arbitrary bytes in filenames. (The first commit to beets happened a bit before Python 3.0 was released, so perhaps the project can be forgiven for not adopting this approach in the first place.)

A few lingering details still worry me about surrogate escapes:

Migrating people’s databases containing old bytes paths to surrogate-escaped strings won’t exactly be fun.
Might surrogate escapes tie us too much to the Python ecosystem? What happens when you try to send one of these paths to another non-Python tool that interacts with the same filesystem?
People in the Python community have misgivings about the current implementation of surrogate escapes. Nick Coghlan summarizes. We’ll need to investigate the nuances ourselves.

Require UTF-8 Everywhere

One day, I believe we will live in a world where everything is UTF-8 all the time. We could hasten that glorious day by requiring that all paths be UTF-8 and either rejecting or fixing any other filenames as they come into beets. For now, though, this seems a just tad user-hostile for a program that works so closely with your files.

Pathlib

We could switch to Python 3’s pathlib module. We’d still need to choose a uniform representation for putting these paths into our database, though, and it’s not clear how well the Python 2 backport works. But we do have a ticket for it.

flexible attributes, in which schemaless and schemaful coexist peacefully

2013-09-02T00:00:00+00:00

Since the beginning, beets has been conceptually built around an object–field data model. Your library is made up of item objects, which represent audio files, and each item has a bunch of fields corresponding to tags on those files: “artist,” “title,” and “year,” for example.

Over the years, beets has grown to support a long list of fields: by my count, we currently support 59. We’ve used an informal policy of adding almost every reasonable field that a user requests. But even this liberal approach can be too constricting: adding a new field at least requires getting the attention of a developer and then waiting for another beets release to roll around.

But what if users could add any field they could think of, without needing to rely on changes to beets itself? Decoupling the list of fields from the beets schema would enable awesome new use cases, great new plugins that aren’t possible today, and even simplify the development of some oft-requested features.

Flexible Attributes

The next version of beets, 1.3, overhauls the internals to support flexible attributes (affectionately known as flexattrs). You’re no longer constrained to the fields that beets ships with: you can now store information in any field name at all. For example, maybe you’d like to start classifying your music based on its mood. Even though beets itself doesn’t use a mood field, you can set it on your music using the modify command:

$ beet modify mood=sexy artist:miguel

Then, later, if you want to see all your music whose mood is “sexy,” just use the list command:

$ beet ls mood:sexy

This is exciting enough on its own—you can classify music based on any criteria you can think of—but it also means many more possibilities for plugins in the future. (Here a are few ideas.)

How it Works

I procrastinated for a long time in implementing flexible attributes because a clean, maintainable implementation was surprisingly tricky to get right. (This post is going to get a little nerdy from here on out; if you’re interested in the technical details, read on.)

As background, beets libraries are stored in SQLite databases. The schema is simple: there’s an items table and an albums table, and each field is a column on these tables. To support flexattrs, I initially considered switching to a schema-free database: abusing SQLite as a key–value store, serializing the whole database to JSON, LevelDB, or the like. With the help of other developers, however, I eventually realized that we could tack on flexible attributes alongside the existing fixed attributes and minimize the disruption.

Here’s what it looks like from a developer’s perspective. If you write:

item.artist = 'Rhye'

beets updates the fixed artist attribute, which is backed by an SQLite column. If you make up your own field name:

item.mood = 'ethereal'

then beets detects that you’re defining a flexattr and updates a separate key–value table instead. The idea here is that code that accesses fixed or flexible attributes should look identical—that way, we can prototype code using flexible attributes and eventually promote the attribute to built-in field with zero changes to the client code. Three cheers for sound abstractions!

Queries, Fast and Slow

Things get slightly more complicated when querying. For example, to query the database for songs by Miguel, beets constructs a SQL query like:

SELECT * FROM items WHERE artist = 'Miguel';

This query can be fast because it’s performed entirely by the native SQLite library; no Python is involved. But to match on a flexible attribute, we’d need to generate complex query with a JOIN on the flexattr table. This gets especially hairy when wildcard-matching on any field or using a special query type like regex queries or fuzzy queries. To avoid this complexity, beets instead evaluates queries involving flexattrs in Python. It fetches every item in the database and “manually” checks the query against each row. This is probably slower than the SQLite query used for purely fixed attribute queries, but it’s probably not much slower—if at all—than the complex JOIN on the flexattr table that would otherwise be required.

To support flexattr queries, beets now classifies all queries into fast and slow queries. As always, Query objects have a clause() method that returns the SQLite WHERE expression that evaluates them and a match(item) method that returns a boolean indicating whether a particular item passes the query. With the flexattr changes, the clause() method can now return None, indicating that the query is slow and cannot be evaluated in SQLite. The library then falls back to one-at-a-time predication.

This fast/slow distinction also had the knock-on effect of simplifying the way beets handles “special” queries like regexes and fuzzy matching. These query types can’t easily be written in SQLite, so they’re implemented as Python predicates. Previously, to shoehorn everything into a SQLite query, we had to register a Python function as a callback and then name the function in the clause() result. Crossing the Python/C barrier is not great for performance, so sticking to SQLite queries here was likely not buying us much. Now, any query involving a complex predicate implemented in Python is just evaluated using the slow path—eliminating the need for callback registration.

Next Steps

I’m happy with the overall design for flexible attributes, but there’s a laundry list of things we can do better with over time:

Ideally, it should be easy to “promote” a flexible attribute to a built-in fixed attribute. This almost works already, but we’ll need a way to transparently move the data from the flexattr tables to the main item and album tables. We can cross this bridge when some field eventually needs migration.
A query that matches on both a fixed attribute and a flexible attribute, like artist:miguel mood:sexy, is currently classified as “slow”—any slow component makes the whole query slow. This could be optimized by matching the fast component in SQLite and then filtering on the slow component in Python. In general, we need to measure the performance of this feature to see whether optimizations like this are necessary.
I’m not ruling out an eventual move to a completely schema-free design, perhaps using a simple key–value store like LevelDB or a putative SQLite4. Eliminating the distinction between the two types of attributes will pay off in simplicity.
Beets needs a system for specifying value types. Currently, we have an ad-hoc data type classification that, for example, knows to print bitrates in KHz and track numbers with zero padding. We need a more principled way to specify conversions to and from strings—at the same time, we should let plugins specify this type information for attributes they define.

moving from Google Code to GitHub: a horrible, ultimately rewarding odyssey

2013-02-28T00:00:00+00:00

The beets project began on May 14, 2008 with my first commit to a Subversion repository. On that day, GitHub was four days old.

Had I known of GitHub then, I never would have started the project on Google Code. But I didn’t, so I did, and it’s been a hell of a time gradually moving the project over. Moving the code was easy and happened earliest. (The repository is also mirrored on BitBucket as a Mercurial repo.) The wiki was its own ordeal and required manual conversion from Google Code’s proprietary wiki syntax to Markdown (thanks, vim macros). But I procrastinated on the hardest part: the issues. Today, I finally moved that last puzzle piece over.

While I’m excited to finally move from an aging, neglected issue tracker to a shiny, well-supported new thing, the process probably could not have been more painful. Here are some tips for other open-source projects looking to make the same transition.

Start early! This was only made more painful by having more than 500 issues in the Google Code tracker. I wish I had not procrastinated so long.
Use the handy, hacky google-code-issues-migrator script by @arthur-debert on GitHub.
But be aware that this project suffers from a peculiar kind of tragedy of the commons. Everybody needs this script once but no one is motivated to maintain it long term. So take a look at the repository’s forks and choose one by a reasonably recent pilgrim. Here’s mine. As messy as the code and the network of forks is, this little cottage industry is an incredible testament to the open-source bazaar and how GitHub is able to facilitate it.
To test your migration, I suggest creating a temporary, private repository that won’t bug anyone as you open and close a million issues. This way, I was able to debug lots of problems with labels, which you don’t get to see when performing a dry run.
You’re going to need to hack the script for your particular needs. Here are some things I did:
- I cobbled together some features from various forks of the repository, such as a --skip_closed flag. (Beets has hundreds of closed issues that I don’t need clogging up the issue tracker.)
- I added a flag to turn off comments and issue bodies. The issues now just link back to Google Code for their historical context. This works for us because the issues in the old tracker are old and have lots of comments, some of which are totally irrelevant. The original report text is also not usually very relevant out of context (although the title usually is).
- I changed the script to only migrate labels that have a configured mapping since I’ve resolved to fiddle with labels less and use a simpler organization scheme.
Here’s the worst part: since there’s no way to avoid creating each issue in turn, GitHub is going to send an email for every new issue that gets copied over. There’s no awesome way to get around this; you’ll just have to apologize to all your repository’s watchers. (I tried temporarily removing all the repository collaborators, but this does not, unfortunately, unsubscribe them.)
I did, however, use a temporary user to do the migration instead of my main account. This helped avoid cluttering my history with 89 different “created issue” events. Since I deleted that user, the migrated issues are now marked as having been created by a cute little ghost.
Finally, Google code lets you replace a project tab with a wiki page. I used this to redirect users to the GitHub issue tracker. Fortunately, this does not prevent access to old issues (which are linked from GitHub).

As unpleasant as it was (as I write this, I’m still deleting straggling notification spam), I recommend moving from Google Code to GitHub Issues. This is an observation that’s been made umpteen times before, but I remember when Google Code was an incredible relief from the antiquated and convoluted SourceForge—and this move has been every bit as satisfying.

your musical year in review

2013-01-04T00:00:00+00:00

Against my better judgment, I tend to think of music in years. Some part of me feels like it understands music better in the context of its time—what other bands were doing contemporaneously, the year’s major headlines, and, most importantly, the phase of my life when I first heard an album I love.

It’s the same instinct that, over the last few weeks, has flooded the Web with top-N lists. Even if I never write it down, I find myself making a list each winter of the “best” albums of the year. I’m not a music critic, but in January, I sometimes wish I were: as utterly subjective as my yearly mental list is, it helps me close the books on 2012 and file its music away in my memory before moving on to 2013’s releases.

Here’s how beets can help you with your own retrospective. First and foremost, you probably want to see a list of all your albums that were released last year:

$ beets ls -a year:2012

(I use this query—or its randomized counterpart—almost daily throughout the year to decide what to listen to.) You might also be curious to see how many albums you collected in the year:

$ beet ls -a year:2012 | wc -l
85

We can take the chronology-worshiping one step further using full release dates. Since most albums in MusicBrainz have release months and years, we can sort these albums by date:

$ beet ls -af '$month-$day $albumartist - $album' year:2012 | sort -n

That query helps me “replay” the year from start to finish when thinking about each album.

This kind of flexible collection browsing is one of the reasons I originally started building beets. I hope it helps you look back over the music of 2012.

My Favorites

For whatever it’s worth, here’s a sort, predictable list of some albums I remember fondly from last year, copied ’n pasted out of my terminal window in no particular order:

alt-J - An Awesome Wave
Passion Pit - Gossamer
Santigold - Master of My Make-Believe
Jack White - Blunderbuss
Dirty Projectors - Swing Lo Magellan
How to Dress Well - Total Loss
Miguel - Kaleidoscope Dream
Bob Mould - Silver Age
Kendrick Lamar - good kid, m.A.A.d city
Macklemore & Ryan Lewis - The Heist

Everyone should clearly love all of these records as much as I do.

rescheduling 1.0

2012-11-27T00:00:00+00:00

I blogged previously that beets would see two more betas—16 and 17—before hitting 1.0. The first upcoming beta version would include a complete overhaul of the configuration system.

Since that post in August, something awesome has happened: the beets community has contributed an collection of new features and fixes to beets, including:

The Convert plugin for transcoding and embedding album art in copies.
A convenient Fuzzy Search plugin.
The aptly-named The plugin, which helps format strings for sorting.
A Zero plugin for nulling out certain fields.
The new IHate plugin to help you skip over albums.
A completely rewritten and more stable version of the ReplayGain plugin.
Album art image resizing.
… and much, much more.

A million thanks to all the contributors over the last few months (see the changelog for credits).

Work on the as-planned beta 16 has been going on concurrently under the confit branch. But, at this point, it would be unwise to unleash all these great new contributed features in the same release as the major refactoring that Confit entails.

So the release schedule is changing. The current development version will become 1.0 RC 1 in the next few days. After a testing period, we’ll release 1.0. At this point, the Confit branch will be developed as 1.1. This will let us fix bugs in the stable version of beets while we polish up 1.1 as the next major version.

To summarize: 1.0 very soon; Confit right after that. Here goes!

the end of Twitter syndication is nigh

2012-10-06T00:00:00+00:00

Since before beets had a real blog of its own, I’ve been using the @b33ts Twitter account as a simple, zero-effort way to make announcements about the project and keep users up to date. Most of the news that comes out of a small open-source project like beets doesn’t warrant a full blog post, so 140-character summaries are usually enough. And the shorter format encourages me to post more often.

I’ve also always syndicated the Twitter feed to beets’ homepage (you can see it on the right hand side there). I like having the latest news from Twitter up there to show visitors that beets, unlike so many other open-source projects, is under active development. And returning users get to see what’s new at a glance without my needing to cross-post to the blog.

With Twitter’s new API restrictions, this kind of syndication is no longer possible. Twitter is exerting more control over how tweets are displayed and this news feed violates the new display requirements. I’m not shocked by this development. In fact, it always surprised me that this use case was allowed in the first place—I’m making demands on Twitter’s servers and giving them nothing in return. So, while it will be a hassle to switch to a different solution, I understand that it needs to be done.

But what should would-be syndicators do instead? This is an honest question—I don’t know of a good solution that obeys Twitter’s new rules. I like being able to post with the many awesome Twitter clients out there and I’m sure users want to see updates in their streams, so I will keep using the @b33ts account. But the news feed on the homepage is also too useful to give up. Here are the alternatives I know of:

Use Twitter’s new official widgets. These widgets are large, clunky, and image-laden—I’m not sure where Twitter intends these to be used (MargaretAtwood.ca, I guess?), but it’s certainly not in a sidebar element. (The old twitter widgets, which featured a less visually intrusive design, are deprecated.)
Cross-post. Twitter’s new rules restrict how you get data out of Twitter—not how you add data to Twitter. This means that I could “microblog” (is that still a word?) somewhere else—on GitHub Pages, say—and run a cron job to post those updates to Twitter. Syndicate in, not out.
Tent or App.net, I guess?

None of these are as simple as my current setup with Twitter. Do you have a better idea for maintaining this use case? Please send me email or—if you’re not too bitter about it yet—tweet at @b33ts.

beets and Python 3

2012-09-09T00:00:00+00:00

Python 3 is the future. Every day, it seems, I come across a Python 3 feature that I wish I could use in beets (to name a few: lru_cache, yield from, working namespace packages, and my own patch, aliases in argparse). And with the release of Python 3.3 imminent, I think the age of widespread adoption is nearly here.

So when will beets move to Python 3? As with so many other projects, beets’ transition is blocked on its dependencies. Before we can make the jump, all of these libraries, which are required by beets “core” or its included plugins, need to go first:

Mutagen, the audio tag manipulation library. This is beets’ main barrier—Mutagen is an integral part of beets; it will be complex to port due to its Unicode-related logic; and I have no intention of switching to a different library (TagLib being the only viable alternative I know of).
Munkres, a bipartite matching solver. Should be a straightforward port.
python-musicbrainz-ngs, bindings to the MusicBrainz API. I and the other contributors to this library have been moving toward Python 3 compatibility for some time.
pyacoustid and audioread support beets’ acoustic fingerprinting plugin. I wrote both of these libraries and should be able to port them with minimal effort.
python-gstreamer is used by the BPD plugin and the current incarnation of the ReplayGain plugin. I believe that forward-compatibility just entails moving to Gstreamer 1.0 and PyGI.
Flask, the basis of the Web interface plugin, currently does not support Python 3. The author, Armin Ronacher, has been critical of some changes in Python 3, especially with regard to Web applications and frameworks. So Armin may need to help fix Python itself first before Flask and Werkzeug become 3-compatible.

Some dependencies (Unidecode, pyLast, Colorama, and Bluelet) are already compatible.

The principal blockers, then, are Mutagen and Flask. The remaining dependencies either have clear upgrade paths or are simple enough that I should be able to contribute compatibility patches back to them. But Mutagen and Flask are too complex (and popular) to make a unilateral porting effort feasible. This means that, if you want to help bring beets to Python 3, the best place to start is with porting Mutagen.

In the meantime, I’m writing beets to be as forward-compatible as possible. An eventual port will still be a headache—due to the need to explicitly manage a distinction between Unicode and raw-bytes path names—but I’m excited to do the work as soon as it’s feasible.

When the leap does eventually happen, I plan to leave Python 2 behind entirely. Beets is not widely used as a library (Headphones is the only dependent I’m aware of) and most Linux distributions these days ship with some version of Python 3. Going 3-exclusive will make maintaining beets’ bytes/Unicode distinction much simpler.

While beets will almost certainly not make the Python 3 leap until after 1.0, that day is not as distant as it might seem at first.

the SQLite lock timeout nightmare

2012-08-24T00:00:00+00:00

Software has bugs. There are little bugs: the beets release notes are saturated with them. And then there are the monstrous, enormous bugs: the kind that follow you from version to version, from year to year.

This is a story about one of those bugs. It existed in eleven releases of beets over almost two years. The problem stuck around for so long because it seemed to manifest exclusively on other people’s machines. Until the day I finally fixed it, I never reproduced the bug once on my own box.

Here’s what the bug looked like to users who experienced it: beets is running along normally, happily chewing through your multi-terabyte music collection and making corrections. Then, seemingly at random, it crashes and spits out:

sqlite3.OperationalError: database is locked

This is particularly frustrating because there’s no correlation at all between what you do as a user and when this exception comes up. Like appendicitis, OperationalError can strike at any time, which makes it all the more maddening.

The Problem

A little bit of background: beets uses the amazing SQLite database library to store its music catalog. When importing music, multiple threads collaborate to speed up the process and several of the threads have to read from and write to the database. Fortunately, SQLite’s transactions and ACID guarantees make this straightforward: each thread gets to make atomic accesses without bothering the other threads.

But things can go wrong. If a transaction stays open too long, it can block other threads from accessing the database—or, in the worst case, several threads can deadlock while waiting for each other. For exactly this reason, SQLite has a lock timeout built in. If it ever sees that a thread has been waiting for a lock for more than five seconds (by default), it throws up its hands and the user sees the dreaded database is locked error.

So the solution should be simple: somewhere, beets is holding a transaction open for more than five seconds, so we can either find the offending transaction or crank up that timeout. But herein lies the mystery: five seconds is a long time. That beets spends 5,000 milliseconds manipulating the database in a single transaction is indicative of something dark and terrible. No amount of SELECTs and INSERTs at beets’ scale should add up to five seconds, so turning up the timeout parameter is really just painting over the rot.

So I looked at every line in the source where a transaction could start. I made extra-double-sure that filesystem operations happened only outside of transactions. I fastidiously closed every cursor after each SELECT. But all this was to no avail—the bug reports continued to pour in.

At this point, I was almost certain that nothing was wrong with beets’ transactions in themselves. I measured the length of each access and, on my machine, they each took a handful of milliseconds apiece—nowhere near a full five seconds.

The Real Problem

I finally gave up trying to reproduce the problem on my own machine. Eventually, one incredibly helpful user offered to give me guest SSH access so I could see the bug manifest in vitro on his machine.

I again set about measuring the length of each transaction. And again, most transactions were in the one- or two-millisecond range. But, this time, an occasional transaction would sometimes take much longer: 1.08 seconds, say. And, eventually, some errant transaction would take 5.04 seconds and beets would crash: database is locked.

But there was a pattern. Every long-lasting transaction took slightly more than an integral number of seconds. I saw transactions that took 1.02 and 1.04 seconds, but never 1.61 seconds or 0.98 seconds. Something was adding whole seconds to transactions’ latencies.

Digging through the SQLite source code, I looked for places where it could sleep in whole-second increments. I found sqliteDefaultBusyCallback, the function that gets called when SQLite tries to acquire a lock but finds that it’s held by a different thread. In ordinary circumstances, that function uses a simple backoff algorithm to wait a few milliseconds before trying again. But that reasonable behavior is wrapped in a preprocessor conditional like #if HAVE_USLEEP and, if SQLite doesn’t think the system can sleep in millisecond intervals, it sleeps for a whole second each time.

So this was why some users saw this horrible behavior but I never did: all my systems have SQLite compiled with HAVE_USLEEP=1. Disassembling SQLite on my machine and the affected user’s confirmed the difference. Even though usleep is so old that it was obsoleted by nanosleep in 2001, the user’s SQLite had somehow been compiled assuming it did not exist.

The mystery was solved. And while one solution would be to berate the world’s software packagers into compiling SQLite with HAVE_USLEEP=1, we needed a nearer-term solution.

The Solution

A simple solution would be to crank the SQLite lock timeout up to eleven. But I wanted something a little bit more durable and a little less ad-hoc. So beets’ eventual solution to the SQLite Lock Timeout Bug from Hell kills several birds with one Pythonic stone:

Ensure that SQLite locks can never time out because they never contend.
Through a simple coding convention, make it easy to avoid accidentally leaving a transaction open longer than it needs to be.
Use portable synchronization that will work if beets eventually moves to a dumber storage backend that doesn’t have its own concurrency support.

To accomplish all of this, beets uses explicit transactions that make it obvious where database accesses begin and end. And those transactions are made mutually exclusive using Python-level locks to ensure that only one thread accesses the database at a time.

Here’s what it looks like. When a thread needs to access the database, it uses a with block and a “Transaction” context manager to query and manipulate the data. Here’s an example in which a Library object looks up an Item by its ID:

with self.transaction() as tx:
    rows = tx.query('SELECT * FROM items WHERE id=?', (load_id,))

The only way to access the database is via methods on the Transaction object. And creating a Transaction means acquiring a lock. Together, these two restrictions make it impossible for two different threads to access the database at the same time. This reduces the concurrency available in the DB (appropriate for beets but not for, say, a popular Web service) but eradicates the possibility of SQLite timeouts and will make it easy for beets to move to a different backend in the future—even one that doesn’t support concurrency itself.

To make this explicit-transaction approach feasible, transactions need to be composable: it has to be possible to take two correctly-coded transactional functions and call them both together in a single transaction. For example, the beets Library has a method that deletes a single track. The “beet remove” command needs to remove many tracks in one fell, atomic swoop.

The smaller method—Library.remove—uses a transaction internally so it can synchronize correctly when it’s called alone. But the higher-level command has to call it many times in a single transaction, like so:

with lib.transaction():
    for item in items:
        lib.remove(item)

To make all of this work, I want to make the outermost transaction the only one that synchronizes. If a thread enters a transaction and then, before leaving the outer one, enters another nested transaction, the inner one should have no effect. In this case, the transaction that surrounds the for loop needs to synchronize with other threads, but the inner transactions (inside each call to lib.remove) should be no-ops because the thread is already holding a lock.

To accomplish this, each thread transparently maintains a transaction stack that keeps track of all the Transaction objects that are currently active. When a transaction starts, it gets pushed onto the stack; when it finishes, it pops off. When the stack goes from having zero transactions to one, the thread acquires a lock; when the last transaction is popped from the stack, the lock is released. This simple policy allows beets to safely compose transactional code into larger functions.

Takeaway for Other Projects

What can we learn from the vanquishing of this monstrous bug—other than the well-known fact that concurrency bugs are horrifying? I think there are two lessons here: one for everybody who uses SQLite and one developers of any small-scale, desktop application that uses a database.

Assume SQLite Sleeps Whole Seconds

If you use SQLite, you currently need to assume that some users will have a copy compiled without usleep support. If you’re using multiple threads, this means that, even under light contention, some transactions will take longer than five seconds. Either turn the timeout parameter up or otherwise account for this inevitability.

I haven’t seen this particular quirk documented elsewhere, but it should be common knowledge among SQLite users.

Try Explicit Transactions

If you’re writing a small-scale application that doesn’t need highly concurrent access to a database, consider using explicit transactions based on a language-level construct (Python’s context managers are a perfect example).

Without explicit transactions, it’s hard—impossible, in some cases—to see where transactions begin and end. So it’s easy to introduce bugs where transactions remain open much longer than they need to be. There are several advantages to marking the start and end of every transaction:

It’s easy to verify that a transaction ends in a timely manner.
You can add synchronization to unsynchronized datastores like LevelDB or flat files.
You can interpose on transactions for debugging purposes. For example, you might want to measure the time taken by each transaction. (This technique was instrumental to diagnosing this bug in beets.)

And if you’re coding for SQLite in Python, feel free to steal beets’ Transaction implementation—it’s open source!

the road to 1.0

2012-08-17T00:00:00+00:00

Beets has been in beta forever—for small values of forever, at least. I made the first commit to the original Subversion repository (for shame!) on May 14, 2008 and uploaded version 1.0b1 on June 17, 2010. I’m now working on beets’ sixteenth beta. It’s high time that the project hit the big 1.0. I’m excited to give beets the point-zero stamp of approval, but I have two big, backwards-incompatible changes I want to make before sprouting a stable maintenance branch. In this post, I’ll describe my plans for beets 1.0b16 and b17—and beyond.

Beta 16

As nerd-oriented software, beets has a lot of configuration options. The config file, affectionately known as ~/.beetsconfig, has used an INI-like syntax as defined by Python’s ConfigParser since the beginning. More than two years later, beets has outgrown the dated, inflexible ConfigParser system. So the next beta of beets will focus on ripping out ConfigParser altogether and replacing it with an entirely new configuration system that solves many problems at once. Here’s what you can expect in 1.0b16:

YAML syntax. The config file has grown a troubling number of half-working syntax hacks over the years. Lists of things have to be separated by whitespace, which means you can’t use spaces in regular expressions. Egregiously, I had to force users to substitute _ characters for :s in query-based path format keys because ConfigParser splits on colons. Because it has a well-defined and nuanced syntax specification, YAML-based config files will eschew these hacks: lists will look like lists and all characters will be treated as equals.
You’ll have a ~/.config/beets directory (or the equivalent on Windows). No more cluttering your home directory with the configuration file, the state file, the library database, and maybe even the import log.
ConfigParser does not play nice with Unicode. It’s 2012, folks.
You’ll be able to combine multiple configuration files—for example, using a global base configuration with per-location overrides.
Most crucially, programming with the ConfigParser API is getting to be a nightmare. Look no further than the monstrous import_files function signature to witness the pain of threading every config option through the UI to the business end of the code. Every time I add a new config option or command-line switch to beet import, I have to touch at least five files to keep the frontend and backend in sync and update the unit tests. And that’s just to parse the option: the real work for the feature begins after all this. The ConfigParser disaster discourages me from adding new features to beets.

These problems are so basic that I don’t think I’m alone in growing uneasy with ConfigParser. So I’m writing a new configuration library called Confit (that’s pronounced con-FEE). I hope to make Confit into the best available library for configuring Python applications. I’ll have more to say about Confit on this blog as work progresses.

Beta 17

Unlike b16’s configuration overhaul, beta 17’s nightmare is a less user-visible one: the plugin API. If you browse through the code for beets’ standard plugins, you’ll quickly see that they all employ a tragic conflation between module-scoped and object-scoped variables. There are globals all over the place and a nonsensical distinction between decorated functions and specially-named methods.

The plugin API overhaul will consolidate everything into the object scope. Plugins will follow the singleton pattern, so it will no longer be necessary to keep anything module-global or assigned to the class itself. Decorators will be deemphasized; events will use a method naming convention instead.

And, finally, we’ll drop namespace packages. I didn’t know this when I first started using the beetsplug namespace package for beets plugins, but a bug in distribute utterly breaks them when they’re installed with a program like pip. This meant that some users could never use third-party plugins without reinstalling beets. (PEP 420 fixes everything but, alas, won’t be backported to Python 2.x.)

While these changes aren’t particularly exciting for end users, it’s important that I break all the plugins before 1.0 rather than after. The goal is to make beets’ plugin system future-proof—to contain the spaghetti before it spreads.

Beyond

With these two major changes out of the way, it will be time for some release candidates and then a massive party as we release 1.0. At this point, I plan on dividing beets development into a stable/trunk development model: version 1.0 will see bug-fix-only releases while the new features go into a separate 1.1 branch. This will let me—and maybe other developers?—experiment with new stuff without rocking the boat for users who don’t want to be bothered.

I have some exciting plans for new directions post-1.0. But, for now, I have two big betas to work on—we can talk about 1.1 and beyond a little later. Keep that dial right here.

Tomahawk resolver

2012-08-03T00:00:00+00:00

Beets is a music library manager—not, for the most part, a music player. It does include a simple player plugin and an experimental Web-based player, but it generally leaves actual sound-reproduction to specialized tools.

Tomahawk is one particularly exciting new open-source music player. The magic of Tomahawk lies in its ability to consolidate many sources of music into a single player interface. (It’s also a very nicely-designed, cross-platform player even if you don’t count the magic.) To integrate new sources of music with Tomahawk, you just have to provide a resolver: a piece of code that searches for music and gives it to Tomahawk to display and play back.

There’s now a beets Tomahawk resolver that can hook your meticulously organized beets music library into the Tomahawk interface. And it even works remotely, so you can stream music from a server running beets to a different machine running Tomahawk.

To use the resolver, first run the beets Web plugin on the machine with your music. Just add web to your “plugins” line in your config file and then run beet web to start the server.

Then, open the Tomahawk settings and check the beets service. Click the wrench icon next to the beets resolver to configure it:

You’ll need to enter the hostname and port of the beets Web server. (You don’t need to change anything if you’re running the server on the local host on the default port, 8337.)

Tomahawk will now be able to find tracks from your beets library. Type a query into the “global search” box and rock out.

beets has a new blog

2012-08-01T00:00:00+00:00

Welcome to the brand new beets blog! The beets project has had a Twitter account for some time now, but I increasingly have more to say than will fit in 140 characters. This blog will be a venue for more detailed writing about the project, Python, open source development, and music software.

These are some of the posts you can expect as the blog gets rolling:

In-depth descriptions of new features and design changes in beets.
Guides to doing cool, non-obvious stuff with beets.
My plans and proposals for how beets will evolve.
Useful Python snippets and libraries that grow out of the beets project (like Bluelet, Confit, pyacoustid, and audioread).
Opinions on Python development and maintaining open source projects.

You can subscribe to this blog using its Atom feed. The Jekyll source for the site is on GitHub.