Code is Sisyphean

Claude Notify

2026-08-23T00:00:00+00:00

Has this ever happened to you? You write out an elaborate prompt that you know is going to take a while but you have to choose between either

(a) babysitting the session while it churns to occasionally say “yes, proceed” when it asks for confirmation, or
(b) doing something else and checking back on it later only to realize it needed confirmation moments after you clicked away.

I want a third way.

Pass 1: Terminal tab renaming

My first pass on this was to have Claude auto-rename the tab when it required input. It would do something like:

Indicating normal processing: “🟢 { normal tab title }”
Indicating prompt for a decision: “🟡 Decision { normal tab title }”
Indicating prompt for permission: “🔴 Permission { normal tab title }”

I forget the exact prompt I gave Claude, but it was something like:

I want you to change the tab title of my iTerm2 tab depending on whether you are processing a prompt response, needing my input for a decision, or needing my permission. It should use { the options I wrote out above }.

Ultimately, this ended up not working because Claude is normally already modifying the tab title with a small status icon that it animates when it’s doing things (this does not clearly indicate confirmation prompts though).

Forcing this approach seems like it would likely be a lot of effort and maintenance so there’s probably an easier way.

Pass 2: Toast?

Remember “Toast” notifications in MacOS? This was an earlier version, in the early 2010s, I think. Little rectangle pop-ups in the upper right corner that anything could tap into. I wasn’t yet a Mac user then, but on Linux I had some window manager plugins installed that emulated MacOS Toast.

“Toast” is no longer available, but there is the osascript library for AppleScript. This is a MacOS native library (that effectively replaces Toast, I guess?). You can open a terminal right now and enter this:

osascript -e 'display notification "Hello world"'

It Just Works!™

I’ve not written AppleScript before and haven’t needed it enough to justify prioritizing learning it, but this is a pretty simple task so it seems like a good instance of having Claude make it for me.

The prompt I gave was similar to the above, but with the directive to “use osascript to display the notification (providing that URL above)” instead of changing the terminal tab.

Files

After a couple iterations, I got it to produce these two files:

`~/.claude/hooks/claude-notify.sh`

#!/bin/bash
# Posts a macOS notification when Claude Code needs a decision from the user.
# Registered on the PermissionRequest hook, which carries tool_name so we can
# tell a plain tool-permission ask apart from an AskUserQuestion prompt.
# Fires once, immediately, when the prompt first comes up (no idle delay)

input="$(cat)"
event="$(printf '%s' "$input" | jq -r '.hook_event_name // empty' 2>/dev/null)"

notify_permission() {
  osascript -e 'display notification "Claude needs permission to proceed" with title "🔴 Permission Needed"' > /dev/null 2>&1
}

notify_decision() {
  osascript -e 'display notification "Claude is waiting on a decision" with title "🟡 Decision Required"' > /dev/null 2>&1
}

if [ "$event" = "PermissionRequest" ]; then
  tool="$(printf '%s' "$input" | jq -r '.tool_name // empty' 2>/dev/null)"
  if [ "$tool" = "AskUserQuestion" ]; then
    notify_decision
  else
    notify_permission
  fi
fi

`~/.claude/settings.json`

{ 
  #
  # possibly other stuff here, ending with a comma: ,
  #
  "hooks": {
    "PermissionRequest": [
      {
        "hooks": [
          {
            "type": "command",
            "command": "~/.claude/hooks/claude-notify.sh"
          }
        ]
      }
    ]
  }
}

Usage

If you’d like to incorporate this you can either:

Do it yourself
Have Claude do it

DIY

Both the snippets above have filename headings. You probably just want to use those.

touch ~/.claude/settings.json
mkdir ~/.claude/hooks # if this gives an error about it already existing, that's fine
touch ~/.claude/hooks/claude-notify.sh

Open those files in your preferred editor. Copy the contents of the above snippets into their respective files. Read over the contents of these files to have a general understanding of how it works. While I am not familiar enough with AppleScript to write it, I would definitely agree it is very readable.

You can execute the system commands (osascript -e ...) yourself to see what the output looks like. I recommend trying this out. This might all seem like overkill, but I think it’s a good habit to get into to challenge / validate the code that we are given by our LLM tools.

N.B. the settings.json file almost certainly already exists for you. If you’re familiar with JSON, you’ll know how to integrate that segment into it. If you aren’t, I recommend having Claude help you do it (see below).

Once those are saved, in your terminal, you’ll need to make the shell script executable:

chmod +x ~/.claude/hooks/claude-notify.sh

Confirming

Getting Claude to actually help you test it might be a little tricky. When I tried it on this device (I had generated it on a different one, initially, so my current device lacked that greater context), it took a few attempts to get it to actually do something that triggered the confirmation prompt correctly.

A prompt like this should work:

I just changed @~/.claude/settings.json and added @~/.claude/hooks/claude-notify.sh - please review these to see that you understand what they do. I would like you to test them out by giving me three dummy prompts that would cause these triggers. First ask my to confirm a decision, then ask for permission, then ask me to confirm a decision again. Space these requests out by 3 seconds between each.

You should see the three notifications appear, in order.

Prompt

Choose a temporary working folder. I’ll use ~/tmp here but you can use whatever you like. Create a space in it:

mkdir -p ~/tmp/claude-notify

Then create two stub files in it:

touch ~/tmp/claude-notify/settings.json
touch ~/tmp/claude-notify/claude-notify.sh

Open the files in your preferred editor and copy the contents of the above into the corresponding file.

Now start a new session in that folder and authorize it to access the contents:

cd ~/tmp/claude-notify
claude

The prompt you give it might look like this:

I’ve written out two files here: @settings.json and @claude-notify.sh – please review these files to see if you understand them. I want you to integrate these into your settings and hooks, respectively, so that I am then notified anytime a confirmation prompt comes up. Once you’ve integrated them, please confirm with me that they work.

It will ask you for additional authorization, most likely.

Simple Link Decoration using Fuzzy Attribute Selectors

2026-05-29T00:00:00+00:00

Over on my project journal I wrote a post a while back about doing dynamically-styled buttons. They look like this, for example:

Listen on Spotify

The animation is neat and all (go dive into that link if you’re curious) but what I wanted to talk about here was writing selectors that focus on the intent of the document and decouple the content layer from the presentation layer.

On one side of the spectrum would be this very simplistic example:

<a
  href="https://open.spotify.com/user/armahillo"
  class="spotify m-1 pt-1 pb-1 pl-2 pr-2">
  @armahillo on Spotify
</a>
<a
  href="https://github.com/armahillo"
  class="github m-1 pt-1 pb-1 pl-2 pr-2">
  @armahillo on Github
</a>
<a
  href="https://instagram.com/armahillo"
  class="instagram m-1 pt-1 pb-1 pl-2 pr-2">
  @armahillo on Instagram
</a>

This isn’t specifically wrong HTML. It’s valid HTML, well-formatted, and all things considered is fine.

But I think we can do better.

Two issues I see here are:

Having a class “spotify” for a Spotify link is redundant.
Class-stuffing adds noise and conflates content/presentation layers.

What if it looked like this, but displayed identically, instead?

<a
  href="https://open.spotify.com/user/armahillo">
  @armahillo on Spotify
</a>
<a
  href="https://github.com/armahillo">
  @armahillo on Github
</a>
<a
  href="https://instagram.com/armahillo">
  @armahillo on Instagram
</a>

This would be the simplest possible HTML we could write that is still functional. I like this approach because it lets the document remain as unmuddied as possible, letting it stay lithe and adaptable.

The first thing we can do can be done in native CSS, using fuzzy attribute selectors. To add icons for each of these links without using classes, we could use this CSS:

a[href*="spotify.com"]::before {
  background: url(/assets/img/post_images/spotify-icon.png) no-repeat;
  /* remainder of styles */
}
a[href*="github.com"]::before {
  background: url(/assets/img/post_images/github-icon.png) no-repeat;
  /* remainder of styles */
}
a[href*="instagram.com"]::before {
  background: url(/assets/img/post_images/instagram-icon.png) no-repeat;
  /* remainder of styles */
}

The selector a[href*="spotify.com"] means “find any a tag that has the href property containing the substring spotify.com”. You can be as specific as you like, though note that you will likely need to escape your slashes (e.g. a[href*="docs.google.com\/document] for Google Docs).

The next step is most easily done using SCSS. You can accomplish something similar using @apply in Tailwind, as well. Using the styles from the example in the beginning:

@mixin icon-prepend {
  content: '';
  display: inline-block;
  width: 15px;
  height: 15px;
  background-size: contain;
  background-repeat: no-repeat;
  background-position: center;
  vertical-align: middle;
}

This defines a reusable block of styles. We can than incorporate those with the fuzzy-matched selectors:

a[href*="spotify.com"]::before {
  background: url(/assets/img/post_images/spotify-icon.png) no-repeat;
  @include icon-prepend;
}
a[href*="github.com"]::before {
  background: url(/assets/img/post_images/github-icon.png) no-repeat;
  @include icon-prepend;
}
a[href*="instagram.com"]::before {
  background: url(/assets/img/post_images/instagram-icon.png) no-repeat;
  @include icon-prepend;
}

Which would render a link that looks more like this:

@armahillo @armahillo @armahillo

Shovels and Ladders

2026-05-11T00:00:00+00:00

Imagine programming to be like a process of mining a flat-surface by digging straight down. The desired program behavior is somewhere below the surface, and the further down you dig, the more complexity you can mine.

So to dig downwards and add complexity, you need a shovel. But you can’t live in a hole; you need to be able to build ladders to get back out.

In this metaphor, I’m loosely equating “shoveling” with “the application of code” and “building ladders” with “the comprehension of code”.

(I suppose you could also compare it to “speaking a language” vs. “understanding a language”)

All ladder, no shovel

This could be more practically called “tutorial hell.” A dev who has done a zillion or two coding exercises and has a general understanding of what the various parts of a body of code mean. Perhaps you can even read existing code and more or less understand what it’s doing. You’ve spun up a zillion or two projects by following a tutorial.

But you still feel stuck. What do you do with this knowledge? Other than serving as a QA person for a tutorial generating company, people don’t typically get paid to just follow tutorials.

So this user is really good at building ladders, but has no shovel.

All shovel, no ladder

This is typically the metaphor I use to describe folks that are purely vibe coders. These are the users that have the vision, they have an instinct that the gold they’re mining is beneath their feet and they just need to dig far enough to get it. You’re skilled prompt-writers and are comfortable with various tools to generate sites and apps at a quick pace.

But maybe you’re introducing bugs and don’t know it. Or you do know it, but can’t seem to get the LLM to remove / resolve the bugs. Maybe the apps you’re handing off are wrought with technical debt / ugly code that is not extensible / maintainable. More typically, what I see quite a lot is “how do I fix this code?” – users that have dug themselves into a hole with no exit. Or “I’ve generated a site, how do I get it onto the Internet?”

This user is really good at digging, but they’re in over their head and can’t get out.

A measured hybrid approach

I really don’t mean this to sound like “old man yells at cloud,” or “back in my day…”, but:

Before LLMs, before passive video tutorials, we used books, documentation, and iteration.

One of the first websites I ever made was using HTML I wrote in a text editor, saved to file, and uploaded via an FTP client to my webhost, back when your ISP would give you 1MB of free space. Every “Hello World” program involves compilation/execution of the program to complete the lifecycle. This is a fundamental part of the process – we learned how to dig and how to get back out.

This is one of the reasons I advise against beginners using LLMs at all – it’s far too tempting to hop on that LLM excavator and start digging before you’ve learned how to get back out, or to even realize that this is something you’ll eventually have to do.

On the other side of things, you can’t only do tutorials – it’s fine to copy a “Hello World” program because the point is to show you “here’s how you dig a hole and then also climb out of it.” But following a tutorial on making a Tasklist, or a Bookmark manager, or a Phone directory – this is akin to following a map to a digsite and being told that “digging a hole that is 35’ deep” is the goal; you’re missing out on both understanding the how to dig and also the why.

If you’ve got a shovel but need to learn how to build ladders (Vibes only!)

Set the LLM aside for just a minute, or at least start a new session and start with a much simpler approach: How do you make a Hello World and get it where you want it to be?

For web development this means writing a perfunctory page, deploying it somewhere, and seeing it in your browser at a public URL.

For application development this means writing a Hello World program, running the compiler and executing the compiled binary.

Once you’ve got that, start chunking off pieces of your Shovel-ready-program and integrating it into your Hello World. Maybe it uses an external library or collects input or displays a dataset. Start small. This how you learn how to build a ladder, one rung at a time.

Avoid using your LLM at all for this part. It may feel frustrating (at first!), but I promise it will make you a stronger developer.

If you’ve got all the ladders but no shovel for digging (Tutorial hell!)

The typical answer I see (and likely have given) is to “identify a problem and spell out the behavior, then start iterating on it.” But this is maybe a bit too much hand-waving.

If you’ve already done some tutorials, pause on starting any new ones. Go dig up an old codebase and read back through it. What does it do? Run it. What is it missing? If you were a user, would you use it? What additional features does it need to actually be useful? Can you make it look different? Can you add new behaviors?

Example: You have a Tasklist / To-Do app. Add deadlines. Add handle prerequisite tasks (task A cannot be started until task B is completed). Add a task archive for completed tasks but hide them from the main interface. Add priority values and add ranked sorting based on different traits.

You want to have an objective that is just a few feet deeper from where you are right now. It needs to have some depth where you have to evaluate the problem and devise a solution, but you don’t want it to be so deep that it feels impossibly exhausting.

If you’re starting completely fresh and aren’t sure where to begin

In all seriousness, find a book on the language you want to learn. There are still many that are relevant today, and even if it’s a little outdated, programming hasn’t changed that much, so the information will still be transferrable. Ideally, a book that has exercises throughout the chapters or at the end are best – the chapter itself helps you learn how to build ladders, and the exercises teach you how to dig.

If you don’t have access to a library or your library doesn’t have the books (most libraries can do Interlibrary loans, ask your circulation / reference desk person!), look for online resources that both offer instruction and exercises. I personally like Exercism but there are other similar resources. You want to have both instruction and exercises, though, and you will want to do the exercises without looking up how to do it. Definitely do not ask the LLM to do it for you, if you use one.

If you are deadset on using an LLM, then try giving it this prompt:

> I am a new programmer who is trying to learn ___ language. Please give me 5 appropriately-difficult exercises I can do. I want to ask you questions about these exercises but I want you not to give me solutions, so that I can learn to write the programs myself.

Then dig in to the first exercise.

Reflections on LLM usage

2026-05-04T00:00:00+00:00

I suppose this was inevitable.

In the previous posts I made about LLMs, which I’ve condensed down to just three posts with audience focus, I spoke stridently and with great vigor against the usage of LLMs at all, essentially teetotalling.

I’ve re-read it, and I still stand by everything I wrote, which is why it remains up. But I realize I’ve been a bit one-sided. In my programming practice I try to hold pragma over dogma, and on further reflection I came to realize that I was being a bit dogmatic in my resistance.

I am definitely not an “AI drunkard”, but I would say that I am less a teetotaller now and more a “having a drink in moderation is probably ok.” I still have ethical misgivings about how we’ve introduced this technology, particularly the absurd greed-driven fervor that is spawning data-centers with little regard to the communities and various ecosystems. I still think that this technology should not belong in the hands of private companies, nor do I trust any of them to be responsible about its usage.

I still stand by every comment I made about the implicit harm it can induce on the learning process of a budding junior in their journey to becoming a competent senior.

HOWEVER.

In February or March, around the time that Anthropic released Claude 4.6, I tried out using it with a coworker while we were pairing. Previously I had experimented a little bit with Opus 4.3 to mild success. It was neat but still felt unreliable and stunted. Opus 4.6, on the other hand… well. I get it now.

Delegating to Claude

One of my earlier tenets of using LLMs was to “treat it as a junior, not a senior.” Typically, this means delegating work to it and then reviewing / checking that work. Here is a short list of things I’ve done where I’ve delegated work to Claude:

Bug hunting
Data combining and permutating
Data diving / complicated find-and-replace

In all these cases, this is work I could have done myself but was able to turn it over more quickly by delegating it.

Bug hunting

My coworker and I both had a general idea of where the bug lay, so we felt confident in our ability to check Claude’s conclusions. We described the normal behavior, described the bug, described approximately the behavior we were expecting to see, and listed out the files of where it could start looking. It churned for 5-10 minutes and then came back with a specific analysis and proposed fix. Surprisingly, it also found an additional bug that we didn’t know about but that was related.

Realistically, this probably would have taken us about an hour to do and I’m confident we would have figured it out. So there was a substantial time-saving there. The bug was also small enough, conceptually, that we could individually integrate it into our understanding of the application’s function.

Data combining and permutating

There is a tabletop board game I’ve been working on for a while. It involves a lot of traits/qualities that are permutated / combined in various ways, and it had a few holes. I gave Claude my seed list of what I had already, described the resultset I was aiming for, and then told it to fill out the list using that scheme.

This case took 15-20 minutes, but it searched the Internet to research more terms / concepts related to the dataset and came back with some very informed opinions (with actual URLs for support) about how to structure this data. I recorded the findings to a static file for reference, then had it apply this to the dataset.

The end result was very elaborate. This would have probably taken me considerably longer to generate this, and honestly I’m not sure I would have thought to look for some of the approaches it took. This was a very impressive win.

Data-diving

Normally for this kind of task I would use a tool like ripgrep (or just grep), and some regex, maybe. But sometimes the needle I’m looking for isn’t a simple string.

There have been a few cases where I’ve given it tasks like “Find me every test in the test suite that uses an order with two or more items”, or “Look through my YML files and find me every top-level record key that has mixed-case” (that one could be regexed but I wanted to see if it could do it).

I’ve also used it to do textual summaries. There are a few books I own and have read already, but also have the PDFs. By using the pdf skill, I can have Claude convert a PDF to text format, then have it create a summary index of the book, and then ask it questions about things I don’t remember and can’t find in the book’s physical index.

I actually ended up doing this with several books I owned and creating expert skills for both programming and also game design – I am aware that the LLMs have been trained on all of these books already, but that ocean of knowledge is so vast the individual perspectives have been diluted. By concentrating a skill onto 5-7 books specifically, I can get concrete reference examples when asking for opinions.

Two Exceptions

In two cases, I have treated it as a subject-matter-expert:

Converting a web application to an iOS app written in Swift
Creating a de novo multi-platform mobile app using Unity

I have very little prior mobile development experience, so I’m breaking away from the original tenet in this case. I can however, at least read the generated code and leverage my experience in a way that an Engineering Manager might – knowing what questions to ask, how to phrase them, and what things to look for.

I find it a little easier to learn new things if I can see examples that are familiar material so it’s more of a diagonal move upwards instead of an orthogonal one. As the apps have taken shape, I’ve been able to use this to build some competency about how an iOS or Unity app is structured, where files are located, and the general flow of working these apps.

Ideating with Claude

A coworker showed me how to use the /plan feature and I’ve found it to be very useful.

When coding, this has been useful for both bughunting (see above) as well proposing it enhancements or greenfield ideas. In both cases, Claude CLI is capable of seeing the full app tree and its contents.

I find myself pushing back a lot more on this, than with the more more surgical bug-fixing mentioned previously. Particularly in the way it generates code.

Most of the generated code I’ve seen so far has been “technically correct” (the best kind of correct, I’m told), but has always felt too atomic / “contained”. By contrast, what I would like to see is code that more clearly integrates / synthesizes with the existing codebase.

Using /plan does help to alleviate this somewhat, because I can get a better sense of what it’s proposing before it executes. But just to be safe, I always ensure I have a clean git status before beginning, so I can review the changes in detail. Sometimes I treat the Claude contribution as a spike and then go back and rewrite it myself, refactoring it to integrate better into the code, as I prefer it.

Assisting with Claude (rather: being assisted by)

There’s two ways I want to approach this:

How I have it assist me, broadly speaking, via CLAUDE.md
How I use it as an Assistive Technology, as a neurodivergent person

`CLAUDE.md`

Claude, specifically, looks for a file called CLAUDE.md. You can put one in the root of any of your applications. You can also put one in your home folder (~/.claude/CLAUDE.md).

In both cases, it’s a document that describes, in plain english, what you’re trying to do and how you want Claude to behave.

`~/.claude/CLAUDE.md`

I treat this one like a very broad “get to know me” directive. It describes the kinds of work I will be using Claude for, and any specifics about how I want it to help me.

For my development work, I describe the kind of work I do (web development using HTML, CSS, and JS, as well as doing a lot of development in Ruby and Rails). I describe my coding practices and preferences in broad terms, as well as what level of specificity I would like it to have in its responses.

For my game design work, I describe the types of games I often make, some general thoughts about my approach towards design, and also the document structure of my design documents. I have some ruby scripts I wrote up a while ago that I use to convert YML files to CSV files that I can ingest into dextrous.com.au to export cards from. Additionally it has designated areas for ideation docs, playtest logs, etc. All of this is described in the doc.

Finally it has some general statements of the kind of attitude I want it to have (TL;DR - avoid sycophancy and be concise but with clarity).

`.../{repo}/CLAUDE.md`

In these files, typically for development projects, I will generally first summarize the project at the 10,000 foot level, then do subheadings with more specific details about implementation that it would need to know. I treat this as a sort of “onboarding” document.

My best advice here, if you do this, is to create the file, write a summary of a few sentences about the project, and then spin up a Claude CLI session in that directory and give it this prompt:

> Please read through @CLAUDE.md and the rest of this project and let me know what more you need me to clarify.

Even better, do it in /plan mode so it can give you a Q&A and come up with all the edits it needs to make, and then it can write out the updated doc itself. (Proofread it!)

As an Assistive Technology

Other examples of “Assistive Technologies” would be things like screen-readers for vision impaired, subtitles for hearing impaired, curb-cuts for people in wheelchairs / mobility issues, etc.

For me, the issues I have are cognitive. When I have challenging days, it feels like the desktop in my brain is very cluttered / has little free space. Sometimes this means it can be harder to lay out a lot of different moving parts in abstraction to synthesize the ideas together. I’ve found Claude to be very helpful with this, particularly in /plan mode – I can lay out a problem, reference files, give it additional contexts, and have it help me better visualize the whole problemspace more effectively.

I can tell this is helpful on these days because I feel a sense of mental relief where normally it would be anxiety/frustration.

In the past, the way I would approach this is typically through writing a lot more things down, drawing diagrams, etc. It helped, though with limitations. I can also take transcripts from calls and import those transcripts into the Claude dialogue and integrate those into the abstraction.

Honestly, this relief alone is worth the $20 a month I’m paying.

Code is Sisyphean

2025-11-03T00:00:00+00:00

This eponymous post applies to developers of all skill levels, but especially to newer developers.

The metaphor I keep coming back to is to imagine your journey as a developer as similar to one as a mountain climber, or perhaps more on-brand for this blog: the Sisyphean task of pushing a boulder up a mountain.

Pushing a Boulder

If we can imagine getting the boulder to the top of the mountain as “solving a problem with technology”, then the act of pushing the boulder up is “identifying the problem, charting a path towards a solution, and writing the code to solve the problem”.

Using an LLM would, in its most ideal case, be something like using a cable-pulley system to pull the boulder up, or using a tractor to push it, etc. In reality maybe the LLM takes you up to the peak of the wrong mountain, or to a sub-summit peak, or pulls only part of the boulder, etc.

In this metaphor, the benefit of pushing the boulder yourself is that it builds your leg and arm strength. It builds your determination. It builds your self-confidence (noted in the CMU study, above). It hones your ability to identify the smoothest path up the mountain.

Utilizing external forces – the cable pulley, the tractor – to do this work for you, that’s not going to build those traits that you need to do it on your own. Worse yet, the underutilization of those traits will likely weaken them.

Every mountain is different, even if they all have a summit. Some mountains are not going to be able to utilize the cable-pulley or tractor for some or all of the ascent. In those scenarios, you will be forced to push it yourself, relying on your bodily strength, situational path awareness, and force of will to continue that challenging ascent.

Becoming & Staying Strong

You aren’t going to forget everything you know because you had the boulder dragged up the mountain once, or even twice. But be vigilant about your habits: are you finding yourself consistently taking the easy way out?

For newer devs, my advice is to completely eschew all LLMs while you are still learning. “Treat them as a junior, not a senior” means that while you are a junior yourself, you don’t use them because there aren’t any tasks you can delegate that you’ve yourself mastered.

You are probably going to see your peers that are doing things with generative LLMs appear to be advancing more quickly. These will be the same peers that inevitably post on reddit

Start strong. Stay strong.

Advice to Seniors regarding LLMs

2025-10-27T00:00:00+00:00

Beware skill atrophy

By “skill atrophy” I mean “skill rot”, “becoming rusty”, “dulling your blade” or any of the other colorful metaphors people use. It’s analogous to a runner getting out of practice and losing speed, or a litigator who hasn’t seen the courtroom in a long time and doesn’t have quite the same edge.

Cell Phone Contacts

A more banal comparison that those of us of a certain age might know:

Those of us that grew up without cellphones probably used to have many phone numbers memorized. Since cellphones (“dumb” phones, included!) have contact lists that reference numbers by name instead of digits, our brains have no reason to memorize new phone numbers.

I still remember most phone numbers from my childhood, but only have a handful of phone numbers memorized now (close family members).

In development, it might look like forgetting syntax more frequently, or finding it harder to refactor, or even as severe as getting executive dysfunction when confronting a programming problem.

Self-confidence and critial thinking in the age of “AI”

In this 2025 Carnegie Mellon study The Impact of Generative AI on Critical Thinking: Self-Reported Reductions in Cognitive Effort and Confidence Effects From a Survey of Knowledge Workers

Specifically, higher confidence in GenAI is associated with less critical thinking, while higher self-confidence is associated with more critical thinking. Qualitatively, GenAI shifts the nature of critical thinking toward information verification, response integration, and task stewardship.

The study is careful to acknowledge:

Our analysis does not establish causation. However, based on our evidence, it is possible that fostering workers’ domain expertise and associated self-confidence may result in improved critical thinking when using GenAI. Task confidence significantly influences how users engage with AI tools, particularly in the context of human-AI “collaboration” (not withstanding objections to that term).

LLM-driven development is still a very novel concept, so we really don’t have any longitudinal studies objectively showing skill atrophy.

Opportunities for growth

Personally, I enjoy programming. The depth of knowledge I have about my particular slice of the coding world is something that is consistently useful at my job, especially with bug hunting and resolution. I would be disappointed to lose that edge or become reliant on a third party service to be able to perform my job.

I have yet to be convinced that using an LLM to generate a solution will save me a significant amount of time, when you consider the “total cost of ownership” for that solution – factoring in review time, corrections, and implementation time. It feels more jarring and stunted than approaching it more conventionally.

Given that, I see every opportunity to program as an opportunity for me to hone my skills a bit more, to become a bit sharper, and to learn new things.

Over-reliance on an LLM means undermines your value-add as an employee

This applies more for juniors, but seniors should beware too:

I see a lot of organizations that have “LLM-required” mandates from management. I see this as a huge red flag that those developer jobs are in danger of being replaced or reduced. For whatever acceleration they may tout they are targeting, the real danger here is that if your job becomes purely prompt engineering, you are being forced into potential skill atrophy and also your role can be replaced by someone who can be paid less for doing the same work.

“AI”-driven Layoffs

Recently, Amazon laid off 14,000 corporate (read “white collar”) employees. It included employees in “Amazon’s cloud, grocery, video games, human resources, sustainability and communications, ads and devices”. Management at Amazon very much believes humans can be replaced by LLMs, and is acting on that belief.

Fortune magazine quotes Jason Schloetzer:

“it’s not so much AI directly taking jobs, but AI’s appetite for cash that might be taking jobs”

The article notes these large corporations and their planned / executed layoffs:

Company	Layoffs	% of workforce	Source
Amazon	14,000	4%	AP News
UPS	34,000	-	AP News
Target	1,800	8%	AP News
Nestlé	16,000	-	AP News
Lufthansa Group	4,000	-	AP News
Novo Nordisk	9,000	11%	AP News
ConocoPhillips	2,600-3,250	20-25%	AP News
Intel	~22,000	15%	AP News
Microsoft	15,000	-	AP News
Proctor & Gamble	7,000	6%	AP News

Note that not all of these cuts are strictly due to “AI”, but this was cited expressly as a factor by more than a few of them.

LLM-driven Skill Caps

If you’re a junior and all of your development is based on generated code, then the limitation of your capabilities is capped at what you are able to produce with an LLM. You will speed out of the gate in a vehicle that has rocket-fast acceleration but a lower overall max speed.

The corrollary to this, of course, is that a developer who chooses NOT to use LLMs to drive their development will have a higher ceiling cap, though it may take longer to get to that point. Your metaphorical vehicle has a lower acceleration but ultimately a much higher skill ceiling. By choosing the slower route, you will ultimately be a stronger developer and ostensibly more competitive in the marketplace.

Market Recovery

I sincerely believe that when the hype finally dies down a bit, as it did for Blockchain, there will be a demand for skilled developers again.

The abilities of LLMs are impressive, but they are predominantly better at creation than maintenance, because creation can ostensibly be done de novo whereas maintenance requires synthesis of context. As things get more and more complex and as systems grow, it will eventually be too large or costly to perform these functions, even with RAG (Retrieval Augmented Generation).

This may or may not be partly behind the ravenously hungry appetite that the various “AI” companies have for expansion and growth – in order to make good on the bets that are being made by corporate America, there will need to be an expansion of computing power.

Inevitable Enshittification of LLMs

If you are a heavy user of LLMs, or are considering it, I would like you to consider:

How much are you spending per month on LLM services?
How effective would you be if they were suddenly unavailable?
How much of a cost increase could you realistically absorb before it became untenable? How much would be a hardship?

Enshittification

In Cory Doctorow’s 2022 (and recently re-printed!) book Enshittification, he proposes 3 stages that every online platform goes through:

The platform is good to users
The platform is bad to users, good to businesses
The platform is bad to users, bad to businesses, good to itself

The progression occurs as a platform reaches critical mass for user lock-in (enough people are established as users that they cannot easily walk away). Consider, for example, the friction you would feel from walking away from using any particular online platform.

We are somewhere between #1 and #2.

When ChatGPT first came out, there was a rapid expansion of users finding different use-cases for it. The future seemed bright and exciting. But we are also seeing LLMs being the driving factor behind layoffs. We are seeing the job market dry up for Junior Devs, which AWS Ceo Matt Garman calls the dumbest thing he’s ever heard in light of entry level jobs disappearing because of “AI”.

Mark my words: as soon as the AI companies feel they have enough users and businesses dependent on their services, they will jack up prices, significantly.

Every single AI company is still operating at a loss (overall) despite raking in massive amounts of revenue. OpenAI and Oracle have signed a 5-year $300 Billion cloud deal. For a company that is operating at a loss, their belief that they will be able to convert that to a net gain of $300 Billion (whether through revenues or investment rounds) means they have a plan for something.

Additionally, they are planning on spending $1 Trillion over the next decade to expand their computing infrastructure.

By comparison, they are currently claiming 800 million regular users, with 5% (40 million) contributing to their $13 Billion of annualized revenue.

Covering the shortfall

Hypothetically speaking, let’s just say that OpenAI alone wants to satisfy their $60 Billion annual payments to Oracle. They need to make up for a $47 Billion shortfall. They would need to increase their subscription prices / costs PER ALL USER (even the 95% of the free tier users!) by $59 / user to recover that difference.

Looking at their current plans:

Tier	Current price	Adjusted price
Free	$0	$59
Plus	$20	$79
Pro	$200	$259

Granted, there are probably other ways they could re-distribute those values, perhaps by introducing a new tier between Free and Plus that costs $20 and requires a substantial portion of Free users to convert, but one way or another, the company needs to either extract $47 Billion from its 800 million users, find more users and also raise prices, or start screwing its business customers over too.

Enshittification points towards some combination of all of those.

Dependency

This brings me back to my original point:

If you’re currently using the Free tier, could you walk away? Could you afford a $59 increase?

If you’re using Plus or Pro, could you afford a 300% or ~35% (respectively) increase in fees? Could you do your job if you had to walk away?

The cynic in me thinks that the companies demanding their employees use LLMs at their jobs are trying to get users dependent on these services, fattening them up for the inevitable slaughter.

It then becomes an act of resistance to continue to be a strong developer in the absence of these generative tools.

Advice to Juniors / Newbies regarding LLMs

2025-10-20T00:00:00+00:00

I feel like a bit of a broken record about this topic at this point.

I started my programming journey as a young kid, doing the LOGO Turtle/Terrapin programming on the Apple LC ][ computers in my elementary school. I didn’t really get into programming in earnest until I was a teen, starting out rather ambitiously with the C and C++ languages. A high school class taught me some fundamentals in Pascal, and I was learning how to do some Bash scripting as well, as a hobbyist linux user. I was a TA for my “Computer Programming & Methods” course in HS my senior year, because I had learned C++ well enough to help teach it.

This isn’t to brag, but just to provide some background illustration.

The point of learning is to learn. Reaching the solution is really only a small part, maybe 20-25%?, of software development. The real skills are in the research, hunting, and experimentation.

I’ve compared this to using Google Translate while you’re trying to learn a new spoken language. Or using a calculator to get the answer to a practice set in a math class.

The process of finding the answer is critical early on. You’re learning how to find answers, getting the correct answer itself is less important.

If you’re using an LLM to tell you how to get to the answer, you’re missing out on the necessary exploration.

Additionally, and this applies to experienced devs as well: if you use an LLM to fast track you to the answer, you’re denying yourself the time you would have spent reading through API documentation, potentially picking up additional details that are unrelated. Anecdotally, I’ve definitely learned about new methods / modalities by spelunking API docs to find an answer.

Every “wrong” answer to your problem is still something new you can learn and may be useful later. The LLM response is going to fast-track you directly to the actual answer.

In the following posts, I am going to reiterate these themes:

Learning “the hard way” is going to help you learn better
Use it or lose it: skills can atrophy
LLMs are imperfect assistants

Treat an LLM as a Junior Dev, not a Senior Dev

One significant gripe I have with LLMs is that they are equally confident about their responses whether they are spot on or completely wrong. When you point out they’re mistaken they say “you’re correct, how about this?” and give you another answer that may or may not be correct. I’ve heard it called “mansplaining as a service” because it’s high-confidence in an answer that looks convincing but without actual basis of expertise.

Spot the error

For example, if you were shown this after asking for some PHP code that would replace the word ‘fizz’ with the word ‘buzz’ from a POSTed param named ‘input’:

<?php
  if (strpos('fizz', $_POST['input'])) {
    $result = str_replace('fizz', 'buzz', $_POST['input']);
  } else {
    $result = $_POST['input'];
  }
  echo $result;
?>

Looks convincing, right? It’s even got semicolons in the right places!

What’s the mistake here? hint 1, hint 2.

Answer

In my decade+ span of using PHP, the argument order of `str_replace` and `strpos` was perennially confusing to me. In this case, they are both reversed, the correct order: `str_replace($needle, $replace, $haystack)` and `strpos($haystack, $needle)`. So it _should_ be: `strpos($_POST['input'], 'fizz')`

Anecdotal example

I recently had a coworker throw up a quick PR to fix some Javascript that we had written for one of our intranet forms. There was a bug in the existing code and this fix was to address that new behavior. My coworker generated a fix using an LLM service because they weren’t comfortable with their level of Javascript skill.

Upon review, it was pretty clear there were some issues. It added a new global-space class (appending to window), for one. The fix was an atomic unit that technically fixed the issue, but did so by adding a bunch of extra cruft that essentially “tacked-on” the solution instead of actually correcting the underlying buggy behavior.

It took me a couple hours, but I managed to detangle the generated Javascript, refactor the behavior into modules, and then integrate it into the existing code to both resolve the bug and add the enhanced behavior.

Trust, but verify

For this reason, my policy, and advice, around the use of LLMs in development is to treat it as you would a junior dev: trust it to do what you ask, but verify its correctness.

By contrast, don’t treat it as a senior dev (or at least, one that is “senior to you”) – that means don’t ask it to do things for you that you do not know how to do yourself.

In other words, if you want to delegate it work that you could have done yourself but will save you time by having it generated, where your time spend is just on the reviewing (and change-requests / modifications), then fine. But avoid delegating it work that you don’t know how to do yourself. There be dragons.

An LLM is not a Rubber Duck

Harvard’s CS50 course, widely lauded as a great intro for nascent devs, and they have created a companion LLM product for it called CS50.ai, which is presented as a tool for “rubber duck debugging.”

This may seem superficially correct, but if you have ever done actual rubber duck debugging, you might be able to detect why it is NOT that. From the wikipedia article, which I think gives a great summation:

Rubber duck debugging (or rubberducking) is a debugging technique in software engineering. A programmer explains their code, step by step, in natural language—either aloud or in writing—to reveal mistakes and misunderstandings.

More specifically:

Programmers often discover solutions while explaining a problem to someone else, even to people with no programming knowledge. Describing the code, and comparing to what it actually does, exposes inconsistencies. Explaining a subject also forces the programmer to look at it from new perspectives and can provide a deeper understanding. The programmer explaining their solution to an inanimate object (such as a rubber duck) has the advantage of not requiring another human, but also works better than thinking aloud without an audience.

I want to specifically call attention to:

“explaining to an inanimate object”
“describing the code … exposes inconsistencies”

A critical aspect of this is that the duck does not speak back to you!

Using an LLM as a coding-pair is another topic entirely, but for this case? Not a rubber duck.

Why is Rubber Ducking helpful?

The benefit of rubber ducking is that

By presuming no knowledge of the listener, you are forced to distill your understanding down to real words that can be verbalized, forcing abstractions to become concrete
By receiving no feedback, you are also forced to consider how that information is received by the listener, acting as their proxy, which means you are now re-ingesting your understanding of the problem, creating a feedback loop.
By re-ingesting the information in that feedback loop, you then do the comparison yourself, identifying gaps and inconsistencies in your understanding, driving you towards the solution.

This is similar to Jungian psychological principles about mental pluralism or the cognitive “senate” in parts work. You gain the most benefit by embodying all the roles.

When using an LLM, you are now only performing the first step (expressing the problem in plain language) but are not doing the latter parts (receiving and synthesizing that information).

I completely understand why someone might think that using an LLM as a rubber duck is an improvement – again, superficially it seems like “wouldn’t it be better if the duck could give you feedback?” But you will become stronger and more competent by embodying that full feedback loop yourself.

Beware XY Problems with LLMs

An XY problem is

where the question is about an end user’s attempted solution (X) rather than the root problem itself (Y or Why?).

This is a common consequence of situations where the attempted solution is borne from an incomplete understanding of the problem. These are usually spotted by being very specific inquiries that are devoid of context, and the correct way to address them is to first ask for the greater context, to understand why they chose to solve it in this way.

The wikipedia article gives this example:

Asking about how to grab the last three characters in a filename (X) instead of how to get the file extension (Y), which may not consist of three characters

This becomes a problem with LLMs because LLMs generally do not push back to look for XY problems. They will give you their best guess at what you are asking for, whether it’s the correct approach or not.

Anecdotal example

On another occasion, a different coworker used an LLM to generate a script that performed a series of commands over SSH to a remote server. We reviewed the script together, focusing on what we felt were the riskiest parts (the SSH commands, predominantly) and found it to be acceptable.

However we ran into a bug that wasn’t immediately apparent. Most server IDs had 5 digits (the vast majority did). Some of them had 4 digits, though. In the generated script, one of the early steps was to parse that string to split off the last 3 digits to be used separately from the first digits.

This happened because, in the initial prompt, the sample data provided had 5 digits so the generated code only accounted for parsing metadata from those 5 digits.

This approach broke for server IDs with 4 digits because it was peeling off the first 2 (always) rather than the last 3 (always) – right solution for the wrong problem. e.g.:

12345 => 12, 345 (Correct)
1234 => 12, 34 (Incorrect, should be: 1, 234)

We overlooked it in the initial pass because we gave it too much leeway and focused on the wrong aspect of the solution. We found an existing script that had already solved this:

part1 = serverID / 1000
part2 = serverID % 1000

This corrected the issue.

This is perhaps a soft XY issue, but highlights the point about LLMs giving you exactly what you ask for, monkey’s paw style.

The Rubygems / RubyCentral Fiasco

2025-09-19T00:00:00+00:00

IMPORTANT: Please review gems.coop as a replacement for source 'rubygems.org' in your Gemfile.

Updates

07 Sep 2026 - A Year Later

A lot has happened in the last year. Leadership has changed. Per that update:

We have parted ways with our Executive Director, our PR agency, our CFO, and concluded several contractor engagements. These were not easy decisions, but they were necessary to ensure the long-term sustainability of Ruby Central.

Given that these were all (likely) the key players in the disastrous mishandling of these events, I think this was a wise move by RubyCentral.

One notable addition was Brandon Weaver (as Secretary), aka KeystoneLemur. I had the pleasure of catching a car ride with him at Ruby By the Bay many years ago, and have kept up with his conference talks / blog posts. I think he’s a genuinely good person.

Additionally, they have changed from a governing board to a volunteer working board.

“That means instead of the board advising a full-time Executive Director, our board members have taken on direct roles and responsibilities alongside our hardworking team of staff and volunteers. We voted for some difficult personnel cuts and to reduce discretionary spending.”

While I still think an apology is due, it isn’t due to me personally, and perhaps it was extended privately to the affected contributors. I had said previously that either a change in leadership or governance structure would be necessary for things to heal – it appears they have done both.

Here’s hoping the future trends positively for the organization, and for Ruby at large.

29 Oct 2025 - Final Update

As of last week, ownership of RubyGems has been transferred to Matz. This decision was made by RubyCentral, who frankly shouldn’t really have the authority to make this decision anyways.

But that’s where we’re at.

RubyCentral published an absurdly lengthy Q&A Response where they continued their line of “sorry you’re mad, bro.”

I am content with RubyGems (and Bundler) landing on Matz’s lap, but RubyCentral has done nothing to restore any confidence or goodwill, as far as I’m concerned. I’m disappointed and annoyed. I don’t intend to support them or anything they do in the future. I will advise others in the same way. I don’t expect things to change until there is a change in leadership or governance structure.

10 Oct 2025

André Arko responds to RubyCentral’s “incident report”. He provides a lot more transparency and clarity than RubyCentral did in their incident report, and it seems pretty apparent that his actions were aligned with his DevOps role in being an on-call for RubyCentral. He responsibly disclosed his access to Marty Haught.

This quote struck me in particular:

I have also noticed I am still, as of September 30, the owner of the GitHub organizations named “rubycentral” and “rubytogether”. I am unable to transfer the HelpScout or PagerDuty accounts, as you have disabled my andre(at)rubygems.org Google account.

Had RubyCentral acted responsibly, they would have (a) known all of the various administrative services being used by the team, and (b) coordinated with the existing team to ensure a seamless transfer of service before making that impossible by disabling their e-mail accounts.

RubyCentral also apparently had accused André of fraudulent access:

Ruby Central’s attorney was sending my lawyer a letter alleging I had committed a federal crime, on the theory that I had “hacked” Ruby Central’s AWS account.

I do not have confidence nor faith in RubyCentral’s leadership to be able to participate positively in this community.

9 Oct 2025

RubyCentral posts an “AWS Root Access Event” security incident post-mortem. Joel Drapper had reported previously how André Arko still had root access to the AWS instance for RubyGems, because RubyCentral failed to cover all their bases when they did their takeover.

My editorialized opinion is: had RubyCentral acted in good faith and executed this seizure of power ethically, they could have avoided this by having everyone work collaboratively.

5 Oct 2025

Most of the maintainers behind the original rubygems repository have just created gem.coop, which is a functional replacement for rubygems.org as a source entry in your Gemfile. This is built on a governance model similar to the “Homebrew” project, aided by the lead maintainer of that project.

Since RubyCentral refuses to take accountability or even acknowledge any kind of misconduct here, this feels like the absolute best path forwards. I strongly encourage any Ruby users to make this change.

9:20AM ET (26 Sep 2025)

Based on the summary fact-checking followup by Joel Drapper (see link below), specifically:

HSBT broke with the existing pact when he added Marty Haught as an owner. As a result, Ruby Central, at the request of Shopify, seized the RubyGems open source properties from their maintainers.

and

I have read the merger document signed by Evan Phoenix on 16 August 2021 and André Arko on 20 August 2021. Bundler and RubyGems never belonged to Ruby Together and they were not transferred to Ruby Central as part of the merger.

The transgressions appear to be:

Github user @HSBT was a rogue actor who violated community trust through a forcible change in ownership of a community resource
The maintainers were removed unjustly and as a rogue act
RubyCentral does not have the de jure authority to justify their de facto possession of this community resource.

The correcting actions seem to be:

Explicitly extend an offer to restore their positions as Repo maintainers to: André Arko, Colby Swandale, David Rodríguez, Ellen Dash, Josef Šimánek, Martin Emde and Samuel Giddins.
Removal or reduction of administrative privileges for HSBT, since this user has shown they cannot be trusted to act in accordance with the community
Reduction of Marty Haught’s Organizational role from Administrator to Member, and setting his privileges to one where he is able to actively contribute to the repository (acting as an advocate for Ruby Central) but unable to act on any hostile demands by RubyCentral or Shopify.

These items need to be satisfied to reset things to even. To repair some of the harm that has been done to community trust, I think these things should also happen so that we can heal and move forwards:

An apology from Shan Cureton (on behalf of RubyCentral) explaining in detail that she understands how and why these actions were inappropriate within the context of this open source community.
An acknowledgement by Shan Cureton that she understands that the assertions made in the press release Ruby Central released were in fact untrue and misguided. (See Joel Drapper’s fact checking post)
An apology from HSBT to the maintainers that were removed and to the community as a whole.

12:46PM ET (19 Sep 2025)

Still not a whole lot of clarity about the situation, but additional links added below.

The people I’ve spoken with that know many of the technical folks involved (Marty et al.) have vouched for the character of those technical people. Rhiannon Payne’s commentary suggests this is a leadership issue. Based on the rather tonedeaf and apparent doublespeak of the official word of Ruby Central, that seems to track pretty well.

There is an ongoing discussion about governance initiated by Martin Emde that is being actively mediated by Mike McQuaid (from Homebrew). However,

The arms-length appearance of this is that it was a perhaps well-intentioned but horribly-executed maneuver. The security / safety of RubyGems as a service seems presently intact. Since the last update by Marty Haught, Emde appears to be dissatisfied with the state of things and is, perhaps, also withdrawing in solidarity with Ellen Dash? This is unclear as there has not been a formal statement outside of that update.

10:15AM ET (19 Sep 2025)

Added a link to the open letter from RubyCentral. They are hosting an open Q&A Zoom on 23 September🔀.

9:59AM ET (19 Sep 2025)

Will update with more info as I learn more. But here’s everything I could find so far.

Officially / Openly Stated withdrawal from RubyGems / RubyCentral

Ellen Dash (RubyGems)
André Arko (RubyGems)
Martin Emde(?) (RubyCentral)
Rhiannon Payne (RubyCentral)
Sam Giddins (RubyCentral)
Josef Šimánek (RubyCentral)

Discussion and relevant links happening presently

Ellen Dash (@puppy / @duckinator)’s initial open letter

## Ruby Central’s Attack on RubyGems

Hi! I’m Ellen, but you probably know me as duckinator or puppy.

I really wish I didn’t have to write this, but I feel the Ruby community needs to know it.

I have been part of the Ruby community since I was 13, and one of the RubyGems maintainers for the last decade.

This community has helped me through very hard times, and you mean the world to me.

One of the most important lessons I learned from y’all is this:

> A person’s character is determined not only by their actions,

> but also the actions they stay silent while witnessing.

## This Month Has Been A Fuck Of A Year

This is what unfolded between September 9 2025 and September 19 2025, as I understand it.

On September 9th, with no warning or communication, a RubyGems maintainer unilaterally:

renamed the “RubyGems” GitHub enterprise to “Ruby Central”, added non-maintainer Marty Haught of Ruby Central, and removed every other maintainer of the RubyGems project.

He refused to revert these changes, saying he would need permission from Marty to do so.

On September 15th, this maintainer said he restored the previous permissions after talking with Marty. Marty stated the deletion was a “mistake” and “should never have happened”.

The “restoration” kept a notable change: Marty was now an owner of the GitHub enterprise.

The RubyGems team responded by immediately began putting in place an overdue official governance policy, inspired by Homebrew’s.

On September 18th, with no explanation, Marty Haught revoked GitHub organization membership for all admins on the RubyGems, Bundler, and RubyGems.org maintainer teams.

By doing this, he took control for himself and other full-time employees of Ruby Central.

Later that day, after refusing to restore GitHub permissions, Ruby Central further revoked access to the bundler and rubygems-update gems on RubyGems.org

I will not mince words here: This was a hostile takeover.

## My Stance On This

I consider Ruby Central’s behavior a threat to the Ruby community as a whole.

The forceful removal of those who maintained RubyGems and Bundler for over a decade is inherently a hostile action. Ruby Central crossed a line by doing this.

When called out, these changes were mostly reverted. Then, it was done again.

By crossing that line a second time after being called out for it, Ruby Central has made it extremely clear to me that they are not engaging in good faith.

Ruby Central’s behavior has forced my hand. I refuse to watch this without speaking up.

I am resigning from my position at Ruby Central, effective immediately.

To remove any doubt: Ruby Central unilaterally, with no explanation, revoked all access to RubyGems against both my wishes and the wishes of the entire RubyGems team.

Ellen Dash (@duckinator)

Adding accessible Dark Mode to your site with prefers-color-scheme and CSS variables

2025-03-28T00:00:00+00:00

Dark mode is about accessibility, not aesthetics

OK, it does have a sleek look to it, also. But this is an accessibility issue, primarily.

My eyes are pretty light sensitive. In most of the photos of me as a child, if it was taken outdoors, I was probably squinting. I can see pretty well in the dark, to where I don’t tend to turn lights on unless I need to read something.

Needless to say, when I see a website that is a sea of white with black text on it, especially if the text is small, this is often my experience:

Can’t I just turn the brightness of my monitor down? Sure, but that indirectly also reduces contrast, which makes it harder to read. This can be particularly challenging when the font is also small. This isn’t just inconvenient, it has caused physical pain (like when you look directly at a bright light), and has also induced headaches.

To be absolutely clear, I am not saying that the whole Internet needs to be made dark-mode. I am saying that the whole Internet needs to be adaptive to dark mode.

Most modern devices, including desktops, are capable of setting a color-scheme preference for light or dark mode.

Users think about dark mode at the operating-system level, not at the individual application level. Once they’ve set their device to dark mode in the system settings, they generally assume that all applications and websites will automatically display in dark mode — that is, they basically expect to see a lot more black than white. They do not expect to have to enable dark mode within individual applications. (via NNG)

Thankfully, there is a fairly easy way to implement this, using CSS.

Prefers Color Scheme

The prefers-color-scheme media query is fully supported in all major browsers, presently. It is a CSS media query that works like this:

@media (prefers-color-scheme: dark) {
  /* styles to use if the user is in dark mode */
}
@media (prefers-color-scheme: light) {
  /* styles to use if the user is in light mode */
}

That’s it! You don’t have to add anything to your document templates, no javascript or anything else. This just works(*).

Just kidding, there’s a little more to do. Thankfully, this implementation is also still pretty smooth.

(* ok, so if your particular site does styling in your javascript, for example, or does not use traditional stylesheets for styling, it may be more complicated for you. But this is another reason why you should use conventional methods!)

CSS Variables for light and dark modes

What we want to do here is kind of like polymorphism, but in CSS. Modern CSS allows us to define CSS variables that we can then reuse. This does not require SASS/SCSS anymore, and is part of the CSS3 standard syntax.

They look are defined like this:

:root {
  --black: #000;
}

and are used like this:

body {
  background-color: var(--black);
}

We have to use the :root pseudo-selector because the variable definitions are technically style properties (custom properties) and need to be contained within a selector. Using :root makes them globally avaiable in your document.

The Labors

We’re going to approach this in a few phases. It shouldn’t take “very long”, but honestly this depends a lot on the complexity of your site and how you organized your CSS. I am a long-time SCSS user who uses variable names for most styles, so my CSS is organized very modularly.

Gather all the colors you use
Define them as CSS variables
Define CSS variables for light mode
Replace all colors with your new variables
Define a dark-mode variant for those purposes

That’s it!

Part 1: Gathering your colors

OK so this can be a little time-consuming. The first thing you have to do is collect all of the colors you use. You can do this pretty quickly with a RegEx and RipGrep(*):

$ rg -o "#[a-fA-F0-9]{3,6}" _scss/ | uniq

_scss/base/_utilities.scss:#000
_scss/base/_utilities.scss:#666
_scss/base/_utilities.scss:#999
_scss/base/_utilities.scss:#333
_scss/base/_reset.scss:#ff0
_scss/base/_reset.scss:#000
_scss/base/_variables.scss:#e0e1e6
_scss/base/_variables.scss:#f8f3d2
// ...and so on...

Part 2: Define your CSS color variables

From this, you can copy and paste it directly into your CSS and trim it down to something that looks more like this:

:root {
  --color-black: #000;
  --color-gray: #666;
  --color-gray-light: #999;
  --color-gray-dark: #333;
  --color-yellow: #ff0;
  --color-silver: #e0e1e6;
  --color-cream: #f8f3d2;
  --color-dodger-blue: #1E90FF;
}

Note: if you are using darken() or lighten() (SCSS functions), or if you were defining your values using rgb() or rgba() or hsl() or hsla() you will need to search your CSS for those separately. I’ll leave that as an exercise to the reader.

For right now, the goal is to name the colors. If you need inspiration or are bad with color names, you can check the extended colors list or color names list. Or just make up your own.

Don’t use “primary-color”, “background”, etc. right now. That’s next. Just name the colors.

Part 3: Defining default usage (light mode)

We’re going to treat light mode as the default case here. The goal here is to define your purposeful uses for these colors.

Here’s a boilerplate example:

:root {
  /* color definitions, from above */

  /* Currently, the media query is commented out because we don't want to enable it quite yet */
  /* @media (prefers-color-scheme: light) { */
  --background-color: var(--color-silver);
  --text-color:       var(--color-black);
  --link-color:       var(--color-dodger-blue);
  --link-hover-color: var(--color-yellow);
  /* } */
}

Consider this a living list – you will likely need to add additional purposeful colors here.

Part 4: Replace your color variables with your purposeful variables

Re-using our ripgrep query from earlier:

$ rg "#[a-fA-F0-9]{3,6}" _scss/

_scss/base/_utilities.scss
   3px  3px 0 #000,
  -1px -1px 0 #000,
   1px -1px 0 #000,
  -1px  1px 0 #000,
   1px  1px 0 #000;
   3px  3px 0 #666,
   2px 3px 0 #999,
   5px 5px 0 #333;

_scss/base/_reset.scss
 background: #ff0;
 color: #000;
 border: 1px solid #1E90FF;

This gives you the filenames and line numbers of all of your colors. The job is now this process:

Open the file and find the line number indicated by the report
Find the corresponding CSS variable from the “purposeful” list you made in Part 3.
If there is no good variable already defined, then consider the usage define a new one
Repeat

For example, you’ll have something like this:

body {
  color: white;
  background-color: black;
}

and change it to:

body {
  color: var(--text-color);
  background-color: var(--background-color);
}

When you’re done, you should not see explicit color codes in the scan of your CSS. It should look something like this:

$ rg "#[a-fA-F0-9]{3,6}" _scss/
_scss/base/_variables.scss
19:    --border-color:      #e0e1e6;
20:    --smoke-color:       #fafafa;
26:    --bg-color:          #333;
29:    --border-color:      #e0e1e6;

$

Check your site in your development environment and make sure it looks the same, and that it loads correctly. If you’re using a CSS preprocessor that checks your syntax, it may give you helpful feedback if you have any errors.

Fix any errors, then continue.

Part 5: Define your dark mode variant

:root {
  /* color definitions, from above */

  /* OK, let's enable it! */
  @media (prefers-color-scheme: light) {
    --background-color: var(--color-silver);
    --text-color:       var(--color-black);
    --link-color:       var(--color-dodger-blue);
    --link-hover-color: var(--color-yellow);
  }
  @media (prefers-color-scheme: dark) {
    --background-color: var(--color-black);
    --text-color:       var(--color-silver);
    --link-color:       var(--color-yellow);
    --link-hover-color: var(--color-dodger-blue);
  }
}

This is the thing that does the thing.

So the way this works for a user that has set their device-level setting to Light Mode, is this:

Given this HTML:

<body>
  <p>Lorem ipsum dolor....</p>
</body>

The CSS that would style it:

body {
  color: var(--text-color);
  background-color: var(--background-color);
}

will consult the definitions for those variables:

@media (prefers-color-scheme: light) {
  --text-color: var(--color-black);
  --background-color: var(--color-white);
}

And you would see black text on a white foreground.

But if the user has their device-level settings set to Dark Mode, then it would instead use the other block:

@media (prefers-color-scheme: dark) {
  --text-color: var(--color-white);
  --background-color: var(--color-black);
}

They’ll see white text on a black background instead.

Success!

Final thoughts

If you want it to be toggleable, you can add that separately. But the point of this is to transparently adapt the display of your site to align with the preference of the user. The whole point of ubiquitous accessibility is that it fades into the background. You don’t have to make a big fanfare about offering dark mode, and ideally, your users realize, sometime after using the site “oh wait, it looks like they added dark mode!”

Using prefers-color-scheme with CSS variables is a compromise for maintainablility of development with providing support for users that have opted into having lower luminosity.

–

Minesweeper

2024-06-14T00:00:00+00:00

The Problem: Minesweeper

This exercise, one of the ones designated “Medium”, is called “Minesweeper”. It’s an exercise to populate a minesweeper grid with the number values for the eponymous computer game.

Your task is to add the mine counts to empty squares in a completed Minesweeper board. The board itself is a rectangle composed of squares that are either empty (‘ ‘) or a mine (‘*’). For each empty square, count the number of mines adjacent to it (horizontally, vertically, diagonally). If the empty square has no adjacent mines, leave it empty. Otherwise replace it with the adjacent mines count.

https://exercism.org/tracks/ruby/exercises/minesweeper/solutions/armahillo

Pass 1: Scaffolding

I usually like to start with creating a separate class, to keep the “execution flow” separate from the data modeling. From some preliminary test runs, I knew that the inputs would get passed in as a string and get passed out as a string. But I also knew that in order to leverage a lot of the Ruby enumerable methods, I would need to convert to Arrays. Since we care about setting a number, we’re going to initialize any blank space to a 0, and then reverse that if it’s still 0.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
class Minefield
  attr_accessor :grid

  def initialize(input)
    @grid = input.map! { |i| i.chars.map! { |c| c == ' ' ? 0 : c  } }
  end

  def to_s
    @grid.map { |i| i.map(&:to_s).join.gsub('0', ' ') }
  end
end

class Minesweeper # this is the default class the tests expect
  def self.annotate(input)
    field = Minefield.new(input)


    field.to_s
  end
end

Due to some junk inputs, there was a little bit of robustness that had to be added to the initialize and to_s methods:

# In
@grid = input.map! { |i| i == '' ? '' : i.chars.map! { |c| c == ' ' ? 0 : c  } }

# Out
@grid.map { |i| i.is_a?(Array) ? i.map(&:to_s).join.gsub('0', ' ') : i || '' }

Once I had this, it gave me an abstraction I could work with, and also get back out of.

Pass 2: Finding neighbors

One of the early starts I had involved trying to find each empty square and then calculating how many mines were around it, but for most boards that would be more iterations since there are generally more empty spaces than mines.

Instead, what I decided to do was find each mine and then consider all of its neighbors. A “neighbor” is any adjacent square (orthogonal, diagonal), like this (the 💣 is the mine):

🔘🔘🔘
🔘💣🔘
🔘🔘🔘

The cool thing here is that any neighbor is adjacent, meaning it’s only 1 unit away on either row or column, so we can define any given neighbor set as:

[row-1, col-1], [row-1, col], [row-1, col+1]
[row,   col-1],      💣       [row,   col+1]
[row+1, col-1], [row+1, col], [row+1, col+1]

If we iterate through the field as a 2D array (rows, columns), we then can consider each individual cell (a ‘*’ is a mine, here):

1
2
3
4
5
6
7
8
9
10
11
12
13
class Minesweeper # this is the default class the tests expect
  def self.annotate(input)
    field = Minefield.new(input)

    field.grid.each.with_index do |row, row_index|
      row.each.with_index do |col, col_index|
        field.update_neighbors(row_index, col_index) if col == '*'
      end
    end

    field.to_s
  end
end

But what about when we’re looking at cells on the edge? Well, we know that we don’t want any negative values for either. We can also ensure that we’re not using values that are equal or greater than the number of rows / cols. I called the method that determines this “viable_neighbors”.

1
2
3
4
5
6
7
8
9
10
11
def viable_neighbors(row, col)
  [
    [row-1, col-1], [row-1, col], [row-1, col+1],
    [row,   col-1],               [row,   col+1],
    [row+1, col-1], [row+1, col], [row+1, col+1]
  ].reject { |neighbor|
    neighbor.any?(&:negative?) ||
    neighbor[0] >= @grid.size ||      # @grid.size is the number of rows
    neighbor[1] >= @grid.first.size   # @grid.first.size is the number of columns
  }
end

This then returns an array of tuples that we can increment:

1
2
3
4
5
def update_neighbors(row = 0, col = 0)
  viable_neighbors(row, col).each do |neighbor|
    @grid[neighbor[0]][neighbor[1]] += 1
  end
end

Pass 3: Implementing Enumerable

This is perhaps some syntactic sugar, but I really wanted to have a simpler API where I can just use :each instead of a nested loop. So I included Enumerable, and then added this method to Minefield:

1
2
3
4
5
6
7
def each
  @grid.each.with_index do |cells, row|
    cells.each.with_index do |cell, col|
      yield(cell, row, col)
    end
  end
end

This then let me change the .annotate method to:

1
2
3
4
5
6
7
8
9
def self.annotate(input)
  @field = Minefield.new(input)

  @field.each do |cell, row, col|
    @field.update_neighbors(row, col) if cell == '*'
  end

  @field.to_s
end

Because of a bad-input gotcha, had to add this inside the outer loop of :each:

@grid.each.with_index do |cells, row|
  next unless cells.is_a?(Array)

At this point, the tests passed.

Pass 4: Polish

At this point I made two final changes:

class Minefield
  MINE_CHAR = '*'

The “mine” character is defined as a constant. Also:

neighbor.any?(&:negative?) || !@grid.dig(*neighbor).is_a?(Integer)

Array gives the :dig method, which lets you specify a series of args and it will attempt to safely find its way into that element. All we care about is whether or not the field is an integer already. If the attempted cell is out of bounds, we just ignore it.

A gotcha here is that we have to keep the “negative” check because technically you can access negative indices in an array, it just wraps around in reverse; but not in the other direction.

irb(main):001> array = [1,2,3]
=> [1, 2, 3]
irb(main):002> array[2]
=> 3
irb(main):003> array[-1]
=> 3

Tests all passed. I added a couple comments to clarify, and submitted!

Integration with Patreon subscription tier via OAuth

2023-07-31T00:00:00+00:00

Overview

I have a side-gig doing some tech stuff for a TTRPG. This TTRPG is backed partly by Patreon subscriptions. Patreon is a service that allows content creators to receive regular monthly “subscriptions” from their fans. It provides an API (GraphQL) that allows a consumer to receive information about the platform as well as functioning as an OAuth provider.

We intended to allow users to sign-in via Patreon, use the membership information of their Patreon account to gatekeep access to certain features (persisting data to the database, early access to content and new features, etc.)

This article details the specific platform-agnostic process for how to retrieve a dataset that will allow you to verify membership for a patreon tier, as well as discussion about how the data nodes fit together.

The last section provides some sample Ruby classes that I wrote that demonstrate one approach to implementing a response handler for this API response.

Part I: the data nodes

Definitions

member

A “member” is a Patreon user. In this context, it is a Patreon user who has decided to subscribe, for any amount of money including nothing at all, to a content creator.

campaign

A “campaign” is the other end of the membership: it equates to the content creator. I don’t know if it’s possible for a content creator to have multiple campaigns or not, but for this article it doesn’t really matter.

tier

A “tier” is the level of the membership. It has general traits (a name, a subscription fee, etc). It does not represent the individual user memberships, though.

membership

This is the record that connect “member” to “tier”. This has almost all of the data from the tier (and more) but does not have descriptive data like the tier name.

Raw response data

This has been sanitized and also truncated to only show the relevant data nodes we care about when parsing user identity data. Note that the included node has three objects under it: “member”, “campaign”, and “tier”. Locations where there would be additional chaff have been noted with { ... }:

@data= {
  "data" => {
    "attributes" => {
      "email" => "alice@babbage.com", 
      "first_name"=>"Alice", 
      "is_email_verified"=>true, 
      "last_name"=>"Babbage", 
      "vanity"=>"alice_babbage"
    }, 
    "id"=>"1234567", 
    "relationships" => {
      "memberships"=> {
        "data"=> [
          {
            "id"=>"ab1c23de-f45a-6b78-90c1-2d3ef4567890", 
            "type"=>"member"
          }
        ]
      }
    }, 
    "type"=>"user"
  }, 
  "included"=> [
    { ... }
    {
      "attributes"=>{
        "pledge_url"=>"/join/stillfleet", 
        "summary"=>"<< html description >>", 
        "url"=>"https://www.patreon.com/stillfleet", 
        "vanity"=>"stillfleet"
      }, 
      "id"=>"1234567",
      "type"=>"campaign"
    }, {
      ...
    }, { # Stillfleet "member" node
      "attributes" => {
        "campaign_lifetime_support_cents" => 69420,
        "currently_entitled_amount_cents" => 900,
        "is_follower" => false,
        "last_charge_date" => "2023-07-01T12:00:00.000+00:00",
        "last_charge_status" => "Paid",
        "lifetime_support_cents" => 69420,
        "next_charge_date" => "2023-08-01T12:00:00.000+00:00",
        "patron_status" => "active_patron",
        "pledge_cadence" => 1,
        "pledge_relationship_start" => "2020-01-01T00:00:00.000+00:00",
        "will_pay_amount_cents" => 900
      },
      "id" => "ab1c23de-f45a-6b78-90c1-2d3ef4567890",
      "relationships" => {
        "campaign" => {
          "data" => {
            "id" => "1234567", # Stillfleet campaign ID
            "type" => "campaign"
          },
          "links" => {
            "related" => "https://www.patreon.com/api/oauth2/v2/campaigns/1234567"
          }
        },
        "currently_entitled_tiers" => {
          "data" => [
            {
              "id" => "3456789", # Stillfleet Tier ID (Sr. Archivist)
              "type" => "tier"
            }
          ]
        }
      },
      "type" => "member"
    }, {
      ...
    }, {
      "attributes"=>{
        "amount_cents"=>900, 
        "published"=>true, 
        "remaining"=>nil, 
        "title"=>"Senior Archivist", 
        "unpublished_at"=>nil, 
        "url"=>"/join/stillfleet/checkout?rid=3456789", 
        "user_limit"=>nil
      }, 
      "id"=>"3456789", 
      "type"=>"tier"
    },
    { ... }
  ], 
  "links"=>{
    "self"=>"https://www.patreon.com/api/oauth2/v2/user/1234567" # from above
  }
}

Part 2: Parsing the response data

This object is a wrapper that does some data preparation after consuming the raw patreon response data. The source is a graphQL query.

This is the shape of a sample Patreon::Response object that has parsed raw response_data.

Summary of Important values

Type	Location inside `includes[]`	Meaning
campaign	`{ node['id']`	The campaign ID for this campaign (1234567)
member	`{ node['relationships']['currently_entitled_tiers'][*]['id']`	The Stillfleet campaign ID
member	`node['attributes']['patron_status']`	whether or not they are an `active_patron`
tier	`node['attributes']['title']`	Title of their tier

Overview of response parsing

Patreon uses a graphQL API. We query against the “identity” endpoint, and have it include the “member”, “campaign” and “tier” data. The main “data” node includes identifying data about the user themselves. The “includes” data node includes all of the Stillfleet data.

The unique identifier of the user is the membership UUID:

{
  "data" => {
    "attributes" => { ... },
    "id" => "XXXXXXXX", # This is a red herring!
    "relationships" => {
      "memberships" => {
        "data" => [
          { ... } # There will be others that are similar
          {
            "id" => "ab1c23de-f45a-6b78-90c1-2d3ef4567890", # UUID
            "type" => "member"
          }
          { ... }
        ]
      }
    },
    type => "user"
  }
  "included" => [ ... ]
}

This UUID is then reused in the “member” node.

⚠️ N.B. As there is no identifying information within this node to indicate which campaign is associated with this, you have to look to the included data first. A single user response will very likely have many nodes under "included", "campaign", "tier" and "member" nodes among them, from various campaigns the user may be a part of.

To find which UUID we need, we have to work backwards.

Starting with the "campaign" nodes:

{
  "data" => { ... },
  "included" => [
    { ... }
    { # Stillfleet "campaign" node
      "attributes" => {
        # ...
        "vanity" => "stillfleet"
      },
      "id" => "1234567",
      "type" => "campaign"
    },
    { ... }
  ]
} 

There is one node in included[] where node has: node['type'] == "campaign", and also node['attributes']['vanity'] == "stillfleet". When we find this node, we extract the id attribute: node['id'] and save that away and never forget it.

Stillfleet Campaign ID: 1234567

This id is thankfully universal for all memberships to this campaign, specifically.

With that membership, we can now query for the "member" node with details of the subscription tier.

{
  "data" => { ... },
  "included" => [
    { ... }
    { # Stillfleet "member" node
      "attributes" => {
        # ...
        "patron_status" => "active_patron", # <-- IMPORTANT
        # ...
      },
      "id" => "a159f0de-7689-4b8b-ae89-6b59562589e6", # membership UUID
      "relationships" => {
        "campaign" => {
          "data" => {
            "id" => "1234567", # Stillfleet campaign ID
            "type" => "campaign"
          },
          "links" => { ... }
        },
        "currently_entitled_tiers" => {
          "data" => [
            { ... } # there should not be other tiers here, but it's still a collection
            {
              "id" => "3456789", # Subscription tier ID (Sr. Archivist)
              "type" => "tier"
            }
            { ... }
          ]
        }
      },
      "type" => "member"
    },
    { ... }
  ]
} 

Again, we are looking in a sea of objects that include "member", "tier", and "campaign" nodes, and the only way to tell them apart is by looking into them. This time, we want to look through the includes[] collection for a node that has: node['type'] == "member" and node['relationships']['campaign']['data']['id'] == "1234567".

With that node, we can now look through node['relationships']['currently_entitled_tiers'] and gather all the currently_entitled_tier['id']s. There should only be the one, but good to verify. There is a different tier ID for each subscription level, so this is less globally useful, and should any tiers be changed we would need to know that beforehand, so we don’t store this permanently, only transiently.

Stillfleet subscription tier ID: 3456789

Additionally, while examining this node we should also note the node['attributes']['patron_status'] value – it should be "active_patron" if they are current.

Now we can gather the actual tier level by looking at the "type" => "tier" nodes:

{
  "data" => { ... },
   "included" => [
      { ... }
      { # Stillfleet "tier" node
      "attributes" => {
        "amount_cents" => 900,
        # ...
        "title" => "Senior Archivist",
        # ...
      },
      "id" => "3456789",
      "type" => "tier"
    },
    { ... }
  ]
} 

This time we are looking for a node that has "type" => "tier" and also has node['id'] == "3456789" (the subscription tier ID from earlier). From this we can extract both the node['attributes']['amount_cents'] (if that matters to us) and also the node['attributes']['title'] which gives us the diegetic name of the tier.

This this data collected, we can now assemble this into a meaningful determination of whether or not the user is (a) a subscribing patron, and (b) actively supporting.

Part 3: Storing & using the data

The last phase of this is to take the data we are

Persisting in the Patreon::Response parsed object

I created a Response class under a Patreon module to persist the data for future requests.

These are the fields I stored:

@status= :active_patron,

@user_data    = {
  "patreon_id"        =>  "1234567",
  "email"             =>  "alice@babbage.com", # Not all users have emails provided!
  "first_name"        =>  "Alice", 
  "is_email_verified" =>  true, 
  "last_name"         =>  "Babbage", 
  "vanity"            =>  "alice_babbage"
},

@campaign_id  = "1234567",

@pledge_data  = {
  "campaign_lifetime_support_cents" =>  10000, 
  "currently_entitled_amount_cents" =>  900, 
  "is_follower"                     =>  false, 
  "last_charge_date"                =>  "2023-07-01T12:00:00.000+00:00", 
  "last_charge_status"              =>  "Paid", 
  "lifetime_support_cents"          =>  69420, 
  "next_charge_date"                =>  "2023-08-01T12:00:00.000+00:00", 
  "patron_status"                   =>  "active_patron", 
  "pledge_cadence"                  =>  1, 
  "pledge_relationship_start"       =>  "2020-01-01T12:00:00.000+00:00", 
  "will_pay_amount_cents"           =>  900
},

@subscription_tier = {
  "amount_cents"      =>  900, 
  "published"         =>  true, 
  "remaining"         =>  nil, 
  "title"             =>  "Senior Archivist", 
  "unpublished_at"    =>  nil, 
  "url"               =>  "/join/stillfleet/checkout?rid=3456789", 
  "user_limit"        =>  nil
}>

Sample `Patreon::Response` class definition

Here is the class used that provides the fields above. It’s written in Ruby and is mostly lifted verbatim from what we’re using. There are some small modifications made for readability.

module Patreon
  # These custom exceptions are for slightly better granular
  # feedback on failures. There are many different ways to run
  # into issues during this processing phase, and some of them
  # are handled differently, or with different error messages.

  class Patreon::NotAValidResponseError < StandardError; end
  class Patreon::NotAPatronError < StandardError; end
  class Patreon::NotAnActivePatronError < StandardError; end
  class Patreon::DataParsingError < StandardError; end


  class Response
    attr_reader :data, :campaign_id, :user_data, :pledge_data, :subscription_tier, :status

    # @param data - a raw Patreon API response object
    def initialize(data)
      @status = :unknown
      @user_data = {}
      @pledge_data = {}
      @subscription_tier = {}

      # This will raise a JSON::ParserError if it is not JSON data
      @data = data.is_a?(Hash) ? data : JSON.parse(data)

      # The strategy here is to bail out of execution if we encounter
      # any showstopping errors. Some of these exceptions are handled
      # upstream.
      raise Patreon::NotAValidResponseErrorunless has_user_data?
      find_user

      raise Patreon::NotAPatronError unless has_historical_subscription_data?
      find_pledge_data

      raise Patreon::NotAnActivePatronError unless has_current_subscription_data?
      find_subscription_tier
      
      # This status is set implicitly if we have not encountered any
      # showstopping errors so far.
      @status = :active_patron  
    
    # These statuses cover those other circumstances. In the real
    # code in prod, there is some additional debug dumping happening
    # within these blocks as well, but the only required part is 
    # that the statuses are set correctly.
    rescue Patreon::NotAPatronError => _
      @status = :not_a_patron
    rescue Patreon::NotAnActivePatronError => _
      @status = :not_an_active_patron
    end

    # Technically, this is a known value but we're memoizing it 
    # dynamically as an early smoke test.
    # the approach of "rescue StandardError, re-raise as a more
    # specific error" is a strategy used throughout this class.
    def campaign_id
      @campaign_id ||= campaign['id']
    rescue StandardError => e
      raise Patreon::NotAPatronError, e
    end

    private

    def has_user_data?
      data.dig('data', 'id').present?
    end

    # In these two has_*_data? methods, we're using `rescue false`
    # because both helper methods that are called previously _may_
    # raise exceptions, depending on their behavior or the status of
    # the response. As this is a predicate, we always want to respond
    # with a true or false value.
    def has_historical_subscription_data?
      historical_subscription_data.present?
    rescue
      false
    end

    def has_current_subscription_data?
      current_subscription_data.present?
    rescue
      false
    end

    # The find_* methods all encapsulate the response object
    # traversal. Initially, I considered memoizing these methods
    # instead, but that approach didn't fit the flow above.
    def find_user
      # This might read awkwardly as two steps, here. The reason
      # it's broken up is because there was a step between them
      # that I did for debugging reasons that isn't germane to this
      # article.
      @user_data = { 'patreon_id' => data['data']['id'] }
      @user_data.merge!(data.dig('data','attributes'))
    rescue StandardError => e
      raise Patreon::DataParsingError, e
    end

    def find_pledge_data
      @pledge_data = membership['attributes']
    rescue StandardError => e
      raise Patreon::DataParsingError, e
    end

    # Because of how ruby behaves, the "current" data will be 
    # applied on-top-of (ie. it will overwrite) any colliding
    # keys.
    def find_subscription_tier
      @subscription_tier = historical_subscription_data.merge(current_subscription_data)
    rescue StandardError => e
      raise Patreon::DataParsingError, e
    end

    def historical_subscription_data
      data['included'].find do |node|
        node['type'] == 'member' &&
          node.dig('relationships', 'campaign', 'data', 'id') == campaign_id
      end['attributes']
    end

    def current_subscription_data
      data['included'].find do |node|
        node['type'] == 'tier' &&
          node['id'] == subscription_tier_id
      end['attributes']
    rescue StandardError => e
      {}
    end

    def subscription_tier_id
      @subscription_tier_id ||= membership.dig(
                                  'relationships',
                                  'currently_entitled_tiers',
                                  'data'
                                ).first&.fetch('id')

    rescue StandardError => e
      raise Patreon::DataParsingError, e
    end

    # This method is memoized.
    def membership 
      @membership ||= data['included'].find do |node| 
        node['type'] == 'member' &&
          node.dig('relationships', 'campaign', 'data', 'id') == campaign_id
      end

    rescue StandardError => e
      raise Patreon::DataParsingError, e
    end

    def campaign
      @campaign ||= data['included'].find({}) do |node| 
        node['type'] == 'campaign' &&
          node.dig('attributes', 'vanity')&.downcase == 'stillfleet' 
      end
    rescue StandardError => e
      raise Patreon::DataParsingError, e
    end
  end
end

In OAuth response controller action

I’ll have to write a separate post about using Patreon for OAuth specifically, but here’s the important steps of the controller action that Patreon OAuth webhook hits when the user signs-in via Patreon:

class OAuthController < ApplicationController
  # ...

  def patreon_redirect
    # ... do the OAuth request via an API client
    # these first two steps are included only for context, but 
    # use bespoke code. :fetch_identity returns the JSON response.
    api_client = Patreon::PatreonClient.new(access_token)
    response_data = api_client.fetch_identity

    # this the class from above.
    response = Patreon::Response.new(response_data)
    # Do stuff with the response object -- instantiate objects, etc.

    # IRL, this is a little more complicated. Not every user puts
    # their name in the first/last name. So I actually have a method
    # that checks name, userID, email, etc and uses the best one it
    # finds.
    redirect_to
      root_path,
      notice: "Welcome, #{response.user_data['first_name']}!"
    
    rescue Patreon::DataParsingError, Patreon::NotAPatronError => e
      redirect_to
        root_path, 
        notice: "Welcome, guest! Please consider subscribing to our Patreon!"
    rescue Patreon::NotAnActivePatronError => e
      redirect_to
        root_path, 
        notice: "Welcome back, friend! Please consider subscribing to our Patreon again!"
    rescue StandardError => e
      redirect_to
        root_path,
        notice: "We had trouble signing-in."
    end
    # ...
  end

Creating a Multimedia Promotional Site

2023-04-08T00:00:00+00:00

Overview / TLDR

I have a hobby playing records at dance events. I needed a promotional site that I could direct people to which would have all my relevant information and media. The requirements were:

A page of social media links
A page of general info
A page of upcoming and past events
The ability to host large audio files (50-80MB)
The ability to play the audio files on the site, without allowing hotlinking from elsewhere
Minimal maintenance

In the past, I probably would have used something like Wordpress on a VPS for this. Both VPSes and Wordpress both require regular maintenance and updates to keep them patched to harden them to attackers. A VPS also costs money.

For this site, the content won’t change very often, and it’s read-only content so it doesn’t need a database or the ability to do request processing. Similar to this site, static files felt like the right path forward. Plus, render.com allows for free hosting of a limited number of static sites.

The plan was:

Create the site using Jekyll
Deploy the site to render.com
Host the large audio files on AWS S3
Use native HTML5 multimedia capabilities to play audio

Bootstrapping

There are literally hundreds of static-site generators. I’m not even sure if Jekyll is being actively maintained, but since it’s only used on the development side and the output is static content, I’m OK using deprecated-yet-familiar code. (If you want something actively maintained, I hear BridgetownRB is pretty good ;))

Jekyll

I use the standard jekyll setup process. I brought some configs and boilerplate over from another site to expedite the work. Nothing fancy.

For layouts, I like to create a bare layout file and then build others off of that. This ensures that my HTML head and general structure templating is universally consistent.

I considered doing Events as “Posts”, and may later decide to leverage that feature, but for right now I just want static files, so I created stub files about.md, index.html, events.md, demos.md and links.md.

Initial Redirects

For the time being, I don’t have root-level content, so I initially wanted it to always redirect to the Social Links page. In index.html:

1
2
3
4
5
6
7
8
9
10
<!doctype html>
  <head>
    <meta http-equiv="refresh" content="0;URL='events.html'" />
  </head>

  <body style="background-color: black">
    <a href="//events.html">For now, you'll be redirected here.</a>
  </body>
</html>
<% end %>

Very simple, classic meta-refresh HTTP redirect. Using this method let’s us avoid worrying about any kind of server-side request processing (where we typically do redirects now). The black background color reduces the flickering from the “flash of unstyleed content” (FOUC).

Event widgets

A month or so after those initial redirects, I added a “past events” section that shows a table of past bookings (with flyers, if possible). I also created a partial for both “upcoming” and “past” events that is rendered on the homepage and the “past events” page.

<figure class="event">
  {% if include.url %}
  <a href="{{ include.url }}" title="{{ include.title }}">
  {% endif %}
    {% if include.img %}
      <img src="{{ include.img }}" alt="event thumbnail" />
    {% else %}
      <img src="/assets/img/save-the-date.png" alt="event thumbnail" />
    {% endif %}
  {% if include.url %}
  </a>
  {% endif %}

  <figcaption>    
        <a class="event-link" href="{{ include.url }}" title="More details for {{ include.title }}">{{ include.title }}</a>
    
      <p>{{ include.text }}</p>
      <time datetime="{{ include.datetime }}">{{ include.time }}</time>
    
  </figcaption>
</figure>

and then it’s used with:

<li>{% include upcoming_event.html 
url="https://url.to/the/event"
img="/assets/img/events/YYYY-MM-DD-title.jpg"
title="The event title"
text="subtitle / location"
datetime="2023-01-01T12:00:00-04:00"
time="1 January 2023; 12p - 1pET" %}</li>

Design

I had previously designed signage and chosen a color scheme (sort of a synthwave / vaporwave vibe).

Since the sound aesthetic is Retro (we vinyl DJs aren’t very common anymore!), I wanted the visual aesthetic to reinforce that brand.

Stylesheet basis

I really, really like Tania Rascia’s Primitive UI boilerplate. It’s clean, basic, and easily extended. I’ve used it on several sites (including this one) so I have a good workflow built around it.

I modified the _variables.scss file to integrate my color choices:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
$aquamarine: #8bd4e1 !default;
$magenta: #ed5ca1 !default;
$amethyst: #73499d !default;
$rich-black: rgba(3,3,3,0.8);

$background: url('/assets/img/background.jpg') no-repeat fixed black !default;
$primary-color: $aquamarine !default;
$secondary-color: $amethyst !default;
$accent-color: $magenta !default;
$alternate-background: $rich-black !default;
$alternate-color: $magenta !default;
$link-color: $magenta !default;
$link-hover-color: darken($link-color, 15%) !default;
$highlight: $aquamarine !default;

Typography

Google offers many web fonts for free. I found one called “Permanent Marker” that had the look I wanted.

I used a standard sans-serif font for the body copy.

Imagery

This happened over two iterations, but to reinforce the “retro” aesthetic of the brand, I knew that I wanted to incorporate a record sleeve (for the navigation background) and a cassette tape / case (for the navigation on the demos page).

Record sleeve

I used my scanner and scanned an actual record sleeve liner (White), then cropped and resized it in Photoshop. I placed it inside the <header> tag like this:

1
2
<header>
  <div id="record-sleeve">

and in the SCSS with:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
header {
  #record-sleeve {
    display: flex;
    flex-direction: column;
    margin: 0;
    padding-top: 1rem;
    background: white;

    @include medium-breakpoint {
      position: fixed;
      height: 100%;
      right: -3rem;

      margin-top: 2rem;
      height: 100%; 
      transform: rotate(3deg);

      background: url('/assets/img/record-sleeve.png') no-repeat;
    }

The transform: rotate(3deg) gave it a subtle angle that helped it stand more prominently and broke up the layout’s linearity, visually.

The individual links are written in $rich-black color using the “Permanent Marker” font. I applied CSS rotation to each one using nth-child() to make them seem more organic / hand-written.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
ul {
  display: flex;
  flex-direction: row;
  list-style: none;
  margin: 20px 100px 0;
  padding: 0;
  flex-wrap: wrap;
  justify-content: space-between;
  column-gap: 1rem;

  @include medium-breakpoint {
    flex-direction: column;  
  }

  li {
    font-family: 'Permanent Marker', cursive;
    display: inline;
    font-size: 3rem;   
    
    a {
      color: $rich-black;
    }

    &.current {
      background-color: $rich-black;
      padding-left: 1rem;
      padding-right: 1rem;
      a {
        color: $primary-color;  
      }
    }

    @include medium-breakpoint {
      margin-bottom: 1rem;
      &:nth-child(odd) {
        transform: rotate(4deg);
      }

      &:nth-child(even) {
        transform: rotate(-6deg);
      }

      &.current {
        a {
          color: $primary-color;  
          text-shadow: -6px 4px 0 $rich-black,
                        6px 4px 0 $rich-black,
                        4px -4px 0 $rich-black;
        }
      }
    }
  }
}

I used flex-box to make them evenly spaced and more responsive. For “odd” items it tilts them slightly one way, and for “even” slightly the other way.

Given that I was anticipating people to get to the site via a QR code, I needed to ensure it was usable on mobile browsers. I added breakpoints to coerce the navigation to present in a stacked way at narrower breakpoints. (The record sleeve graphic is also hidden at smaller breakpoints).

Cassette case

The vision I had for the demos page was to have the list of individual demos resemble a stack of cassette tapes (back in the 90s and early 00s, we traded promos on cassette tapes). I scanned the spine of a cassette. It had a completely black background and I wanted it to have some transparency so it fit into the page more. In Photoshop, I carved the outline out and applied transparency to the background, then along the ends I applied a subtle gradient of transparency.

The SCSS for this section was initially:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
  #mixtapes {
    li.tape {
      background: url('/assets/img/cassette-case.png') no-repeat;

      width: 588px;
      height: 75px;
      
      list-style: none;
      margin-bottom: 1px;

      @include large-breakpoint {  
        &:nth-child(2n) {
          margin-left: 1rem;
        }
        &:nth-child(3n) {
          margin-left: -0.2rem;
        }
        &:nth-child(5n) {
          margin-left: 0.5rem;
        }
      }

The nth-child() lines cause the tapes to have a staggered effect and breaks up the visual linearity of the layout.

Cassette SVG

I had this idea to have a graphic of a cassette tape indicate the demo that was being played. I initially considered using a scan of a cassette, but I had these lofty ideas of some kind of cool animation, and that would be more difficult.

I found a CodePen demo of a cassette-SVG used for an audio player and lifted the SVG code from it. It seemed a bit verbose, and I wanted to tweak it slightly, so I imported it into Illustrator.

In Illustrator, I changed some of theh paths to add transparency so there was a transparent window in the center (to see the tape reels), then I placed two dark-brown circles on the bottom layer to be the tape reels, adjusted the overall size, and cropped it tight. After exporting, the resulting SVG was much more brief and cleaned up.

In the SVG code itself, I added a few classes to some of the <path> elements so that I can manipulate them with CSS. Initially I thought I could do it as a separate SVG file, but I had to inline it.

Two key parts here were the reels:

Note the “r” attribute, which is the “radius” of the circle. The class was attribute was added as a hook for Javascript to attach to later.

1
2
<circle cx="17.03" cy="18.03" r="5" style="fill: #513000;" class="left-reel" />
<circle cx="42" cy="18.03" r="5" style="fill: #513000;" class="right-reel" />

and the gears:

Here it was just the classes, left-gear and right-gear, which are used as hooks for the CSS.

1
2
3
4
5
6
7
8
9
10
11
12
<g class="left-gear">
  <path d="M15,15l4,6" style="stroke: #30a996; stroke-width: .4px;"/>
  <path d="M19,15l-4,6" style="stroke: #30a996; stroke-width: .4px;"/>
  <path d="M13,18h8" style="stroke: #30a996; stroke-width: .4px;"/>
  <circle cx="17" cy="18" r="2" style="fill: #f6dfda;"/>
</g>
<g class="right-gear">
  <path d="M44,15l-4,6" style="stroke: #30a996; stroke-width: .4px;"/>
  <path d="M40,15l4,6" style="stroke: #30a996; stroke-width: .4px;"/>
  <path d="M38,18h8" style="stroke: #30a996; stroke-width: .4px;"/>
  <circle cx="42" cy="18" r="2" style="fill: #f6dfda;"/>
</g>

The SCSS for the gears to animate theme spinning was also lifted from the CodePen, but I modified it slightly. The original code would add the animation if the .spin class was added to the gears (so hitting “play” would add the classes to the elements).

I modified it to instead spin by default, but not animate if the parent SVG has the class “pause” added to it.

I also added a desaturation effect to the cassette when it was “paused” and set the transition for that to give it a soft fade effect when activated. (The classes .plastic and .label are classes I assigned to a few of the parts of the cassette SVG so that it didn’t look flat universally when applied).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
svg#svgCassette {
      width: 100%;
      height: 300px;
      display: block;

      .left-gear, .right-gear {
        transform-origin: center;
        transform-box: fill-box;
        animation: rotate-wheel 4s infinite linear;
      }

      path {
        transition: fill 1s;
      }

      &.pause {        
        path {
          transition: fill 2s;
        }
        path.plastic {
          fill: desaturate(#30a996, 45%) !important;
        }
        path.label {
         fill: desaturate(#e3dcaa, 45%) !important; 
        }

        .left-gear, .right-gear {
          animation: none;
        }
      }

      @keyframes rotate-wheel{
        from {
          transform: rotate(0deg);
        }
        to {
          transform: rotate(-360deg);
        }
      }

Polaroid photo

On the /about page, I wanted to have a photo inlay that had a Polaroid appearance. I used the <figure> tag as a wrapper for the image:

1
2
3
4
5
6
<figure class="polaroid pull-right tilted">
  <div class="frame">
    <img src="/assets/img/highstone-bio.jpg" alt="20-something DJ playing records" class="large" />
  </div>
  <figcaption>ca. 2005, Dayton OH</figcaption>
</figure>

MDN docs say, about the <figure> tag:

The <figure> HTML element represents self-contained content, potentially with an optional caption, which is specified using the <figcaption> element. The figure, its caption, and its contents are referenced as a single unit.

and

Usually a <figure> is an image, illustration, diagram, code snippet, etc., that is referenced in the main flow of a document, but that can be moved to another part of the document or to an appendix without affecting the main flow. A caption can be associated with the <figure> element by inserting a <figcaption> inside it (as the first or the last child). The first <figcaption> element found in the figure is presented as the figure’s caption.

The caption was important to me. I could have done the image only and applied a wide solid-white border to the edges, but since I wanted to also have a caption I knew I would need to wrap it, and the <figure> tag is the most semantically-appropriate for this use-case.

The SCSS looks like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
figure.polaroid {
  margin: 2rem;
  background-color: white;
  border-right: 3px solid rgba(6,6,6,0.7);
  border-bottom: 4px solid rgba(4,4,4,0.7);

  .frame {
    border: 1.5rem #EEE solid;
    padding-bottom: 0;
    overflow: hidden;
    width: 300px;
    height: 300px;

    img {
      height: 100%;
    }   
  }

  figcaption {
    width: 300px;
    height: 3.5rem;
    background-color: #EEE;
    font-family: "Permanent Marker";
    text-align: center;
    color: $rich-black;
  }

  &.tilted {
    transform: rotate(15deg)
  }

  &.pull-right {
    float: right;
  }

}

Again, using the “Permanent Marker” font for brand consistency, and tilting it to break up the linearity and add more visual interest.

It’s right-aligned via float (At this point, using float feels almost archaic, but it’s the right thing to use in this case, for the behavior I wanted).

The photo is technically a landscape-oriented photo, and significantly larger, but it’s cropped with the overflow: hidden. It could have also been cropped in Photoshop, pre-emptively.

Hosted Media

Long ago, I used to host all my mixes on my website directly. Occasionally they would get hotlinked from elsewhere. While the traffic is nice, if they’re only consuming your audio, you’re basically a CDN and I’m not paying for that level of service.

What I wanted to do here was self-host the files so that I’m not beholden to another business’s constraints (if an individual artist / label wants to send me a DMCA takedown notice for one of theh tracks on a mix, so be it, but it’s less likely if it’s not on a big website). Also self-hosting is cheaper and avoids having to deal with other people pre-rolling ads.

AWS S3 configuration

AWS S3 storage is cheap and easy, and it allows for some more complicated configurations that prevent hotlinking. So I wrote this bucket policy for the S3 bucket.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::djhighstone/*",
            "Condition": {
                "StringLike": {
                    "aws:Referer": [
                        "http://www.djhighstone.com/*",
                        "http://djhighstone.com/*",
                        "https://www.djhighstone.com/*",
                        "https://djhighstone.com/*"
                    ]
                }
            }
        }
    ]
}

Superficially, what this does is ensure that any requests to any of my media files originates from my website domain. Obviously this isn’t foolproof and it’s still technically possible to spoof that. If someone wants to go through all that trouble to do it, well that’s just fine.

One gotcha with S3 is that it defaults to very limited permissions for any uploaded file, so after uploading a file, you have to set it’s permissions to make it publicly accessible.

Deployment

Having built this site with Jekyll, the deployment process with render is well-documented and straightforward.

Next Steps

The next thing I want to add to this is the ability to jump forward / backwards to pre-determined timestamps for each mix (so that it works better as a demo), and to add playlists that display while the mix plays.

Implementing Modular CSS in Ruby on Rails

2021-06-22T00:00:00+00:00

The Problem

CSS cascades. This is a great feature of it! But it can be tricky when you have a single monolithic CSS file and want some styles to apply to certain pages but not to other pages.

For example, let’s say we had an article summary block on the homepage of the site:

1
2
3
4
5
6
7
<% @posts.each do |post| %>
  <article>
    <h2><%= post.title %></h2>
    <p><%= post.summary %></p>
    <%= link_to post, 'Read More' %>
  </article>
<% end %>

and wanted to apply these styles to the heading:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
article {
  h2 {
    background-color: $brand-primary;
    font-size: $large;
  }
  p {
    padding: 1rem;

    & + a {
      display: block;
      margin-top: 1em;
    }
  }
}

(I threw this up on a codepen)

Cool! But maybe on the :posts resource itself we want to display multiple paragraphs at a larger width, and also use a different background color:

1
2
3
4
5
6
7
8
9
10
article {
  width: 50%;
  
  h2 {
    background-color: #654321;
  }
  p {
    padding: 0;
  }
}

(codepen)

Looks great! But uh-oh….

With those changes, we now have the :posts resource styles applying to the homepage styling as well. Context collapse!

(codepen)

Of course, this specific situation can be resolved by adding an article class (perhaps <article class="summary"> for the homepage styles and <article class="full-text"> for the full view version). This situation is also oversimplified, to illustrate the context collapse. What happens when you start adding in !important styles, or want to do a style override that can’t be solved easily by changing load order or merely applying a class?

What follows is a solution I developed a few years back and it will:

Allow you full modular control over all your CSS in an intuitive way
Be more or less future-proof (when you add a new resource, the SCSS file created for that resource is what you’ll use)

Bonus: you can technically use this to address similar modularization of JS snippets for simple JS scripts (this is a bit harder to integrate with more complicated JS frameworks).

Modularity Hooks

To gain modularity in our CSS we need to give our application hooks that are contextual to the controller’s action that we’re viewing.

We begin with these assumptions:

Every HTML page has a <body> element
The <body> element has both a class and an id attribute (and we can modify those)
The Rails app uses the vanilla resource-oriented approach (see below for clarification)
The Rails stack will provide us with some params set at load-time, indicating which Controller and Action are being used for this request.

The reductive TL;DR is: We set the body#id to the Controller and the body.class to the Action.

I think using the body tag makes the most sense, semantically, because CSS (practically) only really applies to the content, and the body is the root of the content in the document. You could, in theory, use the <html> tag itself, though I personlly think that makes less semantic sense for this.

In the code, the trivial / naive approach, that we will harden up a little later, is like this:

1
2
3
<!-- app/views/layouts/application.html.erb -->
<body id="<%= params[:controller] %>" 
   class="<%= params[:action] %>">

And then in your SCSS, you can now create controller- and even action-specific styles:

1
2
3
4
5
6
7
8
9
10
11
12
13
<!-- app/assets/stylesheets/users.scss -->
body#users {
  &.new {
    header#banner {
      background-image: url('new-user.jpg');
    }
  }
  &.edit {
    header#banner {
      background-image: url('edit-user.jpg');
    } 
  }
}

And those styles won’t conflict with other controller actions:

1
2
3
4
5
6
7
8
9
10
11
12
13
<!-- app/assets/stylesheets/projects.scss -->
body#projects {
  &.new {
    header#banner {
      background-image: url('new-project.jpg');
    }
  }
  &.edit {
    header#banner {
      background-image: url('edit-project.jpg');
    } 
  }
}

Boom.

For very simple Rails applications that only have shallow top-level resources, this is it; you’re done.

Dealing with Namespaces

Sometimes you want to do namespaces, like an /admin/* namespace. Specifically, I’m talking about situations where you have something like app/controllers/admin/*.rb, the routes have namespace :admin do # ..., and the controllers within that each are like Admin::SomeController:

In these cases, the params[:controller] will give us: admin/users, which doesn’t evaluate so cleanly in SCSS. So let sanitize that using :gsub:

1
2
3
<!-- app/views/layouts/application.html.erb -->
<body id="<%= params[:controller].gsub(/[^\w\d]+/,'--') %>" 
   class="<%= params[:action] %>">

And now in the SCSS, we can do things like:

1
2
3
4
5
6
7
8
9
<!-- app/assets/stylesheets/admin/users.scss -->
body#admin--users {
  &.edit {
    header#banner {
      background-color: orange;
      background-image: url('new-user.jpg');
    }
  }
}

Security Considerations

Since we’re dumping the params[:action] directly into the source, we could pre-emptively apply a similar sanitization to it just to be safe:

1
2
3
<!-- app/views/layouts/application.html.erb -->
<body id="<%= params[:controller].gsub(/[^\w]+/,'--') %>" 
   class="<%= params[:action].gsub(/[^\w]+/,'') %>">

Generally, you don’t want to reveal more about your surface than you need to. Exposing the name of the controller resource and the name of the action might feel like a security risk. Anyone interested in attacking your rails application can probably ascertain your controller names by looking at other resources, so the marginal exposure from using these in the HTML source is very likely minimal.

That said, if your application’s routes do not directly mirror your controller names (eg. you have /players pointing to UsersController, rather than PlayersController), and if you are doing that intentionally because you think this obscurity gives you additional security (it very likely does not), then this approach for modular assets likely won’t fit your needs.

Attribute Collision

It’s possible you are already using body#id and body.class for other purposes – this is totally understandable but thankfully there is an easy work-around: Use data-attributes. You can even go this route anyways if you prefer to keep your data presentation more organized and are put off by using id and class attributes in this way.

It might look like this:

1
2
3
4
5
<!-- app/views/layouts/application.html.erb -->
<body         id="whatever_you_need_it_to_be" 
           class="this-too" 
   data-resource="<%= params[:controller].gsub(/[^\w]+/,'--') %>" 
     data-method="<%= params[:action].gsub(/[^\w]+/,'') %>">

and then referencing it in your CSS like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
<!-- app/assets/stylesheets/users.scss -->
body[data-resource="users"] {
  &[data-method="new"] {
    header#banner {
      background-image: url('new-user.jpg');
    }
  }
  &[data-method="edit"] {
    header#banner {
      background-image: url('edit-user.jpg');
    } 
  }
}

This accomplishes the same thing but with a little more specificity at the cost of a little more typing. I actually like this one a lot because the intention is a lot more semantically clear, though I also admit it’s not quite as elegant as the initial solution.

Non-Resourceful Routing

If your Rails app uses non-resourceful controllers, e.g. UserSignupsController, but those controllers are manipulating a different resource, e.g. User, you can still use this approach though you’ll need to do a bit more maintenance to keep it up.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
<!-- app/assets/stylesheets/users.scss -->
body#users,
body#user-signups {
  &.new {
    header#banner {
      background-image: url('new-user.jpg');
    }
  }
  &.edit {
    header#banner {
      background-image: url('edit-user.jpg');
    } 
  }
}

Personally, I dislike non-resourceful controllers. I would rather have a :signup action on the UsersController than a :new action on a UserSignupsController, unless I’m creating a UserSignup model…but in that case, it’s now a resourceful route again.

This also assumes you’re wanting to present UserSignups and Users the same – but if UserSignups manipulates both User and also, say, Account models, you could be creating a limited context-collapse. This is part of the reason I dislike non-resourceful routes, though.

Implementation Summary

Step 1: Add this to your main layout

Whichever file begins with <html>, and has a big, expectant, <%= yield %> somewhere within the <body> tag, that’s the one you want to modify:

If you aren’t currently setting the id attribute of the Body tag:

1
2
3
<!-- eg. app/views/layouts/application.html.erb -->
<body id="<%= params[:controller].gsub(/[^\w]+/,'--') %>" 
   class="<%= params[:action].gsub(/[^\w]+/,'') %>">

If you have additional classes, append those after the ERB insert.

If you are currently setting the id attribute of the Body tag:

1
2
3
4
5
<!-- app/views/layouts/application.html.erb -->
<body         id="..." 
           class="..." 
   data-resource="<%= params[:controller].gsub(/[^\w]+/,'--') %>" 
     data-method="<%= params[:action].gsub(/[^\w]+/,'') %>">

Step 2: Wrap your resource SCSS files with the appropriate context

For “some_resource”, that has actions: “new”, “edit”, and “register”, you would use these wrappers:

If you aren’t currently setting the id attribute of the Body tag:

1
2
3
4
5
6
7
8
9
10
11
12
<!-- app/assets/stylesheets/some_resource.scss -->
body#some_resources {
  &.new {
    // ...
  }
  &.edit {
    // ...
  }
  &.register {
    // ...
  }
}

If you are currently setting the id attribute of the Body tag:

1
2
3
4
5
6
7
8
9
10
11
12
<!-- app/assets/stylesheets/some_resource.scss -->
body[data-resource="some_resources"] {
  &[data-method="new"] {
    // ...
  }
  &[data-method="edit"] {
    // ...
  }
  &[data-method="register"] {
    // ...
  }
}

That’s it!

If you already have a bunch of SCSS written and it’s scattered all over the place, this can be a non-trivial migration, but I promise it’s worth it for the organization alone.

Good luck!

HTTP Statuses: The Bad Ones

2020-11-24T00:00:00+00:00

I recently learned that Salesforce’s API does this for one of its response payloads:

1
2
3
4
5
6
7
8
9
10
11
12
// HTTP/1.1 200 OK
{
    "requestId": "524059b7-bf60-4fa6-9271-32aaf3479091",
    "resultMessages": [
        {
            "resultType": "Operational",
            "resultClass": "Error",
            "resultCode": "CustomObjectNotFound",
            "message": "Failed to resolve the Custom Object from the provided ObjectReferenceIdentifier [Id: ed9b5f8e-ebb7-e711-8103-005056b37304, Key: ]."
        }
    ]
}

(The key parts to look at are lines 1 (200 OK) and 8 (Not Found).)

400s (You messed up)

In part 1, we covered the status codes in the 200-300 range; the good ones. This article deals with “The Bad Ones”, the 400-series, also known as “You Messed Up”.

These statuses are for when your request wasn’t formed correctly or the server is unable to process your request because you didn’t ask in the right way. This one has a lot more utility and can help steer your user towards providing better input.

400 (Bad Request)

A 400 is simple: “Bad Stuff Happened.”

This means the user submitted a request that the server wasn’t able to correctly parse into an action. I typically see this when the user submits form data to a valid target, but their parameters are in a weird shape. It can also be used when the user is potentially submitting a malicious payload and the server is being polite in response.

In a similar way to a 200 being a universal “All good!” response, a 400 can be a catch-all.

401 (Unauthorized)

The 401 is one of two status codes around permissions.

This requires a bit of definition.

Authentication is when you identify yourself
Authorization is when your identity is assigned permissions in a system

So a 401 code is called “Unauthorized”, but really, it means “Unauthenticated”; the user hasn’t identified themselves and the resource they’re attempting to access requires authorization – but to authorize, you need an identify authenticated first.

For the user, this is remedied by logging in.

403 (Forbidden)

By contrast, a 403 is “Forbidden”, but this one is actually “Unauthorized.” The user has already weighed, measured, and found wanting.

For the user, this is… well, they can’t really do anything about it. Ask nicely, maybe?

404 (Not Found)

Here’s the one everyone knows and loves, the 404!

Your request was formed correctly, but the resource you’re looking for isn’t available. The important thing for this one is that the server doesn’t know whether or not the resource is intentionally missing, just that it’s missing.

The 404 status is a little different from the next one (410). Think of 404 as like if you called a mall’s customer service line and asked if your friend was there, giving customer service your friend’s name. They can tell you that can’t find your friend, but they wouldn’t really have any way of knowing if your friend was there and was now “Gone”, or if they hadn’t ever been there at all.

405 (Method Not Allowed)

This is a slick one that you can use if you are doing resourceful routing. If you aren’t familiar with REST, then this error message might be a little confusing.

What the error message means is that a user has tried to use an HTTP action verb (GET, PUT, PATCH, POST, or DELETE) on a URL that doesn’t support that action. In a RESTful resource, a url that looks like /articles/ can support different kinds of requests – if you send POST /articles/ {...data...} that usually maps to creating a new resource. Sending GET /articles/ would probably retrieve a collection of the Article resource.

In that example, if the server was only prepared to handle GET /articles/ and not POST /articles/ {...data...} then sending the latter would warrant a response of 405.

Note that if you’re going to send a 405 you’re required to also send an Allow header that lists the actions that are allowed for that resource. eg. Allow: GET, in that example.

410 (Gone)

The 410, by comparison, is when the server knows that something is gone. To extend the analogy above, it’s when you call the business where your friend works and they know that your friend’s “Gone” for the day.

Practically speaking, this is the response you want to send when you “take an article down” and want to stop getting inbound traffic from search engines. When the search engine spiders crawl your site, if they go to a page they thought existed, but you send back a 410, that informs the search engine to delete that from their index. A 404 response might penalize the link as being flaky, but may not delete it outright.

418 (I am a teapot)

Including this one here mainly for trivia, as it was an April Fool’s Joke, though it’s also an official header with an RFC (RFC 2324) and everything! This is the response you send back when your IoT coffee machine can’t brew coffee because it is, in fact, a Teapot.

420 (Enhance Your Calm)

Yes, the Cannabis reference in 420 is intentional. This is an unofficial response (as far as I know, there is no RFC documenting its official use) and I don’t think Twitter even uses it anymore. The contextual meaning, though, was as a response for when the client is making too many requests too rapidly, and so the server was throttling them.

Officially, you should use 429 (see below).

422 (Unprocessable Entity)

For those of you building APIs, you’ll want to know about 422. We usually use these in API responses, and it kind of sits at the intersection of 400 and 500. It’s like a 400 because the request can’t be processed, but the request isn’t malformed. It’s like a 500 because the response is ultimately useless to the client, except the server isn’t encountering an error. It means “I don’t know what to do with what you gave me, so …. here.”

One key difference here between a 422 and a 500 is that the 422 has not bailed and is still at the helm of the execution. So you have access to your in-memory objects like variables and other retrieved data. The typical response is just an empty hash with this status code.

429 (Too Many Requests)

This is the official way to respond when throttling. A 429 response indicates to a client that it is requesting too much. You can also send back a Retry-After header (with a parameter that is the number of seconds to wait) to tell the client how long to wait.

HTTP Statuses: The Good Ones

2020-11-02T00:00:00+00:00

Most people are familiar with what “404” means, at this point. Web browsers used to explicitly say “404 Not Found” when you’d type in a web address that was no good.

Modern design tends towards omitting the status code, but it’s been made into memes and I think has a fair amount of cultural diffusion.

HTTP Statuses are a portion of the data that’s sent back from the web server as part of the response. You request a document with your web browser, and it kicks back a response. They’re all 3-digit codes, and each code corresponds with a different kind of response. 200 means “success”, 404 means “not found”, 500 means “whoops!”

Using the correct status is important because you get some behaviors for free when you use it. Using the correct redirect or “not found” status code will cause search engines to update their indexes. Web browsers implicitly understand certain certain status responses, as well.

Proper HTTP statuses are also just the “right thing to do.” It’s part of the HTTP RFC 7231. The web works because we follow the same standards, so the robots know how to talk with one another. It’s important that we keep sticking with these!

This is not an exhaustive list.

I’m curating this list based on my experience as a backend engineer, sitting between the server and the application.

100s (Informational)

In my 15 years of web development, I’ve literally never explicitly used one of these.

200s (Success!)

If you were to use only a single response code to indicate a successful request, it would be 200. This is, I imagine, the most common HTTP status response since everytime a web page is loaded successfully, it’s almost certainly the status that’s sent back.

200 (OK)

A 200 response means “OK”, as in “here’s your page, as requested!”

When you’re writing backend code, and you’re covering the happy path, and everything is happy, this is most likely the code you send back. More than likely, though, the server software you’re using will send it back for you.

201 (Created)

A 201 response means that the resource you were trying to create was successfully created. The expectation is that new resources are created via a POST verb.

204 (No Content)

A 204 is almost like a 200 except that there’s no body (content) to go with it, just the headers. When you are updating an existing resource, it’s generally done via that PUT verb, and this would commonly be the response.

206 (Partial Content)

A 206 is also similar to a 200 except this one requires a Range header to be sent. This one is used when the client is requesting a large bit of content in piecemeal, such as a large video file, 500kB at a time.

300s (Your princess is in another castle)

These are for when the request is valid but you’re not sending back actual content. Typically these are seen as the “redirects” though the lazy 304 (“you already know it”) also falls into this group.

301 (Moved Permanently)

A 301 status is the code you send back when the document is somewhere else. This is what you use when you move a page from one site to another, or if you do a restructuring and everything is in a new place now. When you send this status back, be sure to also send a Location header back too.

(There is another similar status code, 308, that is almost identical to this except the request headers are immutable. A 301 can change the HTTP verb used, but not with a 308. I can’t think of anytime I’ve ever seen or used a 308 in real life, though)

302 (Found)

This one is kind of funny. It’s called “Found” but we more commonly just call it a “Temporary Redirect”. A 302 is almost the same as a 301, otherwise. You have to also send a Location header with it. The content you were looking for is over here for now, but hang on to the old link. It’s called “Found”, but my coworkers and I always call this one the “temporary redirect” (even though that is technically 307 – similar to 308, a 307 has immutable request headers, but is for a temporary redirect.)

304 (Not Modified)

A very important status code to know, for performance! This requires some additional headers sent in with the request (If-None-Match or If-Modified-Since). When the client requests a resource and provides one of those headers, if the resource hasn’t been modified since the date provided, it responds with this status code and nothing else! Much faster, and much more performant on the server-side.

Using These Headers

Most frameworks come equipped with short-hand methods, or automatic means, of sending back the appopriate status message. Here are some examples for languages that I’ve used. You’ll want to check your docs for how to do it in your language / framework, or find out how yours handles status messages.

PHP

One strength that PHP has here is that the Apache web server actually fields a lot of the statuses. Since PHP is a file that is run through a pre-processor before responding (but it’s an actual file), if you request a file that doesn’t exist, Apache kicks back a 404 and will respond with whatever you tell it to use for 404s. If the document is found, it sends back a 200.

For redirects, you can either use .htaccess for your redirects, or you can use direct PHP:

<?php
  header('Location: /foo/bar/baz.php', 301); // a 301 (permanent) redirect
  exit;
?>

<?php
  header('Location: /foo/bar/baz.php', 302); // a 302 (permanent) redirect
  exit;
?>

You can technically use the header() method to respond with pretty much any status code (Ref), including the 4xx and 5xx headers (covered in Part 2), but most commonly I see this used for redirects. One thing you could do that I don’t see very often is to use header() to send back a 201/204 when handling a POST or PUT, respectively. Typically it just falls through as a 200.

Ruby on Rails

Rails is a little more complicated because every request at least hits the Rails stack before it’s handled. All the routes are dynamically generated and handled, so before it can know if the response should be a 200, it has to see what’s available.

200, 201, and 204 are all sent back implicitly with resourceful actions. If you do a POST#create, the response will be a 201. If you do a PUT#update, the response will be a 204.

If you have caching enabled, and a request is cached, it will automatically respond to the request with a 304. No additional action required.

Redirection is offered with the redirect_to method.

# using the literal status code
redirect_to(alternate_resource_path, :status => 301) and return

# using the friendly-looking text version
redirect_to(alternate_resource_path, :status => :moved_permanently) and return

and for temporary redirects:

# using the literal status code
redirect_to(alternate_resource_path, :status => 302) and return

# using the friendly-looking text version
redirect_to(alternate_resource_path, :status => :found) and return

Input Checkbox Arrays

2020-10-27T00:00:00+00:00

This post covers an unobvious method for submitting checkbox selections, but it also covers, conceptually, how to shape data sent in an HTTP Request params so that it is received by the server as a collection, instead of as a separate value in parallel with the other values.

Checkboxes as a List

A while back, I saw a collection of checkboxes that looked like this:

  <input type="checkbox" name="colors_blue" value="1" />
  <input type="checkbox" name="colors_green" value="1" />
  <input type="checkbox" name="colors_red" value="1" />
  <!-- ... -->

Whether or not you see what the problem is depends a lot on how closely you’ve worked with posting form data.

My first server-side language I used, professionally, was PHP. For all of its faults, it actually does teach you a lot about how form data is posted (per HTTP standards) and how you can work with it.

For those of you who don’t see the issue, let me first show what those checkboxes would look like when posted to a PHP handler. Let’s say we want it to respond back with what the color(s) selected were. Obviously there are many ways to do this, and this is only one possibility.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
<?php
  $blue = $red = $green = "";
  if ($_POST['colors_blue'] == 1) {
    $blue = "blue";
  }

  if ($_POST['colors_green'] == 1) {
    $green = "green";
  }
  
  if ($_POST['colors_red'] == 1) {
    $red = "red";
  }

  echo "Your choices were: $blue $red $green";
?>

Maybe you’re clever and realize you can refactor that into a function:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
<?php
  function get_color($color) {
    if ($_POST["colors_$color"] == 1) {
      return $color;
    } else {
      return "";
    }
  }

  $blue = get_color("blue");
  $red = get_color("red");
  $green = get_color("green");

  echo "Your choices were: $blue $red $green";
?>

It’s actually the same number of lines of code, but will scale better with more colors added: Each color added to the array will add 3 lines to the first example, but only one line to the second.

Another approach would be to iterate over ALL the values in $_POST and use a switch() statement to determine the handling.

Those are all “wrong” solutions.

What I mean by “wrong” is that they’re not utilizing the tools correctly. It would be similar to say that it’s “wrong” to hang a framed picture by hammering the frame itself to the wall (instead of using the picture string on the back and hanging that from a nail). Yes, it technically works, but you’re making your life harder by doing it that way.

Checkboxes as an Array

If you do the HTML like this and they select one more options:

  <input type="checkbox" name="colors[]" value="red" />
  <input type="checkbox" name="colors[]" value="blue" />
  <input type="checkbox" name="colors[]" value="green" />

Then you can write your PHP handler like this:

1
2
3
4
5
6
7
8
9
<?php
  $color_choices = "";

  foreach($_POST['colors'] as $color) {
    $color_choices .= "$color ";
  }

  echo "Your choices were: $color_choices";
?>

Five lines. And really, you could shrink it even further if you don’t mind a bit of sloppy implied string interpolation.

1
2
3
<?php
  echo "Your choices were:" . join(','. $_POST['colors']);
?>

One line.

And it will scale infinitely, no matter how many colors you add.

This works because when you pass a named value that is suffixed with [] it is treated as a collection and allows for multiple values to be associated with it. $_POST['colors'] == array("red", "blue", "green")

Were you to do this, and someone checked all the boxes:

  <input type="checkbox" name="colors" value="red" />
  <input type="checkbox" name="colors" value="blue" />
  <input type="checkbox" name="colors" value="green" />

(Note the lack of []s on the name)

The posted array would only contain the last chosen value. $_POST['colors'] == "green"

The best way to learn more about this is to experiment with processing forms. PHP is probably the best way to do this because it sits immediately on the server layer and is readily available on most hosts (and easy to install on your own local machine!). But any server that expects you to handle the requests directly would work.

Between the Server and the Stack

2020-10-25T00:00:00+00:00

My Background

I’ve been doing web development professionally since 2004, though I’ve been working with HTML since… I want to say 1995. When NCSA Mosaic launched, and Netscape 1.0 shortly after that (they would later evolve into Microsoft Internet Explorer and Mozilla Firefox, respectively), I had an O’Reilly book on the HTML 1.0 spec and was learning how to make my own web pages.

My first server-side language was PHP, and like many PHP web developers, I was self-taught. In the early 00s, we didn’t really have a lot of web frameworks; Wordpress was still in 1.0 and was almost exclusively for blogging (it didn’t even have pages yet!).

If you wanted to write form processing, or handle user interaction with a database-backend, or do any kind of session management, you had to basically write it yourself (or grab copypasta from someone else’s code, which was also very common).

More Recently

In the past 5 to 10 years, with the emergence of web frameworks, a lot of the work happening between the server layer and the application layer gets abstracted away. All of the awesome new folks who are entering the wonderful world of web development aren’t forced to learn those things that we had to learn initially.

The upside is that they can get into writing features for awesome web applications sooner, but the downside is the lack of knowledge about what’s going on under the hood.

This Series

What I would like to do with this series is to write some articles to share some of this knowledge. Initially, it will be driven by things I’ve noticed in working with other people.

The content is intended to fill in that area between the web server (where the initial request is received) and the stack that’s running the application. I might note a framework if there’s something relevant to a specific one, but largely this information should be stack-agnostic. After all, behind any stack or server is a platform-agnostic, open source, HTTP and HTML specification!

The target audience for this is probably people who are either working through a coding bootcamp or self-teaching and are focusing primarily on a “framework-first” curriculum (e.g. Wordpress, React, Rails, .NET). Other web developers, even seasoned ones, may find some of this information useful, as well.

Introducing common API with Aliases

2020-08-22T00:00:00+00:00

Scenario

Let’s say that you have an application that has three very similar models, CompactDisc, CassetteTape, and VinylRecord. They all have very similar behaviors, and are all related to one another, though they each have separate tables. STI may or may not be possible.

Code smell Similar to “data clump”, “feature envy”; though in this case it’s models that are similar enough to always be aggregated together.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
# app/models/compact_disc.rb
class CompactDisc < ApplicationRecord
  has_many :tracks
end

# app/models/cassette_tape.rb
class CassetteTape < ApplicationRecord
  has_many :songs, through: :sides
end

# app/models/vinyl_record.rb
class VinylRecord < ApplicationRecord
  has_many :grooves, through: :record_sides
end

They might be collected together like this (where it’s happening isn’t really important):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
@compact_discs = CompactDisc.all
@cassette_tapes = CassetteTape.all
@vinyl_records = VinylRecord.all

(@compact_discs + @cassette_tapes + @vinyl_records).each do |album|
  case album.class
    when CompactDisc
      render partial: "tracklist", collection: album.tracks
    when CassetteTape
      render partial: "liner_notes", collection: album.songs
    when VinylRecord
      render partial: "tracklist", collection: album.grooves
  end
end

Whether or not this could actually be refactored into an STI relationship is unknown, but we will assume it cannot be, for one reason or another. A case statement, particularly with type-checking, is generally a code smell on its own.

Let’s assume that what we would like the block above to look like is this:

1
2
3
4
5
6
7
@compact_discs = CompactDisc.all
@cassette_tapes = CassetteTape.all
@vinyl_records = VinylRecord.all

(@compact_discs + @cassette_tapes + @vinyl_records).each do |album|
  render partial: "album", object: album
end

and that the _album.html.erb partial looks like this (just to show that we’re not just hiding the type checking). This is simplified for clarity.

1
2
3
4
# app/views/albums/_album.html.erb
album.songs.each do |song|
  render partial: "song", object: song
end

The most direct change would be to modify the association name, but because this is a legacy app, we want to make the least-impactful change. That We can use the :alias method here. So we might change the models from earlier to look like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
# app/models/compact_disc.rb
class CompactDisc < ApplicationRecord
  has_many :tracks
  alias :songs, :tracks
end

# app/models/cassette_tape.rb
class CassetteTape < ApplicationRecord
  has_many :songs, through: :sides # unchanged
end

# app/models/vinyl_record.rb
class VinylRecord < ApplicationRecord
  has_many :grooves, through: :record_sides
  alias :songs, :grooves
end

We can assume that Track and Song quack in similar enough ways that they can function interchangeably in the _song partial. (If not, additional alias can be used)

Introducing EmojiSub

2020-08-22T00:00:00+00:00

I wrote a gem. It’s called EmojiSub (the similarity to gsub is intentional).

You can find the gem on rubygems: rubygems.org/gems/emoji_sub, or see the source on github.com/armahillo/emoji_sub. As of this writing, it is at version 0.2.2.

It’s a very simple script that takes a text string and does a global substitution of any emoji shortcodes (a la Slack, Github, etc) and replaces them with the actual HTML entity code for the emoji.

In the process, I’ve had to learn a lot about Emoji, unicode, and how they work, it’s been a surprising topic that’s far deeper than I anticipated!

I’ll write more about the gem a bit later, once it’s more stable. There’s been a lot of steps involved in this whole process that I’ll talk about too.

Book Store

2020-07-20T00:00:00+00:00

I’ve recently been doing some coding drills to create some opportunities for code refactors. I was reminded of Exercism.io and, having not checked them out in several years, was delighted to see how much they’ve improved since then.

Feel free to follow / friend / stalk me over there: https://exercism.io/profiles/armahillo

I would like to do a blog series on some of the exercises I work through over there, to discuss my thought process on how I approach the problems. I’m currently working through the Ruby track, so all these problems will be using Ruby code, but software design discussion is largely language agnostic, and you should be able to follow along even if you prefer a different language.

The Problem: Book Store (Checkout)

This exercise, one of the ones designated “Hard”, is called “Book Store”. This problem felt very realistic because, at first glance, it seems almost trivial (I solved the primary use cases in about 5 minutes), but there’s an optimization requirement underneath the surface that is a bit insidious.

To try and encourage more sales of different books from a popular 5 book series, a bookshop has decided to offer discounts on multiple book purchases.One copy of any of the five books costs $8.If, however, you buy two different books, you get a 5% discount on those two books.If you buy 3 different books, you get a 10% discount.If you buy 4 different books, you get a 20% discount.If you buy all 5, you get a 25% discount.Note: that if you buy four books, of which 3 are different titles, you get a 10% discount on the 3 that form part of a set, but the fourth book still costs $8.Your mission is to write a piece of code to calculate the price of any conceivable shopping basket (containing only books of the same series), giving as big a discount as possible.

https://exercism.io/my/solutions/08d30003d7f347639f31d31930ff4b1e

Pass 1: Naive Interpretatation

Not too complicated, right? My initial reading of it was “oh, look at how many books are in the cart, and calculate the discount from that, easy peasy”. The input you get is an array (this much is given), and looks like: [2,3,5]

My first solution looked a bit like this:

1
2
3
4
5
6
7
8
class BookStore
  DISCOUNTED_TOTAL = [ 0.0, 1.0, 0.95, 0.90, 0.80, 0.75]
  NOMINAL_PRICE_PER_UNIT = 8.0

  def self.calculate_price(basket)
    NOMINAL_PRICE_PER_UNIT * basket.size * DISCOUNTED_TOTAL[basket.size]
  end
end

A few of the tests passed. My approach here was a trivial (and naive, as I later discovered) approach: multiply the nominal cost times the number of books, and then apply a discount.

I used a class constant array to represent the discount – the discount level is a scalar that I can have serve double-duty as the index in the array. If you have 1 book, you pay 100% (1.0); if you have 3 books, you get a 10% discount (1.00 - 0.10 => 0.90), etc.

I had overlooked one important thing: the discount wasn’t on how many books were purchased overall, it was applied individually to books in a set. A set was 5 possible books, represented as the numerals 1 through 5. The 5-book discount was if you made your set from all 5, the 3-book discount was from if you had 3 of the books, etc.

The test that first failed was given input like this: [1,2,3,3,4,5,5]

Whoops!

Pass 2: Allowing for multiple sets

So the problem then was to calculate the largest possible sets, group the purchase together into those sets, and then calculate the discount and apply it to each group, and total up the result.

This was what I came up with:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
class BookStore
  DISCOUNTED_TOTAL = [ 0.0, 1.0, 0.95, 0.90, 0.80, 0.75]
  NOMINAL_PRICE_PER_UNIT = 8.0

  def self.calculate_price(basket)
    grouped = basket.group_by { |i| i }
    all_sets = []
    while (!grouped.values.flatten.compact.empty?) do
      all_sets << grouped.values.map!(&amp;:pop).compact
    end
    all_sets.collect do |set|
      calculate_price_for_set(set.size)
    end.sum
  end

  def self.calculate_price_for_set(size)
    NOMINAL_PRICE_PER_UNIT * size * DISCOUNTED_TOTAL[size]
  end
end

So this got a bit more interesting.

The first thing I did was to extract the price calculation to a separate method, calculate_price_for_set. I applied a dependency injection refactor to it where it only needs to know the size, instead of knowing that the argument it receives is an argument that can receive a :size message.

After that, my thinking for the main algorithm was this: I wanted to produce the largest possible sets possible. I envisioned making multiple passes over the collection, where you make a maximized combination on each pass.

There is no provision that the arguments will be ordered. The example from above could also be written: [3,1,2,5,3,4,5]

Applying a group_by to the collection will create the necessary clustering. This will convert the above example into:

1
2
irb> [3,1,2,5,3,4,5].group_by { |i| i }
=> {3=>[3, 3], 1=>[1], 2=>[2], 5=>[5, 5], 4=>[4]}

If we have it clustered, then we can ask each type to give us 1 book, at a time, sort of like peeling off a layer at a time. Asking an empty book type for an element will yield nil, which can be compacted away easily.

The powerhouse is here:

1
2
3
while (!grouped.values.flatten.compact.empty?) do
  all_sets << grouped.values.map!(&amp;:pop).compact
end

The while conditional is verbose (and not technically a Law of Demeter violation because each method returns self), but straightforward: if there’s anything left in the groupings, keep chugging along.

The inside of the block is a bit trickier, so let’s break it down.

grouped.values is referencing the Hash values method.

Applying map! to it means “edit this in-place” by applying the block to it. Typical usage of map! would look like this:

1
2
3
4
5
animals = ["dog", "cat", "mouse"]
animals.map! do |element|
  element.upcase
end
puts animals.inspect

The end result would yield: ["DOG", "CAT", "MOUSE"]: The original array with each element being transformed in the same way. The ! does in-place modification, so that we don’t have to do:

1
2
3
4
5
animals = ["dog", "cat", "mouse"]
animals = animals.map do |element|
  element.upcase
end
puts animals.inspect

The :pop method, when applied to an array, does a LIFO desertion of an element from the collection. Though this does not have a ! on it, it is destructive and does modify in-place:

1
2
3
4
animals = ['dog', 'cat', 'mouse']
pet = animals.pop
puts animals.inspect
=> ['dog','cat']

Since each array in the groupings is homogeneous, we don’t care about the order (I think :pop is more fun-sounding than :shift, but they’re effectively the same here.) What we want to happen here is this:

1
2
3
all_sets << grouped.values.map! do |set|
  set.pop
end

Since the same operation is being applied directly to each element, uniformly, with no conditions or additional operations, we can use the Proc shorthand. The above code can be re-written like this, and it will behave identically:

1
all_sets << grouped.values.map!(&amp;:proc)

With the array input of: [3,1,2,5,3,4,5], the contents of all_sets will be [[1,2,3,4,5], [3,5]]. With this, we pass it on to the sum calculator, which is similar to before:

1
2
3
all_sets.collect do |set|
  calculate_price_for_set(set.size)
end.sum

:collect is similar to :map, in that it’s an iterator; I think they may actually just be aliases for one another. Contextually, I like to use :collect when I’m gathering the results, and :map when I’m applying a transformation, but that’s a personal preference.

This block of code does the :calculate_price_for_set method, discussed earlier, gathering the result into an array result. The end.sum at the end means “take the result and send the :sum message to it”

Narratively, the original code for this pass means:

Cluster the list into like terms
Peel one entry off of each and store it in one set
Repeat 2 if there are any entries left
Take the sets and calculate their price (using the formula from the first pass), then sum up the total for all the sets

I have to admit, I felt pretty clever here. It’s concise and elegant. And almost passed all the tests.

Pass 3: Optmization

Remember the insidious problem underneath the surface I mentioned earlier?

One way of grouping these 8 books is:

 - 1 group of 5 --> 25% discount (1st,2nd,3rd,4th,5th)
 -- +1 group of 3 --> 10% discount (1st,2nd,3rd)

 This would give a total of:

  - 5 books at a 25% discount
  -- +3 books at a 10% discount
  - Resulting in:5 x (8 - 2.00) == 5 x 6.00 == $30.00
  -- +3 x (8 - 0.80) == 3 x 7.20 == $21.60

For a total of $51.60

However, a different way to group these 8 books is:

 - 1 group of 4 books --> 20% discount (1st,2nd,3rd,4th)
 -- +1 group of 4 books --> 20% discount (1st,2nd,3rd,5th)

This would give a total of:

 - 4 books at a 20% discount
 -- +4 books at a 20% discount

 Resulting in:4 x (8 - 1.60) == 4 x 6.40 == $25.60

 -- +4 x (8 - 1.60) == 4 x 6.40 == $25.60

For a total of $51.20

And $51.20 is the price with the biggest discount (confirmed by the tests).

These were the tests that were failing.

This is also a slightly different, and significantly more complicated, problem: We now have to teach the script how to optimize the groupings. Comparing the grouping costs once they’re assembled is trivial, but establishing the groupings first is not.

Allocation algorithms is a well-documented and explored facet of computer science (and mathematics). There’s the Travelling Salesman problem (optimize the best path given different weights along the edges), the Knapsack problem (optimize inclusion in a final set based on limited space), etc. This problem is a little similar to both of those.

The first thing I did was to look at the original definition of the discount breakdown to understand why there might be a difference:

1 of any of the five books gets 0% discount ($8)
2 different books, gives a 5% discount on those two books ($8 + $8 = $16 * 95%).
3 different books, you get a 10% discount.
4 different books, you get a 20% discount.
5, you get a 25% discount.

The jump from 1 to 2 is a 5% decrease in cost. The difference from 2 to 3 is also a 5% decrease. However, the difference from 3 to 4 is a whole 10% decrease. Going from 4 to 5 is only an additional 5% decrease.

So really, any situation where we can get a 4-book set is potentially a bigger discount. Designing for this scenario specifically will still be somewhat complicated and would likely result in over-fitting (making it harder to modify in the future).

The strategy I was imagining was one where we try to balance out the sets as much as possible. One of the tests is named: “two groups of four is cheaper than a group of five and a group of three”, which further supports this hypothesis.

The question then became, how do I turn this:

1
{3=>[3, 3], 1=>[1], 2=>[2], 5=>[5, 5], 4=>[4]}

Into this:

1
[[1,2,3,5],[3,4,5]]

My first pass on this strategy was very brutish. I don’t have the code anymore and it would be a pain to recreate it, but basically it set up two pointers, a “set pointer” and a “book pointer”. The set pointer was responsible for pointing to which set we were currently adding to, and the book pointer pointed to the book that was a candidate for addition. Each iteration of the loop would advance the set pointer if a book had been added, but would always advance the book pointer.

I also realized that the number of sets we would need was (perhaps naively) the size of the largest collection of books. Logically, we would at least need that many; given [1,1,1,1,1,2,3,4,5] even the “largest set” algorithm would yield:

1
[[1,2,3,4,5],[1],[1],[1],[1]]

This would prove important later.

Overall, though, this was clunky. It was immediately apparent that, in all the ways the “largest set” solution was elegant, this was the opposite. I kept running into “wait, but what if…” cases.

“What if the set already has the book that’s pointed to?”, “What if the set is full already?”, “What if the book that’s pointed to is all gone?”

I put it away for the evening and returned to it in the morning.

Sleep helped! I realized that I didn’t need to pre-group like with the largest set strategy. The approach for “largest set” was to focus on the set first and fill it up with as much as possible; what we needed to do here was focus on each book type and spread them out as much as possible.

If we began with the book types sorted by how many there were, highest quantity first, we would get this:

1
2
order_by_quantity([3,1,2,5,3,4,5])
=> [3,3,5,5,1,2,4]

This tells us that we will need 2 sets (we have two 3s), and if we process this list in order, we will ensure that the highest available books will be distributed before the rest of the books are distributed. In pseudocode, this might look like:

Order the list by its quantity, highest quantity first
Initialize a number of sets equal to the highest number of books of a single type
Starting with the highest quantity books in the ordered list: a. Add the book to the current set b. Move to the next set in the series (wrapping around if necessary)

Step 1 could be approached a few different ways. I did it like this, using the grouping from the “largest set” strategy:

1
2
3
4
5
def cluster(pile)
    pile.group_by { |i| i }
end
# ...
grouped = cluster(pile).sort_by { |b| b.size }.to_h

We’re using the clustering approach from “largest set”, and then applying :sort_by to it. We’re sorting based on the size. This puts the largest groups first.

Step 2, initializing the arrays, is a trivially normal, though there is a gotcha here that I learned (continuing from previous sample):

1
2
number_of_sets = grouped.values.first.size
balanced_sets = Array.new(number_of_sets) { |_| [] }

Array.new can technically accept two arguments: how many elements to initialize, and a second argument that is an initial value for each.

HOWEVER, look at this:

1
2
3
4
5
6
7
8
9
10
11
12
13
2.7.1 :001 > cloned_elements = Array.new(3,[])
2.7.1 :002 > cloned_elements.inspect
 => "[[], [], []]"
2.7.1 :003 > cloned_elements[0].object_id
 => 180
2.7.1 :004 > cloned_elements[1].object_id
 => 180
2.7.1 :005 > cloned_elements[2].object_id
 => 180
2.7.1 :006 > cloned_elements[0] << 1
 => [1]
2.7.1 :007 > cloned_elements
 => [[1], [1], [1]]

Whoops! That’s no good. The documentation for Array.new notes:

When sending the second parameter, the same object will be used as the value for all the array elements. Since all the Array elements store the same hash, changes to one of them will affect them all. If multiple copies are what you want, you should use the block version which uses the result of that block each time an element of the array needs to be initialized.

1
2
3
4
5
6
7
8
9
10
11
12
13
2.7.1 :008 > unique_elements = Array.new(3) { |i| [] }
2.7.1 :009 > unique_elements
 => [[], [], []]
2.7.1 :010 > unique_elements[0].object_id
 => 200
2.7.1 :011 > unique_elements[1].object_id
 => 220
2.7.1 :012 > unique_elements[2].object_id
 => 240
2.7.1 :013 > unique_elements[0] << 1
 => [1]
2.7.1 :014 > unique_elements
 => [[1], [], []]

This looks like what we want. I actually got tripped up by this behavior initially – I was adding a single book to one set and it was adding to all the sets. How confusing!

Step 3 from above is a straightforward approach, instead of using book pointers and set pointers:

1
2
3
4
set_pointer = 0
ordered.each do |book|
  # stuff coming soon
end

We’re using ordered from the previous example. We are going to use a set pointer here, but we no longer need the book pointer as well.

Steps 3a and 3b will use that set pointer to distribute the book:

1
2
3
4
ordered.each do |book|
  balanced_sets[set_pointer % balanced_sets.size] << book
  set_pointer += 1
end

The set_pointer modulo math is a trick I learned in C, many years ago. You can continuously increase and ensure that you wrap around with the right cycle by taking the modulus against the size of the array.

1
2
3
4
5
6
7
8
9
10
11
> 0 % 3
=> 0
> 1 % 3
=> 1
> 2 % 3
=> 2
> 3 % 3
=> 0
> 4 % 3
=> 1
# ...

Since arrays are 0-indexed, this trick maps well, and balanced_sets[set_pointer % balanced_sets.size]should never land out-of-bounds.

This solution worked. I wrapped the whole business in a method called “balanced” and put it alongside the other method “largest”.

The last thing to do was to apply both strategies to the same basket and see which one produced the lower cost:

1
2
3
[:largest, :balanced].map do |strategy|
  calculate_price_for_combinations(book_pile.apply_strategy(strategy))
end.min

The approach I took here was to do some meta-programming: :apply_strategy accepts a label of some kind, and then invokes the method that matches that name:

1
2
3
4
5
6
def apply_strategy(strategy)
  return [] if @pile.empty?
  send(strategy.to_sym, @pile.dup)
rescue NoMethodError => e
  puts "#{e} isn't a supported strategy"
end

This pre-emptively resolves any situation where the inbound array (@pile) is empty. It uses the :send method, passing the strategy provided, with a duplication of the @pile. Using a duplication ensures we don’t accidentally produce mutations as side effects.

One advantage of this approach is that we can add additional optimization strategies (maybe the store offers additional discounts on odd-numbered volumes) and add them alongside these other two, and minimal changes will be required to the overall structure – add a new method :discount_odd_volumes, add that to the array of methods, and that’s it!

Similarly, it separates the cost calculation from the grouping algorithm, so changes to the former will not affect the latter, and vice versa.

I’m happy with this result, and so was the test suite.

Check out my solution and let me know what you think.

I’ll try to post some more of these in the future.

Header Image Credit: https://www.needpix.com/photo/download/928828/books-book-bookshelf-library-read-literature-shelf-free-pictures-free-photos

Modifying a has_many to include nils

2017-10-26T00:00:00+00:00

I had this interesting use case come up.

We have a multi-tenant app that has an Organization model. An Organization has many BarcodeItem.

Originally, the query was direct: We only wanted an instance of Organization to return the barcode_items that belonged to it. Then we decided to pivot on this particular feature and have an organization include both its own barcode_items but also any BarcodeItem that was explicitly set to nil . The idea here was that some BarcodeItem would be “global”, and the least complicated way to do that would be to de-associate them from any Organization.

So I started with a spec to test this out:

1
2
3
4
5
6
7
8
9
10
describe "barcode_items" do
  it "returns both this organizations barcodes as well as global ones" do
    # Test that it can see globals
    create(:barcode_item, organization_id: nil) # global
    expect(organization.barcode_items.count).to eq(1)
    # Test that it can also see its own items
    create(:barcode_item, :for_organization, organization: organization)
    expect(organization.barcode_items.count).to eq(2)
  end
end

My first thought was to use an association model, something like:

1
2
3
4
5
6
7
8
9
10
class Organization < ApplicationRecord
  # ...
  has_many :barcode_items do
    def all
      # this doesn't work but it's the idea I started with
      where(organization_id: [self.id, nil])
    end
  end
  # ...
end

But even after just writing that it just smelled wrong. Surely there’s an easier way to do this. Consulting the docs, I was reminded of using association scopes. So I tried this instead:

1
2
3
4
5
class Organization < ApplicationRecord
  # ...
  has_many :barcode_items, -> (organization) { organization_id: [nil, organzation.id] }
  # ...
end

Still fails. I needed to see what it’s actually doing behind the scenes, so I dumped the SQL before the test:

1
2
3
4
5
6
7
it "returns both this organizations barcodes as well as global ones" do
  # ...
  create(:barcode_item, organization_id: nil) # global
  puts organization.barcode_items.to_sql
  expect(organization.barcode_items.count).to eq(1)
  # ...
end

RSpec displayed the query as:

1
2
3
4
5
SELECT "barcode_items".* FROM "barcode_items" 
  WHERE "barcode_items"."organization_id" = 4 
  AND ("barcode_items"."organization_id" = 4 
    OR "barcode_items"."organization_id" IS NULL)
-- returns both this organizations barcodes as well as global ones (FAILED - 1)

A-ha!

The additional scoping added to the association is a boolean AND, but we need a boolean OR. After several failed attempts at mimicking that functionality, I found a StackOverflow answer that suggested unscoping the query first.

1
2
3
4
5
6
7
class Organization < ApplicationRecord
  # ...
  has_many :barcode_items, ->(organization) { 
    unscope(where: :organization_id).where(organization_id: [nil,organization.id])
  }
  # ...
end

Success!

Authenticating against WordPress in Rails 3

2012-03-13T00:00:00+00:00

N.B. Note the date on this article. It was written before Wordpress Version 5. I have not worked with Wordpress much since writing this, so I don’t know if it still works.

That said, I think the post still has value in its cross-application methodology.

I’ve been working on a Rails app that needs to integrate with an existing userbase found in a WordPress installation.

One problem I’ve encountered is that her main website that manages subscriptions and users is in WordPress. Users will register with that main website and may or may not have access to the Rails application, separately (depending on their subscription level). What I needed was a method to handle User Authentication in Rails that slaves to the subscription / account information held in WordPress. Ideally, this should all keep with the DRY principle to make it nice and Rails-y.

I wanted to avoid using any messy and potentially unsecure session cookie sharing, and I also wanted to avoid doing any database shadowing. I suspected that since all the data is up on databases on the same server, I should be able to just query it out, right?

Right!

Ok, so this was a little tricky and it took a couple hours to suss it all out, but here’s what I did. For this example, we’ll say that all of your databases are on the same host (mysql.host.com), though they don’t need to be.

Defining the settings

In your config/database.yml file, you need to specify separate entries for your WordPress database. Thanks to fnichol for this gist.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
development:
  adapter: mysql2
  encoding: utf8
  reconnect: false
  database: rails_proj_development
  pool: 5
  username: rp_dev
  password: wootwoot
  host: localhost

wp_development:
  adapter: mysql2
  encoding: utf8
  reconnect: false
  database: wordpress
  pool: 5
  username: wp_user
  password: hackable
  host: localhost
  table_prefix_name: wp_dev_

  # Warning: The database defined as "test" will be erased and
  # re-generated from your development database when you run "rake".
  # Do not set this db to the same as development or production.

test:
  adapter: mysql2
  encoding: utf8
 reconnect: false
  database: rails_proj_test
  pool: 5
  username: rp_test
  password: wootwoot
  host: localhost

wp_test:
  adapter: mysql2
  encoding: utf8
  reconnect: false
  database: wordpress
  pool: 5
  username: wp_user
  password: hackable
  host: localhost
  table_prefix_name: wp_test_

production:
  adapter: mysql2
  encoding: utf8
  reconnect: false
  database: rails_proj_production
  pool: 5
  username: rp_prod
  password: wootwoot
  host: localhost

wp_production:
  adapter: mysql2
  encoding: utf8
  reconnect: false
  database: wordpress
  pool: 5
  username: wp_user
  password: hackable
  host: localhost
  table_prefix_name: wp_

Note that in the wp_* entries, I added a table_prefix_name entry. This will help keep things clean (inclusion alone does not do all the heavy lifting, but it puts it in an obvious place that we can reference later). The prefixes allow you to optionally put all your WordPresses into one database, prefixing them with different prefixes. I wouldn’t say this is necessarily advisable but it’s at least an option, if your configuration is already like that.

Creating the Model

We can use ActiveRecord to connect to this database, so that everything is seamless. You will need to create a different model for each table you want to connect to (this is consistent with the ActiveRecord pattern anyways). You can either create the model manually, using your favorite text editor, or you can use the rails generator, but be sure to pass it –skip-migration, since you don’t want it to create a duplicate table in your Rails DB.

Here is the model I used for “User”. (fnichol does one for Post on his gist).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
class User < ActiveRecord::Base

  # define any "has_many" relationships here

  establish_connection "wp_#{Rails.env}"

  @prefix = Rails.application.config.database_configuration["wp_#{Rails.env}"]['table_prefix_name']

  set_table_name "#{@prefix}users"

  alias_attribute  "id",             "ID"
  alias_attribute  "login",          "user_login"
  alias_attribute  "pass",           "user_pass"
  alias_attribute  "nicename",       "user_nicename"
  alias_attribute  "email",          "user_email"
  alias_attribute  "url",            "user_url"
  alias_attribute  "created_at",     "user_registered"
  alias_attribute  "activation_key", "user_activation_key"
  alias_attribute  "status",         "user_status"
  alias_attribute  "display_name",   "display_name"

  def readonly?
    return true
  end

  def before_destroy
    raise ActiveRecord::ReadOnlyRecord
  end
end

So, first things first, do it just like you would any Rails AR model. Just make it a new class inheriting from AR:Base.

This first line creates the connection:

establish_connection "wp_#{Rails.env}"

If you used the naming scheme from the example database.yml file above, AND if your webhost handles Rails environments properly (mine sadly doesn’t 🙁 ), this should “just work.”

The prefix we created in the database.yml file earlier will now be pulled in:

@prefix = Rails.application.config.database_configuration["wp_#{Rails.env}"]['table_prefix_name']

We then use that prefix to specify the table name. If you were wanting to create a Post model, you would obviously just change it to posts. A taxonomy model might use terms.

set_table_name "#{@prefix}users"

The lines in the middle are just aliases to make the model more Rails-y. The only critical one is the first — you need to alias ID to be id so that Rails can locate the id field when doing relationships.

The last part is optional, though I recommend including it. Since Rails is a guest in the house of WordPress, you don’t want it knocking over vases, making out with WordPress’s daughter, or eating all the expired cupcakes and cookies in their cupboards like I used to do to my best friends in grade school (only that last part, not the second one and the first one only once).

Example:

irb(main):005:0> u = User.last
=> #
irb(main):006:0> u.url = "foo"
=> "foo"
irb(main):007:0> u.save
ActiveRecord::ReadOnlyRecord: ActiveRecord::ReadOnlyRecord
...(stacktrace)...

irb(main):008:0> u.destroy
ActiveRecord::ReadOnlyRecord: ActiveRecord::ReadOnlyRecord
...(stacktrace)...
irb(main):009:0> u2 = User.new
=> #
irb(main):010:0> u2.user_login = "test"
=> "test"
irb(main):011:0> u2.save
ActiveRecord::ReadOnlyRecord: ActiveRecord::ReadOnlyRecord

Keeping the foreign table read-only will help ensure your Rails App doesn’t accidentally eat a Swiss Cake Roll that wasn’t yet expired.

Authenticating Against the Database

Password “hashing” is the process of encrypting the password to make it so bad guys and curious monkeys can’t easily tell what your passwords are.

It gets a little tricky here. If your wordpress was created AFTER version 2.5, then the account passwords will be hashed using a third-party library called “Phpass“. If they were created before, they will use an MD5 hash.

Since you are more than likely upgraded to the current version (because you are an awesome and smart wordpress user), bear in mind that even if your old accounts were created prior to v2.5 (because you’re old school, too!), if those passwords are changed or if new users are created, the new passwords will be hashed with Phpass. (If you login to your blog with your account, it will automatically update your password to the more secure encryption).

Like I said, tricky.

Either way, you’ll need Phpass for ruby. Fortunately, someone has already made a Gem for it: phpass-ruby. Add gem 'phpass-ruby' to your Gemfile, then bundle install, and you should be all set with that.

MD5 digests can be done using

Digest::MD5.hexdigest("plain text password").

Consider this an incomplete solution, yet at least a starting point.

Add this block to your User model.

1
2
3
4
def check_password plaintext_password
  require 'phpass'
  return Phpass.new(8).check(plaintext_password, pass)
end

I’m still researching the code to see if this will need to factor in the presence of any salts that you specify in your wp-config.php. Supposedly those salts are factored into the passwords, but in pluggable.php, the only place I see wp_salt() being used is in cookies and nonces. All this means is that you shouldn’t have to do anything extra provided your password is updated.

Testing it out

Once you’ve got that done. Test it out by jumping into your rails console.

Loading development environment (Rails 3.0.3)
irb(main):001:0> u = User.last
=> #
irb(main):002:0> u.check_password "bad_password"
=> false
irb(main):003:0> u.check_password "good_password"
=> true

If you cannot get the password to verify and you are absolutely positive you’re using the correct password, it likely means that your password is old-skool and needs to be updated. An MD5 password will appear in your database as only 0-9 and a-f, if you see any other characters, then it’s been modernized) You can make it hip by simply logging into your dashboard; WordPress will fix it when it checks.

If you’re getting errors about a “table not existing”, you can check to see if it’s using the correct table_name with this console command:

irb(main):008:0> User.table_name
=> "wp_dev_users"

My model took some tweaking here mainly because my host has a very dysfunctional relationship with its dev/test/prod environments.

Pitfall of Belongs_to

I have tested it with a has_many (which requires no modification to the table), but not with a belongs_to. Since the latter requires that the table actually have an additional reference column for the foreign key, do this at your own risk. WordPress may modify the user table without you knowing and that could break your relationships.

My suggestion would be to create a proxy ownership model (“liaison”?) that belongs_to your User model. That proxy ownership model could then handle all other Rails relationships autonomously from the WordPress database. It makes your AREL a little less pretty (eg. some_group.liaisons.first.nicename, assuming a Group model has_many :liaisons)

Applying it

In the real world context, you can now use it in whatever authentication framework you like. Since it’s using ActiveRecord, you can employ it any place you would if it was in a Rails DB instead.

Don’t forget you also have access to the other details in the WP user table, such as “nice name”, their e-mail, etc. If you want to include their profile details, you can create a usermeta model:

1
2
3
4
5
6
7
8
9
10
 
class User < ActiveRecord::Base
  has_many :usermeta
  ...
end

class Usermeta < ActiveRecord::Base
  belongs_to :user
  ...
end

Usermeta already contains a user_id column in Rails-friendly format. You’ll have to write some model methods to extract all the data easily. I suggest making that one readonly as well.

Using Virtualbox with an Existing Partition

2010-01-27T00:00:00+00:00

N.B. This is another throwback to an old blog I ran. It was the most-commented post on that blog, and a local friend recently found it useful, so I’m re-posting it!

I don’t know how specifically relevant it is, but feel free to contact me if you’ve found updates. I will credit you!

Last week, a friend of mine needed me to do an audio file conversion, but the app that I use is installed on my windows partition. I really don’t ever boot into Windows unless I have a good reason for it — I’m much happier tooling around in Linux — there’s just something satisfying and comfortable about being able to pop open a shell at any time.

Anyways – it got me thinking: I’ve booted into a Windows XP image, why can’t I use VirtualBox to boot from a whole partition? Surely that is possible…

Tonight I finally got to play with it. And as you can see from the image here, I got success. :)

It’s a little challenging, but it’s doable. I had to spend some time to iron out the kinks, but you can reap the benefits!

Legacy Update notices

UPDATE: Sandeep has submitted screenshots with instructions on getting this to work with Windows 7, see below, at the very end.

UPDATE: If you are getting the error message: Offset must be a number: rce I have found the fix for it. See the instructions below.

UPDATE: Bogdan (see comments) was able to get Windows Vista working under Virtualbox OSE, using the method below. See his comments for specifics on Windows Vista.

UPDATE: Dan has found some tricks for getting this to work with Win7 if you are getting a BSOD on bootup.

UPDATE: Julian has successfully gotten it to work with Windows 8 Pro, see below, near the end of the post content.

UPDATE: user PCWorld has submitted some suggestions to get this to work with Windows 7

UPATE: user ComputingFroggy has cross-posted some tips on Windows 7 integration with Lubuntu.

Install Virtual Box

Ok, for starters, you need the CLOSED SOURCE version of VirtualBox. As in — do not install the one from the Ubuntu Software Center. Go directly to ~Sun~ Oracle’s website and download the appropriate version for your OS.

Ubuntu users should use the PPA instead of downloading the binaries. Be sure to check the link above for the most recent PPA. This sample applies to 12.04, Precise Pangolin, but there will quite possibly be a newer version on the website when you read this.

Open up Ubuntu Software Center
Click on the Edit top menu, and select “Software Sources“
Choose the “Other Software” tab
Click on “Add…“
Paste: deb http://download.virtualbox.org/virtualbox/debian precise contrib
Click on “Add Source”

Now you’ll want to update your repo cache and get the key:

wget -q http://download.virtualbox.org/virtualbox/debian/oracle_vbox.asc -O- | sudo apt-key add -
sudo apt-get update

And then install the package of your choice.

sudo apt-get install virtualbox-4.2

Whether you do binary-based install or package based install — install the most recent version of the personal edition (NOT “OSE”, which is the open-source version.) As of this post, the current version for Precise Pangolin (12.04) is VirtualBox 4.2.

The Virtualbox documentation also indicates that Ubuntu/Debian users should install the dkms package.

sudo apt-get install dkms

Install `mbr`

In order to successfully trick Windows into booting into a confined space, you need to fake your mbr (no grub). Fortunately this is WAY easier than it sounds:

sudo apt-get install mbr
install-mbr --f ~/.VirtualBox/FAKE.mbr

That’s it! We’ll use that later.

Your system will NOT be affected by this — all it does is copy the MBR from your computer, dump it into a file and that’s it.

NOTE: Originally, I had used the full name flag (–force) which uses two dashes. Due to the way my stylesheet renders italicized fonts, it looked like it was a single dash, understandably confusing some people. If you are getting the error message:

install-mbr: Offset must be a number: rce

Then either use just -f instead, or ensure that you are using two -s before “force”.

Debugging an issue with install-mbr by using the -e1 option

H/T to Ken D:

$ install-mbr -e1 --force ~/.VirtualBox/FAKE.mbr

His windows partition is /dev/sda1, so if yours is not, you need to use -eN where “N” is the number of the partition where windows is installed. Or not. (More info)

Create a VDMK file

This is actually the trickiest part. A VDMK, which I am not sure what that is an acronym for, is essentially a micro-image that contains instructions to tell MBR where it’s booting from. If your computer is modern, your hard drives are likely SATA drives (and thus represented as /dev/sda). If you’re unsure, just go into a shell and type fdisk -l (that last part is a “dash lowercase-L” not “dash one”). What you want is the device name for your harddrive… mine is /dev/sda — depending on how many hard drives you have and what type they are, it might be /dev/hda/ or /dev/sdb/ etc.

It’s worth noting that if you are currently mounting your windows partition in Linux (I do) so that you can access your Windows filesystem while in Linux, you will need to unmount (eg. sudo umount /windows) it first.

You’ll want to determine which partition contains Windows, so we can restrict Windows to ONLY using its own partition — this is actually a Linux-exclusive ability (the Windows version of Vbox can’t do this, because Windows is a wuss). So to do this, first you need to know what partition(s) Windows is on. This command will tell you:

$ VBoxManage internalcommands listpartitions -rawdisk /dev/sda

For the last part, use whatever you determined from above. It’s PROBABLY /dev/sda for you too, that’s pretty common.

The command will produce output that looks something like this:

VirtualBox Command Line Management Interface Version 3.1.2
(C) 2005-2009 Sun Microsystems, Inc.
All rights reserved.

Number  Type   StartCHS       EndCHS      Size (MiB)  Start (Sect)
1       0x83  0   /1  /1   1023/254/63         53348           63
2       0x82  1023/254/63  1023/254/63          4000    109258065
3       0x83  1023/254/63  1023/254/63        155998    117451215
4       0x07  1023/0  /1   1023/254/63         91895    436935870

(I have to admit, I feel a little naked showing the whole Internet my partition table)

The last line is the one you want to look for, although yours may not be your last line. Whichever one(s) have 0x07 as their “Type” are the one(s) you want. Jot down the number(s) for those. (In this case, my number is just 4).

Now there’s one last thing we have to do that may make you a little uncomfortable, if you’re a paranoid person. We need to change the permissions on the hard drive device nodes. If you don’t know what that means, then you are probably not paranoid about that. :)

Fortunately, we only need to change the permissions slightly, and only on the Windows partitions. So it’s not THAT big of a deal.

In a terminal, type:

sudo chmod 666 /dev/sda
sudo chmod 660 /dev/sda4

That second line should reflect whichever hard drives you are using (remember that fdisk -l we did?) You’ll need to repeat the command for each hard drive that you want Windows to be able to access.

For those of you that are paranoid: You should be able to change the permissions back to 600 after we’ve created the VDMK file — VirtualBox needs access to your partition table so it can do its job.

There’s one last thing we need to do, and that’s adding you to the “disk” group, so you can access the partitions you just opened up. You’ll need to logout and log back in after doing this, so that your permissions are reset. (Previously forgot to include the username — thanks Ken!)

sudo usermod -a -G disk yourusername

Ok, now you’re ready to actually create the VDMK file. Get ready because this is a handful.

$ VBoxManage internalcommands createrawvmdk -filename ~/.VirtualBox/winxp.vmdk -rawdisk /dev/sda -partitions 4 -mbr ~/.VirtualBox/FAKE.mbr -relative

First off, sda is whatever you determined you’re using, from earlier. It’s probably sda, as noted.

The number next to the partitions is the numbers you jotted down from the previous “listpartitions” command (the one where I said I felt naked). If you are using more than one partition, you will need to list them as comma-separated values. So if you wanted partitions “1” and “2” for Windows, you would replace the 4 I wrote there with 1,2. Partitions 2, 3 and 4 would be 2,3,4 and so on.

The “partitions” parameter is what tells VirtualBox “only give Windows access to these partitions, and no where else!”

Got a Win7 64bit Home Professional, with an ACER Recovery Partition as partition 1, the “boot loader”(?) partition as 2, and the actual win7 partition as 3, so i did a “-partitions 1,2,3″. Had to use the .iso of the win7 recovery disc afterwards. H/T to Heix.

Just to break down what’s going on here:

VBoxManage internalcommands createrawvmdk

this portion runs the program to “Create Raw VMDK”. Self-explanatory. DO NOT USE SUDO HERE (h/t to Javier, below!)

-filename ~/.VirtualBox/winxp.vmdk

this is the output file that actually holds the micro-image. You can call it whatever you like, and you can put it whereever you like. If you have multiple Virtual Machines, you may like to file them away in a separate folder. Whatever you like.

-rawdisk /dev/sda -partitions 4

This tells it that the raw Windows installation is located on device /dev/sda and that you only want to use partition 4 (yours may be different, as noted above)

-mbr ~/.VirtualBox/FAKE.mbr -relative

the -mbr parameter tells it to use that Fake MBR we created earlier. It won’t work without this. -relative is a parameter that works in conjunction with the -partitions parameter, to allow you to explicitly specify which partition you want to use.

-register

This used to tell VirtualBox to register it in its database of available images, but has since been deprecated. Do not use this with version 4 of VBox. (Thanks, Ken D!)

That’s it. Chances are, if you are going to get errors anywhere, it’s likely to be here.

Possible causes, based on my experience and what I’ve read about on the Internet, is that you didn’t set permissions correctly, didn’t unmount your device, are using the Open-source edition (which doesn’t have createrawvdmk) instead of the closed-source personal edition. If you get an error, you can post it in a comment and I’ll see if I can help you, but google around — there are a LOT of resources online for this. Below, I’ll include links to the places I looked.

Load your new VDMK into Virtual Box

Load up Virtualbox, if you installed it through the automated installation (rather than compiling it manually), you should see it by opening dash (click on the Unity icon) and typing “Virtual”. If you don’t see it there, try logging out and logging back in.

In VirtualBox, click on New, and follow the instructions for the first couple of screens. Name it whatever you like and select your OS (these instructions have all applied to Windows XP so far — caveat operor on any other versions / OSs). Once you get to the “Virtual Hard Disk” screen (pictured) select Use existing hard disk and you should see the VDMK file we just created. If you don’t, click the little folder icon next to the drop down list. It will allow you to “add” your vdmk file that you created.

Once that’s done, you’ll be back at the main screen.

Click once on your new Virtual Machine, and click the Settings icon at the top. You need to do a few more things first.

Configuring your Virtual Machine In the System area, be sure to check Enable IO APIC in the Motherboard tab. In Processor, specify how many CPUs you want to use (ie. I had a Core 2 Duo, so I could choose more than one). You really only need 1 — if you’re doing stuff that requires more, you should probably boot into Windows natively.

In the Acceleration tab — some CPUs have the native instructions built for virtualization — modern Intel CPUs do. If you paid a little extra for your CPU and you bought it in the past year or two, you might have these. I forget the specific models that have it, but it’s something that’s easy to overlook. :)

Under Display area, adjust the slider to specify how much Video RAM you want to give your VM. I picked 64MB for mine… very modest. You can also check Enable 2d Acceleration and Enable 3d acceleration simply because it won’t hurt to do so… but again, if you’re doing things that require hardware acceleration, a VM probably isn’t the best way to do it. :) (sidebar: with computing power as strong as it is nowadays, you can realistically run 3D accelerated games in a VM, provided you can devote enough resources to it)

That’s it — the rest of the stuff is just icing. You can tinker with it if you want, but you’re good to go.

Click on OK to get out of there, then click Start (green arrow at the top). The first time you boot up, it may take a little while longer than usual.

This is the other point that is prone to errors. The first time I did this, I had errors because I didn’t do the -partitions parameter in my VDMK creation, which gave me “disk read” errors. You might get a blue screen of death or some other nonsense. If you get any errors — google it. Like I said earlier, there’s lots of support for it.

Additional Notes for Windows 7 Users

To anyone struggling to get this running with Windows 7, I might be able to save you some time:

If Windows bombs straight to a blue screen of death on boot, in your VM go to settings->storage. If your .vmdk file is listed under SATA, then remove the attachment, then add a new hard disk under IDE, selecting your vmdk file. Next – and this is the crucial bit, in the storage settings again, select the IDE controller, then change its type to ICH6.

Save your settings, and your Windows 7 partition should boot – given you’ve followed all the previous instructions (particularly the windows 7 disc repair option part, which will get you past the MBR 1FA problem.)

Thanks to Dan (@lazydan) for the feedback!

User-submitted Walkthrough for Windows 7 (German) describes a solution for Windows 7 so that you won’t need to do a “Repair Installation” each time you want to switch from native to the VM and vice versa (you won’t need any Windows 7 Installation Disk at all)

First, run

sudo fdisk -l

on your host system, you’ll need the value behind “Disk identifier:” of the hard drive your Windows 7 is on. After you’ve created the VM (leave out the step telling you to boot from the Windows 7 Installation Disk), attach an Ubuntu (or any other Linux) live CD image to the VM:

Settings -> Storage -> Controller: IDE -> Add CD Device (first icon) -> Choose Disk -> select the image of your live CD Then boot the VM, it should boot the live CD image. Once you’re in there, enter:

sudo fdisk /dev/sda

to start fdisk’s interactive mode (replace /dev/sda with the hard drive your Windows 7 is on). First, enter

then

Put in the disk identifier you’ve gathered in the first step, then press

to rewrite the MBR and quit fdisk.

After this, shut down the VM and remove the live CD image from the VM’s storage devices. And note that the device name (e.g. /dev/sda) you have to use on the guest system might differ from its name on the host system.

Now you should be able to boot Windows 7 both natively and through the VM without Windows complaining.

To everyone else that has problems, I suggest you to read the wiki article linked at the top of my comment, it’s very comprehensive and has many troubleshooting tips at the end.

UPDATE: For Windows 8 Professional users Julian says:

I’ve managed to get Windows 8 Pro (64bit) to boot in my 64bit version of Ubuntu 12.04.

I had a few issues getting it booting to start with, but using the windows 8 installation disk to get into command prompt and run

 bootrec.exe /fixboot
 bootrec.exe /fixmbr
 bootrec.exe /rebuildBCD

got this working. I couldn’t access my other SATA HDD that carries all my media, (And adding this as a separate vmdk in my virtual machine’s options made it borky and not detect the original Windows installation), but I’ll try and work it out.

Once You’re In Windows Some of the sites I’ve read have suggested creating an alternate hardware profile for when you boot into Vbox. This is a good idea if you plan on booting into it normally once in a while (for games or whatnot). To create a new Hardware Profile, right click on “My Computer” and click “Properties”. Select “Hardware” then at the bottom, select “Hardware Profiles”. Create a new one and name it “Raw Boot”, and rename the current one to “VBox Boot”.

Windows will go through some growing pains this first boot sequence, as it maps all the hardware drivers to the VirtualBox extensions provided — You will likely need to restart. No guarantees on all the hardware working as it does in a native boot. You should be able to use things like Word, browse the web, use Photoshop (if you allocated enough RAM…you can always change that later), etc. Anything relatively lightweight should be fine.

Working with Windows 7 (Update!) One of the commenters below, Sandeep (“Sandy”), has figured out how to get a Windows 7 partition to work with this method. There are a couple small steps that need to be done, and he was cool enough to take screen shots for this blog:

Before you begin, be sure your VDMK is configured, per the instructions above [screenshot]. You will also need to have the Windows 7 Rescue ISO loaded, which can be done within the VirtualBox configuration (before starting a VM instance) Once that’s ready, boot up the Windows 7 VM [screenshot] Sandeep is using a newer version of VirtualBox, apparently, but AFAIK, it should still work even if yours says “Sun” instead of “Oracle” When prompted, boot to the rescue CD by pressing the appropriate key. [screenshot] At the menu, select “Repair Installation” [screenshot] Select “Repair & Restart” [screenshot] I have not personally done this, but based on the comments below, this seems to work for most users. I gather that the problem is an issue of Windows 7 not liking being second banana in the MBR.

I am pretty sure that if you alternately boot raw and boot virtual, you will have to do this rescue disk procedure each time. Be careful with doing a repair on a raw boot, though, if you use Grub. Currently, I do not know of any way to use the partition for both Windows 7 raw and virtual boots without using the rescue disc fix.

Congratulations! That’s it! Tinkering and optimizing aside, you’re done.

Comments

N.B. Including the comments from the original blog post, at it included some additional contributions, questions, ideas, and feedback:

Javier - February 2, 2010

I’m getting stuck at one point:

sudo usermod -a -G disk

I’m using ubuntu Karmic, and I get an error message:

usermod: user ‘disk’ does not exist

Is there an equivalent user in 9.10?

Aaron - February 2, 2010

I did this in Karmic as well — the “disk” is actually a group, not a user. what was the command you typed, exactly? was it verbatim?

Javier - February 2, 2010

I copied and pasted from the article:

sudo usermod -a -G disk

Thanks for the quick answer!

Javier - February 2, 2010

Nevermind, I got past that. I added my user to the disk group using the user and group manager on gnome. However, now I’m stuck at adding the VDMK file: When I hit add, I find two files. I tried with both, and I get a

“Failed to open the hard disk /home/javierd/.VirtualBox/winxp-pt.vmdk.
Could not open the medium ‘/home/javierd/.VirtualBox/winxp-pt.vmdk’.
VD: error VERR_ACCESS_DENIED opening image file ‘/home/javierd/.VirtualBox/winxp-pt.vmdk’ (VERR_ACCESS_DENIED).”

Best regards

Aaron - February 2, 2010

Could you paste the command you typed? I think I know what you did wrong, but knowing your command would help me know for sure. :)

IIRC, the issue might simply be a permissions issue with your /dev/sd*devices (your hard drives). Show me the command you used to create the VDMK and I should be able to identify the problem.

Javier - February 2, 2010

Nevermind that last comment too… :) Just a correction for this CLI:

sudo VBoxManage internalcommands createrawvmdk -filename ~/.VirtualBox/winxp.vmdk -rawdisk /dev/sda -partitions 4 -mbr ~/.VirtualBox/FAKE.mbr -relative -register

Must be run WITHOUT the sudo… otherwise, the permissions will be for root only 😉

Aaron - February 2, 2010

Ah!

thank you — I will correct that above.

cheers for that!

Javier - February 2, 2010

Sorry to pester again, but I run into another issue: My Windows now asks me for activation!! >:( I have tried manually assigning the MAC ID’s in order to have them match the same as the fisical ones… but no luck! :(

Native XP still boots up without issues, only when I start it through VB I get prompted to activate it…

Any ideas?

Aaron - February 2, 2010

hmm… I didn’t have that problem.

Can you tell me a little more about your situation? You’re using an existing Windows XP installation, right?

On my system, I had a Windows partition that I was already using for dual-booting — that’s like what you have, right?

Javier - February 3, 2010

Yes, exactly the same.

Hmmm… It may have to do with licensing… mine is a OEM with a laptop. Maybe you have a retail version? For OEM, it checks some hardware serial numbers and uses that as a fingerprint for a machine… I’m guessing that because it doesn’t recognize the new values it believes I may have reinstalled it in another machine. I tried matching the MAC ID’s of the virtual interfaces, but didn’t work.

The other thing I thought of, was having my VB use both processors (I have a core 2 duo laptop) instead of 1 as I did it at the beginning… but for some reason, it didn’t work with 2, only accepted one… I may just erase the windows partition and start running everything from VB, it may solve my problem, and I don’t really use Windows for much, just to access a Microsoft CRM web… ies4linux and IE with wine are not stable.

Aaron - February 3, 2010

Ahhhh… yeah — I have a retail (well, semi-retail — I bought it from my univ.) version.

Not sure how to address that problem, but perhaps someone else on the Internet has had that experience? Windows is such a fickle beast w/r/t hardware and whatnot… it’s pretty ridiculous.

The only time I ever really boot into windows is to play the 2 games I can’t get to work in Wine, use Reason 4, and use adobe CS4. that’s it. And I rarely do those things, even.

Balwinder S Dheeman - February 6, 2010

s|winxp|/HardDisks/&|g

Matt - February 11, 2010

Great post, Aaron. I’ve had a similar setup on my desktop / workstation since I first installed Linux a couple years ago. I must admit, you make the process seem much easier than I remember it being! Definitely useful though, as I do have a couple programs that from time to time I need from that partition. I’ve created a “true” virtual XP machine for the one program I use most often, mainly because when you run the physical WinXP install you don’t get features like snapshots and saving the machine state.

Javier, I’ve encountered the exact same problem as you. I did the same thing with my laptop, using the WinXP key it came with. For some reason it just doesn’t like being booted up both physically and virtually. It always asks for activation when I’ve switched from one to the other, even though I have set up separate hardware profiles. So frustrating. For a while I was having to call Microsoft each time (over some threshold I guess) but now I’ve gotten back to where I just tell it to activate each time and it works. Annoying, but usable in a pinch. My desktop (which has an OEM licensed but self-purchased version of XP) has never once had this problem. Go figure. Good luck!

Aaron - February 11, 2010

Matt:

I can totally see why those features would be helpful — really the only reason I am doing this is so that I can pop into InDesign, Dreamweaver, or SoundForge in a pinch. :) I’ve heard that Vbox can translate 3d hardware to the vm, but never actually experimented with that yet — I’m kind of skeptical.

I’m guessing that you and Javier’s problem with activation is that you were using an OEM copy of WinXP, rather than a purchased copy? The OEM copies tend to be nerfed slightly, since they’re subsisdized.

Peter - February 11, 2010

Hello Aaron:

I followed all the steps trying to boot my Win7 partition with a few issues of typing but no big deals. The last 2 steps were:

peter@Mymachine:~$ install-mbr -f ~/.VirtualBox/FAKE.mbr
peter@Mymachine:~$ VBoxManage internalcommands createrawvmdk -filename ~/.VirtualBox/win7.vmdk -rawdisk /dev/sda -partitions 1 -mbr ~/.VirtualBox/FAKE.mbr -relative -register
VirtualBox Command Line Management Interface Version 3.1.2
(C) 2005-2009 Sun Microsystems, Inc.
All rights reserved.

RAW host disk access VMDK file /home/phoboz/.VirtualBox/win7.vmdk created successfully.

When i get the VirtualBox GUI i only found an existing hard disk named Win7.vdi so i took it and follow all the next steps.

When i started the VM i get this:

FATAL: No bootable medium found! System Halted

Could you tell me which is my mistake?

Regards

Peter - February 11, 2010

Hello Aaron:

I followed all the steps trying to boot my Win7 partition with a few issues of typing but no big deals. Some of the important steps were:

peter@Mymachine:~$VBoxManage internalcommands listpartitions -rawdisk /dev/sda
VirtualBox Command Line Management Interface Version 3.1.2
(C) 2005-2009 Sun Microsystems, Inc.
All rights reserved.

Number Type StartCHS EndCHS Size (MiB) Start (Sect)
1 0x07 0 /32 /33 1023/254/63 84434 2048

peter@Mymachine:~$ install-mbr -f ~/.VirtualBox/FAKE.mbr
peter@Mymachine:~$ VBoxManage internalcommands createrawvmdk -filename ~/.VirtualBox/win7.vmdk -rawdisk /dev/sda -partitions 1 -mbr ~/.VirtualBox/FAKE.mbr -relative -register
VirtualBox Command Line Management Interface Version 3.1.2
(C) 2005-2009 Sun Microsystems, Inc.
All rights reserved.

RAW host disk access VMDK file /home/peter/.VirtualBox/win7.vmdk created successfully.

When i get the VirtualBox GUI i only found an existing hard disk named Win7.vdi so i took it and follow all the next steps.

When i started the VM i get this:

FATAL: No bootable medium found! System Halted

Could you tell me which is my mistake?

Regards

Aaron - February 11, 2010

I don’t know how much help I can be, since I havent used Win7 at all — my first guess is that Win7 has different reqs than XP as far as the MBR is concerned; Windows tends to be very jealous and wants a monogamous relationship with your hard disks. XP wont run unless you spoof its presence in the mbr (as you’ve seen). If Win7 is even more jealous, then your best bet is to start googlin’! :) If you DO find anything interesting regarding this particular issue, I’d love to hear about it!

On the off chance its something else, can you post the result of fdisk -l for me? I might be able to help if I can see your partitiion table.

Peter - February 12, 2010

Tks Aaron!! Sure… here it is:

Disk /dev/sda: 120.0 GB, 120034123776 bytes
255 heads, 63 sectors/track, 14593 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x57c6d2a2

Device Boot Start End Blocks Id System
/dev/sda1 * 1 10764 86460806 7 HPFS/NTFS
/dev/sda2 10765 14593 30756442+ 5 Extended
/dev/sda5 10765 14429 29439081 83 Linux
/dev/sda6 14430 14593 1317298+ 82 Linux swap / Solaris

Disk /dev/sdb: 41.1 GB, 41110142976 bytes
255 heads, 63 sectors/track, 4998 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0xa8418303

Im googling and changing some parameters, if i success ill be happy to share… :)

jean.s - February 24, 2010

Hey,

Some users having problems to activating their windows-xp after migrating it to virtualbox due to connectivity problems. The original windows seems to remember the MAC-Address of your network adapter, be sure to set the mac-address of your virtualbox adapter to the one you physically have – worked for me.

Another note: If you experience Bluescreen after migration, reboot into windows, apply the MergeIDE drivers to your current windows installation. See MergeIDE. MergeIDE populates your windows installation with standard drivers which should work with Virtualbox. After having MergeIDE successfully installed, boot your windows with Virtualbox.

Cheers

Aaron - February 24, 2010

Awesome!

Thanks for the notes, Jean — very helpful!

Bogdan - March 30, 2010

Thanks for this tutorial !!

Based on it I was able to load an existing Windows Vista partition in VirtualBox OSE !!

I don’t know why you said not to use OSE. I work to a company and OSE was so much required because it is free and even if it’s old it does it’s job. Using grub image didn’t work for me (would receive “Unknown partition type 0x7″`) and I removed it from booting sequence. After creating the vmdk file I just power the machine (no grub).

The only thing though I need to repair Vista boot with Vista’s CD using Repair option. Otherwise would say that it needs repaired and cannot boot. Another thing was that I needed to activate Windows again after booting in VirtualBox with the same key it was before (key written on my computer box). I had some problems (blue screens) with the programs that had attached all kinds of drivers to them (drivers that failed in VirtualBox). After removing them from the system Vista worked, I used them anyway on Linux.

So, I have a Vista licensed activated and working in Linux. By the way I hate Windows…

Aaron - March 30, 2010

Terrific! I’m glad you were able to do it!

My reason for advising against OSE is for two reasons:

Sun does not keep the OSE up to date as much as the closed source version, so it’s going to be older.
Options such as Guest Additions are not available in OSE (this is more of an extension of #1, since I think it’s simply a version issue.)

OSE is perfectly functional, but unless you plan on hacking the code or modifiying it in some way, there’s really no reason to limit yourself to the OSE version.

I think the reason you needed to repair the boot was because Vista is very jealous about the MBR — it doesn’t like not being top dog.

Your hardware drivers issue:

Try creating a separate hardware profile. I don’t know how to do it in Vista, but in XP it’s under My Computer -> Properties. This step is somewhat critical if you plan on being able to boot into Vista directly still — separate hardware profiles will keep the drivers that work fine in solo boot working, but also allow you to disable the drivers that DON’T work in a vbox boot.

Glad to hear you got it working though, I’ll put a note up in the post.

Yazlanka - April 11, 2010

My question for you is this: How do you make a fake MBR file in Fedora 12? There is no MBR program for it that I was able to find. I have a good feeling that’s why my Windows XP will not start. I get the windows splash, but then the blue crash screen seconds later. If you can tell me where to find a program to generate the FAKE.mbr file for Fedora, please let me know.

Aaron - April 17, 2010

The program is pretty basic, you could probably download the .deb of mbr from one of the debian / ubuntu repositories, then use “alien” (yum install alien, I think) to convert it to an RPM.

I haven’t used Fedora core since FC9, unfortunately, and I don’t have it installed on any systems right now. :/

Try that out, though — I don’t think it should give you any problems. Let me know if you have any success — I’ll update the blog post with your findings. Thanks!

funicorn - April 21, 2010

I tried the above method and failed. All the preparations go well. And the MBR seems working because I can see the boot loader interface. But when booting the windows XP error shows up with file ntldr and a standard error code 0xc000000e. I think the reason is VM can not read any information from the VMDK file winxp.vmdk, because I am sure the ntldr file is there in the windows partition.

I have both windows xp and win7 installed on different partitions, and use win7 boot loader. Could that be the reason ?

Aaron - April 21, 2010

I don’t know that the instructions above will work for Windows 7, my guess is that they revised the bootloader for Windows 7 — since it’s not based on the 2000 kernel, that’s a pretty strong hunch. :)

I unfortunately don’t have Windows 7, so I have no way of testing out a new solution for it. :/

HOWEVER

The bootloader is only necessary to fake the MBR for the win XP partition — I’m pretty sure you can fake it for just the XP partition and have Win7 be completely unaware of it. What is your partition scheme for your computer? I bet we can get a VBox working for you with WinXP, but I can’t promise anything for win7.

funicorn - April 24, 2010

OK. Here is my partition list

Device Boot Start End Blocks Id System
/dev/sda1 * 1 1912 15358108+ 7 HPFS/NTFS
/dev/sda2 1913 35089 266494221 5 Extended
/dev/sda3 35090 38913 30716280 7 HPFS/NTFS
/dev/sda5 1913 15332 107796118 7 HPFS/NTFS
/dev/sda6 15333 17320 15968578+ 83 Linux
/dev/sda7 17321 21654 34812823+ 83 Linux
/dev/sda8 21655 21859 1646631 82 Linux swap / Solaris
/dev/sda9 21860 30384 68477030+ 7 HPFS/NTFS
/dev/sda10 30385 35089 37792881 83 Linux

sda1 is where windows xp is , and you see it’s also the default boot partition. I also have windows7 install in sda9, and the win7 loader is installed on sda1.

Now I use grub as the global boot loader, which means I have grub installed on MBR.

Actually thre is one thing I can not understand. I tried the above method and I can see the win7 boot loader interface. But I believe the real MBR in my disk should be the grub boot code and it’s stage1. So where does this win7 loader boot interface come from ? Or maybe the real question is : what did this command install-mbr -f ~/.VirtualBox/FAKE.mbr really do ?

Aaron - April 24, 2010

I think the fake MBR file tricks Windows XP into thinking that it was booted up normally. IIRC, without the fake MBR it will begin booting but ultimately fail. The documentation for install-mbr said that it just copies your existing MBR into a file. VBox will then use that file to spoof the bootup process.

It’s not like creating the VDMK where you have to specify the partitions and all — the fake MBR is basically a dummy file that simply needs to be there to reassure windows that “it’s ok, you’re not being booted into a virtual console, I promise,”

As far as Win7 is concerned — I don’t know if this will work or not, particularly since it’s built off of a different kernel.

One thing that puzzles me about your situation is that I think you said you are getting the boot loader (GRUB, right?) interface? I don’t get that when I do mine, even though I do have GRUB loaded for conventional dual-booting. FWIW, My linux partition is sda1 and winXP is sda4. I don’t know that will make a difference, though.

The error code you’re getting sounds really familiar, and I wish could remember it.

I just reviewed some of my resources and did a quick google search — you’re kind of pioneering new territory with the Win7 hybrid machine.

How much different were the steps listed above from those that you had to take to get to where you’re at? Were there any deviations?

when mine boots up, I see the Sun Vbox splash screen, then a black screen with “MBR” in the top left corner, then the windows XP loading splash — no grub or anything.

funicorn - April 24, 2010

First let me clarify some points

I’m using grub as the default boot loader, and have windows xp ,win7 and Ubuntu installed.
When I boot in VM, I first see the black screen with “MBR”, then the win7 boot loader interface, not the grub’s.

After reading your explanation I understand why that happens: The fake mbr boots the system to trigger the boot sector of the vmdk disk, which is exactly where win7 boot loader is located. In your case the fake mbr triggered the winxp boot loader, and if you only have one system installed, there should be no loader interface, you got into windows xp directly. If you add to the file boot.ini another line of boot entry, I think you’ll also see the loader interface, i.e., the loader interface of windows xp.

When I follow the steps, I do encountered one deviation. I ran the command sudo usermod -a -G disk, and the system said “no user disk”. So I open the users-admin gui, and added myself into the disk group. Everything else is OK.

Puzzled - June 4, 2010

At the very beginning, I am having difficulty with install-mbr.

install-mbr –force ~/.VirtualBox/FAKE.mbr

Offset must be a number: rce

Robert - June 25, 2010

Hey there, thanks a ton for the guide, and the laughs :) Only problem is I’m hitting a road block and am thinking it’s because Oracle might have changed VB around a bit since you wrote this? anyway, i’m pretty sure i’m doing everything right, but am getting this error:

Number Type StartCHS EndCHS Size (MiB) Start (Sect)
1 0x07 0 /1 /1 1023/254/63 152626 63
root@ubuntu:/home/owner# sudo chmod 666 /dev/sda1
root@ubuntu:/home/owner# sudo chmod 660 /dev/sda1
root@ubuntu:/home/owner# VBoxManage internalcommands createrawvmdk -filename ~/.VirtualBox/wixxxxxnxp.vmdk -rawdisk /dev/sda -partitions 1 -mbr ~/.VirtualBox/FAKE.mbr -relative -register
WARNING: The vboxdrv kernel module is not loaded. Either there is no module
available for the current kernel (2.6.32-22-generic) or it failed to
load. Please recompile the kernel module and install it by

sudo /etc/init.d/vboxdrv setup

You will not be able to start VMs until this problem is fixed.
WARNING: The compilation of the vboxdrv.ko kernel module failed during the
installation for some reason. Starting a VM will not be possible.
Please consult the User Manual for build instructions.
Oracle VM VirtualBox Command Line Management Interface Version 3.2.4
(C) 2005-2010 Oracle Corporation
All rights reserved.

RAW host disk access VMDK file /root/.VirtualBox/wixxxxxnxp.vmdk created successfully.

and upon following their instructions, sudo /etc/init.d/vboxdrv setup, i get this error

Number Type StartCHS EndCHS Size (MiB) Start (Sect)
1 0x07 0 /1 /1 1023/254/63 152626 63
root@ubuntu:/home/owner# sudo chmod 666 /dev/sda1
root@ubuntu:/home/owner# sudo chmod 660 /dev/sda1
root@ubuntu:/home/owner# VBoxManage internalcommands createrawvmdk -filename ~/.VirtualBox/wixxxxxnxp.vmdk -rawdisk /dev/sda -partitions 1 -mbr ~/.VirtualBox/FAKE.mbr -relative -register
WARNING: The vboxdrv kernel module is not loaded. Either there is no module
available for the current kernel (2.6.32-22-generic) or it failed to
load. Please recompile the kernel module and install it by

sudo /etc/init.d/vboxdrv setup

You will not be able to start VMs until this problem is fixed.
WARNING: The compilation of the vboxdrv.ko kernel module failed during the
installation for some reason. Starting a VM will not be possible.
Please consult the User Manual for build instructions.
Oracle VM VirtualBox Command Line Management Interface Version 3.2.4
(C) 2005-2010 Oracle Corporation
All rights reserved.

RAW host disk access VMDK file /root/.VirtualBox/wixxxxxnxp.vmdk created successfully.

i think this is the good stuff from the log:

echo ” ERROR: Kernel configuration is invalid.”;
echo ” include/linux/autoconf.h or include/config/auto.conf are missing.”;
echo ” Run ‘make oldconfig && make prepare’ on kernel src to fix it.”;

anyway, i’m looking forward to your response! hopefully sooner than later because i would love to see this baby work 😉

Aaron - June 25, 2010

Hi! Glad you enjoyed it :)

Ok — first off, which version of VBox are you using?

I had to use Virtualbox closed-source, downloaded directly from them. If you are using the Vbox OSE package, I can’t guarantee this will work.

Did you try doing: sudo /etc/init.d/vboxdrv setup as it suggested? What was the result?

Robert - June 25, 2010

Nice! Wasn’t expecting such a timely response :)

I am using this version downloaded direct from VB’s website, I believe this not to be the OSE edition as provided in the Ubuntu Software Repository. Specifically, this is what I installed: http://download.virtualbox.org/virtualbox/3.2.6/virtualbox-3.2_3.2.6-63112~Ubuntu~lucid_amd64.deb

Second, this is the error in the log file:

i think this is the good stuff from the log:

echo ” ERROR: Kernel configuration is invalid.”;
echo ” include/linux/autoconf.h or include/config/auto.conf are missing.”;
echo ” Run ‘make oldconfig && make prepare’ on kernel src to fix it.”;

Thanks!

Aaron - June 25, 2010

Let me check on that when I get home — I want to see what version I’m running. It sounds like it may just be a dependencies issue — my first guess is that you need the kernel-source package installed — If you check in synaptic for them, you should be able to load those.

I’d start there. If it doesn’t fix the problem, no harm done — it’s just space on your drive. :)

Robert - June 25, 2010

I will definitely do some more strolling around the site sometime soon, I am sure to find some more awesome content :)

So not that I am intending on being spoon fed or anything as I opened the Synaptic Package Manager and searched for kernel-source, but see a ton. Some make sense to me, like the ones with VSE, but I’m not sure which ones to enable/install.

Thanks again

Aaron - July 3, 2010

The package you want is linux-source, sorry — Try installing that package (apt-get install linux-source) and that should do it. Let me know if it doesn’t work.

Stephen - July 1, 2010

Aaron, Thanks for publishing this, I’m running into the same problem as Puzzled in comment 31:

At the very beginning, I am having difficulty with install-mbr.

install-mbr –force ~/.VirtualBox/FAKE.mbr

Offset must be a number: rce

Could you let me know how to get around that?

Thanks

Aaron - July 3, 2010

Stephen – please see my most recent comment below, or check the original article. I have updated it with the fix.

Shauna - July 3, 2010

Hey there I’ve run into a problem when I try and set up the VMDK file. Here is the command I ran pretty sure its exactly the same as what you have listed above and then I also included the output:

VBoxManage internalcommands createrawvmdk -filename ~/.VirtualBox/winxp.vmdk -rawdisk /dev/sda1 -partitions 1 -mbr ~/.VirtualBox/FAKE.mbr -relative -register

ERROR: VMDK: incorrect partition data area ordering set up by the caller in ‘/home/shauna/.VirtualBox/winxp2.vmdk’
Error code VERR_INVALID_PARAMETER at /home/vbox/vbox-3.2.6/src/VBox/Devices/Storage/VmdkHDDCore.cpp(3492) in function int vmdkCreateRawImage(VMDKIMAGE*, VBOXHDDRAW*, uint64_t)
Error while creating the raw disk VMDK: VERR_INVALID_PARAMETER
The raw disk vmdk file was not created

Thanks for any help you could provide.

Aaron - July 3, 2010

@Shauna - Can you please comment back with your partition table data? Type: fdisk -l and paste the output here. If your partition table is different from mine (certainly possible), you will have to setup that VMDK file differently.

Aaron - July 3, 2010

To everyone experiencing the offset must be a number: rce problem — I realized why this was:

In my instructions, the font doesn’t make it clear, but you must use TWO dashes (-) before the “force” for it to work. I will update my instructions above to reflect that.

Alteranately, and this is what I will change it to above, you can just use a single dash and use the “f” flag.

install-mbr -f ~/.VirtualBox/FAKE.mbr

I just tested it and it works. :)

What was happening was that it was seeing: -force and interpreting it as if you had typed: -f -orce

the -o flag specifies an offset, and rce is clearly not a number.

Hope that helps!

Shauna - July 3, 2010

Here you go. Thanks

Disk /dev/sda: 80.0 GB, 80026361856 bytes
255 heads, 63 sectors/track, 9729 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x95aa95aa

Device Boot Start End Blocks Id System
/dev/sda1 * 1 5100 40959968+ 7 HPFS/NTFS
/dev/sda2 9025 9729 5654880 c W95 FAT32 (LBA)
Partition 2 does not end on cylinder boundary.
/dev/sda3 5100 9025 31531009 5 Extended
/dev/sda5 5100 8858 30186496 83 Linux
/dev/sda6 8858 9025 1343488 82 Linux swap / Solaris

Aaron - July 3, 2010

Based on what I read on the Vbox forums here: http://forums.virtualbox.org/viewtopic.php?p=26912

It looks like you aren’t the only one with this problem. I think the crux of your problem is the fact that you’re using an extended partition, but the note that “Partition 2 does not end on a cylinder boundary” seems to be more damning.

See this comment from that link above, written by one of Vbox’s site moderators: http://forums.virtualbox.org/viewtopic.php?p=26912&sid=5a7f106e3c227378f9323e67d9b51486#p26912

The problem that you guys are having is actually because of a bug in src/VBox/Frontends/VBoxManage/VBoxInternalManage.cpp. Function partReadbuilds an internal partition table from the partition sector and then adds the extended partitions. It then does a simple exchange sort to order these, checking for overlaps etc. as it goes (lines 595-641).

What it doesn’t do is to exclude the Extended Partition Entry from these checks, and of course this does overlap and start at the same place as other partitions, so of course this validation fails. So why doesn’t fail all the time? Ah, that’s because a second bug means that these checks aren’t always applied, so if the Extended Partition is the last one, we have a case of two wrongs making a right, or maybe in this case a write

It’s a bug in VBox, I guess.

As far as what you can do:

You’re only really using 4 partitions, so the extended partition is unnecessary — If you have the luxury of being able to dust off the whole drive and starting fresh, that would be ideal.

If that’s not the case, and you are able to backup / re-install Linux, what I’d suggest is to boot into gParted [gparted.sourceforge.net], blow away partitions 3, 5, 6, then create partitions 3 (type 83) and 4 (type 82) as primary partitions.

That should fix the problem. :) Sorry it’s not something easier :/

Sandy - July 5, 2010

I’m new to open source..

I want to use my other partitions too while using virtual box… will i be able to use the other partitions…

PS:I have installed Windows 7 in virtual box Currently using Ubuntu 10.04

thnx in advance

Aaron - July 5, 2010

There is a way to do that, but it’s somewhat tricky, and potentially dangerous.

The step where you create the VDMK, you can specify other partitions with the -partitions flag. I’ve never done that PERSONALLY, but I think it would be something like: -partitions 1 2 3 4 instead of just -partitions 4 if you wanted the VBox guest to have access to partitions 1-4.

the danger here is that you should not give it access to to any partitions that are currently mounted — ie. do not specify your active linux partitions!

If, for example, your partition table was something like this:

– Linux Native
– Swap
– Linux Native
– [extended]
– NTFS (Win XP)
– FAT32 (Win 95)
– FAT16 (Win 3.x)

And you wanted the VBox session to have access to partitions 5, 6, and 7 — that’s probably OK to do. But do not give it access to partitions 1-3 (assuming those are all mounted).

If you are simply wanting to share files between Linux & windows, your best bet is to use the internal VBox “Share a folder” feature, then map a network drive within the Windows VBox session — that’ll allow you to easily copy stuff to/from the host OS (Linux). I strongly discourage mounting additional partitions unless they are absolutely necessary for the guest OS to function, however.

Gianpaolo - July 5, 2010

Hi Aaron,

Great blog post! All of the steps work very smoothly, except for the step where I was supposed to add myself to the disk group- for some reason it complained there wasn’t a group called disk. I added myself via the gui group manager though. I’m getting back to linux after a long, dark night in Windows-land (long story), and bloggers like you are one of the reasons this has been easy and fun! Thanks for taking the time to share what you know.

Aaron - July 5, 2010

Glad it’s worked well for you so far!

A number of people have had issues with the usermod command — the GUI group manager seems to be the more successful route to take; glad you were able to figure it out on your own. I’ll update my original post to reflect that.

Oh, and be sure to look into WinE — it’s changed a LOT in the past few years and with the easy installation of DirectX you can play many modern games in linux — they actually run better without all the Windows overhead. :)

Sandy - July 6, 2010

WOW dat was a quick reply… Basically all i want is to access all my installed softwares(Installed in WIndows 7) in the Windows7( guest OS), which will help me in not to reinstall the softwares again….

hope u r getting my pt. :)

i did tried d 2nd option dat u suggested(sharing) but came of no use… & plz can u ellaborate on “then map a network drive within the Windows VBox session…” thnx *_*

Aaron - July 11, 2010

Once you have the stuff in the original post set up (with your VDMK and all that), you can do this:

There is an option inside of the Virtualbox software (load it in Linux from the main VirtualBox interface). Click on your Virtualbox guest OS (Windows XP in my case — Windows 7 in yours) and click “Settings” then choose the last option, “Shared Folders”

Click on the folder that has a “+” on it, on the right side, to add a shared folder. On my system, I use /home/aaron/Public — but you can use whatever you like (/home/sandy/Documents/ for example). Once you’ve selected all the folders you want to share, click “OK” to get out of there.

when you start the Virtual Machine (not when you boot into Windows from a clean boot), you will see those shared folders under your Network Places — it should show up as if it were another computer on your home network. :)

Which Linux OS are you using? Ubuntu? Fedora core? I can give you more specific information to help you if I know what you’re running. :)

Shauna - July 7, 2010

Aaron,

Thanks for all your help I went a head and blew away my linux and the extended partition as well here is what I get when I run fdisk -l

Disk /dev/sda: 80.0 GB, 80026361856 bytes
255 heads, 63 sectors/track, 9729 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x95aa95aa

Device Boot Start End Blocks Id System
/dev/sda1 * 1 5100 40959968+ 7 HPFS/NTFS
/dev/sda2 5100 9476 35155968 83 Linux
/dev/sda3 9476 9730 2034688 82 Linux swap / Solaris

I’ve tried to get the command to run:

VBoxManage internalcommands createrawvmdk -filename ~/.VirtualBox/winxp.vmdk -rawdisk /dev/sda1 -partitions 1 -mbr ~/.VirtualBox/FAKE.mbr -relative -register

But I still get the same error output:

Error code VERR_INVALID_PARAMETER at /home/vbox/vbox-3.2.6/src/VBox/Devices/Storage/VmdkHDDCore.cpp(3492) in function int vmdkCreateRawImage(VMDKIMAGE*, VBOXHDDRAW*, uint64_t)
Error while creating the raw disk VMDK: VERR_INVALID_PARAMETER
The raw disk vmdk file was not created

Anything else I could try?

Thanks

Aaron - July 11, 2010

Yes, I see one tiny little thing you can do differently that might make it work:

You typed:

VBoxManage internalcommands createrawvmdk -filename ~/.VirtualBox/winxp.vmdk -rawdisk /dev/sda1 -partitions 1 -mbr ~/.VirtualBox/FAKE.mbr -relative -register

But try this instead (changing sda1 to just sda)

VBoxManage internalcommands createrawvmdk -filename ~/.VirtualBox/winxp.vmdk -rawdisk /dev/sda -partitions 1 -mbr ~/.VirtualBox/FAKE.mbr -relative -register

You only need to specify the device itself for the “rawdisk” option — the “partitions” option takes care of specifying the partitions on that disk. :)

Try that out — I bet it’ll work for you.

Good luck!

Sandy - July 11, 2010

Awesome tutorial…. Well For WIndows 7 users all you have to do is

“To use your Windows Installer and select Repair option…”

Nice work Aaron

Aaron - July 11, 2010

That will more than likely restore the actual MBR to boot Windows primarily, but will probably not help with getting the VirtualBox VDMK set up — what did you use that for, in your situation?

Sandy - July 11, 2010

It works for me…

Rebooted my system…& no changes were made to the actual MBR…

I have a query & i want to post a screenshot of my query…

Aaron - July 11, 2010

I think you might be misunderstanding me —

This post isn’t about doing a “dual boot” setup, where you have both Windows and Linux on the same computer. In those cases, yes, doing a “Repair windows installation” would help to fix the problem by resetting the MBR to be windows-friendly.

This post is about using VirtualBox to run an existing Windows partition while currently booted into Linux. ie. I boot into Linux, click a single icon in my applications menu, and then Windows boots up in a separate window. Check out here and here for examples of what I mean.

Normally, you have to create a separate “disk image” to use with VirtualBox, but with the instructions above, you can use an existing partition completely, without modifying it.

As far as how to post a screen shot — the comments area unfortunately doesn’t support uploading, but if you can find some way to post it online (twitpic.com, imageshack, and other online storage systems will all work) you can link to it from here and ask your question. :) I’d be happy to take a look!

Sandy - July 12, 2010

http://picasaweb.google.com/sandyridgeracer/DropBox?authkey=Gv1sRgCJLU8Kbnt97oNA#5492938771966452978

I want the VBox to utilize the whole screen..i.e. i dont want those white void spaces…is there any way by which i can chage the “resolution”.

in previous posts I meant “to boot your windows7 installer dvd in VBox and then select repair ”

Aaron - July 12, 2010

Ah! Ok — glad we cleared that up. That’s good to know, I’ll add it to the original post above!

As far as VBox utilizing the whole screen, there are a couple things you can try:

– While in your session, click on “Machine” then “Fullscreen Mode” – Install “Guest Additions” (requires you to not use the OSE version), then enable “Seamless Mode” (also found under “Machine”

Fullscreen mode may require Guest Additions to enable it, I’m not sure.

Fullscreen mode will use up all your screen real estate, but it might take a couple minutes before the Windows desktop auto-resizes to fit the screen.

Fullscreen mode works kind of like Remote Desktop in Windows — there’s a tab at the top that will emerge if you move your mouse to the top of the screen. You can also just press the Host key (Right-Ctrl on my system) and “F” at the same time, that’ll exit out of it.

Hope that helps!

Sandy - July 12, 2010

thnx a lot….

Sony AK - July 30, 2010

I am using Ubuntu 10.04 that installed using Wubi on my Windows Vista (double booting). Now mostly I am working in Ubuntu and I want to access Windows Vista using VB. I just share Ubuntu and Windows on the same partition /dev/sda2 mounted to /host, so how can I run my existing Windows Vista using VB inside my Ubuntu? Please help :)

Aaron - July 30, 2010

My advice here will be somewhat limited as I have never personally loaded Windows Vista in Linux VBox — but I know a couple of the commenters have — be sure to read through the comments for some extra insight. (Check Bogdan’s comment specifically)

As far as how to do it:

Start out by going through the steps listed above — AFAIK, the process of getting a Vista VDMK registered with Virtual box is nearly identical to doing it with Win XP except there are some last minute finessing steps w/r/t getting Vista to play nice with being second banana.

Once your VDMK is registered with VBox, you can adjust the settings to establish a specific folder / network drive inside of your Vista guest session so that you can access the host (Linux) — this is useful for when you want to copy files into or out of the VBox guest session.

Let me know where you’re at with the process. Don’t be too intimidated by it — the commands listed above aren’t really capable of inflicting any damage on your system. :) You can do it!

Priss - August 4, 2010

Hi, I tried everything and can’t get passed the winxp splash screen. It just freezes there, no error messages or anything. When I disabled the intelppm in the registry I got the blue loading bar to appear, but nothing happened. Any help for me?

Aaron - August 14, 2010

You can’t get past the splash screen in a VBox boot? How long have you waited? For some reason, it occasionally takes me 5 minutes to boot completely — other times it’s much quicker.

I recommend “Saving the Machine State” when you quit, rather than shutting down outright — although I don’t know what kind of effect that would have if you saved the machine state and then booted into it normally. I don’t think it would be a problem, but I have not experimented with that yet.

So for now — try booting it and just let it sit for a while — give it 10-15 minutes (go make some coffee or something :) and if nothing changes STILL, we can try something else.

oh btw — be sure you “Create a new Hardware Profile” — I think it’s in the same area as Device Manager? (Right-click My Computer, select “Properties”, then “Add new hardware profile”). Name one of them “VirtualBoxBoot” and the other “NormalBoot” — set it to default to whichever one you are likely to use more often (I generally select the VBoxBoot one simply for convenience).

VirtualBox uses some different hardware adapter configurations than a normal boot will, since it has some of its hardware virtualized. Having separate hardware profiles should decrease your boot time substantially, I think. You will need to boot into it successfully once before you can get the VBox configuration made though.

martin - August 14, 2010

yo, hey, nice tutorial so far, but for me it seems not to be as easy as it is for some guys. i have windows 7 on partitions 1, 2 and 3 and all commands in linux worked without any flaws. still i cant get past the windows-start-manager which appears directly after the mbr-screen.

@ sandy: how did you repare inside the vb? thx

Aaron - August 14, 2010

Make no mistake — it’s challenging for everyone. :) Success seems to be a direct function of persistence, in this case.

IIRC, Sandy got it to work by booting the Virtual Machine and telling it to “Repair the MBR”? If we can get a final verified answer on this, I’ll add it to the original post.

martin - August 14, 2010

Well, reparing with a virtual cd-image works very well, as long as I stay inside the vb. Have to test starting the system from grub though…

Aaron - August 14, 2010

Martin — please post back here if it was successful, and if you remember, the exact steps you took — I’ll add it to the OP and credit you and Sandy for figuring it out. :)

martin - August 14, 2010

Hey Aaron, it’s not the final solution we hoped to find.

But as long as you only want to migrate it’s what you need. If you still want to use it natively from time to time, then you will have to repair every time you change the way of starting the system.

The steps:

Produce an iso-image from your recovery disk or get one in the net.
Go to the virtualbox settings, storage (I don’t know the exact name, have the german language installed), select the cd symbol, go to the drive dialog and choose your image.
Start the virtual machine –> a few options pop up, choose the one one top which is the repair option, restart and you are there.

When I started windows from scratch it updated the 27k entries in the registry and proposed to repair itself.

The only thing which is left is to skip repairing and see if the virtual way is still working that way.

Cheers

Aaron - August 14, 2010

That’ll work for now – that sounds like what Sandeep did too. Will add to OP later today, thanks!

martin - August 14, 2010

“Use recovery tools that can help fix problems starting windows” 😀

martin - August 14, 2010

My persistence is at it’s end.

It would be good to somehow find the diff between the windows-system before and after repairing, but I dont have the knowledge nor the time to do so atm.

Greetings

Albert - August 17, 2010

Thanks a lot for your good, plain and so fun explanation, the better is that worked.

The only one thing I made different was to make me a disk user: sudo usermod -a -G disk says me that there wasn’t a user called : disk , but i made through the GUI .

Well be sure whenever there is an award for better explanation on that subject I will vote you.

Aaron - August 17, 2010

Other people have had trouble with that too, I think I will add some additional instructions to the post to cover that.

pim - August 17, 2010

Hello

I tried this tutorial but once I have finished everything and I try to launch my windows 7 into VirtualBox, I only see :

MBR 1FA:

What’s wrong ? Thanks

Aaron - August 17, 2010

Pim — Please see the previous comments by both Martin and Sandy — I think those will provide you with the info you need to solve it. :)

Sandy - August 17, 2010

oops!!!

Thanx to aaron, i hv been using windows 7 in the virtual box…

One of the problem with this technique is dat u hv to keep on repairing..

By repair I mean :suppose if u r using windows 7 in VB and den u plan to use the actual windows installed(by rebooting and selecting windows 7 from d grub),

den ur windows will automatically repair it once u boot it.

Now one more scenario is dat u r using d actual windows and u plan to use win 7 in VB den u hv to manually repair it.This is done by creating an WINDOWS INSTALLER ISO file and boot it in VB and boot windows in VB using dat ISO file and finally repair it.

queries are welcome… can also post screenshot if required…

Sandy - August 17, 2010

@pim: just create an iso file of windows installer and boot it in your VB and repair it.This will fix the error… :)

ur emal-id…

Mohammad - August 28, 2010

I tried this with windows 7 and got it working. Only had to repair once. Also have successfully created an image from it and can use it as a regular vm image with everything in it. Thanks a lot for this post.

Ken Ng - September 23, 2010

very nice tutorial! Thanks! :)

just a small correction: missing user-name from usermod

sudo usermod -a -G disk

then perfect!

Aaron - September 23, 2010

Thanks for the feedback, I’ve fixed it and credited you for the spot!

dim - September 23, 2010

First of all this is a great tutorial, the most informative in the web regarding how to boot an already installed windows partition while at the same time being able to boot it in the normal way when you want. I am also trying to implement this,however i get the following

VBoxManage internalcommands createrawvmdk -filename ~/.VirtualBox/winxp.vmdk -rawdisk /dev/sda -partitions 1 -mbr ~/.VirtualBox/FAKE.mbr -relative -register

Oracle VM VirtualBox Command Line Management Interface Version 3.2.8
(C) 2005-2010 Oracle Corporation
All rights reserved.

ERROR: VMDK: could not open raw partition file ‘/dev/sda1′
Error code VERR_DEV_IO_ERROR at /home/vbox/vbox-3.2.8/src/VBox/Devices/Storage/VmdkHDDCore.cpp(3649) in function int vmdkCreateRawImage(VMDKIMAGE*, VBOXHDDRAW*, uint64_t)
Error while creating the raw disk VMDK: VERR_DEV_IO_ERROR
The raw disk vmdk file was not created

I am running Red Hat Enterprise 5.5, and i am trying to boot my native windows xp partition. (my hard drive is RAID with another hdd in a mirror configuration). Windows are indeed in partition sda1 while i have fixed the permission according to the instructions provided here, however i get the message above. Any ideas?

thanks a lot

Aaron - September 24, 2010

Dim, Thanks for stopping by!

The first thing I would check is to see whether or not you have that partition already mounted. I forget if that makes a difference or not, but it’s worth trying.

The second thing I would check is to (a) ensure that you have added yourself to the disk users group (or the equivalent for RH), and (b) that the disk users group (or its RH equivalent) has full access to that particular partition.

It’s been a few years since I’ve used RH / FC, so the idiosyncratic differences between Deb/RH systems aren’t so fresh in my mind now. Check those things first then come on back. :)

Rodimus - September 30, 2010

To poster #9 ‘dim’

I had the same error, and it was because I forgot to log out then log back in to reset my permissions. After that everything was good.

BobC - October 19, 2010

WinXP is asking me to activate it again. Do I need to do this, or is there a way to make it see the activation I’ve been using for my physical boot?

Aaron - October 20, 2010

Are you using the OEM version? I think some other people were having similar versions.

You can try activating again, I don’t know firsthand what the consequence will be, however.

Marco - October 29, 2010

Aaron thanks a lot for this article.

Everything went fine, including Windowns 7 32bit Rescue CD iso image.

But when it goes to starting the Win7 itself, the startup crashes after the initial logo, gives a super fast green screen saing something about the display (wasn’t able to take the screenshot despite tried several times). I’ve tried changing display settings both on VB and Win7, no success :-(

The strange thing is the rescue CD flows just fine in X mode

Using Ubuntu 10.10, VB 3.2.10, Windows 7 Enterprise

Any idea?

Cheers!

Karlabob - November 25, 2010

Hi!

Thanks for this awesome and very easily readable guide. I have the same problem as pim with the text MBR 1FA, I have no ideal how to create a windows installer .iso or a windows system rescue CD, any help?

Aaron - November 27, 2010

Karlabob,

Most CD Burning applications (Nero, Brasero, etc) should allow you to “create a disc image” of an existing CD. This is only really necessary if you are using Windows 7 (or, based on some comments I’ve read, Windows Vista). A Windows XP installation shouldn’t need it, but it’s a good idea to have one around just in case anyways.

Can you be more specific about the error you’re getting?

Randy - December 2, 2010

Your guide was very easy to follow and I received no errors until I tried to load the vmdk file in VirtualBox. I am new to this, so I may have miss something.

I keep getting the following:

Failed to open the hard disk
/home/xxxxxx/.VirtualBox/HardDisks/winxp.vmdk.

Could not open the medium
‘/home/xxxxxx/.VirtualBox/HardDisks/winxp.vmdk’.

VD: error VERR_ACCESS_DENIED opening image file
‘/home/xxxxxx/.VirtualBox/HardDisks/winxp.vmdk’
(VERR_ACCESS_DENIED).

I believe my problem is that it appears to be two Virtual directories on my system. (not sure, but that is how it looks to me) if I issue a ~/.VirtualBox it takes me to a directory where the files I created are there. There appears to be a hidden directory under my home/xxxx/.VirtualBox/HardDisks it does not contain the files I created. I then went back to ~/.VirtualBox and under that directory there is not HardDisks directory like the one VirtualBox sees when adding hard drives.

Am I missing something from you guide? If I do have two different directories, what is the best way to correct?

Aaron - December 2, 2010

Randy let me take a look at my configuration and ill get back to you here.

Ahmed - December 9, 2010

WOW. It worked!

Professional Windows 7, 64 bit, existing installation running on Ubuntu Linux 10.04, 64 bit.

To Aaron and Sandeep: major MAJOR props.

Cris - January 7, 2011

Hi guys,

amazing tutorial!! Got Win7 Home 32bit running within Ubuntu 10.10 ALMOST flawlessly.

Almost? Here’s why…

As I’m pretty new to Win7 I didn’t know that it creates it’s boot record on a partition seperate from the actual Windows partition. So what you actually have to do for creating a raw VMDK for a Win7 system is to integrate this pretty small boot partition to the vmdk file as well.

For me for example the boot record was located on /dev/sda1, the Windows partition on /dev/sda2. When creating a vmdk only for /dev/sda2 I could actually start the virtual machine and everythin with it, but when trying to repair the windows partition, the rescue CD said it wasn’t able to repair.

Creating a VMDK for both /dev/sda1 and /dev/sda2 everything went fine, I could repair my Windows partition and run my Win7 virtual box.

Now the only thing I have to do is see what happens if I reboot the whole system… Hopefully the Windows repair didn’t mess things up and destroy my GRUB or something…

I’ll get back to you as soon as I tried this. Wish me luck and thanks again for the great tutorial!

Sudev - January 20, 2011

You created two vmdk as two commands or one command?

On by Netbook partition 1 is the small patition and then partition 2 is the main win7 partition

I have tried using -partitions 1,2 parameter but no dice. Either I get failure with terminal showing MBR:

Or I get Win7 repair disk message. Since the netbook does not have internal DVD/CDROM and external USB one is not getting recognised as a boot device I am stuck.

On positive side Win7 continues to dual boot from grub.

Aaron - January 24, 2011

IIRC, the -partitions 1,2 flag doesn’t create two VMDKs, it just specifies which partitions the VMDK should have access to. All partitions accessed must NOT be mounted at the time the VMDK is launched.

Can you paste specifically what your error message was?

Thanks!

Alex - May 31, 2011

Here’s what I did for Win 7 Home Premium (64-bit running on a Lucid 32-bit PAE host) for VirtualBox 4.0.8 from Oracle (on an Asus UL30A, if that matters)

Follow all steps above, except Instead of creating one VMDK, you create two for Win 7: One with an MBR, one without an MBR.

For example, if your Win 7 partition is /dev/sda2, you would issue the following commands:

VBoxManage internalcommands createrawvmdk -filename ~/.VirtualBox/HardDisks/Win7x64-mbr.vmdk -rawdisk /dev/sda -partitions 2 -mbr ~/.VirtualBox/FAKE.mbr -relative

AND

VBoxManage internalcommands createrawvmdk -filename ~/.VirtualBox/HardDisks/Win7x64-nombr.vmdk -rawdisk /dev/sda -partitions 2 -relative

Then, you attach the Win7x64-mbr.vmdk as a primary IDE device, and Win7x64-nombr.vmdk as a primary SATA device (assuming you have a SATA controller, that is).

Here’s how (potentially) to avoid the issue of needing to reactivate your Windows installation: http://forums.virtualbox.org/viewtopic.php?t=9697

Follow the DMI decoding steps, then apply the DMI info to your virtual machine using the shell script there. It looked like it worked at first, but now I’m being told that my copy of windows is not genuine :(

It seems that 4.0.8 is painfully slow compared to vbox-ose 3.2.8 using a raw disk partition. It took about 5 minutes for explorer.exe to launch after finally booting. (Then again, what should I expect from running a 64-bit VM on a 32-bit host with a measly SU7300 processor?)

Of course, if anyone sees problems with using the same partition on two different controllers, please reply – I’m pretty sure that the IDE with an MBR just acts as a stepping stone to get the non-MBR SATA one working.

Cris - January 7, 2011

So, GRUB still there and switching between VirtualBox and DualBoot is fine.

However there seems to be a problem with booting WIn7 directly from the scratch (=no virtualbox) when the VirtualBox Guest additions are installed. WIndows boots fine, however the mouse seems to stick to the lower left corner and after a minute or so Windows shuts down with a bluescreen of death.

After booting from VirtualBox and uninstalling the guest additions however I could start Win7 from the scratch as if nothing happened… (apart from repairing of course 😉 )

Did anyone else try to use the guest additions for such a dual boot configuration and run into similar problems? Not being able to use them seems like a pretty big deal breaker to me, but maybe there’s a way to circumvent this problem… Maybe it is possible to temporarily disable the additions before booting from the scratch or something?

Sudev - January 20, 2011

Doing this for Win7

Okay I get to boot but then Win asks for boot repair disk.

Problem: I have a netbook as only way I can connect DVD drive is by USB cable. This is not recognised as boot device in BIOS.

Sandy - February 2, 2011

Does anybody has windows 7 ultimate recovery iso file?(pls upload the file and post the link)

I dont want to use my 2.3GB windows installation iso file…

Aaron - February 2, 2011

Provided it’s not a copyright/license violation to post that ISO online, I would be happy to host it here and put the link in the main blog post.

Anyone that has it should comment here with an email (put it in the email field) where they can be contacted and we can work out the logistics.

Michael - February 4, 2011

I did all the steps and set up the machine without any hiccups, until I tried running the machine. I am getting a terrible blue screen error on boot within VirtualBox. It’s issuing a stop command, which looks like this: STOP: 0x0000007B (0xF78A6524, 0xC0000034, 0x00000000, 0x00000000)

I did a google search but couldn’t find anything concrete pertaining to virtualbox.

Aaron - February 6, 2011

Hi Michael!

Can you give us some more information about your setup? What’s the host OS (the one that’s running Virtualbox) and which Windows OS are you running as a guest OS? also — Are you using VirtualBox open source edition, or closed-source, and which version? Do you have anything funky with your partition table?

deps - February 17, 2011

Excellent tutorial!

I’m trying to configure VB to run windows from ubuntu 10.4

Initially I had windows installed, then I partitioned my hard drive to run ubuntu.

I need to run windows software from ubuntu using VB.

I went through the setup smoothly until I got to the repair phase. Everytime I click repair I get an error telling that my current iso image is not compatible with my windows installation. But I’m using my installation CD!

Any help will be greatly appreciated :)

I print my list of partitions:

Oracle VM VirtualBox Command Line Management Interface Version 3.2.12
(C) 2005-2010 Oracle Corporation
All rights reserved.

Number Type StartCHS EndCHS Size (MiB) Start (Sect)
1 0x07 0 /32 /33 12 /223/19 100 2048
2 0x07 12 /223/20 1023/254/63 152485 206848

Aaron - February 17, 2011

What version of Windows are you using? This will matter greatly. Windows XP seems to be, in my experience, far easier (or at least far more cooperative) with this method than Windows 7 has been.

Tony - February 25, 2011

Great post!!! But I tryed everything and the best I acchieved was a Starting Windows screen running forever.. :S

Any tip? I’m running dualboot with Windows 7 Pro 32bits and Ubuntu 10.04 32bits, with VirtualBox 4.0.4.

Before you ask me to change the VB version I advice you that I’ve already it done… I tryed 3.12, 3.10 and 3.8 build releases, and OSEs too.

Thanks, Tony

Here are some extra informations:

$ fdisk -l /dev/sda
Disk /dev/sda: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x08000000
Device Boot Start End Blocks Id System
/dev/sda1 1 14 112423+ de Dell Utility
/dev/sda2 * 15 1292 10258432 7 HPFS/NTFS
/dev/sda3 1292 12118 86961668+ 7 HPFS/NTFS
/dev/sda4 12118 60802 391053313 5 Extended
/dev/sda5 12118 59676 382009344 83 Linux
/dev/sda6 59676 60802 9042944 82 Linux swap / Solaris

$ VBoxManage internalcommands listpartitions -rawdisk /dev/sda
Number Type StartCHS EndCHS Size (MiB) Start (Sect)
1 0xde 0 /1 /1 13 /254/63 109 63
2 0x07 14 /5 /56 1023/254/63 10018 225280
3 0x07 1023/254/63 1023/254/63 84923 20742144
5 0x83 1023/254/63 1023/254/63 373056 194666496
6 0x82 1023/254/63 1023/254/63 8831 958687232

About partitions:

sda2 –> Windows 7 Pro 32bits (RecoveryImage)
sda3 –> Windows 7 Pro 32bits
sda5 –> Ubuntu 10.04 LTS 32bits

Aaron - March 1, 2011

Hi Tony!

I really can’t make any warranty about this method working with Windows 7, unfortunately. I was able to get it successfully working with Windows XP, and it seems that some others have had luck with getting a Win7/Ubuntu hybrid going, but it’s rather inconsistent. Windows 7 is a bit more greedy w/r/t how it deals with booting virtually/normally.

Have you tried Sandeep’s fix with the recovery disk?

Maho - March 4, 2011

Hi, I followed up all the instructions and everything looks OK. but right after the Boot window ( the one with the blue bar) I got a black screen and nothing happens.

I’m using VB 4.0.4 on Ubuntu 10.10 and Windows XP as guest

Any ideas?

Aaron - March 5, 2011

How long did you wait? Sometimes it would take me as much as 5 or 10 minutes to boot into Win XP (either normally OR vbox). Was there any feedback at all aside from the screen being black?

Maho - March 7, 2011

I leave it booting for a couple of hours. Checking the logs I found this

aRC=E_ACCESSDENIED
aComponent={Console} aText={The virtual machine is being powered down}

Maho - March 7, 2011

More information I’ve got from logs

VBox.log

18:10.223 Changing the VM state from ‘DESTROYING’ to ‘TERMINATED’.
18:10.408 ERROR [COM]: aRC=E_ACCESSDENIED (0x80070005) aIID={515e8e8d-f932-4d8e-9f32-79a52aead882} aComponent={Console} aText={The virtual machine is not powered up}, preserve=false

VBox.log.1
00:10.047 Guest Log: BIOS: Booting from Hard Disk…
00:16.709 RTC: period=0x200 (512) 64 Hz
00:16.888 Display::handleDisplayResize(): uScreenId = 0, pvVRAM=ae7e2000 w=640 h=480 bpp=0 cbLine=0x140, flags=0x1
00:24.733 PIIX3 ATA: LUN#0: IDLE IMMEDIATE, CmdIf=0xef (-1 usec ago)
00:24.733 PIIX3 ATA: LUN#0: aborting current command
00:29.421 OHCI: Software reset
00:29.421 OHCI: USB Reset
00:29.421 OHCI: USB Operational
00:31.304 OHCI: USB Suspended

VBox.log.2

01:52:36.581 Changing the VM state from ‘DESTROYING’ to ‘TERMINATED’.

Gary - March 11, 2011

THANKS! for the great instructions.

I just installed virtual box 4.0.4 from the Oracle website.

Make sure your type 7 partitions are not mounted and that the correcponding /dev/sd.. devices have the permissions as specified in the instructions.

I ran into problem when creating a vmmk file. My C and D drive were partitions 2 and 5, so I thought that I could get away with -partitions 2,5. When I got all done I got a MBR 25FA that I was not able to get it to boot from. But I also had a restore partition, partition 1, that was also type 7. So I created my vmdk again using -partitions 1,2,5 and this worked great. So I would suggest using all of your type 7 partitions in the -partitions option or at least including the windows restore/diagnostic/whatever_it_is partition usually in partition 1.

One thing you will notice is that there is no longer a -register option to VBoxMange. Just use the command in the instructions and leave this option off. I called my vmdk file ~/.VirtualBox/winxp.vmdk and noticed that it created this file along with another file ~/.VirtualBox/winxp-pt.vmdk

When you get to the “Load your new VDMK” step you will need to use the browse option and select the winxp.vmdk file, not the winxp-pt.wmdk.

This resulted in windows booting right into vVrtualBox.

While booting I got a couple of “installing new hardware” messages and complaints about the video not being 32 bit. I would suggest ignoring all these and install the guest addtitions instead. Selecting “devices->Install guest additions” from the VirtualBox menu. This will install drivers for your video (and something else I don’t remember) and thus clean up your device manager yellow “?” that result from the driver changes that running in VirtualBox requries will cause. It will set your resolution to 800×600, but in windows just right click on the background, choose properties and then settings and set the resolution to what you want it to be.

If you need a “ctrl-alt-del” to login to windows, use the “Machine->Insert Ctrl-Alt-Del” from the VirtualBox menu.

I had previously run windows natively before I started the process and set up a second hardware profile for VirutalBox which I selected during boot. You do this by running “sysdm.cpl” from the run menu, selecting “hardware” and then “hardware profiles”. Use the “copy” button and copy your current profile and name the new one “virtualbox” or whatever you want. Then when windows boots in virtualbox, you will get a boot menu where you can chose the old one (which you use when you boot natively) or the “virtualbox” profile when you boot from VirtualBox. This keeps windows from wanting to switch your drivers back and forth. You can order your profiles using the up and down arrows on the “hardware profiles” pop-up and select the “select the first profile listed if I don’t select a profile” if you rarely use one of the profiles. It is safest to chose “Wait until I select a hardware profile” so that you won’t miss the menu and end up booting into the wrong profile and have windows mess with your drivers.

Steps - April 13, 2011

Hi there!

First of all: great tutorial!

But then: I followed your steps exactly, including the steps recommended for Windows 7 (as this is what I want to use in my VirtualBox, hosted by Ubuntu 10.10 64 bit), and finally got to boot the system. But right after the “Repair & Restart” part, I got a BSOD (which causes the system to reboot immediately) that won’t disappear (well, it re-appears every time I boot).

I then tried to boot the system for real (I read somewhere that the MergeIDE-fix offered by VB could help), but the system seems to be broken beyond repair (the recovery won’t do anything).

Can you please help me with that? Do you need any more info?

Keep up the great work!

heix - April 26, 2011

Great tutorial! Worked like a charm.

Got a Win7 64bit Home Professional, with an ACER Recovery Partition as partition 1, the “boot loader”(?) partition as 2, and the actual win7 partition as 3, so i did a “-partitions 1,2,3″. Had to use the .iso of the win7 recovery disc afterwards.

Btw: On Kubuntu 10.10 64bit it seems to work with Virtualbox-OSE 3.2.8, no need to use the “non-OSE” .deb from virtualbox.org. Also, I didn’t do the chmod on the device nodes, only added myself to “disk”.

Aaron - April 26, 2011

Thanks, Heix! I’ve added your comments to the guide, above!

Fran - May 10, 2011

when I try to boot appears black window saying:

Windows failed to start. A recent hardware or software change might be the cause. To fix the problem: ….

My system uses 3 partitions, I assume 1 and 3 belongs to the HP recovery system. So I guess I may use the win7 .iso recovery disc as you said.

How should I do that?

Fran 0 May 10, 2011

Thanks for the tutorial

My system have 3 windows partitions, two of them I guess should be the HP recovery system.

I did not reach to boot, Windows said:

“Windows failed to starts…
1. Insert your windows installation disc and restart
2. Choose your language settings
3. Click “repair your computer”
If you do not have this disc, contact your system adminstrator or computer manufacturer for assistance.
File: windwossystem32bootwinload.exe
Status: 0xc000035a
Info: Attempting to load a 64-bit application, however this CPU is not compatible with 64-bit mode.”

Thanks in advance.

Aaron - May 10, 2011

To do the ISO method, you’ll need an ISO file of the Recovery disc, then specify to Vbox that you want to “load it” in the settings for your virtual machine (before starting the session).

The “Info:” error at the bottom is puzzling, I’m not sure what that means, exactly, other than that Win7 is 64-bit and for some reason it thinks the virtualmachine is not.

Bear in mind that when you boot into native Win7 (ie. not in a virtual machine) you will need to do the recovery disc thing again; it seems to require it anytime you switch the means which you boot into win7 with.

€quiman - May 2, 2011

i had the same problem… i just restart and the problme it’s solved. Take a look again to your windows partition… and if they are mounted automatically, then unmount it.

DP: Sorry my english isn’t vey good.

Aaron - May 2, 2011

It’s no problem, thanks for stopping by. :)

JoseFrancisco - May 5, 2011

Hi, I have installed suse 11.4 and vbox 4.0.6 r71344, as i didnt have install-mbr, I did the mbr file with

sudo dd if=/dev/sda of=/home/myuser/VirtualBox.mbr bs=512 count=1
sudo chmod 666 /home/myuser/VirtualBox.mbr
sudo chown myuser:users /home/myuser/VirtualBox.mbr

then i used it for the vmdk file and everything else, but when i start the winxp in virtualbox i just have black screen with

Error No operating system
FATAL: INT18: BOOT FAILURE

any idea pls?????

Aaron - May 10, 2011

Hola Jose,

I’m really not sure how to fix this problem; I haven’t used Suse in a LONG time and haven’t experimented with methods outside of those listed in the original article. Perhaps another commenter will be able to pick that up, though?

JayBofMA - January 25, 2012

@jose – Were you able to resolve the mbr issue under SUSE?

John Westerdale - May 11, 2011

Awesome tutorial, Aaron. When would you pick Wine vs Virtual box approach?

Am assuming the VMDK-based OS can’t see the Ubuntu file systems.

Aaron - May 11, 2011

Wine is preferable, because it doesn’t block out system resources like VM’ing does, but it’s not always an option for certain software apps that require libraries that aren’t well-implemented yet.

VM’ing Windoze as a guest OS works well when you have (a) an abundance of resources (ie 2 or more CPU cores, although Ideally you want at least 4) and (b) applications that can only natively run in Windows. You can configure most VM programs to make the host OS (Linux, in this case) expose one or more directories to the guest OS, often as “mapped drives” (allowing internal copying, which is much faster) — if you configure the network access for the guest OS to be bridged, the guest appears as its own entity on your network, and can also access any network shares on the guest OS, although it will do that via the router first.

I’m also working on a tutorial for VMWare Express, which I’ve found to be a little better performing than VBox.

Ted - May 3, 2013

We’ve been using this quite successfully, but would prefer VMware Player. Do you have any instructions for that? I found several dead links to an answer on advicesource.org, but those all depend on downloading a vmdk file that is no longer available. Any idea when you’ll have your guide ready?

Aaron - May 3, 2013

Ted,

I would think that you could use the instructions in this guide to generate the VMDK file, and then reference that with VM Ware Player? I don’t know for certain that all VMDKs are created equally, but it seems possible.

Brynley McDonald - May 15, 2011

Whenever I try to add the vmdk to a new machine I get this

Result Code: NS_ERROR_FAILURE (0x80004005)
Component: Medium
Interface: IMedium {9edda847-1279-4b0a-9af7-9d66251ccc18}
Callee: IVirtualBox {d2de270c-1d4b-4c9e-843f-bbb9b47269ff}

I have no idea what it means. I’m using virtualbox on 11.04 and trying to get a Win 7 x64 image working. I’ve already tried making the image again.

Howard - May 15, 2011

I am also getting this error, attempting to load a 64-bit version of windows 7 on 11.04.

Howard - May 15, 2011

EDIT: I figured it out. Re-do the whole process from the chmod 660, 600 step. but make sure you’re not the root user at any point.

Aaron - May 16, 2011

Howard, can you clarify what you mean by that? I will incorporate it into the original post. Do you mean that you did not use sudo? Or something else?

Stef - May 29, 2012

It seems that the device (e.g. /dev/sdaX) that contains the Windows partition must have permission 660, and make sure your user is actually in the group disks.

You can do ls /dev/sd* -als to make sure of this.

Hari - February 3, 2012

I too get this same error. Anyone figured out a solution? Using Arch 64bit as host and XP 32bit as guest. VirtualBox is verson 4.1.8*

Mahkoe - July 4, 2011

It says Invalid parameter ‘-register'`.

It also says (when I enter the create vmdk command without -register, forcing me to create a new disk) that

The medium ‘/home/username/.VirtualBox,winxp-pt.vmdk’ can’t be used as the requested device type.

I also get this message for the other vmdk file.

Aaron - July 4, 2011

Did you use a , in the filename? That might be why. Also please paste the full command you typed so i can help :)

Mahkoe - July 6, 2011

No, that was just a typo in the comment. The whole command I used (to create the VMDK file) was

VBoxManage internalcommands createrawvmdk -filename ~/.VirtualBox/winxp.vmdk -rawdisk /dev/sda -partitions 1,2,3 -mbr ~/.VirtualBox/FAKE.mbr -relative

Neither of the two vmdk files created are accepted by vb.

Aaron - July 11, 2011

hmm…. that’s really weird.

Are you using a very recent version of VBox? Since Oracle bought out Sun, it’s possibly they’ve changed things around, although that would be really strange.

You can certainly leave off the -register flag and just register the VMDK with the VBox gui manually — though it looks like you’re still having problems even without that.

One thing I find strange is that the error message is saying: /home/username/.VirtualBox,winxp-pt.vmdk when your command was ~/.VirtualBox/winxp.vmdk

In the former, there is a “,” between “Box” and “winxp”, and also a “-pt” appended to “winxp”. Was that a typo or was that the actual output?

Göker Acar - July 13, 2011

Same problem here

Raymond - March 27, 2013

Same problem as well, except I’m using windows 7

if i use the vdmk file without the -pt at the end I get this error:

The medium ‘/home/tasty/.VirtualBox/win7.vmdk’ can’t be used as the requested device type.

if i use the pt vdmk file I get this error:

Failed to open the hard disk /home/tasty/.VirtualBox/win7-pt.vmdk.
Could not get the storage format of the medium ‘/home/tasty/.VirtualBox/win7-pt.vmdk’ (VERR_NOT_SUPPORTED)

Raymond - March 27, 2013

I figured it out, you get this error when you create a VDMK without logging in and out after changing your permissions.

aman sharma - August 22, 2011

thanks a lot for the post.

Jspelunker - August 30, 2011

I’m using a 64-bit 10.04 Ubuntu host and attempted for two days the process described above for a 32-bit Vista guest, but all I get is a black screen of failure in starting the virtual machine in VirtualBox. Is it possible to virtualize a Vista OS?

Chrystian Townsley - September 24, 2011

Im also gettin the


FATAL: INT18: BOOT FAILURE

error using debian and win7 ult x64

Ken - November 20, 2011

On Ubuntu 11.10 I get:

$ install-mbr –-force ~/.VirtualBox/FAKE.mbr
Usage: install-mbr [options]
Try ‘install-mbr –help’ for more information.

$ install-mbr –-force /tmp/FAKE.mbr
Usage: install-mbr [options]
Try ‘install-mbr –help’ for more information.

Aaron - November 21, 2011

Be sure that you are using the correct number of dashes for the option flags. If you use the full name “force”, you need 2 dashes. If you use just “f”, you only use one dash.

Rob - December 26, 2011

Hi, tutorial goes well up until i try to create the VMDK, at which point I use the command:

VBoxManage internalcommands createrawvmdk -filename /home/gamedrift/.VirtualBox/winxp.vmdk -rawdisk /dev/sda1 -partitions 5 -mbr /home/gamedrift/.VirtualBox/FAKE.mbr -relative

and get the following result:

VBoxManage: error: VMDK: incorrect partition data area ordering set up by the caller in ‘/home/gamedrift/.VirtualBox/winxp.vmdk’
VBoxManage: error: Error code VERR_INVALID_PARAMETER at /home/vbox/vbox-4.1.8/src/VBox/Storage/VMDK.cpp(3656) in function int vmdkCreateRawImage(VMDKIMAGE*, VBOXHDDRAW*, uint64_t)
VBoxManage: error: Cannot create the raw disk VMDK: VERR_INVALID_PARAMETER
VBoxManage: error: The raw disk vmdk file was not created

My box is Gamedrift running Ubuntu 10.10 Closed source version in use.

i look forward to your comments

Aaron - January 1, 2012

Rob,

I haven’t really been keeping up with VBox (see the top of the OP) so I’m not sure how to debug your problem. IIRC, though, you will want to make sure that the numeric argument for -partitions flag is the number of the partition where you have windows installed (ie. 5 corresponds to /dev/sda5). I forget whether -rawdisk wants /dev/sda or /dev/sda1 — you may want to try both.

Dan - January 5, 2012

To anyone struggling to get this running with Windows 7, I might be able to save you some time:

Many thanks for this article and to everyone who has also contributed – all this plus a bit of fiddling finally got me there!

Aaron - January 5, 2012

Thanks for the contribution, dan :) ill add this above to the OP!

fuck - January 13, 2012

jesus fucking christ. everyone posts instructions for LINUX even though I searched WINDOWS. I want to mount a physical drive in virtual box running on windows in windows but there is no such thing as /dev/drive bullshit in windows. GOD DAMNIT

Aaron - January 16, 2012

Hi there!

Figure it out, piece together the instructions, and write your own how-to guide for windows users. Before this post, there really wasn’t any comprehensive instructions detailing all aspects of the process.

I have no idea what is required to get it to work in windows, but I will assume that the process is roughly similar. I would probably start by looking through the VirtualBox application for something that lets you create a VMDK — you may need to do this at the command line, which surely has documentation somewhere.

Good luck! If you do end up writing a post, please send me the link and I will link to you from this post. Thanks!

Maximiliano Carrizo - February 4, 2012

Very important issue:

If you’re receiving this error message in boot:

MBR
A disk read error occurred
Press Ctrl+Alt+Del to restart

You should edit your generated file in step “Create a VDMK file” and change line:

ddb.geometry.biosHeads=”255″ to ddb.geometry.biosHeads=”240″

Regards !

Ravi - February 7, 2012

im hitting this error..

The boot selection failed because required device is inaccesible

I followed the steps mentioned above let me know if there is any workaround for this problem

Ravi - March 6, 2012

can someone help me with this error.. thanks in advance

Aaron - March 6, 2012

Ravi, is the booting error occuring during VBox bootup or machine bootup?

Rick - February 18, 2012

You da man! This worked for me. I’m now running a WinXP VM (booted from an existing NTFS partition) in VBox on Linux. I have read and tried many other methods posted in Ubuntu and VBox forums, but this is by far the simplest.

I did have to uninstall Symantec Endpoint Protection (SEP) to allow XP to boot in Normal mode. With SEP, it would boot in Safe mode, but would fail in Normal mode with exceptions in services.exe and lsass.exe. (Now to find Windows AV that is compatible with VBox!)

My setup:

Toshiba Tecra M3 2Ghz single-core processor & 2 GB RAM
VMDK created with 2 NTFS partitions: C: & D:
Ubuntu Lucid Lynx 10.04 LTS
VirtualBox 4.1.8 r75467
Guest settings: 768 MB RAM, 64 MB video memory – Motherboard Chipset: PIIX3, IO APIC enabled – IDE Controller Type: PIIX3, Use host I/O cache
Guest OS: Windows XP Pro SP3

Aaron - February 23, 2012

Rick

Thanks for stopping by! Glad it was helpful for you — apologies for not keeping it current, but thankfully there are other VBox enthusiasts who are gracious enough to post their progress and hacks on here. :) If you had to overcome any difficulties or situations with any clever hacks or discoveries, please feel free to post here and I will add it (with attribution) to the OP above if I can fit it in.

cheers!

Greg - March 20, 2012

Aaron: A million thanks to you and to everyone who contributed to these instructions as I finally got my VM up and running! These were the first instructions I found that were clear and didn’t leave pertinent details out.

Gratefully, Greg

Aaron - March 20, 2012

Glad to hear it’s still working for people :)

If you run into any other problems or difficulties, please feel free to post here and I will incorporate the changes into the document.

Justathought! - May 21, 2012

Excellent tutorial! Thank you Aaron. I found this late, and it clarified some of the finer points I was missing from another explanatiion.

Aaron - May 21, 2012

Excellent! Glad you found it useful. If you discover anything I overlooked, please comment here and I will integrate it into the article.

Sean - June 20, 2012

I created the VMDK but when I goto create the virtual disk and point virtualbox to the VMDK, I get the following error:

Failed to open the hard disk /home/myusername/.VirtualBox/win7.vmdk
Permission problem accessing the file for the medium ‘/home/myusername/.VirtualBox/win7.vmdk’ (VERR_ACCESS_DENIED)

A Specialist - August 2, 2012

You should update this so that, after adding the disk group to the user, that they should log out and back in so that this change takes effect.

Jorge - August 31, 2012

thanks! reloging solved this, you should put it into the tutorial :) thanks for the work Aaron

Aaron - September 3, 2012

Can you be more specific about the problem you encountered and what you did to resolve it? I’m happy to include it, thanks!

Julian - October 10, 2012

Aaron, I love you. I haven’t hammered out all the problems yet, (e.g awful awful booting speeds) but I’ve managed to get Windows 8 Pro (64bit) to boot in my 64bit version of Ubuntu 12.04.

I had a few issues getting it booting to start with, but using the windows 8 installation disk to get into command prompt and run bootrec.exe /fixboot, /fixmbr and /rebuildBCD got this working. I couldn’t access my other SATA HDD that carries all my media, (And adding this as a separate vmdk in my virtual machine’s options made it borky and not detect the original Windows installation), but I’ll try and work it out.

Thanks again!

Aaron - October 10, 2012

Julian, that’s excellent!

I will add your bootrec.exe command to the post — please post here with anything else you might find and I’ll incorporate it into the post.

Thanks! :)

Julian - October 10, 2012

Aye of course, I shall. At the moment it’s completely broken it seems but I’m working on it. Also, just a heads up, the commands you quoted from me are to be executed seperately, e.g;

bootrec.exe /fixmbr
bootrec.exe /fixboot
bootrec.exe /rebuildBCD

Just to avoid any confusion!

Aaron - October 10, 2012

OK, thanks for the clarification — I haven’t used windows in a couple years now. :)

I went through and updated the post with some newer info, providing PPA details instead of binary-only, and re-formatted the text to highlight the terminal commands better.

Thanks for helping to keep this post alive. :)

Julian - October 10, 2012

No problem. Happy to help, and I hope this page stays exactly where it is because it’s been a lot of help to me too.

On a related note, I managed to completely murder my bootsector and grub so much to the extent that I ended up reinstalling windows and ubuntu. I’m gonna play it much safer this time, though.

Any ideas on how to give the initial .vmdk access to an additional SATA drive?

Oh, and one more thing, you MUST also include the ‘System Reserved’ partition in the vmdk file for win8 to boot.

I’m pretty much done with this for now I think.

I followed exactly the same steps as before, created the fake mbr, and the vmdk image, but also created an additional vmdk image for my shared drive. My windows image contained both the main body and the system reserved partition, and I executed the bootrec.exe commands to repair the mbr and managed to boot successfully and also access my shared drive. I cannot however boot natively into windows 8, as expected, but I hope some time in the future someone will be able to find a workaround for this, since repairing the installation and then also repairing grub every single time is not a valid option for me.

Gustavo - May 4, 2013

Hi Aaron,

Thanks for this great tutorial. I have been struggling for the last days and I got to a very bizarre point.

So I got a new laptop with Windows 8 and Ubuntu installed with dual boot. I followed the instructions, included the recover partition as Julian said, and I was stuck with the MBR FA error. I tried running my Win recovery CD in the virtualbox, but I does not recognize the data (to be precise, I ask virtualbox to boot from there and it says there are no boot files. It runs ok from raw outside any OS).

I also tried mounting the recovery files in a USB pendrive, and creating a VDMK for this partition and adding it to the fake.mbr. This actually DID work, and I could start the repair mode, went to the command prompt option and run the three options for bootre.exe. After that, I could boot into Windows, but only is the VDMK for the USB exists (if I remove the USB or the VDMK drive I created onVB I get again the MBR FA error). I also trying creating a partition with the content of the USB pendrive and a VDMK, but it does not work.

So, Do you have any idea how can I run Win without the USB? The weird thing is that I have no way of getting again into the recovery mode in the USB. Besides, the dual boot still works perfectly (without any other change).

Thanks

Aaron - May 9, 2013

Gustavo,

I wish I could be more helpful with this, but honestly, the last version of windows I used (and own a legit copy of) was XP. I know MS has changed a lot of things with how the FS works, particularly with regard to the MBR and booting process.

As I essentially use Linux full time now, save for those extremely rare times when I boot into a pure VM to debug a web app in IE, I’m not sure I can be much help. :( perhaps other commenters might be more helpful?

Jacopo Alabiso - July 30, 2013

Hi Julian, how did you finally manage to access the shared drive? I am booting win7. When booting nativelly, it can access the shared partition. But when booting in VirtualBox, it says the disk is RAW, cannot access it (and neither can format it).

Any clue on how to make it see the shared partition? thanks!

Kowalikus - October 30, 2012

Hi i’ve got BSOD with my win7 on ubuntu 12.10

http://imgur.com/6NpkU

00:03.379450 Display::handleDisplayResize(): uScreenId = 0, pvVRAM=0000000000000000 w=720 h=400 bpp=0 cbLine=0x0, flags=0x1
00:08.499302 Display::handleDisplayResize(): uScreenId = 0, pvVRAM=00007f0de4000000 w=1024 h=768 bpp=24 cbLine=0xC00, flags=0x1
00:14.695398 RTC: period=0x200 (512) 64 Hz
00:16.140139 AHCI#0: Reset the HBA
00:16.220738 Display::handleDisplayResize(): uScreenId = 0, pvVRAM=00007f0de4000000 w=640 h=480 bpp=0 cbLine=0x140, flags=0x1
00:16.859567 Reset initiated by ACPI
00:16.859671 Changing the VM state from 'RUNNING' to 'RESETTING'.
00:16.922379 CPUMSetGuestCpuIdFeature: Enabled APIC
00:16.922473 CPUMClearGuestCpuIdFeature: Disabled x2APIC
00:16.922532 PIT: mode=3 count=0x10000 (65536) - 18.20 Hz (ch=0)
00:16.970444 AHCI#0: Reset the HBA
00:16.970530 PIIX3 ATA: Ctl#1: finished processing RESET
00:16.970637 PIIX3 ATA: Ctl#0: finished processing RESET
00:16.970803 PDMR3Reset: after    48 ms, 1 loops: 1 async tasks - piix3ide/0
00:16.971010 TM: Aborting catch-up attempt on reset with a 89 839 813 ns lag on reset; new total: 89 839 813 ns
00:16.971065 Changing the VM state from 'RESETTING' to 'RUNNING'.
00:16.975508 Guest Log: BIOS: VirtualBox 4.2.4
00:16.975688 PIT: mode=2 count=0x10000 (65536) - 18.20 Hz (ch=0)
00:16.986508 PIIX3 ATA: Ctl#0: RESET, DevSel=0 AIOIf=0 CmdIf0=0x34 (-1 usec ago) CmdIf1=0x00 (-1 usec ago)

Aaron - November 17, 2012

Hi Kowalikus,

Win7 is kind of pioneer territory with this. I just had some new comments come in with info, I’m going to integrate it into the main guide. Perhaps that will help?

Charles Kozler - November 3, 2012

Hi Aaron,

I am wondering – if I boot virtual and have to do the rescue disk option it is probably safe to assume that I will need to do the rescue disk again if I boot raw, correct? You seem to make that implication in your post but I just want to be 100% clear before I go this route. I don’t boot in to windows often but when I do it is for gaming and I also do not want it corrupting grub every single time either. Can you tell me the possible overhead for someone who would boot consistently between virtual and raw? Would it be worth doing this or no because you have to effectively repair it each time?

Also, I would like to be clear – if I have to only do the rescue option on only virtual boots then I am ok with that but I want to understand the impacts it may have on a raw boot (eg: conflict with grub, have to do a repair each raw boot as well as each virtual boot, etc…)

Thanks again for the tutorial!

pcworld - November 17, 2012

(Use at your own risk.) http://wiki.ubuntuusers.de/Dualboot-Windows_virtualisieren (translated: http://translate.google.de/translate?sl=auto&tl=en&js=n&prev=_t&hl=en&ie=UTF-8&layout=2&eotf=1&u=http%3A%2F%2Fwiki.ubuntuusers.de%2FDualboot-Windows_virtualisieren&act=url) describes a solution for Windows 7 so that you won’t need to do a “Repair Installation” each time you want to switch from native to the VM and vice versa (you won’t need any Windows 7 Installation Disk at all):

Solution #1 (though that didn’t work for me): Right after install-mbr, but before VBoxManage: Run

sudo fdisk -l

, you’ll need the value behind “Disk identifier:” of the hard drive your Windows 7 is on.

Then:

echo YOURDISKIDENTIFIER | xxd -r | dd of=~/.VirtualBox/FAKE.mbr bs=1 seek=440 conv=notrunc

Then continue with VBoxManage etc.

Solution #2 (should work for everyone): First, run

sudo fdisk -l

on your host system, you’ll need the value behind “Disk identifier:” of the hard drive your Windows 7 is on.

After you’ve created the VM (leave out the step telling you to boot from the Windows 7 Installation Disk), attach a Ubuntu (or any other Linux) live CD image to the VM:

Settings -> Storage -> Controller: IDE -> Add CD Device (first icon) -> Choose Disk -> select the image of your live CD

Then boot the VM, it should boot the live CD image. Once you’re in there, run

sudo fdisk /dev/sda

to start fdisk’s interactive mode (replace /dev/sda with the hard drive your Windows 7 is on).

Enter

then enter

Put in the disk identifier you’ve gathered in the first step, then press

to rewrite the MBR and quit fdisk.

EDIT (added by admin): After this, shut down the VM and remove the live CD image from the VM’s storage devices.

And note that the device name (e.g. /dev/sda) you have to use on the guest system might differ from its name on the host system.

Now you should be able to boot Windows 7 both natively and through the VM without Windows complaining.

Feel free to update the article with this information. To everyone else that has problems, I suggest you to read the wiki article linked at the top of my comment, it’s very comprehensive and has many troubleshooting tips at the end.

Something else that took me a few hours to find out: I also have a Windows XP partition on my hard drive, and I didn’t realize that the “Boot” folder (which is needed for Windows’ bootloader) was only on that partition but not on the Windows 7 partition (I, naturally, installed Windows 7 after I installed WinXP). So I didn’t add WinXP’s partition number to VBoxManage’s “-partitions” list, which prevented the VM from booting.

This wiki article also describes how to prevent Windows from asking you to activate again (translated [see section “Align virtual hardware of the physical hardware”; yes, that’s Google Translator’s English :P])

Aaron - November 17, 2012

Thanks for the info! I’ve integrated it into the guide. :)

Ned - September 12, 2014

just a precision : after entering the new disk identifier, you have to press ENTER, then m to return to FDISK’s main menu, and then press w to rewrite MBR.

My win guest doesn’t repair anything anymore when I reboot it in “real/host” mode.

But it goes BSOD in guest mode, no matter what I do (and I did respect ALL instructions here). I just can’t figure out what’s happening :(

Ned - September 15, 2014

theres’ a slight error here : after entering the new disk identifier + ENTER, you have to press m (back to main menu) before being able to press w

Prateek - January 8, 2013

Very helpful tutorial. Everything worked till booting windows on VM.

But on boot, windows boots till “Windows is loading..” stage and progress bar loads twice and restarts and gets in this loop forever. Anyone know what could be the issue.? I have been trying with different settings but still no solution.

Aaron - March 3, 2013

Thanks, Prateek — which version of Windows are you using?

Shane - January 8, 2013

Followed this recipe and it seemed to work out. Question: I would ALSO like to be able to run the Ubuntu when I boot up in the windows partition (raw disk method). How might one do this?

Aaron - March 3, 2013

Shane — I haven’t tried to do that yet — I’m not sure if Windows is capable of doing this. The tutorial was intended mainly for people that use Ubuntu as their primary OS but need to occasionally switch to Windows (anymore, I just do VM’ing only).

If you use Windows primarily, you could use Wubi or Cygwin to run Ubuntu in parallel.

Ashley Moore - April 25, 2013

Great Howto. Got my W7 install working perfectly thru VB 4.2. It boots natively as well without a hitch.

DiQ - June 8, 2013

vboxmanage not VBoxManage, and I think it needs root access too

Erhannis - June 11, 2013

I’ve managed to get Win7 working (though it complains the copy of windows isn’t valid, and uses generic display drivers). I kept getting the BSoD on boot, even though I’d tried changing the hard drive chipset to several combinations of settings. What finally worked was taken partially from http://sanbarrow.com/p2v2008r2.html.

The steps were as follows:

In virtualbox, go to Settings.
Go to Storage.
Replace all controllers with one SAS controller (I don’t know if it’s necessary to remove all the others or what).
Add your vmdk under that controller, and hit ok.
Now open a terminal. (This is assuming installation under Ubuntu; if not, try analagous steps in whatever OS you’re using).
Install the registry-tools package, either with apt-get or whatever.
Mount your windows partition, say /mnt/WIN/.
Navigate to /mnt/WIN/Windows/System32/config/`.
regshell -F SYSTEM
cd ControlSet001
cd services
cd LSI_SAS
set “Start” REG_DWORD 0x00000000
exit
Then unmount the partition and try booting up. Hopefully it will work.

I guess the idea is to get windows to use the right drivers for however virtualbox is letting it access the drive.

Please note that I haven’t tried actually booting my computer into Win7 yet (after the registry change), so it’s possible that I borked it for normal booting up.

Erhannis - June 11, 2013

Tried it; still works.

Ned - September 15, 2014

Thank you very much !!!!

all this article + PCWorld’s comment + your comment saved my day

Amir Marandi - July 14, 2013

One problem I ran into was access denied when I did

VBoxManage internalcommands listpartitions -rawdisk /dev/sda

command, so I did the next step whis is

sudo chmod 666 /dev/sda

and then executed that command and it fixed the problem.

Karthik - July 19, 2013

I have been searching for this kind of solution for a long time now. Every site I visited always suggests creating a Hardware Profile and goes on to say, HPs are not present from Vista on onwards and thus native partiton boot in VM is not possible.

But, your soultion worked like a charm, upto the details of each and every error message that you had mentioned we might get. followed your instructions and every step worked as expected. Now able to boot my Windows partition from VirtualBox.

Thanks a lot.

Jacopo Alabiso - July 30, 2013

great tutorial, it all worked well.

One problem: when I boot Win7 within virtualbox, it cannot access the primary partition I use for documents, it says it’s RAW and needs to be formated (but if I try formatting it, it’d fail).

The same partition is accessible when booting win7 natively.

Any clue?

Karthik - August 13, 2013

I was constantly getting VERR_ACCESS_DENIED error. If all else fails follow the following procedure.

Do sudo chmod 666 /dev/sda before the start of all commands
For /dev/sda1 do chmod 666 instead of 660

berkov - August 24, 2013

HI, all configured as above but i get stuck when i run the VB. a new, black emulation window gets open with and a sec later one word appears: “MBR”. any ideas why?

Anonymous - October 19, 2013

I did this on another harddrive rather than the same harddrive but different partition. If you get a uuid mismatch, you could just open the ``.vmdk` and change it to match the expected uuid.

Leo - November 5, 2013

Question: If you have a Windows 7 recovery disc, but not an install disc, can you still repair the installation once you’ve done this?

Aaron - January 9, 2014

Leo, As far as I know, the VM treats the guest OS pretty transparently, so if you use the recovery disc it should work in whatever way the recovery disc was intended to work. (ie. if the recovery disc repairs installations on a normal computer, it should also repair the installation on a VM, I think.)

No guarantees, though.

beemoon - January 9, 2014

Hi, nice job !!! I manage to launch my win seven 7 Pro N (natively installed) in my CrunchBang (wheezy) with the last stable version of virtualbox v4.3.6

thanks

John - January 9, 2014

So I used WUBI to install my ubuntu alongside windows. I am now trying to use this to access my windows while in ubuntu.

Will this still work? I think my windows is on /dev/sda2 because when I use fdisk -l it is the only bootable drive, and when I use df -h it says /host/ which contains my windows files is mounted from /dev/sda2`.

However when I run the createrawvmdk it says cannot read partition information from /dev/sda2

Am I doing something wrong here? Also I cannot unmount /host/ apparently because thats also where my linux files are cause thats apparently how wubi works.

PLEASE HELP.

v1adimir - January 26, 2014

Hi, just passing through; curious: as to why you’ve switched-over to the VMWare Player.

Any particular reason, to do so?!??

Aaron - March 15, 2014

I’ve actually switched BACK to VirtualBox, now that Canonical finally updated their repos to use the more current version.

The repo version is pretty solid and there is now even a package for guest additions. :)

Laura Frasca - April 22, 2014

Thanks so much for maintaining this document! I am now able to run Windows 7 Pro (64-bit) on Debian jessie/testing in VirtualBox 4.3.10. I did have to repair the installation (ok by me because I am planning never to boot Windows raw again).

I also found that Windows would not give back the mouse after capturing it. This was exceedingly frustrating since VirtualBox displayed a couple of popups to me that obscured my view of Windows and that I couldn’t dismiss. I had to SSH into my machine and kill the VM a couple of times before I got it sorted.

The mouse capture issues were resolved by installing the Windows additions. Add /usr/share/virtualbox/VBoxGuestAdditions.iso as a CD drive before starting the VM, and then run VBoxWindowsAdditions.exe from the CD/DVD drive inside the guest. Extra points for navigating Windows with the keyboard so you don’t get your mouse captured.

Jonathon - May 16, 2014

Thanks to Aaron and to all who posted this excellent tutorial!

I got everything to work fine (in terms of accessing the partition through VB), but I have a couple VB problems:

Guest Additions CD doesn’t install properly
Host Key and Disable/Enable Capture Mouse function not working.

I realize these are more about VB than about virtualizing a windows partition, but can anyone help me out?

I’m still very happy that it works, though! :)

Raz - June 3, 2014

Excuse me,

I would like to ask the way to modify disk identifier, if applicable windows8.1?

ComputingFroggy - June 19, 2014

Hi,

Great post … but it did not completely work for me. I am running Lubuntu 14.04 on a Asus EeePC 1215B with Windows 7 pre-installed. I followed the given instructions including the instructions for Windows 7.

The instructions with fdisk (User-submitted Walkthrough for Windows 7) did not work for me. I had to download a repair disk ISO and to boot with it (and rebooted a couple of time) … to eventually get a Blue Screen of Death with an error message quickly disappearing !

Then I remembered I started with another post that gave a solution for that: running a registry fix.

I give you the post (the registry fix is at the end): http://in-the-attic.com/2010/06/05/booting-windows-7-from-an-existing-partition-inside-ubuntu-virtual-box/

And now, it works: I can run Windows from my Lubuntu … all of that on a small netbook !

Thanks again for all your work !

Aaron - June 19, 2014

Thanks for your contribution!

The original guide was for winXP with Ubuntu, so I make no warranty about it working with any other configuration. Glad you were able to figure it out:)

ComputingFroggy - June 19, 2014

I have created a post, on my blog (in French ) summarizing all of this http://computing.travellingfroggy.info/article171/double-boot In the English version, I will just point out to your post (and a few others), if that’s ok with you.

Aaron - June 20, 2014

Absolutely! I’ll give you a link near the top here, too :)

ComputingFroggy - June 28, 2014

Strangely, I had my partition table garbled: my first Ubuntu partition has been moved from sda7 to sda9. Thankfully, that was a Ubuntu 12.02 that I am not using any more. I am running from a Lubuntu 14.04 (on sda8) where I have installed VirtualBox. Anyway, that’s just a note in case, someone is interested.

But I have now a problem, on my VirtualBox I have no network, when it worked before ! Any ideas on how to fix it ?

Aaron - June 28, 2014

Which style of networking are you trying to use? I typically either go NAT (if I want it strictly internal) or Bridged (if I want to interface with other computers on LAN)

ComputingFroggy - June 29, 2014

I left everything to the default: NAT with Intel PRO/1000 MT Desktop !

I think the network went pear shape when I installed the guest additions (I wanted to share the clipboard between Ubuntu and Windows).

ComputingFroggy - June 30, 2014

An interesting forum thread on virtualbox.org: https://forums.virtualbox.org/viewtopic.php?f=2&t=31759 with a how-to similar to yours !

It’s claimed they’ve got a method avoiding to use of the repair CD … although I haven’t tried it, yet.

ComputingFroggy - July 3, 2014

I have added another network card with bridge access. That triggered drivers installation, and the network is working now.

Igor Melnyk - July 10, 2014

Thanks a lot!!

Markham - January 10, 2015

I successfully followed the steps to open my Win7 partition in VirtualBox — thanks for the brilliant walkthrough — but when I try to load a Netflix video in my browser in the Win7 VM, I get an error:

Whoops something went wrong… Digital Rights Management (DRM) Error. Error Code: N8156-6013.

Do you (does anyone) have any idea how I can get around that? (Using Silverlight was the only reason I wanted to access my Win7 installation from my Linux installation.) “Repairing” Silverlight doesn’t seem to help.

Markham - January 19, 2015

Nevermind. Tried it in Chrome with success.