Reasonable Deviations

Man and machine: GPT for second brains

2023-02-05T00:00:00+00:00

In this post, I discuss how I used GPT embeddings to build a smart search tool for my second brain note-taking system.

Try this if the video isn't working for you

Introduction

I spend a lot of time thinking about how to improve my processes for learning and knowledge management; I’ve written extensively about my second-brain note-taking system, Molecular Notes.

I use Molecular Notes on a daily basis but am still constantly on the lookout for improvements. As my collection of notes grows, I have recently been running into a retrieval issue: sometimes I’m trying to find an idea I know I’ve written about but I can’t remember the name of the concept or where I came across it.

For example, let’s say I remember making some notes on failure modes in large organisations, but I can’t remember any specific names or the source. I can try searching for relevant keywords in Obsidian search, but this doesn’t help much. The problem is that Obsidian’s default search is lexical – it tries to match words in the search query with words in the documents. It would be generous to call this “hit or miss”: as you can see below, when I search for “failure mode”, it matches anything with the string “mode”, including words like “model” which are not relevant. I could do a literal search using double quotes, but that only returns exact matches.

(Molecular Notes, through its emphasis on linked notes, provides other means of finding content; in this case I could look at the local graph of notes connected to a relevant topic, like Management, and hopefully find what I’m looking for that way).

I was in dire need of semantic search, where the search tool understands the content of my second brain enough to return notes that are similar in meaning rather than just having the same words.

Basics of GPT

I don’t claim to have a good understanding of the internals of large language models. So here is my “black box” understanding of GPT, which emphasises the inputs and outputs rather than the internal function.

GPT (which stands for “Generative Pre-trained Transformer”) is a Large Language Model (LLM) – a fancy deep learning model suited to a variety of language tasks. GPT-3 is the third iteration of GPT, with 10x the number of parameters of v2, and presumably some multiplier of the linguistic power.

The main task of GPT-3 is text completion – feed it some input text and it tries to return the most appropriate output text. This is why we’ve seen amusing screenshots of GPT saying things that are completely wrong – it has no inbuilt truth-seeking, it’s only trying to make stuff sound right (in fairness, I don’t think humans are very different on this point).

Here is an overview of the different tools OpenAI has built on top of GPT-3:

ChatGPT: a variant of GPT-3 optimised for dialogue. It abstracts away the text completion aspect by making it feel like GPT is answering your questions, and wraps it in a nice UI.
Text completions API: feed in text programmatically and get back a completion. For example, check out this demo where the completions API is used to fill in a spreadsheet based on the surrounding cells. You could use the text completions API to build your own version of ChatGPT.
DALL-E: an image generator based on GPT-3, which spits out images instead of text completions in response to some text prompt
Embeddings API: this is what I used in the project. It is less intuitive than the others, so I’ll explain more below.

Embeddings

The core idea of embeddings is to represent a block of text as a point in space (specifically, a vector in high dimensional space).

We mostly want to do this because once text has some spatial location, we can derive useful information based on its location. For example, we might be able to see that “dog” and “cat” are spatially closer than “dog” and “trumpet”.

Also, vectors are easier to mathematically manipulate than blocks of text: adding and subtracting vectors is just a matter of following arrows. So once we’ve represented words as vectors, we can more rigorously say things like:

king = queen + man
tiger = cat + stripes
slower - slow + fast = faster

Samy Zafrany’s word2vec tutorial.

None of this is particularly new – word2vec was published in 2013. But the hope is that with LLMs like GPT-3, the embeddings can become even more expressive, encoding paragraphs or documents rather than just words and identifying abstract relationships between text blocks.

GPT Embeddings for semantic search

The core workflow for semantic search is very straightforward – it’s basically the “hello world” of GPT projects.

Take a bunch of text blocks and feed them to the OpenAI embeddings API. This returns a 1536-dimensional vector for each text block.
Likewise, embed the search query using the same API to get a query vector.
Find the most similar vectors among my notes. There are several options to compute the similarity between vectors, but most people default to cosine distance, which measures how closely two vectors align.
Return the top n notes by similarity.

Here’s ChatGPT’s explanation, which is a bit more detailed.

The main nuance is deciding what constitutes a text block. There were two main considerations here:

The OpenAI embeddings API limits the token size to 8191 (~6000 words). So each text block has to be smaller than that.
What output do I want to be returned? It’s not super useful if the search points me to a large note as I’d still have to scroll through that to find what I’m after.

Fortunately, my long notes are written in a fairly structured way, separated by headings. So I just fed in each section as a text block.

Technical implementation

The code is on GitHub, though the implementation is tied quite closely to my particular system and I don’t intend to put in any effort to generalise it.

Processing input

Because Obsidian notes are flat files, it’s simple to write a python script to walk through the Obsidian vault to get all the notes into python. I asked ChatGPT to do it for me:

The result of this step is a dictionary representation of my entire Obsidian vault, mapping the note name and chapter to the note text.

notes = {('Some Podcast', 'Episode 1'): 'text',
         ('Some Podcast', 'Episode 2'): 'other text',
         ('Some book', 'Chapter 1'): 'my notes'} 

For optimal embedding performance, I cleaned some of the text:

Removed some of the metadata I use in each note (e.g. author tags, metadata).
Remove newlines and special characters
Remove URLs and just keep the name of the link
Removed Obsidian backlinks (double square brackets).

I left mathematical notation untouched, because ideally I want the search engine to understand maths. It seems to have some understanding – when I query “expectation of a function versus the function of an expectation”, it does suggest Jensen’s inequality, even though my note on Jensen’s inequality doesn’t mention the word “expectation” except in mathematical terms. But I can’t be sure, perhaps it’s just picking up on the word “function”.

Embeddings

This is the easiest part of the project: OpenAI does the heavy lifting. The embedding function (which I took from one of their cookbooks) takes in a text block and outputs a vector.

def get_embedding(block: str) -> list[float]:
    return openai.Embedding.create(
                    input=block, model="text-embeding-ada-002"
                )["data"][0]["embedding"]

I simply iterate through the dictionary of notes, applying the get_embedding function and storing the vectors in a pandas dataframe – this is almost certainly not the best structure but I’m familiar with them and the data is small enough where performance isn’t an issue.

def embed(notes: dict[(str, str), str]) -> pd.DataFrame:
    # Embeds the notes into openAI and returns a dataframe containing the vectors.
    res = {}
    showfunc = lambda n: f"{n[0][0]} {n[0][1]}" if n else ""
    with click.progressbar(notes.items(), item_show_func=showfunc) as note_items:
        for (note, section), text in note_items:
            block = section + ". " + text
            n = num_tokens_from_string(block)
            # Truncate if too long
            if n > EMBEDDING_CTX_LENGTH:
                warnings.warn(f"{note} {section} exceeded token limit. Truncating.")
                block = truncate_text_tokens(block)
            try:
                embedding = get_embedding(block)
            except Exception as e:
                print(f"Error for {note} {section}", e)
                continue
            res[(note, section)] = embedding
            time.sleep(0.1)
    df = pd.DataFrame(res)
    return df

I wrote a utility function to compute the cost of embedding a list of notes and requesting user confirmation before executing. Training on 1000+ notes only cost 7 cents!

I then used the same get_embedding function to process the query string, before computing the cosine distance against the note embeddings in the dataframe:

qvec = get_embedding(query_string)
cos_sim = np.apply_along_axis(lambda x: cosine_similarity(x, qvec), axis=0, arr=df)
results = pd.Series(cos_sim, index=df.columns).sort_values(ascending=False)

Finally, I sorted the resulting notes by similarity and returned them.

User interface

I built a CLI (command-line interface) using python’s click library. This allows for easy creation of user confirmation prompts, progress bars, and styling.

Best of all, with click.launch(link) one can actually launch Obsidian, though it took some effort to get the link in the right format.

Firstly, I installed the Advanced URI extension in Obsidian, which allows you to link to text blocks within notes rather than just the note itself (a URI is like a URL but can link to things other than web pages). I then reverse-engineered the format of the URI string, allowing me to convert my (note_title, chapter_title) pairs (which also keys the dataframe of embeddings vectors) into a proper URI. In the CLI, when I enter the note index, the URI is built and passed to click.launch which opens Obsidian.

Next steps

There are several other ways I think GPT-3 can be used to augment my second brain experience. Here are some ideas, in descending order of importance.

Link recommendation

I currently have an Obsidian extension that uses various distance metrics to list similar notes. This is based on the graph structure and backlinks: e.g. if note A links to note C, and note B links to note C, maybe A and B are similar.

But perhaps the GPT embeddings can capture more nuance in meaning. I plan to write a small script to sort all unlinked notes by GPT-similarity (for linked notes, I have already manually identified that there is a similarity so these should be excluded). I’m curious to see if there are any abstract relations I have missed: for example, maybe there is some idea from a particular event in Roman History that is a nice anecdote for a financial blowup in the 21st century.

Summarisation

There are many projects out there using GPT-3’s text completions for Obsidian, for example, obsidian-gpt allows users to generate text in Obsidian.

I feel quite strongly that this is a bad idea for second brains – writing content in my own words is a crucial aspect of Molecular Notes. Though it’s possibly useful if you use Obsidian as part of your content creation workflow.

Instead, I’m intrigued by the idea of using GPT-3 as a kind of “Jarvis”. Instead of entering queries to search, with GPT-3 I could make requests like:

Show me my notes on investment frameworks
Summarise the above notes
Highlight any disagreements between the notes

Conclusion

This was a surprisingly easy project and it’s certainly not because of my coding skills. OpenAI has done all the heavy lifting, exposing a huge amount of functionality with some simple API calls.

I think there are some generalisable lessons: much of the effort involved getting my data into a suitable format for OpenAI to ingest and creating a user experience. I was lucky with the former because my data (small text blocks) is essentially the perfect input, whereas for some other projects people are trying to come up with clever ways of chunking up large pdf documents so they can be embedded. The UX side was also simple because I’m happy to use a plain CLI. I assume it’s a lot more effort to build a consumer-facing web application.

This notwithstanding, I think the barrier to building a cool product has never been lower. I strongly encourage readers to examine their current processes to identify areas ripe for AI integration. I don’t want to be replaced by AI, I want to be enhanced by it!

Molecular Notes: Practice

2022-06-12T00:00:00+00:00

In this Part 2, I discuss the practical implementation of Molecular Notes in Obsidian. I explain how I organise my Second Brain (Tags, Folders, Topics) and present detailed workflows for ingesting different types of content. I also explain how one can extend Obsidian, giving the example of Polymer – a spaced repetition application that I built on top of my Second Brain.

After reading this post, if you would like to clone my setup, head over to the Molecular Notes GitHub repository and follow the instructions!

Choosing software
Organising my Second Brain
- Tags
- Folders
- Topics
- Authors
Learning from different sources
Synthesis
Hacking Obsidian
Areas of improvement

Choosing software

I split the post into Part 1 and Part 2 (this post) to emphasise the difference between philosophy and implementation. Molecular Notes (as a note-taking philosophy) is not tied to any particular software tool. The only non-negotiable constraint is strong support for bidirectional links – when you link from note A to note B, note B should be aware that it has just been linked to.

Depending on your workflows, you may have other requirements for an app. For me, some other minimum requirements are:

Flexibility: because I’ve spent a lot of time thinking about how I want to take notes, I need a system that allows me to structure notes how I see fit. Flexibility may not be desirable for everyone! Constraints can help you think in a structured way and avoid “blank page syndrome”.
Speed: when making notes, I jump around a lot to explore connections (this is encouraged by Molecular Notes). I need a tool that doesn’t slow me down, otherwise I’ll get fed up.
Support for mathematical notation since many of my notes contain formulae.
Graph view: a convenient visual representation of a Second Brain.
Sparks joy! This is completely subjective – I want a tool whose interface gels with me. In my case, this means something lightweight and hackable.

I use Obsidian, which nails all of these requirements and offers many other great features. Obsidian is built for linked notes and is naturally suited to Second Brains. It is hackable, offering deep customisation and flexibility. Since it is essentially just a wrapper around simple text files, I don’t have to worry about being “locked-in” to Obsidian (in fact, I often interact with my notes from other apps). Obsidian works perfectly well offline, while still providing benefits like cloud-syncing.

That said, I can completely understand that Obsidian may not be the right app for everyone. It has the same vibe as a programming environment / IDE, which is great for me, but perhaps annoying for non-programmers. In the rest of this section, I briefly comment on several other options for software (I’ve tried most of them). For readers who are already sold on Obsidian, feel free to skip ahead.

Notion

I have written extensively about Notion elsewhere. I once used Notion to implement Zettelkasten – my first iteration of a Second Brain inspired by Sonke Ahrens’ How to Take Smart Notes (my highlights here).

Notion is an extremely powerful tool – it’s certainly possible to implement Molecular Notes in Notion and I know several people who happily use it for their Second Brain. I am a heavy user of Notion for PKM (personal knowledge management), but I moved my Second Brain away from Notion for several reasons:

Notion is a swiss-army knife, not a precision tool. It offers a tonne of functionality but does each thing less well than a pure-play app. For example, Notion’s Kanban boards are great but probably not as good as Trello’s. Likewise, unlike Obsidian, Notion is clearly not designed specifically for linked note-taking.
Following on from the above, there are many nice-to-have Second Brain features like graph view that are absent from Notion. This is not Notion’s fault; it is a general-purpose PKM tool, not a Second Brain tool!
Historically Notion has had pretty poor offline support – a constant reminder that all of your notes are stored on Notion’s servers!
Slow: Notion feels slow for Second Brains. For example, if you start trying to create a backlink using “[[some note“, there is a noticeable delay before that note pops up. In Obsidian, it feels instantaneous.

RemNote

RemNote is a note-taking app that treats spaced-repetition as a first-class citizen, while also providing strong support for bidirectional linking. I used RemNote religiously during university and the spaced-repetition functionality was a game-changer – it allowed me to memorise content in the most efficient way possible.

The main reason why I shifted to Obsidian for my Second Brain is that Obsidian is more hackable and extensible. With RemNote, there’s not much you can do to customise the interface and all the notes are stored on RemNote’s servers. Additionally, for my Second Brain, spaced repetition (at least in the way RemNote does it) isn’t terribly important.

Roam Research

Roam has a credible claim to being the app that sparked the renaissance in Second Brains. It is heavily optimised for linked thinking and would be a great app for Molecular Notes.

My main criticisms:

It is not as flexible as Obsidian and feels a bit more prescriptive. For example, in Roam, every note consists of bullet points.
It is primarily web-based. I think this makes it harder to have a clean and focused note-taking experience. (EDIT: there is now a desktop app, which possibly nullifies this point).
All the notes are files on Roam’s servers.
There is no free plan. I have no qualms with paying for something as valuable as a Second Brain, but there are many excellent free options out there.

That said, there are several great things about Roam:

Arguably because it’s paid, you can have more faith that the company will stay around for a while and implement new features (though Obsidian’s team has proven to be highly capable in that respect).
Excellent graph view.
Links to blocks rather than just to notes. In Molecular Notes, this doesn’t matter much, but for other systems it’s very useful to be able to link to specific paragraphs.
Unlinked references: Roam probably beats Obsidian on this particular point – with one click you can turn all unlinked references into proper backlinks.

VSCode

VSCode is an IDE (integrated development environment) that I use for most of my software development. I often use VSCode to interact with my Obsidian notes (e.g. when I want to do special find-and-replace operations across the entire database).

However, it turns out that there are VSCode extensions (like Foam and Dendron) that essentially turn VSCode into an Obsidian clone. I use Dendron at work because I don’t have access to Obsidian behind the company firewall.

These apps are quite similar to Obsidian but the UX is slightly less polished because of the IDE’s constraints. The learning curve was a lot steeper since the target audience for these apps are software developers (not many “normies” have VSCode installed!). Unless you are obsessed with VSCode (not unreasonable!), I think Obsidian offers a better note-taking experience.

OneNote/Evernote

Avoid! The difficulty of linking internal notes means that these apps cannot be used as effective Second Brains. I don’t think they even support bi-directional links!

Evernote was the first note-taking app I used (from 2015-2019), so I did have some sentimental attachment to it. But ultimately they failed to innovate and were left in the dust by competitors like Notion (there are many great articles on the failure of Evernote).

OneNote is actually rather good for general note-taking but not for Second Brains. Microsoft seems to be aware of this and is launching a product called Loop, which looks like a Notion clone and should therefore work as a Second Brain (presumably with all the problems of Notion).

Summary – Choosing Software

Broke: Notion
Woke: Roam
Bespoke: Obsidian
Baroque: Evernote/OneNote

All told, you need to pick a note-taking app that gels with your philosophy. The core tenets outlined on Obsidian’s website are exactly what I want:

Linked thinking as the primary objective
Lightweight and extensible
Local markdown notes

Organising my Second Brain

People who have tried to get started with note-taking (either digitally or with pen and paper) will quickly appreciate the pain of “organisation paralysis”: it can be hard to decide how to use tags, folders, categories etc, and you often end up with a convoluted system that is impossible to maintain.

As I argued in Part 1, this doesn’t mean that we should avoid categorisation! It just means that we should focus on a simple and flexible structure.

Folders

Within my main Obsidian vault, I use a simple folder structure (there is no further nesting):

The folders with an underscore in front of them are “system” folders that are used by Obsidian – they don’t contain my notes.

All Atoms go in the main Obsidian vault, not in subfolders. This decision was based on the observation that opening Atoms is probably the action I do most often, so I should reduce the number of clicks as much as possible.

I use folders to house specific types of notes: Authors, Molecules, Sources, and Topics. A reasonable question: doesn’t this create duplication between folders and tags? Why bother creating a folder where everything inside it is tagged a certain way? The answer is that folders speed up navigation; they are great for “skimming” your Second Brain. For example, I often want to skim through a list of my Molecules. While I could do this using a search for “tag:#molecule”, this is such a common task that it’s more efficient to be able to simply click “Molecules” in Obsidian’s file explorer and be able to quickly look through:

The downside of this approach is that I have to remember to move a new note to the correct folder. I wrote a python script to do this for me (i.e. move all notes tagged “#molecule” to the Molecules folder), but even without that, it can be done quickly using the command palette (⌘P “move” ⏎) or you can set a custom shortcut.

Topics

Topics are the main “semantic” source of structure (tags and folders are more organisational). They are a powerful tool in helping me to identify connections across domains:

I don’t have a set procedure to decide when something should be given an explicit Topic. As a very rough rule of thumb, I make something a Topic if I think more than ~5-10 notes will link to it. Some of my most-used Topics are Finance, Statistics, Philosophy, Volatility (in the financial sense), and Management (notes related to running businesses).

Volatility is an interesting example because it is actually a Subtopic of Finance, in the sense that any note related to the Volatility Topic should also be related to the Finance Topic conceptually. But in these cases, I will only link to the Subtopic. Hence if I am writing a new Atom about volatility (e.g. some new options concept), even though it is indeed related to Finance, given that it is only related to Finance via its relation to Volatility, I will only link it to the Volatility Subtopic. This results in a clearer graph structure.

Don’t worry too much about deciding the best Topics. I often restructure and rename the Topics – this iterative procedure helps me remain abreast of the contents of my Second Brain.

Authors

At the top of all my Source notes, I write down the author. But rather than doing this as plain text, I make an empty Author note (containing only Type: #author) and link to that.

The advantages of this approach are threefold:

The Authors show up on the graph.
Because the Authors are notes tagged #author, I can give them a different colour on the graph.
If I ever need to note down information about the author, I can easily do so on the Author page.

However, these advantages only materialise when there are several Sources from the same Author in your Second Brain (or you reference the Author in Atoms/Molecules). If this is unlikely to be the case, e.g. if I’ve read a one-off blog post from some random author, I will leave the author as plain text.

One downside of making placeholder Author notes is that one ends up having to create a lot of notes which then need to be filed away into folders (at least according to my organisation principles). I have written a python script that does this all for me – all I have to do is make an unlinked reference e.g. author: [[Firstname Lastname]] then my helper script will create a Firstname Lastname.md note and move it to Authors/.

Learning from different sources

As I explained in Part 1, a key goal of Molecular Notes is to help me learn from a variety of resources: books, textbooks, podcasts, YouTube videos etc. In this section, I explain how I make Source notes for different media types.

Before that, let me review some key principles:

Avoid duplication:
- If two Sources are explaining the same concept, that concept should be extracted into an Atom and linked to by both Sources.
- But if each Source provides different commentary on the same concept, I can and should make a note of both.
- Before making notes on a Source, I first pull up similar Sources so I can get a better grip on what’s already in my Second Brain. Often, I end up restructuring my old notes at the same time as making new notes!
Review-oriented structure:
- Notes should be hard to write but easy to review – put in the effort to rearrange things until the structure makes sense!
- I’m a big fan of using nested bullets because they let me look at my content on different levels of abstraction, allowing for efficient review. When I’m looking over my Source notes later, I can scan the headings and the top-level bullets to find what I’m looking for (or maybe I’m just browsing), before diving into a particular point in more detail if needed.
- Contrast this with a paragraph of text, where I essentially have to read the whole paragraph to figure out what the paragraph is discussing. Converting textbook prose to nested bullets isn’t always easy, but having to re-arrange concepts helps me internalise the material.
Naming conventions:
- I’m not at all consistent with how I name Source notes – sometimes I use the original title, sometimes I append the author e.g. Complexity, Mitchell. For journal articles, I will either use the title or the reference, e.g. Avellaneda and Stoikov (2008).
- How do I decide? I name the note whatever I would like to reference it as. This is my Second Brain, so I let the note name reflect whatever symbol I have in my mind.
- I generally avoid abbreviations because the marginal effort of linking a long name is tiny due to autocomplete. Again, there are some exceptions to this – if the abbreviation is strongly embedded in my mind (e.g. GEB for Godel, Escher, Bach: an Eternal Golden Braid), I will just use the acronym!

Books

I have discussed at length how I consume books in a previous post. For completeness, here is a summary:

My book review and highlights are contained in Notion (they can be viewed here).
I then copy the key highlights into a Source note.
I extract “known” concepts into Atoms and crystallised pieces of insight into Molecules.
I look through other notes from the same Topic (using the local graph) to find potential connections.

Textbooks and online courses

As discussed in Part 1, Sources are composed of concepts, context, and a thesis. The key distinction between books and textbooks (one can find/replace “textbook” with “online course” in this section since the principles are identical) is that textbooks focus far more on concepts and facts, whereas non-fiction books contain a core thesis and supporting examples. Thus I expect a textbook to result in far more Atoms than a non-fiction book.

Here are some guidelines when making notes on textbooks:

Consume the content in at least two passes:
- I normally read over a textbook (or watch lecture videos) once on 2x speed just to get an idea of what ideas it contains, before making proper notes on a second pass.
- This helps me have a sense of what’s important – or indeed if the textbook is worth reading!
Mimic structure:
- I generally follow the structure of the textbook because it makes it much easier to refer back to the original textbook when I need additional details.
- Obsidian lets you link to headings, so in other notes, I could refer to Volatility Trading Chapter 2 with a proper backlink (see the Obsidian docs for an explanation)
- But one must be careful not to fall into the trap of completionism: just because we are following the textbook’s structure does not mean that we need to take detailed notes on every chapter! I sometimes leave chapters as just a heading if the ideas aren’t immediately relevant to me.
Utility over comprehensiveness:
- We don’t need to re-explain everything in the book! In fact, we should think of Source notes as a collection of pointers. The note should just tell you what is in the Source.
- Unless you expect a concept or idea to be particularly important for your thinking (in which case you should extract it into an Atom or Molecule), feel free to write nothing more than a minimal phrase to remind you of its existence.

Probably the most differentiated aspect of my textbook workflow (compared to typical textbook note-taking) is that only 70% of the time spent making textbook notes is spent on that particular textbook. The other 30% is spent extracting relevant Atoms from other textbooks. For example, I was recently making notes on Ben Lambert’s excellent course on undergraduate econometrics. I had previously read a textbook called Forecasting – Principles and Practice that had a lot of overlap, especially on the subject of time series. So I kept this open in the right pane, and whenever there was a concept that was discussed both by Lambert and Forecasting, I would extract it into an Atom.

With reference to the above image, previously my Source notes in Chapter 9 – ARIMA models looked quite similar to Chapter 10 – Dynamic regression models: lots of text with few links. But as I was making notes on Lambert’s course, I progressively extracted Atoms like “AR model”, “MA model”, “ARMA model” – though note that there is still some Source-specific commentary under in Chapter 9 – ARIMA models, like that within the “Stationary time series” bullet.

This process leaves me with a far better understanding of the concepts because I have to:

View the same concept in different contexts
Merge two explanations for the same concept
Reconcile different notation to strike at the underlying idea

Information media

I generalise blog posts, YouTube videos, Twitter threads etc to “information media” (infomedia).

The main difference between these types of infomedia and books/textbooks is that they typically focus on a much narrower range of concepts – often just one!

In these cases, I will often directly create an Atom to explain the concept (in my own words of course) then link to the infomedia directly (not to a Source note of it). In the example below, I read a Farnam Street post about Chesterton’s fence and made an Atom for it. I didn’t create a Source note for the blog post.

Podcasts

Podcasts have been an incredibly important learning vector for me. They are an excellent way to add incremental learning to my day – as a rule of thumb, I listen to podcasts any time my body is in use but my brain is not: when I’m at the gym, walking from A to B during a commute, cooking and cleaning, etc.

There are several types of podcasts, each of which I treat differently (a list of the podcasts I consume can be found here).

General news and current affairs (e.g. All-In): I don’t make notes on these – I listen to them in the gym and they are primarily for entertainment.
General interviews, e.g. Invest Like the Best, Masters in Business: I occasionally take notes if there is an insight that strongly resonates with me.
Evergreen interviews, e.g. Flirting With Models: these are podcasts that have been carefully designed to be dense in evergreen information. I make notes on these as if they are online courses (i.e textbook-equivalent).
“Long-form”, e.g. the History of Rome, Huberman Lab: as above, I treat these as online courses.

I use an app called Airr that allows me to make audio highlights; when I hear something interesting, I press a button to record an AirrQuote and if convenient I make a text note (these are optional).

At certain intervals (e.g. every few episodes, or every season if I’ve gotten lazy) I compile all the highlights into Source notes – either one note per podcast or one note per podcast season depending on the information density.

Journal articles

I first want to caveat by saying that I am not an academic, so my workflow for journal articles is primarily from the perspective of a “keen amateur” – I generally care more about learning techniques from the articles, as opposed to citing and building on top of them.

My Source notes for journal articles are most similar to my Source notes for books: they are often very short, highlighting just the key ideas that are relevant to my thinking.

I name the note whatever feels right – sometimes that’s the formal citation, sometimes it’s the name of the paper (if it’s particularly memorable). In the front matter for the note, I give the formal citation as a link to Zotero, which I use to store and arrange the PDFs of the papers.

Synthesis

Synthesis is one of the important parts of a Second Brain, but it is also the aspect I am least able to convey in a blog post. While I’ve built a system that aids synthesis, the truth is that it’s a process that must occur in one’s (primary) brain.

My approach isn’t to sit down and say “right, it’s time to be creative”. Most often, the process starts while I’m making notes on a particular Source: perhaps the author discusses a concept that feels vaguely familiar or has a similar flavour to something I’ve come across before. I then gently poke around my Second Brain to try and see if there’s a deeper link.

The local graph is useful here because I have configured it to show both first-degree and second-degree neighbours. The second-degree neighbours are often a fertile hunting ground for connections. In the example below, the “Goal preservation” Atom links to the AI Topic. The local graph shows not just the neighbours of “Goal preservation”, but the neighbours of “AI” too. I can then skim these to see if there are additional links to “Goal preservation”

Occasionally, I do put on my “linking cap” – I set aside some time to go through my Second Brain to remind myself what’s in there and to specifically try and make connections between Atoms via new Molecules.

Hacking Obsidian

In this section, I discuss how one can “hack” Obsidian. This section is entirely optional; it is mainly here for the benefit of the advanced users and geeks (you know who you are!) who want to push the boundaries of Obsidian.

In essence, because Obsidian is made of plain markdown files and a simple folder structure all stored locally, it’s very easy to interact with your Obsidian notes via a different interface. Here are some examples.

Third-party extensions

There are a tonne of third-party extensions available for Obsidian which require very little effort to set up. For example, the Mind Map extension lets you view your notes as a mindmap (provided you have structured them properly, using nested bullets):

Viewing notes from a text editor

Notes are just plain text in folders, which means you can interact with them via your file system. For example, on macOS, this means that your Obsidian notes show up in Spotlight search, which can be quite useful. It also means that you can reorganise your notes in Finder or File Explorer.

Slightly more advanced: I often interact with my Obsidian vault via VSCode, a programming environment.

Backlinks don’t work here, but there are several reasons why this is useful:

You can use VSCode’s powerful text editing functionality. For instance, in Obsidian there isn’t an easy way to do a global find and replace across all notes, but this is trivial in VSCode (plus you can do regex find/replace rather than just plaintext).
Better support for git backups. Obsidian does have a git extension, but in VSCode you can look at diffs and see exactly what is being changed.
More convenient to write python scripts that interact with my Second Brain (see below).

Python helper script

As I’ve mentioned several times in this post, there are a couple of “routine tasks” that I’ve decided to automate using python. My helper script does the following:

Moves files with a given tag into the appropriate folder (e.g. Authors, Molecules, Sources)
Creates Author pages: if I’m making a new Source note and write author: [[New Author]], it doesn’t actually create a page for the author (unless I click on the backlink). My script looks through these linked-but-not-created pages and makes an Author page (empty apart from type: #author).
Creates Topic pages: same as for Authors.
Makes a list of notes that I need to review:
- Notes that I’ve tagged #todo.
- Orphan notes that aren’t linked to by anything and don’t link to anything.

I then set up an alias where if I run obsidian in terminal, it cleans up my library and gives me a report:

The code to do all this is remarkably simple – check out the Molecular Notes GitHub repo.

Polymer

One can also build more advanced applications. I built a spaced repetition (flashcard) tool called Polymer to help me review my notes in a different way.

Like other flashcard tools, Polymer flashes the title of an Atom or Molecule and I try to recall the contents. I then press “Show” to check my understanding, before clicking one of the Difficulty buttons to tell Polymer how hard I found it to recall the note. It then uses a dumb spaced repetition algorithm (explained below) to decide when it should next show me the note.

At a high level, the tool is structured as follows:

Streamlit dashboard as a frontend.
A simple JSON “database” to track my stats for different notes: when I last reviewed it, what my recall level was, etc.
A very dumb spaced repetition algorithm to construct a queue from the database: if I “fail” to recall the note, it gets pushed back 10 places in the queue; if I recall the note with difficulty, the note gets pushed back 20 places in the queue etc.

Areas of improvement

As we reach the end of the post, I want to briefly discuss some areas where I am currently improving my workflows.

Maps of Content

I currently don’t have good entry points into most of my Topics. For example, if I want to refresh myself on concepts related to Management, I tend to wander around the local graph.

In future, I would like to have better indexing via “Maps of Contents” (MOCs). A MOC is simply a semi-structured way to explore a particular Topic. Rather than leaving the Topic note empty, I should use it to organise notes. Here is an example of a MOC I have for the Volatility Topic.

I need to eventually make MOCs for other Topics too!

Integration with PDF highlights

I’m still trying to figure out if there’s a better workflow for integrating PDF highlights with my system. This could be useful when taking notes on journal articles and textbooks.

Currently, I do all of my highlights on the first pass, then manually type notes into my Second Brain on the second pass. This has some advantages (avoiding completionism, filtering for importance) but also requires a bit more effort.

MarginNote looks promising (thanks to Lyra for the suggestion) and there are some workflows which connect Obsidian to Zotero (see this video by Artem Kirsanov).

Ongoing events

Obsidian is very well suited to creating a garden of permanent notes. But in fields like finance, one must also track ongoing events and examine how certain narratives play out over time. This is a task that traditional systems like OneNote, MS Word, or even email threads are actually quite good at because by its nature, event-tracking is more “linear” with the flow of time.

One of my exploration areas: can we design a system that combines linear event tracking with an interlinked Second Brain? I have been exploring some workflows based on Daily Notes (a popular concept in Second Brain circles), but haven’t settled on anything concrete yet. There are some tools like reflect.app which aim to solve this but I haven’t really explored them.

Molecular Notes in teams

For me, working on my Second Brain is a highly individual activity. That said, it’s natural to wonder whether Molecular Notes could be useful in group settings as an alternative to a wiki.

I think one could make this work, provided:

The team is small: if there’s too much content it ends up being difficult to use
Everyone is on the same page about what “deserves” to go into the collective brain
The team puts a lot of effort into maintaining maps of content (see above) – this would be essential to allow someone to quickly review what’s in the second brain without having to wander around other peoples’ notes.

Conclusion

Having published these posts on Molecular Notes, along with my two previous posts on How I Use Notion and How I Read Books, I think I’ve said all that I want to say about productivity tools for the time being. Writing about personal productivity always feels somewhat self-indulgent; it inevitably comes down to “this is how I do things – you should do it too!”.

Thanks to Niamh Q, Callum M, Shiv G, Lyra G, Cedric C, Joseph C for the discussions and inspiration. Writing these posts has given me a great excuse to nerd out about productivity tools and human-computer interaction with you clever people!

Molecular Notes: Principles

2022-04-18T00:00:00+00:00

In this post I present Molecular Notes, a note-taking system I created to help me learn from diverse sources (books, textbooks, articles, courses), distil insights, and synthesise new ideas. Molecular Notes is how I approach my Second Brain – the body of concepts and ideas that are relevant to my understanding of the world, both personally and professionally.

In general, I try to seek out mental models rather than specific facts; when I talk to people, I want to understand what models they are using. In a way, this post is my attempt to convey the most useful thing I can: my mental model on thinking, accumulating information, and building mental models – a meta mental model so to speak. As much as I enjoy writing my little explorations into miscellaneous bits of finance and mathematics, I am under no illusion as to their value. They are designed to pique curiosity (primarily my own), explore a neat quantitative technique, or outline an interesting connection. This post has much more grandiose ambitions: to outline my philosophy and practice of learning.

Part 1 introduces the concept of Second Brains and the principles of Molecular Notes. I also compare Molecular Notes with Zettelkasten and Evergreen Notes, two other Second Brain systems. Part 2 covers the implementation: I discuss the software options and provide an in-depth walkthrough of my actual setup in Obsidian, including the practical workflows I use, along with a forkable GitHub repo so you can mimic my setup.

This will be a detailed post; I spend a fair bit of time introducing the subject, as well as laying the groundwork. Use the table of contents to navigate!

Thanks to Niamh Q and Shiv G for reading the drafts.

What is a Second Brain
Objectives
Molecular Notes
Why Molecular Notes matters to me
Who is Molecular Notes for?
Lay of the land – other note-taking systems
Failure modes
Conclusion

What is a Second Brain

My definition:

A Second Brain is a system for consuming and interacting with information (specifically, concepts and ideas) that I care about.

The way I define it, a Second Brain is a subset of “personal knowledge management” (PKM) – my definition does not include stuff like todos, goals, and plans. I bucket those under task management and project management instead (see my post on using Notion for these aspects of PKM). Note: this isn’t universal. Some people choose to include task management within their Second Brain, but I think it’s useful to think of them separately because the techniques and software tools used for Second Brains are quite different to the other verticals within PKM.

Let me break my definition down further:

“System”: I should have a process, rather than making ad hoc decisions about how to incorporate new information into my thinking.
“Consuming and interacting with”: my brain is not just a hard drive: it computes across a knowledge graph. A proper Second Brain must provide both functions.
“Information (specifically concepts and ideas)”: as mentioned above, my Second Brain does not deal with project/task management. It purely deals with concepts, insights, and ideas.
“That I care about”: the goal is not to have faithful representations of every learning resource I’ve ever encountered. Instead, I want a map of concepts that are relevant to my thinking (personal or professional).

I have been working on my Second Brain for several years, switching between software tools, ripping up my workflows, and evolving my approaches. I am writing this post now because I feel that I have reached a point of equilibrium for the philosophy of my system, although the mechanics are in “perpetual beta” – I’m always tweaking for improvements.

Additionally, I am confident that I can use the system to create value, both for personal development and professional learning. I maintained the system during my internships, using it to effectively learn about complex subjects and derive actionable insights.

Objectives

I, like most of my audience, am a knowledge worker. As Wikipedia puts it, the job of a knowledge worker is to “think for a living” – to consume information, process it intelligently (which may require drawing on existing information or learning new techniques), and then communicate the output to stakeholders.

The underlying goal of a Second Brain is to improve my ability to be a knowledge worker. This manifests itself in several specific objectives:

Rapidly learning complex topics from mixed media forms
Retaining insights and understanding
$O(N^2)$ growth
Synthesising new ideas
Encouraging positive learning habits

I address each in turn.

1. Rapidly learning complex topics from mixed media forms

I summarise it thus:

Having a process for consuming sources of information,

In a way that allows me to see how the pieces fit together,

While pointing to more detailed resources should they be needed.

The notes in my Second Brain should not contain all the details – they should only deal in the key ideas. In the internet age, what matters is knowing how concepts connect and knowing what tool to use for the job. If I ever need the details, I can simply refer to the original resource.

2. Retaining insights and understanding

On many occasions (particularly for university), I have put in the effort to truly understand a particular concept – grokking it at an intuitive level. This normally involves reading lots of different articles and scribbling ideas out on a piece of paper until the intuition clicks in my mind.

Then, a few weeks/months later, I realise that I have forgotten the understanding that I worked so hard to gain. This is an incredibly frustrating situation: it’s not trivial to rebuild this understanding since it required me to consult many different sources. A key motivation for the development of Molecular Notes was to prevent this from ever happening!

Specifically, a good Second Brain should allow me to rapidly achieve the same level of understanding about something that I once had in the past. This may be more difficult in situations where that piece of understanding is conditional on many other bits of knowledge, but even then, I should be able to rapidly traverse the local landscape of related concepts so that I may reconstruct intuition.

3. $O(N^2)$ growth

I don’t want my knowledge and capability to grow merely linearly in the number of things I consume. I want my knowledge to grow $O(N^2)$ – quadratically. Doubling the number of things I read should quadruple what I know.

How does one achieve this? With linked thought. Rather than treating each piece of knowledge in isolation, I want to try and link it with my existing body of knowledge. Each new piece of knowledge can potentially connect with all the other things I know.

This is quadratic because the total number of possible handshakes in a room of $N$ people is of order $N^2$ – specifically, $\frac 1 2 N (N-1)$. I want each new piece of knowledge to “shake the hands” of every other piece of knowledge.

Note: it is neither possible nor desirable to have true $O(N^2)$ growth – some pieces of knowledge are simply not related. Rather, this goal exists to emphasise the importance of linked thinking and the super-linear growth that it provides.

4. Synthesising New Ideas

Figuring out how a machine works is important. Better still is to be able to propose modifications and improvements to a system, to invent and innovate.

My Second Brain should not only enable me to get to grips with a complex topic but to have novel perspectives on the topic and to identify new directions to expand into. In the language of educational psychology, creation is at the top of Bloom’s taxonomy.

To be sure, synthesis is not entirely within the purview of the Second Brain system itself – it depends on the creativity of the user’s actual brain, so I don’t treat this as a sine qua non for a Second Brain system.

5. Encouraging positive learning habits

Simply put, the system should spark joy. This largely comes down to the choice of tool: it should be a tool that is customisable and that I enjoy using.

However, there are valuable insights deriving from the literature on addictive technologies. Consider Nir Eyal’s Hook model, described in his book Hooked. Eyal posits that Habit-forming technology requires:

Trigger: either external (e.g. notification) or internal (e.g. routinely checking the news when you wake up)
Action: behaviour done in anticipation of reward
Variable reward
Investment: user invests time into the product

To the extent I can build a system that follows the Hook model, it will be enjoyable and habit forming.

Summary – Objectives

That’s a lot of different things. But really what I want to strike at is this:

A Second Brain system should allow me to learn from textbooks, online courses, and articles in a way that helps me understand the concepts, and enables me to apply those concepts in my personal and professional life.

Molecular Notes

Over the past several years, I have experimented with different note-taking approaches, ripping my systems to shreds and starting again on numerous occasions. I finally settled on a system that I have come to call Molecular Notes.

I want to emphasise that Molecular Notes is an emergent system – it evolved from many little refinements over the past couple of years, rather than being designed top-down to meet the objectives I laid out earlier. On the one hand, this leads to a more stable and robust system (as is generally true for emergent systems). On the other hand, in writing this post I have found that the emergent nature of the system often makes it surprisingly hard to explain. I will do my best, but if you find some aspects of it unintelligible, the fault is my own. Please send me an email with the offending paragraph and I’ll try to reword it!

The three primitives in Molecular Notes are:

Sources: any source of information: a book, textbook, online course, podcast, youtube video, academic paper, presentation, etc.
Atoms: existing concepts or techniques. As a rule of thumb, these will often have Wikipedia articles.
Molecules (similar to “zettels” in the Zettelkasten system): crystallised insights, pieces of intuition, links between Atoms, my ideas.

These are organised with Topics.

The overall workflow of Molecular Notes is:

Mine Atoms out of Sources, make Molecules out of Atoms, organise everything with Topics.

Sources

A Source is any medium of information that I want to learn from. The main types of Sources that I deal with are non-fiction books, textbooks, podcasts, online courses and YouTube videos.

As I have written previously, my mental model is that Sources (especially books) are typically composed of:

Concepts: either original to the author or well-established concepts that the author is referencing as part of their thesis
Additional details, context, and examples to illustrate the concept.
Thesis: the author gives their commentary/explanation regarding how concepts link to other concepts.

My note-taking approach reflects this model. I want to capture (as briefly as possible) the core thesis of a Source, with reference to some key concepts, along with perhaps one or two of the best examples since these are often good mnemonic tools.

For example, in NNT’s The Black Swan:

Concept: Black Swans – events that are rare, extremely impactful, and intrinsically unpredictable.
Examples: NNT provides several examples from finance and history.
Thesis:
- Most of history is determined by Black Swans
- We should be cautious in learning from the past
- We should collect positive Black Swans

This model is not universal: some Sources, particularly technical ones like textbooks, focus on presenting concepts and techniques, without having a thesis. Other Sources, like opinion pieces, focus primarily on a thesis rather than explaining concepts. Additionally, we need to be a bit loose with our definition of “concept”; for example, Sources about history discuss events rather than concepts, but I mentally lump this under “concepts” nonetheless.

In any case, what I’m getting at is that Sources contain some pieces of knowledge that are “unique” to the Source, and some pieces of knowledge that exist separately to the Source (even if they are frequently referred to by the Source, or indeed invented by the author). Molecular Notes supports this distinction.

I will try to make this more concrete: when taking notes from several Sources, I find that many concepts are duplicated across Sources. For example, the Black-Scholes model is discussed in every textbook about options; the narrative fallacy is explained in many books on decision-making; Kuhn’s paradigm shifts crop up all the time. If I was to re-explain these concepts in every Source note that references the concepts, I would end up with a lot of duplication. Duplication hampers discoverability: the next time I want to re-acquaint myself with “paradigm shifts” by searching for it in my note-taking system, I would be overwhelmed as the definition shows up in 10 sources.

This leads to one of the key tenets of Molecular Notes: we should extract concepts into individual notes – Atoms – and link to them from the Source (modern note-taking software like Obsidian makes this trivial).

I will explain the philosophy of Atoms in detail in the next section, but I think an example of a Source note will be illustrative. Below is my note for Adaptive Markets by Andrew Lo:

The purple text denotes a link to an Atom I’ve made. For example, the Efficient markets hypothesis is a concept discussed in several Sources, so I created an Atom for it. In the bottom-right of the screenshot, you can see all the other books that point to the Efficient markets hypothesis.

Some judgment is needed to decide what should be extracted as an Atom and what shouldn’t be. As a simple rule of thumb, I extract an Atom if I think it is likely that other Sources will discuss it, or if I think I will link ideas to that Atom specifically. The process of deciding whether to extract a concept is useful in itself because it makes me constantly reason about my notes on a meta level – debating whether something is likely to be useful for my thinking and looking at a Source on a higher level of abstraction.

It is also important to allow for flux. Often my initial notes won’t contain many extracted Atoms until I later realise that a particular concept discussed by the author (for example, black swans), comes up in several other Sources. At that point, to prevent duplication, I extract the concept into an Atom and make all of the Sources link to that Atom. So even when I’m taking notes from textbook A, I’ve got panes containing my notes from textbooks B and C, trying to identify which aspects I can extract into Atoms. For example, I was recently working through an online course on econometrics (left-most pane) which referenced moving-average models. This sounded familiar and I realised I had covered it before in a textbook (the right-most pane). So I combined the explanations from the textbook and the online course into an Atom (middle pane), then made both Sources link to that Atom.

There are certain Sources (especially blog posts and Twitter threads) that only present one concept. In this case, it would be wasteful to create a Source note since it would only link to one concept. So I “cut out the middleman”, making an Atom and directly linking to the Source.

However, if this Twitter thread merely referenced Berkson’s paradox as a part of a broader thesis, I’d be tempted to make it a Source note that would link to the Berkson’s paradox Atom.

In Part 2 of this post, I provide a detailed workflow for how I create organised Source notes for different Source types. But for now, here are some guiding principles:

The notes should mention all the key concepts discussed by a Source, but should not explain all of these concepts (the explanation should go in the Atom for the relevant concept).
Aim to reduce the author’s thesis to its simplest form – I shouldn’t feel bad about reducing a book to 1-3 bullet points!
In essence, the note should tell me what is inside the Source, rather than repeating all the contents. My notes store pointers.
Regarding structure, I aim for simplicity and flexibility. Consistency is useful but overrated. I like to use nested bullets because they visually emphasise that there are different levels of abstraction. By scanning the top-level bullets, I should be able to quickly get a feel for the thrust of a particular Source. If I need more information, I can start descending into the bullets. If I need more information still, I should just open the original Source!

Summary – Sources

I consume Sources by making notes on them. While making notes, I extract concepts into individual notes called Atoms. These Atoms can be linked to by several Sources.

Atoms

An Atom represents an existing concept, technique, formula, framework, historical event, or historical figure. It is a piece of knowledge that doesn’t “belong” to me – it already exists, and the Atom is merely an explanation of it in my own words.

As a rule of thumb, Atoms are pieces of knowledge that could have their own Wikipedia page (indeed, many of my Atoms do have corresponding Wikipedia entries).

The functions of an Atom are threefold. It must:

Explain the concept to me, such that I can rapidly resume a level of understanding I once had (Objective #2 – Retaining insights and understanding)
Show how that piece of knowledge fits in with other Atoms (Objective #1 – Rapidly learning complex topics; Objective #3 – $O(N^2)$ growth).
Point to resources should I need more detail. Attached to each Atom should be a link to some Source (or even just a wiki page / URL) with the “best” explanation I can find for the concept.

Conveying understanding

I always write Atoms in my own words. This is a critically important part of understanding something. I aim to keep the Atoms very brief, such that I can rapidly get up to speed. In particular, I am not aiming to capture all the details: only the parts of it that are interesting or relevant to my thinking.

For example, the Wikipedia page for the Battle of Alesia (during Julius Caesar’s campaign in Gaul) contains about 4000 words and covers the historical context, details of the Siege, casualty numbers etc. Yet my particular interest lies in Julius Caesar’s tactics regarding the construction of concentric walls, so my note is a mere 90 words (in fairness, the context has been offloaded to another Atom for the Gallic Wars).

Showing how a piece of knowledge fits in with other atoms

I liberally link my Atoms to other Atoms. Additionally, most Atoms refer to a Source and are categorised with Topics, both of which allow me to better understand the context of a particular Atom.

Atoms as pointers

For each Atom, I provide a Reference. This is usually the Source note that discussed the Atom, but more generally, I want a link to the “best” resource for learning more. In the example above, if I wanted to know more about BLUEs, I would click into the Reference (in this case, my Source note for Ben Lambert’s online course in Econometrics), and look at the context in which BLUE was referenced. If that’s not enough, my Source note links to the original resource.

Summary – Atoms

In essence, my library of Atoms forms something like a personal Wikipedia, with several crucial differences:

My library only contains Atoms that are relevant to my thinking (in the sense that they originated from a Source I felt was worth reading).
Wikipedia pages aim to be comprehensive, Atoms do not. Atoms should only cover the aspects of a concept that are interesting or relevant to me.
Because an Atom is written in my own words, it should be easier for me to regain lost understanding.
Once an Atom is created, it can link to other Atoms and be linked to by other Atoms. This creates a network of ideas that can graphically demonstrate how concepts link to one another.

Molecules

Thus far, Molecular Notes has been nothing more than a system for principled note-taking that seeks to reduce duplication. This is good, but we still haven’t achieved all of the objectives.

1. Rapidly learning complex topics from mixed media forms
2. Retaining insights and understanding
3. $O(N^2)$ growth
4. Synthesising new ideas
5. Encouraging positive learning habits

The solution: Molecules (a.k.a “permanent notes” in the Zettelkasten system).

It’s difficult for me to explain what exactly a Molecule is. The best I can do: a Molecule is an insight. This insight should be self-contained and discrete (in the sense that it can stand on its own). My Molecules are typically only a couple of sentences – a paragraph at most. In most cases, a Molecule will “build on” one or more Atoms (which is why I called it a Molecule in the first place).

To be a little more concrete, here are several types of Molecules along with examples.

Original insights I have, often inspired by some Source. (When I say “original”, I mean “originated by me”. I doubt any of my ideas are original in the sense that they haven’t been thought of before).

Original insights I have, related to an Atom:

Links between two Sources, e.g. when I notice that two Sources are talking about the same concept in different ways:

Links between two Atoms:

Intelligent pieces of commentary related to a Source:

All of the above examples are “original” ideas or links between several pieces of knowledge. However, Molecules don’t need to be original. Many of my Molecules are crystallised insights based on a Source: moments of clarity I’ve achieved after stumbling through the conceptual thicket. I make Molecules like these when I feel that I know exactly the point an author is trying to get across and I can explain it succintly in my own words.

From Black Box Thinking by Matthew Syed:

From Euan Sinclair’s textbook Volatility Trading:

From Patrick O’Shaughnessy’s Invest Like the Best podcast, with guest Marissa King:

I will typically only make something a Molecule (rather than just a bullet point in the Source) if I feel like I have internalised the idea to the point where I would be comfortable calling it a piece of “my” intuition.

Summary – Molecules

Molecules are insights – either my own, or an insight I have internalised from a particular Source. Creating Molecules is the primary goal of Molecular Notes!

Topics

Along with the three primitives – Sources, Atoms, Molecules – Molecular Notes uses Topics (a.k.a categories) as an organising principle.

Categorisation is relatively controversial among note-takers. I want to briefly outline both sides of the debate.

The “spreadsheet” school of thought (my terminology) posits that the whole purpose of note-taking is to turn unstructured thoughts into something highly structured, presentable and reviewable (hence “spreadsheet”) – the more categorisation and organisation, the better. But many Second Brain apologists reject top-down categorisation (see e.g. Why Categories for Your Note Archive are a Bad Idea) in favour of emergent structure – they think that structure should emerge organically from the links.

I think the truth is somewhere between these two extremes: I think that both deliberate and emergent structure – categories and links – are useful. In my view, structure aids understanding because it allows me to build a better map of where a piece of knowledge belongs. Pure emergence (i.e entirely unstructured note-taking) makes it harder to revisit the content.

The problem with categories in traditional note-taking systems is that they often encourage a structure that is too rigid to support the concepts I am trying to learn. For example, in a traditional “files and folders” note system, it is hard for a note to belong to several categories: you’d have to duplicate a file in multiple folders, which seems wasteful. This leads to the “Categorisation paralysis”: when the inability to decide where to categorise some knowledge reduces efficiency (I am reminded of this scene in Wall-E, as he struggles to categorise a spork).

But in modern software, this concern melts away. The way I use Obsidian, categories are merely empty notes that I can link to: I can link to as many categories as I would like. For that reason, I typically think of these as Topics rather than categories – “category” feels more exclusive, whereas “Topic” just means that there is a related subject.

The advantage of Topics is that they give structure to my Second Brain and help me figure out which area of my graph to hunt for a particular concept or link. If I’m learning something about statistics, I will often have a look at the local graph of the statistics Topic to see if there are any interesting links:

If over time, I see that a particular Topic isn’t useful to me (e.g. I later find it was too specific or too broad), I just delete it!

In Part 2, I will go into more detail about how exactly I organise my Second Brain: in particular, how I use Topics, folders, and tags.

Summary – Topics

Topics are a simple and flexible way of introducing structure to my Second Brain. They make it easier to search my Second Brain and to find possible links.

Why Molecular Notes matters to me

Generally I’m quite critical of “productivity YouTube”. The classic archetype is a YouTuber explaining some productivity tool when it’s clear that that they are mostly using the tool to administrate their YouTube videos – something that is presumably not applicable to the majority of viewers. Or a Zettelkasten video where the YouTuber’s three zettels are “Niklas Luhmann”, “Zettelkasten”, and “Linked thinking” – i.e they are using the zettelkasten system as a prop in a video rather than an actual note-taking system. Andy Matuschak puts it thus: “people who write extensively about note-writing rarely have a serious context of use”. Cedric C brings up a similar point regarding mental models (link): there are far more people who write about mental models than people who effectively apply mental models.

These concerns were a real barrier when it came to writing this blog post – the prospect of becoming a “productivity YouTuber” terrified me. Even as I write this, I realise that I’m still unable to clearly articulate what Molecular Notes enables me to do, though I unequivocally maintain that I’ve derived immense value from it.

I guess it comes down to this: to the extent that you think I’ve got interesting ideas or a thought-provoking blog, or you like the way I think about things, all of that has been significantly enabled by my Second Brain. Perhaps not originated by the Second Brain, but at least amplified. Subjectively, I was a keen learner even before using Second Brain and I was decent at spotting connections between different things I had read. But codifying my workflows into Molecular Notes has dramatically accelerated up my pace of learning, even given the constraints of adulthood. I’ll admit this is a weak explanation. I’m essentially saying that I like using my Second Brain because I think it makes me more productive – a highly subjective and vaguely circular argument.

To be slightly more concrete, Molecular Notes gave me a tangible benefit during my internship at a hedge fund (as it continues to do in my full-time job at the same hedge fund) because it allowed me to learn about highly technical areas quickly, while spotting the manifold links to related fields – trading is surprisingly interdisciplinary at the end of the day, drawing on mathematics, computation, psychology, statistics, and economics (among others). I would love to be more specific, but unfortunately the constraints of discretion prevent me from doing so. Nevertheless, the key point I’m trying to convey is that I believe Molecular Notes has helped me in practice. It isn’t just “blog material” – in fact, I only wrote this post after several people independently asked for it.

Who is Molecular Notes for?

Inevitably, Molecular Notes evolved around my particular needs and interests: it is different to the Zettelkasten and Evergreen note-taking systems because my needs are different to those of Luhmann and Matuschak, their creators. It thus stands to reason that what works for me may not work for you, so you should approach Molecular notes with the same philosophy I approached the existing note-taking systems: a source of inspiration; a basket of features, some of which may fit your use case.

I want to highlight several areas in particular where I think Molecular Notes is not well suited (without modification).

For students taking closed-book non-essay exams (particularly STEM subjects), I recommend using a note-taking system that better matches the exam format and treats spaced repetition as a first-class citizen. In my university programme, it was important for me to have all of the concepts and techniques at my fingertips and to practice applying them. Knowing about the concepts on a meta level (in particular, finding connections between different techniques) was not terribly important – perhaps this says something about the exam format! Obsidian does provide spaced repetition functionality via extensions, and as I will discuss in Part 2, I coded up my own little spaced repetition tool to work with Molecular Notes, but in truth I think you are better off with a dedicated spaced repetition tool like RemNote. It was specifically designed for spaced repetition and they have full-time developers maintaining and improving the functionality.

Secondly, while I think many of the ideas in Molecular Notes are highly relevant to academics – particularly the ability to look for connections across Sources – more emphasis would need to be placed on collating references from articles. In academic writing, it is sometimes useful to have verbatim quotes from another article, while in Molecular Notes this is rare. If one’s goal is academic writing, the original Zettelkasten system may be a better fit.

Lastly, Molecular Notes is entirely optimised for evergreen content, as opposed to tracking ongoing events or dealing with new situations (remember, my Second Brain is not a project management tool). In Part 2, I will discuss how Molecular Notes could be modified to address these applications.

Lay of the land – other note-taking systems

Molecular Notes is based on a lot of prior art. Having presented the principles of Molecular Notes, I want to compare and contrast it to two other related systems: Zettelkasten (the OG) and Evergreen notes.

Perhaps after reading this section, you will feel that one of those is a better fit for you. As far as I’m concerned, that’d be an excellent outcome! We must each find the tools that best suit our goals and learning style.

Zettelkasten

How to Take Smart Notes by Sonke Ahrens (my highlights here) may end up being the most important book I’ve read because it significantly accelerated my interest in note-taking systems, thus acting as a force multiplier on everything I’ve learnt since. In the book, Ahrens outlines Zettelkasten, a note-taking system used by the sociologist Niklas Luhmann. Zettelkasten, implemented in Notion, was my first “thoughtful” note-taking set-up – though my system later began to deviate for reasons discussed below.

There is so much about Zettelkasten online (this is a good starting point) that I hesitate to add to the noise. But for the sake of contrasting it with Molecular notes, the main primitives in the Zettelkasten system are:

Reference notes: contain bibliographic information about a source, along with a brief summary of the source.
Literature notes: a summary of a particular point made in a Source. When you read a Source, you will generate many separate Literature notes.
Permanent notes (zettels): your insights.

Eva Keiffenham has a helpful diagram outlining the difference between literature notes and permanent notes in Zettelkasten:

If you would like to see a Zettelkasten system implemented in Obsidian, this video by Artem Kirsanov is the best walkthrough I’ve seen (be warned: many/most of the outlines of Zettelkasten on YouTube are mediocre at best).

The main reason I needed to evolve beyond Zettelkasten is that it lacks a good primitive to deal with established concepts. For example, let’s say I am learning about chemistry and I have just learnt about the Haber process to manufacture ammonia. This is important to my thinking, so I want to include it in my note-taking system. Zettelkasten would include this either as a literature note (referencing the textbook you’re reading) or in a permanent note. But neither seems ideal. If it’s in a literature note, then when I read another textbook with a slightly different discussion of the Haber process, am I going to have a separate literature note for it? Putting it in a permanent note (explained in my own words) makes more sense, but it feels weird to put a “known” concept with an unoriginal explanation into the Zettelkasten.

Secondly, many aspects of Zettelkasten arose out of the physical medium that Luhmann used (literal slips of paper in boxes). The decision to have Literature notes as separate individual notes makes sense in the context of physical slips of paper (writing a large document would make it hard to restructure), yet in a digital system I think it’s much easier to have all of my notes (or at least pointers to notes) from a given Source in one document. Luhmann’s original Zettelkasten also uses a nontrivial naming convention to represent backlinks – something that is clearly unnecessary with a digital Zettelkasten.

We should remember that Luhmann devised the Zettelkasten system to be a more productive sociologist. Sociology, as I understand it, largely involves the analysis of different qualitative theories – it deals with a smaller body of known facts/concepts and a larger body of qualitative discussion (this is not meant to denigrate sociology in any way). For STEM subjects, on the other hand, there are textbooks full of known concepts and techniques. I want to learn when to use these concepts and how they all connect – Zettelkasten is not naturally suited to this.

Evergreen Notes

Evergreen Notes is a system developed by Andy Matuschak, similar in many ways to Zettelkasten. Maggie Appleton has a nice visual explainer, but the ground truth is surely Matuschak’s own writing on the topic. If you are interested in note-taking, this is an absolute must read – I suggest you set aside an hour to trawl through his insights. I arrived at many of the same ideas independently, but I would have saved many hours had I simply read his thoughts to begin with.

I agree with most of what Matuschak writes, but there are a couple of areas where his system didn’t quite work for me.

Firstly, there are no Source notes. As Matuschak explains, his system is designed to optimise for creative thought rather than ingesting other peoples’ ideas so there isn’t a great way to take linear notes on a textbook, for example. I’m sure this is valid for someone as erudite as Matuschak, but given where I am in life’s learning curve, it’s very important to me that I have a good system for taking notes on educational resources.

Secondly, he doesn’t use categories of any kind (this criticism applies to Zettelkasten as well). He explains this in a delightfully-titled permanent note “Prefer associative ontologies to hierarchical taxonomies”, but for the reasons I described earlier, I think categories (Topics) are a good way to index my knowledge.

Following on from the above, the lack of structure makes his system (at least the public version) very hard to navigate, especially since it lacks search functionality and a graph view. Matuschak himself points this out.

Summary – Molecular Notes in context

I don’t want to overstate the novelty of Molecular Notes: it was heavily inspired by Zettelkasten, and incorporates many of Matuschak’s innovations (though most were arrived at independently).

The main additions of Molecular Notes that I feel aren’t well-captured by the existing systems:

Good workflows for taking linear notes from some resource (Sources)
Primitives for established concepts (Atoms)
A top-down structure, albeit a flexible one (Topics)

Failure modes

A recurring theme of this post is that you should modify Molecular Notes to work for you. But before wrapping up, I want to outline some failure modes that you should take care to avoid!

Categorisation hell:
- Don’t spend too much time trying to come up with a fancy categorisation scheme.
- Pick something simple and flexible – something that you can build on top of as you get a better understanding of your particular needs
Completionism:
- Don’t add irrelevant things to your Second Brain.
- It can be tempting to try and be comprehensive but this leads to notes that are harder to review
- Not every concept deserves its own note!
Complicated metadata:
- Don’t try to overcomplicate the metadata – when I create a new note I only need to put three fields: “Topics”, “Reference” and “Type”.
- Anything complicated starts to be a barrier to creating notes.
Copy-pasting content:
- Copy-pasting is a no-go: it defeats the purpose of a Second Brain.
- Further, it dramatically reduces retention while simultaneously encouraging completionism
Not being intentional about note-taking:
- Don’t slip into autopilot and make traditional notes!
- Molecular Notes requires you to actively engage with the content, restructure old Sources even as you’re consuming a new Source, and constantly be on the lookout for opportunities to link ideas.

Conclusion

Part 1 of the series has outlined my objectives for a Second Brain system and proposed Molecular Notes to address them.

I apologise if this post has been excessively abstract. In Part 2, the pendulum will swing the other way as I present my actual Obsidian setup in a sickening amount of detail. You have been warned!

Convexity in DCFs

2022-02-25T00:00:00+00:00

In this post, we revisit classical DCFs through the lens of convexity. This leads to the counterintuitive finding that increased uncertainty about an asset’s fundamentals can sometimes be a good thing!

Overview of DCFs

DCF analysis is used to value cash flow generating assets. The key idea is that we prefer cash today to cash a year from now, such that £100 next year is only worth $£100 \nu$ to us today, where $\nu = 1 / (1+r)$ is the discount factor and $r$ is the discount rate (for $r > 0$ we have $\nu < 1$).

In a simple case, consider a perpetuity: an asset that pays out a fixed cash flow CF every year for eternity. Because of discounting, this is a geometric series with finite value: $V = CF / r$. If the perpetuity is growing, the value is still finite as long as the annual growth rate g of the cash flows is less than the discount rate. In this case, we have $V = CF / (r - g)$.

Unlike fixed income products, the cash flows for equities are not contractually determined; they depend on the uncertain profits of a business. A stock analyst must estimate DCF inputs like the cash flow growth rate. In practice, it’s easier to break down “cash flow” into more tractable components, the most important being revenue growth, operating margins, and capital expenditure:

\[\text{revenue} \times \text{op. margin} - \text{capex} = \text{free cash flow}\]

Analysts tend to input point estimates into DCFs but because of the difficulty of estimating inputs like the future revenue growth rate, I’ve often been intrigued by the idea of using distributional inputs instead, e.g. using a revenue growth rate of $30\% \pm 20\%$ rather than $30\%$.

This is not an original thought: there are tools such as Oracle crystal ball that allow you to construct a valuation distribution using distributional inputs. Aswath Damodaran, NYU’s “valuation guru”, often includes such analysis in his valuations (a good walkthrough can be found on his blog). Yet my impression is that analysts struggle to derive actionable insights from the resulting value distribution.

Sensitivity analysis – re-running the DCF with different inputs – is commonly used by equity analysts. But I think even in this case, emphasis is placed on the specific numbers (e.g. “if revenue growth is 20% instead of 30% my price target is \$12 rather than \$20) rather than the shape of the DCF function.

An investment puzzle

To motivate this exploration, let’s use a little puzzle.

Consider two cash-flow-generating private companies Aco and Bco. Assume that each has the following characteristics:

Current cash flow is £10m/yr
Cost of capital is 10%
Each will have a high-growth phase for 10y, before settling down and growing at the rate of the overall economy (2.5% per year)
We expect the growth rate in the growth phase to be 30%

The only difference between the two companies is as follows:

We are very confident that Aco will grow at 30% per year
But for Bco, while 30% is our expected growth rate, we are uncertain: we have identified two equally likely paths for the company, one in which the growth rate is 35%, and another in which the growth rate is 25%

Which company would we rather buy?

Perhaps intuition suggests that we take the relatively certain option, after all, uncertainty is normally a bad thing. This is wrong. At least in this simplified model of reality, Bco is a better choice. In the next section, we will do some arithmetic to illustrate this.

(If this feels wrong and the phrase “volatility drag” comes to mind, worry not: this point is addressed later on).

Two-stage DCF model

The setup of the problem invites the use of a two-stage DCF model, in which a cash-flow generating asset is modelled as the present value of future cash flows, with the cash flows being generated in two phases: a high-growth phase and a perpetuity phase (still growing, but at the same rate as the overall economy).

This is simple to implement in a spreadsheet: using the input assumptions from above, we find that the intrinsic value of Aco is £1.02bn. If this seems crazy for a company that only has £10m cash flow initially, remember that the cash flow is compounding at 30% over 10y.

For Bco, we run a DCF separately for the bad case (25% growth) and the good case (35% growth).

Because the good and bad cases are equally likely, the expected intrinsic value of Bco is simply the average of the intrinsic values in each case. But at this point we notice something strange:

In expectation, Bco is 6% more valuable than Aco, despite having the same expected growth rate as Aco. For some reason, increased uncertainty in Bco’s growth rate has led to a higher expected intrinsic value!

DCF convexity

Uncertainty in the growth rate leads to a higher expected DCF value because the DCF formula is convex with respect to the growth rate.

Convexity

The high school definition of convexity: $f(x)$ is convex if its second derivative is positive, i.e. it curves upwards. An equivalent definition is that any straight line between two points is always above $f(x)$.

In a financial setting, $x$ is normally the price of some security (e.g. a stock price), and $f(x)$ is the value of some derivative contract whose payoff depends on the price of the underlying. For example, the plot below shows the price of a call option as a function of the underlying stock price (holding other inputs constant). The payoff diagram “curves upwards”, so the payoff is convex.

Yet the mathematical definition of convexity doesn’t entirely capture what traders mean when they talk about “convex bets”. Colloquially, a convex bet is one that “wins bigger than it loses”. Options possess convexity; they have limited downside (the premium you paid for the option) but potentially unlimited upside.

Stock prices go up and down a lot. The reason why options are “good” is that they make more money when the stock moves in one direction than they lose when the stock goes the other way. For example, we may have the following situation for a call option:

Stock up 10%, option up 50%
Stock down 10%, option down 20%

If you own an option, because of its convex payoff, you want uncertainty in the underlying. Considering the numbers above, given the choice between a stock that stays flat and a stock that goes up and down a lot, you would want the latter, because it “wins bigger than it loses”. This is why traders often say they are seeking convexity (though in reality what they mean is mispriced convexity – you have to pay to own an option’s convexity).

The main takeaway from this section: convex payoffs benefit from uncertainty/volatility. This is essentially the entire thesis of NNT’s Antifragile (my review and highlights here).

If you want to read more about convexity, I refer you to Kris Abdelmessih’s excellent post, which builds intuition and illustrates Jensen’s inequality.

The DCF formula

Because DCFs are almost always done in spreadsheets, you rarely see the DCF formula in the wild. Having written out the two-stage DCF formula below, I suppose that’s probably for the better:

\[PV = \frac{CF(1+r)}{r - g_0} \left[ 1 - \left(\frac{1+g_0}{1+r}\right)^{n+1}\right] + \frac{CF(1+g_0)^n (1+g)}{(r-g)(1+r)^n}\]

($CF$ is the initial cash flow, $g_0$ is the growth in the first stage, $n$ is the number of years in the first stage, $g$ is the perpetuity growth).

To show this is convex with respect to the growth rate, I could differentiate it twice with respect to $g_0$ and prove that the resulting expression is positive. That’s a lot of effort. Instead, here is a plot of the DCF value of an asset for different growth rates, fixing $CF = 10, r = 10\%, g=2.5\%, n=10$:

Note: the above analysis merely shows that the DCF formula is convex with respect to the growth rate. In principle, the DCF formula could be linear, concave, or convex with respect to the other value determinants. The plot below shows that DCFs are convex also to the discount rate, so uncertainty in the cost of capital is a good thing.

Is volatility good or bad?

The argument I have made so far can be summarised as follows:

The expected value of a convex function increases with uncertainty in the input variable. In other words, convex payoffs like uncertainty.
The DCF formula is convex with respect to growth rates.
Hence, uncertainty in the growth rate increases the expected intrinsic value of an asset.

Another statement is also true: volatility in the growth rate decreases the expected intrinsic value of an asset.

How can it be that uncertainty in the growth rate increase the expected intrinsic value, but volatility in the growth rate decrease the expected intrinsic value?

Arguably there are some semantical shenanigans going on in my distinction between uncertainty and volatility, but there is an important point in how I laid out the puzzle at the beginning.

For Bco, the two equally likely outcomes are:

Fixed growth rate of 35% for 10y
Fixed growth rate of 25% for 10y

This is equivalent to saying that we flip a coin whose outcome determines whether we are in a universe where cash flows grow at 35% every year for 10y, or a universe where cash flows grow at 25% every year for 10y. Critically, once the universe has been chosen, there is zero volatility in the cash flow growth rate.

Now consider a modified puzzle, where we choose between:

Aco: as before, cash flows grow 30% every year for 10y before entering perpetuity growth.
Cco: cash flows grow at $30\% \pm 20\%$, i.e. every year we draw the growth rate from a normal distribution $N(30\%, 10\%^2)$.

To value Cco, we can’t simply draw growth rates from the normal distribution and substitute them into the DCF formula (this is the whole point!): we must simulate different paths and calculate the DCF value of each path, before taking the mean across all paths.

The below chart shows how the expected intrinsic value of the asset varies with the volatility of the cash flow growth rate:

We can see that volatility in the growth rate (slightly) reduces the expected intrinsic value of the company.

This phenomenon is closely related to variance drag – mathematically, for growth rates drawn from a normal distribution $N(\mu, \sigma^2)$, the expected CAGR (geometric mean growth) is $\mu - \frac 1 2 \sigma^2$. This concept is nicely explained in one of Benn Eifert’s classic tweet threads. The situation is not quite the same as the typical variance drag setup because we are not considering the compounding growth of an investment: in particular, we are not saying that the company’s NPV is growing at a compounding rate, only its cash flows.

To summarise, we need to make the subtle distinction between an uncertain cash flow growth rate and a volatile stream of cash flows. Because of convexity, the former is good. But because of variance drag, the latter is bad. In real life, both will show up at the same time and pull in different directions!

Conclusion

The main lesson of this post is that you should pay attention to convexity because it turns uncertainty into increased expected value. When you own a convex payoff, you want uncertainty in the input parameter!

For equity analysts reading this, a more tangible takeaway is as follows: if you can see a credible blue-sky scenario for a company (e.g. a scenario that leads to a much higher growth rate or lower cost of capital), because of the convexity of DCFs, that blue-sky scenario might be so accretive to the intrinsic value of an asset that the bear case is irrelevant.

More generally, this post is a teaser for the type of insight that can be gleaned when we start considering distributional inputs to DCFs. I’m fascinated by the idea of using old tools in new ways and hope to explore other aspects of “probabilistic valuation” in future posts.

How I Read Books

2022-01-24T00:00:00+00:00

I recently launched a website to open-source some of my book reviews. To accompany this, I’d like to share some thoughts on my philosophy of reading books, and how my current workflow reflects this philosophy.

This post primarily focuses on non-fiction because it is an area in which I think I have “edge” – I am confident in my ability to extract value from non-fiction books. I do enjoy fiction (and discuss it briefly at the end), but I don’t feel qualified to teach someone how to appreciate it.

Motivation
Interlude: spaced repetition
Spaced repetition and reading
What to read
Reading the book
Processing highlights (a week after finishing the book)
Writing a review
Second brain
Fiction
Conclusion

Motivation

What do I want from a non-fiction book? I distil this into my reading triplet:

Retention: remembering what key ideas the book contains.
Understanding: internalising the ideas well enough to explain them to someone.
Contextualisation: knowing how the book relates to other books and how its ideas fit into my “second brain”.

My reading workflow has evolved over several years and hundreds of books to optimise for these goals. It has not evolved to maximise the number of books read, nor my retention of an individual book. I say “evolved” rather than built because it’s been an organic and iterative process, that is still being improved!

I truly believe that reading books, when done properly, can be a competitive advantage.

Interlude: spaced repetition

Before continuing, I will take a slight detour to discuss spaced repetition, a key component underlying my reading philosophy. If you are already familiar with the concept, feel free to skip this section.

After learning something new, we tend to forget it over time. In the late 19th century, psychologist Hermann Ebbinghaus tried to quantify this experimentally (rare for psychologists at the time!). One of the results of his research was the Ebbinghaus forgetting curve, which plots how retention changes as a function of time (it is roughly $e^{-kt}$).

More importantly, Ebbinghaus figured out that active recall of the content (testing yourself) not only refreshes the forgetting curve but makes it decay more slowly subsequently.

Image from Bo Ae Chun

This leads to the idea of “spaced repetition” – refreshing your knowledge at particular intervals to maximise retention. Spaced repetition is the core philosophy of flashcard systems like Anki and RemNote, which I heavily relied on during my time at university and truly were a meaningful contributor to whatever academic success I may have had.

Spaced repetition and reading

How does the idea of spaced repetition apply to reading books? In a personal productivity workflow, repetition is a bad thing. It takes up time, adds no value, and should be automated away. But if retention is a variable for which you are optimising, repetition is a good thing. As such, my reading workflow is deliberately inefficient (more on this later).

This is why I refuse to use Blinkist or read summaries in place of reading books (despite being the kind of person who consumes youtube/podcasts at 1.5-2x speeds). In a typical non-fiction book, the author spends several chapters building up the thesis and providing motivation for it, with loads of examples along the way. The repetition aids retention and the examples help you contextualise the information. This philosophy also suggests that you shouldn’t necessarily read books in one sitting – so I typically choose to read several books in parallel rather than reading sequentially.

Note that while I would not use book summaries to replace books – they can be excellent accompaniments. In the spirit of spaced repetition, listening to Blinkist or reading a summary a couple of weeks after you’ve read the main book is a great way of recalling the core ideas of the book.

What to read

I use Goodreads to track all of my reading: books I want to read, am reading, and have read.

I’m liberal when it comes to adding books to my “Want to Read” shelf: I add a book whenever someone gives me a recommendation, a book is referenced by people I respect, or I come across it randomly and it piques my interest. I’ve currently got about 100 books on this list.

When I’m ready to start a new book, I look at the reading list and consider some of the following factors to decide what to read:

Has the book been strongly recommended by credible people?
Is the book being referenced by other books I’ve enjoyed? This was why I reluctantly read Taleb’s books.
I keep a rough mental list of topics I’m interested in (e.g. finance, decision making, science fiction). Any classics in these areas are always fair game.
If I’ve read a couple of books on the same topic in a row, I’ll try to rotate to something else.
It’s important to sprinkle in some “rogue” books, completely unrelated to my interests. For example, I recently picked up a book on Danish interior design called The Book of Hygge (my review here).

Ultimately, I’m not as deliberate as I could be with book selection. I think it’s fine to be somewhat organic because it allows your minds to explore “threads” and build connections. Perhaps more importantly, it ensures that reading remains an enjoyable activity. If you have a predefined list of books that you are making yourself read, it becomes a chore – the most dangerous thing one could do in building a reading workflow is to become sick of reading.

Once I’ve decided what to read, I create a blank Notion page for the book (literally just the title).

Reading the book

As much as I love physical books, I read most of my books on Kindle. Even if I had a physical book, I would probably prefer to read it on Kindle (though I will sometimes get the physical copy to adorn my shelf!). The reason for this is the highlighting feature – I can’t understate how much of a game-changer this has been for me.

My guiding philosophy when making highlights is the “funnel”. I highlight very liberally when I’m reading the book:

Interesting new concepts
Good explanations for concepts that I was already familiar with
Anything that feels like a good summary of a key idea
Phrases I like (e.g. particularly well-constructed sentences)
Things that make me laugh
Things I don’t understand

But highlighting something once doesn’t obligate me to care about it in future. In the subsequent stages of my reading workflow, I delete lots of highlights, winnowing them down to only the key ideas.

While reading the book, I also make brief notes in the Notion page if something strikes me as a key idea (though I should be able to reconstruct this from my highlights). I also start thinking about the shape of my book review, though at this stage, the notes tend to be quite scattered.

Processing highlights (a week after finishing the book)

After finishing the book, I let it rest for a while, focussing on one of the other books I’ve been concurrently reading, or just cracking on with other things in life.

About a week later, I start processing the highlights. Firstly, I get the highlights from my Kindle onto my computer, using a little python script I wrote.

I then manually copy these highlights into Notion. Note: there are services like clippings.io or Readwise that can do all this for you, but I enjoy my hacky process and again, in the spirit of spaced repetition, the inefficiency of manually copying highlights isn’t a bad thing.

At this point, in my Notion page for the book, I’ll have some miscellaneous bullet points as well as a tonne of highlights.

The next order of business is to go through the highlights, culling the redundancies. For example, I may have highlighted the author’s first reference to a concept, only to realise that they offer a better summary at the end of the chapter. I only keep the latter.

After doing this, I try to group the highlights into different themes. This is a difficult task because it requires you to think on a more abstract level about the key points in the book and how they link together. I am often unable to do it one go; I try for a bit, get frustrated and give up, then return later. I strongly believe that this process, though uncomfortable, significantly improves my understanding of the book. Moreover, it makes it much easier for me to find a specific idea months later.

I also put in the effort to annotate some of the highlights, making a note of why it was impactful to me, what concept it reminds me of, what part of my life it applies to, etc. I use purple to emphasise certain parts of the highlight and write my annotations in italics below.

Writing a review

Writing a review is one of the hardest parts of this process. It entails wrestling with the content and mulling over the categorised highlights to not only understand the concepts the author is presenting but to critically review their originality and how they fit in more broadly with other books.

I don’t have much advice about writing reviews. It’s something that comes with practice and having written 100+ reviews I still struggle. But for each review I try to include:

The core idea of the book and maybe a couple of my favourite quotes
How this book is similar/different to others in the genre
Any interesting links I observed between this book and others (see the section on Second brain below)
How much I enjoyed the book

At the end of this process, I post the book to Goodreads, and now to my review site as well.

These little things gamify the process, providing a little dopamine hit. Remember – dopamine is the chemical for wanting more, not the chemical for satisfaction (that’s serotonin), so there’s value in associating dopamine with something productive.

Second brain

At this point, I have a book review and categorised highlights (see my book reviews site for examples). To maximise the value I get from the book as per my reading triplet, I then try to integrate the book with my second brain.

A second brain is a network of concepts and ideas that are relevant to your life and interests. I run my second brain in Obsidian. To fully describe my system would require a separate blog post, but it is loosely based on the Zettelkasten method, best described in Sonke Ahrens’ book How to Take Smart Notes (my review here).

Concretely, my process for integrating the book with my second brain is as follows:

Summarise the book
Extract atomic concepts
Growth

Summarise the book

The goal is to distil the author’s core thesis (maybe with a couple of examples to aid memory) in your own words. The points should not be original, but it’s important to use your own words, otherwise it is easy to copy-paste ideas without internalising them.

Below is an example of my summary for Adaptive Markets (my review here). I write this summary in Obsidian and sometimes duplicate it on my Notion site for the benefit of other readers.

Extract atomic concepts

My mental model is that a non-fiction book is made of concepts, examples, and commentary. The concepts can either be original to the author, or well-established. Examples are provided by the author to illustrate the concepts. In the commentary, the author discusses how those particular concepts relate to other ideas, or provides a critical perspective on those concepts.

My note-taking approach reflects this model. For example, I try to extract the concepts and examples into individual notes. This avoids unnecessary duplication because they often show up in other books. In the Adaptive Markets screenshot above, the purple text denotes a link to an individual note I’ve made. The efficient markets hypothesis is a concept discussed in several books, so I created an atomic note for it; the bottom-right of the screen shows all the books that point to this concept. Likewise, the Gazzaniga split-brain experiment is frequently used as an example in books on decision-making to illustrate our powers of rationalisation, so it deserves its own note.

This approach also allows me to better understand how the concept fits in with the idea landscape, for example, by looking at Obsidian’s graph view (in the top-right of the screenshot).

Growth

Now that the book is in the second brain, the process doesn’t finish. It instead becomes a “seed” in the garden, whose nourishment is derived from linking it to other concepts.

Using the graph tool, I look through related notes – book reviews, summaries of different web articles, ideas I’ve had – to try and find links. For example, after reading Ask Your Developer, a book about managing software firms written by Twilio CEO Jeff Lawson (my review here), I noticed that there were some interesting links to the design of decentralised systems, and therefore a link to German military strategy (auftragstaktik). I add these ideas to the summary page, or if they are meaningful enough, create a separate note for them.

This is the key to building $O(N^2)$ knowledge growth – each incremental piece of content you consume has N other ideas/atoms/sources to connect to.

A key part of growth is discussion! Nothing sharpens your understanding of a book, highlights your misunderstandings, or inspires new ideas like a discussion with someone else. They don’t need to have read the book – the process of trying to convey the key message of the book to someone else really helps your mind distil it, as per the Feynman method. Oftentimes after discussing a book with my friends I’ll come back and modify my summary to reflect newfound understanding.

Fiction

There are several reasons to read fiction.

Firstly, for “culture”. A cynical take is that the only purpose of reading the classics is to tell people you’ve read them. This probably used to be a source of motivation for me when I was younger, but I’ve matured a bit: it’s important to read classic/popular books so that you can understand references, and so that you can enrich conversations by pointing out a parallel to a well-known tale.

Secondly, fiction books often do a good job of illustrating philosophical concepts. Rather than reading a political philosophy textbook to learn the definition of an authoritarian state, you could instead read a dystopian classic. 1984, Brave New World, and Fahrenheit 451 give three completely different perspectives on an authoritarian state; having read all three allows you to make intelligent commentary on what characteristics the real world shares with each of them, and perhaps even gain insight into where we may be headed. I have learnt more about Japanese history from reading fiction (e.g. the work of Yukio Mishima), more about Russian history from reading Dostoevsky, than I would have learnt from non-fiction history books. My perspectives on AI, technology, and the long term future of humanity have been influenced by science fiction.

The last reason, of course, is for the entertainment value. This is purely subjective, but I think reading fiction books provides a different kind of entertainment to watching TV. It requires more effort but can provide deeper appreciation. In any case, it’s not a binary decision: one should read fiction books and watch TV!

Conclusion

My processes are in “perpetual beta” (a concept from Superforecasting by Philip Tetlock) – I’m constantly trying to improve my system for reading books. For example, I’m terrible at giving up books; I feel a kind of petty competition where I refuse to let the author “defeat me”, so once I start a book I feel obliged to finish it. This is clearly suboptimal – there are poorly written books out there and I could unlock a lot of value by giving up and starting a better one. To improve on this, as part of my 2022 reading goals, I have resolved to give up a certain number of books.

The last thing I’ll say is that (to paraphrase Orwell) you should ignore all of this advice sooner than do something barbarous – not reading! Even reading books without making highlights or taking notes is valuable; I didn’t really make highlights or take notes for the first couple of hundred books I read, and while I certainly would have gotten far more out of them had I applied my current workflow, I don’t regret reading those books.

Probability matching and Kelly betting

2022-01-10T00:00:00+00:00

In this post, we discuss a cognitive bias called probability matching, explaining how it is rational from a population perspective. We then make an analogy to the Kelly criterion, a betting strategy that finds widespread use in both gambling and finance.

Probability matching

I probably don’t need to tell you that humans aren’t perfectly rational utility-maximisers – thanks to the work of heavyweights like Kahneman and Tversky, the existence of cognitive biases is widely appreciated (in theory, if not in practice). Books like Thinking, Fast and Slow catalogue some of these biases.

Today I want to discuss one in particular: probability matching.

Consider a weighted coin that comes up heads 75% of the time. I am going to flip this coin many times; before each flip, you can guess whether it will be heads or tails. Your goal is to maximise the number of correct guesses. What is your strategy?

The optimal strategy is to guess H every time, which should mean that 75% of your guesses are correct.

But animals (humans included) seldom choose this strategy¹. Rather, they guess H about 75% of the time, matching their guessing frequency to the probability – probability matching.

Let the coin have a known $P(\text{heads}) \equiv p > 0.5$ and consider a strategy in which we choose heads with frequency f. Assume that we receive a payout of +1 for a correct guess and -1 for an incorrect guess. The expected value per coin toss is $EV = 1 - 2(f+p-2fp)$.

In the “optimal” case $f=1$ (i.e we always guess H provided $P(\text{heads}) > 0.5$), we gets $EV = 2p-1$. Probability matching, in which we set $f=p$, yields $(2p-1)^2$.

Hence probability matching is a cognitive bias to the extent that it is a mathematically suboptimal strategy that results in a lower expected value.

The rationality of probability matching

In their 2011 paper The Origin of Behaviour, Brennan and Lo make the wonderful insight that while probability matching is suboptimal from an individual level, it may be rational from a population level.

To motivate this, we’ll move from coin flips to a more ecological example.

You are an arctic fox trying to decide whether or not to change your coat from brown to white for the winter. If you have the wrong coat colour, i.e you turn white but it doesn’t snow, or you stay brown and it does snow, you get eaten by predators (sorry, I don’t make the rules). If you have the correct coat colour you survive and reproduce. The probability of no snow in winter is 75%. What is your strategy?

The best strategy for an individual fox is to always stay brown as long as $P(\text{no snow}) > 0.5$.

But consider the population perspective: if every fox stayed brown, then it would only take one winter with snow for the entire fox population to be wiped out. So the optimal strategy at a population level cannot be for every fox to stay brown, even though that is optimal for an individual!

Hopefully, this example illustrates intuitively why probability matching is a plausible strategy at the population level. In the next section, we’ll explore this idea mathematically.

Optimising population growth

Let’s build out the fox model a little more. We make the following assumptions:

The probability that it doesn’t snow in winter is $p$; the probability it does snow is $1-p$
If a fox survives the winter (by having the right coat colour), it reproduces to make $W-1$ offspring, such that one fox becomes W foxes (W for “win”!)
The “objective” is to maximise the population.
Before winter, we have a population of $n_0$. A fraction $f$ of this population remains with brown coats, the other $1-f$ change to white coats.

How does the population evolve after one winter? If it doesn’t snow, the $fn_0$ brown foxes survive and reproduce to become $fn_0 \cdot W$ foxes. If the winter is snowy, the $(1-f) n_0$ white foxes reproduce to become $(1-f)n_0 \cdot W$ foxes.

To summarise, the population dynamics are:

\[n_0 \to \begin{cases} f n_0 \cdot W, & \text{no snow} \\ (1-f)n_0 \cdot W, & \text{snow} \end{cases}\]

Over T winters, we expect snow in $(1-p)T$ winters and no snow in $pT$ winters. The overall population after these T winters is then:

\[n = n_0 \cdot (fW)^{pT} \cdot ((1-f)W)^{(1-p)T}\]

The key question – what fraction $f^* $ of the population should stay brown to maximise population growth – is then just a calculus problem. We just differentiate n w.r.t f and set it to zero to find the optimal $f^*$ (though for mathematical convenience, we first take logs):

\[\begin{aligned} \ln n &= \ln n_0 + pT (\ln f + \ln W) + (1-p)T (\ln (1-f) + \ln W) \\ \frac{\partial \ln n}{\partial f} &= \frac{pT}{f} - \frac{(1-p)T}{1-f} = 0 \\ &\implies \frac p f = \frac{1-p}{1-f} \\ &\implies \frac{p}{1-p} = \frac{f}{1-f} \\ &\implies f^* = p \end{aligned}\]

So a fraction $f^* = p$ of the population should stay brown, leaving them best adapted to a winter without snow (which occurs with probability $p$) – this is exactly the probability matching rule!

Kelly betting

We now modify the model slightly.

In a snowy winter: brown foxes die, white foxes survive but don’t reproduce.
In a non-snowy winter: all foxes survive, but only the brown foxes reproduce (to make $W-1$ offspring as before)

This model is admittedly somewhat contrived, but is vaguely plausible:

A snowy winter is likely a tougher environment with less food available (so even the well-adapted white foxes may not be able to reproduce easily)
In a non-snowy winter, adaptation still matters, so the maladapted white foxes may not be desirable mates.

With this model, the population dynamics are as follows (f is again the fraction of brown foxes):

\[n_0 \to \begin{cases} n_0 (1 + fW), &\text{no snow} \\ n_0 (1-f), & \text{snow} \end{cases}\]

The population growth equation is:

\[n = n_0 (1+fW)^{pT} (1 - f)^{(1-p)T}\]

We repeat the procedure as before, choosing the $f^*$ that maximises $\ln n$.

\[\begin{aligned} \ln n &= \ln n_0 + pT \ln (1+fW) + (1-p)T \ln (1-f) \\ \frac{\partial \ln n}{\partial f} &= 0 \implies \frac{pW}{1+fW} - \frac{1-p}{1-f} = 0 \\ &\implies f^* = p - \frac{(1-p)}{W} \end{aligned}\]

This is identical to Kelly betting, with the following mapping:

n is the value of the bankroll
f is the fraction of the bankroll that we bet
W is the odds of the bet – the factor by which the stake grows if we win.

We can then apply the plethora of known results for the Kelly betting scheme² to understand the long-run growth trajectory of the population, the time it takes for this strategy to dominate others, etc.

However, note that in this model, probability matching is never optimal, since there is no finite W such that $f^* = p$. That said, we see behaviour similar to probability matching: for $p = 0.75$ and a reproductive factor $W = 5$, the optimal fraction is $f^* = 0.7$.

To take a step back, the reason why this ecological model is similar to Kelly betting is that in both cases the objective is to maximise the geometric growth rate – while Brennan and Lo (2011) observe the same, they don’t elucidate the link between probability matching and the Kelly frequency.

Conclusion

We’ve explored some simple models to illustrate that probability matching is rational from a population perspective. The population seeks to maximise its geometric growth rate, and therein lies the link to Kelly betting.

To clarify, probability matching is still irrational for an individual – if you are a fox you should change to a white coat if you think that there is a greater than 50% chance of a snowy winter. But your genes would be telling you otherwise!

This topic is so interesting to me because it is a first-principles derivation of a cognitive bias: it would be awesome if we could explain the full spectrum of cognitive biases at a biological level (be that neuroscience or evolution). Prospect theory is really only a phenomenological theory – it uses the S-shaped utility curve to explain several empirically observed biases, without rigorously explaining where that utility function might come from.

Lastly, a comment on an important extension. In this post, we found that the probability matching rule arises from uncertainty about the future state of the world, but we have completely neglected parameter uncertainty. In other words, we assumed that we know p, while in reality we may not know the bias of the coin, or the probability that it snows – these are parameters that must be estimated under uncertainty. In this case, answering questions about optimal strategies becomes difficult. It is an explore vs exploit tradeoff – you need to estimate p while simultaneously making decisions whose payoff depends on p! Techniques like reinforcement learning become applicable here, and ultimately these are likely to be a key part of understanding where cognitive biases come from.

References

Benoit Hardy-Vallee gives a nice overview of some of the evidence in this blog post. ↩
see Thorp’s classic paper ↩

How I use Notion

2021-09-18T00:00:00+00:00

This post is a fairly comprehensive discussion of how I use Notion (a free personal knowledge management app) to organise various aspects of my life: project management, reading, academics, plans/goals, investing, and more. The post is not designed to be read linearly – pick and choose the bits that are relevant to you.

Inevitably, this is going to sound like one massive ad for Notion. I have no affiliation with Notion, I’m just a very happy user. It’s certainly not a perfect piece of software, but it has radically improved my productivity and efficiency.

The philosophy of Notion

I firmly believe that a good piece of personal knowledge management (PKM) software is an integral aspect of productivity and general efficiency in the modern day. The reality is that we consume and generate vast quantities of knowledge, whether that’s personal plans, projects, or work stuff. PKM apps aim to help you organise this, freeing up RAM in your brain.

We are fortunate to be living in the golden age of PKM. For several years, Evernote and OneNote were the only real players (and these were essentially just linear extensions of the classic “folders and files” paradigm). Recently, however, many apps have sprung up which have dramatically expanded the horizons of PKM – apps like Notion, Roam, Obsidian, RemNote, and many others.

Although each of these apps serves a slightly different purpose, one commonality is that they have evolved far beyond simple note-taking. Notion provides extremely flexible “databases”, which can be viewed as tables, lists, or Kanban boards. Roam and Obsidian emphasise the linking of ideas, allowing you to build (and visualise!) your second brain. RemNote automatically generates flashcards based on your notes, allowing for optimal learning via spaced repetition.

I use Notion, Obsidian, and RemNote – each for slightly different purposes. This post focuses on Notion. I don’t expect that my entire workflow will be interesting to everybody, so please use the table of contents below to navigate to the specific areas that are relevant to you. The list is in approximate descending order of how much I use each piece of functionality.

A caveat: I am constantly refining my systems. There is no guarantee that my workflow will look the same a year from now. Similarly, if you are new to PKM, don’t be daunted by the apparent complexity – Rome wasn’t built in a day. I started with a blank slate, then incrementally added things that I thought would be useful; I watched youtube videos in which various productivity-tubers explained their workflows, and picked different parts that I like; I went down rabbit holes setting up complex Notion tables, then scrapped them in favour of simpler setups.

Ultimately, the core philosophy of PKM is that you need to find a system that helps you achieve what you want to achieve. It is unlikely that what works best for me will also be optimal for you, though there may be certain motifs that come in handy.

With all this out of the way, let’s begin!

Prelude: intro to my Notion workspace
How I use Notion for projects
How I use Notion to track books
- Other content consumption
How I use Notion for short term plans
How I use Notion for long term goals
How I used Notion for academics
- Tracking lecture progress and problem sheets
- Exam revision
How I use Notion as a second brain
How I use Notion for investing
How I use Notion to track my learning
Conclusion

Prelude: intro to my Notion workspace

Before diving into specific use cases, I will briefly explain how Notion works and give a high-level overview of my setup.

The fundamental unit in Notion is a content block. Notion has a huge list of possible content blocks. Here is a small subset of them:

text and text derivatives, e.g. headings, bulleted lists, toggle-able bulleted lists
all kinds of media (embedded images, videos, audio etc)
code and equations (typed in LaTeX)
tables and Kanban boards (popularised by Trello)
calendars

A page contains as many content blocks as you want, arranged however you want. What makes Notion so powerful is that pages can be arbitrarily grouped and nested. For example, each of the items in a Kanban board is itself a page, which you can click and open. If you wanted, you could therefore have a Kanban board where each item contains its own Kanban board etc (though I can’t think why you’d want that!). Below is an image of my Projects and Learning page (more details later), which illustrates nested pages and mixed block types.

The page I visit the most is the Dashboard – a home page of sorts. It links to many of my other pages (which house subpages and/or link to other pages).

Each page can be given an emoji and cover photo. I initially thought this was gimmicky, but it’s a little thing that truly sparks joy and encourages me to stay organised.

I also have an Inbox page, which is essentially where I dump stuff that I may want to revisit later. This includes:

Media to process e.g. URLs, papers, podcasts episodes etc.
Useful web tools
Cool projects/startups I’ve come across
Random thoughts or ideas
A list of blogs that I like

How I use Notion for projects

One of Notion’s key strengths is project management. This is because administrating a project requires you to keep track of a tonne of information in different formats: important tasks, long-term goals, strategy brainstorming, admin details, historical progress, etc.

Notion’s flexibility means that you can view each of these data types in the most natural way, for example: visualising tasks using Kanban (Trello) boards, creating tables to store a list of documents, or having simple text files to store deployment details.

In this section, I give three examples of projects that I manage using Notion.

PyPortfolioOpt

PyPortfolioOpt is an open-source python library that I developed. I use a Kanban board in a Notion page to keep track of the features I am working on.

Each item in the Kanban board is itself a page, which can contain any other content blocks.

In addition to the Kanban board, I use toggle lists to keep track of PyPortfolioOpt’s professional users and citations, and have various subpages to draft publicity or explore derivative projects.

The “Admin details” subpage contains instructions for deployment, builds etc. After a few months away from the project, this is the kind of stuff I forget!

This blog

Notion is very useful for managing the writing process of blog posts. At any given time I have a bunch of ideas, half-written posts, etc. A Kanban board helps me prioritise which to work on.

Within each Kanban subpage, I write out a rough plan for the post and jot down any relevant notes or ideas.

Collaborative projects

Notion is also excellent for collaborative projects. In the free plan, a page can be shared with up to 5 guests.

Earlier this year I was exploring a business opportunity with some colleagues (related to crypto market making). We used Notion to assign tasks (with due dates) to different members, and we could comment on each other’s pages and blocks.

I also used Notion to build a resource library for the project. Each row represents a piece of content (e.g. a book or journal article). Once read, you add your name to the “read by” column. We can then perform complicated filtering on the database. In the example below, I filtered to find the resources that nobody had yet read and saved this as an “unread” view – allowing us to track our collective blind spots.

For the same resource database, I created another view (same data but different way of looking at it), filtered to find the resources that I hadn’t read (sorted in descending order of priority).

Although I’ve only used Notion for one collaborative project, it was a fantastic experience.

How I use Notion to track books

I am obsessed with reading books. But I don’t just want to read, I want to retain the information I consume and incorporate the ideas into my thinking. My overall workflow for reading books is as follows:

Read the book on Kindle, highlighting as much as possible. (When reading physical books, use sticky notes instead).
After reading the book, collate all the highlights. For kindle highlights, I’ve got a python script that pulls the notes into text files, though if most of your books are from Amazon it may be easier to use something like Readwise.
Copy-paste these highlights into a Notion page for the book.
Reorganise the highlights, grouping them by topic so it’s easier to revisit them in future.
Write a book review within the Notion page and post this to Goodreads.
Summarise the book in my own words (ideally in ~5 bullet points) and put this summary into Obsidian.

This workflow is not as efficient as it could be – for instance, I could probably automate the transfer of highlights from my Kindle to Notion. But actually, I appreciate the “slowness” because it forces me to pay more attention to the highlights, improving my retention.

This is my Books database: it contains all of the book pages, tagged by genre (along with some other metadata).

Below is a page for The Success Equation by Michael Mauboussin. Some things to note:

I try to write a one-sentence headline summarising my takeaway from the book. This can be difficult, but it forces me to internalise the key message from the book.
Notice that within the book review there are clickable links to other books. Making links between different things I read is incredibly important to me – it makes me more creative, and invites a deeper contextual understanding of ideas.
The act of categorising quotes by topic can also be quite draining but it’s very worthwhile retrospectively when I’m trying to resurface a particular idea/theme.
Using author tags and genre tags allows the book to show up in filtered searches.

The final step of my workflow is to write a summary (different to the book review) in an Obsidian page. Obsidian is my “second-brain”, containing all my linked ideas: the graph view shows how the particular piece of content fits into my network of content/ideas.

How I use Notion for short term plans

There are two primary ways I use Notion for short term plans: daily plans and daily intentions (the two are linked)

A daily intention is one well-defined thing that I hope to achieve on a given day. This can be work/school-related (tends to be the case for me), but occasionally I put in fun ones like “meet a new person” or “check in on a friend”.

Caveat: I have likely bastardised the “daily intentions” concept. If you google “daily intentions” you may see things like “I intend to be present / to have fun today / to be valued today”. Rather than having such ambiguous and saccharine intentions, I prefer to have specific practical goals and I aim for a hit rate of about 50% – that is, I choose goals that are sufficiently stretching that I only achieve about half of them. This can be discouraging, but my view is that if you’re achieving all your goals, the goals aren’t ambitious enough.

These daily intentions link with my daily plans. Every evening, I reflect on the previous day, write a comment, and give it a quality rating. I then create tomorrow’s daily plan and daily intention. The example below shows one of my daily plans shortly before final exams (“tripos Qs” means past exam questions).

How I use Notion for long term goals

I mentally break out long term goals into three categories:

Specific things I want to do (e.g. preparing for an internship, planning a holiday). I view this as a project management exercise, so use the structure described above.
General long term goals e.g. New Years resolutions. I use simple text pages for this (could also be done in any other software).
Blocks of time that I want to fill productively (e.g. term holidays, lockdowns etc). This merits some further discussion.

The below table is a list of things I wanted to get done during the “Rona holiday” – the period in May-July 2020 when Easter term at Cambridge was “cancelled” due to Covid.

Tasks were tagged by “purpose” – what long term area of my life do they contribute to? This encouraged me to stay balanced and approach personal development from a more holistic perspective (rather than solely focusing on e.g. careers).
I gave each task a deliverable. Instead of passively watching an online course or leafing through a textbook, I challenged myself to transmute that content into a blog post, set of public notes, or code.

How I used Notion for academics

Notion played a critical role in helping me achieve my academic goals. For context, this section pertains to the Cambridge Part II Astrophysics tripos (i.e 3rd-year Astrophysics).

My overall study plan was as follows:

Watch lectures and make handwritten notes based on them (these notes need not be comprehensive).
Apply this knowledge to the weekly problem sheets, paying special attention to weaknesses.
Make comprehensive flashcards with everything that needs to be memorised.
Do a tonne of past papers and flashcards.
Profit

Some caveats:

I don’t use Notion for taking class notes – for that, I use Notability on my iPad to make handwritten notes, and RemNote to make flashcards. I only used Notion to help administrate the execution of the above plan.
The overall philosophy of a physics course may be quite different to that of other subjects, so the above plan may not be relevant to you (in which case the Notion setup will be even less relevant).
Even if you’re studying the same course, my usage of Notion was heavily optimised for the particular format of the Cambridge course.

Here is a basic overview of the structure of the Astro course:

Astrophysics is split into eight compulsory lecture courses.
- At the end of the year, each exam paper contains one question from every course. You may only submit up to six questions, but three complete questions can get you a 1st class.
- You can thus choose not to study certain courses – the only downside is that you’ll have less choice in the exam.
- Exam marks are nonlinear, rewarding complete answers. Getting 16/20 is much more valuable than getting 14/20. This is a key factor in exam strategy, encouraging judicious specialisation before the exams.
In terms of teaching, there were 12 lectures a week, supplemented by two supervisions. A supervision (supo) is a small group session wherein 2-3 students discuss their problem sheet attempts with a supervisor (normally a PhD student or professor). Supo work is handed in several days before the supo and represents the bulk of the workload during term.

Given these rules, my goal was to “game the system” to maximise my academic performance for a fixed amount of effort, strategising around the constraints to see where I could find a competitive advantage.

Tracking lecture progress and problem sheets

For each lecture course, I used Notion to track how many lectures I had watched, as well as how many lectures were required for me to complete the next problem sheet. This meant that even as I slipped behind in lectures (a common occurrence) I always knew how far behind I was.

Each lecture course had its own page, which I used to track:

Areas that I wanted to revisit, e.g. concepts/techniques from lectures that I didn’t understand the first time around.
Important derivations that I’d need to be able to reproduce in an exam.
Which subtopics within the course I would “bin” (i.e not answer an exam question on), and which subtopics I was weak at (would only choose that exam question if there was nothing better).

To organise my progress in making flashcards, I created a new view on the same table to track how many pages of my notes I had converted into flashcards. This screenshot was taken after the exams, hence the 100% progress.

Exam revision

When it came to exam preparation, I needed a system that allowed me to:

Understand what type of questions tend to come up. Is there any pattern or periodicity that can be exploited? For example, perhaps every year there is always a question on perturbation theory, or every three years there’s a question about phase transitions. Are there certain derivations that frequently show up?
Figure out which questions I’m good at – in particular, which questions I can do quickly and accurately (time tends to be the limiting factor in the Astro exams).
Identify what I need to remember and what I should be able to derive.
Highlight weaknesses – either so that I can work on them, or so that I can avoid these questions in the exams.

To that end, I built a big Notion database that tracked my attempts at past exam questions.

Each exam question is a subpage in which I tracked:

Metadata, like what course and what year/paper the question belongs to
The course chapter to which the questions belong
How long it took me to do the question and how hard I found it
Whether I did the question “properly” (exam conditions)
Whether I had subsequently revisited the question. There is no point in doing exam questions if you don’t revisit your attempts and learn from them.

Furthermore, I summarised what each part of the question is asking for, and noted down anything specific I needed to revisit.

Having a rigorous tracking methodology helped me get more out of the past paper questions by ensuring that I actively reflected on the question content and my attempts, building up a mental picture of my strengths and weaknesses.

However, the game-changer was that after a certain point, I had a large Notion database containing both the content of the questions as well as my performance on these questions. I could then perform complex queries on this data to refine my exam strategy.

For example, I could filter by course and by paper to identify patterns. The screenshot below shows a filter of the paper 1 quantum mechanics questions (across all years). This filter helped me identify that paper 1 QM questions typically involve harmonic oscillators, which were normally easy or medium difficulty.

Another example: after completing the first couple of exams, I wanted to find all questions within the galaxies course related to topics that hadn’t yet come up, since these topics had a very high probability of coming up on the last papers. This is a simple filter:

The unfortunate truth is that anytime an exam format gives you discretion (e.g. about which questions you can answer), exam strategy becomes just as important as knowing the content. Notion is a fantastic tool for refining this strategy and making data-driven decisions.

How I use Notion as a second brain

After reading How to Take Smart Notes by Sonke Ahrens, I became aware of the “zettelkasten” note-taking method and built it in Notion.

There is a tonne of information about zettelkastens out there, so I will just link to a youtube video by Mukul Khanna that implements zettelkasten in Notion (I basically duplicated his setup).

Full disclosure: I have since migrated my zettelkasten to Obsidian. Perhaps I’ll do a separate write-up on this in future!

How I use Notion for investing

Notion helps me manage several aspects of investing/trading:

Tracking ideas related to specific themes/styles (e.g. covid, spinoffs), as well as reflect on my trading performance.
Conducting investment research
Storing investment pitches for posterity, so I can better understand my strengths/weaknesses as a trader.

Tracking themes

I use subpages to track specific themes. For example, covid obviously created major dislocations and I made a Notion subpage to track my thoughts on it. Separately, I became interested in spinoff stocks so created a table to track active situations:

Investment research

To manage my investment research, I have a table containing subpages for each asset (tagged by asset class).

Each subpage contains links to resources, as well as my thoughts on the investment.

For equities, I use the structure discussed in my Equity Investing Checklist.

Investment pitches

After doing my research, if I want to add an idea to my portfolio, I write a pitch. This contains:

One sentence introduction to the company (industry, current share price, current multiples)
Investment thesis and key levers.
How my view of the levers differs from consensus
Risk factors (identify risks using a pre-mortem analysis)
How the company fits into my portfolio (in terms of diversification or sector risk)
Price target and exit conditions

Putting pitches into writing is a fantastic habit to get into. It keeps you accountable and helps you distinguish between skill and luck.

I keep all of these pitches in a table partitioned into open and closed positions. For the closed positions, I tag it based on whether it was a success or failure (note: if a stock I bought went up but for reasons other than what I pitched, I flag it as “neutral”).

How I use Notion to track my learning

I use Notion to track things I want to self-study (e.g. using online courses or textbooks). There is nothing special here: I use a Kanban board, and each page contains links to various learning resources (not notes though – those would go in Notability, RemNote, or Obsidian).

Sadly, free time to self-study things is hard to come by – hence the low position of this section within the post.

Conclusion

I hope this post has given you some ideas as to the kind of things Notion can be useful for! Writing this post has made me realise just how important the app is for my productivity.

At a philosophical level, I think Notion is a wonderful example of how “software eats the world”. In the past, note-taking apps like OneNote or Evernote were just digital equivalents of existing systems: folders, notebooks, pieces of paper. But at some point there was a phase change: apps like Notion let you do things that are impossible in the physical world.

I consider myself to be a Notion power user, but at times I feel like I’m only scratching the surface of what the app can do. For example, you can build and host websites in Notion (with no code), and with the recently-released API, the spectrum of possibilities has widened even further.

Hypothesis testing in quant finance

2021-06-17T00:00:00+00:00

At its core, science is about making falsifiable hypotheses about the world (Popper), testing them experimentally, then using the experiment outcomes to refute or refine the hypotheses. The scientific method is an integral part of quantitative finance; it provides a framework we can use to identify and analyse trading signals or anomalies.

In this short post, we discuss a general method for hypothesis-testing in finance, using Monte Carlo simulations to compute the probability that an observed signal can be explained by random chance.

This post is an expanded version of a Twitter thread I wrote. The supporting code can be found in this notebook.

Hypothesis testing and p-values

As a necessary prelude, we first discuss a somewhat controversial topic: the p-value.

Wikipedia defines the p-value as:

the probability of obtaining test results at least as extreme as the results actually observed, under the assumption that the null hypothesis is correct

With a definition like this, it’s no surprise that p-values are so widely misinterpreted. We can try and understand the intuition behind p-values by considering an example.

Let’s imagine that we want to find out whether some new teaching method X (e.g. forcing students to make flashcards) improves test scores versus the default method Y. A simple approach is to split the cohort of 100 students into two groups, making 50 of them do X and the other 50 do Y. We can then compare the average score between the two cohorts.

The problem arises when we try to interpret the results. If the X students had an average grade of 70%, while the control group had an average of 60%, did X really improve test results or was it a random fluke? How about if it was 60.1% vs 60.0%? Or 80% vs 50%?

This is where hypothesis testing comes in. We set up a null hypothesis and test whether there is sufficient evidence to reject the null hypothesis. In our example, the null might be that there is no difference between the mean scores of the X and Y groups.

To test this, we assume that the null hypothesis is true – that X doesn’t actually improve test scores – and compute the probability that we would see X have an average score of 70% vs the control group’s 60%. This probability is the p-value.

At the risk of being imprecise (and thereby perpetuating the misinterpretations of p-values), I think of the p-value as the probability that random data can explain your results.

The trouble with p-values

One of the major criticisms with p-values is that they answer the wrong question. We aren’t truly after “the probability that random data can explain your results”. We actually want to know the probability that X causes test results to improve, given the observed data. To answer questions like this, a different box of tools is needed – I’d highly recommend Judea Pearl’s The Book of Why if you’re interested in learning more – but for now we will proceed under the assumption that the p-value is indeed a statistic of interest.

Even then, there is a subtle issue that we glossed over above. The p-value can only be calculated if we assume that the quantity of interest is drawn from a particular probability distribution. In our example, we can’t actually “compute the probability that we would see X have an average score of 70% vs the control group’s 60%” unless we make some assumption about the distribution of test score (typically a normal distribution or t-distribution is used).

In finance, these types of assumptions rarely hold. Furthermore, we are often looking to test more complicated hypotheses, such as “this time series is stationary” or “this data is normally distributed”. You might be lucky enough to find that someone has already figured out how to test a particular hypothesis – for instance, the augmented Dickey-Fuller test can be used to test for stationarity and Jarque-Bera for normality. But generally, properly calculating p-values is hard!

The good news is that we often don’t need to properly calculate a p-value. Monte Carlo simulations can give us “quick and dirty” estimates of the p-value that might be good enough.

Case study: ETF seasonality

I came across a tweet from Mitchell Rosenthal in which he observes that the iShares Singapore ETF (ticker EWS) shows positive returns in 89% of April months from 2002-2020. This seems like an interesting anomaly, but with only 19 Aprils, it certainly seems plausible that this anomaly could be generated by random chance alone (remember: the probability of this is quantified by the p-value).

Firstly, let’s replicate his findings. I downloaded data for the EWS ETF going back to 1996 and calculated how often each month had a positive return.

def compute_pct_up_months(df):
    # df is a dataframe of daily close prices
    df_mth = df.resample("M").last()
    month_up = df_mth.pct_change().dropna() > 0
    pct_up = month_up.groupby(month_up.index.month).mean()
    return pct_up

A bar chart showing the fraction of up-months is shown below:

The percentage of years in which a given month showed a positive return (EWS)

81% of April months showed positive returns. There is a discrepancy with respect to Mitchell’s results of 89%, likely because I used more data and a slightly different methodology for calculating returns. However, I think it is fair to say that we have replicated the anomaly.

With this in mind, a suitable null hypothesis might be “there is no seasonality in EWS returns”. To estimate a p-value, we need to generate many alternate EWS price series assuming there is no seasonality, then count the number of runs in which we nevertheless see an 81% seasonality. It is worth taking a moment here to convince yourself that this procedure does indeed give you the p-value.

The hard part here is simulating new EWS price series under the null hypothesis that there is no seasonality. How can we do this without assuming a distribution of returns? How can we create alternative realities in which everything is the same about EWS, except that it has no seasonality?

Here’s one solution: for each Monte Carlo run, construct a price series from the shuffled daily returns of the EWS time series. This generates alternative paths of EWS with the same mean and volatility as the original EWS time series (since these formulae are invariant under reorderings), but by shuffling, we are ensuring that there is no seasonality in the data-generating process.

def generate_path(rets):
    # rets is the daily EWS returns
    ret_shuffled = rets.sample(frac=1)
    ret_shuffled.index = rets.index
    px_new = (1 + ret_shuffled).cumprod()
    return px_new

Note that this doesn’t mean that any given Monte Carlo run will lack seasonality – the whole point of this exercise is to see how often a path will show seasonality given that there is no actual seasonality (i.e assuming the null hypothesis is true).

I generated 10,000 paths by reshuffling returns. For each path, I stored the maximum seasonal return (i.e for each month, calculate the percentage of months which had positive returns, then take the maximum percentage).

100 price series made by shuffling EWS returns

Histogram showing the distribution of maximum seasonalities across 10,000 paths

It turns out that only 72 out of the 10,000 paths had seasonalities as high as 80% (remember that we observed 81% in real life).

We therefore see that the p-value – the probability that we observe 80% seasonality purely by random chance (no actual seasonality) – is approximately equal to 0.7%.

Using the standard 5% significance level, we reject the null hypothesis and conclude that this anomaly is statistically significant.

Advice on setting up experiments

There is a slight art in figuring out how to set up these Monte Carlo hypothesis tests – it may not be immediately obvious how to generate data assuming the null hypothesis is true.

The question you need to answer is as follows: how can I randomly generate many paths of the time series with the same statistical properties as your observed data, except removing one particular property (whichever property you’re testing).

For anything to related to autocorrelations or seasonal effects, shuffling the data is appropriate because it removes any phenomena related to ordering, while preserving the moments of the distribution.

Testing regime shifts is simple – you just resample from the “old” distribution and see if it can explain the properties of the “new” distribution. For example, if you want to test whether post-GFC volatilities are different to pre-GFC volatilities, generating data under the null hypothesis is as easy as resampling from the pre-crisis return distribution, computing volatilities, and seeing how many paths generate have volatilities as high as the observed post-GFC volatility.

Conclusion

The key takeway from this post is that we only have access to one run of history. It is immensely difficult to figure out which events/anomalies are idiosyncratic to this run of history, vs “true” anomalies that exist in the actual data-generating process.

Distinguishing between the statistical artefacts and true anomalies is critically important because the former cannot be relied upon to reoccur in the future, so any strategy designed to capture them will look fantastic in backtests but fail in live trading (unfortunately a common occurrence in algo trading).

To summarise: embrace counterfactuals. Figure out as much as you can about alternative realities if you want to have a better idea of how you’re going to fare in this one.

COVID-19 Beta

2020-11-09T00:00:00+00:00

In this short post, we compute and visualise “COVID-19 betas” for stocks in the S&P500 index, to quantitatively and visually understand which companies were most affected (positively and negatively) by COVID-19.

For those of you who just want to see the (interactive!) result, here it is. Click on any sector to zoom in on its constituents:

If you would like to generate this plot for yourself, perhaps using a different basket of stocks, the Jupyter notebook is here.

Note: most of this post was written before November 9th 2020, the day on which Pfizer announced incredibly encouraging results regarding their vaccine.

Motivation

Let’s go back in time to the start of 2020. Despite aggressive trade rhetoric, 2019 has been a great year for markets, with the S&P500 up about 30%. Thanks to low rates (and the Fed’s commitment to loose monetary policy), the “blip” of 2018Q4 is nothing more than a distant bad dream. Big tech is killing it; multiples are expanding; volatility is at a comfortable low.

On January 4th, the World Health Organisation (WHO) tweets that there is a cluster of pneumonia cases in Wuhan, China. As far as the West is concerned, this is a non-event, happening way “over there” in the East. On January 13th, a case is recorded in Thailand. What follows is two months of health officials gathering and mulling over the evidence (on January 23rd an independent committee reports that there is insufficient evidence to make a decision), while the rest of the world continues on blissfully unaware of the worsening situation. The markets continue their steady rise, largely unperturbed.

In February, the situation becomes hard to ignore; by February 13th, COVID-19 is present in 25 countries, with more than 60,000 cases (source). On the 14th of February, market participants suddenly seem to appreciate the gravity of the situation; in one short month, the S&P500 loses 1/3 of its value. People who have been diligently following the prevailing personal finance advice and investing their money in the stock markets are suddenly faced with an unprecedentedly rapid loss of net worth. Speculators, who have been riding the rally with leverage, are caught with their trousers down.

By March, there is full-on panic as the worldwide number of cases hits 100,000 and nations start to impose heavy travel restrictions. What had started as whispers of a viral cough “far away in the East” has now blown up into the terrifying spectre of an omnipresent contagion – $R_0$ numbers suggest that before long, the majority of the world population will be infected by a virus whose severity is still unknown. Readers of mainstream news could be forgiven for thinking that it is the end of the world as we know it.

Cut to 18th August – 6 months later. The S&P500 has just posted a new all-time high, having appreciated about 50% from the low of March 20th.

I have written this dramatised account largely as a note-to-self. With the rose-tinted spectacles of hindsight, it is now clear that the widespread and indiscriminate risk-off in March created a fantastic opportunity for level-headed investors to pick up high-quality companies, well suited to a social-distanced society, at steep discounts. Many of us are now kicking ourselves for not having bought more, but we must remember that in the moment, the future was murky indeed.

All this said, things certainly aren’t back to where they were. Only a handful of companies (mostly big tech) have been responsible for the majority of the market’s rebound. Whether you are optimistic or pessimistic about how the pandemic plays out from here, as long as you think COVID-19 is a key driver for stock markets it is important to know how different sectors or companies are affected by the virus. Optimists might be keen to identify which stocks have been hardest hit, to play the rebound, while those who think that the situation will stay unresolved for longer than consensus expects may want to remain overweight the companies that thrive in a pandemic environment.

The goal of this investigation, therefore, is to develop a highly intuitive tool to allow the viewer to understand, at a glance, which companies respond well and badly to COVID-19 news.

Methodology

The standard approach is “armchair reasoning”: sitting down and logically thinking through what the impacts of COVID-19 have been / will be. On April 2nd 2020, I made this brainstorming mindmap to reason about some of the second-order implications of the pandemic. Several of the companies I mentioned, which may seem obvious in hindsight, have done very well:

Alternatively, we might adopt the “data-driven” approach: let market performance tell you which stocks benefitted most and least from COVID-19. Naively, we could simply look at which stocks/sectors have performed best/worst this year. This is a reasonable starting point but contains a lot of noise because stocks move up and down for all sorts of reasons.

A better methodology for understanding how some factor (in our case, COVID-19) affects stock prices is is to compute the beta of the asset returns to the factor. Concretely, we examine the correlation between the daily change in stock prices and the daily increases in COVID-19 cases: if a stock tends to have negative returns whenever COVID cases rise, we may believe that there is some association between COVID and the stock. The nice thing about this approach is that we can remove the effects of as many other variables as we want by introducing them as additional regression variables.

In this post, we will regress stock returns against both COVID-19 cases and the overall S&P500 index (the latter being what people typically call the beta, though it is really just a beta). This serves to identify the effect of COVID-19 on the stock in excess of the overall market effects, which should give a more accurate picture of how COVID-19 is affecting a stock.

I pulled COVID data from the New York Times’ GitHub repo, and as usual, I used Yahoo Finance (via the yfinance python library) for stock/index pricing data. The “heavy lifting” – regressing stock returns against SPY returns and the change in daily cases – was done by the linear regression class within scikit-learn.

Analysis and Visualisation

A higher COVID beta means that a stock’s returns were positively correlated with the change in COVID cases, i.e. more COVID cases helped the stock. Perhaps unsurprisingly, among the stocks with the highest COVID betas were Netflix, Walmart, Hasbro, Intel, and Citrix (enterprise technology). Conversely, the stocks with the most negative COVID betas include hard-hit companies in the energy and consumer discretionary sectors. This is encouraging, as our methodology passes the basic sanity check. Tech companies like Netflix were clear winners from national lockdowns, while energy companies faced a massive demand shock as people no longer drove to work or travelled overseas.

To summarise the betas in one clear diagram, I drew inspiration from the Bloomberg terminal heatmap, which gives an instant “pulse” of the market with respect to one key variable (with the size of a rectangle representing market caps):

(Note: we don’t see a large blue rectangle corresponding to Zoom because Zoom isn’t yet in the S&P500 index)

At a first glance, this graphic seems to accurately describe much of our intuition regarding what COVID has benefitted/harmed. Tech is largely blue (benefitting), as are healthcare and consumer staples. Energy has been particularly hard hit, with financials having a tough time also. But the effect of COVID on other sectors may not be as obvious, and it is here where the visualisation becomes especially helpful.

Note regarding the Pfizer news of November 9th 2020

On 9 November, while I was halfway through writing this post, Pfizer announced very promising results from their vaccine. I found the heatmap to be a very useful reference – the red companies/sectors posted stunning returns, while many of the blue companies (e.g. big tech) had a lacklustre day – and actually ended up using the heatmap to help identify some stocks to trade. Based on my qualitative judgment post-hoc, the betas on the heatmap do seem to properly reflect the economic link between COVID-19 and the companies.

Conclusion

In this post, we have constructed a “quick and dirty” method of understanding and visualising the effects of COVID-19 on different companies. There are several important caveats which should be considered before you use the heatmap for anything important.

Firstly, betas are essentially correlations, so we must be careful about using them to retroactively create narratives to explain why certain stocks did well/badly with respect to COVID-19. One must exercise judgment in determining which betas represent broad economic impacts due to COVID-19, rather than idiosyncratic company effects. For example, I noticed that within the financials sector, Goldman Sachs stood out as having a positive COVID beta – I reasoned that this was because GS doesn’t have significant commercial banking exposure, instead being focused on trading (which benefits from volatility) and investment banking. However, this narrative is somewhat contradicted by the fact that Morgan Stanley also lacks commercial banking exposure, yet had a negative COVID beta anyway.

Secondly, this analysis treats beta as if it is a static parameter. In reality, to compute beta (or any other time-series property), one must decide on a rolling window for the calculation. In this post, we used all year-to-date stock price data, but the plot below shows how the 2-month rolling beta for the 10 highest/lowest beta stocks (averaged) varied over time.

Lastly, we have all likely heard the many complaints about the market diverging from reality, along with the standard response that “markets are forward-looking”. Our methodology is not at all forward-looking, as it is simply regressing returns against the daily change in cases. A more meaningful calculation would be to calculate the beta with respect to some estimate of future COVID cases – possibly the beta to $R_0$.

These issues aside, I have personally found the heatmap to be a useful visualisation tool; at the very least, the plot is somewhat pleasant to look at and play around with.

Option-implied probability distributions, part 2

2020-10-10T00:00:00+00:00

In Part 1 of this series, we demonstrated that the prices of option butterfly spreads imply a probability distribution of prices for the underlying asset. In this post, we will first examine the limiting case of butterfly spreads. Then, we will tackle the industry-standard approach for constructing PDFs from option prices: interpolating in volatility space to generate a volatility surface, converting this into a continuous set of option prices, then applying the Breeden-Litzenberger formula to find the PDF.

This post will be slightly more technical than Part 1, but as before, I endeavour to explain the motivation behind the maths.

The supporting code for this post can be found in this Jupyter notebook.

The limiting case of narrow butterflies

As mentioned in the conclusion to Part 1, deriving the PDF from butterfly prices is not a perfectly accurate method, even assuming we had perfect pricing data with negligible bid-ask spreads. This is because we made the approximation that the payoff of a butterfly is binary: either the maximum payoff $\Delta K$ if the expiration stock price is on the central strike, or zero otherwise. This payoff, of course, is not binary as it contains a sloped portion.

However, the approximation gets better and better as we use narrower butterflies because the sloped portion becomes less significant. For example, to calculate the probability that the expiry price of the stock is \$340, it would be better to use the 339/340/341 butterfly (as we did in the previous post) than to use the 335/340/345 butterfly, even though both of them are valid butterflies centred on \$340. Unfortunately, in the real world, we are not able to construct arbitrarily narrow butterflies because there is a limit to the granularity of strike prices on exchanges (with a \$1 granularity being typical for retail brokerages).

Let’s pretend we could construct arbitrarily narrow spreads. We will now show that in this limit $\Delta K \to 0$, the butterfly-implied probability converges to the PDF. The argument is rather mathematical and can be skipped, but do take a moment to appreciate the result. I find this little fact to be extremely exciting because it’s a clear case in which an initial intuitive approach (inferring the probability from the prices of butterfly spreads) actually converges to the right answer.

Recall that for a butterfly with price X and maximum payoff $\Delta K$, the probability $p_K$ that the underlying is at the central strike K at expiry (in which case the maximum payoff $\Delta K$ is received) can be found by an expected value calculation:

\[p \Delta K = X \implies p = \frac{X}{\Delta K}\]

Probability density is not the same as probability; the general intuition is that probability is given by the area under a PDF – in fact, we can define the PDF as the function $f(x)$ such that:

\[P(A \leq x \leq B) = \int_A^B f(x)dx.\]

With this in mind, the probability that the underlying price is within a small band $\Delta K$ of a given strike $K_0$ at expiry is:

\[P(K_0 \leq K \leq K_0 + \Delta K) \approx f(K_0)\Delta k.\]

The approximation becomes an equality in the limit of small $\Delta K$. Hence, denoting $P(K_0 \leq K \leq K_0 + \Delta K)$ as $p_K$, we have:

\[p_K = \lim_{\Delta K \to 0} \left\{f(K)\Delta K \right\} \implies f(K) = \lim_{\Delta K \to 0} \frac{p_K}{\Delta K}.\]

But we have an expression for $p_K$ in terms of the price, and can express the price of the butterfly in terms of the prices of the underlying call options (two short calls with strike $K$; long calls at strikes $K \pm \Delta K$):

\[\begin{aligned} f(K) &= \lim_{\Delta K \to 0} \left\{\frac{X / \Delta K}{\Delta K} \right\} = \lim_{\Delta K \to 0} \left\{ \frac{X}{(\Delta K) ^2} \right\} \\ &= \lim_{\Delta K \to 0} \left\{ \frac{C(K + \Delta K) - C(K) - C(K) + C(K- \Delta K))}{(\Delta K) ^2} \right\} \end{aligned}\]

The astute reader will note that the RHS of this expression has a very familiar structure – it is the second (partial) derivative of the call price with respect to the strike price. The PDF can thus be expressed as:

\[f(K) = \frac{\partial^2 C(K)}{\partial K^2}\]

This is an incredibly powerful result that relates the PDF directly to the prices of call options; furthermore, it shows that the limit of narrow butterfly spreads does indeed give the correct answer.

An alternative derivation

For those of you who are not intellectually satisfied by the intuitive derivation above, there is a slightly more rigorous argument (equally elegant, in my view).

By definition of the PDF, $P(S_T > K) = \int_K^\infty f(x)dx$ (here we are using x as a dummy variable for the strikes). As discussed in Part 1, the fair value of a call option (with time $\tau$ until expiry) is the present value of the expected payoff:

\[\begin{aligned} C(K, \tau) &= e^{-r\tau} \mathbb{E}(\max[S_T-K, 0]) \\ &= e^{-r\tau} \int_0^\infty \max[S_T-K, 0] f(S_T) dS_T \\ &= e^{-r\tau} \int_K^\infty (x-K)f(x)dx \end{aligned}\]

The goal is to invert this to find $f(x)$ as a function of C. This can be done by expanding the integral and taking partial derivatives with respect to strike.

\[\begin{aligned} C(K, \tau) &= e^{-r\tau} \left[ \int_K^\infty xf(x)dx - K\int_K^\infty f(x)dx\right] \\ \implies \frac{\partial C}{\partial K} &= -e^{-r \tau} \int_K^\infty f(x)dx \\ &= e^{-r \tau} \left[ \int_{-\infty}^K f(x)dx -1 \right] \end{aligned}\]

We now observe that $\int_{-\infty}^K f(x)dx$ is the definition of the cumulative distribution function $F(x)$, the derivative of which gives the PDF. Hence:

\[\frac{\partial^2 C}{\partial K^2} = e^{-r\tau} \frac{d}{dk} F(K) \implies f(K) = e^{r\tau} \frac{\partial^2 C}{\partial K^2}\]

We have now derived the same result as before, generalised to include the time to expiry. The proper name for this result is the Breeden-Litzenberger formula (derived in their 1978 paper), a very useful tool which can convert call option prices into an implied PDF.

A detour into volatility space

Unfortunately, we cannot immediately apply the Breeden-Litzenberger formula to call prices for the same reason that butterfly spreads can’t be made arbitrarily narrow: strike prices come in discrete increments. To numerically calculate the second derivative of the call price function with respect to strike, we need our call prices to be near-continuous with respect to the strike, requiring interpolation (possibly some signal smoothing first).

At this juncture, two roads diverge in a yellow wood. The obvious thing to do is to interpolate in price-space, i.e join the dots in the below plot:

This seems straightforward, but there is a catch. It is dangerously easy for interpolation to result in arbitrage opportunities (which are not realistic). For example, any straight-line interpolation, such as setting the price of the \$105-strike option to be exactly between the prices of the \$100- and \$110-strike options, leads to a free butterfly spread. In fact, with the Breeden-Litzenberger formula in mind, any interpolation must always be convex (positive second derivative with respect to strike) since the PDF must be positive. If you are a market maker, mispricing the options to allow for arbitrage opportunities would be a cardinal sin, because you would make a guaranteed loss while your counterparty makes a risk-free profit.

It is essential, therefore, to be cautious when interpolating. One reason for the instability of interpolation in “price space” is that the price can be thought of as the dependent variable (i.e output) of an option pricing model which has several inputs:

The price of the underlying and the strike (further OTM is less valuable)
Time to expiration (more time is more valuable)
Volatility (more volatile is more valuable)
Interest rates (higher rates means calls are more valuable)
Dividends (higher dividends means calls are less valuable)

(The last two are much less important than the first three). The argument goes that it makes more sense to interpolate on the inputs to this function versus the output (price) because inputs should be “smoother” than outputs. The moneyness of the option and the time to expiration are pre-determined when entering a contract, so the only meaningful free input parameter which we might consider interpolating is the volatility.

Black-Scholes and implied volatility

I can’t avoid bringing up Black-Scholes, but this is certainly not the place to give a detailed introduction to the theory.

The Black-Scholes model values an option – it does not price an option any more than a discounted cash flow analysis prices a stock – by making several assumptions: the volatility of the underlying is constant, the returns of the underlying are normally distributed, efficient markets, etc. As anyone who has witnessed Nassim Taleb’s Twitter vitriol will know, these assumptions are often far from the truth.

Instead of proceeding with this definition, for the purposes of this post, let’s forget everything we know about Black-Scholes and redefine it as follows:

There is a function called Black-Scholes which converts implied volatilities (IVs) – to be defined – into option prices.
There is a function called Black-Scholes Inverse which converts option prices into IVs.

That’s it. Nothing about pricing or valuing options. No assumptions about the distribution of asset returns. Just a mathematical identity.

The reason we can get away with this bold claim is that there is an embedded circularity. “Implied volatility” is defined to be the volatility which results in the current option price when input to standard Black-Scholes. It might seem like we have just done some Lewis Carroll-esque wordplay, but for reasons we shall soon see, this circularity can be much more workable than the alternative of trying to define IV along the lines of the “expected future volatility”.

In summary, price and IV can be interchanged by applying the appropriate function. This allows us to make the key insight: rather than interpolating in price space, we can convert the prices to IVs using Black-Scholes Inverse, interpolate in IV space, then convert the interpolated IVs back to prices using Black-Scholes.

Interpolating in IV space

Using the same SPY dataset as in Part 1 (collected when $S = \$332$ with about 3 weeks to expiry), the first step is to use Black-Scholes Inverse to bring us from price space into IV space.

There is no closed-form inverse to the Black-Scholes formula, so Black-Scholes Inverse is a numerical method – I chose to use Newton’s method, because it is easy to code up, but be aware that even the topic of computing IVs has been deeply researched. Excluding some pathological deep ITM/OTM contracts, the result of Black-Scholes Inverse is shown below:

There are some problems around the \$365-strike contracts, but after applying a Gaussian filter and clipping out additional deep ITM/OTM contracts, we have something workable:

We can then apply the same cubic spline that we used with the butterfly probabilities to create a smooth function – a cross-section of the volatility surface.

With a moment’s reflection, this plot is rather unintuitive. Rather than having a constant IV, we observe the characteristic volatility smile. This does not make much sense if we think of IV as the “expected wiggliness” of returns in the underling between now and expiry – the future volatility should be independent of the strike – and is in clear contradiction to the Black-Scholes assumption that volatility is constant.

Instead, using the “circular” definition that IV is the input volatility to Black-Scholes that results in the observed call price, we can rationalise the volatility smile as follows: buyers are willing to pay a premium (relative to the Black-Scholes “fair value”) for out-of-the-money options. A classic explanation for this is that asset returns typically have fat tails, meaning that OTM options are more valuable than Black-Scholes would suggest.

Constructing the PDF of underling prices

Having interpolated in volatility space, we can convert back to price space using Black-Scholes, then take the second partial derivative of the call function with respect to strike (numerically) to find the PDF, as per the Breeden-Litzenberger formula:

Comparing PDFs

We can now compare the butterfly PDF that we generated in the previous post with the Breeden-Litzenberger PDF. Firstly, as seen above, the Breeden-Litzenberger method gives us a larger range of strikes (\$300-\$370 vs \$320-\$360 for butterfly PDFs), which makes sense because constructing each butterfly requires three price quotes.

Secondly, even after restricting the range of strikes to be the same as for the butterfly PDF, note that the Breeden-Litzenberger PDF is better-behaved, lacking the kinks at $355 < K < 360$ (though as the previous plot shows, it does start to wobble as we move away from at-the-money):

The last thing we ought to do is compare the two roads that diverged in the yellow wood: interpolation in IV space vs interpolation in price space – recall that these are two methods of generating a continuous call function, after which Breeden-Litzenberger can be applied.

Using the same Gaussian filter and cubic spline, we can smooth and interpolate the call prices directly. The resulting PDFs is compared with the IV-space interpolation PDF below:

The IV-space approach seems to give much better results in the left tail, but slightly worse results in the right tail. Overall, there does not appear to be strong evidence that interpolating in IV-space is superior to interpolating in price-space, at least for this particular example, but I do think that interpolating in IV-space is logically more defensible.

Conclusion

In conclusion, this pair of posts has given an introductory overview of how we might construct price PDFs by looking at option prices.

There is much more that could be said about this subject. For example, we have completely ignored half of the available data – the prices of put options. A common technique is to use put-call parity to generate more data points for fair prices of call options, improving our estimate of the volatility surface. Furthermore, this post has taken a non-parametric approach, i.e we have not assumed that the volatility smile has any shape. In practice, parametric models are popular because they lead to smooth and analytically tractable surfaces.

I realise that I have not spent a great deal of time explaining why we might want the option-implied PDF. The obvious answer is that if you know the option-implied PDF, you can understand the market’s probabilistic expectations of the future and make trades when your view differs. What is not obvious is how one might generate probabilistic views in the first place. This is a topic that I will be revisiting in a subsequent post – this series was written solely to lay the groundwork for an upcoming piece, in which I am hoping to approach fundamental valuation from a probabilistic perspective.

Option-implied probability distributions, part 1

2020-10-01T00:00:00+00:00

The key idea in Expectations Investing, a well-known book by Alfred Rappaport and Michael Mauboussin, is that to profitably invest in stocks one needs to find situations in which your view is variant to consensus expectations – a difficult task when you don’t know what consensus expectations are. There are several ways to figure out what the Street is thinking, the most common of which is to read sell-side research reports and get on the phone with analysts. However, earlier this year I wrote a post investigating how the current price of an asset can be used to infer implied expectations of future performance. In Part 1 of a two-part series, we look to extend this concept by examining what the pricing of option butterfly spreads can tell us about the implied expectations of the probability distribution of returns. Part 2 will consider a more advanced approach, closer to what is used by industry practitioners.

In writing this post, I have had to make a difficult decision regarding the level of background knowledge to assume; there is no escaping the fact that the subject of derivatives is highly mathematical. Therefore, in each section, I aim to present the intuition first and the mathematics second so the reader may decide how much of the mathematics they follow.

The supporting code for this post can be found in this Jupyter notebook.

A European call option gives you the right, but not the obligation, to buy an underlying asset (whose price at time t is denoted by $S_t$) at a particular date in the future (the expiry date, T), at a particular price (the strike price, K). If stock XYZ is currently trading at \$100 a share, one might think that an option to purchase it at \$120 next month is worthless. If the option were expiring today, that would be true – you wouldn’t want to buy something for \$120 when the current price is \$100. The option expiring next month, however, is not worthless because it’s possible that in the next month, the stock price could rise to \$130, in which case at expiry we could exercise the option and buy the asset at \$120 before selling it at \$130 for a neat \$10 profit.

We have established that this option is not worthless, which means that it has some value. Intuitively, its value is equal to the (present value of the) expected payoff at expiry. The payoff at expiry is the difference between the stock price and the strike price (\$130 – \$120 in the example above), or zero if the stock price is less than the strike price since you would not exercise the option. Hence, to value a call option C, we might write:

\[C(K, \tau) = e^{-r\tau} \mathbb{E}(\max[S_T-K, 0]),\]

where $\mathbb{E}$ is the expectation operator, $\tau \equiv T - t$ is the time period between now ($t$) and expiry ($T$) and r is the risk-free rate. $S_T$, the price at expiration, is a random variable – if we knew the future stock price with certainty life would be easy. The goal of this post is to find the probability density function (PDF) that describes $S_T$, which we will denote as $f(S_T)$; loosely speaking, this would tell us the probability that the stock trades near a certain price on the expiration date. With this in mind, to construct an expression for the expected payoff, we essentially need to sum the “probability that the stock price is $S_T$ at expiry” multiplied by the “payoff if the stock price is $S_T$ at expiry” for all possible $S_T$. This is done by integration since prices are assumed to be continuous:

\[C(K, \tau) = e^{-r\tau} \int_0^\infty \max[S_T-K, 0] f(S_T) dS_T.\]

We shall revisit this expression later on. For now, the key point is that there is a mathematical relationship between call option prices and the PDF of future returns.

A first approximation: option butterflies

As we established previously, the probability that the underlying stock will close at a particular price next month is somehow baked into the prices of options expiring next month. Intuitively, the price of calls should be positively related to the probability of an upside move – people would only want to buy an out-of-the-money (OTM) call, e.g. a call with strike \$120 when the underlying is trading at \$100, if there is some chance that their option will be in-the-money at expiration.

However, we cannot simply say that the probability of a certain price move is proportional to the price of call options with that strike because the price of a call embeds the expected profit from all expiry prices above the strike. In other words, the fair price of the K-strike option is related to the entire portion of the probability distribution above K, as the integral above demonstrates.

To proceed, we need to consider some combination of options that only profits if the underlying trades in a small range around a given strike at expiry. The price of such a structure would indeed be proportional to the probability of the associated price move. One such structure is the long butterfly, which consists of two short calls at a given strike combined with a long call at a higher strike and a long call at a lower strike, “bracketing” the strike of the short calls. The payoff diagram explains this better than I can:

(Note that these payoff diagrams are shown on a gross basis – they have not been shifted down to account for the price of the structure).

The maximum payoff of a butterfly spread is equal to half the width of the spread (highest strike minus lowest strike). We shall denote the maximum payoff as $\Delta K$ – this payoff is realised when the stock price is equal to the central strike price at expiry.

Now, an important question. What is the fair price X of this butterfly spread?

Consider a biased coin which pays $\Delta K$ if the coin shows heads, which happens with probability p, and zero if the coin shows tails, which happens with probability $1-p$. The expected value of the coin toss is then $p\Delta K$. A rational individual who is neither risk-averse nor risk-seeking would pay exactly $p \Delta K$ to throw this coin.

We can apply the same reasoning to the butterfly, making the approximation that the payoff of the structure is either the maximum payoff $\Delta K$ or zero (i.e ignoring the sloped portion of the payoff function). In this case, requiring that the price of the structure is equal to the expected value of its payoffs, there is a simple relation between the maximum payoff of the structure $\Delta K$, the probability p that this maximum is received (equivalent to the probability that the price ends up at the central strike), and the price X of the structure.

\[p \Delta K = X \implies p = \frac{X}{\Delta K}\]

Let’s apply this to a real-life example. I downloaded some SPY option price data from my broker (about 3 weeks to expiration):

The probability that the SPY closes at \$340 on expiry (at the time of the snapshot, it was trading at \$332) will be approximated by considering the price of the 339/340/341 butterfly. For this structure, we have $\Delta K = 1$ and

\[X = 1.670 - 2 \times 1.365 + 1.100 = 0.04,\]

meaning that it would cost 4¢ to open this position (assuming we could fill at mid-market). Based on the formula derived above, the probability that the SPY closes at \$340 on expiry is $p = 0.04/1 = 4\%$.

In principle, we can now repeat this process for every possible butterfly to build out the entire distribution.

A kaleidoscope of butterflies

Unfortunately, there are several added complexities when working with real data. Let’s repeat the above calculation to find the probability that the SPY closes at \$349. The price of the 348/349/350 butterfly is

\[X = 0.205 - 2 \times 0.165 + 0.125 = 0.\]

Something is wrong here because we are receiving potential profits (with zero downside) for free! In an arbitrage-free market, this should not exist. Furthermore, the fact that the price of the butterfly is zero implies that there is no chance that the stock will trade at \$349 on expiry, which does not seem plausible, particularly because the same calculation tells us there is a 2% chance that the expiry price of the stock will be \$350.

This problem arises because of the illiquidity of some of the options contracts. There are far more people trading the \$350-strike option (because it’s a nice round number) than there are for the \$349-strike. Hence, for the latter, we cannot be certain that the price we see on the screen is reflective of the true price at which an order would be executed.

The plot below shows the butterfly-implied probabilities vary with the central strike price. Notice how noisy the data seems to be; there are even negative probabilities!

We need to clean this data up somehow. One approach would be to filter out the options with low volume, but that might result in us losing a lot of valuable data. Instead, I have decided to smooth the signal using a Gaussian filter. This works by convolving the noisy signal (in this case, butterfly-implied probabilities) with a Gaussian kernel – intuitively, each noisy data point is replaced with a weighted average of surrounding points (with weights determined by the Gaussian), resulting in a “blurring”. The result is clearly a major improvement:

We are not quite done yet, because all we have is a set of discrete points. We want a continuous PDF, so we need to interpolate between the points. To do so, we shall use a cubic spline, which fits piecewise cubic functions to the discrete points. This results in a relatively smooth PDF (indeed, by changing a parameter, we can decide exactly how smooth we want it to be):

We have now managed to meet the objective of this post. We are now able to answer questions like “what is the probability that the stock trades above \$350 at expiry” by integrating the PDF.

The implied distribution of SPY prices has a characteristic log-normal shape, consistent with a bell-curve return distribution. This does not have to be the case. There are certain situations in which the return distribution can be highly asymmetric. For example, on September 13th Gilead announced that they would acquire Immunomedics for \$88 per share in cash, resulting in IMMU stock jumping up to near \$88. Once this happens, the probability of a move to the upside is low (though it could happen if a bidding war begins), while a downside move is definitely on the table – acquisitions may fall through for a number of reasons, be it antitrust, shareholder disapproval, or force majeure.

Conclusion

To sum up, the pricing of butterfly spreads can be a useful intuitive method for deriving the implied PDF of returns. That being said, the resulting distribution is not completely accurate, because it relies on the approximation that butterflies payoff in a binary manner. Furthermore, we have seen that dealing with real-life data requires several post-processing techniques to make the PDF “sensible”. In Part 2, we will examine a more sophisticated industry-standard approach to finding the option-implied PDF. Stay tuned!

Equity Investing Checklist

2020-08-20T00:00:00+00:00

This page represents a work-in-progress checklist for equity investing. It is continuously being refined based on my reading, as well as my actual experience in researching and investing. The hope is that this provides transparency and accountability (for myself), to better understand my cognitive biases and weak spots.

An important idea underlying this checklist is that time is finite, and there are thousands of public companies that we could analyse. Hence it’s ok to be trigger-happy with your vetos, and to stop researching a company if there’s a minor red flag. In the checklist below, I write “veto” whenever the point is a sufficient reason to stop researching the company.

Investment universe
Preliminary research
In-depth research
Modelling and valuation
Third-party research
Pull the trigger

Investment universe

The first step is to narrow down the investment universe. This is a very vague process, and can be done in many different ways. The quality of these companies really doesn’t matter at this point; we are just trying to reduce our universe from ~5000 companies to ~100 companies. Some methods:

My machine learning screen
Greenblatt/Lynch screeners
CapIQ screens
Random stock ideas
Sell-side aggregated lists
13F filings or insider buying

Preliminary research

In this section, we aim to understand whether a company is worth doing more research on. Throughout the process, we take brief notes of any questions we have about the company or any preliminary pros/cons, which will be revisited later if we continue researching the company.

Understanding the business

How does the company make money?
- can just use Yahoo Finance, Atom Finance, or CapIQ.
- company wikipedia page or website.
What business is the company in?
- Geography?
- Do I understand the basics of the business?
- What is the top-level trend in the business?
Where does the company fit into my macro outlook? (veto if it clearly doesn’t)

Quick look at financials

What is the trend in the stock price?
Quick look at financial ratios, and how it compares with competitors
- use Atom Finance or CapIQ for comps
- note anything surprising (e.g. very low/high multiples)
Quick look at the trends in revenue, margins, cash flows.
Quick look at the balance sheet. (veto if much too complicated for me to understand)

Meta issues

Do I find the company/sector vaguely interesting? (veto if I really have no interest in it, e.g. Oil and Gas, there is no point moving further)
What edge do I have trading this company?
- is it small enough that major analysts don’t cover it? (a la Lynch)
- do I have special sector knowledge?
- not necessarily a veto factor because at this stage in my career there aren’t a huge number of companies I have an edge on.

In-depth research

Once I have decided that it is worth investing the time to deeply research a company, it’s time to pull up the recent financial statements to try and properly understand the company.

Resources

Read and make notes on the following sections from the 10K, with an emphasis on red/green flags and points that need to be revisited:
- Introduction and company overview (we should have a rough idea of what will be in here)
- Skim through the risk factors to see if there are any that aren’t boilerplate
- Management Discussion & Analysis
- Look over financial statements
- Read the footnotes (veto if company has many accounting complexities, e.g. cross-holdings, weird leases, unusual capital structures; it is likely not worth the time given my current ability)
Read through some recent earnings calls transcripts, in particular the Q&A section, to try and understand what the market is paying attention to.
Read through the proxy statement to understand the compensation model.
Read the latest news on the company and third party research
- Very important to judge the third-party research on the thesis and not the name
- Use the third-party research to inform data points rather than the overall story

Key questions

Some of the key questions to answer are as follows (based on Aswath Damodaran’s framework):

What are the key levers for the sector? Do I have any deviant views on these?
- How are supply and demand changing?
- What are the tactical (short-term) and strategic (long-term) considerations?
What are the key levers for this company? Do I have any deviant views on these?
- What non-GAAP metrics are people paying attention to?
- What questions are people asking in the Q&A section of the earnings calls?
How will the company’s investments drive sales growth or operating margin improvement? What is the cost of this growth in terms of required reinvestment?
What are the major risks to the company that may inhibit its sales growth or operating margin improvement?
What is management’s strategy, and how successful have they been at executing on their goal? How has the company evolved over time?
Are management incentives aligned? What is their compensation model? Have they been buying stock?
What non-operating assets does the company have?
Will the company remain a going concern? (veto if I am not confident that the company will remain solvent)

Regarding strategy and competition:

Revisit comps in more detail to understand competitive landscape
SWOT, Porter’s five forces.

Story

Having answered the questions above, it is time to summarise all of the information thus far into a story, that links the facts together. It is advisable to amass all the facts then try to come up with a consistent story, rather than building a story on the go (prone to confirmation bias).

veto if I can’t come up with a simple story that explains the facts, or if the story is not attractive.

Modelling and valuation

Valuation is quite a deep area, so the depth you go depends on how much time you are willing to dedicate to it. The bare minimum is do a relative valuation.

Relative valuation

Using CapIQ, Bloomberg, or a similar tool, construct a list of comparable companies. There is an important tradeoff between having good quality (i.e very similar) comps and having enough of a sample for the mean/median to be meaningful.
- important to keep an eye on variables that are not captured by the valuation multiples, for example, leverage and cash flow
- if you notice any outliers among the compset, investigate why, but consider removing them.
We can then compute the average value for each metric (e.g. EV/EBITDA, P/E) across multiple values.
If our company has multiples much lower than the median, it is important to critically think about why this is the case. Some common reasons for a low P/E are: lower growth than comps, more levered, different risk exposures (geography).
If there is a very good reason that the company is trading at a big discount to comps, and hence you don’t feel the comps are a fair sample, then you can just use multiples from the company’s own history along with your own forecasts of the denominators to estimate the value.

DCF valuation

We should already have most of the inputs ready to go, based on the prior section. The key job here is to tie the numbers to the story and to come up with some idea of the intrinsic value of the company.

Use one of Damodaran’s many spreadsheets. In most cases, we should use his model selector and the relevant Ginzu spreadsheet.
Conduct a sensitivity analysis

If we are having trouble with the inputs, we may need to do some work regarding market-implied variables, or inferring DCF inputs from comparable companies. Useful questions to consider:

How do my assumptions differ from consensus?
Is the company trading at an unjust discount to peers?
What is the market-implied forecasting period (cf Expectations Investing)? Do I think this period of cash flow growth is sustainable?

Margin of safety and catalysts

Using either of the above methods (or a combination of the two), we should have a range of valuations. A useful valuation tool is the football field chart, which plots the valuation range from a number of sources, and compares it to the current share price.
It is a positive sign if there is a high degree of agreement between the different signals – if not, we should analyse why the market is treating one metric very differently.
It is a very positive sign if all of the metrics are significantly above the current share price – this is a margin of safety, and serves to give us some leeway to account for model risk. Nevertheless, it is important to be realistic and really think about what we know that the market doesn’t.
Lastly, it is critical to think about catalysts. If we are arguing that a stock is completely undervalued, what will make the market re-value it?
- the most common catalyst is probably an earnings report, since many people pay attention to these, but for small-cap stocks even these might not be enough to cause a significant re-valuation.
- when dealing with shorts, regardless of how good your valuation is, a catalyst is a necessity. Many legendary investors (Tiger Cubs) got killed in the dot-com bubble because they were shorting incredibly overvalued companies that just went on to become even more overvalued. “The market can stay irrational longer than you can stay solvent”.

Pull the trigger

At this point, if we decide that a company is worth investing in, we prepare a dated memo according to the following template:

Brief introduction to the company (industry, current share price, current multiples)
Investment thesis and key levers.
How my view of the levers differs from consensus
Risk factors (identify risks using a pre-mortem analysis)
How the company fits into my portfolio (in terms of diversification / sector risk)
Price target and stop loss.

This is really for accountability and self-improvement, so that 6 months from now I can understand whether I was:

Right for the right reasons (yay!)
Right for the wrong reasons, i.e lucky.
Wrong but on the right lines, e.g. identified the key lever but called it the wrong way.
Completely wrong, or black-swanned.

After the trade

For posterity, it is good to note down the date and price of your fill (although this is tracked by your broker).
It is inevitable that we will continue to learn new things about the stock once we own it – many investors say that this is the best time to learn about your company because you now have skin in the game. It is very possible that you may come across new information that invalidates your thesis, or affects your entry/exit conditions. This is fine, as long as you note it down very clearly. Hence 6 months from now, you will be able to retrospectively diagnose whether you were constructing a narrative to justify your beliefs, or the new information truly did affect your thesis.
When we eventually close the position, again, we should write down the date and the fill. This is important so that in future we can analyse what happened to all the stocks that we sold – did we sell too early?

A critical look at Greenblatt's Magic Formula

2020-06-08T00:00:00+00:00

As the saying goes, when something sounds too good to be true, it probably is – all the more so when it comes to investing. In this short post, we look at the Magic Formula of Joel Greenblatt, as described in The Little Book That Still Beats the Market, critically examining the strategy and attempting to quantify its alpha.

What is unusual about this Magic Formula is that the person behind it is highly respectable. Joel Greenblatt is a value investing expert with an impressive performance track record; Gotham Capital’s persistent outperformance of the S&P500 speaks for itself. His book You Can Be a Stock Market Genius, despite the awful title, reignited my interest in discretionary stock picking and the chapter on spinoffs has led to some profitable picks. Consequently, this post is not seeking to debunk the Magic Formula, but rather qualify the claims made both by Greenblatt and the many investors who have since adopted it.

The supporting code is here, but it may not be much use on its own as it is intended to be run within the Quantopian research/algorithm environment.

How the Magic Formula works

The “Magic Formula” is a simple procedure to find good companies to invest in. It encompasses the philosophy that the ideal company needs to be high quality, and it needs to be cheap. That’s all there is to it.

There are many different ways these factors can be quantified, leading to several variations of the Magic Formula. Nevertheless, we will look at the canonical approach discussed in The Little Book that Still Beats the Markets. For the quality factor, Greenblatt uses the Return on Capital (ROC), which he calculates as the operating earnings (EBIT) divided by the total value of the fixed assets (property, plant, and equipment) and net working capital. By defining capital this way, instead of the more conventional sum of debt and equity, Greenblatt isolates the assets that are used to generate operating income. Hence, the ROC quantifies how efficient the company is at using its capital to generating earnings.

Cheapness is more straightforward. Greenblatt uses the ratio of operating earnings to the enterprise value (EBIT/EV) but you could instead use the earnings yield (inverse of the P/E ratio) or free cash flow yield. We should formulate the cheapness metric formulated as a yield, such that a higher quantity is better – this is useful when we are combining the cheapness rank with the ROC rank since both rankings will be in the same direction.

With this in mind, the overall procedure as suggested in The Little Book is as follows:

Construct the universe of stocks: we require market capitalisations of more than \$100m and remove stocks in the Financials or Utilities sectors (since they have slightly different accounting).
Calculate the ROC and the EBIT/EV
Rank each company in the stock universe by ROC
Rank each company in the stock universe by EBIT/EV
Add the two ranks to get a combined score
Each month, buy 2-3 positions from the top 20-30 companies
Hold each position for one year (timing the sales/purchases for tax benefits)

The Little Book has a table showing the annual performance of the Magic Formula approach: from 1988 to 2004, Greenblatt claims that the Magic Formula returned an annualised 33% (compared to 14% for the S&P500). This is a phenomenal return – compare it with the scatterplot below showing the performance and track record of many famous investors:

Graph by Frederik Vanhaverbeke

I certainly don’t suggest that Greenblatt has fabricated the results but it is difficult to believe that his simple procedure has consistently outperformed all these other superstar investors by such a large margin. In fairness, we do not have access to any other information about the strategy; those returns might be realistic (but less impressive) if they were achieved by taking twice as much risk.

Preliminary analysis of the ranking

Before running a full backtest, it is useful to start by analysing the information content of whatever signal we aim to capture. In the case of the Magic Formula, the signal is the combined factor score. Quantopian’s research environment (along with the excellent Alphalens library) is the ideal tool for the job – it’s incredibly well-suited to cross-sectional equity factors like this and is easy to pick up for someone with a bit of python experience. Below is a brief snippet of the core ranking logic:

# Filter sector and volume
sector = RBICSFocus.l1_name.latest
min_mcap = 100e6
sector_mask = (sector != "Finance") & (sector != "Utilities")
mask = (Fundamentals.mkt_val_public.latest > min_mcap) & sector_mask

# Quality
ebit = Fundamentals.ebit_oper_ltm.latest
ev = Fundamentals.entrpr_val_qf.latest
earnings_yield = ebit / ev
ey_rank = earnings_yield.rank(mask=mask)

# Cheapnesses
net_fixed_assets = Fundamentals.ppe_net.latest
working_capital = Fundamentals.wkcap_qf.latest
roc = ebit / (net_fixed_assets + working_capital)
roc_rank = roc.rank(mask=mask)

# Compute the score
combined_score = ey_rank / ey_rank.max() + roc_rank / roc_rank.max()

We can then feed this into a boilerplate function which computes the forward returns for each stock and creates a report on the predictive power of the signal. The fastest way to get an initial indication of the signal’s potential is to look at the quantile plot, which shows how stocks with the high ranking scores perform compared to stocks with lower ranking scores.

These quantiles are quite good; we see an overall monotonic increase, suggesting that high ranking scores are indeed associated with higher returns.

After the quantile plot, the next thing to look at is the information coefficient (IC), also known as the Spearman’s rank correlation coefficient, which measures the degree to which the ranking score is correlated with future returns. With a one-year time horizon, the Magic Formula score had a decent mean IC of 0.08. But rather than just looking at a point estimate of the value, it should be remembered that the predictive value of a signal may not remain constant with time. This is demonstrated in the plot below:

The Magic Formula score’s predictive power over a time period from 2004-2015 varied quite a lot, even becoming negative in 2009-2010. However, especially in more recent years, the Magic Formula has a robust IC.

As a brief interjection, you may wonder why we are only testing using data up until 2015. The reason is that in quantitative research it is vital to leave a few years “untouched” to act as the final validation step before you deploy a model live.

Overall, the results of this preliminary analysis are rather encouraging. With good quantiles and a reasonably high IC, the Magic Formula ranking does seem to have some predictive value. However, we shouldn’t get too excited; not all predictive signals are monetisable since the signal may not have sufficient magnitude to be profitable after transaction costs. The only way to find out is to run a proper backtest.

Backtesting the Magic Formula

For this backtest, we will follow Greenblatt’s procedure as closely as possible. Each month, we will pick the top two stocks according to the ranking (provided we don’t already own them) and allocate 1/24 of the total capital to each. After the first year, there will thus be 24 equally-weighted stocks in the portfolio. Each subsequent month will require us to liquidate the oldest two stocks to make room for two new entrants.

The only part of the Magic Formula we are not capturing is the tax optimisation – selling losers before the end of the tax year and winners at the start of a new one. For a fair comparison, we won’t consider taxes on the benchmark portfolio either. Quantopian’s default backtest includes transaction costs in the form of a 5 basis point (0.05%) slippage incurred every time a trade is made. The algorithm will have 100% turnover each year by design so the slippage should not be a major factor but it is good to incorporate it anyway.

Between July 2003 and December 2015, the Magic Formula strategy returned an annualised 11.4% (Sharpe ratio 0.60), versus 8.7% for the S&P500 (Sharpe ratio 0.54). This is clearly an outperformance of the benchmark (3% alpha) but by nowhere near as much as the Little Book claims.

We can gain a more nuanced insight by considering different subsets of the backtesting period. Prior to mid-2007, the strategy was performing very well, achieving an annualised return of about 26% compared to about 18% for the benchmark – this is in line with Greenblatt’s claims. These returns are not just a result of taking more risk – notice the steadily growing specific return (the component of performance independent of the market’s movement, shown in red).

However, 2007-2010 tells a different story. The strategy had a drawdown of 57% (compared to 55% for the SPY), and the cumulative specific return went negative, meaning that despite the strong performance in the prior years, a “pure” (beta-hedged) version of the strategy would have lost money.

Post-2010, the strategy shared in the considerable market recovery, but its 12% annualised return was a slight underperformance of the benchmark’s 13%.

Over the backtest period, the Magic Formula was almost always more volatile than the market (which is to be expected since it holds only 24 stocks) and additionally had steeper drawdowns.

Improving the Magic Formula

Overall, the Magic Formula did indeed outperform the S&P500 between 2004 and 2015 but not by a large margin. For those of you who may be interested in building on top of the Magic Formula for your own investing, we now discuss some potential areas for improvement.

Composite factors: rather than using a single quantity to measure cheapness or quality, it might be better to aggregate different metrics, for example, earnings yield or FCF yield for cheapness. Jim O’Shaughnessy’s research in What Works on Wall Street suggests that a composite value factor is much more robust than single factors so we might have similar luck with quality and cheapness. As for the exact aggregation procedure, we could simply take the mean ranking across multiple different factors to start with but there is a lot of room here for sophistication – alpha combination can be seen as a portfolio optimisation problem so there is a large box of tools out there.

Trend: one danger with buying cheap assets is that there is nothing to stop them from getting cheaper. This could be mitigated using a momentum filter, only buying a highly ranked stock when we can see that it has positive momentum. Wesley Gray has done some interesting work in this area.

Long/short: if we believe that the Magic Formula score can identify both outperforming and underperforming assets, we can use a long/short basket strategy to hedge out market movement. The performance of this backtest should give a better indication of the quality of the ranking methodology compared to a long-only version. To make this more advanced, we can also consider hedging out various style exposures.

Before you boldly go forth testing these modifications, a word of warning. In the context of algorithmic trading, it is crucial to limit the number of backtests you run. What might seem like the “scientific” approach – running a backtest one at a time for each proposed modification to isolate its effects on performance – is poor practice because it can easily lead to overfitting and hence reduce the out-of-sample validity. To minimise this risk, a better approach is to sit down and carefully think about all the modifications you would like to make and ensure that you have a strong economic hypothesis behind each. For example, rather than carelessly adding a new fundamental factor (Quantopian has hundreds) you should argue why the factor should be predictive. If you later find out in a backtest that the signal is not predictive, or is predictive in the other direction, you should probably bin it.

I read an interesting discussion on the Quantopian forums where someone was trying to use the debt/equity ratio as a quality factor, with the hypothesis that a higher debt/equity ratio is associated with a lower quality company. The backtest looked great so it attracted a fair bit of attention but in a shocking twist it was revealed that there was a sign error in the code; in fact, a high debt/equity ratio was predictive of positive future returns. In this case, there is indeed a believable economic hypothesis to explain the observed effect – leverage amplifies returns (incidentally, this was a key point in Joel Greenblatt’s thesis on Host Marriott in You Can Be a Stock Market Genius) – but generally, you should be very careful about creating ex-post justifications for observed phenomena.

Conclusion

Based on our backtest, the surprising conclusion is that the Magic Formula is indeed a simple procedure that beats the markets on a risk-adjusted basis, with an annual alpha of 3% between 2003 and 2015. However, there are a couple of major caveats.

Firstly, the magnitude of outperformance is nowhere near what Greenblatt describes in The Little Book (33% vs 15%). This is likely due to the shift towards systematic equity ETFs over the past two decades, which arbitrage away inefficiencies like this. Furthermore, I hate to repeat the statement that “past performance is not indicative of future returns” but underlying this hackneyed phrase is the deeper concept of regime change. The performance of the Magic Formula in the post-crisis bull market was nowhere near as good as the performance in the 2004-2007 period; it is very possible that the 2008 GFC and subsequent central bank action represented a fundamental sea change, reducing the efficacy of certain value-oriented strategies. Hence, allocating capital to the Magic Formula strategy is implicitly making a bet that over your investment horizon, the market will be in a regime that rewards cheap high-quality companies.

Secondly, following the Magic Formula would still incur a significant psychological risk – the tendency for investors to let their emotions result in bad decision-making, for example, selling at the bottom of a crash or buying at the top of a bubble. It is very easy to look at the high annual return on a backtest and wish that you had used the strategy sooner, but would you really have been able to sit through a 56% drawdown like in 2008 without pulling your money out? Sure, in this case the market also experienced a great drawdown, but how about situations in which the market is going up but the strategy is losing money – in 2012, the Magic Formula strategy lost 5% while the SPY appreciated by more than 10%.

Although this post represents neither a confirmation nor dramatic refutation of the Magic Formula strategy, I hope that it at least provides a framework for investigating cross-sectional equity strategies and highlights some of the important pitfalls in backtesting.

The Big Shorts: what is the smart money betting against?

2020-05-27T00:00:00+00:00

Hedge funds in the UK are legally obligated to disclose to the Financial Conduct Authority (FCA) whenever their net short position in a particular listed company reaches 0.5% of the issued equity capital. In this post, I investigate a publicly available dataset containing information about these large institutional short positions in UK equities and attempt to understand the value that this data contains.

This is a true data science project, in that the main difficulties I had were not so much related to the analysis (there is no machine learning involved), but rather the cleaning and preprocessing of data. Usually when I write posts like this, I explain the code as I go along, but today I’m experimenting with a new format aimed at readers who are more interested in the actual outcomes of the analysis rather than the process. To that end, I have put all of the code (with brief explanations) in an accompanying Jupyter notebook; this post will focus on the motivation and results of the investigation, rather than the methods.

Short selling

A short sale is a relatively simple transaction in which you borrow shares of a company, then immediately sell them. If the share price drops, you can buy them back at a cheaper price, return them to the lender, and pocket the difference. Rather than buying low and selling high, shorting is equivalent to selling high and buying low. This simplicity notwithstanding, shorting is often associated with more experienced investors. One reason could be that short selling has a theoretically unlimited downside – if you shorted Tesla at \$300 a share and covered at \$800, you would be out of pocket by \$500, whereas if you bought at \$300 and it went to zero, you would only be out of pocket by the \$300 you put in.

Short selling is often demonised in the media, construed to be one of the many evil things that hedge funds get up to at the cost of society. However, there are many good reasons why you may want to bet against a company; the famous short-seller Jim Chanos aggressively shorted Enron in the lead up to its collapse in 2001 (and more recently, Luckin Coffee prior to its accounting scandal). Furthermore, it is not possible for short selling in and of itself to cause a company’s equity value to collapse because of the uptick rule (a stock can only be shorted after it ticks up). I am of the belief that short selling is an important market mechanism which may help to counteract the “irrational exuberance” that often plagues markets; it is fundamentally no better or worse morally than buying a stock. In both cases, capital is allocated based on your views of future performance.

The significance of the FCA dataset

As stated in the introduction, the Financial Conduct Authority in the UK mandates that investors disclose short positions exceeding 0.5% of a given public company’s float. These disclosures make for a fascinating dataset because hedge funds tend to be notoriously secretive about their positions since disclosing positions (long or short) may allow competitors to cotton-on to the fund’s strategy and accelerate alpha decay. In the case of short selling, there is an added dimension: disclosure leaves a fund susceptible to targeted short squeezes, in which other market participants start to buy up shares, driving up the price and sometimes forcing short-sellers to cover their positions at a loss (further driving up the price).

Nevertheless, because of the FCA, we can get a rare glimpse into the holdings of the smartest players in the industry. Furthermore, the FCA additionally requires that once companies are above the 0.5% mark, they must disclose any changes in their position of more than 0.1%. Hence we not only know whether an institution has a large position but also how they are modifying their position in response to external factors.

The dataset, which updates every day, can be downloaded from the FCA website. This is what the top of the dataset looks like:

With this in mind, the main hypothesis that this post will be investigating is as follows: companies with a lot of institutional short interest (i.e companies for which many smart players are shorting the stock) should be more likely to experience a decline in the share price.

Data cleaning and exploration

One problem with the FCA dataset is that it gives the company names and International Securities Identification Number (ISIN) rather than symbols like AAPL or TSLA. ISINs are “better” in the sense that they uniquely identify securities (symbols are not unique across different exchanges), but the difficulty is that most of the free data providers only let you query historical data by symbol. I had to write a web scraper to convert the ISINs to tickers, which was a real pain, but it allowed me to download historical prices for most of the present securities.

Having done the dirty work, I was able to start exploring various aspects of the data. I started by looking for the most active investment, which turned out to be Marshall Wace’s position in Sainsbury’s (at one point, the net short position was 2% of the float, more than £100 million). The plot below shows the net percentage short position (green) against the rescaled price (blue).

The flat regions for the short position are expected because below 0.5% Marshall Wace does not have to disclose anything. This plot is rather interesting because the big spikes in the short position do not seem to be associated with decreases in the stock price (a statement that will be refined later on).

Next, I was interested to find out which investors were most active in short selling:

To get a rough indicator for the most shorted stocks, I took the mean short position in a given stock across each investor, then summed these over the time frame. By this measure, the most-shorted stock was Ocado, the UK grocery delivery company.

Next, to start exploring the hypothesis, I plotted the 1-day forward returns against the short position (naively, without grouping by investor or ticker). If the hypothesis were true, we would expect a general downward-sloping pattern (larger short position means lower returns).

Eyeballing it, there does not seem to be any meaningful relationship. The Spearman rank correlation coefficient (also known as the information coefficient) can be used to quantify this – I found that there were only statistically significant correlations across the longer time horizons. Perhaps more interestingly, the sign of the correlation was positive ($r \approx 0.015$). This means that short positions are correlated with future price appreciation, the opposite of what was hypothesised.

Intelligent short interest as a trading signal

The above analysis is very crude because it does not perform any grouping. To improve it, I aggregated the net short position in each stock across all investors for each given date, which I will henceforth refer to as the “intelligent short interest” since it is the short interest from large investors who we hypothesise to be knowledgeable.

We can then use the wonderful Alphalens library, built by Quantopian, to properly evaluate the predictive capability of this feature. One particularly useful tool is the quantile analysis, which evaluates the performance of different feature quantiles (e.g. high/median/low values of the short interest). For example, if the intelligent short interest were indeed predictive of negative returns, we would expect the 20% of assets with the highest intelligent short interest to do much worse than the 20% of assets with the lowest intelligent short interest. Using 5 quantiles, we can look at the information coefficients (read: Spearman correlation coefficients) to understand the predictive value of the feature:

This tells us a slightly different story to the initial analysis. There are indeed statistically significant negative correlations between the intelligent short interest and the future return, although the signal isn’t particularly strong. An especially damning plot is the mean forward return for each quantile:

Colourful though the plot may be, it’s pretty terrible from a signal perspective. It is telling us that regardless of which quintile we are in (low vs high short interest), forward returns are positive and there is no clear pattern. By contrast, below I have attached a quantile plot for another trading signal I have been researching, which shows excellent discriminatory power between the extreme quantiles:

I wasn’t ready to give up just yet – sometimes, a bit of feature engineering can be used to bring life to a seemingly useless signal. Rather than looking at the absolute intelligent short interest, I thought it might be interesting to consider the change in the intelligent short interest, with the hypothesis that an increase in the intelligent short interest is predictive of negative future returns. In fact, this signal was even worse than the absolute intelligent short interest:

In summary, the results of this section are mixed at best. Based on the statistical significance of the information coefficient (even after applying a Bonferroni correction to account for multiple comparisons), we would conclude that the hypothesis is correct – formally, that there is sufficient evidence at the 1% level to reject the null hypothesis that there is no correlation between the intelligent short interest and the future returns. However, statistical jargon aside, the predictive power is rather weak and the quantile plots are a clear indication that the intelligent short interest is not going to be a useful trading signal.

The smartest smart money

This section is a slight detour from the hypothesis about the predictive power of the dataset. I noticed that in principle we had enough data to evaluate the performance of different investors, using the time-weighted rate of return (TWRR). The result of this calculation is tabulated; here is a segment of it:

This is interpreted as follows: AHL Partners was involved in shorting ASOS (ASC.L) over a period of 448 days in total, during which time its overall TWRR on the position was -96%. Of course, this is only based on the time during which AHL’s position was more than 0.5% of ASOS’s float, so this TWRR will not be perfectly accurate.

The performance data was further cleaned to exclude total involvement periods of less than a year since I am more interested in identifying which managers are consistently making positive returns with their large short positions. A histogram of the annualised short returns is shown below:

The distribution itself is quite interesting, displaying long tails (longer on the left) and a sharp peak – note that returns can be more than -100% because we are dealing with the TWRR. The summary statistics paint a richer picture:

Statistic	Value
Mean	-0.8%
Median	-5.0%
Skew	-0.02
Kurtosis	85.9

The high kurtosis means that the distribution has much fatter tails than a normal distribution, suggesting that the “outliers” are quite important, while the negative skew implies that the left tail is longer. However, the thing that jumped out at me was the huge difference between the median and the mean. The median short investment returns an annualised -5%, while the mean returns about 0%. This suggests that there is a small group of investments massively pulling up the average.

We can go one step further – rather than just evaluating the performance of investments, we can look at the performance of investors. To do this, I used a simple aggregation procedure, taking the investor’s mean return across all positions. This is certainly not the best way of doing things because it assumes that the shorts are all of equal value. The dataset actually contains enough information for us to calculate the value of each investor’s shorts, and come up with a dollar performance. Nevertheless, proceeding with the simplified analysis, the histogram of each investor’s average performance is shown below:

This provides evidence that it is not just a small group of investments pulling up the average, it is a small group of investors. The top 10 investors, their average period of involvement, and their annualised return on the large short positions are as follows:

Out of courtesy I will not name and shame the bottom 10, but they can be found in the Jupyter notebook if you’re curious. I was surprised to see many big names, but in retrospect, because this analysis does not take into account dollar returns, the playing field is tilted in favour of the smaller players.

Lastly, we can try to get some sense of the different investment styles (and their relative successes) by plotting both the average annualised TWRRs and the average involvement period. The size of the points is proportional to the number of different companies being shorted (a good proxy for the assets under management of the investor), while darker colours correspond to higher frequencies.

The plot can be used to comment that, for example, although D1 Capital Partners and Arrowgrass Capital Partners both have a similar annualised return of about 65%, D1’s investment style involves much shorter involvement periods with a given company. Unsurprisingly, it is also seen that the larger players also trade with higher frequency.

Conclusion

In conclusion, the short disclosure dataset is almost certainly not a basis for a simple copycat strategy. Although the net intelligent short interest has a statistically significant negative correlation with future returns (over time horizons of more than a month), the predictive power is weak, highlighted by the quantile plot. This may seem strange – the hypothesis was grounded on the reasonable assumption that the bets of hedge funds, who we consider to be the “smart money”, would be pretty good predictors of future returns.

However, we must remember that this dataset only represents half of the story. Hedge funds (the clue is in the name) often hedge their long positions with shorts. These shorts may not be expected to have positive returns when the overall market is going up, as long as they outperform when the market is going down. In particular, a common strategy for large players like Marshall Wace is the long/short basket strategy, in which you go long a large group of companies and short another group, based on some ranking methodology. In this case, you almost expect most of your shorts (and longs) to do badly – it is the law of large numbers that results in a net profit.

Another possible explanation for the weak predictive effect is short squeezes, in which an initial small increase in prices can result in a positive-feedback loop of liquidation as investors cover their positions.

Despite the mixed results, this kind of analysis may provide a useful datapoint for discretionary equity portfolio managers because it shines light onto the landscape of marginal buyers/sellers. For example, if you see that a certain fund holds a large short position in a small company, you know that at some point in future there might be significant demand for the shares as the fund seeks to close its short position.

We have also drawn some other interesting insights from the dataset, for example, the fact that a small group of investors seem to be pulling up the average. This could be an area for further investigation – rather than blindly copying from the person sitting next to you, it might make sense to first identify who the smartest person in the classroom and copy from them.

Statistical arbitrage in closed-end funds

2020-05-10T00:00:00+00:00

Sometimes, it is cheaper to buy a basket of assets than it is to buy the assets in the basket. In this post, we discuss closed-end funds and why they often trade at a discount to their net asset value. Furthermore, we explore whether this could be the basis for an algorithmic trading strategy.

Closed-End Funds vs ETFs

Most of us are probably familiar with exchange-traded funds (ETFs) – baskets of securities where the basket itself is tradable. The ETF industry has grown rapidly over the past few decades because ETFs provide a low-cost way to get exposure to a certain market, sector, or strategy.

In this post, we will instead be discussing closed-end funds (CEFs). On the surface, these may seem similar to ETFs: CEFs are also baskets of securities, and they can also be listed and traded on an exchange. A pedant might point out that by definition, a CEF is an ETF since it is a fund that trades on an exchange – this is unfortunately not correct, as there is a true conceptual distinction.

We often like to think of ETFs as upscaled equivalents of a personal trading portfolio. One gives money to a manager, who in return issues a share of the ETF to represent the ownership and invests that money in a basket of assets. This mental model is, in fact, a much better description of CEFs. ETFs are a little bit more complicated. For both ETFs and CEFs, an important point is that the share price is not “naturally” guaranteed to be equal to the value of all of the securities that the fund owns, i.e the net asset value (NAV). This is because the share price of the fund is determined by supply and demand – if everyone in the world suddenly wants to buy the shares of a fund, the share price can shoot up independently of the value of the underlying assets. However, ETFs have an interesting mechanism for ensuring that these deviations between the price and NAV are very short-lived.

ETF fund managers are seldom the people who buy and sell the securities. They instead turn to authorised participants (APs), typically investment banks, who transact in the underlying securities. The fund manager then issues shares of the ETF to the AP, in return for the securities that were bought. This process is known as creation because shares of the ETF are being issued. The AP is then free to hold these shares, or more often, trade them on a stock exchange. However, the AP’s role goes far beyond the initial purchase of the securities to set up the ETF. They play a critical role in ensuring that the ETF share price never strays too far from the NAV. Any momentary supply-demand imbalances can be arbitraged away – for example, if there is suddenly less demand for the ETF, and it trades at a discount to the NAV, the AP can buy ETF shares on the open market and redeem them from the ETF fund manager. This locks in an arbitrage profit and drives the ETF price back towards the NAV.

This is why ETFs are open-end funds: the fund manager can issue/redeem however many shares they want. Conversely, CEFs issue a fixed number of shares when they first IPO, and there is no subsequent creation or redemption. As a result, we might expect there to be a larger spread between the CEF price and the NAV (compared with an ETF’s price and its NAV).

NAV Arbitrage and the Closed-End Fund Puzzle

Although CEFs don’t have the same creation and redemption mechanism as ETFs, there is still a fundamental link between the NAV and the price.

Let’s take a very simple case where we have a “basket” of two stocks, say AAPL and FB. As of the time of writing, AAPL has a price of \$289/share and FB has a price of \$202/share. The fair price of this basket is then $\$289 + \$202 = \$491$. This is intuitively obvious, but to drive home the point, we can ask what would happen if the basket weren’t priced at \$491. If the basket was instead trading at \$400, we would be correct in thinking that the basket is “cheap” – we’d certainly rather buy the basket than the two stocks individually. We can take this a step further and go long the basket and short the individual stock, which results in a risk-free profit once other market participants realise that the basket and the two stocks should be priced identically.

If we are a majority shareholder in the fund, we don’t even have to rely on the rationality of other market participants; we can force a liquidation event, in which the fund manager sells all the securities, distributes the proceeds as dividends, then delists the shares. At such a time, the price of the shares must be equal to the NAV, less the cost/slippage from selling the securities.

Thus, even without APs, the price of a CEF should track its NAV. However, empirically, this is not the case. Below, we show the price vs NAV for two large funds, the Eaton Vance Global Diverse Equity Fund (EXG), and the Aberdeen Asia-Pacific Income Fund (FAX), using data from CEFConnect:

For both of these funds, the share price trades at a persistent discount to the NAV – in the case of FAX, this discount is almost 20%. The existence of this spread (typically a discount) between the price and the NAV is known as the Closed-End Fund Puzzle.

There are several responses to the closed-end fund puzzle, though I would refer interested readers to Lee, Shleifer and Thaler (1991) for a detailed investigation:

Cost of transacting – we already touched on this. In a liquidation event, we will not receive the full NAV because of the transaction costs, which may be rather significant if we are dealing with assets that are hard to sell. We can verify that this is indeed a contributing factor by observing that there seems to be a broad negative correlation between asset liquidity and the spread. Equity CEFs typically have a smaller discount compared with bond CEFs, because equities are easier to sell.
Difficulty of arbitrage – in practice, it is hard to lock in a risk-free arbitrage because it may be very difficult to transact in the underlying (which is sometimes why the CEF was created in the first place). Furthermore, the fund managers are free to enter and exit positions as they desire, so an arbitrageur would have to constantly make changes to adequately replicate the portfolio.
CEFs have a much lower fraction of institutional ownership than the underlying – it is possible that the pricing of CEFs is less rational as a result, with price movement driven by liquidity and sentiment rather than value.

The rest of this post will investigate the hypothesis that the spread between the CEF price and its NAV is mean-reverting, or more intuitively, that larger discounts are predictive of a future price increase and larger premiums are predictive of a future price decrease.

Statistical arbitrage

Statistical arbitrage (stat arb) is a fancy term describing the process of buying assets that are statistically cheap and selling assets that are statistically expensive, hoping to lock in the difference. The key is that these arbitrage opportunities are not at all risk-free, they are only really arbitrage opportunities in the sense that if we make a sufficiently large number of such bets, we will have a positive expected value (the law of large numbers).

The classic stat arb strategy is pairs trading, which involves finding two assets that are highly correlated and trading the spread between them. The standard example is Coca-Cola and Pepsi. Rather than making a bet on the overall direction of either stock, we can eliminate idiosyncratic risk by trading the difference between the price of Coca-Cola shares and Pepsi shares. To ensure that our correlations aren’t spurious, it is desirable for there to exist some fundamental economic reason why the two assets are related. In the case of Coca-Cola and Pepsi, the argument goes that because they are essentially substitute goods, any widening of the spread should be temporary. So if Coca-Cola stock appreciates while Pepsi’s stays flat, the statistical arbitrageur may long Pepsi and short Coca-Cola.

Stat arb is most effective when the spread is stationary, a technical term which describes the fact that the data-generating process for the time series does not change (its statistical properties are constant in time). This is a logical requirement – if the statistical properties were changing, nothing stops the spread from widening forever. Stationary series, as a consequence of the definition, are almost always mean-reverting – wider spreads are more likely to be followed by narrower spreads and vice versa.

In many ways, the spread between the price and NAV is the ideal candidate for a stat arb strategy. There is an incredibly strong economic link between the NAV and the price; we even know that at liquidation they must converge (less fees). The only wrinkle is that the NAV isn’t actually a tradable security. This doesn’t completely preclude the development of a trading strategy, but it wouldn’t be a proper pairs trade since we would only be acting on one of the legs. For example, if the CEF were trading at a 20% discount to NAV, a pairs trade would involve going long the CEF and “short the NAV”. In this case, even if both the NAV and the price of the CEF decrease, your position would be fine because the short side hedges you. But since you can’t trade the NAV, although you are still able to make bets on convergence (by going long the CEF when it is trading at a discount), such bets would not be pure bets on the spread.

It may be possible to work-around this by computing the beta of the NAV with respect to a comparable asset/ETF then trading the correct amount of that comparable. For example, if we were dealing with a CEF that invests in high yield bonds, we could compute the beta of the NAV to a high yield bond ETF (e.g. HYG). Let’s say $\beta = 2$, in which case we could bet on the spread converging by going long \$10k in the CEF and short \$5k of HYG. But again, this is not a perfect bet on the spread.

This important caveat notwithstanding, we shall now examine the stationarity of the price-NAV spread and investigate whether the spread is predictive of future returns.

The price-NAV spread

We will look at a single CEF, namely, the Liberty All-Star Equity Fund. This was chosen because it is large and liquid (and has a memorable ticker, USA). Using an appropriate colour scheme, we can plot the price and NAV:

From 2000 to 2009, the CEF alternates between trading at a discount/premium to NAV, before settling into a persistent discount from 2009 onwards. Below is a plot of the spread, i.e $\text{price}/\text{NAV} - 1$ (positive is premium, negative is discount).

Now, we can perform an Augmented Dickey-Fuller test to determine whether the spread is indeed stationary. It turns out that although there is not enough statistical evidence to reject the null hypothesis that the spread is non-stationary, the price and NAV are significantly cointegrated (this intuitively means that some linear combination of the price and NAV is stationary).

Stationarity aside, let’s press on with a somewhat crude estimate of the predictive value of the spread. Rather than doing any kind of proper backtest, we will just compute the correlation between the current spread and the subsequent change in the spread. To be specific, we shall use the Spearman rank correlation coefficient, sometimes referred to in finance as the information coefficient (IC), which is more robust to nonlinearity than the Pearson product-moment correlation coefficient.

We find that the correlation between the current spread and the change in the spread over the following week is $-0.06$ (statistically significant at the 5% level). It is comforting that there is a negative sign because it supports the hypothesis that the spread mean-reverts: premiums are correlated with decreasing premiums, discounts are correlated with narrowing discounts.

However, as we discussed previously, the NAV isn’t tradable so we instead want to find out the correlation between the current spread and the price return of the CEF. The same Spearman rank calculation gives a correlation of $-0.07$. This is quite a decent IC – according to a paper by JP Morgan Equity Research, investors can “achieve significant risk adjusted excess returns with information coefficients between 0.05 and 0.15” (the sign doesn’t matter here). We can visualise the relationship with a scatterplot:

Conclusion

There is clearly a lot more exploration that needs to be done – we have only looked at the spread for a single fund, and have chosen a simplistic information coefficient analysis rather than conducting a full backtest. Nevertheless, the results thus far have been quite encouraging. In addition to the reasonably high information coefficient, we have a clear economic hypothesis about why the discount to NAV may be a persistent predictive factor. In a recent MacroVoices podcast, Eric Peters shared a great insight (emphasis is mine):

Our approach to anything that we ever do in markets, is to first ask the question: why do you get paid to do something?

Without this economic understanding, it’s much more likely that any pattern you’ve discovered is a statistical spectre that haunts the historical data but will disappear in the daylight of future data.

To answer the question in the introduction, I do think that this could form the basis of a trading strategy and hope that over the next few weeks I will have a chance to investigate further.

A Tanker Trade

2020-04-24T00:00:00+00:00

April 2020 has been a volatile month for oil. Last week, the May WTI contract traded at a low of minus \$40 a barrel. In a desperate search for storage space, people have been chartering oil tankers to use as floating storage units, leading to a price surge in shares of tanker companies like Nordic American Tanker (46%), Teekay (30%), and Scorpio Tankers (59%). In this post, we aim to build a framework for forecasting the revenue of DHT Holdings (NYSE:DHT), a tanker company.

The Excel models and accompanying python code are here. Please use them responsibly!

It takes two to contango

I am certainly not an expert on energy markets, but over the past couple of weeks, I’ve seen a lot of misinformation regarding oil prices. Since this is relevant to the content of this post, I want to briefly clarify a few of misunderstandings (cognisant of the fact that I probably have a few of my own).

As a retail trader, it is very difficult for you to trade spot/physical oil. Instead, you might use oil futures, or more likely, an ETF like USO (which owns short-term futures). In usual times, USO does a decent job of tracking spot oil, so it’s a cheap way for you to get exposure to oil.
Because of the coronavirus, there has been a massive demand shock. The resulting surplus in oil has lead to a lot of the storage capacity being filled up, resulting in a much higher storage cost.
Futures contracts are thus more expensive the further out you go because they bake in the storage cost. This situation is known as contango, or in this case super contango owing to the magnitude of the effect.
NYMEX WTI futures are physically settled, so when it comes to expiry, owning a long contract obligates you to receive delivery of 1000 barrels of oil. When the storage tanks are full, the piece of paper that gives you the right (and obligation) to receive oil becomes a liability – you are willing to give it away, or even pay someone to take it off your hands. Hence “negative oil prices”.

All this leads to the surprising fact that if you buy oil at \$5 a barrel and it trades at \$20 a barrel two months later, you have not quadrupled your investment. You might have actually lost money on this trade – it’s impossible to say with just the information I have given you since you need to factor in the cost of rolling your contract to the next month. When the futures curve is in super contango, this roll can be incredibly costly. We will later revisit this concept when we try to infer the storage cost from the futures curve.

Modelling revenue for DHT Holdings

The main goal of this post is to build a framework for forecasting the revenue of a tanker company. Though I don’t intend for this post to be a stock pitch, our discussion will be much more tangible in the context of a particular listed company. I decided on DHT Holdings because it is a pure-play on tankers: DHT owns 27 VLCCs (Very Large Crude Carriers – yes, that is their technical name) and their business does nothing but rent these out. In 2019, they had \$535m in revenue, all of which was from shipping. Of their expenses, less than 4% were administrative; everything else was related directly to the tankers. This makes the revenues rather easy to forecast in principle. If you know how much the daily rate that DHT charges you to charter a tanker you can multiply this by 365 days then by the number of tankers to ballpark their annual revenue.

The model essentially consists of an Excel spreadsheet containing estimates of the annual revenue contributions from each of the 27 tankers. The main complexity we will be accounting for in this simple model is that you can either time-charter a tanker, paying a fixed price to rent it for a longer period of time (6 months to a few years), or you can do a spot charter, renting the boat for a short period on short notice. As always, we turn to DHT’s most recent annual report (in particular, the notes to the financial statements therein), which provide a wealth of information regarding how DHT’s vessels were used in 2019.

Time-charters

As of the end of 2019, some of DHT’s ships had already been chartered for 2020. Unfortunately for DHT (and luckily for the people renting the boats), these rates are fixed so DHT cannot raise the prices even though demand is high. The annual report tells us that the 2020 revenues from the time-charters will be \$47.4m. I divided this by the four boats that will be chartered for all of 2020 (the DHT Lotus will only be chartered until Q1), then divided by 365 to estimate the daily rate for these four ships. The result comes to about \$32,000 a day.

On April 1 2020, DHT announced time charters for another 6 VLCCs. These time charters are all for 12 months with a daily rate of \$67,300. As we can see, this is more than double the daily rate of the existing charters, though this premium is not entirely due to supply and demand – you would expect longer contracts to have a cheaper daily rate because it reduces risk for DHT.

Spot charters

Sadly, once we have dealt with the time charters, things become a lot more difficult. This is because the spot charter rate depends on supply and demand – if everybody wants to charter a ship, DHT can raise the daily rates. We will model the situation as follows. Because of the WTI supply glut, there will be a certain number of high-demand days for which people will be willing to pay the high-demand daily rate. For the rest of the year, we use the low-demand daily rate.

We may further assume that the low-demand daily rate will be similar to what it was last year. We are told that in 2019 there was \$478m in revenue from spot chartering (between 22 ships), i.e an average daily rate of \$60,000. Hence the two key variables are the number of high-demand days and the high-demand daily spot rate. This is where the estimates start getting pretty subjective, but we will later do a sensitivity analysis that somewhat mitigates this.

I’ve been reading about people trying to hire tankers for storage since at least mid-March, so we know that the number of high-demand days is going to be greater than 30. A quick look at the futures curve shows that the storage cost is going to be very high for the month of May and possibly even June. Let’s err on the side of caution and say that there are 60 days of high demand.

Now for the hardest part – how do we quantify what “high demand” means in terms of the spot rate? Well we, know that in 2019 the average spot rate was about \$60,000 compared to \$30,000 for the time charter. The most recent time-charter had a rate of \$67,300, so assuming 2019’s time:spot ratio holds in 2020, an initial estimate of the spot rate is about \$130k. However, this was on April 1st, about 3 weeks before the extent of the supply constraints came to a head with negative oil. A number of different sources (not particularly credible) estimate a spot rate of over \$200k a day.

With these assumptions, we can build a simple table adding up the revenue from each VLCC. As a baseline, I am using CapIQ’s consensus revenue estimate of \$646m (updated as of 21st April), which is 20% higher than 2019’s revenue.

It’s nice to see that our forecasted revenue is in the same ballpark as the CapIQ consensus estimate (if it were 10x higher or lower, it’d be more likely that we were doing something wrong).

The futures curve and the market-implied cost of carry

In the initial analysis, we gave a very haphazard estimate of the high-demand spot charter rate. Let’s see if we can do a little bit better by examining the futures curve for oil:

The curve is said to be “in contango” because it is sloping upwards – this is associated with the cost of carry (the cost of storing the underlying), because you’d be willing to pay a little more (or in these times, a lot more) to avoid having to hold the physical oil.

Now for some maths. Let’s say the cost of carry is $s$ per year (expressed as a percentage). So if $s=0.10$ and the oil costs \$20 a barrel, the storage cost is $2 a year. Hence by a simple no-arbitrage argument, the price F of a futures contract expiring 1 year from now relative to the spot price $S_0$ is:

\[F = S_0 (1+s)\]

This is called a no-arbitrage argument because if it weren’t true, you could make a risk-free arbitrage profit. Note that I am ignoring the interest rate because it is so low (money printer go brrrr). With the above formula, we see that the implied storage cost (in dollars) between any two months is the price difference of their futures contracts. For example, as of 24/4/20, the June contract is trading at \$16.48 and the July contract is trading at \$21.80. Hence, a reasonable estimate for the storage cost in June is $\$21.80 - \$16.48 = \$5.32$. I was also interested in calculating the annualised cost of storage for a given month, so derived the formula below, but it’s not relevant to our discussion.

\[\begin{align*} F_1 &= S_0 e^{sT_1} \\ F_2 &= S_0 e^{sT_2} \\ \implies \frac{F_2}{F_1} &= e^{s(T_2 - T_1)} \\ \end{align*}\] \[\implies s = \frac{1}{T_2- T_1} \ln \left(\frac{F_2}{F_1} \right)\]

Below is a plot that shows the implied cost of carry for different months, overlayed on the futures curve.

Using the June cost of \$5.32 a barrel, we can put an upper bound on the high-demand spot charter rate. It can be calculated, using the density of oil and the deadweight tonnage, that a VLCC can carry about 2 million barrels of oil. Thus, a cost of carry of \$5.32 per barrel for a month is equivalent to a daily rate of $\$5.32 \times 2 \times 10^6 ~/~ 30 = \$355,000$. This is only an upper bound because in reality there will be a nontrivial cost of freight, so it is not fair to allocate all of the \$5.32 to the cost of the VLCC. In any case, it is nice to see that our initial estimate of \$200k a day lies between our supposed lower and higher bounds. In fact, \$200k seems like quite a conservative estimate for the high-demand rate, because it is the June storage cost. If we calculate the May storage cost based on the settlement price of -\$40, we get the ludicrous spot charter rate of \$3.8m a day. I’m not saying that this is anywhere close to the true May storage cost, but it’s just to illustrate that \$200k is very much on the low end.

The astute reader will note that we could just use each month’s market-implied spot charter rates directly, rather than our unsophisticated “high-demand low-demand” model. However, I think that the current procedure of reverting to the 2019 cost for the low-demand spot rate is more robust and has significantly less model risk, so we’ll stick with that.

Sensitivity analysis

Before we conclude, let’s conduct a sensitivity analysis. This is a critically important part of any model because it emphasises the intrinsic uncertainty of trying to forecast the future. Concretely, we make Excel recompute the quantity of interest (in this case, FY20 revenue) under a range of different values of our two key inputs: the high-demand spot rate and the number of high-demand days.

Because one of the objectives of this exercise is to understand the downside, I have centred the high-demand rates around the initial lower estimate of \$200k. For the high-demand days, we use a range of values from 45 days to 75 days, centred around 60.

I’ve applied special formatting (dark red with white text) to represent the revenue forecasts that are below the CapIQ estimate. You’ll notice that these are in the top-left as expected, corresponding to a lower spot rate and a shorter period of high demand. However, under a broad range of other assumptions, the model is forecasting positive earnings surprises relative to the CapIQ estimate.

Encore

This section was added a few days after the original post, building on the feedback I received from Reddit.

I posted my initial analysis on Reddit and it generated some nice discussion. Someone pointed out that you can actually find spot charter rates online; in particular, there is a Twitter user that posts nothing but charter details:

This is very useful because it removes a lot of the subjectivity in our estimation of spot charter rates. To that end, I wrote a python script that does the following:

Downloads and parses the tweets using tweepy
Extract the ship name, daily charter rate, the number of days chartered, and start date
Combine this with the charter data from their annual report and press releases (which I had to input manually)
Build a pandas dataframe with 365 columns (one for each day) and 27 columns (one for each ship)
Fill in the data for the days we know
Output to excel

A section of the spreadsheet is shown below:

All of the zeros correspond to days for which we don’t have any explicit information. We can then fill these in with whatever estimation model we see fit. Rather than using the two-stage model from before, I’m going to do something much simpler and replace the zeros with the mean charter rate of non-missing entries:

This results in a forecasted 2020 revenue of \$652m, very close to CapIQ’s estimate. I think the true revenue will lie somewhere between this value and the more optimistic forecast from the first model.

Conclusion

In this post, we have built a very simple model for DHT Holdings’ 2020 revenue by combining their fixed time charter rates with our estimates of the spot charter rate, in light of the recent supply glut. Under fairly conservative estimates, I initially forecasted a 10% revenue beat on the upside. However, with more refined data based on actual spot charter rates this year, the CapIQ consensus estimate seems fair.

There are many variables that I have conveniently chosen to ignore. For example:

Freight, which we mentioned but didn’t quantify, may be quite expensive. Rail freight costs about \$5 a barrel, which would constitute the lion’s share of the \$5.23 June cost of carry.
It takes a couple of weeks to get a VLCC from the Suez to the Texan coast. There may be complications with space at the ports (which may not have enough space to park a huge number of VLCCs).
There may be secular headwinds, such as the new 2020 International Maritime Organisation regulations regarding fuel sulphur content, which may lower the low-demand spot rate.
We have deliberately tried to minimise the number of explicit assumptions regarding the direction of the oil market, because it is not an area that I have any edge in.

However, I think that the Excel model nevertheless provides a reasonable starting point for people to input their own estimates, perhaps factoring in the nuances. Besides, the sensitivity analysis is a nice tool to show you how varying your inputs affects the model output, and in this particular case, shows a reasonable margin of safety. I want to emphasise that this post is not a stock pitch; at best, it is a component of a stock pitch. Ultimately, I suggest you put in the time to come up with your own estimates of the different inputs, and in so doing, tell your own story about the company.

Understanding the market's expectations of COVID-19

2020-03-25T00:00:00+00:00

One of the reasons why I find markets fascinating is that they are capable of integrating huge amounts of information, misinformation, hope, fear and uncertainty into a single number – the price of an asset. In this post, we work backwards, quantitatively examining what the current price of an asset can tell us about its future prospects.

The importance of expectations

For an investment to be successful, it is not enough that you make a correct prediction. You must make a correct prediction that hasn’t already being priced in by the market. Amateur investors quickly realise that buying Google stock because “everyone uses Google!” is a bad idea – the market realised the same thing long ago and has therefore rewarded Google with a share price that is 25 times its annual earnings.

The mechanism for this is as follows: assuming an efficient market (we will comment on this assumption later), if market participants expect a company’s earnings to go up next year, they will buy the stock in advance. So once all market participants have formulated and traded on their views, the market price of the asset encodes future expectations for earnings (or to be precise, cash flows).

To make a successful investment, it really helps to have some idea of what these expectations are so that you can understand whether or not your perspectives deviate from consensus. Only then will you be able to profit – if your deviant perspective is correct.

Price-implied expectations

Unfortunately, although we have just argued that the current price incorporates expectations, the reverse procedure of figuring out the expectations implied by a price is not trivial. Imagine I have a basket of fruit containing some apples and bananas (quantities A and B respectively). I tell you that there are 5 pieces of fruit in the basket. How many apples do I have?

Obviously, there isn’t a (unique) answer to this question because I haven’t given you enough information. There are two unknowns (A and B), but only one equation relating them, $A + B = 5$.

This silly analogy goes quite far in explaining the difficulty of working out the price-implied expectations. There are many variables that determine a stock price, e.g. future sales growth, profit margins, reinvestment rates, interest rates – and for each of these variables, market expectations matter. In the same way that knowing the total quantity of fruit was insufficient to calculate the number of apples, the price is insufficient to calculate the expectation for a particular variable, like the future sales growth.

However, although in the fruit example we weren’t able to calculate a unique value of A or B, we were able to restrict the possibilities to those values of A and B that satisfy $A+ B = 5$. This relationship can be visualised as a straight line in “fruit space”:

We will revisit this idea in due course. However, we must first understand how assets are priced (how the total number of fruits in the basket is related to the numbers of apples and bananas).

Discounted Cash Flows (DCFs)

Would you rather have \$1000 today or \$1000 a year from now? The correct answer is to take the \$1000 today because you can invest it and earn some return on it. Following on from that, we can calculate how much a guaranteed \$1000 n years from now should be worth to us today, in other words, the present value of the \$1000:

\[PV = \frac{1000}{(1+i)^n}\]

Here, i is the “discount rate”, equal to the rate of return you could get on an asset of similar risk. In the above example, we would use the risk-free rate since I said that the \$1000 is guaranteed to be paid.

This simple formula is the heart of all intrinsic valuation methodologies, but it rapidly becomes more complicated when we deal with assets whose future cash flows are uncertain. In particular, there are many idiosyncrasies when it comes to valuing equities (i.e shares in companies).

Corporate earnings are very different to cash flows.
If we own an equity stake in a company, how much cash do we actually receive?
Earnings and cash flows are anything but static – management can deploy capital effectively to improve future growth or conversely can make investments that are value-destructive.

In the next section, we discuss how some of these issues are resolved, to value the S&P500.

Valuing the S&P500

The S&P500 index consists of the largest 500 companies (measured by market capitalisation) in the US. When people are talking about how “the markets” are performing, more often than not they are referring to the S&P.

The framework I am using to value the S&P500 was built by Aswath Damodaran, the “Dean of Valuation”. A brief overview is given below, but I would encourage you to read his blog post (or watch the accompanying YouTube video).

Remember, at its core, a DCF only needs two things: future cash flows and a discount rate. To estimate the future cash flows of the S&P, we estimate the earnings growth in the next five years, as well as what fraction of earnings is returned to shareholders in the form of dividends and buybacks. The discount rate depends on the equity risk premium and the risk-free rate of return. At the end of year 5, the index has some terminal value that is calculated by assuming that the S&P grows in line with the economy in perpetuity.

The cash flow in year $n$ (for $n \leq 5$) can be written in terms of the base-year earnings $E_0$, the growth rates $g_n$, and the cash return fractions $f_n$:

\[CF_n = f_n \cdot E_0(1+g_0)(1+g_1)...(1+g_{n-1})\]

The initial cash return fraction $f_0$ can be set to whatever number we want, but we will ensure that by year 5 they tend towards a stable long-term quantity.

The terminal value in year 5 depends on the perpetual growth rate $\tilde{g}$:

\[TV = \frac{CF_5 (1 + \tilde{g})}{(i - \tilde{g})}\]

Hence the present value of the S&P index is:

\[PV = \sum_{n=1}^5 \frac{CF_n}{(1+i)^n} + \frac{TV}{(1+i)^5}\]

The base-year earnings is a known quantity, and it is possible to find the consensus estimates of near-term growth rates $g_n$ online. Thus, plugging these numbers into the formula should give a value near the current price of the S&P500.

The effect of COVID-19

To model how COVID-19 affects the value of the index, we can think about how it changes the following variables:

Fiscal year (FY) 2020 earnings growth – this will definitely take a hit as consumers spend less.
Earnings growth in FY2021-2024 – this depends on the fraction of lost earnings that we think will be recouped in the subsequent years.
The fraction of cash returned – companies may become more conservative and retain more cash, lowering the cash return fraction.

Damodaran argues that any effects on the discount rate are minimal because the decrease in the risk-free rate is offset by the increase in equity risk premiums. Also, as we will later verify, the fraction of cash returned is nowhere near as important as the first two variables (because regardless of the initial fraction, by the end of year 5 it converges to a stable value). Thus, we can make the approximation that price depends on two variables: the drop in FY20 earnings and the fraction of earnings recouped.

Here we have a minor problem: the model requires us to specify the earnings growth in FY21-24 – while this is indeed related to the fraction of earnings recouped, converting between the two is not trivial. It is intuitively clear that if you think a high fraction of earnings will be recouped, subsequent growth will have to be greater to make up for the lost earnings. But to quantify this, you essentially have to project out the cash flows under the pre-coronavirus consensus growth estimates ($g_0’ = 5.52\%$ in FY20 and $g’ = 3.36\%$ in FY21-24), with $1-f$ of the FY20 earnings lost permanently. The resulting formula is as follows, where $g_0$ (no dash) denotes our estimate of the FY20 earnings growth (negative since we are expecting a drop):

\[\hat{g} = \left[\left(1+f \cdot \frac{g_0' - g_0}{1 + g_0}\right)\cdot (1+g')^4\right]^{1/4} -1\]

EDIT: thanks to Ryan Kho for pointing out that I had transcribed the formula incorrectly. The formula is not at all intuitive, so I have uploaded a spreadsheet to demonstrate where it comes from.

With this out of the way, we are now in a situation very similar to that of the fruit basket. We have two variables $f$ and $g_0$, that combine to form the price. Given the price alone, we do not have enough information to find either of $f$ and $g_0$, but we can certainly plot their relationship visually.

Python implementation

If you just want to see the results, feel free to skip over this section. In any case, there’s nothing particularly fancy going on – it’s just a python translation of the maths from above:

# Function to value the S&P given drop and growth.
def valuation(
    drop, growth,
    base_earnings = 163,
    f0 = 0.855,  # base cash return
    rf = 0.0134, # risk free rate
    erp = 0.0525,  # equity risk premium
    lt_growth = 0.0134,  # perpetual growth
    lt_roe = 0.15  # perpetual return on equity
):
    # Stable cash return fraction
    lt_f = 1 - lt_growth / lt_roe
    # Calculate discount factor 1/(1+i)
    discount_factor = 1/ (1 + rf + erp) 
    
    pv = 0
    earnings = base_earnings
    for n in range(1, 6):
        if n == 1:
            earnings *= (1 + drop)
        else:
            earnings *= (1 + growth)
        ret_frac = f0 + n * (lt_f - f0)/5
        cash_flow = earnings * ret_frac
        pv += cash_flow * discount_factor ** n

    tv = earnings * (1 + lt_growth) * lt_f / (rf + erp - lt_growth)
    pv_terminal = tv * discount_factor ** 5

    return pv + pv_terminal
    
# Function to convert a recoupment fraction to growth rate
def recoup_to_growth(f, drop, g0=0.0552, g=0.0336):
    # confusingly, I have relabelled g0' as g0.
    return ((1+f*(g0-drop)/(1+drop))*(1+g)**4)**0.25-1

# Function to value the S&P given drop and recoupment fraction.
def valuation_recoup(drop, frac, **kwargs):
    return valuation(drop, recoup_to_growth(frac, drop), **kwargs)

To generate the data used in the plots:

import numpy as np

n_res = 100  # resolution
recoup = np.linspace(0, 1, n_res)
drop = np.linspace(-0.5, 0, n_res)

X, Y = np.meshgrid(recoup, drop)
Z = np.zeros((n_res, n_res))

for i, x in enumerate(recoup):
    for j, y in enumerate(drop):
        Z[i, j] = valuation_recoup(y, x)

The plots themselves were just made using matplotlib and the mplot3d toolkit.

Results

Since we have two independent variables, we need three dimensions to visualise how they affect the price:

While this visualisation is pretty cool, it’s not that useful. We can sacrifice some of the visual appeal for utility by projecting it down onto two dimensions (i.e a contour plot):

This chart might look daunting, but I want to stress that it’s the same concept as the original straight-line we had for apples and bananas, except with more lines so that we can see the relationship for different prices (“total number of fruits in the basket”).

There are several ways you can use this chart. Most obviously, you can use it to convert your estimates of the FY20 earnings drop and the fraction recouped into a price. For example, if you think that these values are -40% and 60% respectively, reading off the graph shows that the corresponding value of the S&P is \$2550.

However, the point of this post is to see what the current price is implying. As of 16/4/20, the S&P index is trading at about \$2783. This number is pretty close to \$2750, so let’s consider that contour for now. Specifically, the contour describes all possible combinations of earnings-drop and fraction-recouped that result in a value of \$2750. If, for example, you thought that earnings would drop -33% (as Goldman Sachs recently suggested), then the price-implied recoupment fraction is something like 75%. If you think this value of 75% is optimistic (and that Goldman’s estimate is accurate), the logical trade would be to short the S&P index.

You might be disappointed by the fact that there are only two levers you can play with (I personally think “the fewer the better”), but it turns out that these really are the most important variables. What about the cash return fraction? In the above plots, I have set it at 85%, the 10y average (much lower than the 92% returned in 2019). Let’s say you strongly disagree with this, in one of two ways:

You think COVID-19 is going to be so terrifying that companies will reduce the return fraction to 2009-2010 levels of about 70%.
You think the Fed’s bazooka will encourage companies to revert to their 2019 cash return fraction of 92%.

The plot below shows how much these differing assumptions (red and green respectively) affect the price contours (hint: very little).

For example, taking a look at the new contours next to the original \$2750 curve, we see that these two major changes to the return fraction have only moved the price by about \$30 either way.

Conclusion

One of the underlying assumptions in this post is that markets are broadly efficient. Since the original work of Fama and French on the efficient market hypothesis (EMH) and the subsequent popularisation by books like A Random Walk Down Wall Street, there has been an increasing rift between the traditional academics, who happily apply the EMH to many problems and claim grand results, and the practitioners, who vehemently reject the EMH. But as Andrew Lo puts it in his book, Adaptive Markets, the truth is somewhere in between. The efficient market hypothesis is incorrect, but it is a very good first approximation. When market prices move, your first instinct should be that the price movement is justified by a change in expectations of future cash flows.

All this isn’t to say that markets can’t be wrong. But in fact, that is somewhat the point of this post. Between 19 Feb and 22 Mar, the S&P500 index dropped by 33%. Was this an overreaction? Or was it justified by the expectations of greatly diminished future earnings? With the above plot, you can quantitatively understand the “fair” value of the index given a certain combination of fundamentals. This post is clearly not offering a “magic formula” that tells you if something is overvalued or undervalued – rather, it is a first step in building the toolkit that allows you to even approach such questions.

Rebuilding PyPortfolioOpt: an open source adventure

2020-03-19T00:00:00+00:00

A few weeks ago, a user raised an issue on the GitHub repository for PyPortfolioOpt, my open-source portfolio optimisation software library. In this nontechnical post, I discuss why a seemingly innocuous error resulted in a ground-up rebuild of a large chunk of PyPortfolioOpt, and share some reflections on open-source in general.

The actual bug that was reported is not particularly important, but it is quite easy to explain. PyPortfolioOpt offers a piece of functionality to construct a portfolio with minimum volatility (i.e minimal risk) – aptly called min_volatility. The problem was that it wasn’t working. One of my users, going by the name edgetrader, had provided a sample dataset and some code, which he claimed was producing incorrect output. I hate to say it, but issues like these are often caused by the user not formatting the data correctly or using the wrong value for a parameter. However, edgetrader’s code looked fine, and running it on my machine did nothing but confirm the issue – min_volatility was failing to produce the minimum volatility portfolio (for a painfully simple dataset too!).

I had a very good guess of where the error was coming from (later confirmed by another GitHub user, gatorliu). It was as bad as it could possibly be. To use an automotive analogy, rather than being a minor issue with an indicator light or windscreen wiper, the problem here was coming from deep within the engine. In PyPortfolioOpt, this corresponds to the backend optimiser – the part of the package that does the mathematical legwork of actually crunching the numbers to optimise a portfolio.

What is a backend?

The terms “frontend” and “backend” are often thrown around within the software engineering space, so let’s clarify exactly what we mean. In this context, I think of PyPortfolioOpt as being split into two broad chunks: the “frontend” API (application-programmer interface), i.e how users interact with the software, and the “backend”, which is the part that actually solves the optimisation problem. In the car analogy, which we will continuously revisit, we might say that the frontend API is the steering wheel, gear stick, pedals, etc. (whatever the user is interacting with), while the backend is what goes on “under the hood” (engine, transmission, electronics).

Now, as mentioned before, the “engine” of PyPortfolioOpt is the optimiser. Generally, optimisation is a very difficult mathematical task; many PhD-holders are paid a lot of money to build software routines for optimising different functions under different scenarios. For an open-source project (especially one not directly related to numerical computing), the norm is to pick from a handful of open-source optimisers, which, to be absolutely clear, have also been developed by extremely talented people. The problem is that it is rather difficult to decide on which one. So I put it to you – you have three choices:

A general-purpose optimiser that can optimise for all different kinds of user objectives and constraints, with a very simple syntax.
An optimiser that is highly specialised for a certain type of optimisation problem (which includes financial portfolio optimisation), but is not very flexible to user constraints, and requires strong mathematical competence to interact with.
An optimiser that builds on top of choice #2, which provides a much nicer interface.

What would you do? Bear in mind that it’s only really feasible to choose one and that it will take a large amount of development effort so the cost of swapping is very high. One further point is that although in theory, we like to treat the frontend API and backend as completely separate, your choice of backend will inevitably impact what the frontend looks like.

Technical debt

In March 2018, when I first started building PyPortfolioOpt, the choice seemed like a no-brainer. I chose the first option, which corresponds to scipy. A flexible optimiser that can work with all sorts of user objectives? Sounds fantastic! However, even then I knew that I was making a tradeoff; as I said, optimisation is a tough problem, so while option #1 in principle works with many different kinds of optimisation problems, it is actually less likely to produce the true optimal solution (it might, for example, get stuck in a “good” but suboptimal solution).

Contrast this with #2, which is a package called cvxopt. cvxopt is an efficient solver for convex optimisation problems (the definition of ‘convex’ doesn’t matter here; all you need to know is that it is a narrow subset of all possible problems). Luckily, portfolio optimisation is a convex problem, at least with standard objectives and constraints, so it should produce much better solutions. With that in mind, why did I choose scipy? Here are a couple of reasons:

The syntax for cvxopt was much harder to understand and required a greater level of mathematical sophistication (in March 2018 I hadn’t started university so I was somewhat constrained in that regard). Here’s a quick example to illustrate the difference between syntaxes – note how it is quite clear what everything is doing in scipy, whereas for cvxopt you have to be quite familiar with the library and the mathematical formulation of the problem:

 # Option 1: scipy
 result = scipy.optimize.minimize(
     fun=volatility_objective,
     args=(cov_matrix,)
     x0=np.ndarray([1/n] * n),
     bounds=[(1, 0), * n]
     constraints=[{"type":"eq", "fun" lambda w: np.sum(w) - 1}]
 )
 weight = result["x"]

 # Option 2: cvxopt
 G = cvx.matrix(0.0, (n,n))
 h = cvx.matrix(0.0, (n,1))
 A = cvx.matrix(1.0, (1,n))
 b = cvx.matrix(1.0)
 result = cvx.qp(mu * cov_matrix, -ret, G, h, A, b)['x']

I didn’t think the performance different would matter very much.
I was lazy.

I think you’d agree that the first two reasons seem fair; the third merits more of a discussion.

Technical debt is a very important concept in software development, be it open-source or in business. It is the idea that by choosing “quick and easy” solutions that seem to meet your needs now, you incur a debt that will eventually need to be repaid once your product reaches a certain stage of growth. While it has negative connotations, it is not always a bad thing. In the same way that large companies might borrow money to invest in a new project, incurring technical debt can be a great way to get your software up and running. This is a fundamental tenet of the Lean Startup methodology, according to which the most common reason why startups (and open-source projects) fail is that they spend a lot of time architecting a perfect solution to a problem that nobody has. To that end, it is critical to get a “minimum viable product” out the door as soon as possible so you can start receiving user feedback and iterating. Perhaps this is my way of rationalising a lazy choice, but given my constraints, I think it was probably the right decision at the time.

I released the first version of PyPortfolioOpt in May 2018 and received a strong positive response from the community, with surprising organic growth. Before long, it had a couple of hundred GitHub stars and surpassed most of the existing portfolio optimisation libraries. I believe this success was almost entirely due to the strong emphasis I placed on having an intuitive frontend API design and very detailed documentation, making it very easy for people to get up and running with the library. I did implement some semi-novel mathematics, for example, adding certain constraint functions to produce more diversified portfolios and creating new risk models, but this wouldn’t have mattered if the API were difficult to use.

One thing I did not anticipate was an influx of professional users, who were applying PyPortfolioOpt to real portfolio allocation problems. I had assumed that most professionals would use advanced in-house optimisation suites, but it turns out that there are many discretionary fund managers who were looking for a way to start systematising their allocation process and PyPortfolioOpt happened to come in at the right level of abstraction. For example, a typical workflow in PyPortfolioOpt is:

S = risk_models.sample_covariance(prices)
rets = expected_returns.mean_historical_returns(prices)
ef = EfficientFrontier(rets, S, weight_bounds=(0, 0.3))
weights = ef.min_volatility()

This requires a rough knowledge of what is going on (which most investment professionals have) but abstracts away all of the mathematics.

However, from the more quantitatively-inclined users (for example on the algotrading subreddit), I receive repeated (constructive) criticism on one area in particular – the use of the scipy optimiser on the backend. Many people shared their negative experiences with scipy and suggested I migrate to cvxopt. I took note of this feedback, but remained unconvinced that there was a significant performance difference, and decided to focus on building new features rather than working on the optimisation backend.

The tipping point

The issue that edgetrader raised was ultimately a strong impetus to completely re-evaluate my decision regarding the choice of the backend optimiser. Here I was, confronted with an astoundingly simple example (there were no weird constraints or objectives) for which the existing scipy backend was failing miserably. I could no longer credibly make the case that the worse performance was outweighed by the cleaner syntax, particularly with the thought that my library was being used to allocate real money.

I decided that it was time to “grow up”, and move on from scipy. But where would I go? I still disliked the cvxopt syntax and didn’t feel it was suitably extensible to provide my users with enough options. Additionally, although I was willing to make a few breaking changes to the frontend API (i.e changes that aren’t backwards compatible), I wanted to minimise this when possible, particularly for the basic user.

You may notice that after initially describing the three different options for a backend optimiser, I associated option #1 with scipy and option #2 with cvxopt. What about the last option?

An optimiser that builds on top of choice #2 [cvxopt], which provides a much nicer interface.

I didn’t elaborate on it further because in March 2018 I didn’t think such a thing existed. However, I recently came across a library that got me truly excited – a viable alternative to cvxopt that provided a fantastic interface which I could weave into PyPortfolioOpt. The library is called cvxpy, a modelling language for convex optimisation problems courtesy of the clever folks at Stanford. There are two aspects of cvxpy that instantly sold me. Firstly, the syntax is wonderfully expressive:

w = cp.Variable(n)
risk = cp.quad_form(w, cov_matrix)
prob = cp.Problem(objective=cp.Minimize(risk),
                  constraints=[cp.sum(w) == 1, w >= 0])

Secondly, while cvxpy is restricted to convex optimisation problems, it has an incredible type-checker that can actually explicitly verify if a problem is convex and noisily fail if not. “Noisy failure” sounds terrible, but in fact, it is a gift for a developer – the reason being that if you know there is an error, you can programmatically fix it or make suggestions to the user. By contrast, the bug that edgetrader raised was a silent failure. The only way we found the bug was by manually checking the output.

The actual migration

Having decided to make the switch from scipy to cvxpy, it was time to rearchitect the project. People unfamiliar with the industry jargon sometimes find “software architecture” to be a curious phrase. In this context, it just means thinking at a high level about how all of the parts will interact with each other before actually coding it up – in the same way that architects design buildings before actually laying down the bricks.

I mentioned that I initially wanted to keep the changes entirely backwards compatible, so it would just be swapping out the engine while keeping the cockpit identical. However, when thinking about the architecture and reading cvxpy’s documentation, I realised that this would be incredibly limiting. One of my major goals with PyPortfolioOpt from day one was to have a highly modular design – that is, all the optimisation objectives are kept in one place, risk models in another, optimisers again elsewhere. This makes the codebase much simpler to understand, test, and build on, but more importantly, it allows for advanced users to swap in certain parts of the pipeline according to their own expertise. For example, if you are an investor who has a great model for predicting the expected returns of stocks, you should be able to plug that into the PyPortfolioOpt pipeline with all the other steps taken care of. The thoughtful design of cvxpy allowed PyPortfolioOpt to achieve the truly modular structure I always wanted. See if you can spot the lego bricks in the following example:

ef = EfficientFrontier(returns, cov_matrix, weight_bounds=(0, 0.3))
ef.add_objective(transaction_cost)
ef.add_constraint(lambda weights : weights[2] <= 0.05)
weights = ef.min_volatility()

Ultimately I felt that this modularity was well worth making a breaking change to PyPortfolioOpt. A major consideration was that cvxpy also provides a much better platform for future development effort. I suppose this was the foresight that I was lacking when I initially decided to use scipy on the backend.

All told, the actual migration went rather well. It’s always quite stressful to make breaking changes to the core part of a library because you have to have a very good idea of the interdependence between different parts of a project – even though I am the sole author, this is quite a task! However, two things in particular were an absolute blessing, and I would strongly advise other open-source developers to take note.

Firstly, documentation. The documentation is the “User Manual” which should detail how everything works. It may sound silly, but whenever a user asks me a question about PyPortfolioOpt, I normally don’t know the answer off the top of my head and end up relying heavily on the docs. “Self-documenting” is a popular myth; even in-line comments are not a suitable replacement for proper dedicated (separate) documentation. No user should have to dig through your source code to figure out how to use something.

Secondly, at the risk of saying “eat your vegetables”, having a comprehensive suite of automated tests made things so much easier. I wrote tests for PyPortfolioOpt mostly because it was “suggested best practice”, but they are truly an incredible resource when it comes to software development. Since each piece of functionality is independently tested, I no longer have to maintain an exact mental map of all of the details of the package. If I make a change in one place that happens to affect another function, you can be sure the tests will greet me with an angry red cross. An especially useful practice has been building new tests after every bug fix – for example, in PyPortfolioOpt v1.0.0 the tests include edgetrader’s bug, so we can both be sure it is well and truly resolved.

Conclusion

One thing that I want readers to take from this post is that software design, as with many things in life, is a game of tradeoffs.

You have to make a multitude of decisions (some small, some critical) about the frontend API, the set of features, backend design, etc. Doing this correctly results in a package that people actually enjoy using, leading to organic growth by referrals. Otherwise, you end up putting a lot of effort into something that nobody will use.

I realise that this post has made a lot of use of the ‘perpendicular pronoun’, whereas the truth is that I am indebted to the many contributors who have lightened my load by providing a steady flow of feedback, second opinions, feature suggestions, and code reviews. A special shoutout to Felipe Schneider, who often responds to issues much faster than I and has been indispensable when it comes to bug/feature triage and architecture suggestions.

PyPortfolioOpt is a living thing now, and we are well beyond the days when I can make whatever changes I like. PyPortfolioOpt is being used by real people to allocate real money, and to that end, I have a responsibility to develop responsibly, with a thought to both future progress as well as backwards compatibility. To my current users, I ask for your forgiveness. I can personally attest to the hassle of having to change your code because another library changed its interface. Please know that the change was made with performance and long-term usability in mind.

Having released version 1.0.0, I won’t be making any breaking changes for the foreseeable future. It is my hope that the recent bout of “creative destruction” has laid the groundwork for PyPortfolioOpt to continue a steady and sustainable growth trajectory, and maintain its position as the “go-to” python portfolio optimisation library.

Black-Litterman allocation in algorithmic trading

2020-01-04T00:00:00+00:00

In December 2019, I released a major update to PyPortfolioOpt, my python portfolio optimisation package. The most significant addition was an implementation of the Black-Litterman (BL) method. Although BL optimisation is commonly used as part of a pipeline to optimise a multiasset/equity portfolio, in this post I argue that BL is particularly well suited to the problem of optimally weighting signals in an algorithmic trading context.

Overview of the Black-Litterman model

The “standard” way of doing portfolio optimisation is to start with a group of assets from which you’d like to make a portfolio, construct a vector of expected returns (usually by taking the mean historical returns of a group), then feed this vector along with the covariance matrix into a mean-variance optimiser (MVO). This sounds great in theory, except that the resulting portfolios tend to perform shockingly badly in practice, worse than if you were to just equally-weight the assets. As it happens, the major cause of problems is the use of mean historical returns to estimate future returns (see this post for more). The Black-Litterman model attempts to improve on this by using a clever Bayesian scheme to construct the expected returns vector.

BL allocation revolves around the concept of a “view”. A view is just a forecast you have of the future returns of an asset. It is different to the expected returns vector required by Efficient frontier optimisation in two key ways:

You don’t need to specify views on all the assets. This is much more realistic, since you may only have an opinion on a handful of assets, while still wanting to construct a diverse portfolio.
BL lets you input view confidences, which translate predictably into the resulting allocation. For example, for a positive view, a higher confidence (ceteris parabus) will result in a greater percentage allocation to that stock (if this sounds trivially obvious, note that the same isn’t always true in efficient frontier optimisation due to instability).

Formally, BL is a Bayesian model which essentially provides a way of combining your views with some prior estimate of returns (“prior” has a particular meaning in the context of Bayesian statistics). Part of the insight of Black and Litterman’s 1991 paper was that the current market-capitalisations of stocks can be used to construct a reasonable prior estimate of returns. For example, if company A has double the market cap of company B (assuming they have the same volatility), it can be argued that company A has double the expected return of company B. Maybe this isn’t a great estimate, but it doesn’t have to be – it is just a prior, which we will later update with our sophisticated views. The Black-Litterman expected returns are then a Bayesian weighted average of the prior and your views, with a weighting based on your confidence. That is all I want to say about the theory behind BL – the interested reader should refer to PyPortfolioOpt’s documentation and the links therein (for the more adventurous, I have presented the mathematical derivation here). This post will be kept relatively non-technical, or at least not explicitly mathematical, though it is clearly best supplemented with an understanding of the mathematics.

Black-Litterman in algorithmic trading

Although BL is typically used to optimise multiasset or equity portfolios, during a quantitative research internship this past summer I became quite interested in the applications of Black-Litterman allocation to algo-trading. Having done a little bit more research, I found that BL provides a very natural way of optimally weighting trading signals. A signal, in the most general sense, is a trigger for some marketplace action, usually based on the output of some model. For example, we might have multiple time series that we have found to be predictive of future price movements, or a machine learning model that predicts the performance of all the stocks in a universe.

There is a subtle difference between these two examples. In the time series case, it is possible that we have multiple signals on the same asset. It could be the case that we have found many factors that all seem to be predictive (to different degrees) of the price of the GBPUSD exchange rate – some technical factors (e.g. a moving average indicator, an oscillator) as well as some “fundamental” ones (e.g. yield curves, consumer spending). By contrast, in the machine learning recommender case we have a single model that makes forecasts on many assets.

What is common across both of these scenarios is that we need a way of combining the individual signals into a resulting portfolio. As far as I can see, there are two obvious configurations in which we can use Black-Litterman to attack this problem, each one more suited to the respective scenario.

Method 1: treat the signals as assets

The first method, which is more suitable for signals based on time series, is quite simple – we model each signal as an asset with a certain expected return and volatility, then optimise a portfolio of these assets. In fact, this approach can be done without the Black-Litterman model, using standard mean-variance optimisation. We would just have to construct a covariance matrix of signal returns as well as an estimate for expected returns (most likely based on historical returns), before optimising some objective function. The covariance matrix is there to ensure that we consider how signals move together, cognisant that we don’t want to excessively allocate to signals that are moving the same way. That said, we have already seen that a pitfall of MVO is that it is highly sensitive to return estimates because it assumes 100% confidence. Black-Litterman doesn’t.

A fair criticism of BL is that it is usually difficult to quantify the confidence in views – formally, BL requires the confidence to be specified as the variance of expected returns (this is different to the volatility of returns). There are some solutions to this, for example, Idzorek (2003) provides a method for mapping percentage confidence estimates to the required form of the BL input, though this is still rather unwieldy in practice. However, this is much less of a problem when it comes to algorithmic trading because we normally collect a wealth of data during backtesting regarding various performance metrics. In many cases, we are explicitly able to provide a confidence estimate, leading to a rigorous input for the BL model.

One seemingly thorny issue is what we should use for a prior. The prior is meant to be the estimate of returns for each asset in the absence of any particular information. I would argue that the most appropriate prior for a time series signal is zero; in fact, this is perfectly acceptable since the BL formula still works with no prior. Some might say that this defeats the whole point of BL, but I would counter that in this case BL still provides a logical way of combining views that have different confidence levels.

Method 2: construct a portfolio on signal recommendations

Method 1 is best suited to constructing portfolios of signals where each signal pertains to a small basket of assets. For the second scenario we mentioned, in which we have a model that looks through an asset universe and identifies a subset that may outperform, the application of Black-Litterman is a lot more straightforward. We use the model’s estimate of returns along with its confidence (mapped to a variance) as inputs to the Black-Litterman formula and construct a portfolio.

For this method, there is more choice for the prior. You can either use the relevant market prior for that universe (e.g. use the S&P500 if your universe is S&P500 equities) or go with no prior. The interesting thing about this is that BL is a fantastic way to mix the predictions of different models. There is nothing stopping you from combining multiple views (each with their own confidences) on the same asset – model A might think AAPL will return 20%, while model B thinks it will only return 2%. Who should you trust? A naive meta-model might just take the mean, but the BL machinery allows us to take a Bayesian weighted average, which incorporates your estimates of confidence.

Conclusion

This post has shown how Black-Litterman allocation finds a natural application in algorithmic trading, since backtests give us information on both expected returns and the confidence in those estimates. One major downside with BL (which is also a downside for all kinds of mean-variance optimisation) is that it is naturally a single-period optimiser, meaning that it doesn’t automatically adjust to new information. You will need to decide on some period over which views will be aggregated and return vectors constructed, then use BL to make an allocation. At the end of this period, you will have to repeat the process (including your new data) and rebalance positions. This introduces another degree of freedom into your resulting algo-trading pipeline, which is never a good thing. However, making the switch from single-period optimisation to multi-period optimisation is a massive step up and requires significantly more theoretical machinery.

An asymmetric bet on interest rates

2019-12-05T00:00:00+00:00

In a classic scene of No Country For Old Men, Javier Bardem’s character ominously asks a shopkeeper: “what’s the most you ever lost on a coin toss?”. The shopkeeper says that he doesn’t know – this is probably quite a reasonable response given that for a fair coin, one has little reason to make a bet since your expected value (EV) is zero. Yet retail investors seem to make coin-toss bets all the time: they conclude that based on their analysis, a certain stock is a buy (i.e it has positive expected value). But they often forget to account for model risk – the risk that their analysis itself is faulty.

There is plenty of research to show that in actual fact, the buy/sell decisions of (retail) investors may as well be coin flips. Steve Cohen, the legendary hedge-fund manager, points out that his best portfolio managers get the direction of stock movement right only 56% of the time. However, the key to their success is the asymmetric payoff. When they are right, they ride the stock up to significant gains, but when they are wrong they are quick to cut losses. Let’s assume that whenever they are right, the stock goes up 30%, but if the stock is going down they will cut their losses at 5%. The EV is then calculated as follows:

\[\text{EV} = 0.56 \times 30\% + (1-0.56) \times (-5\%) \approx 17\%\]

Hence despite the “low accuracy”, the expected return on a single trade is an admirable 17%. We can use EV calculations to understand the profitability of all kinds of trading setups – some funds choose to make bets that have a less than 5% chance of being right, but will return 100x if they are. Other players, particularly market makers and option sellers, choose to make fractions of cents on every trade, but often have significant tail risk. In this post, I want to present an idea I had at the start of summer for an asymmetric bet on US interest rates.

Outline of the pitch

It was July 2019: the height of the trade war, Brexit uncertainties alive as ever, people wondering if the decade-long bull market was going to come to an end (having forgotten about December 2018) – all eyes were on the Federal Reserve and it seemed that every day financial news sources had a different analysis of what would happen to interest rates. I was reading the Financial Times when a statement popped up that really excited me:

the market is pricing in an 100% probability of a rate cut of 25bps or more

This statement provides an interesting opportunity to make a positive EV bet because of the asymmetry it presents. When the market is so confident that interest rates will be cut, you are getting excellent odds to make a bet on the opposite outcome (i.e rates held constant or increasing), even if you think that’s quite unlikely.

Concretely, my proposal was to make a hawkish bet, i.e betting that interest rates will either stay the same or go up. Because the market thinks that there is an “100% chance” of a cut, if the rates are indeed cut, then I wouldn’t pay much for being wrong. However, on the off chance I’m right, my payoff could be large.

In the rest of the post, we will discuss how to actually bet on interest rates, how the market-implied probability of a rate cut is calculated, and a post-mortem of my pitch.

The Federal Reserve and interest rates

Banks in the US are legally required to keep some cash in the Federal Reserve. Amounts in excess of this minimum reserve can be lent as an unsecured loan to other market participants; the interest rate on these loans is the Federal funds rate.

These loans are negotiated by the borrowers and lenders (who are usually financial institutions). Why, then, do people always talk about the Fed hiking/cutting rates? The answer is that although the Federal Open Market Committee (FOMC) doesn’t set interest rates, it does set a “target rate”. This is more than just words, because the Fed can conduct open-market actions to help move the overnight rate towards its target – this typically consists of either buying or selling government bonds. For example, when the Fed buys government bonds, there is more money in the system so people will pay less to borrow money (i.e interest rates go down).

Fed funds futures

A forward contract is an agreement between a buyer and a seller to transact in a particular quantity of some item, at a specified date and price in the future. For example, a soybean farmer might want to lock-in a sale price for their soy beans before the harvest and might thus agree to sell 5,000 bushels exactly 6 months from now (May 2020), at a price of 8.85 per bushel. Come May 2020, the farmer delivers the soybeans to the buyer and is paid $\$8.85 \times 5000 = \$44,250$.

A futures contract is essentially the same, except that the contracts are standardised, allowing them to be traded on an exchange. A good way of thinking about this is that there are pieces of paper going round saying “by holding this paper, I agree that on the 1st of May 2020, I will deliver 5000 of bushels of soybeans for the price of $8.85 per bushel” – these pieces of paper can then be bought and sold. Note that you don’t actually have to physically deliver the underlying asset (in most cases) – the usual practice is to settle in cash, for example by paying the deliveree the current market price of the asset.

There is a lot more that could be said about futures, both on the theory side (no-arbitrage pricing) to the practicalities (margin, roll, and covering positions), but the key takeaway is that a futures contract is a way of speculating on the price of some underlying asset. This is a useful way of thinking about it, particularly in the context of underlying assets that are not physical commodities and hence cannot be delivered.

In this vein, a Federal funds futures contract is a piece of paper saying that you will be paid an amount of money that depends on the Federal funds rate at a certain date in future. For example, let’s say that our simple conversion scheme is to take the Federal funds rate as a percentage, multiply it by \$100, and pay out that amount e.g. on the day that the contract expires, if the Fed funds rate is 2.1%, you will be paid \$210. How much would you pay for this piece of paper today? Clearly that depends on what you think is going to happen to interest rates. This is really the heart of a Fed funds futures contract – it’s a piece of paper whose value depends on the Federal funds rate at some date in the future.

There is one minor annoyance, however. For historical reasons, futures brokers prefer standardised scales, so the Fed funds futures trade with the “IMM price”, i.e the price index of the contract is given by:

\[\text{Futures price} = 100 - \text{interest rate}\]

Thus a value of 97.5 corresponds to a Fed funds rate of 2.5%. There is some additional nuance that I don’t particularly want to go into (for example, the interest rate used in the above calculation is actually the 30 day rolling mean of Fed funds rates), but any interested readers can have a look at the actual contract specification.

This notwithstanding, the essential point is that Fed funds futures give us a direct way to speculate on US interest rates. In the above example, if we think the Fed is going to cut rates from 2.5% to 2.25%, we would go long a Fed funds future if it were trading below 97.75, because after the rate cut, the contract would settle at 97.75 (more than what we paid for it). Likewise if we have a hawkish view, we would short the Fed funds future.

The market-implied probability of a rate cut

Clearly, if the current interest rates are 2.5% and the Fed funds futures are priced at 98 (corresponding to a 2% rate), we know that the market thinks a rate cut will happen. But how was financial news quantifying the market-implied probability of a rate cut?

As it happens, the CME group runs a wonderful web app called the CME FedWatch Tool, which provides a whole host of analytics regarding FOMC meetings – including the probability of a certain rate movement. Their methodology is detailed on the website, but a brief summary is as follows.

The first thing to note is that Fed Funds futures contracts can be bought with expiries on calendar months up to about 3 years from now. That sentence was terribly worded, but I really can’t think of a better phrasing, so I’m just going to use a picture:

Data from the CME group

This is incredibly useful because it allows us to isolate the effect of a given month’s FOMC meeting. For example, if there is a meeting on 31 July, we need only consider the contract expiring 1 July and the contract expiring 1 August to understand the market’s view of the meeting. Let’s go through a very simple example of a probability calculation to estimate the probability of a 25bps cut (using made-up data):

Fed funds future (exp 1 Jul) = 97.60
Fed funds future (exp 1 Aug) = 97.70

implied rate (1 Jul) = 100 - 97.60 = 2.40
implied rate (1 Aug) = 100 - 97.70 = 2.30

P(25bps cut) = (2.40 - 2.30) / 0.25 = 40%
P(no cut) = 1 - P(cut) = 60%

The only mysterious step is possibly the P(cut) step, but this can be easily reasoned out. If there is a 25bps rate cut, the Fed-funds-implied rate will go from 2.40 to 2.15 by expiry. But the market has priced this at 2.30 instead. How far “along” is 2.30 between 2.40 and 2.15? Ten twenty-fifths of the distance. This must be the market-implied probability.

Of course, there is added complexity in that the above analysis has only dealt with a very narrow question: what is the probability of a 25bps cut vs no cut. In reality, the Fed can leave rates the same, or cut/raise them by any multiple of 25bps. The full calculation is essentially the same as the above, just with a bit of mathematical trickery to adjust for the different branches.

Now that we understand how these probabilities are calculated, we can look at the data. This chart shows all of the market-implied probabilities leading up to the 31/7/2019 FOMC meeting. For context, I was pitching this idea at the start of July 2019 and the Fed’s target rate was 225-250bps:

Chart from the CME FedWatch Tool

We can see that by mid-June, the probability that rates would be held at 225-250bps went to zero (the nosedive of the purple line). This means that Fed funds futures were priced such that there was an 100% chance of a rate cut.

My estimate of the probability of a rate cut

Having seen what the market expects, we can now begin to think about some reasons that the market could be wrong. I think there are three plausible reasons:

In July 2019, the US economy was chugging along perfectly well. In June, nonfarm payrolls rose 224,000 (well above market expectations); the nonfarm payroll numbers are one of the many key job statistics that people keep an eye on because they give a great look into the health of the US economy.
Related to the above, a major component of the Fed’s mandate is to keep inflation at around 2%. When the economy is “heating up” too much (with high inflation), the Fed may want to raise rates to cool things down. Likewise, if the economy is cooling down, the Fed may want to lower the target rate to encourage people to take money out of their bank accounts and spend! Between January and June 2019, US inflation had steadily climbed from around 1.5% to 2.0%, in absence of any open-market action from the Fed. Thus it could be argued that there is no reason for them to cut rates in the July FOMC meeting.
This is a “softer point”, but one thing I have always found interesting is Jerome Powell’s levelheadedness in the face of Donald Trump, who has issued many scathing comments (over Twitter, no less) of the Fed for not being looser with monetary policy. It is quite possible that Powell would want to remain defiant (and assert the Fed’s independence) by not “giving in” to Trump’s demands.

Very classy.

I haven’t gone into too much detail for the above points because it doesn’t really matter. The fundamental argument in the post is that as long as there exists a nonzero probability of a rate-hold (or rate hike), we are able to make a positive EV bet. To elucidate this, let’s say that based on my analysis above I’m willing to say that there’s just a 2% chance of a rate cut.

Interest rate	Market-implied probability	My estimated probability
225-250 (hold)	0%	2%
200-225 (25bps cut)	97%	96%
175-200 (50bps cut)	3%	2%

Let’s assume that we are dealing with a contract that expires right after the FOMC meeting. Given the market-implied probabilities, we can estimate the price of the future’s contract let’s say the Fed funds futures contract (which expires right after the FOMC meeting) is trading at 97.883 – a number I calculated as below:

\[\text{market price} = 100 - (0.97 \times 212.5 \text{bps} + 0.03 \times 187.5 \text{bps}) = 97.883\]

Using my estimated odds (different from the market’s!), the fair price of the Fed funds future (ignoring pedantry regarding the time-value of money) is:

\[\text{fair price} = 100 - (0.02 \times 237.5\text{bps} + 0.96 \times 212.5 \text{bps} + 0.02 \times 187.5 \text{bps}) = 97.875\]

Hence my hawkish bet (shorting the futures contract) would result in a profit of about \$33 per contract (according to the contract spec, each contract settles for \$4167 times the price index). You might think that 0.08% profit sounds paltry, but it should be remembered that the beauty of futures is leverage. For this particular contract, you would only have to put up about \$400 of maintenance margin, so the actual percentage profit would be about 8%. This isn’t bad at all, considering the time horizon of this bet is a few weeks.

Conclusion

Unsurprisingly, the Fed decided to cut rates by 25bps, citing weak global growth and trade tensions. However, I don’t think this pitch was incorrect. The point is not that I thought rates would be held steady, but that I was willing to assign this outcome a higher probability than the market was.

I ended up pitching this idea, together with three other students, to a panel of managing directors at J.P. Morgan as part of an industry insight event. They were sceptical initially, but were quite intrigued once I had made it sufficiently clear that this was really a play on market expectations rather than interest rates directly. On a side note, I knew that a successful pitch here could result in a front-office summer internship offer at JPM so perhaps there was another dimension to this asymmetric bet…

If there’s anything you should take away from this post, it is this: I am certainly not claiming to know more than economists about what the Fed is going to do. What I am saying is that whenever the market is so sure that something is going to happen (or never going to happen!), there could be a good opportunity for a highly asymmetric bet.

How predictive is the historical volatility?

2019-11-01T00:00:00+00:00

One of the things that makes markets exciting (or frightening) is that prices move around a lot. It is important to be able to describe and predict the range of possible price movements over a given time horizon since some investors might desire assets whose prices don’t move up and down too much. We can quantify this by computing the volatility, which is commonly defined to be the standard deviation of the asset’s (log) returns. This post examines how well we can predict future volatility and why that matters.

Volatility is a rather important quantity because it is the default way for investors to quickly assess the amount of risk that a stock might add to the portfolio. It also shows up in many different parts of finance, from the traditional Capital Asset Pricing Model used in investment banking to option-pricing. One of the most tangible applications, however, is in portfolio optimisation, which attempts to constructs portfolios that meet certain investor objectives, for example, finding the that minimises the total volatility. The general procedure for doing this is to estimate the future returns and covariance of a group of stocks (normally basing these estimates on historical values) then running an optimisation procedure – I have written about this extensively in the documentation for my portfolio optimisation software library.

However, the key to this procedure is that we are assuming that the past is a reasonable predictor of the future. A large amount of research has shown that historical returns are a very poor predictor of future returns, but I haven’t seen many comments on whether historical volatility is a good predictor of future volatility. This post is a brief investigation into the matter.

Methodology and hypothesis

We are going to keep this post extremely simple and examine the predictability of the S&P500’s volatility. Firstly, we will collect adjusted closing prices for the S&P500 over a large timeframe. We will then form 1y, 3y, 5y, 10y, 20y rolling windows and compute the rolling volatility for each. These will then be correlated with the realised volatility.

The reason for this is that I am quite interested to know how much data is required to make a better estimate of future volatility. My hypothesis is that there will be some intermediate optimum – too little data and you can’t make a good estimate, but too much data dilutes recent datapoints with ancient ones (which may be less relevant).

Preparing data

We start by using pandas_datareader to acquire some free price data from Yahoo Finance. Remarkably, we can get daily OHLCV (open, high, low, close, volume) data all the way back to 1927. I am of the opinion that it is good practice to save whatever data you download to the disk, in case you need it later.

import pandas as pd
from pandas_datareader import data as web

sp500_df = web.DataReader("^GSPC", "yahoo", 
                          datetime.datetime(1950,1,1), 
                          datetime.datetime(2019,12,19))
sp500_df.to_csv("data/sp500_prices.csv")

We can then read the data back in and clean it up. In particular, for this exploration we only need the Adj Close column, which contains the daily close prices adjusted for dividends and stock splits.

sp500_df = pd.read_csv("data/sp500_prices.csv", parse_dates=True)
px = sp500_df.set_index("Date")["Adj Close"]

It is straightforward to compute returns thanks to the pct_change() methods, which calculates the percentage change between dataframe rows.

rets = px.pct_change().dropna()
rets.head()

Date
1950-01-04    0.011405
1950-01-05    0.004748
1950-01-06    0.002953
1950-01-09    0.005889
1950-01-10   -0.002927
Name: Adj Close, dtype: float64

Because the price data excludes weekends and holidays, a trading year is actually only 252 days. To calculate the rolling volatility for a given window, we can just rely on the magic of pandas:

import numpy as np

rolling_vol = rets.rolling(time_period * n_days).std() * np.sqrt(252)

The np.sqrt(252) arises because volatilities are most commonly expressed on annual terms, whereas the method calculates a daily volatility. Combining all of this into a loop so that we can easily repeat the calculation for multiple time periods:

n_days = 252  # trading days in a year
time_periods = [1, 3, 5, 10, 20]  # in years
stds = []
for t in time_periods:
    stds.append(rets.rolling(t * n_days).std())
std_df = pd.concat(stds, axis=1)
std_df.columns = [f"{t}y" for t in time_periods]
std_df *= np.sqrt(252)  # annualise

The result of this is a dataframe where each column represents a different window length.

Date		1y	        ...     20y			
2019-12-13	0.143483	...     0.188674
2019-12-16	0.142234	...     0.188642
2019-12-17	0.140569	...     0.188635
2019-12-18	0.140573	...     0.188622
2019-12-19	0.139656	...     0.188622

Having now computed our historical volatility estimates, we need to calculate the realised volatilities (this is what we are trying to predict). This can be done with a nice pandas trick, using the shift() method to “look into the future”:

# Compute realised annual volatilities
realised_vol = rets.rolling(n_days).std().shift(-n_days) * np.sqrt(252)
realised_vol = realised_vol.rename("realised_vol")
realised_vol.head()

Date
1950-01-04    0.148951
1950-01-05    0.148979
1950-01-06    0.149594
1950-01-09    0.150306
1950-01-10    0.150330
Name: realised_vol, dtype: float64

Having collected and processed the data, we are now ready to conduct an analysis.

Analysis

Firstly, let’s plot all of the rolling volatilities that we calculated.

The historical volatility of the S&P500 using different rolling windows

Clearly, the wider the rolling window, the more stable our estimate of volatility. This is reflected in the fact that the yellow line moves around much less than the blue line, which exhibits many spikes. On the subject of spikes, it is interesting to note that the spikes seem to correspond to market crises: 2008 MBS, 2002-03 DotCom, 1987 Black Monday, 1973-74 oil crisis, and the 1962 Kennedy Slide (along with many other smaller ones). I find it equally interesting that most of the volatility spikes are extremely short-lived, which suggests to me that markets don’t take too long to re-price assets following a major shift.

But that is almost a tangent – the main point of this post is to understand how the historical volatility correlates with future realised volatility. The correlation coefficients are as follows:

std_df.corrwith(realised_vol)

1y     0.429631
3y     0.286037
5y     0.176046
10y    0.179328
20y    0.058464

These results are quite shocking. Firstly, I find the overall trend to be surprising, as I expected either the 3y or 5y to strike the right balance between recency and stability. It is quite surprising that using only the last year’s volatility to estimate future volatility has the most predictive power, but there you go. Secondly, the actual values are quite low. To put it differently, even for the 1y data, historical volatility only explains 18% ($0.4296^2$) of the variance in the realised volatility.

However, the explained variance is often not as intuitive as you might expect. I believe that a more tangible number is the Mean Absolute Error (MAE), which tells us how far away our prediction is from the realised volatility, on average.

mae = np.abs(std_df.subtract(realised_vol, axis=0)).mean()

1y     0.043003
3y     0.048608
5y     0.050167
10y    0.050909
20y    0.049677

For the 1y data, historical volatility had a MAE of about 4.3%, which is quite large! However, looking through the data I saw that it was a few far-off predictions that caused such awful results. This is corroborated by a histogram of the realised volatilities, for which we can observe a fat right tail:

The distribution of realised volatilities for the SPY benchmark using data from 1950-2019

All of these findings notwithstanding, it is reassuring to see that the errors between historical and realised volatilities are almost centred around zero:

(std_df["1y"] - realised_vol).hist(bins=100)

The error of the 1y historical volatility predictions

Conclusion

In conclusion, we see that historical volatility is far from a perfect forecaster of future volatility, explaining less than 20% of the variance in realised volatilities in the best case (out of our experiments). It is not entirely surprising that this is the case, given that volatility forecasting is a very heavily-researched topic, since a better estimate of the volatility gives you a competitive advantage when it comes to pricing options or trading volatility instruments like the VIX. There are numerous sophisticated econometric models for predicting future volatility, not least the GARCH family, so it is naive to assume that simply using the historical volatility would give good results.

When it comes to the topic of portfolio optimisation, these findings should be a real wake-up call that you need to consider the uncertainties in both your return estimates and volatility/covariance estimates. It is true that the mean-variance machinery is not able to take this into account – all the more reason to swap to a confidence-aware Bayesian method such as Black-Litterman optimisation. However, even then, it is not straightforward to understand the potential effects of the skewed distribution of realised volatilities.

It should be noted that not everyone agrees that the volatility is a number worth paying attention to in the first place. A famous critic is Warren Buffet, who rightly points out that a good company whose stock price suddenly drops due to temporary factors (e.g. a media scandal) could be a bargain purchase – in which case it is actually less risky than it was before, although its volatility will be higher as a result of the drop. Perhaps this post lends some evidence to this point of view. Given the distribution of realised volatilities, it seems that the converse may also be true: even if everything looks calm and you predict low volatility, a right-tail event could make things go horribly wrong. As is always the case in finance, caveat emptor applies!

The full Jupyter notebook can be found on my GitHub

Implementing k-means clustering from scratch in C++

2019-10-02T00:00:00+00:00

I have a somewhat complicated history when it comes to C++. When I was 15 and teaching myself to code, I couldn’t decide between python and C++ and as a result tried to learn both at the same time. One of my first non-trivial projects was a C++ program to compute orbits – looking back on it now, I can see that what I was actually doing was a (horrifically inefficient) implementation of Euler’s method. I just couldn’t wrap my head around fixed-size arrays (not to mention pointers!). In any case, I soon realised that juggling C++ and python was untenable – not only was I new to the concepts (such as type systems and OOP), I was having to learn two sets of syntax in addition to two flavours of these concepts. I decided to commit to python and haven’t really looked back since.

Now, almost 6 years later (tempus fugit!), having completed the first-year computer science course at Cambridge, I feel like I am in a much better place to have a proper go at C++. My motivation is helped by the fact that all of the second-year computational practicals for physics are done in C++, not to mention that C++ is incredibly useful in quantitative finance (which I am deeply interested in).

To that end, I decided to jump straight in and implement a machine learning algorithm from scratch. I chose k-means because of its personal significance to me: when I was first learning about ML, k-means was one of the first algorithms that I fully grokked and I spent quite a while experimenting with different modifications and implementations in python. Also, given that the main focus of this post is to learn C++, it makes sense to use an algorithm I understand relatively well.

Please let me add the disclaimer that this is certainly not going to be an optimal solution – this post is very much a learning exercise for me and I’d be more than happy to receive constructive criticism. As always, all code for this project can be found on GitHub.

What is k-means clustering?
The k-means algorithm
Implementation
Testing
Conclusion

What is k-means clustering?

I have decided to give four brief explanations with increasing degrees of rigour. Nothing beyond the first explanation is really essential for the rest of this post, so feel free to stop whenever.

k-means clustering allows us to find groups of similar points within a dataset.
k-means clustering is the task of finding groups of points in a dataset such that the total variance within groups is minimised.
k-means clustering is the task of partitioning feature space into k subsets to minimise the within-cluster sum-of-square deviations (WCSS), which is the sum of quare euclidean distances between each datapoint and the centroid.
Formally, k-means clustering is the task of finding a partition $S = \{S_1, S_2, \ldots S_k\}$ where $S$ satisfies:

\[\arg \min \sum_{i=1}^k \sum_{x \in S_i} \lVert{x - \mu_i}\rVert^2\]

The k-means algorithm

The k-means clustering problem is actually incredibly difficult to solve. Let’s say we just have $N=120$ and $k=5$, i.e we have 120 datapoints which we want to group into 5 clusters. The number of possible partitions is more than the number of atoms in the universe ($5^{120} \approx 10^{83}$) – for each one, we then need to calculate the WCSS (read: variance) and choose the best partition.

Clearly, any kind of brute force solutions is intractable (to be specific, the problem has exponential complexity). Hence, we need to turn to approximate solutions. The most famous approximate algorithm is Lloyd’s algorithm, which is often confusingly called the “k-means algorithm”. In this post I will silence my inner pedant and interchangeably use the terms k-means algorithm and k-means clustering, but it should be remembered that they are slightly distinct. With that aside, Lloyd’s algorithm is incredibly simple:

1. Initialise the clusters

The algorithm needs to start somewhere, so we need to come up with a crude way of clustering points. To do this, we randomly select k points which become ‘markers’, then assign each datapoint to its nearest marker point. The result of this is k clusters. While this is a naive initialisation method, it does have some nice properties - more densely populated regions are more likely to contain centroids (which makes logical sense).

2. Compute the centroid of each cluster

Technically Lloyd’s algorithm computes the centroid of each partition of 3D space via integration, but we use the reasonable approximation of computing the centre of mass of the points in a given partition. The rational behind this is that the centroid of a cluster ‘characterises’ the cluster in some sense.

3. Assign each point to the nearest centroid and redefine the cluster

If a point currently in cluster 1 is actually closer to the centroid of cluster 2, surely it makes more sense for it to belong to cluster 2? This is exactly what we do, looping over all points and assigning them to clusters based on which centroid is the closest.

4. Repeat steps 2 and 3

We then repeatedly recompute centroids and reassign points to the nearest centroid. There is actually a very neat proof that this converges: essentially, there is only a finite (though massive) number of possible partitions, and each k-means update at least improves the WCSS. Hence the algorithm must converge.

Implementation

Our goal today is to implement a C++ version of the k-means algorithm that successfully clusters a two-dimensional subset of the famous mall customers dataset (available here). It should be noted that the k-means algorithm certainly works in more than two dimensions (the Euclidean distance metric easily generalises to higher dimensional space), but for the purposes of visualisation, this post will only implement k-means to cluster 2D data. A plot of the raw data is shown below:

By eye, it seems that there are five different clusters. The question is whether our k-means algorithm can successfully figure this out. We are actually going to cheat a little bit and tell the algorithm that there will be five clusters (i.e $k=5$). There are methods to avoid this, but they essentially involve testing different values of k and finding the best fit, so they don’t add much value to this post.

C++ preambles

Firstly, we need to define our imports and namespace.

#include <ctime>     // for a random seed
#include <fstream>   // for file-reading
#include <iostream>  // for file-reading
#include <sstream>   // for file-reading
#include <vector>

using namespace std;

In general, using namespace std is not considered best practice (particularly in larger projects) because it can lead to ambiguity (for example, if I define a function or variable called vector) and unexpected behaviour. However, the alternative is to have things like std::cout or vector::vector everywhere – for an educational post, the loss in clarity is worse than the potential ambiguity.

Representing a datapoint

To represent a datapoint for this program, we will be using a C++ struct. Structs caused me a great deal of confusion when I was learning about C++ because I couldn’t quite figure out how they differ from classes. As it happens, they are really quite similar – possibly the only relevant difference is that members of a struct are public by default. In any case, I would think of a struct as a way of defining a more complicated data type, though it is more than just a container for primitive datatypes because you can also define some functionality.

struct Point {
    double x, y;     // coordinates
    int cluster;     // no default cluster
    double minDist;  // default infinite dist to nearest cluster

    Point() : 
        x(0.0), 
        y(0.0),
        cluster(-1),
        minDist(__DBL_MAX__) {}
        
    Point(double x, double y) : 
        x(x), 
        y(y),
        cluster(-1),
        minDist(__DBL_MAX__) {}

    double distance(Point p) {
        return (p.x - x) * (p.x - x) + (p.y - y) * (p.y - y);
    }
};

The first few lines are self-explanatory: we define the coordinates of a point, as well as the cluster it belongs to and the distance to that cluster. Annoyingly, you can’t directly set the default value in the struct (e.g. double x = 0) – you need to do this via initialisation lists. Initially the point belongs to no cluster, so we arbitrarily set that to -1. Accordingly, we must set minDist to infinity (or the next best thing, __DBL_MAX__).

We also define a distance function, which computes the (square) euclidean distance between this point and another. Our Point struct can be used as follows:

// Define new point at the origin
Point p1 = Point(0.0, 0.0);
cout << p1.x << endl;  // print the x coordinate

// Define another point and compute square distance
Point p2 = Point(3.0, 4.0);
cout << p1.distance(p2) << endl;  // prints 25.0

If we wanted to represent a datapoint in p-dimensions, we could replace the x and y members with a vector or array of doubles, with each entry corresponding to a coordinate in a given dimension. The distance function would similarly need to be modified to loop over the vectors/arrays and sum all of the squared differences.

Reading in data from a file

Having decided how we are going to store datapoints within our C++ script, we must then read in the data from a CSV file. This is rather unexciting, but actually took me a long time to figure out. Essentially, we loop over all the lines in the CSV file and break them down based on the commas.

vector<Point> readcsv() {
    vector<Point> points;
    string line;
    ifstream file("mall_data.csv");

    while (getline(file, line)) {
        stringstream lineStream(line);
        string bit;
        double x, y;
        getline(lineStream, bit, ',');
        x = stof(bit);
        getline(lineStream, bit, '\n');
        y = stof(bit);

        points.push_back(Point(x, y));
    }
    return points;
}

Note that the readcsv function returns a vector of points. I decided to use a vector instead of an array because vectors handle all of the memory management for you (though are slightly less performant) and are functionally quite similar to python lists.

Pointers: an old enemy revisited

Suppose your friend wants to visit your house. You have two options (the relevance of this thought experiment will be clear shortly.)

Give them your postcode and let them find your house.
Hire a team of builders to replicate your house brick-for-brick right outside their front door.

The readcsv function returns a vector of points. One might assume that we can then just pass this to whatever k-means function we define and be done with it.

vector<Point> points = readcsv();  // read from file
kMeansClustering(points);          // pass values to function

However, we must be aware that depending on the size of our dataset, points might take up quite a large chunk of memory, so we must handle it carefully to be efficient. The problem with the above code is that we are passing the values of the points to the function, i.e we are making a copy of them. This is inefficient from a memory perspective. Luckily, C++ offers a way around this, called pass by reference. Essentially, instead of giving the value of the points vector to the function, we pass the location (read: postcode) of the points vector in memory.

vector<Point> points = readcsv(); 
kMeansClustering(&points); // pass address of points to function

The prototype of our kMeansClustering function is then as follows:

void kMeansClustering(vector<Point>* points, int epochs, int k);

Because we are now passing an address (and thus not technically a vector<Point>), we must include an asterisk. Read the first argument as “a reference to a vector of Point objects”.

I have also added two other arguments:

epochs is the number of iterations over which we will do our main k-means loop
k is the number of clusters.

Initialising the clusters

We first need to assign each point to a cluster. The easiest way of doing this is to randomly pick 5 “marker” points and give them labels 1-5 (or actually 0-4 since our arrays index from 0).

The code for this is quite simple. We will use another vector of points to store the centroids (markers), where the index of the centroid is its label. We then select a random point from the points vector we made earlier (from reading in the csv) and set that as a centroid.

vector<Point> centroids;
srand(time(0));  // need to set the random seed
for (int i = 0; i < k; ++i) {
    centroids.push_back(points->at(rand() % n));
}

One brief C++ note: because points is actually a pointer rather than a vector, in order to access an item at a certain index we can’t do points[i] – we have to first ‘dereference’ it by doing (*points)[i]. This is quite ugly, so fortunately we have the syntactic shortcut of: points->at[i].

Once the centroids have been initialised, we can begin the k-means algorithm iterations. We now turn to the “meat” of k-means: assigning points to a cluster and computing new centroids.

Assigning points to a cluster

The logic here is quite simple. We loop through every datapoint and assign it to its nearest centroid. Because there are k centroids, the result is a partition of the datapoints into k clusters.

In terms of the actual code, I had to spend some time thinking about the best way to represent that a point belonged to a certain cluster. In my python implementation (now many years old), I used a dictionary with cluster IDs as keys and a list of points as values. However, for this program I decided to use a quicker solution: I gave each point a cluster attribute which can hold an integer ID. We then set this ID to the index of the cluster that is closest to the point.

for (vector<Point>::iterator c = begin(centroids); 
     c != end(centroids); ++c) {
    // quick hack to get cluster index
    int clusterId = c - begin(centroids);

    for (vector<Point>::iterator it = points->begin();
         it != points->end(); ++it) {
             
        Point p = *it;
        double dist = c->distance(p);
        if (dist < p.minDist) {
            p.minDist = dist;
            p.cluster = clusterId;
        }
        *it = p;
    }
}

Computing new centroids

After our first iteration, the clusters are really quite crude – we’ve randomly selected 5 points then formed clusters based on the closest random point. There’s no reason why this should produce meaningful clusters and indeed it doesn’t. However, the heart of k-means is the update step, wherein we compute the centroids of the previous cluster and subsequently reassign points.

As previously stated, we are going to majorly simplify the problem by computing the centroid of the points within a cluster rather than the partition of space. Thus all we really have to do is compute the mean coordinates of all the points in a cluster.

To do this, I created two new vectors: on to keep track of the number of points in each cluster and the other to keep track of the sum of coordinates (then the average is just the latter divided by the former).

vector<int> nPoints;
vector<double> sumX, sumY;

// Initialise with zeroes
for (int j = 0; j < k; ++j) {
    nPoints.push_back(0);
    sumX.push_back(0.0);
    sumY.push_back(0.0);
}

We then iterate through all the points and increment the correct indices of the above vectors (based on the point’s cluster ID). Importantly, now is a convenient time to reset the minDist attribute of the point, so that the subsequent iteration works as intended.

// Iterate over points to append data to centroids
for (vector<Point>::iterator it = points->begin(); 
     it != points->end(); ++it) {
    int clusterId = it->cluster;
    nPoints[clusterId] += 1;
    sumX[clusterId] += it->x;
    sumY[clusterId] += it->y;

    it->minDist = __DBL_MAX__;  // reset distance
}

// Compute the new centroids
for (vector<Point>::iterator c = begin(centroids); 
     c != end(centroids); ++c) {
    int clusterId = c - begin(centroids);
    c->x = sumX[clusterId] / nPoints[clusterId];
    c->y = sumY[clusterId] / nPoints[clusterId];
}

Now that we have the new centroids, the k-means algorithm repeats. We recompute distances and reassign points to their nearest centroids. Then we can find the new centroids, recompute distances etc..

Writing to a file

One final detail: after all of our k-means iterations, we would like to be able to write the output to a file so that we can analyse the clustering. This is quite simple - we will just iterate through the points then print their coordinates and cluster IDs to a csv file.

ofstream myfile;
myfile.open("output.csv");
myfile << "x,y,c" << endl;

for (vector<Point>::iterator it = points->begin(); 
     it != points->end(); ++it) {
    myfile << it->x << "," << it->y << "," << it->cluster << endl;
}
myfile.close();

Testing

In order to test that my k-means implementation was working properly, I wrote a simple plotting script. I am somewhat embarrassed (in the context of a C++ post) to say that I wrote this in python.

import matplotlib.pyplot as plt
import pandas as pd
import seaborn as sns

# Before clustering
df = pd.read_csv("mall_data.csv", header=None)
df.columns = ["Annual income (k$)", "Spending Score (1-100)"]
sns.scatterplot(x=df["Annual income (k$)"], 
                y=df["Spending Score (1-100)"])
plt.title("Scatterplot of spending (y) vs income (x)")

# After clustering
plt.figure()
df = pd.read_csv("output.csv")
sns.scatterplot(x=df.x, y=df.y, 
                hue=df.c, 
                palette=sns.color_palette("hls", n_colors=5))
plt.xlabel("Annual income (k$)")
plt.ylabel("Spending Score (1-100)")
plt.title("Clustered: spending (y) vs income (x)")

plt.show()

The result is quite pretty and it shows that – bar a few contentious points around the centre cluster – the clustering has worked as expected.

Conclusion

In conclusion, we have successfully implemented a simple k-means algorithm in C++. Obviously there is much that could be improved about my program. Firstly, many simplifications were made, for example, we restricted the problem to two dimensions and also pre-set the number of clusters. However, there are more subtle issues that we neglected to discuss, including the random initialisation which may result in suboptimal clusters. In fact, there are algorithms like k-means++ that offer major improvements over k-means by specifying better procedures to find the initial clusters.

It is also worth mentioning the fundamental difficulties with k-means: it acutely suffers from the ‘curse of dimensionality’, as data becomes more sparse in high dimensions, and it is relatively inefficient since there are four loops (over iterations, points, clusters, and dimensions). However, k-means is often a great solution for quickly clustering small data and the algorithm is just about simple enough to explain to business stakeholders.

In any case, the merits/disadvantages of k-means aside, writing this program has given me a lot more confidence in C++ and I am keen to develop a more advanced understanding. I think it’s a good complement to my current interests in scientific/financial computing and it is pleasing to see that I am making more progress than I was a few years back.

What we learnt building an enterprise-blockchain startup

2019-09-01T00:00:00+00:00

It has been almost a year since the idea of HyperVault was first conceived. In that time, we built HyperVault up from a single sentence, gained and lost team members along the way, developed a functional proof-of-concept over the short winter holidays, crashed out of a few competitions (also won a couple of prizes), and finally decided to open source. This post aims to be an honest reflection on the journey – highlighting both the good bits and the times we wanted to give up.

What is HyperVault?

At a high level, the idea behind HyperVault is to use distributed ledger technology (we preferred to avoid the B-word) to provide a secure access layer to sensitive digital resources. If your company has some super-secret documents, what’s the best way of sharing them with other people? From our market research, we found that most people would just chuck it into an email as an attachment. This is terrible from a security standpoint: system administrators can see everything that goes through and you have absolutely no access record or control whatsoever once the email goes into your ‘sent’ mailbox. HyperVault provides a provably tamper-proof system with smart sharing features, such as scheduled or location-based access.

The Beginning: October-November 2018

HyperVault was born at a Hackbridge cluster – a fortnightly meet-up for Cambridge students who want to build side projects. Li Xi had been thinking about the applications of private blockchains in file sharing and met Robert and a group of other undergrads who were interested in joining the team.

By far the hardest part of our HyperVault journey was balancing our desire to build something with the Cambridge term structure. An 8-week term, packed with lectures and supervisions, doesn’t give you much time for entrepreneurial activities. However, we were excited enough about the idea to have weekly team meetings in a comfortable room at Trinity College, where we focused on defining what exactly we wanted the product to be.

At the same time, we were on a constant lookout for startup competitions that we could enter. Fortunately, Li Xi and Robert were also on the Cambridge Blockchain Society – one of whose sponsors (Stakezero Ventures) was launching the Future of Blockchain competition. This proved to be critical to HyperVault’s progress because it gave us a “fixed point” on the timeline – whatever happened, we needed to have a working product and polished pitch by March 2019, when the competition was scheduled to happen.

Winter, literally and figuratively.

December was tough. We had to transition from talking at a high level about what kind of features we wanted to offer, how cool it’d be if we could do XYZ, into actually building a working prototype. To ensure that everyone was on the same page regarding the expected time commitments, in the last couple of weeks of term, we made a detailed project plan that outlined the deliverables, due dates, and expected time commitments (we concluded that it’d be at least 12 hours a week each).

Three team members dropped out at this point. There weren’t any hard feelings – it is understandable for people to have conflicting priorities and it is better that they can say so upfront. This left Li Xi and Robert the considerable task of building a proof of concept, with Andrew taking the lead on the business plan. With the reduced manpower (and a good helping of classic underestimation), the weekly commitment was closer to 30 hours. It was a real hustle having to balance this with academics (and of course, winter festivities), but by the end of the holidays, we had our proof-of-concept. The main lesson to be drawn from this is the importance of having that “hard conversation” with your team. Give people the option to call it quits at the start, but have a “no ifs, no buts” understanding that come what may, your prototype/MVP must be done by a certain date.

The build-up to the competition (January – March)

At this stage, our proof-of-concept was something like a Google Drive or Dropbox, except with a blockchain backend. We were on track, so moved into “Phase 2” – doing real market research and nailing down a business plan. Andrew tapped into his network at the Judge Business School to get feedback from his peers, which was incredibly useful. One of them mentioned that our product offering was quite similar to something they had used in a previous job. This was very worrying, and after some further googling we were deeply troubled to find out that there was a whole class of products – virtual data rooms – that were doing exactly what we had offered. Robert and Li Xi were a little discouraged by this, but Andrew reminded us about a very important fact of entrepreneurship: you never have to be first, you just have to do things better.

Additionally, one of the oft-repeated pieces of advice from successful entrepreneurs is the need to focus on a very specific niche (a ‘vertical’) rather than trying to build a one-size-fits-all product. We saw that the incumbents had dominated the market for financial companies, so we decided to devote our attention to legal companies. With this in mind, we sat down as a team to really think through our business plan – we found the concept of a ”lean business canvas” enthralling – its almost scientific approach to iterating your product via hypothesis and experimentation really appealed to us (indeed, all three of us have a STEM background).

Competition time (March, April)

Coming up on March, our academic commitments were noticeably picking up. But there was no time to waste, as it was competition-season. We agreed as a team that for any competition, there would need to be at least two of us there (three was unrealistic).

The first competition we entered did not go so well – we failed to convince the judges, who traditionally favoured biotech ideas, that blockchain was anything more than a fad. Despite making it to the finals, it was clear that there was very little engagement with our idea. At the time, we blamed the judges for being too closed-minded, but in retrospect, we see that it’s really our responsibility in the pitch to sell the vision we have. Nevertheless, we used the feedback to refine our business plan and tidy up our proof-of-concept.

The main competition in March, however, was much more engaging and challenging (more than 100 teams participated). Due to a last-minute change of competition dates, Li Xi cut short his travel plans and flew back from Geneva for the Finals – luckily, he made it back just in time for the pitch with only a few hours to spare and the pitch went quite smoothly. The judges were intrigued by the idea and asked us some particularly challenging technical questions regarding encryption, which Li Xi dealt with deftly. Coming out of it, we won the £2000 Future of Blockchain Nucypher prize, which was a strong vote of confidence.

Q&A at the Future of Blockchain Finals

The last competition we attended was the R3 Global Pitch Competition. R3 is an enterprise blockchain software firm working with a large ecosystem of more than 300 of the worlds’ largest companies. Even before the submission of our deck, R3 provided excellent guidance by connecting us to their legal teams and having numerous calls with us to refine our business plan, for which we were immensely grateful. We did well in the competition, winning a place in the global finals and subsequently being offered free office space at R3’s office in London to continue building HyperVault.

The end?

With this opportunity in mind, we were talking to VCs about the idea of scaling up. In particular, we had good chemistry with the folks at Stakezero and Wilbe Ventures, and they gave us a lot of very thoughtful advice, emphasising the importance of not just identifying a vertical, but also targeting the size of the customer and the key stakeholders in a company that could make things happen. We hadn’t put too much thought into this before; I guess we were implicitly making the naïve assumption that if we built a cool product, it would sell itself. Up until that point, in our team meetings, we had just said that we’d target “SMEs” (small and medium-sized enterprises). But after a lot more research and discussion, we came to the troubling realisation that our ideal customer was, in fact, an “elephant” – an organisation that would spend more than $100,000 a year on our product (see this excellent short blog post for more) – they are the only ones who have 1) enough secure documents and 2) a need to share these documents with scalability.

Image from Christopher Janz's blog

Regardless of our successes in talking with smaller companies, big companies are an entirely different ball game. You need enterprise sales teams and a polished, audited product. In principle, there was nothing stopping us. We could have used R3’s office space to turn our proof of concept into a sleek MVP and earnestly begin the search for venture capital on the back of that. But ultimately, none of us felt particularly excited by this prospect. A startup is hard work – many hours spent fixing bugs, refining pitches, cold calling hundreds of companies, etc. The only thing keeping it together is the deep desire to build something and a fundamental belief that your product can “change the world”. Faced with the prospect of an enterprise sales cycle and a product which is really an improvement over current technology instead of something brand new (don’t get us wrong, we do still think it’s a major improvement), we realised that HyperVault as a commercial startup had run its course, at least until we had more experience.

What comes next

We truly believe that distributed ledgers applied to file-sharing could be a significant improvement over current technologies. To that end, we decided as a team that the best way forward would be to open-source our codebase, such that it becomes an example of a real-world enterprise blockchain app for others to build on.

We published most of our code on GitHub a few months ago and have already had someone reach out to us expressing interest in incorporating HyperVault into their product. We plan to continue cleaning up the codebase within the next couple of months, including comprehensive documentation and a contributors’ guide. Do leave a clap or comment below if you think this is something we should devote more time to!

In any case, rather than inserting some banal Edison quote about failure, we’d like to end this post simply by saying that we are extremely grateful for the friendships we made, the people we met, and the chance to tell ourselves that we built something from nothing.

This post has been cross-posted on medium

Graph algorithms and currency arbitrage, part 2

2019-04-21T00:00:00+00:00

In the previous post (which should definitely be read first!) we explored how graphs can be used to represent a currency market, and how we might use shortest-path algorithms to discover arbitrage opportunities. Today, we will apply this to real-world data. It should be noted that we are not attempting to build a functional arbitrage bot, but rather to explore how graphs could potentially be used to tackle the problem. Later on we’ll discuss why our methodology is unlikely to result in actionable arbitrage.

Rather than using fiat currencies as presented in the previous post, we will examine a market of cryptocurrencies because it is much easier to acquire crypto order book data. We’ll narrow down the problem further by making two more simplifications. Firstly, we will focus on arbitrage within a single exchange. That is, we’ll look to see if there are pathways between different coins on an exchange which leave us with more of a coin than we started with. Secondly, we will only be considering a single snapshot of data from the exchange. Obviously markets are highly dynamic, with thousands of new bids and asks coming in each second. A proper arbitrage system needs to constantly be scanning for opportunities, but that’s out of the scope of this post.

With all this in mind, the overall implementation strategy was as follows:

For a given exchange, acquire the list of pairs that will form the vertices.
For each of these pairs, download a snapshot of the bid/ask.
Process these values accordingly, assigning them to directed edges on the graph.
Using Bellman-Ford, find and return negative-weight cycles if they exist.
Calculate the arbitrage that these negative-weight cycles correspond to.

The full code for this project can be found in this GitHub repo.s

Star

Raw data

For the raw data, I decided to use the CryptoCompare API which has a load of free data compiled across multiple exchanges. To get started, you’ll need to register to get a free API key.

As mentioned previously, we will only look at data from Binance. I chose Binance not because it has a large selection of altcoins, but because most altcoins can trade directly with multiple pairs (e.g. BTC, ETH, USDT, BNB). Some exchanges have many altcoins but you can only buy them with BTC – this is not well suited for arbitrage.

Firstly, we need to find out which pairs Binance offers. This is done with a simple call (AUTH is your API key string):

import requests
import json 

def top_exchange_pairs():
    url = (
        "https://min-api.cryptocompare.com/data/v3/all/" + 
        "exchanges?topTier=true&api_key=" + AUTH
    )
    r = requests.get(url)
    with open("pairs_list.json", "w") as f:
        json.dump(r.json(), f)

This is an excerpt from the resulting JSON file – for each exchange, the pairs field lists all other coins that the key coin can be traded with:

"Data":{  
   "Binance":{  
      "isActive":true,
      "isTopTier":true,
      "pairs":{  
         "ETH":["PAX", "TUSD", "USDT", "USDC", "BTC"],
         "ONGAS":["BTC", "BNB", "USDT"],
         "PHX":["ETH","BNB","BTC"]
      }
   },
   "Coinbase":{  
      "isActive":true,
      "isTopTier":true,
      "pairs":{  
         "ETH":["DAI", "USD", "USDC", "EUR", "GBP", "BTC"],
         "BCH":["BTC", "GBP", "EUR", "USD"]
      }
   }
}

I then filtered out coins with fewer than three tradable pairs. These coins are unlikely to participate in arbitrage – we would rather have a graph that is more connected.

def binance_connected_pairs():
    with open("pairs_list.json", "r") as f:
        data = json.load(f)
    pairs = data["Data"]["Binance"]["pairs"]
    return {k: v for k, v in pairs.items() if len(v) > 3}

We are now ready to download a snapshot of the available exchange rates for each of these coins.

import os
import tqdm  # progress bar

def download_snapshot(pair_dict, outfolder):
    if not os.path.exists(outfolder):
        os.makedirs(outfolder)

    # Download data and write to files
    for p1, p2s in tqdm(pair_dict.items()):
        url = (
            "https://min-api.cryptocompare.com/data/" 
            + f"ob/l1/top?fsyms={p1}&tsyms={','.join(p2s)}"
            + "&e=Binance&api_key=" + AUTH
        )
        r = requests.get(url)
        with open(f"{outfolder}/{p1}_pairs_snapshot.json", "w") as f:
            json.dump(r.json(), f)

We can then run all of the above functions to produce a directory full of the exchange rate data for the listed pairs.

top_exchange_pairs()
connected = binance_connected_pairs()
download_snapshot(connected, "binance_data")

"EOS": {
    "BNB": {
        "BID": ".2073",
        "ASK": ".2077"
    },
    "BTC": {
        "BID": ".0007632",
        "ASK": ".0007633"
    },
    "ETH": {
        "BID": ".02594",
        "ASK": ".025964"
    },
    "USDT": {
        "BID": "7.0441",
        "ASK": "7.046"
    },
    "PAX": {
        "BID": "7.0535",
        "ASK": "7.07"
    }
}

This excerpt reveals something that we glossed over completely in the previous post. As anyone who has tried to exchange currency on holiday will know, there are actually two exchange rates for a given currency pair depending on whether you are buying or selling the currency. In trading, these two prices are called the bid (the current highest price someone will buy for) and the ask (the current lowest price someone will sell for). As it happens, this is very easy to deal with in the context of graphs.

Preparing the data

Having downloaded the raw data, we must now prepare it so that it can be put into a graph. This effectively means parsing it from the raw JSON and putting it into a pandas dataframe. We will arrange it in the dataframe such that it constitutes an adjacency matrix:

Column ETH row BTC is the bid:
- i.e someone will pay x BTC to buy my 1 ETH
- this is then the weight of the ETH $\to$ BTC edge.
Column BTC row ETH is the ask:
- i.e I have to pay y BTC to buy someone’s 1 ETH
- the reciprocal of this is the weight of the BTC $\to$ ETH edge.

I chose this particular row-column scheme because it results in intuitive indexing: df.X.Y is the amount of Y gained by selling 1 unit of X, and df.A.B * df.B.C * df.C.D is the total amount of D gained by trading 1 unit of A when trading via $A \to B \to C \to D$.

The column headers will be the same as the row headers, consisting of all the coins we are considering. The function that creates the adjacency matrix is shown here:

def create_adj_matrix(pair_dict, folder, outfile="snapshot.csv"):
    # Union of 'from' and 'to' pairs
    flatten = lambda l: [item for sublist in l for item in sublist]
    keys, vals = pair_dict.items()
    all_pairs = list(set(keys).union(flatten(values)))

    # Create empty df
    df = pd.DataFrame(columns=all_pairs, index=all_pairs)

    for p1 in pair_dict.keys():
        with open(f"{folder}/{p1}_pairs_snapshot.json", "r") as f:
            res = json.load(f)
        quotes = res["Data"]["RAW"][p1]
        for p2 in quotes:
            try:
                df[p1][p2] = float(quotes[p2]["BID"])
                df[p2][p1] = 1 / float(quotes[p2]["ASK"])
            except KeyError:
                print(f"Error for {p1}/{p2}")
                continue
    df.to_csv(outfile)

Putting the data into a graph

We will be using the NetworkX package, an intuitive yet extremely well documented library for dealing with all things graph-related in python.

In particular, we will be using nx.DiGraph, which is just a (weighted) directed graph. I was initially concerned that it’d be difficult to get the data in: python libraries often adopt their own weird conventions and you have to modify your data so that is in the correct format. This was not really the case with NetworkX, it turns out that we already did most of the hard work when we put the data into our pandas adjacency matrix.

Firstly, we take negative logs as discussed in the previous post. Secondly, in our dataframe we currently have NaN whenever there is no edge between two vertices. To make a valid nx.DiGraph, we need to set these to zero. Lastly, we transpose the dataframe because NetworkX uses a different row/column convention. We then pass this processed dataframe into the nx.Digraph constructor. Summarised in one line:

g = nx.DiGraph(-np.log(df).fillna(0).T)

Bellman-Ford

To implement Bellman-Ford, we make use of the funky defaultdict data structure. As the name suggests, it works exactly like a python dict, except that if you query a key that is not present you get a certain default value back. The first part of our implementation is quite standard, as we are just doing the $n - 1$ edge-relaxations where n is the number of vertices.

But because the ‘classic’ Bellman-Ford does not actually return negative-weight cycles, the second part of our implementation is a bit more complicated. The key idea is that if after $n-1$ relaxations, there is an edge that can be relaxed further then that edge must be on a negative weight cycle. So to find this cycle we walk back along the predecessors until a cycle is detected, then return the cyclic portion of that walk. In order to prevent subsequent redundancy, we mark these vertices as ‘seen’ via another defaultdict. This procedure adds a linear cost to Bellman-Ford since we have to iterate over all the edges, but the asymptotic complexity overall remains $O(VE)$.

def bellman_ford_return_cycle(g, s):
    n = len(g.nodes())
    d = defaultdict(lambda: math.inf)  # distances dict
    p = defaultdict(lambda: -1)  # predecessor dict
    d[s] = 0

    for _ in range(n - 1):
        for u, v in g.edges():
            # Bellman-Ford relaxation
            weight = g[u][v]["weight"]
            if d[u] + weight < d[v]:
                d[v] = d[u] + weight
                p[v] = u  # update pred

        # Find cycles if they exist
        all_cycles = []
        seen = defaultdict(lambda: False)

        for u, v in g.edges():
            if seen[v]:
                continue
            # If we can relax further there must be a neg-weight cycle
            weight = g[u][v]["weight"]
            if d[u] + weight < d[v]:
                cycle = []
                x = v
                while True:
                    # Walk back along preds until a cycle is found
                    seen[x] = True
                    cycle.append(x)
                    x = p[x]
                    if x == v or x in cycle:
                        break
                # Slice to get the cyclic portion
                idx = cycle.index(x)
                cycle.append(x)
                all_cycles.append(cycle[idx:][::-1])
        return all_cycles

As a reminder, this function returns all negative-weight cycles reachable from a given source vertex (returning the empty list if there are none). To find all negative-weight cycles, we can simply call the above procedure on every vertex then eliminate duplicates.

def all_negative_cycles(g):
    all_paths = []
    for v in g.nodes():
        all_paths.append(bellman_ford_negative_cycles(g, v))
    flattened = [item for sublist in all_paths for item in sublist]
    return [list(i) for i in set(tuple(j) for j in flattened)]

Tying it all together

The last thing we need is a function that calculates the value of an arbitrage opportunity given a negative-weight cycle on a graph. This is easy to implement: we just find the total weight along the path then exponentiate the negative total (because our weights are the negative log of the exchange rates).

def calculate_arb(cycle, g, verbose=True):
    total = 0
    for (p1, p2) in zip(cycle, cycle[1:]):
        total += g[p1][p2]["weight"]
    arb = np.exp(-total) - 1
    if verbose:
        print("Path:", cycle)
        print(f"{arb*100:.2g}%\n")
    return arb


def find_arbitrage(filename="snapshot.csv"):
    df = pd.read_csv(filename, header=0, index_col=0)
    g = nx.DiGraph(-np.log(df).fillna(0).T)

    if nx.negative_edge_cycle(g):
        print("ARBITRAGE FOUND\n" + "=" * 15 + "\n")
        for p in all_negative_cycles(g):
            calculate_arb(p, g)
    else:
        print("No arbitrage opportunities")

Running this function gives the following output:

ARBITRAGE FOUND
===============

Path: ['USDT', 'BAT', 'BTC', 'BNB', 'ZEC', 'USDT']
0.087%

Path: ['BTC', 'XRP', 'USDT', 'BAT', 'BTC']
0.05%

Path: ['BTC', 'BNB', 'ZEC', 'USDT', 'BAT', 'BTC']
0.087%

Path: ['BNB', 'ZEC', 'USDT', 'BAT', 'BTC', 'BNB']
0.087%

Path: ['USDT', 'BAT', 'BTC', 'XRP', 'USDT']
0.05%

0.09% is not exactly a huge amount of money, but it is still risk-free profit, right?

Why wouldn’t this work?

Notice that we haven’t mentioned exchange fees at any point. In fact, Binance charges a standard 0.1% commission on every trade. It is easy to modify our code to incorporate this: we just multiply each exchange rate by 0.999. But we don’t need to compute anything to see that we would certainly be losing much more money than gained from the arbitrage.

Secondly, it is likely that this whole analysis is flawed because of the way the data was collected. The function download_snapshot makes a request for each coin in sequence, taking a few seconds in total. But in these few seconds, prices may move – so really the above “arbitrage” may just be a result of our algorithm selecting some of the price movements. This could be fixed by using timestamps provided by the exchange to ensure that we are looking at the order book for each pair at the exact same moment in time.

Thirdly, we have assumed that you can trade an infinite quantity of the bid and ask. An order consists of a price and a quantity, so we will only be able to fill a limited quantity at the ask price. Thus in practice we would have to look at the top few levels of the order book and consider how much of it we’d eat into.

It is not difficult to extend our methodology to arb between different exchanges. We would just need to aggregate the top of the order book from each exchange, then put the best bid/ask onto the respective edges. Of course, to run this strategy live would require us to manage our inventory not just on a currency level but per currency per exchange, and factors like the congestion of the bitcoin network would come into play.

Lastly, this analysis has only been for a single snapshot. A proper arbitrage bot would have to constantly look for opportunities simultaneously across multiple order books. I think this could be done by having a websocket stream which keeps the graph updated with the latest quotes, and using a more advanced method for finding negative-weight cycles that does not involve recomputing the shortest paths via Bellman-Ford.

Conclusion

All this begs the question: why is it so hard to find arbitrage? The simple answer is that other people are doing it smarter, better, and (more importantly) faster. With highly optimised algorithms (probably implemented in C++), ‘virtual colocation’ of servers, and proper networking software/hardware, professional market makers are able to exploit these simple arbitrage opportunities extremely rapidly.

In any case, the point of this post was not to develop a functional arbitrage bot but rather to demonstrate the power of graph algorithms in a non-standard use case. Hope you found it as interesting as I did!

Graph algorithms and currency arbitrage, part 1

2019-03-02T00:00:00+00:00

Arbitrage is the holy grail for traders and the bedrock of financial academia. Let’s say you are in an open marketplace with Alice selling oranges for \$1 each and Bob buying them for \$2. As a cunning trader, you realise you can buy an orange from Alice and immediately sell it to Bob for \$1 of “risk-free” profit. However, as you keep reaping this \$1 profit by buying up Alice’s oranges, she raises her prices. Eventually, Alice would be charging \$2 per orange and the arbitrage would be shut. In other words, acting on arbitrage opportunities causes the market to become more efficient, with everything as close to its fair price as possible. Because of this, the only way to profit from true arbitrage is to find and execute opportunities faster than everybody else.

Currency exchange is a natural place to search for arbitrage because there are many different pairs that we can trade. For example, assume we are able to convert GBP to USD, then USD to EUR, then EUR to AUD for an effective rate of 1:2 GBP:AUD. If we were then able to directly swap AUD back to GBP in the ratio 2:1.1, this would be an arbitrage opportunity because we end up with more GBP than we started with while taking zero risk.

Our goal is to develop a systematic method for detecting arbitrage opportunities by framing the problem in the language of graphs. This is a reasonably large topic, so we’ll split it into two parts. Part 1 (this post) will lay the theoretical groundwork, introducing graph algorithms and giving an overview of their application to currency arbitrage. In Part 2 we will present an actual implementation of these ideas in python, applied to cryptocurrencies.

Graphs

Formally, we define a graph as a set of vertices (also called nodes) and edges. These edges can be directed (i.e have little arrows on them) and may have weights. It is completely up to us to decide what the vertices, edges, and weights represent. For example, if we are trying to model a road network we might say that the vertices are cities (or junctions), the edges are the roads themselves, and the weights are the length of the roads. The diagram below shows how a weighted directed graph is typically depicted.

It is useful to formulate real-world problems in terms of graphs because we have about three centuries worth of relevant theory – they were first investigated by Euler in 1736. In particular, there exist many efficient algorithms related to finding the shortest path along a graph, which have widespread applications e.g. in mapping.

The Bellman-Ford algorithm finds the minimum weight path from a single source vertex to all other vertices on a weighted directed graph. For the purposes of this post, we don’t need to know anything other than the fact that once the algorithm terminates, each vertex is labelled with the total weight of the minimum weight path from the source to that vertex.

Actually, the above statement has an incredibly important caveat. What happens if we try to find the minimum weight path from vertices C to D in the above graph? The obvious guess would be that it is the direct path along the CD edge, which has total weight 2.1. However, notice that we can in fact reduce this further by following the path CDECD, which would have a total weight of -1. But why stop there? If we go around the loop again, we can reduce it stil further. In fact, each time we traverse the loop, our minimum total weight will reduce by 1. Hence this is called a negative-weight cycle, the existence of which means that the shortest path between C and D is not defined.

This is actually the main reason why we are choosing to use Bellman-Ford over another shortest path algorithm like Dijkstra, despite the latter being asymptotically more efficient. Bellman-Ford is able to detect if there is a negative-weight cycle and as will be seen shortly, this will be the key to detecting arbitrage opportunities.

Graphs for currency arbitrage

How can graphs be used to model currency markets? At a very high level, we will assign currencies to different vertices, and let the edge weight represent the exchange rate. A simple example is presented below: in this market, we can convert 1 GBP to 1.27 USD, 1 USD to 1.43 AUD, etc.

To find the exchange rate between two currencies that do not share an edge, we simply find a path between the two currency vertices and walk along it, multiplying by each successive edge weight (exchange rate). If there are multiple possible paths, choose the path that has the greatest weight product, which corresponds to the most favourable rate.

What would happen if we start with 1 GBP and convert it along GBP $\to$ USD $\to$ AUD $\to$ GBP? Well, multiplying out the edges we see that our initial 1 GBP becomes $1.27 \times 1.43 \times 0.55 \approx 0.999$ GBP. That is to say, we lose a tenth of a penny on the round trip. However, if this number were greater than 1, it would mean that by following this path we end up with more GBP than we started with – this is textbook arbitrage.

So the problem of finding arbitrage can thus be formally posed as follows: find a cycle in the graph such that multiplying all the edge weights along that cycle results in a value greater than 1. In fact we have already described an algorithm that can find such a path – the problem is equivalent to finding a negative-weight cycle, provided we do some preprocessing on the edges.

Firstly, we note that Bellman-Ford computes the path weight by adding the individual edge weights. To make this work for exchange rates, which are multiplicative, an elegant fix is to first take the logs of all the edge weights. Thus when we sum edge weights along a path we are actually multiplying exchange rates – we can recover the multiplied quantity by exponentiating the sum. Secondly, Bellman-Ford attempts to find minimum weight paths and negative edge cycles, whereas our arbitrage problem is about maximising the amount of currency received. Thus as a simple modification, we must also make our log weights negative.

With these two tricks in hand, we are able to apply Bellman-Ford. The minimal weight between two currency vertices corresponds to the optimal exchange rate, the value of which can be found by by exponentiating the negative sum of weights along the path. As a corollary, we have the wonderful insight that: a negative-weight cycle on the negative-log graph corresponds to an arbitrage opportunity.

Conclusion

We have seen that graphs provide an elegant framework for thinking about currency exchange. Concretely, by representing different currencies as vertices and using the negative log of the exchange rate as the edge weight we can apply shortest-path algorithms to find negative-weight cycles if they exist, which correspond to arbitrage opportunities. In the next post, we code up the methodology presented in this post and apply it to some real-world data. Should be fun!

Portfolio optimisation: lessons learnt

2018-09-27T00:00:00+00:00

Over the past few months I have been busy doing a mixture of blockchain consulting and quantitative finance research. In particular, I have had the opportunity to investigate the interesting problem of portfolio management for cryptoassets – it was not my first experience with portfolio optimisation, having implemented efficient frontier portfolios at a roboadvisor startup, but this time I took the opportunity to do a deep dive into the subject.

I’m nowhere close to being an expert on the subject of portfolio optimisation – I’m just recounting some of the things I’ve learnt along the way.

1. Be careful with in-sample vs out-of- testing

For most people, ‘portfolio optimisation’ basically refers to mean-variance optimisation (MVO) in the style of Markowitz. MVO will tell you the mathematically optimal portfolio allocation, which sounds great, but the caveat is that it requires the (future) expected returns and the covariance matrix as inputs. For those of us who aren’t clairvoyant, these quantities are inaccessible, so we must make do with estimates. The quality of these estimates will be discussed in the next section, but for now I would like to highlight a common mistake, which is backtesting with a forward-looking bias.

The standard way (in textbooks or pedagogical materials) to estimate the expected returns is to take the mean annual return over the past few years. However, it is surprisingly easy to accidentally let future data worm its way into the estimate, resulting in vastly overinflated performance. One of the early mistakes I made was to optimise an equity portfolio using data from 2006-2010, then testing it from 2006-2015. The problem here is the overlap – the 2006 test has had access to data up to 2010, so performance in the 2006-2010 portion of the backtest will be very good.

To be fair, this is often quite an obvious mistake to diagnose, because your portfolios will outperform the benchmark by a ridiculous margin and you will realise that something is wrong. But there are also more subtle ways that you can include future data (e.g. survivorship bias), in which case it may not be so clear.

2. Be extra careful about expected returns

Despite their strong theoretical guarantees, efficient frontier portfolios often have poor real-life performance owing to estimation errors in the inputs.

Anyone who has ever tried to pick stocks will know how absurd it is to expect that a stock’s mean return over the past few years will be a good indicator of its future returns. Such simple relationships have almost certainly been arbitraged away – I therefore contend that mean historical returns are almost pure noise. The problem with this is that MVO has no way of distinguishing noise and signal, so often what happens is that the optimiser highlights the noise in the input. This is why there is a ‘running joke’ in the portfolio optimisation literature that a mean-variance optimiser is really just an “error maximiser”¹.

In practice, I have found that standard MVO can perform at least in line with the benchmark for equities, but for other asset classes (particularly those which are primarily driven by speculation, like cryptocurrency), the mean historical return is a poor estimator of future return (at least for medium-long term time horizons). However, in the next section we will see a simple way round this.

3. You can do better than 1/N

A significant body of research suggests that, in light of the aforementioned failures of MVO, we should instead use 1/N portfolios² – it is noted that they often beat efficient frontier optimisation significantly in out-of-sample testing.

However, an interesting paper by Kritzman et al. (2010) finds that it is not the case that there is anything special about 1/N diversification: it is just that the expected returns are an incredibly poor estimator and any optimisation scheme which relies on them will likely go astray.

The easiest way to avoid this problem is to not provide the expected returns to the optimiser, and just optimise on the sample covariance matrix instead. Effectively we are saying that although previous returns won’t predict future returns, previous risks might predict future risks. This is intuitively a lot more reasonable – the sample covariance matrix really seems like it should contain a lot of information. Empirical results support this, showing that minimum variance portfolios outperform both standard MVO and 1/N diversification.

In my own work I have found that the standard minimum variance portfolio is a very good starting point, from which you can try a lot of new things:

Shrinkage estimators on the covariance matrix
Exponential weighting
Different historical windows
Additional cost terms in the objective function (e.g. small-weights penalty)

4. Don’t ignore rebalance costs

When it comes to investment strategy or algorithmic trading, most people leave the “realities” like commission, slippage, and latency to the last stage. However, because of the general high turnover nature of efficient frontier portfolios, this could be quite a dangerous mistake.

In the context of portfolio management, the two main realities that must be accounted for in all backtests are commission and slippage. Latency is not very important because of the longer time horizons involved. Commissions are not hard to analyse: just multiply your broker’s percentage commission by the turnover. Slippage, on the other, hand is a little bit more subtle (though it is only really an issue in less liquid markets). There are a number of variables that affect slippage, but the important ones are transaction volume and urgency (refer to models such as that of Almgren for more).

Specifically, in the case of cryptoasset portfolios, one major problem is the difference in liquidity between bitcoin and a small altcoin like OmiseGo. A portfolio that shifts positions too much among the altcoins may look like it performs well in backtests, but when traded in real life the 0.5% slippage will be a killer.

Once all transaction costs have been included, you may find that it can be difficult to outperform a buy-and-hold market-cap weighted benchmark. Incidentally, this is another failure with 1/N portfolios: for reasonably volatile assets, there is high turnover at each rebalance and thus a large transaction cost.

5. Overfitting

One thing that I respect a lot about machine learning educators is that they are fundamentally honest about one of the major issues in the subject: overfitting. Essentially, it is possible to encourage most classifiers to wiggle their decision boundary to accommodate all of the training examples at the cost of generalisation ability. This is done by adjusting the hyperparameters (which do things like specify learning rates) and seeing which ones perform better. Hyperparameter tuning is not a bad thing, but it must be done responsibly lest we simply choose the best parameters that fit the particular train/test setup.

When it comes to standard portfolio optimisation, one doesn’t really have to worry about this because there aren’t any hyperparameters. However, once you start adding different terms to the cost function (each scaled by some parameter), things get a little bit more complicated. For example, let us consider the simple example of minimum variance optimsiation with an L2 regularisation term.

\[\underset{w}{\text{minimise}} ~ \left\{w^T \Sigma w + \gamma w^T w \right\}\]

The performance of this portfolio varies greatly depending on the choice of $\gamma$. A natural instinct is to run the optimisation for 10 different values of $\gamma$, but if this is not done carefully, we are probably fitting to the noise within the train and test sets.

Remember that ceteris paribus, a simpler model should be preferred. Black-Litterman, or optimisation with an exotic objective with multiple parameters may be seem to improve performacne on the backtest, but the proof will always be in the out-of-sample pudding. Approach portfolio optimisation like you would any financial machine learning task: with a healthy dose of skepticism.

Conclusion

I still have a lot to learn about portfolio optimisation, particularly with respect to optimisation of higher moments (skew and kurtosis) and things like copula, but the lessons highlighted in this post definitely still apply. If you found this interesting, check out PyPortfolioOpt on GitHub.

References

Michaud, R. (1989). “The Markowitz Optimization Enigma: Is Optimization Optimal?” Financial Analysts Journal 45(1), 31–42. ↩
Demiguel, Victor & Garlappi, Lorenzo & Uppal, Raman. (2009). Optimal Versus Naive Diversification: How Inefficient is the 1/N Portfolio Strategy?. Review of Financial Studies. 22. 10.1093/rfs/hhm075. ↩

Exponential Covariance

2018-08-15T00:00:00+00:00

For the past few months, I have been doing a lot of research into portfolio optimisation, whose main task can be summarised as follows: is there a way of combining a set of risky assets to produce superior risk-adjusted returns compared to a market-cap weighted benchmark? The answer of Markowitz (1952) is in the affirmative, with some major caveats. Given the expected returns and the covariance matrix (which encodes asset volatilities and correlations), one can find the combination of asset weights which maximises the Sharpe ratio. But because we don’t know the expected returns or future covariance matrix a priori, we commonly replace these with the mean historical return and sample covariance matrix. The problem is that these are very noisy estimators (especially the mean return), so much so that a significant body of research suggests that a naive diversification strategy (giving each asset equal weight) outperforms most weighting schemes. However, work by Kritzman, Page and Turkington (2010) affirms the intuition that there must be some information in the sample covariance matrix; accordingly, they observe that minimum variance portfolios can beat 1/N diversification.

In my own research, I have found this to be true. Minimum variance optimisation and its variants (no pun intended) can significantly outperform both 1/N diversification and a market-cap weighted benchmark. The nice thing about minimum variance optimisation is that success largely depends on how well you can estimate the covariance matrix, which is easier than estimating future returns. The most common methods are

Sample covariance – standard, unbiased and efficient, but it is known to have high estimation error which is particularly dangerous in the context of a quadratic optimiser.
Shrinkage estimators – pioneered by Ledoit and Wolf, shrinkage estimators attempt to reduce the estimation error by blending the sample covariance matrix with a highly structured estimator.
Robust covariance estimates – estimators that are robust to recording errors, such as Rousseeuw’s Minimum Covariance Determinant .

I have been experimenting with a new alternative, which I call the exponential covariance matrix (to be specific, the exponentially-weighted sample covariance matrix). In this post, I will give a brief outline of the motivation and conceptual aspects of the exponential covariance. However, because I am currently using this professionally, I will be intentionally vague regarding implementation details.

Motivation

In many technical indicators, we see the use of an exponential moving average (EMA) rather than the simple moving average (SMA). The EMA captures the intuition that recent prices are (exponentially) more relevant than previous prices. If we let $p_0$ denote today’s price, $p_1$ denote yesterday’s price, $p_n$ denote the price n days ago, the exponentially weighted mean is given by:

\[\alpha \left[ p_0 + (1-\alpha) p_1 + (1-\alpha)^2 p_2 + \ldots + (1-\alpha)^n p_n + \ldots\right]\]

$\alpha$ parameterises the decay rate ($0 < |\alpha| < 1$): by observation we note that higher $\alpha$ gives more weight to recent results, and lower $\alpha$ causes the exponential mean to tend to the arithmetic mean. Additionally, because $1/\alpha = 1 + (1-\alpha) + (1-\alpha)^2 + \ldots$, we note that ‘weights’ sum to one.

In practice, we do not compute the infinite sum above. Rather, observing that the weights rapidly become negligible, we limit the calculation to some window. This window is not to be confused with the span of the EMA, which is another way of specifying the decay rate – a good explanation can be found on the pandas documentation.

The EMA is useful because it ‘reacts’ to recent data much better than the SMA owing to the exponential weighting scheme, while still preserving the memory of the timeseries.

Covariance

Covariance, like correlation, measures how two random variables X and Y move together. Let us think about how we would define this metric. For variables that have high covariance, we would expect that when $x$ is high, so is $y$. When $y$ is low, $x$ should be too. To capture this idea, we can proceed as follows.

For each pair $(x_i, y_i)$ in the population, we measure how far away $x_i$ and $y_i$ are from their respective means, then multiply these distances together. If these differences have the same sign, i.e $x_i$ is greater than $\bar{x}$ and $y_i$ is greater than $\bar{y}$, there is a positive contribution to the covariance. If $x_i$ is less than $\bar{x}$ but $y_i$ is greater than $\bar{y}$, there is a negative contribution. We can then sum over all observations, and divide by the number of observations to get some kind of ‘average co-variation’. In fact, this is the exact definition of the population covariance:

\[Cov(x,y) = \frac{1}{N} \sum_{i=1}^N (x_i - \bar{x})(y_i - \bar{y})\]

Please note that in practice you would always use the sample covariance instead, which has a factor of $1/(N-1)$ rather than $1/N$, but this is not something we will worry about here.

Covariance of asset returns

One would think that the easiest approach is to take two price series (e.g. stock prices for AAPL and GOOG), then compute the daily percentage change or log returns, before feeding these into a covariance calculation. This does work, and is the standard approach. But in my view, this is carelessly throwing away a good deal of information, because:

Covariance does not preserve the order of observations.

You will get the same covariance whether you provide $(x_2, y_2), (x_{17}, y_{17}), (x_8, y_8), \ldots$ or $(x_1, y_1), (x_2, y_2), (x_3, y_3), \ldots$.

However, in the case of time series, the order of the returns is of fundamental importance. Thus we need some way of incorporating the sequential nature of the data into the definition of covariance. Fortunately, it is simple to apply our intuition of the EMA to come up with a similar metric for covariance.

We will rewrite our previous definition as follows:

\[Cov(x,y) = \frac{1}{N} \left[ (x_1 - \bar{x})(y_1 - \bar{y}) + (x_2 - \bar{x})(y_2 - \bar{y}) + \ldots + (x_N - \bar{x})(y_N - \bar{y}) \right]\]

Rather than letting $(x_i, y_i)$ be any observations from the dataset, let us preserve the order by saying that $(x_i, y_i)$ denotes the returns of asset X and Y $i$ days ago. Thus $(x_1 - \bar{x})(y_1 - \bar{y})$ specifically refers to the co-variation of the returns yesterday.

The next step should now be clear. We simply give each co-variation term an exponential weight as follows:

\[Cov(x,y) = \frac{\alpha}{N} \left[ (x_1 - \bar{x})(y_1 - \bar{y}) + (1-\alpha)(x_2 - \bar{x})(y_2 - \bar{y}) + \ldots + (1-\alpha)^N(x_N - \bar{x})(y_N - \bar{y}) \right]\]

Or more simply:

\[Cov(x,y) = \frac{\alpha}{N} \sum_{i=1}^N (1-\alpha)^{i-1}(x_i - \bar{x})(y_i - \bar{y})\]

And we are done! This simple procedure is all that is required to incorporate the temporal nature of asset returns into the covariance matrix.

Conclusion

This post has presented a modification of the covariance matrix especially suited to time series like asset returns. It is simple to extend this to an exponential covariance matrix and use it in portfolio optimisation – it is reasonable to suggest that this matrix will be positive definite if the sample covariance matrix is positive definite.

I have used the exponential covariance to great effect regarding portfolio optimisation on real assets. Backtested results have affirmed that the exponential covariance matrix strongly outperforms both the sample covariance and shrinkage estimators when applied to minimum variance portfolios.

At some stage in future, I will consider implementing this in my portfolio optimisation package PyPortfolioOpt, but for the time being this post will have to suffice. Please drop me a note if you’d like to use the ideas herein for any further research or software.

Update: as of 20/9/18, exponential covariance has been added to PyPortfolioOpt!

Stormy Seas for Proof of Work

2018-06-25T00:00:00+00:00

In this post we will be examining one of the main problems with Proof of Work (PoW) – not the energy inefficiency (as it is debatable how much of a problem this really is), but something more fundamental with the consensus process. In the past couple of months we have seen a number of cryptocurrencies fall victim to 51% attacks. Verge, Bitcoin Gold, ZenCash, and Electroneum are just a few coins that have been targeted, resulting in a total equivalent theft of $5 million (not to mention the subsequent loss in market value of the coins).

51% attacks are a basic problem in distributed ledger technology, covered in any crypto 101 course. Essentially, each individual node in a decentralised network is responsible for validating transactions and optionally submitting a block of these transactions to the blockchain – doing so requires the node to solve hash puzzles (this is why it is called Proof of Work). The beauty of this system is that each node has a say in what happens, proportional to the amount of hash power they contribute – thus the system is a democracy of sorts. However, a natural corollary of this is that any node or group of nodes that achieves a majority of the hash power can ‘outvote’ the rest of the network, allowing them to conduct a 51% attack.

Standard theory dictates that if there are enough independent nodes on a distributed ledger, we can reap the benefits of democracy while knowing that it would be immensely costly for a malicious party to achieve 51% of the hash power. This may be true for cryptocurrencies with many active nodes (like Bitcoin and Ethereum), but with the proliferation of multitudinous altcoins, we may not be able to say the same for more obscure tokens. It is thus the case that 51% attacks have gone from being a textbook problem to something very real, actively destroying both reputation and value in the crypto space.

In this post, we will explore how an attacker might go about conducting a 51% attack, examine the features that make a coin most vulnerable, and comment on prevention and mitigation. An obvious disclaimer: nothing in this post should be construed as a recommendation or a practical guide on conducting a 51% attack.

Malicious actors

It is no secret that cryptocurrency has attracted a large number of malicious parties, ranging from exchange-hackers to phishing scammers. Part of the reason for this is the underlying anonymity/psuedonymity of the system. If someone were to steal one million USD, they would likely require a network of offshore bank accounts to get away with it. However, if you were to steal 200BTC, all you’d have to do is convert it to a privacy coin like monero on any one of the exchanges that doesn’t require KYC, and law enforcement would have a very hard time catching up with you.

In this post we are dealing with a narrow subset of malicious actors: those who play by the rules. There are multiple ways a 51% attacker can ‘legally’ (in terms of the blockchain protocol) use their majority hash power to behave maliciously, one example being the denial of service to users in a network. But by far the most direct way of benefiting at the cost of others is to double spend, which is the when the same coin is sent to multiple parties in a network.

How a 51% attacker might double spend

Once an attacker has identified a suitable target (more on this later), this is a rough sketch of how they might profit. Again, this is purely educational and hypothetical – by no means is it a practical guide.

I will refer to the target coin as TCOIN. Many of these steps also require an anonymous “exit coin” – I will use Monero (XMR) as an example.

Acquire some TCOIN anonymously, e.g. via an offline swap of fiat for XMR then XMR for TCOIN.
Set up an account with exchange A and exchange B. Clearly these exchanges should have minimal KYC.
Send 1000 TCOIN from your TCOIN address to that of exchange A, then immediately cash it out to XMR.
Acquire 51% of the TCOIN network’s hash power, then make a new TCOIN transfer to exchange B. This transaction should be put into a block that orphans the previous block, so although exchange A thinks they have received your TCOIN and you have cashed it out to XMR, in reality it is exchange B that has received the TCOIN.
On exchange B, convert TCOIN to XMR and send it to your monero wallet.

In general terms, this describes how the double spend lets you manufacture 1000 TCOIN from thin air (at the expense of the first exchange). An optional additional step is to first short TCOIN, because we have seen that 51% attacks severely reduce public trust in the coin’s development team and tend to lead to a sudden price drop.

Thus far this has all been theoretical – we have presented a textbook situation of how a double spend might work in theory. In the next section, we will understand the worrying practical reality of the situation.

Vulnerability of different coins

One would think that there should be a perfect correlation between the market cap of a coin and its overall hash rate – that is, the demand for a coin should be correlated with the ‘strength’ of the network validation. However, empirically this is not the case. Because of the rise of ill-informed speculation, we see that some coins with a high market cap (a few hundred million USD) have a surprisingly low hash rate. This is the clear yet disturbing message of crypto51.app, a simple site which displays the cost of becoming a 51% attacker (over 1 hour) for different PoW cryptocurrencies.

I submit that there are four main features an attacker would look for in an ideal target coin, apart from the obvious factor of having a smaller hash rate (and thus a lower attack cost).

High market cap (as a proxy for interest/liquidity in the coin)
Low transaction time (i.e very few block confirmations)
Support on KYC-free exchanges
A common hashing algorithm, or at least one that is supported on a mining marketplace like NiceHash. The advantage of this for an attacker is that they can just rent hash power anonymously rather than having to acquire hardware.

Using python’s requests and beautifulsoup, I scraped the data from crypto51 in order to do my own analysis (available on a Jupyter notebook) of which coins were most vulnerable:

import requests
from bs4 import BeautifulSoup

r = requests.get("https://www.crypto51.app/")
soup = BeautifulSoup(r.text, "lxml")
data = []
headers = []
table = soup.find('table')
column_names = table.find('thead').find_all('th')
headers = [i.text.strip() for i in column_names]
rows = table.find('tbody').find_all('tr')
for row in rows:
    data.append([i.text.strip() for i in row.find_all('td') if i])
df = pd.DataFrame(data, columns=headers)

I then processed the data and generated a plot of the attack cost versus the market cap, coloured by NiceHash-ability (all log-transformed). This is very easy to do using pandas plotting:

df.plot.scatter("Log Market Cap", "Log Attack Cost", 
                c="Log NiceHash", colormap="plasma")

In a graph like this, a target coin should be as close to the lower-right quadrant as possible (high market cap but low attack cost). The lighter the datapoint, the easier it is to attack via NiceHash, which may or may not be important to an attacker.

A linear trendline fits the data with $R^2 = 0.65$, and because a straight line on log-log axes corresponds to a power law relationship, we can calculate the coefficients as follows:

\[\text{1h attack cost} = 8.3 \times 10^{-5} \cdot (\text{mcap})^{0.92}\]

Actually, it is not the general relationship that matters but rather the specific outliers – altcoins below the trendline are those with especially low attack costs relative to their market cap.

The top portion is self explanatory. In the bottom-left quadrant are coins that are easy to attack, but also have such small market caps that they are likely not worth attacking (e.g. very poor exchange support). I thus determine the danger zone to be coins with slightly higher market caps (in the order of USD 10 million to 100 million). However, I happened to notice that some of the recently attacked altcoins (listed in the introduction) formed a narrow band slightly above my danger zone. Perhaps this is a liquidity sweet-spot that makes it easier for attackers to exit onto another exchange, or perhaps it’s a coincidence.

In any case, if I were holding something like BTCP (Bitcoin Private), I’d be a little bit worried. It has a market cap of more than \$200 million, and reasonably liquid trading pairs, but costs less than \$500 an hour to attack. If the attacker were unwilling to set up hardware, they might prefer to target something like EMC2 (Einsteinium), which has a higher NiceHash-ability.

EDIT: 3 months after writing this post, BTCP was 51%-attacked!! I hope this post had nothing to do with it.

Now, in a perfectly efficient market, one would imagine that the possible financial benefit from conducting a 51% attack would not outweigh the financial costs. However, recent attacks show that this is clearly not the case (especially because one can now short the target coin prior to the attack). So there is a clear problem in the crypto markets.

What can we do about it?

An obvious solution is to ditch PoW and go with Proof of Stake (PoS). Much easier said than done. Arguably the most mature PoS development effort is that of Ethereum, in the form of Casper. Yet despite the undeniable talent of the dev team, solving the nothing-at-stake problem and ironing out the wrinkles in the implementation is not proving to be a straightforward task. Delegated PoS may be a stepping stone, but the concessions with regard to decentralisation may be a bit off-putting. So assuming that PoW is still the de-facto consensus mechanism, what can the ecosystem do to reduce 51% attacks? Here are some ideas.

Speculators/investors should consider the hash rate of any PoW coins they are looking to invest in, and be aware that a low hash rate makes that blockchain vulnerable for a 51% attack – certainly not good for their investment.

New PoW altcoins should not use the same hashing algorithm as big coins, even if it’s easier to implement – a large miner can just point their hash power at the new blockchain and immediately become a majority. Dev teams for these projects should implement programmatic checks for 51% attacks: a quick response can be critical. Fiat reserves should be maintained, which can quickly be used to add hashing power to the network if a potential 51% attack is detected.

Exchanges should be wary of what tokens they list, and should increase the required block confirmations for transactions, making it more costly for attackers to rewrite the recent history of a blockchain. Yes, this would make it slightly more inconvenient for users, but 51% attacks can directly lead to material losses.

Conclusion

Proof of Work is a genius solution to the distributed consensus problem, and I often have to remind myself just how amazing the original bitcoin protocol is. But I believe that in PoW, 51% attacks should be possible – they are a direct consequence of democracy. For large decentralised networks like that of bitcoin or Ethereum this is not a problem, but owing to the speculative interest in cryptocurrencies, the demand for some altcoins has separated entirely from the security of their network, resulting in a kind of arbitrage which allows malicious actors to profit by conducting a double spending 51% attack.

This post’s analysis of coin vulnerability has not been perfect: the main flaw is our use of market cap as a proxy for the profitability of a 51% attack. In reality, the profitability is a function of block confirmation times and exchange liquidity. Market cap is arguably correlated with the two, but it is not as direct.

I don’t claim to have all the solutions to the problem of 51% attacks, but it is evident that no single party can solve the issue alone. Until the different players (miners, users, exchanges, investors) in the ecosystem contribute their part, coins with low hash rates will continue to be low hanging fruit for the aspiring 51% attacker; I’d wager that we’ll see a few more big attacks before the year ends.

Evolving cellular automata to solve problems, part 2

2018-06-01T00:00:00+00:00

We will be picking up where the previous post left off. As a brief summary, we are attempting to replicate the results of Evolving Cellular Automata with Genetic Algorithms (Mitchell, Crutchfield and Das 1996), dealing with the density classification task for 1D binary cellular automata (CAs). To put it simply, we are trying to design a ruleset such that the final configuration of a cellular automaton after M iterations is either all 1s or all 0s depending on which class was more common in the initial state. The caveat is that each cell in the universe can only make its decision based on the three neighbours to the left and right.

We examined the failures of the majority ruleset, an ‘obvious’ solution in which each cell updates to the local majority among its neighbourhood, and thus see the need for something more sophisticated. At the same time, we saw that the space of possible solutions is in the order of $10^{38}$, meaning that any kind of brute force method will not be feasible. This invites the use of genetic algorithms (GAs), with which we could evolve a CA to perform this task without ever explicitly devising a ruleset.

Genetic algorithms in the context of CAs

A complete treatment of genetic algorithms is not in the scope of this post (the wikipedia page is a good start), but I will attempt to cover the salient points:

We start with a population of different possible rulesets. This is a very important point – the individuals in the population are not the cellular automata, they are what we are trying to optimise, which in this case is the ruleset.
Rulesets in the population have different fitnesses, corresponding to how well they solve the $\rho_c = 1/2$ task. We will define the fitness F to be the proportion of random ICs on which the ruleset succeeds (becomes all 1 or all 0 correctly). So if we test a ruleset on 100 random ICs and 35 are classified correctly, $F = 0.35$
In each generation, we compute the fitness for each ruleset in the population. The fittest rulesets (the elites) are cloned to the next generation, but we also crossover (i.e breed) these rulesets to form offspring, which are then mutated before joining the next generation.
Over many generations, we should see the fitness increase, as the traits of the favourable rulesets are passed onto the next generation.

The GA acts as a more efficient way to traverse the huge space of possible rulesets in search of one that can suitably solve the density classification task – it does not guarantee that the global optimum will be reached.

Implementation

For the most part, the functions used in this post are the same as in part 1, though took care to make obvious optimisations because the evolution of the CAs will involve repeated calls of the same functions.

The initial population consists of random rulesets. To generate a single random ruleset, we zip the the rule inputs (7-bit binary strings) with random 1s or 0s which are obtained from the uniform_random_binary function we defined. For convenience, we can uniquely refer to a given ruleset with a hexadecimal ID, which is made by converting the binary string of all rule outputs (assuming ordered inputs) into hexadecimal.

def random_ruleset(n_rules):
    rule_values = uniform_random_binary(n_rules)
    hex_id = hex(int(rule_values, 2))
    return hex_id, dict(zip(rule_keys(n_rules), rule_values))

To generate the initial population of 100 (or more generally, n_ic) ICs, it is not as simple as running random_ruleset 100 times because we want exactly half of the ICs to have a majority of 1s.

def generate_random_ICs(universe_size, n_ic=100):
    ics = []
    cutoff = int(0.5 * universe_size) + 1
    # Exactly half of the initial conditions will have p < 0.5
    for _ in range(n_ic // 2):
        minority = uniform_random_binary(universe_size, 0, cutoff)
        majority = uniform_random_binary(universe_size, cutoff, 
                                         universe_size + 1)
        ics.append(minority).append(majority)
    return ics

Prior to implementing crossover and mutation, we will define a helper function that converts a hex ID into a ruleset, by converting the hexadecimal to binary and zipping this with a list of 7-bit rule inputs:

def hex_to_ruleset(hex_id, n_rules=128):
    rule_values = bin(int(hex_id, 16))[2:].zfill(n_rules)
    return dict(zip(rule_keys(n_rules), rule_values))

As discussed in the paper, crossover involves pointwise swaps between two parent rulesets, and mutation involves randomly switching bits.

def crossover_and_mutate(hex1, hex2, n_rules=128, 
                         n_crossover_points=1, n_mutations=2):
    bin1 = list(bin(int(hex1, 16))[2:].zfill(n_rules))
    bin2 = list(bin(int(hex2, 16))[2:].zfill(n_rules))
    # Crossover: get indices of unequal, then
    # randomly select an index to swap
    unequal_indices = [i for i, digit in enumerate(bin1) if 
                       bin2[i] != digit]
    if len(unequal_indices) > n_crossover_points:
        crossover_points = random.sample(unequal_indices, 
                                         n_crossover_points)
        for c in crossover_points:
            bin1[c] = bin2[c]
    # Mutation
    for _ in range(n_mutations):
        idx = random.randrange(n_rules)
        # Flip bit
        bin1[idx] = str(int(not bool(bin1[idx])))
    return hex(int("".join(bin1),2))

Metaparameters

As with most optimisation methods, genetic algorithms have a number of metaparameters that can drastically change the resulting behaviour, for example the number of generations, the mutation rate, the size of the initial population of chromosomes, etc.

GEN = 50
N_CHROMOSOMES = 100  # number of rulesets in each generation
N_CROSSOVER_POINTS = 1
N_MUTATIONS = 2

In addition, we shouldn’t forget the CA parameters:

N = 59  # universe size
MAX_ITER = 100  # max number of iterations to perform the task
N_IC = 100  # number of ICs each ruleset is tested on
N_RULES = 128  # number of rules in each ruleset

These are mostly similar to the choices of Mitchell et al., though I have opted for fewer generations and a much smaller universe (they use 100 and 149 respectively), which should be faster to compute and serve as a proof-of-concept.

It is important to note that genetic algorithms can be quite sensitive to initial conditions, which is why we will parameterise each run by a random seed.

Evolution

We are now ready to implement the actual genetic algorithm. The first thing to do is to set the random seed and generate the initial population. We will use a dictionary, with the keys being hex IDs and values being the rulesets.

random.seed(0)
population = {}
for _ in range(N_CHROMOSOMES):
    id, ruleset = random_ruleset(N_RULES)
    population[id] = ruleset

In each generation, we will first compute the fitness of each ruleset on 100 random ICs:

generation_ics = generate_random_ICs(N, N_IC)
fitness_log = []

for hex_id, chromosome in population.items():
    # Evaluate performance on random ICs.
    fitness = 0
    for config in generation_ics:
        # Count the majority
        majority = config.count("1") > len(config) // 2
        # Iterate automaton
        hist = iterate_automata(config, chromosome, MAX_ITER, 
                                verbose=False)
        # If the CA has classified correctly, increment fitness
        if hist[-1] == str(int(majority)) * len(config):
            fitness += 1
    print("hex_id", hex_id, "  Fitness", fitness)
    fitness_log.append((hex_id, fitness))

Next, we determine who are the elites (the top 20 rulesets):

# Find top 20 in population
fitness_log.sort(key=lambda x: x[1], reverse=True)
elites = [i[0] for i in fitness_log[:20]]
print("Generation best:", fitness_log[0])
print("Least fit elite:", fitness_log[20])

As suggested by the paper, we will copy these elites directly to the next generation, then form the rest of the population by breeding these elites with replacement.

# Copy the top 20 to the next generation
new_population = {}
for i in elites:
    new_population[i] = hex_to_ruleset(i)

# Crossover and mutate random pairs of elites
for _ in range(N_CHROMOSOMES - 20):
    parent1, parent2 = random.sample(elites, 2) 
    child_hex = crossover_and_mutate(parent1, parent2, 
                                     N_CROSSOVER_POINTS, 
                                     N_MUTATIONS)
    new_population[child_hex] = hex_to_ruleset(child_hex)

population = new_population

We then repeat the above for many generations, and hopefully should see the fitnesses increase.

Initial observations

Judging by the success of the previous post, I was expecting the script to work immediately and yield interesting results. Alas, programming is rarely so simple. The genetic algorithm was slow (I’m used to the 5-10 seconds it takes to train most sklearn models), but what is worse, it was not able to solve the task. After 50 generations, the GA has not managed to beat random guessing:

Gen 49 
==========
hex_id 0x455d652de3a5534054ed51a12093d25   Fitness 50
hex_id 0x455d652de3a5534054ec51a12093d27   Fitness 50
hex_id 0x455d652de3a5534054ed51212093d25   Fitness 50
hex_id 0x455d652de3a55b40146d51a12093d27   Fitness 50
...
Generation best: ('0x455d652de3a5534054ed51a12093d25', 50)
Least fit elite: ('0x455d652de3a55b40146d51a12093d277', 50)

A closer analysis revealed that many of the final rulesets corresponded to the naïve strategy of mapping any possible input to a 1, which results in the ruleset being correct about 50% of the time.

It should be noted that Mitchell et al. trained their genetic algorithm across 300 different runs, so it could be the case that this was one of the unlucky runs in which the GA fails to learn. But testing multiple random seeds yielded no further improvements.

A possible fix

I hypothesised that the initial population contained too many rulesets that had a clear majority of 1s or 0s, disproportionately mapping their input to one of the two regardless of the IC. Remember that this was by design, we chose to sample rulesets and ICs from a distribution uniform over the density of 1s rather than the unbiased distribution which is binomial over the density of 1s. In light of the poor empirical results, I modified the random_ruleset function to use the unbiased distribution instead:

def random_ruleset2(n_rules):
    random_binary = bin(random.randint(0, 2**n_rules - 1))
    rule_values = str(random_binary)[2:].zfill(n_rules)
    hex_id = hex(int(rule_values, 2))
    return hex_id, dict(zip(rule_keys(n_rules), rule_values))

Because it seemed that I would have to test multiple parameters, I also refactored the GA script into a function:

evolve_automata(59, n_crossover_points=1, n_mutations=6,       
                random_seed=0, n_ic=100, n_gen=50, 
                ruleset_generator=2)

Using this new ruleset generator, we are much less likely to have naïve outputs of all 1s or all 0s. My guess was that although this would result in the GA starting with poorer fitnesses, they would not get ‘stuck’ with naïve random-guessing strategies. However, testing revealed that fitness still converged to 50%, just at a much slower rate.

To combat this, I reasoned that the simplest way to escape local optima is to pump up the mutation rate in the hope that elites will be found that beat random guessing, and these will pass on their traits to the next generation.

Finally, we achieved some progress. I noticed that the combination of a high mutation rate and an unbiased random ruleset meant that the GA didn’t always get stuck with a 50% fitness. For N_MUTATIONS = 6, tested over 20 different seeds, mean elite fitnesses in the last 10 generations were:

[83.5, 52.2, 92.6, 51.2, 51.1, 50.0, 51.5, 51.5, 51.7, 79.2, 
 83.1, 51.7, 73.5, 57.2, 51.6, 51.2, 52.1, 50.0, 78.7, 82.9]

Eight of the 20 runs resulted in fitnesses beyond the 50% level, however only one of the 20 beat 90%. This falls very far short of Mitchell et al’s claim to have at least 90% fitness on all 300 of the runs, but it is certainly progress.

Interestingly, increasing N_MUTATIONS to 12 produced worse results, with the maximum mean elite fitness being 80.2 and only six of the 20 runs resulting significantly beating 50%. N_MUTATIONS=18 does even worse, with only one of the 20 beating 50%. I think this is because having many mutations reduces the impact of crossover (which is only at a single point), so we don’t get the benefits of ‘meiosis’.

Analysis of evolved CAs

We will examine one of the rulesets which achieved a fitness of above 90%. For some different initial configurations, this is what we observe:

Remember that purple is alive and white is dead; I have displayed the actual majority class above each subplot. Thus the fifth subplot represents an error, because despite the majority of 1s, the final state is all 0s.

As in the paper, the ruleset works by quickly reaching a steady state of all 0s unless large blocks of 1s are found – it is not really the sophisticated emergent behaviour that I was hoping to see. The only exception is the first subplot, in which some semblance of the particle strategy is achieved.

Conclusion

Though we did achieve some interesting results, I don’t think that this exploration overall was a success. I could not replicate the results of Mitchell et al., even using a simple universe of 59 cells rather than 149. The default setup used in the paper kept getting stuck at 50% fitness, and it was only after I deviated from their suggestions that I managed to do better.

I have become somewhat disillusioned regarding the scientific value of the paper (though I still find it fascinating). The sophisticated particle strategy discovered by Mitchell et al. is remarkable, yet they seem to downplay the fact that it was observed only in a few of the 300 runs of the genetic algorithm. This is hardly reproducible, and really feels to me like grabbing at patterns in the random darkness.

Additionally, the evolution of CAs with genetic algorithms is a very slow process (I had to leave my code running over a few nights), and the results are very inconsistent. I can only imagine how long it would have taken for Mitchell et al. to conduct 300 runs with a universe size of 149 (before the turn of the millennium!). This severely limits the practical applications, and the prospect that evolved CAs could tackle more complex tasks like image processing (as suggested in the paper’s conclusion) seems chimerical.

Nevertheless, I am happy to have attempted to replicate the results, because it has been a while since I’ve played around with genetic algorithms. It was also some extra practice with python’s standard language features, of which I make extensive use. Perhaps some time in future I may realise that my implementation had a flaw and author a part 3, but for now I’m content to stay in this local optimum.

The code for this post, along with plots and data, is available as a Jupyter notebook on GitHub.

Evolving cellular automata to solve problems, part 1

2018-05-25T00:00:00+00:00

Recently I finished reading Complexity: A Guided Tour, by Melanie Mitchell, which reminded me a lot of Gödel, Escher Bach (indeed, the book is dedicated to Douglas Hofstadter). It has reminded me that emergence is an incredibly fascinating concept – simple individual units somehow coming together to result in complex behaviour that cannot really be explained in terms of the components. In this post, we analyse the emergent behaviour of a cellular automaton and attempt to use it to solve a simple problem.

One of the first examples of emergence discussed in the book is the army ant: even in groups of 100, these ants are completely useless and will walk in circles until they die of exhaustion. But in a colony, they show remarkable sophistication and a “superintelligence”. I would highly recommend the article Army Ants: a Collective Intelligence, by Nigel Franks, and this short David Attenborough clip for more information. Apart from ants, Mitchell also makes consistent reference to cellular automata, the study of which really led to the growth of complexity science as a distinct field.

Cellular automata (CAs) have managed to divert the attention of many great scientists – Stephen Wolfram considers them the foundation of a “new kind of science”. I am clearly not one of these great thinkers, because I have never really understood what the fuss is all about. Although I have written about Conway’s Game of Life (a special case of CAs) in a previous post), I admit that I was more interested in the implementation than in the CAs.

But Complexity has given me a fascinating new perspective. In Chapter 11, Mitchell introduces the idea that CAs can be designed (or should I say evolved) to solve computational tasks. Although it is widely known that the Game of Life is Turing-complete (arguably more interesting than Mitchell’s idea), this always felt far too abstract for me to appreciate – the proof relies on a demonstration that some logic gates can be emulated, rather than actually constructing a nontrivial program. However, in Evolving Cellular Automata with Genetic Algorithms (the journal article on which Chapter 11 is based), Mitchell and coauthors tangibly demonstrate that genetic algorithms can be used to evolve a CA to perform a simple but nontrivial task.

The paper is very interesting, but does not come with source code (in their defense, it was written in 1996 and they refer to the internet as the “World Wide Web”). I think that their work deserves a modern recreation and analysis, with the benefit of intuitive code in python and more processing power.

Because this is quite a large topic, I will split it into two parts. Today’s post will set up the problem, dealing with notation and terminology, as well as discussing the nature of the CAs considered. We will leave it to a future post to actually apply genetic algorithms to evolve CAs.

A quick introduction to cellular automata
Computation with cellular automata
The majority ruleset
Conclusion

A quick introduction to cellular automata

A cellular automaton can be thought of as a set of units (cells) whose states change based on simple rules. In the Game of Life, the universe is a 2D grid of binary cells (either alive or dead), with the update rules as follows:

A living cell (black) will die if it has fewer than two live neighbours or more than three live neighbours – roughly corresponding to underpopulation and overpopulation respectively
A dead cell (white) will come alive if it has exactly three living neighbours (captures the idea of reproduction).

Note that to begin applying the rules, some initial condition must be specified – in practice, small changes in this starting point can drastically affect the resulting behaviour.

For our purposes we will be dealing with an arguably simpler model: a 1D binary CA. In this case, the universe consists of a row of N cells, each of which can either be dead (0) or alive (1), and is updated by looking at the radius R of surrounding cells. So if $R=3$, any update rules must map a 7-bit input (the 3 cells on either side, and the cell itself) to a 1-bit output (whether the cell will be dead or alive). A given rule might look something like this, bearing in mind that the output bit determines the new state of the middle cell:

A ruleset is the set of mappings between all possible input configurations and their output states. We may as well begin to speak in specific terms referring to $R=3$, in which case a ruleset is a mapping between all possible 7-bit binary strings to a binary output:

Input	Output
0000001	1
0000010	0
…	…
0100101	0
0100110	1
…	…
1111110	1
1111111	0

If we assume that the input is ordered, then as a convenience of notation we can say that a ruleset is equal to the binary string formed by concatenating the output bits – in the above example, this is $10…01…10$.

It is very important to be comfortable with this concept: we are representing rulesets with binary strings of length 128, because there are 128 possible 7-bit inputs. Later on, we will also be using binary numbers to represent configurations of cells, so please remember to distinguish the two.

Because we are dealing with long 1D universes, animation is probably not the visualisation tool of choice (though it works very nicely for 2D). Instead, we will use a space-time diagram, which essentially stacks the generations on top of each other (with time going down the page). Here’s an example, with a simple rule that switches input bits:

Computation

I am going to conveniently sidestep the issue of properly defining “computation”. As far as this post is concerned, computation is what a computer does – a CA “computes” if it solves some task for a variable input. I would refer you to Complexity if you’d like a meaningful discussion.

The paper’s main objective is the density classification task, in which the CA must decide whether the ‘density’ of 1s in the initial configuration is greater than some cutoff proportion. Because density is typically denoted by $\rho$, this is more precisely referred to as the $\rho_c = 1/2$ task.

To put it simply, the CA must decide whether the initial setup had more living or dead cells (1s or 0s). If there were more 1s, then after some number of iterations, we want the CA to display all 1s. Otherwise, if there were more 0s, we want the CA to display all 0s. An example mapping is shown below:

You may be questioning the difficulty of that task. After all, it’s as easy as iterating over the universe and counting the number of 1s, then determining the output based on this value! But remember the catch: each cell only has access to a fixed neighbourhood of three cells on either side. There is nowhere we can store the count: each cell can only make a decision based on its neighbours and change its state accordingly.

The majority ruleset

Before we bother with genetic algorithms, it is worth checking how a simple solution performs. For this, we will use the majority ruleset, in which each cell will just change its state to match the majority of its 7-cell neighbourhood.

Although it is conceptually straightforward, it is at this stage that we need to consider the details of the implementation in python. We will be using strings of binary digits to store configurations; they are efficient and easy to work with. At a high level, we need to do the following:

generate the majority ruleset (of course we are not going to hard code all 128 rules)
generate a random initial configuration (i.e a random binary number)
iterate the initial configuration according to the majority ruleset
plot the space-time diagram

Here are some of the global constants that we need for now, though we will introduce many more when we get onto genetic algorithms:

R = 3  # size of neighbourhood
N_RULES = 2 ** (2 * R + 1)  #  number of rules in a ruleset
N = 79  # number of cells in universe

The number of rules is $2^7 = 128$ (the number of possible 7-bit inputs). Incidentally, this means that the number of possible rulesets is $2^{128}$ – a clear indication that we will not be able to find a solution by brute force.

Generating the majority ruleset

We are going to treat rulesets as python dictionaries. This is quite natural, since a ruleset is essentially a bijective mapping between all possible 7-bit inputs (keys) and the output bits (values). We will generate the keys and values separately.

def rule_keys(n_rules, radius=3):
    rule_input = []
    for i in range(n_rules):
        binary = str(bin(i))[2:].zfill(2*radius + 1)
        rule_input.append(binary)
    return rule_input

This could also be refactored into a one-line generator expression:

rule_keys = (str(bin(i))[2:].zfill(7) for i in range(N_RULES))

But out of respect for Guido I have kept it as a function with a loop inside (which is slightly faster anyway). The [2:] is used to slice away the ugly 0b that python prepends to a binary number, and zfill pads the string with leading zeroes.

To complete the majority ruleset, we just need to know if there are more 1s or 0s in the input. We will use a bit of python trickery to make this more concise:

majority_ruleset = {}
for rule_input in rule_keys(N_RULES):
    # 1 if the majority of the neighbourhood is 1.
    out = int((sum(d == "1" for d in rule_input) > R))
    majority_ruleset.update({rule_input: str(out)})

The statement (sum(d == "1" for d in rule_input) > R is True if there are more than 3 living cells in the input (remember that R = 3), and False otherwise. We then cast this to an int (1 or 0) then a str (“1” or “0”), and update the ruleset with this key-value pair. This is an excerpt from the resulting ruleset:

 {
     '1010010': '0',
     '1010011': '1',
     '1010100': '0',
     '1010101': '1',
     '1010110': '1',
     '1010111': '1',
     '1011000': '0',
     '1011001': '1',
 }

Generating a random initial configuration

One might think that the best way to generate a random configuration is just to combine N random bits where N is the size of the universe. However, consider the density of 1s in such a configuration – it is a (literal) textbook example of a binomial distribution with $\rho \sim B(N, 0.5)$.

Notice the strong peak at 0.5, meaning that most of the initial configurations will have a roughly equal split of 0s and 1s. This is not ideal for us to explore the density classification task, because we do want to have enough easy cases (i.e clear majorities).

As such, we will adopt a slightly different strategy:

Randomly choose a density of 1s that we want in the configuration (sampled from a uniform distribution).
Form an ordered universe with that number of 1s, and the rest 0s
Shuffle the universe

A small snippet that does this is as follows:

n_ones = random.randrange(0, N)
binary = "1" * n_ones + "0" * (N - n_ones)
result = "".join(random.sample(binary, N))

We could use random.shuffle() in the third line, but that works in-place (and requires a list), so the chosen solution is cleaner. In the end, I decided to generalise the above slightly, letting you change the range of the uniform sample and specify the length of the output string:

def uniform_random_binary(length, lower=0, upper=None):
    # The defualt upper bound is the length
    if upper is None:
        upper = length
    n_ones = random.randrange(lower, upper)
    binary = "1" * n_ones + "0" * (length - n_ones)
    return "".join(random.sample(binary, length))

Iterating an automaton

To remain general, our iteration function should take in a configuration, a ruleset, and a radius. One iteration consists of applying the appropriate rule to each bit in the universe’s configuration, which is done by getting a cell’s neighbours as a 7-bit binary string, then using the ruleset dictionary to find the output bit.

def single_iteration(config, ruleset, radius):
    next_iteration = ""
    # Support wrapping by extending either end
    extended_config = config[-radius:] + config + config[:radius]
    for i in range(radius, len(config) + radius):
        neighbours = extended_config[i-radius:i+radius+1]
        next_iteration += ruleset[neighbours]
    return next_iteration

The complexity in the above implementation arises because we need a circular universe, so we must extend the configuration to allow for ‘wrapping’.

To iterate multiple times, we repeatedly apply the single_iteration() function:

def iterate_automata(initial, ruleset, n_iter, radius=3,
                     verbose=False):
    config = initial
    history = [config]
    for _ in range(n_iter):
        config = single_iteration(config, ruleset, radius)
        if verbose:
            print(config)
        history.append(config)
    return history

This function also keeps track of the CAs history, which is necessary if we want to use a space-time diagram for visualisation.

Space-time diagrams

If we run iterate_automata() with verbose=True, it’ll print the universe at each time step, so in a sense that is a space-time diagram. However, it is fair to ask for something more stylish, for which we will use matplotlib’s imshow:

def plot_spacetime_diagram(binary_history, filename=None):
    arr = np.array([[int(digit) for digit in row]
                     for row in binary_history])
    plt.figure()
    plt.imshow(arr, cmap="Purples_r")
    if filename is None:
        plt.show()
    else:
        plt.savefig(filename)

Tying it together

Now that we have implemented the pieces, testing the majority ruleset is very simple:

random.seed(0)
initial_config = uniform_random_binary(N)
majority_history = iterate_automata(initial_config,
                                    majority_ruleset,
                                    n_iter=N, 
                                    radius=R)
plot_spacetime_diagram(majority_history)

Looking at the space-time diagrams for different random seeds shows that in general, the majority ruleset does not consistently solve the $\rho_c = 1/2$ task:

The above results are very similar to what Mitchell et al. found (though they use $N = 149$), which gives us a good indication that our implementation thus far has been accurate.

Conclusion

In this post, we have laid the framework for using cellular automata to solve a computational task, namely, classifying the density of living cells in the initial conditions.

We have examined the failings of a ‘local’ solution that naively treats a cell’s immediate neighbourhood as a proxy for the entire universe, and thus see the need for emergent computation, in which individual cells somehow learn to appreciate the rest of the universe and come up with a global solution.

In the next post, we discuss how genetic algorithms can be applied to evolve the cellular automata to solve the density classification task, rather than having to supply a hand-crafted rule, or do a brute force search over all possible rulesets (which we have shown to be infeasible). Stay tuned.

Classifying financial time series using Discrete Fourier Transforms

2018-04-19T00:00:00+00:00

A financial time series represents the collective decisions of many individual traders; it seems reasonable to me that the nature of these decisions may differ based on the underlying asset. For example, a company with a higher market cap may be more liquid, and subject to larger individual buy/sell orders including institutional investment. Thus, there is a case to be made that information such as the market cap of a company should be ‘encoded’ into its price movements. While these characteristics may be difficult to pinpoint on a chart, in principle it may be possible for a machine learning algorithm to find statistical relationships between the time series and the market cap of the company. This post will investigate that claim.

Concretely, a raw dataset will be constructed, consisting of 100-day price charts which belong to companies that are respectively in the top and bottom 1000 tickers in the Russell 3000 ordered by market cap. We will then run feature extraction (which is the core of this post), before applying a standard XGBoost classifier.

This post thus represents an intuitive (if naive) first approximation for classifying whether a given price chart belongs to a stock with high or low market cap. The hypothesis is that this problem is learnable, so the goal is to show that a machine learning model can classify a time series with greater than 50% accuracy. Though accuracy often has flaws as a metric, in this case it is sufficient because the classes will be almost perfectly balanced.

A discussion on classifying time series

Because much of real life data is temporal in nature, classifying and clustering time series has widespread importance, with such applications as:

speech recognition
medical data such as ECGs
cyber-kinetic prosthetics: classifying electrical activity of muscles in terms of what action it corresponds to
meteorological data
seismic data

Some general approaches to classifying time series are as follows (a thorough survey can be found in Rani & Sikka 2012):

Directly clustering with an algorithm like k-Nearest Neighbours. What is important here is a relevant distance metric between two different time series. Euclidean distance may not be sufficient as it has trouble dealing with timeseries that have been shifted slightly (either vertically or horizontally).
Hidden Markov Models, which model the timeseries as a Markov Process, whose transition matrix may be used as input to a classifier.
LSTM Neural Networks, which are fed in a time series and contain a sequential ‘memory’. The output of a cell can be run through the sigmoid function to convert it into a binary class value.
Extracting summary statistics, then passing these results into a standard classifier.

This post will explore the last method. On the surface, it seems to be the most straightforward approach, but the difficulty is in choosing which aspects of the time series should be used. There are existing libraries that can extract a multitude of features, e.g. tsfresh, but although it is very likely that there is predictive power somewhere in here, it would be nice to find a simpler (and perhaps faster) approach.

Thus, I have decided to take the Discrete Fourier Transform (DFT) of the time series, and to use the largest terms as features. Such a method is really under the domain of spectral analysis, a well established subfield of time series analysis. However, we will adopt a rather informal approach. It should be noted that the Fourier Transform certainly shouldn’t be used to forecast the future of a financial time series because of the strong assumptions of periodicity. But it should be able to extract the main (sinusoidal) signals from a time series, which we can then put into a classifier.

Preparing the raw data

The data used in this project will be:

keystats.csv: parsed yahoo finance key statistics from which the latest market cap of securities will be ascertained. In order to reproduce something like this dataset (though only for the S&P500), please refer to my repo on GitHub
My stock_prices database, which I discussed in this post.

After extracting the data from the csv files and MySQL database, I was left with a pandas dataframe sorted_tickers_by_mcap, containing tickers and their market caps, sorted by market cap in ascending order.

print("Number of tickers with mcap data available:",
       len(sorted_tickers_by_mcap))
print(sorted_tickers_by_mcap.head())
print(sorted_tickers_by_mcap.tail())

Number of tickers with mcap data available: 3382
Ticker
heat      67830.0
aryx     103730.0
gmet     336260.0
coco    1090000.0
dvox    1140000.0
Name: Market Cap, dtype: float64
Ticker
xom      3.632700e+11
msft     4.304200e+11
goog     4.942700e+11
googl    4.944500e+11
aapl     5.413300e+11
Name: Market Cap, dtype: float64

I then extracted the top and bottom 1000 tickers (removing the tickers for which data was not available), resulting in two lists of tickers: top_mcap and bottom_mcap:

Available top market cap: 831
Available bottom market cap: 862

These tickers are then sorted by market cap, and 100-day windows are made from the last 750 datapoints (roughly corresponds to 3 market-years). 3 years was chosen as the cutoff because market cap values from more than three years ago are likely to be significantly different from the latest values, as companies grow or shrink over time. A solution would be to use the keystats dataset to find the market cap for a company in a given year, but I have opted for a simpler workaround in this post.

Here is the function I wrote to create the time series snapshots from the raw price data.

def create_snapshots(ticker, n_days=750, window_size=100, 
                     window_shift=50):
    """
    Returns list of time series of length windowsize, shifted
    by window_shift, for the last n_days.
    """
    df = pd.read_sql(f"SELECT adj_close_price FROM daily_price "
                    f"WHERE ticker_id={ticker_id[ticker]}", conn)
    df = df.iloc[-n_days:]
    snapshots = []
    for i in range(0, n_days, window_shift):
        window = df.iloc[i:i +window_size]
        window = window['adj_close_price'].values
        if len(window) == window_size:
            snapshots.append(window)
    return snapshots

These snapshots are then stacked into a numpy array as follows:

dataset = []

for i, ticker in enumerate(top_mcap):
    try:
        ticker_snapshots = create_snapshots(ticker)
    except Exception as e:
        print(str(e))
        continue
    ticker_snapshots = np.c_[np.array(ticker_snapshots),
                        np.ones(len(ticker_snapshots))]
    dataset.append(ticker_snapshots)

for i, ticker in enumerate(bottom_mcap):
    try:
        ticker_snapshots = create_snapshots(ticker)
    except Exception as e:
        print(str(e))
        continue
    ticker_snapshots = np.c_[np.array(ticker_snapshots), 
                        np.zeros(len(ticker_snapshots))]
    dataset.append(ticker_snapshots)

dataset = np.vstack([a for a in dataset if a.shape[1] == 101])

The first 10 rows of dataset are presented below. Note how the last column is always 1 – this is because the first half of the dataset consists of the top market cap tickers (classified as 1).

array([[35.639999, 35.610001, 39.220001, ..., 37.189999, 37.060001,
         1.      ],
       [40.369999, 40.369999, 40.419998, ..., 36.189999, 35.889999,
         1.      ],
       [37.810001, 37.849998, 38.419998, ..., 34.68    , 34.959999,
         1.      ],
       ...,
       [41.369999, 40.02    , 39.860001, ..., 35.279999, 35.27    ,
         1.      ],
       [38.5     , 38.669998, 38.599998, ..., 41.169998, 41.27    ,
         1.      ],
       [35.360001, 35.439999, 36.34    , ..., 42.740002, 43.349998,
         1.      ]])

We have now finished preparing the raw data for analysis. However, we still need to extract the features and preprocess the data for machine learning specifically. This will be covered in the next section.

Methodology

It is a remarkable fact of mathematics that periodic functions (at least the well-behaved ones) can be decomposed into sines and cosines. Although price data certainly isn’t periodic, if we treat the whole time series as one period, we can apply a Discrete Fourier Transform (DFT) to extract the main ‘signals’ in a time series. My motivation for using a DFT is that it can be used to de-noise a time series by ignoring the smaller terms, and that the coefficients of the resulting terms can be fed into a classifier. Additionally, there exist very efficient implementations in numpy, such as np.fft.fft(). One potential problem is that the results of the DFT are actually complex numbers:

array([ 3.86450001e+02+0.j        , -7.77637336e+00+2.22326054j,
       -3.85445493e+00+2.69405956j, -2.39862764e+00+4.72442769j,
       -1.07054807e+00+1.46787384j,  1.49997000e-01+0.j        ,
       -1.07054807e+00-1.46787384j, -2.39862764e+00-4.72442769j,
       -3.85445493e+00-2.69405956j, -7.77637336e+00-2.22326054j])

This begs the question as to how we can input the DFT terms into a classifier. We will implement and compare two possible methods.

Taking the modulus (np.abs) of each term and using that as a feature
Splitting up the real and imaginary parts to use as separate features.

Clearly, we would expect the latter to have more predictive power, but at the cost of using twice as many terms.

There is another decision we have to make, namely, how many terms of the DFT we want to use. As it is impossible to know a priori how much information the Fourier terms hold, we’ll examine the first 5, 15, and 30 terms.

So in total, we have 6 datasets to test. Now, in order to avoid the data mining bias, we will apply a Bonferroni correction. The idea is that each additional test increases our chance of finding a spurious yet statistically significant relationship, so to counteract this, the significance level must be decreased to $\frac{\alpha}{m}$, where $m$ is the number of tests. So if the original significance level was $\alpha = 0.05$, the Bonferroni correction would imply a new significance of $\frac{\alpha}{6} = 0.00833$.

Feature extraction and data preprocessing

Adopting standard scikit-learn notation, X will denote the feature array and y the target vector. Using the dataset that we prepared before:

X = dataset[:,:-1]
y = dataset[:, -1]

Prior to putting our data through a DFT, I am first going to standardise the time series (scale everything to the range [0,1]) to ensure that we don’t actually learn some boring correlation between the share price of a company and its market cap. To maximise the efficiency of this computation, bearing in mind that this computation needs to be done for more than 20k rows, it is important to (ab)use numpy’s broadcasting:

numerator = X - X.min(axis=1).reshape((len(X), 1))
denominator = X.max(axis=1) - X.min(axis=1)
denominator = denominator.reshape((len(X),1))
X_scaled = numerator/denominator

We will now define functions to take the Fourier Transform, then slice for the first k terms, before either taking the modulus or splitting into real and imaginary parts.

def generate_modulus_dataset(X, k):
    return np.abs(np.fft.fft(X)[:, :k])

def generate_complex_dataset(X, k):
    fourier = np.fft.fft(X)[:, :k]
    return np.column_stack((fourier.real, fourier.imag))

With these methods ready, creating all of the datasets is simple:

modulus_5 = generate_modulus_dataset(X_scaled, 5)
modulus_15 = generate_modulus_dataset(X_scaled, 15)
modulus_30 = generate_modulus_dataset(X_scaled, 30)
complex_5 = generate_complex_dataset(X_scaled, 5)
complex_15 = generate_complex_dataset(X_scaled, 15)
complex_30 = generate_complex_dataset(X_scaled, 30)

Machine Learning

We are finally ready to apply machine learning using python’s wonderfully intuitive sklearn. I have chosen to use an XGBoost classifier – it trains quickly and normally produces good results with the default parameters. It should be noted that no hyperparameter optimisation will be done: the point of this experiment is to see if this problem is learnable (rather than trying to achieve excellent results). I have written the classification script as a function which can be easily applied to each of the datasets.

from sklearn.model_selection import train_test_split
from xgboost import XGBClassifier
from sklearn.metrics import precision_score, accuracy_score


def classify(features, target, return_acc=False):
    X_train, X_test, y_train, y_test = train_test_split(
                                            features, target)
    clf = XGBClassifier()
    clf.fit(X_train, y_train)
    y_pred = clf.predict(X_test)

    acc = accuracy_score(y_test, y_pred)
    prec = precision_score(y_test, y_pred)
    if return_acc:
        return acc
    print(f"Accuracy:\n {100* acc:.2f}%")
    print(f"Precision:\n {100* prec:.2f}%")

It is then a simple matter of applying the function to any of our datasets:

classify(modulus_5, y)

Running the above resulted in a cross-validated accuracy of 58%. Given balanced classes, this is clearly a significant difference, so in principle the experiment could be stopped here and we can conclude that classifying stock charts based on market cap is a learnable problem. This result was especially surprising to me because this particular example is just using the moduli of the first 5 terms of the Fourier Transform, which really ignores a lot of the price chart. To see what I’m talking about, this is what a time series made of the absolute values of the first 5 Fourier terms looks like:

Although you can see that some of the main trends are captured, it is clearly a crude approximation that ignores a lot of variation.

Despite having achieved our initial goal of demonstrating the learnability of this problem, seeing as we have already prepared the datasets it may be interesting to do further analysis.

Comparison with a benchmark

What happens if we try to naively classify based on the whole time series, i.e using each of the 100 days’ prices as a feature?

%%time
classify(X_scaled, y)

Accuracy:
 65.34%
Precision:
 64.48%
CPU times: user 18.6 s, sys: 135 ms, total: 18.7 s
Wall time: 19.2 s

So actually the naive benchmark has a much better accuracy. But note the relatively long compute time of 18.7s. The question is whether any of our other datasets can reach comparable accuracies more efficiently.

%%time
classify(complex_5, y)

Accuracy:
 60.96%
Precision:
 60.17%
CPU times: user 1.96 s, sys: 18.5 ms, total: 1.97 s
Wall time: 2.04 s

It makes sense that the complex dataset has a higher accuracy than the modulus dataset for the same number of Fourier terms: bear in mind that it includes twice as many features (since we have split the real and imaginary part).

%%time
classify(modulus_15, y)

Accuracy:
 59.45%
Precision:
 57.72%
CPU times: user 3.77 s, sys: 62.8 ms, total: 3.83 s
Wall time: 4.33 s

This is an interesting result. 15 modulus terms (15 features) underperform 5 complex terms (i.e 10 features). This tells us that the actual complex number holds a lot of information.

%%time
classify(complex_15, y)

Accuracy:
 64.97%
Precision:
 63.75%
CPU times: user 3.02 s, sys: 26.1 ms, total: 3.04 s
Wall time: 3.09 s

Using 15 complex terms (i.e 30 total features), comparable accuracy to the benchmark has been achieved, in about a sixth of the time!

%%time
classify(modulus_30, y)

Accuracy:
 59.74%
Precision:
 57.98%
CPU times: user 3.16 s, sys: 35.1 ms, total: 3.19 s
Wall time: 3.33 s

%%time
classify(complex_30, y)

Accuracy:
 66.63%
Precision:
 65.24%
CPU times: user 11.9 s, sys: 136 ms, total: 12.1 s
Wall time: 12.6 s

We can observe the diminishing returns: using more Fourier terms increases accuracy only slightly. However, it is a pretty remarkable result that using 30 complex terms (i.e 60 features), we can beat the benchmark while also being 1.5x faster.

To better visualise how accuracy varies with the number of terms, we can generate multiple datasets and run our classification method on them:

scores = []
for i in range(0, 100, 5):
    print(i)
    dataset = generate_modulus_dataset(X_scaled, i)
    acc = classify(dataset, y, return_acc=True)
    scores.append(acc)

Doing the same for the complex datasets and plotting the results yields:

It is interesting to note that the modulus datasets never beat the benchmark: it seems that taking the modulus of the complex terms throws away too much information. The complex datasets are most predictive at around 40 terms, but the plateau and gradual decrease after that is a clear sign that overfitting occurs.

Conclusion

In this post, we have seen that we can determine whether a given 100-day time series of stock prices belongs to a high market cap or low market cap stock, with more than 65% accuracy. This was done by extracting the main signals from a time series with a Discrete Fourier Transform, which were then used as features in a classifier. The DFT is not only able to outperform the naive solution of using each day’s price as a feature, but can also do so with a large speedup. The implication of this is that the benchmark overfits to noise, and thus this is a practical example of how reducing the information available to a classifier can actually improve performance.

I think this is an important lesson for anyone applying machine learning to a real-world problem: throwing data at a classifier or deep learning model is not a solid approach. First try to extract important features or at least discard those that only contribute noise.

Future work on this topic could involve comparing the classification performance of the DFT method to a classifier that uses different summary statistics, or perhaps even a clustering methodology like k-Nearest Neighbours. But for now I am satisfied that this simple and intuitive method is sufficient to do significantly better than random guessing.

DIY MachineLearningStocks

2018-03-10T00:00:00+00:00

I recently released a machine learning stock prediction project on GitHub, unimaginatively named MachineLearningStocks. It is a project that I’ve put quite a lot of time into, and is in fact a simplified version of a system that I’ve been using to live trade. This post doesn’t really offer anything on top of the existing readme, but I figured it would be good to have a copy (with some minor changes) here as well.

MachineLearningStocks is designed to be an intuitive and highly extensible template project applying machine learning to making stock predictions. My hope is that the project will help readers to not only understand the overall workflow of using machine learning to predict stock movements, but also to appreciate some of the many subtleties. I’ve also provided a long list of possible improvements you could make, or ideas for moving forward, to put some spice into the vanilla project.

Concretely, we will be cleaning and preparing a dataset of historical stock prices and fundamentals using pandas, after which we will apply a scikit-learn classifier to discover the relationship between stock fundamentals (e.g. PE ratio, debt/equity, float, etc) and the subsequent annual price change (compared with the an index). We then conduct a simple backtest, before generating predictions on current data.

While I would not live trade based off of the predictions from this exact code, I do believe that you can use this project as starting point for a profitable trading system – I have actually used code based on this project to live trade, with pretty decent results (around 20% returns on backtest and 10-15% on live trading).

This project has quite a lot of personal significance for me. It was my first proper python project, one of my first real encounters with ML, and the first time I used git. At the start, my code was rife with bad practice and inefficiency: I have since tried to amend most of this, but please be warned that some minor issues may remain (feel free to raise an issue, or fork and submit a PR). Both the project and myself as a programmer have evolved a lot since the first iteration, but there is always room to improve.

As a disclaimer, this is a purely educational project. Be aware that backtested performance may often be deceptive – trade at your own risk!

Star

Overview
Quickstart
Preliminaries
Historical data
- Historical stock fundamentals
- Historical price data
Creating the training dataset
Backtesting
Current fundamental data
Stock prediction
Unit testing
Where to go from here
Contributing

Overview

The overall workflow to use machine learning to make stocks prediction is as follows:

Acquire historical fundamental data – these are the features or predictors
Acquire historical stock price data – this is will make up the dependent variable, or label (what we are trying to predict).
Preprocess data
Use a machine learning model to learn from the data
Backtest the performance of the machine learning model
Acquire current fundamental data
Generate predictions from current fundamental data

This is a very generalised overview, but in principle this is all you need to build a fundamentals-based ML stock predictor.

Quickstart

For readers who want to throw away the instruction manual and play immediately, just clone the project, then download and unzip the data file into the same directory. Then, open an instance of terminal and cd to the project’s file path, e.g

cd Users/User/Desktop/MachineLearningStocks

Then, run the following in terminal:

pip install -r requirements.txt
python download_historical_prices.py
python parsing_keystats.py
python backtesting.py
python current_data.py
pytest -v
python stock_prediction.py

Otherwise, follow the step-by-step guide below.

Preliminaries

This project uses python 3, and the common data science libraries pandas and scikit-learn. A full list of requirements is included in the requirements.txt file. To install all of the requirements at once, run the following code into terminal:

pip install -r requirements.txt

To get started, clone this project and unzip it. This will become our working directory, so make sure you cd your terminal instance into this directory.

Historical data

Data acquisition and preprocessing is probably the hardest part of most machine learning projects. But it is a necessary evil, so it’s best to not fret and just carry on.

For this project, we need three datasets:

Historical stock fundamentals
Historical stock prices
Historical S&P500 prices

We need the S&P500 index prices as a benchmark: a 5% stock growth does not mean much if the S&P500 grew 10% in that time period, so all stock returns must be compared to those of the index.

Historical stock fundamentals

Historical fundamental data is actually very difficult to find (for free, at least). Although sites like Quandl do have datasets available, you often have to pay a pretty steep fee.

It turns out that there is a way to parse this data, for free, from Yahoo Finance. I will not go into details, because Sentdex has done it for us. On his page you will be able to find a file called intraQuarter.zip, which you should download, unzip, and place in your working directory. Relevant to this project is the subfolder called _KeyStats, which contains html files that hold stock fundamentals for all stocks in the S&P500 between 2003 and 2013, sorted by stock. However, at this stage, the data is unusable – we will have to parse it into a nice csv file before we can do any ML.

Historical price data

In the first iteration of the project, I used pandas-datareader, an extremely convenient library which can load stock data straight into pandas. However, after Yahoo Finance changed their UI, datareader no longer worked, so I switched to Quandl, which has free stock price data for a few tickers, and a python API. However, as pandas-datareader has been fixed, we will use that instead.

Likewise, we can easily use pandas-datareader to access data for the SPY ticker. Failing that, one could manually download it from yahoo finance, place it into the project directory and rename it sp500_index.csv.

The code for downloading historical price data can be run by entering the following into terminal:

python download_historical_prices.py

Creating the training dataset

Our ultimate goal for the training data is to have a ‘snapshot’ of a particular stock’s fundamentals at a particular time, and the corresponding subsequent annual performance of the stock.

For example, if our ‘snapshot’ consists of all of the fundamental data for AAPL on the date 28/1/2005, then we also need to know the percentage price change of AAPL between 28/1/05 and 28/1/06. Thus our algorithm can learn how the fundamentals impact the annual change in the stock price.

In fact, this is a slight oversimplification. In fact, what the algorithm will eventually learn is how fundamentals impact the outperformance of a stock relative to the S&P500 index. This is why we also need index data.

Preprocessing historical price data

When pandas-datareader downloads stock price data, it does not include rows for weekends and public holidays (when the market is closed).

However, referring to the example of AAPL above, if our snapshot includes fundamental data for 28/1/05 and we want to see the change in price a year later, we will get the nasty surprise that 28/1/2006 is a Saturday. Does this mean that we have to discard this snapshot?

By no means – data is too valuable to callously toss away. As a workaround, I instead decided to ‘fill forward’ the missing data, i.e we will assume that the stock price on Saturday 28/1/2006 is equal to the stock price on Friday 27/1/2006.

Features

Below is a list of some of the interesting variables that are available on Yahoo Finance.

Valuation measures

‘Market Cap’
Enterprise Value
Trailing P/E
Forward P/E
PEG Ratio
Price/Sales
Price/Book
Enterprise Value/Revenue
Enterprise Value/EBITDA

Financials

Profit Margin
Operating Margin
Return on Assets
Return on Equity
Revenue
Revenue Per Share
Quarterly Revenue Growth
Gross Profit
EBITDA
Net Income Avi to Common
Diluted EPS
Quarterly Earnings Growth
Total Cash
Total Cash Per Share
Total Debt
Total Debt/Equity
Current Ratio
Book Value Per Share
Operating Cash Flow
Levered Free Cash Flow

Trading information

Beta
50-Day Moving Average
200-Day Moving Average
Avg Vol (3 month)
Shares Outstanding
Float
% Held by Insiders
% Held by Institutions
Shares Short
Short Ratio
Short % of Float
Shares Short (prior month)

Parsing

However, all of this data is locked up in HTML files. Thus, we need to build a parser. In this project, I did the parsing with regex, but please note that generally it is really not recommended to use regex to parse HTML. However, I think regex probably wins out for ease of understanding (this project being educational in nature), and from experience regex works fine in this case.

This is the exact regex used:

r'>' + re.escape(variable) + r'.*?(\-?\d+\.*\d*K?M?B?|N/A[\\n|\s]*|>0|NaN)%?(</td>|</span>)'

While it looks pretty arcane, all it is doing is searching for the first occurence of the feature (e.g. “Market Cap”), then it looks forward until it finds a number immediately followed by a </td> or </span> (signifying the end of a table entry). The complexity of the expression above accounts for some subtleties in the parsing:

the numbers could be preceeded by a minus sign
Yahoo Finance sometimes uses K, M, and B as abbreviations for thousand, million and billion respectively.
some data are given as percentages
some datapoints are missing, so instead of a number we have to look for “N/A” or “NaN.

Both the preprocessing of price data and the parsing of keystats are included in parsing_keystats.py. Run the following in your terminal:

python parsing_keystats.py

You should see the file keystats.csv appear in your working directory. Now that we have the training data ready, we are ready to actually do some machine learning.

Backtesting

Backtesting is arguably the most important part of any quantitative strategy: you must have some way of testing the performance of your algorithm before you live trade it.

Despite its importance, I originally did not want to include backtesting code in this repository. The reasons were as follows:

Backtesting is messy and empirical. The code is not very pleasant to use, and in practice requires a lot of manual interaction.
Backtesting is very difficult to get right, and if you do it wrong, you will be deceiving yourself with high returns.
Developing and working with your backtest is probably the best way to learn about machine learning and stocks – you’ll see what works, what doesn’t, and what you don’t understand.

Nevertheless, because of the importance of backtesting, I decided that I can’t really call this a ‘template machine learning stocks project’ without backtesting. Thus, I have included a simplistic backtesting script. Please note that there is a fatal flaw with this backtesting implementation that will result in much higher backtesting returns. It is quite a subtle point, but I will let you figure that out :)

Run the following in terminal:

python backtesting.py

You should get something like this:

Classifier performance
======================
Accuracy score:  0.81
Precision score:  0.75

Stock prediction performance report
===================================
Total Trades: 177
Average return for stock predictions:  37.8 %
Average market return in the same period:  9.2%
Compared to the index, our strategy earns  28.6 percentage points more

Again, the performance looks too good to be true and almost certainly is.

Current fundamental data

Now that we have trained and backtested a model on our data, we would like to generate actual predictions on current data.

As always, we can scrape the data from good old Yahoo Finance. My method is to literally just download the statistics page for each stock (here is the page for Apple), then to parse it using regex as before.

In fact, the regex should be almost identical, but because Yahoo has changed their UI a couple of times, there are some minor differences. This part of the projet has to be fixed whenever yahoo finance changes their UI, so if you can’t get the project to work, the problem is most likely here.

Run the following in terminal:

python current_data.py

The script will then begin downloading the HTML into the forward/ folder within your working directory, before parsing this data and outputting the file forward_sample.csv. You might see a few miscellaneous errors for certain tickers (e.g. ‘Exceeded 30 redirects.’), but this is to be expected.

Stock prediction

Now that we have the training data and the current data, we can finally generate actual predictions. This part of the project is very simple: the only thing you have to decide is the value of the OUTPERFORMANCE parameter (the percentage by which a stock has to beat the S&P500 to be considered a ‘buy’). I have set it to 10 by default, but it can easily be modified by changing the variable at the top of the file. Go ahead and run the script:

python stock_prediction.py

You should get something like this:

21 stocks predicted to outperform the S&P500 by more than 10%:
NOC FL SWK NFX LH NSC SCHL KSU DDS GWW AIZ ORLY R SFLY SHW GME DLX DIS AMP BBBY APD

Unit testing

I have included a number of unit tests (in the tests/ folder) which serve to check that things are working properly. However, due to the nature of the some of this projects functionality (downloading big datasets), you will have to run all the code once before running the tests. Otherwise, the tests themselves would have to download huge datasets (which I don’t think is optimal).

I thus recommend that you run the tests after you have run all the other scripts (except, perhaps, stock_prediction.py).

To run the tests, simply enter the following into a terminal instance in the project directory:

pytest -v

Please note that it is not considered best practice to include an __init__.py file in the tests/ directory (see here for more), but I have done it anyway because it is uncomplicated and functional.

Where to go from here

I have stated that this project is extensible, so here are some ideas to get you started and possibly increase returns (no promises).

Data acquisition

My personal belief is that better quality data is THE factor that will ultimately determine your performance. Here are some ideas:

Explore the other subfolders in Sentdex’s intraQuarter.zip.
Parse the annual reports that all companies submit to the SEC (have a look at the Edgar Database)
Try to find websites from which you can scrape fundamental data (this has been my solution).
Ditch US stocks and go global – perhaps better results may be found in markets that are less-liquid. It’d be interesting to see whether the predictive power of features vary based on geography.
Buy Quandl data, or experiment with alternative data.

Data preprocessing

Build a more robust parser using BeautifulSoup
In this project, I have just ignored any rows with missing data, but this reduces the size of the dataset considerably. Are there any ways you can fill in some of this data?
- hint: if the PE ratio is missing but you know the stock price and the earnings/share…
- hint 2: how different is Apple’s book value in March to its book value in June?
Some form of feature engineering
- e.g, calculate Graham’s number and use it as a feature
- some of the features are probably redundant. Why not remove them to speed up training?
Speed up the construction of keystats.csv.
- hint: don’t keep appending to one growing dataframe! Split it into chunks

Machine learning

Altering the machine learning stuff is probably the easiest and most fun to do.

The most important thing if you’re serious about results is to find the problem with the current backtesting setup and fix it. This will likely be quite a sobering experience, but if your backtest is done right, it should mean that any observed outperformance on your test set can be traded on (again, do so at your own discretion).
Try a different classifier – there is plenty of research that advocates the use of SVMs, for example. Don’t forget that other classifiers may require feature scaling etc.
Hyperparameter tuning: use gridsearch to find the optimal hyperparameters for your classifier. But make sure you don’t overfit!
Make it deep – experiment with neural networks (an easy way to start is with sklearn.neural_network).
Change the classification problem into a regresion one: will we achieve better results if we try to predict the stock price rather than whether it outperformed?
Run the prediction multiple times (perhaps using different hyperparameters?) and select the k most common stocks to invest in. This is especially important if the algorithm is not deterministic (as is the case for Random Forest)
Experiment with different values of the outperformance parameter.
Try to plot the importance of different features to ‘see what the machine sees’.

Contributing

Feel free to fork, play around, and submit PRs. I would be very grateful for any bug fixes or more unit tests.

This project was originally based on Sentdex’s excellent machine learning tutorial, but it has since evolved far beyond that and the code is almost completely different. The complete series is also on his website.

Creating a stock price database with MariaDB and python

2018-02-01T00:00:00+00:00

One of my interests is exploring the applications of machine learning to financial markets. As part of this hobby, I’ve spent many more hours parsing and processing data than I have actually applying machine learning. I’ve worked broadly with two datasets in particular: historical financial statistics (e.g. P/E ratio, price/book) make up the features that my algorithms learn from, but the actual backbone of any strategy is historical price data.

My main data source has been Yahoo Finance. Although they’ve deprecated their official API, they still have the same data on their website, meaning that it can be scraped if you can be bothered. I discovered a crude but functional way of doing this (detailed in this post), but then discovered an extremely convenient python library that does the same thing much more efficiently, with a direct pandas-datareader interface.

However, it remained a concern for me that one day the winds would change and Yahoo Finance would deprecate this hidden API permanently. So I decided to make a hoard of this data, in the form of a stock price database. I think creating your own securities database is an important step for anyone looking to get into algorithmic investing more seriously, so I’ve decided to share how I’ve done so. As this is my first financial database, there may be inefficiencies in the schema, but overall I believe that the solution presented here is relatively robust, and definitely sufficient for my purposes.

Edit as of Feb 2022: I probably wouldn’t recommend following the steps in this post – there are better resources elsewhere. Nevertheless, I’m leaving this post up for personal nostalgia.

Choosing a database management system
Prerequisites
Database schema
Populating the database
Conclusion

Choosing a database management system

When it comes to choosing a database system, there are a somewhat distressing number of decisions that you have to make. Do you want a relational database or NoSQL? If you choose relational, which system are you going to go with? Within that system, what storage engine should you use?

I think this chart from nuodb sums up the various options quite well:

It was quite clear to me that a SQL relational database was what I wanted, after all, price data is highly structured and I need very quick read speed. But even after narrowing it down to a traditional RDBMS, you still have to choose the exact system. A quick look cut my options down to SQLite, MySQL, or PostgreSQL.

I decided to go with MySQL, because I felt that I didn’t need the application-embedding that SQLite offered, nor did I need all the advanced features of PostgreSQL. In the end though, I chose MariaDB, which is an open-source fork of MySQL that seemed to offer all of the features that MySQL did, with a few minor improvements (see an interesting discussion here). Lastly, you have to choose a storage engine. The default for anything after MariaDB 10.2 is InnoDB, and I felt that the burden of proof was on the alternatives to demonstrate superiority for my purposes. They didn’t do so, hence I stuck with InnoDB. The official documentation gives a very clear exposition on the topic of storage engines.

Prerequisites

A complete guide to setting up MariaDB/MySQL is outside of the scope of this post. Please refer to the official webpages for more. However, if you’re on a mac and already have homebrew, it’s really quite easy:

brew install mariadb

Then it should be a simple matter of starting the MariaDB server and logging in:

mysql.server start
mysql -u root

MySQL and MariaDB are pretty much interchangeable (at least for now), which is why you will interact with MariaDB via mysql commands.

The requirements for this project, which can be installed easily via pip, are as follows:

pandas==0.22.0
PyMySQL==0.8.0
pandas_datareader==0.5.0
fix_yahoo_finance==0.0.21

Lastly, although it’s completely up to you, I would recommend having some other GUI database software on your computer to help visualise things. I like Sequel Pro (macOS only), which is open-source, elegant, and designed specifically for MySQL/MariaDB.

Database schema

A database schema sets out how all of the information is going to be organised in your database. Optimal schema design is a really huge topic that is often the subject of university courses, as well as being a typical interview task for prospective database administrators. I wasn’t naive enough to think that I could come up with the perfect schema from nothing, so I took to google to learn from other people’s mistakes. This was especially important because I knew I would end up with quite a lot of data: I was imagining at least 15 years of daily data for 5000 tickers, which is about 30 million rows. There are lots of conflicting opinions on financial database schema, but in the end I decided to follow the advice of an article from Quantstart. The main advantage of their proposed schema over some of the other suggestions I saw online is flexibility: it really makes no assumptions about what securities you’re interested in. As long as they can be represented as a time series of prices, they’ll fit into the database nicely.

Before getting started generating the tables, we mustn’t forget to generate the database! After logging in to MariaDB, just run the following in the console:

CREATE DATABASE stock_prices;

We can then proceed to generate the tables. As per the linked article, we will have four tables: exchange, security, data_vendor, and daily_price.

Just a quick note: if you’re worried about the eventual size of the database, you could consider setting up on an external drive. On unix systems, there is a simple ‘hack’ for this: navigate to your MySQL folder (for me this was /usr/local/var/mysql), find the correct database folder, then drag it to your external drive. Now, create a symbolic link in the mysql directory, by running in terminal:

ln -s [path to drive] [database folder path]

However, I think in most cases this is unnecessary: my 23 million datapoints only take up 3.65 GB.

Table: exchange

The exchange table contains information (the name and currency-traded) of the exchange. This is a parent of table security.

CREATE TABLE IF NOT EXISTS `exchange` (
  `id` INT(11) NOT NULL AUTO_INCREMENT,
  `name` VARCHAR(10) NOT NULL,
  `currency` CHAR(3) NULL DEFAULT NULL,
  `created_date` DATETIME NULL DEFAULT CURRENT_TIMESTAMP(),
  `last_updated` DATETIME NULL DEFAULT CURRENT_TIMESTAMP() ON UPDATE CURRENT_TIMESTAMP(),
  PRIMARY KEY (`id`))
ENGINE = InnoDB
DEFAULT CHARACTER SET = utf8;

Table: security

security holds all of the relevant information about the actual companies for which we will collect data. Actually the only thing we really need here is the ticker (and which exchange it is traded on), but I thought it might be nice to have some other data like the sector and industry.

DROP TABLE IF EXISTS `security` ;

CREATE TABLE IF NOT EXISTS `security` (
  `id` INT(11) NOT NULL AUTO_INCREMENT,
  `exchange_id` INT(11) NOT NULL,
  `ticker` VARCHAR(10) NOT NULL,
  `name` VARCHAR(100) NULL,
  `sector` VARCHAR(100) NULL,
  `industry` VARCHAR(100) NULL,
  `created_date` DATETIME NULL DEFAULT CURRENT_TIMESTAMP(),
  `last_updated` DATETIME NULL DEFAULT CURRENT_TIMESTAMP() ON UPDATE CURRENT_TIMESTAMP(),
  PRIMARY KEY (`id`),
  INDEX `exchange_id` (`exchange_id` ASC),
  INDEX `ticker` (`ticker` ASC),
  CONSTRAINT `fk_exchange_id`
    FOREIGN KEY (`exchange_id`)
    REFERENCES `exchange` (`id`)
    ON DELETE NO ACTION
    ON UPDATE NO ACTION)
ENGINE = InnoDB
DEFAULT CHARACTER SET = utf8;

The main features of interest are the foreign key on the exchange_id, and the fact that we have added an index on ticker.

Table: data_vendor

Because I was planning to get all of my data from Yahoo Finance, I didn’t think I would need a table like this. However, I suppose that as I cast my net wider in future, it may be useful to have this flexibility.

DROP TABLE IF EXISTS `data_vendor` ;

CREATE TABLE IF NOT EXISTS `data_vendor` (
  `id` INT(11) NOT NULL AUTO_INCREMENT,
  `name` VARCHAR(32) NOT NULL,
  `website_url` VARCHAR(255) NULL DEFAULT NULL,
  `created_date` DATETIME NULL DEFAULT CURRENT_TIMESTAMP(),
  `last_updated` DATETIME NULL DEFAULT CURRENT_TIMESTAMP() ON UPDATE CURRENT_TIMESTAMP(),
  PRIMARY KEY (`id`))
ENGINE = InnoDB
AUTO_INCREMENT = 3
DEFAULT CHARACTER SET = utf8;

Table: daily_price

This is the real heart of the database. In fact, we could probably denormalise and squish the other tables into this one, but in accordance with the Zen of Python:

Explicit is better than implicit. Sparse is better than dense.

DROP TABLE IF EXISTS `daily_price` ;

CREATE TABLE IF NOT EXISTS `daily_price` (
  `id` INT(11) NOT NULL AUTO_INCREMENT,
  `data_vendor_id` INT(11) NOT NULL,
  `ticker_id` INT(11) NOT NULL,
  `price_date` DATE NOT NULL,
  `created_date` DATETIME NULL DEFAULT CURRENT_TIMESTAMP(),
  `last_updated` DATETIME NULL DEFAULT CURRENT_TIMESTAMP() ON UPDATE CURRENT_TIMESTAMP(),
  `open_price` DECIMAL(11,6) NULL DEFAULT NULL,
  `high_price` DECIMAL(11,6) NULL DEFAULT NULL,
  `low_price` DECIMAL(11,6) NULL DEFAULT NULL,
  `close_price` DECIMAL(11,6) NULL DEFAULT NULL,
  `adj_close_price` DECIMAL(11,6) NULL DEFAULT NULL,
  `volume` BIGINT(20) NULL DEFAULT NULL,
  PRIMARY KEY (`id`),
  INDEX `price_date` (`price_date` ASC),
  INDEX `ticker_id` (`ticker_id` ASC),
  CONSTRAINT `fk_ticker_id`
    FOREIGN KEY (`ticker_id`)
    REFERENCES `security` (`id`)
    ON DELETE NO ACTION
    ON UPDATE NO ACTION,
  CONSTRAINT `fk_data_vendor_id`
    FOREIGN KEY (`data_vendor_id`)
    REFERENCES `data_vendor` (`id`)
    ON DELETE NO ACTION
    ON UPDATE NO ACTION)
ENGINE = InnoDB
DEFAULT CHARACTER SET = utf8;

Notice the indexes (I believe ‘indices’ is incorrect in this context) on price_date and ticker_id: these are important, because for the most part I know I will be wanting to select stock prices either by date or by ticker.

Putting it all together, we have the following ER diagram:

Populating the database

Let’s start by connecting to the database via python. We could do this via an ORM, but it might be a bit simpler to use a MySQL interface. As seems to be a recurring theme, there are many options for interacting with MariaDB from python. In previous projects I had used MySQLdb, which is generally straightforward but a hassle to install. In order to make this tutorial simpler, I’ve chosen to use a popular pure-python alternative, PyMySQL – functionality is almost identical, but it is much easier to install. Let’s proceed. The top of our python file will begin with the relevant imports.

import time
import pandas as pd
from pandas_datareader import data as pdr
import pymysql.cursors
import fix_yahoo_finance as yf
yf.pdr_override()

Connecting to the database is very simple; if you are having a problem here, it is likely because of the MySQL login details.

conn = pymysql.connect(host='localhost', 
                       user='root', 
                       db='stock_prices')
cursor = conn.cursor()

Just a quick note: it is quite difficult to format the code properly for these blog posts, so if you are copy-pasting (which I don’t recommend!) then please be aware that there are some forced line breaks below that will raise errors in python (particularly within long SQL statements).

Adding an exchange and a list of securities

We will start by manually writing an exchange to the database:

sql = "INSERT INTO exchange (name, currency) VALUES ('NYSE', 'USD')"
cursor.execute(sql)
conn.commit()

As a rule of thumb, after every set of like database operations (e.g. after you’ve added/deleted a few things), you should throw in a conn.commit() which makes the changes permanent.

After adding the exchange, we need to give it some children (i.e securities). Luckily, there are official lists in csv format containing all of the tickers in the NYSE, NASDAQ and AMEX. Head to that link and download the NYSE list. Within my project directory, I put this in a subfolder called data, and named it nyse_tickers.csv. Parsing these tickers is not very difficult:

nyse = pd.read_csv('data/nyse_tickers.csv')
print("Number of NYSE tickers:", len(nyse))
nyse.drop(['LastSale', 'MarketCap', 'IPOyear', 'Summary Quote',
           'Unnamed: 9', 'ADR TSO'], axis=1, inplace=True)
nyse.columns = ['ticker', 'name', 'sector', 'industry']
nyse['exchange_id'] = 1
nyse = nyse[cols[-1:] + cols[:-1]]

Notice that I’ve done some weird manipulations with the column names, and I’ve added a new constant column corresponding to the exchange ID of NYSE (in our exchange table). This allows for an elegant snippet to add the data to our database:

for row in nyse.itertuples(index=False):
    try:
        cursor.execute("INSERT INTO security (exchange_id, 
        ticker, name, sector, industry) VALUES (%s, %s, %s, %s, %s)", 
        row)
    except:
        # Assume that the exception is because sector
        # and/or industry are missing
        cursor.execute("INSERT INTO security (exchange_id, 
        ticker, name) VALUES (%s, %s, %s)", row[:3])
conn.commit()

Adding price data

Before we add price data, we must manually add a data vendor:

sql = "INSERT INTO data_vendor (name, website_url) VALUES " + \
      "('YahooFinance', 'https://finance.yahoo.com')"
cursor.execute(sql)
conn.commit()

YAHOO_VENDOR_ID = 1

Next, we list the securities that have been added, so that we know the tickers for which we should download data.

all_tickers = pd.read_sql("SELECT ticker, id FROM security", conn)
ticker_index = dict(all_tickers.to_dict('split')['data'])
tickers = list(ticker_index.keys())

In principle, now all we have to do is iterate over the list of tickers, download data from pandas-datareader, then write to daily_price. This can be implemented very simply as follows:

for ticker in tickers:
    # Download data
    df = pdr.get_data_yahoo(ticker, start=start_date)
    # Write to daily_price
    for row in df.itertuples():
        values = [YAHOO_VENDOR_ID, ticker_index[ticker]] + list(row)
        cursor.execute("INSERT INTO daily_price (data_vendor_id, 
        ticker_id, price_date, open_price, high_price, low_price, 
        close_price, adj_close_price, volume) VALUES 
        (%s, %s, %s, %s, %s, %s, %s, %s, %s)",
        tuple(values))

In practice, however, we need to add some fault tolerance. For example, I have found that pandas-datareader inconsistently throttles. One solution is to split the download into chunks, and keep track of failed downloads. Throwing a couple of try/except statements in there too, we get the following two methods:

def download_data_chunk(start_idx, end_idx, tickerlist, 
                        start_date=None):
    """
    Download stock data using pandas-datareader
    :param start_idx: start index
    :param end_idx: end index
    :param tickerlist: which tickers are meant to be downloaded
    :param start_date: the starting date for each ticker
    :return: writes data to mysql database
    """
    ms_tickers = []
    for ticker in tickerlist[start_idx:end_idx]:
        df = pdr.get_data_yahoo(ticker, start=start_date)
        if df.empty:
            print(f"df is empty for {ticker}")
            ms_tickers.append(ticker)
            time.sleep(3)
            continue

        for row in df.itertuples():
            values = [YAHOO_VENDOR_ID, ticker_index[ticker]] + \
                    list(row)
            try:
                sql = "INSERT INTO daily_price (data_vendor_id, 
                        ticker_id, price_date, open_price,
                        high_price, low_price, close_price, 
                        adj_close_price, volume) 
                        VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s)"
                cursor.execute(sql, tuple(values))
            except Exception as e:
                print(str(e))
        conn.commit()
        return ms_tickers


def download_all_data(tickerlist, chunk_size=100,
                      start_date=None):
    # Hacky snippet to get the ceiling
    n_chunks = -(-len(tickerlist) // chunk_size)

    ms_tickers = []
    for i in range(0, n_chunks, chunk_size):
        # This will download data from the earliest possible date
        ms_from_chunk = download_data_chunk(i, i+chunk_size, 
                                            tickerlist,
                                            start_date)
        ms_tickers.append(ms_from_chunk)
        
        # Check for possible throttling
        if len(ms_from_chunk) > 40:
            time.sleep(120)
        else:
            time.sleep(10)
    return ms_tickers

After this, you can do a second pass over the missing tickers to try to get as much data as possible.

Updating data

At specified intervals (perhaps via a cron job), you may want to update the prices in the database. Here’s a quick script to do that:

def update_prices():
    # Get present tickers
    present_ticker_ids = pd.read_sql("SELECT DISTINCT ticker_id 
                                    FROM daily_price", 
                                    conn)
    index_ticker = {v: k for k, v in ticker_index.items()}
    present_tickers = [index_ticker[i]
                       for i in list(present_ticker_ids['ticker_id'])]
    # Get last date
    sql = "SELECT price_date FROM daily_price WHERE ticker_id=1"
    dates = pd.read_sql(sql, conn)
    last_date = dates.iloc[-1, 0]
    download_all_data(present_tickers, start_date=last_date)

Conclusion

I hope this post has given you an idea of how you might go about creating your own securities database. In future, I’m quite keen to add to this database new tables for stock fundamental data as well, but this would be much more complicated unless I’m only interested in a specific list of features. Otherwise, NoSQL might be a better option.

But for now, I’m happy with the price database. Looking back, I think it was a very worthwhile time investment – I have already made quite significant use of the database and am enjoying the convenience. I can now prototype algorithms much faster, without the hassle of having to re-download data every time or find the csv files somewhere on my filesystem. Not to mention the fact that I won’t be as heavily affected by the whims of some Yahoo Finance team-leader who decides to change the UI again.

Learning Machine Learning

2017-12-28T00:00:00+00:00

Two years ago I was an absolute novice at machine learning: I had read around the subject a little bit, could probably rattle off a few of the buzzwords, and had some appreciation of the general idea, but there was no way I could have developed a predictive model beyond linear regression. I was somewhere near the peak of Mount Stupid (from a great chart by SMBC):

Fast forward to the present day – I’m just about escaping the local minimum on the other side. Although I definitely wouldn’t claim anything close to mastery of the subject, a few months ago I took a step back to look at my progress and realised that I was definitely not a beginner any longer: I had built a couple of extensive projects, created some educational resources, and started applying machine learning models to my professional work. I therefore think that I am in a reasonable place to share the mistakes I made when learning ML, so that other people don’t have to make them!

Edit as of Jan 2022: in retrospect, this post was probably written near the Dunning-Kruger peak. I will happily say that I am in one of the valleys, though I’m not sure which one. That said, I still stand by the advice in this post!

It is worth noting that two years is a very long time; I was far too occupied (with the army, among other things) to commit fully to machine learning, but I think that most people could achieve near-mastery in 6 months if they so pleased.

In any case, from the perspective of a fellow learner who has had a small head start, and without any guarantees of it being the most efficient way to learn the subject, I’m going to present some advice. The rough overview is as follows:

Learn to code
Learn the basics of ML
Pick a subject you are interested in
Pick an interesting algorithm, and dive deep
Review your achievements

For each subsection, I’ll try to provide some resource recommendations, and some tips.

Learn to code

For better or worse, programming is basically a prerequisite for machine learning. It is true that there are some GUI environments, most notably Weka and the Azure ML studio, but users of these tools are definitely in the minority. Programming gives you far more control over the whole machine learning process. I know from experience that ‘control’ is often the last thing a beginner wants, because with great power comes the responsibility to actually write functional code, but it’s honestly hard to appreciate ML without programming. The one exception I would cautiously make is if you have experience with statistical analysis (e.g. in Excel), in which case Azure might not be a bad place to start.

For those new to programming, a natural question is what language to use. This is a big decision, but spending too long deciding will hold you back. I really recommend just picking python 3 and sticking with it: it has a very mature environment for machine learning, with simple syntax and an abundance of tutorials online. It has the added benefit of being a general-purpose language, so you can really do anything you want with it.

I can’t remember exactly how I learnt python (it was a long time ago): I recall it being a dark and uncomfortable time, probably because I tried to navigate my own way through. I suggest you choose one of the many tutorials available. Here are three suggestions:

Automate the Boring Stuff with Python, by Al Sweigart. Very highly regarded, although it doesn’t directly address data science. It teaches the syntax, and just how useful python can be. Even if you know for a fact that you’re only interested in data science, the first 12 chapters are critical.
A Byte of Python. A great online book which teachers you quite a lot of the language. I used this a lot while I was learning.
Python course from Rice University. Comprehensive and beginner-friendly, but perhaps with an excessive focus on GUI programming.

Some tips:

You don’t need to know all that much about writing classes in python, at least for ML. Understanding the basic language features, along with pandas dataframes, is probably sufficient.
Please use python 3.
I’d recommend against trying to learn about ML and learning the python language at the same time.

Learn the basics of ML

Machine learning is a very mathematical subject, and though it’s better if you can learn it rigorously (with all the maths), it’s not necessary. But be aware that you will need a firm grasp of your high school maths if you want to progress further.

One of the seemingly unanimous recommendations is Andrew Ng’s Coursera course. My honest opinion is that this is likely to put you off ML unless you are a very quick learner (and good at maths). I didn’t finish the course, and after giving up, I avoided ML for a long time. One of the problems is that he uses Octave/Matlab as a programming language, which nobody uses. That being said, the course does give amazing insight (provided you can follow the maths), and he covers many relevant algorithms. If you feel up to it, it may be worth following along with the videos but doing all of the assignments in python. Then you can compare your work with other people’s solutions on GitHub.

My personal recommendation is Georgia Tech Machine Learning. This course spends a lot more time explaining the intuition behind the algorithms before going into the maths or the code, which is invaluable as a learner.

I also bought the Udemy Machine Learning course for $10. While Udemy has a pretty bad rep nowadays, this course is actually decent: it gives you a basic understanding of a wide range of algorithms. There was a time when this lack of depth frustrated me, but looking back, I realise how useful it is to be given a practical tour across the whole ML scene. To be fair, you can probably learn this directly from the scikit-learn documentation , but it does help to have video explanations. I’ve put my code for this course onto GitHub.

Tips:

If the maths is too hard, don’t worry about it. At this stage, it’s far more useful to understand intuitively how the algorithm works, and more importantly how to apply the algorithm in python. If you truly appreciate how useful ML can be, and become excited by it, you will be much more inclined to learn the maths.
Ensure that you pick up the necessary skills with numpy, pandas and sklearn. These libraries are fast, and knowing the common usage patterns will save you from reinventing the wheel.
Keep a folder with lots of different python files, each containing an example application of a particular machine learning algorithm to a dataset (dummy or real).

Pick a subject you are interested in

After acquiring the basic skills, it is critical to find a subject that you are interested in and to build a project. Machine learning is a widely applicable tool, and I’d wager that there’s an application of ML in whatever field you’re interested in.

Once you decide, it’s probably a good idea to look for a tutorial series on that particular topic. I was most interested in finance, so I followed an excellent youtube playlist from Sentdex, called Machine Learning for Investing.

At this stage, your goal is to build a fully working project, so that you come into contact with the whole machine learning stack, all the way from raw data to eventual predictions (and preferably some visualisations too). Performance isn’t the most important thing right now, though it would be nice if you can do better than random guessing :)

Tips:

Try not to copy and paste code, at the very least type it out. You’ll learn faster this way.
Don’t spend too long trying to figure out what project to do: indecision is the worst form of procrastination when it comes to learning new subjects. If you’re really struggling to think of something, try kaggle.com. Don’t be put off by the fact that it’s a ‘competition’ – it’s also a good place to learn, and lots of people share notebooks showing what they’ve tried. Seeing all these different approaches is useful for beginners, as you can quickly try experimenting with existing solutions to see what works and what doesn’t.

Pick an interesting algorithm, and dive deep

If you’ve followed the steps thus far, you should have a crude, but functional, project in a field that you are interested in. Now is the time to ascend to the next level, by really trying to understand what you are doing and improving your machine learning performance.

Learn about cross validation, particularly about the very important bias-variance tradeoff. Improve your project’s methodology.
Examine the different methods of sampling to make training, validation, and test sets.
Explore different performance metrics, for example the ROC curve. Which one is the most relevant for your problem? Learn about what the F-score is, and why it’s commonly used instead of accuracy.
Evaluate your feature importance, and maybe even try some feature selection/engineering.

After you’ve done all of this, it may be that your performance has actually gotten worse (for example, if you’ve changed to a more relevant metric). This was the case for me, as I changed from accuracy to precision – precision is much more pertinent to stock predictions because it’s the false positives that lose you money. But at least I had a logically sound methodology off of which to base future endeavours.

Now, you should do a quick search on Google Scholar to see what algorithms are most successful in your field. For example, if you chose to deal with computer vision, you’d find that Convolutional Neural Networks are the de facto standard. Then, really hit the books – try to understand the algorithm in detail, to the point where you could recreate it from scratch. This naturally involves a lot of maths, but hopefully at this stage you’re excited enough about ML to weather it. You may find The Elements of Statistical Learning to be useful: it is a legendary textbook that basically defines the field of machine learning. Supplement this with alternative explanations and tutorials from youtube or stackexchange, and you will be in a good place. In particular, you should look out for the actual meaning of all of the different metaparameters. This is very important, because it will guide your future search for optimal metaparameters to increase your performance.

In my case, SVMs and neural networks were common in the literature on stock prediction, but I wasn’t too keen to proceed with them. I found in a more recent study that boosted decision trees were having quite a lot of success in a number of different fields, so I was interested to see how they would do in stock prediction. I’ve written a post on my investigation into boosted trees, so that you can get an idea about how deep I went into the subject. I also found the seminal academic papers to be illuminating, but you should probably leave them for last.

Once you understand the algorithm, you should be able to go back to your original project with fresh eyes, applying your new knowledge to increase performance. But even if your performance won’t increase, you’ll have a much better idea as to why.

Review your achievements

Although it is true that there is always more to learn about a subject, sometimes you do need to take a step back and acknowledge that you’ve come a long way. Look over your projects, your notes, and take some pride in what you have learnt. I’m most proud of an extensive tutorial I made, MachineLearningStocks, which is meant to help beginners get started with applying ML to stock prediction.

Conclusion

Machine Learning is a subject that is not hard to pick up, but has a learning curve that steepens the further you get into it. Just being able to apply machine learning puts you ahead of most of the world. It is constantly evolving, with a startling amount of research released daily. But don’t let this put you off: with concerted effort and the right resources, it’s definitely feasible for you to contribute to that ever-expanding corpus.

Gradient tree boosting and XGBoost

2017-10-10T00:00:00+00:00

Decision trees make for pretty vanilla classifiers: they do an unspectacular job with most machine learning tasks, and you’d be forgiven for overlooking them when deciding on a classification algorithm. But decision trees happen to be the cornerstone of a powerful class of learning algorithms: gradient tree boosting methods. I will try to elucidate the (short) history of gradient tree boosting, starting with the pioneering implementation of boosted trees and ending with the state-of-the-art. I’m afraid there will be a hearty amount of mathematics, but it is really a very intuitive topic at its core.

Note that I will not be covering the actual functioning of decision trees; for that, you may either refer to my notes on the original paper, or to the excellent Wikipedia page.

Weak learners and boosting

As I have mentioned, I don’t think decision trees on their own are especially useful. You might even say that they are ‘weak classifiers’, or generally, ‘weak learners’. This, in fact, is a technical term which refers to predictive models which only do a little bit better than random guessing.

However, Robert Schapire (currently a Princeton professor) discovered in his classic paper The Strength of Weak Learnability (1990), that it is possible to combine a number of weak learners into a strong learning algorithm – a process called boosting. The first tangible algorithm for doing so came in the form of Adaptive Boosting, or AdaBoost, invented by Freund and Schapire in 1997 (their paper now has about 15,000 citations). Essentially, the idea behind AdaBoost is that you can build a strong algorithm by making weak learners learn from their mistakes. This is done by sequentially fitting new classifiers, each one giving more weight to the datapoints for which their predecessors made incorrect predictions. In the end, you take the weighted ‘vote’ of all the weak classifiers, et voilà.

I won’t go into the details of AdaBoost, because it is slightly tangential to the goal of this post, but Chris McCormick has an excellent exposition of the algorithm here.

Additive models

Around the same time, a lot of research was going on into additive expansions, which are linear combinations of basis functions $b(x;\gamma)$ with coefficients $\beta$:

\[f(x) = \sum_{m=0}^{M} \beta_m b(x;\gamma_m)\]

Here, $x$ is an input vector (e.g. training data), and $\gamma_m$ denotes the parameters of the mth basis function.

What is interesting is that additive expansions are fundamentally equivalent to boosting methods, as shown by Friedman, Hastie and Tibshirani (2000). If you take $b(x; \gamma)$ to be a weak learner, then it is clear that $f(x)$ presents a weighted vote of the weak learners, just as in AdaBoost. So now instead of dealing with the somewhat vague concept of boosting, all we have to do is figure out how to fit an additive expansion $f(x)$ to some training data, denoted by $\{(y_i, x_i)\}_1^N$.

A first attempt to do this is to minimise some loss function $L(y_i, f(x_i))$ for each training example, such that the parameters are:

\[\arg\min_{\{\beta_m, \gamma_m \}_1^M} \sum_{i=1}^N L \left( y_i, \sum_{m=1}^M \beta_m b(x_i;\gamma_m)\right)\]

That looks to be quite a monstrous summation, but it is only the cumbersome notation: if you carefully look at the expression, you will see that it is really just choosing $\beta_m$ and $\gamma_m$ values that will minimise the loss of $f(x)= \sum_{m=1}^M \beta_m b(x_i;\gamma_m)$ over all training examples.

While the above optimisation may not be conceptually difficult, it is computationally difficult. Fortunately, additive expansions invite a forward stagewise approximation. Instead of trying to optimise the parameters all at once, we do it in stages, slowly building up an approximately optimal model. At each iteration $m$ we find an optimal $\beta_m$ and $\gamma_m$, and add the new term to the existing predictor function $f_{m-1}(x)$.

The Forward Stagewise Additive Modelling (FSAM) algorithm

Set $~f_0(x) = 0$

For $~m=1,\ldots, M$

Compute:

\[(\beta_m, \gamma_m) = \arg \min_{\beta, \gamma} \sum_{i=1}^N L(y_i, f_{m-1}(x_i)+\beta b(x_i;\gamma))\]

Set $~f_m(x)=f_{m-1}(x)+\beta_m b(x;\gamma_m)$

Return $~f_M(x)$

Gradient tree boosting

Unfortunately, the difficulty with the FSAM algorithm is the actual optimisation step:

\[(\beta_m, \gamma_m) = \arg \min_{\beta, \gamma} \sum_{i=1}^N L(y_i, f_{m-1}(x_i)+\beta b(x_i;\gamma))\]

For all but the simplest loss functions, this presents a prohibitively difficult optimisation problem. One solution is gradient boosting, as described by Friedman in his 2001 Paper Greedy Function Approximation: a Gradient Boosting Machine.

It is worth mentioning at this point that although we can use a large variety of models for $b(x;\gamma)$, a decision tree is a common choice. We will adopt the notation conventions from Elements of Statistical Learning. A tree with $J$ terminal nodes (or ‘leaves’) is represented as follows:

\[T(x; \Theta) = \sum_{j=1}^J \gamma_j I(x \in R_j), \qquad \Theta = \{R_j, \gamma_j \}_1^J\]

Such a tree partitions the feature space into $J$ disjoint regions/leaves, each having an associated constant $\gamma_j$ which acts as a weight on that leaf. $I$ is the indicator function, returning 1 if its argument is true, and 0 if false. $\Theta$ encodes both the leaf weights and the ‘structure’ of the tree.

Thus the optimisation step in the FSAM algorithm becomes:

\[\Theta_m = \arg \min_{\Theta} \sum_{i=1}^N L(y_i, f_{m-1}(x_i) + T(x_i; \Theta))\]

Note that we no longer need to consider the constants $\beta_m$, as they are effectively absorbed into $\gamma_{jm}$.

One method of optimising the above is to borrow the idea of classical gradient descent, a wonderfully intuitive method which realises that the negative gradient of the loss function points in the direction of fastest decreasing loss, so updating parameters in the direction of the gradient must also decrease loss! Of course, in the case of boosting we are optimising in function space rather than parameter space – we are trying to find the optimal tree to add to $f_{m-1}(x)$. The components of the gradient vector $\mathbf{g}_m \in \mathbb{R}^N$ are thus given by:

\[g_{im} = \frac{\partial L(y_i, f_{m-1}(x_i))}{\partial f_{m-1}(x_i)}\]

Again, because $-\mathbf{g}_m$ points in the direction of fastest decreasing loss, we can simply fit the tree to the negative gradient instead of trying to optimise with respect to some loss function – in both cases, the tree will be one that decreases loss, but in the former case we have the computational benefits of fitting with squared error:

\[\Theta_m = \arg \min_{\Theta} \sum_{i=1}^N (-g_{im} - T(x_i;\Theta))^2\]

I think it is definitely worth reading over the last few paragraphs again; it is a beautifully subtle piece of reasoning. The full algorithm is presented below: we start by initialising the model with the best constant weights. Then, in each round, we fit a tree to the negative gradient before choosing optimal leaf weights.

Gradient tree boosting algorithm

Set $~f_0(x) = \arg \min_\gamma \sum_{i=1}^N L(y_i, \gamma)$

For $~m=1,\ldots, M$

For $~i=1, \ldots, N$

Compute:

\[g_{im} = -\frac{\partial L(y_i, f_{m-1}(x_i))}{\partial f_{m-1}(x_i)}\]

Fit tree to targets $-g_{im}$ giving:

\[R_{jm}, \qquad j = 1,2,\ldots,J_m\]

For $~j=1,\ldots,J_m$:

Compute:

\[\gamma_{jm} = \arg \min_\gamma \sum_{x_i \in R_{jm}} L(y_i, f_{m-1}(x_i) + \gamma)\]

Set $~f_m(x)=f_{m-1}(x)+ \sum_{j=1}^{J_m} \gamma_{jm} I(x \in R_{jm})$

Return $~f_M(x)$

Regularisation

A popular method of regularising boosted tree models, proposed in the original paper by Friedman, is shrinkage, wherein additional trees are multiplied by a small shrinkage parameter $\nu$ before being added to the existing model. This would change the update step in the gradient tree boosting algorithm to:

\[f_m(x) = f_{m-1}(x) + \nu \sum_{j=1}^{J_m} \gamma_{jm} I(x\in R_{jm})\]

Note that we cannot absorb $\nu$ into $\gamma_{jm}$: even though the latter are the best parameters (having the least loss on the training data), we must shrink their contribution to the final model to reduce overfitting. The meta-parameter $\nu$ is intuitively the same as the learning rate and is often referred to as such: smaller values (in the order of $0.1$) lead to slower learning, but improved generalisation ability.

Another method that has previously been employed with success is subsampling, in which a certain subset of the training data is randomly selected (without replacement) at each boosting round. Stochastic gradient boosting subsamples by row; not all available training data is used in a given round. These methods have been found to give computational benefits and improved generalisation.

XGBoost

XGBoost (‘eXtreme Gradient Boosting’) is as an open-source gradient boosting software package that won renown for simply being the algorithm you used if you wanted to win Kaggle competitions.

XGBoost provides an alternative perspective on the optimisation problem in gradient tree boosting (reproduced below):

\[\Theta_m = \arg \min_{\Theta} \sum_{i=1}^N L(y_i, f_{m-1}(x_i) + T(x_i; \Theta))\]

Instead of making the ‘logical leap’ of fitting to gradients, we simply find the second order Taylor expansion of the loss function:

\[L(y_i, f_{m-1}(x_i) + T(x_i; \Theta)) = L(y_i, f_{m-1}(x_i)) + g_{im} T(x_i; \Theta) + \frac 1 2 h_{im}T(x_i; \Theta)^2\]

where

\[\begin{aligned} g_{im} &= \frac{\partial L(y_i, f_{m-1}(x_i))}{\partial f_{m-1}(x_i)} \\ h_{im} &= \frac{\partial^2 L(y_i, f_{m-1}(x_i))}{\partial (f_{m-1}(x_i))^2} \end{aligned}\]

It looks pretty nasty, but it is really just a standard Taylor expansion of the form:

\[f(x+ k) \approx f(x) + kf'(x) + \frac{k^2}{2}f''(x)\]

Following on from this, XGBoost’s learning objective, or cost function, in boosting round $m$ is given by

\[\begin{gathered} C_m = \sum_{i=1}^N \left[L(y_i, f_{m-1}(x_i)) + g_{im} T(x_i; \Theta_m) + \frac 1 2 h_{im}T(x_i; \Theta_j)^2 \right] \\ +~\Omega(T(x; \Theta_m)) \end{gathered}\]

The ultimate goal is to minimise this cost with respect to $\Theta_m$. The first term is simply the sum over all training examples of the loss approximation. The second term is a complexity penalty on the $m$th tree. Many different functions could work here, but the following is an intuitive choice which results in an elegant algorithm (we will subsequently drop the $m$ subscripts for clarity):

\[\Omega(T(x; \Theta)) = \alpha J + \frac 1 2 \lambda \sum_{j=1}^J \gamma_j^2\]

Recall that $J$ is the number of terminal nodes on the tree, each having weight $\gamma_j$. This complexity penalty punishes having more leaves (controlled by $\alpha$), as well as leaves with greater weight (controlled by $\lambda$). If we substitute this back into the cost function, and at the same time remove terms that are constant in $\Theta$, we get:

\[C = \sum_{i=1}^N \left[g_{i} T(x_i; \Theta) + \frac 1 2 h_{i}T(x_i; \Theta_j)^2 \right] + \alpha J + \frac 1 2 \lambda \sum_{j=1}^J \gamma_j^2\]

At this stage, one must recall the the mathematical defition of a tree: $T(x; \Theta) = \sum_{j=1}^J \gamma_j I(x \in R_j)$. By substituting this into the above, and regrouping the terms, we can rewrite the cost function:

\[C = \sum_{j=1}^J \biggl[ \underbrace{\biggl( \sum_{x_i \in R_j} g_i \biggr)}_{G_j} \gamma_j + \frac 1 2 \biggl(\underbrace{\sum_{x_i \in R_j} h_i}_{H_j} + \lambda \biggr) \gamma_j^2 \biggr] + \alpha J\]

Essentially, instead of counting the contributions to the cost over each training example, we count over each leaf on the tree, summing the weighted gradient statistics for all training examples in that leaf. New variables $G_j = \sum_{x_i \in R_j} g_i$ and $H_j = \sum_{x_i \in R_j} h_i$ are defined to simplify the notation.

Remembering that we are trying to minimise the cost with respect to the parameters $\Theta = \{R_j, \gamma_j \}_1^J$, we can first find the optimal weights $\gamma_j$ by noting that the cost function above is a sum of independent quadratics in $\gamma_j$. The optimal $x$ in a quadratic $ax^2 + bx + c$ is given by $x = -b/2a$. Likewise, the optimal weights are:

\[\gamma_j^* = -\frac{G_j}{H_j+\lambda}\]

Our final cost function is then:

\[C = -\frac 1 2 \sum_{j=1}^J \frac{G_j^2}{H_j + \lambda} + \alpha J\]

We can use this cost function to greedily grow trees. To evaluate a potential split into ‘left’ and ‘right’ nodes, we consider the gain, the amount by which a given split would decrease the cost:

\[\text{gain} = \frac 1 2 \left[ \frac{G_L^2}{H_L + \lambda} + \frac{G_R^2}{H_R + \lambda} - \frac{(G_L+G_R)^2}{H_L + H_R + \lambda} \right] - \alpha\]

Then, we do a linear scan over the training examples for each feature to decide on the split. What is amazing about XGBoost is really just its cleverness: none of the above mathematics is difficult – it is mostly algebraic sleight-of-hand. However, these ‘tricks’ do really improve the computational efficiency, which is partially why XGBoost is such a popular learning algorithm.

Regularistion in XGBoost

Apart from the inbuilt complexity penalty, XGBoost offers shrinkage and row subsampling, but also column subsampling – the regularisation method of choice for Random Forests. In column subsampling, each boosting round neglects a certain portion of the features. This increases the overall model’s robustness by reducing the emphasis placed on one or two key variables, giving other variables ‘a chance to speak’, so to speak.

Conclusion

This post turned out to be more detailed than I originally intended. While the logic behind the mathematics is not hard to follow, I concede that the notation is often quite arcane: there are many different indexing values to keep track of. However, it’s very rewarding to be able to understand just how clever XGBoost is, even more so because with it, you can achieve amazing performance on many learning tasks.

I did a fair bit of research into the theory of gradient tree boosting while writing a paper about using XGBoost to classify potentially outperforming stocks (more on that another time, perhaps). For me, the biggest issue was synthesising a clear and consistent notation from all the conflicting sources (if you think the notation in this post is confusing, try three different sets of notation!). Nevertheless, some resources I found useful are as follows:

The Elements of Statistical Learning, which is often the gold-standard reference for machine learning theory. I made notes on the relevant chapters, which I have put up here.
If ESL is too rigorous for your liking, there is a slightly easier (though obviously less in-depth) version by the same authors, called Introduction to Statistical Learning. I also have notes on the Tree-Based methods chapter.
The original XGBoost paper, detailing the theory and implementation. My notes on the paper are here: XGBoost: A Scalable Tree Boosting System
The XGBoost docs offer a nice high-level overview of the theory, and are definitely a must-read when it comes to implementations and APIs.
There is a great youtube video by Tianqi Chen explaining his algorithm.

I personally can never go back to the ‘standard’ Machine Learning algorithms such as SVM, kNN etc, because in my experience they are just crushed by XGBoost. However, even XGBoost may no longer be the state-of-the-art. Competing software packages such as LightGBM and CatBoost; after playing with those libraries, even XGBoost seems slow.

8-bit Julia set art in python

2017-09-03T00:00:00+00:00

You may have heard a mathematician or physicist (or more likely your maths teacher) describe mathematics as beautiful. What could they mean by this? There is just something mysteriously attractive about the purity, complexity, interconnectedness, and underlying truth of it all (“Beauty is truth, truth beauty” - Keats). I can’t really say more than that, so I will leave you with a quote from the ‘man who loved only numbers’:

It’s like asking why is Ludwig van Beethoven’s Ninth Symphony beautiful. If you don’t see why, someone can’t tell you. I know numbers are beautiful. If they aren’t beautiful, nothing is. - Paul Erdős

But occasionally, mathematics can be beautiful in the traditional sense of the word.

What is this image, and how did I make it?

The Julia set

The Julia set is a type of fractal, a complex shape with a certain kind of evolving symmetry. What is interesting about fractals is that they are often based on very simple rules, which really begs the question as to where the dazzling intricacy comes into the picture.

While the Julia set comes in many flavours, we will consign ourselves to examine the easiest case, which is more than sufficient to produce pretty images; this is the Julia set defined by:

\[f_c(z) = z^2 + c\]

where $c$ and $z$ are complex numbers. To get from this equation to an image, we follow a relatively simple procedure

Choose a complex $c$ as the ‘seed’. This value of $c$ will determine the final picture, so choose wisely.
Draw a blank Argand diagram, i.e a set of axes with vertical being imaginary and horizontal being real.
Each point on this diagram is a complex number $z$. Repeatedly apply the above formula.
For each pixel, count the number of iterations it takes for the absolute value of the resulting complex number to exceed a certain limit. If it hasn’t exceeded this limit after a specific maximum iteration value, we terminate at this maximum.
Colour the pixel based on the number of steps it took.
Repeat this for all the points on the diagram.

Perhaps I have botched the explanation, but in essence, we count how many iterations of the above formula are required to increase $|z|$ beyond a certain bound, and colour accordingly. This is a remarkably simple procedure, and is rather easy to implement in python.

Python produces art

We will start by importing the necessary libraries. We need numpy for array manipulations, and matplotlib to produce our graphs (or art, if you prefer).

import numpy as np
import matplotlib.pyplot as plt
import matplotlib.cm as cm

In addition to deciding the resolution and the size of the plot, we must also settle the mathematical parameters. Firstly, we need a suitable $|z|$ to terminate the iterations; secondly, a value for the maximum number of iterations to compute. Quick experiments have suggested that values of 10 and 1000 respectively produce decent results.

# Resolution and plot size
x_res, y_res = 200, 200
xmin, xmax = -1.5, 1.5
width = xmax - xmin
ymin, ymax = -1.5, 1.5
height = ymax - ymin

# The mathematical parameters.
z_abs_max = 10
max_iter = 1000

You should definitely play around with some of the above parameters. In general, I left everything untouched except the resolution – I will demonstrate the effects of adjusting this later on.

x_res and y_res are the number of x and y pixels for which we will iterate the complex function. At some stage, we will have to loop over all of these pixels. But let’s first examine what happens for one pixel with coordinates (ix, iy).

We first map the pixel position to a point on the complex plane. ix ranges from 0 to 200, but our real axis ranges from -1.5 to 1.5. We thus have to ‘scale’ the ix range onto our Argand diagram.

real_part = ix / x_res * width + xmin
imaginary_part = iy / y_res * height + ymin
z = complex(real_part, imaginary_part)

This encodes the logic that we want to loop over all values between xmin and xmax, with increments of width/x_res. The same reasoning applies to the imaginary parts and iy.

Now that we have our $z$ and a $c$ (assuming the latter has already been chosen), we can proceed with the iterations. What I love about python is that the code practically speaks for itself – continue iterating as long as $|z|$ is smaller than our chosen limit, if we haven’t yet exceeded the maximum number of iterations.

iteration = 0
while abs(z) <= z_abs_max and iteration < max_iter:
    z = z**2 + c
    iteration += 1

So for each (ix, iy), we will have a certain value of iteration which should determine the colour. It will be easiest to implement the colouring if we first map all of the iterations to the domain [0, 1]. One shortcut for doing this is to just write:

iteration_ratio = iteration / max_iter

Because iteration is always greater than zero but less than max_iter, this value will be between zero and one, and will be proportional to the number of iterations it took to escape the $|z|$ circle.

All together now…

import numpy as np
import matplotlib.pyplot as plt
import matplotlib.cm as cm

# Parameters
x_res, y_res = 300, 300
xmin, xmax = -1.5, 1.5
width = xmax - xmin
ymin, ymax = -1.5, 1.5
height = ymax - ymin

z_abs_max = 10
max_iter = 1000


def julia_set(c):
    # Initialise an empty array (corresponding to pixels)
    julia = np.zeros((x_res, y_res))

    # Loop over each pixel
    for ix in range(x_res):
        for iy in range(y_res):
            # Map pixel position to a point in the complex plane
            z = complex(ix / x_res * width + xmin,
                        iy / y_res * height + ymin)
            # Iterate
            iteration = 0
            while abs(z) <= z_abs_max and iteration < max_iter:
                z = z**2 + c
                iteration += 1
            iteration_ratio = iteration / max_iter    
            # Set the pixel value to be equal to the iteration_ratio
            julia[ix, iy] = iteration_ratio

    # Plot the array using matplotlib's imshow
    fig, ax = plt.subplots()
    ax.imshow(julia, interpolation='nearest', cmap=cm.gnuplot2)
    plt.axis('off')
    plt.show()
    fig.savefig('julia_set.png', dpi=500)

Running the code for $c = -0.1 - 0.65i$ gives the image at the top of this post.

Your turn

It is rather fun to play around with the different $c$ values. Here is a quick animation I made, using an adaptation of the above code:

Or, if this colourscheme isn’t your thing, try playing around with the cmap parameter of imshow. Here is a list of the available colourmaps (for each one, you can append ‘_r’ to the cmap name to reverse the spectrum). I’ve shown a few different cmaps below:

It is worth noting that the fractal we produced is actually infinitely complex. If you want to see some more of that complexity, try increasing the resolution. Be warned, though: the computation time grows as the square of x_res and y_res, so it might take some time to render. This is a render of the same initial image, except with the resolution set to 2000.

Conclusion

Clearly, this post doesn’t present the most efficient computation of a Julia set. For speed, we would use array manipulation rather than iterating over each pixel. However, the code is very easy to understand and we can still get some very pretty results.

There is a lot of room for personal preference: I have demonstrated the effects of choosing different c values, colourmaps, resolutions. However, you can even do stuff like ‘zoom’ onto a certain area of the fractal by changing xmin, xmax, ymin, ymax.

But my personal favourite is the image I showed you at the start of this post. That lovely rich purple combined with the 8-bit pixelation gives the image an other-worldly charm.

Play around with the code on GitHub to experiment some more. The code therein is very similar to what I’ve shown in this post, except I’ve also included the code to generate the animations and test different colourmaps.

Retrieving historical stock prices from Yahoo Finance with no API

2017-07-30T00:00:00+00:00

Yahoo Finance has long been an excellent free financial resource with a wealth of data and a convenient API, allowing open source programming libraries to access stock data. But not any more. As of May 2017, they have discontinued their API, probably as a result of Yahoo’s pending acquisition by Verizon. This means that excellent tools like pandas-datareader are now broken, much to the dismay of many amateur algorithmic traders or analysts. It turns out that there is a rather hackish workaround which allows us to download the data as CSV (i.e spreadsheet) files, which of course can then be read into excel, pandas dataframes etc.

Just a legal disclaimer. This method involves making a large number of requests on Yahoo Finance, which may ‘look like’ a DDOS. Clearly, I am not trying to conduct a DDOS – I am merely trying to parse data for educational purposes. I am not responsible for how you use the methods demonstrated in this post.

Update as of 10/2/18: there is now a library on GitHub that puts this post to shame, with a direct pandas-datareader interface. I will leave this post up for legacy, but for any serious implementations, I wholeheartedly recommend the aforementioned tool.

Update as of 20/5/18: it seems that fix-yahoo-finance is becoming very inconsistent. So I guess this post does have some utility after all!

Update as of 9/1/20: looking back on this post, I don’t think my solution is very good. A much better way would be to use requests directly and parse the csv bytes, instead of physically downloading files moving them.

Overview

Although the API doesn’t work, you can still manually download CSV files containing historical price data for a given ticker. To do so, you go to finance.yahoo.com, and enter a ticker in the search. I entered ‘AAPL’.

After navigating to the Historical Data tab, you will end up with a screen like this:

After editing the time period and applying it, you can click the Download Data button (which I emphasised in red above), and a CSV file will promptly start downloading on your browser. But to manually do this for 3000 stocks, as I would have to do for my MachineLearningStocks project, would be tantamount to water torture. I thus needed a good way of automating this process, and luckily I found one.

The method

The Download Data button is actually a hyperlink, so I decided to have a quick check as to the url of this link. If you right click a link in chrome, there is an option to ‘Copy Link Address’. Doing so for AAPL with a time period of Jan 01 2001 – Aug 06 2017, the link is:

https://query1.finance.yahoo.com/v7/finance/download/AAPL?
period1=946656000&period2=1501948800
&interval=1d&events=history&crumb=BkT/GAawAXc

Actually, the structure of this url is all that one needs to crack the problem of automation. I don’t know what time system yahoo finance uses internally, but it’s a pretty fair guess that 946656000 corresponds to Jan 01 2001, and likewise that 1501948800 corresponds to Aug 06 2017.

I tried just changing ‘AAPL’ to ‘GOOG’ in the above url and entering it onto chrome, and suddenly GOOG.csv had downloaded onto my machine. I did some digging as to what the ‘crumb’ is, and to my knowledge it is some sort of API token / cookie which will most likely be different for you.

In any case, I figured that I had basically solved the issue at this point, so the rest was a matter of coding it up in python.

Implementing the solution in python

To interact with my browser (Google Chrome), I used the webbrowser library in python. It is extremely intuitive to use: opening a URl is as simple as

webbrowser.open('https://www.google.com')

I wanted to download a list of tickers that I had saved in a text file called russell3000.txt. To read this into a python list, we can use an elegant list comprehension:

russell_tickers = [line.split('\n')[0].lower() 
                   for line in open('russell3000.txt')]

For each ticker, we will want to open the URL as specified earlier. The time period will be the same in all of them – Jan 01 2001 to Aug 06 2017. The code for this is as follows:

import webbrowser 

for ticker in tickers:
    link = ("https://query1.finance.yahoo.com/v7/finance/download"
            "/{}?period1=946656000&period2=1501948800&interval=1d"
            "&events=history&crumb=BkT/GAawAXc".format(ticker))
    webbrowser.open(link)

And that’s it. This will download all of the CSV files straight to your downloads folder.

Or will it? I have found the hard way that web-related code often runs into errors regarding the web queries – often, pages that you expect to exist have been taken down or corrupted etc. When webbrowser tries to open this URL, it will encounter an error, but instead of continuing with the rest of the tickers, it’ll just give up. The way around this is to use python’s exception handling:

for ticker in tickers:
    try:
        link = ("https://query1.finance.yahoo.com/v7/finance/"
                "download/{}?period1=946656000&"
                "period2=1501948800&interval=1d&events=history"
                "&crumb=BkT/GAawAXc".format(ticker))
        webbrowser.open(link)
    except Exception as e:
        print(ticker, str(e))

This way, if there is ever an error, the code will just print the ticker and the error, then carry on. In principle, we are finished. However, it is a bit annoying that all of these CSVs are in the downloads folder, rather than the directory for my project. I wrote a few lines of python to fix this:

import os

# Change these as appropriate
project_path = "/Users/Robert/Project/"
download_path = '/Users/Robert/Downloads/'

# Look at the contents of your download folder
for item in os.listdir(download_path):
        # If the file is a csv, move it to the project folder
        if item[-4:] == '.csv':
            print(download_path + item)
            print(project_path + item)
            os.rename(download_path + item, project_path + item)

For AAPL.csv, the code would print:

/Users/Robert/Downloads/AAPL.csv
/Users/Robert/Project/AAPL.csv

and the file would be moved from the former location to the latter. Note that it will also move any other CSVs, but I didn’t need to fix this as I don’t normally have CSVs in my downloads.

Conclusion

It is always a nice feeling when you can use some python to save you hours of manual labour, and this was certainly the case here. Using this code, one can automate the downloading of a large number of datafiles which contain historical stock prices. This was a very important aspect of my MachineLearningStocks project, as it is meaningless to try to learn from data without having the data.

As is often the case, the code I’ve showed here just scratches the surface. You will find that some of the CSVs didn’t actually download, or they are empty. Thus, to increase the quality of your training data, some troubleshooting is in order. Some of the improvements I’ve made:

troubleshoot non-alphabet characters in the ticker names
use Quandl, and its python API, to download data that Yahoo Finance doesn’t have
use pandas-datareader to download data from Google Finance, if both Yahoo and Quandl don’t have the data

The Leibniz integral rule in electrodynamics

2017-07-25T00:00:00+00:00

I have been slowly working through David Griffith’s widely-used textbook, Introduction to Electrodynamics. It is known for having a large number of reasonably difficult exercises, which are instrumental in conveying some concepts not directly addressed in the main text. I found an elegant shortcut to one of the questions, which was not noted in the solution manual. The shortcut involves (ab)using a very versatile, if slightly overlooked theorem in calculus – the Leibniz integral rule. I will briefly discuss this, before working through the problem.

The Leibniz integral rule

The Leibniz rule concerns derivatives of integrals; the most basic form is as follows:

\[\frac{d}{dx} \left( \int_a^b f(x,t) dt \right) = \int_a^b \frac{\partial}{\partial x} f(x,t) dt\]

This can be proven by letting $I(x) = \int_a^b f(x,t) dt$, then using the definition of the derivative to evaluate $I’(x)$, which will be left as an exercise for the reader. However I hope you can get an intuitive feel for this: the integral measures the total variation with respect to $t$, whereas the derivative measures the rate of change with respect to $x$, so it makes sense that the order of operation doesn’t really matter.

A slightly more general version concerns the case where the limits of the integral are variable:

\[\frac{d}{dx} \left( \int_{a(x)}^{b(x)} f(x,t) dt \right) = f\left(x, b(x)\right)\frac{d b(x)}{dx} - f(x, a(x))\frac{d a(x)}{dx} + \int_{a(x)}^{b(x)}\frac{\partial}{\partial x} f(x,t) dt\]

Effectively, it is the same as what we had before, plus some additions to take into account of the variable limits. This formula can be explained in terms of the fundamental theorem of calculus, but the derivation is besides the point. For now, I would urge you to just accept the above rule (it is surprisingly useful). Let’s go through an example to evaluate the following expression:

\[\frac{d}{dx} \int_{\sin x}^{\tan x} \sqrt{1+ t^3}dt\]

Hint: do not do it by evaluating the indefinite integral, substituting in the limits, then differentiating. Applying the Leibniz rule, we see that:

\[\begin{aligned} \frac{d}{dx} \int_{\sin x}^{\tan x} \sqrt{1+ t^3}dt &= \sqrt{1+ \tan^3 x}\frac{d}{dx}\tan x - \sqrt{1 + \sin^3 x} \frac{d}{dx} \sin x\\ &= \sec^2x\sqrt{1+ \tan^3 x} - \cos x \sqrt{1 + \sin^3 x} \end{aligned}\]

The electrodynamics problem

I say ‘electrodynamics’, but really it is an electrostatics problem. Below I have reproduced the question from Chapter 2 (Electrostatics) of Griffith’s Introduction to Electrodynamics. It has been edited slightly.

Problem 27 Find the electric field on the axis of a uniformly charged solid cylinder, at a distance z from the centre. The length of the cylinder is L, its radius is R, and the charge density is $\rho$.

Our general strategy will be to first find the potential (or rather, an expression for the potential), by decomposing the cylinder into simpler shapes. Then we will find the field using $\mathbf{E} = - \nabla V$.

A cylinder can be thought of as many discs; a disc can be thought of as many rings; a ring can be thought of as many point charges. We will thus build a cylinder.

1. A single point charge

Recall the formula for the potential at a point $P$:

\[V = \frac{1}{4\pi \epsilon_0} \frac{q}{r}\]

In this formula, $r$ is the displacement between the charge and the test point. Applying Pythagoras’ theorem to the diagram, we see that $r^2 = R^2 + z^2$.

Plugging values into the formula:

\[V = \frac{1}{4\pi \epsilon_0} \frac{q}{\sqrt{R^2 + z^2}}\]

2. A ring of charge

In order to find the potential for a ring of charge given the potential of a point charge, we consider a charge element $\lambda dl$. But the length of a segment on a circle is given by $R \theta$, so we can say that $dl = R d\theta$.

Instead of a positive point charge $q$, we now have a charge element $\lambda R d\theta$. Using our previous result, the potential at P from one of these charge elements is:

\[dV = \frac{1}{4\pi \epsilon_0} \frac{\lambda R d\theta}{\sqrt{R^2 + z^2}}\]

The total potential for the ring is the sum of the contributions from each of the charge elements.

\[V = \int_0^{2\pi} \frac{1}{4\pi \epsilon_0} \frac{\lambda R d\theta}{\sqrt{R^2 + z^2}} = \frac{1}{4\pi \epsilon_0} \frac{\lambda R }{\sqrt{R^2 + z^2}} \int_0^{2\pi} d\theta\] \[\therefore V = \frac{1}{4\pi \epsilon_0} \frac{2 \pi R \lambda}{\sqrt{R^2 + z^2}}\]

3. A disc of charge

We can build up a disc of charge with radius R from an infinite number of rings of charge of radius r. Note that this r is different to the one briefly mentioned earlier, which was the separation between P and the charge.

Because we are going from 1D to 2D, we must let $\lambda = \sigma dr$, where $\sigma$ is the surface charge. Thus the potential from one ring is:

\[dV = \frac{1}{4\pi \epsilon_0} \frac{2 \pi r \sigma dr}{\sqrt{r^2 + z^2}}\]

We could arrive at this otherwise, by noting that the total charge in the ring is

\[\sigma \cdot \text{area} = \sigma \cdot 2\pi r dr\]

Anyway, let us continue.

\[V = \frac{\sigma}{2 \epsilon_0} \int_0^R \frac{r}{\sqrt{r^2 + z^2}} dr\]

As is rarely the case in electrodynamics, this integral is not at all difficult to evaluate. We use the ‘reverse chain rule’ to see that:

\[\int_0^R \frac{r}{\sqrt{r^2 + z^2}} dr = \int_0^R r \left( r^2 + z^2\right)^{-1/2} dr =\left[ (r^2+z^2)^{1/2} \right]_0^R\]

The total potential is then easily found.

\[V = \frac{\sigma}{2\epsilon_0} \left(\sqrt{R^2+z^2} - z \right)\]

4. A cylinder of charge

We are now ready to tackle the original problem. Here it is again:

Find the electric field on the axis of a uniformly charged solid cylinder, at a distance z from the centre. The length of the cylinder is L, its radius is R, and the charge density is $\rho$.

I’ll set up the solution in the same way Griffiths does – this image is taken from the solution manual.

A cylinder is made up of an infinite number of discs. We want an expression for the potential at the test point, a distance z from the centre. We will let x denote the distance between the test point and a given disc. The potential from one such disc is given by the formula we found earlier, with the substitution $z \rightarrow x$ and $\sigma \rightarrow \rho dx$:

\[dV = \frac{\rho dx}{2 \epsilon_0} \left( \sqrt{R^2 + x^2} - x \right)\]

Since L is the length of the cylinder, and z is the distance from the centre of the cylinder, it is clear that there are discs all the way from $x = z - \frac L 2$ to $x = z+ \frac L 2$. The total potential at our test point is just the integral over all of these discs:

\[V = \frac{\rho}{2\epsilon_0} \int_{z-L/2}^{z+L/2}(\sqrt{R^2 + x^2} - x) dx\]

Despite the simple-looking expression, this integral will soon become a nightmare to evaluate. And don’t forget – we are trying to find the electric field, so we will then have to take the gradient of the whole mess.

To be clear, because of the cylinder’s symmetry, ‘taking the gradient’ here just means differentiating with respect to z, but nevertheless this will not be a fun calculation.

\[\mathbf{E} = - \nabla V = - \mathbf{\hat{z}} \frac{\partial V}{\partial z}\]

Leibniz to the rescue

Instead of immediately evaluating the integral to calculate the potential, then differentiating, why don’t we write it in one expression first:

\[\mathbf{E} = - \mathbf{\hat{z}} \frac{\rho}{2\epsilon_0} \frac{\partial}{\partial z} \int_{z-L/2}^{z+L/2}\left( \sqrt{R^2 + x^2} - x \right) dx\]

This might look more complicated, but in fact it makes life much easier. Because we are taking the derivative of an integral, we can simply apply the Leibniz integral rule. The limits are not constant, but the integrand is independent of z, so the rule tells us that:

\[\begin{align*} \mathbf{E} = - \mathbf{\hat{z}} \frac{\rho}{2\epsilon_0} &\bigg[ \left( \sqrt{R^2 + (z+L/2)^2} - (z+L/2) \right) \frac{\partial}{\partial z}(z+L/2)\\ & - \left( \sqrt{R^2 + (z-L/2)^2} - (z-L/2) \right) \frac{\partial}{\partial z}(z-L/2) \bigg] \end{align*}\]

Intuitively, there is some kind of ‘cancellation’ between the differentiation and integration, so we simply substitute the limits into the integrand (and multiply by the derivative of the limit since it is not constant), then subtract. In this case it is even simpler because the derivatives of the limits are unity:

\[\mathbf{E} = \frac{\rho}{2\epsilon_0} \left[L + \sqrt{R^2 + (z-L/2)^2} - \sqrt{R^2 + (z+L/2)^2} \right] \mathbf{\hat{z}}\]

Conclusion

In many electrodynamics problems, one finds the electric field by first calculating the potential, then taking the gradient. I hope I’ve given you a reasonable motivation to not just do this mechanically – keep an eye out for shortcuts that will save you time, and reduce the chances of you making a minor algebraic error.

Intuiting the gamma function, part 3

2017-07-21T00:00:00+00:00

We ended Part 2 with the stunning result that

\[x! = \int_0^\infty t^x e^{-t}dt\]

Of course, this is not exactly the same as the gamma function, because the gamma function is defined as:

\[\Gamma (x) = \int_0^\infty t^{x-1}e^{-t} dt\]

A direct observation of the above leads to the conclusion that:

\[\Gamma (x) = (x-1)! \qquad \text{or alternatively} \qquad x! = \Gamma (x+1)\]

This ‘shift’ is a minor annoyance and is largely a historical artifact, but don’t let it distract you from the fact that the gamma function is a superset of the factorial. That is quite a bold statement (no pun intended), so it is worth proving. To show it, we must simply demonstrate that the gamma function can satisfy the definition of the factorial which we discussed in Part 1, reproduced here:

$1! = 1$
$n! = n(n-1)!$

In terms of the gamma function, we thus need to show that

$\Gamma (2) = 1$, the boundary condition
$\Gamma (x+1) = x \Gamma (x)$, the recursion condition

The boundary condition

To evaluate $\Gamma(2)$, we can start by substituting $x=2$ into the definition of the gamma function:

\[\Gamma(2) = \int_0^\infty te^{-t}\]

We can use standard integration by parts (IBP) to evaluate this:

\[\begin{align*} \therefore \int_0^\infty t e^{-t} dt &= \left[ -te^{-t} - e^{-t} \right]_0^\infty \\ &= \left[ \lim_{t \rightarrow \infty} \left( -\frac{t}{e^t}\right) \right] + 1 \end{align*}\]

I can tell you very quickly that this limit is equal to zero – look at what happens to the expression within the limit as $t$ grows. The $e^t$ on the denominator clearly grows much faster than the $t$ on the numerator, so as $t \rightarrow \infty$, this expression will tend to zero. In fact this is the intuition behind L’Hopital’s rule, which we could apply by differentiating the top and the bottom of the fraction.

In any case, we can conclude that

\[\Gamma(2) = \int_0^\infty t e^{-t} dt = \left[ \lim_{t \rightarrow \infty} \left( -\frac{t}{e^t}\right) \right] + 1 = 1 \qquad QED\]

The recursion condition

As a quick reminder, we need to show that:

\[\Gamma(x+1) = \int_0^\infty t^x e^{-t}dt\]

This proof is really rather elegant, so I hope you’ll be as impressed with it as I was when I first understood it.

We will start by evaluating the left hand side (LHS) using IBP, though we will do it without our table method. Choosing an $f$ and a $g’$ then substituting them into the IBP formula:

\[\begin{aligned} f = t^x \implies f' = xt^{x-1}\\ g' = e^{-t} \implies g = -e^{-t} \end{aligned}\] \[\therefore \int_0^\infty t^x e^{-t} dt = \left[ -t^xe^{-t} + x\int e^{-t} t^{x-1} dt\right]_0^\infty\]

To evaluate the RHS, we can split up the expression (noting that x can be treated as a constant)

\[\begin{aligned} \left[ -t^xe^{-t} + x\int e^{-t} t^{x-1} dt\right]_0^\infty &= \left[ -t^xe^{-t}\right]_0^\infty + x \left[\int e^{-t} t^{x-1} dt\right]_0^\infty\\ &= \left[ -t^xe^{-t}\right]_0^\infty + x\int_0^\infty e^{-t} t^{x-1} dt\\ \end{aligned}\]

Let us evaluate the first term (in square brackets):

\[\left[ -t^xe^{-t}\right]_0^\infty = \lim_{t \rightarrow \infty} \left( -\frac{t^x}{e^t} \right)\]

I could just apply the same intuition as before to find this limit, but in this case it’s not immediately obvious that $e^t$ grows faster than $t^x$. To show this, we just repeated differentiate the top and bottom as per L’Hopital’s rule, until we can ‘substitute’ $t = \infty$ with sensible results (please don’t show that statement to any mathematicians):

\[\lim_{t \rightarrow \infty} \left( -\frac{t^x}{e^t} \right) = \lim_{t \rightarrow \infty} \left( -\frac{xt^{x-1}}{e^t} \right) = \lim_{t \rightarrow \infty} \left( -\frac{x(x-1)t^{x-2}}{e^t} \right) = \ldots = \lim_{t \rightarrow \infty} \left( -\frac{x!}{e^t} \right) = 0\]

To summarise:

\[\begin{aligned} \Gamma(x+1) &= \int_0^\infty t^x e^{-t} dt\\ &= \left[ -t^xe^{-t}\right]_0^\infty + x\int_0^\infty e^{-t} t^{x-1} dt\\ &= 0 + x\Gamma(x)\\ &=x\Gamma(x) \qquad \qquad QED \end{aligned}\]

We have successfully shown that the gamma function satisfies both the boundary condition and the recursion condition of the factorial function. This means that it will always give the same result as the traditional factorial (ignoring the shift). For example, we know for sure that $1729! = \Gamma(1730)$ without having to calculate either quantity.

Extending the factorial

It is a fact that $\Gamma(x)$ is defined for all complex numbers except the negative integers (and zero), but unfortunately, its values are rarely ‘nice’ and can often only be evaluated with a computer. However, I’d like to show you one example that does turn out nicely: $\Gamma(\frac 1 2)$.

We start by substituting 1/2 into the definition of the gamma function:

\[\Gamma (\frac 1 2) = \int_0^\infty t^{-1/2}e^{-t}dt\]

To evaluate the RHS, we make a substitution:

\[u = t^{1/2} \implies \frac{du}{dt} = \frac{1}{2\sqrt{t}} = \frac{1}{2u}\] \[\begin{aligned} \therefore \int_0^\infty t^{-1/2}e^{-t}dt &= \int_0^\infty \frac 1 u e^{-u^2} \cdot 2u~du\\ &= 2\int_0^\infty e^{-u^2}du \end{aligned}\]

You may recognise this very special integral, which is called the Gaussian integral. Although the indefinite integral has no elementary antiderivative, this definite integral has a beautifully simple result (the derivation of which I won’t show):

\[\Gamma(\frac 1 2) = 2\int_0^\infty e^{-u^2} du = \sqrt \pi\]

I hope it’s clear just how unexpected and weird this result is. We are saying that $\left( -\frac 1 2 \right)! = \sqrt \pi$, which is when you should realise that this whole series of posts has been a practical joke…

Not really.

Anyway, note how we can use this value to find $\frac 1 2 !$ We see that $\frac 1 2 ! = \Gamma(\frac 3 2)$, and we can easily write this in terms of $\Gamma(\frac 1 2)$ using the recursion condition.

\[\Gamma(\frac 3 2) = \frac 1 2 \Gamma(\frac 1 2) = \frac{\sqrt{\pi}}{2}\]

This is a result with which I like to surprise people at parties, because it’s slightly more believable than the result of $(- \frac 1 2)!$. Maybe this is why nobody invites me to parties.

Conclusion

We have reached the end of the series on understanding the intuition behind the gamma function. I hope you now see that the weird integral really does ‘contain’ the traditional factorial, which you can verify at any time by doing repeated integration by parts.

Unfortunately, despite my efforts, I can offer no intuition regarding the freaky results like $(1/2)! = \sqrt{\pi}/2$. But fortunately, you don’t have to take these results on faith – you can prove them to be true (as we have done).

Intuiting the gamma function, part 2

2017-07-16T00:00:00+00:00

In Part 1, we showed that repeated differentiation gives rise to a factorial. In this second post of Intuiting the gamma function, we are going to show that integration by parts can also produce a factorial – an instrumental step in generalising the idea of a factorial to non-integers.

Just a couple of minor points regarding the presentation of this post. I will abbreviate ‘integration by parts’ to IBP, since we will be mentioning it rather often. I will also consistently exclude the constant of integration, because it is just clutter.

A quick IBP refresher

If we are integrating a function $f(t)g’(t)$, the IBP formula tells us:

\[\int fg'dt = fg - \int f'g~dt\]

Notice some things:

There is both differentiation and integration involved. $f(t)$ is differentiated, while $g’(t)$ is integrated.
There is a minus sign, which will later be very important.

As we saw in the previous post, in order to end up with a factorial, we will have to repeatedly differentiate something like $x^n$. As such, I am going to let $f(t) = t^n$. This is because $f(t)$ is always differentiated in the IBP formula.

We also need to choose a $g(t)$, which will be always be integrated. I will choose the exponential function because it stays the same and makes everything easier. Maybe you think that I’m just choosing it because I know for a fact that the gamma function includes an exponential, but even if you think so, I urge you to read on as this doesn’t affect the intuition behind the gamma function – it just makes this post an explanation instead of a derivation.

A shortcut for repeated IBP

It may not be clear what I am trying to say at first, but it will make the eventual moment of understanding that much better. Follow along with the steps for now, and hopefully I will be able to make it all come together.

Consider the integral $\int t e^t dt$. We set up the parts required:

\[\begin{aligned} f = t \implies f' = 1\\ g' = e^t \implies g = e^t \end{aligned}\]

Now we apply the IBP formula:

\[\begin{aligned} \therefore \int t e^t dt &= te^t - \int e^t dt\\ &= te^t - e^t \end{aligned}\]

What about $\int t^2 e^t dt~$? We can still use IBP, though we have to do it twice. The first time:

\[\begin{aligned} f_1 = t^2 \implies f_1' = 2t\\ g_1' = e^t \implies g_1 = e^t\\ \therefore \int t^2 e^t dt = t^2e^t - \int 2te^t dt. \end{aligned}\]

We already know what $\int te^tdt$ is, so we conclude that:

\[\int t^2 e^t dt = t^2e^t - 2te^t + 2e^t\]

Look carefully at the structure of the above answer. What can you notice? First, there is an alternating sign between each term. Secondly, each term is made up of a differential and an integral. For example, in $t^2e^t - 2te^t + 2e^t$, notice how the first part of each term is the derivative of the previous term’s first part ($t^2 \rightarrow 2t \rightarrow 2$).

These two facts aren’t just coincidences, they are a direct result of the IBP formula: the $f(t)$ that we choose is always differentiated and the $g(t)$ is always integrated. The alternating sign between the terms is explained by the fact that there is a minus sign in the IBP formula, so when we apply the IBP formula multiple times, some of the negative signs cancel to become positives, while some stay negative.

In fact, we can derive a useful shortcut from this. Let us now try to evaluate $\int t^4 e^t dt$. This would normally be very tedious, as we would have to apply IBP four times! But let’s be clever about it. The integrand has two parts, $t^4$ and $e^t$. As we saw, the former will always be differentiated while the latter will always be integrated. Not forgetting the alternating signs, we can set up a small table:

The D denotes the part of the integrand which we will always differentiate, i.e the f that we choose for the IBP formula. Every term in the D column is the derivative of the term above it.

The I denotes the part of the integrand that we will always integrate, i.e the $g’$ that we choose. Every term in the I column is the integral of the term above it (this is why we chose $g’(t) = e^t$; it is easy to integrate).

The +/- column represents the alternating sign.

Combining these facts provides an easy shortcut to integrate by parts. We simply lattice multiply according to the arrows, and include the sign.

\[\therefore \int t^4 e^t dt = t^4e^t - 4t^3e^t -24te^t + 24e^t\]

This is reminiscent of the Maclaurin expansion in the sense that we are differentiating a power of t many times. When we did that $n$ times, the result was a factorial. Could we apply the same logic here? Why don’t we just go ahead and evaluate $\int t^n e^tdt$?

A minor problem

Due to the alternating signs in the first column of our table, we will not be able to properly evaluate the integral $\int t^n e^tdt$ – how would we know whether the nth term is positive or negative? However, there is an elegant workaround: we change our $g’(t)$ from $e^t$ to $e^{-t}$. Successive integrations of $e^{-t}$ have alternating sings, so when we multiply out with the alternating signs in the first column of our table, they will all cancel so that everything will be negative!

To show this clearly, consider the table to evaluate $\int t^n e^{-t}dt$:

Lattice multiplying the first few terms gives

\[t^n(-e^{-t}) - nt^{n-1}e^{-t} + n(n-1)t^{n-2}(-e^{-t})-\ldots\] \[= -t^ne^{-t} - nt^{n-1}e^{-t}-n(n-1)t^{n-2}e^{-t}-\ldots\]

Every term has a negative sign, because we have two sets of alternating signs that cancel to a negative! Because we know that all the terms are negative, we can proceed.

Integrating by parts n times

We will now evaluate $\int t^ne^{-t}dt$:

Thus we see that:

\[\begin{multline*} \int t^n e^{-t}dt = -t^ne^{-t} - nt^{n-1}e^{-t}-n(n-1)t^{n-2}e^{-t}-\ldots \\ - n(n-1)(n-2)(\cdots)(3)(2)te^{-t} - n!e^{-t} \end{multline*}\]

Note very carefully the last term – we have managed to produce an $n!$ (I would put another exclamation mark there, but…)

This is good news, but we now need to isolate the $n!$ by removing all of the other terms. The easiest way to do this is to cleverly choose limits on our integral such that all of the other terms, as well as the $e^{-t}$ in $-n!e^{-t}$ disappear to leave us with solely $n!$

To do this, we can first notice that if the lower limit of the integral were zero:

all of the terms that contain powers of x will disappear (since anything multiplied by zero is zero)
the last term would reduce to $n!$ because $e^0=1$.

Hence choosing a lower limit of 0 produces $n!$ and nothing more. Wouldn’t it be great if we could now choose a suitable upper limit that will make all other terms disappear? This can be done. Note that as t grows, $e^{-t}$ shrinks. In particular, as $t \rightarrow \infty$, $e^{-t} \rightarrow 0$. Thus if our upper limit where $\infty$, every single term would equate to zero. Just what we need, since our lower bound has already produced $n!$ Adding these limits to the integral:

\[\begin{aligned} \int_0^\infty t^n e^{-t}dt = \left[ -t^ne^{-t} - nt^{n-1}e^{-t} - \ldots - n!te^{-t} - n!e^{-t} \right]_0^\infty\\ = [0] - [-n!] = n! \end{aligned}\]

By using our well-chosen limits, we have produced a factorial (and nothing more). To reiterate:

\[\int_0^\infty t^n e^{-t}dt = n!\]

This is a hugely significant result, as we have shown that the factorial can be written as the result of an integral. But integrals don’t have to have integer components – we have managed to de-couple factorials from integers, which is what we set out to do! We shall therefore redefine the factorial, and change the variable from n to x to reflect the fact that we don’t just have to deal with integers any more:

\[x! = \int_0^\infty t^x e^{-t}dt\]

Doesn’t this look familiar?

Conclusion for part 2

We have shown that repeated integration by parts can produce a factorial, and in fact that

\[x! = \int_0^\infty t^x e^{-t}dt\]

Compare this to the gamma function, and you will see that there is only a slight difference:

\[\Gamma (x) = \int_0^\infty t^{x-1}e^{-t} dt\]

The first integral has an $x$ in the exponent, while the second has an $x-1$. You can connect the dots here on your own, but in Part 3 I will explicitly show how the gamma function related to the factorial.

Intuiting the gamma function, part 1

2017-07-11T00:00:00+00:00

What is the factorial of a half? This series of posts builds up from middle-school algebra to the enigmatic gamma function in an attempt to answer this simple question.

When high-school students study the Binomial Theorem, a classic problem is the expansion of a binomial with a fractional power, such as $(1+x)^{1/2}$. Speaking from experience, a student’s first attempt is often to naïvely substitute values into the formula for the Binomial Theorem, leading to

\[(1+x)^{1/2} = 1 + \binom{0.5}{1}x + \binom{0.5}{2}x^2 +\ldots+x^n\]

The problem arises when trying to evaluate $\binom{n}{r}$. We could just try to use the standard formula:

\[\binom{n}{r} = \frac{n!}{r!(n-r)!}\]

but the student will soon encounter the troubling question of how to evaluate $0.5!.$ Clearly, the traditional definition of the factorial does not make any provision for non-integers.

However, it turns out that there is a natural extension of the factorial to all complex numbers excluding the nonpositive integers called the gamma function, $\Gamma (x)$, which is defined using an integral:

\[\Gamma (x) = \int_0^\infty t^{x-1}e^{-t} dt, \qquad x \in \mathbb{C}| x \not\in \left \{ \{0\} \cup \mathbb{Z}^- \right \}\]

Don’t let the t distract you: it is just a dummy variable. The gamma function is a function of x. But the question is, what on earth has this got to do with factorials? As it happens, there is a wonderful intuitive reason for the form of the gamma function, and how it relates to $x!$, which I aim to explain. Following along will require the reader to have a reasonable understanding of differentiation and integration, but I will try to assume little else.

Just a disclaimer: the maths used here isn’t going to be formally rigorous, but it will definitely be valid and hopefully intuitive.

The Factorial

For $\forall n \in \mathbb{N}$ (this means ‘for all natural numbers’), the factorial, which will henceforth be referred to as the ‘traditional factorial’, is defined by the following two conditions:

$1! = 1$
$n! = n(n-1)!$, the recursion condition

These two conditions alone are sufficient to calculate any integer factorials.

For example, if we needed to find $4!$, we would simply apply the recursion condition multiple times until we see a $1!$, at which point we apply condition 1:

\[4! = 4 \cdot 3! = 4 \cdot 3 \cdot 2! = 4 \cdot 3 \cdot 2 \cdot 1! = 4 \cdot 3 \cdot 2 \cdot 1 = 24\]

Now the problem is that there is no obvious way to apply these two conditions to non-integers. We could conceivably apply the recursion condition

\[2.5! = 2.5 \cdot 1.5! = 2.5 \cdot 1.5 \cdot 0.5!\]

But we are now left with the conundrum of working out $0.5!$. Nevertheless, it is important to discuss this, because any function that claims to extend the factorial must also meet these conditions. We shall eventually prove that the gamma function does have these two properties, but first we must understand its shape.

The Gamma Function

Just a reminder that our final goal is to explain:

\[\Gamma(x) = \int_0 ^\infty t^{x-1}e^{-t}dt\]

We will approach this in a slightly round-about way. To begin with, we need to find an area of mathematics in which the factorial is used then see if we can tweak the method to work with non-integers. The most common use of the factorial is in permutations and combinations, but these deal with integers so are unlikely to be of use. However, another area in which the factorial appears is in Taylor’s theorem. Consider the Maclaurin series (a simpler special case of the Taylor series) for $e^x$:

\[e^x = 1 + x + \frac{1}{2!}x^2 + \frac{1}{3!}x^3 + \ldots + \frac{1}{n!}x^n + \ldots\]

Actually, this just begs the question: why is there an $n!$ in the Maclaurin series? This is interesting enough to warrant a tangent, though it will soon become apparent that it is not so much a tangent as a key to understanding the gamma function.

Deriving the Maclaurin series

We will assume that a function $f(x)$ can be written as a polynomial of degree n plus some remainder term $R(x)$ (which we will ignore):

\[f(x) = a_0 + a_1x + a_2x^2 + \cdots + a_nx^n + R(x)\]

This is a reasonable assumption, because you’ve probably seen that polynomials can often be made to look like other functions. Here is a graph on which I’ve plotted $y = \sin x$ as well as $y = x - \frac{x^3}{3!} + \frac{x^5}{5!}$, which are the first three terms of the Maclaurin series for $\sin x$.

But I digress. Going back to our expression for a general function $f(x)$, all we have to do is find the values of our coefficients $a_0, a_1, a_2, \ldots, a_n$, and we will have our approximation. Well for starters, we can easily find $a_0$ because it is the constant term. If we substitute in $x = 0$, everything except that constant will disappear, so we can see that:

\[f(0) = a_0\]

A nice trick is that we can actually also use similar reasoning to find every other coefficient. How? By differentiating. Notice that:

\[f'(x) = a_1 + 2a_2x + \cdots + na_n x^{n-1}\]

Now, substituting $x=0$ will kill all the terms except $a_1$. Hence:

\[f'(0)=a_1\]

To find $a_2$, there is a slight difference. Note that $a_2$ was initially the coefficient of the $x^2$ term, so when we differentiate $a_2x^2$ twice we arrive at $2a_2$.

\[f''(0)=2a_2 \implies a_2 = \frac{f''(0)}{2}\]

We can repeat this process, differentiating as we go, to find $a_n$. Just to make this clear, let us examine do this step by step (but only consider the last term):

\[\begin{gather*} f(x) = [ \ldots ] + a_n x^n \\ f'(x) = [ \ldots ] + na_n x^{n-1} \\ f''(x) = [ \ldots ] + n(n-1)a_n x^{n-2} \\ f'''(x) = [ \ldots ] + n(n-1)(n-2)a_n x^{n-3} \\ \cdots \\ f^{(n)}(x) = n(n-1)(n-2)\ldots(3)(2)(1)a_n = n!a_n\\ \end{gather*}\]

So for our polynomial, we conclude that:

\[f^{(n)}(0) = n!a_n \implies a_n = \frac{f^{(n)}(0)}{n!}\]

Substituting these coefficients back into $f(x)$:

\[f(x) = f(0) + f'(0)x + \frac{f''(0)}{2!}x^2 + \ldots + \frac{f^{(n)}(0)}{n!}x^n\]

which completes our derivation of the Maclaurin series. If you want to find the Maclaurin series for a particular function, you just have to compute it’s nth derivative and substitute in $x=0$.

Conclusion for Part 1

I know that this is an abrupt end, and that I haven’t even started to explain the form of the gamma function, but in this post I showed that factorials have a natural relation to calculus, when one repeatedly differentiates a power of x. I trust that because of this, the reader may be getting a sneaking suspicion that there might actually be a link between factorials and integrals, which would explain why the gamma function looks the way it does.

In fact, there is a certain type of integration that does involve repeated differentiation. If you did A-level mathematics or an equivalent, you will almost certainly have come across it, though you may not see right now how it relates to factorials. I am referring to Integration by Parts. Head to Part 2 to find out more.

Combinatorial optimisation with a pseudo-genetic algorithm

2017-07-05T00:00:00+00:00

a python approach to XKCD’s Social Seating problem

Social groups are remarkably complex affairs, and on many occasions this complexity can lead to awkward situations. Randall Munroe humorously portrays one such example in XKCD #173. This comic naturally begs the question – how do we find the optimal linear seating arrangement for a given social group?

If you’re not a fan of people trying to logically dissect non-mathematical things, now would be a good time to cut your losses (at least you got to see the XKCD comic!). Otherwise, don your cringe-protection glasses and we can move on.

The precise problem here is that the nonlinear ‘social graph’ has to be compressed onto a linear arrangement, which, as the comic points out, is computationally hard. Nevertheless, we will attempt to find a good arrangement (not necessarily the optimal one) by using a genetic algorithm of sorts. But before that, we must firstly quantify the various parameters in this problem.

Edit as of Jan 2020: reading back on some of my old posts, I typically realise that many of my solutions are excessively convoluted or poorly-explained. This post is no exception. One particular highlight is my rediscovery of adjacency matrices!

1. Developing the initial model
- Examining a social network
- A quick example
2. Refining the model
- The total enjoyment for a general arrangement
- Back to the XKCD network
3. Optimising the arrangement
- Genetic algorithms
4. Implementing the (pseudo-)genetic algorithm in python
Application to the XKCD network
Conclusion

1. Developing the initial model

What does it mean for a seating arrangement to be optimal? I am going to take a disgustingly utilitarian approach, by simply calculating the enjoyment per individual for a certain arrangement then summing over all individuals to find the total enjoyment. The optimal arrangement is then just the one with the highest total enjoyment (duh). Before going further, we must first decide on a way of describing social networks which will allow us to do this analysis.

Following the XKCD comic, each individual be denoted by a circular node. We will consider four types of relationships between individuals: friends, one-way ‘crushes’, romantic relationships, and acquaintances. I’m going to slightly modify the notation of the comic though:

The major advantage of this notation is that it encourages us to only consider the outgoing connections from a particular individual, which will be a big simplification when we come to considering an individual’s enjoyment. But it is far from perfect. For example, this notation implies that a romantic relationship is the same as a mutual crush, i.e. that an individual would get as much enjoyment from sitting next to their crush as they would with their partner.

Proceeding anyway, we assign each type of outgoing arrow a weighting, corresponding to how much enjoyment an individual experiences when sitting next to a person of that relation. Here is one proposed weighting:

Crush (arrowed line): weight = 2
Friend (solid line): weight = 1
Acquaintance (dotted line): weight = 0.5
Stranger (no line): weight = 0

Of course, the actual values are quite arbitrary – what matters is their relative ordering. I think it is fair to say that an individual’s seating preferences follow the above list in descending order. That is, given free choice, you would enjoy most to sit next to your crush, and least enjoy sitting next to a stranger. Choosing precise weightings would require you to quantify, for example, exactly how much more you would prefer to sit with a crush than with a friend – good luck with that!

A quick example

Now that we can describe a social network, let’s try to calculate the total enjoyment for a seating arrangement. We may as well use the network given in the comic. Here it is reproduced with our new notation (labelling each individual with a letter of the alphabet):

If this group has to sit down in a line, they will form an arrangement that will be denoted by a sequence of letters. For example, the arrangement BFGHECDA represents F sitting to the right of B, G to the right of F, etc. For each of the 8! possible arrangements, we could calculate a total enjoyment, denoted by $\epsilon_{\text{total}}$, which is done as follows. Consider the arrangement $ABCDEFGH$:

Assuming that an individual can only be affected by the individuals immediately to their left or right, each individual can have up to two outgoing connections, which can assume values of 2, 1, 0.5, or 0 depending on the relationship. For each individual, we add up these values, then to find $\epsilon_\text{total}$ we just sum over all the individuals.

This is easy to do with just one arrangement, but we need a robust way of finding $\epsilon_{\text{total}}$ for any arrangement.

2. Refining the model

We will define the system as follows:

Let the number of individuals in a network be denoted by $n$. In the XKCD network, there are 8 individuals, so $n=8$.
Each individual will be arbitrarily labelled $1, 2, 3, \ldots , n$, rather than using letters of the alphabet.
In order to define an individual’s relationships with other members of the group, we will associate a vector $x_i$ with each individual in the group, where $x_i \in \mathbb{R}^n$.
- This vector contains one element for each member in the network (including itself).
- The $m$th entry of the vector $x_i$ encodes the enjoyment produced for individual $i$ by individual $m$.
- We will denote the $m$th row of the vector $x_i$ as $[i, m]$.
- Let me reiterate that $x_i$ does not mean the i th element of a vector $x$. $\mathbf{x_i}$ is the vector!

That is a lot to take in, but hopefully this notation will become clearer with an example. Here is the XKCD network, with the letter labels replaced by numerals.

Corresponding to the sixth individual, we have: $ x_6 = \langle 0,0,0,2,1,0,0.5,0 \rangle $. Let’s break this down to make sure we are on the same page. Recalling our relationship weightings from earlier on:

The first three entries (and the final one) are equal to zero, because individual 6 has no relationship with individuals 1, 2, 3, and 8 (i.e they are strangers to him). We could also write $[6,1] = [6,2] = [6,3] = [6,8] = 0$.
The fourth entry is equal to two, because individual 6 has a crush on individual 4. We could also write $[6,4] = 2$.
The fifth entry is equal to one, because individual 6 is friends with individual 5. We could also write $[6,5] = 1$.
The seventh entry is equal to a half, because individual 6 is an acquaintance of individual 7. We could also write $[6,7] = 0.5$.

You may notice that I neglected to mention the sixth entry of $x_6$. $[6, 6]$ represents the enjoyment produced for the sixth individual by the sixth individual. I suppose that some people do enjoy being alone, but seeing as we are dealing with a group of friends at a cinema, I will set this entry to zero. That is to say, if $i=m, [i,m] = 0$. In our model, individuals get no enjoyment from sitting next to themselves.

We can write a vector $x_i$ for every individual in our network. After doing this, we do not need the graph any more – our network is fully defined by these n vectors!

The total enjoyment for a general arrangement

We will denote a general linear arrangement of individuals as $u_1, u_2, u_3, \ldots, u_n$. (earlier we had ABCDEFGH). You may be wondering why I have decided to add yet another set of variables – am I just trying to be novel in my notation? As it happens, there is a very important distinction between the individual’s location $\left(u\right)$ and the label of the individual ($i$). $\mathbf{u_1}$ is not the same as individual 1. $u_1$ could be any of the people in the network – but what matters is that this individual is in the first position of the arrangement. The u terms are a way of representing the order of people in their seats. Strictly speaking, I should therefore be saying things like “the individual at $u_1$”, but I will often just say “the individual $u_1$” or just “$u_1$”. This is an important point, so I’ll give you a concrete example:

\[\begin{align*} u_1 &= 3 \\ u_2 &= 7 \\ u_3 &= 1 \\ u_4 &= 2 \\ u_5 &= 5 \\ u_6 &= 8 \\ u_7 &= 6 \\ u_8 &= 4 \end{align*}\]

corresponds to a seating arrangement 3, 7, 1, 2, 5, 8, 6, 4. Remember, each numeral is the label of a person, so this could alternatively be thought of as CGABEHFD.

In order to calculate $\epsilon_\text{total}$, we need to calculate the enjoyment for each $u$ term then take the sum of all these. Because of our simplification that an individual is only affected by the people immediately to their left or right, a neat trick is to group the arrangement into pairs. So we will think of the arrangement as:

\[(u_1, u_2), (u_2,u_3), (u_3, u_4), \ldots, (u_{n-2}, u_{n-1}),(u_{n-1}, u_n),\]

where each pair of parentheses encloses the relationship between two individuals.

It is at this stage that we need to recall the previously-defined vector notation. Just as $x_6$ represents the relationship of individual 6 to the network, the vector representing individual $u_1$’s relationship to the network is $x_{u_1}$.

If we wanted to find how much individual 4 contributes to the enjoyment of individual 6, we would look in the 4th entry of $x_6$. Likewise, the enjoyment produced in individual $u_1$ by individual $u_2$ can be found by looking in the $u_2$th entry of the vector $x_{u_1}$. This subscript-within-subscript notation is very confusing, which is why we defined an alternative notation, the square-bracket notation wherein $[i, m]$ represents the $m$th row of the vector $x_i$.

With this notation, the $u_2$th entry of the vector $x_{u_1}$ is simply written as $[u_1, u_2]$. However, note that in the pair $(u_1, u_2)$, individual $u_1$ also produces some enjoyment in $u_2$. Thus the total enjoyment produced within the pair $(u_1, u_2)$ is equal to:

\[[u_1, u_2] + [u_2, u_1]\]

Now all we need to do is sum over all the $n-1$ pairs. Therefore:

\[\epsilon_\text{total} = \sum_{r = 1}^{n-1} \left( [u_r, u_{r+1}] + [u_{r+1}, u_r] \right),\]

where $r$ is a dummy variable.

Back to the XKCD network

Let’s calculate the $\epsilon_\text{total}$ of a random arrangement. I’ll choose this with my custom arrangement-randomiser:

def random_arrangement():
	# please note that this is a joke
    return [3, 2, 8, 5, 4, 6, 1, 7]

To begin with, we need to know the vectors $x_i$. These unfortunately have to be hard-coded based on the network. Earlier we showed the full form of $x_6$ and how we got there. Let’s repeat the same process to find $x_i, i = 1,2,3,…,8$ (because $n=8$).

\[\begin{align*} x_1 &= \langle 0,2,0,0,0,0,0,0 \rangle \\ x_2 &= \langle 2,0,1,0,0,0,0,1 \rangle \\ x_3 &= \langle 0,1,0,1,0,0,0,0.5 \rangle \\ x_4 &= \langle 0,0,1,0,0.5,2,0,0 \rangle \\ x_5 &= \langle 0,0,0,0.5,0,1,0,0 \rangle \\ x_6 &= \langle 0,0,0,2,1,0,0.5,0 \rangle \\ x_7 &= \langle 0,0,0,0,0,0.5,0,2 \rangle \\ x_8 &= \langle 0,,0.5,0,0,0,1,0 \rangle \end{align*}\]

In order to calculate $\epsilon_\text{total}$, we just need to apply our formula:

\[\begin{align*} \epsilon_\text{total} &= \sum_{r = 1}^{n-1} \left( [u_r, u_{r+1}] + [u_{r+1}, u_r] \right) \\ &= [3, 2] + [2, 3] + [2, 8] + [8, 2] + \ldots + [1, 7] + [7, 1] \\ &= 1+1+1+1+0+0+0.5+0.5+2+2+0+0+0+0 \\ &= 9 \end{align*}\]

It is worth commenting on the fact that all of the above values come in identical pairs (that is, $[u_r, u_{r+1}] = [u_{r+1}, u_r]$), which is not true generally. However, for this particular network, there is only one asymmetrical relationship, the one-way crush between individuals 7 and 8 – and in our particular arrangement, 7 is not next to 8.

We could have found $\epsilon_\text{total}$ without our vector notation, and simply with arithmetic by looking at the linkages on the graph. However, the process that we have described in this section does not require actually looking at the networks, meaning that we can write code to do it for us.

3. Optimising the arrangement

Now that we can effectively calculate $\epsilon_\text{total}$ for any given arrangement. We are ready to find the optimal arrangement.

A brute force search (i.e, calculating $\epsilon_\text{total}$ for every possible arrangement and returning the maximum), is not feasible due to the computational complexity, which is $\Theta(n!)$. For those not familiar with Big O notation and it’s variants – this is equivalent to saying that it’ll take one hell of a long time.

So we need an alternative. The problem is that many optimisation algorithms rely on calculus, which requires a continuous independent variable. Our independent variable is the arrangement, which is certainly not continuous, meaning that we cannot use many of these algorithms. How, then, do we approach this problem?

Genetic algorithms

A genetic algorithm is an optimisation heuristic (shortcut) that mimics Darwinian natural selection.

A brief summary of Darwin’s theory is as follows:

There is a population of phenotypes, individual members with different genetics and hence different characteristics.
This variety means that certain individuals are better suited to their environment.
The ‘fitter’ individuals survive and reproduce, passing on their genetic information to the next generation.
During the process of reproduction, random mutations may occur, resulting in slight changes to the child phenotype.
The process repeats, and over time, the species gets increasingly better adapted to the environment.

A genetic algorithm follows the same steps, but in computer science terms:

There is a ‘population’ of different possible arrangements.
Some of these arrangements have a higher $\epsilon_\text{total}$.
These phenotypes survive, and some of the survivors ‘reproduce’ – their arrangements are somehow combined. In computer science, this step is known as the crossover.
Random mutations occur, i.e. some arrangements may be changed slightly.
The process repeats, and over time, the mean $\epsilon_\text{total}$ of the population should increase.

4. Implementing the (pseudo-)genetic algorithm in python

In this section we will write python code to find the optimum arrangement for the XKCD network, though of course we could use it for any other network.

Let us represent our network as a list of lists, where each sublist corresponds to a vector $x_i$.

network = [[0, 2, 0, 0, 0, 0, 0, 0],
           [2, 0, 1, 0, 0, 0, 0, 1],
           [0, 1, 0, 1, 0, 0, 0, 0.5],
           [0, 0, 1, 0, 0.5, 2, 0, 0],
           [0, 0, 0, 0.5, 0, 1, 0, 0],
           [0, 0, 0, 2, 1, 0, 0.5, 0],
           [0, 0, 0, 0, 0, 0.5, 0, 2],
           [0, 1, 0.5, 0, 0, 0, 1, 0]]

network_size = len(network)
population_size = 16

However, don’t forget that python indexing starts from zero. So to find the enjoyment that individual 5 produces in individual 3, we look at network[2][4], not network[3][5]. We will also define two global variables: the network_size (number of individuals in the network) and the population_size (number of arrangements in the population).

Generating the population

Creating our population of arrangements is a simple matter of shuffling a list of numbers.

import random

def generate_population(network_size, population_size):
    """
    :param network_size: the number of individuals in the network
    :param population_size: the number of different arrangements
    """
    # Empty list which will contain the arrangements
    population = []
    # Produce the simplest arrangement, which is just the
    # individuals in order of their index, i.e [1,2,3,4,5,6,7,8].
    test_phenotype = list(range(1, network_size + 1))

    # Shuffle the test_phenotype to generate the population
    for i in range(population_size):
        random.shuffle(test_phenotype)
        population.append(list(test_phenotype))

    return population

generate_population(network_size, population_size)

This function will return the following:

[[3, 4, 5, 6, 8, 2, 1, 7],
 [1, 5, 2, 3, 7, 8, 6, 4],
 [1, 5, 6, 3, 2, 8, 4, 7],
 [7, 2, 6, 4, 3, 1, 5, 8],
 [1, 5, 2, 4, 6, 8, 3, 7],
 [5, 6, 3, 1, 4, 7, 8, 2],
 [4, 5, 2, 1, 3, 8, 6, 7],
 [5, 8, 4, 1, 7, 6, 2, 3],
 [7, 2, 5, 1, 3, 8, 4, 6],
 [5, 3, 1, 2, 6, 7, 4, 8],
 [7, 1, 2, 4, 6, 8, 3, 5],
 [7, 8, 2, 4, 6, 1, 3, 5],
 [2, 4, 3, 7, 1, 8, 6, 5],
 [1, 8, 7, 5, 3, 6, 4, 2],
 [3, 8, 5, 7, 2, 1, 6, 4],
 [3, 1, 6, 8, 7, 4, 5, 2]]

Evaluating a phenotype’s fitness

We now need to implement a function to calculate the $\epsilon_\text{total}$ of a particular arrangement. We do this by initially setting $\epsilon_\text{total} = 0$, then cumulatively adding the contributions from each pair.

def evaluate_etotal(arrangement):
    etotal = 0

    for r in range(network_size - 1):
        # u_r1 is meant to represent u_{r+1}
        # When defining u_r and u_r1, we subtract 1,
        # because indices in python start from zero.
        u_r = arrangement[r] - 1
        u_r1 = arrangement[r + 1] - 1

        # network[u_r][u_r1] is the same as [u_r, u_r1]
        # epair is the contribution from each pair
        epair = network[u_r][u_r1] + network[u_r1][u_r]
        etotal += epair

    return(etotal)

We can test this function on the random arrangement that we examined earlier, by running: evaluate_etotal([3, 2, 8, 5, 4, 6, 1, 7]). This returns a value of 9.0, which agrees with our calculation.

Survival of the fittest and reproduction.

Now that we have a function to measure the fitness of an arrangement, we need to find a way to remove the ‘weak’ individuals, leaving only survivors. Then, we need to let these survivors reproduce. But what does this mean in terms of arrangements and etotals? The first part is simple: we will consider an arrangement to be weak if it’s $\epsilon_\text{total}$ is less than the median $\epsilon_\text{total}$ (which I think is reasonable).

However, once we have the survivors, how can we get them to ‘reproduce’ to form new arrangements? The important aspect of reproduction, when it comes to genetic algorithms, is that the offspring receives features from both parents (who are themselves ‘fit’) – there is a crossover of genetic material. However, the individuals of our population are arrangements we cannot just take the first half of one and combine it with the second half of another.

I thought for a long time about this, and came up with a rather unsatisfactory solution. Instead of mixing genetics from two parents, I decided to define a child as half of a parent’s arrangement, plus the shuffle of the other half.

Yes, I know. It’s hardly reproduction – more like a glorified form of mutation, but this is why I have titled the post as a pseudo-genetic algorithm. If anyone can come up with a better idea for what reproduction could mean in the context of seating arrangements, I’m all ears.

import statistics

def survive_and_reproduce(population):
    # We will make a new list that stores the 'fitness' of each
    # phenotype in the population, where 'fitness'
    # depends on the e_total.
    population_fitness = [evaluate_etotal(phenotype)
                          for phenotype in population]

    # If a phenotype has an etotal lower than the median etotal,
    # it is WEAK.
    median = statistics.median(population_fitness)
    weaklings = [i for i in range(len(population))
                 if population_fitness[i] < median]

    # Destroy the weak arrangements by excluding them
    # from the survivor list
    survivors = [population[i] for i in range(len(population))
                 if i not in weaklings]

    number_removed = len(population) - len(survivors)

    # Now, the survivors 'breed'. This for loop maintains
    # the population_size.
    for i in range(number_removed):
        # Pick a parent from the survivors
        parent = random.choice(survivors)

        # A child is the first half of a parent,
        # plus the shuffled second half (estranged_father).
        estranged_father = parent[int(network_size / 2):network_size]
        random.shuffle(estranged_father)
        child = parent[0:int(network_size / 2)] + estranged_father

        # The child is now part of the new generation
        survivors.append(child)

    return survivors

Mutation

Mutations are random changes to the genetic material. With regard to seating arrangements, I will consider a mutation to be the swapping of any two members within the arrangement. A required parameter is the mutation rate, which encodes how often mutations should occur. A value of 0.05 (which is what we shall use) means that, on average, 5 in every 100 arrangements will contain a swap.

def mutate(generation, mutation_rate=0.05):
    # Each member of the population has a chance of being mutated
    for i in range(len(generation)):
        # random.random() returns a float between 0 and 1, so 
        # this is equivalent to a probability of the mutation_rate.
        if random.random() < mutation_rate:
            # A mutation entails two elements of
            # the arrangement swapping with each other.
            a, b = random.sample(range(network_size), 2)
            population[i][b],population[i][a] = \
                                    population[i][a],population[i][b]

Iteration

Evolution requires many generations to happen. In the context of our algorithm, this just means that we need to repeat the process of survival, crossover, and mutation over and over again.

That being said, we also want to get a feel for how our algorithm is doing, so I will write a progress function to do that. The idea is that at every iteration, we will run progress(generation), which will log the best $\epsilon_\text{total}$ of that generation, and the corresponding arrangement.

progress_list = []
hall_of_fame = []

def progress(generation):
    population_fitness = [evaluate_etotal(phenotype) for phenotype in
                          population]
    progress_list.append(max(population_fitness))
    max_location = population_fitness.index(max(population_fitness))
    hall_of_fame.append(population[max_location])

Application to the XKCD network

Now that we have everything ready, we are ready to find the optimal arrangement for the XKCD network.

network = [[0, 2, 0, 0, 0, 0, 0, 0],
           [2, 0, 1, 0, 0, 0, 0, 1],
           [0, 1, 0, 1, 0, 0, 0, 0.5],
           [0, 0, 1, 0, 0.5, 2, 0, 0],
           [0, 0, 0, 0.5, 0, 1, 0, 0],
           [0, 0, 0, 2, 1, 0, 0.5, 0],
           [0, 0, 0, 0, 0, 0.5, 0, 2],
           [0, 1, 0.5, 0, 0, 0, 1, 0]]

network_size = len(network)
population_size = 16
population = generate_population(network_size, population_size)

gen_number = 500

while gen_number:
    next_generation = survive_and_reproduce(population)
    progress(next_generation)
    mutate(next_generation)
    population = next_generation

    if gen_number == 1:
        champion_location = progress_list.index(
                    max(progress_list))
        print("The optimal arrangement is {}, with an etotal of: {}"
              .format(hall_of_fame[champion_location],
                      max(progress_list)))

    gen_number -= 1

import matplotlib.pyplot as plt

plt.plot(progress_list)
plt.title('Evolution of etotal')
plt.xlabel('Generation number')
plt.ylabel(r'$\epsilon$ total')
plt.show()

One example of running this gave the output:

The optimal arrangement is: [4, 2, 8, 7, 3, 1, 6, 5],
with an etotal of: 17

I ran this a number of times on my system, and plotted all the results at once.

Notice how it is quite volatile: many generations have a lower maximum $\epsilon_\text{total}$ than their parent’s generation. This is probably because our reproduction step is really just another form of mutation. However, because our code keeps track of best arrangement so far, all that matters is that our optimisation algorithm hits the maximum – it doesn’t have to stay there.

Conclusion

So there you have it. Using pseudo genetic algorithm, we have managed to find good linear seating arrangements given a friendship network.

This problem of optimising seating arrangements actually belongs to the much broader (and actively researched) field of combinatorial optimisation – so it is a cousin of the famous Travelling Salesman problem, as we could interpret our investigation as trying to find the optimal route through the social network. Perhaps we could initialise many salesmen and let them evolve! Food for thought.

I hope you enjoyed what turned out to be a rather long article. All the code is on my GitHub, feel free to fork it and try out your own networks.

Conway's Game of Life in python

2017-06-10T00:00:00+00:00

In this short post, I explain how to implement Conway’s Game of Life in python, using numpy arrays and matplotlib animations. I emphasise intuitive code than performance, so it could be a useful starting point for somebody to understand the logic before implementing a more efficient version.

Cellular automata

A cellular automaton (pl. automata) consists of a grid of cells; each cell has a state — either alive or dead. I will refer to this grid as the universe, though this is non-standard terminology. The automaton then iterates forward in time, with the new state of each cell being decided by a set of rules.

One of the most widely-studied cellular automata is the Game of Life, sometimes just called ‘Life’, which was discovered (or ‘invented’ if you prefer) by John Conway. It is so named because its rules can be interpreted as a very simplistic model of living cells.

Firstly, let us define the the eight grid squares around a cell as its neighbours. The rules of Life are then as follows:

1. A living cell will survive into the next generation by default, unless:

it has fewer than two live neighbours (underpopulation).
it has more than three live neighbours (overpopulation).

2. A dead cell will spring to life if it has exactly three live neighbours (reproduction).

Clearly, no cells can come to life unless there are already some living cells in the universe (Aquinas’ primum movens?). Thus, we have to provide the universe with a seed, i.e a set of initial living cells. The massive variety and complexity of the results, often starting from simple seeds, is what makes cellular automata so interesting.

A python implementation

This is a rough overview of our plan of attack:

Initialise an empty universe
Set the seed in the universe
Determine if a given cell survives to the next iteration, based on its neighbours
Iterate this survival function over all the cells in the universe
Iterate steps 3–4 for the desired number of generations

1. The universe

I am using numpy arrays to represent the universe, as they are easy to initialise and manipulate. Let us start with a small universe consisting of a six-by-six grid. In our universe, each cell will either be a 1 (alive) or 0 (dead). For now, they are all dead, so we initialise with np.zeros

import numpy as np
universe = np.zeros((6, 6))

2. The seed

We must now decide on the seed. I will choose a simple oscillator, known as the beacon, which evolves as follows:

If you are struggling to see how this works, simply count the number of living neighbours for each cell then apply the rules of Life as presented earlier.

In python, we will set the seed like so:

beacon = [[1, 1, 0, 0],
          [1, 1, 0, 0],
          [0, 0, 1, 1],
          [0, 0, 1, 1]]

universe[1:5, 1:5] = beacon

It is worth discussing at this point how we plan on visualising our universe. The easiest solution I have come across is to use matplotlib’s imshow, which outputs a grid of colours corresponding to the value of every entry in an array. Using the binary colourmap, we have:

import matplotlib.pyplot as plt
plt.imshow(universe, cmap='binary')
plt.show()

3. Survival

We now need a function which will do the following:

Take as arguments the x and y coordinates of a cell
Examine the neighbours of this cell
Decide whether the cell survives
Write this into a new universe

The last point is rather subtle — we cannot update the cell in place, because this will affect the surrounding cells in the same iteration. Instead, we need to compute the survival of every cell based on a snapshot of the current universe, then simultaneously write all of these new cells into our universe. A cheap shortcut is to write the new values into a copy of the universe called new_universe, while leaving universe as a read-only. Then, later on, we can set the universe to be equal to new_universe.

# Create an identical copy of the universe, which will be the next generation.
new_universe = np.copy(universe)

def survival(x, y):
    """
    :param x: x coordinate of the cell
    :param y: y coordinate of the cell
    """
    # Find the number of living neighbours by taking
    # the sum of the 8 surrounding grid squares
    num_neighbours = np.sum(universe[x - 1:x + 2, y - 1:y + 2])
                     - universe[x, y]

    # If the cell is alive
    if universe[x, y] == 1:
        if num_neighbours < 2 or num_neighbours > 3:
            new_universe[x, y] = 0
    # Otherwise, the cell is dead...
    elif universe[x, y] == 0:
        # ... but it will rise again
        # IF there are three living neighbours
        if num_neighbours == 3:
            new_universe[x, y] = 1

I know that the if-else statements can be simplified drastically, like such:

if universe[x, y] and not 2 <= num_neighbours <= 3:
    new_universe[x, y] = 0
elif num_neighbours == 3:
    new_universe[x, y] = 1

But it is much more intuitive to keep the logic in its ‘expanded’ form, wherein one can clearly see the rules of Life.

4. Compute one full generation

This is a simple matter of applying our survival function to every cell in the universe, then setting universe to be equal to new_universe.

def generation():
    # We are inside a function, so we must specify
    # that we are referring to the global variable 'universe'.
    # Otherwise python will think that we are trying to
    # create a new variable also called 'universe'.
    global universe

    # Simple loop over every possible xy coordinate.
    for i in range(universe.shape[0]):
        for j in range(universe.shape[1]):
            survival(i, j)

    # Set universe to be equal to new_universe.
    universe = np.copy(new_universe)

5. Iterate and animate

There is no point iterating unless we can visualise the changes. To do so, we can simply use matplotlib’s animation module (see some examples here).

import matplotlib.animation as animation

fig = plt.figure()

# Remove the axes for aesthetics
plt.axis('off')
ims = []

for i in range(30):
    # Add a snapshot of the universe, then move to the next generation
    ims.append((plt.imshow(universe, cmap='Purples'),))
    generation()

# Create the animation
im_ani = animation.ArtistAnimation(fig, ims, interval=700,
repeat_delay=1000, blit=True)

# Optional to save the animation
im_ani.save('beacon.gif', writer="imagemagick")

Putting it all together, the output is:

Yes, I changed the colour to purple. Get over it.

Where here to go from here

In principle, we are done. Now it is up to you to experiment with different seeds! But I can’t resist one more demonstration, using the R-pentomino seed:

Perhaps you’d expect its behaviour to be vaguely similar to that of the beacon. But the wonderful thing about Life is the dazzling complexity that can result from a simple seed. See for yourself:

Check out my PyGameofLife repository on GitHub, which includes a slightly more professional refactoring with a number of built-in seeds allowing a user to generate gifs from the command line. The R-pentomino animation above can be created with:

python life.py -seed r_pentomino -n 500 -interval 50

Feel free to submit a pull request if you’d like to add more seeds or functionality.

The Cambridge Natural Sciences Interview

2017-05-01T00:00:00+00:00

In December 2015, just after finishing my IB exams, I went to Cambridge for my (Physical) Natural Sciences Interview. The interview has a reputation for being incredibly demanding and intense, and you’ve probably seen on the internet some of the ‘crazy’ questions that people get asked. Although it is definitely stressful, the Cambridge interview is almost purely academic (and quite reasonable). So without further ado, these are the questions I was asked, along with my thoughts at the time and the interviewer’s reactions.

Interview 1 – physics and chemistry

In your Personal Statement, you mentioned that you had done an experiment to determine the Planck constant. Tell us a bit about this.

Prior to the interview I had thought about this a lot, because I knew they were going to ask me about it. So I just described the experiment I did and what I learnt from it. The interviewers seemed very impressed by this, and they didn’t ask anything more about it.

What does F = ma mean

Of course it’s one of Newton’s laws, so with abounding confidence I said “that’s Newton’s first law!”. Something about that sounded weird, but the damage was done and I will never forgive myself for failing to identify Newton’s 2nd. I was mortified, but the interviewer chuckled and said not to worry about it. I recovered this by giving a half-decent explanation in terms of inertia, i.e reluctance to move. I think this is one of the setup questions to test your fundamental understanding before moving onto harder stuff.

Imagine we have a box, hanging by a light inextensible rope, in an elevator. The box weighs 5kg. The lift is accelerating upwards at 4$ms^{-2}$. What is the tension in the rope?

We will now connect another box, via a light inextensible rope, to the bottom of the box in the previous question. What is the tension in this new piece of rope?

I was familiar with what happens when you put weighing scales in lifts, and this problem was similar. But I got quite confused doing the question, especially with the signs of acceleration, force etc. Frustrating, because I know that if it were an exam I’d be able to do it easily. I now see that the whole question would have easily (and impressively) been done by considering the principle of equivalence. I think I did eke out the correct answer in the end, but it wasn’t pretty.

For the second question, as soon as the question was asked, I made an intuitive observation regarding the system, and it turned out that this is exactly what he wanted. I can’t remember exactly what I said, but I think it was something regarding how we can consider the bottom of the second box as the new ‘roof’ of the elevator, so to speak.

At this point, the second interviewer (who I believe specialised in chemistry) took over. I was really not looking forward to this part, because on my personal statement I said that I had read Why Chemical Reactions Happen – I did read it, but it was a lot harder than I expected it to be, and I wasn’t too sure that I could answer questions about it.

Tell me about hydrogen bonding

I explained it in terms of electron density. When a hydrogen atom moves near an oxygen atom (for example), the high positive charge density of the oxygen draws toward it the hydrogen electron cloud, leaving the hydrogen atom with a partial positive charge. I think you have to be careful not to rattle off textbook definitions, you have to actually understand what you’re saying.

If I mix some lithium ions in water, what happens?

I said something about coordinate complexes. Although lithium isn’t a transition metal, it has a very high charge density, so I suggested that it could form some kind of ligands. I drew a little diagram with curly arrows going from the lone pairs on $OH$ or $H_2O$ to the lithium ion.

If I mix some aluminium ions in water, what happens?

This one threw me off, because I would have thought that aluminium behaved the same way as lithium. So when they asked me about aluminium after asking me about lithium, I figured something was up. I tried to make up some stuff about Lewis acid-base character (when I say ‘make up’, I mean that I brought up facts that I knew were true, but of questionable relevance).

In the end, after some fidgeting, I decided to be honest. I said “I don’t see how it’s different to the lithium case”. She then asked the next question (very closely related).

Is the aluminium-ligand bond stronger or weaker than the lithium-ligand bond?

I first stated the relevant features. The ionic radius, and the charge. Aluminium obviously has greater charge, but then on the other hand I thought it had a larger radius (I was wrong). I didn’t know how to compare these opposing factors. I reasoned that, because of Coulomb’s law, the radius would be more important (because it is an $r^2$), but she informed me that the charge was much more significant.

In hindsight, I could have worked that out. Aluminium has 3/2 times the charge. For the ionic radius to be more significant, aluminium’s radius would have to be at least $\sqrt{3/2}$ (about 1.2) times greater than that of lithium.

What happens when $Al_2Cl_6$ reacts with water?

I think a description would have sufficed, but I wrote the chemical equations

\[Al_2Cl_6 + 6H_2O \to [Al(H_2O)_6]^{3+} + 6Cl^{-}\] \[[Al(H_2O)_6]^{3+} \to [Al(H_2O)_5OH]^{2+} + H^+\]

I also mentioned something about the pH decreasing.

In the above equilibrium, how would I produce more of the metal ion

I had no idea what I was being asked. I clarified once, but still didn’t get it. But I figured that it was something to do with Le Chatelier’s principle, so I talked about how adding a strong acid or strong base would affect the equilibrium. This seemed to satisfy them.

Any questions?

I asked “what do you think is the next upcoming field in your subject?”. Physics guy said ‘energy’, chemistry person agreed. Obviously I asked this question because everyone told me that you have to ask something good at the end. I figured I had nothing to lose by asking.

All in all, I felt that the first interview was disastrous, especially regarding chemistry, even though I wasn’t asked anything about Why Chemical Reactions Happen.

Interview 2 – physics and maths

I felt that I was more prepared for this interview, because it didn’t involve chemistry.

Have you heard of Felix Baumgartner?

Yes, he jumped off some balloon. I was informed that he had jumped from 40,000m.

Tell me about the forces involved in his jump

Obviously, the force of gravity, and the force of air resistance. I said that I couldn’t find his equation of motion, because I didn’t know about the air resistance. We came back to this shortly.

What value of g are you using?

I perceived that the question was about the variation of g. I wrote down $g=GM/R^2$. I knew that the radius of the earth was about 6400km, so I did a quick mental calculation to note the difference in order of magnitude between 40,000m and 6400km. Therefore, I concluded that the variation of g was not relevant. He agreed, but asked me to quantify it.

I replaced $R$ with $R+h$, ($R$ is the Earth’s radius and $h$ is the height above the surface) and expanded, but didn’t know how to proceed. I think the key point was that the $h^2$ term vanishes in comparison to the $R^2$ and the $2Rh$.

\[g = \frac{GM}{(R+h)^2} =\frac{GM}{R^2 + 2Rh + h^2} \approx \frac{GM}{R^2 + 2Rh}\]

How does air pressure vary as a function of height?

This was quite a frustrating question. Everyone knows that there is an inverse relationship, but I had never stopped to consider the exact function.I said that I could work probably work it out if I knew the distribution for pressure or density.

It turns out that it can be derived using the ideal gas equation to find the density as a function of height and $dP$, then substituting this into $P = \rho g h$ to get a differential equation.

Have you heard of the Boltzmann distribution?

Yes, it is a distribution for the velocities of particles in an ideal gas. I don’t know the precise form of the equation, except that it has an $\exp(\frac 1 T)$ in it.

This was a weird question, and seemed a bit out of place. But it related to the previous question. Since I didn’t know the form of the Boltzmann distribution, the guy just said “assume that the pressure halves every 5000m”. As he said this I wrote down $\rho = \rho_0 (\frac{1}{2})^{h/5000}$, and he seemed satisfied.

With the above in mind, could Felix have broken the sound barrier?

I thought that he wanted me to use the pressure function from the previous question to do the differential equation. But when I started, the interviewer got a bit impatient and told me that the pressure thing was a separate part of the question. I think that was a bit unfair of him, but never mind.

What they wanted was much more simple. I had already shown that changes in g were pretty much irrelevant. Likewise, because of the exponential relationship of air pressure, air resistance was also negligible. So it reduces to a simple SUVAT equation. I did it mentally and told him the answer, and he was surprised that I got it correct.

At this point, the second interviewer took over. I was thankful to switch interviewers (the first had been a bit grumpy), but I knew that the second interviewer was the author of my IB maths textbook, which is a bit of a daunting prospect.

Integrate $f(x) = \arcsin(x)$

Standard trick, use integration by parts with $u=\arcsin(x)$ and $dv = 1$.

Differentiate $f(x) = \arcsin (\cos x)$

I was very happy to get some calculus, because I thought it was my strong suit.

I started off by saying that I would simplify the function, but this is actually a mistake because it can’t be simplified – I got $f(x)$ confused with something similar, $\cos(\arcsin x)$ (which can be simplified). So I had to backtrack and just differentiated it normally.

Easy application of the chain rule, so as usual I skipped a few steps (which I later regretted).

\[\frac{d}{dx} \arcsin (\cos x) = -\frac{\sin x}{\sqrt{1-\cos^2 x}} = -1\]

(Do you spot the mistake here? I didn’t)

Find $\int_0^1 f(x)dx$

I had absolutely no clue how to integrate it (at first). I started off by suggesting substitutions, but knew that these wouldn’t work.Luckily I had a mini ‘eureka’ moment – the derivative is constant!. This means that the $f(x)$ ALWAYS has the same negative gradient – it’s just a straight line.

So I knew that $f(x)$ could be rewritten as $–x + c$. I didn’t know how to proceed at first, but the interviewer suggested “how might you find c?” and I worked it out. $c = f(0) = \pi/2$. The integral is then just the area of the triangle. The interviewer seemed very pleased that I had managed to spot this method. However, he then smiled and asked the next question.

That’s a nice method. But the answer is wrong. What did you miss?

As soon as he said I missed a step, I knew exactly where my error was - I felt that I had messed up the square root of $(1 - \cos^2 x)$ in the derivative. He agreed, but after this it took me a while to figure out exactly what it was. I suggested that I had forgot a +/-, but it wasn’t that. We then had a really sad discussion (sad and embarrassing for me) about square roots.

What is the square root of 16?

+/- 4, but the sqrt sign technically means only the positive root, so 4.

What is the square root of 4?

What is the square root of $( -2)^2$?

I got really flustered, said -2, then quickly corrected to 2.

What is the square root of $x^2$?

Thank goodness I realised what he was getting to, the answer is clearly $|x|$. I’m ashamed it took me so long. Having figured this out, I quickly realised that the function would look like a sawtooth, and said so.

Overall, I was pleased that I figured out the integral, but very humbled by the square root discussion. Of course, I missed an obvious trick to instantly figure out the integral:

\[\int_0^1 \arcsin (\cos x)dx = \int_0^1 \arcsin (\sin(x + \pi/2))dx\]

What is the factorial of a half?

I smiled and said $\sqrt{\pi/2}$. (I mentioned the Gamma function in my personal statement).

In your maths exploration, how did you show that the gamma function worked as a factorial?

Repeated integration by parts. I asked if he wanted me to do it, he said no and seemed to be content with this answer.

Is the Gamma function the only function that can model a factorial?

I said that it wasn’t: given any (finite) set of coordinates, there are an infinite number of functions that can model it (polynomials).

However, I said that I had proved that the gamma function satisfies the recursion principle, and that $\Gamma(1)=0$. This means that it is at worst, a superset of the factorial. I also said that I was aware that the Gamma function had an extra property which meant that it was the only function that satisfactorily extended the factorial to non-integers, but I was honest and said I didn’t know exactly what it was.

He told me that the name of the property was analytic continuation. Thanks mate.

Why do you want to study natural sciences?

This is a question that everyone should prepare for, but I mucked it up. In my personal statement, I had already talked about the philosophical aspects – I love to find simple explanations, reductionism etc. I hinted at that in this answer, but ended up saying “these are where my strengths lie”. The grumpy interviewer said “so you’re basically saying that you’re good at this subject”. I gave a cautious “yeah”

How are you going to keep your academics up to scratch prior to entering?

I said that I had already planned which books I would study, mentioning the Feynman lectures. I think this was a smart choice, because I know that physicists and mathematicians love Feynman. They were happy, and said that it was a good book.

Any questions? You don’t have to try to impress us with good questions, we’re genuinely asking if you have any questions.

I asked something administrative about UCAS.

THE END

Conclusion

I really thought that I had screwed it up afterwards, but everyone had told me that that’s how you’re meant to feel. The interviewers obviously know more than you, and they want to probe you until you don’t know something, so it makes sense that you will feel out of your depth. That being said, even thinking about objectively I knew that I had made a poor showing: getting Newton’s laws mixed up, and getting destroyed by some simple square roots.

However, I decided not to dwell on it and to enjoy my Christmas break: it had been a rough couple of months of IB and interview prep. In the end, I did get in, so I guess some of my better answers must have made up for the failures.