Unless you believe the premise that there is something metaphysical about the general nature of human intelligence, there is nothing stopping us from engineering an intelligence superior to our own.

We’ve long since built machines far more physically capable than humans.

We have machines that can lift weights far beyond human capability (BelAZ 75710 truck, Liebherr LR 13000, Bagger 293…), that can unshackle from the gravitational pull and go supersonic (SR-71 Blackbird, Concorde…), that can dive miles beneath the ocean’s surface (Deepsea Challenger, Project 941 Akula submarines…), that can leave Earth’s orbit and head for the stars (Voyager probes, Starship…), or orbit around the Earth for many years (ISS, Starlink…).

We’ve engineered sensors that stretch the boundaries of perception itself.

With LIGO, we can detect ripples in spacetime — gravitational waves — triggered by black hole collisions over a billion light-years away. Using IceCube, a cubic kilometer of ice buried in Antarctica, we can sense elusive neutrino particles that pass straight through entire planets. Our instruments span the full electromagnetic spectrum: from radio telescopes that capture long-wavelength signals to gamma-ray detectors that see the universe’s most energetic events — far beyond the narrow 380–700 nm band visible to the human eye. We’ve also built detectors that reveal invisible ionizing radiation — alpha, beta, and gamma particles — using tools like Geiger counters.

Just as we’ve extended our muscles and senses with steel and silicon, we now stand poised not merely to extend, but to surpass our own cognition.

Engineering general intelligence (AGI) is the new frontier.

But what exactly do I mean by AGI (artificial general intelligence)?

There are hundreds of definitions with varying levels of formality, but most boil down to two camps: one that’s useful but informal, and one that’s formal but not especially helpful.

Here is a formal definition from Legg & Hutter’s Universal Intelligence: A Definition of Machine Intelligence [1] paper:

Roughly: compute the above score for your AI agent and once it crosses the “human score” (e.g. median human’s score) you have an AGI.

It’s not particularly useful because it relies on an infinite sum over all environments of interest, weighted by an incomputable Kolmogorov complexity function. Elegant, but impractical.

A useful, informal definition (inspired by OpenAI’s definition [2]) goes like this: we’ve achieved AGI once we have an intelligence that’s useful across a broad set of economically valuable tasks.

How do we define useful?

-> Customers are willing to pay for that AI system/agent’s services because it performs those tasks on par with human experts. :)

We could also define AGI in terms of its economic impact — for example, by tying it to global GDP growth (e.g. “we’ve built AGI once we see 10% annual global GDP growth”). But structural bottlenecks — societal (e.g., resistance to adoption), physical (e.g., shortage of humanoid robots), and others — might still prevent that from happening, even after AGI is achieved.

As a natural extension, artificial superintelligence (ASI) is defined as being strictly Pareto-dominant over humans across all economically relevant tasks.

We already have datapoints of artificial superintelligence (ASI) in narrow, constrained domains — games being a notable example. DeepMind’s AlphaGo family of models is a well-known case, built to master Go, a complex strategic board game originating from China.

AlphaGo began with imitation learning from human gameplay (much like how infants first learn), training on hundreds of thousands of expert games (~30 million moves in total [3]).

The next iterations in the AlphaGo family — AlphaGo Zero, AlphaZero and MuZero in particular — eliminated human gameplay data entirely, allowing the systems to learn from first principles using only the reward signal and the environment (i.e., the rules of the game).

Through self-play — and a massive amount of compute — the first Go superintelligence emerged.

AlphaGo’s unexpected creativity took Lee Sedol — and the global Go community — by surprise. The now-legendary “Move 37” [4] entered the history books.

Subsequently, AlphaGo Zero defeated the original AlphaGo 100–0 [5]! And AlphaZero and MuZero took it even further — generalizing beyond Go to master chess and shogi (AlphaZero) and extending to Atari games as well (MuZero).

This happened nearly a decade ago (back in 2016), and for years afterward, there seemed to be no clear path to achieving that level of superintelligence in open-ended, world-like environments — given their immense complexity.

Until recently — when we discovered Generative Pretraining of Transformers (GPT) [6], a particular model class trained to predict the next word in a sentence (i.e., language modeling objective) across tens of trillions of words from the internet.

The GPT family was born when Alec Radford pushed the scale of the transformer architecture beyond what Vaswani et al. had originally demonstrated in their seminal Attention Is All You Need paper [7].

It all boils down to two core ideas (The Bitter Lesson [8] was truly prescient):

• Use a hardware-friendly model architecture — one that is highly parallelizable (the transformer). This yields a strong scaling curve (with a steep slope), and remarkably, we’ve needed only minor architectural tweaks since 2017.
• Scale ad infinitum — data, parameters, compute. The scaling laws (Kaplan et al. [9]) and follow-ups like the Chinchilla paper [10] formalized this intuition.

My intuition for why this works is the following:

• I’ll skip the mathematical rigor and keep it simple. As the number of parameters in your transformer goes to infinity, the probability of getting stuck in a strict local optimum — a point where the loss increases in every direction — drops to zero. Take a 1-trillion-parameter LLM (not hypothetical — GPT-4 reportedly had ~1.8T). You’re doing gradient descent in a trillion-dimensional space. For a point to be a strict local optimum, the loss must curve upward along every single dimension. Assume the probability that the loss increases along any one direction is p. Then the chance it increases along all directions (under simplifying assumptions) is roughly p¹⁰⁰⁰⁰⁰⁰⁰⁰⁰⁰⁰⁰ (i.e., p raised to 1 trillion). Even if p = 0.999, this evaluates to essentially zero. Consequently, gradient-based optimization is overwhelmingly unlikely to terminate in a spurious local minimum; instead it almost surely lands in wide basins separated by saddle points, which dominate high-dimensional landscapes. The loss keeps going down! :)
• The world model emerges from next-token prediction. I really liked Ilya Sutskever’s example here: Imagine a 100-page crime novel. After a complex plot, on the very last page, the detective gathers everyone and says, “I know who the murderer is. The murderer is _.” How do you maximize the probability of that final token (or equivalently, minimize the loss)? You do it by understanding the entire story that came before. You have to grok the characters, the relationships, the clues — everything. In other words, you have to build a world model. The world model is not explicitly trained — it emerges as a byproduct of the prediction objective.

Today, we pretrain our large language models (LLMs), with trillions of parameters, on tens of trillions of multimodal tokens (both human and synthetic). Then, during post-training, we teach them to follow instructions (via imitation learning) and to align with the “median human preference” (via RLHF — reinforcement learning from human feedback).

This was the new AlphaGo moment — Proto-AGI, if you will.

The problem is, imitation learning from the internet can only take us so far — it can match human capability, but not surpass it. At best, it gets us to AGI. So how do we reach the AlphaZero moment for real-world, open-ended intelligence?

We need an equivalent of self-play — and some sort of reward mechanism — for the open-ended world.

As OpenAI’s Noam Brown (who works on reasoning models like o3) pointed out [11], self-play converges to a minimax equilibrium in two-player zero-sum games like Go and chess — where that outcome is ideal. But in general-sum, multiplayer environments, that same (self-play) policy breaks down and fails to generalize.

This is where RLVR — reinforcement learning from verifiable rewards — enters the picture. It’s the approach behind models like o1, o3, and DeepSeek R1.

The idea is to create RL environments where outputs can be easily evaluated against ground truth. Domains like mathematics (where you can compare against the correct answer), and programming (where you can check against test cases) are ideal.

Reward model traces that lead to correct solutions; penalize those that don’t. It turns out this setup is quite powerful — various cognitive and metacognitive skills, such as self-verification, backtracking, planning, and more, begin to emerge in LLMs.

The field has evolved from a “scale only data, parameters, and compute” paradigm to one that also scales the quantity and diversity of RL environments — and it’s already showing incredible promise.

We seem to be going from “Yann LeCun’s world”:

to an entirely new one as described by OpenAI’s Dan Roberts in his recent talk [12]:

The Welcome to the Era of Experience [13] paper by David Silver and Rich Sutton goes a step further — proposing to drop pretraining (i.e. imitation learning) altogether and train solely through RL. Not surprising, given that David led the AlphaGo project.

In theory that sounds promising — and with infinite compute, it would likely outperform models biased by human language. But in practice (finite compute regime :) ), it may simply be too inefficient for today’s hardware.

The idea is to let LLMs evolve their own internal language (since human language is by no means optimal) as long as they can ultimately solve arbitrary problems. If a model can solve cancer, do we even care how it thinks?

Well, we certainly should — for reasons of safety, alignment, and interpretability. But if the choice is between an alien artifact that cures cancer (but we don’t understand it) and a familiar one that can’t… we might just take the risk.

In a similar vein, a recent paper from Meta — Training Large Language Models to Reason in a Continuous Latent Space [14] — modifies the LLM to output latent tokens directly, skipping the decoding step into human language. As long as it can ultimately solve the problem, that might be good enough.

These new research directions seem like an incredibly promising path to ASI. Combined with geopolitical dynamics involving China, massive capital investment into ever-larger compute clusters (on the order of millions of GPUs [15]), and a growing concentration of talent in the field — we might get there sooner than most expect.

Some people are bullish, believing we’ll reach ASI by 2027 [16]. My estimate falls between 2027 and 2030, with the highest likelihood somewhere in the middle.

The list of “machines can never do X” has been shrinking for centuries. From Ada Lovelace’s objection [17] “The Analytical Engine has no pretensions to originate anything. It can do whatever we know how to order it to perform” to Gary Marcus’s [18], and I’m paraphrasing, “AI is dumb — it can’t even draw a horse riding an astronaut” we’ve now reached: “Well, sure, but can it get a gold medal at the IOI? Is it even AGI?” [19]

Even if you believe human intelligence has some kind of metaphysical quality, it seems we’re getting closer to engineering it. But on the flip side — how much understanding and intelligence do humans truly possess?

Note: The following subsections reflect my current understanding of reality, along with a fair amount of speculation. Strong opinions, loosely held.

Human understanding is local — much like in the Chinese Room Argument

Do we truly understand? Or are we merely experiencing the illusion of understanding?

When we stub our pinky toe against a wall, it feels as though the pain is located in the toe itself. But in reality, the sensation is generated in the brain — the mind simply creates the illusion and projects the pain back onto the physical location where the impact occurred.

When a person loses a limb, they often still feel sensations, sometimes even pain, as if the missing part were still there. This phenomenon, known as phantom limb pain, is another striking example of the brain generating and projecting bodily experiences.

If there are no retinal cells at the point where the optic nerve exits the eye — our so-called blind spot — why don’t we notice a hole in our vision? The brain doesn’t just leave it blank; instead, it inpaints the missing information by filling in the gap using surrounding visual cues. Once again, perception isn’t a direct feed from reality — it’s a constructed experience, patched together by the brain to create a seamless illusion of completeness.

Ever since I read Alan Turing’s seminal paper Computing Machinery and Intelligence [20] followed by John Searle’s Chinese Room Argument [21], I’ve been struck by how closely the latter mirrors what I believe is the nature of human understanding. Rather than refuting machine intelligence, it seemed to describe us.

I believe the illusion of understanding arises from the magical performance of System 1 thinking being interpreted — and often misinterpreted — by a puzzled System 2 [22].

Neurons clearly aren’t intelligent. They don’t understand language.

And when we zoom out beyond the individual neuron, there’s still no true understanding in System 1. It’s just neural blobs performing classification, and pattern matching. Complex functions living in high-dimensional space, optimized by the evolutionary processes.

So the understanding should then reside in System 2?

But understanding always seems to be constrained to local computations — like that of the human in the Chinese Room.

We don’t truly understand how our own body works — it’s far too complex of a system. What we do understand is how to drive it, with varying degrees of success. If I want to grab an apple from the table, all I have to do is set a mental objective — fetch that object — and the rest unfolds automagically through neural circuitry I don’t understand.

Nor do we understand how neural networks work — biological or artificial. We know how to make useful functions emerge through gradient descent and the right data, but we don’t really understand what’s going on inside those functions. That’s the challenge mechinterp researchers are trying to tackle.

Going beyond the individual, we don’t understand how large companies work. Take Alphabet, for example. It’s too complex for any single human to truly comprehend, including Sundar Pichai. He operates with a low-resolution view across a broad range of areas. One could argue that while Alphabet’s employees collectively run the company, none of them truly understand it. They don’t grasp the full system — they just know how to drive their part of it.

And of course, we don’t truly understand how the economy works, a vast, dynamic graph of superorganisms we call companies, and individuals acting within them.

Our understanding feels like an illusion — much like phantom limb pain or our failure to notice the blind spot in our vision.

Do we truly understand, or do we just feel like we do?
I believe it’s the latter.

Humans are carbon-based biological pattern-matching machines

Introduce a third copy of chromosome 21 instead of the usual two, a glitch in the matrix, and you get Down syndrome [23].

Mute the body’s response to growth signals and you get growth hormone deficiency (GHD) [24], leaving adults with childlike proportions.

With Hutchinson-Gilford Progeria Syndrome [25], it’s the opposite — the body speeds through aging, as if someone hit fast-forward on the biological timeline.

Each disorder is a bug report from our wetware — proof that underneath it all, we’re all just biological machines.

I remember watching the sci-fi movie A.I. Artificial Intelligence as a kid, where a boy eventually discovers he’s actually a robot. That idea really unsettled me. Imagine living your whole life thinking you’re human, deeply believing and feeling as one — only to find out you’re just a machine.

But are we really so different from the hosts in Westworld, or from David in A.I. Artificial Intelligence? Or is it merely a difference in substrate — biological versus silicon?

We tend to overestimate the average human’s intelligence

A few weeks ago, while riding toward San Francisco, my Uber driver gave me what he called a “difficult puzzle.” With pride in his voice, he said it had taken him 30 minutes to solve. The puzzle? “What’s the square root of 0.1?”

I gave him a symbolic answer: “1 divided by square root of 10”. He shook his head — “That’s not the answer.” Realizing he was looking for a non-symbolic answer, I said, “1 divided by roughly 3.33” and he lit up: “That’s it!”.

In return, I gave him a logic puzzle that required no domain-specific knowledge — no need to understand what a square root is, or even what a fraction means.

After 5–10 minutes, he gave up.

Notably, the confounding factor might have been grit — he may have had the intelligence to solve the problem, but he “terminated with an EOS token” before reaching a solution.

In general, I don’t think most people have truly internalized the shape of the IQ bell curve. We tend to surround ourselves with a very narrow slice of it. We forget that half of the population scores below the median on that bell curve.

LLMs display jagged intelligence — and so do humans

Jagged intelligence is a coinage introduced by Andrej Karpathy [26] in one of his banger tweets:

Jagged Intelligence — The word I came up with to describe the (strange, unintuitive) fact that state of the art LLMs can both perform extremely impressive tasks (e.g. solve complex math problems) while simultaneously struggle with some very dumb problems…

Chess provides a compelling case study of jagged intelligence in humans.

For generations, world champions have been held up as icons of extraordinary intellect, yet their brilliance is highly spiky/jagged. It’s possible for a grandmaster to sit in a hall, cover their eyes, and keep track of dozens of games at once. The current record [27] is 48 simultaneous blindfold games, set in 2016, with the grandmaster scoring 35 wins, 7 draws, and 6 losses.

But the more we studied how their minds work, the more it became clear: it’s largely pattern matching built on having observed and memorized thousands of games, both their own and those of other masters. What sets them apart is exceptional domain-specific memory.

But that memory doesn’t generalize well beyond the chessboard. In fact, studies show [28] that grandmasters’ recall advantage disappears entirely when the board is randomized and the familiar structure is removed.

E.g. in the World Championship of Fischer Random Chess, which randomizes the starting position of the pieces on the back rank, Wesley So decisively defeated former (and four-time) World Champion Magnus Carlsen (4 wins, 0 losses, 2 draws) to become the inaugural world Fischer Random Chess champion.

They’ve overfitted to a benchmark — sound familiar?

Note: None of this is to say that System 2 thinking — deliberate reasoning and planning — doesn’t matter. It still plays a crucial role, especially in longer-duration chess variants.

Similar patterns show up elsewhere — for example, elite Rubik’s Cube solvers who can solve the standard 3x3x3 cube in just 3.05 seconds [29].

But none of them go on to solve the Poincaré Conjecture à la Grigori Perelman. And conversely, I’m pretty sure Terence Tao isn’t one-shotting his way to grandmaster status either.

My goal here isn’t to argue that the present-day LLMs are less jagged than humans — only that this same quality exists in us too. But unlike humans, as AI continues to advance at a break-neck pace, its jaggedness will steadily smooth out.

LLMs can extrapolate

Critics often make the argument that LLMs can’t extrapolate, that they can only interpolate within their training data and therefore can’t possibly generate new scientific breakthroughs.

But that argument seems increasingly moot.

Interpolation at the knowledge level probably won’t lead to groundbreaking discoveries. But interpolation at the reasoning level just might.

LLMs could uncover new heuristics for understanding the world by combining existing ones (simple interpolation) — opening up entirely new branches on the tree of knowledge (i.e., extrapolation).

To summarize my claim: interpolation in the reasoning space can look like extrapolation in the knowledge space.

Humans can engineer an Artificial Superintelligence (ASI)

No matter how you look at it:

a) if human intelligence is pristine and mystical -> well, it seems we have found a way to engineer this mystical emergent entity

b) if it’s just pattern matching -> turns out we’ve already built some pretty damn good pattern matchers

The biggest bottlenecks I currently see on the path to ASI are:

1. lifelong memory (~READ)
2. continuous learning (~WRITE)
3. catastrophic-forgetting (~DELETE)
4. multi-agent training environments

The first three are prerequisites for a single ASI; the last is for societies of ASIs.

I believe all of these can be solved with more compute, and maybe a few modelling breakthroughs (or by dusting off one of Schmidhuber’s papers from the ’90s — iykyk).

Here is a highly simplified model I hold. It seems to me that humans primarily collect data and experiences during the day. Some additional data collection happens at night, through mental simulations. Dreams act as generators of synthetic data. During sleep, we initiate fine-tuning runs (brief micro-replays during quiet wakefulness add incremental updates, but the heavy lifting happens overnight). Some memories are consolidated and burned into the neural weights, while others are selectively erased. Catastrophic forgetting isn’t a bug — it’s a feature.

It’s currently just too expensive to fine-tune massive 10-trillion parameter models every single night. To a large extent it’s an infra problem. That said, we’ll likely need some algorithmic breakthroughs to fully solve it. We still don’t fully understand the shape of forgetting during these fine-tuning runs. But I’m optimistic — whether through mechanistic interpretability or sheer scale, we’ll eventually figure it out.

As maximum context length approaches infinity and new methods for compressing information emerge (e.g. DeepSeek’s new Multi-head Latent Attention (MLA) mechanism [30]), we may eventually solve lifelong memory without relying on external memory structures like vector databases or Neural Turing Machines [31].

Things are accelerating.

xAI is already building Colossus III: >1 million GPUs in a single cluster [32]. Other players (including Chinese companies) are taking notice and closely following suit. This is triggering an unprecedented wave of compute infrastructure buildouts.

Compute is ultimately bottlenecked by two factors: energy input on one end, and the risk of rendering the planet uninhabitable through waste heat on the other. Even if we approach the Landauer limit [33], there are only so many orders of magnitude we can scale before Earth heats up by several degrees — a threshold that’s likely catastrophic. Scaling will continue, but the physical limits are real. At least until we become a multiplanetary species and start building planet-scale computers [34].

With media attention and capital pouring in like never before, more bright minds than ever are working on bold new research ideas.

The first signs of superintelligence in open-ended domains have already begun to emerge:

• Models like ChatGPT, Gemini, and Claude are already remarkably powerful and knowledgeable across a wide range of domains.
• OpenAI’s o3-based DeepResearch [35] already outperforms most human analysts — and rivals the best.
• OpenAI’s Operator [36] isn’t quite there yet, but with a few more iterations, we’ll have a system that can use a computer as well as most humans.

This is part of a broader trend.

It’s obvious to me that the digital AI workforce is coming.

Over the next decade, everyone will have digital team members — from software engineers (with early examples like Cognition’s Devin [37]) to hardware and product designers shaping the physical world.

What do humans do in this new world?

In the short term, we augment ourselves. We augment our workforce. We become more efficient.

What about the long term — beyond the Singularity horizon?

Note: What follows ventures into speculative, science-fiction territory — but every step is grounded in established psychology, sociology, game theory, and current scientific understanding.

Neocambria. Humanity Post-Singularity — The Great Variance Explosion.

Definitionally, once we build an ASI, we will have created a system that is in principle Pareto-dominant across practically every task of economic value.

Thus, by definition, “Humans 1.0” can’t compete in this new economic reality. Whatever task we choose to compete on, there exists a cheaper, more efficient, and higher-performing silicon-based (embodied) intelligence ready to fulfill it.

The only solution is a human condition variance explosion (see also this related tweet [38]) — humanity forks into multiple branches. A new Cambrian event. Neocambria.

Game-theoretically, I believe Homo sapiens diverges into one of a few main branches:

1. Humans 1.0 — Reversionists. They reject both genetic and biomechatronic upgrades. They opt out of the brand new world, choosing instead a life in deep nature, supported by just enough self-sustaining background technology to ensure a pleasant existence. A near-biblically long, undisturbed life — until the garbage collection (death) is finally invoked. Their version of the promised Eden (or Jannah, Vaikuntha, Tian, Pure Land). All watched over by machines of loving grace [39]. Today’s Amish offer a gentle preview.

2. Humans 1.5 — The Immortals. They embrace germline edits but shun biomechatronic add-ons. Imagine you could detect a disease like diabetes at the embryonic stage of your child — and cure it. Would you? Of course you would (most of us would). Assuming humans aren’t cursed by a hard biological version of the No Free Lunch theorem [40] — and there is some evidence they are (see, for example, the Ashkenazi Jewish [41] population that have IQ/health tradeoff) — then why stop at disease prevention? Why not ask enhance vision, stamina, cognitive ability, physical beauty, even lifespan? (Dario Amodei calls it “biological freedom” [42]) And if the genome allows — wings? (Angels, perhaps?)

3. Humans 2.0 — Cyborgs. Building on the bio-engineered baseline of Humans 1.5, this branch replaces select biological organs and limbs with superior prosthetics, augments the body with advanced materials, and integrates deeply with synthetic sensors. Their augmented sensorium spans infrared, ultraviolet, infrasound, etc. — spectra once invisible to flesh alone.

4. Humans 3.0 — Homo Astra. Post-biological entities. These beings retain no biological substrate, yet they are the intellectual offspring of organic minds — their lineage traceable to humanity. These superintelligent entities exist as vast, planetary- or even galaxy-scale computational systems. They command self-assembling factories, producing distributed swarms of perfectly aligned machines that can act in the world.

Humans 1.0 were the bootloader for everything that followed. The four branches are just trunk lines; each will ramify into sub-species as technology, access, and culture diverge.

Consider Humans 1.5. If germline editing remains costly, IQ-boosting dynasties will arise beside unmodified households, spawning fresh caste dynamics inside the “Immortals.” Game-theoretic outcomes are murky: will high-IQ clans subsidize the rest or harden the divide?

Cyborg 2.0s and post-biological 3.0s wield overwhelming leverage. Do they conquer, curate, or cooperate? 1.0s — living museums — pose no threat; we 0.9s don’t extinguish harmless fauna, so a similar détente seems plausible.

But as 1.5+ lineages accelerate and 1.0s forget their own origins, the latter may come to revere the former as gods.

The only path to minimizing conflict is to escape zero sum dynamics.

Expansion is the safety valve.

Climb the Kardashev scale [43].

Build Dyson spheres [44].

Flood the void with compute.

Dilute conflict across the cosmos.

Our destiny lies among the stars. Lunar super-computers, asteroid foundries, interstellar relays.

And ultimately: to learn how it all began…and how it ends.

Open Questions:

• Governance OS? What operating systems handle a multi-species polity?
• Post-scarcity finance?
• Techno-capital markets?
• New laws of physics that are undiscovered as of now?

References

[1] Shane Legg & Marcus Hutter, “Universal Intelligence: A Definition of Machine Intelligence”, https://arxiv.org/abs/0712.3329 (2007).

[2] “Our structure”, https://openai.com/our-structure/

[3] David Silver et al, “Mastering the game of Go with deep neural networks and tree search”, https://www.nature.com/articles/nature16961 (2016).

[4] “AlphaGo versus Lee Sedol”, https://en.wikipedia.org/wiki/AlphaGo_versus_Lee_Sedol

[5] “AlphaGo Zero: Starting from scratch”, https://deepmind.google/discover/blog/alphago-zero-starting-from-scratch/

[6] Brown et al, “Language Models are Few-Shot Learners”, https://arxiv.org/abs/2005.14165 (2020).

[7] Vaswani et al, “Attention Is All You Need”, https://arxiv.org/abs/1706.03762 (2017).

[8] “The Bitter Lesson”, http://www.incompleteideas.net/IncIdeas/BitterLesson.html

[9] Kaplan et al, “Scaling Laws for Neural Language Models”, https://arxiv.org/abs/2001.08361 (2020).

[10] Hoffmann et al, “Training Compute-Optimal Large Language Models”, https://arxiv.org/abs/2203.15556 (2022).

[11] “Scaling Test Time Compute to Multi-Agent Civilizations: Noam Brown”, https://www.latent.space/p/noam-brown

[12] https://x.com/danintheory/status/1920898285142716782

[13] David Silver, Richard S. Sutton, “Welcome to the Era of Experience”, https://storage.googleapis.com/deepmind-media/Era-of-Experience%20/The%20Era%20of%20Experience%20Paper.pdf (2025).

[14] Hao et al, “Training Large Language Models to Reason in a Continuous Latent Space”, https://arxiv.org/abs/2412.06769 (2024).

[15] https://x.com/elonmusk/status/1830650370336473253

[16] “AI 2027”, https://ai-2027.com/

[17] A. M. Turing, “Computing Machinery and Intelligence”, https://courses.cs.umbc.edu/471/papers/turing.pdf (1950).

[18] “Horse rides astronaut”, https://garymarcus.substack.com/p/horse-rides-astronaut

[19] Twitter hivemind :)

[20] A. M. Turing, “Computing Machinery and Intelligence”, https://courses.cs.umbc.edu/471/papers/turing.pdf (1950).

[21] John Searle, “The Chinese Room Argument”, https://plato.stanford.edu/entries/chinese-room/ (1980).

[22] “Thinking Fast and Slow”, https://en.wikipedia.org/wiki/Thinking,_Fast_and_Slow

[23] “Down syndrome”, https://en.wikipedia.org/wiki/Down_syndrome

[24] “Growth hormone deficiency”, https://en.wikipedia.org/wiki/Growth_hormone_deficiency

[25] “Progeria”, https://en.wikipedia.org/wiki/Progeria

[26] https://x.com/karpathy/status/1816531576228053133

[27] “Most simultaneous blindfolded chess games (total games)”, https://www.guinnessworldrecords.com/world-records/72345-most-simultaneous-blindfolded-chess-wins

[28] Chase & Simon, “Perception in Chess”, https://www.sciencedirect.com/science/article/abs/pii/0010028573900042 (1973).

[29] “Xuanyi Geng”, https://www.worldcubeassociation.org/persons/2023GENG02

[30] DeepSeek-AI, “DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model”, https://arxiv.org/pdf/2405.04434 (2024).

[31] Graves et al, “Neural Turing Machines”, https://arxiv.org/abs/1410.5401 (2014).

[32] “Elon Musk plans to expand Colossus AI supercomputer tenfold”, https://www.ft.com/content/9c0516cf-dd12-4665-aa22-712de854fe2f

[33] “Landauer’s principle”, https://en.wikipedia.org/wiki/Landauer%27s_principle

[34] “The Moon Should Be a Computer”, https://www.palladiummag.com/2025/04/18/the-moon-should-be-a-computer/

[35] “Introducing Deep Research”, https://openai.com/index/introducing-deep-research/

[36] “Introducing Operator”, https://openai.com/index/introducing-operator/

[37] “Introducing Devin, the first AI software engineer”, https://cognition.ai/blog/introducing-devin

[38] https://x.com/karpathy/status/1846448411362709980

[39] “All Watched Over By Machines Of Loving Grace”, https://allpoetry.com/All-Watched-Over-By-Machines-Of-Loving-Grace

[40] “No free lunch theorem”, https://en.wikipedia.org/wiki/No_free_lunch_theorem

[41] “Ashkenazi Jews”, https://en.wikipedia.org/wiki/Ashkenazi_Jews

[42] “Machines of Loving Grace", https://www.darioamodei.com/essay/machines-of-loving-grace

[43] “Kardashev scale”, https://en.wikipedia.org/wiki/Kardashev_scale

[44] “Dyson sphere”, https://en.wikipedia.org/wiki/Dyson_sphere

Update: Cracked Engineers has been sunset as of December 2nd, unfortunately not enough interest. Feel free to ignore this post. :)

Super excited to share this: I’ve just launched a new job platform for technical roles called “Cracked Engineers”! LinkedIn, Twitter)

If you want to land a job with some of the world’s best AI/tech startups — my goal is to make this the go-to place for you!

Here it is: https://www.crackedengineers.com/

And here is a YouTube walk-through of the platform:

Some of the best AI startups (rising stars & established ones) already posted their jobs posts! A huge thank you to:

• Weaviate (Bob van Luijt & team, best OSS vector DB!)
• Unsloth AI (Daniel Han & Mike, one of the hottest YC startups)
• OpenPipe (David Corbitt, amazing fine-tuning engine)
• Julius AI (Rahul Sonwalkar, over 1M users!)
• Jimini AI (Luis F Voloch, ex co-founder of immunai (unicorn))

for using the platform while still in beta and posting first paid posts!

As of yesterday there are now 774 people who signed up for the newsletter!

And first feedback seems to be great!

Well, these are good problems to have! :)

And talking about the newsletter — I’m also announcing a weekly tech jobs newsletter that will be sent directly to your email inbox every Thursday! The best thing is you can subscribe on your dashboard (once you login via GitHub) only to job post tags you care about!

e.g. “CUDA”, “Triton”, “MLOps”, “Software Engineering”, “LLMs”, etc.

On the flip side — for companies that want to share a job post: i’ll be sending you weekly updates on the number of views/applicants, etc. to help you hire the most cracked engineering talent!

It’s also super simple to post a job: AI helps you fill up the whole form in a minute and you’re ready to post! AI also helps you review your job post and catch occasional typos, etc. You also have a job post preview before you hit that submit button, so you know exactly what you get before you pay.

if you need any help please dm me (see my social at the bottom of this post), or email me and I’ll make sure to help you asap. Also, if you have any feedback I’d love to hear from you!

This is a great place to provide deeper feedback: https://crackedengineers.canny.io/cracked-engineers

Or join my Discord channel: https://discord.gg/peBrCpheKE

I will be sharing job posts there directly from the platform, and we have a #crackedengineers channel for any feedback, help needed, etc.

Quick note on platform’s design/style/vibes: I wanted to make it the least corporate as much as possible. This should be a place where best tech startups hire the best tech talent. Period.

Why did I build it?

Well, over the past period a lot of fellow founders were asking me to help them with hiring ML/tech talent. Since I have access to pools of extremely talented engineers and researchers I managed to help some of them, but it really doesn’t scale, so I wanted to automate as much as possible.

Last but not least I also want to thank Jina AI (Han Xiao), SmartCat (Nenad Bozic), Luminai (Harsh Gupta), Nous Research (Karan), in no particular order, and many friends who helped test the platform while in alpha/beta stage — thank you! I learned a lot, and you made the platform so much better!

Onto $10k MRR! :)

1. Connect and reach me on 💡 LinkedIn and Twitter
2. Subscribe to my 🔔 YouTube channel for AI-related content️
3. Follow me on 📚 Medium and 💻GitHub
4. Subscribe to my 📢 monthly AI newsletter and join the 👨‍👩‍👧‍👦 Discord community!

And if you find the content I create useful consider becoming a Patreon!

The goal of this blog post is to explain flash attention in such a way that hopefully anyone who already understands attention will ask themselves:

“Why didn’t I think of this before?” followed by “It’s so easy”.

We’ll start from the first principles. We’ll first understand how the standard/vanilla attention is implemented and then we’ll address the inefficiencies one by one — as if we were to independently discover flash attention ourselves.

Also, my sub-goal is to demystify some of the lingo from the compilers folks’ community: kernel, kernel fusion, materialization, etc.

Note: I won’t be explaining attention itself, for that refer to Jay Alammar’s awesome blog or my implementation of the original transformer paper.

Without further ado let’s start by breaking down the paper title:

“FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness”

The takeaway is that FlashAttention is:

• Fast — excerpt from the paper: “We train BERT-large (seq. length 512) 15% faster than the training speed record in MLPerf 1.1, GPT2 (seq. length 1K) 3x faster than baseline implementations from HuggingFace and Megatron-LM, and long-range arena (seq. length 1K-4K) 2.4x faster than baselines.”
• Memory-efficient — compared to vanilla attention, which is quadratic in sequence length, O(N²), this method is sub-quadratic/linear in N (O(N)). We’ll see later why & how.
• Exact — meaning it’s not an approximation of the attention mechanism (like e.g. sparse, or low-rank matrix approximation methods) — its outputs are the same as in the “vanilla” attention mechanism.
• IO aware — compared to vanilla attention, flash attention is sentient.

Joking :) — it just means that it does not treat the underlying hardware as a black box. Instead, it leverages the knowledge of the memory hierarchy of the underlying hardware (e.g. GPUs, but other AI accelerators should work as well, I’ll be using GPUs as the running example).

Let’s expand on this IO awareness part a bit more. “IO” is the reason more FLOPS doesn’t necessarily translate into longer wall-clock time (maybe somewhat counterintuitively, but obvious if you know how the HW works).

Relevant excerpt from the paper:

“Although these [approximate] methods reduce the compute requirements to linear or near-linear in sequence length, many of them do not display wall-clock speedup against standard attention and have not gained wide adoption. One main reason is that they focus on FLOP reduction (which may not correlate with wall-clock speed) and tend to ignore overheads from memory access (IO).”

What’s the trick?

It’s the hardware:

Over the years GPUs have been adding compute capacity (FLOPS) at a faster pace than increasing the memory throughput (TB/s).

It doesn’t matter if you can compute at exaFLOPS speeds if there is no data to be processed. These 2 need to be closely aligned, and since the hardware lost that balance we have to make our software compensate for it.

Hence “IO-aware”.

Depending on this ratio between computation and memory accesses, operations can be classified as either:

• compute-bound (example: matrix multiplication)
• OR memory-bound (examples: elementwise ops (activation, dropout, masking), reduction ops (softmax, layer norm, sum, etc.)…)

Note on the terminology: this ratio is commonly measured by the arithmetic intensity, which is the number of arithmetic operations per byte of memory access.

Note 2: I strongly recommend reading Horace’s blog post https://horace.io/brrr_intro.html — it’ll help further clarify differences between being compute/memory/overhead-bound.

It turns out attention is (on current AI accelerators) memory-bound.

Why?

Because it “mostly consists of elementwise ops” or more accurately the arithmetic density of attention is not very high.

Let’s zoom in on this diagram from the paper:

You can see on the left bar, that masking, softmax & dropout are the ops that are taking the bulk of the time and not matrix multiplication (even though bulk of the FLOPS is in matmul).

But all is not lost. Memory is not a monolithic artifact, it’s hierarchical in its nature and the general rule is: the faster the memory, the more expensive it is, and the smaller its capacity.

Let’s zoom in on this part of the diagram:

Being “IO-aware” in practice boils down to exploiting the fact that SRAM is so much faster than HBM (“high bandwidth memory” — unfortunate name) by making sure to reduce the communication between the two.

To make things a bit less abstract here is a concrete example:

A100 GPU has 40–80GB of high bandwidth memory (HBM, the thing that gives you lovely CUDA OOMs) with a bandwidth of 1.5–2.0 TB/s and 192KB of on-chip SRAM per each of 108 streaming multiprocessors with bandwidth estimated around 19TB/s.

Similar ratios still hold for H100 and other accelerators.

Now, let’s see the computations behind the standard attention implementation:

You can see how the standard implementation shows the utmost disrespect for the way HW operates. It’s basically treating HBM load/store ops as 0 cost (it’s not “IO-aware”).

Let’s now think from the first principles about how we could make this implementation more efficient (time & memory-wise).

The lowest hanging fruit is to remove redundant HBM reads/writes.

Why write S back to HBM only to (re)load it again in order to compute the softmax? Let’s keep it in SRAM instead, perform all of the intermediate steps, and only then write the final result back to HBM.

This is what compilers folks refer to as “kernel fusion”, one of the most important low-level optimizations in deep learning:

No, not that one, but simply this:

A kernel is basically a fancy way of saying “a GPU operation”.

Fusion means you’re fusing/combining multiple ops together.

So, you are loading from the HBM only once, you execute the fused op, and only then write the results back. By doing this you reduce the communication overhead.

On a side note I honestly think people should stop naming concepts with words just because they sound cool. Kernel is the most overloaded word (2nd only to maybe “model”) in the computer science world. It can mean anything: Linux kernel (core software components of Linux OS), neural tangent kernel, SVM kernel, GPU operation, etc. It’s the HIV Aladeen of the CS world. 😂

One final piece of terminology you’ll find floating around is “materialization”. It refers to the fact that in the above standard attention implementation, we’ve allocated full NxN matrices (S, P). We’ll soon see that that’s the bottleneck flash attention directly tackles reducing the memory complexity from O(N²) to O(N).

Now that the complete background context is set, let’s now dig deeper into the flash attention algorithm.

Flash attention basically boils down to 2 main ideas:

1. Tiling (used during both forward & backward passes) — basically chunking the NxN softmax/scores matrix into blocks.

2. Recomputation (used in the backward pass only — if you’re familiar with activation/gradient checkpointing, this will be trivial to understand)

That’s it.

Here is the algorithm:

My job here is done. Hope you enjoyed this blog post, subscribe for more content in the future! 🚀

Just kidding.

Let’s understand a couple more ideas that are needed to get the tiling to work, and then I’ll explain the algo line by line.

FlashAttention algorithm

The main hurdle in getting the tiling approach to work is softmax. In particular, the fact that softmax couples all of the score columns together. Here is how we compute the i-th output of a softmax.

You see that denominator?

That’s the issue.

To compute how much a particular i-th token from the input sequence pays attention to other tokens in the sequence you’d need to have all of those scores readily available (denoted here by z_j) in SRAM.

But let me remind you: SRAM is severely limited in its capacity. You can’t just load the whole thing. N (sequence length) can be 1000 or even 100.000 tokens. So N² explodes fairly quickly.

So here’s the trick, we can actually chop the softmax computation down into smaller blocks and still end up with precisely the same result.

Here are the main formulas:

We can grab only the first B scores (x_1 through x_B) and compute the softmax for them.

These numbers are, at least for now, incorrect. But bear with me, through iterations, we’ll “converge” to a correct result.

Note: you can ignore the m(x) part, at least for now while we’re still in Plato’s world of ideas. Its purpose is solely to avoid numerical instabilities. On some hypothetical hardware from the future that’s more precise (e.g. we represent our data using more bits) this would not be needed. m(x) does not change the final result in any way.

Note: also check out the original papers that introduced the online softmax: https://arxiv.org/abs/1805.02867
https://arxiv.org/abs/2112.05682

Now the trick is that we can combine those per-block partial softmax numbers in a smart way such that the final result is actually correct. Here is the main idea:

So basically, in order to compute the softmax for the scores belonging to the first 2 blocks (of size B), you have to keep track of 2 statistics for each of the blocks: m(x) (maximum score) and l(x) (sum of exp scores).

And then you can seamlessly fuse them together using the normalizing coefficients.

Note: if you do some super basic algebra you’ll easily convince yourself that coefficients make sense. By expanding the f(x) and l(x) terms and multiplying with e^x some terms will simply cancel each other out, that’s the basic manipulation.

This logic continues recursively all the way up to the last, (N/B)-th, block, at which point you have the N-dimensional correct softmax output!

Ok, we now have all of the ingredients we need in order to understand the forward pass of the flash attention algorithm.

Note: the algo below assumes we have a batch of size 1 (i.e. single sequence) and a single attention head, we’ll easily scale it up later (by simply parallelizing across GPU’s streaming multiprocessors — more on that later). Also we ignore dropout & masking for the time being, trivial to add it later.

Let’s now break it down step by step!

Step 0: HBM’s capacity is measured in GBs (e.g. RTX 3090 has 24 GBs of VRAM/HBM, A100 has 40–80 GB, etc.) so allocating Q, K, and V is not an issue.

Step 1: Let’s compute the row/column block sizes. Why ceil(M/4d)? Because query, key, and value vectors are d-dimensional, and, we also need to combine them into the output d-dimensional vector. So this size basically allows us to max out SRAM capacity with q, k, v, and o vectors.

Toy example: assume M = 1000, d = 5. In this example, the block size is (1000/4*5) = 50. So in this example, we would load blocks of 50 q, k, v, o vectors at a time, to make sure we’re reducing the number of reads/writes between HBM/SRAM.

Worth keeping this image in your mind (it will make more sense soon):

As for B_r, I’m not exactly sure why do they perform a min op with d? If anyone knows feel free to leave a comment!

Step 2:

We initialize the output matrix O with all 0s. It’ll act as an accumulator hence that init value. Similarly for l (remember: its purpose is to hold the cumulative denominator for the softmax - the sum of exp scores). m (that holds row-wise maximum scores) is initialized with -inf because we’ll be doing a max operator over it so whatever the first block’s max is — it’ll certainly be larger than -inf — hence this is the natural init value.

Step 3:

We split the Q, K, and V into blocks using the block sizes from Step 1. See also the diagram above.

Step 4:

Similarly split O, l, m into blocks (same block size as Q).

Step 5:

Let’s start looping across the columns i.e. across key/value vectors (outer loop in the diagram above).

Step 6:

Let’s load the K_j and V_j blocks from HBM to SRAM. Remember because of the way we constructed the block sizes we still have 50% of the SRAM unoccupied at this point in time (dedicated to Q and O).

Step 7:

Start the inner loop across the rows i.e. across query vectors (again, see the diagram).

Step 8:

Load Q_i (B_r x d) and O_i (B_r x d) blocks, as well as l_i (B_r) & m_i (B_r) into SRAM.

How do l_i & m_i fit into the SRAM (including all of the intermediate variables) when we computed block size in such a way that we only have enough space for K_j, V_j, Q_i & O_i? I think the answer is: registers (see this CUDA video series to get some intuition on GPU memory hierarchy). But I might be wrong, someone who’s actually implemented this in CUDA please correct me. 🙏 I’m sure I’m missing out on important implementation details by just analyzing the pseudo-algorithm.

Step 9:

Compute the dot product between Q_i (B_r x d) and K_j transposed (d x B_c) to get the scores (B_r x B_c). As you can see we don’t have the whole NxN S (scores) matrix “materialized”. Only a fraction of it (S_i_j)!

Toy example: assuming the outer loop index is j (j=3), inner loop index is i (i=2), N is 25 and the block size is 5 this is what we just computed (assuming 1-based indexing):

Basically the attention scores for tokens 6–10 with tokens 11–15 of our input sequence. But, importantly, these are exact scores, they’ll never change (as opposed to softmax results that will gradually get refined).

Step 10:

Compute m~_i_j, l~_i_j, and P~_i_j using the scores computed in the previous step. It’s trivial.

m~_i_j is computed row-wise, find the max element for each of the above rows.

We get P~_i_j by applying elementwise ops:

1. Normalization — take the row max and subtract it from row scores
2. Exp

l~_i_j is simply a row-wise sum of the matrix P.

Step 11:

Compute m_new_i and l_new_i. Again fairly simple, let’s reuse the diagram from above:

m_i contains row-wise maximums for all of the blocks that came before (j=1 & j=2, colored in green). m~_i_j contains the row-wise maximums for the current block (colored in yellow). To get the m_new_i we just have to apply a max between m~_i_j & m_i. Similarly for l_new_i (it additionally requires multiplying by coefficients as we saw previously in formula 2).

Step 12 (the most important step):

This is the hardest part of the algorithm but still not that complicated, esp. once you internalize the formulas 1 & 2 for partial softmax computation.

Let’s break down the diag(l) part first.

It basically just allows us to do row-wise scalar multiplication in a matrix form. If you have a list of scalars s (N) and a matrix A (NxN), if you do diag(s)*A you’re basically doing elementwise multiplication of rows of A with those scalars.

Next up notice the similarity between step 12 and formula 1 (pasting it here again for convenience):

So what the 1st term of step 12 does (underlined in green) is it updates the current softmax estimate for the blocks before the current block in the same row of blocks. In case j=1 (that is the first block in this row) the 1st term will be 0 and we’ll just end up with the 2nd term.

The multiplication of the 1st term by diag(l_i) is there to cancel the division by that same constant from the previous iteration (this constant is hidden inside of O_i).

The 2nd term of the expression (underlined in yellow) doesn’t require this canceling of terms because as you can see we’re directly multiplying the P~_i_j matrix with the block of V vectors (V_j).

The e^x terms are there to modify the matrix P~_i_j & O_i by canceling out the m from the previous iteration and instead updating it with the latest estimate (m_new_i) that contains the row-wise max so far.

The easiest way to convince yourself this makes sense is to just simulate a couple of iterations yourself — in case you still didn’t quite get it.

It literally takes 5 minutes. Here is my step-by-step analysis (hope it helps!):

Recall: this is just a current estimate of the final O_i. Only after we iterated through all of the red blocks in the diagram above will we end up having the exact result. And that’s it!

Step 13:

Write the newest cumulative statistics (l_i & m_i) back to HBM. Notice these are of dimension B_r.

Steps 14, 15, 16:

Once the nested for loop is over, O (Nxd) will contain the final result: attention-weighted value vectors for each of the input tokens!

That’s it guys. That’s the forward pass of the flash attention!

This algorithm can easily be extended to “block-sparse FlashAttention”, a sparse attention algorithm that is 2–4 faster than even FlashAttention, scaling up to a sequence length of 64k! The idea is we use a block form mask matrix and we simply skip certain loads/stores from the above nested for loop and by doing so we can save proportionally to the sparsity coefficient.

Now let’s briefly touch on the complexity.

Complexity analysis

Space: We’ve allocated Q, K, V, O (Nxd), l & m (N) in HBM. That’s 4*N*d + 2*N. Dropping the constants (big O stuff), and knowing that d is also a constant and usually much smaller than N (e.g. usually d={32, 64, 128}, N={1024, …, 100k}) we get O(N) for space. Big win! That helps us scale transformers to 64k sequence lengths “easily” (add to that a couple of other “tricks” like ALiBi which I’ll cover in one of the follow-up blog posts).

Time: we won’t strictly do a time complexity analysis, instead we’ll use a good proxy: the number of HBM accesses.

Here is an excerpt from the paper:

How do they get to that number? Well, let’s analyze the nested for loop:

• Our block size is M/4d. This means that the vectors are split into N/(M/4d) blocks.
• Raise that to the power of 2 (because we’re looping across row/column blocks) and we get O(N² d² / M²)
• Now here I have M² and they only have M¹ — my hypothesis is that we probably need M/4d memory accesses inside of the loop to actually fetch all the vectors, i.e. we can’t fetch the whole block all at once. That would explain the missing M? (d can be ignored as a constant in big O).

If we were to do a big O analysis that could lead us to think this is not much better than standard attention, but for typical numbers, this leads to up to 9x fewer accesses (as per the excerpt above).

And that’s it, you now (hopefully) understand the flash attention!

Let’s wrap it up by closing the gap with the real world. So far we were analyzing the pseudo algorithm focusing on a single attention head assuming a batch size of 1. And we also glossed over the backward pass.

batch_size > 1, num_heads > 1, backward pass

Let’s start with the low-hanging fruit. Extending the implementation we saw to support batch_size > 1 and the num_heads > 1 is actually not that hard.

So far the algorithm we saw is basically handled by a single thread block (CUDA programming lingo). This thread block is executed on a single streaming multiprocessor (SM) (e.g. there are 108 of these on A100). To parallelize our computation we just run batch_size * num_heads threadblocks in parallel on different SMs. The closer that number is to the number of available SMs on the system the higher the utilization will be (ideally a multiple as each SM can run multiple thread blocks).

What happens when that number is bigger than the number of available SMs? I’m not sure but I assume there is a queue that keeps track of the waiting kernels (update: apparently the CUDA runtime takes care of that and it is using some sort of queues to implement that logic).

Next up let’s briefly address the backward pass.

The backward pass relies on the same set of concepts + recomputation.

To demonstrate the concept of recomputation I’ll use the example of “activation/gradient checkpointing” method.

We know that we need to have the activations computed during the forward pass readily available during the backward pass in order to compute the gradients w.r.t. our loss function.

The trick here is to not store them during the fwd pass (as they have a huge memory footprint), but instead, recompute them de novo during the backward pass. There is a built-in tradeoff here: we’re slowing down the backward pass in order to reduce the memory footprint.

Note: This tradeoff is a spectrum, e.g. you can store the activations every n layers, and then when computing the activations for the i-th layer you don’t have to start from the input but instead from the closest stored activations.

The same concept of recomputation is re-used here — but with a twist! Luckily for the flash attention, we don’t have to sacrifice neither runtime nor memory!

By storing the output O (Nxd) and the softmax normalization statistics (N) we can recompute the attention matrices S (NxN) and P (NxN) in the backward pass directly from blocks of Q, K, and V (Nxd) in SRAM! Thus keeping the memory at O(N). I encourage you to read the paper if you’re curious about the details, but I assure you, you’re equipped with all of the tools you need to understand it now.

Lastly, let’s see some of the issues one could expect implementing flash attention.

The real world is…messy

The same thing that gives flash attention its power is the root cause of its issues. Let’s see this excerpt from the paper:

“Our current approach to building IO-aware implementations of attention requires writing a new CUDA kernel for each new attention implementation. This requires writing the attention algorithm in a considerably lower-level language than PyTorch, and requires significant engineering effort. Implementations may also not be transferrable across GPU architectures. These limitations suggest the need for a method that supports writing attention algorithms in a high-level language (e.g., PyTorch), and compiling to IO-aware implementations in CUDA…”

As a consequence, the original flash attention supported only a subset of GPUs. For example, V100 is not supported. See this issue on their GitHub:

Note: here is one more relevant issue: https://github.com/HazyResearch/flash-attention/issues/190

To make this issue a bit more visceral, here is the actual CUDA code from the original implementation:

As you can see, writing CUDA is…messy. Additionally — this is a research codebase which doesn’t help, but even if it wasn’t for someone coming from an ML research background, who is only comfortable with Python this is (potentially) a deal breaker.

This is where projects like OpenAI’s Triton could be game changers (see their FlashAttention implementation). Triton is basically a DSL (domain-specific language) between CUDA & other DSLs (e.g. TVM) in its level of abstraction. You can write Python code that’s super optimized (once compiled) instead of having to deal directly with CUDA. That very same Python code could then be deployed on an arbitrary accelerator (that responsibility lies on Triton devs and HW manufacturers).

Triton has recently been integrated with PyTorch 2.0 so definitely keep an eye out for this project! Shout out to Philippe Tillet who started building Triton during his PhD and later became a part of OpenAI.

Finally, it’s worth mentioning that for certain use cases, you might still prefer other methods. E.g. for sequence lengths beyond 1K, some approximate attention methods (e.g., Linformer) start to become faster. But ultimately, to the best of my understanding, the block-sparse implementation of the flash attention outperforms all other methods.

Outro

You might ask yourself: why didn’t anyone invent FlashAttention before? Given how crucial this piece of computation is to all modern ML workloads. Given how many engineering hours are spent, across different tech organizations, trying to squeeze the last ounce of performance out of these systems. Why did a Stanford student (shout out to Tri Dao for the amazing work!) come up with this and not e.g. NVIDIA engineers?

There is a couple of possible explanations that I can see:

• FlashAttention is, as far as I can tell, easier/only possible to implement on the latest GPUs (hence V100 is not supported in the original codebase). Couple this with the phenomenon that oftentimes “outsiders” are those who see the problems with beginner’s eyes and tackle issues from the first principles and we get to the possible explanation.
• Ultimately I really like Nat Friedman’s perspective on the efficiency of the world:

Finally, let me wrap up with some food for thought:

Given how much it costs to train these models (see this blog from MosaicML for the most optimistic estimates) being able to shave off 15% from BERT-large training, or speed up GPT training 2/3x has such a tremendous economic impact when considering all of the current, and future, model trainings at a global scale.

Imagine if we lived in a world where Y (value captured) was actually correlated with X (value produced). The author would be a trillionaire over the next few years. :)

Alas, researchers rarely capture that value, for better or worse (imagine if Pythagoras patented his theorem :P). Yet people who oftentimes just repackage stuff are those who capture most of the value. Going on a complete tangent here, until next time! ;)

Further resources

• Flashier Attention blog — https://www.adept.ai/blog/flashier-attention -> They show how to further optimize flash attention for highly distributed settings where batch sizes become super small (pipeline parallelism) and sequence length super long.
• Tri Dao’s talk: FlashAttention — Tri Dao | Stanford MLSys #67
• Tri Dao’s talk: MedAI #54: FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness | Tri Dao
• Paper: FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

Acknowledgments

Thanks to Tri Dao, Horace He, and Amrit Sahu for reading earlier drafts of this blog post and providing feedback!

BibTeX Citation

@article{gordiceli5flashattention,
  author={Aleksa Gordic},
  title={ELI5: FlashAttention},
  year={2023},
  url={https://gordicaleksa.medium.com/eli5-flash-attention-5c44017022ad},
}

Connect with me

Last but not least feel free to drop me a message or:

1. Connect and reach out on LinkedIn and/or Twitter
2. Subscribe to my YouTube channel for more ML
3. Follow me on Medium and GitHub
4. Subscribe to my monthly AI newsletter and join the Discord community!
5. 📄 Website

And if you find the content I create useful consider becoming a Patreon!

Much love ❤️

This blog is the second one in my series of blogs covering the fascinating world of cryptocurrencies and beyond! You can check out the first blog here, which covers the “Digital Gold” book.

In this one, I’ll be sharing my thoughts, learnings, and notes from the book “The Infinite Machine: How an Army of Crypto-hackers Is Building the Next Internet with Ethereum” by Camila Russo.

Similar to the last blog, you should read this book if you’re interested in the history, people, stories, and a high-level overview of what Ethereum is and the inspiring story that led to its development.

Before I share my notes and some people you should probably check out on Twitter, let me give you a short summary of the book…🌈🦄

A TL;DR

Ethereum’s white paper was published back in 2013 by Vitalik Buterin, a 19-year-old “kid programmer”.

He got in the crypto-space back in 2011, when he was only 17 years old. In these early days, he was trying to learn as much as he possibly could about Bitcoin, blockchains in general, and how these ideas may be leveraged to improve the world.

His passion was subsequently channeled into what came to be known as the Bitcoin Magazine, which he jointly started together with Mihai Alisie, whom Vitalik had never met personally until that moment of time.

After spending some time in crypto he started his Bitcoin tour by traveling around the world attending various Bitcoin events (he’d basically stay at friends’ apartments to spare the money he collected). This gradually got him connected with the developers of Mastercoin and Colored Coins projects.

Their idea was to build on top of the Bitcoin blockchain and extend it in various different ways so that it can support creating new coins and manipulating real-world assets, and not just currencies.

After his deep engagement with Bitcoin, Mastercoin, and Colored Coins projects the idea was born in his mind on how he could further generalize all of their ideas — that’s how Ethereum came to be.

The name “Ethereum” itself was inspired by the disproved concept of “ether” that was hypothesized to be the substance that carries the light waves through space, hinting at the ambition that Ethereum could, one day, become a decentralized global computer that would support all of the future distributed applications.

He basically thought that instead of building one feature at a time on top of Bitcoin, which obviously wouldn’t scale up as it’s impossible to anticipate all the things that’ll eventually be built on top of it, it’d be better to create a programming language (Solidity) that can simply be used to program arbitrary functionality on top of a blockchain (inside the so-called smart contracts).

The initial days of Ethereum development were hectic. There were more than 10 “co-founders” and most of them were building Ethereum from a spaceship-looking house in Zug, Switzerland.

After certain internal fights, Vitalik decided that there’ll be “only” 8 co-founders.

Since then people have been proposing various ideas that could be built on top of Ethereum, stuff like:

• DAOs (decentralized autonomous organizations) — the idea is to create a trustless organization whose rules will be executed by the code and not by fallible and corruptable humans.
• DeFi (decentralized finance) — again the idea is to do fundamental financial operations without the unnecessary intermediaries.

And many others.

The most ambitious project, back in 2016, and one of the first DAOs was the so-called “The DAO”.

It was a decentralized funding organization where investors could invest their money (via Ether), get some “dai” tokens in return, and then use those to vote for certain whitelisted organizations. The actual Ether distribution would be done automatically without any intermediaries.

The idea was beautiful and pure, but the actual implementation taught us a lot about the potential problems with blockchains and open-source code in general.

The code was open-source and developed by humans, and as such, it had a fatal flaw — a bug that made it possible for malicious hackers to start draining the funds out directly into their own wallets. Additionally, thanks to the anonymity property of blockchains, the attackers ran away without any consequences.

All of this opened up an important question — what happens if a vital smart contract has an undiscovered bug?

The problem was solved by a largely controversial hard fork, that corrected the code in such a way that the original owners kept their coins and the attacker was left empty-handed. Most miners decided to pull the newest code changes.

The event was controversial because it showed that even in the world of cryptocurrencies some organizations are “too big to fail” and can be bailed out by the centralized authority. Similar to how governments bail out banks during severe recessions, the very system that cryptocurrencies were trying to replace.

Interestingly enough, some miners didn’t agree with this decision so they continued mining using the pre-hardfork code. In this parallel universe, the attacker managed to escape with the stolen money. The new-old blockchain came to be known as the Ethereum Classic.

Fast forward to today, the main improvements that Ethereum still has on its backlog can roughly be split into 2 classes:

• Proof of Stake
• Scaling

The Proof of Stake is an alternative consensus algorithm in which instead of using the amount of compute (Proof of Work) as a proxy for who should have more “voting power” in the blockchain, it uses the number of coins that a user has as a proxy. So it’s a plutocratic system in a way.

The why behind PoS is fairly simple — PoW is warming up the planet. 😂

On the other hand, scaling is much more important if Ethereum is ever to become a global computer. Compared to payment systems such as Visa, and MasterCard which can process thousands of transactions per second, Ethereum is in the double-digit realm.

Scaling techniques can approximately be divided into 2 categories:

• Level 1 techniques will be directly implemented on the Ethereum blockchain itself. The main one is sharding i.e. allowing the nodes in the blockchain to only keep a fraction of the whole blockchain/ledger, without (seriously) jeopardizing the security of the system.
• Level 2 techniques are those that’ll be implemented in systems built on top of the Ethereum blockchain. Those include rollouts (ZK-rollouts, optimal rollouts, etc.), plasma, and others.

By combining the level 1 and level 2 techniques together, Ethereum can theoretically achieve a 100x speedup.

Note: I didn’t digress into what each of the concepts mentioned mean, as otherwise this summary would be worthy of a book.

Similar to the last blog in the series, next up, I’ll share my private notes, in the hope their structure or the very fact I’m doing it may help somebody. You can also just extract some keywords from my notes and do your own (re)search.

Finally, I’ll share a list of Twitter influencers I extracted from the book.

Super important tip reading the book: take notes while reading it!!! Because:

a) You’ll be a more active learner which will further help consolidate the knowledge from the book.

b) There is a lot of new names/actors that you’ll need to remember to follow along the story. Thank me later.

The Infinite Machine book notes

Chapter 1 (Mooning)

The hype of 2017, lush parties everywhere, scammers, hackers.

Vitalik as the voice of reason — is the value we produced truly worth a trillion $ market?

Chapter 2 (Cypherpunk’s Fever Dream)

The ideas of cypherpunks (Jude Milhon), open-source software (Richard Stallman), P2P, shaped the crypto community.

The problems of “double-spend” and “Sybil attack” (duplicating money and identities basically).

Proof-of-Work as the solution to the above attacks.

The story of the monetary crisis in Argentina.

A high-level description of how Bitcoin works and the story behind it.

“The Times 3 January 2009 Chancellor on brink of second bailout for banks.” <- a message hidden inside the 1st block on the Bitcoin blockchain! Basically, the reason why we need decentralized, trustless systems such as Bitcoin.

Vitalik Buterin is introduced, when he was 17 he wrote his 1st post on the Bitcoin forum.

Chapter 3 (The Magazine)

Mihai Alisie and Vitalik create the first (printed) Bitcoin magazine.

Chapter 4 (The Rabbit Hole)

The story of Anthony (who had rich parents and would go on to become Ethereum co-founder) and the meetup he organized that Vitalik accidentally attended.

Mihai invites Vitalik to Europe (who was in Waterloo, Canada) to work on Bitcoin Magazine and his crypto startup Egora. Vitalik says yes.

They end up in Calafou community/hacklab in Spain, founded by Enric Duran the inventor of the pyramid scheme.

Chapter 5 (The Swiss Knife)

Vitalik’s Bitcoin tour (He “couch-surfed” in order to save up some money!).

During the tour, he got invited to Israel by a Colored Coins’ dev Amir Chetrit.

Mastercoin — the 1st blockchain startup that had raised funds by selling its digital token (coins not created by work/mining but out of thin air). It allowed others to easily create novel currencies, plus it used smart contracts.

The term “smart contract” is mentioned (Nick Szabo coined it back in the 1990s). A nice metaphor is that a smart contract is like a “digital vending machine”.

Vitalik writes the white paper for Colored coins and contributes to Mastercoin as well.

He notices problems with both of these, generalizes the approach and this is how Ethereum is born.

Chapter 6 (The white paper)

Since Vitalik was a kid he was exposed to maths by his parents (who were in software engineering (dad) and finance (mum)).

He wrote his first “white paper” while he was still 7, he had a private maths tutor, and he was less sociable than other kids.

During his middle school, he joins a specialized private school. He also joins the debate team that must have played a role in his exceptional ability to explain complicated concepts in a more approachable way).

Travels all around the world to compete in maths and debating.

In 2011 Dmitry (his father) mentioned Bitcoin to Vitalik.

Starts writing in exchange for bitcoins (BitcoinTalk forum).

In Ripple’s office in SF (end of his Bitcoin tour) he coins the name Ethereum based on the word “ether”.

November 27, 2013 — the Ethereum white paper is written.

Chapter 7 (The First Responders)

You have to pay ether to run a piece of code on the Ethereum blockchain.

2 types of accounts exist on Ethereum:

• external accounts (controlled by someone’s private key)
• contract accounts (controlled by code)

A brief story of Charles Hoskinson (founder of Cardano), and how he created Bitcoin courses that made him famous in the crypto community which eventually ended up with him meeting various Bitcoin celebs and getting funding!

Charles builds Invictus with Dan and Stan who he “met” on the forums.

Decentralized Autonomous Organization (DAO) is introduced as a concept by Dan and Stan.

Charles gets fired and at that same time joins Ethereum as the co-founder.

All the cofounders gather in Miami.

Chapter 8 (The Miami House)

The story of Gavin Wood and how he wrote the 1st C++ implementation of Ethereum (invited to Miami with other Ethereum “co-founders”).

Discussions on the ownership of Ethereum.

Story of Jeff Wilcke (who worked for Mastercoin) and his Go implementation of Ethereum in Amsterdam.

Chapter 9 (The announcement)

Story of Joseph Lubin (another co-founder) and the 1st demo of Ethereum by Gavin.

Initial fights as Gavin (rightfully) wants to become the co-founder as well.

Vitalik’s talk at the Bitcoin conference was followed by a lot of hype.

Chapter 10 (The Town of Zug)

Early days, more than 10 co-founders worked from a tiny flat, doing the legal stuff, building the website, etc.

Charles was initially the CEO.

Chapter 11 (The Spaceship)

They are working in a weird building in Zug that looks like a spaceship.

Small fights start happening, Charles, the CEO, starts pushing for the for-profit structure and hints that he is Satoshi Nakamoto (others get annoyed).

Vitalik is in Silicon Valley at that same time.

Chapter 12 (The White-Shoe Lawyers)

Legal problems in defining Ether: is it a security or a commodity?

Gavin publishes the “yellow paper” that contains technical details described in Vitalik’s white paper.

Pryor Cashman, a white-shoe law firm, makes a letter stating that Ether is not a security — a green light!

Chapter 13 (The Red Wedding)

Big fight bursts out as the crowd sale was approaching — the co-founders sit at the table in Zug and share their grievances.

Vitalik gets Peter Thiel’s grant.

The decision falls to Vitalik to decide on the co-founders and the future of Ethereum.

He decides on 8 co-founders leaving out Amir and Charles (the CEO).

Chapter 14 (The (Non)Investment)

The presale heat and hype are building up — many are accusing them of scamming.

Lots of effort is put into convincing people that ether is not a security.

Chapter 15 (The Ether Sale)

The buyers only got a wallet and a password to avoid speculative trading before the platform was up and running.

Criticisms because of the pre-mining (creating coins out of thin air). The control is, contrary to cypherpunks’ spirit, centralized among “insiders” who participated in a presale.

July 22, 2014 — They raised 18.3M$!!! (60M ether priced at 30 cents).

Co-founders and the Ethereum Foundation additionally got (out-of-thin-air) 9.9% of the total amount of coins sold (60M) during the ICO.

Ethereum’s supply strategy: Uncapped supply of Ether with the declining rate (contrast that with Bitcoin’s perfect scarcity).

Lots of controversy about the presale — it seems that the wealth was very unevenly distributed and the charts were “unusually smooth” as if somebody was tinkering with the buying process (a bot).

Chapter 16 (Takeoff)

The story of Aurel, Paul, and the mining operation before Ethereum went live.

Ethereum had 9 proof of concepts and rigorous testing since the presale event!

A bigger team is forming around Ethereum.

12 seconds for a transaction to get accepted vs Bitcoin’s 10 minutes.

Stages of Ethereum development: Olympic, Frontier, Homestead, Metropolis, and Serenity (PoS, sharding).

Vitalik gave a 25k ether reward for the community to try to crash the Olympic release (9th proof of concept).

Joe Lubin (Ethereum co-founder) started building a startup on top of Ethereum — ConsensSys.

July 30, 2015 day of the launch!

March 2016 Ether saw a 30/40x increase in value which made people like Ken Seiff very rich.

Chapter 17 (The Shrinking Runway)

The only co-founders left were Vitalik, Gavin, and Jeff (the initial C++/Go Ethereum devs) — others started their own companies.

After Bitcoin plummeted, and since the foundation kept all of its money in Bitcoin, their finances became very weak.

Ming was introduced to try to keep the Ethereum foundation afloat.

Tribalism between Jeff’s (Go) and Gavin’s (C++) teams.

Gavin is removed from the foundation because he started a for-profit company Ethcore.

Chapter 18 (The First Dapps)

Jack and Joey build Augur — a Dapp built on top of the Ethereum blockchain for betting (“prediction market” platform). Augur raises 5.3 M$.

ERC20 was born — a template to create coins on top of Ethereum (20 because it was the 20th issue on Ethereum’s GitHub!)

Rune creates MakerDAO on top of Ethereum — the value prop is a stable coin pegged to USD ($).

MakerDAO brings “the best of both worlds “— decentralized but stable!

March 14, 2016 — a production-ready version released, dubbed “Homestead”.

DigixDAO — similar to MakerDAO they peg the token to 1 g of gold.

Chapter 19 (The Magic Lock)

Slock.it was supposed to be a decentralized alternative to Uber, Airbnb, etc.

The DAO, a decentralized venture fund, from Slock.it raises 150M$ in ether!!!

Chapter 20 (The DAO Wars)

A bug in The DAO logic was exploited (the fund that had over 200M$ in ether at that time!).

Soft fork proposed by Vitalik and the team as the last solution, and even worse, a hard fork that would undermine Ethereum’s legitimacy (centralized authority is bailing out a company — sounds familiar).

The white hat hackers team (“The Robin Hood group”) starts draining the main DAO using the same attack. The attacker also infiltrated their child DAOs — the “DAO wars” start.

Chapter 21 (The Fork)

The soft fork is out of the question because of the potential DDoS attack.

After doing a brief voting session (holders of only 5.5% of the total ether supply participated) — they decided to go for a hard fork.

The hard fork saved Ethereum and The DAO but some miners decided to keep using the pre-fork software version, and thus even the old blockchain was kept alive— it became the so-called Ethereum Classic.

The Robin Hood group wanted to return the money back to the rightful owners in ETH (Etherum), and not ETC (Ethereum Classic), but the exchanges didn’t allow the transaction to go through.

In the end, the attacker cashed out and gave a 1k ETC donation to the ETC Foundation.

Chapter 22 (The Shanghai Attacks)

The Geth client (Go implementation) had a bug that was exploited by the attackers just before the Shanghai Devcon2 Ethereum conference.

It was quickly patched and the community was very happy with the responsiveness of the dev team behind Ethereum.

More attacks on Ethereum followed — hard forks had to be done (this time these were not controversial as they’re not bailing out other companies).

The market and miners will decide which one of the 2 versions (after the hard fork, and if there is a disagreement) is more valuable in their opinion — that was the adopted model.

Chapter 23 (The Burning Wick)

The ICOs era — most new tokens were built on top of Ethereum because of ERC20 (which is a template for building new coins).

Crazy amount of hype — a gaming company raises money with a 1M$/minute rate, and subsequently closes down their sales after only 5 minutes (5.5M$)!

Taylor and Kosala are building an intuitive MyEtherWallet. During the Golem ICO they get too many buy requests and many angry responses from investors whose transactions didn’t go through — a glimpse of what is to come in 2017.

Chapter 24 (Accidentally Ether Rich)

It’s 2017 — event by event the prices of Bitcoin, Ether, and other altcoins are spiraling up.

Big companies like Microsoft, IBM, JPMorgan, etc. come on board the Enterprise Ethereum Alliance.

More money is being raised by selling tokens than by selling stocks.

Chapter 25 (The New IPO)

Brave (the browser founded by a guy who created Mozilla and JavaScript) sells 35M$ worth of BAT tokens in 30 seconds!!!

In order to get the ICO process under some control, Bancor adds gas fee limits (so that rich investors can’t skip the line), a 1-hour investing window, and an additional mechanism that’ll return all of the outstanding Ethers that went above their target funding amount.

The demand during Bancor ICO was crazy, 550M$+ worth of Ether was sent — by they took only a small amount of that as previously agreed.

A similar story with Status (https://status.im/)

“Ethereum wasn’t able to handle traffic from its first killer app: fundraising.” <- indication of scaling problems.

Chapter 26 (The Friendly Ghost)

Vitalik had a meeting with Vladimir Putin! 😂

He was attending many conferences, meetups, hackathons, meetings with CEOs and government officials — that helped him further improve his presentation and debating skills.

Vitalik hates wastefulness on every level and is pushing for proof of stake (PoS) since Ethereum’s inception.

He and Vlad Zamfir create the first PoS implementations called Casper CBC and FFG respectively.

Vitalik is also thinking about scaling (methods include: sharding, plasma, zero-knowledge proofs, state channels).

Sharding is the only method that’ll be implemented in layer 1, i.e. directly to Ethereum base blockchain, other methods will be in layer 2.

PoS and Scaling should be part of the Serenity release of Ethereum.

Chapter 27 (The Boom)

People took photos with Vitalik and subsequently used those to pump up the prices by faking that “they have Vitalik’s support”.

Ethereum’s market cap caught up with Bitcoin (more than 25% of the market cap came from Ethereum).

Lots of scam/troll coins (Jesus Coin, TrumpCoin, etc.).

More hype coming from influencers creates a positive feedback loop.

Bitcoin millionaires flexing on the internet with Lambos.

An explosion of memes and hashtags like “hodl”, #Tothemoon, #whenmoon, #whenlambo.

Ethereum captures 32% of the market cap in June 2017 (Bitcoin at 38%).

Multi-million $ Ether sell orders start triggering stop-loss orders — the price plunges and bounces back. Somebody said that Vitalik is dead — the price plunges, Vitalik tweets with a picture of the latest Ethereum block id.

SEC makes a statement that tokens are indeed securities — lawsuits and fees are possible.

Protocol Lab organizes an unfair/undemocratic ICO where only rich investors could participate (even in the public sale, in the private sale they had Andressen Horowitz and others), as is common in IPOs. The largest ICO so far at 257M$.

Ethereum’s Metropolis release is scheduled for October 2017. Miners’ block rewards are reduced so as to incentivize the transition to PoS.

A zero-knowledge proof feature is added.

Cryptokitties, collectible cats, are born (cats because they knew that the internet loves cats, so that’ll boost the virality coefficient) during the Waterloo hackathon (the idea itself was born before the actual hackathon during a brainstorm session).

ERC721 standard was created and it made it easy for others to build similar projects. Individual non-fungible “digital things” can now be traded, not just fungible currencies. NFTs (non-fungible tokens) are born.

Chapter 28 (Futures and Cats)

Bitcoin futures announced by the CME group — a big deal since that promises institutional investors money. ETFs were also rumored to come. Price skyrockets.

Dedicated funds for cryptocurrencies started popping up (hedge funds, etc.).

Rich crypto guys buying fancy houses.

Gavin Wood starts Polkadot (the idea is to connect all blockchains to create a decentralized web).

A bug leaves Polkadot without the money collected during their ICO.

Cryptokitties are launched in November 2017, the gaming aspect made them additionally viral.

At one point Cryptokitties accounted for 15% of total Ethereum transactions (some were sold for as much as 100k$!).

Cboe futures launch (CME a bit later as well).

PoS testnet (Casper FFG) launches on pyethereum.

If you bought Ether for 250$ back in the ICO days (when it was 30 cents) you’d be a millionaire in early 2018 (during the peak).

Chapter 29 (The Crash)

SEC starts charging ICO startups like Munchee (decentralized Yelp), which were legit, and fraudulent ones like AriseBank and others.

In January 2018 Bitcoin and Ether prices start plummeting — all the crypto enemies came down on crypto. The positive feedback loop.

The 2.6B$ Bitconnect scam.

Legit Ethereum projects like MakerDAO do exist.

Tether not backing up every USDT token with $ — they’re accused of pumping prices and producing coins out of thin air.

Exchanges accused of wash trading and thus of inflating the trade volume (some by as much as 95%!).

An exception to the downward slope of ICO funding was in June 2018, 4.2B$ was raised by Block.one’s EOS blockchain!!! (the idea is to replace Ethereum by having a much higher throughput)

It was the 2nd largest “IPO” in the USA just after Spotify at 9.2B$.

Accused of centralization (10 addresses account for 50% of the tokens) and market manipulation (they could trade tokens during the sale). Also, 20ish miners control the network.

The advisor to the project, Brock Pierce, was accused of sexual assaults back in the early 2000s.

Many other platforms were trying to replace Ethereum like Cardano, etc.

The bear market starts.

Chapter 30 (The Party)

DeFi starts getting its foothodl. An example of Ripio wallet with a “lend” button was described where you’d get the money from some random investor (who was probably somewhere in South Korea at that point in time) via a smart contract — no intermediaries!

Vitalik is sending out some of his Ethers to help out startups — VCs argued how this structure is not a sustainable way to fund startups.

“Collateral became a popular way for platforms to issue loans without having to screen each users’ credit score.”

MakerDAO is a good example of a successful DeFi platform.

The hype was reducing and the value that Ethereum was bringing was increasing — Vitalik happy, happy.

Early Ethereum dapps: Slock.it, The DAO, DigixDAO, MakerDAO, Augur.

That’s it! Hopefully, you found these somewhat useful. I know I did.

Now drum rolls.

People you should (probably) check out on Twitter

Here is a list of people I started following after reading the book, in no particular order (note I’ve omitted people from the last blog including Vitalik):

• gavofyork (Gavin Wood) — co-founder of Ethereum, wrote the 1st C++ implementation of Ethereum.
• IOHK_Charles (Charles Hoskinson) — Cardano (and Ethereum) co-founder.
• ethereumjoseph (Joseph Lubin) — co-founder of Ethereum and ConsenSys.
• CryptoCobain — co-founder of Ethereum (didn’t find him in the book, but he claims to be the co-founder, maybe just a meme ❤️).
• justinsuntron (Justin Sun) — founder of TRON.
• ummjackson, BillyM2k — Dogecoin founders.
• ChrJentzsch — founder of Slock.it
• roham — lead the group who started the NFT revolution.
• beeple (Michael Winkelmann) — his NFT “everydays” was sold for 69M$. He created the art that went into “everydays” over the span of 14 years. (5000 pictures in a mosaic is what was sold, i.e. the “ownership”)
• peter_szilagyi — the main Ethereum Go dev.
• tayvano — co-founder of MyEtherWallet.
• el33th4xor (Emin Sirer )— a CS prof who noticed The DAO bug.
• nayibukkele — president of El Salvador or CEO as he calls himself haha.
• aantonop (Andreas) — co-author of the “Mastering Ethereum” book.
• novogratz — an ex-Fortress investor that went into crypto.
• BitcoinMagazine (Vitalik stopped writing for it a long time ago)

Try and explore their feeds, and based on your preference start following those people whose ideas resonate with you.

But, also try and follow some people with whose ideas you don’t necessarily agree — you need a healthy dose of different opinions in your life. Who knows maybe you’re wrong and they are right. Be open-minded.

Finally, worth mentioning, while I was reading through the book, I came across some concepts that I thought were important enough to justify the digressions I made. ❤️

E.g., I wasn’t familiar with the term “Austrian economics”, so I did some YouTube search and came across these super valuable videos:

A second example would be Ayn Rand’s name. Her name triggered me to do some further research because I’ve stumbled upon her name way too many times. A spike occurred in my decision-making neuron — the rest is history. Here is a cool interview with her:

I’ve also done a mini-research on the phenomenon of Bitconnect, I found this great interview with Vitalik, plus I’ve previously listened to him on Lex Fridman’s and TimFeriss’s podcasts.

That’s it, I hope you learned something new!

If there is something you would like me to write about — write it down in the comment section or DM me. I’d be glad to write more about mathematics & statistics, ML/deep learning, CS/software engineering, landing a job at a big tech company, getting an invitation to prestigious ML camps, finance, etc., anything that could help you!

Also feel free to drop me a message or:

1. Connect and reach me on 💡 LinkedIn and Twitter
2. Subscribe to my 🔔 YouTube channel for AI-related content️
3. Follow me on 📚 Medium and 💻GitHub
4. Subscribe to my 📢 monthly AI newsletter and join the 👨‍👩‍👧‍👦 Discord community!

And if you find the content I create useful consider becoming a Patreon!

Much love ❤️

This blog is a first in my series of blogs on crypto (both currencies and dapps (decentralized apps) more broadly).

In this one, I’ll be sharing my thoughts, learnings, and notes from the book “Digital Gold: Bitcoin and the Inside Story of the Misfits and Millionaires Trying to Reinvent Money” by Nathaniel Popper.

Let me open it up with a question.

(when) should you read this book?

You should read this book if you’re interested in the history, people, stories, and a high-level overview of what Bitcoin is and how it came to be.

Even though you will, arguably, learn a bit about the blockchain tech itself (the underlying tech behind all of the existing cryptocurrencies including but not limited to Bitcoin), the book itself is not designed to teach you anything about the underlying tech per se.

I found some prerequisite knowledge to be of help, but having said that, this can definitely be your first encounter with the wor(l)d (of) bitcoin.

But, before I start, let me tell you a short story…🌈🌈🌈

My journey into crypto

Aside from dwelling in the depths and breadths of artificial intelligence, finance, and more broadly economics, is something that deeply excites me for its potential to reform society for the better.

Economics deals with the question of how to satisfy the unlimited demands of humans with the limited resources that we have at our disposal.

Because even if AI was to bring a huge economic value to society, if the value is not distributed effectively and efficiently it’ll cause even greater inequalities.

In my mind’s eye, I imagine the society (both global and local) as a seesaw, a delicate, and fairly fragile system. The purpose of economics, finance, and politics is to keep the seesaw in balance even when the weight (the value produced) is simultaneously changing on both sides (rich and poor).

Let it lean too much to one side and we have recessions, coups, and/or wars.

One of my main life goals is to maximize my positive societal impact. That’s a common thread that goes throughout almost everything that I do, and that naturally brought me to the topic of crypto.

From all the different trends that we’ve been seeing over the last decade or so, the crypto space (i.e. apps built on top of the blockchain, like currencies but more broadly any type of a dapp (distributed application)) introduced the most interesting ideas (decentralized, trustless systems) that could play an important role in a (more) fair distribution of wealth.

The federal reserve doesn’t have an equivalent in the world of Bitcoin. There is no centralized authority that can secretly change the distribution of wealth, based on their own plutocratic decisions, by simply printing more money.

People often complain how cryptocurrencies are a “Ponzi scheme” since they’re “not backed up by anything of value”, but, IMHO, they fail to realize that:

1) Money is inherently a social construct. A $ bill has value only because you and I are willing to accept it as such. So $ is also not backed up by anything of “value” anymore, aside from the fact that the USA government will gladly accept your $ bills when it’s tax collection time.

2) Being backed up by gold is no more real than being backed up by belief.

Why is gold valuable again?

Because it’s shiny and visually appealing? Because we put it in jewelry? Or is it’s the other way around — we use it in jewelry because it’s valuable.

But where does this value come from? From its scarcity? Well, gold is not special in that regard, we have plenty of scarce material on this Earth.

If you were freezing to death and you had to pick between a gold bar and a warm coat I’m fairly sure I could guess what’s more valuable for you in that context.

So in which context is gold more valuable? Ok, we do use it to produce electronics, but again, gold is not special in that regard, we use other scarce materials as well.

No, the reason I think that gold is valuable is that it has a couple of nice properties. It’s a mix of its mythological appeal (gold ~ Sun), relative scarcity (we still keep excavating gold), the fact that somebody had to work to excavate it, etc.

All of these properties now have a perfect digital equivalent because we’re now operating in Plato’s world of ideas.

Bitcoin is perfectly scarce (21M coins, mathematically guaranteed), somebody had to “work” to get it (mining), it’s faster to transport than gold (the speed of light in coaxial/optical fibers), it can be divided into arbitrarily small pieces (e.g. 1 satoshi = 0.00000001 btc), to name a few.

My personal journey with crypto started back in 2017, during the bull market, which was the first time ever I heard of the word Bitcoin.

Again, it’s funny how unequal our opportunities are. We need to constantly take that into account evaluating someone’s end results, without being judgemental.

Here I was hearing about this awesome technology for the first time, almost 9 years after Satoshi’s original white paper came out (back then I wasn’t surrounded by a lot of tech-savvy people like I am right now, even though I studied EE).

On the other hand, Vitalik Buterin (the (co)founder of Ethereum) had the luck (the purpose of this reasoning is not to undermine his achievements) that his dad was a skilled software engineer who told him about Bitcoin way back in the early days.

His childhood was also fairly special. He had private math tutors, and a home education by 2 ambitious, driven, and smart parents. They were throwing arithmetic problems at him since he was 3 or 4 years old, just to give a couple of examples.

This, aside from the Polgar sisters stories and many others, further strengthens my belief that extraordinary nurture can create people that we call “geniuses”.

Back in 2017, I didn’t have any serious money (I was still studying) that I could use to invest in Bitcoin, but I did do my research both from the angles of technology (how blockchains work — Merkle trees, hash codes, etc.), people (heard of Vitalik, Satoshi, btc millionaires), and trends (ICOs, centralized exchanges, lots of interesting irrational behavior in general).

Since then I’ve been reading finance/business books on and off but this time I’m sharing my learnings and thoughts. That’s what changed.

Over the past 2 weeks I’ve read 2 very interesting books:

• “Digital Gold” by Nathaniel Popper (or is it Digital Godl?)
• “The Infinite Machine” by Camila Russo

In this blog, I’ll give a very short overview of the book, or more precisely I’ll share my private notes, in the hope their structure or the very fact I’m doing it may help somebody.

I’ll also share a list I compiled of influential Twitter people to follow (or just check out) to stay up to date with the newest stuff happening in the crypto space. The list is very raw (I collected names from the book and did some link following) so I’ll gradually prune it as I get a better grasp for what these people are sharing, based on my own interests.

Super important tip reading the book: take notes while reading it!!! Because:

a) You’ll be a more active learner which will further help consolidate the knowledge from the book.

b) There is a lot of new names/actors that you’ll need to remember to follow along the story. Thank me later.

Digital Gold book notes

Intro

A secret meeting at lake Tahoe with Erik Voorhees (btc evangelist), hedge fund managers like Dan Morehead, and other early adopters — the “btc millionaires”.

The origin story of this mystical money called Bitcoin (Satoshi Nakamoto introduced).

Bitcoin was introduced in the middle of the financial crisis. It has nice properties (universal — can be used across borders, low fees (back then), decentralized, anonymous, divisible, scarce, etc.).

Part 1

Chapter 1

The story of Satoshi Nakamoto and Hal Finney.

Hal was the first who believed in the idea and a couple of months later he was iterating on the initial codebase together with Satoshi.

The story of cypherpunks/extropians (concerned with privacy & security).

1970/80 mathematics at Stanford/MIT enabled everyone to safely encrypt messages.

Hal was more ideologically driven rather than financially.

Chapter 2

Hashcash was invented by Adam Back.

Bit gold by Nick Szabo.

The origin story of Bitcoin and S.N.

The inner working of Bitcoin.

Hal Finney as the main evangelist and believer in the early days of Bitcoin

Chapter 3

Martti Malmi introduced. A young programmer, early Bitcoin dev.

Satoshi starts promoting btc by pushing political ideas and not just technical spec.

The problem of getting users to use btc — a need for an app (the first mention of Bitcoin exchanges)

Btc logo design.

The word cryptocurrency is born.

First $ transaction — 5050 btc was exchanged between Martti and NewLibertyStandard for 5$. The price was set based on the electrical bill needed to mine btc back then by NewLibertyStandard.

Chapter 4

Laszlo Hanecz was the first who started mining using GPUs!

He famously sold thousands of Bitcoins for a pizza— the 1st time somebody wanted to sell a real thing to get Bitcoins.

Gavin Andresen is introduced, as the 2nd core developer (after Martti) — he used his real name and pic on the forums that turned out to be quite important. He became the face behind the community.

Slashdot news on Bitcoin gave rise to a new influx of users.

Chapter 5

Jed McCaleb is introduced. He created Mt. Gox (Bitcoin exchange) out of frustration because he couldn’t buy Bitcoin during nighttime.

No deposit insurance and no regulators overseeing the safety of operation in the world of crypto.

Artzforz found a serious Bitcoin bug where somebody could spend money from another person’s account!

Wikileaks blockade (the government ordered PayPal and other providers not to allow transfers to/from Wikileaks) further increased Bitcoin’s popularity.

Satoshi stopped posting to the forum (December 2010) — the mystery of who is behind that name escalates.

Chapter 6

Mt. Gox ownership transfer (from Jed to Mark).

Silk Road is born (Ross Ulbricht).

Chapter 7

Roger Ver is introduced — a successful businessman operating in Japan.

Forbes and Times cover the phenomenon of Bitcoin.

Satoshi disappears (Martti, Gavin continue being lead devs).

Bitcoin starts getting a bad reputation: hacker attacks, Silk road, etc.

Chapter 8

A big breach of Mt. Gox.

Jesse Powell — Roger Ver’s friend introduced helping to fix Mt. Gox.

Chapter 9

Private keys on Bitomat were accidentally deleted. This further emphasizes the tradeoff — centralized platforms vs directly operating on the blockchain.

Centralized platform MyBitcoin’s CEO steals everyone’s Bitcoin and disappears.

Mike Hearn creates an email list in Google for Google employees to get involved.

Charlie Lee (creator of Litecoin) is introduced.

1st Bitcoin conference.

Chapter 10

Erik Vorhees is introduced (approached Roger on the conference) — Bitcoin evangelist same as Roger Ver.

Various occupy/anti central bank movements (occupy wall street, free state project) emerge that benefit Bitcoin.

Chapter 11

Dread Pirate Roberts is born — the idea was for Ross Ulbricht to have an alibi — “he wasn’t running the site on his own the whole time, he was just 1 man that ran it for a 1 period of time” would be the argument.

Vendor feedback/rating was keeping the Silk Road in check.

Marco Polo operation started, an agent by the name nob was about to form a relationship with Ross.

Part 2

Chapter 12

Roger the Bitcoin evangelist!

Casascius coin was first manufactured thanks to Roger (this is how media visualizes btc coin even today).

Charlie Shrem of BitInstant (“add-on” to Mt. Gox — makes things faster and easier) was introduced (Roger invested into BitInstant).

Aside from the ideas of decentralization and anonymity Bitcoin offered faster payments back then (they gave an example of how a Japanese bank had to write a 9B$ paper check for JPMorgan because they had holidays and the e-transaction would be too slow).

David Azar introduced (Syrian Jewish as well, invests into Charlie’s BitInstant).

Chapter 13

Erik’s gambling site SatoshiDice is mentioned and his friend Ira.

Peter Vessenes and Patrick Murck introduced (Bitcoin Foundation)

Mt. Gox’s Mark problem stated (micromanagement issues, speed, etc.)

Roger is called the “Bitcoin Jesus”.

Chapter 14

Barry Silbert NYC tech guy introduced.

Tyler and Cameron Winklevoss introduced, David Azar met them and offered to invest in Charlie’s BitInstant.

David & twins vs Barry and FirstMark as investors into BitInstant.

Chapter 15

Wences Casares — a successful Argentinean entrepreneur introduced.

The story of extreme inflations in Argentina.

David Marcus president of PayPal introduced (Wences converted him).

Wences’s friend Fede introduced (cofounder of Lemon).

Peter Briger of Fortress introduced (meetup at his place).

Chapter 16

Cops (user nob) are getting closer to Ross. First big arrests.

Chapter 17

Erik and Ira create a side company Satoshi Ltd. (they were still in BitInstant).

Fights with David and Winklevoss twins.

Erik leaves, BitInstant’s bank stops cooperating, forcing them to shut down.

Chapter 18

Fortress, Winklevoss, and others start investing heavily into Bitcoin.

Big secret tech leaders conference happens — Marc Andressen, Reid Hoffman (LinkedIn), Twitter representatives, and others.

Wences was the biggest evangelist.

Skepticism arises because PayPal also had big dreams before regulations kicked in.

Jed (the original creator of Mt. Gox) created Ripple — A.H., and PayPal invest.

Chapter 19

The dawn of ASICs. Dedicated mining machines. (CPU -> GPU -> ASICs)

Bitcoin pools emerge — you can be sure you’ll get a piece of the cake. Centralization concerns.

The full fork problem. BTC Guild downgrades to version 0.7 (Bitcoin once again shows that it has correct incentives in place — if BTC Guild hadn’t done it the legitimacy of Bitcoin would be at stake!)

USA government’s 1st statement on the legality of Bitcoin.

btc market cap crosses 1B$.

Chapter 20

Lack of trust in Mt. Gox. Wences and others put their private keys in cold storage in a bank.

Competition to Mt. Gox and BitInstant: Coinbase, Bitstamp (exchange).

Mt.Gox has very bad rates because they’re slow. Obscure code changes break clients.

Chapter 21

The biggest public investment into a Bitcoin company (Coinbase) to date happens (2013).

Bitcoin Foundation’s 1st conference (twins, Charlie, and others gave talks).

Chapter 22

Big names are buying (Pete Briger, Dan Morehead, etc.).

Operators of Liberty Reserve were arrested.

Banks are very reluctant to deal with Bitcoin (even big players like Peter Briger can’t get Wall Fargo to cooperate).

People are incentivized to hold Bitcoin since it’s truly scarce.

Charlie closes Bitinstant for legal reasons. Erik sells SatoshiDice.

Ross pays for hits, more heat on him, and on Silk Road in general.

Part 3

Chapter 23

Patrick Murck (a pro-gov guy) keeps the Bitcoin Foundation floating, regularly chats with regulators.

Regulations kick in — Bitstamp and Coinbase introduce various validations/id verifications (and they overtook Mt. Gox’s and Bitinstant’s lead).

Roger and blockchain.info (wallet company — private keys are encrypted so that the company has no info on its users) as “the last stand” against centralized Bitcoin companies.

Argentina is a great use case where anonymity can be helpful, and companies such as blockchain.info can’t be subpoenaed by the gov.

Wences sells Lemon and starts his new Bitcoin company Xapo.

Chapter 24

Ross is finally arrested.

btc price plummets, gov statements don’t indicate that they want to stop btc from happening — smart people (like Winklevoss twins) start buying.

Secrete gathering on lake Tahoe organized by Dan Morehead.

Chapter 25

BTC China exchange is formed.

Bobby Lee becomes a co-founder (his brother Charlie Lee mentioned as the founder of Litecoin (“silver to btc’s gold”)).

Tencent becomes the payment processor (and then buying became really easy in China).

Mining in China is getting bigger (rather than buying/selling because of China’s gov).

Various news in China/USA drive price changes (Baidu, PayPal).

Chapter 26

Patrick and other Bitcoiners present to the USA senate — very positive reactions.

Again gov dictates what happens.

Roger and other libertarians keep pushing for anonymous Bitcoin projects.

In China we have a different story: Tencent stops collaborating with BTC China (gov has declared btc as a digital commodity and thus banks/payment processors like Tencent can’t deal with btc exchanges).

Chapter 27

BitPagos solves the problem of receiving USA payments in Argentina. Another great application for Bitcoin.

“Bitcoin’s evolution in the United States and China was showing how the technology could become dependent on the official financial system and government approval. Argentina, on the other hand, was showing how it could develop without any of that”.

Xapo is started, 1st feature was the “vault” — safely storing private keys offline.

The wild West of China — Tencent stops collaborating but banks are still collaborating with Bitcoin businesses.

Federal reserve praises some properties of btc, more and more companies start using btc (like Overstock).

Chapter 28

A.H. invests heavily (Coinbase, 21e6 (mining company), etc.).

Wences’ Xapo gets more high-profile investments.

Charlie is arrested and bailed out for 1 million $.

New hearings, the day after Charlie’s arrest (twins, Barry Silbert, Fred Wilson, etc.)

Banks are increasingly reluctant to deal with btc companies.

Divesh Makan is mentioned as the “spider of Silicon Valley”. He manages money for many billionaires. The so-called family offices.

Chapter 29

Mt. Gox problems — Mark tells the world that Bitcoin has a bug (transaction malleability, a known “quirk”) -> price plummets, but quickly bounces back.

Huge Mt. Gox “attack” (they lost 750k Bitcoins! due to the aforementioned bug). The news is leaked by the twobitidiot.

Chapter 30

Press finds random people and claims they are Satoshi (Dorian Nakamoto was the name of this one guy who got very famous).

The real Satoshi Nakamoto writes on the forum that he is not Dorian.

Big players meet secretly in a private conf at Goldman Sachs.

Internally banks and federal reserves are looking for innovative ways to use the blockchain concept.

Bitfury (a Ukrainian mining company) is introduced — they promised that they won’t have more than 40% of computer power (they were that powerful!).

Marginalization of the early Bitcoin community (ideologically inclined people).

Big conference in Austin (Silicon Valley people + celebrities).

Chapter 31

Meeting at lake Tahoe (Dan Morehead).

Nick Szabo may be Satoshi Nakamoto.

Bitcoin conference with huge players like Bezos & Gates.

Wences converts Bill Gates! 😅

And that’s it! Hopefully, you found it somewhat useful. Now drum rolls.

People you should (probably) check out on Twitter

Here is a list of people I started following after reading the book (in no particular order):

• twobitidot — a guy who leaked the confidential document from Mt. Gox
• nealstephenson — a guy behind the Cryptonomicon Sci-Fi book (that talks about universal money), fairly respected in the digital currency community in general
• pwuille, jgarzik, marttimalmi, gavinandresen— early Bitcoin developers
• NickSzabo4 — a guy for whom many believe to be Satoshi Nakamoto because of his engagement in the early digital cash communities, the fact that he modified some dates on forum posts just before Bitcoin whitepaper was published, and that his permutated name acronym is the same as Satoshi’s.
• adam3us (Adam Back) — proposed some fundamental ideas in the cryptography space
• jespow — Kraken co-founder, introduced as Roger Ver’s friend in the book
• CharlieShrem — a guy behind the BitInstant platform (made it easier to buy Bitcoins back in the days of Mt. Gox)
• jedmccaleb — original founder of Mt. Gox and Ripple!
• bobbyclee (Bobby Lee)— founded the most famous Chinese exchange
• satoshilite (Charlie Lee) — founder of Litecoin and Bobby’s brother
• virtuallylaw (Patric Murck)— a lawyer who was vital for the survival of Bitcoin foundation in the early days
• BarrySilbert — successful NYC entrepreneur involved in btc since the early days
• nejc_kodric — co-founder of Bitstamp (exchange)
• tylercowen — prolific blogger in the space
• niccary — co-founder and CEO of blockchain.com, anonymous wallet
• TimDraper, fredwilson VCs involved in Bitcoin since the early days
• valeryvavilov — co-founder of Bitfury (involved in mining business)
• FEhrsam (Fred Ehrsam)— co-founder of Coinbase
• balajis (Balaji Srinivasan)— founder of 21e6, a famous, secretive ASIC company (producing dedicated mining chips)
• RealRossU — creator of Silk Road (a platform where you could sell/buy illegal products/services). He’s still in prison.
• ErikVoorhees, rogerkver — biggest btc evangelists back in the days
• tyler, cameron — Winklevoss twins, early Bitcoin adopters. Probably most famous for their dispute with Mark Zuckerberg around the Facebook idea.
• davidmarcus — president of PayPal back then
• bgarlinghouse — CEO of Ripple
• paulkrugman — Nobel prize winner in economics
• reidhoffman — founder of LinkedIn
• sama (Sam Altman) — YCombinator, OpenAI guy
• naval (Naval Ravikant) — founder of AngelList
• cdixon (Chris Dixon) — partner at Andreessen Horowitz
• pmarca, bhorowitz — Andressen Horowitz founders
• APompliano (Anthony Pompliano)— btc maximalist/evangelist
• saylor (Michael Saylor)— founder of MicroStrategy and btc maximalist
• RayDalio — founder of Bridgewater hedge fund
• jack (Jack Dorsey)— founder of Twitter
• VitalikButerin — main co-founder of Ethereum
• BillyM2k (Billy Markus)— co-founder of Dogecoin

Try and explore their feeds, and based on your preference start following those people whose ideas resonate with you.

But, also try and follow some people with whose ideas you don’t necessarily agree — you need a healthy dose of different opinions in your life. Who knows maybe you’re wrong and they are right. Be open-minded.

If there is something you would like me to write about — write it down in the comment section or DM me. I’d be glad to write more about mathematics & statistics, ML/deep learning, CS/software engineering, landing a job at a big tech company, getting an invitation to prestigious ML camps, finance, etc., anything that could help you!

Also feel free to drop me a message or:

1. Connect and reach me on 💡 LinkedIn and Twitter
2. Subscribe to my 🔔 YouTube channel for AI-related content️
3. Follow me on 📚 Medium and 💻GitHub
4. Subscribe to my 📢 monthly AI newsletter and join the 👨‍👩‍👧‍👦 Discord community!

And if you find the content I create useful consider becoming a Patreon!

Much love ❤️

Note from Aleksa from the future (November, 2025): joining a prestigious company is less important than joining a great, tightly knit, ambitious team. In retrospect I had much better work experiences than that at DeepMind (I left in 2023 to start my own startup).

The company was already bloated (~1200 people, now it’s more like 5000–7000), the culture significantly broken (e.g. memes were being removed from internal Slack channel because they “offended” someone), and I couldn’t freely move around to work on stuff that actually interested me.

Having said that having that DeepMind badge opened up many doors for me — so i will not discount that nor discourage you from chasing “the current thing”. :) And I’m honestly grateful for the opportunity.

If I were you, on the job market right now, I’d look at the smaller companies, the next DeepMinds of the world.

I recently landed a job at DeepMind as a Research Engineer! It’s a dream come true for me, I still can’t believe it! 😍 (if you feel like an imposter sometimes, trust me, you’re not alone…)

I don’t have a Ph.D. in ML.

I don’t have a master's in ML.

In fact, I don’t have any type of degree in ML — I did my bachelor’s in EE.

That’s it. That’s my formal education. I dropped out of my EE master’s as I figured out I could learn stuff much more effectively and efficiently on my own.

My first exposure to ML was back in 2018. And my first exposure to the world of programming was when I was 19 (I’m 27 now).

So how in the world did I pull it off? (hint: I’m not that smart, you can do it!)

In this blog, I’ll try to tell you the whole story. I’ll be very transparent in order to help you guys out as much as possible. This blog even contains a snapshot of my resume (very much suboptimal, I feel uncomfortable sharing it 😅), the information on how I got referred to DeepMind, and many more details that will hopefully make this blog much closer to your heart.

I’ll structure this blog post into roughly 3 parts:

1. I’m going to give you some context on who I am, and what my learning journey so far was! As I don’t want to set false expectations that you can pull this off in a short period of time, without an ML degree, if you don’t already have solid foundations and experience with self-education.
2. I’m going to explain to you, in nitty-gritty details, how my ML curriculum looked like since my first exposure to ML back in 2018.
3. Finally, I’m going to tell you how exactly did my final preparations for DeepMind look like (+ more info on how I got a referral).

Note: for all practical purposes you can read DeepMind in this post as “any top-notch AI lab/company like DeepMind, OpenAI, FAIR, AI2, etc.” because I’m fairly sure that the same (or at least similar) principles would apply there as well.

Well, let’s go!

Update: if you prefer video format I made a video based on this blog:

Part 1: My background (no ML degree)

It’s the summer of 2017. I‘m doing my first ever internship at a small German startup called Telocate, as an Android dev. I’m about to graduate (in September, once I return back to Belgrade, Serbia) from an EE program that had a strong focus on digital/analog electronics and less so on CS.

Sometime around the end of my studies (end of 2016), I realized I actually want to completely pivot to the software industry. It’s exciting, it’s much more open compared to the hardware industry, there are all these hackathons and datathons, and it offers amazing salaries.

So earlier that year (2017) I started learning Android on my own and I landed this internship in Freiburg, Germany. I was so excited — the world was mine.

Snap back to the summer of 2017, Freiburg. I’m chatting with a friend whom I knew was a very successful software engineer. He’s done multiple internships at companies such as MSFT, Facebook, Jane Street, etc. and he was my peer.

I realized I’m falling behind by a lot (along the SE dimension at least)! Some of my friends were doing competitive programming since their high school days. Algorithms and data structures were their second nature at this point.

I wanted to apply for big tech companies, and I suddenly realized that all of the skills I accumulated throughout the years are not that important when you’re interviewing at FAANG.

Nobody cared that I speak 5 languages, that I know a bunch about how microcontrollers work in the tiniest of details, how an analog high-frequency circuit is built from bare metal, and how computers actually work. All of that is abstracted away. You only need…algorithms & data structures pretty much.

I felt confused and angry at the same time. Some kid who’s 16 (I was 23 back then), could go and do some Leetcode problems for 9 months and he is already ahead of me in this respect. Even though he/she lacks so much of the fundamental engineering knowledge.

It’s a nasty feeling, I very much remember it. I was very ambitious throughout whole my life, and hardworking, and yet I’m falling behind. I remember hating my life for not figuring out about this earlier on in my childhood.

But, what can you do, this world is obviously not fair. Some kids get to grow up in Silicon Valley and have rich and educated parents whereas some kids are born in a “third world” country, sometimes starving to death.

This strong feeling of injustice was giving me energy. I thought to myself — I’m going to hustle really hard over the next years. Warp speed on.

I came back home from my internship in Germany. I successfully graduated 3 days after that. I remember some of my friends were celebrating their graduation for 3+ weeks, drinking 🍺 and everything that goes along. Me? I celebrated my success over the weekend and I immediately started planning ahead. I felt I was on a mission.

I enrolled in a master’s program “symbolically” (you get some perks as a student) but I knew that I really wanted to land a job at a big tech company.

And so I made my own SWE curriculum, plus, I started attending an algorithms course at my university without even being enrolled. I was sitting there with students that were 2 years younger than me.

Over the next couple of months I was working like crazy, and I immediately started applying. Soon after, I got myself an interview with Facebook in December 2017 and I failed miserably on my first interview.

I kept learning and I got an invitation from Microsoft — the tests are in March 2018 they said. Got it, I’m gonna work hard until then. I was learning, I was attending hackathons and datathons, and I started doing competitive programming (mostly Topcoder).

Then, in February 2018, I heard about this ML summer camp being organized by Microsoft people (we have an office in Belgrade, the only big tech company we have). I applied and after nailing a 3-day long entrance test I got accepted!!! (my decent digital image processing background helped)

I had a strong gut feeling that this may be my opportunity to show that I’m good enough to work at Microsoft! 💡

Around that same time, I landed an internship in Ouro Preto, Brazil via a student organization called IAESTE. It ended up being more of an amazing life experience (I lived in a fraternity with 11 Brazilians and I had to learn to speak Portugueses) rather than a tech internship. It was scheduled right after my ML summer camp. It’s as if the stars aligned!

March 2018 came, and I did my first tests with Microsoft. I failed the written entrance tests. Luckily 2 months later they gave me another opportunity. I tried again, passed the written entrance tests, did a round of 4 back to back interviews on-site, and 3 weeks later…they told me that there currently are no positions opened for me. I interpreted that as a rejection. Again.

So if there is somebody that can empathize with newcomers to this field that’s me. I know that it can be very mentally draining, this whole job hunt process.

Fast forward to the summer of 2018. I attended the ML summer camp and I nailed it. I was very engaged, I learned a lot and our final project was internally voted as the best one (solving Rubik’s cube with deep RL).

Immediately after the camp, I flew to Brazil in early August 2018.

3 days in and I get a call from Microsoft. You got an offer. We need you back here asap.

I was super happy and super sad at the same moment. I had to leave this beautiful country of Brazil after a bit over a month instead of after 3 months. It was either that or I wouldn’t get the job. They told me that the team that wants me is the “HoloLens team“!!! The same team that ran the summer camp. The most famous, exciting team we had in Belgrade — I couldn’t miss this opportunity. I took it.

September 2018, I’m in Belgrade working at Microsoft as a software engineer! I pulled it off. I was incredibly happy.

And that’s where my SE and ML story really started.

But, ever since I attended that summer camp the idea of DeepMind and the feeling that it is almost unreachable for me was deeply in my subconsciousness. The only folks I knew there (some of them were lecturing at the camp) studied at Oxford or Cambridge.

But there is a part of me that’s very attracted to impossible challenges. I thought to myself I nailed Microsoft even though I once thought that’s very hard. Why should this be any different?

And so that idea was buried into my subconsciousness. I really enjoyed the fact that I’m at Microsoft now.

Throughout 2018 and 2019 I was focusing really hard on learning as much as I can about software engineering while working at Microsoft. I was reading coding books such as the C++ book from Scott Meyers, etc.

Other than that I started doing ML in my free time.

I finished all of Andrew Ng’s Coursera courses before the end of 2018. I attended internal ML hackathons, I was heavily involved around the organization of the aforementioned ML camp, and I started reading research papers!

I still remember how they made me feel (and they still do!). 😂 I felt very dumb. I was like, really? I studied EE for 4 years and I can still feel this way? (Little did I know; a semidefinite chuckle)

Somehow, around that same time, I also discovered Lex Fridman’s podcast. That’s 3 years ago before he even had 50k subs on his YouTube channel! I remember religiously watching all of the 20ish podcasts he had back then.

I remember I started feeling increasingly confident with my ML knowledge. And so beginning of 2019 I shared my first blog ever on how to get started with ML:

Get started with AI and machine learning in 3 months

(My writing was terrible back then, and it still probably is, I apologize for that. I’ll try to update it over the next couple of days after publishing this blog (I won’t change the content as I want to keep it as a historical document))

End of 2019 I internally made a switch to an ML role (due to my religious engagement that my manager noticed). I was sent to ICCV 2019 conference.

I returned and I got the task of implementing a paper from scratch in PyTorch. I didn’t know any PyTorch back then. Needless to say — I learned a lot.

And this is where another warp speed moment happened. But this time, I was smarter, this time I’m going to share my journey online as I progress.

So what happened? What made me change my mindset? What made a push from this old Aleksa, that was learning and working hard but kept everything for himself, to this new Aleksa?

Well, throughout 2019, as a “side quest”, I was reading a lot about entrepreneurship, marketing, and personal finance. And along the way, I found this guy called Gary Vaynerchuk who motivated me to start sharing stuff I do online. Thanks Gary! ❤️

I know most of you probably don’t like him (if you are even aware of him in the first place), but he had a positive influence on my thinking. And for that I’m grateful. He definitely wasn’t the only person that influenced my mindset (I was sharing my code even back in 2018) but he finally made it click.

You probably may say (if you don’t know me): “Well, this story is inspiring and all but how did you create your own SWE/ML curriculum in the first place?”

Well, before all of this happened, I already had a strong track record of learning on my own. If it still feels like magic I strongly recommend you go ahead and read this blog:

5 Tips to Boost your LEARNING

I’ll leave a TL;DR here: through my workout routines, learning human languages throughout my life, and maths (all of that on my own) I got skillful at learning. Coursera’s “Learning how to learn course” helped as well.

Now that you have all of the necessary context — let me tell you about the ML curriculum I followed to land a job at DeepMind!

Part 2: ML curriculum (January 2020 — June 2021)

New Year 2020 came along. I just finished reading “Crushing it” from garyvee and decided I should start my own YouTube channel sharing my ML learning journey along the way.

The pandemic struck and I was in warp speed again. The stars aligned. ⭐

I decided that over the next year I want to cover various subfields of AI, and I roughly decided that I want to dedicate 3 months to each one of these.

But things weren’t that smooth in the beginning, I was still figuring stuff out. My first subfield, neural style transfer (NST), took longer than 3 months because I found it enjoyable and I thought why rush it?

Along the way, I found a perfect strategy for myself.

I structured my learning the following way. I had “macro” cycles (3-month periods where I tackle a new subfield) and I had multiple micro-cycles interspersed over macro.

The micro-cycles are of 2 types:

• Input mode: I’m ingesting information. The goal is to either get a high-level understanding of the structure of the subfield (blogs, videos) or an in-depth understanding of the topic at hand (research papers, books, etc.)
• Output mode: I’m sharing information I accumulated during my input mode. Teach others! Create public artifacts like YouTube videos, GitHub projects (1 project in the middle of macro), and blogs (1 blog at the end of a macro). As well as higher-frequency updates on LinkedIn. Much later I started sharing on Twitter and Discord as well.

Now the tricky part was combining all of this with my full-time job at Microsoft! It took a significant amount of willpower. I’d hit the program as soon as I wake up for 2h, I’d go for a quick stroll, I’d do my job at Microsoft and a 30-minute power nap after I wrap it up, and finally I’d be working for additional 2–3 hours before going to sleep. The pandemic helped me maintain this crazy rhythm.

The time management and stress handling probably deserve a separate blog post, but it mostly boils down to willpower, correct mindset, as well as taking smart breaks (power naps are golden — let me know in the comments if you’d like a separate blog on the topic of time management).

Now let me go into the specifics of the macros I did on my way to DeepMind! Luckily I started writing blogs sometime around my transformers macro so if you want to know all the nitty-gritty details and the strategy I took, I’ll link the blogs in every single macro section.

For the first 3 macros (NST, DeepDream, and GANs) I, unfortunately, didn’t write a dedicated blog, but the intro section of the transformers blog below as well as the YouTube videos and projects I created can fill in the gaps:

Deep learning journey update: What have I learned about transformers and NLP in 2 months

Having said that, here are the TL; DRs of every macro I did:

[1] NST (Neural Style Transfer)

During this period I was learning about neural style transfer. I ended up reading a pile of research papers (though a lot less than later when I became proficient), implementing 3 different projects, which I then open-sourced on my GitHub, and I made an NST playlist on YouTube.

I think that this is a great topic to start with as it’s highly visual and visceral. I recently wrote some of my thoughts on a similar topic in this LinkedIn post. Also, digital image processing and computer vision were the fields I felt the most comfortable with and so that’s why this was a great starting point for me.

I perfected my PyTorch knowledge, learned a lot about CNNs and optimization methods, improved my presentation skills (just check out those README files 😂), became more proficient at reading research papers, and improved my SWE skills in general — to name a few.

Here is an NST image synthesized using the code I wrote:

[2] DeepDream

Ever since I first saw the pictures created with DeepDream I was fascinated. I knew I had to learn it. Every single detail. I couldn’t just use online generators and not understand what was going on.

I was reading blogs, analyzing DeepDream subreddits, and exploring various codebases. Most of the original code was written in Torch & Lua combo. I remember losing a lot of time trying to set it up on Windows. Then I switched to Linux, got it to work, only to realize that I won’t be using that repo either way. 😂

My output was a single video on the theory behind DeepDream and an open-source project. I learned a lot by doing this and I enjoyed the process. I used my code to generate these beautiful images, among them the current visual identity of The AI Epiphany. Here is an example:

[3] GANs (Generative Adversarial Networks)

GANs were still very popular in early 2020. I felt like I’m lacking the necessary background and so I decided to tackle them next.

I’ve read all of the seminal papers (as well as other less prominent papers) and I decided to implement vanilla GAN, cGAN (conditional GAN), and DCGAN models. Here is an output from the DCGAN model I trained:

At this point in time, I refined my strategy. I realized that:

• I need to write a blog at the end of each macro, summarizing what I’ve learned. That will help future Aleksas (and Aleksa from the future) tackle topic X.
• I need to cover papers on my YouTube channel as I’m reading them. A similar thought process — by doing this I’ll learn better and I’ll help others.
• I need to open-source a project in the middle of the macro. After you implement something you understand it way better. After that, all the papers I read made more sense.

Unfortunately, I don’t have GAN paper overviews from this period. Back then I was experimenting with videos such as “PyTorch or Tensorflow” which turned out to be the most popular ones (proving a hypothesis I had in my mind — higher level, beginner-friendly content always does better).

But I didn’t seek popularity (not that I would mind in general), I was seeking a highly relevant audience. I’d much rather have 1 guy/gal from “DeepMind” following my work than 100 beginners — because I knew what my goals are.

[4] NLP & transformers

I knew I wanted to learn more about BERT and the GPT family of models. Transformers were everywhere, and I didn’t quite understand them.

This time I did everything correctly and executed the plan I sketched above. I ended up implementing the original transformer paper from scratch and learned a ton! I decided to create an English-German machine translation system. Since I speak those languages, I thought that’ll make debugging easier, and it did.

I’m so grateful I did it back then — every now and then you need to understand the QKV attention, and it’s second nature to me now. You can find out much more about this journey here:

Deep learning journey update: What have I learned about transformers and NLP in 2 months

[5] Graph/Geometric ML

Even before I started doing transformers I knew I wanted to dig deeper into Graph ML (before it became as popular as it is nowadays).

And, in retrospect, I’m very glad I covered transformers/NLP first since the field of NLP had a huge influence on this field. GAT was directly inspired by the original transformer paper, DeepWalk by Word2Vec, and so on.

At this point in time, I was already chatting with Petar Veličković on a fairly regular basis (I’ll tell you a bit later more about how this came to be) and he knew that I’ll be covering GNNs in the next period. He was very receptive and he told me that I can ping him whenever I felt stuck!

I realized that this is a great opportunity! I’ll be learning Graph ML, a topic that fascinated me and I’ll be communicating with Petar, a researcher at DeepMind, who was among the best researchers in this particular field.

Needless to say, I learned a lot! I made a “popular” YouTube GNN series that was shared by influential Graph ML researchers like Michael Bronstein, Petar, and others.

I also made a popular PyTorch implementation of GAT (Petar is the first author of GAT). The reason it became so popular is that it was, according to others, the most beginner-friendly resource out there. With 2 Jupyter notebooks, PPI and Cora datasets support, a nice README, and accompanying videos it filled the gap and made it easy for beginners to enter the field.

(It became a recommended repo for a Cambridge lecture on GNNs and I even got citations. 😅)

Once again, I leaned heavily on being able to reach out to Petar whenever I got stuck, and I saw this as a great opportunity for us to collaborate.

For the whole story check out this blog:

How to get started with Graph Machine Learning

[6] RL (Reinforcement Learning)

Drum roll. Surprise, surprise! I really wanted to learn RL either way but it so happens that DeepMind is famous for its RL breakthroughs. 😅

There were so many papers and projects I wanted to understand! Like AlphaGo, DQN, OpenAI’s robotic hand that could solve Rubik’s cube, etc.

This field proved to be very different compared to other ML subfields. The data point independence assumption doesn’t hold anymore, a “bad” random seed can cause your agent to fail to converge, etc.

I’m very glad I already had a decent amount of experience before tackling RL, otherwise, I might have gotten demotivated. I started with stuff that’s very close to my heart — computer vision — and I used it to ramp up my ML knowledge. Then, I slowly progressed toward topics I knew little about.

I again had some help here! Petar connected me with Charles Blundell (who’s also a DeepMinder) and so from time to time, I used that opportunity, although a lot more frugally.

For the whole story check out this blog:

How to get started with Reinforcement Learning (RL)

And those are pretty much the main macros I had. Throughout the whole journey I was making notes in my OneNote:

(If there is enough interest I could maybe convert those to pdfs and share them on a public drive. Let me know in the comments!)

After RL, I started focusing on the latest and greatest papers. You can clearly notice that trend looking at my newest YouTube videos over the past 4 months.

And all of you who were following along my journey knew that these were my plans — I was very transparent throughout the whole process! (except for the fact that I wanted to apply for DeepMind the whole time).

Aside from this main quest, early in 2020 — when I first started reading research papers — I realized that I was having difficulties with mathematics so I took some time to read the “Mathematics for Machine Learning” book (and I went over 3B1B’s awesome playlists).

That turned out to be a great move. I’m so happy I did it similarly to how I took a step back and took the “Learning How to Learn” course in September of 2019. All of these actions helped boost my learning speed.

Here are some other books I also read in this period:

• Python Data Science Handbook, chapter 5 — helped me get a visceral feeling for SVMs, PCA, linear regression, etc.
• Automate the Boring Stuff with Python, skipped first 11 chapters — I felt like going through at least one book on Python since I learned the language on the fly. It also shaped my mindset which improved my productivity in general (in a nutshell: “automate the repetitive stuff”).
• Deep Learning— I only read the first part as I realized I don’t really need all of that theory. It should be treated as a reference manual.
• The Book of Why — I read it because I was genuinely interested in causality and Judea Pearl’s work! Here is my LinkedIn summary.

In parallel with everything I mentioned above, I was also working full time at Microsoft on a variety of SWE and ML engineering projects. Unfortunately, I can’t tell you a lot about all of that but here are some of the high level highlights I can talk about:

• Developed a glasses detection algorithm as a part of the eye-tracking subsystem on the HoloLens 2 device. The purpose of the eye-tracking subsystem is to predict the user’s eye gaze vector in real time and enable the instinctual interaction with the holograms. It also helps render graphics correctly for a particular user (everybody has different eyes, thus different IPD), etc.
• Played with video encoders so as to add the foveated rendering functionality to next generation VR/MR devices. That way you save up power as you don’t need to render the details in the peripheral vision. I ended up reading the whole reference manual on H.264 encoders (that was, again, probably an overkill 😅).
• Implemented a research paper from scratch, in PyTorch to track human body landmarks, validating a new sensor technology under consideration. This project is how I learned PyTorch (late 2019/early 2020).
• Wrote various scripts that verified labelers are doing their job correctly, developed a part of our internal metrics pipeline, dealt with rendering and improving our synthetics data (which is great, check out the paper here!), made various quantization and perf-vs-compute experiments (because I was working with edge devices), and so on.
• Interviewed candidates for SWE/ML roles at Microsoft. This made me feel more comfortable with the interview process, and gave me a perspective on how it looks like to be on “the other side of the table”.
• Taught and mentored students at the ML summer camp, including delivering lectures and running CNN workshops.

In summary that’s: one year of intense SWE study, followed by 1.5 years of slower-paced ML learning, and then another 1.5 years of warp-speed ML study; the rest is captured in my “5 Tips to Boost Your Learning” blog.

And that’s my story.

Tip: an additional thing you could do to see more details of this whole period is to just scroll through all of my LinkedIn posts, that’s where I was (and still am) regularly sharing my updates.

Part 3: How did the “DeepMind channel” open up?

Middle of the year 2020 I reached out to Petar on LinkedIn. It turned out he was already following the content I was posting there and found it interesting, so that made things a bit easier! ❤️

During my ICCV19 conference I met other cool DeepMinders, that were also from Serbia, like Relja Arandjelović (who did his Ph.D. with professor Andrew Zisserman at Oxford) and Jovana Mitrović (Oxford girl as well!).

Even though this was my first conference I immediately realized that the value of these conferences lies in:

• 1–1 conversations with the paper authors during the poster sessions
• Meeting other likely-minded people

IMHO, many people get this wrong. By the time you attend the conference most of the papers were already published to arXiv months ago (and chances are somebody already covered them and shared the summary online 😂).

That’s why virtual conferences will never be a good replacement.

Then sometime around September of 2020, I made it clear to Petar and Relja that sometime next year, probably in April, I’ll start applying for DeepMind and asked them for any tips on how I could prepare! In the meanwhile, both of them offered to refer me! ❤️

Fast forward to Friday, April 2021. I was casually chatting with Petar, mentioned that I’m finally ready to apply for DeepMind, and while we were chatting he just said (without me even realizing): “ok, I just finished submitting the referral!” 😂

He then pinged a recruiter (Cameron Anderson, I didn’t know him back then but I can tell you now that he’s amazing at his job) on LinkedIn, mentioned who I was, and asked whether it’s fine for me to submit my LinkedIn/YouTube/GitHub instead of my resume. It turned out that Cameron was already following my work! And he said that it’s fine! My God, I was so happy.

Everything happened so fast. I just sent my LinkedIn profile and I already had my interview scheduled for Monday.

So summarizing. Having high quality connections does matter! BUT. Please don’t spam people. Instead, create genuine connections. Ask what you can do for them and not what they can do for you. None of them, including me, won’t recommend a person unless we’re familiar with that person’s work already and we can make some guarantees.

(Pro tip: contribute to an open-source project of the company you care about — that’s a great way to get noticed! That way some of them will be familiar with your work and you can later ask for a referral —that is if they don’t already offer it themselves! Why is nobody using this strategy? It beats me.)

At one point an automated email came asking for my resume (although Cameron confirmed that it’s not necessary). After some quick edits, I decided to send them my resume without investing too much effort into it (thinking that they probably won’t even read it).

Which in retrospect could have been a big mistake. A well-written resume matters! And my interviewers did in fact use it. It’s easier to ask questions looking at a 1/2 page resume than at a LinkedIn profile.

I’m leaving it here for legacy:

There are many things wrong with it. For a start, I should have written a lot more about my latest work at Microsoft. The thing is that my team at Microsoft was very secretive (for quite some time my other colleagues from Microsoft didn’t know what I was working on), and so I took the safest strategy — of not communicating anything to the outside world. 😂

Also, my resume was always a 1-pager, I should probably cut all of these older projects out. I have many other ideas on how to improve it.

Finally, let’s see how my last preps for DeepMind looked like!

Final interview preparations

Ok, at this point I managed to build up solid foundations and I was referred for an RE position. You do need a solid background to land a job at DeepMind. I’m fairly sure I wouldn’t even get a chance to interview if I didn’t have all of my GitHub projects, prior experience, and other public artifacts.

(I don’t have any paper publications but I compensated in different, original ways. REs don’t necessarily need publications, as you can see. But the game may be different for RSs (research scientists). I’m fairly sure they need a strong record in publishing to top-tier conferences and a Ph.D. Take all of this with a grain of salt, I’ll update this part when I start working at DeepMind and get myself better informed.)

But the battle wasn’t won. As all of you know, the interviewing processes are…well, kind of dark magic.

Even when you’re really good, big tech companies may reject you. It’s safer for them to reject a great candidate (by a mistake) than to accept a really bad one (they want to reduce the false positives). Repeat after me — hiring pipelines are never perfect — this will save you from a lot of unnecessary stress!

Secondly, you’re often asked stuff you won’t actually be using in your daily job and that’s why you have to explicitly prepare for the interviews.

Let me now tell you how I pulled it off.

First, here are some general tips:

• Research every single one of your interviewers. Understand where they are coming from and what they do, so that you can ask relevant questions. Don’t waste their time. Check out their LinkedIn/Twitter profiles (they always have at least one of those) and their Google Scholar page. Read all of their papers if possible or the ones with the most citations/the ones where they are the first author. For some of my DeepMind interviewers, I read more than 10 papers. 😅 That’s probably not needed, but I was passionate enough (read nuts) to do it.
• Research every single one of your resume projects. You should be able to talk in-depth about all of those projects. Be those your GitHub projects or the ones you did at some previous company.
• Do some mock interviews. Find friends who do AI at whatever company and are ideally interviewers themselves.

For DeepMind I also did serious research around AGI. I read some of the seminal AI papers like:

• A Collection of Definitions of Intelligence (my LI summary)
• Universal Intelligence: A Definition of Machine Intelligence
• On the Measure of Intelligence (my LI summary)

I’ve previously also researched Alan Turing’s seminal “Computing Machinery and Intelligence” paper as well as follow-up papers from John Searle, etc. You probably don’t need these. (in case you want a summary here are my blogs, parts one and two)

And I also watched YouTube videos featuring some of the following people: Demis Hassabis, Shane Legg, Marcus Hutter, F. Chollet, Murray Shanahan.

(Flex: Demis liked a couple of my tweets 😍)

The reason I did this research, is because it’s DeepMind’s mission — cracking AGI and using it to solve difficult scientific problems. You can also expect “behavioral” questions around these topics so I wanted to have a sound background.

Research as much as you can about DeepMind. For me, it came naturally since I was covering papers on YouTube, implemented some of them, knew people, culture, etc.

Those were some general tips. Replace DeepMind with your dream company and map a concept like “AGI” to “whatever my dream company cares about” and do your research on it.

Now let’s see how DeepMind’s hiring pipeline looks like for an RE position at the “Core” part of the team (more on DeepMind’s org a bit later).

The pipeline consists of the following interviews (at the time I’m writing this, it may change later in the future but you should still have a general idea):

• The initial chat with your Recruiter. This is supposed to give your recruiter an idea of whether you’re more interested in applying for the Applied or the Core part of the team. As with any other interview, treat it with respect, there may be a behavioral component to it. (1 interview)
• The “quiz”. The idea is to test your understanding of fundamental topics in maths, stats, CS, and ML. (2 back to back interviews)
• The coding interview. Your classical SWE interview if you ever interviewed for Google, Microsoft, or any other big tech company. (1 interview)
• Team Lead interview. You’ll be discussing various topics. Anything that’s on your resume may come up, as well as open-ended ML problems, and behavioral questions. (1 interview)
• Senior Team Lead interview. Very similar in structure to the above interview. Additionally, this interview more explicitly assesses whether you’re a fit for the team. (TBH, you should treat every interview with that in mind) (1 interview)
• People & Culture interview. Your classical HR interview. (1 interview)

Let me now give you some tips on how to prepare for these!

My goal here is not to give you an unfair advantage or a way to game the hiring system, but rather guidelines on how you can get better at these topics, and thus hopefully a better engineer (and a better human being)! 😜

I’m writing this for my past self — the Aleksa who would have been grateful to have this guidance.

Interview #1: A chat with the recruiter

Do your due diligence and understand what exactly it is that you want to do at DeepMind (as much as you can!). DeepMind has 2 “teams” (it’s a soft type of a division, both “teams” are working towards a common mission):

• The “Core” research team. They do more pure research types of projects. Examples of more “production-ready” projects would be AlphaFold (I guess the Applied team was involved here as well), AlphaGo, etc. Other projects that REs in this team do may involve helping implement certain research ideas in collaboration with research scientists.
• The applied team. This one is self-explanatory. Some of the projects they did are the Data Centre energy reduction project, improving the RecSys for Google Playstore, WaveNet, saving power on Android, and much more (they’re not only Google-focused).

Go over to DeepMind’s website, read their blogs, and research all of the available positions you care about.

Also, I strongly recommend you read the first (behavioral) part of the “Cracking The Coding Interview” (CTCI) book. As I said every interviewer will be assessing your personality and whether you’re fit for the team — so some preps in that regard definitely won’t hurt!

Interviews #2 and #3: The “quiz”

This is probably the hardest part to prepare for (although TBH, it wasn’t for me, it really depends on where you’re coming from). You need solid technical foundations to pass it.

I’ll tell you how I prepared myself, although again, it might be overkill:

• [CS] I read an Algorithms & Data structures textbook that was written by a Serbian professor so it won’t be that relevant to you. The main point is — I picked a book I was already familiar with (I went through it back in 2017 preparing for big tech companies, remember?). This knowledge will also come in handy later during the SWE interview so your time won’t be wasted. It’s a 400-page book. I skipped some parts that I knew are “too advanced” like B-trees (external-memory data structures) and certain hash function details (I probably read ~340 pages). I covered topics such as linear/non-linear data structures (graphs/trees), and various sorting and searching algorithms.
• [CS] Skimmed through OS course materials at my faculty. Stuff like deadlocks, threads, virtual memory, etc. Also, I read this blog on threading in Python to brush up on my knowledge of the terminology.
• [CS] Skimmed through the big O-notation formal theory from the “Introduction to Algorithms”(CLRS) book.
• [Maths, classical ML] Read the “Mathematics for ML” book, again. (As you recall the first time I read it was in 2020 so I was way faster this time).
• [ML] Brush up on my RL theory. I just googled stuff and found some ad hoc materials. What’s an RL framework (env-agent-reward), what is an MDP, etc. My “Getting Started with RL” blog is a good starting point.
• [Statistics] YouTube videos (channels such as jbstatistics and StatQuest with Josh Starmer helped me), googling for unknown concepts, skimmed through this material (sheets 1 and 2) from Cambridge. I made some reasonable assumptions on what they could ask me.

That should be enough. If you already feel confident with some of these — just skip it — treat this as a biased guideline rather than as ground truth.

My interview process already started at this point so this was more of a sprint rather than a month-long preparation. I strongly recommend you start preparing before you kick off the interviewing process.

Interview #4: The coding challenge

• I solved some of the CTCI problems. I didn’t go through every single section of the book — it takes too much time. I solved most of the problems from the first 8 chapters (minus chapters 5, 6 where I skimmed the solutions or quickly solved some of them in my mind). I skimmed through the solutions of section 9 on system design, I skipped chapter 10 as I have already implemented some of the most popular sorting/searching algorithms while preparing for the quiz, and I’ve completely ignored chapters 11–15.
• I read a chapter on dynamic programming from the CLRS book.

A couple of tips on how to read through CTCI:

• First, try to solve a problem yourself. Only if you’re stuck, take a look at the first hint. If that doesn’t help see the next one, and so on. In the end, even if you managed to solve the problem, go through the solutions. You’ll definitely gain some new perspectives on how to approach that problem.
• Make sure to always write down the memory and time complexity of your solution before looking at the solution section.
• Pick a language that you feel the most comfortable with. The first time I went through this book (back in 2017) I used Java (as I was doing Android back then and the book itself contains solutions in Java). This time I used Python. You can find solutions in any language online or just use the GeeksForGeeks website.

You really don’t need anything else other than CTCI! Just that. 😅

Interviews #5, #6 and #7: team lead interviews and people & culture

Just read the behavioral/first part of the CTCI book and follow my general tips (like be ready to talk about your projects and be familiar with your interviewer’s research).

They’ll ask you questions such as: “What do you like about DeepMind?”, “Why DeepMind and not some other company?”, “Tell me something about your favorite project”. Stuff like that.

And that’s it! You should be good to go now!

Finally, let me briefly tell you what happened to me. Some of you may recall from this video that I mentioned failing to land a job at DeepMind:

I failed the 6th (Senior Team Lead) interview. IMHO, it was literally just a misunderstanding more than anything else (I received the feedback via email — they were very kind to provide it). His feedback was that I came across as more passionate about research than engineering — and i was applying for an RE position! Funny enough, I only emphasized research because I already felt strong on the engineering side.

As a “technically strong” candidate, I was rerouted to the Incubation/Applied part of the team. I again prepared myself following the general tips I gave you guys and got 4 times YES. I was accepted. Hurray!!!

The interviews were very conversational in nature. They told me what the team does, there were some behavioral questions, some ML system design, and open-ended problems. The best way to prepare for those is to have genuine work experience.

And that’s it. I guess the rest is history.

And these 2 viral posts on Twitter and LinkedIn. 😂

My next steps

I’ll be joining DeepMind later this year (December 2021) as a Research Engineer. I’m very excited about this next period of my life!

Many new things are waiting for me. I’ll be moving to London, UK. I’ll have to build new social circles. I’ll be exposed to a new culture, new languages (as London is quite multicultural), a new company, and even a new OS! 😅

As for what you can expect from me over the next period, I’ll be:

• Sharing everything I do on my Twitter, LinkedIn, and Discord!
• Covering more geometric DL papers on my YouTube channel!
• Explaining various open-source ML tools and research code as well.
• Learning JAX and this time sharing the journey with you guys!

On a personal note, I’ll try and travel a bit more before December and I’ll be doing some investing (I’ll be reading the “The Intelligent Investor” book).

I got so many kind words and requests from my community at The AI Epiphany:

And so I felt obliged to share more about my journey to DeepMind in order to help you guys become better. And hopefully, by sharing this deeply personal journey I managed to help at least some of you.

If there is something you would like me to write about — write it down in the comment section or DM me. I’d be glad to write more about mathematics & statistics, ML/deep learning, CS/software engineering, landing a job at a big tech company, getting an invitation to prestigious ML camps, etc., anything that could help you!

Also feel free to drop me a message or:

1. Connect and reach me on 💡 LinkedIn and Twitter
2. Subscribe to my 🔔 YouTube channel for AI-related content️
3. Follow me on 📚 Medium and 💻GitHub
4. Subscribe to my 📢 monthly AI newsletter and join the 👨‍👩‍👧‍👦 Discord community!
5. 📄 Website

And if you find the content I create useful consider becoming a Patreon!

Much love ❤️

This blog is the last one in my series of “deep learning update” blogs, where I was sharing my learnings as I was intensively researching various ML/deep learning subfields over the last 16 months.

Aside from sharing my learnings, the idea of these blogs is to give you the general learning-how-to-learn tips & tricks, which I’ve built up over the last many years of self-study that started way before “my AI story” was even a thing, as well as the specific advice peculiar to a particular subfield which I’m currently working on. And as they say:

To help others you only need to be one step ahead. — some{one, when, where}

The goal of these blogs is to get you started really fast so that you can build up solid foundations in the ML/deep learning subfield X, where so far this variable X = {Computer Vision (CV), NLP, Graph ML, RL}.

Now, you may ask yourself (but probably not 😂), why do I make these? Well, there are, as always, multiple reasons. The main ones I can think of are:

• I genuinely love helping others and hearing from you. All the feedback I get from you fuels me even more.
• I wished there was a blog post like this one when I started learning each of these AI subfields, that structured the knowledge of the field in a way that’s as linear as possible guiding me through the ocean 🌊 of information (or is it a graph of information? 😜 (with cycles of course!)).
• Future me will be grateful as well — in case I need to quickly overview a particular topic again.
• Writing blogs helps me further consolidate my knowledge and crystalize my thoughts — I can’t recommend it enough! Start writing your own blog.

In this one, I’ll give you some tips on how to get started with RL, but before that, as usual, let me give you a super short review of my previous content if you’re new here!

Blogs 📚

Here are some of my previous blogs 📚 you may want to check out if you’re interested in other areas of AI as well:

1. How to get started with transformers and NLP
2. How to get started with Graph ML
3. How to get started with AI and ML in 3 months

Unfortunately, I made a mistake and didn’t write these blogs during the first 6 months when I was working on Neural Style Transfer (NST), DeepDream, and Generative Adversarial Networks — GANs.

Having said that, I did briefly describe that part of my journey in the “How to get started with transformers and NLP” blog, so do check it out if that’s something of interest to you!

If you need tips on how to learn more effectively and efficiently I recommend this one: “5 Tips to Boost Your Learning”.

GitHub projects 🤓

Also, I couldn’t be happier with the projects I open-sourced over the last 16 months. You may find them useful in your learning journey, they are super easy to get started with, just read through the setup section of the README:

• NST (Gatys et al.), NST-fast (Johnson et al.), NST for videos (naive)
• DeepDream (Mordvintsev et al.)
• GANs (original, cGAN, DCGAN)
• The original transformer (Vaswani et al.)
• Graph Attention Network (Veličković et al.)
• Deep Q-Network (Mnih et al.)

All of these were written in PyTorch. I also strongly recommend creating your own deep learning projects. You’ll learn a lot by doing — trust me.

YouTube videos 🔔

I also started my YouTube channel last year, The AI Epiphany 🔔, whose main goal over the last period was to share my story along the way, as I was learning myself. Going forward its main focus will be on the latest and greatest ML papers and beyond!

[New!] The monthly AI newsletter 📢 & a Discord community 👨‍👩‍👧‍👦

If you want to keep up with latest & greatest stuff happening in the AI world subscribe to this monthly AI newsletter and join the Discord community!

Aside from Medium, YouTube, and GitHub where I have a somewhat lower frequency of upload, I started sharing my daily AI updates on Twitter and LinkedIn — so if that’s something of interest to you check it out! ❤️

Before I did this ~1.5 year long deep learning exploration, my strongest background lied in computer vision, but after 16 months of strategically exploring other areas, honestly, I don’t identify myself as a “computer vision guy” anymore that’s for sure.

My goal was to build up knowledge breadth in AI (and depth as much as possible given the time I had, BFS basically 🤓) and I think I might have just done it. I decided not to pursue a PhD and to do it my way — and I’m happy with how things rolled out, although this path is definitely not for everybody.

Note: I haven’t been sharing everything I’ve been doing there was a lot of mathematics and classical ML involved as well. I found the book “Mathematics for ML” — Deisenroth et al. extremely valuable (I’ve read it twice). I started learning ML back in 2018 — so it’s not a 16-month journey in total obviously.

If you were following along, you knew that my next interest lied in RL, and so 2-3 months later here we are!

With that out of the way let’s start with the question “WHAT is RL?”.

✅ Intro

What is Reinforcement Learning? (RL 101)

Simply put, RL is a framework that gives our models (more commonly referred to as agents in RL) the ability to make “intelligent” decisions that help them achieve their goal which is approximately this one:

Maximize the expected amount of reward. (more on this later)

It’s super common to see a diagram like this one explaining, on a high level, what the framework is all about:

You have an agent interacting with the environment. It makes some actions and the environment sends back the reward for that particular action and also the next state of the environment.

Let me make it a bit less abstract! 😅 Abstraction is a great thing to have once you’re grounded in real examples otherwise it’s just poor pedagogy.

In the game of Pong (Atari), which I’ll be using as the running example:

• The state could be a couple of last frames (images of the screen)
• The agent could be a simple CNN (like the famous DQN agent)
• The actions could be: “move the pad up”, “move the pad down”, and “don’t move at all”.
• The rewards could be +1 if you win the game else -1.

And the goal of the Pong agent would be to learn how to maximize this scalar reward which hopefully correlates with our goals — and that is for it to become a great player at the game of Pong in this case.

You can immediately see some of the problems in RL (definitely not a comprehensive list, trust me on this one):

• The sparse rewards problem (+-1 only at the end of the episode)
• The credit assignment problem (which actions led to win/loss?)
• The problem of effectively communicating our goals to the agent
• The problem of exploration vs exploitation

Let me try to break these down for you. ❤️

Sparse reward

Unlike in supervised learning where the “reward” is dense i.e. for every single data point you have the correct solution (e.g. in image classification image-class tuples), here you don’t have a dataset that tells your agent where to move the pad in each state of the game.

For that matter, you don’t have the correct label (ground truth) in any of the states! You merely observe some rewards and you devise an algorithm that will map those rewards into some pseudo-labels.

Hence RL is considered a separate category of learning alongside supervised and unsupervised (there are also self-supervised and semi-supervised paradigms and there are overlaps between these, it’s not a clear cut).

The main criterion of this categorization is the following question: “Were the humans in the loop? Did somebody had to create the annotations/labels for the ML algorithm?”.

Credit assignment

Say your Pong agent accidentally won early in the training process, which of the moves it performed led to it winning the game?

The “move up” action 40 steps ago? The “don’t move at all” action 100 steps ago?

And that’s the credit assignment problem — figuring out a strategy to basically create good pseudo-labels for each data point you collect.

Effective communication with the RL agent

Choosing the right goal may seem trivial in the case of Pong, but let me prove you wrong. 😅 (don’t hate me for this I know everybody wants to be right!)

Here is a super famous example where we implicitly stated a goal by asking the agent to maximize its game score, with an actual intention/goal being that the agent learns how to become great at winning races, but it ended up just learning a pathological behavior:

It did accomplish the goal we gave it but that wasn’t the goal we meant.

Now imagine a system way more complex than a simple game. Also, imagine people’s lives being at risk and you’ll understand the weight of this problem.

You could choose to shape the reward in order to be more explicit in your communication but that won’t lead us anywhere over the long run, plus it’s hard and time-consuming. Take a look at this reward function (the goal was for a robot to learn to pick up stuff):

As you can imagine designing this reward was hard and it doesn’t generalize.

All this reward chit-chat leads us to the next important question.

Exploration vs exploitation

Once the agent finds high reward states should it continue to visit them (exploit) or should it allow for some exploration which may lead to a discovery of even higher-reward states?

That’s one of the main dilemmas in RL, and in humans for that matter!

Say you’ve been to 10/40 restaurants in your neighborhood and you’ve found 3 of those which you really enjoy. Will you continue visiting them from now on or will you give yourself a chance to find a better restaurant? Potentially risking eating much worse food on that exploratory journey but with a promise of something truly great (like Kentucky Fried Chicken 🐥).

One of the classical methods for balancing exploration with exploitation is epsilon-greedy. Basically with an epsilon probability you pick a random action otherwise you pick an action that promises the highest reward.

Finally, you may have asked yourself how does this maximization of a sum of scalar rewards (reward ∈ R) translate to a desired intelligent behavior?

Well, basically, RL relies on this hypothesis (aka the “reward hypothesis”):

That all of what we mean by goals and purposes can be well thought of as maximization of the expected value of the cumulative sum of a received scalar signal (reward). — Richard Sutton

All of the words in the above sentence are there with a purpose, loaded with meaning, like expected and cumulative, but the goal of this blog is for it to be a meta-learning resource so I obviously won’t get into the nitty-gritty details and maths of RL but merely point you in the right direction and hopefully give you some good tips along the way.

The way we capture these cumulative sums of scalar rewards is usually via the Q-function (action-value function) or via the V-function (value-function). Q tells us the amount of reward we can expect starting from a specific state and picking a specific action whereas V tells us the amount of reward we can expect in a state. RL agents which use these are called value-based agents.

There are also policy-based agents which instead of trying to estimate the value that certain state/action will bring just try to learn which action to pick.

And there are those that combine the 2 — actor-critic methods. We could also have a model of the world or not (model-based RL vs model-free RL). The model tells you the state-transition probabilities but you don’t need to worry about it for now.

With that, you’re “good enough” to go. Don’t worry if you didn’t grasp everything in the first pass, you’ll have plenty of time.

With the WHAT question out of the way, let me give you a reason WHY you should even bother learning about RL! ❤️

Note (on how you should (probably) read the following chapters): I’d recommend not digressing too much into the resources I linked during your first read, do check out the shorter videos though. After that, I’d recommend exploring all of the resources I’ve linked in the following 2 sections just to get you hyped and to get a feeling for what RL has accomplished as well as the problems it’s facing. After that start following the resources I linked, approximately in the order I’ve linked them.

✅ RL 101

The awesome world 🌍 of RL

RL is an exciting and interesting paradigm and the only one I know of that promises the creation of an AGI. For that matter, it’s the only one which at least dares to try.

An RL agent can readily use the latest and greatest advancements from the worlds of CV, NLP, and Graph ML.

Computer vision gives it the “eyes”, the ability to represent the visual parts of its environment and to abstract the visual concepts. The breakthrough that the DQN made happened thanks to the breakthroughs in the computer vision world (more on DQN later).

NLP gives it the power to construct good representations of language concepts. NLP (transformers, etc.) enabled AlphaStar (more on it later) to better “understand” the game of StarCraft II.

Graph ML, e.g., allows it to understand the interconnectedness of visual and language concepts which I’ve mentioned above. They can be modeled as nodes in a heterogeneous graph.

Over the past decade, we had amazing accomplishments, previously unimaginable, happening in precisely the field of RL. Let me tell you about a few of those!

Note: Deep RL = RL + deep neural networks

Deep RL — the early days 🌌⭐

It all started with Deep Q-Network back in 2013. Before it, RL was mostly using the tabular setting for the state space (the collection of all states in the environment), which couldn’t scale to applications we actually care about since you need a slot of memory for every state in your environment.

DQN created a “new” field that we call deep RL. Deep stems from the fact that these RL algorithms are using deep neural networks as function approximations. With DNNs we only need a finite amount of parameters to capture huge state spaces (they are parametric models).

DQN managed to play ~29 Atari games at a human or super-human level. And all that with the same architecture, the same minimal setup, and directly from the raw images — something completely unimaginable in the world of RL prior to that.

Basically, the agent only receives an input frame (image), the number of actions for a particular game (which defines the number of outputs of a CNN) and the rest (the learning) is done via game specific scalar rewards.

You throw this very same setup in any one of the Atari games and it learns to play them (most of them at least — some of them like Montezuma’s revenge were a tough challenge up until recently).

A similar trend in the initial deep RL era was also made in robotics showing that we can train robots end-to-end (e2e) to perform certain dexterous tasks, like screw a cap onto a bottle, placing a coat hanger on a rack, etc., using raw vision and robot’s state (robot’s joint angles, etc.) as an input!

CNNs were powering this breakthrough as well.

Next came the almighty AlphaGo. The first program that won against the best human player in the world, Lee Sedol, in the game of Go. It’s also the only algorithm that came even close to doing that back at the time.

AlphaGo lineage 🐒 -> 👶

It was a big deal. Most AI experts didn’t expect that to happen for at least 10 more years. It was considered the “holy grail” of AI, solving Go.

The huge search space of the game (~250¹⁵⁰) seemed unsolvable. Chess, the ex-drosophila of AI, is funny compared to it (35⁸⁰) and yet decades of research went into cracking chess.

Yet in 2016, AlphaGo, trained with a combination of supervised learning, RL (self-play) and Monte Carlo Search Tree (MCST) did it.

This feat was such a big deal that millions of people were following the event. DeepMind even made a movie on it. It’s got 24 million views at the time I’m writing this (and no it’s not even a Netflix series 😂). I strongly recommend it, it’s a great story even if you’re not into AI per se:

Training AlphaGo took a massive amount of compute, more than a month on 50 GPUs to train the DNNs and I’m not even counting in the hyperparam search that led to this final model!

After AlphaGo even better agents appeared, the so-called AlphaGo lineage of agents: AlphaGo Fan/Lee/Master (incremental improvements), AlphaGo Zero, AlphaZero and finally MuZero.

AlphaGo Zero ditched the supervised learning part that relied on human play datasets and managed to beat the previous best AlphaGo 100–0! Throughout its training it reinvented the collective knowledge which humans had to collect over the past 2000+ years. And all of that through self-play RL.

AlphaZero further generalized the AlphaGo Zero agent and learned to additionally play Chess and Shogi (the so-called “Japanese Chess”) with pretty much the same model.

Finally, MuZero appeared. It had to additionally learn the rules of the game (none were given to it!) and it additionally could play all of the Atari games (on top of Go, Chess and Shogi). A new step towards more general agents.

All is not rosy though — the BIG problem with all of these is the fact they have to be retrained from scratch for every new game they play! There is no generalization that happens across the games.

But I’ll leave the problems RL faces for the next section — once the board games became “too easy” the community turned their eyes onto tougher challenges, like multiplayer real-time strategy games such as Dota 2 and StarCraft II.

The story of OpenAI Five and AlphaStar 🎮

The really hard part about these games is that:

• They are partially observable compared to previous board games we looked at (e.g. the fog of war doesn’t allow you to see the parts where your opponent is building his/her infra unless you have your troops there)
• The action and observation spaces are huge! The breadth of the decision tree is somewhere around 8k-80k in Dota 2, contrast that with Go’s ~250 or Chess’s 37 legal moves on average! Funny, right?
• Also, the time horizon of the game is very long. Humans play these games in ~45-minute sessions which means a huge number of steps for an RL agent. E.g. the decision tree’s depth =~20k in Dota 2, contrast that with Go‘s 150 moves.

Again, experts thought we’ll need lots of time to solve these. Some were more optimistic after Go was taken down. But soon enough we had bots winning at professional eSports games!

OpenAI started working on their Dota 2 bot called OpenAI Five back in 2016. A couple of years later in 2019, they have beaten the best human teams in the game of Dota 2.

They imposed certain restrictions on the game — like the pool of heroes humans could pick from was smaller than normally, certain parts of the bot logic were scripted instead of learned, but all in all, given more compute none of these change the fact that bots are truly better than humans in these specific constrained domains we call e-games.

You could also see a clear curriculum — they first started beating humans in the 1 vs 1 setting and only later tackled the 5 vs 5 setup. Which is smart, if you can’t win in the 1 vs 1 setting why even bother with 5 vs 5.

If you’re interested you can watch the finals here (I personally skimmed through the interesting parts mostly on 2x playback speed):

The main RL algorithm behind this was PPO which was invented a couple of years before. So it wasn’t that much the algorithmic innovation that got us here but the sheer compute and engineering, RL at scale!

An interesting parameter they had was “team spirit”. It basically was annealed from 0 to 1 during the training and it controlled the agent’s selfishness (0 — maximally selfish, only my score matters, 1 — only the score of team matters even if I die).

Compared to AlphaGo lineage what I found interesting here is the generalization capability — the bots were able to play against humans (compete) as well as to play with humans (collaborate) in a mixed-team setup even though they’ve never seen that play-style during their long self-play careers.

In this LinkedIn post, I’ve linked additional cool resources if you want to learn more about OpenAI Five. Now onto AlphaStar!

A similar story happened with StarCraft II and DeepMind’s AlphaStar bot. They trained AlphaStar using various RL and game-theory techniques — bunch of compute and engineering later it was beating the best humans at StarCraft II.

Arguably AlphaStar team tried really hard to make the game absolutely fair. The APM (action-per-minute) of their bot, the way it could see the world (initially it could see the complete map but later it could only see parts of the map — same as humans), etc. all mimicked the way humans play the game.

Again, if you’re interested here is a cool demonstration of the finals against one of the best StarCraft II players in the world (I watched the whole thing some time back and enjoyed it):

The main idea of the AlphaStar paper is the concept of the league of players, because the pure self-play, that worked so well with Go, lead to a circular progress (read no progress) whereby new agents can easily be exploited by simpler agents.

The league consists of:

• Main agents — these are the ones we ultimately care about
• Main exploiters — their goal is to find the weaknesses of main agents
• League exploiters — their goal is to find systematic weaknesses

In the interplay of main agents with themselves (SP) with exploiters as well as with older agents (so as to minimize forgetting), in the so-called fictitious self-play (FSP) they grow very robust.

Check out this LinkedIn post for more details.

Personally, one thing that I find weird is that they haven’t tried to beat the best of the best player in the world — like in Go. I can only assume why that was, and I guess it has to do with resources — even DeepMind occasionally gets to taste the reality like we, mere mortals! 😅

Shifting the focus from games onto robotics here are some amazing things that happened in robotics — RL intersection!

Robots, robots, robots 🤖

Learning Dexterous In-Hand Manipulation by OpenAI (codenamed Dactyl): This is definitely one of the projects that sparked my love for RL!

The goal of the project was to orient a cube into a specified target orientation using a robotic hand (Shadow Dexterous Hand) + RL.

Now the cool thing about it is that the RL agent was trained purely in a simulation, it was deployed to a real-world and it actually worked!

Check out this short demonstration:

The main algorithmic idea behind it was to use heavy randomization during the simulation so that the real-world seems like yet another incarnation of a simulation (wait, is it not? laughs in Elon Musk).

In the follow-up work they managed to improve it so that it can solve the Rubik’s cube! One of my favorite papers ever, TBH. It was such a multi-faceted project — a combination of mechanical engineering, electronics, robotics, algorithms, computer vision, RL, you name it!

Now, be careful here, the feat is not solving the Rubik’s cube per see (they use an off-the-shelf solver for that), but dexterity/controlling the hand when you know on a high-level what your next steps are (e.g. rotate the upper face 90 degrees clockwise).

Check out this awesome short demonstration:

They improve upon Dactyl by introducing an automatic domain randomization algorithm that automatically changes the ranges of randomization params in the simulation — like gravity.

It starts being calibrated to Earth (~9.81 m/s²) and depending on how much the policy is struggling the range will, eventually, expand to say 4–12 m/s². Thus this provides the agent with an implicit curriculum learning as the envs become harder and harder.

The agent had 13 000 years of cumulative experience! (counting in all the compute needed for the hyperparam search, training, etc. and keeping in mind that 1 step of simulation corresponds to 80 ms).

The part that blew my mind about this paper was the emergent meta-learning behavior. The hand learned to encode the params of the environment it is in, directly into LSTM’s hidden state.

They tested this by deleting the state at one point of the execution and noticed how the hand takes longer to perform the next few steps and later catches up to its previous speed.

They also showed that they can infer env params (like the size of the cube, gravity, cube mass, etc.) through LSTM’s hidden space — again showing that the hand learns to encode the parameters of the env it is in!

Finally, check out this short blog on how RL was used at Google to reduce the power consumption by a massive amount (~40%). The paper never came out and I’m decently sure that they needed to add bunch of hacks to make it work — so it’s not just RL that made this happen, keep that in mind.

You may have noticed that most of these accomplishments happened in the domain of super constrained problems like constrained robotics and board/video games, and you may have asked yourself why is that?

Well, that nicely brings us to the next section.

✅ Cool RL stuff (the WHY)

RL is not only roses (RL ∉ {🌹, 🌸, 🌻})

There are still many problems that are making it hard for RL to be useful outside of very controlled and/or simulated environments.

For start, RL is sample inefficient, a fancy way to say that RL requires a LOT of data & compute for it to work. The original DQN agent required ~200M frames for every single Atari game!

Check out this blog by OpenAI to get a feeling for the compute needed for some of the agents I mentioned in the previous section:

AI and Compute

Pretty much only big tech companies can afford this much compute.

RL sucks at generalization. E.g. even MuZero has to be retrained from scratch for every single game in order for it to work. The fact that a MuZero agent can play Breakout on a super-human level doesn’t help it to learn to play Pong — which is weird, from the human perspective at least.

Somewhat related is this quote:

Deep RL is popular because it’s the only area in ML where it’s socially acceptable to train on the test set. — Jacob Andreas

Made me laugh the first time I read it. 😂 But it tells a story. We’re testing our agents on the very same environments in which we’re training them. How can we expect generalization to emerge? The lack of generalization in somewhat baked-in into the current deep RL paradigm.

Also, as I mentioned reward function design is hard, agents can learn pathological behaviors that don’t match the goal we envisioned, and even when they have a reward function that matches our goal they’ll often get trapped in a poor local optima.

Other than that RL agents are not very robust to changes in hyperparams, i.e. they are utterly unstable and thus the papers are hard to reproduce. Even a random seed can make a difference between your agent working and not!

Sometimes, just changing the random seed can make a difference between the agent learning how to accomplish a goal and it failing miserably.

Rather than trying to list everything that could go wrong I strongly recommend this blog by Alex Irpan who works at Google Brain on the Robotics team, he’s done the work for me (thanks Alex!):

Deep Reinforcement Learning Doesn't Work Yet

Read it carefully, and let it sink in.

All those problems make it hard to reproduce RL algorithms. The “But it works on my machine” force is especially strong in RL.

Lastly, here is a video by Joelle Pineau from FAIR/McGill additionally explaining some of the main problems that RL is facing:

I had a first-hand experience with RL being hard to reproduce— I’m still debugging my implementation of the DQN agent, I’ll update the blog as soon as I get it working!

Note: TBH I wasn’t invested in it 100% because of the lack of time. But it definitely is harder than your average DL project, that much I can claim.

Nice, you got the yin and yang perspective of RL!

Having hopefully answered the question WHY, let me give you the tips on HOW you can get started with RL.

✅ Problems with RL

Getting started with RL 🚦

By listing all the stuff in the previous sections you’ve already started getting familiar with RL. Here I’ll try to structure it in a way that’s more suitable for building up solid foundations — I’m going to give you a strategy that worked for me.

I’m a huge fan of the top-down approach. You should start with high-level resources such as high-level YT videos and blogs and slowly work your way all the way down to reading papers.

Related to that I think it’s super useful to get exposed to the terminology of the field early on, so that you can let it sink in. Many confusions happen just because we don’t know the actual language, the syntax.

Tips out of the way, let’s start!

Out of all the YT videos I’ve watched I’d start with these ones:

• Intro to Deep RL and Reinforcement Learning from MIT

They are very basic and nicely structured, next up I’d watch this one:

Nuts and Bolts of deep RL by John Schulman from OpenAI will tell you about the common problems and strategies you could use to debug your RL algorithms.

In this stage of your learning process don’t beat yourself down for not understanding everything. Feel free to stop the video, google some concept and get right back into it. You should use the benefits modern tools provide us with.

Never be passive while learning. Open up a OneNote (or whatever your favourite note app is) on the other screen and write down concepts you found important. I also found it useful to write down timestamps and make a comment, others and future me will be grateful.

Also if some parts of the video are already familiar, hit that playback speed button — I often watch content on 1.5-2x speed (especially if I’m already familiar with the topic).

The bottom line is — you shouldn’t be watching YouTube videos like you’re watching your favourite Netflix series. You’re trying to learn, remember?

Lastly, I found this short video by Pieter Abbeel interesting as it gave hints of what it takes to make RL actually work in the real world.

After the videos, go through some high-level blogs:

• Pong from Pixels — Karpathy, amazing as always. You’ll learn a lot from it.
• Spinning Up — getting started as an RL researcher, you’ll find some nice ideas in here as well!
• This (get exposed to terminology), this (survey of RL) and this (build a simple snake app, nice ablations) to get you started reading about RL.

Then go through some simple code. I strongly recommend that you get exposed to RL code early on in your learning process. Code can sometimes be the best explanation of the concept you’re trying to understand:

• Vanilla Policy Gradients gist by Karpathy, and a great accompanying video, also this blog as well as this one can further help you understand Karpathy’s Vanilla PG implementation.
• RL adventure and RL adventure 2 — I’ve consulted the code snippets in these repos after reading a paper whenever there was an implementation of that paper. The code is often times overly simplistic, there are some bugs, but it paints an overall picture!

Finally take 4–7 days to go through David Silver’s course:

After that you’ll have heard most of the fundamental RL algorithms, problems and terminology, and David is a great lecturer so I believe you’ll enjoy it.

A word of advice here, I’ve said 4–7 days since I don’t recommend you binge-watch all of these in a day or two (even though you could). You’ll simply forget everything, you have to let it sink in.

Same goes with anything, if you hit the gym and do 4 trainings in a day that won’t make you stronger that will probably get you injured. Doing 4 trainings over a week on the other hand can bring you some progress.

Finally you can watch this Deep RL lecture by John Schulman (again)!

Note: If you really know that RL is something you want to do for the years to come I’d strongly recommend the following. Instead of jumping into reading papers in the next section, I’d invest a month of my time and I’d go through the famous Sutton’s and Barto’s book with a very intensive pace. IMHO, that’s a much better strategy than reading it over a prolonged period of time.

Sutton is one of the most influential figures in RL (if not THE most). They focus on the tabular RL case and will teach you all the necessary mathematics and formalism you need in order to deeply understand RL (no pun intended).

As an accompanying resource this repo has solutions to all of the problems in the book. Do use it while reading the book — again it’s all about being active while learning if you want to be more productive.

Having said that, I personally haven’t read the book yet (I did go through all of the other resources I mention) and I didn’t notice any bigger problems reading through the papers in the next section. There were some papers which I had problems understanding but that’s normal!

✅ A high-mid level understanding of RL

Let’s go deeper — reading papers

Once you have the high-level overview of the field and the terminology in place it’s time to go deeper!

Paper collection strategy 🔍

I’ll tell you the approach I took researching RL!

I roughly found the biggest names in the field and I’ve collected their most cited papers. I then sorted them according to the time they were published. Finally, I roughly split those into 2 batches again according to the timestamp.

Some of the main names I used were:

• Volodymyr Mnih and David Silver from DeepMind
• John Schulman from OpenAI
• Sergey Levine and Pieter Abbeel (now at Covariant) from Berkley

Now this is far from perfect and if left like this it would bias me towards 2, 3 research groups (but frankly the power law is real). But, along the way as I was reading through the papers I organically found some other interesting resources and I’ve added them to my list. That was the strategy I was using.

Also, I was lucky enough to be in an email correspondence with Charles Blundell and Petar Veličković from DeepMind 🧠. Charles shook up the list I compiled with the above strategy and proposed some other good papers.

Finally, here is a glimpse of how one of my directories with papers looks like:

I split the papers I read into the following groups:

• The ones I still haven’t read (currently empty)
• The ones I’ve read and understood
• The ones I’ve read and had problems understanding

• The ones I’ve skimmed (only R2D2 paper)
• And finally the miscelaneous read — those are the papers I read along the way even though they are not necessarily related to RL. These either help me catch up on the prerequisite knowledge or they help me keep up my energy levels high! 🍎

Also as you can notice the first group contains papers like “Concrete Problems in AI Safety” or Evolutionary Strategies paper which arguably belong to this miscelaneous group but yeah it’s not a deal breaker.

You can also see that I’ve organized papers by the timestamp. I guess there are also platforms which could do all of this for me but this just does the job.

Note: don’t worry about the specific papers I’ve listed in there, in the next section I’ll list everything that you really need.

Hopefully this glimpse into my mind helps you a bit. And now!

RL papers list 📜📜📜

Let me try to introduce some structure into a pile of RL papers that’s out there.

Note: I’ll label the papers I had hard time reading with (#hard-read). This doesn’t necessarily mean it’s hard — it was so for me given the context I had at the time.

You should probably start by reading the DQN paper (Mnih et al.) which was a huge breakthrough at the time. I read both the arxiv (13') and Nature (15') versions, and I’ve covered DQN in this video:

Note: If you want to get a feeling for how RL is used in other subfields, like in CV, check out Recurrent Models of Visual Attention which has a nice example of how RL could be used in vision to emulate the way humans parse the visual space around them (multiple fixations interspersed by saccades and information integration throughout). It uses the simple REINFORCE (policy gradient) algo.

After the DQN was published it literally created an ImageNet-like moment for the RL world. A lot of research happened trying to improve upon DQN. Here are some of those papers:

• Deep Recurrent Q-Learning for Partially Observable MDPs — instead of stacking frames why not integrate the information over single frames using some sort of memory, LSTM in the case of this paper. They showed this is a viable alternative and that the model gains better robustness if the quality of observations (game frames) changes during the game.

The following papers led to the famous rainbow DQN paper:

• Dueling Network Architectures for Deep Reinforcement Learning — basic innovation lied in its dual-stream architecture, where one stream was estimating the value-function, the other one normalized advantage function and the fusion gave the action-value function.
• Prioritized Experience Replay — the main idea was to improve upon the uniform sampling of experience used in DQN’s replay buffer. Instead absolute TD-error was used as a proxy to choose pieces of experience which are the most valuable in the sense that they’ll accelerate learning.
• Deep Reinforcement Learning with Double Q-learning — deals with overestimation of Q-values by the regular DQN by separating the selection and evaluation stages of DQN (the idea was actually proposed in a tabular setting years before).
• Noisy Networks for Exploration — instead of using epsilon-greedy like the original DQN they baked the exploration noise directly into DNN layers.
• A Distributional Perspective on Reinforcement Learning — instead of finding the expectation of the reward (point estimate) this paper proposes to keep the whole value distribution during the training. (#hard-read)

And finally the rainbow DQN, “Rainbow: Combining Improvements in Deep Reinforcement Learning”, which combined the above research in order to engineer the best-performing DQN-like model back in 2017.

Learning tip: some of these papers use hardcore math and you may lack the intuition and fail to understand the mathematical formalism. Try and grasp the main idea, the semantics behind the paper if you’re having problems with the formalism. Watching YouTube videos before you start reading a paper is a strategy I sometimes find useful — although for most of these, unfortunately, it’s hard to find a (good) video.

Also if you feel you’re getting tired of the theory presented in a super formal way you need to find a way to bring your mental energy up! 🍎

Try reading up some interesting paper out of chronological order or watch an interesting video. I’d suggest you read this paper by Joshua Tennenbaum:

• Building Machines That Learn and Think Like People

If you’re passionate about the whole AGI endeavor it’ll provide you with some refreshment! Or go ahead and read something lightweight so that you think you’re smart (😂) like the OpenAI gym paper. Power naps also help.

After that I’d recommend you read these papers from 2016:

• Asynchronous Methods for Deep Reinforcement Learning (A3C) — seminal paper, introduced the multi-threaded advantage actor-critic setup. The parallel actors stabilize the training without any use of replay buffer. In RL scale plays an important role and so detaching from the single-threaded setup of DQN was very important.
• Continuous Control with Deep RL (DDPG) — they generalize the success of DQN to the continuous domain such as robotics tasks using deterministic policy gradient + DNNs stabilized via similar ideas like DQN. It’s an actor-critic algorithm as well.
• Benchmarking Deep RL for Continuous Control — (introduces RLLib and gives a nice overview of the common tasks and overview and comparison of RL agents like TRPO, DDPG).

I’d wrap up this part with “RL with Unsupervised Auxiliary Tasks” (UNREAL agent) paper. The main idea is to cope with the sparse reward problem that RL agents are facing. So aside from maximizing the cumulative reward (extrinsic) they introduce pseudo-rewards (intrinsic) given by unsupervised auxiliary tasks like learning to maximize the pixel intensity over the input image (as a proxy to an important event in the env). It drastically improved the sample efficiency of A3C agent (~10x less data).

Now, If you’re like me one of the papers you were excited to understand is AlphaGo:

• Mastering the Game of Go with DNNs and Tree Search

It is one of those projects which couldn’t have been accomplished using any other method, where RL really shined to its fullest!

I’ve covered it in this video as well:

Yes, I love this picture with those glorious mustaches. Now let’s continue.

One of the main components of AlphaGo is Monte Carlo Tree Search or MCTS for short. If you’re not familiar with it I found this video was great at explaining it.

While the AlphaGo knowledge is still fresh in your mind you can go ahead and read the whole saga (for my CS nerds: before any context switching occurs!):

• Mastering the Game of Go without Human Knowledge (AlphaGo Zero )— a step towards more general RL agents, it ditched the SL part of training that relied on human play datasets and focused purely on the self-play.
• Mastering Chess and Shogi by Self-Play with a General RL Algorithm (AlphaZero )— further generalized the algorithm to play Chess and Shogi.

Here is a video covering both papers (since they’re very similar):

And finally, MuZero which additionally learned to play the Atari games as well as the rules of the game!

• Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model

The part with how the hell can it learn the rules of the game used to bug me, I’ve explained the paper as well as that specific part in this video:

There are many additional cool resources out there on AlphaGo lineage of agents. I’d recommend reading all of DeepMind’s blogs you can find like the one on MuZero and this one on AlphaGo Zero.

After getting back from this tour of the AlphaGo world you can now return back to reality and start reading RL papers for mere mortals. Here are some additional papers you should read:

• “Target-driven Visual Navigation in Indoor Scenes using Deep RL” — the cool thing about this paper is that they try to tackle the generalization problems present in RL. Instead of baking the goal directly into DNN’s parameters they can flexible change the goal — the target location where they want the robot to go. They test the agent in progressively harder generalization tasks. Kudos.
• “Emergence of Locomotion Behaviours in Rich Environments” (aka DeepMind parkour paper) —instead of reward shaping they show that the agents can learn interesting behavior on their own when given progressively harder environments (implicit curriculum learning). Definitely check out this paper and the video below:

• Hindsight Experience Replay (HER) — similarly to UNREAL they tackled the sparse reward problem. The main idea is the following, say you want your robot to move to a position (x1, y1) but it instead moved to (x2, y2). Instead of dropping this trajectory you can do the following, imagine your goal was precisely (x2, y2) that means that whatever robot did was correct we just need to change the goal post hoc! Hence the “hindsight” part.
• Curiosity-driven Exploration by Self-supervised Prediction — proposes the Intrinsic Curiosity Module which acts as a source of intrinsic reward (extrinsic reward comes from the environment like in Atari the score you get after every move, intrinsic comes from well, within). Check out this post for more details.
• Survey of RL — you should occasionally read these to get the breadth and organic growth of your paper list as you’re learning. Just be careful not to inflate your list just for the sake of it. (Charles recommended this one)

Finally before you get to the arguably most popular and interesting RL projects (aside from AlphaGo) you’ll need to read 3 more papers:

• Proximal Policy Optimization (PPO, OpenAI) — very impactful paper used in many RL papers. Even DeepMind parkour paper used it. OpenAI used it for OpenAI Five bot and robotic hand project.
• Domain randomization paper from OpenAI — showed how smart randomization in a simulation can help models generalize fairly well to the real-world.
• IMPALA (DeepMind) — used in e.g. AlphaStar. Allows for amazing scalability, more compute = more results, remember? 😅

Skill unlocked. Given infinite compute you could now solve anything in this world! That’s just an implementation detail, it shouldn’t be a deal-breaker…

Right?

Now, you could first research the AlphaStar saga by reading these 2 papers:

• StarCraft II: A New Challenge for RL — describes the API and some details you’ll find useful reading the AlphaStar paper I suggest you at least skim it.
• Grandmaster level in StarCraft II using multi-agent RL (AlphaStar)

After you’ve read through the AlphaGo and AlphaStar papers I’d also recommend this video by David Silver.

Next up you could investigate OpenAI’s robotic hand saga, start with the official blogs Dactyl and Rubik. I also covered both of these in this video:

as well as this one by Marcin Andrychowicz, one of the co-authors from OpenAI.

and only then head over and read the actual papers (remember? top-down):

• Learning Dexterous In-Hand Manipulation (Dactyl)
• Solving Rubik’s Cube with a Robot Hand

Last but not least OpenAI Five!

• Dota 2 with Large Scale Deep Reinforcement Learning

I wrapped up my RL exploration with Agent 57 lineage. The first RL agent to crack (go above the human baseline) all of the 57 Atari games, including Pitfall, Solaris, Skiing and Montezuma’s revenge:

Agent57: Outperforming the Human Atari Benchmark

Search for the blogs and read these papers (Agent 57 builds right on top of R2D2 and NGU so make sure to check those out):

• Recurrent Experience Replay in Distributed RL (R2D2)
• Never Give Up: Learning Directed Exploration Strategies (NGU)
• Agent57: Outperforming the Atari Human Benchmark

Charles Blundell, the co-author, covered these beautifully in this video:

Also, something from 2021, the recently published Dreamer v2 is the first single-threaded, model-based agent that outperformed the Rainbow DQN that took years of incremental research to craft:

Mastering Atari with Discrete World Models

Check out the paper here:

• Mastering Atari with Discrete World Models (Dreamer v2)

Yannic Kilcher also made a nice video on it.

Wrapping up, I’ve skipped some papers which are more theoretical, harder to read, and not necessarily needed in order to understand the resources I’ve linked above, but are still a good read!

• Addressing Function Approximation Error in Actor-Critic Methods (TD3) tackled the overestimation bias in TD methods that was present in the continuous action spaces with actor-critics (similar to what double DQN paper did for the discrete action spaces).
• Unifying Count-Based Exploration and Intrinsic Motivation — another intrinsic motivation method, achieved great results with Montezuma’s Revenge at the time.
• High-Dimensional Continuous Control Using Generalized Advantage Estimation — introduces GAE a novel RL method that pulled ideas from TD(lambda) and TRPO to cope with data efficiency and stability.
• Trust Region Policy Optimization (TRPO)— PPO simplified it and since then I haven’t seen TRPO mentioned a lot (I may be wrong!). This blog by OpenAI helped me better understand it.
• End-to-End Training of Deep Visuomotor Policies and Guided Policy Search — lots of robotics/control theory inside that’s what makes it a harder read I have little experience with those subfields.
• Deterministic PG — devised a method that outperforms its stochastic counterparts in high-dimensional, continuous action spaces. And it’s much more efficient as well.

✅ RL papers!

And now, as always, I’m a big fan of actually coding something up and not just focusing on the theory and papers. The devil is in the details as they say!

Here are some learnings from the DQN project I did!

Deep Q-Network project from scratch

I didn’t have that much time lately to focus on the project but nonetheless, I can say that debugging and creating RL projects is harder than your average ML project — for the same reasons mentioned in the “RL is not only roses” section.

Nonetheless I learned a lot! And I’ll keep doing so until the project is done! ❤️

You can check out the project here:

gordicaleksa/pytorch-learn-reinforcement-learning

Here are some of the problems off the top of my head which I’ve encountered with OpenAI’s gym library:

• Had to downgrade to 0.17.3 because Monitor had a video recording bug
• Installing atari on Windows is not smooth but there is a simple workaround.
• You sometimes need to dig into issues and source code, e.g. to figure out those cryptic env naming conventions like “PongNoFrameskip-v4”.
• Gym random seeds do not work as expected.
• Gym’s atari preprocessing has no frame-stacking, it’s simply a max-pooling over the last 2 observations.

And there are some problems with all of the DQN implementations that I’ve looked at in general:

• Toy replay buffer — saving (1M, 4, 84, 84) bytes instead of (1M, 1, 84, 84 (even stable baselines 3 is having problems with replay buffer! — my issue)
• Most of the repos don’t have a readme stating the achieved results. Have they managed to get results comparable to the published results?

Debugging this thing is hard and slow. It takes me ~5 days to train the DQN on my machine (although I may have a bug) and so I understand why regular people haven’t been posting their results — they probably haven’t achieved the published results in the first place.

Check out this LinkedIn post for more rant! ❤️

Another great blog with which I can now completely relate to is this one by Amid Fish on reproducing deep RL projects.

Note: If you want to understand Atari preprocessing a bit better check out this blog.

I’ll update this section of the blog as soon as I get it working.

✅ DQN project

My next steps

If you stuck with me until here, congrats! ❤️

Hopefully this blog will inspire you to take your first step into the world of reinforcement learning! Or in general, I hope it’ll motivate you to keep self improving and learning every single day!

One thing you’ll notice after studying RL for a while is how many related subfields there are like (similarly to Graph ML):

• AI safety — imagine this example: A robot knocks over a vase because it learned that it can move a box faster to another location by doing that. Can we achieve the goal without specifying EVERYTHING that the robot SHOULDN’T do? Those are the types of problems we have to think about. You could start exploring this field by reading the Concrete Problems in AI Safety paper from OpenAI. Check out my summary.
• Robotics and control theory in particular
• Inverse RL and Imitation Learning
• NAS (neural architecture search) — the original NAS paper used RL. Since then other more compute-efficient methods have been advised but RL still has its place in the NAS subfield. I did a “detour” into the NAS field and I enjoyed it, here are some seminal papers you should read: NASNet, MnasNet, Efficient Net v1 and v2 (just recently published!), NAS survey paper, MIT OFA, Mnet v1, v2, v3, DARTS and this video and this high-level overview. Also, I’ve recently covered Efficient v2 in this video:

• Evolutionary algorithms —An alternative to RL, a type of black-box optimization method, i.e. you don’t need to know the exact structure of your loss function (like you need to when you do backprop). Perturbing the weights, evaluating them in the environment and forming a weighted average using those weights and rewards leads to better agents. Evolution strategies paper is a good starting point and my summary. This blog as well+ code snippets should be enough to get you started.

Also if you’re super curious and you want to understand more about this great moment from the game history skim the DeepBlue paper. I’ve made a summary in this LinkedIn post as well.

And as I said in my previous blog, I’ll tell you that exploring can feel overwhelming at times. You think you’re starting to grasp everything and all of a sudden a whole new subfield emerges. And I’ve been doing this for a while after I’ve finished my EE studies. So take your time and try to enjoy the process!

As for my next steps, I now feel I’m ready to start keeping up with SOTA across different AI subfields. That means that I’ll completely focus on building the AI community of people passionate about AI and tech in general.

I’ll:

• Start keeping up with the newest stuff in AI on a daily basis. My plan is to invest 3x15 mins daily to skim Twitter, LinkedIn and Facebook groups for AI news, which I almost never have the time to do. Consequently I’ll read the SOTA papers as they come out and I’ll cover them more regularly on my YT channel.
• Make regular (hopefully daily) paper/AI news overviews on Twitter and LinkedIn so if that’s something of interest to you feel free to follow me/connect with me there!
• Periodically (every 2–3 months) develop my own interesting and creative AI projects instead of implementing other people’s work.
• Occasionally write on Medium, especially when I have something super interesting to tell you or something long-form.

There is one more thing that I’m currently working on, which I can’t yet share, but I’m super, super excited about it. Hopefully, I’ll soon be able to share it! If that happens I can only tell you it’ll be a dream come true for me pretty much.

Hopefully, this inspires you to start creating your own plans and to keep on (deep) learning!

If there is something you would like me to write about — write it down in the comment section or send me a DM, I’d be glad to write more about maths, statistics, ML & deep learning, computer science, how to land a job at a big tech company, how to prepare for an ML summer camp, anything that could help you.

Also feel free to drop me a message or:

1. Connect and reach me on 💡 LinkedIn and Twitter
2. Subscribe to my 🔔 YouTube channel for AI-related content️
3. Follow me on 📚 Medium and 💻GitHub
4. Subscribe to my 📢 monthly AI newsletter and join the 👨‍👩‍👧‍👦 Discord community!

And if you find the content I create useful consider becoming a Patreon!

Much love ❤️

This blog is a part of my “deep learning update” series and I want to open it up with a question: What have I learned about Graph ML in 2+ months?

Nothing? If that was your first thought, no worries, it’s probably true. 😅

(Bad) jokes aside, my “relative knowledge” (the knowledge I “possess” vs the knowledge I’m aware of) is asymptotically approaching 0 whereas my “absolute knowledge” did grow by quite a lot over the last 2 months.

But you never see all the amazing things you’ve learned, you see that 0. You have that weird feeling that 0 is staring at you. At least I have it, sometimes.

It’s always like that. You approach a new field and you have this delusional feeling that you’ll only need to learn A (e.g. word2vec/node2vec), B (e.g. GPT/GCN), and C (e.g. BERT/GAT) and you’ll master the field.

It’s finite. You can complete it. It’s like a game.

But, as you keep on learning and exploring you discover a whole new world of ideas and terminology, of people and history. And that feeling is beautiful and frightening at the same time, and I assume it’s the reason many people give up.

Research and learning can get intimidating because it makes you feel so small. The best advice I can give you is — never give up. I mean it. I’ll delete your GitHub (or Fortnite?) account if you do. I swear.

Nice, after giving you a micro-existential crisis and then bringing you up ❤️ , let us continue.

If you were following along my journey you knew that I’m going to tackle Graph ML next. In my last blog update on transformers, 2 months ago, I mentioned that my next field of exploration is Graph ML, so here we are.

But first, what is Graph ML?

Simply put Graph ML is a branch of machine learning that deals with graph data. Graphs consist of nodes, that may have feature vectors associated with them, and edges, which again may or may not have feature vectors attached.

Graphs come in different kinds, we can have undirected and directed graphs, multi and hypergraphs, graphs with or without self-edges. There is a whole field of mathematics aptly named graph theory that deals with graphs.

And you don’t need to know all of the above definitions for now.

Graph data is abundant all around us. You name it! Social networks are obviously graphs, molecules can be represented as graphs (atoms are nodes and bonds are edges), any kind of interaction in physics can be modeled as a graph, and so forth!

So what can we do with graphs?

Common tasks include graph classification/regression and node/edge classification/regression. Talking about abstraction, right? 😅

Let me make one of these less abstract. Maybe the nodes in your graphs are tweets and you want to regress the probability that a certain tweet is spreading fake news. Your model would associate a number between 0 and 1 to that node and that would be an example of a node regression task.

But how do we do that?

Usually, we have neural networks that deal with graphs and we call these Graph Neural Networks (I’m simplifying it a little bit for now). It’s ok not to understand how exactly these work for the time being, just stick with me.

Now let me tell you why you should learn more about this field. ❤️🌍

Personal update: just created a monthly AI newsletter and a Discord community! Subscribe/join those to keep in touch with the latest & greatest AI news!

The awesome world of Graph ML

There are so many exciting applications. I mean it. I’ve never seen a field with this many exciting applications!

All throughout science, you’ll find GNNs (Graph Neural Networks) being deployed as we speak. Anything you can model as a graph (and that’s pretty much everything! 😅) and you can throw a GNN at it. 💻

Massive-scale recommender systems, particle physics, computational pharmacology/chemistry/biology, traffic prediction, fake news detection, and the list goes on and on.

You still don’t believe me? That’s great. I don’t want you to take my word for granted, you’re a critical thinker let me prove it to you!

Uber Eats uses GNNs to recommend you food and restaurants.

Pinterest deployed PinSage GNN at least 2 years ago to their platform and it’s now curating your homefeed and it’s suggesting you the most relevant pins.

Twitter uses GNNs to detect (and hopefully filter out) fake news.

This is possible thanks to the fact that fake and real news spread differently on social media! We can thus exploit the propagation pattern of fake news over time as a valuable signal.

On the above image, you can notice an interesting pattern that likely-minded people tend to cluster together — nothing new there 😎. When the graph has that property (connected nodes share a similar label) we call it homophilic (from Ancient Greek: “homos”-same, “philie”- love/friendship).

Let’s continue on into fundamental science!

We’re detecting neutrino particles on the South Pole at the IceCube lab, and we’re classifying them using GNNs!

IceCube is a 1 km3 neutrino observatory located at the South Pole and its primary purpose is to look for high-energy (above 100 gigaelectronvolts (GeV)) neutrinos!

It basically consists of a hexagonal grid of light-sensitive sensors that capture the Cherenkov light emerging from the interactions between charged particles with the ice.

You can nicely model this sensor structure as a graph! Each light sensor is a vertex and it has associated with it a 6-dimensional feature vector (3 XYZ coordinates + some energy and time of arrival information).

But let’s leave the tech details for a bit later and continue the Odyssey. ❤️

We can use GNNs to help beat cancer! By taking the protein/gene and drug interaction data and modeling the problem as a graph and throwing some GNN weaponry at it we can find a list of molecules that help beat/prevent cancer.

Once we have the most effective CBMs (cancer-beating molecules) we can create a map of foods that are richest in these CBMs, also called hyperfoods.

Here is a TL;DR: the hyperfoods are: tea, grape, carrot, coriander, sweet orange, dill, cabbage, and wild celery.

But, don’t overdose with these, I beg you! 😂

I know I started drinking green tea regularly after reading this paper. Who says that reading research papers is not healthy!? 🍎

On a side note, I take my nutrition very seriously. I’m regularly pushing both my body (I lift weights, etc.) and my mind to its limits so it’s very important to think of the recovery and rest as well. High quality sleep, omega-3s, nuts, vitamin D, green tea are some of my tricks.

But if you’re not a fan of nutrition (and causality in general 😜) then the following research may excite you even more — predicting (polypharmacy) drug side effects using GNNs!

Polypharmacy (using multiple drugs at the same time) side effects is something pretty much everyone will experience and we know that our grandparents are already experiencing it.

We basically can’t tell with certainty what will happen to you if you take 2+ random drugs.

Experimentally screening every single combination of drugs for side-effects is impossible due to the combinatorial nature of the problem — so computational methods such as GNNs to the rescue!

Aside from the polypharmacy side effects problem we’re facing an ever-growing threat from antibiotic-resistant bacteria that are evolving thanks to our irresponsible use of antibiotics.

Earlier in 2020, a super famous application of GNNs was published in Cell — A deep learning approach to antibiotic discovery.

Note: no free-access pdf, unfortunately (afaik), which makes me sad

Basically, by training a GNN on a dataset of molecules known to inhibit the growth of Escherichia coli, and later doing inference on a dataset of drugs under investigation, they “discovered” halicin-a molecule previously tested for its anti-diabetic properties now repurposed to be used as a potent antibiotic!

I’ve left out many exciting apps like predicting quantum properties of molecules, predicting molecule’s X chemical property (e.g. solubility), etc.

It’s an infinite list.

You can now see how dull and homogeneous the computer vision and NLP fields are! 😅

NLP: I manipulate text. CV: I manipulate images. Graph ML: hold my beer! 🍺

Jokes aside, there are some nice connections between CV/NLP and Graph ML. You can treat transformers as Graph Attention Networks operating on fully-connected graphs (but more on that later) and you can treat images/videos as regular graphs (aka grids).

Through this lens, everything is a graph. Now whether this abstraction makes actual sense is problem specific. Specialized models such as CNNs have some useful biases ingrained into them and you probably wouldn’t want to apply a GNN directly to image pixels (although GNNs are getting some serious traction in the CV world in one way or another).

Hopefully, I got you hyped up! Let me now show you some resources that I’ve found useful learning about Graph ML.

Note: I often mention GNNs but they are only a subset of the Graph ML field. Aside from GNNs there are graph embedding methods and many others.

Note 2 (on how you should (probably) read the following chapter): first read it without digressing into the resources which I’ve linked and only then start using it as a reference manual — because that’s what this blog is, it’s not your one-shot blog, read it and forget it, noup.

Fantastic GNNs and Where to Find Them

I don’t aim to provide you with an extensive overview of the Graph ML literature here, and I don’t think that would be useful at all for that matter.

Dumping a bunch of resources (which I often see people do in GitHub repos) is not that useful — especially not for beginners.

What I’ll try to do is structure some of the resources, which I’ve found useful, in a way such that 2-month younger Aleksa would be super grateful. ❤️

Let’s start! I’ll try and give you some more generally applicable research tips along the way as well.

Graph embedding methods

If you’re trying to dive into a new field it’s super useful to organize your learning resources chronologically (for nerds: the timestamp feature of the research paper is highly valuable — don’t ignore it).

Before GNNs became mainstream people used methods analogous to Word2Vec which enjoyed a great deal of popularity in the NLP world.

Most of these were actually directly inspired by the Word2Vec method and to be honest, I’m happy I built my NLP and CV backgrounds before jumping into Graph ML. The field pulled a lot of inspiration from CV and NLP fields as you’ll soon see.

Here are some papers I recommend you start your journey with:

• DeepWalk — it’s reduced to Word2Vec if you do the following: sample “sentences” from the graph by doing random walks.
• Node2Vec — literally the same idea as DeepWalk with the additional control of how you’ll sample from your graph (BFS/DFS). If you give more weight to BFS the tightly coupled nodes will have similar embeddings.
• Planetoid — a semi-supervised setting. Aside from random walks (unsupervised), you can also leverage the labels (supervised). They additionally made sure that their method can be used in an inductive setting (check out my GAT Jupyter Notebook for the short explanation of transductive vs inductive learning, just ctrl+f it ❤️).

A brief clarification on DeepWalk: if you treat the nodes in the graph as words then if you do a random walk you’re basically sampling a random sentence from your graph.

Once you have the sentence you can do the Word2Vec magic and learn your node embeddings such that those nodes that tend to be close in your sentences have similar embeddings.

✅ Graph embedding methods

All right! Now you’re ready for some GNN magic! 😍

Graph Neural Networks

Historically there were 2 main approaches to developing GNNs:

• Spectral methods
• Spatial (message passing) methods

Both tried to generalize the mathematical concept of convolution (that worked so nicely in the CV world) to graphs.

The spectral methods tried to preserve the strict mathematical notion of convolution and had to resort to the frequency domain (Laplacian eigenbasis) for that.

On the other hand, the spatial methods are not convolutions in the strict mathematical sense of the word, but we still informally call them convolutions, as they were very loosely inspired by the spectral methods.

The long story short, spectral methods “died out” because they are:

• Computationally expensive
• Inherently transductive

Having said that they are definitely still worth researching and you should build up at least some basic foundations on spectral methods, IMHO.

But, before we continue with spectral/spatial GNN papers I strongly suggest you take some time off to do a superficial, high-level overview of what Graph ML is. Build up that knowledge skeleton first before you start filling in the blanks. The top to bottom approach, I’m a big fan of it.

I recommend watching this high-level overview of GNNs by Microsoft Research folks, as well as this one by Xavier Bresson (one of the big names in the Graph ML field):

Additionally here are some blogs to get you exposed to new terminology and information, let them sink in:

• Tutorial on Graph Neural Networks for Computer Vision and Beyond (a beginner-friendly intro to GNNs)
• A Tale of Two Convolutions: Differing Design Paradigms for Graph Neural Networks
• A high-level overview of some important GNNs (MoNet falls into the realm of geometric deep learning though, but more on that later)

Nice!

✅ A high-level overview of Graph ML

You’re now ready to dive into the world of Graph Neural Networks. 🌍

The spectral methods

The main “tool” behind the spectral methods is something called the graph Laplacian. Additionally, the undirected graph assumption is made so that we can have a symmetric graph Laplacian which has some nice mathematical properties — namely the fact that we can find an orthonormal Laplacian eigenbasis*.

* it’s a generalization of the Fourier eigenbasis with which you’re familiar if you took any signal processing course, check out 3B1B’s video for a gentle intro. In a special case where the underlying graph is a grid (like image), we get sinusoids i.e. Fourier eigenfunctions!!!

Now the main idea is to project the graph signal into that eigenbasis, filter the projected graph signal directly in the spectral domain by doing an element-wise multiplication with the filter, and reproject it back into the spatial domain.

It’s ok not to understand all of this right here and right now. You’ll have your epiphany moment sooner or later. Here are the main spectral papers:

• Spectral Networks and Locally Connected Networks on Graphs — first introduced the concept of the convolution to graphs.
• ChebNets — One of the main problems with the above method is that it’s not localized i.e. the updated node features can be affected by any node in the graph.

The ChebNets paper made it so that only k-hop neighborhoods can impact the feature representation of any particular node and also made it computationally efficient to do so — by using a Chebyshev polynomial of the graph Laplacian there is no need to compute the Laplacian eigenbasis which is an expensive operation (~O(n³)).

Note: in spectral method papers you’ll be seeing some “heavy” mathematics which is not a prerequisite for understanding the spatial methods (and they are way more popular so don’t beat yourself down if you don’t get this the first time).

Aside from the papers here are some other super useful resources that will help you better understand these spectral methods:

• Spectral Graph Convolution Explained and Implemented Step By Step
• The Smallest Eigenvalues of a Graph Laplacian
• A tutorial on Graph Laplacians

And finally, this video made it click for me when it comes to understanding the spectral clustering method which leverages the Laplacian eigenvectors to cluster the nodes (any additional intuition you can collect will be useful).

That’s it. I’ve left out some less important papers to avoid the clutter (and btw. I’m only recommending the resources which I’ve read and found useful).

✅ Spectral methods!

Now let’s continue to the realm of spatial methods.

Spatial/Message passing methods

Here is a high-level, super informal, explanation of message passing methods.

The goal is to update each and every node’s representation. You have the feature vectors from your node and from his neighbors at disposal. You then do the following:

1. You somehow transform the feature vectors (maybe a linear projection)
2. You somehow aggregate them (maybe weighing them with attention coefficients, voilà, we get GAT (you see what I did there))
3. You update the feature vector (somehow) of the current node by combining its (transformed) feature vector with the aggregated neighborhood representation.

And that’s pretty much it, you can fit many different GNNs into this framework.

Let’s now explore some of the most famous spatial methods:

• GCN (Graph Convolutional Network) — simplified the ChebNets paper to operate on 1-hop neighborhoods and made GNNs much more efficient and applicable.

I’ve covered the GCN paper in this video:

• Graph Attention Network (GAT) — made GCN more expressive by allowing it to learn the neighborhood aggregation coefficients (GCN uses constants instead of learnable weights).

As you’ll later see I’ve even implemented GAT from scratch and Petar Veličković (the first author) himself recommended this video so I guess it should be good enough! 😅

Now the problem with GAT and GCN is they were originally not scalable which is needed for many real-world, massive-scale, graph datasets like any social network’s graph.

The following 2 papers made GNNs applicable to huge graphs:

• GraphSAGE — SAmple and aggreGatE, it introduced the concept of sampling your neighborhood and a couple of optimization tricks so as to constrain the compute/memory budget. It’s a direct precursor to PinSage which is deployed at Pinterest as the recommender system!

Here is my overview of GraphSage. Btw. if you were wondering, Graph ML is a pretty young field so you won’t find many YT overviews of these papers that’s why I’m linking the ones I created. I only wish to link the best content — if that’s sometimes my content — well let it be. 😅 Anyways! The second paper:

• PinSage — One of the most famous real-world applications of GNNs — recommender system at Pinterest.

And here is my video overview:

Those were some of the most important spatial GNNs. Going further here is a paper that generalized all of the above nets into a so-called Message Passing framework:

• MPNN (message passing neural net) — aside from having a theoretical significance of unifying the spatial GNNs it also introduced an awesome application of GNNs: predicting quantum properties of molecules e.g. fundamental vibrations, the energy needed to break up the molecule in certain states, etc.

Two more seminal papers you should check out are:

• Gated Graph NN — used GRU for neighborhood aggregation and applied the method to program verification via the separation logic. It’s not an easy read, but I think it’s worth it, it played an important role in kick-starting the spatial method revolution.
• Learning Convolutional Neural Networks for Graphs — gave an idea of how we could impose some order onto the graph neighborhood (via labeling) and apply a convolution that resembles CNNs much closer. I guess it could be considered as a third way to introduce convolution to graphs, but this approach didn’t get any serious traction though.

Wrapping up with one more amazing paper which showed that we clearly don’t know what we’re doing 😂:

• Simplifying Graph Convolutional Networks (SGC)

Jokes aside, what it showed is that for certain graph families GNNs are totally unnecessary and a much simpler baseline (that acts as a simple low-pass filter) can do an even better job!

It’s a single layer GNN that first does the aggregation over the k-hop neighborhood (the majority of the benefit arises from this) followed by a simple linear projection and a softmax.

In these early days of spatial methods we were missing answers to some fundamental theoretical questions like why are GNNs so shallow (compared to their extremely deep CNN cousins), what are their limitations, etc.

That brings us to the following section.

✅ Seminal spatial GNN papers

Let’s go!

GNN expressivity

Early spatial methods, such as the famous GCN paper, started making some links between GNNs and old graph theory methods such as the Weisfeiler Lehman algorithm.

Weisfeiler Lehman (1-WL) also known as the color refinement algorithm is a simple graph isomorphism test: given 2 graphs verify they are the same even though their node ids may be permuted. WL can tell you with certainty whether your 2 graphs are non-isomorphic and it can tell you, with high probability, whether they are isomorphic.

Check out this awesome blog for a great explanation of how WL works.

Now there are certain graph families that can “confuse” the WL test. Here is one example of where the WL algorithm outputs the same color histograms (and thus it “thinks” that these 2 graphs are isomorphic — but they’re not):

Ok, now you may think this is all nice Aleksa, you’re brushing up our fundamental graph theory knowledge, but what the heck? Well.

This seminal paper showed that MPNNs are, at best, as powerful as 1-WL:

• GIN — Graph Isomorphism Network

It introduced certain requirements needed for the MPNN to be as expressive as 1-WL like using sum as the aggregation function (hint it’s injective) instead of the popular max and mean aggregators.

Michael Bronstein has a nice series of 3 blogs on this topic so do check it out!

After GIN, many methods tried to unleash GNNs and give them even more expressive power, and so various methods were advised which were as powerful as k-WL algorithms.

Also, other families of methods were advised which broke up with the WL hierarchy altogether like this one:

• Improving GNN expressivity via subgraph isomorphism counting

The main idea is that since some previous research showed that GNNs lack the subgraph structure counting ability we can add those explicitly as additional node/edge features.

And another recent paper called Principal Neighborhood Aggregation further improved upon GIN and made its theory applicable to realistic use cases (since the original GIN had some simplifying assumptions like only using one-hot vectors as feature vectors).

PNA showed that we need to use multiple aggregators, they used n-th statistical moments — like mean, std deviation, and higher, in order to get the theoretical guarantees of being able to discriminate between continuous multisets as the theorem from the paper suggests:

In order to discriminate between multisets of size n whose underlying set is R, at least n aggregators are needed.

Where multisets are sets where the elements may appear more than once (feature vectors of a node is a multiset in our specific case).

Note on expressivity: if a model is more expressive that means that it would produce different embeddings for 2 graphs that are non-isomorphic with higher probability than some less expressive model (as we saw some graphs confuse 1-WL test (and thus MPNNs) and so in those cases we’d get the same embeddings even though the 2 graphs are non-isomorphic!)

As you can see you could be doing just this subfield of Graph ML research as it’s fairly complex and there are still many open questions!

✅ GNN expressivity

Let’s take a step back from the hard-core theory and investigate some exciting app papers, some of which I’ve mentioned in the previous chapter.

Papers on exciting GNN applications

I’ve already mentioned some of these:

• GNNs for IceCube signal classification
• Hyperfoods paper (anti-cancer molecules and foods) — it’s a bit harder to read than the other papers but you could skim it to get a rough idea of what they did.
• Modeling polypharmacy side effects with GNNs — models the problem as a multi-modal graph (proteins/drugs), with the goal of multi-relational link prediction (given a pair of drugs there can be up to 964 different edges in between each corresponding to a different side effect!)
• Fake News Detection on Social Media using Geometric Deep Learning
• Convolutional Networks on Graphs for Learning Molecular Fingerprints — beautiful paper it replaced the old heuristics with a learnable GNN and gave awesome results on predicting different properties of molecules.
• Modeling Relational Data with GCNs — applying GNNs to knowledge base entity and link prediction.
• Graph Structure of Neural Networks — uses network/graph analysis knowledge to do a smarter NAS (neural architecture search). Neural networks themself are graphs, right? So let’s apply some Graph ML knowledge! Note this is less of a GNN paper but still an interesting read.

Aside from all of these and the stuff I mentioned in the previous chapter I also suggest these 2 blogs for more awesome applications:

• Geometric ML becomes real in fundamental sciences (Michael Bronstein)
• Top Applications of Graph Neural Networks 2021 (Sergey Ivanov)

As well as this paper for a holistic overview of different methods used in the Graph ML field:

• Representation Learning on Graphs: Methods and Applications

✅ Exciting applications of GNNs

We’re getting closer to the end. 💪 Go drink a green tea and get back. 🍵❤️

Handling dynamic graphs

So far all of the papers I linked deal with the so-called static graphs i.e. graphs that don’t evolve over time. But the truth is many graphs of interest do change over time. Take Twitter/Facebook/<your fav social net if any> graph as an example.

Here are 2 nice papers dealing with continuous-time step dynamic graphs:

• Temporal Graph Networks for Deep Learning on Dynamic Graphs — this paper came straight from Twitter with M.Bronstein being one of the coauthors.
• Inductive Representation Learning on Temporal Graphs (TGAT) — fairly similar to TGN above, it can actually be considered a special case.

And again here is my overview of TGN paper (Rossi endorses it ✅❤️):

This was just to tickle your imagination. You can imagine dedicating a couple of years of research to only this subtopic of Graph ML. There are still many open questions and engineering problems left.

The 2 “main” tricks are:

• Find a way to represent time and include it as an important feature
• Define a temporal neighborhood over which to aggregate

✅ Dynamic graphs

Finally, let me leave you with some totally optional papers.

Note: pretty much every subsection, including this one on dynamic graphs, is orthogonal to other subsections. Just saying.

Go unsupervised young ML human

If you’re passionate about unsupervised learning you’ll find these 2 interesting:

• Deep Graph Infomax (DGI) — combines the deep infomax theory with graphs.
• VGAE — combines the VAE (variational auto-encoder) with GCN.

Aside from the unsupervised learning, you may wish to place your foot into the Geometric-DLandia (Geometric DL mostly deals with manifolds although there are many connections with the Graph ML field):

Geodesic CNNs on Riemannian manifold (GCNN) — this one broke my mind 😂, working on manifolds involves so many new details that I haven’t seen anywhere else across all the different deep learning fields I’ve explored so far.

Geometric deep learning: going beyond Euclidean data — again, really math-heavy and includes a complex survey on manifolds, differential geometry, graphs, different methods, and different applications.

MoNets — Geometric DL on graphs/manifolds using mixture models. It generalizes some of the previous methods like the above-mentioned GCNN.

Again, note that there is some heavy mathematics, new jargons, and terminology involved with Geometric DL — promise that you won’t get overwhelmed!

You don’t need to understand these in order to understand Graph ML.

✅ Misc

Let’s wrap up this brain dump of mine. 🧠⚡

Graph visualization toolkit, datasets, where to next?

Here I’ll list some of the other things you should care about. It’s not only about papers. Tooling is super important. Data is arguably even more important than the algorithms. Here we go!

This blog contains a nice overview of different graph drawing packages:

Large Graph Visualization Tools and Approaches

I ended up using igraph as it seemed like the most powerful graph drawing tool which I could use directly from Python.

Here is a nice beginner-friendly intro to igraph and here is the official tutorial (it’s pretty good I’ve used it)

And again, graph drawing is a subfield for itself! You can dedicate some of your research years to researching only this topic. Similarly to how tokenization in transformers is a subfield for itself even though we usually use these “tools” as a black box.

While we’re talking about visualizations I think I should mention the graph datasets!

First of all, you’ll be seeing Cora, Citeseer, PPI, and Pubmed everywhere those are the classical benchmarks (like MNIST in CV). The problem with these is, as people later found out, that they are treacherous — we can’t exactly discriminate between more expressive methods and the less expressive ones.

So people started advising new graph benchmarks, and we should all be cognizant of how important this is. If you’re coming from the CV background, as I do, you know what ImageNet did for the CV world.

It even became a metaphor! People say that NLP got its “ImageNet moment” with the arrival of big data, transfer learning, and deep contextualized representations. So data is a big thing! (pun probably intended)

So what’s happening with ImageNet in Graph ML?

Open Graph Benchmark initiative looks very promising, the time will tell!

I’d like to wrap up this mind-breaking blog, which had a bold attempt to cover the Graph ML literature, with 2 additional resources you should take into consideration.

If I were you I’d read every single blog from Michael (as I did):

Michael Bronstein - Medium

And you should follow Sergey’s newsletter. Sebastian Ruder showed that this format works wonders for the NLP world, so let’s see whether Sergey makes the same thing for the Graph ML world! 👑

Graph Machine Learning

We’re done with the theory! Put your relaxed hat on and lemme walk you through my GAT project. 😎

Edit May 2022: I strongly recommend you also check out Distill.pub’s articles on GNNs: A gentle introduction to GNNs and Understanding Convolutions on Graphs.

Graph Attention Network (GAT) Project

As always, I’m a strong believer that reading through theory is just not enough. You also need to test your understanding, because, how else would you know you’re not just overfitting the training dataset? 😅

And what better way to do it than to engineer it from scratch?

If you can’t build it, you don’t understand it — somebody, somewhere, somewhen

With that mindset, I decided to implement the Graph Attention Network. It just felt like a natural step since they are a generalization of transformers and I’ve previously implemented Vaswani’s transformer.

I mean: attention + GNNs = ❤️ , right?

Additionally, I am lucky enough that I get to chat with Petar Veličković (the first author of the GAT paper) on a regular basis — so that made it easier for me to figure out why certain design decisions were made.

So after ~3 weeks of my free time (including weekends and aside from my daily job at Microsoft, I’m crazy I know), I pulled it out:

gordicaleksa/pytorch-GAT

I’ve really learned a lot.

I never played with graphs before (aside from implementing some classical CS graph algorithms in C++ back in the days) so this made me really excited.

Understanding graph data and visualizing it gave me a lot of satisfaction. I’ve broadened my toolkit with new packages such as igraph and NetworkX, and I created beautiful visualizations such as this one:

I’ve used the Kamada Kawai layout to draw this one. Obviously, I got some background in network analysis and graph drawing.

And of course, I’ve learned how to implement GAT in many different ways (I finally ended up with 3 implementations!). The main tradeoff was between the ease of understanding and efficiency — as it often is.

After I’ve trained the model I decided to add as many visualizations as possible so that I can better understand what’s happening. The 3 main visualizations I did were:

• Embedding visualization via t-SNE
• Neighborhood attention visualization via edge thickness
• Attention pattern visualization via entropy histograms

Once you train the GAT on the Cora dataset here is how a single neighborhood looks like (I’ve cherry-picked a high-degree node here):

The first thing you notice is that the edges are of the same thickness — meaning GAT learns constant attention on Cora.

For more details, you can check out the README and the accompanying Jupyter Notebook! (this one covers Cora the other one covers PPI)

If you need additional help understanding the project I created a project walkthrough video:

I’ve put an emphasis on explaining the hardest-to-understand implementation (implementation #3 as I’ve dubbed it) so hopefully, in addition to Jupyter Notebook, this will help you get a deep understanding of how GAT works.

I also created this project update video, while I was working on GAT, which you may find interesting if you’re interested in my journey:

Aside from my GAT implementation, I want to mention that you have at your disposal an awesome framework called PyTorch Geometric which you should use if you don’t necessarily want to dig into the nitty-gritty details, but you just want to quickly test some GNN out-of-the-box.

PyTorch Geometric is to Graph ML field what HuggingFace is to NLP. 👑

Another lib worth mentioning is DGL whose PPI dataset I end up using.

My next steps

Phew. If you made it till the very end, congrats! ❤️

I hope this blog will inspire you to start exploring Graph ML on your own! Or deep learning/machine learning in general for that matter.

One thing you notice when you start exploring Graph ML is how many related fields (but with a completely new set of terminology, equations, and theory) there are.

Examples would be:

• Geometric deep learning (learning on manifolds) — which is closely related to Graph ML since both are concerned with learning on non-Euclidean domains (graphs/manifolds).
• Equivariance deep learning (exploiting symmetries to make your models statistically efficient i.e. use less data to achieve the same perf) — related to CNN and geometric deep learning fields (I’ve mostly seen it in this context: group CNNs, spherical CNNs, gauge CNNs (they work on manifolds)) but is also much more generally applicable (equivariant transformers, etc.).
• KG field (knowledge graphs) — which lies somewhere in between graph ML and NLP.

I did some “digressions” into these fields and I tell you — it can get overwhelming. It’s not that it’s hard it’s just that there is a lot of new information and it takes time.

Especially interesting are GDL and equivariance DL where you’ll start seeing physics theory perturbing its tentacles and injecting ideas from quantum/particle/classical physics into deep learning.

You find yourself asking questions like how did the general theory of relativity end up here, but there you have it.

When it comes to my next steps nothing has changed since my last blog post update. In a couple of days, I’ll start exploring the (deep) RL field!

I’m super excited as some of the most famous AI breakthroughs happened exactly in RL like AlphaGo/AlphaZero/MuZero/AlphaStar line of research coming from DeepMind.

Be sure that I’ll extensively cover all of these on my YT channel and, again, I’ll probably implement one of these from scratch, although I’m still not aware of how complex it is to implement something similar — so I’ll stay flexible here. 😅

Gratitude

I also want to say that I’m super grateful and humble about all the beautiful things that have been happening over the last couple of months in my life.

I had many opportunities sent my way, like being offered to do various Graph ML research with people from CERN and many different industries. I got offered to hold presentations, webinars, podcasts, and even a meetup in London (that one will probably happen so stay tuned!).

People are asking me to mentor them, Sergey Ivanov linked my video series in his newsletter, Michael Bronstein, Thomas Kipf, and many other influential AI people were sharing my work.

I’ll have to say no to many opportunities as I simply don’t have the time to manage all of them (even though I’d love to!). I’m working full-time at Microsoft and I’m following this path of mine pretty much full-time as well.

I also get tons of DMs and emails and I just want to state here that I apologize in advance if my replies are delayed (and sometime in the not-so-distant future I simply won’t be able to handle all of it myself).

I received many beautiful messages of support and love from many of you and I truly appreciate it. ❤️ It keeps me working even harder.

If there is something you would like me to write about — write it down in the comment section or send me a DM, I’d be glad to write more about maths, ML, deep learning, software, landing a job in a big tech company, preparing for ML summer camps, electronics (I actually officially studied this beast), etc., anything that could help you.

Also feel free to drop me a message or:

1. Connect and reach me on 💡 LinkedIn and Twitter
2. Subscribe to my 🔔 YouTube channel for AI-related content️
3. Follow me on 📚 Medium and 💻GitHub
4. Subscribe to my 📢 monthly AI newsletter and join the 👨‍👩‍👧‍👦 Discord community!
5. 📄 Website

And if you find the content I create useful consider becoming a Patreon!

Much love ❤️

As I often do, I’d like to share my personal story before jumping into the “core part” of the blog — if you only came to read about the transformers/NLP resources please skip this section (I still love you! ❤️). So here we go!

This year I decided to structure my deep learning journey really neatly. It all started in February 2020 with Neural Style Transfer.

After studying it for months, reading many research papers, blogs, watching videos, etc. I started my own YouTube channel “The AI Epiphany” and I kicked off the NST series:

This was the first-ever video I made — not the one I am the proudest of but it was the first step and a step in the right direction!

My idea for 2020 was basically this:

I am going to take a full year to cover the 6 areas of deep learning (I’ll mention which ones a bit later) that are super interesting to me, and I’ll gain an even stronger understanding of deep learning, ML, and mathematics.

I’ll do that by covering both the theory (usually via the top-down approach) like reading blogs, watching videos, reading books, and all the relevant research papers particular to the current topic I’m working on, and by coding relevant projects from scratch (mostly reimplementing papers) in PyTorch.

This is where many make a mistake — until you’ve tried to get your hands dirty by developing some project on your own, you don’t know how many details you actually don’t know! Believe me on this one…

The devil is in the details

Aside from that, I won’t make the same mistake I made throughout my life, that is I’ll share my learning journey with the world along the way.

Until that point, I always thought I should first become a world-class expert before I start teaching others (and it also seemed like an unnecessary distraction at that point in time), but then I realized (at least) 3 things:

1. You learn a looooot by teaching others (that’s 5 ‘o’s yes, a lot.)
2. You are best positioned to help people one step behind you (probably)
3. A whole world of opportunities opens up once the broader community becomes aware of your existence (most certainly)

So, how I like to think about it is I have 3 main content platforms, namely:

1. My YouTube channel The AI Epiphany where I’ll keep on sharing everything AI related that I find important (I’ve been mostly covering deep learning research papers lately and I’ll continue doing that)
2. GitHub where I open-source all of my projects (I love this platform ❤️)
3. Medium where I occasionally write (I definitely spend the least time here)

And I reshare that content on a lot of places like Facebook AI groups, Reddit, etc., but my central social media locations where I also share my thoughts are definitely LinkedIn (I was really bullish about it in 2020) and Twitter.

I only just recently started sharing my thoughts more actively on Twitter — as of today actually (the day I wrote this blog 😅).

The AI community is super strong on Twitter. I’ve been following AI people there for 2 years but I wasn’t active myself. That’s about to change!

So long story short my Neural Style Transfer field exploration ended up with me reading 20+ research papers, uploading 7 videos in the NST series, and open-sourcing 3 GitHub projects:

• My implementation of the original NST paper (Gatys et al.)
• My implementation of the fast NST approach (Johnson et al.)
• My pipeline for creating NST videos (I used it to create my YT videos)

Here is a nice neural style transfer pic for you!

I later covered the Deep Dream algorithm (GitHub Deep Dream). And yeah a fun fact — I used this one to create my YouTube/GitHub/<fill in the blank> banners so I do use my own software you know!

Here is a psychedelic pic I generated (I find DeepDream truly fascinating):

But. I hear you asking:

“But why NST and DeepDream, those things are useless?”

And they are.

I’m joking a short answer would be: they’re fun & awesome and are definitely useful to artists, you’ll get really familiar with ConvNets and a DL framework of your choice, and you’ll probably learn a bit about interpretability.

Also, many ideas from NST appear elsewhere. Like the famous StyleGAN paper used an idea from the NST literature (adaptive instance normalization).

I love art and I’m a highly visual type of learner (I think so at least) so in my probably overly biased opinion, I think these 2 projects are a great way to get your hands dirty with deep learning.

The third area I explored was Generative Adversarial Networks (GitHub GANs). Here is some latent space exploration for you:

I learned a lot about GANs and they are a fascinating field. So you can probably expect me doing something related to GANs in the future.

And finally, I just “finished” exploring transformers and NLP field in general! Unfortunately, I haven’t been sharing my journey on Medium from the very onset, but I still have 2 topics left (see my next steps chapter at the end of the blog) so I’ll hopefully be at 50% coverage! 😅

Note: Aside from NST which took me much longer and Deep Dream which took less time, it usually takes about 2 months of intensive learning for me to feel good about my understanding of the topic. I also work full time at Microsoft — so it’s pretty intensive 😆, but I like it a lot — I’m probably sick.

OK! That was a short attempt to compensate for the lack of 3 thorough blogs. Hopefully, some of you found it useful or motivating I dunno.

Now let’s jump to NLP and transformers. I’ll share the theory I found useful, some videos I created so far (which could further help you), and what I’ve learned reimplementing the original transformer paper from scratch.

A lofty goal. Here we go!

Personal update: I just created a monthly AI newsletter and a Discord community! Subscribe/join those to keep in touch with the latest & greatest AI news!

The trigger point — Vision Transformer (ViT)

I’ll tell you how I approached learning this subfield of deep learning. You’ll probably get the most out of it if you already have a decent background in some other subfield of deep learning. For me, that was computer vision.

So I initially wanted to cover GNNs first but on October 3rd the Vision Transformer paper (ViT) was published — and since I’m really bullish on computer vision and this paper pretty much wrecked CNNs I decided I should get a solid grasp of transformers sooner rather than later.

So I went ahead and read the Vision Transformer (ViT) paper. At least I tried. In the first go I didn’t understand the details (I did understand the main point of the paper though) as it had a hard dependency on the original transformer paper so I took a step back and I read the original transformer paper:

Attention Is All You Need.

Now at this point, I realized many gaps in my knowledge so I took a further step back and decided I needed to learn about attention/self-attention as well, word embeddings, tokenization, etc.

I did have some notion of all of those concepts — but it was far from a deep understanding (no pun intended).

Enter the dependency hell — Aleksa Gordić, October-November 2020

I also knew I wanted to understand the main trends of NLP. BERT/GPT/Transformers were the recurring topics I kept hearing about, even though I was nicely cushioned in my small bubble world of computer vision.

So, on one hand, I wanted to understand NLP and its main trends and on the other hand, I really wanted to understand the Vision Transformer. Those interleaving goals helped shape my strategy.

Over the next section, I’ll group the relevant papers, blogs, and in general different resources, which helped me in achieving those goals.

I warn you — there was a lot of back and forth between different “groups” of resources and subtopics. Learning is far from a linear process.

Sometimes you’ll read something and you won’t understand anything. So you’ll go to a completely new subtopic, new connections will form, the old ones will sink in, you’ll return back, re-read that something again and understand it a bit better.

Sometimes you’ll quickly skim papers just to get the bigger picture and form that initial skeleton of the knowledge graph and only later will you thoroughly read it.

Sometimes you’ll read the paper, then you’ll take a step back, read some blogs or watch videos that contain higher-level explanations, and get back to the paper again.

There are lots of back and forth both vertically, between the low level and high-level explanations (across the knowledge pyramid), and horizontally across different subtopics (e.g. attention/tokenization) of the subfield (e.g. NLP) you’re learning.

That’s the reason you shouldn’t consider the list of resources that follow as a chronological list which you should linearly follow in order to learn NLP.

Although that approach might just work for you — who knows — the trick is to never give up!

NLP/Transformers learning resources

Here we go. Let’s start with attention — that’s a key concept that you’ll be seeing everywhere going forward. Make sure you deeply understand it.

Attention/self-attention/50 shades of attention

Blogs (sorted approx. from the best ones towards less valuable ones IMO):

• OG of transformers — Jay Alammar’s blog on attention
• Understanding attention mechanism
• Illustrated attention
• Self attention mechanisms in NLP
• Illustrated self-attention

The first link will probably suffice, it’s an amazing resource. Feel free to go at your own pace and check out the other links as well (I’m only linking stuff I read myself and found to be of a good/great quality).

How I would probably go about reading this blog if I were you is I’d first read it without stopping, top to bottom, and then I’d return back to sections/links that I think are the most important for me. Nice! Let’s continue!

Low level (papers) that introduced attention to the field:

• Neural Machine Translation By Jointly Learning to Align and Translate
• Effective Approaches to Attention-based Neural Machine Translation

Once you grasp these 2 papers you’ll understand attention.

Next, I’ll introduce a couple more concepts that are used in transformers.

Tokenization

Tokenization is a world for itself. In most transformers papers it’s just mentioned in one line maybe but it’s really important. That being you can treat it as a black-box most of the time. Here is the best resource I found:

• https://blog.floydhub.com/tokenization-nlp/

This one from the famous Stanford professor Christopher Manning is also really nice.

Word Embeddings

This one is necessary to better appreciate the deep contextualized token representations we have nowadays with pretrained transformers.

Word embeddings played an important role in NLP history.

The drawback these representations have is that they aren’t contextual. What I mean by that is that e.g. Word2Vec or GloVe would give you the same embedding vector for the word/token “queen” in both of these sentences:

1. “Queen Elizabeth II has ruled for longer than any other Monarch in British history.”
2. “The Queen Bee plays a vital role in the hive because she is the only female with fully developed ovaries.”

Even though the semantics i.e. the meaning of these words are totally different. Here are the 2 best high-level explanations I found:

• Illustrated Word2Vec (Transformers OG Jay Alammar)
• What are word embeddings (Jason Brownlee)

And here are some papers worth reading (pick only one, you don’t need both I skipped GloVe, you just want to get the idea of how these were trained):

• Word2Vec paper
• Distributed Representations of Words and Phrases and their Compositionality (Word2Vec second paper)
• GloVe

At this point you could probably take a look at this seminal paper written in 2003 by the Turing award laureate Yoshua Bengio:

• A neural probabilistic language model

It’ll help you further consolidate your knowledge.

Contextualized shallow representations

I mentioned the “queen” example and how we're failing to capture the meaning of the word based on the context.

Can we do better than Word2Vec and GloVe?

And so, contextualized word-embeddings were born. The first one that I’m aware of was this method called CoVe:

• Learned in Translation: Contextualized Word Vectors

These CoVe embeddings were contextual but they still had to develop task-specific architectures and they only used the last layer information to solve the downstream NLP tasks.

Then came ELMo which further refined the idea:

• Deep contextualized word representations

They noticed that different layers contained different representations of semantic/syntax information and so they used all of those representations to solve the downstream NLP tasks and that helped.

ELMo gained some serious traction, TechCrunch even made a blog about it:

• Machines learn language better by using a deep understanding of words

Now around this point in time, things were getting really intensive — lots of interesting research was happening in parallel. Transformer happened in 2017, a bit later GPT and BERT came to the scene — the rest is history.

Preceding those events, there are 2 more important dots we need:

• Universal language model ULM-FiT
• Semi-supervised seq learning (earlier in 2015)

Somewhere along this point of time when ULM-FiT appeared, the NLP world started shifting. Something strange was happening. The NLP field was entering its “ImageNet moment”. All of the previous research slowly enabled the power of full-blown transfer learning in the field of NLP.

Researchers have figured out that we can do transfer learning the same way it was previously being done in the computer vision world since 2012 — when the famous AlexNet/ImageNet moment happened. Basically, the CNNs (re)appeared and left every other approach in the dust.

You could see the saturation of the “top-5” error metric happening just around 25. Those were the old hand-crafted ML methods. Then CNNs made a quantum jump in 2012 and this wasn’t just the start of a new era in computer vision, this was the start of the deep learning era/hype as well.

We’ve known about CNNs since the early 70s so what was different this time? Well, powerful hardware and lots of data (like the ImageNet dataset). And a handful of people with a huge dose of “belief” that this time CNNs will shine.

Sebastian Ruder nicely documented this moment of NLP history:

NLP's ImageNet moment has arrived

Sebastian also has (had? the last one happened on April 2020) an awesome monthly newsletter so do check out that one (or subscribe) as well:

• NLP news by Sebastian Ruder

He’s a scientist at DeepMind and really famous in the NLP community, it’s always a good idea to surround yourself with the best from the field if you want to stay up to date.

And now put out the red carpet.

Transformers

Mighty transformers. This was the paper that helped start the revolution — The “ImageNet moment” of NLP:

Attention Is All You Need

I created a detailed explanation of the paper that can help you understand all of the nitty-gritty details (I’ve got a lot more transformers-related resources that I’ve created scattered across the rest of this blog post):

And going further, This Blog Is All You Need:

• The Illustrated Transformer (Again Jay Alammar ❤)

Here are a couple of nice videos as well:

• AI Language Models & Transformers — Computerphile
• Transformer Neural Networks — EXPLAINED! (Attention is all you need)
• Illustrated Guide to Transformers Neural Network: A step by step explanation
• Attention Is All You Need (Yannic Kilcher)

For a really nice recap of attention I’d recommend this blog:

• Attention Attention

It’ll also introduce you to the concepts of Neural Turing Machines and Pointer networks. And finally, if you want to get your hands dirty with some code, here is the project I developed and open-sourced:

• PyTorch The Original Transformer (I’ll mention it again a bit later)

BERT

Let’s continue our journey through the magic world of NLP.

I hear you cry “but BERT is just a transformer!”. No, it’s not! They even coined a separate term for exploring BERT-like models — “bertology”. Nice, right?

Anyways BERT is architecture-wise the same thing as the above transformer (it took the encoder portion of the architecture).

What it introduced were some details like e.g. the “class” token, and a new pretraining objective masked language modeling objective or MLM for short.

It was also bidirectional compared to the unidirectional/causal nature of the original transformer. That gives it an edge in many NLP tasks where it’s useful to have both the left and right context — the most obvious one predicting a word in the middle of the sentence (fill in the blank task).

And again this blog is the best resource I found:

• Illustrated BERT

Also, Google published 2 nice blogs on it:

• Open-sourcing BERT
• BERT is powering the Google Search

If you like the video format this one is a nice high-level overview:

• BERT Neural Network — EXPLAINED!

That’s BERT. Now, the second really famous family of models was of course the GPT family.

GPT (Generative Pretrained model)

GPT has a totally different approach and different goals from BERT. It’s an autoregressive language model. It’s causal and it’s unidirectional. It’s really good at generating text — especially long stories humans have trouble distinguishing from human-written stories (particularly the GPT-3 model).

Architecture-wise GPT basically took the decoder part of the transformer model (contains causal masks — i.e. tokens can’t look ahead) and pushed it to its limits.

The final iteration of the GPT family, the famous GPT-3 model, is basically competing for the position of the most general language model. No fine-tuning is needed for downstream NLP tasks. Pretrain once and use it for everything.

Again this could easily be a separate blog post. Here are the papers:

• GPT 1 (Improving Language Understanding by Generative Pre-Training)
• GPT 2 (Language Models are Unsupervised Multitask Learners)
• GPT 3 (Language Models are Few-Shot Learners)

I recently also covered the monstrous (75 pages long) GPT-3 research paper on my YouTube channel as well:

I only recently started doing these paper distillation videos so if you found them useful I’d appreciate your feedback and I’ll start doing even more of those. Yannic also did a nice video!

Again Jay did a marvelous explanation on his blog:

• Illustrated GPT-2 and a little about GPT-3

And TBH I really like OpenAI’s blogs:

• OpenAI’s GPT-1 blog
• OpenAI’s GPT-2 blog
• AI and compute (excellent one!) and malicious uses of AI

The AI development and compute power relationship is rather interesting so I thought it’d be nice mentioning that blog as well.

Finally here are some blogs covering the hype-anti-hype spectrum.

Verge blog — a nice collection of what happened on social media and in general around GPT-3 after it was first published.

This one nicely explains the topology of the hype-spectrum, this one gives a nice overview of the hype also (and model limitations, etc.).

You can see some of the “GPT-3’s pitfalls” here and finally by far the most thorough resource out there is that by Gwern:

GPT-3 Creative Fiction

Gwern’s text is a bit more difficult to parse IMO, but it’s got all of the details you’d want to know about GPT-3's real-world behavior (compared to non-intuitive objective metrics like perplexity, etc.)

Finally, here are a couple of GPT apps you could try out:

• AI Dungeon (it’s an interesting interactive game!)
• Talk To Transformer
• Write With Transformer
• IDE autocompletion

Is there life after BERT and GPT?

You bet there is! First, it’d be worth checking out these 2 nice overview blogs.

• NLP papers you should read (Nikolai is an ex NVIDIA researcher)
• Overview of transformers by HuggingFace

Here are some of the interesting papers I’ve read:

• TransformerXL (Google) (smart way to widen the context of LM)
• XLNet (Google) (proposes a way to unite BERT’s MLM and GPT’s LM objectives, uses Transformer XL as a backbone, 512 TPU v3s)
• RoBERTa (FAIR) (strong dependency on BERT and XLNet, 1024 V100s)
• ALBERT (Google) (smart way to get more efficient BERT, ≤512 TPU v3s)
• T5 (Google) (the best overview of the transformers landscape, TPU pods!)
• Uni LM (Microsoft) (introduced prefix-causal masking, 8 V100s)

I intentionally wrote the affiliation next to the paper just to paint the picture of how the current NLP landscape looks like. 90% of the highly influential research came from Google (including the original transformer).

I’ll stop here there is a lot of things that can be said on that topic! 😅

Next, I wrote the amount of compute that’s needed. Again I won’t go there either! (nervous chuckle).

You should probably readTransformerXL and XLNet in a batch. XLNet is super hard to understand IMO (and others share my opinion haha) and it’s also hard to reproduce. T5 later showed that it’s debatable whether we need those complications in the first place, but it’s still worth a read.

RoBERTa did a combinatorial space exploration around the original BERT and showed that with more data, compute and some hyperparam exploration you can gain much more out of BERT.

ALBERT experiments with parameter sharing, embedding matrix decomposition, and gradient checkpointing (to save up some memory). All of this leads to a much smaller memory footprint.

Finally, If there is one paper you should read that would be T5. Google also published a nice blog on it. I also really liked the Uni LM paper (not because I work at Microsoft but it’s truly a nice read!)

The second group of papers I’ll mention here have to do with making transformers more efficient (memory/time complexity):

• SparseTransformer (OpenAI) (introduced new attention patterns, O(n*sqrt(n)) compare that to original transformer’s O(n²), 8–64 V100s)
• Longformer (Allen AI) (introduced new attention patterns, 8 RTX8000)
• Reformer (Google), (introduces LSH, chunking, gradient checkpointing, etc. Here is the official blog, a nice overview (cool animations) and a nice in-depth explanation, 8 TPU v3 cores)
• Linformer (FAIR) (low rank approximation, O(k*n)~ O(n)) (64 V100s)

You can see that the necessary HW for training these ones is way smaller than for the above group of papers (that’s why I’ve put ALBERT in this other group).

Phew let me stop here. That was a lot of theory to go through. I explained a part of the process of how I approached learning all of this in this video:

What I’ve learned implementing the transformer

Reading through a bunch of excellent theory I still knew there are details that I don’t understand and so as always I wanted to implement at least 1 project in the field I’m interested in.

So after ~3 weeks of non-stop work I created this one:

gordicaleksa/pytorch-original-transformer

I learned a looot. One thing that used to confuse me was how do we start the generation process in the autoregressive architectures like the original transformer? Which is a detail none of the above blogs/papers covered.

It’s one of those “tribal knowledge” details. If you’re long enough in the field you know that stuff so it feels redundant to say it out loud.

And the answer is you trigger it with the start of the sentence (commonly denoted as <sos>) token’s embedding vector. Whatever comes out at the topmost layer you feed back as the second token and voila!

Super simple, as most things truly are, it’s just that rarely who has an incentive to make it easier for others, for whatever reason that may be.

I was also interested to see how the data is exactly fed to these transformers and how the exact training procedure looks like (KL divergence over soft targets in the case of the original transformer).

I got a much better feeling for different decoding methods like greedy, beam search, sampling, top-k, top-p (nucleus), and most importantly I understood the exact details of the attention mechanism and how to visualize it.

I had fun along the way, although it was also rough at times — I especially had problems with PyTorch’s torchtext and BucketIterator class specifically. Their torchvision lib for computer vision is far more sophisticated.

Anyways if you’d like to learn more I created a Jupyter Notebook here. It’s really long I know but once you understand it you’ll be in a good position to understand much of the exciting research that’s happening across the deep learning field.

I also did a video where I explained the process I went through developing this project:

Finally, if you’d like to get a feeling for what you can do with transformers do check out HuggingFace’s official and community notebooks and try out their pipelines API— it’s super easy to use and you’ll get some toy projects under your belt.

My next steps

Phew, that was a long blog! I, unfortunately, can’t afford (time-wise) to make more than one of these for every deep learning topic that I cover.

If you’ve read the first-ever Medium blog, that I wrote ~2 years ago, you could have expected me doing all of this. I mentioned that I am a strong believer in mixing theory with practical, hands-on projects.

So what are my next steps?

I’ll start exploring GNNs (Graph Neural Networks) starting from tomorrow pretty much! ❤️And after ~2 months I’ll switch my attention (pun intended) to RL (Reinforcement Learning).

You can definitely expect more YT videos during my exploration of these 2 subfields, at least 2 more (GitHub) projects, and 2 (Medium) blogs like this one.

And I’ll also try changing my DL framework a bit. One look at my GitHub repo and you know that I love PyTorch. I’ll try experimenting with JAX & GNNs.

After I finish that I’ll probably take one more month to cover a couple of topics that interest me: adversarial attacks, VAEs (variational auto-encoders), knowledge distillation and I want to build some foundations in symbolic AI.

That way I’ll (hopefully) have a holistic overview of the whole field.

Finally going longer into the future (longer into the future = 5+ months from now 😜), I’ll:

• Shift my focus from reimplementing other people’s work (papers) to developing my own creative/funny/useful projects.
• Start covering SOTA research papers on my YouTube channel as soon as they come out and I’ll also try to cover the most exciting stuff that’s happening/trending in the AI world be it some AI breakthroughs (like I recently covered DeepMind’s AlphaFold2), social topics or something else.
• Still occasionally write on Medium

Also, there are some plans which I, unfortunately, can’t yet disclose — so stay tuned for those! ❤️

A personal update September, 2021: I recently landed a job at my dream company Google DeepMind! I’ll be working there as a Research Engineer starting December, 2021. That was one of those things I couldn’t disclose back then as I was still working at Microsoft.

A small tangent on the topic of Ph.D. as that’s something I often think about. I don’t have a Ph.D. in ML/deep learning as I didn’t want it. It just doesn’t align with my personal goals.

I think I have much more freedom like this but I’m also 100% aware that not everybody can work in a fully unsupervised manner.

People say Ph.D. gives you freedom. And that’s true — compared to undergrad/grad studies for sure. But I do believe it would be a “lesser freedom” to the thing I have. At least for me.

There are lots of tradeoffs of course and this could be an entirely separate blog post (Chris Olah wrote a beautiful blog on the topic).

Briefly in my opinion the top pros of doing a Ph.D. are:

• Credentials (or “academic” badge as I like to call it) (oh he finished Stanford, oh he’s been to Cambridge, MIT, <fill in the blank>)
• Networking with fellow researchers (although you definitely don’t need a Ph.D. to do this, I have a bunch of friends who are researchers at Microsoft, DeepMind, Stanford (just throwing fancy names around))

Top cons:

• It takes a lot of your best years of life when you’re the most entrepreneurial you’ll ever be (most people at least)
• No relevant experience in the industry, worse coding/engineering skills (unless you do an industrial Ph.D. that’s a nice tradeoff IMO)
• Less freedom (again relative thing)
• You tradeoff depth for breadth
• No money (relatively speaking compared to the industry)

So IMO, If you’re certain that your goal in life is to become a researcher in a particular field then you should do a Ph.D. by all means — otherwise, you probably should not.

I just feel that many people (some of which I know) are doing a Ph.D. in deep learning not because they’re interested in pursuing a research career but quite the opposite actually.

They do it to get the highest paying jobs in the industry or to become more attractive to VCs (omg he’s been to Stanford). And that’s probably a good strategy, who am I to judge — but it’s not what Ph.D. should be about IMO, it’s just hacking/exploiting the current system.

If there is something you would like me to write about — write it down in the comment section or send me a DM, I’d be glad to write more about maths, ML, deep learning, software, landing a job in a big tech company, preparing for ML summer camps, electronics (I actually officially studied this beast), etc., anything that could help you.

Also feel free to drop me a message or:

1. Connect and reach me on 💡 LinkedIn and Twitter
2. Subscribe to my 🔔 YouTube channel for AI-related content️
3. Follow me on 📚 Medium and 💻GitHub
4. Subscribe to my 📢 monthly AI newsletter and join the 👨‍👩‍👧‍👦 Discord community!

And if you find the content I create useful consider becoming a Patreon!

Much love ❤️

The last blog of this series covered Turing’s famous “Computing Machinery and Intelligence” paper from 1950 and introduced his main ideas like The Imitation Game, nowadays better known as the Turing Test. It also covered the main objections that Turing himself imagined being used against his theory.

As a short recap Turing strongly believed in the computational nature of our minds, meaning it will be entirely possible for a digital computer to mimic the human mind and by doing so develop a (digital) mind of its own.

In this blog, I’m going to cover, in more depth, the strongest objections, as well as the new proposals for the potentially better tests, that came in the decades after the original Turing’s paper was published.

As well as some fun stuff I found along the way! Let’s jump into it!

Personal update: I just created a monthly AI newsletter and a Discord community! Subscribe/join those to keep in touch with the latest & greatest AI news if that’s something you care about!

Minds, brains, and programs

Written in 1980 by John Searle, this paper is considered as one of the strongest objections to the Turing Test.

It’s more famously known as the “Chinese room” argument. But before describing how the thought experiment goes let me first explain some terminology that is helpful to have in mind (whatever that is).

Searle distinguishes between 2 types of AI:

• Weak AI (a useful tool that helps us, humans, study the mind)
• Strong AI (programmed correctly the digital equivalent of the mind itself)

And we can all agree about the weak AI. Without a doubt AI is making our lives easier and is helping us better understand the world around us, minds included (there are certainly bad sides to technology but those are more of social problems than ones of technology).

On the other hand, strong AI implies that those, correctly-programmed, digital computers have cognitive states and from that, it follows that programs explain cognition. This is something that Turing strongly believed in.

And here is how Searle refuted this idea:

Imagine you have a person inside a room. That person doesn’t speak Chinese* nor has any, whatsoever, understanding of it. Now outside of the room is a native Chinese speaker, he/she writes down something in Chinese and passes the paper to the person inside of the room.

The person inside the room has been given some rules (think of computer instructions) on how to react to every input (Chinese characters) that he/she gets. So the person sees “squiggle squiggle” and just blindly following the rules, without any understanding, forms the output that is also in the correct “squiggle squiggle” form and passes it outside of the room.

From the point of a native speaker, if the rules were properly engineered, it seems like he/she is communicating with another native Chinese speaker, but we know that that’s not the case and that the result was created by a blind symbol manipulation without any understanding of the Chinese language.

*- I’m aware that there is no such thing as “Chinese” that would be equivalent to me saying ‘I speak Slavic’ instead of ‘I speak Serbian’. For my Chinese friends out there suppose we implicitly think of Mandarin Chinese when we say Chinese.

So the person in this thought experiment took the role of the computer and we see that without any understanding it is able to imitate a native Chinese speaker. We are allowed to do this since the “hardware implementation” does not matter — mind exists independent of its implementation details as per the theory we’re trying to refute.

And that is precisely what the Turing Test is all about — imitation.

Now there are some BUTs here that may pop into your mind. It’s one thing to make a thought experiment where the Chinese output that the person produces is perfect, but it’s a different thing whether that’s at all possible to do following a finite list of instructions.

My belief is the following. If and when we engineer a machine who’s behavior is truly indistinguishable from humans it will probably have to use some “novel” technology and it will produce some form of cognition in the process.

I don’t believe that it can be done using symbol manipulation that Turing proposed, which we use even today, including the learning methods that I’m aware of. We may get to ‘hey close enough’ in a 20-minute conversation, which would, from an economic standpoint, be damn useful! but that’s not it. That’s not the real AI that we’re trying to build.

Reiterating, what Searle was trying to say is that dualism* is a wrong idea to start with. You have to duplicate the causal powers of the brain in order to achieve cognition, running a program is just not enough. This idea, although it’s in conflict with a mathematician inside of me who’s prone to abstraction and simplification, resonates with me.

*- theory that we can split the mind and the brain in the same way that we split hardware and software. The opposite theory is that you can’t split them — mind requires this specific implementation that we call the brain.

I’ll leave you with an interesting formulation that Searle made:

No one supposes that computer simulations of a five-alarm fire will burn the neighborhood down or that a computer simulation of a rainstorm will leave us all drenched. Why on earth would anyone suppose that a computer simulation of understanding actually understood anything?

Now let’s see some of the most famous proposals on how to replace the Turing Test with something better.

Total Turing Test (TTT)

Dubbed like this by Stevan Harnad, the earliest mentions of TTT I’ve found originate in his paper Minds, Machines and Searle (1989), but was covered at least 1 more time in his Other Bodies, Other Minds paper (1991).

The basic idea of TTT is this, TT only tests for the linguistic capacities of humans whereas TTT also includes our “robotic” capacities. Moving, interacting with physical objects in our environment, seeing, and all the other sense-modalities (hearing, etc.) we have.

Or as Harnad puts it:

It is hard to imagine, for example, that a TT candidate could chat with you coherently about the objects in the world till doomsday without ever having encountered any objects directly — on the basis of nothing but “hearsay,” so to speak. Some prior direct acquaintance with the world of objects through sensorimotor (TTT) interactions with them would appear to be necessary in order to ground the candidate’s words in something other than just more words.

If by the word “world” he also thinks of the simulated world I’m happy. Now this may sound obvious and you may ask yourself why didn’t Turing include all of this, why did he abstract these “details” away?

Surely 1 reason was this:

Bodily appearance clearly matters far less now than it might have in Turing’s day

Back in the day, they weren’t accustomed to seeing robots everywhere. Nowadays in the post Star Wars era, we got really accustomed. We see them in cinemas, on television, in our favorite movies and series and we are quite comfortable with the idea that robots can have intelligence.

Turing thought that if the judge was to see the robot it needed to look exactly like a human (to have the same skin as we do, etc.) — which was an unfair and redundant constraint according to Turing and rightfully so, but we don’t hold the same biases anymore.

There is 1 more assumption that Turing implicitly made and Harnad made that explicit in his paper:

It is unlikely that our chess-playing capacity constitutes an autonomous functional module, independent of our capacity to see, move, manipulate, reason, and perhaps even to speak. The TT itself is based on the pre-emptive assumption that our linguistic communication capacity is functionally isolable.

The hypothesis that Harnad is making is that it may not even be possible to pass the TT without passing TTT, because in order to develop linguistic capability we’d have to develop the robotic capability in the first place:

It is highly unlikely that our linguistic capacities are independent of our robotic capacities: Successfully passing the teletype version of the Turing Test alone may be enough to convince us that the candidate has a mind (just as written correspondence with a never-seen pen-pal would), but full robotic capacities (even if only latent ones, not directly exhibited or tested in the TT) may still be necessary to generate that successful linguistic performance in the first place.

A further benefit of introducing TTT is that we’re more likely to converge to a “right” solution in the sense that the machine will more likely have a mind if it passes the TTT.

But, an interesting question is what’s the evolutionary advantage of having a mind in the first place? How did evolution produce the mind? Natural selection “knows” as much as the judge in the Turing Test. Better performing individuals have a greater chance of survival that’s it.

Then why aren’t we all mindless robots with amazing capabilities?

Harnad as well as Turing both acknowledge that there is no way to know whether others (humans or robots) have a mind (an age-old problem, known in philosophy as the “other minds” problem) and so accepts TTT as the best thing we can do at the moment.

You know that one:

“If it looks like a duck, walks like a duck, quacks like a duck…it’s a duck”

But there is always this one guy that still doesn’t buy it. And it’s Searle again.

Searle’s paper, discussed in the previous section, also had an objection against TTT, called “The Robot Reply (Yale)”. The argument goes like this:

By adding cameras and other sensors and actuators (fancy name for anything that moves objects in the physical world like a robotic hand) to a robot you don’t really add anything new in the sense of understanding.

The processing part of the robot still only sees a stream of meaningless formal symbols (this time coming from the camera or some other input sensor and not from a native Chinese speaker) which it just mindlessly manipulates and then sends back to actuators in the form of yet another meaningless stream of symbols.

Harnad's contra-argument is that this time Searle cannot do the job of a transducer (which converts, in the example of the camera, photons into “symbols”) and thus the “systems reply” holds — namely that although the person in the Chinese room experiment has no understanding, the system as a whole has.

I think I’m with Searle on this one but I still do think that from an engineering standpoint we want to go for TTT and develop amazing robots (even if we don’t unlock a true cognition by doing so).

Now if anyone is pushing robots beyond our imagination that’s Boston Dynamics — I guess most of you heard of them — take a look at this video:

A while back I listened to Marc Raibert, Boston Dynamic’s CEO and he said at that point in time that they had 0 machine learning in their systems.

Now I don’t know whether that’s true, but if it is, it’s utterly amazing how much can we accomplish by only using clever engineering and a bunch of domain expertise.

Fun fact: There is also a TTTT (Total Total Turing Test), where you also have to perfectly imitate the neuromolecular structure of humans.

And now for the final test!

Lovelace Test (LT)

Written in 2001 by Bringsjord and his colleagues, LT builds on the famous Lady Lovelace’s argument that we covered in the last blog that roughly states the following:

Computers can’t create, they can only do stuff that we program them to do i.e. only the things that we know how to do.

So the informal version of LT is pretty simple:

A machine will pass the Lovelace test if it manages to surprise us.

Now the surprise is a really vague term, so they formalize it explaining that if the architect of that AI agent can’t explain by any possible mean and given enough time how the agent produced some result, we say that the machine has passed the LT test.

They went on and dissected the state-of-the-art creative systems that were available back then, like BRUTUS (this one was developed by the authors of LT!) — which generated “creative” stories and COPYCAT, and they showed that these fail the Chinese Room argument by Searle (that guy again!).

They further took into account that Turing mentioned learning as the potential way to overcome Lovelace’s argument. So they ended up equating learning with deep learning by tackling only neural nets, showing that these, once fully trained, can be considered as a knowledge base, and thus they still don’t originate anything.

That’s an interesting way to look at the neural networks, but what they missed, in my opinion, is that the “creativity” happened during the training. We really can’t explain how the knowledge base was built in the first place, especially if we use some kind of stochasticity (randomness) in the optimization procedure*, which is a usual thing to do.

*- neural networks are usually “trained” i.e. their “neurons” are slightly modified in this procedure we call the optimization procedure, where they get better at the task of interest.

As a final desperate attempt to figure out how we can “engineer” these LT-passing machines they considered something called the oracle machines.

These are fictional machines (consider them as a helpful imaginary construct) that can basically solve problems that are proven to be unsolvable (like the halting problem, etc.) by the Turing machines *— we say that they can go past the so-called Turing Limit.

*- without going deeper into the theory of what Turing machines are, consider your computer as one specific implementation of these Turing machines.

Most popular of these oracle machines are analog chaotic neural nets, trial-and-error machines, and Zeus machines.

In an attempt to make it less abstract let me briefly explain the Zeus machine. It basically enumerates all of the natural numbers in say 1 second — that’s what makes Zeus an oracle machine. It outputs 0 in 1/2 second, 1 in 1/4 second, 2 in 1/8 second, etc. and if you’re familiar with this sequence it adds up to 1. So 1 second elapses and you’ll end up with all of the natural numbers “printed to your screen” (better crank up those fps). Khm. Yeah.

So using this one you can solve otherwise computationally unsolvable problems. But even Zeus will fail in the Chinese Room experiment. Imagine Searle mindlessly manipulating Chinese symbols only this time he does it unbelievably fast, that’s it. But where does understanding emerge?

Well, nowhere.

They wrap up the paper saying well this didn’t go by the plan. 😃 And suggest that the possible way may be hidden in the agent causation — which would imply that certain mental activities (like when you make a decision/free will) happen without ordinary physical causal chain ever happening.

But I think we went too far into metaphysics at this point. Maybe they’ve “proved” that pure computation can not explain how the mind works and we’ll have to seek answers in quantum computing or these microtubules that Roger Penrose believes may hide some hints.

But let's step back and revisit the original question.

Have machines managed to surprise us?

Well starting in 1997 DeepBlue has beaten the world grandmaster in chess, Garry Kasparov, for the first time in history. And those chess-engines have gotten much better ever since. They are so good now that Magnus Carlsen, who is the current chess champion of the world, compares to them the same way as me and you compare to Magnus (except if you’re a grandmaster).

Take a look at this amazing video with Garry Kasparov where you’ll hear him explain how the whole story with DeepBlue went and much more:

But I hear you say “but DeepBlue used the brute force approach”, so the “architect” would be able to exactly tell you what happened and thus DeepBlue wouldn’t pass the LT by definition. And I’d agree.

But what about AlphaGo or better yet AlphaZero and AlphaStar?

I don’t know, I’d like to know whether it’s possible for David Silver and his crew to know exactly why their algorithm did the “Move 37” for example? The famous move that AlphaGo did that made all of the best Go players in the world go what??? And it won the game, of course.

I listened to David’s talk here, but I got the impression that he hasn’t got the faintest clue — not to say anything bad about David or anything he’s one of my heroes.

I know, that’s still not good enough for LT. Could they, the DeepMind team that was behind this marvelous feat of engineering, explain how it exactly came to be? I’ve got some friends at DeepMind I guess I’ll have to ask or hopefully somebody more knowledgeable than I am could comment on this blog, I’d really like to know.

I want to finish this blog series by recommending a movie. I know. Plot twist. You really expected to get some definite answers here, right?

I really can’t recommend it highly enough (please go and watch it!)

And as a fun fact, the algorithm that played Go against Lee Sedol lost 100–0 against the next iteration of the very same algorithm, and this went on in cycles.

They stopped developing it further because it would have turned into a massive black hole that would gulp down the whole Milky Way galaxy.

Just kidding it just wasn’t economical anymore, but what’s interesting is that they didn’t see any diminishing returns, the algorithm just kept on improving.

Than came AlphaZero which was even better and trained solely using self-play, no human intervention, no domain knowledge injection from human experts. And the very same algorithm generalizes to chess, Go and shogi. MuZero is their latest flagship (afaik)— and it’s even better than AlphaZero and can generalize to Atari games.

Nothing hints of AGI (artificial general intelligence) more than the stuff that DeepMind develops. These algorithms are, at the moment, deployed only in these constrained environments, but still, I’m damn impressed.

Final thoughts v2.0

Phew, we went all the way from 1950 to 2020, I hope you liked the journey!

I just want to wrap this journey up by mentioning that there are some concrete tests that we used and still use to test the machine intelligence.

• The Loebner prize — which is most similar to what the original Imitation Game was, but the competitors ended up using pure trickery to win, so the community lost interest and so this competition died off.
• Alexa prize — organized by Amazon obviously, is another test where the goal is to have a natural voice conversation with a bot for X minutes. But it’s only for students. It’s like a really weak form of TTT.
• Hutter prize — which is totally non-intuitive at a first glance, where the goal is to do a lossless compression of some knowledge base such as the Wikipedia content — Mark Hutter says this correlates with intelligence.
• Winograd schema challenge — where you’re trying to solve problems that require common-sense knowledge.

Go ahead and explore these at your own pace.

Also, take a look at this video covered by Lex Fridman, who probably has the best tech/AI podcast in the world at the moment. I’ve been following him since 2018. It inspired me to finally take the Turing’s paper “Computing Machinery and Intelligence” off the backlog and read it!

In one of the next blogs, I’ll cover Francois Chollet’s “On the Measure of Intelligence” that’s one of the most beautiful tests that I’ve recently seen.

Take care, stay safe, and never stop learning and improving! ❤️

If there is something you would like me to write about — write it down in the comment section or send me a DM, I’d be glad to write more about maths, ML, deep learning, software, landing a job in a big tech company, preparing for ML summer camps, electronics (I actually officially studied this beast), etc., anything that could help you.

Also feel free to drop me a message or:

1. Connect and reach me on 💡 LinkedIn and Twitter
2. Subscribe to my 🔔 YouTube channel for AI-related content️
3. Follow me on 📚 Medium and 💻GitHub
4. Subscribe to my 📢 monthly AI newsletter and join the 👨‍👩‍👧‍👦 Discord community!

And if you find the content I create useful consider becoming a Patreon!

Much love ❤️