Ever since I bikepacked through the surreal, high-altitude landscapes of Ladakh, I’ve been chasing that same feeling, of being dwarfed by wind-carved canyons, of coasting through meadows that roll into desert valleys, of waking up at elevation with the crisp bite of thin air in my lungs. For years, that pursuit took me through the American West, but nothing quite captured Ladakh’s raw magic. Until now.

California, Refracted

I spent three days bikepacking near Independence and Lone Pine, in California’s Eastern Sierra with my friend Chandu. We followed the Owens Valley Ramble route from bikepacking.com. The moment I caught the first view though, I thought: this might be it. I might have found the Ladakh of California.

The landscape here is compressed into an astonishing density. Within a few hours of pedaling, we passed through arid desert floors, climbed into grass-carpeted meadows, traced the base of jagged granite walls, and stared out at snow-capped peaks rising abruptly from the valley. The Owens Valley, marked by the Sierra Nevada to the west and the Inyo Mountains to the east, might be one of the most geographically dramatic places in the United States, or at least in the lower 48 states.

A Landscape in Tension

Part of what makes this place so compelling is its geologic drama. The Eastern Sierra region lies along a massive fault zone. I could see how the eastern face of the valley rises sharply from the desert floor, while the western slope descends more gradually. My friend Curtis, who researches fault lines, once explained that the Sierras themselves occupy a giant fault block, making the Owens Valley a sunken rift.

The valley floor sits at about 4,000 feet, which is where we spent most of the time (except for a day hike to an alpine lake). Just a short distance away, peaks like Mount Whitney soar to over 14,000 feet, creating one of the greatest vertical reliefs in North America. We even saw a road go about the near-vertical cliff, apty named Whitney portal, as if portaling one from the desert floor to the mountain peaks instantly.

Over time, glaciers scoured the high elevations, carving U-shaped valleys and cirques into the granite. Meltwater and snowmelt continue to feed creeks that tumble down to the desert below, shaping canyons and alluvial fans that resemble something extraterrestrial. It felt like wind, water, and time had collaborated to create this rare juxtaposition of alpine and desert, verdant and barren, all in one continuous view.

Everything Aligned

The landscape alone would have made this trip memorable. But what made it feel magical was that somehow nothing went wrong.

On past trips, something always goes wrong. Sometimes multiple things. I’ve broken a bottom bracket in the middle of a desert during a heatwave in Utah, forcing me to ditch my bike and hike out 18 miles through the night. I’ve battled headwinds so strong near Mojave that I only managed 20 miles in a day instead of my usual 60. I’ve faced impassable terrain near the Salton Sea, and I’ve been chewed alive by mosquitoes in Oregon.

But this time? Nothing. No weather surprises. No mechanicals. Not even any flat tyres. Even the notorious valley winds that I heard gust up to 50 kmph decided to take a break for us. The bugs? Barely a nuisance, just a few hovering at dusk. We found frequent river crossings to refill our bottles and cool our bodies. To top it all, even my apparel choice was perfect. My cotton kurta sheltered me from the Sun and let me feel the breeze too. It was almost unsettling how smoothly everything went. We kept wondering whether the odds are steadily stacking up against us to hit us with a bang in the end (they din’t).

Ride or Glide

One of the best parts of the trip was biking with my friend Chandu. He’s one of the very few people I never have to worry about on these kinds of adventures. Equally clueless, equally happy-go-lucky, and equally okay with adapting plans on the fly — or tossing them out altogether. He doesn’t mind discomfort (although nothing tested us on this trip), and he doesn’t fuss over details. Which makes him the perfect riding partner in places like this.

There’s a particular joy in moving through wild landscapes with someone who shares your rhythm, not just in pace, but in spirit. No debates over where to camp, no panic when a trail vanishes, no complaints when breakfast is made of bland, tasteless mush of oats. Just easy camaraderie, shared awe, and the occasional spontaneous yodel.

Ironman

Yodeling wasn’t our only brush with cinematic drama. There were movies, too! Just outside Lone Pine lies the Alabama Hills- a strange and beautiful cluster of rounded granite boulders set against the jagged backdrop of the Sierras. We biked through them, soaking in the strange, sculptural terrain.

What made it even more lively was realizing how often this landscape has been used in movies. I found it funny that the Alabama Hills have stood in for everywhere except America: Afghanistan in Ironman, Mexico in countless Westerns, even distant planets in sci-fi flicks. Something about the terrain is just ambiguous enough I guess that filmmakers can turn it into whatever they need.

A Familiar Echo

The Eastern Sierras may not be Ladakh — but for a few days, they felt like its echo. A version in a different register, but still part of the same song.

If you’ve been chasing that same wild, high-desert solitude and you live on the West Coast of the USA, you might not have to fly halfway across the world. It’s waiting just a few hours from home.

Finding Ladakh in the Sierras was originally published in Bicycle Touring on Medium, where people are continuing the conversation by highlighting and responding to this story.

Since graduating from college, I’ve tracked my career along a single axis: my impact on sustainability. There have certainly been ups and downs — my first job was in oil and gas extraction, for example — but from my perspective, the journey has been steadily upward. Today, I work in a climate-tech startup building AI systems for helping companies measure and reduce their carbon emissions.

Lately, I’ve been thinking a lot about AI: the transformative impact it has had on my work, the broad excitement surrounding it, and the fact that nearly everything I do now involves it.

Building AI technologies also seems to attract incredible talent. Many of my close friends have joined the behemoths or launched their own ventures. Working on something so transformational seems extremely exciting to me — even if it’s not directly focused on sustainability or climate. The astronomical salaries are a draw, too. I tell myself I don’t care about the money, but when I hear the TC figures, I must admit I get second thoughts.

Then I read devastating news about the climate crisis, or the recent dismantling of climate policy, and it jolts me — “Focus on sustainability, Krishna. There’s not enough time.”

I’m currently reading Doing Good Better by William MacAskill, and there’s a compelling argument in there about “earning to give.” It’s the idea that I can still have an impact on sustainability — even if I’m not working on it directly — by donating large sums to effective causes. MacAskill talks about Dr. Greg Lewis, who donates significant amounts while working as a doctor in Britain, even though he’s not directly involved in medical efforts in Africa. Should I do the same?

In my case, however, I believe I can have a greater impact by working directly on the cause:

1. I’m uniquely positioned to contribute because of my climate science knowledge from my PhD and my data science skills from past work.
2. I have an extremely supportive family — my wife and parents.
3. I think I’ll be more motivated and work harder if I’m contributing directly.
4. Considering the counterfactuals, I believe I’ll do a better job building sustainability-related technologies than someone else, but a worse job building general AI than whoever would take my place.
5. I’m already earning quite a bit and giving proportionally.

I waver on that last point sometimes. So, today — on Earth Day — I’m making a pledge:

I pledge to work on sustainability for the foreseeable future.

Announcing something so broadly can seem boastful. I believe, if anything, it’s the opposite. I once heard a talk by Rajinikanth, my childhood favourite hero, where he noted that human minds are fickle. What’s in my mind today can change tomorrow. But words — once spoken — cannot be undone. So I’m asking for help from all of you to keep me honest. Please remind me if you see me drifting from this pledge:

I pledge to work on sustainability for the foreseeable future.

Will you join me?

Earth day, AI, Effective Altruism, and Rajinikanth was originally published in Age of Awareness on Medium, where people are continuing the conversation by highlighting and responding to this story.

Ground truth is never perfect. From scientific measurements to human annotations used to train deep learning models, ground truth always has some amount of errors. ImageNet, arguably the most well-curated image dataset has 0.3% errors in human annotations. Then, how can we evaluate predictive models using such erroneous labels?

In this article, we explore how to account for errors in test data labels and estimate a model’s “true” accuracy.

Example: image classification

Let’s say there are 100 images, each containing either a cat or a dog. The images are labeled by human annotators who are known to have 96% accuracy (Aᵍʳᵒᵘⁿᵈᵗʳᵘᵗʰ). If we train an image classifier on some of this data and find that it has 90% accuracy on a hold-out set (Aᵐᵒᵈᵉˡ), what is the “true” accuracy of the model (Aᵗʳᵘᵉ)? A couple of observations first:

1. Within the 90% of predictions that the model got “right,” some examples may have been incorrectly labeled, meaning both the model and the ground truth are wrong. This artificially inflates the measured accuracy.
2. Conversely, within the 10% of “incorrect” predictions, some may actually be cases where the model is right and the ground truth label is wrong. This artificially deflates the measured accuracy.

Given these complications, how much can the true accuracy vary?

Range of true accuracy

The true accuracy of our model depends on how its errors correlate with the errors in the ground truth labels. If our model’s errors perfectly overlap with the ground truth errors (i.e., the model is wrong in exactly the same way as human labelers), its true accuracy is:

Aᵗʳᵘᵉ = 0.90 — (1–0.96) = 86%

Alternatively, if our model is wrong in exactly the opposite way as human labelers (perfect negative correlation), its true accuracy is:

Aᵗʳᵘᵉ = 0.90 + (1–0.96) = 94%

Or more generally:

Aᵗʳᵘᵉ = Aᵐᵒᵈᵉˡ ± (1 — Aᵍʳᵒᵘⁿᵈᵗʳᵘᵗʰ)

It’s important to note that the model’s true accuracy can be both lower and higher than its reported accuracy, depending on the correlation between model errors and ground truth errors.

Probabilistic estimate of true accuracy

In some cases, inaccuracies among labels are randomly spread among the examples and not systematically biased toward certain labels or regions of the feature space. If the model’s inaccuracies are independent of the inaccuracies in the labels, we can derive a more precise estimate of its true accuracy.

When we measure Aᵐᵒᵈᵉˡ (90%), we’re counting cases where the model’s prediction matches the ground truth label. This can happen in two scenarios:

1. Both model and ground truth are correct. This happens with probability Aᵗʳᵘᵉ × Aᵍʳᵒᵘⁿᵈᵗʳᵘᵗʰ.
2. Both model and ground truth are wrong (in the same way). This happens with probability (1 — Aᵗʳᵘᵉ) × (1 — Aᵍʳᵒᵘⁿᵈᵗʳᵘᵗʰ).

Under independence, we can express this as:

Aᵐᵒᵈᵉˡ = Aᵗʳᵘᵉ × Aᵍʳᵒᵘⁿᵈᵗʳᵘᵗʰ + (1 — Aᵗʳᵘᵉ) × (1 — Aᵍʳᵒᵘⁿᵈᵗʳᵘᵗʰ)

Rearranging the terms, we get:

Aᵗʳᵘᵉ = (Aᵐᵒᵈᵉˡ + Aᵍʳᵒᵘⁿᵈᵗʳᵘᵗʰ — 1) / (2 × Aᵍʳᵒᵘⁿᵈᵗʳᵘᵗʰ — 1)

In our example, that equals (0.90 + 0.96–1) / (2 × 0.96–1) = 93.5%, which is within the range of 86% to 94% that we derived above.

The independence paradox

Plugging in Aᵍʳᵒᵘⁿᵈᵗʳᵘᵗʰ as 0.96 from our example, we get

Aᵗʳᵘᵉ = (Aᵐᵒᵈᵉˡ — 0.04) / (0.92). Let’s plot this below.

Strange, isn’t it? If we assume that model’s errors are uncorrelated with ground truth errors, then its true accuracy Aᵗʳᵘᵉ is always higher than the 1:1 line when the reported accuracy is > 0.5. This holds true even if we vary Aᵍʳᵒᵘⁿᵈᵗʳᵘᵗʰ:

Why does this happen? When errors are independent, a model with Aᵐᵒᵈᵉˡ > 0.5 tends to be correct on some examples where the ground truth is wrong. Remember that our derived expression for Aᵗʳᵘᵉ accounts for both:

1. Cases where model is penalized unfairly (labels wrong, model right)
2. Cases where model gets credit unfairly (labels wrong, model wrong)

For Aᵐᵒᵈᵉˡ > 0.5 and Aᵍʳᵒᵘⁿᵈᵗʳᵘᵗʰ > 0.5, the first effect tends to outweigh the second when errors are truly independent.

Error correlation: why models often struggle where humans do

The independence assumption is crucial but often doesn’t hold in practice. If some images of cats are very blurry, or some small dogs look like cats, then both the ground truth and model errors are likely to be correlated. This causes Aᵗʳᵘᵉ to be closer to the lower bound (Aᵐᵒᵈᵉˡ — (1 — Aᵍʳᵒᵘⁿᵈᵗʳᵘᵗʰ)) than the upper bound.

More generally, model errors tend to be correlated with ground truth errors when:

1. Both humans and models struggle with the same “difficult” examples (e.g., ambiguous images, edge cases)
2. The model has learned the same biases present in the human labeling process
3. Certain classes or examples are inherently ambiguous or challenging for any classifier, human or machine
4. The labels themselves are generated from another model
5. There are too many classes (and thus too many different ways of being wrong)

Best practices

The true accuracy of a model can differ significantly from its measured accuracy. Understanding this difference is crucial for proper model evaluation, especially in domains where obtaining perfect ground truth is impossible or prohibitively expensive.

When evaluating model performance with imperfect ground truth:

1. Conduct targeted error analysis: Examine examples where the model disagrees with ground truth to identify potential ground truth errors.
2. Consider the correlation between errors: If you suspect correlation between model and ground truth errors, the true accuracy is likely closer to the lower bound (Aᵐᵒᵈᵉˡ — (1 — Aᵍʳᵒᵘⁿᵈᵗʳᵘᵗʰ)).
3. Obtain multiple independent annotations: Having multiple annotators can help estimate ground truth accuracy more reliably.

Conclusion

In summary, we learned that:

1. The range of possible true accuracy depends on the error rate in the ground truth
2. When errors are independent, the true accuracy is often higher than measured for models better than random chance
3. In real-world scenarios, errors are rarely independent, and the true accuracy is likely closer to the lower bound

Handling Label Noise: How to Measure Real Model Accuracy was originally published in Data Science Collective on Medium, where people are continuing the conversation by highlighting and responding to this story.

Imagine a national park without the crowds.

Imagine a bike ride/tour without cars on the road.

Imagine exploring giant volcanoes, still lakes, buzzing meadows, and hushed forests all in just one ride.

Starting today and for the next few weeks, a once-in-a-year opportunity awaits all bikers in Northern California. The Lassen Volcanic National Park has opened Highway 89 (the only road through the park) exclusively to hikers and bikers. Read on to learn more about my bikepacking trip through the park last year.

Kohm Yah-mah-nee or snow mountain is exactly how one might describe Lassen Peak. And the Kohm Yah-mah-nee visitor center is where we started our 3-day bikepacking tour. We biked through Lassen National Park and Lassen National Forest, briefly pausing at small towns along the way. Good thing we started at the visitor center — the park rangers helped us plan the route, and potentially saved us from dehydrating on the very first day!

“No, you can’t drink water from the lakes”

When we were at the visitor center getting ready for our ride, we saw clear blue skies and warm weather of 25C (or 80F). We saw plenty of streams crisscrossing the mountainous terrain in front of us. The map also showed a few lakes on our route (the closest one being just 10 miles away) which led us to believe we would have plenty of water refilling points.

However, when we discussed our route with the park ranger, she immediately warned us that it would be hard to find water for the first half of our day. I asked her about the lakes on our way pointing to the map, and she said, “No, you can’t drink water from the lake. They are frozen under at least 6 feet of snow.” Confused, I looked out of the window at the warm weather and wondered if she was joking.

Not wanting to risk running out of water (knowing how that feels) we filled up all our bottles and set away on our ride.

Good that we listened! Within a few miles, the road climbed, and with it, the terrain and the weather changed drastically. Bustling meadows gave way to snow-covered terrain. Warm weather gave way to cool winds. Small streams gave way to frozen lakes. And gentle slopes gave way to volcanic mountains … many of them.

“That’s not Lassen … is it?”

With every switchback on the winding climb, we saw a new peak and my friend Abby wondered “That’s not Lassen … is it?” Since we hadn’t done any Instagram research, we didn’t know what Lassen looked like. And all peaks looked about the same — a sharp snow-covered peak with steep slopes arising from relatively flat ground covered with black volcanic rocks. Except one.

When we finally saw Lassen, we immediately knew that was it. It was so gigantic, that it dwarfed all the other peaks surrounding us. It was so tall that a few clouds were permanently stuck on it. For a few minutes, Abby and I stopped riding and just stared at it, in awe of its beauty. We were oblivious to the violent volcanic activities hidden beneath its peaceful outer appearance.

Uh Oh

Lassen last erupted in 1915 when rocks and ash rained down from the sky, illuminated by the fiery glow of the molten lava. This eruption dramatically reshaped the landscape, creating today's rugged terrain. The volcano is dormant but active today.

The 1915 eruption of Lassen Peak was not just a local event; its effects were felt far and wide. At the Loomis Museum, we learned that the explosion hurled rock fragments (like the black rock pictured below) as far as 20 miles away. The ash cloud reached as far as 200 miles to the east. In some areas, the ash layer was several inches thick. The roar of the eruption was reportedly heard up to 150 miles away, a testament to the immense power unleashed by Lassen Peak.

Long before the arrival of European settlers and the eruption of 1915, the area now known as Lassen Volcanic National Park was home to Native American tribes, including the Atsugewi and Maidu. These tribes lived in harmony with the land, utilizing its resources for food, shelter, and spiritual practices.

Today, we can witness the remnants of the catastrophic eruption. The ground is strewn with black volcanic rocks, a stark contrast to the green forested areas. The Devastated Area, aptly named for the destruction wrought by the 1915 eruption, is a prime example. There, we saw trees flattened by the blast, and large volcanic boulders scattered across the landscape. The park’s hydrothermal areas, such as Bumpass Hell, Sulphur Works, and Devil's Kitchen, continue to steam and bubble, a living reminder of the volcano’s power. A series of lava domes formed by the previous eruptions are now at Chaos Crags. These features are not just relics of the past; they are active reminders of the park’s volcanic activity. They also serve as active habitats for a variety of flora and fauna.

Animal Allure

Plenty of animals kept us company during the tour, day and night. From tiny birds to horses, we saw many animals. It reminded me of my friend Blake who used to rate his bike rides not by the miles ridden or elevation climbed, but by the number of animal sightings. And I think Blake would rate our ride highly.

Seen below: Horses, ducks, deer, and blue jays we encountered on our ride.

The route

lassen loop

Day 1: We started at the Kohm Yah-mah-nee visitor center and rode north on Highway 89 until Old-station where we camped at Hat Creek Campground for the night and enjoyed a refreshing dip at Hat Creek.

Day 2: We went southeast on the Volcanic Legacy Scenic Byway (Highway 44) before we turned right on Mooney Road (Forest Route 21) until Westwood. We then turned west on Highway 36. We camped on the shore of Lake Almanor in a dispersed campsite.

Day 3: Finally, we continued west on Highway 36 and made it back to the visitor center on the third day at approximately noon. Highway 36 was a little busy with vehicles and semi-trailers. At times, it had narrow shoulders too, so we had to be careful on the road.

Alternate routes:

1. One could do just the Highway 89 section from the Kohm Yah-mah-nee visitor center to the Loomis museum and back. It is about 60 miles long out and back and free of cars.
2. On day 2 from Old Station to Lake Almanor, we considered a different route via Butte Lake, Soliver Lake, and Echo Lake. It was through the Lassen National Forest on unpaved Forest roads. However, a few miles into Forest Route 32N21, we saw our thin tires biting into the loose soil on the fire road. So we decided against it and took the longer way to Lake Almanor via Westwood.

This tour was partially supported by Vargo Outdoors.

Pedaling Through Fire and Ice: Bikepacking Lassen Volcanic National Park on opening day was originally published in Bicycle Touring on Medium, where people are continuing the conversation by highlighting and responding to this story.

Hidden correlations can mislead optimization strategies

p99, or the value below which 99% of observations fall, is widely used to track and optimize worst-case performance across industries. For example, the time taken for a page to load, fulfill a shopping order or deliver a shipment can all be optimized by tracking p99.

While p99 is undoubtedly valuable, it’s crucial to recognize that it ignores the top 1% of observations, which may have an unexpectedly large impact when they are correlated with other critical business metrics. Blindly chasing p99 without checking for such correlations can potentially undermine other business objectives.

In this article, we will analyze the limitations of p99 through an example with dummy data, understand when to rely on p99, and explore alternate metrics.

The Correlation Conundrum

Consider an e-commerce platform where a team is tasked with optimizing the shopping cart checkout experience. The team has received customer complaints that checking out is rather slow compared to other platforms. So, the team grabs the latest 1,000 checkouts and analyzes the time taken for checking out. (I created some dummy data for this, you are free to use it and tinker with it without restrictions)

import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
sns.set(style="ticks", font_scale = 1)
order_time = pd.read_csv('https://gist.githubusercontent.com/kkraoj/77bd8332e3155ed42a2a031ce63d8903/raw/458a67d3ebe5b649ec030b8cd21a8300d8952b2c/order_time.csv')
fig, ax = plt.subplots(figsize=(4,2))
sns.histplot(data = order_time, x = 'fulfillment_time_seconds', bins = 40, color = 'k', ax = ax)
print(f'p99 for fulfillment_time_seconds: {order_time.fulfillment_time_seconds.quantile(0.99):0.2f} s')

As expected, most shopping cart checkouts seem to be completing within a few seconds. And 99% of the checkouts happen within 12.1 seconds. In other words, the p99 is 12.1 seconds. There are a few long-tail cases that take as long as 30 seconds. Since they are so few, they may be outliers and should be safe to ignore, right?

Now, if we don’t pause and analyze the implication of the last sentence, it could be quite dangerous. Is it really safe to ignore the top 1%? Are we sure checkout times are not correlated with any other business metric?

Let’s say our e-commerce company also cares about gross merchandise value (GMV) and has an overall company-level objective to increase it. We should immediately check whether the time taken to checkout is correlated with GMV before we ignore the top 1%.

from matplotlib.ticker import ScalarFormatter
order_value = pd.read_csv('https://gist.githubusercontent.com/kkraoj/df53cac7965e340356d6d8c0ce24cd2d/raw/8f4a30db82611a4a38a90098f924300fd56ec6ca/order_value.csv')
df = pd.merge(order_time, order_value, on='order_id')
fig, ax = plt.subplots(figsize=(4,4))  
sns.scatterplot(data=df, x="fulfillment_time_seconds", y="order_value_usd", color = 'k')
plt.yscale('log')
ax.yaxis.set_major_formatter(ScalarFormatter())

Oh boy! Not only is the cart value correlated with checkout times, it increases exponentially for longer checkout times. What’s the penalty of ignoring the top 1% of checkout times?

pct_revenue_ignored = order_value.loc[order_time.fulfillment_time_seconds>order_time.fulfillment_time_seconds.quantile(0.99), 'order_value_usd'].sum()/order_value.order_value_usd.sum()*100
print(f'If we only focussed on p99, we would ignore {pct_revenue_ignored:0.0f}% of revenue')
## >>> If we only focussed on p99, we would ignore 27% of revenue

If we only focused on p99, we would ignore 27% of revenue (27 times greater than the 1% we thought we were ignoring). That is, p99 of checkout times is p73 of revenue. Focusing on p99 in this case inadvertently harms the business. It ignores the needs of our highest-value shoppers.

df.sort_values('fulfillment_time_seconds', inplace = True)
dfc = df.cumsum()/df.cumsum().max() # percent cumulative sum
fig, ax = plt.subplots(figsize=(4,4))
ax.plot(dfc.fulfillment_time_seconds.values, color = 'k')
ax2 = ax.twinx()
ax2.plot(dfc.order_value_usd.values, color = 'magenta')
ax.set_ylabel('cumulative fulfillment time')
ax.set_xlabel('orders sorted by fulfillment time')
ax2.set_ylabel('cumulative order value', color = 'magenta')
ax.axvline(0.99*1000, linestyle='--', color = 'k')
ax.annotate('99% of orders', xy = (970,0.05), ha = 'right')
ax.axhline(0.73, linestyle='--', color = 'magenta')
ax.annotate('73% of revenue', xy = (0,0.75), color = 'magenta')

Above, we see why there is a large mismatch between the percentiles of checkout times and GMV. The GMV curve rises sharply near the 99th percentile of orders, resulting in the top 1% of orders having an outsize impact on GMV.

This is not just an artifact of our dummy data. Such extreme correlations are quite common. For example, the top 1% of Slack’s customers account for 50% of revenue. About 12% of UPS’s revenue comes from just 1 customer (Amazon).

A Balanced Approach

To avoid the pitfalls of optimizing for p99 alone, we can take a more holistic approach.

One solution is to track both p99 and p100 (the maximum value) simultaneously. This way, we won’t be prone to ignore high-value users.

Another solution is to use revenue-weighted p99 (or weighted by gross merchandise value, profit, or any other business metrics of interest), which assigns greater importance to observations with higher associated revenue. This metric ensures that optimization efforts prioritize the most valuable transactions or processes, rather than treating all observations equally.

Finally, when high correlations exist between the performance and business metrics, a more stringent p99.5 or p99.9 can mitigate the risk of ignoring high-value users.

Recap

It’s tempting to rely solely on metrics like p99 for optimization efforts. However, as we saw, ignoring the top 1% of observations can negatively impact a large percentage of other business outcomes. Tracking both p99 and p100 or using revenue-weighted p99 can provide a more comprehensive view and mitigate the risks of optimizing for p99 alone. At the very least, let’s remember to avoid narrowly focusing on some performance metric while losing sight of overall customer outcomes.

Written with the help of Perplexity (for definitions and background research, chat here) and ChatGPT (for spellcheck, chat here).

Perils of Chasing p99 was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Paying attention to autocorrelation in data can help you build better predictive models

First, a quiz.

In the two examples below, is the reported accuracy trustable?

1. Shivram gets a brilliant idea to predict heart rate just from iPhone movements. He collects time-synchronized data from iPhone movements and heart rate from Apple watch from thousands of consenting users. He then splits the data randomly second by second into training, validation, and test sets. After he is happy with his model, he reports that he is able to predict heart rate from iPhone movements with a whopping 98% accuracy on the test set!
2. Abhilash wishes to use satellite imagery to find locations of forests. He obtains some training data of satellite images and human-drawn geolocated maps of forests. He then splits the pixels randomly into training, validation, and test sets. After he is happy with his model, he reports his test accuracy as 99%!

Is the reported accuracy trustable in the above two cases?

NO!

In this article, we will learn why they aren’t. We will also learn some basic pre-processing principles one can follow to avoid such pitfalls in the future.

Why care if data is i.i.d.?

Independent and identically distributed data (i.i.d.) have a lot of good properties in predictive settings where knowing one data point does not tell you anything about another data point. When we split data for model training, knowing if data is i.i.d. is a must.

Splitting data into training, validation, and test sets is one of the most standard ways to test model performance in supervised learning settings. Even before we get into the modeling (which receives almost all of the attention in machine learning), not caring about upstream processes like where is the data coming from, are they really i.i.d., and how we split them can have bearing consequences on the quality of predictions.

This is especially important when data has high autocorrelation. Autocorrelation among points simply means that the value at a point is similar to values around it. Take temperature for instance. The temperature at any moment is expected to be similar to the temperature in the previous minute. Thus, if we wish to predict temperature, we need to take special care in splitting the data. Specifically, we need to ensure that there is no data leakage between training, validation, and test sets that might exaggerate model performance.

By how much can model performance be exaggerated with information leakage?

After reading the above, it is natural to ask, is this an important enough problem for me to care about? Through an example of highly autocorrelated data, we will see that the answer is certainly yes! We will break the example into two parts. First, we will split the data randomly into training and validation sets and achieve very high accuracy on the validation set. We will then split the data using stratified random sampling, thus reducing information leakage. We will then see how the same model has almost zero accuracy.

Interactive example

If you wish to follow this example interactively, you can use this colab notebook.

Let’s first import the relevant packages.

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns
import sklearn.model_selection
import sklearn.linear_model
import sklearn.ensemble

Let’s make some synthetic data with high autocorrelation in the response variable.

# number of examples in our data
n = int(100*2*np.pi)
# Seed for reproducebility
np.random.seed(4)
# make one feature (predictor)
x = np.arange(n)
# make one response (variable to predict) which has high autocorrelation. Use a
# sine wave.
y =  np.sin(x/n*7.1*np.pi)+np.random.normal(scale = 0.1, size = n)
# merge them into a dataframe to allow easy manipulation later
df = pd.DataFrame({"x":np.array(x), "y":np.array(y), "y_pred":np.nan})
# visualize the response versus feature
sns.set(style = "ticks", font_scale = 1.5)
sns.regplot(x="x",y="y",data=df)

Random splitting of data

Let’s split the data randomly into training and validation set and see how well the model does.

# Use a helper to split data randomly into 5 folds. i.e., 4/5ths of the data is chosen *randomly* and put into the train set, while the rest is put into

# is chosen *randomly* and put into the train set, while the rest is put into

# the validation set.

kf = sklearn.model_selection.KFold(n_splits=5, shuffle=True, random_state=42)

# Use a random forest model with default parameters.

# The hyperaparameter of the model are not important for this example because we

# will use the same model twice- once with data split randomly and (later) with

# data split with stratification

reg = sklearn.ensemble.RandomForestRegressor()

# use k-1 folds to train. Predict on the kth fold and store in the dataframe

for fold, (train_index, test_index) in enumerate(kf.split(df)):

reg.fit(df.loc[train_index, "x"].values.reshape(-1, 1), df.loc[train_index, "y"])

df.loc[test_index, "y_pred"] = reg.predict(df.loc[test_index, "x"].values.reshape(-1, 1))

# visualize true y versus predicted y

fig, ax = plt.subplots(figsize = (5,5))

sns.kdeplot(

data=df, x="y_pred", y="y",

fill=True, thresh=0.3, levels=100, cmap="mako_r",ax=ax

)

ax.set_xlim(-2,2)

ax.set_ylim(-2,2)

ax.set_xlabel(r"y$_{\rm predicted}$")

ax.set_title("Exaggerated predictive ability\nassuming data is i.i.d.")

r2 = sklearn.metrics.r2_score(df.y, df.y_pred)

ax.annotate(f"R$^2$ = {r2:0.2f}", xy =(0.95,0.95), ha = "right", va = "top", xycoords = "axes fraction")

print(f"[INFO] Coefficient of determination of the model is {r2:0.2f}.")

Whoa!! We achieved an R2 of 97%! Seems like our model does a fantastic job in modeling the sinusoidal response function.

But … is the model really able to understand the response function between x and y? Or is it just acting as a nearest-neighbor interpolation? In other words, is the model just cheating by memorizing the training data, and outputting the y value of the nearest training example? Let’s find out by making it hard for the model to cheat.

Stratified splitting of data

Now, rather than splitting the data randomly, we will separate the data into 5 chunks along the x (feature) axis. We will then put 4 chunks into the training data and 1 chunk into the validation set.

By stratifying the data along the feature which is autocorrelated, we respect the non i.i.d. nature of the data.

Let’s see if the model has the same accuracy.

# How many chunks to split data in? 
nbins = 5
df["fold"] = pd.cut(df.x, bins = nbins, labels = range(nbins))

# Split the data into training and validation data based on the chunks.
# Train on 4 chunks, predict on the remaining chunk.
for fold in sorted(df.fold.unique()):
  train_index = df.loc[df.fold!=fold].index
  test_index = df.loc[df.fold==fold].index
  reg.fit(df.loc[train_index, "x"].values.reshape(-1, 1), df.loc[train_index, "y"])
  df.loc[test_index, "y_pred"] = reg.predict(df.loc[test_index, "x"].values.reshape(-1, 1))
# Visualize true y versus precited y.
fig, ax = plt.subplots(figsize = (5,5))
sns.kdeplot(
    data=df, x="y_pred", y="y",
    fill=True, thresh=0.3, levels=100, cmap="mako_r",ax=ax
)
ax.set_xlim(-2,2)
ax.set_ylim(-2,2)
ax.set_xlabel(r"y$_{\rm predicted}$")
ax.set_title("True predictive ability")
r2 = sklearn.metrics.r2_score(df.y, df.y_pred)
ax.annotate(f"R$^2$ = {r2:0.2f}", xy =(0.95,0.95), ha = "right", va = "top", xycoords = "axes fraction")
print(f"[INFO] Coefficient of determination of the model is {r2:0.2f}.")

Now, we see that our model has below-random performance (side-note: wondering how can coefficient of determination be negative? Read more here)! This shows that our initial model was not really using x as an informative predictor for y, rather only to find the nearest x from the training set and spit out the corresponding y. Thus if we are not careful about autocorrelation in our data, we may have exaggerated model performance.

Worse, we may have erroneously inferred the importance of x, and went on to draw several scientific conclusions. Whereas, our model was using x only to interpolate/memorize the response. This is unfortunately not a made-up example. This paper shows that several papers in the geosciences attempting to predict vegetation biomass (similar to Abhilash’s example at the beginning of this article) are riddled with this problem.

Conclusion

Splitting data can have huge consequences. If there is any evidence for data to be autocorrelated or more generally non-i.i.d., stratified splitting or other techniques for decorrelating the data using signal decomposition can be useful. At the very least visualizing your data before jumping in to modeling can be tremendously beneficial. So the next time you meet Shivram, Abhilash, or anyone else who claims to achieve very high modeling performance after randomly splitting their data, you are well equipped to help them come up with better predictive models without exaggerated model performance.

Want to ruin your model? Assume data is i.i.d. was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Bicycling on opening day- pictures, itinerary, and planning tips

TLDR- If you are here for just the links:

1. Check here for the opening date or subscribe to https://tiogawatch.wordpress.com/.
2. The starting point of the ride (free car parking) is Tuolomne Grove Trailhead. No water available.
3. Camp 4 reservations are here (lottery on the previous day only, 10% success rate). If unlucky, camp on Evergreen road or Hardin Flat road anywhere by the roadside (USFS land allows dispersed camping).
4. Strava route

Yosemite has always held a special place in my heart. From the mammoth granite peaks to the crystal clear alpine lakes, from the beautiful wildlife to the giant sequoia trees, from the starry nights to spectacular sunsets, each time I visit the park, it captivates me in a new way.

In the past, I have explored the park by vehicle and on foot. Although I love to bicycle, I had heard that the park was not well-suited for bicyclists because of the large amounts of (speedy) vehicular traffic and because most scenic roads lacked a shoulder. Naturally, I thought I would never get to explore it on my bicycle. Until yesterday.

“Krishna, you should join us for the ride to Tioga”, my friend Sam said, just as he pulled up to me during one of our Friday coffee rides. That was the first time I heard about it, so I enquired a little more to understand the ride. As he explained, everything about the ride seemed enticing- a bicycle ride through one of the most popular roads in Yosemite National Park WITHOUT any vehicles, mild gradients, spectacular views, and some pleasant weather. It didn't take much convincing after that for me to make up my mind.

The itinerary

The plan was fairly simple. We would ride the Tioga Pass Road (also known as Highway 120) on one of the three days reserved for bicyclists only before the road opened to all vehicular traffic (the road is typically closed for six months during the winter because of snow).

1. We drove to the entrance of Yosemite on the day before the opening
2. We camped at Evergreen road which is just a few miles before the entrance to the park. The road was not a National park, but part of a National forest, where dispersed camping was free. But remember to bring your own water!
3. The next day morning, we packed up our tents and drove for approximately 20 minutes to the Tuolumne Grove Trailhead. That was the starting point of our bicycle ride.
4. We rode from 8am to 7pm on Tioga Pass Road (more details in the next section). Sunset was at 8:15pm, so that gave us a sufficient buffer in case of hiccups on the ride.
5. After finishing our ride, we drove back home the same day.

The ride

The road had a considerable amount of uphill- 8,200 feet over 100 miles. I had never done that much climbing or riding in a single day, so I was quite hesitant to commit to such a treacherous ride. Thankfully, the whole ride was an out-and-back on Tioga Pass Road. So I could turn back sooner if I wanted to (I did). There were about four natural options for turning around based on the terrain:

1. Olmstead point (50 miles)
2. Tenaya Lake (60 miles)
3. Tioga Pass (90 miles)
4. Lee Vining (120 miles)

The weather and the euphoria kept me going until Lee Vining, and although I find the back side (past Tioga pass) much more beautiful because of its raw, rugged, martian terrain, I wouldn’t have regretted turning around earlier if I had to — it’s all beautiful!

The first section until the Olmstead point has winding roads as you make your way up the coniferous mountains. You are gifted with a grand view of the Cloud’s Rest vista at the top.

The second section until Tenaya lake is all downhill. Depending on the sun and clouds, the lake water keeps changing colors, so keep an eye out for that!

The third section until Tioga Pass is again uphill, but this time you will be biking through rich green meadows.

The final section until Lee Vining is a hair-raising downhill through the canyon carved by Lee Vining Creek. Be careful- the winds can be quite turbulent in the canyon (as reported by my friends).

Planning tips

When does the road open?

Typically in late May or early June. The exact opening date will appear on this website depending on snowpack and plowing progress.

Weather

The temperature can swing wildly during the ride. I observed significant shifts between the leeward and windward sides of the mountains. Ditto when clouds covered the sun. Altitudes also play a role. So does the time of day. Bottom line- carry layers! I packed a base layer, cycling jersey, a windbreaker, a thermal layer, and gloves. I used all except the thermal layer at different stages of the ride. I might carry an additional layer of socks or use woolen socks if I do the ride again. I did not have to face other weather elements like rain or snow, so if you see that in the forecast, plan accordingly.

Nutrition

I consider this a long ride so I carried four spicy sandwiches, three tubes of energy gels, and two energy bars. For breakfast, I had had three energy bars (if you are looking for recommendations, “peanut butter banana choco-chip” from CLIF is my favorite!). After the ride, I ate a bagel and a couple of chocolate cookies. I was not disappointed with my choices! I think it is sufficient to say you won’t be disappointed too if you carry a few extra grams of food than risk biking hungry. That is never fun.

Water

There are plenty of creeks, waterfalls and lakes along the ride. The longest I rode without seeing a water source might have been ~8 miles. So I carried two bottles of water and iodine tablets to purify the water. Thankfully, I did not have to use my iodine (I hate its taste) as my friends carried squeezy water filters.

We had some extra water in the car to drink before and after the ride as there was no water filling stations at the car park.

When to turn around?

We have all been there. We are enjoying our ride on a new landscape and the curiosity of what is around the next bend propels us forward. At some point, we glance at the watch or the sun and it strikes us that we should turn around soon to make it back safely before it gets dark. Despite doing some simple math to estimate the time required to get back, we disregard it and keep biking forward to avoid FOMO. Here are the times when we turned around to help you plan your ride.

We started the ride at 8:30 am from the Tuolomne Grove Trailhead. I turned around at 3:30 pm from Lee Vining. On the way back I stopped to take plenty of photos and made it back to the trailhead at 8:30 pm. The sunset was at 8:15 pm, so there was plenty of daylight even after I returned. I hope this helps you land on a reasonable worst-case turnaround time.

Permissions

To drive up to the Tuolomne Grove Trailhead (starting point of the ride), you will need to purchase park entry permit. Annual passes are valid. Parking at the trailhead is free. Biking on Tioga Road requires no permission.

If you are going to camp on Evergreen road the night before the ride, that requires no reservations too. Just pick a safe spot away from the road.

Self-care

Finally, remember to stretch after your ride!

Once-in-a-year ride in Yosemite on Tioga Pass Road (updated 2024) was originally published in Bicycle Touring on Medium, where people are continuing the conversation by highlighting and responding to this story.