And so the mythical hero Theseus sailed home to Athens after defeating the Minotaur. To honor him, the Athenians preserved his wooden ship in their harbor down the centuries. As the old planks rotted, builders replaced them with identical new ones. Eventually, they replaced every single original piece of wood.

This sparked a famous debate among philosophers: With every part replaced, is it still the same ship?

Like the ship of Theseus, what endures also undergoes many changes. That idea applies not only to the Data Science + AI at Microsoft online publication you’re reading now, but also to my own career journey. After two stints totaling more than 32 years at Microsoft, and more than six years leading Data Science + AI at Microsoft, I am turning the page on both chapters this month as I prepare to depart. As I mark this transition, I feel many emotions, but one that stands out most is gratitude — for the people, opportunities, and experiences that have shaped both the work and me.

Leading Data Science + AI at Microsoft (which we call DS@M internally) since its inception has been a highlight of my working life. From its initial focus on data science in early 2020 to its evolution to encompass AI starting in 2023, DS@M continues to attract an audience as it approaches 10,000 followers. For me, this is especially remarkable given some of the internal pushback that came in the time before we launched, reflecting skepticism that it was possible, much less advisable, to share our data science expertise outside the company without crossing proprietary lines. That we moved forward — and ultimately succeeded — was possible only because of the many authors and collaborators who believed in the idea and were willing to contribute their work. More than 340 articles later, we’ve amply proved we could do it, and do it well.

In many ways, this work reflects the culmination of the paths that have shaped my career at Microsoft: technical content development, business analysis, and, most recently, as a Principal Data Scientist focusing on data visualization and information design. Each phase has added something new while replacing something that came before — different roles, different problems, different ways of thinking — while what has remained is a consistent focus on communicating complex ideas in ways that drive business impact. What I’ve come to appreciate most is not any single experience, but how all of them have combined — often in unexpected ways — with the influences of colleagues, mentors, and partners who’ve shaped how I’ve approached each transition.

One lesson I’ve taken from these experiences is that technical excellence matters most when people can understand why it matters. This means that data and AI scientists must do more than excel at the technical work itself — to have impact, they must also tell a story about that work in terms non-practitioners can understand. I think of that story in three parts: the What, the So What, and the Now What. The What explains the business problem, the stakeholders, customers, partners, and influencers involved, and the deliverable and capabilities it provides or enables. The So What explains why the work matters by showing how it increases desirable outcomes such as revenue, share, audience, or users, or reduces undesirable ones such as cost, time, redundancy, or waste. The Now What explains the call to action, the next planned capabilities, and the timeline for what comes next.

In this storytelling sequence, there is no How, and that is by design. Among practitioners, the How matters deeply. For stakeholders and senior leaders, however, what matters most is the What, So What, and Now What, because together they provide a clear understanding of business impact and direction.

DS@M has served as a forum to develop this kind of storytelling across audiences. For articles aimed at non-practitioners, we emphasize the What, So What, and Now What. For articles aimed at practitioners, we do the same but also explore the How in greater depth. Through an author-driven model, contributors bring forward topics and perspectives, and we shape them using this storytelling methodology for their intended audience. The result is not a single, definitive voice, but a collection of perspectives — each one distinct, with its own point of view, while contributing to a broader dialogue. That collective effort has been fundamental to what DS@M has become.

This is where the analogy of the ship of Theseus returns. DS@M has never been a fixed artifact. Over time, its authors, topics, and technologies have all changed. Individual parts have evolved, and new ones have taken their place. Yet the purpose of DS@M — to share knowledge, perspectives, and progress in data science and AI — has remained. The same could be said of my career at Microsoft: the roles, projects, and teams have changed, but with an underlying thread of connection through data and information design that has endured even as each new “plank” has taken its place.

At its heart, DS@M is its authors. Working with them, among the best in the industry from around the world, has been one of the great privileges of my career. Many people deserve thanks for helping make DS@M what it is today, but above all I want to recognize these authors, whose expertise, curiosity, and willingness to share their work beyond Microsoft have defined DS@M. Their contributions will continue to endure long after my role in this work has concluded.

It is for that reason I’m delighted to announce that my colleague Alexandra Savelieva, Principal Applied Research Scientist, will be taking over my role running DS@M after my farewell from Microsoft. Alexandra is one of the pillars behind our internal AI & ML Connected Community, and she brings both great depth and breadth of experience to all of her work. I depart knowing that DS@M is in fine hands with Alexandra’s leadership.

In this burgeoning age of AI, when information is easier than ever to access and AI is poised to take on more of the intellectual work long done by human beings, it is fair to ask whether DS@M, or something like it, still has a reason to exist. I believe it does. Data and AI scientists will always benefit from sharing what they are building, the frontiers they are pushing forward, and the latest developments from their peers. That exchange reflects a uniquely human spark within our professional community, and I believe it will remain indispensable to innovation. Whether embodied by DS@M or by other forums, our metaphorical ship will endure while being rebuilt, piece by piece — not because its parts remain the same, but because its purpose does. As I step into my own next chapter of change after Microsoft, I do so with gratitude for having been part of building this one, and with confidence that the work and the community will continue to both endure and evolve.

Casey Doyle is on LinkedIn.

What, so what, and what comes next was originally published in Data Science + AI at Microsoft on Medium, where people are continuing the conversation by highlighting and responding to this story.

Six years ago, on January 9, 2020, we embarked on what we thought to be an ambitious journey: launching the Data Science at Microsoft online publication you’re reading today. At the time, the idea of sharing insights from our internal data science work with the broader public felt both bold and unconventional. We grappled with a fundamental question — how could we openly discuss our data science practices and discoveries without crossing the boundaries of proprietary information? This challenge shaped our editorial philosophy from the outset.

Our solution, refined over the course of more than 300 published articles, has been to focus on process and principles, always presenting a distinct point of view. By emphasizing the “how” and “why” behind our work, we contribute to the advancement of the global data science discipline while ensuring that each article offers a unique perspective. This approach allows us to revisit topics from multiple angles, recognizing that no single piece can ever be the definitive word on a subject in a field as dynamic as data science.

Over these six years, the field of data science has undergone profound transformation. The rise of Artificial Intelligence has not only accelerated the pace of innovation within our discipline but has also catalyzed a convergence of fields — statistics, engineering, business, and beyond — around the possibilities of AI. Data has become the lifeblood of this new paradigm, and the interplay between AI and data science continues to enrich both domains. We have witnessed — and contributed to — a rapid succession of breakthroughs, each one building on the last, and each one opening new avenues for exploration and impact.

In response to this evolution, we are excited to announce an update to our publication’s charter and name: from this point forward, we are Data Science + AI at Microsoft and feature a new site banner image — though we’ll continue to use the familiar DS@M (pronounced “d-sam”) shorthand. This new identity reflects our expanded focus and acknowledges the central role that AI now plays in our mission. It’s a recognition that AI is not just a tool for advancing data science, but a transformative force that is reshaping the very questions we ask and the solutions we pursue. Notably, more than 83 percent of our articles published in 2025 have explored AI, either directly or as a critical context, underscoring the remarkable progress our community has made in integrating AI into our work over the past three years. For these amazing advancements, we thank our remarkable group of Microsoft employee authors, who create all the content we feature in DS@M based on their outstanding work.

None of this progress would be possible without you — our dedicated and engaged readers. As reflected in DS@M having reached 9,100 followers, your curiosity, feedback, and enthusiasm have been essential to our growth and evolution. When we launched DS@M, we could not have predicted how quickly AI would become central to our mission, nor could we have foreseen the vibrant readership that would grow around this online publication. Looking ahead, the future promises even more rapid change and opportunity. Six years from now, the landscape of data science and AI will undoubtedly look very different, but we are confident that we will continue to learn, adapt, and lead.

Thank you for being an integral part of this journey. Your readership inspires us to push boundaries, share knowledge, and foster a culture of innovation. As we step into this new chapter as Data Science + AI at Microsoft, we invite you to join us in exploring the exhilarating and enriching possibilities that lie ahead.

Casey Doyle
Principal Data Scientist, Microsoft
Leader, Data Science + AI at Microsoft online publication

Celebrating our sixth anniversary while embracing the AI era was originally published in Data Science + AI at Microsoft on Medium, where people are continuing the conversation by highlighting and responding to this story.

Today, January 9, marks the fifth anniversary of the Data Science at Microsoft online publication on Medium.com, which we refer to internally at Microsoft as DS@M (“d-sam”). In that time, we’ve published nearly 250 articles by more than 162 authors and co-authors working full-time at Microsoft (at the time of publication) across a range of topics, including “bread and butter” data science methods, Generative AI, data visualization, data engineering, data science program management, data science leadership, data science organizational development, and data science career development. Our purpose is to publicly share our Microsoft data sciences best practices, discuss the value of our work, show our impact on the business, and connect with and contribute to the dialog in the broader data sciences community.

How do we determine what to publish? Because we follow an author-driven model, we rely on our Microsoft data science colleagues to come to us with articles and ideas for articles instead of us commissioning specific topics at specific times. This enables us to reflect a wide range of the latest thinking on data science across the company from those motivated to write about it, and it enables us to share multiple points of view and perspectives — even on the same or similar topics. We believe this approach keeps our material fresh and, hopefully, relevant over time as the many developments in our field continue to rapidly unfold.

To help keep our readers engaged, we aim to publish a new article every week. Articles tend to arrive in our editorial inbox in clumps, however, instead of being evenly spread out — more evidence of the “inherent clumpiness of randomness,” as described elsewhere by Mike Loukides. What this means for us is that we’re constantly working to have a pipeline of about a month’s worth of articles ready to publish at any one time as article arrivals naturally ebb and flow. That way, if any proposed article is delayed in being ready to publish for any variety of reasons, we can easily publish another in its place and keep to our weekly publishing schedule. If we sometimes have more articles than we anticipated, we can dial back our acquisitions efforts for a little while to even things out — or even publish more than one article per week for a time if we have the bandwidth to process them.

We ask our contributors to start with a business problem whenever possible instead of jumping right into the technical side of things, for ours is an applied science, and this additional business context helps show how the data science methods and techniques covered in an article can actually be applied in the real world. We also ask that articles “show” more than they “tell,” describing the author’s actions and thinking in ways that help the reader to share in them, which also helps to bring the material to life while making the narrative more vivid and engaging. Some articles may hit this mark more than others, but we believe that developing data science storytelling skills is a journey, that all journeys start from somewhere, and all stories have value.

We edit all articles for content and style in ways that we believe help to put contributors’ work in the clearest light that we can, while hopefully allowing their individual voices to shine through.

As I’ve said in previous anniversary commemorations, DS@M would not exist without the fine work of our many data science colleagues at Microsoft and their desire and ability to share it with a wider audience outside the company. For that we are grateful, just as we are so very deeply grateful for you, our wonderful readers, who now include nearly 7,000 followers.

Here’s to looking forward to more great contributions to share with you on Data Science at Microsoft in the year ahead!

Casey Doyle
Principal Data Scientist, Microsoft
Leader, Data Science at Microsoft

Celebrating our fifth anniversary of Data Science at Microsoft was originally published in Data Science + AI at Microsoft on Medium, where people are continuing the conversation by highlighting and responding to this story.

Those of you who have been reading us for a while may have seen me write before that when we were discussing getting started, it wasn’t even clear that we could do what we have spent these last four years doing on DS@M. After all, how could we write about our work while still keeping proprietary what must be kept proprietary? Who would be interested in hearing from us? How would we even have enough topics to write about data science every week while keeping things fresh?

One of my biggest learnings during this time in running DS@M is that point of view matters a lot. It is because of point of view — having multiple points of view, that is — that we don’t have to worry about exhausting a given topic. As long as each contributor comes ready to express a point of view — informed by training, tempered by experience — we can address a given topic in multiple ways and over multiple periods of time.

This is not to say that we set out to repeat ourselves on DS@M. My point is that with so many highly trained data scientists of such deep experience, naturally different points of view abound, coupled with our collective desire to not try to proclaim something as the “last word” on a given topic, which I doubt is even possible in the fast-evolving, ever-changing world of data science. Capturing this richness is a big part of what DS@M is all about, bringing it to you, the reader, in the hope that it informs your work while it fosters and contributes to an ongoing discussion that helps to continuously advance the state of the art in our discipline.

Of course, without our many contributors bringing these multiple points of view, none of this would be possible. I’ve had the privilege to work with some of the most brilliant minds in our profession, and I’ve learned so much along the way. From some of our data scientists, I’ve learned new ways to think about things long familiar, and from others, new things altogether new.

Who among these contributors helps to evolve our ongoing data science dialog the most? The truth is, it’s everyone, whether at either end of the spectrum of familiar to new or somewhere in between. We operate in a truly multi-disciplinary field, after all, and we are at our strongest when we seek, recognize, and learn from data scientists at every career stage and representing every education, background, specialty, and experience. It is for this reason that data science is one of the most dynamic and engaging fields of modern knowledge work, with so much to offer both practitioners and clients — because it’s all about the uncountable ways that a diverse body of data scientists can challenge, contribute to, and hopefully improve the world.

So it is with great pride and deep humility that I offer my sincerest of thanks to our many contributors both recent and past as well as to you, dear reader, for devoting one of your most valuable commodities to us and what we talk about here — your precious attention. As we embark on our fifth year, our luminous contributors and I will do our best to continue to bring you something worthy in exchange.

Casey Doyle leads Data Science at Microsoft and is on LinkedIn.

Celebrating our fourth anniversary of Data Science at Microsoft was originally published in Data Science + AI at Microsoft on Medium, where people are continuing the conversation by highlighting and responding to this story.

I’ve written for past DS@M anniversary articles about how gratifying it’s been to play a part in founding and developing this online publication, going from our small beginnings to where we are today as we rapidly approach 150 articles published from Microsoft data scientists and related professionals for other data scientists, decision-makers, customers, partners, and students. With 3,500 followers, I think — I hope — that we’ve shown that we have something relevant and meaningful to contribute to the ongoing dialog in the data science community.

This year, we’ve continued to expand our partnership with data scientists on a variety of teams throughout Microsoft, who’ve written on topics spanning how to get data science work done to advanced data science techniques — and everything in between!

It’s particularly gratifying to write about our journey this year as I welcome my Microsoft colleague, Riesling Walker, who joins me in running DS@M. Riesling is a Senior Data & Applied Scientist who has been a co-worker for most of the last year, and whom some of you might know as Riesling Meyer — in addition to all she’s been doing at Microsoft, she was married this autumn! Besides her data science “day job,” she’s stepped up in a tremendous way to add her voice to our data science narrative on DS@M. As part of this, she wrote our most-read article in 2022, on writing a resume for a data science role, a fine piece I recommend that everyone read. We are so happy to have you with us, Riesling!

Riesling is one among many data science professionals across Microsoft whose work contributing articles based on their hard-won expertise has formed the backbone of Data Science at Microsoft. My deep thanks to all of our many contributors, both current and past, who’ve helped get us to where we are today! And my thanks to you, our dear readers, for whom we present this work in the hope that we are advancing the state of the art in our data science community by describing the work we do on behalf of our customers, partners, and stakeholders to improve Microsoft products, services, and the materials that help users learn more about them.

Here’s to a great 2023 for all of us!

Casey Doyle is on LinkedIn.

Celebrating our third anniversary of Data Science at Microsoft was originally published in Data Science + AI at Microsoft on Medium, where people are continuing the conversation by highlighting and responding to this story.

When we began, some were skeptical. They said we wouldn’t have enough to write about to sustain regular publication, or that there wouldn’t be enough interest among potential readers, or that we couldn’t write about how data science at Microsoft is done and protect what is proprietary to Microsoft. I’m happy to say that more than 120 published articles later, with contributions from data science professionals throughout Microsoft and a steadily growing readership now exceeding 3,200 followers, I believe we have shown that there is indeed a desire to read what we have to share about our work in all of the data sciences.

DS@M, as we call it internally, is open to all professionals in the data sciences who work full-time at Microsoft at the time of article publication, and indeed they have heard our call: Our articles are written by data scientists, ML scientists, data engineers, and data science program managers, among others, from across the company. Although the focus of their work may vary from person to person, they all play an integral role in shaping the development and success of data science as a profession both inside Microsoft and outside of it across the wider data sciences community.

As we proceed into the second half of 2022, I’ve put together this summary to highlight the articles that have resonated the most with our readers, in terms of views, during the first half of the year. Each of these articles has achieved at least 1,000 views since being published. I’ve done no further adjustments or monthly averaging to these metrics, so this does mean that the ones published earlier in the year have had more time to build their audiences than the ones published later. The vast majority of DS@M articles gain the bulk of their views in the first month, however, and so I do not believe that this presents an especially distorted view. We’ve had at least one article eclipse this threshold for each of the first six months of this year. What follows is a list of these seven articles, along with a brief synopsis and link to each one so that you can seek them out and read them if you have not had the opportunity to do so earlier.

My deep thanks to the authors of all our DS@M articles, the data science professionals who take time in their busy working and personal lives to commit to sharing what they know with our readership.

And now the highlights of the most viewed articles:

Comparing matrix factorization with transformers for MovieLens recommendations using PyTorch-accelerated by Chris Hughes, published January 4. Chris notes the importance of recommendation systems in preventing us from being overwhelmed by choice and exposing us to content that would be difficult to otherwise discover and demonstrates how transformers can be used to predict ratings from sequences of past behavior, as well as seeing how this compares with the more widely known matrix factorization approach. Chris also notes that transformers are driving significant advancements in NLP, vision, and time series domains, so it’s about time they came to recommenders as well.

Autosuggestion services in web search by Tezan Sahu, published February 8. Tezan’s first article this year reviews the fundamentals of autosuggestion services in web search and delves into some key aspects of modern autosuggestion services, such as relevance and coverage. Tezan also discusses some techniques to improve the UX of these services, along with performance metrics to evaluate the effectiveness of these techniques.

Visual question answering with multimodal transformers by Tezan Sahu, published March 8. As Tezan explains in his second article this year, recent years have seen significant advancements not only in the respective domains of Natural Language Processing (NLP) and Computer Vision (CV) but also in tasks involving a combination of these modalities. Among the various tasks, Visual Question Answering (VQA) has particularly drawn the interest of several researchers. In this article, Tezan illustrates some of the basic concepts related to Visual Question Answering and Multimodal Models for performing such a task. He presents a detailed PyTorch implementation of VQA models using text and image transformers from the huggingface library, and also compares the performance of several models using different text and image transformers for this task.

Scalable time series forecasting and anomaly detection by Sourav Khemka, published March 15. As Sourav explains in his first article this year, time series forecasting and anomaly detection is important for many businesses in today’s data-intensive world. It finds use across diverse industries including IT, manufacturing, retail, health care, banking, and finance. It also has applications for sales forecasting, inventory analysis, intrusion detection, fraud detection, and production system monitoring, among others. Sourav writes that his team has multiple use cases for time series forecasting and anomaly detection in the spaces of Azure finance and commerce, involving a significantly large number of time series. Forecasting and detecting anomalies in a univariate time series is a well-known problem and many solutions exist that are quite effective. But as the number of time series increases, these solutions often do not scale well. Sourav provides a solution that scales up to a volume of 100,000 time series.

The role of a technical program manager in AI projects by Nik Sachdeva, published April 5. In this article, Nik relays a report from Venture Beat that 87 percent of data science projects fail and never move to production. He notes that Technical Program Managers (TPMs) can drive change to this statistic and help data science and engineering teams build successful AI projects, and he walks through PM considerations for ML projects and a learning path for Technical PMs to increase their skill set for projects that have an ML component.

Estimating customer churn based on usage data by Haribabu Inuganti, published May 31. In this article, Hari explores how to identify churned or retained customers by looking at product usage patterns to understand from the data whether a churn problem exists, covers how to identify a churned customer based on usage data, discusses how to understand churn patterns using Kaplan–Meier curves and heatmaps, and shows how to use statistical testing to determine whether there is a churn problem.

Scalable time series forecasting by Moid Hassan and Sourav Khemka, published June 21. In this article, Moid and Sourav demonstrate the use of Temporal Fusion Transformers (TFT) as one way to solve the problem of scalable time series forecasting. This article builds on the article that Sourav wrote solo on scalable time series forecasting and anomaly detection that was published March 15 and is described above, as it also has been viewed more than 1,000 times.

If you have any ideas for articles that you would like to see covered in Data Science at Microsoft, please include them in the Comments section below. We thank you for your readership!

Casey Doyle is on LinkedIn.

Resonating with readers: Our most viewed articles during the first half of 2022 was originally published in Data Science + AI at Microsoft on Medium, where people are continuing the conversation by highlighting and responding to this story.

It’s common for data scientists to hear about the need for “data storytelling.” But what does that mean? Put simply, it means — contrary to what is often said — that data doesn’t speak for itself: It requires humans to construct a narrative about it, to tell a story about it.

Telling stories comes naturally to human beings. This ability helps us make sense of the world around us. But too often, data scientists may become so focused on the details of their analysis — or so mindful of the challenges they have overcome — that they talk only about those aspects of their work, forgetting to put the work in the context of why it was likely commissioned in the first place: to solve a business problem.

This is not trivial. Data scientists who can’t communicate the business impact of their work — typically by describing the “So what?” and “Now what?” in addition to the “What” — may find their careers unintentionally limited, no matter how great their technical expertise. But constructing a relevant, meaningful narrative about the business impact of their work is only part of what is needed. To be successful, data scientists must understand all the parts of their story and how each part appeals — or does not appeal — to different audiences and stakeholders, and then deliver accordingly.

Whether a data science deliverable consists primarily of some type of long-form writing, a presentation, or something else — or in cases where a talk track is being built to accompany the unveiling of an ML model, for example— they can all benefit from having a defined narrative structure. In this article I share ways to think about building a data science storyline using an example of narrative structure from Star Trek, and I suggest ways to approach writing what may be the most necessary and significant but often the most overlooked part of it: the executive summary.

A data science storytelling framework

Just as Shakespeare wrote his stage plays in five acts, so, too, there are five main structural components of a data science story. They include data exploration, analysis, findings, conclusions, and recommendations. Furthermore, a “wrapper,” supplementing this core and consisting of an introduction, appendixes, and — crucially — an executive summary, round out a complete data science storyline.

I start with the core five sections not only because they are central to a well-told data science story, but also because they must be constructed before the introduction, appendixes, and all-important executive summary can be put together.

The five core sections

To illustrate the five core sections of the data science storytelling framework, I follow a typical Star Trek storyline. In a classic episode from the original series, “The Devil in the Dark,” the Enterprise arrives at mining planet Janus VI to investigate the recently impaired production of a key mineral vital to the populations of other worlds. As the show unfolds, Kirk, Spock, McCoy, and their fellow crewmembers follow a story structure used in many Star Trek episodes that also serves to illustrate a meaningful data science narrative structure.

First, they explore the data as presented to them by the miners on the planet: Ever since reaching a particularly deep subterranean level, some equipment has been damaged and some miners have been killed, leaving only acid-charred remains. Some miners claim to have seen a “monster” lurking near the carnage. Fearing for their lives, the miners have been reluctant to venture back to the levels where the deaths are occurring, reducing production of the mineral. Also in that vicinity, the miners say they have discovered thousands of spherical silicon nodules — unique in form and composition, but of no commercial value — that they have not encountered elsewhere.

With these data points in mind, Kirk and Spock investigate the deep tunnels for clues to help them analyze the situation. They confirm that deaths are occurring at the lower levels when one of their own crewmembers is killed there. They confirm the presence of the silicon nodules and also notice that many have been smashed by the miners. They also observe that additional tunnels have recently been cut into the rock at the lower levels — but not by the miners. Ultimately, they encounter the creature glimpsed by the miners, and after a brief confrontation ends in a draw, Spock establishes telepathic communication with it, learning about the situation from the creature’s point of view — and even that the creature calls itself a Horta, indicating intelligence.

From their analysis, including the information gleaned from Spock’s telepathic communication, they come to their findings: The Horta is the last of her kind, a sentient being and natural maker of tunnels through deep rock who has laid a large number of eggs at the lower levels — the spherical silicon nodules — and is watching over them. Enraged by the miners destroying her progeny and doing what she can to stop them, she has been attacking and killing the miners and destroying their equipment.

Based on these findings, Kirk and Spock come to their conclusions: If they can stop the miners from destroying the Horta’s eggs, the killings will stop, and mineral production can resume. Furthermore, because the Horta is a natural burrower — and because thousands of her eggs are about to hatch — they could possibly reach an accord for the miners to make use of the tunnels naturally made by the Hortas in the course of their lives while otherwise leaving each other alone. This would provide the miners with even greater sources of the mineral as well as other valuable raw materials that would be too difficult to reach otherwise.

Finally, based on their conclusions, the Enterprise team makes their recommendations to the miners: Stop destroying the silicon nodules, make peace with the Horta, welcome her offspring when they emerge, and seek a mutually beneficial relationship in which the Hortas live in peace as they tunnel through the depths, making minerals more accessible to the miners along the way. With Spock acting as intermediary, the miners and the Horta come to an agreement, and then the Enterprise departs, with Kirk, Spock, McCoy, and other crewmembers having succeeded not only in resuming mineral production, but also increasing and expanding it while protecting and respecting the Horta and her coming offspring.

The importance of a complete narrative

Now imagine for a moment that the Enterprise crew did what many data scientists do — stop with their findings, or even less helpfully, keep their focus on how they did their analysis, instead of moving to conclusions and recommendations. The miners would learn only the following: 1.) extensive subterranean cave exploration by Kirk and Spock revealed a creature burrowing tunnels in the deep rock and recently motivated to murder because of the destruction of her eggs, and 2.) Spock has an ability to establish telepathic communication with the creature by having the courage and fortitude to physically and mentally make contact with it.

In this case, the creature’s existence and motivation, while interesting, provide an explanation for what the miners have experienced but no deeper insights that could result in a direction to take for the miners to remedy the situation. And the details of Spock’s telepathic communication — no matter how much of a significant personal risk and display of abilities for Spock — are irrelevant to the miners. It is only when Kirk and Spock explain their conclusions (the meaning of their findings, namely their opinion based on their findings that there is an opportunity for mutually beneficial cooperation) and make their recommendations (what they believe the miners should do based on their conclusions, namely to come to an agreement that gives both sides what they want) that the miners can take action to solve the problem they called on the Enterprise to help with in the first place.

What works as a narrative structure in Star Trek also works in data science. A complete data science storyline can be said to require explanation not only of the What (the findings), but also the So What (the conclusions) and the Now What (the recommendations). Regarding the extent to focus on the How — the details of the analysis itself — while it indeed has its place and audience, it’s not everywhere and for everybody, as I explore next.

Creating a wrapper for your data science story

The five core components I’ve walked through above that are critical for a complete data science narrative can be further enhanced by the addition of an introduction, appendixes, and — most importantly — a well-crafted executive summary.

Although the finished data science deliverable, when finally presented, starts with an executive summary that is followed by an introduction, these two sections are not the parts of the overall narrative for the data scientist to construct first. That’s because the five core parts of the data science narrative must already be built so that there is something to say in the executive summary and introduction. In fact, the executive summary should be prepared last, as I describe shortly.

The introduction, in much the same way that each episode of Star Trek begins with a voiceover of Kirk speaking into his log to set the stage for the story to come, provides background on the business problem, situation, or question that led to the data science work being done, along with who commissioned it and any other key stakeholders involved. It also provides a high-level overview of the sections that follow it, spanning data exploration to recommendations, without going into specific details. (Those are found instead in the executive summary; see more on this below.)

The appendix, akin to the closing credits of a Star Trek episode, is the place for anything related to the work that doesn’t belong in one of the other sections. In this way it is typically not a formal part of the deliverable, but more of an ancillary component. As such, it is optional. It may also be a place for work or “roads not taken” that are related to the data science work but outside the flow of the overall narrative.

The executive summary

Aptly named, the executive summary is literally what is shared with executives and other decision-makers, and I describe it last while going into some detail about it for a reason: It is often the most challenging section of the data storytelling framework to write. That’s because it must distill the essence of the key information the decision-makers need — the What, So What, and Now What — without going into the How.

Many data scientists stumble here because they have been so focused on the How for so long — and have overcome so many obstacles along the way for which they are justifiably proud — that they are eager to demonstrate to executives and key stakeholders their smarts, perseverance, and skill in getting to their results. So, they focus on the How.

But if we pause to consider the point of view of the decision-maker, we can see how this good intention can lead to significant missed opportunity. The decision-maker is likely far removed from data science work and probably does not have a background in data science. Moreover, the decision-maker is likely hearing from many different data scientists about a number of different projects on a regular basis. If those communications are primarily about process — the How — that the decision-maker is not well positioned to appreciate (or possibly even understand), and if those communications lack what the decision-maker needs to do their job — the What, the So What, and especially the Now What — then there is a significant gap between what the decision-maker needs and what the data scientist is delivering.

Multiply this across many data scientists taking a similar approach, and the decision-maker may lose confidence in the ability of data science to provide practical help in making better business decisions, and data scientists may lose standing and influence with the decision-maker, who may then seek help elsewhere or may even reconsider organizational investments made in data science.

Once this dynamic is understood, it’s more straightforward to write an effective executive summary, and easy to see why it should be the last section to write, even though it appears at the beginning of the finished deliverable. This executive summary, necessarily brief just as its title indicates, relays the highlights in no more than two or three sentences each of the findings (the What), the conclusions (the So What), and the recommendations (the Now What). It does not include references to data sourcing, exploration, or the details of the techniques or processes of the analysis itself.

It is the executive summary that is primarily what is presented to the decision-maker, with all the other portions of the work either clearly subordinated or included in an appendix for later reference if needed or requested.

In this way, the executive summary is meant primarily to help business decision-makers and other key stakeholders do their work, which is to decide whether to undertake the direction recommended by the data scientist based on the findings and conclusions presented. The executive summary may also be helpful for other audience members either as a useful overview of the material without diving deeply into its particulars, or as a reference to the material after it has been presented.

The other sections of the narrative framework and its wrapper — the introduction, data exploration, analysis, findings, conclusions, recommendations, and appendixes — are in their fullness primarily for other data scientists, data science managers, program and project managers, and other interested parties who want or need the details in these sections to do their work and advance the practice of data science as a discipline.

By understanding the different needs of various audience members for different parts of the deliverable and providing material accordingly, data scientists can give these audiences what they need to do their jobs while promoting excellence in data science.

For anyone looking to complete the Star Trek analogy, where does the executive summary fit in? It is akin to the pitch made to entertainment executives after the TV series is developed, and perhaps after the pilot episode has been produced, summarizing what will appeal to the audience for the series to earn viewership and, ultimately, revenue.

Data storytelling framework summary

The following table summarizes the parts of the data storytelling framework:

Conclusion

It’s not difficult to see why data scientists sometimes encounter obstacles in making business impact with their work. Data science is challenging, complex, and can be difficult to explain to non-specialists. But doing so is critical to realizing the ultimate success of data science, which depends on making business impact.

To get there, data scientists who understand and include the five core parts of a data science narrative and the three wrapper sections (collectively, the executive summary, introduction, data exploration, analysis, findings, conclusions, recommendations, and any appendixes), who can prepare an effective executive summary based on the five core components, and who can present the relevant parts of their work to the right audiences in the form of a data science narrative that’s applicable and relevant will not only communicate about their work more effectively, they will increase their ability to influence decision-makers and key stakeholders, leading to effective and lasting business impact on their organizations — and gain enhanced career success along the way.

Casey Doyle is on LinkedIn.

Using “Star Trek” to help understand what “data storytelling” means — and how you can do it, too was originally published in Data Science + AI at Microsoft on Medium, where people are continuing the conversation by highlighting and responding to this story.

Data Science at Microsoft — the online publication on Medium.com you’re reading now — is two years old on January 9. What a journey it’s been!

When we started, the pandemic was not yet a fact of daily life in much of the world. The organization we were a part of was a much smaller one, and with some different people. Some were skeptical about our undertaking with DS@M, as we call it internally. How could we write about our data science work publicly without exposing proprietary information? How would we be able to establish and maintain an online publication that would require regular content? How receptive would those outside Microsoft be to reading about our data science work?

Two years later, the world has changed quite a bit — and so has DS@M. Many people, if they work for specific companies that have required it, have been away from their offices (working “virtually,” as well call it at Microsoft) for nearly two years. Everyone on the planet, regardless of walk of life, has experienced unexpected twists and turns from the pandemic that have affected lives and loved ones in ways large and small. Contributors to DS@M, both on the authoring and editorial side, have changed, just as our sponsoring organization, now called Microsoft Cloud Data Sciences, has changed. But through it all, we’ve built up DS@M from scratch to the point that we are now approaching publication of our hundredth article — and we now have more than 2,500 followers. Our content contributors come from across Microsoft, eager to share their expertise and experience in the data sciences with a wide audience. If we can do this during a pandemic, who knows what else we can do when the world finally emerges from it?

As I wrote one year ago for our first anniversary, our mission for DS@M remains to share our data sciences best practices, discuss the value of our work, show our impact on the business, and connect with the broader data sciences community. The social journalism platform enabled by Medium enables us to tell the story of Microsoft data science and share it with business decision-makers, customers, partners, other data scientists, students, and even prospective employees, whether the topic is on data science methods, data visualization, data engineering, organization, professional development, or people.

If DS@M has provided you, our reader, with ideas about how to pursue your own work in the data sciences, provided a window into ways that data science at Microsoft is done, or helped you feel connected with the larger data sciences community, then we’ve accomplished at least part of what we set out to do. Just as we would have no online publication without our scores of contributors, the editorial collaborations of many, or without support from our executives, we would only be talking to ourselves without you, for whom we publish this work. I’m very grateful to each of you for reading DS@M, I look forward to what we will be sharing with you here in the coming year, and I wish you and all of us the very best as we begin 2022 together.

Casey Doyle is managing editor of Data Science at Microsoft.

Celebrating our second anniversary of Data Science at Microsoft was originally published in Data Science + AI at Microsoft on Medium, where people are continuing the conversation by highlighting and responding to this story.

One year ago, on January 9, 2020, members of the Customer Growth Analytics team at Microsoft launched this online publication on Medium: Data Science at Microsoft. Our objectives were to share our data science best practices, discuss the value of our work, show our impact on the business, and connect with the broader data science community. These objectives remain the same today. In these ways, we’ve wanted to use the social journalism platform enabled by Medium to tell the story of Microsoft data science, and share it with business decision-makers, data science practitioners, customers, and partners.

One year later, we’ve published 40 articles — nearly one per week — on a variety of topics related to data science, in the areas of data science methods, data visualization, data engineering, organization, professional development, and people. We’re approaching 1,000 followers and our stories have garnered 50,000 views.

What have we learned about our readership during this time? Articles on data science methodology — cutting edge ones like causal inference and foundational ones like calculating customer lifetime value — have done very well, suggesting deep interest in the nuts and bolts of doing data science work. Writing about data engineering and what we’ve learned about it along the way has also found a wide audience, as have articles on how to think about data science as a discipline and as it relates to the larger organization. Articles on data visualization have received a warm welcome, too.

I was particularly gratified by the response to our article by Alex Blanton on building a machine learning and data science community at Microsoft, which suggested a hunger for connection during a year that many of us have been physically and socially isolated from colleagues. This also marked a key turning point for us as we started hearing from data scientists across Microsoft who expressed interest in adding their voices to our online publication, which we envisioned as encompassing all of Microsoft from the start.

As we prepare to begin our second year of this online publication, and as you’ve had a chance to see the range of articles and topics we’ve published, we would like to extend an invitation to you to compare your interests to ours. What would you like to see us cover more of? Less of? What would make Data Science at Microsoft — “DS@M,” as we refer to it internally — more interesting and relevant to you? I invite you to leave your comments below.

Whether you’re a data scientist, machine learning scientist, data engineer, customer, partner, business decision-maker, student, or other interested party, here’s to a great new year together in 2021, both for DS@M and our professional interests!

Casey Doyle is managing editor of Data Science at Microsoft.

Celebrating our first anniversary of Data Science at Microsoft was originally published in Data Science + AI at Microsoft on Medium, where people are continuing the conversation by highlighting and responding to this story.

But color brings challenges, too. Because it can dominate the visual field, it may distract instead of enhance, taking center stage over the information it is intended to convey. In extreme cases the use of color can interfere with comprehension, leading to misinterpretation or at least ambiguity. Some differences in color may also not be discernible by those who are color-blind.

Used effectively, color increases the informative value of a visualization — and even the appeal of the how the information is presented — without dominating. Achieving this means that color fades into the background, along with the rest of the design or data visualization elements, so that the message is what stands out and is remembered.

Metaphorically speaking, instead of bursting into a crowded room and shouting “COLOR!” so that everyone stops and stares, I want to enter the room, join and help guide the conversation, and mention color as part of the dialogue. In this article I share ways to think about color and show some ways to use color to achieve specific information design goals.

An example of color use in wayfinding

Consider roadway signs. Although ubiquitous, they are such an integrated part of the roadside landscape that it’s easy to notice them without actually seeing them. They guide, inform, advise, and warn — all in a small amount of space in which they can be noticed, decoded, understood, and acted upon quickly, at a glance and at a distance. These purposes are probably already familiar: Almost all of them are the same as for information design and data visualization.

Their use of color is especially relevant. According to the Manual on Uniform Traffic Control Devices used in the United States, the specific colors of a roadway sign indicate the information contained. Regulatory signs use only black, white, and sometimes red. Warning signs use only yellow and black. Guide signs use only green and white. Color is applied with a clear intent for motorists to quickly take away specific information so they can act on it, as shown at left in Figure 1 below.

Just as color can communicate purposefully in roadway signs, it can do the same for us in information and data visualization. In this article I look at ways I have used color to communicate key information in dashboards and data visualizations I have developed.

Start with a single color

Color choices abound in modern tools, whether, for example, in a package such as Power BI Desktop or in code with R and ggplot2. Default settings are often set to display color from across the spectrum or may otherwise make it easy to use several colors in a single visualization. But starting with multiple colors just because they’re available may increase the user’s cognitive load (the working memory resources necessary to process information) by diminishing clarity.

In my last article, using grids to improve information dashboards, I recommended considering information design and user experience goals when starting information design and data visualization problems. A significant consideration in this approach is to determine what users should take away from the visualization being constructed. Color is a powerful way to help accomplish this. When I use it to direct attention or make a distinction, I move from merely reporting data to telling a story about what’s important, and I increase the impact of the information so it can be acted upon.

To start down this path, I begin with a single color (which, not incidentally, also makes the information design process easier as there’s one fewer initial design variable to manage). I choose a medium level of shading for that color — i.e., not over- or under-saturated. In addition to being easier on the eyes, starting with a medium shade enables other shades of that color to be used for contrasts. I choose a darker or more saturated version of the same color to make a distinction or draw attention, and a lighter or less saturated version of the same color to de-emphasize an element or help it fade into the background. This approach is shown is shown in the first two graphs in Figure 2 below.

An additional advantage of this approach is aesthetic: It eliminates concern about whether the colors look good together — what color theorists call color harmonies, of which there are several. This is because shades of a color combine to form what is called an analogous color scheme, which naturally forms a visually comfortable presentation.

Contrast this approach with one, shown at right in Figure 2, using the ggplot2 scale_colour_hue() default color scheme with evenly spaced colors from the HCL color wheel, as described by Hadley Wickham in his book ggplot2, second edition. This graph increases cognitive load, taking the user longer to process, not only because the multiple colors themselves don’t suggest what’s important — in fact, the use of red in one of the bars might be misinterpreted as noteworthy, which is not true in this case — but also because of the additional element (the red box) employed as a result to draw attention.

Adding a second color

Using shades of a single color not only provides a way to make useful graphical distinctions, it also sets the stage for introducing a second color for a defined purpose. For example, Stephen Few describes the utility of using a red dot to flag items in data visualizations requiring user action in his book Information Dashboard Design.

I show an illustration of the red dot in Figure 3 below at left, which indicates the user is to take action for item E, which is below target. (Building on this approach, if I consistently combine this use of red with a specific shape, it helps serve the needs of color-blind users who are unable to distinguish red but can rely on seeing the shape.) Of course, red is often used to signal alarm as well. If no warning is intended, I either use a more “neutral” color for the dot, shown in Figure 3 in the middle, or I change other elements of the visual to highlight the element needing attention, shown in Figure 3 at right.

I can use a second color to draw useful distinctions in other ways, too. Consider a line graph in which the lines overlap, as shown below in Figure 4 at left. If the graph contains only a couple lines, I can use a baseline color of medium shading and a darker shade where I want to show what’s important. If the graph contains many lines, as shown in Figure 4 in the middle, I can introduce a second color for this purpose. By indicating emphasis in this way, I am helping direct attention and telling the user a story of what’s important while minimizing cognitive load — enabling the viewer to focus on the impact of the information itself instead of trying to figure out how to read the visualization, as is the case in the graph shown in Figure 4 at right.

Using multiple colors

A natural question might be when to use multiple colors. I would answer by saying there are better questions to ask, namely by going back to my information design and user experience goals. For every color I want to use, starting with the first one, I ask whether it serves a purpose. In cases where that purpose is not obvious — or perhaps nonexistent — I stay with as few colors as I need to tell the story about what’s important. This eases comprehension and enables action.

An illustrative example of this approach can be seen in two presentations of a graph showing cumulative views of a set of web-based articles over time. In the graph at left in Figure 5 below I use blue to call attention to the trendline of one particular article. In this way I form a narrative that contrasts the performance of that article with the others, which are presented more for context than for their individual details. In the graph at right in Figure 5, I use multiple colors from the default in Excel, but this time with purpose: The aim of the narrative I construct here is to focus not on the performance of any one particular article, but on comparing and contrasting all of them. Note that I would not use the default Excel color scheme in this way without a specific intention like this one in mind, after first formulating my information design and use experience goals for the data visualization.

Another example of using color deliberately can be seen in two presentations of a heatmap. The purpose of a heatmap is to graphically show contrasts in a set of underlying data, which it achieves with a shading matrix. Heatmaps are often presented in multiple colors, but they can be interpreted more easily in cases when shades of a single color are used for the gradient, as shown in Figure 6 below at left.

This is because of the natural perceptual ordering offered by a single-color gradient. Because the purpose of conveying the information is best served with shades of a single color in a heatmap, there is no reason to introduce additional ones. In fact, the use of additional colors, as shown in Figure 6 at right, increases cognitive load and impedes the comprehension the heatmap is meant to convey.

Cole Nussbaumer Knaflic discusses use of color in information presentation, and other related topics worth reading, in her book Storytelling with Data.

Summary

An effective approach to color in information design and data visualization is to use it to communicate meaning in ways that are purposeful and deliberate. Start with determining what users should take away from a visualization while considering information design and user experience goals. With these established, begin with a single baseline color and then add other incremental color only as needed, focusing on delivering a visualization that supports the goals you’ve established and that clearly communicates what’s important.

This approach results in transforming data into information that stands out and is easy for the user to discern, leaving the design and color palette to fade into the background. For the data visualizer, it makes the information design process faster because fewer design variables are introduced that need to be managed. The result is an informed viewer who understands where it’s important to direct attention and take action.

For more information on visual information design, see my article on using grids to improve information dashboards:

Using a grid to improve information dashboards

Casey Doyle is on LinkedIn.

Using color in data visualization and information design was originally published in Data Science + AI at Microsoft on Medium, where people are continuing the conversation by highlighting and responding to this story.