<?xml version="1.0" encoding="UTF-8"?>
<rss  xmlns:atom="http://www.w3.org/2005/Atom" 
      xmlns:media="http://search.yahoo.com/mrss/" 
      xmlns:content="http://purl.org/rss/1.0/modules/content/" 
      xmlns:dc="http://purl.org/dc/elements/1.1/" 
      version="2.0">
<channel>
<title>Arthur Turrell</title>
<link>https://www.aeturrell.com/atom.html</link>
<atom:link href="https://www.aeturrell.com/atom.xml" rel="self" type="application/rss+xml"/>
<description>{{&lt; meta description-meta &gt;}}</description>
<generator>quarto-1.11.9</generator>
<lastBuildDate>Thu, 17 Sep 2026 23:00:00 GMT</lastBuildDate>
<item>
  <title>How to save the UK taxpayer £130m a year</title>
  <dc:creator>Arthur Turrell</dc:creator>
  <link>https://www.aeturrell.com/blog/posts/the-case-for-public-sector-os/</link>
  <description><![CDATA[ 





<p>This post is going to enrage some people…</p>
<p>The UK Civil Service <a href="https://www.instituteforgovernment.org.uk/explainer/civil-service-staff-numbers">employs</a> half a million people. Most of them are using Microsoft’s Windows operating system and its associated applications such as Teams and Outlook. Is that value for money for the taxpayer? And are we making the right choice for national sovereignty and security? Let’s explore.</p>
<section id="the-cost-of-the-uks-current-approach" class="level2 page-columns page-full">
<h2 class="anchored" data-anchor-id="the-cost-of-the-uks-current-approach">The cost of the UK’s current approach</h2>
<p>I analysed the UK government’s excellent <a href="https://www.contractsfinder.service.gov.uk">Contracts Finder</a> to find all spending on Microsoft services and products—collectively, there was an eye-watering £576m per year. Some of that is good stuff—I’m pleased to see UK government departments using cloud compute, for example. But even just looking at just Windows, Microsoft365, Teams, SQLServer, PowerBI, and SharePoint, which we’ll call the “core stack”, it was £125m a year.<sup>1</sup></p>
<div class="no-row-height column-margin column-container"><div id="fn1"><p><sup>1</sup>&nbsp;The numbers in this piece are on a best endeavours basis—it is a blog post after all. Depending on how you count, and what time period you look at, there is a fair range. Some examples are £531m–£622m for total spend, and £226–£585 per year-FTE for core stack spending. There’s a cross-check to the experience of one German state later that suggests the potential savings I find are not implausible.</p></div><div id="fn2"><p><sup>2</sup>&nbsp;I also tried to get a ‘matched’ sample of departments’ spends <em>and</em> their number of full-time equivalent (FTE) staff. Using data from various sources on the number of staff employed by the organisations listed by the Contract Finder, I got a spend of £355 per FTE per year on the core Microsoft stack. I’ve used the more conservative number.</p></div></div><p>I probably don’t need to tell you that this is a lot of money in absolute terms, but it might still be a good deal—there are a lot of Civil Servants, after all. It works out to £250 per FTE per year.<sup>2</sup></p>
<p>As a double check, I cross-referenced this with some listed prices for Microsoft’s Enterprise offerings on Microsoft’s <a href="https://www.microsoft.com/en-gb/microsoft-365/enterprise/microsoft-365-plans-and-pricing">own website</a>. The basic version comes in at £402 per user per year, while the premium version is £980 per user per year! The user vs FTE distinction matters because—with more people making use of flexible working patterns—the cost scales with users even if they’re less than 1 FTE.</p>
<p>I’m not someone who advocates that the public sector should use the cheapest and worst version of everything to save money—quite the opposite, and such penny-pinching is self-defeating when you look at value through a broader lens. We’ve all known public sector workers wasting their most valuable asset—their own time—because of poor IT. Not only does this siphon away all enthusiasm for a job, it’s economically non-sensical because of the price of labour relative to the price of computing.</p>
<p>So, are there some cheaper options that are broadly <em>as</em> good, or perhaps even better? We need to look at the products that make up the typical tech stack that the licenses are buying, and ask what out there might do as good a job for free.<sup>3</sup></p>
<div class="no-row-height column-margin column-container"><div id="fn3"><p><sup>3</sup>&nbsp;Free can mean different things. Here, I mean free and open source software (FOSS). This means that while you might have to pay for hosting your data or a server to, well, serve that data up, the software and all its features are free.</p></div></div><ul>
<li>Microsoft Windows operating system. Cheaper option: <a href="https://ubuntu.com">Linux</a> at £0.</li>
<li>Outlook. Cheaper option: <a href="https://www.thunderbird.net/en-GB/">Thunderbird</a> at £0.</li>
<li>Word/Excel/Office suite. Cheaper option: <a href="https://www.libreoffice.org">LibreOffice</a> at £0 (as they say on their website, “no forced AI.”)</li>
<li>SharePoint. More complicated as SharePoint is a few things at once, but a cheaper option for intranet is Django at £0. (hat tip to the Cabinet Office, who already do this), <a href="https://nextcloud.com/files/">Nextcloud Files Community Edition</a> for a shared file system at £0, and <a href="https://github.com/paperless-ngx/paperless-ngx">paperless-ngx</a> at £0.</li>
<li>Teams. Cheaper option: <a href="https://zulip.com">Zulip</a> at £0.</li>
<li>SQL Server. Cheaper option: <a href="https://www.postgresql.org">Postgres</a> at £0.</li>
</ul>
<p>So, the whole software stack could be a towering behemoth of free. I’m not sure how much hosting the data and servers costs, but not nothing, and we need to account for it. Let’s be generous to the UK government’s existing arrangements and assume something like the per user per year cost of a Dropbox account with 2TB of storage and 30 days’ of delete retention. This is a huge amount for most people in the public sector: I’ve had a Gmail account with 2TB of storage since 2004 and I haven’t used 100GB let alone 2TB! The cost per user per year of this basic Dropbox account is around £96. So we’re looking at savings of £154 per year per FTE. Over the 500,000 Civil Servants, that’s a cost saving of £77m per year.</p>
<p>Doing a similar calculation with the range of enterprise offerings listed by Microsoft, we’re talking about saving at least £153—£442m a year.</p>
<p>Don’t like this set of alternative software? Well, the UK state has such size and heft that it could easily hire some software engineers and build a bespoke software application, if it really wanted to. Or even sponsor development of one of the above in the exact direction it desired. AI means it’s now easier than ever to roll your own software. There would be an upfront cost, and some maintenance costs, but likely nothing compared to the Microsoft versions above. And of course, if you felt that only some of these switches away from Microsoft was right for your public sector org, you could choose to do just those.</p>
<p>I’ll also say that some of these options are, at least in my opinion, superior to the MS equivalents: sensible people can disagree but Postgres, Thunderbird, and Linux are genuinely excellent, industry-standard tools.</p>
</section>
<section id="what-happens-if-the-stack-switches-away-from-windows" class="level2 page-columns page-full">
<h2 class="anchored" data-anchor-id="what-happens-if-the-stack-switches-away-from-windows">What happens if the stack switches away from Windows?</h2>
<div class="quarto-figure quarto-figure-center page-columns page-full">
<figure class="figure page-columns page-full">
<p><img src="https://www.aeturrell.com/blog/posts/the-case-for-public-sector-os/tux_visits_no_10.webp" class="img-fluid figure-img" style="width:50.0%"></p>
<figcaption class="margin-caption">Tux, the Linux penguin, takes up residence at No.10</figcaption>
</figure>
</div>
<p>If we did this, would we need to throw out all the Windows laptops that are currently deployed to Civil Servants staff? No! Linux works on most (all?) laptops. So no need for new laptops, which makes this aspect of the transition cheap. And, yes, all of the software alternatives above work on Linux!</p>
<p>Laptops that had been switched to Linux would also likely work better: Linux consumes significantly less RAM at idle, (4.7GB vs 9.3GB <a href="https://www.notebookcheck.net/Windows-11-vs-Linux-RAM-usage-in-web-browsing-and-gaming-reveals-big-difference.1359634.0.html">in one test</a>.) Many public sector laptops might only <em>have</em> 16GB of RAM to play with, so this is a large gain.</p>
<p>Linux also has significantly less bloat: it doesn’t serve ads when you try and search your own computer, it doesn’t force distracting “widgets” on your or AI integrations you didn’t ask for, and it certainly isn’t sending information on your use back to the developers. Linux runs only what you tell it to run, leaving more of your hardware dedicated to your active tasks. When you right-click and are running a standard version of Linux, you instantly get a menu pop-up. My personal experience of the public sector is that Windows stalls, and sometimes crashes, when you have the gall to do something as complex as <a href="https://www.techradar.com/news/annoying-windows-11-bug-finally-gets-fixed">right-clicking a file</a>.</p>
<p>The way that the software is all bundled together is also problematic. There are quite a few people who would say that Excel is the leading spreadsheet software, but you’d be harder pressed to find people who rave about Teams, or enjoy SharePoint’s search function (which <a href="https://blog.adenin.com/sharepoint-search-problems/">doesn’t work</a>.) While the bundling may not be totally forced, it’s <a href="https://www.cnbc.com/2025/09/12/microsoft-avoids-big-fine-as-eu-accepts-deal-to-unbundle-teams.html">certainly encouraged</a>, and you get the good bits with the bad bits. I think our public servants deserve better. There is no such bundling with Linux: you can choose the best tool for each individual job.</p>
<p>If you’ve used Outlook, you’ll know that crashes are just a fact of life. It doesn’t have to be that way, and it shouldn’t. Even if you don’t care for your own time, we should be wiser with the state’s assets. Linux has <a href="https://www.sealevel.com/linux-os">rock-solid stability</a>, which is the reason it’s used for <a href="https://www.linuxfoundation.org/blog/blog/reality-check-supercomputers-still-rule-and-linux-still-rules-them">critical infrastructure and servers</a>. Linux is what Android runs on, it powers the top one million web servers, and it’s the operating system of the world’s fastest supercomputers.</p>
<p>By the way, the time saved is economically meaningful. Let’s run through this. Say the average cost to the Civil Service of 1 FTE is £40k a year, and they lose just 2 minutes a day due to flaky software. Over the year, that’s a cost to the UK of £83m! I think it’s probably more in practice.</p>
<p>Now, of course there will be costs of the switch. The migration will need resources from good IT people, and laptops may be unavailable while they go through the switch-over. Based on the experience of one state in Germany, it looks like a one time switch-over cost of £8m.</p>
</section>
<section id="security-and-sovereignty" class="level2 page-columns page-full">
<h2 class="anchored" data-anchor-id="security-and-sovereignty">Security and Sovereignty</h2>
<p>Windows is known for being prone to viruses, though it’s a target <em>because</em> of its market share, and it’s a lot safer than it used to be. But it’s <a href="https://www.digitaltrends.com/computing/operating-system-security-showdown/">not as secure</a> as the most popular variants of Linux, and there are flavours of Linux that are explicitly focused on security (but would come with painful trade-offs, no doubt.)<sup>4</sup></p>
<div class="no-row-height column-margin column-container"><div id="fn4"><p><sup>4</sup>&nbsp;Qubes OS, OpenBSD, and Kicksecure are examples.</p></div></div><p>The key point is that if the UK government decided that the trade-off between security and productivity/openness of any of the existing variants of Linux didn’t match its needs, it could just create a new one that did.</p>
<p>If you use Windows for critical systems then other people will have deep access to your computer. No single event better demonstrated this than when the unfortunately named American cybersecurity company CrowdStrike launched a flawed update and unleashed the world’s single biggest crash of 8.5 million systems. The crash happened because Microsoft had allowed a very deep form of access to people’s computers by a third party. Such kernel-level access had been closed off by Apple for MacOS in 2020, and it has never been available to third parties running Linux.<sup>5</sup> I think, mostly, the CrowdStrike incident was unfortunate: whether provided by a third party or the creator, any update has the chance to go awry. What’s <em>more</em> concerning with Windows is the deep level of access that can be given to third parties without you even being aware of it. (If you’ve used Windows at all, you’ll also know that an unavoidable update is likely to land just as you’re about to give a presentation or leave the office.)</p>
<div class="no-row-height column-margin column-container"><div id="fn5"><p><sup>5</sup>&nbsp;Microsoft blamed previous EU antitrust rulings that it said <em>made</em> it give kernel-level access to third parties… but the <a href="https://globalcompetitionreview.com/article/eu-pushes-back-claim-regulation-played-role-in-crowdstrike-outages">EU refutes this</a>, and has pointed to the fact that kernel access is not necessary to fulfil the antitrust stipulation.</p></div><div id="fn6"><p><sup>6</sup>&nbsp;However, they presumably have to use Outlook’s search function, so can’t actually find anything.</p></div></div><p>Relatedly, with closed source and proprietary operating systems, there’s <em>always</em> the risk that a foreign state has put a backdoor into the machine. You can see how it would happen: if a company is based in country A, but it sells its operating system to country B, the security services of country A are going to be tempted to use that as a Trojan Horse to take a peek at what’s going on. There is <a href="https://www.theguardian.com/world/2013/jul/11/microsoft-nsa-collaboration-user-data">evidence that the US’ National Security Agency</a> had a backdoor into Outlook<sup>6</sup>:</p>
<blockquote class="blockquote">
<p>Another newsletter entry stated that NSA already had pre-encryption access to Outlook email. “For Prism collection against Hotmail, Live, and Outlook.com emails will be unaffected because Prism collects this data prior to encryption.”</p>
</blockquote>
<p>I don’t know the truth or extent of any access any state has into operating systems or other software. But I do know that if I wanted to protect my operating system from the possibility of a foreign state having a backdoor into it, I would be using an open source operating system that allowed me to know <em>exactly</em> what was going on. And the most mature open source operating system, behind Ubuntu, ChromeOS, and Android, is Linux.</p>
<p>Policymakers are taking note. <a href="https://www.theregister.com/ai-and-ml/2026/07/07/mps-tell-brit-government-sort-out-your-tech-sovereignty-or-get-left-out-in-the-cold/5267076">One UK parliamentary select committee</a> has urged the UK government to do more on the security and sovereignty of its IT.</p>
</section>
<section id="if-this-is-such-a-good-idea-why-havent-more-states-done-it" class="level2 page-columns page-full">
<h2 class="anchored" data-anchor-id="if-this-is-such-a-good-idea-why-havent-more-states-done-it">If this is such a good idea, why haven’t more states done it?</h2>
<p>You got me: it is odd how little switching away from Windows there is. That’s partly because of the bundling, the ubiquity, and that some of the software is loved by millions. I’m personally no fan of Excel, but there are lots of people who love it and—if you want to use spreadsheet software<sup>7</sup>—I’m not sure it has an equal.</p>
<div class="no-row-height column-margin column-container"><div id="fn7"><p><sup>7</sup>&nbsp;Use code instead! See the part of <a href="../public-sector-analytical-efficiency/#getting-a-bad-rap">this post</a> about reproducible analysis.</p></div></div><p>Having said that, there are some examples of organisations getting off Windows. Actually, there are corners of the public sector that use MacOS instead; in the UK, it tends to be the most technical ones. As I’ve <a href="../../../blog/posts/bad-it/index.html">written before</a>, MacBooks are actually a great deal economically compared with what the UK public sector might otherwise provide.</p>
<p>In terms of the switch to Linux, there are a few examples of that too. The French Gendarmerie use a custom variant of Ubuntu called <a href="https://en.wikipedia.org/wiki/GendBuntu">GendBuntu</a>. Germany has been in the vanguard of switching to free and open source software. Schleswig-Holstein estimates that 80% of its work computers run open source software and that it is saving $17.5m a year in licensing costs. They mention <a href="https://www.theregister.com/software/2025/10/31/international-criminal-court-dumps-microsoft-office/680564">40k workers</a>, so this is a saving of £325 per year per FTE—higher than our conservative estimate above. And France’s central government is reportedly looking to move <a href="https://linuxiac.com/france-launches-government-linux-desktop-plan-as-windows-exit-begins/"><em>all</em> ministries from Windows to Linux</a>.</p>
<p>There have been some previous attempts in the UK: <a href="https://github.com/NHSbuntu/nhsbuntu">NHSbuntu</a> was an Ubuntu-based Linux variant developed for the UK’s health service but was taken down by a legal challenge from… the UK Department of Health, who stood to gain from the savings. Apparently, a rename to “National Health Open Source” was not enough to stop the legal challenges. As one of the volunteers <a href="https://www.digitalhealth.net/2018/01/nhos-project-ending-after-legal-challenge/">pointed out</a>, this kind of heavy-handed approach to tinkerers trying to improve public services doesn’t exactly encourage innovation.</p>
<p>There are some other countries that have moved their governments to Linux: the Russian state has developed <a href="https://en.wikipedia.org/wiki/Astra_Linux">Astra Linux</a>, and North Korea has <a href="https://en.wikipedia.org/wiki/Red_Star_OS">Red Star OS</a>. Make of that what you will. I’ll only say that these are countries that are more worried than most about foreign states interfering with their computer systems, and they’ve chosen Linux.</p>
</section>
<section id="conclusion" class="level2 page-columns page-full">
<h2 class="anchored" data-anchor-id="conclusion">Conclusion</h2>
<p>We have a number of estimates of the savings from switching away from Microsoft and its ecosystem now:</p>
<ul>
<li>£77m from looking at historical UK government contracts (a conservative estimate)</li>
<li>£153m—£442m using numbers from Microsoft’s own website</li>
<li>£162.5m based on the experiences of the German state of Schleswig-Holstein</li>
</ul>
<p>Of course, there’s the cost of transition, which Schleswig-Holstein’s experience suggests is a one off cost of £200 per user. Weighed against that there’s the performance benefits at both the intensive (more RAM for what you’re actually doing) and extensive (fewer crashes) margins, worth, extremely conservatively, £83m. One could also argue that the revealed preference of UK voters is that sovereignty is important to them, and though it’s hard to measure the value<sup>8</sup> people might put on this, it’s positive. Security would be tighter too—all the more important given recent high profile hacks.</p>
<div class="no-row-height column-margin column-container"><div id="fn8"><p><sup>8</sup>&nbsp;For context, Brexit—which was partly about sovereignty—probably cost 6 or 8% of UK GDP, so something like £196bn.</p></div></div><p><strong>Using the average of just the headline estimates, there is £131m a year on the table.</strong> This uses the lower end of the range of Microsoft’s <em>own</em> enterprise deals, so, again, is a conservative estimate.</p>
<p>It’s hard to conclude anything but that the UK should look at its free and open source software strategy urgently. £130m a year seems worth having to me.</p>


</section>


 ]]></description>
  <category>public finances</category>
  <category>productivity</category>
  <category>public sector</category>
  <guid>https://www.aeturrell.com/blog/posts/the-case-for-public-sector-os/</guid>
  <pubDate>Thu, 17 Sep 2026 23:00:00 GMT</pubDate>
  <media:content url="https://www.aeturrell.com/blog/posts/the-case-for-public-sector-os/tux_visits_no_10.webp" medium="image" type="image/webp"/>
</item>
<item>
  <title>A politically realistic reform of the triple lock</title>
  <dc:creator>Arthur Turrell</dc:creator>
  <link>https://www.aeturrell.com/blog/posts/politically-realistic-reform-triple-lock/</link>
  <description><![CDATA[ 





<p>The triple lock is in the news again: the UK’s cost of living tsar, appointed by Sir Keir Starmer, <a href="https://www.ft.com/content/21527e4b-c720-4078-9209-61e6314e8dc8?syn-25a6b1a6=1">recently said</a> that it was “mathematically unsustainable and”profoundly unfair.” The <a href="https://obr.uk/frs/fiscal-risks-and-sustainability-july-2026/">OBR estimates</a> that it will cost £15.5bn more a year by the end of this parliament than it would have under average earnings alone; a not insignificant amount (0.5 per cent of GDP) for government spending. <a href="https://www.ft.com/content/05ab5df6-877b-4e5a-a01d-de9f19b52b37?syn-25a6b1a6=1">OBR expects the UK</a> to spend <em>9% of GDP on the triple lock by 2075</em> absent any action. In the short-term, UK spending on state pensions is forecast to hit £180bn in 2030, up from around £146bn in 2025.</p>
<p>However, the political reality of the UK’s ageing voting population is that no party wants to risk saying they will “unlock” this policy.<sup>1</sup> And, equally, many people (sensibly) don’t want to go back to a world where many pensioners live in poverty either.</p>
<div class="no-row-height column-margin column-container"><div id="fn1"><p><sup>1</sup>&nbsp;It’s always true on here, but your regular reminder that these are my views and do not represent the position of any institution.</p></div></div><p>The UK’s triple lock sees the state pension grow at whatever is highest: average earnings, inflation, or 2.5 per cent. Looking at the latest numbers from ONS and <a href="https://www.gov.uk/government/publications/benefit-expenditure-and-caseload-tables-2026">DWP</a>, it looks like spending as a portion of UK GDP and real spending per pensioner are likely to climb from now until 2030.</p>
<p>But what if there’s a way to keep the triple lock but reform it so that it’s more in-tune with what’s actually happening in the economy?</p>
<p>I’m far from the first to suggest something like this. As one example, the IFS’ Heidi Karjalainen <a href="https://www.ft.com/content/330c5f50-c475-4cba-9362-e3430aece4cb">wrote in the FT</a> that an Australian-style “smoothed earnings link” might be better. This essentially targets a fraction of median real full-time earnings. I like this, though its exact mechanism would have to be set out ahead of time to prevent year-to-year tinkering. Something like this is almost certainly first-best ignoring the politics of the triple lock.</p>
<p>My proposal is a bit different, and tries to reflect the <em>political reality</em> around this topic. It’s very hard for someone to get up and say we are getting rid of the triple lock. Even saying that it will be replaced by “smoothed earnings” would likely trigger a backlash because it doesn’t explicitly account for the three variables that the triple lock tracks. <em>Any</em> sense that the triple lock is being broken is likely to generate huge pushback from <a href="https://www.google.com/url?sa=t&amp;source=web&amp;rct=j&amp;opi=89978449&amp;url=https://researchbriefings.files.parliament.uk/documents/CBP-8060/CBP-8060.pdf&amp;ved=2ahUKEwjPh9L17M-VAxUwVKQEHfhdHDIQFnoECCAQAQ&amp;usg=AOvVaw3u0t4K_vGAjVBcEUbAaFyf">the demographic most likely to vote</a>. But a slight change to the maths of the triple lock, while keeping its three components, is much more defensible.</p>
<section id="from-the-max-to-the-geometric-mean" class="level2 page-columns page-full">
<h2 class="anchored" data-anchor-id="from-the-max-to-the-geometric-mean">From the max to the (geometric) mean</h2>
<p>The triple lock of today is</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cmax%5Cleft%5C%7B2.5%5C%25,%20%5Cpi_t,%20%5Comega_t%20%5Cright%5C%7D%0A"></p>
<p>where <img src="https://latex.codecogs.com/png.latex?%5Cpi_t"> is CPI, and <img src="https://latex.codecogs.com/png.latex?%5Comega_t"> is average earnings growth. We could switch that to a different <em>aggregation function</em>. It could be the (arithmetic) mean but we’re comparing three different growth rates here, so the obvious choice is the <em>geometric mean</em>. The growth factors are <img src="https://latex.codecogs.com/png.latex?1%20+%20x_i"> for each series. Here’s the new equation I propose for working out the growth in the state pension:</p>
<p><img src="https://latex.codecogs.com/png.latex?%0Ag(x)%20=%20%5Cleft(%20%5Cprod_%7Bi%7D%20%5Cleft(1%20+%20x_i%5Cright)%20%5Cright)%5E%7B1/%7Cx%7C%7D%20-%201%0A"></p>
<p>where <img src="https://latex.codecogs.com/png.latex?x_i"> are the different growth rates, expressed as fractions. Let’s compare how the year-on-year growth rates look under these two different set ups, in Table&nbsp;1.</p>
<div class="cell" data-execution_count="1">
<div id="tbl-growth-rates" class="cell quarto-float quarto-figure quarto-figure-center anchored" data-execution_count="1">
<figure class="quarto-float quarto-float-tbl figure">
<figcaption class="quarto-float-caption-top quarto-float-caption quarto-float-tbl" id="tbl-growth-rates-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
Table&nbsp;1: Year-on-year growth rates compared
</figcaption>
<div aria-describedby="tbl-growth-rates-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<div class="cell-output cell-output-display" data-execution_count="1">
<div>


<table class="dataframe cell caption-top table table-sm table-striped small" data-border="1">
<thead>
<tr class="header">
<th data-quarto-table-cell-role="th"></th>
<th data-quarto-table-cell-role="th">Current triple lock</th>
<th data-quarto-table-cell-role="th">Geometric mean triple lock</th>
<th data-quarto-table-cell-role="th">CPI</th>
<th data-quarto-table-cell-role="th">Earnings growth</th>
</tr>
<tr class="even">
<th data-quarto-table-cell-role="th">year</th>
<th data-quarto-table-cell-role="th"></th>
<th data-quarto-table-cell-role="th"></th>
<th data-quarto-table-cell-role="th"></th>
<th data-quarto-table-cell-role="th"></th>
</tr>
</thead>
<tbody>
<tr class="odd">
<th data-quarto-table-cell-role="th">2011</th>
<td>5.2</td>
<td>3.4</td>
<td>5.2</td>
<td>2.8</td>
</tr>
<tr class="even">
<th data-quarto-table-cell-role="th">2012</th>
<td>2.5</td>
<td>2.0</td>
<td>2.2</td>
<td>1.5</td>
</tr>
<tr class="odd">
<th data-quarto-table-cell-role="th">2013</th>
<td>2.7</td>
<td>2.1</td>
<td>2.7</td>
<td>1.2</td>
</tr>
<tr class="even">
<th data-quarto-table-cell-role="th">2014</th>
<td>2.5</td>
<td>1.3</td>
<td>1.2</td>
<td>0.5</td>
</tr>
<tr class="odd">
<th data-quarto-table-cell-role="th">2015</th>
<td>2.8</td>
<td>1.3</td>
<td>-0.1</td>
<td>2.8</td>
</tr>
<tr class="even">
<th data-quarto-table-cell-role="th">2016</th>
<td>2.5</td>
<td>1.9</td>
<td>1.0</td>
<td>2.5</td>
</tr>
<tr class="odd">
<th data-quarto-table-cell-role="th">2017</th>
<td>3.0</td>
<td>2.6</td>
<td>3.0</td>
<td>2.2</td>
</tr>
<tr class="even">
<th data-quarto-table-cell-role="th">2018</th>
<td>2.7</td>
<td>2.5</td>
<td>2.4</td>
<td>2.7</td>
</tr>
<tr class="odd">
<th data-quarto-table-cell-role="th">2019</th>
<td>4.0</td>
<td>2.6</td>
<td>1.7</td>
<td>4.0</td>
</tr>
<tr class="even">
<th data-quarto-table-cell-role="th">2020</th>
<td>2.5</td>
<td>-0.2</td>
<td>0.5</td>
<td>-0.9</td>
</tr>
<tr class="odd">
<th data-quarto-table-cell-role="th">2021</th>
<td>8.4</td>
<td>4.1</td>
<td>3.1</td>
<td>8.4</td>
</tr>
<tr class="even">
<th data-quarto-table-cell-role="th">2022</th>
<td>10.1</td>
<td>5.3</td>
<td>10.1</td>
<td>5.5</td>
</tr>
<tr class="odd">
<th data-quarto-table-cell-role="th">2023</th>
<td>8.4</td>
<td>5.3</td>
<td>6.7</td>
<td>8.4</td>
</tr>
<tr class="even">
<th data-quarto-table-cell-role="th">2024</th>
<td>4.4</td>
<td>2.7</td>
<td>1.7</td>
<td>4.4</td>
</tr>
<tr class="odd">
<th data-quarto-table-cell-role="th">2025</th>
<td>4.8</td>
<td>3.6</td>
<td>3.8</td>
<td>4.8</td>
</tr>
</tbody>
</table>

</div>
</div>
</div>
</figure>
</div>
</div>
<p>As expected, the positive geometric mean triple lock is less than or equal to the current triple lock at all times. It is less generous. If the pension bill started at £100 and was only changing due to the lock, you’d expect the final bill in 2025 to be £148.9 under the positive geometric mean triple lock vs £190.9 under the original triple lock. That’s a saving of 22.0% in the level of the state pension today if we had used the positive geometric mean all the way through.<sup>2</sup></p>
<div class="no-row-height column-margin column-container"><div id="fn2"><p><sup>2</sup>&nbsp;The percentage difference won’t match the growth rate of <em>total</em> government spending on the state pension because of changes in the number of pensioners. That, and the <a href="https://www.international-adviser.com/pensions-triple-lock-quietly-shelved-for-2022-23/">triple lock was suspended in 2022/23</a>. But, all else equal, a lower compounding effect—as you get with the geometric triple lock—means a lower cost to the state.</p></div><div id="fn3"><p><sup>3</sup>&nbsp;I’m aware that a negative value for the “uprating” is not politically plausible. In this case, it won’t make too much difference, and it could have a floor of zero.</p></div></div><p>The geometric mean is guaranteed to be less than or equal to the max, so we know this choice would make spending a bit lower on average. I would argue that it also reflects the closer underlying economic reality too. Not as well as explicitly targeting median earnings, for sure. But in a way that gets a fair bit closer to that while still retaining the core of a triple lock that reflects CPI, earnings growth, and 2.5 percent.<sup>3</sup></p>
</section>
<section id="what-it-means-for-policy" class="level2 page-columns page-full">
<h2 class="anchored" data-anchor-id="what-it-means-for-policy">What it means for policy</h2>
<p>The triple lock of today has been acknowledged by many to be fiscally unsustainable. Yet, we do want to uprate the state pension given more elderly people often have no easy way to supplement their income, and we don’t want a country where many pensioners live in poverty. Politically, eliminating the triple lock entirely is extremely difficult—but tweaking it to better reflect the <em>average</em> of CPI, earnings, and 2.5%, rather than the maximum, is—I think—much more politically realistic than the alternatives that have been suggested.</p>
<div id="fig-pensions" class="quarto-float quarto-figure quarto-figure-center anchored page-columns page-full">
<figure class="quarto-float quarto-float-fig figure page-columns page-full">
<div aria-describedby="fig-pensions-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<img src="https://www.aeturrell.com/blog/posts/politically-realistic-reform-triple-lock/pension_under_different_locks.svg" class="img-fluid figure-img">
</div>
<figcaption class="quarto-float-caption-margin quarto-float-caption quarto-float-fig margin-caption" id="fig-pensions-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
Figure&nbsp;1: Spending differences under the two types of lock
</figcaption>
</figure>
</div>
<p>Although the past is not a guide to the future (we’ve had some unusual shocks in recent years), the backward-looking five-year average cumulative growth rate under the geometric triple lock vs the current triple lock is 13.3% vs 23.7%. <strong>Projecting the <em>geometric</em> triple lock out to 2030, it would cost 4.73% of GDP instead of 4.98% under the current triple lock</strong> using DWP estimates.</p>
<p>Figure&nbsp;1 shows the spend over time: this change results in a saving in 2030 of £9.0bn (and £7.0bn a year on average.) Not an insignificant difference, and the average annualised growth rate is still fairly generous to pensioners: it’s a balance. It’s far from perfect. But, given the political constraints, it may be better than any alternative.</p>


</section>


 ]]></description>
  <category>public finances</category>
  <category>public sector</category>
  <guid>https://www.aeturrell.com/blog/posts/politically-realistic-reform-triple-lock/</guid>
  <pubDate>Sun, 12 Jul 2026 23:00:00 GMT</pubDate>
  <media:content url="https://www.aeturrell.com/blog/posts/politically-realistic-reform-triple-lock/pension_under_different_locks.svg" medium="image" type="image/svg+xml"/>
</item>
<item>
  <title>Ten command line tools for 2026</title>
  <dc:creator>Arthur Turrell</dc:creator>
  <link>https://www.aeturrell.com/blog/posts/ten-command-line-tools-for-2026/</link>
  <description><![CDATA[ 





<p>Back in 2020, I wrote a post about <a href="../../../blog/posts/10-lesser-known-python-packages/index.html">ten lesser known Python packages</a>. Coding and data science have changed a lot in those six years: large language models are the most obvious example! But there have been plenty of other developments too. So I wanted to run down <strong>ten command line tools</strong> that are useful for coding and data science in 2026.</p>
<section id="uv" class="level2 page-columns page-full">
<h2 class="anchored" data-anchor-id="uv">1. <strong>uv</strong></h2>
<section id="project-and-dependency-management" class="level3 page-columns page-full">
<h3 class="anchored" data-anchor-id="project-and-dependency-management">Project and dependency management</h3>
<p><a href="https://docs.astral.sh/uv/">Astral’s uv</a> has transformed working in Python. Far from the <a href="https://xkcd.com/1987/">dependency hell</a> of just a couple of years ago, we now have an all-in-one Python package manager, reproducibility tool, and dependency resolver. Data scientists in particular will remember the days of waiting for <code>conda install</code> to work out a dependency tree and install a package, and how tricky reproducible environments are to obtain using conda too. I still use Anaconda’s <a href="https://github.com/conda/conda">conda tool</a> for certain situation because it can install things that aren’t just Python dependencies (its pre-built binaries can be a life saver on Windows.)<sup>1</sup></p>
<div class="no-row-height column-margin column-container"><div id="fn1"><p><sup>1</sup>&nbsp;Top tip: if you need to use conda for anything but are finding it slow, try out its fast drop-in replacement, <a href="https://github.com/mamba-org/mamba">mamba</a>.</p></div></div><p>So why is <strong>uv</strong> so good? Creating the scaffolding for a new project is as simple as <code>uv init</code>, and this gives you a <code>pyproject.toml</code> where all your packages are recorded.</p>
<pre class="text"><code>[project]
name = "hello-world"
version = "0.1.0"
description = "Add your description here"
readme = "README.md"
requires-python = "&gt;=3.10"
dependencies = [
    "pandas&gt;=2.2.3",
    "jupyterlab&gt;=4.3.2",
    "matplotlib&gt;=3.9.3",
    "statsmodels&gt;=0.14.4",
    "graphviz&gt;=0.21",
    "hf-mem&gt;=0.4.4",
]
</code></pre>
<p>Everything works on a per-repo / per-folder basis, with <em>one environment per folder</em> which usually sits in <code>.venv</code>. To add packages is as simple as <code>uv add &lt;packagename&gt;</code>. This is where the magic happens; <strong>uv</strong> is <em>fast</em>:</p>
<div class="quarto-figure quarto-figure-center page-columns page-full">
<figure class="figure page-columns page-full">
<p><img src="https://www.aeturrell.com/blog/posts/ten-command-line-tools-for-2026/uv_speed.svg" class="img-fluid figure-img"></p>
<figcaption class="margin-caption">Chart comparing speed of dependency resolution in uv to alternatives</figcaption>
</figure>
</div>
<p>That speed comes from it being written in Rust.</p>
</section>
<section id="reproducibility" class="level3">
<h3 class="anchored" data-anchor-id="reproducibility">Reproducibility</h3>
<p>While <code>pyproject.toml</code> has high-level dependency information, <code>uv.lock</code>, which gets created when you run <code>uv lock</code> or <code>uv sync</code>, is a cross-platform recipe with exact information about your project’s dependencies. Check it in to version control and you can get as close as you can possibly get to cross-platform reproducibility without what is effectively a one-liner.</p>
<p>When someone else, or you, needs to use the project in a fresh environment, they will get the automatically generated <code>pyproject.toml</code> and <code>uv.lock</code> files. Then, all they need is to navigate to the project folder in the command line, and enter <code>uv sync --frozen</code> to install all of the (same versions of) packages needed for the code. For more on reproducibility with <strong>uv</strong>, look at <a href="https://aeturrell.github.io/coding-for-economists/wrkflow-rap.html#reproducible-python-environments">Coding for Economists</a>.</p>
</section>
<section id="python-versions" class="level3">
<h3 class="anchored" data-anchor-id="python-versions">Python versions</h3>
<p><strong>uv</strong> really is a complete solution because it also comes with the ability to install Python itself, eg:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb2" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb2-1"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">uv</span> python install 3.10</span></code></pre></div></div>
<pre class="text"><code>Installed Python 3.10.19 in 1.26s
 + cpython-3.10.19-macos-aarch64-none (python3.10)</code></pre>
</section>
<section id="standalone-code-scripts" class="level3">
<h3 class="anchored" data-anchor-id="standalone-code-scripts">Standalone code scripts</h3>
<p>One cool but perhaps less well-known feature is to the ability to run self-contained scripts, aka code with script-level dependencies. The use case for this is when you have a single code file that you need to use in different contexts that’s not attached to any wider project. You could make a whole new environment for just that one script but that seems like overkill. Instead, you can declare the dependencies in-line with the script! Let’s say you have a file called <code>example.py</code> containing:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb4" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb4-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># /// script</span></span>
<span id="cb4-2"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># dependencies = [</span></span>
<span id="cb4-3"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#   "requests&lt;3",</span></span>
<span id="cb4-4"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#   "rich",</span></span>
<span id="cb4-5"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># ]</span></span>
<span id="cb4-6"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># ///</span></span>
<span id="cb4-7"></span>
<span id="cb4-8"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> requests</span>
<span id="cb4-9"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> rich.pretty <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> pprint</span>
<span id="cb4-10"></span>
<span id="cb4-11">resp <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> requests.get(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"https://peps.python.org/api/peps.json"</span>)</span>
<span id="cb4-12">data <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> resp.json()</span>
<span id="cb4-13">pprint([(k, v[<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"title"</span>]) <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">for</span> k, v <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">in</span> data.items()][:<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">10</span>])</span></code></pre></div></div>
<p>This can be run with <code>uv run example.py</code>, with the environment and dependencies installed on the fly just to execute the script. I can never remember the syntax at the top of the script, but you can create the file and add dependencies from the command line like so:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb5" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb5-1"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">uv</span> init <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--script</span> example.py <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--python</span> 3.12</span>
<span id="cb5-2"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">uv</span> add <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--script</span> example.py <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'requests&lt;3'</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'rich'</span></span></code></pre></div></div>
</section>
<section id="command-line-tools" class="level3">
<h3 class="anchored" data-anchor-id="command-line-tools">Command line tools</h3>
<p>This is going to get meta because <strong>uv</strong> also really helps with command line tools! There are now a lot of command line tools available as Python packages, which is great. But you don’t always want them associated with specific projects, maybe you want them on hand all the time, ready to jump in and do their thing in whatever circumstance arises. Well, once again, <strong>uv</strong> has you covered.</p>
<p>To install a tool once to use everywhere on your computer, run:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb6" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb6-1"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">uv</span> tool install <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;</span>package<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&gt;</span></span></code></pre></div></div>
<p>To then use the tool, you can do</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb7" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb7-1"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">uvx</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;</span>package<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&gt;</span></span></code></pre></div></div>
<p>Some of the tools feature in the rest of this post are Python packages for the command line and can be installed and used in this way.</p>
</section>
</section>
<section id="sec-hfmem" class="level2 page-columns page-full">
<h2 class="anchored" data-anchor-id="sec-hfmem">2. <strong>hf-mem</strong></h2>
<p>The big story of the last few years has been large language models. Increasingly, <a href="../../../blog/posts/era_local_agentic_llms/index.html">as I’ve argued</a>, we can run them <em>locally</em> too, <strong>as long as they fit in RAM</strong>. <a href="https://github.com/alvarobartt/hf-mem"><code>hf-mem</code></a> is a great little tool you can use to estimate the inference memory requirements for Hugging Face models.</p>
<p>If you didn’t read the previous section, you can install this package once, then use everywhere, with</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb8" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb8-1"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">uv</span> tool install hf-mem</span></code></pre></div></div>
<p>and use it with</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb9" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb9-1"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">uvx</span> hf-mem <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--model-id</span> mlx-community/Qwen3.5-27b-4bit</span></code></pre></div></div>
<p>where you could use any model supported by Hugging Face in place of <code>uvx hf-mem --model-id mlx-community/Qwen3.5-27b-4bit</code>.</p>
<div class="quarto-figure quarto-figure-center page-columns page-full">
<figure class="figure page-columns page-full">
<p><img src="https://www.aeturrell.com/blog/posts/ten-command-line-tools-for-2026/hfmem.gif" class="img-fluid figure-img"></p>
<figcaption class="margin-caption">Using <code>hf-mem</code></figcaption>
</figure>
</div>
</section>
<section id="docling" class="level2">
<h2 class="anchored" data-anchor-id="docling">3. <strong>docling</strong></h2>
<p>The rise of large language models has meant that plain text files (.txt, .md, .qmd, .tex, …) are more important than ever because they can go straight into the context window. Perhaps I’m a simple machine too because, personally, I’ve always liked writing in plain text files over, say, Microsoft Word and usually begin there and convert to other formats later. <a href="https://quarto.org/">Quarto</a> is absolutely excellent for this, and I use it for everything from documents to websites to papers to slides. (I even like svg because you can open it up and read the text inside.)</p>
<p>Not everyone is so keen on plain text as a format though! The internet is a place of PDFs, docxs, Powerpoints, and PNGS. These make it <em>much</em> harder to get the information from them into a format a large language model can work with. But this is where <a href="https://github.com/docling-project/docling"><strong>docling</strong></a> comes in—a truly brilliant package from IBM that can convert PDF, DOCX, PPTX, XLSX, HTML, WAV, MP3, WebVTT, images (PNG, TIFF, JPEG, …), LaTeX, … to plain text files. <strong>docling</strong> is not just a command tool, in fact, for anything complex you probably want to use it from within a Python script. But it does also work as a nifty command line tool when you just want the text out.</p>
<p>As an example, let’s take my <a href="../../../research/dormant-working-papers/hill-bardoscia-turrell/index.html">(dormant) working paper</a> with Ed Hill and Marco Bardoscia on solving heterogeneous agent macro models with deep reinforcement learning, aka <a href="https://arxiv.org/abs/2103.16977">https://arxiv.org/abs/2103.16977</a>.</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb10" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb10-1"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">uvx</span> docling https://arxiv.org/pdf/2103.16977</span></code></pre></div></div>
<p>I’ll just show a segment of the output, which was automatically written to a markdown file</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb11" style="background: #f1f3f5;"><pre class="sourceCode markdown code-with-copy"><code class="sourceCode markdown"><span id="cb11-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">## 2. Background</span></span>
<span id="cb11-2"></span>
<span id="cb11-3">Macroeconomic models seek to explain the behaviour of economic variables such as wages, hours worked, prices, investment, interest rates, the consumption of goods and services, and more, depending on the level of complexity. They do this through...</span></code></pre></div></div>
<p>Yep, in this case, we didn’t even need to download the PDF! <strong>docling</strong> does a good job of navigating the arxiv two-column structure. The equations haven’t come down in latex format, but in symbols, but I reckon an LLM could still make sense of this. For the speed and flexibility of inputs/outputs, this is my favourite to-plain-text tool.</p>
</section>
<section id="skimpy" class="level2 page-columns page-full">
<h2 class="anchored" data-anchor-id="skimpy">4. <strong>skimpy</strong></h2>
<p><em>Full disclosure: I created this package</em></p>
<p><a href="https://aeturrell.github.io/skimpy/"><strong>skimpy</strong></a> is a super-charged version of <a href="https://pandas.pydata.org/"><strong>pandas</strong></a>’ <code>df.describe()</code> that handles a wide range of input data types. You can use it in Python code, but you can also use it as a command line tool. Install with <code>uv tool install skimpy</code> and then, to use it:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb12" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb12-1"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">uvx</span> skimpy some_data.parquet</span></code></pre></div></div>
<div class="quarto-figure quarto-figure-center page-columns page-full">
<figure class="figure page-columns page-full">
<p><img src="https://www.aeturrell.com/blog/posts/ten-command-line-tools-for-2026/skimpy_example.svg" class="img-fluid figure-img"></p>
<figcaption class="margin-caption">Example output</figcaption>
</figure>
</div>
<!-- ```

from skimpy import skim, generate_test_data, skim_get_figure

df = generate_test_data()

skim_get_figure(df, "skimpy_example.svg")
``` -->
<p>You can <code>uvx skimpy some_data.csv</code> too; it works for parquet files, csv files, and even sqlite files. For sqlite files, you’ll need to say which table you want with <code>--table &lt;Your table&gt;</code>; running without this will come back with a list of tables.</p>
<p>Skimpy uses the powerful csv “sniffer” from <a href="https://duckdb.org/">DuckDB</a>, which is why it’s so good at guessing what the data types of the different columns are<sup>2</sup>.</p>
<div class="no-row-height column-margin column-container"><div id="fn2"><p><sup>2</sup>&nbsp;Column data type metadata aren’t carried by CSV files so it’s necessary to guess them. Parquet carries column data types with it.</p></div></div><p>Right now, if you want to save the table, you’ll need to run skimpy through Python code but it’s possible the command line interface will expand in future.</p>
</section>
<section id="vhs" class="level2 page-columns page-full">
<h2 class="anchored" data-anchor-id="vhs">5. <strong>vhs</strong></h2>
<p>Ever wondered how people create those nifty videos of using command line tools? Me too. One really good option is called <a href="https://github.com/charmbracelet/vhs"><strong>vhs</strong></a>.<sup>3</sup> You write a bash script in a “.tape” file that includes recording information and then run it. Here’s the <code>hfmem.tape</code> I wrote for making the video in Section&nbsp;2:</p>
<div class="no-row-height column-margin column-container"><div id="fn3"><p><sup>3</sup>&nbsp;For younger fans of Markov Wanderer, VHS stands for Video Home System. It was a <a href="https://en.wikipedia.org/wiki/VHS">magnetic video tape format</a>.</p></div></div><div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb13" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb13-1"></span>
<span id="cb13-2"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">Output</span> hfmem.gif</span>
<span id="cb13-3"></span>
<span id="cb13-4"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Set up a 1200x600 terminal with 46px font.</span></span>
<span id="cb13-5"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">Set</span> FontSize 25</span>
<span id="cb13-6"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">Set</span> Width 1200</span>
<span id="cb13-7"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">Set</span> Height 700</span>
<span id="cb13-8"></span>
<span id="cb13-9"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Type a command in the terminal.</span></span>
<span id="cb13-10"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">Type</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"uvx hf-mem --model-id mlx-community/Qwen3.5-27B-4bit"</span></span>
<span id="cb13-11"></span>
<span id="cb13-12"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Pause for dramatic effect...</span></span>
<span id="cb13-13"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">Sleep</span> 500ms</span>
<span id="cb13-14"></span>
<span id="cb13-15"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Run the command by pressing enter.</span></span>
<span id="cb13-16"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">Enter</span></span>
<span id="cb13-17"></span>
<span id="cb13-18"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Admire the output for a bit.</span></span>
<span id="cb13-19"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">Sleep</span> 5s</span></code></pre></div></div>
<p>This results in the gorgeous .gif you see earlier in the post. Now, I’d really like the export to be a .svg file. There is a package for that called <a href="https://nbedos.github.io/termtosvg/">termtosvg</a> which is nice too, but it records you as you go and so (for me) looks a lot more clunky.</p>
</section>
<section id="ollama" class="level2">
<h2 class="anchored" data-anchor-id="ollama">6. <strong>ollama</strong></h2>
<p>I’ve said on here that we’ve entered the <a href="../../../blog/posts/era_local_agentic_llms/index.html">era of local agentic AI</a>, and that capability is built on open models. <a href="https://ollama.com/"><strong>ollama</strong></a> provides an easy command line interface to find, download, chat with, and launch open model-based versions of popular CLI versions of Claude and Codex.</p>
<p><img src="https://www.aeturrell.com/blog/posts/era_local_agentic_llms/run_ollama.svg" class="img-fluid"></p>
<p>Follow the download and install instructions on their website. Once you hit <code>ollama</code> in your command line, a text user interface will pop up in your terminal with options to chat to a model directly or launch one of the CLI tools but with a model of your choice (eg Codex with qwen3.5:35b) for agentic coding help. <strong>Ollama</strong> is a game-changer for when you don’t have WiFi or you’ve run out of tokens.</p>
</section>
<section id="pydoclint" class="level2">
<h2 class="anchored" data-anchor-id="pydoclint">7. <strong>pydoclint</strong></h2>
<p>Some would say I’m obsessed with making sure the documentation and code are consistent, or, as someone put it (I forget who) “the documentation is the code.” I like a combination of <a href="https://quarto.org/">Quarto</a> and <a href="https://github.com/machow/quartodoc">QuartoDoc</a> for this; the latter is similar to autodoc: it picks up public functions from your code and automatically creates docs for them drawing on the doc strings and function signatures. But what if your doc strings are stale? You can imagine how this can happen: functions change purpose over time, more types are introduced, and perhaps even the arguments and return types change. Then your function does one thing but says it does something else! Very irritating, not to mention confusing.</p>
<p><a href="https://github.com/jsh9/pydoclint"><strong>pydoclint</strong></a> is a command line tool for checking that your code and your docstrings are consistent across arguments, returns, raises, and more. You can use it as a pre-commit hook on a per-project basis or directly on the command line:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb14" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb14-1"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">uvx</span> pydoclint src/ <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--style</span> google </span></code></pre></div></div>
<p>where <code>--style google</code> sets it to use Google-style docstrings. It’s a Python package, so you can install it with <code>uv tool install pydoclint</code>.</p>
</section>
<section id="ripgrep" class="level2">
<h2 class="anchored" data-anchor-id="ripgrep">8. <strong>ripgrep</strong></h2>
<p><a href="https://github.com/BurntSushi/ripgrep"><strong>ripgrep</strong></a> (<code>rg</code>) is command line tool that recursively searches directories using regex patterns. It’s fast, and you can probably guess why: yes, it’s written in Rust. A nice feature is that it respects your <code>.gitignore</code> file, meaning folders (like <code>.venv</code>) that you probably don’t <em>want</em> to search, are excluded by default.</p>
<p>Here’s a typical example of using it (against this blog):</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb15" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb15-1"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">rg</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"reproducible analytical pipelines"</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--type</span> md</span></code></pre></div></div>
<pre class="text"><code>blog/posts/data-science-maturity/data-science-maturity.md
9:Data science has enormous potential to do good in the public sector. The efficiencies that are possible from automation and reproducible analytical pipelines alone are huge—if you like this is improvement at *existing* tasks. Throw machine learning and advanced analytics into the mix and data science can also complete entirely new tasks, *expanding the horizon of what's possible*. It's an exciting time to be a data scientist.</code></pre>
<p>Although there are built-in types, like <code>md</code>, you can also glob (<code>-g</code>) for any file extension or name you like:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb17" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb17-1"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">rg</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"pydoclint"</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-g</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"*.qmd"</span></span></code></pre></div></div>
<pre class="text"><code>blog/posts/ten-command-line-tools-for-2026/index.qmd
220:## 7. **pydoclint**
224:[**pydoclint**](https://github.com/jsh9/pydoclint) is a command line tool for checking that your code and your docstrings are consistent across arguments, returns, raises, and more. You can use it as a pre-commit hook on a per-project basis or directly on the command line:
227:uvx pydoclint src/ --style google 
230:where `--style google` sets it to use Google-style docstrings. It's a Python package, so you can install it with `uv tool install pydoclint`.</code></pre>
</section>
<section id="cookiecutter" class="level2">
<h2 class="anchored" data-anchor-id="cookiecutter">9. <strong>cookiecutter</strong></h2>
<p><a href="https://github.com/cookiecutter/cookiecutter"><strong>cookiecutter</strong></a> is a cross-platform command-line utility that creates projects from “cookiecutters” (project templates). The classic use case is that you want to create a new project, say in Python, without repeating all the boilerplate like setting up folders for models, notebooks, and different types of data (perhaps with .gitkeeps in.) Aimed at the right template, it can cut out a lot of the effort in getting going with a new repo.</p>
<p>It’s a Python package so you can install it with</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb19" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb19-1"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">uv</span> tool install cookiecutter</span></code></pre></div></div>
<p>Then, to use it, you need a template. Well, luckily, I have two I made earlier! You can read all about my Python <a href="../../../blog/posts/ultra-modern-python-cookiecutters/index.html">research project template and package template in this previous post</a>.</p>
<p>Let’s say you wanted to create a new Python package with all the bells and whistles (like pydoclint.) You would run,</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb20" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb20-1"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">uvx</span> cookiecutter https://github.com/aeturrell/cookiecutter-python-package.git</span></code></pre></div></div>
<p>This will take you through a series of questions about the new project you want to create, then create it: a ready-to-go ultra-modern Python package in just a few commands. I’ve used it myself and it has cut out lots of time and mistakes. I’d say it’s fairly easy to create your own templates too.</p>
</section>
<section id="fzf" class="level2 page-columns page-full">
<h2 class="anchored" data-anchor-id="fzf">10. <strong>fzf</strong></h2>
<p><a href="https://github.com/junegunn/fzf"><strong>fzf</strong></a> is an interactive command-line fuzzy finder: you put a list of stuff in and it lets you fuzzily narrow it down, selecting the line you want. It and <strong>ripgrep</strong> make a good pair: pull out all mentions and then narrow down to the one you’re looking for. In the below example, ripgrep finds all instances of “import pandas” in Python files and then pipes it (<code>|</code>) to fzf for further selection:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb21" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb21-1"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">rg</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"import pandas"</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--type</span> py <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">|</span> <span class="ex" style="color: null;
background-color: null;
font-style: inherit;">fzf</span></span></code></pre></div></div>
<div class="quarto-figure quarto-figure-center page-columns page-full">
<figure class="figure page-columns page-full">
<p><img src="https://www.aeturrell.com/blog/posts/ten-command-line-tools-for-2026/fzf.svg" class="img-fluid figure-img"></p>
<figcaption class="margin-caption">Interactivity of <strong>fzf</strong></figcaption>
</figure>
</div>
<p>Now wait because it gets even better: you can combine these commands with bat<sup>4</sup> too to create a fully-fledged interactive text search with file preview. This is a custom function I’ve defined, called <code>rgf</code>, which combines the three tools.</p>
<div class="no-row-height column-margin column-container"><div id="fn4"><p><sup>4</sup>&nbsp;A command line text file viewer.</p></div></div><p>Here’s what it looks like in practice:</p>
<div class="quarto-figure quarto-figure-center page-columns page-full">
<figure class="figure page-columns page-full">
<p><img src="https://www.aeturrell.com/blog/posts/ten-command-line-tools-for-2026/rgf.svg" class="img-fluid figure-img"></p>
<figcaption class="margin-caption">Example of using “rgf”</figcaption>
</figure>
</div>
<p>Once you’ve found what you’re looking for, hitting return will open the file in Visual Studio Code <em>at the selected line</em>!</p>
<p>To set this up on Mac, you’ll need to <code>brew install</code> <a href="https://github.com/BurntSushi/ripgrep"><strong>rg</strong></a>, <a href="https://github.com/sharkdp/bat"><strong>bat</strong></a>, and <a href="https://github.com/junegunn/fzf"><strong>fzf</strong></a>. Then add the following to your <code>.zshrc</code> file, if you’re using ZSH (amend according to your setup.) You can open the file in an editor with <code>code ~/.zshrc</code>. Then add this to the end of the file:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb22" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb22-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># ripgrep + fzf interactive search → VS Code</span></span>
<span id="cb22-2"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">rgf()</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">{</span></span>
<span id="cb22-3">  <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">local</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">result</span></span>
<span id="cb22-4">  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">result</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">$(</span></span>
<span id="cb22-5">    <span class="ex" style="color: null;
background-color: null;
font-style: inherit;">rg</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--color</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>always <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--line-number</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--no-heading</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--smart-case</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">$@</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">|</span></span>
<span id="cb22-6">      <span class="ex" style="color: null;
background-color: null;
font-style: inherit;">fzf</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--ansi</span> <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">\</span></span>
<span id="cb22-7">          <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--delimiter</span> : <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">\</span></span>
<span id="cb22-8">          <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--preview</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'bat --color=always --highlight-line {2} --map-syntax "*.qmd:Markdown" {1}'</span> <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">\</span></span>
<span id="cb22-9">          <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--preview-window</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'up,60%,border-bottom,+{2}+3/3,~3'</span></span>
<span id="cb22-10">  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">)</span></span>
<span id="cb22-11">  <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">if</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">[[</span> <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">-n</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">$result</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">]];</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">then</span></span>
<span id="cb22-12">    <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">local</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">file</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">line</span></span>
<span id="cb22-13">    <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">file</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">$(</span><span class="bu" style="color: null;
background-color: null;
font-style: inherit;">echo</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">$result</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">|</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">cut</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-d:</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-f1</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">)</span></span>
<span id="cb22-14">    <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">line</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">$(</span><span class="bu" style="color: null;
background-color: null;
font-style: inherit;">echo</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">$result</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">|</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">cut</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-d:</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-f2</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">)</span></span>
<span id="cb22-15">    <span class="ex" style="color: null;
background-color: null;
font-style: inherit;">code</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--goto</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">${file}</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">:</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">${line}</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span></span>
<span id="cb22-16">  <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">fi</span></span>
<span id="cb22-17"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">}</span></span></code></pre></div></div>
<p>Now you can use commands like</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb23" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb23-1"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">rgf</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"import pandas"</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-g</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"*.qmd"</span></span></code></pre></div></div>
<p>to interactively, fuzzily, and quickly search your files.</p>


</section>


 ]]></description>
  <category>code</category>
  <category>data science</category>
  <category>open-source</category>
  <category>python</category>
  <guid>https://www.aeturrell.com/blog/posts/ten-command-line-tools-for-2026/</guid>
  <pubDate>Fri, 13 Mar 2026 00:00:00 GMT</pubDate>
  <media:content url="https://www.aeturrell.com/blog/posts/ten-command-line-tools-for-2026/hfmem.gif" medium="image" type="image/gif"/>
</item>
<item>
  <title>The era of local agents is here</title>
  <dc:creator>Arthur Turrell</dc:creator>
  <link>https://www.aeturrell.com/blog/posts/era_local_agentic_llms/</link>
  <description><![CDATA[ 





<p>Local models—that is, large language models you can run on your own computer—have been okay for chat for a while even if they’re typically some way behind the frontier. I can’t now remember when or how I first got a small large language model running locally, but it was probably via Simon Willison’s <a href="https://github.com/simonw/llm">llm</a> with <a href="https://huggingface.co/mlx-community/Mistral-Small-24B-Instruct-2501-4bit">Mistral-Small-24B-Instruct-2501-4bit</a>, which was released in January 2025. Mistral-Small-24B-Instruct-2501-4bit uses 12.35GB, so it could even run on a machine with 16GB of RAM!</p>
<p>NB: model weights in this post come via the <a href="https://github.com/alvarobartt/hf-mem"><code>hf-mem</code> command line tool</a>, which is really good for quickly checking how much space a particular model needs. To check that Mistral model, I ran <code>hf-mem --model-id mlx-community/Mistral-Small-24B-Instruct-2501-4bit</code>.</p>
<p>Subsequent models really upped the game in terms of performance. Looking through my model history, I can see I also tried the fairly large Llama3.3 model, at 37GB, and the more recent, and pretty-good-actually, gpt-oss-20b-MXFP4-Q8, which weighs in at just 11GB. This low RAM requirement means you can still comfortably run other applications while using it.</p>
<p>By the way, small vision models that can run locally have been phenomenal for a while: see my <a href="../../../blog/posts/radical-new-approach-to-time-use/index.html">previous post on this</a> and the <a href="../../../blog/posts/vision-ai-for-odometer-reading/index.html">one by guest poster Katie Russell</a>.</p>
<p>But it’s been clear for a while that the real value of LLMs, especially for coding, is more agentic behaviours. People have <a href="https://simonwillison.net/tags/agent-definitions/">different definitions</a> of what qualifies as agentic behaviour, but I like LLMs using tools (eg Python interpreter, curl, file reads) iteratively to achieve a goal. Hugging Face has a nice little course on <a href="https://huggingface.co/learn/agents-course/en/unit1/introduction">this</a>.</p>
<p>Agentic models are just so much more useful because they can be intelligent about how they use their context window, they can bring in true information (rather than guessing) when needed, and they can run checks for you. It feels much more like you’re setting an assistant a task, letting them get on with it, and occasionally approving their work. I even wonder if it might make help some managers be clearer as the feedback loop between good instructions and good outputs is so tight.</p>
<p>Claude Code has become <em>the</em> indispensable tool for agentic coding specifically, and it does impressively well on other tasks too. I’ve had it rewrite some 35-year-old Fortran in Rust, build a Python front-end to a Java library, and “vibe-code” entire websites (backend and frontend) from scratch. As many others have noted, the inflection point was around November 2025: that is when being a coder transitioned from “writing code with an LLM aid” to “overseeing code written by an LLM.” I’m personally still finding my way through this transition and how to best work in the world.</p>
<p>But what about running <em>agentic</em> models <em>locally</em>?</p>
<p>Of course, there’s been much excitement about OpenClaw, best expressed by <a href="https://x.com/karpathy/status/2024987174077432126">Andrej Karpathy</a>:</p>
<blockquote class="blockquote">
<p>Bought a new Mac mini to properly tinker with claws over the weekend. The apple store person told me they are selling like hotcakes and everyone is confused :)</p>
</blockquote>
<p>But I’m not sure how many people are <em>actually</em> buying a Mac Mini (£2k for a 64GB version at the time of writing) and then serving up OpenClaw on it over tailscale. Plus OpenClaw is a live dangerously approach. Bing which, in case you had forgotten, is Microsoft’s search engine, has been redirecting anyone who has searched for “OpenClaw Windows” <a href="https://www.theregister.com/2026/03/04/fake_openclaw_installers_malware/">to a virus</a>. That’s before you’ve even installed the real OpenClaw! And there are plenty of <a href="https://smarterarticles.co.uk/the-openclaw-warning-from-viral-sensation-to-security-nightmare">horror stories</a> about security issues. Between the costs and the risks, which need real know-how to work with, this option is not practical for the majority of people using LLMs.</p>
<p>So agentic local models have been off the table for most of us. I haven’t found any local models that can reliably do agentic tool use. Until now.</p>
<p>What has changed is the release of <a href="https://ollama.com/library/qwen3.5:35b">Qwen3.5:35B</a> with “Q4_K_M” quantisation. I know little about quantisation, but it seems like this is an intelligent approach to using 4-bit quantisation that manages to keep most of the performance while cutting out as much as 70% of the RAM requirement. So instead of this being a 67GB model, as with the <a href="https://huggingface.co/Qwen/Qwen3.5-35B-A3B">original on Hugging Face</a>, it’s <strong>only a 24GB model</strong>, and so comfortably fits on a 64GB machine.</p>
<p>Why would you want to run an agentic model fully locally? If you’re anything like me, you may have simply exhausted your subscription tokens and wish to carry on working. Or perhaps you have no WiFi and no phone signal (hello travelling in parts of South West London—you know where you are) and want to get some stuff done. Another motivator is privacy: the recent <a href="https://www.bbc.co.uk/news/articles/c8jxevd8mdyo">leak of users’ confidential emails to Microsoft Copilot</a> show how even enterprise-level security systems can go awry. It’s also just unbelievably cool to turn your own computer into a fully fledged, tool-wielding assistant.</p>
<p>You’re still not going to be blown away by the speed of these mid-tier, &lt;64GB models. In terms of getting through a simple problem, like investigating a test failure or updating docs, they’re simply not as fast as the latest Claude models. But they do have pretty incredible performance nonetheless and, for me, they have now crossed the point where they are <em>fast enough</em> and <em>good enough</em> to be used for serious development.</p>
<section id="getting-started-on-your-own-computer" class="level2">
<h2 class="anchored" data-anchor-id="getting-started-on-your-own-computer">Getting started on your own computer</h2>
<p>You’ll need at least 32GB of RAM, and probably 64GB. As far as I know, the steps work cross-platform though the exact commands will vary (the below commands are for MacOS.)</p>
<p><a href="https://ollama.com/download">Download Ollama</a>, which is an open model serving and management tool. On the command line, run</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb1" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb1-1"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">curl</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-fsSL</span> https://ollama.com/install.sh <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">|</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">sh</span></span></code></pre></div></div>
<p>Download the <a href="https://developers.openai.com/codex/cli/">OpenAI Codex CLI</a>.</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb2" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb2-1"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">brew</span> install codex</span></code></pre></div></div>
<p>You can also use Claude Code CLI but I found Codex worked slightly better with open models (I have no firm evidence for this, it’s just based on experience.)</p>
<p>Next, run Ollama in your terminal and use it to download Qwen3.5:35B</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb3" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb3-1"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">ollama</span> run qwen3.5:35b</span></code></pre></div></div>
<p>This will start an automatic download of the full model then launch a chat when it’s finished. Quit that, then run <code>ollama</code> on your command line. You’ll be given a bunch of options for CLIs. Use the down arrows to get to “Launch Codex” then use the right arrow to switch models. Select <code>qwen3.5:35B</code> and you will see Codex CLI launch with your local model!</p>
<p><img src="https://www.aeturrell.com/blog/posts/era_local_agentic_llms/run_ollama.svg" class="img-fluid"></p>
<p>Now, when you issue a task, you’ll find that the LLM starts using tools to complete that task. All running in your own laptop. Incredible!</p>


</section>

 ]]></description>
  <category>data science</category>
  <category>AI</category>
  <category>open-source</category>
  <guid>https://www.aeturrell.com/blog/posts/era_local_agentic_llms/</guid>
  <pubDate>Mon, 09 Mar 2026 00:00:00 GMT</pubDate>
  <media:content url="https://www.aeturrell.com/blog/posts/era_local_agentic_llms/run_ollama.svg" medium="image" type="image/svg+xml"/>
</item>
<item>
  <title>The Economy As A Complex Evolving System</title>
  <dc:creator>Arthur Turrell</dc:creator>
  <link>https://www.aeturrell.com/blog/posts/economy-complex-evolving-system/</link>
  <description><![CDATA[ 





<p>I have a <a href="https://doi.org/10.37911/9781947864665.11">chapter</a> in a new volume of the <em><a href="https://www.santafe.edu/">Sante Fe Institute</a></em>’s series on <em>The Economy As A Complex Evolving System</em>. Alongside my chapter, which is about what data science and AI can do for understanding increasingly complex economies, there are contributions on:</p>
<ul>
<li>economic complexity analysis (sophistication of product mix)</li>
<li>disciplining economic agent-based models with data (<em>highly recommended</em> if you are building an agent-based model, link <a href="https://doi.org/10.37911/9781947864665.10">here</a>)</li>
<li>how worker flows cause Zipf’s law to in firm sizes</li>
<li>the ways that complexity and network science can help understand and capture systemic financial stability risk</li>
<li>the potential of complexity science for understanding rapid disruptions to labour markets</li>
<li>the (perhaps) missing pieces of firm productivity growth: tacit knowledge and local infrastructure assets that combine non-linearly</li>
<li>how agent-based models can add climate change, and the behavioural responses to climate change, to macroeconomic models</li>
<li>and loads more!</li>
</ul>
<p>In case you’re wondering what complexity economics is, it tries to understand the economy as an adaptive system with emergent behaviours. This is as opposed to seeing everything through an aggregate lens or deriving behaviour from representative (and therefore non-interacting) agents. Many applications see tools from physics, biology, computer science, and network science brought to bear on apparent puzzles in economics. There’s a good introductory paper at the start of the volume.</p>
<p>And, yes, it’s 2026—all the chapters are 100% open access. <a href="https://www.sfipress.org/books/eecs-iv">You can find links to every chapter here</a>.</p>
<p>It really is a treasure trove on what complexity science can add to the study of economics, which, in my view, is a lot.</p>
<p>Oh yeah, and I managed to get an isochrone map of the commute from St Helens in. If you want to know why that’s relevant, you’ll just have to read the chapter!</p>
<div class="quarto-figure quarto-figure-center page-columns page-full">
<figure class="figure page-columns page-full">
<p><img src="https://www.aeturrell.com/research/chapters/turrell-2025/travel_area_web.webp" class="img-fluid figure-img"></p>
<figcaption class="margin-caption">Where can you commute to from St Helen’s, UK, in an hour or less?</figcaption>
</figure>
</div>



 ]]></description>
  <category>economics</category>
  <category>research</category>
  <category>data science</category>
  <guid>https://www.aeturrell.com/blog/posts/economy-complex-evolving-system/</guid>
  <pubDate>Sat, 07 Mar 2026 00:00:00 GMT</pubDate>
  <media:content url="https://www.aeturrell.com/research/chapters/turrell-2025/travel_area_web.webp" medium="image" type="image/webp"/>
</item>
<item>
  <title>Guest post: Experimenting with Vision AI to recognise digits on an odometer reading from a photo</title>
  <dc:creator>Arthur Turrell</dc:creator>
  <link>https://www.aeturrell.com/blog/posts/vision-ai-for-odometer-reading/</link>
  <description><![CDATA[ 





<p><em>Today, I’m excited to feature a guest post from <strong>Dr Katie Russell</strong>, another No10 Innovation Fellow. You can find more information about Katie and her work over at her <a href="https://www.linkedin.com/in/katie-russell-60249b22/?originalSubdomain=uk">LinkedIn page</a>.</em></p>
<p>OCR has been possible since the 1910s. Generative AI brings that capability - and much more - to a chat-like interface to any users with an internet connection, which is inspiring further innovation. Using this technology to assist with meter reading is already at production scale in some energy companies and water companies, e.g.&nbsp;<a href="https://www.goodenergy.co.uk/support/how-to-verify-your-meter/">Good Energy</a>, <a href="https://www.utilityswopshop.co.uk/upload-meter-readings/">Utility SwapShop</a>, <a href="https://anyline.com/news/ocr-meter-to-cash-process#:~:text=How%20OCR%20can%20look%20in,water%20consumption%20for%20conservation%20purposes.">Northumbrian Water</a>.</p>
<p>This article shows how to test this technology on odometer readings from a car’s display, exploring whether lightweight local models are good enough.</p>
<p>Possible use cases would be assisting a mechanic at MOT time, or end users needing an odometer reading when handling insurance, tax or leasing arrangements.</p>
<p>You can test the capability yourself by taking an image of a utility meter or your car’s odometer. Paste it into your favourite (workplace approved, if on workplace IT) consumer AI tool, e.g.&nbsp;Gemini, ChatGPT or other, and ask it ‘what is the meter reading?’</p>
<p>It would be trivial to extend this further, building a demo interface, provide a meter reading, input an optimised prompt, call one of these foundation models and input the clean response directly into a form where mileage estimates were required, say.</p>
<p>However, do we need the full capability of a foundation model with multi‑billion‑parameter weights and cloud‑scale inference infrastructure? Inspired by <a href="../../../blog/posts/radical-new-approach-to-time-use/index.html">A Radical New Approach to Time Use</a>, let’s test a local lightweight model, from the <a href="https://huggingface.co/Qwen">Qwen</a> family created by Alibaba Cloud, but with <a href="https://github.com/Blaizzy/mlx-vlm">MLX</a> tooling to run on my Macbook’s Apple silicon chips rather than the industry server standard NVIDIA CUDA chips. I needed python, torch and xz installed first.</p>
<p>Arthur must have a more powerful Mac than me, as I found Qwen3-VL-4B-Instruct-8Bit hit metal limits, but the smaller <a href="https://huggingface.co/Qwen/Qwen3-VL-2B-Instruct">Qwen3-VL-2B-Instruct-4Bit</a> is deemed to have 98% the performance of the full mode, and worked. I ruled out the older Qwen2-VL-2B-Instruct-4Bit model early in testing as it truncated reads which leads to 10x or 100x under reporting.</p>
<div id="fig-odo" class="quarto-float quarto-figure quarto-figure-center anchored page-columns page-full">
<figure class="quarto-float quarto-float-fig figure page-columns page-full">
<div aria-describedby="fig-odo-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<img src="https://www.aeturrell.com/blog/posts/vision-ai-for-odometer-reading/odometer_one.png" id="fig-odo" class="img-fluid figure-img">
</div>
<figcaption class="quarto-float-caption-margin quarto-float-caption quarto-float-fig quarto-uncaptioned margin-caption" id="fig-odo-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
Figure&nbsp;1
</figcaption>
</figure>
</div>
<p>See the prompt evolution and difference in models for the this image (Figure&nbsp;1):</p>
<table class="caption-top table">
<colgroup>
<col style="width: 28%">
<col style="width: 32%">
<col style="width: 40%">
</colgroup>
<thead>
<tr class="header">
<th>Model</th>
<th>Prompt</th>
<th>Response</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td>Qwen2</td>
<td>Read the odometer</td>
<td>The image shows the dashboard of a motorcycle, displaying the speedometer and tachometer. The speedometer needle is pointing to a speed of 506, and the tachometer needle is pointing to a tachometer reading of 0.</td>
</tr>
<tr class="even">
<td>Qwen2</td>
<td>Read the odometer, look for a number in km or miles and confirm the units</td>
<td>The odometer on the motorcycle shows a number of 506, which is a distance in kilometers.</td>
</tr>
<tr class="odd">
<td>Qwen2</td>
<td>Read the odometer, look for a number in km or miles and confirm the units. make sure you read all the digits</td>
<td>The odometer on the motorcycle shows a number of 506,5 km. The units are kilometers.</td>
</tr>
<tr class="even">
<td>Qwen3</td>
<td>Read the odometer, look for a number in km or miles and confirm the units. make sure you read all the digits</td>
<td>Based on the image provided, the odometer is displaying the number 50670. The units for the odometer are indicated on the display as “km”, which stands for kilometers. Therefore, the odometer reading is 50,670 kilometers.</td>
</tr>
</tbody>
</table>
<p>Images with no odometer reading are also correctly classified, with an example response:</p>
<blockquote class="blockquote">
<p>Based on the image provided, there is no odometer visible. The image is a simple black and white icon of a car. There are no numbers, units of measurement (like km or miles), or any other text that would indicate a distance or odometer reading. Therefore, it is not possible to confirm the units or the number of the odometer.</p>
</blockquote>
<p>I’ve tested fewer than 10 images, as got stuck trying to optimise for this blurry image (Figure&nbsp;2):</p>
<div id="fig-odooriginal" class="quarto-float quarto-figure quarto-figure-center anchored page-columns page-full">
<figure class="quarto-float quarto-float-fig figure page-columns page-full">
<div aria-describedby="fig-odooriginal-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<img src="https://www.aeturrell.com/blog/posts/vision-ai-for-odometer-reading/odometer_original.jpg" id="fig-odooriginal" class="img-fluid figure-img">
</div>
<figcaption class="quarto-float-caption-margin quarto-float-caption quarto-float-fig quarto-uncaptioned margin-caption" id="fig-odooriginal-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
Figure&nbsp;2
</figcaption>
</figure>
</div>
<p>The 8 is misread as a 6 despite careful, and optimised, prompting, e.g.&nbsp;Locate the digital display of the odometer reading. First, describe the shape of the digits you see. Then, read the digits one by one from left to right. Finally, state the full odometer reading and the units.</p>
<p>My next step was to sharpen the image with imagemagick and then run the model against that:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb1" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb1-1"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">magick</span> ~/Downloads/mlxtest5.jpg <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-brightness-contrast</span> 10x40 <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-sharpen</span> 0x2.0 /tmp/temp_contrast.jpg <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">&amp;&amp;</span> <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">\</span></span>
<span id="cb1-2"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">mlx_vlm.generate</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--model</span> mlx-community/Qwen3-VL-2B-Instruct-4bit <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--max-tokens</span> 100 <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--prompt</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Locate the digital display of the odometer reading. First, describe the shape of the digits you see. Then, read the digits one by one from left to right. Finally, state the full odometer reading and the unit"</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--image</span> /tmp/temp_contrast.jpg</span></code></pre></div></div>
<div id="fig-odosharp" class="quarto-float quarto-figure quarto-figure-center anchored page-columns page-full">
<figure class="quarto-float quarto-float-fig figure page-columns page-full">
<div aria-describedby="fig-odosharp-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<img src="https://www.aeturrell.com/blog/posts/vision-ai-for-odometer-reading/odometer_sharp.png" id="fig-odosharp" class="img-fluid figure-img">
</div>
<figcaption class="quarto-float-caption-margin quarto-float-caption quarto-float-fig quarto-uncaptioned margin-caption" id="fig-odosharp-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
Figure&nbsp;3
</figcaption>
</figure>
</div>
<p>Unfortunately the 8 is still misread as a 6 (Figure&nbsp;3).</p>
<p>Note: if you prefer a Chat UI to cli, you can launch one, it’s still running locally:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb2" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb2-1"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">mlx_vlm.chat_ui</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--model</span> mlx-community/Qwen3-VL-2B-Instruct-4bit</span></code></pre></div></div>
<p><img src="https://www.aeturrell.com/blog/posts/vision-ai-for-odometer-reading/odometer_chat.png" class="img-fluid"></p>
<p>I’ll stop there for now, as this was just a couple of hours of experimentation, but it highlights an interesting transition in how we handle ‘physical’ data. I suspect there are specialised models out there even better at reading 7-segment digital displays than the general purpose Qwen family, but the fact that a ‘lightweight’ model can get this close on a standard laptop is awesome for accessibility.</p>
<p>In a world of increasingly smart meters and connected vehicles, we expect mileage, energy and other telemetry will flow seamlessly via APIs directly to insurers, suppliers and mechanics (say). But for now Vision AI is a potentially useful bridge, especially if small and local models can do the job.</p>



 ]]></description>
  <category>AI</category>
  <category>data science</category>
  <category>machine learning</category>
  <category>guest post</category>
  <guid>https://www.aeturrell.com/blog/posts/vision-ai-for-odometer-reading/</guid>
  <pubDate>Thu, 26 Feb 2026 00:00:00 GMT</pubDate>
  <media:content url="https://www.aeturrell.com/blog/posts/vision-ai-for-odometer-reading/odometer_original.jpg" medium="image" type="image/jpeg" height="96" width="144"/>
</item>
<item>
  <title>One trick to stop ticket touts</title>
  <dc:creator>Arthur Turrell</dc:creator>
  <link>https://www.aeturrell.com/blog/posts/one-trick-to-stop-ticket-touts/</link>
  <description><![CDATA[ 





<p>What can be done to stop the ticket resellers?<sup>1</sup> Artists don’t want to charge the maximum they can, perhaps because they want all of their fans to have a chance to attend their gigs, or perhaps because they want to establish a long-term relationship with the fans and feel eye-watering prices would turn some away. But as a consequence, when concert tickets go on sale, resellers—using sophisticated bots—scoop up all the tickets at the low price that the artist intended for fans. Let <img src="https://latex.codecogs.com/png.latex?p"> be the low price the artist intends to charge.</p>
<div class="no-row-height column-margin column-container"><div id="fn1"><p><sup>1</sup>&nbsp;Reselling is not fundamentally bad. Markets are good! Getting the thing to the person who wants it is good! But if you’re an artist who wants ‘true’ fans to come to your concert for less than the market rate, then you want to disincentivise the resellers.</p></div></div><p>In economics terms, the artists are charging below the equilibrium price, ie the one that the fans are willing to bear (even if they would prefer not to!) The artists would like the fans to capture this benefit. But the resellers <em>do</em> charge the equilibrium price, which is <img src="https://latex.codecogs.com/png.latex?p'%3Ep">, and can pocket <img src="https://latex.codecogs.com/png.latex?p'-p"> for every ticket sold (minus the cost of their labour.) By the way, <img src="https://latex.codecogs.com/png.latex?p'/p"> can be substantial: a paper by <span class="citation" data-cites="connolly2018secondary">Connolly and Krueger (2018)</span> found that the mean resale mark-up was 36% but a few 100 percent for superstar artists.</p>
<p>You can make laws to ban reselling but it isn’t easy to enforce them. Might economics have an answer?</p>
<div class="quarto-figure quarto-figure-center page-columns page-full">
<figure class="figure page-columns page-full">
<p><img src="https://www.aeturrell.com/blog/posts/one-trick-to-stop-ticket-touts/gig_demand_supply.svg" class="img-fluid figure-img" style="width:100.0%"></p>
<figcaption class="margin-caption">The economics of gig tickets. The artist wants to charge their loyal fans below the equilibrium price</figcaption>
</figure>
</div>
<section id="using-economics-to-stop-the-resellers" class="level2 page-columns page-full">
<h2 class="anchored" data-anchor-id="using-economics-to-stop-the-resellers">Using economics to stop the resellers</h2>
<p>Artists charge the full, market price for tickets, <img src="https://latex.codecogs.com/png.latex?p'">. But. They then give a rebate at the venue, on the night, of <img src="https://latex.codecogs.com/png.latex?p'-p"> to each person with a ticket. This could be cash or (more safely) money back on a card. This means that the fan has paid <img src="https://latex.codecogs.com/png.latex?p">, and the artist + venue still collect <img src="https://latex.codecogs.com/png.latex?p">.</p>
<p>Resellers get nothing. Here’s why: you’re a reseller, and you buy the ticket at <img src="https://latex.codecogs.com/png.latex?p'">. The only thing you can do is resell it at <img src="https://latex.codecogs.com/png.latex?p'">. But you bought it for that! So your profit is zero.</p>
<p>Now you might worry that resellers buy the tickets anyway, and turn up to the venue to collect their rebate. In this case, they have paid <img src="https://latex.codecogs.com/png.latex?p'"> and get a rebate of <img src="https://latex.codecogs.com/png.latex?p'-p">. So they are down <img src="https://latex.codecogs.com/png.latex?p"> per ticket. The fan doesn’t mind being down <img src="https://latex.codecogs.com/png.latex?p"> per ticket, because the fan only buys one and gets to see the concern of their favoured artist. The ticket reseller, however, probably doesn’t value seeing the artist at <img src="https://latex.codecogs.com/png.latex?p"> for more than a few artists, and certainly doesn’t want more than one ticket. And, for the sake of argument, let’s say the value to them of actually attending any gig is on average zero, then their profit in this case is <img src="https://latex.codecogs.com/png.latex?-p">! They actually lost money by going to the gig themselves and getting the rebate.<sup>2</sup></p>
<div class="no-row-height column-margin column-container"><div id="fn2"><p><sup>2</sup>&nbsp;Even if you sent a gang of people to go and collect the rebates at low labour cost, you’re still down on every ticket.</p></div></div><p>The reason this works is that you need to go to the gig to get the surplus, and the only people who get value from going to the gig are—by definition—fans.</p>
<p>Even better, there’s no need for the government to regulate. The people involved in this market are incentivised to behave themselves. That is far cheaper and more effective than trying to police the market.</p>
<p>There are some limitations of this approach. It might cost the venue more to do the rebates. But they could take this out of the surplus. Let’s say it costs them <img src="https://latex.codecogs.com/png.latex?c"> per ticket to do the rebate. Fans still get <img src="https://latex.codecogs.com/png.latex?p'-p-c"> in surplus! Resellers still get zero. So it’s much closer to what the artist and venue intended.</p>
<p>The real downside of this approach is that fans must have savings enough to purchase an expensive ticket <em>up-front</em>. <span class="citation" data-cites="connolly2018secondary">Connolly and Krueger (2018)</span> found that a Bruce Springsteen ticket was resold at 240% of base price, making a £100 (<img src="https://latex.codecogs.com/png.latex?p">) ticket cost £240 (<img src="https://latex.codecogs.com/png.latex?p'">). So how do you deal with this problem? Well, the counterfactual without the economic trick is that the resellers are still operating and have used sophisticated methods to enable them to resell at <img src="https://latex.codecogs.com/png.latex?p'">. So our fan who can’t up-front a lot of cash has the same initial problem… but without the promise of a rebate at the end!</p>
<p>So there you go, one trick to stop ticket touts… using the power of economic incentives.</p>



</section>


<div id="quarto-appendix" class="default"><section class="quarto-appendix-contents" id="quarto-bibliography"><h2 class="anchored quarto-appendix-heading">References</h2><div id="refs" class="references csl-bib-body hanging-indent" data-entry-spacing="0">
<div id="ref-connolly2018secondary" class="csl-entry">
Connolly, Marie, and Alan B Krueger. 2018. <span>“The Secondary Market for Concert Tickets: Theory and Evidence.”</span> <em>Internat. J. Music Bus. Res</em> 7 (2): 6–51.
</div>
</div></section></div> ]]></description>
  <category>economics</category>
  <category>ideas</category>
  <guid>https://www.aeturrell.com/blog/posts/one-trick-to-stop-ticket-touts/</guid>
  <pubDate>Wed, 22 Oct 2025 23:00:00 GMT</pubDate>
</item>
<item>
  <title>An idea for a radically new approach to time use statistics</title>
  <dc:creator>Arthur Turrell</dc:creator>
  <link>https://www.aeturrell.com/blog/posts/radical-new-approach-to-time-use/</link>
  <description><![CDATA[ 





<p>How we spend our time is one of the most important questions facing any mortal creature.<sup>1</sup> While you can, with effort, earn more money, you cannot—at least in the course of normal life<sup>2</sup>—create more time for yourself. As an extensive recent review of the economics of attention points out <span class="citation" data-cites="loewenstein_economics_2025">(Loewenstein and Wojtowicz 2025)</span>, the time we have is limited and <em>rivalrous</em>: if you spend it on one thing, you cannot spend it on another.</p>
<div class="no-row-height column-margin column-container"><div id="fn1"><p><sup>1</sup>&nbsp;Aren’t all creatures mortal? Perhaps not. <a href="https://en.wikipedia.org/wiki/Turritopsis_dohrnii">Some jellyfish have</a> the ability to revert to an earlier life-cycle stage and start again.</p></div><div id="fn2"><p><sup>2</sup>&nbsp;Astronauts in orbit experience time more slowly than their Earth-bound relatives. It doesn’t mean they get more time overall, however, just that the clock has moved forward more on Earth than they have in space. It’s a <a href="https://www.skyatnightmagazine.com/space-science/age-slower-in-space">small effect</a>: even 1,000 days in space only leads to a clock-difference of 0.027 seconds.</p></div></div><p>How we spend our time is all the more important in an age when competition for our attention is so high. There are many services that have no cost, but which do rely on our attention (and our data too) to make money. In many ways, how we decide to use time is a classic economics problem just like how we decide to use other scarce resources like labour and capital.</p>
<p>This, for me, is why time use is so interesting and why I am such a fan of time-use surveys. Some examples of these are the <a href="https://www.bls.gov/tus/">American Time Use Survey</a> and ONS’ <a href="https://www.ons.gov.uk/peoplepopulationandcommunity/personalandhouseholdfinances/incomeandwealth/datasets/timeuseintheuk">Time use in the UK</a>. To help with the substantial hill that UK public sector productivity has to climb, ONS has helpfully begun a time-use survey specifically targeted at that called <a href="https://www.ons.gov.uk/economy/economicoutputandproductivity/publicservicesproductivity/bulletins/timeuseinthepublicsectorgreatbritain/furtheranalysisfebruary2024">Time use in the public sector</a>.</p>
<p>Time use surveys offer rich and useful information, and one of their hard-to-replace functions is to ask people how they feel or what they intended while they were spending time doing this or that. Here’s a list of just some of the useful info they offer (for more on their usefulness, see <span class="citation" data-cites="joyce1999can">Joyce and Stewart (1999)</span>):</p>
<ul>
<li>quantifying, and establishing the value of, non-market work, helping with inclusivity as more non-market work is performed by women</li>
<li>verifying existing series on hours worked</li>
<li>assessment of changes in quality of life, and helping to understand phenomena like the well-paid working more (apparently contrary to economic theory)</li>
<li>assessing the amount of time spent in education and training</li>
<li>assessing how government policies affect the way individuals spend their non-working time</li>
<li>an aid in measuring well-being</li>
<li>a way to understand the attention economy</li>
</ul>
<p>However, <em>granular</em> time use surveys are expensive to run and may not always be accurate—can you recall what you were doing yesterday at 3pm? How about last week at the same time? Even if you are supposed to be writing down what you’re doing in close to real time, it’s very easy to forget.</p>
<p>Well, very speculatively, and as a complement to those existing surveys, here is an idea for a way to get more granular time-use data.</p>
<section id="an-alternative-and-complementary-approach-to-measuring-time-use" class="level2 page-columns page-full">
<h2 class="anchored" data-anchor-id="an-alternative-and-complementary-approach-to-measuring-time-use">An alternative and complementary approach to measuring time use</h2>
<p>The crux of this idea is to collect time-use data using low-cost wearable computers, or people’s existing devices, and methods from data science/AI. The device would be hung around the neck (as some people do with their phones today) or pinned on (in the manner security guards wear cameras). In this position, it would take regular snapshots of what was happening. Then—really importantly—those snapshots would be processed securely <strong><em>on-device</em></strong> and turned into counts of time use according to a given taxonomy.</p>
<p>We might call this approach of generating numbers on-device <em>federated statistics</em>, in analogy with federated machine learning. The on-device part is designed to protect privacy: we don’t want to know <em>exactly</em> what someone is doing, just how it fits into the time use taxonomy. It may not be possible to boil down a model enough to fit on a small device like a phone or raspberry pi just yet, but it’s likely to happen soon.</p>
<p>There are two options for the wearable device:</p>
<ul>
<li>an app on a mobile phone (as most mobile phones have a camera)</li>
<li>a bespoke device developed expressly for this purpose</li>
</ul>
<section id="mobile-phone-app" class="level3">
<h3 class="anchored" data-anchor-id="mobile-phone-app">Mobile phone app</h3>
<p>This approach has the benefit that a questionnaire that complements the machine learning-driven activity recording could be bundled into the same app as the machine learning itself. It would require the provision of a lanyard that ensures that the mobile phone is front facing, and the development of software that could regularly record photographs, run the images through a machine learning algorithm, and transmit activity counts back to the office. The software could potentially query the participant on the intent behind each activity each day too.</p>
<p>There are a couple of downsides to the mobile phone app approach: first, mobile phone batteries may run out faster than participants are normally used to, which could be annoying or could modify the results; and, second, the transmission of counts and survey responses would require data—which would cost users and may put some off from participating or lead to sample selection issues.</p>
<p>However, most people have a smartphone and they’re increasingly powerful and energy efficient. The phone is already connected to the mobile network for data upload. So this approach has potential. Better yet, the statistical agency would only need to send out lanyards (cheap) and write the software.</p>
</section>
<section id="a-bespoke-wearable-device" class="level3">
<h3 class="anchored" data-anchor-id="a-bespoke-wearable-device">A bespoke wearable device</h3>
<p>A bespoke wearable device is more intrusive than a mobile phone, and the statistical office would have to pay for the build too.</p>
<section id="build" class="level4">
<h4 class="anchored" data-anchor-id="build">Build</h4>
<p>Each wearable device would <a href="https://learn.adafruit.com/raspberry-pi-wearable-time-lapse-camera/overview">consist of</a>:</p>
<ul>
<li>a Raspberry Pi Zero 2 W, a small (65mm x 30mm x 5mm), wifi-enabled, low-cost (&lt;£4.80) computer</li>
<li>a Raspberry Pi Camera Board v2, with 8 megapixel resolution (£21)</li>
<li>a Raspberry Pi Zero Camera Adapter (£5)</li>
<li>an 8GB MicroSD card (£4)</li>
<li>a battery than can last for around 7 hours (Lithium Ion Polymer Battery - 3.7v 1800mAh; £11)</li>
<li>a USB charging device (PowerBoost 500 Charger - Rechargeable 5V Lipo USB Boost @ 500mA+; £15)</li>
<li>an LED light (Adafruit LED Sequins; £1)</li>
<li>a button (Tactile Switch Buttons, 6mm slim; £0.20)</li>
<li>a slide switch (Breadboard-friendly SPDT Slide Switch; £1)</li>
<li>a 3D printed case or produced case that fits a Zero 2 W and camera (eg this <a href="https://uk.pi-supply.com/products/official-raspberry-pi-zero-case">official one</a>; £6)</li>
<li>a lanyard suitable for a Raspberry Pi case (eg this <a href="https://thepihut.com/products/adafruit-double-hook-lanyard-in-adafruit-black-ada3982">Adafruit one</a>; £1.50)</li>
</ul>
<p>The total cost per unit is £70—£80. The components would need to be assembled.</p>
<p>With regards to software, the following need to be installed on the device:</p>
<ul>
<li>the Raspbian Jessie Lite operating system</li>
<li>packages to make the camera take photos (eg <a href="http://wiringpi.com/">WiringPi</a> or <a href="https://picamera.readthedocs.io/">Picamera</a>)</li>
<li>a version of Python</li>
<li>a custom-trained and fine-tuned version of the machine learning tinyYOLO package or <a href="https://github.com/CiscoBlockChain/YOLO-Pi">YOLO-pi</a> package, for turning images into counts of objects. Alternatively a modern, small vision model that can also do object recognition</li>
</ul>
<p>Then there’s the question of how to get the counts back to a central server. There are three options:</p>
<ul>
<li>physically return the units and plug them in. Manual, slow.</li>
<li>use the Wifi connectivity… but this requires some way for users to input their Wifi network passwords</li>
<li>add pre-set mobile connectivity to the devices. This comes at considerable extra expense, at least £80 and likely more. Plus extra for network costs.</li>
</ul>
</section>
<section id="operation" class="level4">
<h4 class="anchored" data-anchor-id="operation">Operation</h4>
<p>After assembly and installation, it need only be turned on for it to begin capturing images at a pre-defined interval. The status LED light will turn ON while the camera is taking a picture.</p>
<p>Participants can turn the camera off by pressing and holding the halt button for 2 seconds. The status LED will turn ON while the system performs a clean shutdown (about 20 seconds), then OFF to indicate it has shut down.</p>
</section>
</section>
<section id="which-device-approach-is-best" class="level3">
<h3 class="anchored" data-anchor-id="which-device-approach-is-best">Which device approach is best?</h3>
<p>It seems pretty clear that using smartphones, which already have everything that’s needed on a device, is more efficient. And has lower barriers to entry: participants need only download an app and receive a (relatively cheap to produce) lanyard that keeps the camera in place, and they’re good to go.</p>
<p>Now the big downside of the smartphone-taking-photos approach is that many people today spend a lot of time actually <em>on</em> their mobile phones. But don’t worry! Most modern smartphones track people’s use of apps. If this data could also be released to the time use app then it could be recorded in broad categories as well, and would show how that time was being spent.</p>
</section>
<section id="how-feasible-is-a-time-use-smartphone-app" class="level3 page-columns page-full">
<h3 class="anchored" data-anchor-id="how-feasible-is-a-time-use-smartphone-app">How feasible is a time use smartphone app?</h3>
<p>Can you get an object detection model working on a smartphone? Yes! Ultralytics has made YOLO models for <a href="https://docs.ultralytics.com/hub/app/android/">Android</a> and <a href="https://docs.ultralytics.com/hub/app/ios/">iOS</a>. And language models are coming on-stream too, for example mobile vision language model <a href="https://github.com/Meituan-AutoML/MobileVLM">MobileVLM</a>. Eventually, the tech will get to a point where this works on devices.</p>
<p>And just to show that it can work at all let’s take this image:</p>
<div class="quarto-figure quarto-figure-center page-columns page-full">
<figure class="figure page-columns page-full">
<p><img src="https://www.aeturrell.com/blog/posts/radical-new-approach-to-time-use/mystery_activity.png" class="img-fluid figure-img"></p>
<figcaption class="margin-caption">A picture from everyday life: but what is happening?</figcaption>
</figure>
</div>
<p>We’ll now use Simon Willison’s <a href="https://llm.datasette.io/">llm</a> package to run Claude 4.5 Sonnet against it to see what’s happening here.</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb1" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb1-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># install llm</span></span>
<span id="cb1-2"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">uv</span> tool install llm</span>
<span id="cb1-3"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># install the anthropic plugin</span></span>
<span id="cb1-4"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">llm</span> install llm-anthropic</span>
<span id="cb1-5"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># set your API key</span></span>
<span id="cb1-6"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">llm</span> keys set anthropic</span>
<span id="cb1-7"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Paste key here</span></span></code></pre></div></div>
<p>Now, with the above image,</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb2" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb2-1"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">llm</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-m</span> claude-sonnet-4.5 <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'what activity is happening in this image?'</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-a</span> mystery_activity.png</span></code></pre></div></div>
<p>Response:</p>
<pre class="text"><code>In this image, someone is **washing dishes** by hand. You can see hands using a blue sponge or cloth to clean a white plate over a kitchen sink. There are other dishes visible in the sink and surrounding area, indicating dishwashing is in progress.</code></pre>
<p>Okay, now we ask the model to put this finding straight into a category of activity using <em>schemas</em>, which are supported by Anthropic and <em>llm</em>. Our schema is going to be defined in JSON below—note that, as an example, we’ll only use three possible options but you could reasonably have more. They do not need to be mutually exclusive either, but for simplicity let’s assume that no-one can run and wash dishes at the same time.</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb4" style="background: #f1f3f5;"><pre class="sourceCode json code-with-copy"><code class="sourceCode json"><span id="cb4-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">{</span></span>
<span id="cb4-2">  <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"type"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"object"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span></span>
<span id="cb4-3">  <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"properties"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">{</span></span>
<span id="cb4-4">    <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"activity"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">{</span></span>
<span id="cb4-5">      <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"type"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"string"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span></span>
<span id="cb4-6">      <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"description"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"The activity being performed in the image."</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span></span>
<span id="cb4-7">      <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"enum"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">[</span></span>
<span id="cb4-8">        <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"running"</span><span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">,</span></span>
<span id="cb4-9">        <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"writing"</span><span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">,</span></span>
<span id="cb4-10">        <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"washing dishes"</span></span>
<span id="cb4-11">      <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">]</span></span>
<span id="cb4-12">    <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">}</span></span>
<span id="cb4-13">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">},</span></span>
<span id="cb4-14">  <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"required"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">[</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"activity"</span><span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">]</span></span>
<span id="cb4-15"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">}</span></span></code></pre></div></div>
<p>I’ll save this as activity.json</p>
<p>Let’s load this in to <em>llm</em> as a schema and save it:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb5" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb5-1"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">llm</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--schema</span> activity.json <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--save</span> activity</span></code></pre></div></div>
<p>And now we can use it!</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb6" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb6-1"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">llm</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-m</span> claude-sonnet-4.5 <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--schema</span> t:activity <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-a</span> mystery_activity.png</span></code></pre></div></div>
<p>Response:</p>
<pre class="text"><code>{"activity": "washing dishes"}</code></pre>
<p>I don’t think it’s at all unlikely that we’ll have this kind of ability running on-device in the near future. And, unlike the raw images, the privacy concerns around sending back to stats HQ a log that the user is “running”, “writing”, or “washing dishes” are minimal.</p>
</section>
</section>
<section id="conclusion" class="level2">
<h2 class="anchored" data-anchor-id="conclusion">Conclusion</h2>
<p>As I’ve <a href="../future-of-statistics/#use-of-ai">written before</a>, AI gives us opportunities to produce statistics in new and more insightful ways. Here’s hoping that time use statistics gets this kind of attention in the near future.</p>
</section>
<section id="update-2025-10-17" class="level2">
<h2 class="anchored" data-anchor-id="update-2025-10-17">Update 2025-10-17</h2>
<p>This post got a lot of engagement. Part of the discussions I’ve had subsequently have been about whether <em>local</em> models would be up to the job of doing the categorisation. After all, as of right now, Claude Sonnet 4.5 is one of the top models out there.</p>
<p>Well, the answer is that an under-7GB model running locally on a MacBook can do this job just as well! Using the very convenient MLX version of Qwen3-VL-4B-Instruct-8Bit (details on the <a href="https://github.com/Blaizzy/mlx-vlm">MLX VL package here</a>, and info on <a href="https://huggingface.co/mlx-community/Qwen3-VL-4B-Instruct-8bit">the model is here</a>), this is the command:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb8" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb8-1"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">mlx_vlm.generate</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--model</span> mlx-community/Qwen3-VL-4B-Instruct-8bit <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--max-tokens</span> 100 <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--prompt</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'output as JSON matching the following schema, returning only the activity field $(cat activity.json)'</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--image</span> mystery_activity.png</span></code></pre></div></div>
<pre class="text"><code>==========
Files: ['mystery_activity.png'] 

Prompt: &lt;|im_start|&gt;user
&lt;|vision_start|&gt;&lt;|image_pad|&gt;&lt;|vision_end|&gt;output as JSON matching the following schema, returning only the activity field $(cat activity.json)&lt;|im_end|&gt;
&lt;|im_start|&gt;assistant

{
  "activity": "washing dishes"
}
==========
Prompt: 288 tokens, 280.687 tokens-per-sec
Generation: 11 tokens, 54.537 tokens-per-sec
Peak memory: 6.248 GB</code></pre>
<p>While a MacBook is not a mobile phone or a bespoke wearable, it’s a step along the way. Impressive stuff!</p>



</section>


<div id="quarto-appendix" class="default"><section class="quarto-appendix-contents" id="quarto-bibliography"><h2 class="anchored quarto-appendix-heading">References</h2><div id="refs" class="references csl-bib-body hanging-indent" data-entry-spacing="0">
<div id="ref-joyce1999can" class="csl-entry">
Joyce, Mary, and Jay Stewart. 1999. <span>“What Can We Learn from Time-Use Data.”</span> <em>Monthly Lab. Rev.</em> 122: 3.
</div>
<div id="ref-loewenstein_economics_2025" class="csl-entry">
Loewenstein, George, and Zachary Wojtowicz. 2025. <span>“The <span>Economics</span> of <span>Attention</span>.”</span> <em>Journal of Economic Literature</em> 63 (3): 1038–89. <a href="https://doi.org/10.1257/jel.20241665">https://doi.org/10.1257/jel.20241665</a>.
</div>
</div></section></div> ]]></description>
  <category>data science</category>
  <category>ideas</category>
  <category>statistics</category>
  <guid>https://www.aeturrell.com/blog/posts/radical-new-approach-to-time-use/</guid>
  <pubDate>Wed, 01 Oct 2025 23:00:00 GMT</pubDate>
</item>
<item>
  <title>TIL: merging PDFs</title>
  <dc:creator>Arthur Turrell</dc:creator>
  <link>https://www.aeturrell.com/blog/posts/til-merging-pdfs/</link>
  <description><![CDATA[ 





<p>I got the following email:</p>
<blockquote class="blockquote">
<p>I was wondering if it is possible to get your excellent book “coding for economists” as a single PDF? I’m trying to improve my coding skills, and move away from just working in Stata, but I would very much prefer to read the book on my ReMarkable reader rather than on Github?</p>
</blockquote>
<blockquote class="blockquote">
<p>Many thanks for putting in the effort to make such a great text available for the public.</p>
</blockquote>
<p>Well it’s always nice to get emails appreciative of your work. But I have no PDF version of the book!</p>
<p>I had a quick look, however, and remembered that <a href="https://aeturrell.github.io/coding-for-economists/intro.html">each page of the book</a> has a “print to PDF” button. If I could do that on each page, and merge the PDFs, then I could help out the email correspondent.</p>
<p>Thanks to Claude Sonnet 4.5, I vibe coded the following self-contained script that <em>worked perfectly</em>. It created a fairly hefty (60MB+) PDF with every page of the book in to read offline and at leisure. What’s more, this approach could work on any series of webpages as it’s just using the “print to PDF” option.</p>
<p>To use it in this instance it was (assuming you have <a href="https://docs.astral.sh/uv/">uv</a> installed),</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb1" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb1-1"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">uv</span> tool run playwright install chromium</span>
<span id="cb1-2"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">uv</span> run download_and_merge_pdfs.py</span></code></pre></div></div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb2" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb2-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#!/usr/bin/env python3</span></span>
<span id="cb2-2"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># /// script</span></span>
<span id="cb2-3"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># requires-python = "&gt;=3.11"</span></span>
<span id="cb2-4"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># dependencies = [</span></span>
<span id="cb2-5"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#   "requests",</span></span>
<span id="cb2-6"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#   "beautifulsoup4",</span></span>
<span id="cb2-7"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#   "PyPDF2",</span></span>
<span id="cb2-8"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#   "playwright",</span></span>
<span id="cb2-9"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># ]</span></span>
<span id="cb2-10"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># ///</span></span>
<span id="cb2-11"></span>
<span id="cb2-12"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""</span></span>
<span id="cb2-13"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">Script to download PDFs from each page of an online book and combine them.</span></span>
<span id="cb2-14"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">Since the site uses window.print() instead of pre-generated PDFs, we use Playwright to render each page and print to PDF.</span></span>
<span id="cb2-15"></span>
<span id="cb2-16"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">First time setup (installs Chromium):</span></span>
<span id="cb2-17"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    uv tool run playwright install chromium</span></span>
<span id="cb2-18"></span>
<span id="cb2-19"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">Usage with uv:</span></span>
<span id="cb2-20"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    uv run download_and_merge_pdfs.py</span></span>
<span id="cb2-21"></span>
<span id="cb2-22"></span>
<span id="cb2-23"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""</span></span>
<span id="cb2-24"></span>
<span id="cb2-25"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> os</span>
<span id="cb2-26"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> time</span>
<span id="cb2-27"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> requests</span>
<span id="cb2-28"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> bs4 <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> BeautifulSoup</span>
<span id="cb2-29"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> PyPDF2 <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> PdfMerger</span>
<span id="cb2-30"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> urllib.parse <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> urljoin, urlparse</span>
<span id="cb2-31"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> playwright.sync_api <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> sync_playwright</span>
<span id="cb2-32"></span>
<span id="cb2-33"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Configuration</span></span>
<span id="cb2-34">BASE_URL <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"https://aeturrell.github.io/coding-for-economists/"</span></span>
<span id="cb2-35">START_PAGE <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"intro.html"</span></span>
<span id="cb2-36">OUTPUT_FILE <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"coding_for_economists_complete.pdf"</span></span>
<span id="cb2-37">TEMP_DIR <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"temp_pdfs"</span></span>
<span id="cb2-38"></span>
<span id="cb2-39"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> create_temp_dir():</span>
<span id="cb2-40">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""Create temporary directory for PDFs"""</span></span>
<span id="cb2-41">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">if</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">not</span> os.path.exists(TEMP_DIR):</span>
<span id="cb2-42">        os.makedirs(TEMP_DIR)</span>
<span id="cb2-43"></span>
<span id="cb2-44"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> get_page_links(start_url):</span>
<span id="cb2-45">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""Extract all chapter/page links from the sidebar navigation"""</span></span>
<span id="cb2-46">    <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">print</span>(<span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">f"Fetching page structure from </span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span>start_url<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">..."</span>)</span>
<span id="cb2-47">    response <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> requests.get(start_url)</span>
<span id="cb2-48">    response.raise_for_status()</span>
<span id="cb2-49">    soup <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> BeautifulSoup(response.content, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'html.parser'</span>)</span>
<span id="cb2-50">    </span>
<span id="cb2-51">    links <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> []</span>
<span id="cb2-52">    </span>
<span id="cb2-53">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Look for the sidebar navigation in Jupyter Book</span></span>
<span id="cb2-54">    nav <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> soup.select_one(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'nav#bd-docs-nav, nav.bd-links, div.bd-sidebar'</span>)</span>
<span id="cb2-55">    </span>
<span id="cb2-56">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">if</span> nav:</span>
<span id="cb2-57">        <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Find all links in the navigation</span></span>
<span id="cb2-58">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">for</span> link <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">in</span> nav.find_all(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'a'</span>, href<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">True</span>):</span>
<span id="cb2-59">            href <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> link[<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'href'</span>]</span>
<span id="cb2-60">            <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Filter for HTML pages, exclude anchors and external links</span></span>
<span id="cb2-61">            <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">if</span> href.endswith(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'.html'</span>) <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">and</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">not</span> href.startswith(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'http'</span>) <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">and</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">not</span> href.startswith(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'#'</span>):</span>
<span id="cb2-62">                full_url <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> urljoin(BASE_URL, href)</span>
<span id="cb2-63">                <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">if</span> full_url <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">not</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">in</span> links:</span>
<span id="cb2-64">                    links.append(full_url)</span>
<span id="cb2-65">                    <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">print</span>(<span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">f"  Found: </span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span>href<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span>)</span>
<span id="cb2-66">    </span>
<span id="cb2-67">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># If we didn't find links in nav, try to find them anywhere on the page</span></span>
<span id="cb2-68">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">if</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">not</span> links:</span>
<span id="cb2-69">        <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">print</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Nav structure not found, trying alternative method..."</span>)</span>
<span id="cb2-70">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">for</span> link <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">in</span> soup.find_all(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'a'</span>, href<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">True</span>):</span>
<span id="cb2-71">            href <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> link[<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'href'</span>]</span>
<span id="cb2-72">            <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">if</span> href.endswith(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'.html'</span>) <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">and</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">not</span> href.startswith(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'http'</span>) <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">and</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">not</span> href.startswith(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'#'</span>):</span>
<span id="cb2-73">                full_url <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> urljoin(BASE_URL, href)</span>
<span id="cb2-74">                <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">if</span> full_url <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">not</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">in</span> links <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">and</span> BASE_URL <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">in</span> full_url:</span>
<span id="cb2-75">                    links.append(full_url)</span>
<span id="cb2-76">                    <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">print</span>(<span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">f"  Found: </span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span>href<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span>)</span>
<span id="cb2-77">    </span>
<span id="cb2-78">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">return</span> links</span>
<span id="cb2-79"></span>
<span id="cb2-80"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> page_to_pdf_playwright(page_url, output_path):</span>
<span id="cb2-81">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""Use Playwright to render page and save as PDF"""</span></span>
<span id="cb2-82">    <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">print</span>(<span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">f"    Converting to PDF with Playwright..."</span>)</span>
<span id="cb2-83">    </span>
<span id="cb2-84">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">with</span> sync_playwright() <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">as</span> p:</span>
<span id="cb2-85">        browser <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> p.chromium.launch()</span>
<span id="cb2-86">        page <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> browser.new_page()</span>
<span id="cb2-87">        </span>
<span id="cb2-88">        <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Navigate to the page</span></span>
<span id="cb2-89">        page.goto(page_url, wait_until<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'networkidle'</span>)</span>
<span id="cb2-90">        </span>
<span id="cb2-91">        <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Wait a bit for any dynamic content</span></span>
<span id="cb2-92">        page.wait_for_timeout(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1000</span>)</span>
<span id="cb2-93">        </span>
<span id="cb2-94">        <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Print to PDF with good settings</span></span>
<span id="cb2-95">        page.pdf(</span>
<span id="cb2-96">            path<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>output_path,</span>
<span id="cb2-97">            <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">format</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'A4'</span>,</span>
<span id="cb2-98">            margin<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>{</span>
<span id="cb2-99">                <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'top'</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'20mm'</span>,</span>
<span id="cb2-100">                <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'right'</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'20mm'</span>,</span>
<span id="cb2-101">                <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'bottom'</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'20mm'</span>,</span>
<span id="cb2-102">                <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'left'</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'20mm'</span></span>
<span id="cb2-103">            },</span>
<span id="cb2-104">            print_background<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">True</span></span>
<span id="cb2-105">        )</span>
<span id="cb2-106">        </span>
<span id="cb2-107">        browser.close()</span>
<span id="cb2-108">    </span>
<span id="cb2-109">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">return</span> output_path</span>
<span id="cb2-110"></span>
<span id="cb2-111"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> merge_pdfs(pdf_files, output_file):</span>
<span id="cb2-112">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""Merge multiple PDFs into one"""</span></span>
<span id="cb2-113">    <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">print</span>(<span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">f"</span><span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">\n</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">Merging </span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span><span class="bu" style="color: null;
background-color: null;
font-style: inherit;">len</span>(pdf_files)<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;"> PDFs into </span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span>output_file<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">..."</span>)</span>
<span id="cb2-114">    merger <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> PdfMerger()</span>
<span id="cb2-115">    </span>
<span id="cb2-116">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">for</span> pdf_file <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">in</span> pdf_files:</span>
<span id="cb2-117">        <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">print</span>(<span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">f"  Adding: </span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span>os<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">.</span>path<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">.</span>basename(pdf_file)<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span>)</span>
<span id="cb2-118">        merger.append(pdf_file)</span>
<span id="cb2-119">    </span>
<span id="cb2-120">    merger.write(output_file)</span>
<span id="cb2-121">    merger.close()</span>
<span id="cb2-122">    <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">print</span>(<span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">f"Successfully created </span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span>output_file<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span>)</span>
<span id="cb2-123"></span>
<span id="cb2-124"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> cleanup_temp_files(pdf_files):</span>
<span id="cb2-125">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""Remove temporary PDF files"""</span></span>
<span id="cb2-126">    <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">print</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span><span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">\n</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">Cleaning up temporary files..."</span>)</span>
<span id="cb2-127">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">for</span> pdf_file <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">in</span> pdf_files:</span>
<span id="cb2-128">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">try</span>:</span>
<span id="cb2-129">            os.remove(pdf_file)</span>
<span id="cb2-130">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">except</span> <span class="pp" style="color: #AD0000;
background-color: null;
font-style: inherit;">Exception</span> <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">as</span> e:</span>
<span id="cb2-131">            <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">print</span>(<span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">f"  Warning: Could not remove </span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span>pdf_file<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">: </span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span>e<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span>)</span>
<span id="cb2-132">    </span>
<span id="cb2-133">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">try</span>:</span>
<span id="cb2-134">        os.rmdir(TEMP_DIR)</span>
<span id="cb2-135">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">except</span> <span class="pp" style="color: #AD0000;
background-color: null;
font-style: inherit;">Exception</span> <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">as</span> e:</span>
<span id="cb2-136">        <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">print</span>(<span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">f"  Warning: Could not remove </span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span>TEMP_DIR<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">: </span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span>e<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span>)</span>
<span id="cb2-137"></span>
<span id="cb2-138"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> main():</span>
<span id="cb2-139">    create_temp_dir()</span>
<span id="cb2-140">    </span>
<span id="cb2-141">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Get all page links from the table of contents</span></span>
<span id="cb2-142">    start_url <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> urljoin(BASE_URL, START_PAGE)</span>
<span id="cb2-143">    page_links <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> get_page_links(start_url)</span>
<span id="cb2-144">    </span>
<span id="cb2-145">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">if</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">not</span> page_links:</span>
<span id="cb2-146">        <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">print</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Could not find page links. Using just the intro page..."</span>)</span>
<span id="cb2-147">        page_links <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> [start_url]</span>
<span id="cb2-148">    </span>
<span id="cb2-149">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># </span><span class="al" style="color: #AD0000;
background-color: null;
font-style: inherit;">TEST</span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"> MODE: Only do first 3 pages</span></span>
<span id="cb2-150">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># print(f"\n{'='*60}")</span></span>
<span id="cb2-151">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># print(f"TEST MODE: Processing first 3 pages only")</span></span>
<span id="cb2-152">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># print(f"{'='*60}")</span></span>
<span id="cb2-153">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># page_links = page_links[:3]</span></span>
<span id="cb2-154">    </span>
<span id="cb2-155">    <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">print</span>(<span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">f"</span><span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">\n</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">Will process </span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span><span class="bu" style="color: null;
background-color: null;
font-style: inherit;">len</span>(page_links)<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;"> pages"</span>)</span>
<span id="cb2-156">    </span>
<span id="cb2-157">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Convert each page to PDF</span></span>
<span id="cb2-158">    downloaded_pdfs <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> []</span>
<span id="cb2-159">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">for</span> i, page_url <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">in</span> <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">enumerate</span>(page_links, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>):</span>
<span id="cb2-160">        <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">print</span>(<span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">f"</span><span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">\n</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">[</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span>i<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">/</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span><span class="bu" style="color: null;
background-color: null;
font-style: inherit;">len</span>(page_links)<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">] Processing: </span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span>page_url<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span>)</span>
<span id="cb2-161">        </span>
<span id="cb2-162">        <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Create a safe filename from the page URL</span></span>
<span id="cb2-163">        page_name <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> os.path.basename(urlparse(page_url).path)</span>
<span id="cb2-164">        page_name <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> os.path.splitext(page_name)[<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span>]</span>
<span id="cb2-165">        filename <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">f"</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span>i<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:03d}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">_</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span>page_name<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">.pdf"</span></span>
<span id="cb2-166">        output_path <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> os.path.join(TEMP_DIR, filename)</span>
<span id="cb2-167">        </span>
<span id="cb2-168">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">try</span>:</span>
<span id="cb2-169">            page_to_pdf_playwright(page_url, output_path)</span>
<span id="cb2-170">            downloaded_pdfs.append(output_path)</span>
<span id="cb2-171">            <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">print</span>(<span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">f"    ✓ Saved: </span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span>filename<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span>)</span>
<span id="cb2-172">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">except</span> <span class="pp" style="color: #AD0000;
background-color: null;
font-style: inherit;">Exception</span> <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">as</span> e:</span>
<span id="cb2-173">            <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">print</span>(<span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">f"    ✗ Error converting page: </span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span>e<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span>)</span>
<span id="cb2-174">        </span>
<span id="cb2-175">        <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Be nice to the server</span></span>
<span id="cb2-176">        time.sleep(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>)</span>
<span id="cb2-177">    </span>
<span id="cb2-178">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Merge all PDFs</span></span>
<span id="cb2-179">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">if</span> downloaded_pdfs:</span>
<span id="cb2-180">        <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Show info about downloaded files</span></span>
<span id="cb2-181">        <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">print</span>(<span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">f"</span><span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">\n</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'='</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">60</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span>)</span>
<span id="cb2-182">        <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">print</span>(<span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">f"Downloaded Files:"</span>)</span>
<span id="cb2-183">        <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">print</span>(<span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">f"</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'='</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">60</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span>)</span>
<span id="cb2-184">        total_size <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span></span>
<span id="cb2-185">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">for</span> pdf <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">in</span> downloaded_pdfs:</span>
<span id="cb2-186">            size <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> os.path.getsize(pdf) <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">/</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1024</span>  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># KB</span></span>
<span id="cb2-187">            total_size <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+=</span> size</span>
<span id="cb2-188">            <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">print</span>(<span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">f"  </span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span>os<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">.</span>path<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">.</span>basename(pdf)<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">: </span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span>size<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:.1f}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;"> KB"</span>)</span>
<span id="cb2-189">        <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">print</span>(<span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">f"  Total: </span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span>total_size<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:.1f}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;"> KB"</span>)</span>
<span id="cb2-190">        </span>
<span id="cb2-191">        merge_pdfs(downloaded_pdfs, OUTPUT_FILE)</span>
<span id="cb2-192">        </span>
<span id="cb2-193">        cleanup_input <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">input</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span><span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">\n</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">Delete temporary PDF files? (y/n): "</span>)</span>
<span id="cb2-194">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">if</span> cleanup_input.lower() <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'y'</span>:</span>
<span id="cb2-195">            cleanup_temp_files(downloaded_pdfs)</span>
<span id="cb2-196">        <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">else</span>:</span>
<span id="cb2-197">            <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">print</span>(<span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">f"Temporary files kept in: </span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span>TEMP_DIR<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">/"</span>)</span>
<span id="cb2-198">        </span>
<span id="cb2-199">        final_size <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> os.path.getsize(OUTPUT_FILE) <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">/</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1024</span></span>
<span id="cb2-200">        <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">print</span>(<span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">f"</span><span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">\n</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'='</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">60</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span>)</span>
<span id="cb2-201">        <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">print</span>(<span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">f"✓ Complete!"</span>)</span>
<span id="cb2-202">        <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">print</span>(<span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">f"</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'='</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">60</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span>)</span>
<span id="cb2-203">        <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">print</span>(<span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">f"Combined PDF: </span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span>OUTPUT_FILE<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;"> (</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span>final_size<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:.1f}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;"> KB)"</span>)</span>
<span id="cb2-204">        <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">print</span>(<span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">f"</span><span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">\n</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">To process all pages, edit the script and remove the [:3] slice"</span>)</span>
<span id="cb2-205">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">else</span>:</span>
<span id="cb2-206">        <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">print</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span><span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">\n</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">✗ No PDFs were created."</span>)</span>
<span id="cb2-207"></span>
<span id="cb2-208"><span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">if</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">__name__</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"__main__"</span>:</span>
<span id="cb2-209">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Check if playwright is installed</span></span>
<span id="cb2-210">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">try</span>:</span>
<span id="cb2-211">        <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> playwright.sync_api <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> sync_playwright</span>
<span id="cb2-212">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">except</span> <span class="pp" style="color: #AD0000;
background-color: null;
font-style: inherit;">ImportError</span>:</span>
<span id="cb2-213">        <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">print</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Error: Playwright not found."</span>)</span>
<span id="cb2-214">        <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">print</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Install with: uv run playwright install chromium"</span>)</span>
<span id="cb2-215">        exit(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>)</span>
<span id="cb2-216">    </span>
<span id="cb2-217">    main()</span></code></pre></div></div>



 ]]></description>
  <category>TIL</category>
  <category>economics</category>
  <category>AI</category>
  <guid>https://www.aeturrell.com/blog/posts/til-merging-pdfs/</guid>
  <pubDate>Tue, 30 Sep 2025 23:00:00 GMT</pubDate>
</item>
<item>
  <title>Thinking about the economic value of open data</title>
  <dc:creator>Arthur Turrell</dc:creator>
  <link>https://www.aeturrell.com/blog/posts/economic-value-of-data/</link>
  <description><![CDATA[ 





<p>How much economic growth can you generate from making a private dataset open? Let’s create a simple model.</p>
<p>Assume firms have total factor productivity <img src="https://latex.codecogs.com/png.latex?A">, with baseline <img src="https://latex.codecogs.com/png.latex?A_0">. When a firm has access to new data relevant to its production, it moves to TFP <img src="https://latex.codecogs.com/png.latex?A_d%3EA_0"> with</p>
<p><img src="https://latex.codecogs.com/png.latex?%0AA_%7B%5Ctext%7Bdata%7D%7D%20%5Cequiv%20A_d%20=%20A_0(1+%5Cgamma),%5Cqquad%20%5Cgamma%3E0%0A"></p>
<p><img src="https://latex.codecogs.com/png.latex?%5Cgamma"> is the proportional TFP boost from using the data, <img src="https://latex.codecogs.com/png.latex?d">. A fraction <img src="https://latex.codecogs.com/png.latex?f%5Cin(0,1)"> of firms can use <img src="https://latex.codecogs.com/png.latex?d"> (<img src="https://latex.codecogs.com/png.latex?f%5Cll1"> for any given data release). The other <img src="https://latex.codecogs.com/png.latex?1-f"> firms do not directly use the data. However, there are some spillover benefits to them, which have strength <img src="https://latex.codecogs.com/png.latex?%5Cvarphi%5Cin%5B0,1%5D">. The non-data firms’ TFP becomes</p>
<p><img src="https://latex.codecogs.com/png.latex?%0AA_%7B%5Ctext%7Bother%7D%7D%20=%20A_0%5Cbig(1%20+%20%5Cvarphi%5Ccdot%20f%5Ccdot%5Cgamma%5Cbig).%0A"></p>
<p>Aggregate TFP <img src="https://latex.codecogs.com/png.latex?A_%7B%5Cmathrm%7Bagg%7D%7D"> is the weighted average of firm TFPs:</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cbegin%7Baligned%7D%0AA_%7B%5Cmathrm%7Bagg%7D%7D%0A&amp;=%20f%5Ccdot%20A_%7B%5Ctext%7Bdata%7D%7D%20%5C;+%5C;%20(1-f)%5Ccdot%20A_%7B%5Ctext%7Bother%7D%7D%20%5C%5C%0A&amp;=%20f%20A_0(1+%5Cgamma)%20%5C;+%5C;%20(1-f)%20A_0%5Cbig(1+%5Cvarphi%20f%5Cgamma%5Cbig)%5C%5C%0A&amp;=%20A_0%5CBig%5B1%20%5C;+%5C;%20f%5Cgamma%20%5C;+%5C;%20(1-f)%5Cvarphi%20f%5Cgamma%5CBig%5D%5C%5C%0A&amp;=%20A_0%5CBig%5B1%20%5C;+%5C;%20f%5Cgamma%5Cbig(1+%5Cvarphi(1-f)%5Cbig)%5CBig%5D.%0A%5Cend%7Baligned%7D%0A"></p>
<p>If <img src="https://latex.codecogs.com/png.latex?f"> is small, then you have</p>
<p><img src="https://latex.codecogs.com/png.latex?%0AA_%7B%5Cmathrm%7Bagg%7D%7D%5Capprox%20A_0%5Cbig%5B1%20+%20f%5Cgamma(1+%5Cvarphi)%5Cbig%5D.%0A"></p>
<section id="what-is-the-one-off-growth-effect-of-releasing-data" class="level2">
<h2 class="anchored" data-anchor-id="what-is-the-one-off-growth-effect-of-releasing-data">What is the (one-off) growth effect of releasing data?</h2>
<p>Assume capital is fixed in the short run (so output changes proportionally to TFP). Then the immediate jump in aggregate output is</p>
<p><img src="https://latex.codecogs.com/png.latex?%0Ag%20%5Cequiv%20%5Cfrac%7B%5CDelta%20Y%7D%7BY%7D%20%5Capprox%20%5Cfrac%7B%5CDelta%20A_%7B%5Cmathrm%7Bagg%7D%7D%7D%7BA_0%7D%0A=%20f%5Cgamma%5Cbig(1+%5Cvarphi(1-f)%5Cbig)%5Capprox%20f%5Cgamma(1+%5Cvarphi).%0A"></p>
<p>What’s going on here? A small fraction <img src="https://latex.codecogs.com/png.latex?f"> of firms each get a boost <img src="https://latex.codecogs.com/png.latex?%5Cgamma">; spillovers <img src="https://latex.codecogs.com/png.latex?%5Cvarphi"> multiply that effect on the rest of the economy.</p>
<p>Let’s see how this would look for some plausible numbers. Let’s say a dataset was made public that affects 10,000 firms. For reference, <a href="https://www.ons.gov.uk/businessindustryandtrade/business/activitysizeandlocation/adhocs/2447estateagentsbysizeandregion">there are around 25,000 estate agents</a> in the UK. You could imagine a dataset relevant to addresses being made open and some number of those estate agents benefitting from it. The UK has 2.72 million businesses, so we’re looking at <img src="https://latex.codecogs.com/png.latex?f%20%5Capprox%200.37%5C%25">. Let’s say the data makes those firms 1% more productive in TFP terms, so <img src="https://latex.codecogs.com/png.latex?%5Cgamma%20=%201%5C%25">.</p>
<p>And what of spillovers? Not so clear but for research and development investments, it seems like spillovers are around 2.5 times the private increase (see <a href="https://www.frontier-economics.com/media/015adtpq/rate-of-return.pdf">this Frontier Economics</a> article). So let’s run with <img src="https://latex.codecogs.com/png.latex?%5Cvarphi%20=%202.5">.</p>
<p>Plugging those numbers in, you could see a whole economy one-off growth benefit of</p>
<p><img src="https://latex.codecogs.com/png.latex?%0Ag%20=%20f%5Cgamma%5Cbig(1+%5Cvarphi(1-f)%5Cbig)%20=%200.37%5C%25%5Ccdot%201%5C%25%5Ccdot%20(1%20+%202.5(1-0.37%5C%25))%20%5Capprox%200.01%20%5C%25%0A"></p>
<p>Recent UK GDP growth year-on-year has been in the 1% to 2% range, so while 0.01 pp extra is small, it’s not nothing. 2024 UK GDP was 2,851 billion GBP, so we’re talking about 285 million pounds. Worth having!</p>
</section>
<section id="what-does-this-framework-tell-us-about-a-real-case" class="level2">
<h2 class="anchored" data-anchor-id="what-does-this-framework-tell-us-about-a-real-case">What does this framework tell us about a real case?</h2>
<p>There’s a lovely paper by <span class="citation" data-cites="loomis_valuing_2015a">Loomis et al. (2015)</span> that uses contingent valuation to work out that the value of opening up the US’ Landsat satellite imagery. The authors find that the value of opening up the data is 1.8 billion USD. Using this and other numbers from the paper, and our equation, we can back out the productivity boost.</p>
<p>Based on the valuation, the one-time economic benefit is equivalent to 0.012% of US GDP,</p>
<p><img src="https://latex.codecogs.com/png.latex?%5Cfrac%7B%5C$1.8%20%5Ctext%7B%20billion%7D%7D%7B%20%5C$15.094%20%5Ctext%7B%20trillion%7D%7D%20%5Ctimes%20100%20%5Capprox%200.012%5C%25"></p>
<p>This is pretty close to the made-up numbers. For the Landsat case study, there were 13,473 users, so let’s use that for <img src="https://latex.codecogs.com/png.latex?f"> with the US’s 32 million firms. Then</p>
<p><img src="https://latex.codecogs.com/png.latex?%0Af%5Cgamma(1+%5Cvarphi)%20=%20%5Cfrac%7B13473%7D%7B32%5Ctimes10%5E6%7D%20%5Cgamma%20(1+2.5)%20=%200.012%5C%25%20%5Clongrightarrow%20%5Cgamma%20%5Capprox%208.1%5C%25%0A"></p>
<p>This makes sense: compared to the made-up example, the fraction of firms directly affected is substantially smaller at 0.04% but the growth is similar—so the productivity must be proportionately higher.</p>
<p>Of course, this is all just back-of-the-envelope, tons of caveats apply, and there are downsides to releasing (some) data. But, to an order of magnitude, open data can have a positive effect on growth.</p>



</section>

<div id="quarto-appendix" class="default"><section class="quarto-appendix-contents" id="quarto-bibliography"><h2 class="anchored quarto-appendix-heading">References</h2><div id="refs" class="references csl-bib-body hanging-indent" data-entry-spacing="0">
<div id="ref-loomis_valuing_2015a" class="csl-entry">
Loomis, John, Steve Koontz, Holly Miller, and Leslie Richardson. 2015. <span>“Valuing <span>Geospatial Information</span>: <span>Using</span> the <span>Contingent Valuation Method</span> to <span>Estimate</span> the <span>Economic Benefits</span> of <span>Landsat Satellite Imagery</span>.”</span> <em>Photogrammetric Engineering &amp; Remote Sensing</em> 81 (8): 647–56. <a href="https://doi.org/10.14358/PERS.81.8.647">https://doi.org/10.14358/PERS.81.8.647</a>.
</div>
</div></section></div> ]]></description>
  <category>data</category>
  <category>economics</category>
  <category>public sector</category>
  <guid>https://www.aeturrell.com/blog/posts/economic-value-of-data/</guid>
  <pubDate>Wed, 10 Sep 2025 23:00:00 GMT</pubDate>
</item>
<item>
  <title>Media mentions of opposition politicians in the UK</title>
  <dc:creator>Arthur Turrell</dc:creator>
  <link>https://www.aeturrell.com/blog/posts/media-mentions/</link>
  <description><![CDATA[ 





<p>You hear a lot of people claiming that Nigel Farage gets an undue amount of coverage given the number of MPs in his Reform UK party.</p>
<p>I wondered if that was true, so I ran a query against <a href="https://www.gdeltproject.org/">GDELT</a> (Global Database of Events, Language and Tone). GDELT monitors mentions of specific people across broadcast, print, and web news.</p>
<p>And…. it is!</p>
<p><img src="https://www.aeturrell.com/blog/posts/media-mentions/mentions_chart_22de8.svg" class="img-fluid"></p>
<p>You often hear this criticism levelled specifically at the BBC, so I did a cut just for that. And it’s true there too.</p>
<p><img src="https://www.aeturrell.com/blog/posts/media-mentions/mentions_chart_d7b1c.svg" class="img-fluid"></p>
<section id="technical-appendix" class="level2">
<h2 class="anchored" data-anchor-id="technical-appendix">Technical Appendix</h2>
<p>I used BigQuery and the GDELT gkg_partitioned database to get the data. The SQL query is below. The lines are smoothed with <strong>statsmodels</strong>’ <code>lowess</code> function (Locally Weighted Scatterplot Smoothing).</p>
<p>The SQL query is below.</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb1" style="background: #f1f3f5;"><pre class="sourceCode sql code-with-copy"><code class="sourceCode sql"><span id="cb1-1"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">SELECT</span> </span>
<span id="cb1-2">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">EXTRACT</span>(<span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">YEAR</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">FROM</span> PARSE_DATETIME(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'%Y%m%d%H%M%S'</span>, <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">CAST</span>(<span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">DATE</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">AS</span> STRING))) <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">as</span> <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">Year</span>,</span>
<span id="cb1-3">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">EXTRACT</span>(<span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">MONTH</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">FROM</span> PARSE_DATETIME(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'%Y%m%d%H%M%S'</span>, <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">CAST</span>(<span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">DATE</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">AS</span> STRING))) <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">as</span> <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">Month</span>,</span>
<span id="cb1-4">  REGEXP_EXTRACT(DocumentIdentifier, r<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'^(https?://[^/]+)'</span>) <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">as</span> Truncated_URL,</span>
<span id="cb1-5">  <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">CASE</span> </span>
<span id="cb1-6">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">WHEN</span> REGEXP_CONTAINS(Persons, r<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'(?i)nigel.farage'</span>) <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">THEN</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'Nigel Farage'</span></span>
<span id="cb1-7">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">WHEN</span> REGEXP_CONTAINS(Persons, r<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'(?i)ed.davey'</span>) <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">THEN</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'Ed Davey'</span></span>
<span id="cb1-8">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">WHEN</span> REGEXP_CONTAINS(Persons, r<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'(?i)kemi.badenoch'</span>) <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">THEN</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'Kemi Badenoch'</span></span>
<span id="cb1-9">  <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">END</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">as</span> Person_Mentioned,</span>
<span id="cb1-10">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">COUNT</span>(<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span>) <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">as</span> Mentions</span>
<span id="cb1-11"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">FROM</span> `gdelt<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span>bq.gdeltv2.gkg_partitioned`</span>
<span id="cb1-12"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">WHERE</span> </span>
<span id="cb1-13">  <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">DATE</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&gt;=</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">20240101000000</span>  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-- Adjust date range as needed</span></span>
<span id="cb1-14">  <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">AND</span> DocumentIdentifier <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">IS</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">NOT</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">NULL</span></span>
<span id="cb1-15">  <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">AND</span> (</span>
<span id="cb1-16">    REGEXP_CONTAINS(Persons, r<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'(?i)nigel.farage'</span>) <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">OR</span></span>
<span id="cb1-17">    REGEXP_CONTAINS(Persons, r<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'(?i)ed.davey'</span>) <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">OR</span></span>
<span id="cb1-18">    REGEXP_CONTAINS(Persons, r<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'(?i)kemi.badenoch'</span>)</span>
<span id="cb1-19">  )</span>
<span id="cb1-20"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">GROUP</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">BY</span> <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">Year</span>, <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">Month</span>, Truncated_URL, Person_Mentioned</span>
<span id="cb1-21"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">ORDER</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">BY</span> <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">Year</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">DESC</span>, <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">Month</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">DESC</span>, Mentions <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">DESC</span>, Truncated_URL</span></code></pre></div></div>


</section>

 ]]></description>
  <category>data science</category>
  <category>visualisation</category>
  <guid>https://www.aeturrell.com/blog/posts/media-mentions/</guid>
  <pubDate>Thu, 28 Aug 2025 23:00:00 GMT</pubDate>
  <media:content url="https://www.aeturrell.com/blog/posts/media-mentions/mentions_chart_22de8.svg" medium="image" type="image/svg+xml"/>
</item>
<item>
  <title>A re-imagining of how to produce statistics</title>
  <dc:creator>Arthur Turrell</dc:creator>
  <link>https://www.aeturrell.com/blog/posts/future-of-statistics/</link>
  <description><![CDATA[ 





<p>National statistics matter: they have a direct bearing on everything from government targets, to funding formulae, to the focus of the media, to the UK’s debt. But there have been many high-profile errors in the nation’s numbers recently, and that’s a cause for concern. Furthermore, the recent <a href="https://www.gov.uk/government/publications/independent-review-of-the-performance-and-culture-of-the-office-for-national-statistics/independent-review-by-sir-robert-devereux-kcb-june-2025">Devereux</a> and <a href="https://www.ft.com/content/a342f058-1b20-4706-8b62-d408bbf88c22">Public Administration and Constitutional Affairs Committee</a> (PACAC) reviews have found deep, systemic flaws in the UK’s current approach.</p>
<p>I’m a <a href="https://rss.org.uk/policy-campaigns/policy-groups/education-policy-advisory-group/rss-william-guy-lecturers/upcoming-william-guy-lecturers-2025-26/">Royal Statistical Society William Guy Lecturer</a> for 2025-2026, and my topic is “Economic statistics and stupidly smart AI.” With my William Guy Lectureship hat on, I’ve been thinking a lot about how innovation could help with the production of statistics.</p>
<p><strong>If we were to start from scratch, what would a statistical institute designed for the world of today—and tomorrow—look like?</strong> How would you rebuild for the era of AI and data science and cloud, take on board the best practices in management, and ensure the right talent was in place to deliver for the nation?</p>
<p>This blog post is my attempt at answering these questions and giving a rough blueprint for a different model for producing statistics. Please note that these are very much my personal views.</p>
<section id="introduction" class="level2 page-columns page-full">
<h2 class="anchored" data-anchor-id="introduction">Introduction</h2>
<section id="background" class="level3 page-columns page-full">
<h3 class="anchored" data-anchor-id="background">Background</h3>
<p>National statistics are critical infrastructure—essential for decisions that will determine the success of the country. However, the current means of their production has been under pressure. To name a few, there have been recent high-profile errors in <a href="https://www.ft.com/content/61679d49-2b87-45ef-8fa2-1b76e393aed4">trade</a>, <a href="https://www.ons.gov.uk/economy/economicoutputandproductivity/productivitymeasures/datasets/internationalcomparisonsofproductivityfirstestimates">productivity</a><sup>1</sup>, <a href="https://www.ft.com/content/a33e505a-b504-46a9-ba08-e1efd8831a69">GDP</a>, <a href="https://www.ons.gov.uk/news/statementsandletters/errorsidentifiedinonsonlinetimeusesurveyotusdata">time use</a>, <a href="https://www.ons.gov.uk/economy/governmentpublicsectorandtaxes/researchanddevelopmentexpenditure/articles/comparisonofonsbusinessenterpriseresearchanddevelopmentstatisticswithhmrcresearchanddevelopmenttaxcreditstatistics/2022-09-29">innovation</a><sup>2</sup>, <a href="https://www.ft.com/content/41a07d42-2518-4ec9-9220-943268c93bb5">migration</a>, <a href="https://osr.statisticsauthority.gov.uk/publication/review-of-statistics-on-gender-identity-based-on-data-collected-as-part-of-the-2021-england-and-wales-census-final-report/">census</a>, <a href="https://www.ft.com/content/197bb24b-01fd-44b1-9ac7-f70ccf987825">health</a> and <a href="https://www.ft.com/content/7d3de81f-b845-490b-804f-af0aa4f76fe3">labour market figures</a>.</p>
<div class="no-row-height column-margin column-container"><div id="fn1"><p><sup>1</sup>&nbsp;The materials accompanying the <a href="https://www.ons.gov.uk/economy/economicoutputandproductivity/productivitymeasures/datasets/internationalcomparisonsofproductivityfirstestimates">productivity revision</a> said “the previous version suggested that the UK’s average output per hour worked growth rate was 5.0% during the coronavirus (COVID-19) period of 2020 and 2021. The corrected version shows that the UK’s average output per hour growth rate was -0.3% over this period…. the previous version suggested that the UK output per hour worked in 2021 (excluding Japan) had the fastest growth of the G7 countries. This has now been corrected to the second slowest. The previous version also suggested that in 2020 the UK had a fall of 12%, which is now corrected to a 1.2% increase.”</p></div><div id="fn2"><p><sup>2</sup>&nbsp;“…the value of expenditure on R&amp;D performed by UK businesses according to ONS’ BERD survey were £15.0 billion, £15.6 billion, and £16.1 billion higher in 2018, 2019 and 2020 respectively than previously estimated.”</p></div></div><p>The media has noticed—here are some recent headlines:</p>
<ul>
<li><a href="https://www.telegraph.co.uk/news/2025/05/21/office-for-national-statistics-can-no-longer-be-trusted/">Why the UK’s official statistics can no longer be trusted</a>, Daily Telegraph, 2025/05/21.</li>
<li><a href="https://www.ft.com/content/f2ef0740-6420-400f-8a78-abc02936aaa9">Faulty data leaves Britain in the dark: The quality of ONS governance and critical statistics needs fixing</a>, Financial Times, 2025/04/05.</li>
<li><a href="https://www.theguardian.com/uk-news/2025/mar/21/troubled-uk-statistics-agency-warns-of-errors-in-its-growth-figures">Troubled UK statistics agency warns of errors in its growth figures</a>, The Guardian, 2025/03/21.</li>
<li><a href="https://www.instituteforgovernment.org.uk/comment/office-national-statistics-fix-data-problems">The Office for National Statistics must change to fix its data problems</a>, Institute for Government, 2025/03/12.</li>
</ul>
<p>It is not all bleak: there have been incredible developments in UK statistics in recent years. The Covid Infection Survey was a triumph. Though it’s early days, <a href="https://datasciencecampus.ons.gov.uk/estimating-geographical-retail-markets-from-card-spending-data/">card data shows promise</a>. The use of scanner data for inflation is perhaps the strongest contemporary example.<sup>3</sup> Scanner data are created as consumers purchase goods at a supermarket checkout (or online), and they give the product name, volume, and price paid. This is a giant leap compared to field agents travelling around supermarkets with clipboards, who were able to only record the price, and then only for a much smaller number of products. Scanner data make consumer price inflation (CPI) more accurate, give more information on price changes across the economy, and allow more analysis of what products are driving price rises. The adoption of scanner data was an extremely significant step forward for how inflation is measured in the UK.<sup>4</sup></p>
<div class="no-row-height column-margin column-container"><div id="fn3"><p><sup>3</sup>&nbsp;Though the UK <a href="https://www.insee.fr/en/information/5014725?sommaire=5014796">wasn’t the first</a>: others who have adopted scanner data include the Netherlands (2002), Norway (2005), Switzerland (2008), Sweden (2012), Belgium (2015), Denmark (2016), Iceland (2016), Luxembourg and Italy (2018).</p></div><div id="fn4"><p><sup>4</sup>&nbsp;This particular example shows that, with the right conditions, substantial improvements in statistics are possible.</p></div></div><p>It’s also fair to say that there have been headwinds in producing good statistics too: one enormous challenge stems from the long-running issue of declining response rates that has afflicted most national statistical agencies <span class="citation" data-cites="stedman_end_2019">(Stedman et al. 2019)</span>.</p>
<p>However, many other issues have come from copy-and-paste errors or the use of spreadsheets, and could have been avoided. The two reviews from 2025, Devereux and PACAC, follow many others<sup>5</sup> and one recurring theme is that the problems with UK statistics go beyond declining response rates.</p>
<div class="no-row-height column-margin column-container"><div id="fn5"><p><sup>5</sup>&nbsp;Turnbull-King 1999, Allsopp 2004, Smith 2006, Smith 2014, Johnson 2015, Bean 2016, and Lievesley 2024.</p></div></div><p>So I thought it would be fun to imagine what a greenfield design for producing the best quality statistics possible would look like.</p>
</section>
<section id="imagining-better" class="level3">
<h3 class="anchored" data-anchor-id="imagining-better">Imagining better</h3>
<p>Why is a better way of doing statistics easy to imagine? We just have much better tools across technology and organisational science to bring to bear on the production of quantitative outputs than we have ever had before, and they should give us huge optimism that there’s a different path here.</p>
<p>Data science, big data, AI, cloud services, better algorithms, faster hardware, innovations in the management of technical and data-heavy organisations; all could fundamentally change and improve how statistics are produced and delivered <span class="citation" data-cites="turrell2025cuttingcomplexitydatascience">(Turrell 2025)</span>. Studies have suggested that in regular organisations, the use of data science and big data can improve productivity by as much as 7% <span class="citation" data-cites="brynjolfsson_strength_2011 muller_effect_2018">(Brynjolfsson, Hitt, and Kim 2011; <span class="nocase">Müller, Fay, and vom Brocke</span> 2018)</span>. That number should surely be higher for tasks that are solely about the collection, processing, and publication of data.</p>
<p>Also, the management of data-first organisations has been the subject of a great deal of improvement and innovation—with firms like <a href="https://www.amazon.co.uk/Working-Backwards-Insights-Stories-Secrets/dp/1529033829">Amazon</a>, Google, <a href="https://handbook.gitlab.com/handbook/company/culture/all-remote/asynchronous/">GitLab</a>, and more developing ways of working that are suited to the kinds of tasks that statistical production also requires: everything from asynchronous working, to a small and highly skilled workforce, to technical career paths, to “single-threaded” leadership. The management and organisational strategies that have been developed in these frontier firms are part of the reason for their success, and there is every reason to think that other data-centred organisations would benefit from trying them.</p>
<p>Let’s now look at some concrete ideas for what to do differently across technology and capital, people and skills, and management.</p>
</section>
</section>
<section id="technology-and-capital" class="level2 page-columns page-full">
<h2 class="anchored" data-anchor-id="technology-and-capital">Technology and capital</h2>
<p>Prof Bart van Ark, managing director of the Productivity Institute, asseses that around one-third of the UK’s productivity ‘puzzle’ is a result of low capital per worker. The UK’s capital per hour worked lags behind Italy and the Czech Republic <span class="citation" data-cites="allas_uks_2025">(Allas and Zenghelis 2025)</span>. If you were redesigning the production of statistics from scratch, you’d want to ensure that top quality technology and capital were in place.</p>
<section id="use-of-ai" class="level3">
<h3 class="anchored" data-anchor-id="use-of-ai">Use of AI</h3>
<p>AI is just a tool, so I almost didn’t honour it with its own section. However, it’s where some of the most exciting developments could emerge. Here are a few ways it could help:</p>
<ul>
<li>detecting mistakes in survey responses. Machine learning and large language models could significantly increase the rate at which this kind of data error is caught.</li>
<li>flagging outliers. (Similarly to above.)</li>
<li>interview transcription—even when the language is not English!</li>
<li>translation to official taxonomies from free text fields <span class="citation" data-cites="turrell20226 soria2025empirical">(Turrell et al. 2022; Soria 2025)</span>. This is currently a burdensome process with much manual labour, but large language models can be targeted to, as an example, turn respondents’ written job description into official Standard Occupational Classification categories.</li>
<li>imputation of missing responses in early estimates. Machine learning excels at matrix completion, and that would be more sophisticated than what is commonly used today (eg ratio implementation.)</li>
<li><em>conducting</em> survey interviews. Yep, you read that right! See <span class="citation" data-cites="geiecke2024conversations">Geiecke and Jaravel (2024)</span>.</li>
<li>computer vision can extract data from satellite imagery for, say, building starts: Statistics Canada and the US’ Census Bureau are both compiling more accurate, more timely statistics using satellite data <span class="citation" data-cites="erman_use_2022">(Erman et al. 2022)</span></li>
<li>similarly, computer vision can be applied to street scenes to assess the cleanliness and desirability of high streets</li>
<li>accessibility enhancement: AI can automatically generate alternative text for visual content, create audio descriptions, and translate content into multiple languages.</li>
<li>natural language processing can assess news articles for the softer indicators of sentimentn that often lead harder indicators <span class="citation" data-cites="kalamara2022making">(Kalamara et al. 2022)</span></li>
<li>making statistics more accessible to the general public. Large language model-based <a href="https://www.llamaindex.ai/">retrieval augmented generation</a> is able to answer users’ questions based only on certified information, and can link back to its sources. This would allow for a fundamental shift in how people discover and consume statistics. It would also potentially reduce the number of ad hoc requests. My former colleague Andy Banks led <a href="https://datasciencecampus.ons.gov.uk/using-large-language-models-llms-to-improve-website-search-experience-with-statschat/">a project demonstrating the principle</a> already.</li>
<li>help with coding via tools such as GitHub Copilot, Claude Code, and Gemini CLI.</li>
<li>drafting articles that are simply summaries of data updates.</li>
</ul>
</section>
<section id="computing-and-data-infrastructure" class="level3">
<h3 class="anchored" data-anchor-id="computing-and-data-infrastructure">Computing and data infrastructure</h3>
<p>Hardware based on Unix, which is well-suited to data and code, would be available to staff for local development. High performing hardware saves labour, so is cost effective, but it’s also necessary to run large language models and other AI locally—allowing you to quickly try out ideas. Recognising that people’s time is usually the biggest cost in an organisation, top-end laptops would be a no-brainer.</p>
<p>The new automated pipelines would be based on secure cloud services, capitalising on the falling cost, increasing functionality, and scalability of the cloud. These products have <a href="https://aws.amazon.com/managed-workflows-for-apache-airflow/sla/">typical service level agreements</a> of 99.9% uptime. As far as possible, the cloud infrastructure used would be service provider agnostic to prevent vendor lock-in and to keep costs competitive. The architecture and services used would follow the “infrastructure-as-code” principle, which means that it can be programmatically deployed, and easily re-deployed, including—at least in principle—via a different vendor. <a href="https://developer.hashicorp.com/terraform">Terraform</a> and related technologies will be used for this.</p>
<p>The data architecture used would be be a ‘data lakehouse’. This combines the ability to work with unstructured data and structured data in databases with a data catalogue and other metadata, including version histories, all of which is useful and goes beyond a simple database. Some options are <a href="https://aws.amazon.com/sagemaker/lakehouse/">Amazon Sagemaker Lakehouse</a>, <a href="https://cloud.google.com/blog/products/data-analytics/extending-the-google-data-cloud-lakehouse-architecture">Google’s open lakehouse</a>, and <a href="https://www.databricks.com/product/data-lakehouse">Databricks Lakehouse</a>. For local development and prototyping, it’s possible to use <a href="https://ducklake.select/">DuckLake</a>.</p>
</section>
<section id="code" class="level3 page-columns page-full">
<h3 class="anchored" data-anchor-id="code">Code</h3>
<p>The computer code used would of course be under version control, and subject to ongoing quality checks, some automatic (eg unit and integration tests.) Tools like <a href="https://sqlfluff.com/">SQLFluff</a> and <a href="https://docs.astral.sh/ruff/">Ruff</a> can flag and even auto-correct some problems in database queries and code respectively, while other tools such as <a href="https://pre-commit.com/">pre-commit</a> and <a href="https://aeturrell.github.io/panoptipy/">panoptipy</a><sup>6</sup> provide ways to flag <em>potential</em> issues to both developers and managers. Of course, this code would be under version control. There’s even a maximalist approach where the code to produce statistics is open source too, so that problems can be found more quickly by more eyeballs.</p>
<div class="no-row-height column-margin column-container"><div id="fn6"><p><sup>6</sup>&nbsp;Full disclosure: I wrote this package! It’s still in beta, but you can get the gist already.</p></div></div><p>The code would mainly be Python and SQL, for many reasons.</p>
<p>Python’s dominance in data science reflects several compelling advantages that make it the natural choice for modern analytical work. The language benefits from <a href="https://www.tiobe.com/tiobe-index/">vast popularity</a> across <a href="https://pypl.github.io/PYPL.html">multiple measures</a>, creating substantial positive network externalities, eg rapid bug discovery, the ability of large language models to write it well, and the sheer volume of available free and open source packages that are on <a href="https://pypi.org/">PyPI</a>. Major conferences like NeurIPS and ICML predominantly feature Python-based research, as do <a href="https://mlcontests.com/state-of-machine-learning-competitions-2024">machine learning competitions</a>.</p>
<p>Python is widely taught and used. It’s <a href="https://www.bcs.org/policy-and-influence/education/bcs-landscape-review-computing-qualifications-in-the-uk/england-computer-science-gcse-as-and-a-levels/">taught in some UK and US high schools</a>, both state and <a href="https://www.isc.co.uk/media-enquiries/isc-blogs/python-becomes-a-modern-language-for-putney-high-school-s-next-generation-of-digital-nomads/">private</a>, meaning that the future pipeline of talent is assured. In the professional sphere, <a href="https://www.kdnuggets.com/2019/05/poll-top-data-science-machine-learning-platforms.html">data scientists in the private sector predominantly work in Python and SQL</a>, with this share growing over time. Beyond the academy, firms that sponsor and support Python include <a href="https://www.python.org/psf/sponsors/">Microsoft, Google, Meta, and Bloomberg</a>. Even <a href="https://www.infoworld.com/article/3668252/rstudio-changes-name-to-posit-expands-focus-to-include-python-and-vs-code.html">RStudio’s rebranding to ‘Posit’</a> reflects a strategic pivot toward Python support.</p>
<p>Cloud infrastructure considerations are increasingly crucial: services like <a href="https://cloud.google.com/run">Google Cloud Run</a> and <a href="https://learn.microsoft.com/en-GB/azure/azure-functions/supported-languages?tabs=isolated-process%2Cv4&amp;pivots=programming-language-csharp">Azure Functions</a> support Python natively.</p>
<p>Just as Python is dominant in the space of analytical code, SQL dominates in database languages. SQL complements Python perfectly, offering flexibility, simplicity compared to distributed databases, wide adoption, and impressive scalability (as demonstrated by Google BigQuery’s massive-scale SQL operations).</p>
<p>Using Python and SQL means being able to piggy-back on the wider innovations in these languages, and being able to attract staff for whom statistical production may not be their only ever job.</p>
</section>
<section id="sec-pipelines" class="level3 page-columns page-full">
<h3 class="anchored" data-anchor-id="sec-pipelines">Automated data pipelines based on code and focused on quality</h3>
<p>Data pipelines—from ingestion to prepared outputs—would be automated, with both automated and human checks throughout.</p>
<p>This would begin with data acquisition. Insofar as is possible, data could be programmatically read in from <a href="https://aeturrell.com/blog/posts/the-prize-with-apis/the-prize-with-apis.html">application programming interfaces</a> (APIs) rather than sourced manually. This may not always be possible depending on the data source. But if the data are coming from, say, a survey, the information can still be acquired initially through digital means and validated at the very first stage. Surveys that are panels could be pre-populated with known answers to try and limit the fall off in responses due to “survey fatique.”</p>
<div class="quarto-figure quarto-figure-center page-columns page-full">
<figure class="figure page-columns page-full">
<p><img src="https://www.aeturrell.com/blog/posts/future-of-statistics/survey_pipeline_dag.svg" class="img-fluid figure-img" style="width:60.0%"></p>
<figcaption class="margin-caption">A schematic of the directed acyclic graph that a data orchestration tool would run.</figcaption>
</figure>
</div>
<p>The progression of data through a pipeline would be run by the data orchestration tools that are fairly common in production grade data science applications. Data orchestration tools schedule, monitor, and catch errors or issues in the processing of data by computer code. Orchestration flows can be triggered on on a schedule or according to an event, for example if a new file lands in a folder. They ensure that data are transformed, cleaned, and delivered efficiently and accurately. Orchestration tools can handle large volumes of data, support various data formats and programming languages, and can make it easier for organisations to monitor their entire estate of processes. Some popular data orchestration tools include <a href="https://airflow.apache.org/">Apache Airflow</a> (first generation) and <a href="https://docs.dagster.io/">Dagster</a> and <a href="https://docs.prefect.io/v3/get-started/index">Prefect</a> (both second generation, with more advanced features). All of these are free and open source. For hosting of orchestration pipelines, <a href="https://cloud.google.com/composer/docs/composer-3/composer-overview">Google Cloud Composer</a> and <a href="https://aws.amazon.com/managed-workflows-for-apache-airflow/">Amazon Managed Workflows for Airflow</a> are both managed Airflow instances that can be configured using a dedicated API or using <a href="https://developer.hashicorp.com/terraform">Terraform</a>.</p>
<p>As data are passed around the workflows, they can be subject to checks that will raise a red flag should a problem be detected. “Type checking” would flag any issues with the format of the data. This can be achieved with data validation tools such as <a href="https://greatexpectations.io/">Great Expectations</a>, <a href="https://pandera.readthedocs.io/">Pandera</a>, and (for unstructured data) <a href="https://docs.pydantic.dev/">Pydantic</a>. Data validation and plausibility checks could look to see whether the data are consistent with historical and other contextual information too—this helps ensure that implausible values are caught early, including at the aggregate level.</p>
<p>Once data and pipelines are in the cloud and automated, it becomes possible to do data lineage tracking. This would allow one to understand how data change from landing, through processing and transformation, and to reports, applications, and statistics. Such lineage tracking can trace an error’s origin, highlight inconsistencies across processes, and show what the downstream consequences of an error might be at the start of a pipeline. Data lineage tracking also enables sensitivity analysis, so that producers can ask questions like, “What would the statistic look like if this or that data point were incorrect?” This gives a sense of whether the overall numbers are driven by one or two responses that, if incorrect, could make the final numbers misleading.</p>
<p>Any changes to critical production-of-stats code would be subject to review, and to tests that would show how the output statistic would change subject to the code change.</p>
<p>At the end of the pipeline, the single source of truth on the final output would be an API, and data available in the traditional way (ie where it could be downloaded manually from a webpage) would be built on top of that API—ensuring that these sources were consistent and that the statistics could be accessed programmatically by others. APIs are the bedrock of online services and are already provided by a wide range of public sector organisations such as <a href="https://api.tfl.gov.uk/">Transport for London</a> (TfL), the Federal Reserve Bank of St Louis’ statistical service <a href="https://fred.stlouisfed.org/">FRED</a>, the <a href="https://datahelpdesk.worldbank.org/knowledgebase/articles/889386-developer-information-overview">World Bank</a>, and the <a href="https://www.oecd.org/en/data/insights/data-explainers/2024/09/api.html">OECD</a>. They are powerful because they allow for integration of downstream services: analysts using the statistics could write their own automated pipelines that directly consume relevant data feeds and businesses could use the data in services they provide (much as CityMapper does with the TfL API.)</p>
<p>Internally, dashboards of KPIs of the quality of statistics and code, and the numbers of manual and programmatic downloads of statistics, and other relevant pipeline data would be produced and displayed as part of the process—more on that in the management section!</p>
</section>
</section>
<section id="labour" class="level2 page-columns page-full">
<h2 class="anchored" data-anchor-id="labour">Skills and labour</h2>
<p>Recognising how technology is <a href="https://www.ft.com/content/8e730692-fd9c-45b1-84dc-7ea16429c5c6">changing the demands on workforces</a>, and the high costs of co-ordination of lots of people, you’d probably want to go for a leaner, more highly skilled and better paid workforce than has hitherto been the case. Fewer, more expert staff can be more agile and make risk-reward trade-offs more succesfully.</p>
<p>Fundamentally, the production of statistics is a technocratic endeavour, and you would want a workforce that reflects this: you’d want them to be highly skilled in relevant technical subject areas; for example, data science, statistics, and economics. The level of knowledge that you would want to see staff demonstrate in these areas would be high—so high that, like those similarly technocratic institutions, central banks, you would want to have a dedicated career offering for people with PhDs in relevant topics and you would make space for them to do research on how to make the statistics better.</p>
<p>To ensure that the workforce did meet these criteria, and to reflect that the use of code, data, and AI would be foundational to the tasks, you might want to require technical staff (most staff) to pass a coding test in Python and SQL.</p>
<p>Inevitably, going for a highly skilled workforce would mean needing to offer high salaries too—though pay would not need to be as high as in the private sector because access to the best data in the country would be a strong draw for economists, data scientists, and statisticians alike, and public service is a great motivator for many. To ensure that the overall budget remained sensible, one would have to have a smaller workforce however.</p>
<p>Getting the right skills is the most important part of any endeavour, and where you site your statistical production should reflect this: it’s got to be where the people you want to work with are. Ideally, you would use a cost-benefit analysis of different locations and look at whether they could supply suitable labour at reasonable cost. A place with a thick labour market for highly educated graduates and post-graduates, but not too high a cost of living, is most likely to deliver this.</p>
<p>Once highly skilled labour is acquired, it would need to be cultivated and retained. Ongoing investment in human capital is all the more necessary in a world where AI means work is changing rapidly. Beyond that, there would need to be a compelling career offering for those who can bring value not just via managerial and organisational skills, but also via technical knowledge, ability to innovate, and professional leadership. Therefore, it would be essential to have technical career paths on offer. Technical career paths are common in frontier technology firms that require somewhat similar skills: Google has a complete parallel technical career ladder for roles up to and including Senior Vice President; Amazon, Meta, AirBnB, Uber, Apple, and Microsoft have similar pathways. This approach is not unprecedented in the public sector.</p>
<p>One advantage of having research skills would be to invigorate innovation. One (extremely imperfect) metric of innovation in statistics is how many research papers are published by national statistical organisations. As you can see in Figure&nbsp;1, world-leading institutes like Statistics Netherlands and Statistics Canada fare well on this measure.</p>
<div id="fig-research" class="quarto-float quarto-figure quarto-figure-center anchored page-columns page-full">
<figure class="quarto-float quarto-float-fig figure page-columns page-full">
<div aria-describedby="fig-research-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<img src="https://www.aeturrell.com/blog/posts/future-of-statistics/mean_papers_per_year.png" class="img-fluid figure-img">
</div>
<figcaption class="quarto-float-caption-margin quarto-float-caption quarto-float-fig margin-caption" id="fig-research-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
Figure&nbsp;1: A crude measure of innovation for selected institutions. INSEE is the French statistical agency. The Federal Reserve Bank of St Louis publishes a wide range of statistics through its <a href="https://fred.stlouisfed.org/">Federal Reserve Economic Statistics website</a>. Data from SCOPUS.
</figcaption>
</figure>
</div>
<p>It is not just highly skilled technical workers that will be required: management and leadership must also be highly skilled in their crafts. It almost goes without saying that poor leadership and bad management practices can undermine any endeavour, no matter how good the rest of the people working on it may be. Therefore, as well as seeking highly skilled technical staff, it is important that senior leadership have vision, the pragmatism to deliver, and the ability to implement the best practices in management. <a href="https://www.productivity.ac.uk/news/do-managers-matter/">Firms with better management practices have higher labour productivity</a>; why should we expect the production of statistics to be any different? And the benefits are significant: moving from the median to the 75th percentile of the distribution of management practices scores increases productivity by 11%. On the topic of management practices…</p>
</section>
<section id="management-practices-culture-and-organisation" class="level2 page-columns page-full">
<h2 class="anchored" data-anchor-id="management-practices-culture-and-organisation">Management practices, culture, and organisation</h2>
<p>The best practices for culture, organisation, and management would need to be used to deliver the best possible statistics. Drawing from sources like <a href="https://www.productivity.ac.uk/news/do-managers-matter/">Do managers matter?</a> and the techniques implemented at Amazon and others, the following innovations would be used:</p>
<ul>
<li><p>A culture of continuous improvement <em>within</em> existing teams. No hiving off improvement work to be done in a different department or team—innovation is everyone’s job! Those improvements can be encouraged, monitored, and celebrated through KPIs…</p></li>
<li><p>The use of Key Performance Indicators (KPIs) for tracking progress and for making key management decisions (eg around prioritisation.) These should be available through the whole organisation, and will help hold people to account for the statistics they are producing.</p></li>
<li><p>Small, highly autonomous teams<sup>7</sup> responsible for elements within the vertically integrated product-structure. Small teams operating with high autonomy within guardrails can help tackle co-ordination costs, remove the need for wide consensus (which leads to inaction), and prevents the diffusion of accountability. It is also much more empowering, which is strongly linked to job satisfaction.</p></li>
<li><p>The use of targets that are stretching, tracked, and reviewed.</p></li>
<li><p>Strict, rapid, and effective performance management.</p></li>
<li><p>An organisational structure that reflects products and outputs rather than functional areas. This means that an entire production pipeline, from survey to final published statistics, will be owned by one area that can be held accountable for it. This is a contrast to having surveys in one department, technology in another, HR another, and statistics processing another. Functional models struggle to ensure aligned incentives. For example, if you have someone responsible for IT they are typically incentivised to minimise how much is spent on technology and to de-risk that technology as much as possible. However, another person who is responsible for processing statistics has quite different incentives: they want analysis to be done quickly (so prefer more expensive IT) and they are willing to take on some risk to get the job done. It can be hard for an organisation to solve this trade-off effectively, especially if these two people have reporting lines that only meet in the head of the organisation, who will have many other concerns on their plate. Instead, a product-oriented structure sees much of technology, HR, etc., embedded right next to the coal face.</p></li>
<li><p>Similarly, “single-threaded” leadership, which means a single person is ultimately responsible for getting a product out. This leader is incentivised to make the optimal trade-off on risk and spend versus delivery. There is clear accountability in this model. Single-threaded means that leaders have end-to-end accountability for outcomes, decision bottlenecks are removed, and everyone under that leader is aligned on the goal.</p></li>
<li><p>Kanban boards for management of work tasks, wherein everyone in the team knows what is happening, and a large number of routine updates are no longer necessary</p></li>
<li><p>Asynchronous working patterns, such as status updates being automatic rather than through meetings (as above.) Synchronous meetings would be reserved for bilaterals and low-latency collaboration on complex issues. Documentation would be used obsessively for code, processes, decisions, and actions (this also lowers initial onboarding costs.) In particular, the documentation and code that processes data in a part of the pipeline will sit together and automated checks will ensure that they do not diverge (a common practice in high end software development.)</p></li>
<li><p>Proposals to be discussed by committees in prose, not PowerPoint, in the form of notes with a page limit. Meetings would start with “silent time” so that everyone gets on the same page (literally) and can have a constructive discussion.</p></li>
<li><p>Decisions counter-signed by all, and the reason for the decision documented.</p></li>
<li><p><a href="https://www.aboutamazon.com/news/workplace/an-insider-look-at-amazons-culture-and-processes">“working backwards”</a> for new products</p></li>
<li><p><a href="https://en.wikipedia.org/wiki/Disagree_and_commit">“disagree and commit”</a>, to avoid consensus bias</p></li>
<li><p>a single source of truth for all information</p></li>
</ul>
<div class="no-row-height column-margin column-container"><div id="fn7"><p><sup>7</sup>&nbsp;Usually defined as those that can be fed with two pizzas.</p></div></div></section>
<section id="conclusion" class="level2">
<h2 class="anchored" data-anchor-id="conclusion">Conclusion</h2>
<p>Our current statistical infrastructure is creaking dangerously. Could this crisis present an opportunity? We now have the tools—AI, cloud computing, modern data science, and battle-tested management practices from the world’s most successful tech companies—to build anew.</p>
<p>Starting from scratch would mean <strong>putting technology first</strong>: automated pipelines running on secure cloud infrastructure, with AI catching errors that humans might miss, and APIs making statistics as accessible as checking the weather on your phone. It would mean <strong>talent over scale</strong>: a smaller, highly skilled workforce of data scientists, statisticians, and economists who can programme this technology to produce the best stats possible. And it would mean <strong>innovation in management</strong>: product-focused teams with single-threaded leadership, clear KPIs, and the management practices that have made Amazon successful but applied to the considerably more important task of measuring the economy and more.</p>
<p>Here’s what should keep us all awake at night: we could be making billion-pound decisions based on statistics produced with clipboard-and-Excel methods while the tools to do this properly are sitting right there, proven and ready to deploy. The question isn’t whether we <em>can</em> rebuild our statistical infrastructure for the modern age—the question is whether we will—and whether it can happen before the next crisis in the nation’s numbers hits the headlines.</p>
<p>The blueprint exists. The technology works. The management practices are proven. And we <em>need</em> statistics fit for the choices that will shape Britain’s future: because a decision based on the wrong data is like a house built on sand.</p>



</section>


<div id="quarto-appendix" class="default"><section class="quarto-appendix-contents" id="quarto-bibliography"><h2 class="anchored quarto-appendix-heading">References</h2><div id="refs" class="references csl-bib-body hanging-indent" data-entry-spacing="0">
<div id="ref-allas_uks_2025" class="csl-entry">
Allas, Tera, and Dimitri Zenghelis. 2025. <span>“The <span>UK</span>’s Capital Gap: A Short-Fall in the Trillions of Pounds That Will Take Decades to Bridge.”</span> <em>The Productivity Institute</em>, no. 55 (May).
</div>
<div id="ref-brynjolfsson_strength_2011" class="csl-entry">
Brynjolfsson, Erik, Lorin M. Hitt, and Heekyung Hellen Kim. 2011. <span>“Strength in <span>Numbers</span>: <span>How Does Data-Driven Decisionmaking Affect Firm Performance</span>?”</span> SSRN. Rochester, NY. <a href="https://doi.org/10.2139/ssrn.1819486">https://doi.org/10.2139/ssrn.1819486</a>.
</div>
<div id="ref-erman_use_2022" class="csl-entry">
Erman, Sevgui, Eric Rancourt, Yanick Beaucage, and Andre Loranger. 2022. <span>“The <span>Use</span> of <span>Data Science</span> in a <span>National Statistical Office</span>.”</span> <em>Harvard Data Science Review</em> 4 (4). <a href="https://doi.org/10.1162/99608f92.13e1d60e">https://doi.org/10.1162/99608f92.13e1d60e</a>.
</div>
<div id="ref-geiecke2024conversations" class="csl-entry">
Geiecke, Friedrich, and Xavier Jaravel. 2024. <span>“Conversations at Scale: Robust Ai-Led Interviews with a Simple Open-Source Platform.”</span> <em>Available at SSRN 4974382</em>.
</div>
<div id="ref-kalamara2022making" class="csl-entry">
Kalamara, Eleni, Arthur Turrell, Chris Redl, George Kapetanios, and Sujit Kapadia. 2022. <span>“Making Text Count: Economic Forecasting Using Newspaper Text.”</span> <em>Journal of Applied Econometrics</em> 37 (5): 896–919.
</div>
<div id="ref-muller_effect_2018" class="csl-entry">
<span class="nocase">Müller, Oliver, Maria Fay, and Jan vom Brocke</span>. 2018. <span>“The <span>Effect</span> of <span>Big Data</span> and <span>Analytics</span> on <span>Firm Performance</span>: <span>An Econometric Analysis Considering Industry Characteristics</span>.”</span> <em>Journal of Management Information Systems</em> 35 (2): 488–509. <a href="https://doi.org/10.1080/07421222.2018.1451955">https://doi.org/10.1080/07421222.2018.1451955</a>.
</div>
<div id="ref-soria2025empirical" class="csl-entry">
Soria, Chris. 2025. <span>“An Empirical Investigation into the Utility of Large Language Models in Open-Ended Survey Data Categorization.”</span> <a href="https://doi.org/10.31235/osf.io/wv6tk_v2">https://doi.org/10.31235/osf.io/wv6tk_v2</a>.
</div>
<div id="ref-stedman_end_2019" class="csl-entry">
Stedman, Richard C., Nancy A. Connelly, Thomas A. Heberlein, Daniel J. Decker, and Shorna B. Allred. 2019. <span>“The <span>End</span> of the (<span>Research</span>) <span>World As We Know It</span>? <span>Understanding</span> and <span>Coping With Declining Response Rates</span> to <span>Mail Surveys</span>.”</span> <em>Society &amp; Natural Resources</em> 32 (10): 1139–54. <a href="https://doi.org/10.1080/08941920.2019.1587127">https://doi.org/10.1080/08941920.2019.1587127</a>.
</div>
<div id="ref-turrell2025cuttingcomplexitydatascience" class="csl-entry">
Turrell, Arthur. 2025. <span>“Cutting Through Complexity: How Data Science Can Help Policymakers Understand the World.”</span> <a href="https://arxiv.org/abs/2502.03010">https://arxiv.org/abs/2502.03010</a>.
</div>
<div id="ref-turrell20226" class="csl-entry">
Turrell, Arthur, Bradley Speigner, Jyldyz Djumalieva, David Copple, and James Thurgood. 2022. <span>“6. Transforming Naturally Occurring Text Data into Economic Statistics.”</span> In <em>Big Data for Twenty-First-Century Economic Statistics</em>, 173–208. University of Chicago Press.
</div>
</div></section></div> ]]></description>
  <category>statistics</category>
  <category>productivity</category>
  <category>public sector</category>
  <guid>https://www.aeturrell.com/blog/posts/future-of-statistics/</guid>
  <pubDate>Sat, 05 Jul 2025 23:00:00 GMT</pubDate>
  <media:content url="https://www.aeturrell.com/blog/posts/future-of-statistics/survey_pipeline_dag.svg" medium="image" type="image/svg+xml"/>
</item>
<item>
  <title>Untangling spaghetti code with smartrappy</title>
  <dc:creator>Arthur Turrell</dc:creator>
  <link>https://www.aeturrell.com/blog/posts/introducing-smartrappy/</link>
  <description><![CDATA[ 





<div class="quarto-figure quarto-figure-center page-columns page-full">
<figure class="figure page-columns page-full">
<p><img src="https://www.aeturrell.com/code/logos/smartrappy.svg" class="img-fluid figure-img" style="width:50.0%"></p>
<figcaption class="margin-caption"><strong>smartrappy</strong> logo</figcaption>
</figure>
</div>
<p>If you’ve ever inherited responsibility for a messy Python codebase, or returned to your own code after a few months, and wondered “what on earth is going on here?”, you’re not alone. Understanding complex code dependencies, data flows, and how different parts of a project interact is a common challenge, especially with analytical pipelines that have gathered a bit of dust or changed substantially since you last used them.</p>
<p>Of course, ideally, it shouldn’t get to this stage: there <em>are</em> ways to auto-document <em>precisely</em> what’s happening in your code, eg via <a href="https://aeturrell.github.io/coding-for-economists/wrkflow-rap.html#make-for-reproducible-analytical-pipelines">a Makefile and Makefile2Dag</a>—see <a href="https://github.com/aeturrell/example-reproducible-research">a fully worked example here</a>—or using a <a href="https://aeturrell.github.io/coding-for-economists/data-advanced.html#data-orchestration">data orchestrator</a>. For everything else, there’s <a href="https://aeturrell.github.io/smartrappy/"><strong>smartrappy</strong></a>.</p>
<p><strong>smartrappy</strong><sup>1</sup> is designed to help you understand and visualise the dependencies in analytical Python projects that haven’t hit the heady heights of having auto-generated their own directed cyclic graph. Let’s be honest, that’s most projects. It’s important to say that <strong>smartrappy</strong> is just sniffing out relationships between code and data and other things in a project, and it does a fairly good job of that, but it’s not perfect and it certainly won’t find everything. For a lot of analytical Python code, though, it should give you a big boost in understanding what’s going on.</p>
<div class="no-row-height column-margin column-container"><div id="fn1"><p><sup>1</sup>&nbsp;smart + RAP (Reproducible Analytical Pipeline) + py. Get it? Get it?? Hilarious.</p></div></div><section id="what-does-smartrappy-do" class="level2">
<h2 class="anchored" data-anchor-id="what-does-smartrappy-do">What does smartrappy do?</h2>
<p><strong>smartrappy</strong> analyses a Python project and infers the directed acyclic graph (DAG) of code and data dependencies. It traces through your codebase to find (as best as it can):</p>
<ul>
<li>Which files are being read from or written to, including data files, databases, configuration files, and quarto files</li>
<li>When data files were last updated, or whether they exist on disk at all</li>
<li>What Python modules are being imported, including ones internal and external (ie packages) to the project</li>
<li>How these all connect together</li>
</ul>
<p>It then visualises these relationships, and quite beautifully I think: let’s take a look.</p>
<p>Here’s the default output, which appears in the console, from running on a test repo I put together. If you haven’t come across <code>uvx</code> it’s an <a href="https://docs.astral.sh/uv/">Astral uv</a> command that does a one-off install just to use a tool (ie without installing that tool in a permanent Python environment). I’m imagining that <strong>smartrappy</strong> gets used in this way.</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb1" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb1-1"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">uvx</span> smartrappy tests/test_set_two</span></code></pre></div></div>
<pre class="text"><code>File Operations, Database Operations, and Import Analysis
================================================================================

File: data/input.csv
Operation: READ
Referenced in:
  - tests/test_set_two/data_processing.py

File: data/processed.csv
Operation: READ/WRITE
Referenced in:
  - tests/test_set_two/data_processing.py
  - tests/test_set_two/visualisation.py

File: equation.tex
Operation: READ/WRITE
Referenced in:
  - tests/test_set_two/model_solver.py
  - tests/test_set_two/slides.qmd

File: output.png
Operation: READ/WRITE
Referenced in:
  - tests/test_set_two/slides.qmd
  - tests/test_set_two/visualisation.py

💽 Database Operations:

Database: mydatabase
Type: mssql
Operation: READ/WRITE
Referenced in:
  - tests/test_set_two/data_processing.py

Module Imports:

Script: data_processing.py
  - import numpy [external]
  - import pandas [external]
  - import pyodbc [external]

Script: model_solver.py
  - from pathlib → pathlib:Path [external]

Script: visualisation.py
  - import pyplot [external]
  - import pandas [external]
  - from data_processing → data_processing:process_data [internal]

Terminal Visualisation
📦 Project Dependencies
┗━━ 📰 slides.qmd
    ┣━━ 📄 equation.tex
    ┃   ┗━━ 📜 model_solver.py
    ┃       ┗━━ 📦 pathlib:Path
    ┗━━ 📄 output.png
        ┗━━ 📜 visualisation.py
            ┣━━ 📄 data/processed.csv
            ┃   ┗━━ 📜 data_processing.py
            ┃       ┣━━ 📄 data/input.csv
            ┃       ┣━━ 💽 mydatabase
            ┃       ┃   ┣━━ Type: mssql
            ┃       ┃   ┗━━ ↻ data_processing.py (circular)
            ┃       ┣━━ 📦 pandas
            ┃       ┣━━ 📦 numpy
            ┃       ┗━━ 📦 pyodbc
            ┣━━ 📦 pandas
            ┣━━ 📦 pyplot
            ┗━━ 🔧 data_processing:process_data</code></pre>
<p>This gives you the flow of data and dependencies across the project. But it’s not so easy to save this console output and it’s still not the clearest ever. We really want, either, something that we can take away and analyse or a bona fide visualisation.</p>
<p>The <code>--format json</code> option gives you the best of both. If you give no output path, then you get json in the console, like this:</p>
<pre class="text"><code>JSON Representation
================================================================================
{
  "nodes": [
    {
      "id": "quarto_document_1193520",
      "name": "slides.qmd",
      "type": "quarto_document",
      "metadata": {}
    },
    {
      "id": "data_file_15482117",
      "name": "output.png",
      "type": "data_file",
      "metadata": {
        "exists": false
      }
    },
    ....</code></pre>
<p>Or you can pipe JSON, or the other output formats, to a folder:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb4" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb4-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Generate a JSON for onward analysis</span></span>
<span id="cb4-2"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">uvx</span> smartrappy /path/to/project <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--format</span> json <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--output</span> path/to/output/folder</span>
<span id="cb4-3"></span>
<span id="cb4-4"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Generate a PDF visualisation using Graphviz—requires the Graphviz application to be installed</span></span>
<span id="cb4-5"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">uvx</span> smartrappy /path/to/project <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--format</span> graphviz <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--output</span> path/to/output/folder</span>
<span id="cb4-6"></span>
<span id="cb4-7"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Generate a Mermaid diagram for embedding in Markdown</span></span>
<span id="cb4-8"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">uvx</span> smartrappy /path/to/project <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--format</span> mermaid <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--output</span> path/to/output/folder</span>
<span id="cb4-9"></span>
<span id="cb4-10"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Generate all formats at once</span></span>
<span id="cb4-11"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">uvx</span> smartrappy /path/to/project <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--all-formats</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--output</span> path/to/output/folder</span></code></pre></div></div>
<p>You can also use it programmatically in your Python code:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb5" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb5-1"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> smartrappy <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> analyse_project</span>
<span id="cb5-2"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> smartrappy.reporters <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> ConsoleReporter, GraphvizReporter</span>
<span id="cb5-3"></span>
<span id="cb5-4"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Analyse the project</span></span>
<span id="cb5-5">model <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> analyse_project(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"/path/to/your/project"</span>)</span>
<span id="cb5-6"></span>
<span id="cb5-7"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Generate console report</span></span>
<span id="cb5-8">console_reporter <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> ConsoleReporter()</span>
<span id="cb5-9">console_reporter.generate_report(model)</span>
<span id="cb5-10"></span>
<span id="cb5-11"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Generate graphviz visualization</span></span>
<span id="cb5-12">graphviz_reporter <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> GraphvizReporter()</span>
<span id="cb5-13">graphviz_reporter.generate_report(model, output_path<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"path/to/your/outputs"</span>)</span></code></pre></div></div>
<p>Although, as noted, there are better and more precise ways to auto-document your code’s DAG, this means you can build an auto-documentation of your DAG to your repo!</p>
</section>
<section id="visualisation" class="level2 page-columns page-full">
<h2 class="anchored" data-anchor-id="visualisation">Visualisation</h2>
<p>The two ‘for keeps’ visualisation formats are mermaid and graphviz. Here’s the latter, converted from a PDF to an SVG image for this post, and generated with</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb6" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb6-1"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">uvx</span> smartrappy path/to/project <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--format</span> graphviz <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--internal</span></span></code></pre></div></div>
<div class="quarto-figure quarto-figure-center page-columns page-full">
<figure class="figure page-columns page-full">
<p><img src="https://www.aeturrell.com/blog/posts/introducing-smartrappy/smartrappy_viz.svg" class="img-fluid figure-img"></p>
<figcaption class="margin-caption">Graphviz and <strong>smartrappy</strong> produced visualisation of a research project</figcaption>
</figure>
</div>
<p>The <code>--internal</code> flag just restricts the visualisation to internal-only code imports, so ignores, eg, <code>import pandas as pd</code>. The repo it was pointed at is a research project where the slides and paper are generated from Quarto .qmd files but the <code>paper.qmd</code> file has yet to be modified much from its default settings. That’s why it depends on a file that doesn’t actually exist; it’s just got a default import in there.<sup>2</sup></p>
<div class="no-row-height column-margin column-container"><div id="fn2"><p><sup>2</sup>&nbsp;As an aside, the research repo was originally generated from <a href="https://github.com/aeturrell/cookiecutter-research-project">this cookiecutter</a>, which you can read more about <a href="../../../blog/posts/ultra-modern-python-cookiecutters/index.html">here</a>. In fact, <strong>smartrappy</strong> was generated from a cookiecutter too—<a href="https://github.com/aeturrell/cookiecutter-python-package">this one</a>, which you can read about in <a href="../../../blog/posts/ultra-modern-python-cookiecutters/index.html">the same post</a>.</p></div></div><p>This might sound like a pretty elaborate solution for something that’s maybe only useful for extremely complex projects, but the reality is that analytical codebases that are messy and not carefully joined up are <em>much</em> more common than ones that are.</p>
</section>
<section id="why-smartrappy-is-useful" class="level2">
<h2 class="anchored" data-anchor-id="why-smartrappy-is-useful">Why smartrappy is useful</h2>
<p>Unlike most production software, analytical code often has a different structure and purpose. Rather than building a long-lived application, we’re often answering specific questions or generating regular reports. Data flows from files to processing scripts to outputs, and understanding this flow is critical.</p>
<p>The problem is that these analytical pipelines can grow organically over time:</p>
<ul>
<li>A new data source gets added</li>
<li>Someone introduces a new intermediate processing step</li>
<li>A report format changes, requiring new outputs</li>
<li>The project outlives the original authors, and new team members have to make sense of it all</li>
<li>The IT of the institution doesn’t support advanced orchestration, or Make, or other automation tools, so manual steps can creep in</li>
<li>Diverse teams (that have lots of benefits in other ways) don’t always have the right skills to build or maintain fully automated DAGs</li>
</ul>
<p>And, before long, what started as a simple pipeline resembles a plate of tangled spaghetti.</p>
<p>By mapping out the dependencies automatically in a “best endeavours” way, <strong>smartrappy</strong> helps to <em>understand legacy code more quickly</em> and <em>verify that a reproducible analytical pipeline (RAP) is connected as expected</em>.</p>
</section>
<section id="more-on-how-it-works" class="level2">
<h2 class="anchored" data-anchor-id="more-on-how-it-works">More on how it works</h2>
<p><strong>smartrappy</strong> basically searches for common patterns found in data analysis projects. The fact that Python has a fairly powerful built-in library to parse Python code (<a href="https://docs.python.org/3/library/ast.html">ast</a> for abstract syntax trees), plus some text analysis, makes this package possible. <strong>smartrappy</strong> can detect:</p>
<ul>
<li>Basic file operations with Python’s built-in <code>open()</code> function</li>
<li>Pandas operations like <code>pd.read_csv()</code>, <code>df.to_csv()</code>, etc.</li>
<li>Matplotlib figure saving with <code>plt.savefig()</code></li>
<li>Database operations with SQLAlchemy, psycopg2, sqlite3, etc.</li>
<li>Latex tables and equations</li>
<li>Module imports, distinguishing between internal and external dependencies</li>
<li>Quarto documents (<code>.qmd</code> files), including with embedded Python code chunks, includes, and markdown images</li>
</ul>
<p>It’s not perfect—it won’t catch every possible way you might read or write data—but it covers the most common patterns you’ll find in analytical code.</p>
</section>
<section id="limitations" class="level2">
<h2 class="anchored" data-anchor-id="limitations">Limitations</h2>
<p>While smartrappy is quite useful, it’s important to understand its limitations:</p>
<ol type="1">
<li>It uses static code analysis, so it won’t catch some dynamic patterns like using <code>**kwargs</code> to pass filenames.</li>
<li>It might miss custom file access methods or unusual patterns.</li>
<li>There are some naive features, for example: all modules containing database access get flagged as having circular imports.</li>
<li>It will never be as accurate in exposing the DAG as making the code and the DAG <em>the same thing</em></li>
</ol>


</section>


 ]]></description>
  <category>python</category>
  <category>data science</category>
  <category>reproducibility</category>
  <category>code</category>
  <category>data</category>
  <guid>https://www.aeturrell.com/blog/posts/introducing-smartrappy/</guid>
  <pubDate>Mon, 21 Apr 2025 23:00:00 GMT</pubDate>
  <media:content url="https://www.aeturrell.com/code/logos/smartrappy.svg" medium="image" type="image/svg+xml"/>
</item>
<item>
  <title>Efficiency in the public sector: analysis and operations</title>
  <dc:creator>Arthur Turrell</dc:creator>
  <link>https://www.aeturrell.com/blog/posts/public-sector-analytical-efficiency/</link>
  <description><![CDATA[ 





<p>I’ve been thinking a lot about efficiency in the public sector recently. This post looks at ideas for increasing the efficiency of analysis and operations through automation, good coding practices, artificial intelligence, and, well, (meta-?) analysis. This is the second post in this series; see the <a href="../../../blog/posts/public-sector-communication-efficiency/index.html">previous post</a> for ideas on efficiency related to communication and co-ordination.</p>
<section id="what-is-meant-by-analysis-and-operations" class="level2 page-columns page-full">
<h2 class="anchored" data-anchor-id="what-is-meant-by-analysis-and-operations">What is meant by analysis and operations?</h2>
<p><strong>What does it mean to make analysis more efficient? It is to produce analysis faster, cheaper, better, or in higher volume.</strong> Automation is key to doing this. And I’m thinking here of both ad hoc analysis and production analysis (eg regularly published statistics). Before we hit the ideas, it might help to give examples of these two types of analysis.</p>
<p>A good example of ad hoc analysis would be if, say, a global event meant that a lot more analysis was suddenly required on trade and trade links, typically in the form of answers to one-off questions. With such ad hoc analysis, you don’t know the <em>specific</em> question in advance—but you might know that, from time-to-time, questions about trade links will arise. So to improve the productivity of analytical operations in this case, you would want analysts or data scientists or economists, whoever it is, to have longitudinal data on bilateral trade flows to hand, with their permissions to access it cleared ahead of time, the data cleaned, the data stored in a sensible format, and the data accessible to suitable analytical tools. Essentially, it’s about putting analysts in the best position to answer the questions in a timely way as they arise, maximising agility and minimising repeated effort for when the next question lands.</p>
<p>An example of regular analysis would be a pipeline that ends in a publication and which involves data ingestion, data checking, processing, and publication of data alongside a report that has a fairly consistent structure. Perhaps this process involves a lot of manual steps, like handovers, emails, someone staring at a spreadsheet, and someone having to serve requests for custom cuts of the data. Improving productivity might look like automating the end-to-end pipeline, cutting out the manual steps, and, finally, serving the data up via an API so that people can take whatever cuts they want. It’s again minimising effort.</p>
<div class="quarto-figure quarto-figure-center page-columns page-full">
<figure class="figure page-columns page-full">
<p><img src="https://www.aeturrell.com/blog/posts/public-sector-analytical-efficiency/automated_stats.webp" class="img-fluid figure-img"></p>
<figcaption class="margin-caption">Automation can take many forms</figcaption>
</figure>
</div>
<p>Turning to operations now; in every public sector institution, there are thousands of actions that are repeated every day by most staff. Something as basic as turning on a laptop or checking a calendar. <strong>If some of the tasks that happen repeatedly can be sped up even a little, then there’s a large saving to be made</strong>. This does require <em>measuring</em> how fast those tasks are today and, to get an idea of what the return on investment (ROI) might be, it also requires one to have a counter-factual in mind of how fast they <em>could</em> be with that investment.</p>
<p>With that background out of the way, let’s move on to some ideas!</p>
</section>
<section id="operations" class="level2 page-columns page-full">
<h2 class="anchored" data-anchor-id="operations">Operations</h2>
<section id="more-productive-capital" class="level3 page-columns page-full">
<h3 class="anchored" data-anchor-id="more-productive-capital">More productive capital</h3>
<p>I touched on computer-task time savings in the context of better hardware in <a href="../../../blog/posts/bad-it/index.html">this post on the false economy of bad IT</a>. TL;DR it <em>always</em> makes sense to buy staff earning over the median UK salary a high-spec laptop if they lose more than 9 minutes a day due to a low- to mid-range laptop being slow or glitchy. As an aside, the UK has relatively low capital stock, as seen in Figure&nbsp;1 (chart by <a href="https://www.mckinsey.com/our-people/tera-allas">Tera Allas</a>).</p>
<div id="fig-capital" class="quarto-float quarto-figure quarto-figure-center anchored page-columns page-full">
<figure class="quarto-float quarto-float-fig figure page-columns page-full">
<div aria-describedby="fig-capital-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<img src="https://www.aeturrell.com/blog/posts/data-science-with-impact/capital_stock_uk.jpeg" class="img-fluid figure-img" style="width:100.0%">
</div>
<figcaption class="quarto-float-caption-margin quarto-float-caption quarto-float-fig margin-caption" id="fig-capital-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
Figure&nbsp;1: Capital is important for productivity, who would have guessed?
</figcaption>
</figure>
</div>
<p>There’s a whole class of repeated tasks that people do on computers that are slower than they could be because of the computer’s hardware or configuration. How can the productivity (loss) be definitively tested? One option is a log of the times when someone’s workflow was interrupted because, say, the computer was turning on, or froze, or was opening an application. Another would be to measure the time it takes to do similar tasks in a perfect reference situation, eg the same task on a high-spec laptop, and use the time that took as the counterfactual. Similarly, one could measure productivity<sup>1</sup> with one software package relative to another to more accurately assess which is better (a lot of software will be chosen for reasons that are orthogonal to productivity, for reasons I sketch out in <a href="../../../blog/posts/public-sector-communication-efficiency/index.html#have-someone-own-the-trade-off">this other post</a>.)</p>
<div class="no-row-height column-margin column-container"><div id="fn1"><p><sup>1</sup>&nbsp;I very purposefully say ‘productivity’ and not just ‘time’ here because the most expensive input to the production function is usually peoples’ time, not the time it takes software to run. Which is why data science only reaches for C++ and Rust, which are slower to write code in but run faster, when it’s absolutely necessary.</p></div></div><p>Now you might say, “ah, but you haven’t done a fair test—that low- to mid-range institutional laptop has lots of important extra security on it that the high-end reference laptop doesn’t.” You’re right, ideally one would do a like for like comparison where you would only change one feature at a time. Pragmatically, though, I would see it as a feature, not a bug—use it as a way to see the overall trade-off that’s being made between security + cost vs productivity.</p>
<p>Regardless of how you do it, once bottlenecks in productive workflows have been identified and assessed, investment resources can focus on improving them in the order of biggest benefit.</p>
<p>A partial alternative to going high-spec is to allow Bring Your Own Device (BYOD). This allows the institution’s data and software to be accessed via the personal devices of staff, which may be faster and may be more suited to staff (by selection). There are security issues around this, and it may not be a equitable solution—more junior staff may not be able to afford the same tech as those at the top of the organisation. Still, if it doesn’t make sense to stump up for a high-spec laptop for everyone this can be a great second option. One particularly secure combination that will be relevant to many public sector institutions is Apple hardware + iOS as an operating system + Microsoft<sup>2</sup> software, which allows people to use corporate account Outlook, Teams, Sharepoint, etc, etc, on iPhone and iPad (including with a keyboard, as long as it’s an Apple one). My iPad has a processor that’s 40% faster than my work laptop’s, so BYOD is a game-changer. Of course, you can’t do everything from an iPad<sup>3</sup>—but you can certainly do a lot.</p>
<div class="no-row-height column-margin column-container"><div id="fn2"><p><sup>2</sup>&nbsp;Throughout, I’ll be referring to whether something or another plays well with Microsoft software. This is not because I think the public sector <em>should</em> be using Microsoft for everything, far from it, but—as I like to say—‘nobody ever got fired for buying Microsoft’, and the reality is that it’s what most public sector institutions use. Key exceptions I’m aware of are the <a href="https://interoperable-europe.ec.europa.eu/collection/open-source-observatory-osor/document/munichs-long-history-open-source-public-administration">Munich government</a>, the <a href="https://en.wikipedia.org/wiki/GendBuntu">French Gendarmarie</a>, who both use custom versions of Linux as part of their own efficiency drives, high security areas of many governments, some national laboratories, and of course computationally intensive systems.</p></div><div id="fn3"><p><sup>3</sup>&nbsp;For all practical purposes, you can’t code directly on an iPad in the way you would a laptop or desktop. Yet.</p></div></div></section>
<section id="free-and-open-source-software" class="level3 page-columns page-full">
<h3 class="anchored" data-anchor-id="free-and-open-source-software">Free and open source software</h3>
<p>Free and open source software (FOSS) definitely isn’t always more efficient, but my rule of thumb is <strong>if there is a widely-used and close free and open source substitute, it will be far more efficient to use that rather than proprietary software</strong>.</p>
<p>Let’s look at a really clear example: Matlab vs Python. The latter is free and open source, does <em>almost</em> everything the former does, does vastly more other things to boot, and can be used in a wider range of places. Now there are some tasks for which Python is <em>not</em> a close substitute, and they really do need Matlab. But for most people, leveraging the massive Python ecosystem is more efficient.</p>
<p>There are many examples where paying for proprietary software makes sense because there is no close FOSS substitute. I am a fan of Obsidian, the markdown-based note-taking app. One of the reasons I think it’s great is that you never lose control of your data or the format it’s in: all of your notes are just markdown files and the Obsidian app just gives a fancy way to edit and view that content. The Obsidian app happens to be free but it’s not open source. I would similarly say there’s no FOSS equivalent of the incredible integration of hardware and software that Apple has achieved with the Macbook. But, equally, there are plenty of areas where direct FOSS substitutes do exist.</p>
<p>In my view, the high efficiency of widely-used FOSS comes through a bunch of channels:</p>
<ul>
<li>The ‘F’ stands for free! Productivity is output per unit of input, so free is pretty good.<sup>4</sup></li>
<li>Widely-used open source software is less likely to have critical security flaws.<sup>5</sup></li>
<li>Again because it’s open source, you actually know what it’s doing!</li>
<li>Very much related to the above, and somewhat counter-intuitively, it’s more private. You can see for yourself if there’s a backdoor. A proprietary software vendor knows, at the very least, that you bought the software: a maintainer of a FOSS package may not even know you exist.</li>
<li>By dint of being free and widely used, there are many ideas for how to improve it, and guides on how to use it.</li>
<li>You don’t get locked in to a proprietary software provider who can subsequently ratchet up the price or drop support for a critical feature.</li>
<li>Missing a feature you’d like for your org in a specific library? If it’s reasonable and within-scope, you can probably get it added by the maintainers. Or your org can simply add it for themselves. <a href="https://www.bloomberg.com/company/values/tech-at-bloomberg/python/">Bloomberg</a> is a good example of a company who has gone heavily down this route.</li>
<li>There’s also a diversity, equality, and inclusion benefit. Lots of students, especially those from poorer backgrounds, can’t afford expensive proprietary software but can use FOSS instead (as a student, I used a version of <a href="https://www.libreoffice.org/">LibreOffice</a>). Those with more resources may have been able to use proprietary software before they start work, which gives them an edge when they land their first job; FOSS helps level the field. Similarly, if you’re trying to be a leading public sector institution and inspire other institutions around the world, you’ll go further if you use FOSS because it’s accessible in every country.</li>
</ul>
<div class="no-row-height column-margin column-container"><div id="fn4"><p><sup>4</sup>&nbsp;I can hear you saying that some FOSS is harder to use. Linux being harder to use than Windows is a good example, although that one is <em>far</em> less true today given the rise of user-friendly Linux distributions like Ubuntu. But if a piece of software is <em>so</em> much harder to use, I would argue it isn’t a close substitute!</p></div><div id="fn5"><p><sup>5</sup>&nbsp;If it did have security issues, they probably—but not certainly—would have already been exploited and fixed <em>because</em> it’s open. Call it the weak open source security efficiency hypothesis.</p></div></div></section>
<section id="operational-efficiencies" class="level3">
<h3 class="anchored" data-anchor-id="operational-efficiencies">Operational efficiencies</h3>
<p>AKA process efficiency. Computer tasks aren’t the only area where one could find operational efficiencies. There’s a whole set of efficiency gains that most organisations could find in their standard processes too. One example is around the permissions required to access data or install certain software, etc. In most public sector organisations I’ve observed, these have to be separately approved for each individual within a team. But another model would be to use team-level permissions—if someone joins Team A, it is assumed they will need access to a set of data used by Team A. This eliminates a lot of tedious time for managees and managers in requesting and approving, respectively, permissions. In another post in this series, <a href="../../../blog/posts/public-sector-communication-efficiency/index.html#asynchronous-working">I wrote about how presuming access and openness within the organisation</a> could also provide a boost—in that case, no permissions are needed!</p>
<p>That’s just one example. But by analysing how staff have to spend their time to get things done, one could find many more—and again try to minimise them.</p>
</section>
</section>
<section id="data-for-analysis" class="level2 page-columns page-full">
<h2 class="anchored" data-anchor-id="data-for-analysis">Data for analysis</h2>
<p>Most public sector organisations, probably most organisations in general, have a legacy systems problem that makes analysing data (particularly big data) painful. There are also legitimate reasons why security is often tight around the analytical tools that most public sector orgs can use. Between these two, it can take analysts <em>orders of magnitude longer</em> to analyse data using an institutions’ systems than it would the same size dataset on their own laptop or cloud account or open system. Analysis would go faster and be able to do more if these datasets were available outside the security fence.</p>
<p>Additionally, there is strong evidence that releasing datasets into the wild has huge benefits (eg <a href="https://www.usgs.gov/news/featured-story/landsats-economic-value-increases-256-billion-2023">Landsat data being released for free by the US government was estimated to have generated 25 billion USD in 2023</a>). You can think of the economic narrative like this: data are an input into lots of potential public goods. People wouldn’t pay for the data directly to make those public goods, as they don’t capture enough of the benefit themselves—but make the data free, and their private capture is enough for them to put the work in and generate public goods worth way more than the total revenue obtained by directly charging for the upstream data.</p>
<p>The form of those public goods may be especially beneficial for public sector institutions. You can think of situations where the public sector doesn’t have enough resources to perform all of the analysis it would ideally have. But, with the data out there, others might do the analysis—co-opting others to help with the overall analytical mission! Those other analysts might just be in another government department. (More on this idea of sharing as an efficiency in <a href="../../../blog/posts/public-sector-communication-efficiency/index.html#publishing-analysis">the previous post in this series</a>.)</p>
<section id="releasing-real-data" class="level3 page-columns page-full">
<h3 class="anchored" data-anchor-id="releasing-real-data">Releasing real data</h3>
<p>The first and most simple response to these pressures and productivity-sapping systems is to simply publish the data publicly, at least for data that aren’t proprietary or sensitive. And perhaps we can all be a bit more radical about what is and is not sensitive. An obvious example is firm-level data: UK legislation currently prohibits the Office for National Statistics from publishing information on individual firms, even though Companies House, another part of the UK state, <em>does</em> publish information on individual firms’ balance sheets.</p>
<p>Having important data be available publicly means staff within an institution can use non-restricted tools to work with it, which would be a massive boost to productivity. It means that any public goods that are possible with it are much more likely to be created, unlocking the true value and, potentially, it means that more relevant analysis would be happening with it than otherwise would be the case. And, especially if the <a href="../../../blog/posts/the-prize-with-apis/the-prize-with-apis.html">data are released via an API</a><sup>6</sup>, it means no more FOI requests or emails from other institutions—people can self-serve the cuts of the data that they need. The substantial time that public sector analysts spend serving Freedom of Information (FOI) and inter-departmental requests for data would simply no longer be needed for any data that are published.</p>
<div class="no-row-height column-margin column-container"><div id="fn6"><p><sup>6</sup>&nbsp;It’s not that hard to build an API using modern cloud tools, by the way. Check out <a href="../../../blog/posts/build-a-cloud-api/build-a-cloud-api.html">this post</a> for an example.</p></div></div></section>
<section id="synthetic-data" class="level3">
<h3 class="anchored" data-anchor-id="synthetic-data">Synthetic data</h3>
<p>However, most datasets that are used within public sector institutions are proprietary or sensitive or both. They simply can’t be shared externally. But what if there was a way to share the data without sharing the data? Well, there is—it’s called <strong>synthetic data</strong>.</p>
<p>Synthetic data are artificially generated data that are made to resemble real-world data in some of their statistical properties but in a non-disclosive way. It has been used by the US Census Bureau. There are mathematical guarantees about how hard it is to reverse the process and get back to the original data (a bit like encryption.) And you can set how strong this encryption is, though there’s a trade-off with preserving aggregate statistical properties.</p>
<p>One way synthetic data could be beneficial is if it was released and people could still get accurate statistical aggregations from it. But I imagine that most institutions wouldn’t be super comfortable with this as some statistics will be accurate but others won’t be. You can imagine instead creating synthetic data with extremely strong protection and releasing that, and allowing people to develop with it. Once an analytical script developed on the synthetic data is ready, staff and even externals could submit it to be run on the sensitive data (on what is probably a more expensive computational setup.) The benefit of synthetic data is that the bottleneck around the sensitivities of the data and access to it via specialist secure platforms are removed for most of the development process (except the last stage.) From a science point of view, there are even some attractions to working blind to the actual results in this way—it’s a bit like pre-registering the analysis for a medical trial.</p>
<p>Of course, there is no free lunch—there is some work to do for the institutional data owner in generating and releasing the synthetic data, and to do it at scale probably needs cloud computing. And it’s not the easiest concept to explain to people who are less in the data science space either, so senior colleagues may struggle to understand the risks and benefits.</p>
<p>There are well-developed packages to work with synthetic data, for example to:</p>
<ul>
<li>generate synthetic data, eg <a href="https://github.com/sdv-dev/SDV">Synthetic Data Vault</a> and <a href="https://github.com/mostly-ai/mostlyai">Synthetic Data SDK</a></li>
<li>evaluate how close synthetically generated data are to the real underlying data, eg <a href="https://docs.sdv.dev/sdmetrics">SDmetrics</a></li>
<li>benchmark how quickly <a href="https://github.com/sdv-dev/SDGym">synthetic data are generated</a> according to different methods</li>
</ul>
<p>I was quite involved in synthetic data at the Office for National Statistics, running a team that tried to use it in practice. They did excellent work. If you’re really interested in using it in anger, you should reach out to ONS’ Data Science Campus. You can find a number of posts on the topic that are by my former colleagues <a href="https://datasciencecampus.ons.gov.uk/category/projects/synthetic-data-and-pets/">here</a>. They also have a recent data linkage package that uses synthetic data so that two public sector institutions can link data without seeing sensitive columns; you can find that <a href="https://github.com/datasciencecampus/pprl_toolkit">here</a>.</p>
</section>
<section id="fake-data" class="level3">
<h3 class="anchored" data-anchor-id="fake-data">Fake data</h3>
<p>Even if the risk appetite of your institution doesn’t extend to synthetic data, perhaps it would extend to entirely <strong>fake data</strong>? If you like this is an extreme form of synthetic data where the encryption cannot be reversed (as opposed to ‘cannot practically be reversed’ with synthetic data). Fake data have only the same data types (eg string) and forms (eg an email address) of original data but are otherwise completely and utterly made up. This means that fake data have all the benefits of synthetic data bar statistical aggregates being approximately similar to those that would have derived from the real data.</p>
<p>There are a number of packages that can generate fake data that are extensible and with which you can create schemas that produce an entire set of fake data in the style you want. For example, <a href="https://faker.readthedocs.io/">Faker</a>:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb1" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb1-1"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> faker <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> Faker</span>
<span id="cb1-2">fake <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> Faker()</span>
<span id="cb1-3"></span>
<span id="cb1-4">fake.name()</span>
<span id="cb1-5"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 'Lucy Cechtelar'</span></span>
<span id="cb1-6"></span>
<span id="cb1-7">fake.address()</span>
<span id="cb1-8"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># '426 Jordy Lodge</span></span>
<span id="cb1-9"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#  Cartwrightshire, SC 88120-6700'</span></span>
<span id="cb1-10"></span>
<span id="cb1-11">fake.text()</span>
<span id="cb1-12"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 'Sint velit eveniet. Rerum atque repellat voluptatem quia rerum. Numquam excepturi</span></span>
<span id="cb1-13"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#  beatae sint laudantium consequatur. Magni occaecati itaque sint et sit tempore. Nesciunt</span></span>
<span id="cb1-14"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#  amet quidem. Iusto deleniti cum autem ad quia aperiam.</span></span>
<span id="cb1-15"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#  A consectetur quos aliquam. In iste aliquid et aut similique suscipit. Consequatur qui</span></span>
<span id="cb1-16"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#  quaerat iste minus hic expedita. Consequuntur error magni et laboriosam. Aut aspernatur</span></span>
<span id="cb1-17"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#  voluptatem sit aliquam. Dolores voluptatum est.</span></span>
<span id="cb1-18"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#  Aut molestias et maxime. Fugit autem facilis quos vero. Eius quibusdam possimus est.</span></span>
<span id="cb1-19"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#  Ea quaerat et quisquam. Deleniti sunt quam. Adipisci consequatur id in occaecati.</span></span>
<span id="cb1-20"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#  Et sint et. Ut ducimus quod nemo ab voluptatum.'</span></span></code></pre></div></div>
<p>And <a href="https://mimesis.name/master/">Mimemsis</a>:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb2" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb2-1"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> mimesis <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> Person</span>
<span id="cb2-2"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> mimesis.locales <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> Locale</span>
<span id="cb2-3"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> mimesis.enums <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> Gender</span>
<span id="cb2-4">person <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> Person(Locale.EN)</span>
<span id="cb2-5"></span>
<span id="cb2-6">person.full_name(gender<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>Gender.FEMALE)</span>
<span id="cb2-7"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Output: 'Antonetta Garrison'</span></span>
<span id="cb2-8"></span>
<span id="cb2-9">person.full_name(gender<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>Gender.MALE)</span>
<span id="cb2-10"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Output: 'Jordon Hall'</span></span></code></pre></div></div>
</section>
</section>
<section id="reproducible-analytical-pipelines" class="level2 page-columns page-full">
<h2 class="anchored" data-anchor-id="reproducible-analytical-pipelines">Reproducible analytical pipelines</h2>
<p>A reproducible analytical pipeline (RAP) is a series of steps that trigger one after the other to produce an analysis from data ingestion all the way to the end product. It’s a form of <em>automation</em>. To illustrate why there are big efficiency gains to be had with them, let’s first look at an end-to-end process that <em>doesn’t</em> use RAP.</p>
<section id="getting-a-bad-rap" class="level3">
<h3 class="anchored" data-anchor-id="getting-a-bad-rap">Getting a bad RAP</h3>
<p>Let’s take a high-stakes analytical pipeline that has lots of potential for improvement and efficiencies—but which is extremely common in the wider public sector. Imagine a process for publishing some key economic statistics that looks like this:</p>
<ol type="1">
<li>Statistical office publishes new data in an Excel file</li>
<li>An analyst downloads the file onto a network drive</li>
<li>Analyst checks the file by eye and copies specific cells into another Excel workbook that has the analysis</li>
<li>Analyst drags out the equations behind the cells to incorporate the new data</li>
<li>Values from the calculations get manually typed into a third spreadsheet, which also has a tab with sensitive data in</li>
<li>The results then get manually copied into a Word document for a report</li>
<li>The final calculations tab from the third spreadsheet is published externally, alongside the report</li>
</ol>
<p>Apart from being heavily manual, this process has multiple failure points:</p>
<ul>
<li>Perhaps there’s an earlier, unrevised version of the same file already on a network drive and that is used by mistake in step 3.</li>
<li>Excel messes with data—it <a href="https://www.theverge.com/2020/8/6/21355674/human-genes-rename-microsoft-excel-misreading-dates">changes dates and names, and data types</a></li>
<li>Excel doesn’t support larger datasets, so rows are missed</li>
<li>Manual copy-paste errors</li>
<li>Equations are different in different cells, and the analyst doesn’t notice because equations are hidden</li>
<li>Calculations break if someone adds/removes rows</li>
<li>Cannot see if there are changes in the calculations since last time the spreadsheet was run</li>
<li>Time consuming manual checks</li>
<li>Any mistake in step 1 means redoing all steps manually again</li>
<li>High risk of the sensitive data being published alongside the other data</li>
</ul>
<p>Perhaps you think this is all a bit implausible. Sadly, it isn’t! Here are some real-world examples of where Excel-based pipelines have gone wrong:</p>
<ul>
<li>On Thursday, 3rd August 2023, an FOI request was made to the Police Service of Northern Ireland (PSNI). It asked for the “number of officers at each rank and number of staff at each grade”. PSNI sent back not just the numbers of officers at each rank in the spreadsheet they returned but every employee’s initials, surname, and unit—even those working out of MI5’s Northern Irish headquarters. Sensitive information on <em>10,000 people</em> was released by mistake, because of an extra tab in the Excel file.</li>
<li>There’s the infamous case of the UK’s Covid Test and Trace programme. Civil servants were tracking cases in an Excel spreadsheet that was limited to 65,000 rows, and thus missed a bunch of infections, not to mention crucial links in the spread of the disease. Excel simply ran out of numbers. Work by economists Thiemo Fetzer and Thomas Graeber has shown that this <a href="https://www.medrxiv.org/content/10.1101/2020.12.10.20247080v1">probably led to an extra 125,000 COVID infections</a>.</li>
<li><a href="https://www.ft.com/content/57acfb54-6a62-46d6-b930-87b19c2ab2d3">Hans Peter Doskozil</a> was mistakenly announced as the new leader of Austria’s Social Democrats (SPÖ) because of a “technical error in the Excel list”.</li>
<li>One government department published a spreadsheet that still had a note in it that said “Reason for arrest data is dodgy so maybe we shouldn’t publish it.”</li>
<li>Mathematical models written in Excel are especially vulnerable to mistakes. In a paper that was highly influential, economists Carmen Reinhart and Ken Rogoff found that economic growth slows dramatically when the size of a country’s debt rises above 90% relative to GDP. But the formula in their spreadsheet column missed out some of the countries in their data. A <a href="https://www.ft.com/content/1c3bae3c-a6ce-11e2-95b1-00144feabdc0">student found the error</a>, and realised that removing it made the results less dramatic.</li>
<li>The Norwegian Soveign Wealth Fund’s 2023 <a href="https://www.ft.com/content/db864323-5b68-402b-8aa5-5c53a309acf1">Excel snafu</a> saw it miscalculate an index due to a “decimal error”. The mistake cost 92 million dollars.</li>
<li>Back in 2012, J.P. Morgan went even further and racked up <em>$6 billion of losses</em> in part due to a “value-at-risk” model being <a href="https://www.ft.com/content/80bc5046-6ed2-30ef-b31a-bcbeac55d8bb">manually copied and pasted</a> across spreadsheet cells.</li>
</ul>
<p>There are many more examples: you can find a list at the delightfully named <a href="https://eusprig.org/">European Spreadsheet Risks Interest Group</a>. Now, good reproducible analytical pipeline (RAP) practices cannot protect you from mistakes, but <strong>reproducible analytical pipelines can substantially lower the risk of errors while also saving time via automation</strong>.</p>
</section>
<section id="good-rap" class="level3 page-columns page-full">
<h3 class="anchored" data-anchor-id="good-rap">Good RAP</h3>
<p>What does a good RAP look like? There are a number of technologies and practices that can help create a RAP that avoids mistakes and saves tons of time and labour to boot. Some of these RAP technologies and practices are listed below in, roughly, the order in which they would be used in a data pipeline.</p>
<ul>
<li>Use an <a href="../../../blog/posts/the-prize-with-apis/the-prize-with-apis.html">API</a> to programmatically access the latest version of the data</li>
<li>Keep a copy of the raw, downloaded version of the data so that you can always come back to it. Use datetime versioning to keep track of download time.</li>
<li>Use <a href="../../../blog/posts/the-data-validation-landscape-in-2025/index.html">data validation</a> checks to ensure that the data looks how you expect it to on receipt</li>
<li>Use code to process the data, rather than Excel spreadsheets. And use code<sup>7</sup> that is:
<ul>
<li>under <a href="https://aeturrell.github.io/coding-for-economists/wrkflow-version-control.html">version control</a>, so that it is completely auditable and all changes are logged</li>
<li>is well-formatted and linted, perhaps by a tool such as <a href="https://astral.sh/ruff">Ruff</a>.</li>
<li>has <a href="https://pre-commit.com/">pre-commit</a> enabled with hooks that check for data, secrets, and other sensitive information submitted by mistake, and which auto-checks quality</li>
<li>has tests so that everyone feels assured the code is doing the right thing</li>
<li>subject to pull requests whenever it changes, so that multiple people sign off on important code</li>
<li>is tested, etc., using runners, eg GitHub Actions</li>
<li>is reproducible, including the set of packages and version of the language (eg via a <code>pyproject.toml</code> file and use of a tool like <a href="https://docs.astral.sh/uv/">uv</a>) and ideally for which the operating system is also reproducible too (eg via <a href="https://www.docker.com/">Docker</a>)</li>
</ul></li>
<li>Use a data orchestration tool to execute the entire code processing pipeline from start to finish. Some popular orchestration tools are <a href="https://airflow.apache.org/">Apache Airflow</a> (first generation) and <a href="https://docs.dagster.io/">Dagster</a> and <a href="https://docs.prefect.io/v3/get-started/index">Prefect</a> (both second generation). See below for an example of Prefect in action. Orchestration flows can be triggered on events, for example if a new file lands in a folder, or on a schedule. (An easier way in to execute-code-on-event for small, local problems is via packages such as <a href="https://pythonhosted.org/watchdog/index.html">watchdog</a> which watches for file changes and takes actions.)</li>
<li>If there is a big overhead to using big data systems, use <a href="https://duckdb.org/">DuckDB</a> for processing fairly large datasets locally (say, up to 1TB). Don’t like writing SQL queries? Don’t worry no-one does—but <a href="https://ibis-project.org/">there are packages that can help</a>.</li>
<li>Store the cleaned data or results in a database.</li>
<li>Use anomaly detection techniques, etc, on the created statistics to check that they are plausible. (With a human in the loop!)</li>
<li><a href="../../../blog/posts/build-a-cloud-api/build-a-cloud-api.html">Build an API</a> to disseminate the created statistics</li>
<li>Create the report using a tool which can reproduce the same charts via code each time, eg <a href="https://quarto.org/">Quarto</a>, which can execute code chunks and be used to produce Word documents, HTML pages, PDFs, and even websites. (I wrote a short guide <a href="https://aeturrell.github.io/coding-for-economists/wrkflow-quarto.html">here</a>.) Oh yeah, it does Powerpoints and Beamer presentations too.</li>
</ul>
<div class="no-row-height column-margin column-container"><div id="fn7"><p><sup>7</sup>&nbsp;Many of these code features are present in the <a href="https://github.com/aeturrell/cookiecutter-python-package">cookiecutter Python package</a>; blog on that available <a href="../../../blog/posts/ultra-modern-python-cookiecutters/index.html">here</a>.</p></div></div><p>Just as an aside, here’s how to orchestrate a simple data pipeline that runs every hour with Prefect:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb3" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb3-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Contents of flow.py</span></span>
<span id="cb3-2"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> typing <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> Any</span>
<span id="cb3-3"></span>
<span id="cb3-4"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> httpx</span>
<span id="cb3-5"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> prefect <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> flow, task <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Prefect flow and task decorators</span></span>
<span id="cb3-6"></span>
<span id="cb3-7"><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">@task</span>(retries<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span>)</span>
<span id="cb3-8"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> fetch_stats(github_repo: <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">str</span>) <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-&gt;</span> <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">dict</span>[<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">str</span>, Any]:</span>
<span id="cb3-9">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""Task 1: Fetch the statistics for a GitHub repo"""</span></span>
<span id="cb3-10"></span>
<span id="cb3-11">    api_response <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> httpx.get(<span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">f"https://api.github.com/repos/</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span>github_repo<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span>)</span>
<span id="cb3-12">    api_response.raise_for_status() <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Force a retry if not a 2xx status code</span></span>
<span id="cb3-13">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">return</span> api_response.json()</span>
<span id="cb3-14"></span>
<span id="cb3-15"></span>
<span id="cb3-16"><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">@task</span></span>
<span id="cb3-17"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> get_stars(repo_stats: <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">dict</span>[<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">str</span>, Any]) <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-&gt;</span> <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">int</span>:</span>
<span id="cb3-18">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""Task 2: Get the number of stars from GitHub repo statistics"""</span></span>
<span id="cb3-19"></span>
<span id="cb3-20">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">return</span> repo_stats[<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'stargazers_count'</span>]</span>
<span id="cb3-21"></span>
<span id="cb3-22"></span>
<span id="cb3-23"><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">@flow</span>(log_prints<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">True</span>)</span>
<span id="cb3-24"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> show_stars(github_repos: <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">list</span>[<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">str</span>]):</span>
<span id="cb3-25">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""Flow: Show the number of stars that GitHub repos have"""</span></span>
<span id="cb3-26"></span>
<span id="cb3-27">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">for</span> repo <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">in</span> github_repos:</span>
<span id="cb3-28">        <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Call Task 1</span></span>
<span id="cb3-29">        repo_stats <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> fetch_stats(repo)</span>
<span id="cb3-30"></span>
<span id="cb3-31">        <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Call Task 2</span></span>
<span id="cb3-32">        stars <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> get_stars(repo_stats)</span>
<span id="cb3-33"></span>
<span id="cb3-34">        <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Print the result</span></span>
<span id="cb3-35">        <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">print</span>(<span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">f"</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span>repo<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;">: </span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span>stars<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="ss" style="color: #20794D;
background-color: null;
font-style: inherit;"> stars"</span>)</span>
<span id="cb3-36"></span>
<span id="cb3-37"></span>
<span id="cb3-38"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Run the flow</span></span>
<span id="cb3-39"><span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">if</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">__name__</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"__main__"</span>:</span>
<span id="cb3-40">    show_stars([</span>
<span id="cb3-41">        <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"PrefectHQ/prefect"</span>,</span>
<span id="cb3-42">        <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"pydantic/pydantic"</span>,</span>
<span id="cb3-43">        <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"huggingface/transformers"</span></span>
<span id="cb3-44">    ])</span></code></pre></div></div>
<p>This flow can be executed on a schedule by running the below (assuming that you have already set up a worker pool—see the Prefect documentation for more):</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb4" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb4-1"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> prefect <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> flow</span>
<span id="cb4-2"></span>
<span id="cb4-3"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Source for the code to deploy (here, a GitHub repo)</span></span>
<span id="cb4-4">SOURCE_REPO<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"https://github.com/prefecthq/demos.git"</span></span>
<span id="cb4-5"></span>
<span id="cb4-6"><span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">if</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">__name__</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"__main__"</span>:</span>
<span id="cb4-7">    flow.from_source(</span>
<span id="cb4-8">        source<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>SOURCE_REPO,</span>
<span id="cb4-9">        entrypoint<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"my_workflow.py:show_stars"</span>, <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Specific flow to run</span></span>
<span id="cb4-10">    ).deploy(</span>
<span id="cb4-11">        name<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"my-first-deployment"</span>,</span>
<span id="cb4-12">        parameters<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>{</span>
<span id="cb4-13">            <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"github_repos"</span>: [</span>
<span id="cb4-14">                <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"PrefectHQ/prefect"</span>,</span>
<span id="cb4-15">                <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"pydantic/pydantic"</span>,</span>
<span id="cb4-16">                <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"huggingface/transformers"</span></span>
<span id="cb4-17">            ]</span>
<span id="cb4-18">        },</span>
<span id="cb4-19">        work_pool_name<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"my-work-pool"</span>,</span>
<span id="cb4-20">        cron<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"0 * * * *"</span>,  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Run every hour</span></span>
<span id="cb4-21">    )</span></code></pre></div></div>
<p>I share this to show that automation of code doesn’t have to be super complicated—with the cloud, it’s achievable in a few lines!</p>
<p>There’s a lot in the list above. But doing all of the above not only massively reduces the risk of mistakes, it also makes it <em>far</em> easier to find mistakes when they do occur. And, the automation saves masses of time—allowing analysts to focus where they can add the most value.</p>
<p>You can find more on reproducibility and RAP <a href="https://aeturrell.github.io/coding-for-economists/wrkflow-rap.html">here</a> and <a href="https://analysisfunction.civilservice.gov.uk/support/reproducible-analytical-pipelines/">here</a>, and <a href="https://aeturrell.github.io/coding-for-economists/auto-research-outputs.html">here</a> in the context of research.</p>
<p>Finally, I realise that having people with the skills to do RAP to this standard is difficult and potentially expensive in itself. The market clearly recognises the benefits too, which is why data scientists are in such high demand! For me, overall, this is a much better way of doing things and even if the input costs are potentially higher, the output quality is likely to be <em>significantly</em> higher—so I contend that you still win out on productivity overall.</p>
</section>
</section>
<section id="technical-communication-and-code-documentation" class="level2">
<h2 class="anchored" data-anchor-id="technical-communication-and-code-documentation">Technical communication and code documentation</h2>
<div class="callout callout-style-default callout-note callout-titled">
<div class="callout-header d-flex align-content-center">
<div class="callout-icon-container">
<i class="callout-icon"></i>
</div>
<div class="callout-title-container flex-fill">
Note
</div>
</div>
<div class="callout-body-container callout-body">
<p>This section is significantly more technical than the others and some readers may want to skip it and go straight to Model Behaviour.</p>
</div>
</div>
<p>I already covered the efficiencies that could be unlocked by <a href="../../../blog/posts/public-sector-communication-efficiency/index.html#asynchronous-working">obsessively using documentation</a> in a previous post on efficiency. Many of the same benefits apply from documenting code obsessively too. As a reminder, those benefits include faster onboarding, having a single source of truth, making it easy to avoid stale documentation, more auditability, less likelihood of (in this case) code being used incorrectly, and, if the documentation is shared, transparency.</p>
<p>However, there are some extra actions you can take with the documentation of code to squeeze every last drop of efficiency out of these benefits.</p>
<p>At a minimum, each code repository should have a markdown README file that sets out what it is about and how to use it.</p>
<p>The code itself should have sensible comments, and use informative variable, function, method, and class names. Functions, classes, and methods should have <a href="https://en.wikipedia.org/wiki/Docstring">docstrings</a> (comment blocks that say what the object does.) I tend to use the Google-style docstring conventions, defined in <a href="https://google.github.io/styleguide/pyguide.html#38-comments-and-docstrings">Google’s Python style guide</a>, but there are other docstring conventions around. Here’s an example of Google-style docstring on a function:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb5" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb5-1"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> _create_unicode_hist(series: pd.Series) <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-&gt;</span> pd.Series:</span>
<span id="cb5-2">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""Return a histogram rendered in block unicode.</span></span>
<span id="cb5-3"></span>
<span id="cb5-4"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    Given a pandas series of numerical values, returns a series with one</span></span>
<span id="cb5-5"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    entry, the original series name, and a histogram made up of unicode</span></span>
<span id="cb5-6"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    characters. However, note that the histogram is very approximate, partly</span></span>
<span id="cb5-7"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    due to limitations in how unicode is displayed across systems.</span></span>
<span id="cb5-8"></span>
<span id="cb5-9"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    Args:</span></span>
<span id="cb5-10"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        series (pd.Series): Numeric column of data frame for analysis</span></span>
<span id="cb5-11"></span>
<span id="cb5-12"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    Returns:</span></span>
<span id="cb5-13"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        pd.Series: Index of series name and entry with unicode histogram as</span></span>
<span id="cb5-14"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        a string, eg '▃▅█'</span></span>
<span id="cb5-15"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    """</span></span></code></pre></div></div>
<p>It’s not just helpful for people looking at the innards of the code, it also helps if people are using functions and methods from the code in other code. If someone were to call <code>help(_create_unicode_hist)</code> then they would get the docstring back. In sophisticated development environments like Visual Studio Code, they would get a pop-up of this docstring even when just hovering their mouse over the function.</p>
<p>For critical code, you may want to consider having code <em>type annotations</em>, aka <em>type hints</em> too. The code example above has those: in this case <code>series</code> has type <code>pd.Series</code> and the function returns <code>pd.Series</code> too (this is what <code>-&gt; pd.Series</code> does.) Type hints are just that, hints to help people understand what types of object are supposed to be going into and out of modular parts of code like functions and methods. Because they are hints, they’re included in this section on documenting code—adding type hints helps others understand what’s supposed to happen.</p>
<p>If you’re really concerned about making sure the code is not used incorrectly, you can even enforce type hints, either at run time, so that if a user runs the code with the wrong data types an error is thrown, or at development time, so that the developer gets a warning that the internals of the code don’t adhere to the given type hints. <a href="https://typeguard.readthedocs.io/">typeguard</a> is a popular package for run-time type checking, while for development time (aka static) type checking, <a href="https://mypy-lang.org/">Mypy</a> and <a href="https://microsoft.github.io/pyright/">Microsoft’s Pyright</a> are the biggest libraries around.</p>
<p>But you can go further! You’ll notice that the type annotations are, in effect, repeated in both the docstring and in the function signature. This means they could say different things, which would be confusing to users. Have no fear, though, because <em>there’s a package for that</em> (there always is): <a href="https://jsh9.github.io/pydoclint/">Pydoclint</a> trawls through your docstrings and type annotations and tells you if they diverge.</p>
<p>But you can go even further! Let’s say you want to help people understand, practically, how to use a function, method, or class. Well, you can also add example code to docstrings. Here’s the format:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb6" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb6-1"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> skim(</span>
<span id="cb6-2">    df_in: Union[pd.DataFrame, pl.DataFrame],</span>
<span id="cb6-3">) <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-&gt;</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">None</span>:</span>
<span id="cb6-4">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""Skim a pandas or polars dataframe and return visual summary statistics on it.</span></span>
<span id="cb6-5"></span>
<span id="cb6-6"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    skim is an alternative to pandas.DataFrame.describe(), quickly providing</span></span>
<span id="cb6-7"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    an overview of a data frame via a table displayed in the console. It produces a different set of summary</span></span>
<span id="cb6-8"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    functions based on the types of columns in the dataframe. You may get</span></span>
<span id="cb6-9"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    better results from ensuring that you set the datatypes in your dataframe</span></span>
<span id="cb6-10"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    you want before running skim.</span></span>
<span id="cb6-11"></span>
<span id="cb6-12"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    Note that any unknown column types, or mixed column types, will not be</span></span>
<span id="cb6-13"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    processed.</span></span>
<span id="cb6-14"></span>
<span id="cb6-15"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    Args:</span></span>
<span id="cb6-16"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        df_in (Union[pd.DataFrame, pl.DataFrame]): Dataframe to skim.</span></span>
<span id="cb6-17"></span>
<span id="cb6-18"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    Raises:</span></span>
<span id="cb6-19"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        NotImplementedError: If the dataframe has a MultiIndex column structure.</span></span>
<span id="cb6-20"></span>
<span id="cb6-21"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    Examples</span></span>
<span id="cb6-22"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    --------</span></span>
<span id="cb6-23"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    Skim a dataframe</span></span>
<span id="cb6-24"></span>
<span id="cb6-25"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        &gt;&gt;&gt; df = pd.DataFrame(</span></span>
<span id="cb6-26"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">                {</span></span>
<span id="cb6-27"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">                'col1': ['Philip', 'Turanga', 'bob'],</span></span>
<span id="cb6-28"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">                'col2': [50, 100, 70],</span></span>
<span id="cb6-29"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">                'col3': [False, True, True]</span></span>
<span id="cb6-30"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">                })</span></span>
<span id="cb6-31"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        &gt;&gt;&gt; df["col1"] = df["col1"].astype("string")</span></span>
<span id="cb6-32"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">        &gt;&gt;&gt; skim(df)</span></span>
<span id="cb6-33"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    """</span></span></code></pre></div></div>
<p>where you can add as many examples as you like.</p>
<p>Now this is helpful in itself because it guides users as to how to use a bit of code. But, again, what if the examples were written 2 years ago and the code has changed? <em>There’s a package for that!</em> Using a package called <a href="https://github.com/Erotemic/xdoctest">xdoctest</a> you can, you guessed it, test that any examples in your docstrings actually run and do what the docstring says they will do.</p>
<p>This is all very nice, but lots of people don’t want to go trawling through code. They want to read a glossy guide. The problem with guides, though, is that they’re often left to go stale while the code itself changes. You may have seen situations where there’s an Excel-based model and somewhere, floating around, a Word document explaining what that Excel model does. But who knows how much they are conjointly kept up to date?</p>
<p>Fortunately, there’s a way to build documentation that sits with your code, and is in lock-step with it. The technology is called <a href="https://quarto.org/">Quarto</a>, and it’s a flexible document publishing system (where “document” includes websites and slides!) Perhaps the greatest power of Quarto is that you can put chunks of code into the source that gets compiled into a document and, in that final product, the outputs from the code will appear too (and only the outputs if you wish.) This means you can create documentation with Quarto (in the form a Word document or website) and keep the source files for that documentation in exactly the same place as your code. <em>Even better</em>, you can include examples of using the code in the documentation and you know it will be <em>exactly</em> the same code as in your software, and that it will work—because if it doesn’t, the documentation simply won’t build!</p>
<p>But wait… you can go <em>even</em> further! You’ve spent all of this effort on lovely docstrings that tell people about how the modular parts of the code work and are meant to be used, and perhaps you even have examples in there too. What if you could automatically include all of that nice within-code documentation in your actual documentation? <em>There’s a package for that</em>! The massively underrated <a href="https://machow.github.io/quartodoc">Quartodoc</a> allows you to automatically pull docstrings from your code base and insert them into your documentation. It really is efficient, helpful, and a lot of fun too.</p>
<p>For an example of this self-documenting behaviour in action, check out the website for my package <a href="https://aeturrell.github.io/specification_curve/">Specification Curve</a> or for econometrics regression package <a href="https://py-econometrics.github.io/pyfixest/pyfixest.html">Pyfixest</a>. In both cases, the writing is interspersed with examples of running real code from the packages. For both websites, there’s a reference page where parts of the code are automatically documented using Quartodoc.</p>
<p>A lot of this is overkill for a lot of projects. But for complex analysis code on the critical path to a major publication that your institution produces? It’s a huge help and likely worth it.</p>
</section>
<section id="model-behaviour" class="level2">
<h2 class="anchored" data-anchor-id="model-behaviour">Model Behaviour</h2>
<p>When it comes to models, there are a lot of bad practices around. For me, the most worrying is that <em>most models are not versioned</em>. This means that if a decision was taken, or an analysis performed, with a model at time <img src="https://latex.codecogs.com/png.latex?T=t"> and it’s now time <img src="https://latex.codecogs.com/png.latex?T%3Et"> and the model is different, you’ve very little chance of getting back to the original model. Yes, this is true even if you’re using version control. That’s because a model is composed of three things: the code, the data, and (often) the random numbers used. You should version control code, and ideally you’d set the seed of your random numbers in that code too. But few people version data, so even if you have code and random numbers under control, you’ll need to version models.</p>
<p>Why does this matter for efficiency? Not knowing why some part of your analysis is why it is, and having to dig back through and do detective work, is very time-consuming when it is needed. And it happens fairly often with models. A classic example would be if you were asked to do a retrospective forecast evaluation—how did the model you used two years ago perform? Has performance degraded since then? Additionally, you can test whether changes you’ve made to the model’s algorithm are actually improving its performance. So <strong>investing in model versioning can reduce time costs further down the line</strong>.</p>
<p>A second part of model efficiency is continuous integration and continuous deployment. Enough has been written elsewhere about what these are that I’ll give the briefest summary of what they mean in this context. Continuous integration for models could involve automatically evaluating performance, or other metrics, when a new model is created. Continuous deployment would mean that when a new version of a model is ready, it automatically gets served up as an API.</p>
<p>The latter idea, that models get served up as an API, is really important for efficiency. Today, if I’m in department X and I want to use a model in department Y to get predictions, I have to email someone in department Y, get them the permissions for the data that will form the input and then the data itself, and persuade them to take time out of whatever else they were doing to run it! If the model is deployed, I skip all that, and can work asynchronously—I simply use my data to ping the API and get the prediction straight back into my analytical tool. There’s more about asynchronous working and its efficiency benefits <a href="../../../blog/posts/public-sector-communication-efficiency/index.html#asynchronous-working">here</a>.</p>
<p>There is a way to get most of these benefits pretty easily on the cloud using a <em>model registry</em>: a repository for privately storing models and their versions (from which those models can also be deployed.) I wrote at length about model registries <a href="../../../blog/posts/why-have-a-model-registry/index.html">here</a> and, rather than repeat myself here, I’ll defer the reader to that post for more information.</p>
</section>
<section id="artificial-intelligence-ai" class="level2 page-columns page-full">
<h2 class="anchored" data-anchor-id="artificial-intelligence-ai">Artificial Intelligence (AI)</h2>
<p>A former boss of mine, now even more heavily invested in the AI world than he was when we worked together, recently told me “don’t have an AI strategy, have a business strategy that makes use of AI.” It stuck with me, partly because there are so many AI strategies around, but mostly because he’s right: AI is a tool and not a business plan. We don’t have a keyboard strategy or an email strategy.<sup>8</sup></p>
<div class="no-row-height column-margin column-container"><div id="fn8"><p><sup>8</sup>&nbsp;Though now I think about it, perhaps there’d be some merit to the latter?</p></div></div><p>That said, AI is also quite new and people are still thinking through how it can be used to improve efficiency. So, in this section, I offer a few ideas around using AI to improve analysis and operations. (In the <a href="../../../blog/posts/public-sector-communication-efficiency/index.html">previous post in this series</a>, which covered communication and co-ordination, some AI uses also featured—for transcription, meeting summarisation and minutes, and for searching the stock of institutional knowledge.)</p>
<p>A huge one is as a coding aid. GitHub Copilot is the leader, and large-institution-friendly in the sense that it’s all owned by Microsoft. I’ve been hugely impressed by, for example, Claude 3.7 Sonnet, as a back-end model. Whether you’re a new developer and need a chat with an artificial coding expert, or an experienced developer who knows what they need and can prompt the AI in the right way, coding aids are a huge benefit—though they are not trivial to use and there is a learning curve. It’s also very possible that, in the near future, AI models will be able to look at issues and raise pull requests without intervention. Coding aids are also helpful in a world where data science talent is scarce. The prolific Simon Willison has written about <a href="https://simonwillison.net/2025/Mar/11/using-llms-for-code/">how he uses LLMs to help him write code</a>, and there are likely few guides as out there today as good as his.</p>
<p>Another area where AI is particularly helpful with efficiency is review and critique. As a manager, you do often end up saying the same kinds of things over and over again—whether that’s about a note, or code, or a presentation. What if I as a manager had a set of prompts that could precede content by staff in a large language model (LLM) query and which gave them a helpful first cut of what I would have said anyway? That means our time together as two humans reviewing the work can be focused on the less obvious, more subtle, and frankly more interesting questions and issues. A simpler way to do this is for the managee to just ask for a general review. If you do research, LLMs also do a reasonably good job of acting as practice reviewers.</p>
<p>One area of LLMs that is likely to grow is auto-analysis. Instead of me as an analyst having to come up with, say, SQL queries to explore the data, I just say to the LLM, start exploring these data and tell me about anything useful. It doesn’t have to be analysis, it could be coding some software up too. Quite a few models now have code execution capabilities, and consumer-facing products in this space will only grow over time. This is going to democratise who can do analysis, which is a good thing, and will be especially helpful for senior leaders who want to make quick queries.</p>
<p>I’m fairly sure I don’t believe it<sup>9</sup>, but the <a href="https://www.theintrinsicperspective.com/p/why-we-stopped-making-einsteins">theory that geniuses stopped appearing because we stopped educating a handful of people with uber-elite tutors</a> (aka aristocratic tutoring) has always stuck with me. However, you don’t have to believe in the hard form of this thesis to think that having an extremely good tutor who is available 24/7 would be helpful for learning. Well, let’s now imagine that everyone in the public sector can have access to the most knowledgeable (if not always the smartest) tutor on the planet! The possibilities for (self-)teaching using LLMs are strong. I’ve found that I can simply say, “write some questions to test me on X”, or “prepare a lesson on Y”, and get something very sensible.</p>
<div class="no-row-height column-margin column-container"><div id="fn9"><p><sup>9</sup>&nbsp;Many <a href="https://www.astralcodexten.com/p/contra-hoel-on-aristocratic-tutoring">others aren’t convinced either</a>.</p></div></div><p>My final category of ways that AI can immediately help with public sector efficiency is perhaps the most obvious one: improving analysis directly. I wrote a lot about the ways that data science more broadly can help with efficiency in my <a href="../../../blog/posts/data-science-with-impact/index.html">post on data science with impact</a>, so I’ll refer you to that to find out more what the possibilities are in this space. I also have a recent paper and book chapter out on this very question too, called <a href="../../../research/chapters/turrell-2025/index.html">“Cutting through Complexity: How Data Science Can Help Policymakers Understand the World”</a>, which is chock full of examples.</p>
</section>
<section id="automating-and-organising-the-office" class="level2">
<h2 class="anchored" data-anchor-id="automating-and-organising-the-office">Automating and organising the office</h2>
<section id="automation-with-code" class="level3">
<h3 class="anchored" data-anchor-id="automation-with-code">Automation with code</h3>
<p>On this blog, we like code, because it’s transparent, usually it’s FOSS, it’s auditable, and it’s reproducible. So, where possible, it’s best practice to automate with code. What kind of office tasks can be done with code?</p>
<p>In the private sector, tools like Slack—with their very powerful code integrations—seem to be more dominant. However, a lot of public sector organisations use Microsoft and its suite of office tools. These are, happily, scriptable with code. In particular, Microsoft has tools that aid with automating the use of its office software, including:</p>
<ul>
<li><a href="https://github.com/AzureAD/microsoft-authentication-library-for-python">Microsoft Authentication Library</a>, for authentication with Microsoft Azure Active Directory accounts (AAD) and Microsoft Accounts (MSA) using industry standard OAuth2 and OpenID Connect. Basically, this logs you in to Microsoft services when using code.</li>
<li><a href="https://learn.microsoft.com/en-us/graph/overview">Microsoft Graph API</a>, for interacting with cloud-based Microsoft software products (eg Outlook.) Although it’s an API, there is an associated front-end Python package too, called <a href="https://github.com/microsoftgraph/msgraph-sdk-python#readme">Microsoft Graph SDK</a>. There’s also an <a href="https://developer.microsoft.com/en-us/graph/graph-explorer">interactive Graph Explorer tool</a>.</li>
</ul>
<p>Third-parties have built other, simpler-to-use packages on top of these. A notable example is the <a href="https://github.com/vgrem/Office365-REST-Python-Client">Office365-REST-Python-Client</a>. That page has lots of examples but, just to give you a flavour, here’s how to send an email in code:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb7" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb7-1"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> office365.graph_client <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> GraphClient</span>
<span id="cb7-2"></span>
<span id="cb7-3">client <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> GraphClient(acquire_token_func)</span>
<span id="cb7-4"></span>
<span id="cb7-5">client.me.send_mail(</span>
<span id="cb7-6">    subject<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Meet for lunch?"</span>,</span>
<span id="cb7-7">    body<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"The cafeteria is open."</span>,</span>
<span id="cb7-8">    to_recipients<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>[<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"hungryfriend@microsoft.com"</span>]</span>
<span id="cb7-9">).execute_query()</span></code></pre></div></div>
<p>There’s support for doing operations with Outlook, Teams, OneDrive, Sharepoint, OneNote, and Planner (a task manager).</p>
<p>Perhaps a couple of examples would give you a sense of how this can be useful.</p>
<p>Imagine you have a RAP that produces a report, as discussed above. Great, you have a report, now what? Now you can use code to do that last step and email it out to stakeholders or put it in a Sharepoint folder.</p>
<p>Another example: like me, you work with your team on a Kanban board where code tasks are assigned to people. You could have a GitHub action set up so that each time a name is assigned to a card/issue on that Kanban, a task is also sent to that person’s Planner app.</p>
<p>One more example: people are always emailing documents to read. As noted in the previous post though, email is a bad way to communicate source information. Instead, you can have a code setup to watch a Sharepoint folder for any new documents dropped into it. When a new document arrives, the people who need to read the document get a notification that it’s there waiting for their review.</p>
<p>If you’re on Google Docs, there’s an <a href="https://developers.google.com/docs/api/reference/rest">API for that too</a>. And there are plenty of other office efficiency code tools not from the big tech firms. Here’s a fairly random selection:</p>
<ul>
<li>how about good quality, person-labelled transcripts of meetings? See this post on <a href="https://den.dev/blog/how-i-automated-podcast-transcription-with-local-ai/">how someone automated their podcast transcription process with local AI</a> (complete with diarization).</li>
<li>rooting out any <a href="https://github.com/butuzov/deadlinks">deadlinks</a></li>
<li>creating <a href="https://github.com/lincolnloop/python-qrcode">QR codes</a></li>
<li><a href="https://github.com/tebelorg/RPA-Python">Automation of generic interactions with your computer</a> (aka robotic process automation)</li>
<li>renaming, sorting, deleting, or moving hundreds of files at once (using tools like <a href="https://realpython.com/python-pathlib/">pathlib</a>, <a href="https://github.com/gorakhargosh/watchdog">watchdog</a>, and <a href="https://docs.python.org/3/library/shutil.html">shutil</a>)</li>
<li>Web-scraping, of course</li>
<li><a href="https://schedule.readthedocs.io/en/stable/">Scheduling tasks</a> to happen while your computer is running</li>
</ul>
<p>A quick example of the latter, scheduling, is:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb8" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb8-1"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> schedule</span>
<span id="cb8-2"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> time</span>
<span id="cb8-3"></span>
<span id="cb8-4"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> morning_task():</span>
<span id="cb8-5">    <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">print</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Good morning! Running your daily automation task..."</span>)</span>
<span id="cb8-6"></span>
<span id="cb8-7">schedule.every().day.at(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"08:00"</span>).do(morning_task)</span>
<span id="cb8-8"></span>
<span id="cb8-9"><span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">while</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">True</span>:</span>
<span id="cb8-10">    schedule.run_pending()</span>
<span id="cb8-11">    time.sleep(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">60</span>)</span></code></pre></div></div>
<p>There are really incredible possibilities in this space.</p>
</section>
<section id="low-or-no-code-automation" class="level3">
<h3 class="anchored" data-anchor-id="low-or-no-code-automation">Low or no code automation</h3>
<p>There’s a huge barrier to automating with code, which is that most people in most organisations can’t code. Overall, for the reasons listed, code-based automation is to be preferred. However, we should not let the perfect be the enemy of the good and <strong>proprietary low or no code tools can democratise the ability to automate tasks</strong>. The most relevant example of such a tool in the public sector is Microsoft’s <a href="https://www.microsoft.com/en-gb/power-platform/products/power-automate">Power Automate</a>, which is described as “A comprehensive, end-to-end cloud automation platform powered by low code and AI.”</p>
<p>Power Automate is not free, but if your institution uses Microsoft then there’s a chance that you have access to it as part of a bundle. Power Automate’s stand out feature is that you can create automated flows simply by pointing and clicking (and sometimes filling in a few details.) For the automations it covers, it’s extremely convenient. Some examples of what can be automated include:</p>
<ul>
<li>Sending a customised email when a new file is added to a folder on Sharepoint</li>
<li>Scheduling a recurring message in a Teams chat</li>
<li>Saving an email to OneNote</li>
</ul>
<p>There are also ‘connectors’ that provide integrations with popular other services and software, including GitHub, Gmail, Slack, etc.</p>
<p>If you have the relevant Microsoft subscription, you can find example templates and connectors <a href="https://www.microsoft.com/en-us/power-platform/templates">here</a>.</p>
</section>
</section>
<section id="conclusion" class="level2">
<h2 class="anchored" data-anchor-id="conclusion">Conclusion</h2>
<p>In this post, I’ve explored a range of ideas to boost efficiency in public sector analysis and operations. Investing in better technology, embracing open-source software, data sharing, and automating analytical processes has the potential to yield significant productivity improvements. The same goes for implementing reproducible analytical pipelines, maintaining comprehensive technical documentation, versioning models, strategically integrating AI tools, and automating routine office tasks.</p>
<p>These practices are likely to boost productivity by being, in some cases, cheaper. But the likely biggest contribution to productivity is in reducing errors, saving time, enabling more analysis, and increasing agility.</p>


</section>


 ]]></description>
  <category>productivity</category>
  <category>public sector</category>
  <category>reproducibility</category>
  <guid>https://www.aeturrell.com/blog/posts/public-sector-analytical-efficiency/</guid>
  <pubDate>Tue, 18 Mar 2025 00:00:00 GMT</pubDate>
  <media:content url="https://www.aeturrell.com/blog/posts/public-sector-analytical-efficiency/automated_stats.webp" medium="image" type="image/webp"/>
</item>
<item>
  <title>Efficiency in the public sector: communication and co-ordination</title>
  <dc:creator>Arthur Turrell</dc:creator>
  <link>https://www.aeturrell.com/blog/posts/public-sector-communication-efficiency/</link>
  <description><![CDATA[ 





<p>I’ve been thinking a lot about efficiency in the public sector recently, particularly how we can improve it. In this post, I’ll focus on some ideas for improving communication and co-ordination between public sector workers.</p>
<p>Quick disclaimer: efficiency, in time or money, is not the only important factor. People’s well-being and satisfaction matter too. (It matters in itself but it’s also true that grumpy staff are not going to be well-motivated.) Most of the time there isn’t going to be a trade-off between efficiency and staff well-being as no-one likes wasting time but it’s always worth considering what the pursuit of efficiency in time and money might be costing elsewhere.</p>
<p><img src="https://www.aeturrell.com/blog/posts/public-sector-communication-efficiency/iso_city.webp" class="img-fluid"></p>
<section id="ideas-for-lowering-communication-and-co-ordination-costs" class="level2 page-columns page-full">
<h2 class="anchored" data-anchor-id="ideas-for-lowering-communication-and-co-ordination-costs">Ideas for lowering communication and co-ordination costs</h2>
<section id="publishing-analysis" class="level3 page-columns page-full">
<h3 class="anchored" data-anchor-id="publishing-analysis">Publishing analysis</h3>
<p>A major source of inefficiency across the wider public sector is the co-ordination costs that result from having everyone involved in topic X try to keep in touch.<sup>1</sup> Across the whole public sector, there are often too many people working on X to get into a single room at the same time. So people resort to many bilateral conversations. With only <img src="https://latex.codecogs.com/png.latex?N"> people working on X across the whole of the public sector, one individual needs to have <img src="https://latex.codecogs.com/png.latex?N-1"> conversations to keep on top of what’s happening and, in total, if everyone does the same, there are <img src="https://latex.codecogs.com/png.latex?%5Cfrac%7BN%5E2-N%7D%7B2%7D"> co-ordination meetings happening. For as few as <img src="https://latex.codecogs.com/png.latex?N=30">, that means 435 meetings! I’m using an extreme case—surely <em>some</em> sub-groups of more than two people will meet. But the point stands that this is not at all efficient.</p>
<div class="no-row-height column-margin column-container"><div id="fn1"><p><sup>1</sup>&nbsp;In a perfect world, you would have very clean lines of responsibility for different topics. But topics that can be so cleanly separated from any other issue are the exception, if they exist at all.</p></div></div><p>An alternative is for public sector institutions to share as much as possible of their analysis across institutional lines, preferably by publishing it openly. You might think sharing it just within the public sector makes more sense but there are legal, technological, and co-ordination barriers to do this—again, you might end up having to do 100s of bilateral sharing agreements.</p>
<p>Of course there will be analysis that cannot be shared. Some questions, and definitely some answers, are inherently sensitive or disclosive. But not as many as most people in the public sector assume (I assert), and for those that aren’t, the benefits of release are substantial.</p>
<p>By the way, by ‘sharing analysis’, I don’t just mean sharing the end report, I mean sharing the data and code too.<sup>2</sup> This gets all the benefits that will be familiar to people who know about free and open source code, and data sharing<sup>3</sup>. Let me give you a specific example: over the last few years there have been a number of errors in UK official statistics, on <a href="https://www.ft.com/content/61679d49-2b87-45ef-8fa2-1b76e393aed4">trade</a> and on <a href="https://www.ft.com/content/95b425c1-7f09-3c14-b423-42da8fa30bfe">productivity</a>, and these are just two examples from the top of my head. I have this dream that the code that is used to create national statistics could be open-sourced, improving transparency, getting more eyes on fixing any errors, and allowing countries to adopt the best practices of the most innovation national statistical offices far more quickly. So the biggest benefit of all of this sharing is that you can co-opt others into improving your analysis! That’s very efficient!</p>
<div class="no-row-height column-margin column-container"><div id="fn2"><p><sup>2</sup>&nbsp;One organisation, GitLab, has even <a href="https://about.gitlab.com/blog/our-handbook-is-open-source-heres-why/">open sourced</a> how they run their organisation.</p></div><div id="fn3"><p><sup>3</sup>&nbsp;I’ll write more about the benefits of data sharing in another post.</p></div></div><p>So, where possible, I believe it makes sense to publish analysis for anyone to find. Then everyone who needs it can find it, benefit from it, and avoid duplicating it. This isn’t just a public good: the people who are able to find it and avoid duplicating it may well be people from your own institution in the future—I’ve <a href="../../../blog/posts/managing-public-sector-knowledge/managing-public-sector-knowledge.html">written before</a> about how the public sector is not good at maintaining the stock of knowledge, but making that knowledge accessible to the most powerful search engines in the world means it is far less likely to disappear or be inefficiently duplicated.</p>
<p>This share-what-you-can approach eliminates the need for lots of communication and co-ordination that would otherwise be costly.</p>
</section>
<section id="asynchronous-working" class="level3 page-columns page-full">
<h3 class="anchored" data-anchor-id="asynchronous-working">Asynchronous working</h3>
<p>As well as disseminating analysis and code widely, you can reduce within-org communication costs by opting for asynchronous ways of working. Those who have worked in organisations that span time zones will be familiar with this idea. GitLab has lots of <a href="https://handbook.gitlab.com/handbook/company/culture/all-remote/asynchronous/">good content on the idea of asynchronous working</a> as they are a 100% remote and international company.</p>
<p>Here’s an excerpt from their Handbook, itself a quote by <a href="https://codahale.com/">Coda Hale</a>:</p>
<div class="page-columns page-full"><blockquote class="blockquote">
<p>A significant source of failure demand<sup>4</sup> for meetings and status updates is the desire of organizational leaders to keep abreast of who’s doing what. This situational awareness is indeed important, but trying to maintain it by calling meetings, messaging people on Slack, and catching people on the hallways is a significant systemic drag on organizational productivity.</p>
<p>A better model for staying informed of developments as the organization scales is for groups to publish status updates as part of the regular cadence of their work. Leaders can asynchronously read these updates and, should the need arise, initiate additional, synchronous conversation to ask questions, provide feedback, etc.</p>
<p>Synchronous meetings should be reserved for low-latency collaboration on complex issues; likewise, collaboration should be reserved for synchronous meetings.</p>
</blockquote><div class="no-row-height column-margin column-container"><div id="fn4"><p><sup>4</sup>&nbsp;Failure demand is the demand on a system or organisation that arises not from delivering work, but from failings of the system and excess co-ordination costs.</p></div></div></div>
<p>The fundamental idea here is that co-ordinating to be in the same meeting at the same time is more difficult, and may create bottlenecks, that are not present when communication is asynchronous. Another aspect is that there’s a “default to action” rather than a default to delay until a conversation can happen.</p>
<p>Asynchronous communication appeals to me because of my observation that the public sector has more meetings than perhaps would be ideal—and that as the ‘cost’ of convening large groups of people decreased due to the availability of virtual meetings in the pandemic, we all opted to ‘buy’ more of them (without an accompanying change in the volume of meetings the work actually requires).</p>
<p>As the GitLab Handbook notes, there are other benefits to asynchronous working:</p>
<ul>
<li>it is better for people who work part time or have unusual working hours</li>
<li>it tends to push people toward greater transparency, which can help junior staff feel more involved</li>
<li>it tends to push people to write down key decisions and actions so that a log of institutional knowledge is created (useful in general, but especially great for onboarding new staff.)</li>
</ul>
<p>So how can you actually get some of the productivity benefits of asynchronicity? A big part is mindset and (again, this line courtesy of the GitLab Handbook), the easiest way to shift into an asynchronous mindset is to ask: “How would I deliver this message, present this work, or move this project forward right now if no one else on my team or organisation were awake?” Here are some more specific ideas for implementing asynchronous working:</p>
<ul>
<li><strong>Default to sharing information</strong>, eg by using the most permissive classification available.</li>
<li><strong>Use documentation obsessively</strong>—for code, for decisions, for processes, and for actions. And make sure there is a process by which that documentation is prevented from becoming stale.<sup>5</sup> Documenting why something didn’t happen is as important as documenting why it did because it helps others understand why certain decisions were made. In practice, documenting obsessively means avoiding putting key decisions or information in emails—anything important should be in documents that are findable by default. If you’re in any kind of managerial position, you’ll know that your bandwidth can be entirely absorbed simply by your email inbox—leaving no time for deeper thought. Amazon has an interesting ban on Powerpoint decks in favour of written memos that goes in this direction; there’s a <a href="../../../blog/posts/managing-public-sector-knowledge/managing-public-sector-knowledge.html">post that mentions Amazon’s approach here</a>. As a concrete example, in my own teams, I have had some success managing code-based work via a <a href="https://en.wikipedia.org/wiki/Kanban_board">Kanban board</a> (we use <a href="https://docs.github.com/en/issues/planning-and-tracking-with-projects">GitHub projects</a> but there are lots of alternatives available.) This provides asynchronicity, transparency, it means that team meetings aren’t just a round table of updates, and (using GitHub’s <a href="https://docs.github.com/en/graphql">GraphQL API</a>) it also allows for automatic compilation of reports on what kinds of projects we’re undertaking.</li>
<li><strong>Have meetings only when they are needed</strong>. This means avoiding scheduling them regularly and holding ad hoc ones only when it would unblock progress faster than written communication. Good reasons for meetings might include:
<ul>
<li>Meetings with external parties</li>
<li>First-time meetings for team members who have not previously worked together</li>
<li>‘One-way door’ decisions (eg when stakes are high and decisions are difficult to reverse)</li>
<li>Complex project initialisations and planning (eg defining the workplan for the next quarter, scoping a major project, etc.)</li>
<li>Sensitive topics (e.g.&nbsp;discussing personal issues, career, performance, giving difficult feedback, etc.)</li>
<li>Supporting and unblocking direct reports</li>
</ul></li>
<li><strong>Meetings should have an agenda</strong>.</li>
<li><strong>Empower people to make ‘two-way door’ decisions</strong>. Most decisions are easy to reverse and not sensitive. In these cases, people should go ahead and make them without approval or needing to have a meeting.</li>
<li>Ensure there is a clear <strong>single source of truth</strong> for every important piece of information. If you have to ask someone else which of two similar pieces of information is relevant, that’s not very efficient.</li>
<li><strong>Ensure that information is easily accessible and can be searched</strong>. I wrote more about that last point <a href="../../../blog/posts/managing-public-sector-knowledge/managing-public-sector-knowledge.html">here</a> but now there’s a new kid on the block for helping to interrogate the combined stock of knowledge of an institution: retrieval augmented generation (RAG). With RAG, you can get the general benefits of a large language model focused on a specific set of documents that you choose. Even better, the LLM can cite the documents it is sourcing its answers from. This seems like a fantastic way to make the considerable stock of knowledge in most institutions more accessible.</li>
<li>When meetings are necessary, record them, or make a transcript, or better than both <strong>create a concise but complete summary of meetings in which it is clear who attended and what decisions were made, and ensure that the information is available for anyone to find afterwards</strong>. Most public sector institutions are using Microsoft Teams, and there are Teams tools that can help with this.</li>
</ul>
<div class="no-row-height column-margin column-container"><div id="fn5"><p><sup>5</sup>&nbsp;More on code documentation in a subsequent post.</p></div></div></section>
<section id="have-someone-own-the-trade-off" class="level3 page-columns page-full">
<h3 class="anchored" data-anchor-id="have-someone-own-the-trade-off">Have someone own the trade-off</h3>
<p>This one is more about the structure of your organisation, but it’s a change that saves a ton of potentially fractious communication too.</p>
<p>It’s probably best to illustrate the idea with a hypothetical example. Let’s say that there is a person in an organisation who is responsible for IT, including software, hardware, and technology security (call them Bob.) And let’s say there’s someone else who is responsible for high quality, efficient, and impactful analysis across the same organisation (call them Alice.) Alice and Bob are far removed from each other—perhaps they only have a single manager in common who is the head of the entire institution or the permanent secretary.</p>
<p>Now Bob is incentivised to minimise costs and maximise security, because he holds the budget for software and hardware, and if there’s an IT security incident, you can bet Bob is going to be in the firing line. Alice, meanwhile, is incentivised to get things done, and likely has substantial risk appetite in order to get important things done. If there’s a major crisis of some sort looming, Alice isn’t going to be much pleased that her staff are unable to access key government websites because of zealous internet security settings, and if she doesn’t get that critical analysis out in time, <em>she’s</em> going to be in the firing line.</p>
<p>Hopefully you can see that Alice and Bob have wildly different incentives here and their communications might, very naturally, end up being fractious. We can’t blame Bob or Alice for this—they’re both doing their jobs, the jobs they have been given. But at an institutional level, the structure has created a natural clash. Co-ordination and communication will be far less good than they could be because Alice is always straining at the limits of what Bob allows her to do, and Bob is always trying to reign Alice in.</p>
<p>The solution here is to have someone who owns this productivity-security trade-off<sup>6</sup>. There are a few options for making this happen:</p>
<div class="no-row-height column-margin column-container"><div id="fn6"><p><sup>6</sup>&nbsp;As an aside, I’m sceptical there’s as much of a productivity-security trade-off as most people think. If you have too high security, people start to find risky work arounds, which isn’t very secure. If you have very lax security and processes, mistakes can go unnoticed (including analytical mistakes). But there are definitely times when the needs of “getting things done” and “avoiding any risk” clash.</p></div></div><ul>
<li>have someone who doesn’t sit so far up the organisation own the trade-off. Maybe it could be that Alice and Bob report to the same person. This helps because, if the person invested with the trade-off decision is too high up in the organisation, then that decision is never going to be top of their list—they will simply always have bigger fish to fry. Whereas if the senior person’s role is explicitly to get that trade-off right, and they’re close to both Bob and Alice, they’re going to be well-placed to make good decisions.</li>
<li>merge Alice and Bob’s jobs. Alice-Bob now owns risk tolerance on IT security, hardware and software choices, and productivity of analysis. Alice-Bob can make the trade-off directly.</li>
<li>Bob’s job disappears, and Bob’s staff become Alice’s staff, but Alice now has more diverse teams that include technology experts as well as analysts. There’s a professional network for technology people, but they are embedded in business areas. Alice owns the trade-off and can make the decision based on what her mixed teams tell her. Some US institutions use this model.</li>
</ul>
<p>Stepping away from the Alice and Bob hypothetical, you can imagine other places where this same dynamic plays out. The choice of where to site a new office? The choice of how to refurbish an office? How complex the HR policies are for bringing new people in? In every case, if your goal is productivity, you need someone to own the trade-off with the other factors at stake.</p>
</section>
<section id="project-management" class="level3">
<h3 class="anchored" data-anchor-id="project-management">Project management</h3>
<p>I already mentioned Kanban boards in the previous section but there’s a bunch of other ideas that can help reduce communication and co-ordination costs within the nitty gritty of individual projects. One is the Amazon ‘working backwards’ method, which you can find more about in <a href="../../../blog/posts/data-science-with-impact/index.html#working-backwards">this post on data science with impact</a>. The same post also talks about the benefits of incremental delivery.</p>
<p>On communication and co-ordination specifically, though, another Amazon-favoured trick is worth mentioning: “disagree and commit”. The basic idea is that individuals can, and should, openly disagree with a decision that is on the table but—once the decision is definitively made—everyone should commit to implementing it. This allows people to speak up and challenge what they don’t agree with, without causing a fear that they will be obstructive. And it also helps avoid what’s known as the ‘consensus trap’, where the feeling of needing to reach consensus turns into a lack of progress. Both of these can be issues in the public sector or, indeed, any large organisation.</p>
</section>
</section>
<section id="conclusion" class="level2">
<h2 class="anchored" data-anchor-id="conclusion">Conclusion</h2>
<p>It seems like there’s plenty of room to improve communication and co-ordination in the public sector. I’ve shared a few suggestions focused on four areas: making your analysis public by default avoids costly bilateral co-ordination, gets you free help from others, and helps avoid duplication; working asynchronously avoids bottlenecks, empowers staff, lowers onboarding costs, and promotes transparency; thinking carefully about who owns key trade-offs can prevent a lot of inefficient back-and-forth, and ensure that everyone is pulling in the same direction; and battle-tested project management practices can promote delivery and action. Got better ideas? Let me know!</p>


</section>


 ]]></description>
  <category>productivity</category>
  <category>public sector</category>
  <guid>https://www.aeturrell.com/blog/posts/public-sector-communication-efficiency/</guid>
  <pubDate>Mon, 17 Mar 2025 00:00:00 GMT</pubDate>
  <media:content url="https://www.aeturrell.com/blog/posts/public-sector-communication-efficiency/iso_city.webp" medium="image" type="image/webp"/>
</item>
<item>
  <title>The data validation landscape in 2025</title>
  <dc:creator>Arthur Turrell</dc:creator>
  <link>https://www.aeturrell.com/blog/posts/the-data-validation-landscape-in-2025/</link>
  <description><![CDATA[ 





<p>What’s going on in the world of data validation? For those of you who don’t know, data validation is the process of checking data quality in an automated or semi-automated way—for example, checking datatypes, checking the number of missing values, and detecting whether there are anomalous numbers. It doesn’t have to be rows in a dataframe though, it could be for validating API input or form submissions. The user provides rules for what should be flagged as an issue, for example saying that all values in a particular column should be within a certain number range. And, depending on the package, data that fail the validation either result in an outright error or in a data validation report that determines (either automatically or manually) what should happen next.</p>
<p>Note: while most of what I say in this post will be relevant in general to data validation tools, I’ll be making a few observations of how this might be useful or not for the public sector specifically. And this is not going to be a comprehensive list—it’s just going to feature some of the more widely-used packages.</p>
<section id="why-is-data-validation-useful" class="level2">
<h2 class="anchored" data-anchor-id="why-is-data-validation-useful">Why is data validation useful?</h2>
<p>In my experience, there are broadly two types of analytical work that public sector institutions undertake. One is ad hoc analysis, the other is regular production of statistics. In the latter case, data validation is extremely helpful because you want to know whether there are problems with, say, the latest data you’ve ingested <em>before</em> it goes to senior leaders or, even worse, is published externally. But even for ad hoc data analysis, if it’s on a standard dataset that is ingested, say, every month, you probably do want to have data validation checks so you’re not caught out by an anomaly that you misinterpret as a real effect.</p>
<p>Ultimately, if you want the analysis you’re doing to be accurate, then data validation is a great way to efficiently remove some of the risks of making a mistake.</p>
</section>
<section id="data-validation-tooling-landscape" class="level2 page-columns page-full">
<h2 class="anchored" data-anchor-id="data-validation-tooling-landscape">Data validation tooling landscape</h2>
<section id="great-expectations" class="level3 page-columns page-full">
<h3 class="anchored" data-anchor-id="great-expectations">Great Expectations</h3>
<p>I think the first data validation tool I came across was <a href="https://greatexpectations.io/">Great Expectations</a>. The landscape has evolved and now there are many other tools; meanwhile Great Expectations has developed into a heavy-duty production-grade data validation tool. The website pushes you quite hard in the direction of their hosted solution, but there is an open source package underlying it all that anyone can use.<sup>1</sup> Perhaps because it has become production-grade, it now seems a little more difficult to configure and get going with than some of the other options on this list, and perhaps less well-suited to teams where not everyone has advanced data science skill (this is common in the public sector; eg teams composed of economists with one data scientist.)</p>
<div class="no-row-height column-margin column-container"><div id="fn1"><p><sup>1</sup>&nbsp;For example, you have to click through a few menus on their website to find their original open source package as they are understandably keen to get you to use their hosted GX Cloud solution.</p></div><div id="fn2"><p><sup>2</sup>&nbsp;Hopefully one could rig up an action to send an alert on a tool that is more likely to be in use in the public sector too.</p></div></div><p>It’s well worth checking out in any case, though, as it has a number of advanced features that other packages on this list don’t have (at least not without significant further effort). For example, if validations fail, you can use nifty integrations to do things like send a message on Slack.<sup>2</sup></p>
<p>If you do use the open source element of Great Expectations, it works a bit like this:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb1" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb1-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Import required modules from GX library.</span></span>
<span id="cb1-2"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> great_expectations <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">as</span> gx</span>
<span id="cb1-3"></span>
<span id="cb1-4"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> pandas <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">as</span> pd</span>
<span id="cb1-5"></span>
<span id="cb1-6"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Create Data Context.</span></span>
<span id="cb1-7">context <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> gx.get_context()</span>
<span id="cb1-8"></span>
<span id="cb1-9"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Import sample data into Pandas DataFrame.</span></span>
<span id="cb1-10">df <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> pd.read_csv(</span>
<span id="cb1-11">    <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"https://raw.githubusercontent.com/great-expectations/gx_tutorials/main/data/yellow_tripdata_sample_2019-01.csv"</span></span>
<span id="cb1-12">)</span>
<span id="cb1-13"></span>
<span id="cb1-14"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Connect to data.</span></span>
<span id="cb1-15"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Create Data Source, Data Asset, Batch Definition, and Batch.</span></span>
<span id="cb1-16">data_source <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> context.data_sources.add_pandas(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"pandas"</span>)</span>
<span id="cb1-17">data_asset <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> data_source.add_dataframe_asset(name<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"pd dataframe asset"</span>)</span>
<span id="cb1-18"></span>
<span id="cb1-19">batch_definition <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> data_asset.add_batch_definition_whole_dataframe(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"batch definition"</span>)</span>
<span id="cb1-20">batch <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> batch_definition.get_batch(batch_parameters<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>{<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"dataframe"</span>: df})</span>
<span id="cb1-21"></span>
<span id="cb1-22"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Create Expectation.</span></span>
<span id="cb1-23">expectation <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> gx.expectations.ExpectColumnValuesToBeBetween(</span>
<span id="cb1-24">    column<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"passenger_count"</span>, min_value<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>, max_value<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">6</span></span>
<span id="cb1-25">)</span>
<span id="cb1-26"></span>
<span id="cb1-27"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Validate Batch using Expectation.</span></span>
<span id="cb1-28">validation_result <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> batch.validate(expectation)</span></code></pre></div></div>
<p>The returned Validation Result object is a json representation of the results. There are numerous options for data sources including, importantly, databases. And there are numerous expectations too—and an ability to define custom ones.</p>
<p>I mentioned the actions that can be triggered when a validation fails. Here’s a couple and how they can be used:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb2" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb2-1">action_list <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> [</span>
<span id="cb2-2">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># This Action sends a Slack Notification if an Expectation fails.</span></span>
<span id="cb2-3">    SlackNotificationAction(</span>
<span id="cb2-4">        name<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"send_slack_notification_on_failed_expectations"</span>,</span>
<span id="cb2-5">        slack_token<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"$</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{validation_notification_slack_webhook}</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span>,</span>
<span id="cb2-6">        slack_channel<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"$</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{validation_notification_slack_channel}</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span>,</span>
<span id="cb2-7">        notify_on<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"failure"</span>,</span>
<span id="cb2-8">        show_failed_expectations<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">True</span>,</span>
<span id="cb2-9">    ),</span>
<span id="cb2-10">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># This Action updates the Data Docs static website with the Validation</span></span>
<span id="cb2-11">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#   Results after the Checkpoint is run.</span></span>
<span id="cb2-12">    UpdateDataDocsAction(</span>
<span id="cb2-13">        name<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"update_all_data_docs"</span>,</span>
<span id="cb2-14">    ),</span>
<span id="cb2-15">]</span>
<span id="cb2-16"></span>
<span id="cb2-17">checkpoint_name <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"my_checkpoint"</span></span>
<span id="cb2-18">checkpoint <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> gx.Checkpoint(</span>
<span id="cb2-19">    name<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>checkpoint_name,</span>
<span id="cb2-20">    validation_definitions<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>validation_definitions,</span>
<span id="cb2-21">    actions<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>action_list,</span>
<span id="cb2-22">    result_format<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>{<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"result_format"</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"COMPLETE"</span>},</span>
<span id="cb2-23">)</span>
<span id="cb2-24">context.checkpoints.add(checkpoint)</span></code></pre></div></div>
</section>
<section id="pointblank" class="level3 page-columns page-full">
<h3 class="anchored" data-anchor-id="pointblank">Pointblank</h3>
<p><a href="https://posit-dev.github.io/pointblank/">Pointblank</a> is the new kid on the block, and another recent Python creation from the org previously known as RStudio, now Posit. Posit has always created really high quality R packages (including one of the same name for R) and it’s wonderful to see them now focusing their attention on creating equally high quality Python packages. The syntax is somewhat reminiscent of Great Expectations, which shows you just how influential that original package has been.</p>
<p>Here’s an example of using Pointblank with a <strong>Polars</strong> dataframe (it also works with <strong>pandas</strong>):</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb3" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb3-1"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> pointblank <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">as</span> pb</span>
<span id="cb3-2"></span>
<span id="cb3-3">validation <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> (</span>
<span id="cb3-4">    pb.Validate(data<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>pb.load_dataset(dataset<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"small_table"</span>)) <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Use Validate() to start</span></span>
<span id="cb3-5">    .col_vals_gt(columns<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"d"</span>, value<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">100</span>)       <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># STEP 1      |</span></span>
<span id="cb3-6">    .col_vals_le(columns<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"c"</span>, value<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">5</span>)         <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># STEP 2      | &lt;-- Build up a validation plan</span></span>
<span id="cb3-7">    .col_exists(columns<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>[<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"date"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"date_time"</span>]) <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># STEPS 3 &amp; 4 |</span></span>
<span id="cb3-8">    .interrogate() <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># This will execute all validation steps and collect intel</span></span>
<span id="cb3-9">)</span>
<span id="cb3-10"></span>
<span id="cb3-11">validation</span></code></pre></div></div>
<div class="quarto-figure quarto-figure-center page-columns page-full">
<figure class="figure page-columns page-full">
<p><img src="https://www.aeturrell.com/blog/posts/the-data-validation-landscape-in-2025/pointblank-tabular-report.png" class="img-fluid figure-img"></p>
<figcaption class="margin-caption">Example output from Pointblank</figcaption>
</figure>
</div>
<p>From the documentation:</p>
<blockquote class="blockquote">
<p>The rows in the validation report table correspond to each of the validation steps. One of the key concepts is that validation steps can be broken down into atomic test cases (test units), where each of these test units is given either of pass/fail status based on the validation constraints. You’ll see these tallied up in the reporting table (in the UNITS, PASS, and FAIL columns).</p>
</blockquote>
<p>Again, a wide range of data sources are supported, including Polars, Pandas, DuckDB tables, MySQL tables, PostgreSQL tables, SQLite tables, and Parquet.</p>
<p>It’s early days for this package—it only had its first release in 2024, compared to (I think) 2017 for Great Expectations, and some functionality you’d expect isn’t quite there yet, but it looks super user-friendly so far and perhaps geared a bit more to the individual institutional user.</p>
<p>One important caveat with Pointblank is that it doesn’t have the action triggers for follow-ups to failed validations. So you may validate your data, but then you either need to code in a next action yourself, or take action manually.</p>
</section>
<section id="pandera" class="level3">
<h3 class="anchored" data-anchor-id="pandera">Pandera</h3>
<p><a href="https://pandera.readthedocs.io/">Pandera</a> also follows an API similar to that of Great Expectations! Here’s an example:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb4" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb4-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># data to validate</span></span>
<span id="cb4-2">df <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> pd.DataFrame({</span>
<span id="cb4-3">    <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"column1"</span>: [<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">4</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">10</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">9</span>],</span>
<span id="cb4-4">    <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"column2"</span>: [<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1.3</span>, <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1.4</span>, <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">2.9</span>, <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">10.1</span>, <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">20.4</span>],</span>
<span id="cb4-5">    <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"column3"</span>: [<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"value_1"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"value_2"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"value_3"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"value_2"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"value_1"</span>],</span>
<span id="cb4-6">})</span>
<span id="cb4-7"></span>
<span id="cb4-8"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># define schema</span></span>
<span id="cb4-9">schema <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> pa.DataFrameSchema({</span>
<span id="cb4-10">    <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"column1"</span>: pa.Column(<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">int</span>, checks<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>pa.Check.le(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">10</span>)),</span>
<span id="cb4-11">    <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"column2"</span>: pa.Column(<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">float</span>, checks<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>pa.Check.lt(<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1.2</span>)),</span>
<span id="cb4-12">    <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"column3"</span>: pa.Column(<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">str</span>, checks<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>[</span>
<span id="cb4-13">        pa.Check.str_startswith(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"value_"</span>),</span>
<span id="cb4-14">        <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># define custom checks as functions that take a series as input and</span></span>
<span id="cb4-15">        <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># outputs a boolean or boolean Series</span></span>
<span id="cb4-16">        pa.Check(<span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">lambda</span> s: s.<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">str</span>.split(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"_"</span>, expand<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">True</span>).shape[<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>] <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>)</span>
<span id="cb4-17">    ]),</span>
<span id="cb4-18">})</span>
<span id="cb4-19"></span>
<span id="cb4-20">validated_df <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> schema(df)</span>
<span id="cb4-21"><span class="bu" style="color: null;
background-color: null;
font-style: inherit;">print</span>(validated_df)</span></code></pre></div></div>
<pre class="text"><code>   column1  column2  column3
0        1     -1.3  value_1
1        4     -1.4  value_2
2        0     -2.9  value_3
3       10    -10.1  value_2
4        9    -20.4  value_1</code></pre>
<p>If the validation doesn’t go through, an error is thrown (in this case it passed through.)</p>
<p>One of the nice features of Pandera is that you can combine it not just with column-level validation questions but also with statistical hypothesis testing too. It might seem a bit niche but I can well imagine cases where this could pop up. Here’s an example,</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb6" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb6-1"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> pandas <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">as</span> pd</span>
<span id="cb6-2"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> pandera <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">as</span> pa</span>
<span id="cb6-3"></span>
<span id="cb6-4"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> pandera <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> Column, DataFrameSchema, Check, Hypothesis</span>
<span id="cb6-5"></span>
<span id="cb6-6"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> scipy <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> stats</span>
<span id="cb6-7"></span>
<span id="cb6-8">df <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> (</span>
<span id="cb6-9">    pd.DataFrame({</span>
<span id="cb6-10">        <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"height_in_feet"</span>: [<span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">6.5</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">7</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">6.1</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">5.1</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">4</span>],</span>
<span id="cb6-11">        <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"sex"</span>: [<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"M"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"M"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"F"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"F"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"F"</span>]</span>
<span id="cb6-12">    })</span>
<span id="cb6-13">)</span>
<span id="cb6-14"></span>
<span id="cb6-15">schema <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> DataFrameSchema({</span>
<span id="cb6-16">    <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"height_in_feet"</span>: Column(</span>
<span id="cb6-17">        <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">float</span>, [</span>
<span id="cb6-18">            Hypothesis.two_sample_ttest(</span>
<span id="cb6-19">                sample1<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"M"</span>,</span>
<span id="cb6-20">                sample2<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"F"</span>,</span>
<span id="cb6-21">                groupby<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"sex"</span>,</span>
<span id="cb6-22">                relationship<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"greater_than"</span>,</span>
<span id="cb6-23">                alpha<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.05</span>,</span>
<span id="cb6-24">                equal_var<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">True</span>),</span>
<span id="cb6-25">    ]),</span>
<span id="cb6-26">    <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"sex"</span>: Column(<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">str</span>)</span>
<span id="cb6-27">})</span>
<span id="cb6-28"></span>
<span id="cb6-29"><span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">try</span>:</span>
<span id="cb6-30">    schema.validate(df)</span>
<span id="cb6-31"><span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">except</span> pa.errors.SchemaError <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">as</span> exc:</span>
<span id="cb6-32">    <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">print</span>(exc)</span></code></pre></div></div>
<pre class="text"><code>Column 'height_in_feet' failed series or dataframe validator 0: &lt;Check two_sample_ttest: failed two sample ttest between 'M' and 'F'&gt;</code></pre>
<p>Just as with the other libraries we’ve seen so far, a wide range of data sources are supported, including (despite the name!) Polars, geopandas, Pyspark, dark, and modin dataframes.</p>
</section>
<section id="pydantic" class="level3">
<h3 class="anchored" data-anchor-id="pydantic">Pydantic</h3>
<p><a href="https://docs.pydantic.dev/">Pydantic</a> is a bit different from the previous examples we’ve seen as the fundamental unit to which it applies validation is not a dataframe, but a dictionary (eg from ingesting a JSON file). It’s more in the business of <em>schema validation</em> than dataframe validation. So this is less useful out-of-the-box for dataframes but more flexible and can be used for non-tabular data. The clever bit is that you define your schema via a class, and then you get a lot of desirable behaviour ‘for free’ that comes from this. On the website, it lists the UK Home Office among its users. It’s fair to say that Pydantic is production-grade too.</p>
<p>It’s easiest to illustrate how it works with an example:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb8" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb8-1"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> datetime <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> datetime</span>
<span id="cb8-2"></span>
<span id="cb8-3"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> pydantic <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> BaseModel, PositiveInt</span>
<span id="cb8-4"></span>
<span id="cb8-5"></span>
<span id="cb8-6"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">class</span> User(BaseModel):</span>
<span id="cb8-7">    <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">id</span>: <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">int</span>  </span>
<span id="cb8-8">    name: <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">str</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'John Doe'</span>  </span>
<span id="cb8-9">    signup_ts: datetime <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">|</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">None</span>  </span>
<span id="cb8-10">    tastes: <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">dict</span>[<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">str</span>, PositiveInt]  </span>
<span id="cb8-11"></span>
<span id="cb8-12"></span>
<span id="cb8-13">external_data <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> {</span>
<span id="cb8-14">    <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'id'</span>: <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">123</span>,</span>
<span id="cb8-15">    <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'signup_ts'</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'2019-06-01 12:22'</span>,  </span>
<span id="cb8-16">    <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'tastes'</span>: {</span>
<span id="cb8-17">        <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'wine'</span>: <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">9</span>,</span>
<span id="cb8-18">        <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">b'cheese'</span>: <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">7</span>,</span>
<span id="cb8-19">        <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'cabbage'</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'1'</span>,</span>
<span id="cb8-20">    },</span>
<span id="cb8-21">}</span>
<span id="cb8-22"></span>
<span id="cb8-23">user <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> User(<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">**</span>external_data)  </span>
<span id="cb8-24"></span>
<span id="cb8-25"><span class="bu" style="color: null;
background-color: null;
font-style: inherit;">print</span>(user.<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">id</span>)  </span>
<span id="cb8-26"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#&gt; 123</span></span>
<span id="cb8-27"><span class="bu" style="color: null;
background-color: null;
font-style: inherit;">print</span>(user.model_dump())  </span>
<span id="cb8-28"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""</span></span>
<span id="cb8-29"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span></span>
<span id="cb8-30"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    'id': 123,</span></span>
<span id="cb8-31"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    'name': 'John Doe',</span></span>
<span id="cb8-32"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    'signup_ts': datetime.datetime(2019, 6, 1, 12, 22),</span></span>
<span id="cb8-33"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">    'tastes': {'wine': 9, 'cheese': 7, 'cabbage': 1},</span></span>
<span id="cb8-34"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span></span>
<span id="cb8-35"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">"""</span></span></code></pre></div></div>
<p>If validation fails, Pydantic will raise an error with a breakdown of what was wrong.</p>
<p>As I mentioned, a strength of Pydantic is that you can validate arbitrarily complex objects. So here, for example, is a a fruit class that has an attribute called <code>bazam</code> that is a dictionary that maps strings into lists of tuples of ints, bools, and floats.</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb9" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb9-1"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> annotated_types <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> Gt</span>
<span id="cb9-2"></span>
<span id="cb9-3"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> pydantic <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> BaseModel</span>
<span id="cb9-4"></span>
<span id="cb9-5"></span>
<span id="cb9-6"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">class</span> Fruit(BaseModel):</span>
<span id="cb9-7">    name: <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">str</span>  </span>
<span id="cb9-8">    color: Literal[<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'red'</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'green'</span>]  </span>
<span id="cb9-9">    weight: Annotated[<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">float</span>, Gt(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span>)]  </span>
<span id="cb9-10">    bazam: <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">dict</span>[<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">str</span>, <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">list</span>[<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">tuple</span>[<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">int</span>, <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">bool</span>, <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">float</span>]]]  </span></code></pre></div></div>
<p>You’ll also notice that there’s a validation check for whether the weight is greater than zero <code>Gt(0)</code>.</p>
<p>Pydantic integrates with some other Python tools such as the extremely excellent <a href="https://fastapi.tiangolo.com/">FastAPI</a> for building APIs. But, unlike the other tools we’ve seen, it doesn’t integrate with dataframe and data source tools without further work by the user.</p>
</section>
<section id="cerberus" class="level3">
<h3 class="anchored" data-anchor-id="cerberus">Cerberus</h3>
<p>From the Cerberus docs:</p>
<blockquote class="blockquote">
<p>Cerberus provides powerful yet simple and lightweight data validation functionality out of the box and is designed to be easily extensible, allowing for custom validation.</p>
</blockquote>
<p>In many ways, Cerberus has a similar feel to Pydantic though, as far as I can tell, it’s a bit less fully featured and it returns True or False for validations rather than passing or throwing an error. Instead of using base classes and Python’s typing functionality, it defines schemas using text-based dictionaries (though these can be extended and customised).</p>
<p>It’s easiest to illustrate via an example:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb10" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb10-1">schema <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> {<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'name'</span>: {<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'type'</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'string'</span>}}</span>
<span id="cb10-2">v <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> Validator(schema)</span>
<span id="cb10-3">document <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> {<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'name'</span>: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'john doe'</span>}</span>
<span id="cb10-4">v.validate(document)</span></code></pre></div></div>
<pre class="text"><code>True</code></pre>
</section>
<section id="jsonschema" class="level3">
<h3 class="anchored" data-anchor-id="jsonschema">jsonschema</h3>
<p>This package, <a href="https://github.com/python-jsonschema/jsonschema">jsonschema</a>, is similar to Cerberus covered above and is specifically focused on validating JSON. Here’s an example:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb12" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb12-1"><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&gt;&gt;&gt;</span> <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> jsonschema <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> validate</span>
<span id="cb12-2"></span>
<span id="cb12-3"><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&gt;&gt;&gt;</span> <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># A sample schema, like what we'd get from json.load()</span></span>
<span id="cb12-4"><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&gt;&gt;&gt;</span> schema <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> {</span>
<span id="cb12-5">...     <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"type"</span> : <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"object"</span>,</span>
<span id="cb12-6">...     <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"properties"</span> : {</span>
<span id="cb12-7">...         <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"price"</span> : {<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"type"</span> : <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"number"</span>},</span>
<span id="cb12-8">...         <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"name"</span> : {<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"type"</span> : <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"string"</span>},</span>
<span id="cb12-9">...     },</span>
<span id="cb12-10">... }</span>
<span id="cb12-11"></span>
<span id="cb12-12"><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&gt;&gt;&gt;</span> <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># If no exception is raised by validate(), the instance is valid.</span></span>
<span id="cb12-13"><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&gt;&gt;&gt;</span> validate(instance<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>{<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"name"</span> : <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Eggs"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"price"</span> : <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">34.99</span>}, schema<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>schema)</span>
<span id="cb12-14"></span>
<span id="cb12-15"><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&gt;&gt;&gt;</span> validate(</span>
<span id="cb12-16">...     instance<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>{<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"name"</span> : <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Eggs"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"price"</span> : <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Invalid"</span>}, schema<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>schema,</span>
<span id="cb12-17">... )                                   <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># doctest: +IGNORE_EXCEPTION_DETAIL</span></span>
<span id="cb12-18">Traceback (most recent call last):</span>
<span id="cb12-19">    ...</span>
<span id="cb12-20">ValidationError: <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'Invalid'</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">is</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">not</span> of <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">type</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'number'</span></span></code></pre></div></div>
</section>
</section>
<section id="so-what-should-you-use-for-data-validation-in-the-public-sector" class="level2">
<h2 class="anchored" data-anchor-id="so-what-should-you-use-for-data-validation-in-the-public-sector">So what should you use for data validation in the public sector?</h2>
<p>That’s up to you, and will depend on your use case. For now, if you are validating a dataframe or database, and you work in a mixed team, and you’re <em>not</em> going to deploy to a production system, I think that <a href="https://pandera.readthedocs.io/"><strong>pandera</strong></a> is a strong choice but keep an eye on <a href="https://posit-dev.github.io/pointblank/"><strong>pointblank</strong></a> to see where it goes. If you’re working in a serious production environment and doing wholesale automation, you might find <a href="https://greatexpectations.io/"><strong>great expectations</strong></a> better, especially because of the built-in functionality that triggers actions. And, finally, if you’re working with validating user input, from an API or form, then <a href="https://docs.pydantic.dev/"><strong>pydantic</strong></a> is probably the strongest choice.</p>


</section>


 ]]></description>
  <category>data science</category>
  <category>data</category>
  <category>public sector</category>
  <guid>https://www.aeturrell.com/blog/posts/the-data-validation-landscape-in-2025/</guid>
  <pubDate>Wed, 05 Mar 2025 00:00:00 GMT</pubDate>
  <media:content url="https://www.aeturrell.com/blog/posts/the-data-validation-landscape-in-2025/pointblank-tabular-report.png" medium="image" type="image/png" height="65" width="144"/>
</item>
<item>
  <title>Ultra-modern Python Cookiecutters</title>
  <dc:creator>Arthur Turrell</dc:creator>
  <link>https://www.aeturrell.com/blog/posts/ultra-modern-python-cookiecutters/</link>
  <description><![CDATA[ 





<p>In January 2020, <a href="https://cjolowicz.github.io/">Claudio Jolowicz</a> published an extremely influential post on <a href="https://cjolowicz.github.io/posts/hypermodern-python-01-setup/">Hypermodern Python</a>. It was extremely influential on me, anyway, because it introduced me to a number of tools that I now consider essential to creating solid, high-quality and low-maintenance Python packages. As part of the article, a <a href="https://github.com/cookiecutter/cookiecutter">cookiecutter</a> template was released to help people create new packages with all of these exciting features.<sup>1</sup></p>
<div class="no-row-height column-margin column-container"><div id="fn1"><p><sup>1</sup>&nbsp;Cookiecutter templates allow people to fill in a few details in the command line, like project name and Python version, to both copy a repo and populate it with their choices. It’s a simple trick that saves a lot of time in creating a basic project structure. The <a href="https://github.com/cookiecutter/cookiecutter">cookiecutter</a> Python package is used to populate the templates.</p></div></div><p>But a lot has changed since 2020, and the brilliant hypermodern python cookiecutter repo hasn’t been updated in two years. There are at least three big changes (we’ll come to these) since then that mean it’s well worth revisiting what a modern (or ultramodern) Python package cookiecutter should look like… and I’ve tried to capture them.</p>
<p>So, in this post, I’m introducing two new, ultramodern repos: a <a href="https://github.com/aeturrell/cookiecutter-python-package">Cookiecutter Python Package</a> and a <a href="https://github.com/aeturrell/cookiecutter-research-project/tree/main">Cookiecutter Research Project</a>. Neither of them are battle tested, far from it, but they do deliver the core goods they promise.</p>
<div id="fig-logos" class="quarto-layout-panel page-columns page-full">
<figure class="quarto-float quarto-float-fig figure page-columns page-full">
<div aria-describedby="fig-logos-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<div class="quarto-layout-row">
<div class="quarto-layout-cell-subref quarto-layout-cell" data-ref-parent="fig-logos" style="flex-basis: 50.0%;justify-content: flex-start;">
<div id="fig-package" class="quarto-float quarto-figure quarto-figure-center anchored">
<figure class="quarto-float quarto-subfloat-fig figure">
<div aria-describedby="fig-package-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<img src="https://www.aeturrell.com/code/logos/cookiecutter_logo_package.svg" class="img-fluid figure-img" data-ref-parent="fig-logos">
</div>
<figcaption class="quarto-float-caption-margin quarto-subfloat-caption quarto-subfloat-fig" id="fig-package-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
(a) Package
</figcaption>
</figure>
</div>
</div>
<div class="quarto-layout-cell-subref quarto-layout-cell" data-ref-parent="fig-logos" style="flex-basis: 50.0%;justify-content: flex-start;">
<div id="fig-research" class="quarto-float quarto-figure quarto-figure-center anchored">
<figure class="quarto-float quarto-subfloat-fig figure">
<div aria-describedby="fig-research-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<img src="https://www.aeturrell.com/code/logos/cookiecutter_logo_research.svg" class="img-fluid figure-img" data-ref-parent="fig-logos">
</div>
<figcaption class="quarto-float-caption-margin quarto-subfloat-caption quarto-subfloat-fig" id="fig-research-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
(b) Research
</figcaption>
</figure>
</div>
</div>
</div>
</div>
<figcaption class="quarto-float-caption-margin quarto-float-caption quarto-float-fig margin-caption" id="fig-logos-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
Figure&nbsp;1: Logos for these dev tools
</figcaption>
</figure>
</div>
<p>The <a href="https://github.com/aeturrell/cookiecutter-python-package">cookiecutter python package</a> is designed to help people (<a href="../../../code/index.html">including me!</a>) build open source software. The <a href="https://github.com/aeturrell/cookiecutter-research-project/tree/main">cookiecutter research project</a> is designed to help people (<a href="../../../research/index.html">especially me!</a>) spend more research time thinking.</p>
<section id="ultramodern-tooling" class="level2 page-columns page-full">
<h2 class="anchored" data-anchor-id="ultramodern-tooling">Ultramodern tooling</h2>
<p>Both of these new cookiecutters take advantage of major developments in Python tooling:</p>
<ul>
<li><p>Astral’s extremely fast linter and formatter, <a href="https://astral.sh/ruff">Ruff</a>, has burst onto the scene since 2020. And in tjhis repo, it replaces Black (which was in the hypermodern setup) as a linter and formatter, and isort for sorting imports.</p></li>
<li><p>The arrival of another tool from Astral, <a href="https://docs.astral.sh/uv/">uv</a>, which replaces a whole host of tools (including Poetry, which was part of the hypermodern setup). At a minimum, this is a blazing fast package manager and drop-in replacement for pip, but it also:</p>
<ul>
<li>resolves dependency conflicts</li>
<li>produces a lockfile</li>
<li>automatically creates virtual environments per-project, per-folder</li>
<li>can spin up an environment just to run a single script<sup>2</sup></li>
<li>builds packages</li>
<li>can install some versions of Python for some systems</li>
<li>creates a valid <code>pyproject.toml</code> file</li>
<li>can install and run tools globally with <code>uv tool install</code> and <code>uv tool run</code>.</li>
</ul></li>
<li><p>The release of <a href="https://quarto.org/">Quarto</a>, which, among many other things, can build websites, and <a href="https://machow.github.io/quartodoc/get-started/overview.html">Quartodoc</a>, which can produce automatic reference documentation based on a code base in concert with Quarto. Quarto means you can write your documentation in a Jupyter Notebook, importing your actual package so that the code and code outputs are in lockstep with the codebase. The addition of Quartodoc means that the API reference on your docs is also <em>exactly</em> what is in your code base.<sup>3</sup> Quarto requires a standalone installation, while Quartdoc is a Python package. You can find more about using Quarto in anger for research over at <a href="https://aeturrell.github.io/coding-for-economists">Coding for Economists</a>.</p></li>
</ul>
<div class="no-row-height column-margin column-container"><div id="fn2"><p><sup>2</sup>&nbsp;Yes, that’s how fast it is!</p></div><div id="fn3"><p><sup>3</sup>&nbsp;Note that Quartodoc is only relevant for the Cookiecutter Python Package.</p></div></div></section>
<section id="cookiecutter-python-package" class="level2">
<h2 class="anchored" data-anchor-id="cookiecutter-python-package">Cookiecutter Python Package</h2>
<p>So what are the key features of the new ultramodern cookiecutter? As of the time of writing, it packs the following:</p>
<ul>
<li><a href="https://nox.thea.codes/en/stable/">Nox</a> for isolated testing</li>
<li>Modern Python dependency management with <a href="https://astral.sh/blog/uv">uv</a></li>
<li><a href="https://docs.pytest.org/en/7.4.x/">pytest</a> for testing</li>
<li>Code formatting with <a href="https://docs.astral.sh/ruff/">ruff</a> (including formatting and import sorting)</li>
<li><a href="https://github.com/Erotemic/xdoctest">xdoctest</a> to check that examples of use in docstrings work as intended</li>
<li><a href="https://github.com/agronholm/typeguard">typeguard</a> for run-time type checking</li>
<li>Git pre-commit hooks for code quality:
<ul>
<li><a href="https://docs.astral.sh/ruff/">Ruff</a> lint/format/sort imports</li>
<li>check for added large files
<ul>
<li>check TOML</li>
<li>check YAML</li>
<li>end of file fixer</li>
<li>trailing whitespace trimmer</li>
<li><a href="https://github.com/kynan/nbstripout">nbstripout</a> for ensuring notebook outputs are not committed. (Notebook outputs are included when Quarto pushes the docs website to GitHub pages, however, as you’d expect.)</li>
<li><a href="https://github.com/jsh9/pydoclint">pydoclint</a> for checking docstrings agree with function definitions</li>
</ul></li>
</ul></li>
<li>Continuous Integration/Continuous Deployment with <a href="https://github.com/features/actions">GitHub Actions</a>
<ul>
<li>covers multiple versions of Python, and all three major operating systems</li>
<li>tests with <a href="https://docs.pytest.org/en">pytest</a> + <a href="https://nox.thea.codes/en/stable/">Nox</a>
<ul>
<li>Test cover with <a href="https://coverage.readthedocs.io/">Coverage.py</a></li>
<li>Automatic <a href="https://github.com/actions/labeler">release labeler</a></li>
<li>Automatic publishing to <a href="https://pypi.org/">PyPI</a></li>
<li>Dynamic docs build and deploy using <a href="https://quarto.org/">Quarto</a> for a docs site, <a href="https://machow.github.io/quartodoc">Quartodoc</a> for automatic API documentation, and <a href="https://pages.github.com/">GitHub Pages</a> for deployment.</li>
</ul></li>
</ul></li>
</ul>
<p>For some of this, you’ll need to get API keys. But all of it is free if your repo is public and you have a paid GitHub account.</p>
<p>In the choice of what’s included, I’ve tried to strike a balance between not bloating the features and providing a serious foundation for a strong package. I took the decision <em>not</em> to include the following, for the following reasons:</p>
<ul>
<li>an automatic updating utility, like <a href="https://github.com/dependabot">dependabot</a>; mostly because I’m not sure if there’s one that works well with uv yet.</li>
<li>a static type checker, like <a href="https://github.com/python/mypy">mypy</a>, because it seemed to be of marginal value-add in other projects and its strictness can be daunting. There was a recent announcement that Astral is working on a static type checker code-named <a href="https://github.com/astral-sh/ruff/pulls?q=is%3Apr+is%3Aopen+label%3Ared-knot">red-knot</a>, and I’m inclined to see what that’s like given the quality of their other tools.</li>
</ul>
<p>Both of these features could be added in future.</p>
</section>
<section id="cookiecutter-research-project" class="level2 page-columns page-full">
<h2 class="anchored" data-anchor-id="cookiecutter-research-project">Cookiecutter Research Project</h2>
<p>The second new dev tool is a <a href="https://github.com/aeturrell/cookiecutter-research-project/tree/main">Cookiecutter Research Project</a> that is Python-oriented and flexible. It has a number of useful features designed to make starting, and maintaining, a research project less of a slog:</p>
<ul>
<li>a well-designed folder structure with folders for data at different stages, models, notebooks, code, and outputs.</li>
<li>sensible defaults on which of these folders are ignored by git (via a <code>.gitignore</code> file). For example, code, references, paper, and slides folders are under version control. But data, logs, outputs, and so, folders are not.</li>
<li>a .env file for storing secrets—researchers are increasingly using cloud computer to do research (see <a href="../../../blog/posts/visual-studio-code-in-the-cloud/index.html">this post</a> for doing this with VS Code and Google Cloud.)</li>
<li>pre-commit with Ruff (linting, formating, import sorting), nbstripout, end of file fixer, large file check, trailing whitespace fixer, and toml/yaml checks.</li>
<li>uv for managing the Python environment, and making it reproducible via the lockfile.</li>
<li>a Makefile with commands for installing the environment (for reproducibility), and for compiling the paper, and the slides.</li>
<li>paper and slides based on Quarto—more detail on these below.</li>
<li>a project config TOML file where global project settings can be stored. For example, you could have all your chart configurations here, or the hyperparameter settings.</li>
</ul>
<p>One of the big innovations is the use of Quarto for the paper and the slides (and for the references that get picked up in both.) <a href="https://quarto.org/">Quarto</a> is a document, slide, and website publishing tool. It uses an extended form of markdown, supports LaTeX equations, and can execute and insert code into the final document.</p>
<p>One small advantage of Quarto over using LaTeX directly is the use of citation style language files. References are in a <code>.bib</code> file as normal for a LaTeX document with a bibliography, but the style of citations is defined in a <code>.csl</code> (csl = citation style language) file, which is clearer and more flexible than other methods (at least in my view.) You can find a very long list of citation styles, including for most major journals, <a href="https://github.com/citation-style-language/styles">here</a>.</p>
<p>Another advantage, and hold your horses, because it’s a big one, is that you can <em>automatically update charts, tables, and </em>even text** when your results change. People have long <a href="https://aeturrell.github.io/coding-for-economists/auto-research-outputs.html">pointed their LaTeX document at their results folder</a> so that when the results change, their paper updates. Some people have done it for slides too, if they’re created with Beamer. But now the <em>text</em> in the document can be updated too.</p>
<p>Quarto can support execution of code <em>within</em> the document itself. So you can have a code block that reads in your results and assigns them to the variables, say, <code>number</code> and <code>big_pen</code>. Then the syntax below shows how to insert these numbers directly into the text.</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb1" style="background: #f1f3f5;"><pre class="sourceCode markdown code-with-copy"><code class="sourceCode markdown"><span id="cb1-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">## Report</span></span>
<span id="cb1-2"></span>
<span id="cb1-3">We find that the heaviest penguin, out of a total of <span class="in" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">`python f"{number}"`</span> penguins, has a mass of <span class="in" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">`python f"{big_pen:.2f}"`</span> kilograms.</span></code></pre></div></div>
<p>Images and equations can be included using the standard markdown syntax, and LaTeX tables can be included in PDF outputs simply by reading and printing a LaTeX table <code>.tex</code> file with the <code>#| output: asis</code> option in a code block. You can see an example in the <code>paper.qmd</code> file in the <a href="https://github.com/aeturrell/cookiecutter-research-project/tree/main">repo</a>. That paper is set up to compile into something that looks like the default arXiv style. It actually compiles by creating a <code>.tex</code> file as an intermediary and there’s a Quarto config option to save that, should you need too (eg arXiv prefers you to submit a <code>.tex</code> over a PDF.)</p>
<p>Currently, the slides use Reveal.js to output a <code>.html</code> file rather than a PDF so while the code/text and charts work in the same way, the command to include LaTeX is slightly different—and I’ve only tested it for equations. The syntax is: <code>{{&lt; include path/to/equation.tex &gt;}}</code>. Anything that has a HTML output should work too, and that’s true of most regression table packages (eg <a href="https://github.com/py-econometrics/pyfixest">Pyfixest</a>.) If you prefer Powerpoint<sup>4</sup>, you can export to that with Quarto too, though I haven’t tested it with typical research outputs.</p>
<div class="no-row-height column-margin column-container"><div id="fn4"><p><sup>4</sup>&nbsp;Some people might. No-one I’ve ever met. Or heard about. But it could happen. Probably.</p></div></div><p>There are a couple of features one could argue for that I have <em>not</em> implemented for the research cookiecutter:</p>
<ul>
<li>a way to re-run all of the analysis. This is quite bespoke to your project so it was hard to include anything meaningful in the template. There is a Makefile in there, so you could extend that to include a part that executes the analysis—you can see an example along these lines <a href="https://github.com/aeturrell/example-reproducible-research">here</a>.<sup>5</sup></li>
<li>a Dockerfile to run your project reliably across systems. It’s possible this will be added in future. Again, you can find an example research project that uses uv and a Dockerfile <a href="https://github.com/aeturrell/example-reproducible-research">here</a>.</li>
<li>a way to automatically generate a DAG (directed acyclic graph) of all the operations that go into producing the outputs. I find this very useful to understand research projects (even if they’re my own.) The <a href="https://github.com/aeturrell/example-reproducible-research">example I keep mentioning</a> uses <code>make2graph</code> to do this if you want to see what I mean and why it might be useful.</li>
</ul>
<div class="no-row-height column-margin column-container"><div id="fn5"><p><sup>5</sup>&nbsp;For more on making your research project a reproducible analytical pipeline, check out <a href="https://aeturrell.github.io/coding-for-economists/wrkflow-rap.html">this page</a> on Coding for Economists.</p></div></div><p>Imagine how much time you will save if new results can be incorporated into your paper and slides simply by recompiling! For me, this is a huge win for the amount of time actually spent on the research question vs doing tedious updating.</p>
</section>
<section id="conclusion" class="level2">
<h2 class="anchored" data-anchor-id="conclusion">Conclusion</h2>
<p>While these templates are both still wet behind the ears, they could be useful to people today—which is why I’m releasing them now. Feedback and pull requests are welcome. Finally, if you use either of them to produce your own content, let me know by raising an issue. I’ll add your work to the relevant repo—showing that these cookiecutters can be part of a recipe for success!</p>


</section>


 ]]></description>
  <category>code</category>
  <category>data science</category>
  <category>research</category>
  <guid>https://www.aeturrell.com/blog/posts/ultra-modern-python-cookiecutters/</guid>
  <pubDate>Sun, 23 Feb 2025 00:00:00 GMT</pubDate>
  <media:content url="https://www.aeturrell.com/code/logos/cookiecutter_logo_package.svg" medium="image" type="image/svg+xml"/>
</item>
<item>
  <title>TIL: How to resume sessions on virtual machines</title>
  <dc:creator>Arthur Turrell</dc:creator>
  <link>https://www.aeturrell.com/blog/posts/til-resuming-vm-sessions/</link>
  <description><![CDATA[ 





<p>Today I learned how to resume sessions on virtual machines while using Visual Studio Code remote.</p>
<p><a href="https://code.visualstudio.com/docs/remote/remote-overview">Visual Studio Code remote</a> is incredible for SSH-ing into remote virtual machines. But sometimes you start off a long-running process in a shell on the remote machine and your connection is interrupted before it can finish; then, when you resume the connection, you’ve lost the progress you made through your code.</p>
<p>But today I learned that you can start off <em>persistent</em> shell processes using a command line tool called <strong>screen</strong> that comes pre-installed in many Linux distributions (eg Ubuntu).</p>
<p>To use it (assuming it is pre-installed), once you’ve opened the folder you’re working in on the remote via SSH from VS Code, simply type</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb1" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb1-1"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">screen</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-S</span> YOUR-SHELL-NAME</span></code></pre></div></div>
<p>to initiate a shell with screen name <code>YOUR-SHELL-NAME</code> that will persist.</p>
<p>Once you reconnect to the virtual machine run <code>screen -ls</code> to see existing running shells and <code>screen -r YOUR-SHELL-NAME</code> to reconnect to the running session! Super useful.</p>
<div class="quarto-figure quarto-figure-center page-columns page-full">
<figure class="figure page-columns page-full">
<p><img src="https://www.aeturrell.com/blog/posts/til-resuming-vm-sessions/dall_e_haynes_style_screens.webp" class="img-fluid figure-img"></p>
<figcaption class="margin-caption">DALL-E rendition of resuming a session</figcaption>
</figure>
</div>



 ]]></description>
  <category>code</category>
  <category>cloud</category>
  <category>TIL</category>
  <guid>https://www.aeturrell.com/blog/posts/til-resuming-vm-sessions/</guid>
  <pubDate>Wed, 01 Jan 2025 00:00:00 GMT</pubDate>
  <media:content url="https://www.aeturrell.com/blog/posts/til-resuming-vm-sessions/dall_e_haynes_style_screens.webp" medium="image" type="image/webp"/>
</item>
<item>
  <title>Converting handwritten, maths-heavy lecture notes to markdown using large language models</title>
  <dc:creator>Arthur Turrell</dc:creator>
  <link>https://www.aeturrell.com/blog/posts/converting-lecture-notes/</link>
  <description><![CDATA[ 





<p>It would be nice to have digital copies of all of those old handwritten lecture notes that I so lovingly put together. Some of them might even still be useful, though I have to admit I don’t have tons of opportunities to use quantum field theory these days.</p>
<p>Recent Large Language Models (LLMs) have stunning vision capabilities and so it occurred to me that they might be able to convert even old notes into beautifully formatted markdown and equations.</p>
<section id="models" class="level2">
<h2 class="anchored" data-anchor-id="models">Models</h2>
<p>I’ll try out two recent models from OpenAI and Google respectively,</p>
<ul>
<li>chatgpt-4o-latest</li>
<li>gemini-exp-1206</li>
</ul>
<p>The latter has a generous free tier, the former cost $0.03 to run one image.</p>
</section>
<section id="code" class="level2">
<h2 class="anchored" data-anchor-id="code">Code</h2>
<p>The Python APIs for both of these couldn’t really be simpler—you can find the example <a href="https://github.com/aeturrell/digitise-lecture-notes">in this repo</a>, but, to give you a sense, here is the complete code to use Google’s Gemini model:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb1" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb1-1"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> os</span>
<span id="cb1-2"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> pathlib <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> Path</span>
<span id="cb1-3"></span>
<span id="cb1-4"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> google.generativeai <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">as</span> genai</span>
<span id="cb1-5"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> PIL.Image</span>
<span id="cb1-6"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> dotenv <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> load_dotenv</span>
<span id="cb1-7"></span>
<span id="cb1-8">load_dotenv()</span>
<span id="cb1-9">genai.configure(api_key<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>os.getenv(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"API_KEY_GOOGLE"</span>))</span>
<span id="cb1-10"></span>
<span id="cb1-11"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Path to image</span></span>
<span id="cb1-12">image_path <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> Path(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"input/test_lecture_note_photo.jpg"</span>)</span>
<span id="cb1-13">sample_file <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> PIL.Image.<span class="bu" style="color: null;
background-color: null;
font-style: inherit;">open</span>(image_path)</span>
<span id="cb1-14"></span>
<span id="cb1-15"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Choose a Gemini model.</span></span>
<span id="cb1-16">model <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> genai.GenerativeModel(model_name<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"gemini-exp-1206"</span>)</span>
<span id="cb1-17"></span>
<span id="cb1-18">prompt <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"The photograph shows some lecture notes. Please transcribe the text in this photograph to markdown, using latex for equations where needed."</span></span>
<span id="cb1-19"></span>
<span id="cb1-20">response <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> model.generate_content([prompt, sample_file])</span>
<span id="cb1-21"></span>
<span id="cb1-22"><span class="bu" style="color: null;
background-color: null;
font-style: inherit;">open</span>(Path(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"output/output_example_gemini.md"</span>), <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"w"</span>).write(response.text)</span></code></pre></div></div>
</section>
<section id="results" class="level2 page-columns page-full">
<h2 class="anchored" data-anchor-id="results">Results</h2>
<p>This is the input file. Yes, I took a bad photograph. In some ways, that’s quite helpful, as it gives a sense of how these models do in a difficult situation. If you were doing this at scale, you’d probably want to find a consistent way of taking high quality photographs, however.</p>
<div class="quarto-figure quarto-figure-center page-columns page-full">
<figure class="figure page-columns page-full">
<p><img src="https://www.aeturrell.com/blog/posts/converting-lecture-notes/test_lecture_note.webp" class="img-fluid figure-img" style="width:80.0%"></p>
<figcaption class="margin-caption">Example input of handwritten lecture note</figcaption>
</figure>
</div>
<p>And here are the outputs:</p>
<div class="quarto-figure quarto-figure-center page-columns page-full">
<figure class="figure page-columns page-full">
<p><img src="https://www.aeturrell.com/blog/posts/converting-lecture-notes/lecture_output_gpt4o.webp" class="img-fluid figure-img" style="width:80.0%"></p>
<figcaption class="margin-caption">Output capture from GPT-4o</figcaption>
</figure>
</div>
<div class="quarto-figure quarto-figure-center page-columns page-full">
<figure class="figure page-columns page-full">
<p><img src="https://www.aeturrell.com/blog/posts/converting-lecture-notes/lecture_output_gemini.webp" class="img-fluid figure-img" style="width:80.0%"></p>
<figcaption class="margin-caption">Output capture from gemini-exp-1206</figcaption>
</figure>
</div>
<p>The GPT-4o model cost around $0.03, while the experimental Gemini model is free up to so many tokens per day. They both do a really great job given the quality of the input and you can see that this would probably work at scale with some better photographs.<sup>1</sup> They each make a couple of mistakes, and Gemini seems more keen on beautiful latex equations than it does on markdown formatting, at least relative to GPT-4o. Bear in mind, there was close to zero effort expended on designing the prompt here too—so it’s likely that this could be improved too.</p>
<div class="no-row-height column-margin column-container"><div id="fn1"><p><sup>1</sup>&nbsp;Garbage in, garbage out.</p></div></div><p>All in all, this looks completely feasible. And there’s some speculation (h/t: <a href="https://simonwillison.net/">Simon Willison</a>) that vision models that can run <em>locally</em> (Llama 3.2, Qwen-VL, and Pixtral) would be able to handle it too.</p>


</section>


 ]]></description>
  <category>AI</category>
  <category>python</category>
  <category>data science</category>
  <guid>https://www.aeturrell.com/blog/posts/converting-lecture-notes/</guid>
  <pubDate>Sat, 28 Dec 2024 00:00:00 GMT</pubDate>
  <media:content url="https://www.aeturrell.com/blog/posts/converting-lecture-notes/lecture_output_gemini.webp" medium="image" type="image/webp"/>
</item>
<item>
  <title>The false economy of bad IT</title>
  <dc:creator>Arthur Turrell</dc:creator>
  <link>https://www.aeturrell.com/blog/posts/bad-it/</link>
  <description><![CDATA[ 





<p>Many of us will have experienced bad hardware or software at work. Applications that freeze when you try and do something. A lag when typing. Some programmes ceasing to work before crashing completely. Maybe it’s another kind of performance that makes you want to throw your laptop out of the window: the battery dies after you’ve only been to a couple of meetings, or the text on the screen screen seems teeny tiny if you’re not plugged into a monitor. All of this is very annoying. But, worst of all, it’s a waste of your most precious resource: your time.</p>
<div class="quarto-figure quarto-figure-center page-columns page-full">
<figure class="figure page-columns page-full">
<p><img src="https://www.aeturrell.com/blog/posts/bad-it/dall_e_laptop_window.webp" class="img-fluid figure-img" style="width:60.0%"></p>
<figcaption class="margin-caption">DALL-E generated image of someone expressing their frustration with their laptop, and window(s)</figcaption>
</figure>
</div>
<p>Employers should care about this because it’s losing them money, particularly if they have staff who use computers for most of their work or their staff are relatively well-paid<sup>1</sup>, or both.</p>
<div class="no-row-height column-margin column-container"><div id="fn1"><p><sup>1</sup>&nbsp;For the sake of making this concrete, let’s take “well-paid” to mean above the current UK median of £35k.</p></div></div><p>Let’s do a back-of-the-envelope calculation to show why bad IT makes no economic sense. The mid-range hardware used in many organisations looks, at best, like a 1TB SSD, 16GB RAM, i7 touch laptop or touch surface running Windows. As examples of these sorts of machines, a Lenovo ThinkPad model <a href="https://www.lenovo.com/gb/en/p/laptops/thinkpad/thinkpadp/thinkpad-p14s-gen-4-14-inch-intel/21hf000uuk">21HF000UUK</a> and a <a href="https://www.microsoft.com/en-gb/d/surface-pro-9/93vkd8np4fvk?activetab=pivot:overviewtab">Microsoft Surface Pro 9</a> with those specifications are a good fit. The laptop has a 14-inch screen; the surface 13 inches. A quick check shows that these come in at £1600 and £2200 respectively at the time of writing. In my experience, the ThinkPad has a battery life of, at best, 2—3 hours; I’m not sure about the Surface.</p>
<p>Now let’s take a high-end laptop: a <a href="https://www.apple.com/uk/shop/buy-mac/macbook-pro/14-inch-space-grey-apple-m3-chip-with-8-core-cpu-and-10-core-gpu-8gb-memory-512gb">14-inch M3-chip MacBook Pro</a> that can run small large language models locally (!) and has 24GB of RAM plus a 1TB SSD. It’s the same size of hard drive but has 8 extra GB of RAM, which is enough to make a substantial difference to the performance (but, also, <a href="https://www.howtogeek.com/865066/its-okay-to-buy-a-mac-with-only-8-gb-ram/">each GB of RAM is more performant</a> in a Mac.) This is all-important because many of the glitches, pauses, and slowness of less good laptops come down to insufficient RAM. It isn’t simple to compare the processors of these machines, but the Mac is very approximately 30% faster. Without going back and forth on claims about battery life, Mac laptops occupy four of the top six laptops with the best battery life as <a href="https://www.tomsguide.com/best-picks/best-laptops-for-battery-life">listed in Toms Guide</a> and personal experience suggests they go for far longer than 2—3 hours. Our high-end laptop comes in at £2300, an extra £100 or £700 relative to the mid-range ones. Now Macs do not have touchscreens but my observation is that extremely few people use the touch feature on a laptop. My other observation is that high-end laptops have problems once in a blue moon—there’s a crash once every 3 months or so. This is so rare that I take it to be zero.</p>
<p>Let’s say the cheaper hardware has three incidents a day where something glitches, crashes, or goes slow enough to pause a workflow and, on average, this causes three minutes of lost time during each occurrence. This seems entirely reasonable because most people will start doing something else, perhaps making a cup of tea, if their laptop becomes unresponsive for any length of time. Now, imagine the average professional costs a business £60.0k a year (with pension, National Insurance, etc, etc)<sup>2</sup> and works 7.5 hours a day, so 51.0p per working minute. This makes the cost of that lost time around £1200.0 per year, compared to spending only £700 to £100 more to get the higher end laptop.</p>
<div class="no-row-height column-margin column-container"><div id="fn2"><p><sup>2</sup>&nbsp;To give a sense of what this is as a headline salary, a base salary of £50k and pension of 7% plus National Insurance of £5644 takes the total cost to the employer to £59k.</p></div></div><p>Switching to a higher-end laptop pays off by at least £500.0 just in the first year. For the laptop, it’s worth going high-end for <em>any</em> all-in salary cost above £35.0k (remember, the UK median headline salary cost is £35k). The benefits are likely bigger when looking at a longer period: as far as I’ve been able to tell, mid-range laptops usually last around three years while higher end laptops last more like five years–with <a href="https://www.businessinsider.com/guides/tech/how-long-do-laptops-last">some reports suggesting</a> Apple laptops last longer than this. So the “payoff” from switching might be a few times this.</p>
<p>There are a bunch of other factors here that are hard to estimate. Staff might be more motivated by better IT, and this is hard to quantify. But if staff surveys consistently rate technology as poor, the benefits may be significant. There could be better staff retention, which is worth a lot. But better IT might also discourage some bad behaviour: if people have to deal with a laptop battery that doesn’t last long enough for two meetings in a row, they will either make no notes or start using personal devices to take notes. If people are faced with a laptop or surface that’s too fiddly to use on a train, they simply won’t work when in transit.</p>
<p>On the other hand, there is likely to be a cost to maintaining two or more separate operating systems in an enterprise. Apple laptops use MacOS, a different operating system to the one often installed on ThinkPads, Surfaces, and similar devices, which is Windows. But most organisations have already crossed the Rubicon on this with their use of iPhones for mobiles, and Linux for critical systems. It’s also really poor practice to hitch your organisation to a single provider of software and hardware. You’ll be backed into a corner when negotiating contracts, and may not be able to easily change things up if the software or hardware quality degrades. (It’s notable that frontier tech firms <a href="https://www.quora.com/What-operating-system-do-the-Google-employees-use-How-much-freedom-do-they-have-with-respect-to-an-OS-usage">allow staff to choose</a> between Windows, MacOS, and Linux.) I don’t know how to estimate the cost of maintaining two or more operating systems in an organisation but there are serious downsides to having just one.</p>
<p>I’m not suggesting that organisations should just buy Apple laptops–this is about efficiency, and the best prescription for that is to <strong>allow staff to choose what works best for them</strong>. The apparently economically optimal choice won’t make sense for everyone: some staff might want to stick with mid-range laptops because they get along better with the operating system or laptop design, and that’s okay. By allowing staff to choose what they get on best with you keep them happy, and you make your organisation more productive. The bottom line is that everybody wins when staff have the option to use a higher-end laptop—except, maybe, glaziers.</p>




 ]]></description>
  <category>productivity</category>
  <category>public sector</category>
  <guid>https://www.aeturrell.com/blog/posts/bad-it/</guid>
  <pubDate>Mon, 16 Sep 2024 23:00:00 GMT</pubDate>
  <media:content url="https://www.aeturrell.com/blog/posts/bad-it/dall_e_laptop_window.webp" medium="image" type="image/webp"/>
</item>
</channel>
</rss>
