Our engineering division used to be able to maintain consistency with zero effort using this one weird trick: just have one engineer. As the team grew, one engineer would do things one way, another that way, but by sharing the same code base, it would passively converge on the true way. Later on, engineering split roughly along the lines of our usage products and account support, and so ended one group of people looking at one codebase. The river between the groups caused a speciation event, and new traits could emerge independently in the two groups. Additional splits occurred later: splitting off inventory, splitting messaging. In each case, new traits could and would arise to fulfill their technological niches.

When I joined the company two years ago, our teams had already speciated. I joined the colony of Telnyxus mercurii (from the Roman god Mercury) and adopted the view from the messaging niche: a very deep, but very narrow view—everything messaging and only messaging. I knew every permutation of options, features introduced that few used, and what happens in that strange edge-case. The considerations I placed on adding features and fields were to be consistent with whatever our group had, but often they were novel, so I just shopped it around my team. However, they may not have been new to other teams. I wasn’t as familiar with what the other teams were doing: time was spent on problems that telephony apps or numbering inventory may have already solved, but worse, inconsistencies arose in our API.

As much fun as I’m having, we don’t want our customers to have to be evolutionary technologists to make sense of what’s going on. Coming to our docs and API in modern-day, interacting with multiple products, they may ponder why widgets are timestamped with “created_at” in Unix milliseconds, and doodads have “date_created” in the RFC 3339 format. One object can be removed with HTTP DELETE, but another would require an HTTP POST with a ?delete=true querystring. These are the remnants of our API’s evolution brought about because individual product teams seldom need to interact with the public API of other teams, and a general view of the API being the “last thing” along the path to develop a new feature.

Time for a change

So we’ve evolved. It was messy and not obvious why there’s that appendix or tailbone. To clean up from our evolution, we decided to release a second major API version and adopt a style guide so our engineers could work towards a common public facade. By providing a consistent API,

1. our documentation can be more consistent,
2. developers integrating can understand new products faster, and
3. we can maintain more SDKs for your language of choice by relying on the uniformity.

…but change is hard.

APIs are difficult to change. Adding fields is generally safe, but clients may be expecting fields in a specific order and may process responses in ‘creative’ ways which could be thrown off by something different. Removing or changing existing fields can be much more difficult, if not impossible. In theory, you could log and check who updates those fields, but you can’t check if they’re being read. Notifying users to update their code via your website might get some, via email some more, but until you make the breaking change, you won’t know. Ramifications are not immediate as well: it could only occur in rare (to the customer) circumstances. The customer may not use that one feature often, or could just be in a lull. Months later you can get questions about “why don’t we see X” anymore.

There are some proactive approaches to mitigate this API ossification. A project I used to work on, OpenStack, generalized their V2 API to use microversions. This would allow clients to declare which microversion they wanted via an HTTP header, and if the server supported it (servers would support a range), it would reply with the mutually supported response schema. However, this in and of itself requires a change to the API. Additionally, the clients and software using them must be periodically updated even if nothing functionally changes, to stay in the supported band of versions. This is OK for a coalition of projects like OpenStack, but when the components are us and our customer’s software, it can be a costly proposition to tell them they need to update every release cycle.

However you change, it should be gradual, allowing for users to migrate by opting-in over time. Taking down your entire version X API over a weekend, then bringing up version X+1 is a way to not do it (but that we’ve seen.)

Guiding principles

OK, so we needed to unify, but how? What would it look like? Adopting an API design guide is good for any organization. As a Pythonista, I try to take many “Pythonic” ideas (which are not necessarily about Python) to heart. For instance, PEP-8 teaches us:

A style guide is about consistency. Consistency with this style guide is important. Consistency within a project is more important. Consistency within one module or function is the most important.

But it should be tempered. The Zen of Python also applies, such as

Readability counts.
Special cases aren’t special enough to break the rules.
Although practicality beats purity.

Whatever you do, choose a style as a team. It ensures a consistent “voice” no matter who is writing parts of your API, with what language, using whatever libraries. If you don’t like something about it, fork it and change it. If it’s silent on a matter that causes inconsistencies, make it explicit, make it yours. Engineers, testers, and users should all be contributing to the guide.

Finding the One True Way

What is it? Does it exist? When we chose to go in our final direction a year ago, and still now, there does not appear to be One True Way. If there was The Way, everyone from AWS, GitHub, and Google would all have been using that way and this wouldn’t be hard. That said, I am severely understating my predecessors' contributions: HTTP, TLS, TCP/IP, and more underpin it all and are all part of The Way.

On the application layer, however, things are a little rockier. REST (or REST-esque depending on how stringent you can be) has provided HTTP with sane semantics unlike the wild west of 90s web dev. JSON isn’t XML, therefore it’s good. However, how to model and structure data still varies. Back in the day, SOAP as a form of XML-RPC would allow auto-discovery of calls and objects, but XML was a bit of a bear to deal with and is now considered more of a dinosaur.

HATEOAS (Hypermedia As The Engine Of Application State) is typically mentioned along with REST and does proscribe a bit more. Links (URLs or paths) to connect objects, define relationships, and show valid actions when in a given state. This has the advantage of being a discoverable API, and you can avoid versioning entirely! But then means that your users must rediscover the API. Unfortunately, lazy developers like me generally make calls by firing a JSON/query string directly to the previously discovered/documented endpoint with a semantically correct HTTP verb and happily getting the data in one round trip. The code it takes to do that is clear and straightforward, but even as a fairly experienced Python programmer, I don’t know what clients are out there that will do HATEOAS discovery for me, and I don’t have the time to develop my own. Even if I did, you’ll need to thwart your users from “misusing” your API and hardcoding paths. In effect, SDKs then turn from an optional feature (to provide and use) to be mandatory.

During our initial search for a guide, JSON-API was brought up. It’s a specification that follows many RESTful and HATEOAS principles, notably extensive linking and referencing of related resources. The decisions against it principally boiled down to a perception of complicatedness and lack of adoption by bigger players. Our initial view of sample specifications and responses seemed overly complicated, for us to provide, and we had a concern that customers may feel the same. I’m still having trouble finding larger players that have a publicly avowed “JSON-API” interface. Many like GitHub V3 make use of Hypermedia but seem more generally HATEOAS.

Our choice

We ended up using the Heroku/Interagent’s API design guide as a starting point and forked and extended it to our domain of telephony. Phone numbers will always be phone_number, not number, pn, tn, tel, telephone_number, or anything else. Local access and transport areas (LATA) are lata. The style was a good balance of being prescriptive in many areas, parsimonious, and able to accommodate a wide variety of workflows required at Telnyx.

Typical additions to the API guide are usually about constraining formats, as what’s “natural” in one language may be slightly different in another language. “Trivia” like how a date-time is formatted, or if query parameters are “starts_with” or “startswith”, don’t take much effort to fix if done as part of the development. The improved fit and finish of the API is something users won’t notice, as they can just get on with what they need to do. My OXO peeler isn’t the sharpest in the drawer, but it’s the one I reach for most often because it’s just easy and feels right.

Looking forward

So now we have a consistent version 2 of our API. That’ll last forever, right? Nope, we’ll need to improve it later. We even needed to improve it between our beta and final and did so in a rolling manner where our customers can switch over at their and our leisure. Tech is constantly evolving and the smoothest, obvious way to do things changes with it. I have hopes that one of the “binary JSON” formats like Protobuf, MsgPack, or Avro take off with a killer design and there’s ubiquitous usage of it in public APIs (I imagine they’re much more common internally). GitHub’s version 4 moved to GraphQL, which has many strengths that I value much more after undertaking our whole saga, and maybe we’ll beat them to it (version 3 is before version 4…haha). Maybe it’ll be something I read about on HackerNews tomorrow.

But I’m done for a bit. It’s your turn to use our API and see if we succeeded. Hit me up @nicktimko on Twitter with what you think.

APIs aren’t perfect, but they can be better was originally published in Telnyx Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

My principal reasons for using Pipenv were twofold. First, was the ability to have reproducible builds through an exhaustively defined lock file, with every package and version specified, then hashed for good measure. Second, I could do the above while maintaining a clean, minimal list of direct dependencies for my application.

Pin everything…

The reason for a lock file should be clear: so you can have reproducible builds. pip install mypackage may work when you build and test on your workstation, and through one release cycle, but what about when you make a small fix later? Why did changing one line of code and rebuilding cause everything to explode? There was a new incompatible release among one of your dependencies. Precisely defining all packages and their exact versions leads to reproducible builds that are easier to diagnose and bisect when they go wrong.

Bonus features in the lock file include package hashes. It can improve security, but the failures I’ve seen are because of using multiple package indexes. At Telnyx we have a private PyPI that we use to distribute Python libraries. We adapted a synchronous library to asyncio, but later someone else did so and released it publically with the same name. The collision led to sometimes downloading the private version, sometimes the public, depending on how the indexes were configured on the user’s machine, their build container, or the CI infrastructure. Our private PyPI also allows re-uploading of the same version, which is also made clear when the hash check fails.

The locked list of dependencies should be deliberately generated and validated through tests. This is often by a developer on their workstation, and they can immediately check before entering a broken commit into the repo. Diffing the lock file will show what new versions were installed, and the version can be pinned to an older version, or source changed to accommodate the change.

…and also only what you need

OK, but why do we need a separate list of dependencies? Imagine a simple cycle of pinning everything in a singular requirements.txt and regenerating it with pip freeze. The file will not track why certain packages are installed, and if you explicitly remove a direct dependency from it, you’ll be keeping around all of its sub-dependencies for no reason. Additionally, you can enter version resolution hell wherein trying to bump one package for a new version means it needs another package, but a different version than you have pinned. Why is that package pinned? Are you using it directly where you care about the changes, or is it just for one of your deps? Or multiple?

In this minimal list, we can pin just what we need, excluding incompatible or buggy versions purposefully, and allowing incidental upgrades where we can. This can provide bug fixes and performance improvements (admittedly there can be regressions…), and preclude integrating with deprecated features in older libraries that will be a large maintenance burden later.

pip-compile: does one thing

Note that none of the features listed include virtual environment management. I use Pyenv for this on my machine, and in production, setting using pip and the built-in venv package is best. Why install anything when you can create a virtual environment in one line, and pip install with one more? They’re ever-so-slightly onerous to call if you use full paths, but you’re typing it once in the Dockerfile or other setup scripts anyways.

python3 -m venv /opt/appvenv
/opt/appvenv/bin/python3 -m pip install requirements.txt

So how do I get that requirements.txt while holding on to the features I liked? Enter pip-tools. Making a wiser decision than Pipenv, pip-tools goes with the UNIX philosophy of “do one thing, and do it well”. It provides a few commands, but one of them: pip-compile can bake one requirements-style file into another, complete with versions and hashes.

pip-compile --quiet \
    --generate-hashes \
    --output-file=requirements.txt \
    requirements.in

Pipenv uses this library to do it — we can cut out the middleman! What’s more, the generated requirements.txt is perfectly readable by pip, so you don’t need to install the tooling in your production system, saving time, space, and exposure to bugs.

How can you make the switch? If you’re willing to shake up the versions from your Pipfile.lock, converting the Pipfile to a requirements.in is sometimes just a slightly-tedious bunch of editing. However, if you have a lot of libraries, and want to keep their versions locked to make sure changing tools didn’t break anything, I wrote a script that will do it for you!

# in a path with a Pipfile and Pipfile.lock
# prereqs: `pip install toml pip-tools`

curl -sSL https://raw.githubusercontent.com/nicktimko/pipenv2tools/master/pipenv2tools | python -

So even if Pipenv came out with a new release fixing all the outstanding bugs and pain points I’ve come across, I won’t go back.

RIP Pipenv: Tried Too Hard. Do what you need with pip-tools. was originally published in Telnyx Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Software is wondrous because it can be distributed for pennies, and with open-source software, licensed for free. However, source code or even binaries don’t do anything unless they’re run. Knowing how to run code can be tricky, you need to have the proper environment. The quality of documentation varies, and what’s obvious to the people with the skills to develop a tool, is not to those that may want to install it.

Case in point, this was my thought process while trying to set up this blog on GitHub Pages. jekyll build? No, need Jekyll installed. Docs say use Bundler. Bundler doesn't support Ruby 2.0. How do I upgrade Ruby on OS X, and should I? Maybe use RVM, that's maybe like pyvenv? OK, kinda works, using the "minimal" theme, still don't know how/why it's hitting GitHub and throwing rate limit errors at me…

This is frustrating. Even from someone who raves about Python, is reasonably skilled with it, and perversely enjoys stepping over those sorts of hurdles. Thankfully, none of that environmental configuration is needed. GitHub provides the service to build your pages for you, no Ruby required. Hell, you don’t even need Git, just a browser and some time.

Providing software as a service (and keeping that service up) makes sense to me as an engineer, loath as I am to bandy about that buzzword. It completely bypasses the problems of installation and configuration. Users don’t need to set up a finicky environment that is liable to break because some library got upgraded past where its version was pinned, and providers are able to get their tools used, seeing how they run and where they break to make them better.

Web apps are king.

They reduce the barrier to entry to a click. Putting your awesome, bespoke tools up on the web for anyone with a browser to use is one of the best forms of advertising, and something I’d strongly suggest my academic colleagues do whenever possible. Despite the protestations of a former advisor, successful academics are heavily engaged in marketing. Reading the trends and trying to fulfill the needs of Nature, et al. to get published. Put a URL in your paper and say “try me now!”

Get that code moving.

Originally published at amaral.northwestern.edu/blog/do-it-live on December 31, 2016.

Prepared with my 100% undeniable facts and smooth delivery, I ended up winning the day. View the PDF.

Mirror

Originally published at https://nicktimko.dev on October 13, 2016.