<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Observability+ - Medium]]></title>
        <description><![CDATA[Publications on Distributed Systems Observability - Medium]]></description>
        <link>https://medium.com/observability?source=rss----eb277599ce4a---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Observability+ - Medium</title>
            <link>https://medium.com/observability?source=rss----eb277599ce4a---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 09:56:05 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/observability" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com" rel="hub"/>
        <item>
            <title><![CDATA[Health, Availability, Debuggability]]></title>
            <link>https://medium.com/observability/health-availability-debuggability-5b0ab300b35c?source=rss----eb277599ce4a---4</link>
            <guid isPermaLink="false">https://medium.com/p/5b0ab300b35c</guid>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[monitoring]]></category>
            <dc:creator><![CDATA[Jaana Dogan]]></dc:creator>
            <pubDate>Thu, 09 May 2019 19:10:51 GMT</pubDate>
            <atom:updated>2019-05-09T19:10:25.432Z</atom:updated>
            <content:encoded><![CDATA[<p>As we collect various observability signals from systems, it fosters a new conversation around the classification of the signals.</p><h3>Jaana B. Dogan on Twitter</h3><p>In observability, we need a new categorization of signals: availability and debuggability signals. Examples of availability signals are metrics, metrics derived from non-downsampled other signals. Debuggability is about logging, tracing, events, etc.</p><p>There is a significant discussion on observability signals and even strong advocacy for one signal over the other. Metrics, events, logs, traces or others? In order to provide some structure to the conversation, it might be productive to provide a high-level breakdown of the signals based on how we utilize them.</p><p>There are three main high-level aspects observability enables:</p><ol><li>Health</li><li>Availability</li><li>Debuggability</li></ol><p>Even though I introduce this classification, it is possible for a single signal to fit all three. It is not the ideal option but imagine how people rely on watching their logs to see a service is up. They are utilizing logs to measure latency for certain operations, and dig into logs to see what’s going on when there is an issue. Even though logs can fit into all three categories, it requires distinct design and planning to make them useful in these categories. And although it is possible to utilize the same signal for all three aspects, we may rely on different signals for each because each signal type has its pros/cons.</p><h4>Health</h4><p>Health checking is fundamentally critical for reliability. We previously wrote on what it means for a system to be <a href="https://medium.com/observability/health-and-availability-b2f32ce28716">healthy</a>. Health is to describe the existence of a system and can be extended to answer its availability to accept work. “Is my server running?”, “Is my worker up?”, “Is my server ready to accept requests?” are the questions you can ask to health signals.</p><p>Health signals are critical for orchestrators such as schedulers and load balancers. Health signals are queried pretty often. They often have to be cheap to produce and easy to query.</p><p>One example of health signals is a health check endpoint served from long-running processes (typically servers):</p><pre>GET /health<br>ok</pre><p>Health signals should be made available quickly at startup. Faster health checking means less overhead for rescheduling, autoscaling, and crash recovery.</p><h4>Availability</h4><p>Availability is the systematic approach to be able to tell if a system is running as users expect. Availability is often quantified and measured by indicators (see <a href="https://landing.google.com/sre/sre-book/chapters/service-level-objectives/">SLIs</a> as an example formal definition). Then, a target value or range of values (<a href="https://landing.google.com/sre/sre-book/chapters/service-level-objectives/#objectives-g0s1tdcz">SLOs</a>) are determined for each indicator to represent the acceptable values.</p><p>A system is having an outage when the values are out of the target value/range for the target amount of user requests. And example SLO, “95% of the requests will complete in 100ms.”, will trigger an alert when more than 5% of the requests have a latency longer than 100ms.</p><p>Availability signals are crucial for reliability because they answer the fundamental question of whether a system is working as expected as user wants. This is why availability signals need to report all instances of tasks. The instrumentation should be cheap enough so it is not downsampled. This is why often aggregated metrics are the preferred availability signal. They are compact, and cheap to collect and transfer on wire.</p><h4>Debuggability</h4><p>Debuggability comes into the play mainly for troubleshooting scenarios. Debuggability is also a major aspect in having self-descriptive systems. The more debuggable a system is, the more you can learn about it without having to rely on documentation and reading the codebase.</p><p>Debuggability signals can be considered more optional compared to health and availability signals because they come into play once you figure out the health and availability. Think about signals like logs and distributed traces. They are often more verbose and more expensive to collect. Downsampling the collection is a common strategy to deal with the cost in production. Dynamically enabling more collection and vice versa is also common. You might want to temporarily turn on some collection, investigate an issue and turn it off again.</p><p>—</p><p>Regardless of how you utilize the existing signals available to you from your systems, it is a more healthy approach to think about their role in these high level aspects. Then, you can easily design towards achieving health checking, availability and debuggability based on what signals you can possibly collect. A signal signal might be useful in multiple categories as well as you may need to invest into multiple signals.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5b0ab300b35c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/observability/health-availability-debuggability-5b0ab300b35c">Health, Availability, Debuggability</a> was originally published in <a href="https://medium.com/observability">Observability+</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Health and Availability in Computer Systems]]></title>
            <link>https://medium.com/observability/health-and-availability-b2f32ce28716?source=rss----eb277599ce4a---4</link>
            <guid isPermaLink="false">https://medium.com/p/b2f32ce28716</guid>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[high-availability]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[sre]]></category>
            <dc:creator><![CDATA[José Carlos Chávez]]></dc:creator>
            <pubDate>Wed, 04 Jul 2018 06:11:48 GMT</pubDate>
            <atom:updated>2019-07-12T08:18:22.661Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*FwHLrfuTamcvqCs1" /><figcaption>Are you alive? Sure you are but, are you operative?</figcaption></figure><p>Operating a system is a complex discipline. Claiming that our system is doing well means nothing if users can perceive an outage. Thus establishing that a system is working properly requires the conciliation of two points of view: what you can say about your system vs what your users can say about your systems.</p><p>If you visit a doctor for a medical check up, the doctor will run a set of tests: electrocardiogram, blood pressure check, cholesterol level check, etc. After those tests, doctor will determine whether you are <strong>healthy</strong> or not based on the results.</p><p><strong>Health</strong> in computer system is not so different. A probe is sent to determine whether a server is able to process an input or not (e.g to process a HTTP request or consume a message from a queue). Such a probe considers two dimensions that describe the basic functioning of the service and its current ability to accept work: liveness and readiness.</p><p><strong>Liveness:</strong> Indicates whether the service is running. If the liveness probe fails there is a big chance the service died or needs to restart. Liveness checks are similar to vital signs as they indicate the status of the body’s vital (life-sustaining) functions. Questions like <em>is there a heartbeat?</em> or <em>is the patient breathing?</em> can be answered.</p><p><strong>Readiness:</strong> Indicates whether the service is ready to receive input. If the readiness probe fails, then whatever controls the traffic to that service (e.g. a load balancer) should stop sending traffic to it to avoid failures. Readiness checks are like reflexes — they verify the integrity of the nervous systems. Questions like <em>are the pupils reactive?</em> or <em>is the plantar reflex present?</em> are supposed to be answered in a readiness test.</p><p>The upstream-to-downstream health checking can happen from point to point (let’s say by enabling a circuit breaker) or via some control system (if a load balancer needs to determine whether to forward request to an instance or not). A service may choose to reply “unhealthy” because it is not ready to take requests, a third party dependency is down, it reached the maximum number of database connections, it is shutting down or for some other reason. The client can act accordingly if the response is not received within some time window or the response indicates an unhealthy service.</p><p>As you can tell, this is a very nagios-ish monitoring style: a service is either up or down and although we have two dimensions, from client standpoint it is a binary answer: service works/does not work.</p><h3>Missing piece in the puzzle</h3><p>Software engineers are pretty good at borrowing concepts from other disciplines. If we look at the <a href="http://www.who.int/about/mission/en/">WHO’s definition of “health”</a>, we will find some similarities to the one we have in software:</p><blockquote>Health is a state of complete physical, mental and social well-being and not merely the absence of disease or infirmity. […]</blockquote><p>However our definition of health lacks one important piece: “social well-being”. Indeed, we are not considering how our services interact with clients or upstream as part of our definition of health, we only look for the “absence of disease or infirmity” from our point of view. In that line we need to introduce another idea to our health definition that represents our system as a social being: <strong>availability</strong>.</p><p><strong>Availability</strong> is the state of the system described from the user perspective, coupled to specific use cases: error responses in an endpoint, timeouts or latencies in a workflow, etc.</p><p>The Google’s <a href="https://landing.google.com/sre/book.html">Site Reliability Engineering book</a> introduced crucial concepts of service level and ways to quantify the availability of a service: <a href="https://landing.google.com/sre/book/chapters/service-level-objectives.html">service level indicators (SLIs), objectives (SLOs), and agreements (SLAs)</a>:</p><blockquote>[…] These measurements describe basic properties of metrics that matter, what values we want those metrics to have, and how we’ll react if we can’t provide the expected service […]</blockquote><p>On top of that, <a href="https://medium.com/@rakyll">JBD</a> describes availability as:</p><h3>JBD on Twitter</h3><p>Availability is not about keeping specific processes or hosts alive. It is about keeping the user experience undisrupted. User doesn&#39;t care how you achieve it. Set your SLOs based on user experience.</p><p>Availability level is described by a set of SLI values and it comes into play when SLIs go beyond SLOs (e.g. when an acceptable error rate, the SLO, is 7%, but the actual error rate, the SLI, is 10%). Most services consider request latency, error rate or system throughput as key SLIs.</p><p>Continuing the analogy, for availability we can set SLIs as:</p><ul><li>Can you stand on one leg?</li><li>How many steps can you take before you go breathless?</li><li>How many words can you remember and repeat afterwards?</li></ul><p>If you can not make two steps due to breathlessness then there is clearly something wrong, and the reason may or maynot be correlated with your vital signs or reflexes, but clearly from the patient’s perspective there is something up. Then you need to measure your availability levels to determine whether it is in the range of acceptance or not.</p><p>It is important to choose appropriate metrics to drive the right action if something goes wrong. SLIs should arise from meaningful user experience, and detrimental changes in user experience should directly influence one or more SLIs in a similar peace.</p><h3>Conclusion</h3><p>In summary, availability is as important as health as it conciliate engineering’s quantifiable model of system health with the user’s perception of system health and, although they focus on different aspects of the system, they are both meaningless without each other.</p><p><strong>Health</strong> helps you focus on static checks of the system itself: <em>Is my service alive? Can it accept requests?</em>, etc. This sort of reasoning can be automated with health endpoints.</p><p><strong>Availability</strong> is use-case centric:</p><ul><li>What is the error rate for requests in this endpoint?</li><li>What is the average processing time for this message type, for this consumer?</li></ul><p>It helps to understand system trends: an increase of latency in certain endpoints, or an increase of error rates at an operational level.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b2f32ce28716" width="1" height="1" alt=""><hr><p><a href="https://medium.com/observability/health-and-availability-b2f32ce28716">Health and Availability in Computer Systems</a> was originally published in <a href="https://medium.com/observability">Observability+</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>