donchev.is

Sealed Secrets UI

Fri, 19 Jul 2024 00:00:00 +0000

Kubernetes is a powerful orchestration platform for microservices or for that matter any kind of workloads. Yet, managing sensitive data such as passwords, API keys, and credentials in Kubernetes can be frustrating. Tools such as Vault, Secret Store CSI and External Secrets Operator exist for this purpose. But there is a simpler way to manage secrets in Kubernetes especially if you are already doing GitOps. Sealed Secrets by Bitnami is one of my favorite tools. I can store my secrets encrypted in Git and have them decrypted in my Kubernetes cluster.

However, managing Sealed Secrets can be a bit challenging especially for beginners. The CLI tool is great but it can be a bit intimidating and don’t get me started with updating / merging secrets into existing Sealed Secrets.

This challenge is precisely what inspired us to create and now open-source the Sealed Secrets UI.

Why Sealed Secrets UI?

Sealed Secrets UI helps creating and updating sealed secrets in Kubernetes. Developed to integrate seamlessly with the Bitnami Sealed Secrets controller, this tool significantly reduces the complexity and potential for errors when managing secrets.

You can now go from this:

kubectl create secret generic awesome-secret \                                                                                                             
--from-literal=username=someUser \
--from-literal=password=somePassword \
--namespace=dev \
--dry-run=client \
-o yaml | kubeseal -o yaml

or this

echo -n PutPasswordHere | kubectl create secret generic awesome-secret --dry-run=client --from-file=apiKey=/dev/stdin -o yaml \
  | kubeseal --merge-into path/to/existing-sealed-secret.yaml

to this

For a better understanding of how and why check out this cool video by kubesimplify!

He did a great job explaining the Sealed Secrets UI and how it can be used to manage secrets in Kubernetes. At least a lot better than I can do here in writing 😅

Features and Benefits

Simple Web Interface: Instead of juggling kubectl commands, users can create and update sealed secrets directly through a user-friendly dashboard.
Seamless Integration: The UI fetches the necessary public keys directly from the Sealed Secrets controller without manual intervention, ensuring that secrets are always encrypted with the correct keys.
Intelligent Management: The UI intelligently checks for existing secrets and updates them accordingly.
Environment Customization: Users can customize settings through environment variables to fit different deployments, such as specifying namespaces or controller names.

Getting Started

The prerequisites for using Sealed Secrets UI are minimal:

The Kubernetes cluster must have the Sealed Secrets controller installed. The Sealed Secrets Helm chart is recommended for ease of installation.

Quick Configuration

Setting up the Sealed Secrets UI involves minor configuration, primarily related to pointing it at the right namespace and controller, if defaults are not used:

SEALED_SECRETS_CONTROLLER_NAMESPACE: kube-system (default)
SEALED_SECRETS_CONTROLLER_NAME: sealed-secrets-controller (default)

Deployment

Deploying the Sealed Secrets UI is straightforward with the provided Kubernetes manifests. Here’s an example of how to deploy it within your cluster:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sealed-secrets-ui
  namespace: kube-system
...

For complete deployment instructions, refer to the deployment section in our repository

As always, if you have any questions or remarks, feel free to ping me on twitter @bobby_donchev.

Introducing Statebin.io

Sat, 02 Dec 2023 00:00:00 +0000

I have been working on a new project called Statebin.io. It is a SaaS tool (free) that provides atomic counters, persistent storage, and locking mechanisms. It is a versatile tool designed to simplify and streamline various data management tasks. It offers a range of functionalities that cater to the needs of developers, system administrators, and tech enthusiasts alike. Think of it as httpbin.org but for state management :)

Features

Currently, Statebin.io offers the following features. More features are planned for the future. If you have any suggestions, please let me know!

Atomic Counters

Atomic counters are used to track the number of times an event has occurred. For example, you can use them to track the number of times a user has logged in to your website or the number of times a particular API has been called. Statebin.io provides a simple API that allows you to create and increment atomic counters. You can also use the API to retrieve the current value of a counter.

Use cases I use it for:

CI Pipelines: Keeping a precise track of the number of builds, ensuring accurate build sequencing and management. I know Azure DevOps provides BuildIDs but its a global counter. I needed a counter that is unique to each pipeline / branch.
GitHub Profile View Tracking: By leveraging the counter functionality, I can easily track the number of views on my GitHub profiles and repositories. I’ll be writing a blog post on how I did this soon.
Link Click Tracking: Haven’t implemented this yet but I can see how this can be easily implemented using the counter functionality.

Persistent Storage

Statebin.io provides a simple API that allows you to store and retrieve data. You can use this API to store and retrieve any type of data, including JSON, XML and binary data. The API is simple with GET, DELETE, and PUT requests.

Use Cases:

Settings files / Config files
Coverage SVG Files
CI Pipeline Files

Locking Mechanisms

Locks have a similar API to Atomic Counters. You can acquire and release locks. Locks by default have a TTL (Time To Live) of 60 minutes.

Use Cases:

Resource Locking in CI Pipelines
Exclusive Access Management

The Architecture

I created statebin.io using a serverless architecture. I used cloudflare workers that run on top of the cloudflare edge network. The workers are written in TypeScript and are deployed using the wrangler CLI tool. The workers are backed by more cloudflare, namely durable objects. Durable objects are a new serverless primitive that allows you to store stateful, long-lived objects in the cloudflare edge network.

For persistent storage, I use R2 with custom metadata detection. Currently the storage is limited to 100MB per object.

The combination of these technologies ensures that Statebin.io is not only globally available but also maintains high reliability. Data integrity and uptime are paramount, making it a dependable choice for your data management needs.

Conclusion

I hope you find Statebin.io useful. I have been using it for a while now and it has been working great. I have a few more features planned for the future.

Go check it out at Statebin.io and let me know what you think!

Securing a publicly accessible Fluentd server

Sat, 06 May 2023 00:00:00 +0000

Securing a publicly accessible Fluentd server: mTLS & Zero-Trust

In this post, I’ll walk you through the process of enhancing the security of a publicly accessible Fluentd server by implementing a Go TCP proxy that leverages mutual TLS (mTLS) authentication. By utilizing client-side self-signed certificates and their fingerprints, the proxy will ensure that only authorized clients can connect to the Fluentd server. Additionally, I’ll explore alternative zero-trust solutions like Cloudflare and OpenZiti.

Why Securing Fluentd Matters

Fluentd may need to be publicly accessible from the internet for a variety of reasons. The specific use case I recently dealt with involved an application deployed across multiple cloud providers. The application needed to send logs to a centralized Fluentd instance for aggregation and analysis. The cloud providers were not peered and did not have a VPN connection between them. As a result, the only way to send the logs was to make Fluentd publicly accessible.

Altough it was convenient, it also left my logging infrastructure vulnerable to potential security threats. As a result, it’s absolutely essential to employ strong security measures like mTLS and at a bare minimum an IP allow-list and Firewall.

Newer zero-trust solutions like Cloudflare and OpenZiti can also be used. I’ll discuss these solutions in more detail later in this post.

A Closer Look at the mTLS-based Go TCP Proxy

I wrote a Go proxy package that takes care of the core functionality of a TCP proxy, including mTLS authentication. Acting as an intermediary between clients and the Fluentd server. This proxy allows connections exclusively from authorized clients with valid self-signed certificates. The proxy relies on the fingerprints of these certificates for validation.

type FingerprintValidator interface {
	CheckFingerprint(fingerprint string) bool
}

type Proxy struct {
	listener  net.Listener
	group     *errgroup.Group
	validator FingerprintValidator
	stop      context.CancelFunc
	ctx       context.Context
	cfg       Opts
}

type Opts struct {
	Validator      FingerprintValidator
	ListenAddr     string
	FluentdAddress string
	CertFilePath   string
	KeyFilePath    string
}

func New(in Opts) *Proxy {
	p := Proxy{
		validator: in.Validator,
		group:     &errgroup.Group{},
		cfg:       in,
	}
	p.listener = p.mustListener()

	return &p
}

The Proxy struct contains the FingerprintValidator interface, which can be implemented depending on where the fingerprints are stored. For example, we can implement the CheckFingerprint() method to check the fingerprints against a database, a file, or any other data source (I recommend redis).

Another thing to note is that the Opts struct contains the CertFilePath and KeyFilePath fields. These are needed because the proxy acts as a TLS server. Meaning the traffic between the client and the proxy is encrypted. I used letsencrypt to generate the certificates for the proxy. But you can use any certificate authority or self-signed certificates.

func (p *Proxy) mustListener() net.Listener {
	cer, err := tls.LoadX509KeyPair(p.cfg.CertFilePath, p.cfg.KeyFilePath)
	if err != nil {
		log.Panic().Err(err).Msgf("Error loading cert/key pair")
	}

	config := &tls.Config{
		Certificates:          []tls.Certificate{cer},
		ClientAuth:            tls.RequireAnyClientCert,
		VerifyPeerCertificate: p.verifyPeer,
		MinVersion:            tls.VersionTLS12,
	}

	l, err := tls.Listen("tcp", p.cfg.ListenAddr, config)
	if err != nil {
		log.Panic().Msg("Could not start proxy")
	}

	return l
}

The full code for the proxy can be found here

The connection handling logic is based on one connection one goroutine. Meaning that for every connection, a new goroutine is spawned. Based on the benchmarks here the performance of this approach is sufficient for most use cases. If you are memory constrained, you can use a pool of goroutines to handle the connections. Just adjust the constructor (New function) and set a limit on the number of goroutines.

func New(in Opts) *Proxy {
  maxGoroutines := 1000
	group := errgroup.Group{}
	group.SetLimit(maxGoroutines)

	p := Proxy{
		validator: in.Validator,
		group:     &group,
		cfg:       in,
	}
	p.listener = p.mustListener()

	return &p
}

Setting Up the Proxy for Fluentd

To set up the proxy, follow these simple steps:

Initialize the Proxy

Adjust the ListenAddr and FluentdAddress in the main.go file and run go run main.go. You also need to have a running Fluentd instance. The easiest way to do this is to use the official docker image:

docker run -it -p 24224:24224 -e FLUENTD_CONF="<source>@type forwardport 24224bind 0.0.0.0</source><match **>@type stdout</match>" fluent/fluentd:latest

Test the proxy

To test the proxy you will need to send a msgpack-formatted message to the proxy. We can send a message using the fluent-cat command line tool:

echo '{"message":"hello"}' | fluent-cat debug.log --host localhost --port 8080

If everything is set up correctly, you should see the message in the Fluentd logs:

2023-05-07 15:20:20.000000000 +0000 debug.log: {"message":"hello"}

Alternative Solutions

While the mTLS-based Go TCP proxy with a Firewall and IP allow list effectively secures a publicly accessible Fluentd instance. It’s not the only solution.

Recently I have been experimenting with Cloudflare’s Zero Trust service and OpenZiti. Both of these solutions allow me to create a zero-trust connection between clients and servers.

Zero Trust Networking

Zero-Trust networking is a security paradigm that shifts the focus from traditional perimeter-based defenses such as firewalls, VPNs, and IP allow lists. Under this model, the network is not assumed to be trusted. Instead, every access request, whether it originates from inside or outside the network, is treated with equal scrutiny and must be verified before granting access to resources. This model is built on the principle of “never trust, always verify,” and requires continuous authentication and authorization of users, devices, and applications. By implementing granular access controls, Zero-Trust networking minimizes the attack surface and reduces the risk of unauthorized access, data breaches, and lateral movement of threats within the network.

In the case of Cloudflare, we can leverage their Zero Trust service to create a zero-trust connection between our clients and the Fluentd server. OpenZiti, on the other hand, is open source and offers a Go SDK that allows to create zero-trust connections from our clients to my backend infrastructure.

Cloudflare

Some of the benefits when using Cloudflares Zero-Trust service are:

It is a managed service - no need to manage the infrastructure and the certificates
WARP client - No custom code needed on the client. It’s available for Windows, Mac, Linux, iOS, and Android.
Scalability and performance - Cloudflare’s global network ensures high performance and scalability. The WARP client connects to the nearest Cloudflare PoP and the traffic is routed to the Fluentd server. This ensures that the Fluentd server is not exposed to the internet and is protected by Cloudflare’s network.

(disclaimer: I have no affiliation with Cloudflare but I do use their services and own stock in the company)

OpenZiti

OpenZiti is another option for implementing a zero-trust architecture. With its Go SDK, I can create secure connections between my clients and the backend without needing to install anything on my clients. The OpenZiti SDK handles authentication, authorization, and encryption.

Obviously, this solution requires more work than the Cloudflare one. But it’s open source and can be customized to fit my needs. Here is an example server and client implementation. It’s basically a custom Go HTTP Client Transport:

zitiDialContext := ZitiDialContext{context: ctx}

zitiTransport := http.DefaultTransport.(*http.Transport).Clone() // copy default transport
zitiTransport.DialContext = zitiDialContext.Dial
zitiTransport.TLSClientConfig.InsecureSkipVerify = true
client := &http.Client{Transport: zitiTransport}

Not much code if you ask me.

Conclusion

Securing a publicly accessible Fluentd instance is crucial. Implementing a Go TCP proxy with mTLS authentication is an effective way to ensure that only authorized clients can connect to the Fluentd server. Additionally, considering alternative zero-trust solutions like Cloudflare Zero Trust and OpenZiti can further enhance the security.

By combining these security measures, we can build a robust and secure public Fluentd deployment. Without having to worry about unauthorized access.

If you have any questions or comments, feel free to reach out to me on Twitter

Reloading Fluentd certificates

Sat, 11 Feb 2023 00:00:00 +0000

Introduction

Fluentd is a great tool for collecting logs. Usually it’s deployed in Kubernetes as a Log Aggregator and / or a Log Forwarder. In this post I will focus on Fluentd as a Log Aggregator. In a usual log aggregation setup we would want to use TLS for encrypting our traffic and even though Fluentd supports TLS certificates, it does not support automatic cert reloads.

Why do we need to reload the certificates? lets-encrypt certificates are valid for 90 days and they are renewed every 60 days. And since I am 99% of the time using lets-encrypt certificates, I will focus on that =)

In this post I will show you how to configure Fluentd to use lets-encrypt certificates. Primarily, I will show you how to reload the certificates when they are renewed.

Configuring Fluentd to use lets-encrypt certificates

Configuring Fluentd to use lets-encrypt certificates is pretty straightforward. We just need to add the following lines to our Fluentd config:

    <source>
      @type forward
      @log_level error
      <transport tls>
        version TLSv1_2
        ciphers ALL:!aNULL:!eNULL:!SSLv2
        insecure false

        cert_path /letsencrypt/tls.crt
        private_key_path /letsencrypt/tls.key
        client_cert_auth false
      </transport>

      bind 0.0.0.0
      port 24224
    </source>

The only thing we need to do for the above config to work is to mount the lets-encrypt certificates to the Fluentd container. In my case I am using cert-manager to manage my certificates. So I just need to add the following lines to my Fluentd deployment:

      volumes:
        - name: letsencrypt
          secret:
            secretName: Fluentd-tls

I am omitting the rest of the Fluentd deployment config for brevity. But if you have any questions, feel free to contact me on twitter.

Checking if the certificates have been renewed

Now that we have configured Fluentd to use lets-encrypt certificates, we need to monitor the certificates for renewal. I wrote a small go application for this purpose. The application is pretty simple. It checks if the certificate served by the Fluentd server has a different expiration date than the certificate created by cert-manager.

I deployed the application as a cronJob in my Kubernetes cluster with a once a day schedule.

Full application code is available on GitHub

An alternative to this approach might be to deploy the service as a kubernetes deployment and watch for changes to the certificate CRD. However, I didn’t want to add another long running service to my cluster just for this purpose. IMO, a cronJob is a much better fit for this use case.

The application is pretty simple. It has a main function that is executed once a day. The main function will call the checkCert function to get the expiration date of the certificate served by the Fluentd server. Then it will call the getCRD function to get the expiration date of the certificate created by cert-manager. Finally, it will compare the two dates and if they are different it will call the reloadFluentdConfig function trigger Fluentd to do a graceful config reload.

Checking the expiration date of a server certificate using the go tls package is pretty straightforward:

// get the expiration date of the certificate served by the Fluentd server
func checkCert(serviceURL string) (time.Time, error) {
	conn, err := tls.Dial("tcp", fmt.Sprintf("%s:443", serviceURL), nil)
	if err != nil {
		return time.Time{}, fmt.Errorf("Server doesn't support SSL certificate err: %w", err)
	}

	err = conn.VerifyHostname(serviceURL)
	if err != nil {
		return time.Time{}, fmt.Errorf("Hostname doesn't match with certificate: %w", err)
	}
	expiry := conn.ConnectionState().PeerCertificates[0].NotAfter
	log.Printf("Issuer: %s\nExpiry: %v\n", conn.ConnectionState().PeerCertificates[0].Issuer, expiry.Format(time.RFC850))

	return expiry, nil
}

The function checkCert will return the expiration date of the current certificate served by the Fluentd server or an error if there is a problem with the server / certificate.

The following code will fetch the certificate CRD (Custom Resource Definition) object created by cert-manager from the Kubernetes API. The CRD object contains the expiration date of the certificate.

func (a app) getCRD() (cmapi.Certificate, error) {
	certificates := cmapi.CertificateList{}
	uri := fmt.Sprintf("/apis/cert-manager.io/v1/namespaces/%s/certificates", a.namespace)
	err := a.client.RESTClient().Get().RequestURI(uri).Do(context.Background()).Into(&certificates)
	if err != nil {
		return cmapi.Certificate{}, fmt.Errorf("failed to get certificates: %w", err)
	}

	for _, cert := range certificates.Items {
		if strings.EqualFold(cert.Name, a.certName) {
			return cert, nil
		}

		log.Printf("Certificate %s is not Fluentd cerificate", cert.Name)
	}

	return cmapi.Certificate{}, fmt.Errorf("failed to find Fluentd certificate")
}

Now that I have the expiration dates I can compare them. Having looked at the cert-manager code, I found that the certificate CRD object has a helper function for comparing expiration dates. So I am going to use that function for the comparison:

t := metav1.NewTime(expiry)
if certificate.Status.NotAfter.Equal(&t) {
  log.Println("The certificates are the same. No need to reload Fluentd config")

  return
}

Reloading the certificates (Fluentd config)

Now that I have the code for checking the expiration dates, I can reload the certificates if they are different. Fluentd has a built-in API for reloading the config. Instead of killing the running pod we can just call the API and Fluentd will reload the config.

First we need to enable the RPC API. This is done by adding the following lines to the Fluentd config:

    <system>
      rpc_endpoint 0.0.0.0:24444
    </system>

Now we can call the API to reload the config:

func reloadFluentdConfig(ips ...string) error {
	for _, ip := range ips {
		log.Println("Reloading Fluentd config on", ip)

		url := fmt.Sprintf("http://%s:24444/api/config.gracefulReload", ip)
		req, err := http.NewRequest("GET", url, nil)
		if err != nil {
			return fmt.Errorf("failed to create request: %w", err)
		}

		client := &http.Client{
			Timeout: 5 * time.Second,
		}
		resp, err := client.Do(req)
		if err != nil {
			return fmt.Errorf("failed to send request: %w", err)
		}
		defer resp.Body.Close()

		// check response
		if resp.StatusCode >= 400 {
			return fmt.Errorf("failed to reload Fluentd config: %s", resp.Status)
		}

		b, err := io.ReadAll(resp.Body)
		if err != nil {
			return fmt.Errorf("failed to read response body: %w", err)
		}

		log.Printf("Response: %s", string(b))
	}

	return nil
}

Conclusion

You can find the full code for the application on GitHub. I hope you found this article useful. If you have any questions, feel free to contact me on twitter @bobby_donchev

References

Working with PostgreSQL in Go using pgx

Sat, 14 Jan 2023 00:00:00 +0000

Introduction

Having had the chance recently to work with SQL again after years of NoSQL (DynamoDB, MongoDB, Redis, CosmosDB), I was surprised to see how much I missed it.

The Project I am working on uses Go and even tough Go has a couple of good ORMs (ent) and bad ones (gorm), I decided to go with the native PostgreSQL driver pgx. Why? Because I needed all the performance I could get and pgx provides some nice features like:

prepared statements
connection pooling
binary decoding - pgx can decode query results directly into Go structs, which can be faster and more convenient than manually parsing rows.
feature rich - pgx supports a wide range of PostgreSQL features, including notifications, large objects, and COPY.

The only downside is the somewhat lack of documentation. I had to go through GitHub issues and read the source code to understand the API. Having invested the time to learn it, I thought it would be nice to share my experience.

In this post I’ll write down some of the things I learned while working with pgx. I will try to keep it as simple as possible, but I will also try to cover some of the more advanced features.

Specifically, I will cover the following topics:

Connection pool setup
Inserts
Bulk inserts
Querying and parsing results into Go structs

Connection pool setup

Setting up a connection pool is the recommended way when we are working with DBs. Instead of creating a new connection for each query a connection pool will reuse connections. To set up a connection pool I need to call pgxpool.New and pass it a connection string:

package pg

import (
	"context"
	"fmt"
	"sync"

	"github.com/jackc/pgx/v5/pgxpool"
)

type postgres struct {
	db *pgxpool.Pool
}

var (
	pgInstance *postgres
	pgOnce     sync.Once
)

func NewPG(ctx context.Context, connString string) (*postgres, error) {
	pgOnce.Do(func() {
		db, err := pgxpool.New(ctx, connString)
		if err != nil {
			return fmt.Errorf("unable to create connection pool: %w", err)
		}

		pgInstance = &postgres{db}
	})

	return pgInstance, nil
}

func (pg *postgres) Ping(ctx context.Context) error {
	return pg.db.Ping(ctx)
}

func (pg *postgres) Close() {
	pg.db.Close()
}

Note that I am using a singleton pattern to make sure that I only have one connection pool. This is not strictly necessary, but it is a good practice to avoid creating multiple connection pools.

Also note that I like to put my database connection code in a separate package. This way I can reuse it in other projects.

Inserting data

We will cover three ways of inserting data. Single row inserts, bulk inserts and COPY inserts.

Single row inserts

Inserting a single row is pretty straightforward. Having initialized a connection pool, we can call Exec on it and pass it a SQL query and the values we want to insert:


func (pg *postgres) InsertUser(ctx context.Context) error {
  query := `INSERT INTO users (name, email) VALUES (@userName, @userEmail)`
  args := pgx.NamedArgs{
    "userName": "Bobby",
    "userEmail": "bobby@donchev.is",
  }
  _, err := pg.db.Exec(ctx, query, args)
  if err != nil {
    return fmt.Errorf("unable to insert row: %w", err)
  }

  return nil
}

Note that I am using named arguments instead of positional arguments (e.g. $1, $2). This is a good practice because it makes the code more readable and it is also more secure. If you use positional arguments, you can easily make a mistake and pass the arguments in the wrong order. With named arguments, you can’t make this mistake.

Bulk inserts

Bulk inserts are a bit more complicated. You need to create a pgx.Batch and add all the queries you want to execute to it. Then you need to call SendBatch on the connection pool and pass it the batch. This will return a pgx.BatchResults object. You can then iterate over the results and check for errors:

func (pg *postgres) BulkInsertUsers(ctx context.Context, users []model.User) error {
  query := `INSERT INTO users (name, email) VALUES (@userName, @userEmail)`

  batch := &pgx.Batch{}
  for _, user := range users {
    args := pgx.NamedArgs{
      "userName": user.Name,
      "userEmail": user.Email,
    }
    batch.Queue(query, args)
  }
  
  results := pg.db.SendBatch(ctx, batch)
  defer results.Close()

  for _, user := range users {
    _, err := results.Exec()
    if err != nil {
      var pgErr *pgconn.PgError
      if errors.As(err, &pgErr) && pgErr.Code == pgerrcode.UniqueViolation {
          log.Printf("user %s already exists", user.Name)
          continue
      }

      return fmt.Errorf("unable to insert row: %w", err)
    }
  }

  return results.Close()
}

This code is a bit more involved but I think it shows a couple nice to knows like how to check for a specific error code and how to use the pgx.Batch struct.

The SendBatch method returns a pgx.BatchResults struct. This struct has a Exec method that is somewhat misleading. It doesn’t actually execute the insert, it just returns the result. I can call Exec multiple times to get the results of all the inserts in the batch. The Close method on the pgx.BatchResults struct is what actually executes the batch. Its idempotent, so I can call it multiple times.

COPY inserts

Bulk inserts a performant way to insert data, but they are not the fastest. If you want to insert a lot of data, you should use the COPY method.

We can do a COPY insert with the CopyFrom method:

func (pg *postgres) CopyInsertUsers(ctx context.Context, users []model.User) error {
  entries := [][]any{}
  columns := []string{"name", "email"}
  tableName := "user"

  for _, user := range users {
    entries = append(entries, []any{user.Name, user.Email})
  }

  _, err := pg.db.CopyFrom(
    ctx,
    pgx.Identifier{tableName},
    columns,
    pgx.CopyFromRows(entries),
  )

  if err != nil {
    return fmt.Errorf("error copying into %s table: %w", tableName, err)
  }

  return nil
}

That isn’t too bad =)

When to use COPY and when to use bulk inserts really for me boils down to wether I need to know if a particular insert failed or not. If I don’t need to know, I will use COPY. If I do need to know, I will use bulk inserts.

Querying data

pgx has a couple of methods for querying data. The most basic one is QueryRow. This method will return a pgx.Row struct. This struct has a Scan method that will scan the result into a struct:

func (pg *postgres) GetUsers(ctx context.Context) ([]model.User, error) {
  query := `SELECT name, email FROM user LIMIT 10`
  
  rows, err := pg.db.Query(ctx, query)
  if err != nil {
    return nil, fmt.Errorf("unable to query users: %w", err)
  }
  defer rows.Close()

  users := []model.User{}
  for rows.Next() {
    user := model.User{}
    err := rows.Scan(&user.Name, &user.Email)
    if err != nil {
      return nil, fmt.Errorf("unable to scan row: %w", err)
    }
    users = append(users, user)
  }

  return users, nil
}

However, what I found out is that since v5 we have generics and can do this instead:

func (pg *postgres) GetUser(ctx context.Context, name string) ([]model.User, error) {
  query := `SELECT name, email FROM user LIMIT 10`

  rows, err := pg.db.Query(ctx, query)
  if err != nil {
    return nil, fmt.Errorf("unable to query users: %w", err)
  }
  defer rows.Close()

  return pgx.CollectRows(rows, pgx.RowToStructByName[model.User])
}

This is a lot simpler and it also has the advantage of not having to know the column names in advance. It will just scan the result into the struct.

Doing SQL in Go got a lot of hate in the past because of interface{} and manually scanning the result into a struct. But with pgx v5, this is no longer the case. I think that libraries like sqlx and scany are great but not necessary anymore.

Conclusion

We covered a lot of ground in this post. We looked at how to connect to Postgres using pgx.Pool, three different ways how to insert data, how to query and parse data. We also looked at how to use the new pgx v5 features like named arguments and the pgx.CollectRows method.

I hope you enjoyed this post and that you learned something new. If you have any questions or remarks, feel free to ping me on twitter @bobby_donchev.

Thanks.

How to authenticate a GitHub App using python

Wed, 28 Dec 2022 00:00:00 +0000

Introduction

Recently I was working on moving a Bitbucket bot to GitHub. With Bitbucket, I can use a personal access token to authenticate with the API. It seems that this personal access token was valid forever? Which can be a good thing, but also a bad thing. With GitHub you can create a bot account and issue an access token for that account. But this token is only valid for max 1 year. After a brief research I encountered GitHub apps. Which are basically webhooks and bot account with granular permissions installed on a GitHub organization. However, getting the authentication right was a bit tricky. In this post I will try to explain how to authenticate with a GitHub app.

Webhook secret, private key, JWT and access token

When I first started to look into GitHub apps I must admit that I was a bit confused. There are a lot of different keys and tokens. Let me try to explain what they are and how they are used.

Webhook secret

Whenever a GitHub app needs to subscribe to events from one ore multiple repositories I must create a webhook secret. This secret is used to verify that the webhook request is coming from GitHub. This is a good thing, because it prevents other people from sending fake webhook requests to my app. I can create a webhook secret in the GitHub app settings. Lets quickly go over how to verify the webhook request.

I created a simple decorator that I can use to verify the webhook request:

import functools
import hmac
import hashlib
import os

API_SECRET = os.environ["GITHUB_WEBHOOK_SECRET"]

def verify(f):
    @functools.wraps(f)
    def wrapper(req, *args, **kwargs):
        logging.info('Verifying request')
        got = req.headers.get('X-Hub-Signature-256')
        if not got:
            raise VerifyError('No signature found')

        payload = req.get_body()
        want = 'sha256=' + hmac.new(bytes(API_SECRET, 'latin-1'), msg = payload, digestmod = hashlib.sha256).hexdigest()
        isCorrect = hmac.compare_digest(got, want)
        if not isCorrect:
            raise VerifyError()
        logging.info('Request verified')
        
        return f(req, *args, **kwargs)
    return wrapper

The decorator is pretty simple. It gets the X-Hub-Signature-256 header from the request and compares it with the signature that we computed using the webhook secret and payload.

If the signatures match we know that the request is coming from GitHub. If the signatures don’t match we raise an (custom) exception.

Note that the decorator was created for Azure Functions, but it can be easily adapted to other frameworks. This is how I use the decorator in my azure function handler:

@verify
def main(req: func.HttpRequest) -> func.HttpResponse:
    return handler(req)

Private key, JWT and access token

Now that I have verified the origin (GitHub) I can start calling the GitHub API with the APP. In order for me to do this I need to create a JWT (JSON Web Token) and use that to authenticate with the GitHub API. The JWT is created using the private key that I downloaded from the GitHub app settings. After I create a JWT I must use that JWT to get an access token. I can then use this access token to call the GitHub API.

Lets see how we can create a JWT and use that to get an access token.

First you’ll need to install a dependency cryptography:

pip install cryptography

Now we can create a class that can create a JWT and use that to get an access token:

import hmac
import hashlib
import requests
from datetime import datetime, timedelta
from cryptography.hazmat.primitives import serialization

APP_ID='YOUR_APPLICATION_ID'
APP_INSTALLATION_ID='YOUR_APPLICATION_INSTALLATION_ID'


class Auth(object):
    def __init__(self, private_key):
        self.private_key = private_key

    def get_jwt(self):
        due_date = datetime.now() + timedelta(minutes=10) # 10 minutes from now
        expiry = int(due_date.timestamp())
        payload = {
            'iat': int(datetime.now().timestamp() - 60), # 1 minute ago
            'exp': expiry,
            'iss': APP_ID
        }
        priv_rsakey = serialization.load_pem_private_key(self.private_key.encode('utf8'), password=None)

        return jwt.encode(payload, priv_rsakey, algorithm='RS256')

    def get_accesstoken(self):
        token = self.get_jwt()
        resp = requests.post(self.url, headers={'Authorization': f'Bearer {token}'})
        if not resp.ok:
            raise Exception('Failed to get access token')

        return resp.json()['token']

Note that we can only create a JWT that is valid for 10 minutes. Also, because of clock skew, I created the JWT 1 minute ago. This way I can be sure that the JWT is valid when I exchange it for an access token.

The private key that I pass to the Auth class can come from the environment:


import os

private_key = os.environ['GITHUB_PRIVATE_KEY']
auth = Auth(private_key)
access_token = auth.get_accesstoken()

# Use the access token to call the GitHub API

I have stored my private key in KeyVault and load it into the environment variable GITHUB_PRIVATE_KEY. Note that if you store your private key in a .env file or KeyVault you need to add newline chars .e.g \n in order for it to work:

GITHUB_APP_PRIVATE_KEY="-----BEGIN RSA PRIVATE KEY-----
.....
.....
"

GITHUB_APP_PRIVATE_KEY="-----BEGIN RSA PRIVATE KEY-----\n....\n....\n"

Conclusion

As you can see there are a lot of steps involved in authenticating with a GitHub app. I am also only scratching the surface here. Theoretically you can also create an Oauth GitHub app and let users log into your app using their GitHub account and issue API calls on their behalf. But that is a topic for another post.

What surprised me the most and what took me the longest to figure out was the ACCESS TOKEN. I tried using the JWT to call the GitHub API, but that did not work. I had to use the access token. I hope this post will help you get started with GitHub apps. If you have any questions or comments you can reach me on twitter @bobby_donchev.

AWS Lambda invoker identification

Mon, 31 Oct 2022 00:00:00 +0000

Introduction

Edit 2023-07-03: It was brought to my attention that the solution I proposed might benefit from using JSON instead of decoding xml payload.

I am building a CLI tool that will invoke AWS Lambda functions. The tool will be used by developers to create AWS resources. You may ask why not give the developers access to AWS CLI?

Even if I did give them AWS CLI access I cannot guarantee that the developers will adhere to the naming conventions and tagging conventions.

Another added benefit of creating a CLI tool is that developers will get the resources pre-configured with the “right” settings. E.g.

# configures rds with correct name, tags, small instance, no backups etc...
$ dev-tool rds create --test

# same but with larger instance, backups etc...
$ dev-tool rds create --prod

Problem

In AWS I have enabled federated AWS access. Developers assume a role and get CLI / Console read access to the AWS account. But what if the developer needs to create secrets, RDS instances, provision state machines etc…?

Thats where the CLI tool comes in. The CLI tool invokes AWS Lambda functions that creates the resources. But how do you know who invoked the Lambda function? How do you know if the developer is allowed to create production resources?

Solution

In the CLI I pre-sign the get-caller-identity call. The presigned URL is then passed to the Lambda functions. I created a lambda middleware that calls the presigned URL and gets the caller identity. Does authentication and authorization and if everything is OK it passes the user information via the context to the Lambda functions.

The solution is similar to how EKS IAM works. It is actually a very simple solution. But I could not find any documentation on how to do it. So I thought I would share it.

package auth

import (
	"context"
	"encoding/json"
	"fmt"
	"io"
	"net/http"
	"net/url"
	"regexp"
	"strings"

	"github.com/aws/aws-sdk-go-v2/aws"
	"github.com/aws/aws-sdk-go-v2/service/sts"
)


type getCallerIdentityResponse struct {
	GetCallerIdentityResult struct {
		Arn     string `json:"Arn,omitempty"`
		UserId  string `json:"UserId,omitempty"`
		Account string `json:"Account,omitempty"`
	} `json:"GetCallerIdentityResult,omitempty"`
	ResponseMetadata struct {
		RequestId string `json:"RequestId,omitempty"`
	} `json:"ResponseMetadata,omitempty"`
}

type IAMService struct {
	aws *sts.Client
	cfg aws.Config
}

func NewIAMService(cfg aws.Config) IAMService {
	return IAMService{
		aws: sts.NewFromConfig(cfg),
		cfg: cfg,
	}
}

func (iam IAMService) GetCallerIdentity(ctx context.Context) (data.Idenity, error) {
	sign := sts.NewPresignClient(iam.aws)
	identity, err := sign.PresignGetCallerIdentity(ctx, &sts.GetCallerIdentityInput{})
	if err != nil {
		return data.Idenity{}, fmt.Errorf("could not get caller identity: %w", err)
	}

	url, err := url.Parse(identity.URL)
	if err != nil {
		return data.Idenity{}, fmt.Errorf("could not parse url: %w", err)
	}

        // only send the query params
	return data.Idenity{
		Authentication: url.RawQuery,
	}, nil
}

func (iam IAMService) VerifyIdentity(ctx context.Context, id data.Idenity) (data.Tenant, error) {
        // create the URL yourself. Do not trust the URL from the client
	authURL := fmt.Sprintf("https://sts.%s.amazonaws.com/?%s", iam.cfg.Region, id.Authentication)
	url, err := url.Parse(authURL)
	if err != nil {
		return fmt.Errorf("could not parse url: %w", err)
	}

	httpReq := &http.Request{
		Method: "GET",
		URL:    url,
		Header: http.Header{
			"Accept": []string{"application/json"},
		},
	}

	response, err := http.DefaultClient.Do(httpReq)
	if err != nil {
		return fmt.Errorf("could not send request: %w", err)
	}

	body, err := io.ReadAll(response.Body)
	if err != nil {
		return fmt.Errorf("could not read response body: %w", err)
	}

	resp := getCallerIdentityResponse{}
	err = json.Unmarshal(body, &resp)
	if err != nil {
		return fmt.Errorf("could not unmarshal response body: %w", err)
	}
  
        // do authentication and authorization here
        // the resp.GetCallerIdentityResult object has accountID, userID, the role assumed etc...

	return nil
}

The GetCallerIdentity function is called by the CLI tool. The VerifyIdentity function is called by the Lambda middleware.

IMPORTANT

The PresignGetCallerIdentity from the AWS STS SDK will return an URL. Do not trust the URL from the client. Create the URL yourself. The client can manipulate the URL and send you a URL that points to a different website.

Also do your own authentication / authorization checks in the VerifyIdentity function. You have access to the accountID, userID, the role assumed etc… You can do whatever you want with that information.

Conclusion

I hope this helps someone. I could not find any documentation on how to do this. So I thought I would share it. As always If you have any questions or comments you can reach me on twitter @bobby_donchev.

State machine using MongoDB changestreams

Fri, 21 Oct 2022 00:00:00 +0000

Introduction

I worked on a project where we needed to implement a process that spans multiple days. The process is dependent on the internet connectivity of our customers IoT device.

Usually I would go for something like temporal.io, Azure Durable Entities or AWS Step Functions. But incorporating new infrastructure is always a challenge and the project wanted to keep the infrastructure as simple as possible. Meaning re-using existing infrastructure.

That narrowed the options down to Kubernetes and MongoDB =)

Building a state machine with MongoDB and Kubernetes

In this post I will show how to build a state machine using MongoDB changestreams and Kubernetes. The state machine will be used to track the progress of a process that spans multiple days.

Each state is represented as a column in a single document. The document is updated by the state machine when the process moves to the next state.

Each state in the state machine is implemented as a Kubernetes statefulset. Why statefulset? Because we need to store a resume token in a durable place. The resume token is used to resume the changestream when the statefulset is restarted (update or crash).

Process manager

I need a process manager that takes in my state business logic and executes it. The process manager needs to keep track of the resume token and make sure its persisted in case of a crash or update. This is what I came up with:

import fs from 'fs';
import { ChangeStreamDocument, Document, MongoDatabase } from '@lib/mongo';
import { config } from '@lib/config';
import log from '@lib/logger';

interface workInterface {
  (item: ChangeStreamDocument<Document>): Promise<void>;
}

export class Process {
  private filter: any[];
  private mongo: MongoDatabase;
  private isShutdown = false;
  private lastEvent: { init: boolean; token: unknown };
  private cursorLocation: string;
  private work: workInterface;

  constructor(filter: any[], cursorFileName: string, work: workInterface) {
    this.filter = filter;
    this.work = work;
    this.mongo = new MongoDatabase(config());
    this.cursorLocation = '/data/' + cursorFileName;
    try {
      this.lastEvent = JSON.parse(fs.readFileSync(this.cursorLocation).toString());
    } catch {
      this.lastEvent = { init: true, token: {} };
    }
  }

  private async start() {
    await this.mongo.initDatabase();
    const cursor = this.mongo
      .getDb()
      .collection(config().mongodb.downgradeCollection)
      .watch(this.filter, this.lastEvent.init ? {} : { resumeAfter: this.lastEvent.token });

    try {
      while (!this.isShutdown) {
        const data = await cursor.next();
        this.lastEvent.token = data._id;
        await this.work(data);
      }
    } catch (err) {
      log.error(err);
      fs.writeFileSync(this.cursorLocation, JSON.stringify(this.lastEvent));
      process.exit(1);
    }
  }

  public async run() {
    const signals = ['SIGTERM', 'SIGINT', 'SIGQUIT'] as const;
    signals.forEach((signal) => {
      process.once(signal, () => {
        this.isShutdown = true;
        if (Object.keys(this.lastEvent.token).length > 0) {
          this.lastEvent.init = false;
        }
        fs.writeFileSync(this.cursorLocation, JSON.stringify(this.lastEvent));
        log.info(`Received ${signal}, will resume after ${JSON.stringify(this.lastEvent)}`);
        process.exit(0);
      });
    });
    await this.start();
  }
}

The process manager takes in a filter, a cursor file name and a work function. The work function is the business logic that is executed when a document is updated. The process manager listens to termination signals and saves the resume token to a file. The resume token is used to resume the changestream when the process is restarted.

In case of an error in the work function the process manager will save the resume token and exit. This will cause the statefulset to restart and resume the changestream. It is important that you implement your business logic in a way that it can be restarted. In other words, the business logic should be idempotent.

Lets look at an example of how to use the process manager:

import { ChangeStreamDocument, Document, MongoDatabase } from '@lib/mongo';
import { config } from '@lib/config';
import log from '@lib/logger';
import { Process, states } from './process';

const filter = [
  {
    $match: {
      operationType: 'insert',
    },
  },
];

const work = async (data: ChangeStreamDocument<Document>) => {
  const now = new Date().toISOString();
  const { operationType } = data;
  if (operationType !== 'insert') {
    log.warn(`Unexpected operation type: ${operationType}`);
    return;
  }

  const collection = MongoDatabase.getDb().collection(config().mongodb.downgradeCollection);

  const { fullDocument: item } = data;
  if (item.state !== states.started) {
    log.warn(`State is not ${states.started} but ${item.state}`);
    return;
  }

  const { _id, state, user } = item;
  log.info(`Processing ${_id} with state ${state} for user ${user}`);
  
  /. . . business logic . . ./


 // Move to next state
  const nextState = states.downloading;

  await collection.updateOne({ _id }, { $set: { state: nextState, updatedAt: now } });
};

const started = new Process(filter, `started.json`, work);
started.run();

I call this the “started” state and will be deployed as it own statefulset in Kubernetes. The statefulset will be configured to restart on failure.

With this setup I can add as many states as I want. Each state will be deployed as a statefulset in Kubernetes. For my use case I needed 5 states. Each state did one thing and then moved process to the next state.

Conclusion

I hope this post was useful. If you have any questions or comments you can reach me on twitter @bobby_donchev.

Switching from gatsby to hugo

Sat, 08 Oct 2022 00:00:00 +0000

Last year I switched to gatsby and firebase hosting from a self made flask app.

Altough I enjoyed the experience, I found myself updating dependencies and plugins and fixing bugs more often than I would like. I also found myself spending more time on the website than I would like.

What I started appreciating about hugo is that it is a tool, a single binary that helps you build a static website. It is not a framework or a library or plugin. There is a real benefit in that. It is a tool that you can use to build a website, not a framework that you have to use to build a website. Using Gatsby to achieve the same result as hugo requires also using various software dependencies, so I have to keep not only Gatsby but also all those packages maintained, which resulted in a lot of time spent on the website.

I also have to admit that some of the plugins in the Gatsby ecosystem are of low quality. Updating Gatsby sometimes wasn’t possible because a plugin was not compatible with the new version.

With hugo I don’t have these problems.

Templating

One remark I constantly hear is that Gatsby is React based and thus creating a website is easy. But this is only true if you define easy as “familiar”. I wouldn’t say that creating a website with Gatsby or React is simple. You need to learn React, JSX, GraphQL, Gatsby, and all the plugins. Since I know this stack, I can say that it is not simple it’s easy for people with prior knowledge but it is a lot of work to learn for somebody who is new.

Hugo on the other hand is based on HTML, CSS (SCSS), JS and go templates. The only thing you need to learn is go templates (which are very similar to Jinja2 templates which I have used for many years in the past).

The argument of which to choose boils down to:

go templates vs react / jsx / graphql
no npm dependencies vs fixing npm dependencies

For now I choose the “simpler” solution even though I chose the “easier” solution in the past =-)

Kubernetes interview questions

Fri, 21 May 2021 00:00:00 +0000

What have all recent projects (last 5 years) have had in common?

Kubernetes. :)

I have been developing applications and leading teams that developed large systems around Kubernetes. Most of the time the developers didn’t want to know much of the inner workings of kubernetes. However, I believe to be a competent developer, that wants to deliver features ontop of kubernetes you need to be familar with kubernetes concepts.

The last years I have interviewed 20+ developers. In this article I want to note down some of the questions I ask and what answers I look for.

Usually I start with docker. I know I know. You might not agree with me. Why ask docker questions? But I believe it still shows the depth the individual has. So I might start with something simple such as

What is the difference between a VM and docker?

I am fishing for kernel here but talk around cgroup, security (SELinux) are also fine.

Whats the differnece between an image and a container?

Fishing for runtime / package but anything around repository / docker hub or describing the docker build vs docker run commands is fine.

Now we can go with some icebreakers for kubernetes questions :)

Whats the difference between the service types, ClusterIP, NodePort and LoadBalancer

There are many ways to answer this question. It should also lead to a discussion that I can use to gauge the depth of the knowledge. My experience is that people usually don’t know much about NodePorts, so let me answer it here:

You want to use NodePort if you want to provide a port number in addition to the virtual IP provided by the ClusterIP type. This way you can access your backend Pods by using the workers (k8s nodes) IP and the port number.

What happens when a readinessProbe fails?

Fishing for availability, k8s service and how it relates to deployments and the probes.

Correct answer: excludes pods from the Service targets.

Whats the difference between statefulsets and deployments?

I am looking not only for the stateful vs stateless discussion (scaling a deployment with a PVC is painfull). But also the networking advantages statefulsets bring to the table, such as the pods have stable network IDs and predictable names.

Can I use a Role to grant access to all Pods in the cluster?

For this you would need a ClusterRole, so no.

I believe these questions to have low to medium difficuly. If you have an unicorn you can ask something more exotic such as:

What is metadata.finalizers and how do they work?

Async pre-delete hook. Will be called whenever the resources that have them gets deleted.

What happens to PVCs once a statefulset has been deleted?

Will not be autodeleted :)

What happens when you only define resources.limits.memory on a deployment?

resources.requests.memory will be equal to the limits you have defined.

What happens when a container consumes more memory than defined in resources.requests.memory

Becomes a target for eviction. If the pod does not have resources.requests.memory and resources.limits.memory, by defualt they are using more than they requested :)

Does kubernetes evict Pods that run on a worker that is low in CPU?

No. Thus it is important to specify resources.requests.cpu in production.

Switching from flask to gatsby

Wed, 17 Mar 2021 00:00:00 +0000

In 2016 when I started this site I wrote about the tech stack I chose.

Yesterday I switched from servers to serverless using Gatsby and firebase hosting. Feels good to leave the db and containers behind and have gatsby build a static website whenever I make changes.

With firebase I can review all my changes before promoting them to production (https://donchev.is) How?

No, I dont have seperate projects for dev / prod in firebase. I just use their recently released feature “hosting channels” ;)

Here is how:

firebase --project donchev-is hosting:channel:deploy $(uuidgen) --expires 1h


✔  hosting:channel: Channel URL (donchev-is): https://donchev-is--a5894fa1-1982-49ce-9513-e3135fd2e866-nw2h141t.web.app [expires 2021-03-17 14:25:01]

Now I can view my changes on the newly created channel URL

https://donchev-is--a5894fa1-1982-49ce-9513-e3135fd2e866-nw2h141t.web.app

(will probably be expired by the time you read this)

I can even run UI tests on that link if I am too lazy doing the testing myself, here is how I do it:

"HEADLESS=false WEBSITE=https://donchev-is--a5894fa1-1982-49ce-9513-e3135fd2e866-nw2h141t.web.app yarn test:e2e",

this triggers the e2e tests written with jest-puppeteer

const puppeteer = require('puppeteer')

describe('donchev-is', () => {
  beforeAll(async () => {
    jest.setTimeout(15000)
    const website = process.env.WEBSITE || 'https://donchev.is'
    await page.goto(website)
    await page.setViewport({ width: 1712, height: 1283 })
  })

  it('should work :) ', async () => {
    // write your e2e tests here
  })
})

My new dev workflow goes something like this:

Check in code
Run yarn build
Run yarn deploy:dev (creates a firebase hosting channel that expires in 1h)
Run WEBSITE=“hostinChannelUrl” yarn test:e2e
Run yarn deploy:prod

No more servers and no more ansible playbooks for patching my servers ;)

Azure SAS tokens on cloudflare workers

Sat, 23 May 2020 00:00:00 +0000

I have been playing with cloudflare workers a lot recently. I fell in love with the tech. Although you need a different attitude when using cloudflare workers than when building NodeJS apps. For starters, cloudflare workers run as service workers on the edge, which means you can’t use all NodeJS libraries out there.

In my opinion this is also gives cloudflare workers an advantage. We are so used to doing npm install in NodeJS world that we almost forgot how to write a left padding algorithm ourself :)

With cloudflare workers we have more constraints and have to think about the libraries we include carefully. These constraints made me start writing the libraries my applications needed myself instead of relying on npm for everything. And it has been refreshing and fun :)

Just recently I wrote a library that generates Azure SAS tokens.

Azure SAS token is a string that gives your clients temporaily access to Azure storage

Microsoft provides a javascript SDK that can generate SAS tokens but unfortunately it only works for NodeJS and not in the service workers and browsers.

I decided to write a library that can generate SAS tokens in service workers and is compatible with browsers and cloudflare edge workers. For this I used the Web Crypto API. You can view the code on GitHub. I wrote it in typescript, so you can use it with javascript or typescript.

The API of the library is also pretty simple. Just call createBlobSas and use the returned string to store / view files in Azure.

For example, Imagine you have someBlob.txt file stored in azure storage and you want to give somebody access to view the file for five minutes. You could use the following code snippet:

import { createBlobSas } from '@neurocode.io/azure-sas-token'

const expireInMin = 5

const { blobSasUrl } = await createBlobSas({
      accountKey: 'yourStorageAccountKey',
      accountName: 'yourAccountName',
      containerName: 'youStorageContainerName',
      blobName: 'someBlob.txt',
      permissions: 'r',
      expiresOn: new Date(new Date().valueOf() + expireInMin * 60 * 1000)
})

As of now the library only issues Blob SAS tokens. If you need the capability to issue container and account SAS token. Let me know! Or event better help out by contributing a PR on GitHub.

Prometheus with Azure Monitor

Thu, 08 Aug 2019 00:00:00 +0000

I have been using Azure AKS for quite some time now and haven’t had many problems with it. I run my own observability stack (Prometheus + Grafana) and logging stack (EFK) on AKS myself. I recently noticed on Azures blog that I can ditch my Prometheus from inside kubernetes and have azure take care of scraping and storing the application metrics. If you have been using Prometheus with AKS than there is not much that will change as a developer. If you, however, have been using Prometheus for DevOps related work and are quite good at writing PromQL than embrace yourself for KQL (Kusto Query Language)

<Queries>
InsightsMetrics
| where Name == "user_management"
| extend dimensions=parse_json(Tags)
| where request_status == "fail"
| where TimeGenerated > todatetime('2019-08-02T09:40:00.000')
| where TimeGenerated < todatetime('2019-08-02T09:54:00.000')
| project request_status, Num, TimeGenerated | render timechart

Note that this feature is still in preview and might change in the future. For all I know maybe Microsoft decides to support PromQL as well as KQL :)

Implementation

In order to try this feature, you’ll need an AKS cluster with OMS agent running version ciprod07092019 or above. Check if you have OMS agent in your AKS cluster and the version by running the following command

k -n kube-system get ds omsagent -o yaml | grep image:
image: mcr.microsoft.com/azuremonitor/containerinsights/ciprod:ciprod07092019

if you get an output with something similar to the above you are good to go. If not, you will need to enable OMS agent on your AKS cluster:

az aks enable-addons -a monitoring -n MyExistingManagedCluster -g MyExistingManagedClusterRG

I will write a follow-up post where I go into details of how to setup a production-grade AKS cluster with terraform. So for this post I assume you have a running AKS cluster.

Activation

Now that we have verified that the OMS agent is running we can apply the configuration and have the OMS agent scrape our services:

kubectl apply -f https://raw.githubusercontent.com/microsoft/OMS-docker/ci_feature_prod/Kubernetes/container-azm-ms-agentconfig.yaml

The OMS agent should restart automatically by running the above command. Also, make sure you have enabled Prometheus metrics by annotating your services:

apiVersion: v1
kind: Service
metadata:
  name: user-management
  namespace: user-domain
  labels:
    app: user-management
  annotations:
    prometheus.io/scrape: 'true'
    prometheus.io/path: /metrics
    prometheus.io/port: '8080'

That should be all. After a minute you can query the service metrics from within azures portal using KQL :)

Streaming kubernetes logs from multiple replicas

Mon, 24 Jun 2019 00:00:00 +0000

I have been really busy these days with kubernetes. Not only managing it but also developing microservices on top of it. One thing that has been particularly annoying is viewing logs in the command line from multiple replicas. First I installed stern and then I tried kubetail. Both are fine but require me to remember some commands on top of all the commands I have to remember using kubectl.

I was reading this document the other day and noticed the paragraph:

Begin streaming the logs from all containers in pods defined by label app=nginx

kubectl logs -f -l app=nginx --all-containers=true

I am not sure when this feature was added but I wanted to try it out and see if it was working the way I expected it. So I tried it out on one of the microservices I am working on.

I have a microservice called user-information it is of kind deployment and has the replica count set to 3. I want to stream all pod logs of this service so I tried:

kubectl logs -f --namespace user-information -l app=user-information

and see there all logs from all pods displayed in my console! Nice. No more external tools, just plain kubectl and grep

Bonus

If your services are structured and all of them have labels (which they should) you can further simplify the above command by adding a function to your ~/.functions file. Open the ~/.functionsfile with your preferred editor and paste the following block inside it:

logs() {
  kubectl logs -f -n $1 -l app=$1 --max-log-requests 5 --all-containers=true --timestamps=true
}

in a new terminal I can then simply view the logs with:

logs user-information

Caveats

Unfortunately, this works only on deployments / statefulSets with replication count <= 5. If you try the above and encounter this

error: you are attempting to follow 6 log streams, but maximum allowed concurrency is 5

You might need to resort to something else then kubectl.

Kubernetes in a private AWS network

Sat, 05 May 2018 00:00:00 +0000

I have been working on setting up a kubernetes cluster on AWS. Usually, the setup isn’t difficult and there are many tools that can assist, for example, kops, spray, conjure-up and probably many others I am forgetting. The problem I had with these tools is that they are configured to create all the resources a kubernetes cluster might need. For example, trying to use kops to create a cluster in a private subnet fails if no Internet gateway exists, thus it will try to create an internet gateway. But what if you are in a corporate environment and no internet gateway has been provisioned? What if the internet breakout should go through your own data center? Now your options are limited to:

Setting up the cluster manually (hard way)
Dig deep into the kops / spray code and modify it to do what you need

Obviously, both options are time-consuming. What if there is another way? The semi-automated way. Actually, the creators of kubernetes thought about the case where flexibility and customizability are needed. That’s why they gave us kubeadm currently my go-to tool for provisioning and managing kubernetes clusters. I call it the semi-automated way because I have to ssh into the master / nodes and issue kubeadm commands and write some config files but with a little bash scripting and terraform knowledge it’s rather easy to automate everything.

I already wrote the automation for you and you can get started by cloning my repo. I’ll explain briefly in this post how you can use the code.

Replace the ssh keys in the ssh/ folder with your own or generate new ones.
Take a look at input.tf and change it to your liking. If you want to install Kubernetes 1.10.1, just change the version variable. Obviously here you will need to define the proxy you want your kubernetes cluster to run behind. Declare the proxy variable like: http://PROXY:PORT
Go into the state folder and provision the resources there, e.g. cd state/ && terraform init && terraform plan / apply
Go back and provision the kubernetes cluster with terraform, e.g. terraform init && terraform plan && terraform apply
The previous step should give you the master node IP, (actually the LB IP). Use the IP to configure kubectl

In the background what the code does is utilize redsocks to forward all the traffic from the containers / host through your corporate proxy. Redsocks will not route internal cluster traffic through your proxy. The traffic rules are configured here if you want to have a look.

Using redoscks is essential in this setup. If you decide not to use it you will need to configure every node and every application / container with proxy configurations. This will quickly lead to maintenance nightmares and hard to debug networking problems.

Cavets

The code assumes you are provisioning a kubernetes cluster with Ubuntu 16.04 as host OS. Further, it assumes that you already have the networking provisioned (VPC, subnets…)

It can also be made better by adding additional checks. It would also be nice to have a choice of CNI, currently it supports only flannel.

Feel free to copy and edit as you like.

Docker for developers: part 2

Fri, 02 Feb 2018 00:00:00 +0000

In a previous post, I wrote about some useful commands developers should know when using docker as a development environment. In this post, I’ll add two more topics, namely multi-stage builds and permission problems with host mounted volumes. Let’s start with docker multi-stage builds.

Docker multi-stage builds

In docker version 17.05 multi-stage builds were introduced which simplified life for us who used docker in production and testing. Back when applications, where deployed to production servers over SCP or similar, production servers were kept lean. Only packages related to the apps where added to the OS and nothing else. Keeping the production OS lean was a way of reducing the attacker surface and thus smaller probabilities of vulnerabilities.

Similar to that logic when you deploy containers to production you want the images to be as lean as possible. Build-essentials in the container? No way.

However, some packages require that you have compilers installed when doing package installs. The way to solve this puzzle prior to docker multi-stage builds was to have two separate Dockerfiles. One for building the binaries and one for running the actual application. With docker multi-stage builds you can now combine the two dockerfiles in one and you will end up with a lean image for production. An example might be useful to show this

Clone my GitHub repo to participate or just look at the code in the repo and read along to better understand the concept.

The repo consists of a fairly simple JAVA app and one test. I’ll use Maven for building the app, for dependencies take a look at the pom.xml file.

Now, let us take a look at the Dockerfile in the repo:

FROM maven:3.5-jdk-8 AS builder

WORKDIR /opt/hello_world

COPY pom.xml .
RUN mvn install

COPY . .
RUN mvn package

FROM openjdk:8-jre-alpine3.7
WORKDIR /usr/src/myapp
COPY --from=builder /opt/hello_world/target .

RUN addgroup -S java && adduser -S -g java appuser
USER appuser

CMD ["java", "-jar", "hello-world-app-0.1.0.jar"]

We start by declaring the builder image, in this case, the maven:3.5-jdk-8 image. This image will be used to build the code and as such only used as an intermediate step. After maven has done installing dependencies and packaging the app in a jar file this image will exit and we move on to the actual image that we will use in production for running our app. This production image copies the jar file from the previous image (COPY –from=builder /opt/hello_world/target .) and runs our app.

When we build the image:

docker build -t simpleapp .

we can check the size of the image:

docker image ls | grep simpleapp | awk '{ print $8 "         " }'
83MB

The image is only 83MB!

If you change the source code in src/main/java/hello/Greeter.java from “Hello World!” to “Hello World!!!!!” and rebuild the image, the rebuilding process only takes 3 secs. Can you spot why it is faster this time? Cache If no, you should re-read my previous post :)

Docker multi-stage builds caveats

If we change the source code a couple of times and build the image after each change we will get many orphaned (dangling) images. For example, I changed the Greeter.java function three times and build the corresponding image three times. If I list the images on my laptop I get:

docker images
REPOSITORY            TAG               IMAGE ID            CREATED             SIZE
simpleapp             latest            057e18566a5c        About an hour ago   83MB
none                  none              396fd17d8ba6        About an hour ago   765MB
none                  none              899f7cde139c        About an hour ago   83MB
none                  none              d28bf47eed96        About an hour ago   765MB
none                  none              c2ebfd088f1d        About an hour ago   83MB
none                  none              8e88d370e323        About an hour ago   765MB`

I can see that I have 5 dangling images (maked none). I need to remove them manually and the caveat with removing them manually is that caching stops to work and the next build will take 3-4 minutes :/

Something you should definitely keep in mind when using multi-stage builds.

Docker volumes and file permission

Having worked with docker in many development environments I have encountered many problems with user privileges and the file system. Usually, the problems are caused because the developers are sharing code folders from the hosts to the container. The main problems I have seen can be categorized as two:

When writing to a docker volume you won’t be able to access the files that the container has written because the process in the container usually runs as root.
You shouldn’t run the process inside as root but even when you hard-code a user it still won’t match the user in your team/jenkins/staging.

Docker provides a –user flag with it’s run command. You can switch to a specified UID during container start like so:

docker run --rm -i -t --user $( id -u $USER ):$( id -g $USER ) simpleapp /bin/sh

This approach works 80% of the time but has some drawbacks. The biggest drawback being that the UID is not present in /etc/passwd file. I know the linux filesystem only cares about the UID and not the username BUT some applications will refuse to start if the user is not present in /etc/passwd

You will get an error like the following:

docker: Error response from daemon: linux spec user: unable to find user uidn6484: no matching entries in passwd file.

An improved approach would be to mount the /etc/passwd and /etc/group files into your container! Like so:

docker run --rm - i -t -v /etc/passwd:/etc/passwd:ro -v /etc/group:/etc/group:ro  --user $( id -u $USER ):$( id -g $USER ) simpleapp /bin/sh

The same solution can be applied to docker-compose files! Now the whole team can work with the same containers!

Docker for developers

Sat, 20 Jan 2018 00:00:00 +0000

With the rise of microservices, the complexity of dev environments has increased. Technology stacks can be used independently and thus the choices of programming languages have increased. Companies used to be proficient in one language and implement their solutions using the tools/libraries available to them in the specific language in a monolithic way. Today with containers it’s totally different. If you are a Ruby on Rails shop but need to implement a real-time service you might dip into NodeJS. Or you are a PHP shop and need to implement a high traffic service you might look into Golang. Or you have been using python2 but want the new service to use asyncio (python3). Now imagine the time it will take onboarding developers and setting up their dev environments. That can be quite time-consuming for the developer and for you! So what is the solution? Docker of course :)

Usually, docker is utilized for CI (building / testing / QA..etc) but I think that every project should begin with a Dockerfile and a docker-compose file. That way when somebody new comes and joins the team he would just need to do a docker-compose up to start his work. NodeJS 6 or 8? No problem just define it in the Dockerfile and the developer doesn’t have to bother with installing anything.

For optimum results, the developer has to know some useful docker commands. In this post, I intend to go over some of the good to know docker commands that I have found myself using on a regular basis while using docker as my development environment.

Spawning a shell inside a container

So you have your containers up and running with docker-compose up but need to drop into the postgreSQL container to issue some psql commands. If your postgreSQL container name is db you can accomplish this with

docker-compose exec db /bin/bash

this works only if the container db is running. To view running containers, do

docker container ls

if you have an image you pulled from the docker hub and want to take a look at the filesystem of this image, you can

docker run --rm -i -t --name db postgres:10 /bin/bash

–rm remove the container after exit
-i keep STDIN open
-t allocate a pseudo-tty
–name name the container

When you run the above command in your shell you will notice that after pulling the image docker starts to execute a bunch of different commands and only in the end of it all you’ll get a bash shell. The reason for this behavior is because the image (postgres:10) was built with an entrypoint script. The entrypoint script allows the creators of the postgres image to configure a container to run as an executable. But what if you don’t want this behavior, what if you just want to look at the files inside the image and the configuration of postgres? Don’t worry you can overwrite this behavior:

docker run --rm -i -t --name db --entrypoint /bin/bash postgres:10

the –entrypoint argument in the docker run command overwrites the entrypoint script used in the official image.

Copying a file from a docker container

There might be some files in your running docker container that you might want to keep. For example, you might decide to keep the postgresql.conf file on your laptop for later reference maybe? Executing a shell inside the postgres container everytime you want to look/change the postgresql.conf isn’t fun so why not just copy it to your laptop?

docker cp db:/var/lib/postgresql/data/postgresql.conf .

This command will copy the config file from the running container to your current directory. Neat, right? :)

Debugging docker builds

In a previous post of mine. I mentioned that when I build docker images I sometimes like to start from the shell and issue the build commands manually and afterward save the running container to an image via docker commit. What I have been doing recently is slightly different but still the same principle. I start with a Dockerfile that I build and if the build fails I will drop into the container to view the filesystem and correct the error and commit the container to an image. An example might be suitable here, imagine the following Dockerfile:

FROM python:alpine3.7

RUN pip install psycopg2
CMD ["python"]

building this Dockerfile will fail

Sending build context to Docker daemon  2.048kB
Step 1/3 : FROM python:alpine3.7
 ---> 4b00a94b6f26
Step 2/3 : RUN pip install psycopg2
 ---> Running in 8a2acbca6301
Collecting psycopg2
  Downloading psycopg2-2.7.3.2.tar.gz (425kB)
    Complete output from command python setup.py egg_info:
    running egg_info
    creating pip-egg-info/psycopg2.egg-info
    writing pip-egg-info/psycopg2.egg-info/PKG-INFO
    writing dependency_links to pip-egg-info/psycopg2.egg-info/dependency_links.txt
    writing top-level names to pip-egg-info/psycopg2.egg-info/top_level.txt
    writing manifest file 'pip-egg-info/psycopg2.egg-info/SOURCES.txt'
    Error: pg_config executable not found.
    
    Please add the directory containing pg_config to the PATH
    or specify the full executable path with the option:
    
        python setup.py build_ext --pg-config /path/to/pg_config build ...
    
    or with the pg_config option in 'setup.cfg'.
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-n3mrzydh/psycopg2/
The command '/bin/sh -c pip install psycopg2' returned a non-zero code: 1

The error is because pg_config wasn’t found. We can have a look at the filesystem of this build with:

docker run --rm -i -t 4b00a94b6f26 /bin/sh

4b00a94b6f26 is the filesystem layer just before the error occured. By opening a shell in this layer we can examine the filesystem and look for clues if pg_config is installed:

~$ find / -name pg_config
~$ 
~$ apk add postgresql-dev 
(1/9) Installing libressl2.6-libtls (2.6.3-r0)
(2/9) Installing pkgconf (1.3.10-r0)
(3/9) Installing libressl-dev (2.6.3-r0)
(4/9) Installing db (5.3.28-r0)
(5/9) Installing libsasl (2.1.26-r11)
(6/9) Installing libldap (2.4.45-r3)
(7/9) Installing libpq (10.1-r1)
(8/9) Installing postgresql-libs (10.1-r1)
(9/9) Installing postgresql-dev (10.1-r1)
Executing busybox-1.27.2-r7.trigger OK: 56 MiB in 44 packages 
~$ find / -name pg_config
/usr/bin/pg_config

Add dependencies before copying your code

Docker uses caches when building / rebuilding images. You should utilize the caching mechanism as much as possible. Instead of doing:

FROM python:alpine3.7
COPY . .
RUN pip install -r requirements.txt

DO THIS:

FROM python:alpine3.7
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .

The latter will create 3 layers. If your dependencies don’t change between builds then the first two layers will be executed from cache! Although I used python and pip as an example remember that this is also valid for npm, maven builds, ruby gems..etc..

This post is already quite long. In a future post, I will discuss two more topics, namely gotchas and caveats around multi-stage builds and permission problems around host mounted volumes.

When to use DynamoDB and when not to

Fri, 22 Dec 2017 00:00:00 +0000

Having seen projects that use DynamoDB as their persistence layer with good results, I have also seen many projects where DynamoDB wasn’t suitable but was used regardless. Upon asking the question why DynamoDB was chosen the answers were usually unclear and defensive. All good Architects and Engineers should ask themselves before using a new technology what the benefits are, the technology will bring and if these benefits warrant the inclusion of the technology.

In this post, I will list a couple of answers to the simple question “Why did you go with DynamoDB” with my remarks based on my experience with DynamoDB. Let’s start with the most common answer:

We don’t have to think about scaling the system it is done for us

Yes, scaling is done behind the scenes by AWS, for every 10GB of data a new Node is added but what people forget is that when you initially provision 30 RCU/WCU and suddenly your data grows and you have 2-3 Nodes the provisioned RCUs/WCUs is a total of all Nodes. Hence if a query throws a ProvisionedThroughputExceededException you can’t provision more throughput on a Node by Node basis you will have to overprovision your throughput or re-design your data model. Also with the newly introduced auto-scaling feature for provisioned throughput, care must be taken to not exceed budget limits because of a bad initial data model design.

It’s so simple getting started with

I agree that it is simple to get going, you throw data in DynamoDB and that’s that. I can see how it is convenient to prototype apps with DynamoDB instead of messing around with schemas and relations in an RDBMS but please don’t confuse the simplicity of interfacing with DynamoDB with the complexity needed to design a DynamoDB architecture. In the beginning, you will probably only use simple key-value lookups and a handful of scans but later in your project, you will encounter the need for doing more involved queries and as your query needs grow, the more the initial design will bite you in the ass. The reason for this is that most devs don’t know how to model a dataset in DynamoDB. You might say that they can learn it and that’s possibly true but once you designed your data model it is really tricky to change. You can’t add local secondary indexes any more hack you can’t even change the originally defined sort key. At this point you might be saying that DynamoDB supports global secondary indexes (GSIs) and that’s true but GSIs are essentially just another table that contains projections of the data that lets you choose different hash/range keys than originally planned. If you are willing to pay double for your data go with GSIs but also remember that the data in the GSIs are not guaranteed to be the latest because DynamoDB uses eventual consistency for projecting the data to the indexes.

Query times are consistent regardless of data amounts

Yes, AWS even states this on their webpage “single-digit millisecond latency at any scale” but the problem is again the design of the data model. You really need to choose a good partition key (primary key) or composition key (partition key and sort key), if you go for the latter, remember you need to include both the partition key and sort key when querying (Although you could use begins_with filter on the sort key and thus wouldn’t need to specify a full value). And don’t forget to use the local secondary indexes you get 5 per table and you can only define them in the beginning.

Takeaway

My takeaway for DynamoDB is to use it in applications with high write load where data access patterns (queries) are known beforehand and don’t change over time. Note that I didn’t say high read load because I am still debating if the “hot key” problem is preventable in the design of the data model or if it’s only preventable with the use of a caching layer in front of DynamoDB. It seems a hard problem and AWS recently introduced DAX an in-memory cache for DynamoDB, probably to help with this problem. Unfortunately, it isn’t available in eu-central-1 (Frankfurt) and only the JAVA and JS SDKs, as of right now, are supported.

As for a final thought checkout this funny but accurate decision tree made by Forrest Brazeal

Aerospike vs Redis vs Tarantool

Tue, 12 Dec 2017 00:00:00 +0000

I have been working on a project where we have to store application data (clicks, views, mentions,…etc) in order to do some analytics later on on the data. There are several ways to architect a solution for this kind of problem and although in the past I architected a similar solution to this problem using Apache Kafka and the Citus extension for PostgreSQL this time we decided to use an in-memory storage engine and implement a batch job that regularly aggregates the data and stores it in PostgreSQL.

The question was which in-memory storage engine should we go for? In this post, I will compare three promising tools and do some performance testing on them in order to better evaluate which one to go with. Before I start I have to mention that I am slightly biased towards Redis because I have used it regularly in the past and know my way around it but after having now worked with both Aerospike and Tarantool I have to say that all of them are worthy candidates.

Test Setup

The tests are run on AWS. I use terraform to provision the infrastructure. The picture below shows an overview of the infrastructure. The tools were provisioned as single instances on their own nodes:

YCSB-tester on a c4.2xlarge (This instance was used to run YCSB tool to perform the tests)

Redis-tester(v. 4.0.6) on a m4.xlarge

Tarantool-tester(v. 1.8) on a m4.xlarge

Aerospike-tester(v. 3.15.0.1) on a m4.xlarge

Elasticache-tester(v. 3.2.4) on a cache.m4.xlarge

YCSB

Yahoo! Cloud Serving Benchmark, or YCSB is a powerful utility for performance measuring of a wide range of databases. It is the de facto tool for benchmarking databases and that’s why I use it here for my tests.

YCSB provides few core workload types that are presented in its own directory as configuration files. There are six major workload types named by letters from A to F.

Workload A is an update-heavy workload. It has a mix of 50/50 reads and writes. An application example is a session store recording recent actions.

Workload B is primarily a read workload. It has a 95/5 read/write mix. Application example: photo tagging; add a tag is an update, but most operations just read tags.

Workload C is 100% read. Application example: user profile cache, where profiles are constructed elsewhere (e.g., Hadoop).

Workload D, new records are inserted and the most recently inserted records are the most popular. Application example: user status updates; people want to read the latest news.

Workload E, short ranges of records are queried instead of individual records. Application example: threaded conversations, where each scan is for the posts in a given thread (assumed to be clustered by thread id).

Workload F, the client will read a record, modify it, and write back the changes. Application example: user database, where user records are read and modified by the user or to record user activity.

On top of the core workloads configurations, YCSB was configured to use 2000000 record counts, 5000000 operation counts, and 100 threads.

Configuration considerations

Tarantool and Aerospike were configured to use only their respective in-memory storages. Log level was set to INFO and wasn’t disabled because I wanted a realistic production comparison. As mentioned before single instances of the tools were run although redis, tarantool and aerospike are capable of being run as multiple instances on the same node (Elasticache?). Another way to push tarantools and aerospike performance would be to attach multiple NICs to the nodes, I didn’t do this in my tests and I didn’t perform any performance tweaks such as IRQ affinity and process affinity.

Since networking will be a real bottleneck in the tests sometimes the tests are run on the tool instances to circumvent the network. However, we don’t plan on running our App on the same server as the in-memory DB but rather in the same VPC. In order to mimic the production setup as best as possible, we decided on placing the YCSB test tool on a sperate instance in a public subnet and the in-memory DBs in a private subnet.

Performing the tests

As mentioned earlier, the code is located on GitHub. If you want to perform the tests yourself you would initiate the infrastructure with the terraform scripts in the infrastructure/ directory. Terraform will ask you for your public IP, it needs the public IP because of the security group, it will add a security group that allows ingress connections to port 22(SSH) and port 2375 (docker) only from the IP you specify.

After terraform has completed the provisioning the infrastructure it will output a message similar to this:

Apply complete! Resources: 19 added, 0 changed, 0 destroyed.

Outputs:

Aerospike-tester = 10.0.16.11
Elasticache-tester = elasticache-ycsb.jkzj1v.0001.euw1.cache.amazonaws.com
Redis-tester = 10.0.16.8
Tarantool-tester = 10.0.16.9
YCSB-tester = 54.171.25.21

Use the IPs from the output to update the ycsb_conf.dat file.

Afterwards, you can run the tests by running:

python ycsb.py run --html=True --stdout=True --config_file='ycsb_conf.dat' --docker_host='tcp://54.171.25.21:2375'

The python script will execute remote docker API calls so that you don’t have to ssh into the YCSB-tester node and do it yourself. It will also output the results to your terminal and write a report HTML file.

After the python script has completed and the results are obtained you can get rid of the infrastructure by running terraform destroy in the infrastructure/ directory.

Results

The results are slightly surprising. It seems either Redis can’t deal with more than roughly 100 threads or the Redis JAVA client included with YCSB has some problems. The results for 100 threads can be seen here and the results for 256 threads here.

We can see that Aerospike and Tarantool are both faster than Redis for Inserts. For workload B, C.. there are no INSERT results because we get those only when loading the data and we only load the data once.

We can also see that the performance degrades when using 256 threads which in my opinion is surprising. After examing the nodes (CPU, Mem, I/O) I think the reason for the degradation is either the YCSB tool or the network.

Conclusions

All of the tools tested passed our requirements of <3ms ops latencies with a throughput higher than 50.000 ops/sec. In the end, the choice boils down to two things:

Ease of use (Learning curve)
Maintainance & Scalability

Redis is already in use in our App so the developers wouldn’t need to learn a new tool. Maintenance is also reduced thanks to AWS managed Redis (Elasticache). Scaling Redis is not as straightforward as scaling Tarantool and Aerospike. Adding nodes can be tricky and requires high knowledge of Redis architecture.

Tarantool isn’t difficult to learn either, it supports SQL commands which is awesome and it can act as a hybrid storage (mem+disk). It supports hot/standby setups and automatic sharding and resharding.

Aerospike is a hybrid DB and has its own query language which is SQL-like. It also comes with a management console and supports automatic sharding and resharding and adding new nodes can be done on a running production cluster.

For a new project, I will definitely be recommending Aerospike or Tarantool but for now the team has to decide :)

AWS Fargate

Thu, 30 Nov 2017 00:00:00 +0000

In re:Invent 2017 AWS announced AWS Fargate among many other new services. AWS describes Fargate as

A technology that allows you to use containers as a fundamental compute primitive without having to manage the underlying instances

Image Source and Credits (AWS Blog)

I have experience with ECS and Kubernetes and the underlying machines haven’t posed a problem for me to manage. In Kubernetes draining a node for maintenance or update is achieved via the CLI

kubectl drain <node name>

This command will tell the kubernetes cluster to mark the node as unschedulable and prevent new pods from arriving. It will also evict running pods and move them to other nodes (except those are static pods which I never use).

After the node is drained you could perform OS upgrades or security patch installations and return the node to the cluster via

kubectl uncordon <node name>

With AWS ECS its a little more involved but still not that difficult. Usually, you would run your ECS cluster with a launch configuration and auto scaling group and the services would be made accessible through a load balancer (AWS ELB). If you wanted to update the underlying ec2 instance to use a newer version of the AWS ECS optimized AMI you would use terraform or CloudFormation to update your launch configuration group and configure it to create new instances before deleting old ones.

With AWS Fargate obviously this burden shifts to AWS and you can focus on your containers (Business Logic) but I have a couple of questions that I haven’t been able to find answers to:

If a container of some customer gets hacked and the hacker obtains escalated host rights what would stop him to shutdown my containers? Or worse break into my container and stealing sensitive data?
- In my ECS clusters I use Sysdig to detect this kind of problems but what does AWS use to keep my containers safe?
The pricing is ambiguous. As of this writing, the official pricing page states both that the pricing is calculated based on the vCPU and memory resources used and that pricing is based on requested vCPU and memory resources. Which one is it? Used or requested? People experienced with Kubernetes know the difference between kubernetes limits vs requests so how are we to interpret these pricings? I guess there is only one way to find out. I have to test it myself :)

Although there are still some questions open with AWS Fargate I still think this service might get popular with companies that use containers. I don’t see companies abandoning AWS ECS totally and switch to Fargate but I can see its use case in CI where I could have different test tasks (Integration, End-End) and perform the tests on Fargate without much Terraform/CloudFormation boilerplate code.

On another note, you should check Google’s recent announcement to cut kubernetes management fees to nil. Now that might pursue me away from AWS at least temporarily :)

Fix slow container shutdown

Mon, 30 Oct 2017 00:00:00 +0000

Today I was working in a dev environment where everything was done within containers and docker-compose was used to define the services. Usually, when I work on my own projects in this way I don’t have to rebuild containers every time I make changes to my code to reflect those changes but in this special case, there was no getting around it. Every time I made changes to the code I had to rebuild the container and run it again. The annoying part was that it took such a long time to re-up the container because the stop process took such a long time. I investigated it and it seems I have been running my apps in the container as the PID 1 process. How does this happen?

There are two ways of running your containers. With CMD or ENTRYPOINT or a combination of both. For example

FROM alpine:3.6

CMD ["ping"]

and

FROM alpine:3.6
ENTRYPOINT ["ping"]
CMD ["8.8.8.8"]

are two ways of defining how your container should later be run.

CMD lets the user that runs the container overwrite the command where ENTRYPOINT does not give this freedom thus making the container a ping only app.

In both cases running the containers, the ping application is run as PID 1 and comes with a bunch of responsibilities and edge cases that we have not anticipated - notably the handling of SIGTERMs.

Lets have a look at the container

docker run --rm -it --name pinger donchev7/alpine-pinger

PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=59 time=10.643 ms
64 bytes from 8.8.8.8: seq=1 ttl=59 time=9.861 ms
64 bytes from 8.8.8.8: seq=2 ttl=59 time=12.448 ms

The donchev7/alpine-pinger image is only 3.97MB, you should have no problem running this container in seconds.

While the container is running issue this command in another terminal window

docker exec pinger ps aux

you should see the following

docker exec pinger ps aux                               
PID   USER     TIME   COMMAND
1     root     0:00   ping 8.8.8.8
7     root     0:00   ps aux

There it is, ping is PID 1

Connecting the dots

When we issue a docker stop command the docker daemon sends a SIGTERM signal to stop our pinger but since it’s PID 1 nothing happens and after a while (10s) the docker daemon loses its patience with the container and sends a SIGKILL. Why does this happen? Obviously, because our application running in the container wasn’t programmed to listen to SIGTERM.

While still having your pinger container running run the following

time docker stop pinger

On my computer it took 10,577 seconds to stop the container. That’s crazy, containers should be lighting fast :)

Solutions

Add STOPSIGNAL SIGINT to your Dockerfile (this is a quick & dirty solution and only takes care of one edge case and shouldn’t actually be used
Avoid being PID 1 when you run ad-hoc docker run commands you can pass the–init flagand your app won’t become PID 1

Let’s test the second solution out. Run the pinger again this time passing the –init flag like so

docker run --rm -it --name pinger --init donchev7/alpine-pinger

and then

time docker stop pinger

Now it takes around 500ms to stop the running container!

Bonus

If you are using docker-compose and a YML version 2.2 or 2.3 you can do the same

version: '2.3'
services:
  pinger:
    image: donchev7/alpine-pinger
    init: true

save this in a docker-compose.yml file and your docker-compose stop command should be lightning fast!

AWS network loadbalancer

Mon, 16 Oct 2017 00:00:00 +0000

The other day I received an email from AWS with their latest announcements. I went through the email and saw that AWS now offers a new kind of ELB namely the Network Load Balancer (NLB). Compared to the ALB and the ELB classic the NLB is a layer 4 load balancer (transport). When I clicked the link, the AWS webpage states the following “The Network Load Balancer for the Elastic Load Balancing service is designed to handle millions of requests per second while maintaining ultra-low latencies” and I thought to myself why didn’t AWS offer this ELB a year ago when I needed a network load balancer :)

A year ago I worked on a project where the requirements were to capture NetFlow traffic and query CPEs (Customer Premises Equipment) of their status through the SNMP protocol. The SNMP part wasn’t that big of a challenge but the NetFlow was more of a challenge. Having 10.000 CPE each with at least 10 interfaces bombing your backend is quite a challenge. One of the requirements was data integrity since the data would be used in a commercial manner. Our initial design consisted of having 5x EC2 instances with NetFlow parsers that would parse and forward (produce) the data to an Apache Kafka cluster. The following picture shows a simplified design

The problem with this design is that you have to guess the number of EC2 instances needed to parse and forward the NetFlow data. The reason we couldn’t use an AutoScalingGroup (ASG) was that in the router itself NetFlow is configured with an IP, for example in Cisco

ipflow-export destination 10.0.0.134 2055

This command would configure a Cisco router to send NetFlow to the EC2 instance with IP 10.0.0.134 on port 2055. The Cisco router can take IPs and hostname records as valid destinations but not DNS names.

What we did was we configured 2000 CPEs per EC2 instance. There was a lot of manual tweaking involved since some CPEs generate more flow data than others.

After reading the announcement from AWS I immediately started digging deeper into their NLB service. As it turns out it can solve all the problems we had. We can position the NLB in front of the EC2 instances and let the NLB handle the routing of the UDP NetFlow data. The NLB also enables us to stop trying to predict the number of required EC2 instances. We can now put the EC2 instances in an ASG and scale in and out with the ingested data! But the biggest win, in my opinion, is the need to manually configure the CPEs with the IPs of EC2 instances. With AWS NLB we get a static IP per availability zone! :=)

I welcome the addition of NLB and think the NLB will enable other people working with IoT or similar customer endpoint devices to abstract a layer out of their architecture and let people focus more on their core problems.

Building and Shipping Docker images

Mon, 12 Jun 2017 00:00:00 +0000

Most of the time we don’t have to deal with building and shipping docker images because the community has already built quite a number of useful images for us. However, if you practice DevOps with Docker (you should) chances are you have to build custom images for your services. Maybe some developers in your organization have a service running NodeJS, maybe others are writing Java APIs, obviously for these services you have to build your own images and today I would like to describe two ways of building and shipping docker images.

Note that I haven’t used the word container in the above paragraph although in many places online the word container is sometimes used interchangeably with the meaning of images. Which is not correct. A Docker container is a running instance of a Docker image. A Docker image can be considered as the “source code” for the container. Docker images are made up of union file systems layered over each other and are build step-by-step using a series of instructions. We will see this in action now by building an image in two ways.

1. Dockerfile

The most common way building Docker images is to define the series of instructions in a Dockerfile. For example, save the following in a file named Dockerfile:

FROM alpine:3.6

RUN apk update && apk add openjdk8


Entrypoint ["java"]

Now we can build this Dockerfile to an image by running:

docker build -t alpine-jdk:8 .

Notice the dot and the space behind the version number 8. This dot tells the docker daemon to look for a Dockerfile in the current directory. If you named the Dockerfile differently you could do:

docker build -t alpine-jdk:8 -f DockerFileName .

To test this image we can spin a one time container off it by running:

docker run --rm -it alpine-jdk:8 -version

This command will run an instance of the alpine-jdk8 image issuing the java version command and exit. After exiting the docker daemon will remove the container. The –rm flag tells the docker daemon to remove the container after running it. You can try to run the container without the –rm flag and with the –name flag giving the container a name, like:

docker run -it --name jdk8 alpine-jdk:8 -version

Now after the version is displayed the container will exit but will not be deleted. We can view the container by running

docker ps -a

and re-run the container

docker start -i jdk8

2. Committing to an Image

The second way we can create a Docker image is by building it interactively from within a base container. If we use the same base image as previously (alpine:3.6) we can build an alpine-java image by running:

docker run -it --name alpine-jdk8 alpine:3.6 /bin/sh

this would drop us into the shell of alpine, from here we can issue the install commands as described in our Dockerfile before:

> apk update
> apk add openjdk8

We can view the changes made to our base file system (alpine 3.6) by running the following command from another shell

docker diff alpine-jdk

C means created, A means Appended Now that we have installed openjdk8 land verified the installation lets save a copy of this container in the form of an image:

docker commit -a YourName alpine-jdk DockerRepo/alpine-jdk8

We can verify the image was created

docker images

should list the DockerRepo/alpine-jdk8 image we just created. We can exit the running container where we installed jdk8 by pressing Ctrl+d on the keyboard.

Some of you might think why would I need to interactively make a Docker image, why not just write the steps to a Dockerfile?

Don’t get me wrong you should be writing all your Docker images in Dockerfiles and keep them in version control, however sometimes when I build Docker images based off alpine-linux I receive strange errors and in order to debug these errors I find it better to be “inside the image”. After I have finished debugging the errors I write the steps to a Dockerfile. So yes most if not all the time you should be writing Dockerfiles and build images of them.

Now that we have seen two ways of building Docker images lets look at two ways of shipping them.

1. Pushing to a Docker repository

This is the most common method of shipping images. You tag the image with the repository URL and issue a docker push command

docker tag IMAGE_ID DockerRepo/image-name:image-version
docker push DockerRepo/image-name:image-version

To find the IMAGE_ID run the docker image ls command.

The problem with this method is that most organizations have their private docker repositories not accessible from the internet. So If I want to deliver an updated version of a service I have written I cannot push my image directly. I could push the image to the Docker Hub and tell the client to pull the image from there but that is also not an option since most organizations don’t want their code somewhere on the internet.

Now I hear some of you shouting VPN and proxies but most of the time I don’t get access to that either so the only solution is to package the images. Luckily for me the Docker daemon makes this really is.

2. Saving Docker images to a Tar Archive

docker save alpine-jdk8 > alpine-jdk8.tar

The tar file can be further processed with gz and shipped to the customer. On the other end the customer just needs to issue a load command

docker load < alpine-jdk8.tar

We have seen two ways of building Docker images one is to write Dockerfiles and the other is issuing the steps directly in a base container running bash / sh and committing the changes made to the container file system with the docker commit command. We then saw two ways of shipping the images. The most common one is to push the image to private or public docker repository and the other method is to save the image to a tar archive and ship it the old way using email / FTP / SCP …etc..

If I forgot something please let me know :)

Implementing DevOps

Tue, 23 May 2017 00:00:00 +0000

DevOps seems to gather some real interest from large organizations here in Germany. After having talked to a couple of companies and asked about what they specifically need, it seems to me that there exists this misconception of what DevOps actually is and how to implement it in a company.

Let me start by stating what DevOps is not

DevOps is not a replacement of Agile

If your company is not practicing Agile it will be more difficult to switch directly to DevOps.
DevOps is not a replacement for ITIL

DevOps can coexist with ITIL processes, however, to support short lead times and higher deployment frequencies many areas of the ITIL processes need to become fully automated. The ITIL disciplines of service design, incident and problem management still remain relevant because DevOps requires fast detection and recovery when service incidents occur.
DevOps is not just Automation

This is probably, in my opinion, the biggest misconception. Companies seem to focus eagerly on people to come in and help with automation. Although I agree that the biggest part of DevOps time should go into enabling Automation (CI/CD, IaC, DR…etc) but the work shouldn’t end there. DevOps requires cultural norms and an architecture that allows for the shared goals to be achieved throughout the IT value stream. This goes far beyond automation.

With this out of the way, let’s start looking at what DevOps is and how to implement it.

DevOps is a practice that enables companies to deliver value quickly, safely and reliably to their customers. Usually, this is implemented by allowing small teams to quickly and independently develop, test, monitor and deploy their code.

The implementation time can differ between companies and is dependent on many factors. Most companies struggle to deploy production changes in minutes or hours and in an age where competitive advantage requires fast time to market, these companies are at a significant disadvantage. This is in large part due to their inability to resolve conflicts within their software development and other IT departments (Operations, QA, InfoSec..etc.). Why? Because for example, IT Operations responsibility is to provide stable and reliable service and the software developers responsibility is to introduce new features or make the solution more accessible to the customers. The goals between departments differ.

Obviously, companies structured this way will suffer from chronic conflicts between departments. These conflicts can often lead to poor software and service quality that can kick off a chain of effects felt by customers, employees, and everybody else involved.

Solutions

Introduce a DevOps Engineer that understands:

The principles of flow, which accelerates the delivery of work from developers to the customers. The principles of feedback, which enables the creation of safer systems of work. The principles of continual learning and experimentation, which enables new ideas to be validated faster leading to a competitive edge in the industry.
Send the IT managers to a DevOps school

Similarly to how IT managers needed to be ITIL schooled one can reason that in our fast paced environment IT managers need now to understand both ITIL and DevOps principles. The best way for IT managers to understand DevOps is reading about how companies such as LinkedIn, Etsy, Amazon, Netflix, Spotify and many others introduced the DevOps principles. Take a look at Spotify’s cultural video and here.

Summary

It seems everybody understands the principles of DevOps differently. I hope I could express my views on the topic in a meaningful way and some of you might even agree. If not that is also fine :Þ Just make sure to email or comment your opinion.

In a future post I might delve more deeply into the above-mentioned principles and how to actually implement them, also a topic for a future post might be, “The perfect DevOps engineer a software developer or a Sys Admin?”

What do you think? Let me know!

Redis, Apache Kafka & RabbitMQ - When to use what

Sat, 06 May 2017 00:00:00 +0000

I recently had to present a design where the designed consisted of Redis, Apache Kafka and RabbitMQ, among other things of course. At the presentation the obvious question came up, can’t we just use one of thesse?

I understand that for novice users visiting the respective websites it must be difficult to understand the differences. On the front page on the redis webpage they state

Redis is an open source (BSD licensed), in-memory data structure store, used as a database, cache and message broker.

On the Apache Kafka webpage you’ll see

Publish & Subscribe to streams of data like a messaging system

and finally on RabbitMQ

RabbitMQ is the most widely deployed open source message broker

“Message” seems to be the keyword for all of them but that doesn’t tell the full story. Let’s have a look at the details and example scenarios where one would choose one over the other.

Redis

The main thing that redis does is being a blazing fast in-memory data structure store and as such it’s mainly seen in caching related scenarios. After release 3 however, many features were added and redis now is also wildly used as a publish/subscribe messaging system where it crosses into RabbitMQ territory.

On this site I use redis for caching the responses. If a user visits a newly created URL that nobody has visited before it will be served by the DB but the HTML response will be cached in redis. Next time somebody visits the same URL the request is served from cache without hitting the DB.

As mentioned before, redis is also widley used as a publish/subscribe messaging system. I have seen it being utilized in simple chat applications as well as a message broker for distributed tasks queues.

Apache Kafka

The main thing kafka does is provide you with the ability to perform real-time processing of streaming data. What is streaming data? Syslog messages send to you, stock price updates send to you, any data clients might generate that your servers receive can be thought of as streaming data.

Like redis, kafka is also blazing fast but it is limited to the I/O of the hard drive however, persistence is higher with Kafka. Kafka can store data on multiple machines in a fault tolerant manner and supports automatic recovery when machine failures occur.

Kafka design is build around the producer-consumer model. Producers write to topics, many topics can be written to a partition and consumers consume these topics. Queuing and publish/subscribe are traditionaly the main two models in messaging and Kafka is flexible enough to incorperate both of them.

RabbitMQ

RabbitMQ provides a fault tolerant messaging queue. It can be used to build publish/subscribe and work queuing solutions. RabbitMQ supports message acknowledgments. An acknowledgement is sent back from the consumer to tell RabbitMQ that a particular message had been received, processed and that RabbitMQ is free to delete it.

If a consumer dies (its channel is closed, connection is closed, or TCP connection is lost) without sending an ACK, RabbitMQ will understand that a message wasn’t processed fully and will re-queue it. If there are other consumers online at the same time, it will then quickly redeliver it to another consumer. That way you can be sure that no message is lost, even if the workers occasionally die.

Redis vs RabbitMQ

Redis main application is in memory storage. For pub/sub related applications I would prefer RabbitMQ over Redis as you get persistence, at least once delivery guarantees and complex topic based routing features out of the box.

Redis vs Kafka

I can see a use case where one could use redis in memory storage for ingestion of streaming short lived data with high consumer capacity on the backend. If you go this route make sure to keep an eye on memory usage. If consumer capacity is a limit I would advice to use kafka since kafka is built to accept and persist vast sums of data regardless of any consumers.

RabbitMQ vs Kafka

The same applies here as with redis. RabbitMQ was not designed for streaming message ingestion. If you want to use it as such make sure to have enough consumer capacity on the backend and preferably really fast ones. If the message queue grows to large RabbitMQ will stop responding which will lead to problems. Kafka on the other hand was specifically designed for this purpose. Kafka also makes it easy for multiple consumers to consume the same topic. If you encountered problems later on in your application you can playback a topic and let the consumer do its work again. With RabbitMQ after it receives an ACK the message is deleted and will not be seen again in the queue.

Summary table

	Redis	Kafka	RabbitMQ
Streaming data		X
ETL		X
Message routing			X
Topics		X	X
HA Storage		X	X
Chat application	X		X
High throughput	X	X

Web app deployment without downtime

Mon, 13 Mar 2017 00:00:00 +0000

Hi there today I’ll show you how I deploy my site to production without incurring downtime. My site isn’t big and is deployed on a single server where I have NGINX, HAproxy and redis running. Here a diagram that shows the setup

All services run in separate Docker containers. Why? Unless you have been living on Mars the past years you know why containers are awesome if not go on and read my post on docker containers.

On an architectural note the reason I am using NGINX and HAproxy is that I let NGINX handle compression, caching, SSL management and zipping static files and I use HAproxy in order to not incur downtime when I deploy a new version of my site. Only during deployments I spin up an additional instance of the web app and HAproxy automatically routes traffic between the instances. The routing decision is based on RR (Round-Robin) but can be configured in the docker-compose file to any load balancing algorithm HAproxy supports (weighted round robin, dynamic round robin, least connection …etc..)

Let’s now look at an example application to see how it all works

If you want to see this in action yourself you can clone my repo:

git clone https://github.com/donchev7/webapp-sample
docker build -t webapp-sample .

For this example to work properly on your computer it is important to tag the docker build as “webapp-sample” if you wish to use a different name update the docker-compose.yml file.

After building the Dockerfile do a

docker-compose up -d

This will bring up the environment. NGINX is running on port 80, if you do a quick

curl http://localhost

you should see the webapp responding with “Hello beautiful world!”. Now imagine you make a change to the webapp and want the changes to be reflected on your website you would need to do a

docker build -t webapp-sample .
docker-compose up -d

That of course introduces service downtime to your website because with the docker-compose up command docker recreates your container from the new image and in the process takes your container down for a few seconds.

Now lets see how to manage this without downtime

Open app.py and make a change to the function route

def route(app):
    """
    Add some routes to our test web app
    
    :param app: Flask application instance
    :return: Flask app
    """
    @app.route('/')
    @app.route('/index') 
    @app.route('/home')
    def index():
        return "Hello beautiful changed world!" 

    return app

As we can see I changed the return statement. Now we save the app.py file and build a new image:

docker build -t webapp-sample .

Now that we have a new image we need the webapp container to switch to this new image without restarting the container. We do this with the help of HAproxy, just write:

docker-compose scale website=2
docker ps

Now we should have two containers running the webapp, webappsample_website_1 is running your old version and webappsample_website_2 is running your newer version (take a look at the image above). To switch to only having one version running (the newer) we would need to just get the name of the container and execute

docker stop webappsample_website_1

Now HAproxy gets notified that it is only linked to one webapp container and thus only routes traffic to your newer version of webapp.

How does this work behind the curtains?

The HAproxy docker image is configured to automatically configure HAproxy depending on the linked containers it has. The solution also works on docker swarm and docker cloud. You can have a look at the source code here.

Awesome Docker

Thu, 23 Feb 2017 00:00:00 +0000

Docker is a disrupting technology for software delivery, infrastructure, and development. Just look at the adoption numbers (courtesy to datadog)

But let us step back for a second and answer the common question, What is Docker?

Docker started as an application build and deployment management system with the idea of having homogeneous application containers. Similarly to the shipping container industry, loading and unloading cargo ships full of containers are made possible because of the homogeneous container sizes. Imagine what would happen if the containers on the ships were all in different sizes and shapes, loading, unloading, and stacking would be a nightmare. And exactly this problem has also existed in the software development and operations space for more than a decade. Usually, a developer would write code and a complementary document detailing how to install, configure and run his code. Operations would pick up a ticket and try to replicate these steps in a production environment with the obvious problems. Docker solves this problem by letting the developer package up his application with all of the dependencies it needs and ship it out as one package in the form of an image.

What are docker images?

Docker images are indexed snapshots of the entire filesystem a container is meant to run. Every time the developer introduces a change in his code a new version of the image is automatically created and assigned a unique ID.

Why are docker images revolutionary?

Every production deployment is re-buildable making rollbacks easy
Is one application using Java 7 and the other Java 8? No problem with docker images
Need 2 isolated instances of MySQL on the same host? Yeah no problem with docker images

Not let us look at a practical example to see docker and it benefits in action. We will go for a WordPress deployment with ELK monitoring. Think for a second from an operational standpoint what would be needed to realize this

Install PHP, Apache, MySQL
Create a database
Setup WordPress and configure it
Copy source files to Apache
Install ELK
Configure ELK
Configure Logstash to read Apache logs

I am probably forgetting something but the steps are quite many! Whit docker you could use docker-compose to bundle and configure every service and just run it. Here an example docker-compose.yml file for spinning up a WordPress site with ELK:

version: '2'

services:
  elasticsearch:
    image: elasticsearch:5.3.1
    restart: always
    ports:
      - "9200:9200"
      - "9300:9300"
    volumes:
      - data:/usr/share/elasticsearch/data<
  kibana:
    image: kibana:5.3.1
    restart: always
    links:
      - elasticsearch
    ports:
      - "5601:5601"
    depends_on:
      - elasticsearch
      - logstash
  logstash:
    image: logstash:5.3.1
    restart: always
    command: logstash -e 'input { stdin {  } udp { port => 5050 type => syslog } tcp { port => 5050 type => syslog } } output { elasticsearch { hosts => [ 'elasticsearch' ] index => "logstash-%{+YYYY.MM.dd}" } stdout { } }'
    ports:
      - "5050:5050/udp"
      - "5050:5050/tcp"
  db:
    image: mysql:5.7
    volumes:
      - db_data:/var/lib/mysql
    restart: always
    environment:
      MYSQL_ROOT_PASSWORD: evil_password
      MYSQL_DATABASE: wordpress
      MYSQL_USER: wordpress
      MYSQL_PASSWORD: sentencepress

  wordpress:
    depends_on:
      - db
    image: wordpress:latest
    ports:
      - "80:80"
    restart: always
    environment:
      WORDPRESS_DB_HOST: db:3306
      WORDPRESS_DB_USER: wordpress
      WORDPRESS_DB_PASSWORD: sentencepress
      SYSLOGSERVER: "udp://logstash:5050"
    logging:
      driver: syslog
      options:
        syslog-address: "udp://logstash:5050"
       
volumes:
  data:
  db_data:

save this to a docker-compose.yml file in a directory and inside the directory run

docker-compose up

You can view your WordPress site by opening your browser and navigate to http://localhost

Kibana is running on port 5601, http://localhost:5601

Now I ask you, would you prefer a document with 300-600 words and pictures on how to install and configure our solution or would you rather receive a docker-compose.yml file and run one line in your command prompt? You decide!

Need professional help with setting up an ELK cluster with redundancy? Send me an email and I’ll gladly help out!

State of the Cloud

Sun, 15 May 2016 00:00:00 +0000

Cloud technologies and services are evolving rapidly and startups are not the only adopters anymore. I can see that many German companies are moving their infrastructure to the cloud, either entirely or partly (hybrid clouds).

In this post I want to take you for a cloud journey! Follow along if you have time.

In 2006 Amazon launched AWS (Amazon Web Services) offering IAAS (Infrastructure As A Service) providing compute and storage capacity to clients. The S3 service was in my opinion a game changer at the time, only paying for storage you actually used without needing to forecast storage needs offered companies to innovate in the storage domain. Think, backup companies and personal data storage (dropbox).

Although a game changer, it wasn’t until Google introduced App Engine in 2008 that the cloud computing space turned into an industry. App Engine was one of the first PAAS (Platform As a Service) solution and at that time a popular one.

With IAAS developers didn’t need to worry about the physical hardware however they still needed to manage the operating system (security patching, os updates..etc..).

With PAAS developers don’t need to worry about the operating system anymore and can focus on their code. PAAS is still going strong and many companies utilize a set of private and public cloud PAAS solutions. Red Hat’s OpenShift and Cloud Foundry are probably the most widespread open source multi cloud PAAS solutions.

FAAS (Function As A Service) or serverless is the next big thing in my opinion. With PAAS even though we don’t have to take care of the operating system we still need to provision one and run it 24/7 and thus pay for the underlying resources. With FAAS we don’t need to provision anything and don’t need to pay anything if the service (function) isn’t executing.

I can already see startups architecting their software to fit these new concepts and also older companies starting to migrate to serverless architectures. One predecessor to be able to adopt serverless solutions is to build decoupled applications and to have developers think in events/triggers as opposed to the classical OOP design.

What FAAS providers are out there?

AWS Lambda is the undisputed leader considering market traction. Other notable solutions are Azure Functions, Google Functions and Bluemix OpenWhisk.

How does FAAS work under the hood?

That’s a topic for a new post but you can go and read how it is implemented in the open source project Bluemix OpenWhisk here

If I want to migrate my app to FAAS what options do I have?

It depends on your current application and the language it is written in. As of now (Mai, 2016) Google functions only supports JavaScript. AWS lambda supports Java, JavaScript, Python and C#. Azure Functions supports JavaScript, C#, F#, as well as Python, PHP, Bash, Batch, and PowerShell.

So if your application is written in Java your options are limited to AWS Lambda, if your application is written in PHP you will need to look at Azure.

My company wants to go serverless on a private cloud how do we do that?

In my opinion if you decide going serverless the real benefits are only seen when you adopt a public cloud. I don’t see no real benefits for companies going for in house serverless solutions. You might as well run your applications in containers on kubernetes.

How do I get started with FAAS architecture?

Their exists several serverless frameworks, the ones with the most traction (number of contributors) are:

Serverless(Support for Java, JS and Python)
Chalice (Python only support)
Zappa (Python only support)
Claudia (JS only support)

If you need help designing your app to have a serverless architecture, send me an email!

About this website

Fri, 08 Apr 2016 00:00:00 +0000

I will use this website mainly as a brain dump for my projects or for projects I would like to start in the near future. Since this is mainly a tech related website I hope that someone out there might find it helpful if they encountered similar problems or have similar questions.

Let me tell you the tech specifics around this website:

NGINX web server
flask web framework deployed with gunicorn
SQLAlchemy ORM wrapper
AWS cloud

The site has its own CI/CD tool chain build on top of Docker and Jenkins but I am thinking of moving everything to AWS lambda. I will make a couple of posts on how to architecture this and how I am planning to use AWS lambda.

On a site note, everything on this site is my opinion (except comments written by others, which are their opinions). If people don’t agree with me that’s fine just let me know and we can have a discussion.

In the tech industry, everything moves at the speed of light and I understand that some of my opinions might be outdated by the time I write them. Readers are encouraged to comment and provide alternative solutions.

UPDATE 2021-03-17

Moved to a static webpage with gatsby hosted on firebase. More info here

UPDATE 2022-09-11

Moved from gatsby to hugo. More info here

About

Thu, 07 Apr 2016 00:00:00 +0000

Hi, my name is Bobby Blagovest Donchev. I live in the Frankfurt metropolitan area, Germany. Originally I am from Bulgaria but have spent the majority of my adulthood in Iceland. I came to Germany to finish my studies (M.Sc. Engineering) and have since decided to stay a little longer.

From a young age I had an urge to tinker with computers and networks and after I found out I could make a living by doing what I love, I was instantly hooked. The software behind this blog is written by me. I tried WordPress and blogger but wanted something fast and lean and thought I might as well do it myself. In the near future I will write a couple of posts about how I programmed and architectured the software.

I will be mainly writing about cloud services, open source software and from time to time programming related subjects.

Thanks for reading my site, I hope you find something interesting or useful.

About me

Thu, 07 Apr 2016 00:00:00 +0000

I will be mainly writing about cloud services, open source software and from time to time programming related subjects.

Thanks for reading my site, I hope you find something interesting or useful.

Search

Mon, 01 Jan 0001 00:00:00 +0000

donchev.is

Sealed Secrets UI

Why Sealed Secrets UI?

Features and Benefits

Getting Started

Quick Configuration

Deployment

Introducing Statebin.io

Features

Atomic Counters

Persistent Storage

Locking Mechanisms

The Architecture

Conclusion

Securing a publicly accessible Fluentd server

Securing a publicly accessible Fluentd server: mTLS & Zero-Trust

Why Securing Fluentd Matters

A Closer Look at the mTLS-based Go TCP Proxy

Setting Up the Proxy for Fluentd

Initialize the Proxy

Test the proxy

Alternative Solutions

Zero Trust Networking

Cloudflare

OpenZiti

Conclusion

Reloading Fluentd certificates

Introduction

Configuring Fluentd to use lets-encrypt certificates

Checking if the certificates have been renewed

Reloading the certificates (Fluentd config)

Conclusion

References

Working with PostgreSQL in Go using pgx

Introduction

Connection pool setup

Inserting data

Single row inserts

Bulk inserts

COPY inserts

Querying data

Conclusion

How to authenticate a GitHub App using python

Introduction

Webhook secret, private key, JWT and access token

Webhook secret

Private key, JWT and access token

Conclusion

AWS Lambda invoker identification

Introduction

Problem

Solution

IMPORTANT

Conclusion

State machine using MongoDB changestreams

Introduction

Building a state machine with MongoDB and Kubernetes

Process manager

Conclusion

Switching from gatsby to hugo

Templating

Kubernetes interview questions

Switching from flask to gatsby

Azure SAS tokens on cloudflare workers

Prometheus with Azure Monitor

Implementation

Activation

Streaming kubernetes logs from multiple replicas

Bonus

Caveats

Archives

Kubernetes in a private AWS network

Cavets

Docker for developers: part 2

Docker multi-stage builds

Docker multi-stage builds caveats

Docker volumes and file permission

Docker for developers

Spawning a shell inside a container

Copying a file from a docker container