ceph-testcontainers-go

ceph-testcontainers-go

Testcontainers for Go modules that run a real, disposable Ceph cluster for integration tests. The cluster runs the official Ceph daemons in containers: one MON, one MGR and two OSDs by default, plus RBD pools, CephFS filesystems with MDS daemons, and RGW gateways when you ask for them. Tests prepare and change the server side through typed Go APIs. Your application talks to the cluster with its own client: go-ceph, librados, librbd, libcephfs, the Ceph CLI or an S3 SDK.

The modules control Ceph through the CLIs inside the containers. They have no go-ceph, librados or cgo dependency and build with CGO_ENABLED=0. The Ceph daemons need no privileged mode, host disks, LVM or systemd: each OSD stores its data in a sparse BlueStore file inside its container by default. MemStore and memory-backed BlueStore are also available for faster disposable fixtures.

Releases follow semantic versioning. Before v1.0.0, a minor release may change the API; CHANGELOG.md lists every change.

Documentation for a published release is available at its Git tag. See Upgrading to v0.3.0 for the new native input API, compatibility and the limits of the protocol receipts. For changes from v0.1.0, see Upgrading to v0.2.0.

Modules

Every package follows the Testcontainers for Go module conventions: a Run function that takes a context, an image and options, and a Container that embeds testcontainers.Container. The service packages build on ceph: their Run starts a cluster with the service's defaults, their options can be passed to ceph.Run together, and their Container embeds *ceph.Container. The API reference for every package is also available with go doc and on pkg.go.dev.

Requirements

• Go 1.26 or later (v0.1.0 needs Go 1.25); use a patched toolchain. The repository selects Go 1.26.9 for its own checks. A consumer must select its own toolchain (see the upgrade guide).
• A Docker engine that runs Linux containers. Docker Desktop works. Host networking has extra requirements, described in the host network guide.
• Access to pull the Ceph image and the Ryuk image.

The default image is the official quay.io/ceph/ceph Tentacle 20.2.x image, pinned by digest as ceph.DefaultImage. Squid 19.2.x images also work, with the differences listed in the releases guide.

Installation

go get github.com/JSYoo5B/ceph-testcontainers-go/ceph@latest

Add rbd, cephfs or rgw the same way when you need them.

Quick start

package app_test

import (
	"testing"

	"github.com/JSYoo5B/ceph-testcontainers-go/ceph"
	"github.com/testcontainers/testcontainers-go"
)

func TestWithCeph(t *testing.T) {
	ctx := t.Context()
	cluster, err := ceph.Run(ctx, ceph.DefaultImage,
		ceph.WithPools(ceph.PoolConfig{Name: "app"}),
	)
	testcontainers.CleanupContainer(t, cluster)
	if err != nil {
		t.Fatal(err)
	}
	if err := cluster.WaitForClean(ctx); err != nil {
		t.Fatal(err)
	}

	// Give a native client the cluster's ceph.conf and an admin keyring.
	conf, keyring, err := cluster.ConnectionConfig()
	if err != nil {
		t.Fatal(err)
	}
	_, _ = conf, keyring
}

The service packages start a cluster with their own defaults:

// An initialized RBD pool named rbd.
volumes, err := rbd.Run(ctx, ceph.DefaultImage)

// The filesystem tc-cephfs with one active MDS.
files, err := cephfs.Run(ctx, ceph.DefaultImage)

// One S3 gateway with a generated test user.
objects, err := rgw.Run(ctx, ceph.DefaultImage)
endpoint, err := objects.S3Endpoint(ctx)
gateway := objects.Gateways()[0] // AccessKey, SecretKey and Region

To serve several services from one cluster, pass their options to ceph.Run:

cluster, err := ceph.Run(ctx, ceph.DefaultImage,
	rbd.WithPools(ceph.PoolConfig{Name: "volumes"}),
	cephfs.WithFilesystems(cephfs.Config{Name: "shared"}),
	rgw.WithGateways(rgw.Config{Name: "s3"}),
)

Documentation

Module pages, one per package, in the format of the Testcontainers for Go module catalogue:

• Ceph: the cluster, its options and every ceph.Container method.
• RBD, CephFS and RGW.

Guides for tasks that span packages:

• Fault injection: stopping, pausing and disconnecting daemons and clients, delaying or dropping messages, fencing, full clusters and damaged replicas.
• Host network: bridge and host networking, and connecting native clients from the test process.
• Releases and images: supported Ceph releases, role images and image requirements.
• Upgrading to v0.2.0: Go requirements, API and error compatibility, Messenger policies and release limitations.
• Upgrading to v0.3.0: bounded native input, new fixture receipts and what remains unverified.
• Contexts and cleanup: how contexts bound calls, what each fixture owns, and how to restore temporary changes.
• Running native commands: how to do anything the typed API does not cover with the Ceph CLIs inside the containers.
• Fixtures for Messenger client tests: the profiles, daemon controls and differences for testing a Messenger client implementation against native daemons.
• Classified errors: telling a refused request, a feature the release lacks, a missing image tool, an engine limit and a failed native command apart with errors.Is and errors.As.

For contributors:

• Testing: running and writing unit and integration tests.
• CI: the GitHub Actions workflows and how to read a failing batch.
• Preparing a release: the scope, evidence and publication checks for a candidate.

Related projects

ceph-testcontainers-images checks Ceph images against the requirements of these modules and publishes smaller role images (control, OSD, MDS, RGW) extracted from the official images.

License

The modules are licensed under the MIT License, like Testcontainers for Go.

They do not contain or link Ceph. Tests pull the Ceph container images at run time and run the daemons and CLIs inside them; Ceph in those images is under its own licenses, mainly LGPL-2.1 and LGPL-3.0 with some GPL-2.0 parts. Images that a test selects with WithOSDImage, rgw.WithImage or cephfs.WithMDSImage keep the licenses of whoever publishes them.