4.25 silently lowers vm.max_map_count from 262144 to 65530, breaking Elasticsearch containers ("ExitCode": 78)

[jamshid]

Description

Elasticsearch containers rely on vm.max_map_count being 262144
https://www.elastic.co/guide/en/elasticsearch/reference/current/vm-max-map-count.html
and that was the default for Docker for Mac up until 4.25.

Reproduce

docker run centos:7 sysctl vm.max_map_count on old and new Docker for Mac. Now it outputs:
vm.max_map_count = 65530
Previously it output:
vm.max_map_count = 262144

Expected behavior

Upgrades shouldn't break existing deployments or lower sysctl values.

docker version

Client:
 Cloud integration: v1.0.35+desktop.5
 Version:           24.0.6
 API version:       1.43
 Go version:        go1.20.7
 Git commit:        ed223bc
 Built:             Mon Sep  4 12:28:49 2023
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.25.0 (126437)
 Engine:
  Version:          24.0.6
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.7
  Git commit:       1a79695
  Built:            Mon Sep  4 12:31:36 2023
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.22
  GitCommit:        8165feabfdfe38c65b599c4993d227328c231fca
 runc:
  Version:          1.1.8
  GitCommit:        v1.1.8-0-g82f18fe
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Version:    24.0.6
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.2-desktop.5
    Path:     /Users/jamshid/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.23.0-desktop.1
    Path:     /Users/jamshid/.docker/cli-plugins/docker-compose
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     /Users/jamshid/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.20
    Path:     /Users/jamshid/.docker/cli-plugins/docker-extension
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v0.1.0-beta.9
    Path:     /Users/jamshid/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/jamshid/.docker/cli-plugins/docker-sbom
  scan: Docker Scan (Docker Inc.)
    Version:  v0.26.0
    Path:     /Users/jamshid/.docker/cli-plugins/docker-scan
  scout: Docker Scout (Docker Inc.)
    Version:  v1.0.9
    Path:     /Users/jamshid/.docker/cli-plugins/docker-scout

Server:
 Containers: 58
  Running: 23
  Paused: 0
  Stopped: 35
 Images: 364
 Server Version: 24.0.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8165feabfdfe38c65b599c4993d227328c231fca
 runc version: v1.1.8-0-g82f18fe
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
 Kernel Version: 6.4.16-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 8
 Total Memory: 15.61GiB
 Name: linuxkit-7ad88f20e5c4
 ID: c6944eec-13eb-4b2b-96b3-beb3e4774da1
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  192.168.1.50:5100
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: daemon is not using the default seccomp profile

Diagnostics ID

76189D97-5C98-4A66-A89B-45AC1DEF5F42/20231027203626

Additional Info

No response

[jamshid]

As a temporary workaround you can run this but idk how to make it permanent on macOS.
docker run --rm -ti --privileged centos:7 sysctl vm.max_map_count=262144

I think there is now a way to permanently adjust sysctl values with Docker for Windows but don't know it offhand.

On linux of course it's just this on the host:
echo 'vm.max_map_count = 262144' > /etc/sysctl.d/98-elasticsearch.conf

[jamshid]

Finally, I'm not sure this is related to this problem but I think this s a new error in 4.25 with my elasticsearch container:
docker-compose logs -f elasticsearch shows a lot of expected logs then quits mysteriously with:
error from daemon in stream: Error grabbing logs: invalid character '\x00' looking for beginning of value

[thomet]

I'm experiencing the same problem on 4.25, quite easy to reproduce it:

❯ docker ps -a
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

❯ docker run -it --privileged alpine:latest
/ # sysctl vm.max_map_count
vm.max_map_count = 65530
/ # sysctl -w vm.max_map_count=262144
vm.max_map_count = 262144
/ # sysctl vm.max_map_count
vm.max_map_count = 262144
/ # exit

❯ docker ps -a
CONTAINER ID   IMAGE           COMMAND     CREATED          STATUS                      PORTS     NAMES
ff0d3a4df7b4   alpine:latest   "/bin/sh"   24 seconds ago   Exited (0) 11 seconds ago             strange_hamilton

❯ docker start -ai strange_hamilton
/ # sysctl vm.max_map_count
vm.max_map_count = 262144
/ # exit

❯ echo 'Now I restart docker-desktop'
Now I restart docker-desktop

❯ docker start -ai strange_hamilton
/ # sysctl vm.max_map_count
vm.max_map_count = 65530

[diepes]

Just logged a support ticket for elastisearch dying with error.

2023-11-03 00:07:11 ERROR: [1] bootstrap checks failed
2023-11-03 00:07:11 [1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

[desmortum]

Same here, reproduced on two machines. Couldn't remedy easily. Rolled back.

[plessbd]

As a temporary workaround you can run this but idk how to make it permanent on macOS. docker run --rm -ti --privileged centos:7 sysctl vm.max_map_count=262144

I think there is now a way to permanently adjust sysctl values with Docker for Windows but don't know it offhand.

On linux of course it's just this on the host: echo 'vm.max_map_count = 262144' > /etc/sysctl.d/98-elasticsearch.conf

Looks like there is currently no way to make the changes perm

"the filesystem of Docker Desktop is readonly and you can’t write or install anything in it"

https://forums.docker.com/t/how-to-enter-docker-vm-terminal-on-macos-14/138316

[jamshid]

FWIW this workaround for compose users came up in:
https://dockercommunity.slack.com/archives/C2TJ8FEJY/p1698948708883499?thread_ts=1698438548.692899&cid=C2TJ8FEJY

Add a privileged compose service, make the elasticsearch service depends_on it.

  fixsysctl:
    restart: "no"
    privileged: true
    deploy:
      replicas: 1
      resources:
        limits:
          memory: 25m
    image: centos:7
    command: "sysctl -w vm.max_map_count=262144"

[haloboy777]

Faced the same issue.

Like @desmortum said, Rolling back to v4.24.2 fixes the issue.

Release note link: https://docs.docker.com/desktop/release-notes/#4242

[Deadvi5]

Same here, in my team we're all affected by this