Empty database after upgrade 2.1.0 -> 3.0.0

[regulartim]

After upgrading GreedyBear to version 3.0.0 on one of my instances all historic data disappeared. After some investigation I noticed that the application uses the wrong docker volume.

> sudo  docker inspect greedybear_postgres --format '{{json .Mounts}}' | python3 -m json.tool                                                                                                                                                   
[
    {
        "Type": "volume",
        "Name": "5ed93ca33747f82767cf26389827376700ca472dbeb05a5fff8ab9f759f87e07",
        "Source": "/var/lib/docker/volumes/5ed93ca33747f82767cf26389827376700ca472dbeb05a5fff8ab9f759f87e07/_data",
        "Destination": "/var/lib/postgresql",
        "Driver": "local",
        "Mode": "",
        "RW": true,
        "Propagation": ""
    },
    {
        "Type": "volume",
        "Name": "greedybear_postgres_data",
        "Source": "/var/lib/docker/volumes/greedybear_postgres_data/_data",
        "Destination": "/var/lib/postgresql/data",
        "Driver": "local",
        "Mode": "z",
        "RW": true,
        "Propagation": ""
    }
]

As can be seen, there are two volumes available for the postgres container. After the update, for some reason the application decided to use the first one, although the data lives in the second one.

[regulartim]

I currently don't have an idea why this happened after this upgrade. For my instance the fix was to add PGDATA=/var/lib/postgresql/data/18/docker to the file env_file_postgres. But that is a fix that we can not push to other instances, as this file is in the .gitignore. Any idea @mlodic ?

[mlodic]

mm that's weird, I checked the changes from the last released version and it seems there's nothing affecting this.
It could be also something related to your own environment like changes in the docker versions, or something messed up in the docker service itself. I think that with some cleanups (while keeping the db volume) it could come to normal again. The volumes should still keep the same naming convention

[regulartim]

Yes, it might just be my environment. I'll invesigate tomorrow, too tired right now.

[regulartim]

Turns out it is not my environment. I investigated the issue and it's really complicated, but I'll try to break it down:

When we migrated to PG 18 last year with version 2.0.0 I missed a change in the Postgres Docker container. They changed the PGDATAenvironment variable, which determines where the container stores its data to /var/lib/postgresql/18/docker. Unfortunately, we mount our Docker volume for the PG container to /var/lib/postgresql/data/ in default.yml. As a consequence, the Postgres data directory is outside of our mount point.

On GreedyBear instances >= version 2.0.0 something strange happened. A anonymous docker volume was created at the correct mount point, but the data still was written to the named volume (greedybear_postgres_data). AFAIK this was because a backwards-compatibility symlink in the container redirected the new data path to the old one.

A few days ago, a new version of the Postgres Docker container was released, that again changes things: a symlink was removed. If you pull this version of the PG container, GreedyBear breaks. It starts using the anonymous volume which contains no data. The old data is still on the named volume but it is not used anymore.

Suggested fix: we change the volume definition from postgres_data:/var/lib/postgresql/data/ to postgres_data:/var/lib/postgresql in the default.yml. This "forces" GreedyBear to use the right volume. As far as I can see, no further action from a user side is required. I could fix both of my instances with this.

Any thoughts on that @mlodic ?

[mlodic]

oh my what a mess, thanks for sorting this out. I checked IntelOwl to see whether we did any change there but it seems that we are still at postgres 16 🤣 for the local instance. I would create an issue there so that we don't have the same problem there too.

Feel free to do the change that you mentioned, I would just mention this in the docs like we usually did for IntelOwl (see: https://intelowlproject.github.io/docs/IntelOwl/installation/#update-and-rebuild). In this way we can explicilty track this upgrades problems for the community.

[regulartim]

Yeah, will change the docs once the release is ready. To my understanding, nothing has to be done from a user perspective, aside from pulling the most current GreedyBear container.

[regulartim]

I would just mention this in the docs like we usually did for IntelOwl (see: https://intelowlproject.github.io/docs/IntelOwl/installation/#update-and-rebuild). In this way we can explicilty track this upgrades problems for the community.

Is that what you meant?