Re: pgsql: Add support for OAUTHBEARER SASL mechanism

Add support for OAUTHBEARER SASL mechanism

This commit implements OAUTHBEARER, RFC 7628, and OAuth 2.0 Device
Authorization Grants, RFC 8628. In order to use this there is a
new pg_hba auth method called oauth. When speaking to a OAuth-
enabled server, it looks a bit like this:

$ psql 'host=example.org oauth_issuer=... oauth_client_id=...'
Visit https://oauth.example.org/login and enter the code: FPQ2-M4BG

Device authorization is currently the only supported flow so the
OAuth issuer must support that in order for users to authenticate.
Third-party clients may however extend this and provide their own
flows. The built-in device authorization flow is currently not
supported on Windows.

In order for validation to happen server side a new framework for
plugging in OAuth validation modules is added. As validation is
implementation specific, with no default specified in the standard,
PostgreSQL does not ship with one built-in. Each pg_hba entry can
specify a specific validator or be left blank for the validator
installed as default.

This adds a requirement on libcurl for the client side support,
which is optional to build, but the server side has no additional
build requirements. In order to run the tests, Python is required
as this adds a https server written in Python. Tests are gated
behind PG_TEST_EXTRA as they open ports.

This patch has been a multi-year project with many contributors
involved with reviews and in-depth discussions: Michael Paquier,
Heikki Linnakangas, Zhihong Yu, Mahendrakar Srinivasarao, Andrey
Chudnovsky and Stephen Frost to name a few. While Jacob Champion
is the main author there have been some levels of hacking by others.
Daniel Gustafsson contributed the validation module and various bits
and pieces; Thomas Munro wrote the client side support for kqueue.

Author: Jacob Champion <jacob(dot)champion(at)enterprisedb(dot)com>
Co-authored-by: Daniel Gustafsson <daniel(at)yesql(dot)se>
Co-authored-by: Thomas Munro <thomas(dot)munro(at)gmail(dot)com>
Reviewed-by: Daniel Gustafsson <daniel(at)yesql(dot)se>
Reviewed-by: Peter Eisentraut <peter(at)eisentraut(dot)org>
Reviewed-by: Antonin Houska <ah(at)cybertec(dot)at>
Reviewed-by: Kashif Zeeshan <kashi(dot)zeeshan(at)gmail(dot)com>
Discussion: https://postgr.es/m/[email protected]

Branch
------
master

Details
-------
https://git.postgresql.org/pg/commitdiff/b3f0be788afc17d2206e1ae1c731d8aeda1f2f59

Modified Files
--------------
.cirrus.tasks.yml | 15 +-
config/programs.m4 | 65 +
configure | 332 +++
configure.ac | 41 +
doc/src/sgml/client-auth.sgml | 252 ++
doc/src/sgml/config.sgml | 26 +
doc/src/sgml/filelist.sgml | 1 +
doc/src/sgml/installation.sgml | 27 +
doc/src/sgml/libpq.sgml | 445 +++
doc/src/sgml/oauth-validators.sgml | 414 +++
doc/src/sgml/postgres.sgml | 1 +
doc/src/sgml/protocol.sgml | 133 +-
doc/src/sgml/regress.sgml | 10 +
meson.build | 100 +
meson_options.txt | 3 +
src/Makefile.global.in | 1 +
src/backend/libpq/Makefile | 1 +
src/backend/libpq/auth-oauth.c | 894 ++++++
src/backend/libpq/auth.c | 10 +-
src/backend/libpq/hba.c | 64 +-
src/backend/libpq/meson.build | 1 +
src/backend/libpq/pg_hba.conf.sample | 4 +-
src/backend/utils/adt/hbafuncs.c | 19 +
src/backend/utils/misc/guc_tables.c | 12 +
src/backend/utils/misc/postgresql.conf.sample | 3 +
src/include/common/oauth-common.h | 19 +
src/include/libpq/auth.h | 1 +
src/include/libpq/hba.h | 7 +-
src/include/libpq/oauth.h | 101 +
src/include/pg_config.h.in | 9 +
src/interfaces/libpq/Makefile | 11 +-
src/interfaces/libpq/exports.txt | 3 +
src/interfaces/libpq/fe-auth-oauth-curl.c | 2883 ++++++++++++++++++++
src/interfaces/libpq/fe-auth-oauth.c | 1163 ++++++++
src/interfaces/libpq/fe-auth-oauth.h | 46 +
src/interfaces/libpq/fe-auth.c | 36 +-
src/interfaces/libpq/fe-auth.h | 3 +
src/interfaces/libpq/fe-connect.c | 48 +-
src/interfaces/libpq/libpq-fe.h | 85 +
src/interfaces/libpq/libpq-int.h | 13 +-
src/interfaces/libpq/meson.build | 5 +
src/makefiles/meson.build | 1 +
src/test/authentication/t/001_password.pl | 8 +-
src/test/modules/Makefile | 1 +
src/test/modules/meson.build | 1 +
src/test/modules/oauth_validator/.gitignore | 4 +
src/test/modules/oauth_validator/Makefile | 40 +
src/test/modules/oauth_validator/README | 13 +
src/test/modules/oauth_validator/fail_validator.c | 47 +
src/test/modules/oauth_validator/magic_validator.c | 48 +
src/test/modules/oauth_validator/meson.build | 85 +
.../modules/oauth_validator/oauth_hook_client.c | 293 ++
src/test/modules/oauth_validator/t/001_server.pl | 594 ++++
src/test/modules/oauth_validator/t/002_client.pl | 154 ++
src/test/modules/oauth_validator/t/OAuth/Server.pm | 140 +
src/test/modules/oauth_validator/t/oauth_server.py | 391 +++
src/test/modules/oauth_validator/validator.c | 143 +
src/test/perl/PostgreSQL/Test/Cluster.pm | 22 +-
src/tools/pgindent/pgindent | 14 +
src/tools/pgindent/typedefs.list | 11 +
60 files changed, 9278 insertions(+), 39 deletions(-)

> Add support for OAUTHBEARER SASL mechanism

Debian still has this experimental port with a GNU userland and a
FreeBSD kernel called kfreebsd. I don't expect anyone to particularly
care about it, but it found an actual bug:

/build/reproducible-path/postgresql-18-18~~devel.20250331/build/../src/interfaces/libpq/fe-auth-oauth-curl.c: In function ‘register_socket’:
/build/reproducible-path/postgresql-18-18~~devel.20250331/build/../src/interfaces/libpq/fe-auth-oauth-curl.c:1317:20: error: ‘actx’ undeclared (first use in this function); did you mean ‘ctx’?
1317 | actx_error(actx, "libpq does not support multiplexer sockets on this platform");
| ^~~~

This should not be a compile-time error; actx is not defined outside
the #ifdef blocks there:

/*
* Adds and removes sockets from the multiplexer set, as directed by the
* libcurl multi handle.
*/
static int
register_socket(CURL *curl, curl_socket_t socket, int what, void *ctx,
void *socketp)
{
#ifdef HAVE_SYS_EPOLL_H
struct async_ctx *actx = ctx;
...
#endif
#ifdef HAVE_SYS_EVENT_H
struct async_ctx *actx = ctx;
...
#endif

actx_error(actx, "libpq does not support multiplexer sockets on this platform");
return -1;
}

https://buildd.debian.org/status/fetch.php?pkg=postgresql-18&arch=hurd-amd64&ver=18%7E%7Edevel.20250331-1&stamp=1743455288&raw=0

Christoph

Re: To Daniel Gustafsson
> > Add support for OAUTHBEARER SASL mechanism
>
> Debian still has this experimental port with a GNU userland and a
> FreeBSD kernel called kfreebsd.

Sorry this part was nonsense, kfreebsd was actually terminated and
obviously I didn't even read the port's name. The failing port (still
experimental and care-is-optional) is hurd-amd64.

The bug is the same, though.

Christoph

On Mon, Mar 31, 2025 at 2:54 PM Christoph Berg <myon(at)debian(dot)org> wrote:
>
> > Add support for OAUTHBEARER SASL mechanism
>
> Debian still has this experimental port with a GNU userland and a
> FreeBSD kernel called kfreebsd. I don't expect anyone to particularly
> care about it, but it found an actual bug:
>
> /build/reproducible-path/postgresql-18-18~~devel.20250331/build/../src/interfaces/libpq/fe-auth-oauth-curl.c: In function ‘register_socket’:
> /build/reproducible-path/postgresql-18-18~~devel.20250331/build/../src/interfaces/libpq/fe-auth-oauth-curl.c:1317:20: error: ‘actx’ undeclared (first use in this function); did you mean ‘ctx’?
> 1317 | actx_error(actx, "libpq does not support multiplexer sockets on this platform");
> | ^~~~
>
> This should not be a compile-time error; actx is not defined outside
> the #ifdef blocks there:

Ah, sorry about that. Thank you for reporting it!

(That means that Windows builds --with-libcurl are similarly broken, I
think. Not that Windows packagers will want to use --with-libcurl --
it doesn't do anything -- but it should build.)

I don't have hurd-amd64 to test, but I'm working on a patch that will
build and pass tests if I manually munge pg_config.h. We were skipping
the useless tests via a $windows_os check; I think I should use
check_pg_config() instead.

We could change how this works a bit for the proposed libpq-oauth.so
plugin, and only build it if we have a workable implementation. I do
like having these other platforms compile the Curl code, though, since
we'd prefer to keep the build clean for a future Windows
implementation...

--Jacob

On Mon, Mar 31, 2025 at 4:09 PM Jacob Champion
<jacob(dot)champion(at)enterprisedb(dot)com> wrote:
> I don't have hurd-amd64 to test, but I'm working on a patch that will
> build and pass tests if I manually munge pg_config.h. We were skipping
> the useless tests via a $windows_os check; I think I should use
> check_pg_config() instead.

Proposed fix attached.

Thanks,
--Jacob

Re: Jacob Champion
> (That means that Windows builds --with-libcurl are similarly broken, I
> think. Not that Windows packagers will want to use --with-libcurl --
> it doesn't do anything -- but it should build.)

Does --with-libcurl still do anything useful if this feature test
fails? From what you are saying, the answer is "no", and I can see
more "not on this platform" error messages in other callbacks.

This should be documented in doc/src/sgml/installation.sgml.

> We could change how this works a bit for the proposed libpq-oauth.so
> plugin, and only build it if we have a workable implementation. I do
> like having these other platforms compile the Curl code, though, since
> we'd prefer to keep the build clean for a future Windows
> implementation...

I would prefer to get an error from configure if the feature doesn't
do anything on my platform. The current way is confusing. If future
users of libcurl change that, the configure test can still be changed.

With the libpq-oauth split, this makes even more sense because
building a library that always throws an error isn't very useful.
(Don't build that file at all if the feature doesn't work.)

Since oauth/curl have some security implications, would it make more
sense to call the switch --enable-oauth (-Doauth) so users could
control better what features their libpq is going to have? Perhaps
some other feature (pg_service as URL?) is going to need libcurl as
well, but it should be configurable separately.

Christoph

> On 1 Apr 2025, at 15:03, Christoph Berg <myon(at)debian(dot)org> wrote:

> With the libpq-oauth split, this makes even more sense because
> building a library that always throws an error isn't very useful.
> (Don't build that file at all if the feature doesn't work.)

After the split, configure/meson should fail if the libcurl dependency isn't
satisfied or if the platform isn't supported.

> Since oauth/curl have some security implications, would it make more
> sense to call the switch --enable-oauth (-Doauth) so users could
> control better what features their libpq is going to have? Perhaps
> some other feature (pg_service as URL?) is going to need libcurl as
> well, but it should be configurable separately.

Perhaps --with-oauth-client for the opt-in libpq-oauth?

--
Daniel Gustafsson

> On 1 Apr 2025, at 02:06, Jacob Champion <jacob(dot)champion(at)enterprisedb(dot)com> wrote:
>
> On Mon, Mar 31, 2025 at 4:09 PM Jacob Champion
> <jacob(dot)champion(at)enterprisedb(dot)com> wrote:
>> I don't have hurd-amd64 to test, but I'm working on a patch that will
>> build and pass tests if I manually munge pg_config.h. We were skipping
>> the useless tests via a $windows_os check; I think I should use
>> check_pg_config() instead.
>
> Proposed fix attached.

Thanks, I agree that this is the right fix. While this is all subject to
change, I will go ahead with this patch in the meantime to make the tree
properly buildable.

--
Daniel Gustafsson

On Tue, Apr 1, 2025 at 6:12 AM Daniel Gustafsson <daniel(at)yesql(dot)se> wrote:
>
> > On 1 Apr 2025, at 15:03, Christoph Berg <myon(at)debian(dot)org> wrote:
>
> > With the libpq-oauth split, this makes even more sense because
> > building a library that always throws an error isn't very useful.
> > (Don't build that file at all if the feature doesn't work.)
>
> After the split, configure/meson should fail if the libcurl dependency isn't
> satisfied or if the platform isn't supported.

Yeah, after sleeping on it I agree. If I want a "canary" buildfarm
animal to opt into compilation on unsupported platforms, I can instead
look into a manual #define or something; it doesn't have to be a
supported configure-time thing.

> > Since oauth/curl have some security implications, would it make more
> > sense to call the switch --enable-oauth (-Doauth) so users could
> > control better what features their libpq is going to have? Perhaps
> > some other feature (pg_service as URL?) is going to need libcurl as
> > well, but it should be configurable separately.
>
> Perhaps --with-oauth-client for the opt-in libpq-oauth?

It started as -Doauth way back when, but was changed as part of the
discussion at [1]. Peter, do you have any objections to switching back
to an OAuth-related name?

--Jacob

[1] https://postgr.es/m/6bde5f56-9e7a-4148-b81c-eb6532cb3651%40eisentraut.org