Adding wolfSSL support as ngtcp2 crypto library and examples.

[icing]

QUIC support has been added to wolfSSL (commit fa979230050dde47f6ac084c6f3b232a5e729eb5)
and ngtcp2 bindings are done as a separate crypto library.

wolfssl has been added as option to configure.

wolfssl based example client/server are build when configured
and have been verified to work against google/cloudflare/fastly
servers.

[tatsuhiro-t]

It looks like root CMakeList.txt lacks wolfSSL detection.

[icing]

It looks like root CMakeList.txt lacks wolfSSL detection.

Me and cmake are not really on good terms. I'd rather learn Japanese!

But let me give this a try...

[tatsuhiro-t]

wolfSSL provides pkgconifg file, so it is relatively easy to deal with.
For example, look at the gnutls detection code:

• ngtcp2/CMakeLists.txt

  Lines 89 to 91 in dfcdcc8

  	if(ENABLE_GNUTLS)
  	find_package(GnuTLS 3.7.2)
  	endif()

• ngtcp2/CMakeLists.txt

  Lines 162 to 171 in dfcdcc8

  	# GnuTLS (required for libngtcp2_crypto_gnutls)
  	if(ENABLE_GNUTLS AND GNUTLS_FOUND)
  	set(GNUTLS_INCLUDE_DIRS ${GNUTLS_INCLUDE_DIR})
  	set(HAVE_GNUTLS TRUE)
  	set(HAVE_CRYPTO TRUE)
  	else()
  	set(HAVE_GNUTLS FALSE)
  	set(GNUTLS_INCLUDE_DIRS "")
  	set(GNUTLS_LIBRARIES "")
  	endif()

If you have hard time or no time to deal with cmake, then leave it to me.

[icing]

Always willing to learn. I have installed all my quic related libs in /opt/quic. How would I invoke cmake to search there first?

[tatsuhiro-t]

cmake understands PKG_CONFIG_PATH environment variable. List the custom install path in the variable. Then cmake will search them first.

[tatsuhiro-t]

With the above comments, I can successfully compile the patch.

After some tries of client and server, wsslclient works regular connection (nice!). But it is unable to send 0-RTT.
wsslserver is unable to get ALPN, that is wolfSSL_ALPN_GetProtocol returns -9.

I built wolfssl with the following configure options: --enable-quic --enable-session-ticket --enable-earlydata --enable-alpn
wolfssl provides fine grained feature options, so I might miss something important.

[tatsuhiro-t]

Right, --enable-all fixes ALPN issue.

[tatsuhiro-t]

wsslserver has some issues on resumption.

[icing]

wsslserver has some issues on resumption.

Manual testing or do you have some scripting?

[tatsuhiro-t]

I use ngtcp2 examples/client to test resumption and 0 RTT.
Adding --session-file session.txt --tp-file tp.txt to client command-line will save session info in session.txt and quic transport parameters in tp.txt. Run the client with the same options again, which will read session info and transport parameters from the files and try to do resumption.

[icing]

I use ngtcp2 examples/client to test resumption and 0 RTT.
Adding --session-file session.txt --tp-file tp.txt to client command-line will save session info in session.txt and quic transport parameters in tp.txt. Run the client with the same options again, which will read session info and transport parameters from the files and try to do resumption.

Thanks. Can reproduce. wsslclient can resume against wsslserver, but client fails. Hmm...

[icing]

It looks like root CMakeList.txt lacks wolfSSL detection.

cmake is now working on my machine. please check.

[icing]

I think I isolated the problem in wolfSSL that leads to failures in resumption with an openssl client. See wolfSSL/wolfssl#5452 for the PR that demonstrates the problem and a possible solution.

[tatsuhiro-t]

It sounds like the same behavior as boringssl does.

ngtcp2/crypto/boringssl/boringssl.c

Line 428 in 26b0d3c

if (SSL_in_early_data(ssl)) {

boringssl SSL_do_handshake returns 1 when it sends server finished to read early data from client. It provides SSL_in_early_data function to determine the actual TLS handshake completed or not. ngtcp2 boringssl binding uses it to decide whether keep calling SSL_do_handshake or not.

[tatsuhiro-t]

cmake is now working on my machine. please check.

Thank you. cmake now works for wolfssl binding library.

[icing]

It sounds like the same behavior as boringssl does.
...
boringssl SSL_do_handshake returns 1 when it sends server finished to read early data from client. It provides SSL_in_early_data function to determine the actual TLS handshake completed or not. ngtcp2 boringssl binding uses it to decide whether keep calling SSL_do_handshake or not.

Yes, indeed.

wolfSSL merged my addition of a wolfSSL_quic_do_handshake() that handles this situation and I pushed the necessary changes here now.

In my local testing, openssl example client resumes against wolfssl server and early data seems to work as well.

[tatsuhiro-t]

Yeah, it is much better now.
Resumption works, but early data is rejected by server: Early data was rejected by server.

It seems that wolfssl server only accepts early data if wolfSSL_read_early_data is called. Inside it, it setups ssl->earlyData = expecting_early_data. QUIC does not use the function. Needs to do this somewhere.

[tatsuhiro-t]

While I was playing around wolfSSL server 0RTT, it looks like it sends early_data extension even if it rejects resumption.
It also happens with the regular TLSv1.3 connection over TCP.

[icing]

I am wrestling with it. Seems that feature could do with some more attention.

[icing]

Ok, just created wolfSSL/wolfssl#5458 which in my tests with openssl/wolfssl based client/server examples here no longer rejects early data.

No changes necessary to this PR other than the git change id update after wolfSSL merged.

[tatsuhiro-t]

Yes, the PR has been merged, and I confirmed that 0RTT now works for both wsslclient and wsslserver. Thank you very much.

The remaining issue is when wsslserver receives a session ticket which it cannot resume.
With the latest wolfssl build, I observed that QUIC handshake encryption and decryption key do not agree between client and wsslserver in this case.

[tatsuhiro-t]

It is interesting that the disagreement of keys happens if session ticket has not been expired yet (you can trigger this condition by restarting server, and use the ticket that the old server issued). After some time, I guess that the ticket has been expired, now both parties agree with the keys, but server still sends early_data extension in full handshake. OpenSSL does not like this and issues TLS alert.

[tatsuhiro-t]

The issue mentioned above happens with regular TLSv1.3 connection over TCP.
Steps to repro:

1. Run wolfssl examples server: examples/server/server -0 -v 4 -d -C10000
2. Run openssl client: openssl s_client -connect localhost:11111 -sess_out session.txt -early_data data.txt
3. Stop openssl client (ctrl-C)
4. Stop wolfssl examples server
5. Start wolfssl server again with the same options.
6. Run openssl client: openssl s_client -connect localhost:11111 -sess_out session.txt -sess_in session.txt -early_data data.txt

Now I get 140059142509952:error:1408F119:SSL routines:ssl3_get_record:decryption failed or bad record mac:../ssl/record/ssl3_record.c:676:

After some time, do step 5 and 6 (needs step 5 because wolfssl example server stops as a result of step 6) and I get 140609374721408:error:1421A06E:SSL routines:tls_parse_stoc_early_data:bad extension:../ssl/statem/extensions_clnt.c:1946:

It seems that these are not QUIC specific. I'm not sure how the fix impacts the QUIC stuff, but I think we can go ahead and just polish up the current PR and merge it.

[icing]

The issue mentioned above happens with regular TLSv1.3 connection over TCP. Steps to repro:

1. Run wolfssl examples server: examples/server/server -0 -v 4 -d -C10000
2. Run openssl client: openssl s_client -connect localhost:11111 -sess_out session.txt -early_data data.txt
3. Stop openssl client (ctrl-C)
4. Stop wolfssl examples server
5. Start wolfssl server again with the same options.
6. Run openssl client: openssl s_client -connect localhost:11111 -sess_out session.txt -sess_in session.txt -early_data data.txt

Now I get 140059142509952:error:1408F119:SSL routines:ssl3_get_record:decryption failed or bad record mac:../ssl/record/ssl3_record.c:676:

After some time, do step 5 and 6 (needs step 5 because wolfssl example server stops as a result of step 6) and I get 140609374721408:error:1421A06E:SSL routines:tls_parse_stoc_early_data:bad extension:../ssl/statem/extensions_clnt.c:1946:

Nice find. I will try to reproduce and then open an Issue at wolfSSL about it.

It seems that these are not QUIC specific. I'm not sure how the fix impacts the QUIC stuff, but I think we can go ahead and just polish up the current PR and merge it.

Excellent! Many thanks for the fast testing. I'll polish based on your feedback and rebase/squash afterwards.

[icing]

Sorry, for having to stumble across all my leftovers.

[icing]

Reported the resumption problem as wolfSSL/wolfssl#5463.

[tatsuhiro-t]

Thank you for fixing all issues. Now LGTM.

[vszakats]

Can someone confirm that the TLS-backend specific headers ngtcp2_crypto.h and ngtcp2_crypto_wolfssl.h are not copied by make install to their destination when using CMake?

UPDATE: PR opened.