以下是对这些方法的详细解释和补充:
检查服务器配置
- 证书有效性
- 证书的有效期至关重要,过期的证书会导致连接被客户端拒绝。除了使用
openssl
工具检查证书细节外,还可以在服务器的证书管理界面查看证书的到期时间。对于生产环境,应设置提醒机制,提前更新即将过期的证书。 - 证书的颁发机构必须是受信任的。如果证书是自签名的,需要将其根证书手动导入到客户端的信任列表中,确保客户端能够验证服务器的身份。
- 证书的有效期至关重要,过期的证书会导致连接被客户端拒绝。除了使用
- 密码套件
- 不同的密码套件提供不同级别的安全性和性能。一些旧的或弱的密码套件容易受到攻击,应避免使用。例如,
RC4
- based密码套件存在安全风险,应禁用。 - 服务器配置文件中通常可以指定支持的密码套件列表。例如,在Apache服务器中,可以通过
SSLCipherSuite
指令来配置。应根据服务器和客户端的兼容性,选择一组安全且高效的密码套件。
- 不同的密码套件提供不同级别的安全性和性能。一些旧的或弱的密码套件容易受到攻击,应避免使用。例如,
- TLS版本
- 确保服务器支持最新的TLS版本,如TLS 1.3,它在安全性和性能方面都有显著提升。同时,为了兼容性,也可以保留TLS 1.2。
- 配置服务器时,要注意不同TLS版本的兼容性问题。有些旧的客户端可能不支持最新的TLS版本,需要进行适当的配置调整,以确保能够与各种客户端进行安全通信。
检查客户端问题
- 证书信任
- 客户端的操作系统或应用程序维护着一个受信任的证书颁发机构列表。如果服务器的证书颁发机构不在这个列表中,连接将被视为不安全。在Windows系统中,可以通过“管理工具” - “证书”来查看和管理受信任的根证书颁发机构。
- 对于一些特定的应用程序,它们可能有自己独立的证书信任管理机制。例如,Java应用程序可以通过
keytool
命令来管理其信任的证书。
- TLS配置
- 客户端的TLS配置应与服务器兼容。这包括TLS版本和密码套件的设置。一些客户端库可能有默认的TLS配置,可能需要根据服务器的要求进行调整。
- 在移动应用开发中,不同的平台(如Android和iOS)有各自的TLS配置方式。例如,在Android中,可以通过
SSLSocketFactory
来配置TLS相关参数。
- 网络设置
- 网络中的防火墙、代理服务器等设备可能会影响TLS连接。防火墙可能会阻止某些端口或协议,需要确保允许TLS流量通过。常见的HTTPS端口是443,应确保该端口在防火墙上是开放的。
- 如果使用代理服务器,要确保代理服务器正确配置了对TLS流量的处理。有些代理服务器可能需要进行特殊的设置,以支持SSL - Tunneling,以便正确转发TLS加密的数据包。
分析连接过程
- 数据包捕获
- Wireshark是一款强大的数据包捕获工具,可以详细分析TLS握手过程中的各种消息。通过分析数据包,可以确定问题出在哪个阶段,例如证书交换失败、密码套件协商不一致等。
- 在捕获数据包时,要注意过滤出与TLS相关的流量,以便更清晰地查看连接过程。可以使用Wireshark的显示过滤器,如
ssl
或tls
来过滤相关数据包。
- 服务器日志
- 服务器日志是排查问题的重要依据。不同的服务器软件有不同的日志格式和位置。例如,Nginx服务器的错误日志通常位于
/var/log/nginx/error.log
。 - 日志中会记录各种与TLS连接相关的信息,如连接请求、证书验证结果、错误代码等。通过分析这些日志,可以快速定位问题所在。同时,要注意日志的级别设置,确保能够记录足够详细的信息。
- 服务器日志是排查问题的重要依据。不同的服务器软件有不同的日志格式和位置。例如,Nginx服务器的错误日志通常位于
更新软件和库
- 服务器端
- 服务器软件的更新通常会包含对TLS相关功能的改进和安全漏洞修复。例如,OpenSSL库的更新可能会修复一些已知的安全问题,提高TLS连接的安全性。
- 在更新服务器软件时,要注意兼容性问题。一些更新可能会导致配置文件的格式发生变化,需要及时调整。同时,要进行充分的测试,确保更新后服务器的正常运行。
- 客户端
- 客户端软件的更新也同样重要。例如,浏览器的更新可能会改进对TLS的支持,提高连接的稳定性和安全性。
- 对于一些应用程序客户端,开发人员应及时更新相关的网络库和TLS库,以确保与服务器的兼容性和安全性。同时,要向用户提供及时的更新提示,确保用户能够使用到最新的、安全的版本。
Troubleshooting TLS - enabled Connections involves several steps to identify and resolve issues that may occur when establishing secure connections using Transport Layer Security (TLS). Here is a comprehensive guide:
Check Server Configuration
- Certificate Validity:
- Ensure the server’s TLS certificate is valid, not expired, and issued by a trusted Certificate Authority (CA). You can use tools like
openssl
to check the certificate details. For example,openssl x509 -in server.crt -text -noout
will display the certificate information. If the certificate is self - signed, make sure the client trusts the self - signed CA.
- Ensure the server’s TLS certificate is valid, not expired, and issued by a trusted Certificate Authority (CA). You can use tools like
- Cipher Suites:
- Check that the server supports appropriate cipher suites. The server should be configured with strong and up - to - date cipher suites. You can use tools like
nmap
with the--script ssl - cipher - suites.nse
option to scan the server for supported cipher suites. If weak cipher suites are found, they should be disabled, and stronger ones like AES - GCM - SHA256 should be enabled.
- Check that the server supports appropriate cipher suites. The server should be configured with strong and up - to - date cipher suites. You can use tools like
- TLS Version:
- Verify that the server is configured to use a supported and secure TLS version, such as TLS 1.2 or 1.3. TLS 1.0 and 1.1 are considered less secure and should be phased out. You can use
openssl s_client -connect example.com:443 -tls1_2
to test a connection using a specific TLS version. If the server does not support the required version, it may need to be updated in the server configuration.
- Verify that the server is configured to use a supported and secure TLS version, such as TLS 1.2 or 1.3. TLS 1.0 and 1.1 are considered less secure and should be phased out. You can use
Inspect Client - Side Issues
- Certificate Trust:
- On the client side, ensure that the root CA certificates used to verify the server’s certificate are installed and up - to - date. If the client does not trust the server’s certificate, the connection will fail. In a browser, you can check the certificate settings and manage trusted CAs. In other applications, the process may vary depending on the programming language and libraries used.
- TLS Configuration:
- Check the client - side TLS configuration. Make sure the client is configured to use the appropriate TLS version and cipher suites that are compatible with the server. Some clients may have default configurations that need to be adjusted. For example, in Java, you can set the
https.protocols
system property to specify the supported TLS versions.
- Check the client - side TLS configuration. Make sure the client is configured to use the appropriate TLS version and cipher suites that are compatible with the server. Some clients may have default configurations that need to be adjusted. For example, in Java, you can set the
- Network Settings:
- Ensure that the client can reach the server over the network. Check for any firewalls, proxies, or network restrictions that may be blocking the TLS - enabled connection. If a proxy is used, make sure it is configured correctly to handle TLS - encrypted traffic. You can try to bypass the proxy temporarily to see if the connection works.
Analyze the Connection Process
- Packet Capture:
- Use packet - capture tools like Wireshark to capture the network traffic during the TLS handshake and connection process. This can help identify any errors or anomalies in the communication. Look for messages related to the TLS handshake, such as certificate exchanges, cipher suite negotiations, and any error messages. For example, an “alert” message in the TLS protocol may indicate a problem like a certificate error or an unsupported cipher suite.
- Server Logs:
- Check the server logs for any error messages related to the TLS - enabled connection. The server logs can provide detailed information about what went wrong during the connection attempt. For example, an Apache server with TLS enabled will log errors in the
error_log
file. Look for messages related to certificate validation failures, connection timeouts, or problems with the TLS handshake.
- Check the server logs for any error messages related to the TLS - enabled connection. The server logs can provide detailed information about what went wrong during the connection attempt. For example, an Apache server with TLS enabled will log errors in the
Update Software and Libraries
- Server - Side:
- Keep the server software, including the web server, application server, and any relevant libraries, up to date. Software updates often include bug fixes, security patches, and improvements to the TLS implementation. For example, if you are using Apache Tomcat, make sure to update it to the latest version to benefit from the latest TLS - related improvements.
- Client - Side:
- Update the client - side software and libraries as well. This includes browsers, operating systems, and any application - specific libraries that handle TLS connections. Outdated client software may have bugs or security vulnerabilities that can affect the TLS - enabled connection. For example, updating a web browser to the latest version can fix issues with TLS certificate validation.
If the problem persists after trying these steps, it may be necessary to consult the documentation of the specific software and libraries being used, or seek help from relevant technical communities or support channels.
以下是一些关于排查 TLS-enabled Connections 问题的方法:
证书相关问题
- 证书不受信任:访问网站时,如果浏览器提示证书不受信任,需将 CA 证书添加到客户端的信任根存储中。例如在 Windows 系统中,可通过管理计算机证书,找到“受信任的根证书颁发机构”下的“证书”,选择“所有任务”中的“导入”,然后选择导出的 crt 文件完成导入。在 Ubuntu 系统中,可使用命令
sudo cp <exported crt file> /usr/local/share/ca-certificates
和sudo update-ca-certificates
来添加新的 CA 证书。 - 证书过期或无效:使用在线工具如 SSL Labs 的 SSL Test 检查 SSL 证书是否过期或无效。如果证书存在问题,应及时更新证书。
协议和加密套件问题
- TLS 版本不匹配:确保客户端和服务器都支持较新的、安全的 TLS 版本,如 TLS 1.2 或 TLS 1.3。可使用 OpenSSL 的 s_client 命令来测试特定的 TLS 版本,例如
openssl s_client -host <server> -port <port> -tls1_2
,根据测试结果调整服务器或客户端的 TLS 版本配置。 - 加密套件不支持:检查客户端和服务器支持的加密套件是否一致。在服务器端,可能需要更新配置以支持客户端使用的现代加密套件。同样可以利用 OpenSSL 的 s_client 命令,通过指定
-cipher
选项来测试特定的加密套件,如openssl s_client -host <server> -port <port> -cipher ECDHE-RSA-AES256-GCM-SHA384
,从而确定双方是否支持共同的加密套件。
网络连接问题
- 防火墙或安全软件干扰:某些安全程序可能会限制加密连接。可暂时禁用安全软件,查看是否能正常建立 TLS 连接,以判断是否是安全软件导致的问题。如果是防火墙拦截了请求,可尝试将相关网站添加到允许列表中。
- 网络连通性问题:使用工具如
ping
或traceroute
检查客户端和服务器之间的网络连通性,确保没有网络设备阻止了 SSL/TLS 端口(通常是 443 端口),同时确认服务器正在正确监听的 IP 地址和端口。
浏览器相关问题
- 浏览器配置或插件问题:有时浏览器配置错误或浏览器插件可能会导致 TLS 握手失败。可以尝试更换浏览器,如果问题依旧,可能是浏览器插件导致,可尝试禁用所有插件后再进行测试。此外,还可以检查浏览器是否使用了最新的 SSL 协议,可通过访问 SSL Labs 的 SSL Client Test 页面进行检查。
- 浏览器缓存和 SSL 状态问题:清除浏览器缓存和 SSL 状态可能有助于解决一些 TLS 连接问题。例如在 Chrome 浏览器中,可通过设置中的“清除浏览数据”选项来清除缓存,还可通过“更改代理设置”中的“内容”选项卡来清除 SSL 状态。
其他问题
- 系统时间和日期问题:如果客户端的系统日期和时间不准确,可能会导致 SSL/TLS 握手失败,因为 SSL 证书有有限的生命周期和到期日期,所以需要确保系统时间和日期正确。
- 服务器配置问题:检查服务器的 SSL/TLS 协议设置,确保其配置正确。如果服务器配置了错误的 SSL/TLS 设置,可能会导致连接失败,可根据服务器的具体类型和配置要求进行相应的调整。
This guide covers a methodology and some tooling that can help diagnose TLS connectivity issues and errors (TLS alerts). It accompanies the main guide on TLS in RabbitMQ. The strategy is to test the required components with an alternative TLS implementation in the process of elimination to identify the problematic end (client or server).
Bear in mind that this process is not guaranteed to identify the problem if the interaction between two specific components is responsible for the problem.
The steps recommended in this guide are:
Verify effective configuration
Verify that the node listens for TLS connections
Verify file permissions
Verify TLS support in Erlang/OTP
Verify certificate/key pairs and test with alternative TLS client or server using OpenSSL command line tools
Verify available and configured cipher suites and certificate key usage options
Verify client connections with a TLS-terminating proxy
And finally, test a real client connection against a real server connection again
When testing with a RabbitMQ node and/or a real RabbitMQ client it is important to inspect logs for both server and client.
Check Effective Node Configuration
Setting up a RabbitMQ node with TLS involves modifying configuration. Before performing any other TLS troubleshooting steps it is important to verify config file location and effective configuration (whether the node has loaded it successfully). See Configuration guide for details.
Check TLS Listeners (Ports)
This step checks that the broker is listening on the expected port(s), such as 5671 for AMQP 0-9-1 and 1.0, 8883 for MQTT, and so on.
To verify that TLS has been enabled on the node, use rabbitmq-diagnostics listeners or the listeners section in rabbitmqctl status.
The listeners sections will look something like this:
Interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Interface: [::], port: 5671, protocol: amqp/ssl, purpose: AMQP 0-9-1 and AMQP 1.0 over TLS
Interface: [::], port: 15672, protocol: http, purpose: HTTP API
Interface: [::], port: 15671, protocol: https, purpose: HTTP API over TLS (HTTPS)
Interface: [::], port: 1883, protocol: mqtt, purpose: MQTT
In the above example, there are 6 TCP listeners on the node. Two of them accept TLS-enabled connections:
Inter-node and CLI tool communication on port 25672
AMQP 0-9-1 (and 1.0, if enabled) listener for non-TLS connections on port 5672
AMQP 0-9-1 (and 1.0, if enabled) listener for TLS-enabled connections on port 5671
HTTP API listeners on ports 15672 (HTTP) and 15671 (HTTPS)
MQTT listener for non-TLS connections 1883
With rabbitmqctl status it will look like so:
% …
{listeners,
[{clustering,25672,“::”},
{amqp,5672,“::”},
{‘amqp/ssl’,5671,“::”},
{http,15672,“::”}]}
% …
If the above steps are not an option, inspecting node’s log file can be a viable alternative. It should contain an entry about a TLS listener being enabled, looking like this:
2018-09-02 14:24:58.611 [info] <0.664.0> started TCP listener on [::]:5672
2018-09-02 14:24:58.614 [info] <0.680.0> started SSL listener on [::]:5671
If the node is configured to use TLS but a message similar to the above is not logged, it is possible that the configuration file was placed at an incorrect location and was not read by the broker or the node was not restarted after config file changes. See the configuration page for details on config file verification.
Tools such as lsof and netstat can be used to verify what ports a node is listening on, as covered in the Troubleshooting Networking guide.
Check Certificate, Private Key and CA Bundle File Permissions
RabbitMQ must be able to read its configured CA certificate bundle, server certificate and private key. The files must exist and have the appropriate permissions. Incorrect permissions (e.g. files being owned by root or another superuser account that installed them) is a very common issue with TLS setups.
On Linux, BSD and MacOS directory permissions can also affect node’s ability to read the files.
When certificate or private key files are not readable or do not exist, the node fail to accept TLS-enabled connections or TLS connections will just hang (the behavior differs between Erlang/OTP versions).
When new style configuration format is used to configure certificate and private key paths, the node will check if the files exist on boot and refuse to start if that’s not the case.
Check TLS Support in Erlang
Another key requirement for establishing TLS connections to the broker is TLS support in the broker. Confirm that the Erlang VM has support for TLS by running
rabbitmq-diagnostics --silent tls_versions
Or, on Windows
rabbitmq-diagnostics.bat --silent tls_versions
The output will look like this:
tlsv1.2
tlsv1.1
tlsv1
sslv3
With versions that do not provide rabbitmq-diagnostics tls_versions, use
rabbitmqctl eval ‘ssl:versions().’
Or, on Windows
rabbitmqctl.bat eval ‘ssl:versions().’
The output in this case will look like so:
[{ssl_app,“9.1”},
{supported,[‘tlsv1.2’,‘tlsv1.1’,tlsv1]},
{supported_dtls,[‘dtlsv1.2’,dtlsv1]},
{available,[‘tlsv1.2’,‘tlsv1.1’,tlsv1,sslv3]},
{available_dtls,[‘dtlsv1.2’,dtlsv1]}]
If an error is reported instead, confirm that the Erlang/OTP installation includes TLS support.
It is also possible to list cipher suites available on a node:
rabbitmq-diagnostics cipher_suites --openssl-format --silent
Or, on Windows:
rabbitmq-diagnostics.bat cipher_suites --openssl-format --silent
It is also possible to inspect what TLS versions are supported by the local Erlang runtime. To do so, run erl (or werl.exe on Windows) on the command line to open an Erlang shell and enter
%% the trailing dot is significant!
ssl:versions().
Note that this will report supported versions on the local node (for the runtime found in PATH), which may be different from that used by RabbitMQ node(s) inspected.
Use OpenSSL Tools to Test TLS Connections
OpenSSL s_client and s_server are commonly used command line tools that can be used to test TLS connections and certificate/key pairs. They help narrow problems down by testing against alternative TLS client and server implementations. For example, if a certain TLS client works successfully with s_server but not a RabbitMQ node, the root cause is likely on the server end. Likewise if an s_client client can successfully connect to a RabbitMQ node but a different client cannot, it’s the client setup that should be inspected closely first.
The example below seeks to confirm that the certificates and keys can be used to establish a TLS connection by connecting an s_client client to an s_server server in two separate shells (terminal windows).
The example will assume you have the following certificate and key files (these filenames are used by tls-gen):
Item Location
CA certificate (public key) ca_certificate.pem
Server certificate (public key) server_certificate.pem
Server private key server_key.pem
Client certificate (public key) client_certificate.pem
Client private key client_key.pem
In one terminal window or tab execute the following command:
openssl s_server -accept 8443
-cert server_certificate.pem -key server_key.pem -CAfile ca_certificate.pem
It will start an OpenSSL s_server with that uses the provided CA certificate bundler, server certificate and private key. It will be used to sanity check the certificates with test TLS connections against this example server.
In another terminal window, run the following command, substituting CN_NAME with the expected hostname or CN name from the certificate:
openssl s_client -connect localhost:8443
-cert client_certificate.pem -key client_key.pem -CAfile ca_certificate.pem
-verify 8 -verify_hostname CN_NAME
It will open a new TLS connection to the example TLS server started above. You may leave off the -verify_hostname argument but OpenSSL will no longer perform that verification.
If the certificates and keys have been correctly created, an TLS connection output will appear in both tabs. There is now a connection between the example client and the example server, similar to telnet.
If the trust chain could be established, the second terminal will display a verification confirmation with the code of 0:
Verify return code: 0 (ok)
Just like with command line tools, a non-zero code communicates and error of some kind.
If an error is reported, confirm that the certificates and keys were generated correctly and that a matching certificate/private key pair is used. In addition, certificates can have their usage scenarios restricted at generation time. This means a certificate meant to be used by clients to authenticate themselves will be rejected by a server, such as a RabbitMQ node.
For environments where self-signed certificates are appropriate, we recommend using tls-gen for generation.
Validate Available Cipher Suites
RabbitMQ nodes and clients can be limited in what cipher suites they are allowed to use during TLS handshake. It is important to make sure that the two sides have some cipher suites in common or otherwise the handshake will fail.
Certificate’s key usage properties can also limit what cipher suites can be used.
See Configuring Cipher Suites and Public Key Usage Extensions in the main TLS guide to learn more.
openssl ciphers -v
will display all cipher suites supported by the local build of OpenSSL.
Attempt TLS Connection to a RabbitMQ Node
Once a RabbitMQ node was configure to listen on an TLS port, the OpenSSL s_client can be used to test TLS connection establishment, this time against the node. This check establishes whether the broker is likely to be configured correctly, without needing to configure an RabbitMQ client. The tool can also be useful to compare the behaviour of different clients. The example assumes a node running on localhost on default TLS port for AMQP 0-9-1 and AMQP 1.0, 5671:
openssl s_client -connect localhost:5671 -cert client_certificate.pem -key client_key.pem -CAfile ca_certificate.pem
The output should appear similar to the case where port 8443 was used. The node log file should contain a new entry when the connection is established:
2018-09-27 15:46:20 [info] <0.1082.0> accepting AMQP connection <0.1082.0> (127.0.0.1:50915 -> 127.0.0.1:5671)
2018-09-27 15:46:20 [info] <0.1082.0> connection <0.1082.0> (127.0.0.1:50915 -> 127.0.0.1:5671): user ‘user’ authenticated and granted access to vhost ‘virtual_host’
The node will expect clients to perform protocol handshake (AMQP 0-9-1, AMQP 1.0 and so on). If that doesn’t happen within a short time window (10 seconds by default for most protocols), the node will close the connection.
Validate Client Connections with Stunnel
stunnel is a tool that can be used to validate TLS-enabled clients. In this configuration clients will make a secure connection to stunnel, which will pass the decrypted data through to a “regular” port of the broker (say, 5672 for AMQP 0-9-1 and AMQP 1.0). This provides some confidence that the client TLS configuration is correct independently of the broker TLS configuration.
stunnel is a specialised proxy. In this example it will run in daemon mode on the same host as the broker. In the discussion that follows it is assumed that stunnel will only be used temporarily. It is also possible to use stunnel to perform TLS termination but that is out of scope for this guide.
In this example stunnel will connect to the unencrypted port of the broker (5672) and accept TLS connections from TLS-capable clients on port 5679:
cat client_key.pem client_certificate.pem > client/key-cert.pem
stunnel -r localhost:5672 -d 5679 -f -p client/key-cert.pem -D 7
stunnel requires a certificate and its corresponding private key. The certificate and private key files must be concatenated as shown above with the cat command. stunnel requires that the key not be password-protected. TLS-capable clients should now be able to connect to port 5679 and any TLS errors will appear on the console where stunnel was started.
Validate RabbitMQ Client Connection to RabbitMQ Node
Assuming none of the previous steps produced errors then you can confidently connect the tested TLS-enabled client to the TLS-enabled port of the broker, making sure to stop any running OpenSSL s_server or stunnel instances first.
Certificate Chains and Verification Depth
When using a client certificate signed by an intermediate CA, it may be necessary to configure RabbitMQ server to use a higher verification depth.
Insufficient verification depth will result in TLS peer verification failures.
Understanding TLS Connection Log Errors
New broker logfile entries will be generated during many of the preceding steps. These entries together with diagnostic output from commands on the console should help to identify the cause of TLS-related errors. What follows is a list of the most common error entries:
Logged Errors Explanation
Entries containing {undef, [{crypto,hash,… The crypto module is missing in the Erlang/OTP installation used or it is out of date. On Debian, Ubuntu, and other Debian-derived distributions it usually means that the erlang-ssl package was not installed.
Entries containing {ssl_upgrade_error, ekeyfile} or {ssl_upgrade_error, ecertfile} This means the broker keyfile or certificate file is invalid. Confirm that the keyfile matches the certificate and that both are in PEM format. PEM format is a printable encoding with recognisable delimiters. The certificate will start and end with -----BEGIN CERTIFICATE----- and -----END CERTIFICATE----- respectively. The keyfile will likewise start and end with -----BEGIN RSA PRIVATE KEY----- and -----END RSA PRIVATE KEY----- respectively.
Entries containing {ssl_upgrade_failure, … certify …} This error is related to client verification. The client is presenting an invalid certificate or no certificate. If the ssl_options has the verify option set to verify_peer then try using the value verify_none temporarily. Ensure that the client certificate has been generated correctly, and that the client is presenting the correct certificate.
Entries containing {ssl_upgrade_error, …} This is a generic error that could have many causes. Make sure you are using the recommended version of Erlang.
Entries containing {tls_alert,“bad record mac”} The server has tried verifying integrity of a piece of data it received and the check failed. This can be due to problematic network equipment, unintentional socket sharing in the client (e.g. due to the use of fork(2)) or a bug in the client implementation of TLS.
Getting Help and Providing Feedback
If you have questions about the contents of this guide or any other topic related to RabbitMQ, don’t hesitate to ask them on the RabbitMQ mailing list.
Help Us Improve the Docs ❤️
If you’d like to contribute an improvement to the site, its source is available on GitHub. Simply fork the repository and submit a pull request. Thank you!