Proxy environment variables shouldn't be passed to apiserver

[andrewshilliday]

What keywords did you search in kubeadm issues before filing this one?

environment, proxy

Is this a BUG REPORT or FEATURE REQUEST?

BUG REPORT

Versions

kubeadm version (use kubeadm version):
kubeadm version: &version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.2", GitCommit:"5fa2db2bd46ac79e5e00a4e6ed24191080aa463b", GitTreeState:"clean", BuildDate:"2018-01-18T09:42:01Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Kubernetes version (use kubectl version):
  Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.2", GitCommit:"5fa2db2bd46ac79e5e00a4e6ed24191080aa463b", GitTreeState:"clean", BuildDate:"2018-01-18T10:09:24Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
  Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.2", GitCommit:"5fa2db2bd46ac79e5e00a4e6ed24191080aa463b", GitTreeState:"clean", BuildDate:"2018-01-18T09:42:01Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration:

• OS (e.g. from /etc/os-release):
  NAME="Red Hat Enterprise Linux Workstation"
  VERSION="7.4 (Maipo)"
  ID="rhel"
  ID_LIKE="fedora"
  VARIANT="Workstation"
  VARIANT_ID="workstation"
  VERSION_ID="7.4"
  PRETTY_NAME="Red Hat Enterprise Linux"
  ANSI_COLOR="0;31"
  CPE_NAME="cpe:/o:redhat:enterprise_linux:7.4:GA:workstation"
  HOME_URL="https://www.redhat.com/"
  BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.4
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.4"

• Kernel (e.g. uname -a):
  Linux [redacted] 3.10.0-693.2.2.el7.x86_64 kubeadm join on slave node fails preflight checks #1 SMP Sat Sep 9 03:55:24 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux

• Others:

What happened?

My cluster sits behind a corporate firewall and as such, we have http_proxy, https_proxy, and no_proxy environment variables defined in order to reach outside severs. When calling kubdadm the enviromnment gets passed along to the apiserver manifest (and to others). However, unless your no_proxy settings are configured to include all of the IP adresses that will be used by kubernetes for pods and services, the api server may incorrectly try to use the proxy settings to access services.

For example, if you start up the dashboard and attempt to access it through the api server through http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/, the service will fail with an EOF error because the api server cannot access the dashboard (because it attempts to use the proxy settings).

What you expected to happen?

To the best of my knowledge, the api server never needs to access the outside world, only local services and endpoints, therefore, I see no reason why the environment should be passed to it when the manifest is created.

How to reproduce it (as minimally and precisely as possible)?

[ See example above ]

Anything else we need to know?

The obvious workaround is to not have proxy settings defined when you call kubeadm init. And this is a workable solution. However, some of the services that start up do expect to call home for various reasons (e.g., kubeadm attempts to fetch the latest version number of kubernetes if it's not passed as a command line), and those requests will fail if required proxy settings are not defined.

[dixudx]

/assign

Sent out PR kubernetes/kubernetes#58698 to fix this.

[kad]

@andrewshilliday while you were initialising cluster with kubeadm, what kind of warning messages it gave to your setup ?

[kad]

@andrewshilliday Any update on the question above ? or should we just close the issue ?

/assign

[andrewshilliday]

There was an initial warning about an ip address pointing to a proxy, which I fixed by adding it to the no_proxy variable. Then it went through without warnings and I started getting the error accessing the dashboard (as detailed above). Now that I understand kubeadm a bit more, I believe that what it wanted during that first check was to ensure that all assignable ip addresses are omitted via the no_proxy setting. However that list could easily be thousands of ip address long, and no_proxy doesn't allow for ip ranges, so that's not really practical.

Regardless, I can see that kubeadm, when it gets pods starts up, seems to pick and choose which pods get the proxy environment variables and which don't (they don't all have them set). The point I was trying to make in this issue is that I don't see a reason why the apiserver pod should have the proxy settings propagated into the pod. If I'm correct that it only ever needs to access local resources (i.e., the pods and services), then it can probably get by without propagating proxy settings.

[dixudx]

no_proxy doesn't allow for ip ranges, so that's not really practical

It is okay to set cidr for no_proxy in Kubernetes.

https://github.com/kubernetes/kubernetes/blob/07e849c98698b19de636d6f7282dbc0b2edebde7/staging/src/k8s.io/apimachinery/pkg/util/net/http.go#L86-L88

Kubernetes does involve a way to handle this circumstance.

@andrewshilliday I think this issue can be closed now.

[kad]

There are reasons why control plane services might need to access external hosts and those hosts can be only accessible over proxy. e.g. #5
kubeadm warns about scenarios that are potentially can lead to incorrectly working cluster, but not preventing sysadmin to put that into that state.

With support of CIDR notation in NO_PROXY, it is easy to control which internal networks will be accessed via proxy or not, including pod/service CIDRs.

/close