Upstream part of smarter way to pick endpoints for dual-stack IPv4/v6 support

[soulxu]

As @mattklein123 said at #16123 (comment) the issue #16123 can be separated into two parts: upstream and downstream.

The downstream part is about Support multiple addresses in the listener #11184, and it is under review.

So it is time to look at the upstream part.

A little background here:

With multiple addresses in listener, it gets rid of the duplicated listeners. but since people want to choose endpoint smartly (ipv4 request choose ipv4 endpoint, ipv6 request choose ipv6 endpoint), still have to add duplicated config.

• One way is using two duplicated filter chains to choose two duplicated clusters.
• Another way is using the endpoint metadata to get rid of the duplicated clusters, it is a little improved but still, has two duplicated filter chains. The example config can be referenced here listener: support multiple addresses #19367 (comment)

So I'm thinking there need three things to reach the final goal:

• Choose a correct endpoint
  This still can base on the endpoint metadata, but it needs to set the different metadata based on the downstream connection's IP version automatically. We can extend the HTTP Set-Metadata filter https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/filters/http/set_metadata/v3/set_metadata.proto to support converting some variables (the variables like this https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_conn_man/headers#custom-request-response-headers) to the metadata. Like a new %IPVERSION% variable, then convert that variable to specific metadata. (maybe add a network filter also to support the TCP use case)
• Choose a correct upstream local address
  Since Istio using the upstream_bind_config in Cluster (https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/core/v3/address.proto#envoy-v3-api-msg-config-core-v3-bindconfig) to set the upstream local address as 127.0.0.6 for the connection to the App, it will do the same thing for the IPv6 also. So we also need a way to choose the correct upstream local address. We can implement this by adding a new policy to the upstream_bind_config, allowing the binding of different local addresses based on the IP version.
• Choose a correct endpoint for the dynamic proxy case or use the DNS name for the endpoint address.
  I think this can be done by the HappyEyeballConnection. The dns_lookup_family will be set as ALL as the HappyEyeballConnection requirement https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/cluster/v3/cluster.proto#envoy-v3-api-enum-config-cluster-v3-cluster-dnslookupfamily
  The HappyEyeballConnection will get both ipv4 and ipv6 addresses from the DNS, then we can add a new policy to allow it to choose the address based on the downstream IP version.
  And this PR is on my radar Add a connection provider to happy eyeball connection implementation and split it into a new class for broader use cases #21539, we can a new connection provider for it.

Appreciate any feedback and comments on this, let me know whether those proposals are enough for this use-case and correct for this usecase!

[soulxu]

@mattklein123 @ggreenway @lambdai

Also, @jacob-delgado @howardjohn looking for your guys' feedback from the Istio view.

Appreciate for any feedback, and thanks in advance!

[lambdai]

My 2 cents:

• downstream metadata
  Now that this v4/v6 selection is across protocols on top of TCP, it might worth encode metadata or filter state in listener.

• upstream bind
  Upstream bind need to be extended to provide multiple bind address for sure. In terms of consuming the new addresses, IMHO it is natual to bind v4 address if the v4 upstream address.

[soulxu]

Thanks for the feedback!

• downstream metadata
  Now that this v4/v6 selection is across protocols on top of TCP, it might worth encode metadata or filter state in listener.

Yes, we can do that in the listener or done by a filter, both works for me. If we want to avoid to hardcode a metadata, then we can do that in the filter.

[howardjohn]

Not sure how much this is "entire driven by Istio" vs "Istio also had a use case", but IIUC we actually hadn't concluded on whether the correct approach was to ensure v4 downstream goes to v4 upstream or if they are decoupled.

[soulxu]

Not sure how much this is "entire driven by Istio" vs "Istio also had a use case", but IIUC we actually hadn't concluded on whether the correct approach was to ensure v4 downstream goes to v4 upstream or if they are decoupled.

Thanks for the feedback! At least I think I shouldn't move this forward before istio has a conclusion here.

@jacob-delgado @zhlsunshine probably needs you guys' help to move the discussion on the istio side.

[jacob-delgado]

Envoy should allow a way for users to pick what's best for their project. There are other projects besides Istio that may need this capability and each should choose a way that fits their needs. I'm not sure if that's possible or if I just increased the scope by a lot, however.

Our customers would like a "native ip" solution. IPv4 downstream should go to IPv4 upstream, if possible. Same for IPv6. There is some concern about underlying network issues by proxying across protocols.

[zhlsunshine]

In my opinion, it will make sense if the solution can help to reduce the consume and improve performance of Envoy for Istio dual stack support.

[soulxu]

Our customers would like a "native ip" solution. IPv4 downstream should go to IPv4 upstream, if possible. Same for IPv6. There is some concern about underlying network issues by proxying across protocols.

I prefer to add a network filter to add metadata, it is generic and can be used for other usecase.

[soulxu]

May I get some feedback from you @mattklein123 @ggreenway ? also @RyanTheOptimist for HappyEyeBallConnection, thanks in advance!