[backend] Add the ability to specify driver labels/annotations as a KFP api server configuration

[mginfn]

Edited by: @HumairAK

Based on the discussion below, this issue is now allowing admin users to configure what labels/annotations are added to driver pods at the KFP API Server level.

---

Environment

• How did you deploy Kubeflow Pipelines (KFP)?
  Kubeflow manifests (kustomize)
• KFP version:
  2.5.0
• KFP SDK version:
  2.13.0

Steps to reproduce

Enable STRICT mTLS on the entire mesh, then run a pipeline.

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: mesh-traffic
  namespace: istio-system
spec:
  mtls:
    mode: STRICT

Expected result

Pipeline runs successfully

Materials and Reference

KFP driver pods run without Istio sidecars, therefore they can't connect to services within the mesh (if mTLS STRICT mode is enabled).

In particular, the default Kubeflow installation is shipped with MinIO object store, the driver pod fails when trying to connect to it:

level=info msg="Saving file to s3" bucket=mlpipeline endpoint="minio-service.kubeflow:9000" key=artifacts/mnist-pipeline-mdtkv/2025/06/25/mnist-pipeline-mdtkv-system-dag-driver-938794099/main.log path=/tmp/argo/outputs/logs/main.log
level=info msg="Transient error: Get \"http://minio-service.kubeflow:9000/mlpipeline/?location=\": read tcp 10.244.2.191:60656->10.108.206.88:9000: read: connection reset by peer"

Notice also the MinIO pod Istio sidecar that reports a mismatch of TLS configuration between
the downstream client (driver pod) and the upstream server (MinIO Istio sidecar):

"- - -" 0 NR filter_chain_not_found - "-" 0 0 0 - "-" "-" "-" "-" "-" - - 10.244.2.152:9000 10.244.2.191:60656 - -

Another service which is impacted is the MLDM gRPC service, the driver pod fails with error:

KFP driver: driver.RootDAG(pipelineName=mnist-pipeline, runID=xyz, runtimeConfig, componentSpec) failed: Failed GetContextByTypeAndName(type="system.Pipeline", name="mnist-pipeline"): rpc error: code = Unavailable desc = connection error: desc = "error reading server preface: read tcp 10.244.2.191:51708->10.105.73.97:8080: read: connection reset by peer"

I could fix the above errors errors by selectively disabling mTLS:

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: minio-traffic
  namespace: kubeflow
spec:
  selector:
    matchLabels:
      app: minio
  portLevelMtls:
    9000:
      mode: PERMISSIVE
---
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: metadata-grpc-server-traffic
  namespace: kubeflow
spec:
  selector:
    matchLabels:
      component: metadata-grpc-server
  portLevelMtls:
    8080:
      mode: PERMISSIVE 

I wonder whether this is the right approach or there's a better solution to address mTLS communication.
Would it be an option to run driver pods within the mesh as well?
Thank you

---

Impacted by this bug? Give it a 👍.

[HumairAK]

I'm not super savvy with istio, would it be sufficient if we gave the option of allowing admins to configure kfp deployment to include istio inject annotations sidecar.istio.io/inject: "true" to driver pods across the board?

Iirc you can test if that works by labelling the namespace where you are running pipelines to auto inject all pods:

kubectl label namespace kubeflow istio-injection=enabled --overwrite

and confirm if that solves the problem, if so I think the annotation solution could be one way around this - in general we want to give admins flexibility of adding annotations/labels to driver pods and this could fall under that

[mginfn]

Hi @HumairAK, I'm running a multi-tenant Kubeflow instance, where all namespaces (i.e. the kubeflow namespace and all profile namespaces) are labeled to enable Istio sidecar injection.
However, KFP add annotation sidecar.istio.io/inject: false to driver PODs, which seems enough to disable Istio sidecar injection (I tried applying a mutating webhook to remove that annotation, but it seems my webhook runs after the Istio webhook and injection is skipped anyhow).
For KFP impl pods, I can manually add the injection label with kubernetes.add_pod_label(...) (taking care of issue #11077).
For driver pods, would it be possible to add labels/annotations using the KFP SDK? Otherwise, yes it could be a good option to allow admins to configure KFP deployments to include Istio inject annotations.
Thank you.

[HumairAK]

For driver pods, would it be possible to add labels/annotations using the KFP SDK? Otherwise, yes it could be a good option to allow admins to configure KFP deployments to include Istio inject annotations.

@mginfn Okay so I think we're agreed on allowing admins to do this at the KFP config level. In general istio is in my mind, an admin level concern and the pipeline author should not have to worry about this, rather their admin should have this configured upon KFP deployment.

For the SDK, we want to avoid having the user think about driver pods as this is an implementation detail, in the future we want to abstract this further and get rid of driver pods entirely.

Where it gets tricky is that driver pods don't have a 1:1 mapping to component execution pods (where user code runs). So when a user does:

add_pod_label(sometask, ...) 

It is not always clear exactly which driver pods should get that label. We could add a feature that allows you to set it at the pipeline config level, so when you execute a run, all pods in that pipeline will always get that label, including all driver pods. Would that be a good alternative?

[mginfn]

Hi @HumairAK, I agree that pipeline authors should not worry about Istio setup, however it seems to me that this isn't always possible: in general, if a pod needs to access a service in the Istio mesh (assuming STRICT mTLS is enabled), that pod must run in the mesh (through Istio sidecar proxy). This applies to both KFP impl pods (in my case, a KFP task that registers a model to the Kubeflow Model Registry) and driver pods (that e.g. need to save artifacts in the Kubeflow MinIO instance).
However, not all impl pods need to access services in the mesh (e.g. a pod that post to a remote out-of-mesh HTTP url), therefore the add_pod_label seems to be the best option to enable Istio sidecar injection only if needed.
Regarding driver pods, as mentioned above, I noticed they need to access the in-mesh MinIO service (at least if you don't configure a different storage) and MLMD gRPC server (which is in-mesh as well), so probably they all need to run in-mesh (at least if STRICT mTLS is enabled) and it makes sense to enable/disable in-mesh for them all with a single setting at pipeline level.

[HumairAK]

Regarding driver pods, as mentioned above, I noticed they need to access the in-mesh MinIO service

IIRC driver pods don't ever access object store, only the executor impl pods. Where did you see them access object store?

You're correct that they do access mlmd gprc server, as well as KFP server, and the k8s server. I believe that should be it, which is why I think this should be a KFP level configuration, since all of these pods probably need to be in the same mesh, right?

[mginfn]

IIRC driver pods don't ever access object store, only the executor impl pods. Where did you see them access object store?

As far as I can see, driver pods save logs file in the MinIO instance.
These are the logs of a "system-dag-driver" pod, which would fail if it can't connect to MinIO (see first issue in this thread).

time="2025-06-27T07:49:31.729Z" level=info msg="No output artifacts"
time="2025-06-27T07:49:31.729Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/main.log, key: artifacts/mnist-pipeline-pwrlr/2025/06/27/mnist-pipeline-pwrlr-system-dag-driver-2611184278/main.log"
time="2025-06-27T07:49:31.729Z" level=info msg="Creating minio client using static credentials" endpoint="minio-service.kubeflow:9000"
time="2025-06-27T07:49:31.729Z" level=info msg="Saving file to s3" bucket=mlpipeline endpoint="minio-service.kubeflow:9000" key=artifacts/mnist-pipeline-pwrlr/2025/06/27/mnist-pipeline-pwrlr-system-dag-driver-2611184278/main.log path=/tmp/argo/outputs/logs/main.log
time="2025-06-27T07:49:31.743Z" level=info msg="Save artifact" artifactName=main-logs duration=14.045825ms error="<nil>" key=artifacts/mnist-pipeline-pwrlr/2025/06/27/mnist-pipeline-pwrlr-system-dag-driver-2611184278/main.log

You're correct that they do access mlmd gprc server, as well as KFP server, and the k8s server. I believe that should be it, which is why I think this should be a KFP level configuration, since all of these pods probably need to be in the same mesh, right?

Yes, they need to be in the same mesh.

[HumairAK]

These are the logs of a "system-dag-driver" pod, which would fail if it can't connect to MinIO (see first issue in this thread).

Ah yes, this is if you are using argo workflows artifact storage, which is an argo level configuration, in this case you are right that driver uses object store if configured via the argo workflow configmap.

[mginfn]

Hi @HumairAK , can we summarize this thread as a feature request that allows pipeline authors to set (at pipeline level) that all driver pods will run within istio mesh (i.e. with istio sidecars injected)?
Thanks!

[HumairAK]

I'm very much open to:

1. allowing KFP admins to set, at the pipeline server, what pod labels/annotations/etc. should be set for drivers.
2. for pipeline authors, we can add a feature that will apply blanked pod labels to all pipeline pods, I think that's a reasonable request, this could maybe be a PipelineConfig option.

Now whether (2) should also apply to driver pods (i.e. the main ask), I think I could use more opinions here, as I'm not 100% sure this is the right call. I also see this as being at odds with (1), and my preference is we go (1) route because again, drivers are very much an infra level concern, and not something pipeline authors are supposed to be thinking about.

@mprahl / @droctothorpe what do you all think?

[droctothorpe]

I lean towards (1), especially because we're planning to centralize the driver anyway -- exposing functionality in the SDK that will soon be deprecated doesn't make sense. Does (1) meet your needs, @mginfn ? It helps that it's a lot easier / faster to implement.

[mginfn]

Hi @droctothorpe , yes (1) is fine: enabling strict mTLS mesh-wide is indeed a system configuration, so settings labels/annotations as (1) accordingly seems to be a coherent option.
Thanks.