Integration with CSI

[peter-facko]

Hello everyone,

I am researching the use of SPIFFE in my organization, and currently, I am investigating whether we can use SPIFFE for authentication of FluxCD to an OCI registry.

I have read #3368, but I am not sure if I understand the situation well.

Wouldn't it be possible for Flux to consume secrets as volume: declarations in pods? That way, integration with SPIFFE would be relatively straightforward.

This is also discussed here spiffe/spiffe#369.

[matheuscscp]

For which OCI registry do you need secret-less authentication support? Does the OCI standard define anything with SPIFFE?

[peter-facko]

Sorry, I didn't think about multi-tenancy because we don't use it. Can you please explain more about how mounting a file breaks it?

In a multi-tenant scenario, couldn't the workload that communicates with the OCI registry be specific to one tenant? Also, each tenant would have their own SPIFFE trust domain, so a workload that got a certificate from one trust domain wouldn't be trusted in the other (unless the tenants configure SPIFFE so they trust each other).

We didn't consider other ways of authenticating, such as OIDC, because for workloads, we are currently generating short-lived passwords in Secrets, and we are fine with that solution for now. That is, unless support for SPIFFE came.

Another use case would be organizations that have a policy "no Kubernetes Secret objects". They would use something like https://secrets-store-csi-driver.sigs.k8s.io/ with HCP Vault, for example. They would have this problem as well, not only SPIFFE users. That is why I changed the discussion title to contain CSI instead of SPIFFE.

[stefanprodan]

Flux is meant to service clusters with hundreds of tenants, what would stop a tenant from pointing a Kustomization to another tenant secret if they are all on disk?

Also mounting secrets in containers means a restart I guess. Flux would be incapacitated by this, as it would mean 1000+ restarts per day on clusters with lots of activity. Not to mention that the CSI driver can not update secrets in running pods according to this https://secrets-store-csi-driver.sigs.k8s.io/known-limitations

[peter-facko]

I am sorry, I am not sure I understand the multi-tenant problems with CSI, but I believe you.

I think that https://cert-manager.io/docs/usage/csi-driver-spiffe/ updates secrets fine.

I just wanted to express that being able to integrate Flux with SPIFFE (or CSI in general) would be appreciated by at least one Flux user. Thank you for the responses, Matheus and Stefan!

[matheuscscp]

@peter-facko Check #5681