Memory usage of Kwok clusters

[adibrastegarnia]

We need to use Kwok to simulate thousands of clusters, and I am seeing each cluster consuming around 300MB of memory. Is there a way to reduce it further?

[wzshiming]

Are you starting the cluster with an all-in-one image? It's basically the apiserver that's taking up all the memory, and I haven't found a way to reduce the apiserver's memory footprint yet.

[adibrastegarnia]

@wzshiming yes I use all in one image. I tried to disable a bunch of controllers that I don't need, it helped a bit but not much. I am not sure if even it is technically possible to reduce it further.

[wzshiming]

Yes, if you look at apiserver's pprof you can see that about 100M+ is taken up by OpenAPI v2/v3, which in many cases are not used at all. You may need to change the code to disable them.

[adibrastegarnia]

@wzshiming thanks do you have any pointer to that part?

[wzshiming]

https://github.com/kubernetes/kubernetes/blob/4bed36e03e7bd699b089d33da6f7d7c9ef9eb661/staging/src/k8s.io/apiserver/pkg/server/genericapiserver.go#L443-L453

I'm not sure which features are disabled when you disable them.