Gerrit UI sporadically fails with 504 bad-gateway

[raghavendrapj]

We are consistently observing 504's on our gerrit. I left JavaMelody to instrument our production gerrit instance. We are running Gerrit on a kubernetes cluster with Dockerfile from openfrontier/docker-gerrit. The data is persisted on a NFS share. From what we have noticed Indexing in Gerrit is handled by a third-party library Lucene, and it does not support NFS mounts. Since the data directory is coming from NFS, This causes Gerrit WebUI to have very bad response times. Can we have Lucene configured to not use NFS instead use local memory and if that is possible can this setting be baked into base dockerfile?

[thinkernel]

Hi @raghavendrapj
I can't find anyway to relocate index storage to other directory though the gerrit.config. If you've already known a config option I'd like to know it. And I'd like to add these options in to the dockerfile.

However if we can't relocate the index out of the volume, the only way I can find right now is to separate data volume.
There's one volume which is mounted on /var/gerrit/review_site by default. It has to be separated to at least 5 volumes which will be mounted on /var/gerrit/review_site/git, /var/gerrit/review_site/index, /var/gerrit/review_site/cache, /var/gerrit/review_site/db, /var/gerrit/review_site/etc. Also, there's a gerrit_version file which is located directly under /var/gerrit/review_site should be moved into one of those new volumes. And I also have to find a way to make sure those existed containers can be migrated to the new one automatically which is tricky. So you can see that I prefer not changing the volume unless someone can propose a better way.
My question is why not use another way to provide your persistent volumes in k8s like ceph rbd? Our gerrit is running in k8s with a ceph rbd as the pv which works perfectly.

[raghavendrapj]

Thank You for your prompt response.

As you rightly pointed out there is no better way to relocate the index and only way to do that is by exposing all the 5 volumes. The official gerrit docker image already does that.

We are also evaluating options like StorageOS for persistent Volume which offers similar benefits as ceph.

Just in case if i come across a better way to do this I will run it through you.