Poor performance and poor results

[astariul]

I'm trying to fine tune BERT on STS-B dataset.

I used the following notebook to fine tune it using BERT-keras.
(As described in the paper, I just added a classification layer using the CLS token of the output of BERT).

However, there is great differences in performance and results between this notebook and the script used in the official version for fine tuning :

	BERT-keras	Official BERT
Pearson	0.0254	0.8956
Spearman	0.0289	0.7942
MSE	2.2691	0.5456
Training time	9h	10min

Note : Pearson / Spearman and correlation metrics used to evaluate the accuracy on the STS-B dataset

---

Why there is such a difference between the 2 approach ?

[Separius]

Hmm, 9 hours compared to 10 minutes? wow, that is horrible, sadly though I'm swamped with work and can't figure out what's wrong (I checked your code and it seems fine). @HighCWu do you have any ideas?

[HighCWu]

I guess it should be caused by a small batch size. Have you used the same training batch size and the same epoch as the official? @colanim large batch size keeps the training steady, but it also means that the training time will be longer.
If it is the same batch size and epoch, then the problem may be in this implementation.

[astariul]

Thanks for the answer. I used same epoch. With the official implementation I used batch size of 32, with this one batch size of 64, so I don't think the problem come from here.

I think there is a problem in my code, because 9h ! Compare to 10 with the tensorflow script, this is weird..

[MrKamiZhou]

I tried this code and https://github.com/hanxiao/bert-as-service to get sentence representation, and tensorflow is much faster, like 200ms vs 2000ms

[Separius]

Thanks @MrKamiZhou, so I'm guessing that something is wrong here because Keras shouldn't be this slow. I will try to figure it out as soon as I have some free time.