performance decrease after finetuning

I finetune the pretrained model using this script: 

python scripts/train.py --batch_size 8 --lr 3e-4 --max_num_frames 40 --param_schedule --datapath data_clevrer/clevrer/ --gt_datapath data_clevrer/clevrer/ --model_name clevrer --T 5  --K 6 --continue_train

however, the ari seems to decrease after a few iteration, Is it because of the learning rate?  