videoMMMU.py failed to unzip the downloaded dataset

[dkmenglie]

videoMMMY.py can download the dataset from Huggingface correctly. But it doesn't unzip them to video subfolder. Accordingly the benchmark will fail.

The root cause is in unzip_hf_zip()
target_dir = base_dir / 'videos'
target_dir.mkdir(exist_ok=True)

if not target_dir.exists():
# unzip logic...
else:
print('The video file already exists.')

The target_dir is always exists since mkdir(exist_ok=True). We should change next line to if target_dir.exists()

[sry2]

Indeed, same problem here.
Seems the line target_dir.mkdir(exist_ok=True) is misplaced and should be inside the conditioned hunk, like the similar implementation in VideoMME, which is correct. Maybe too less using and feedback of VideoMMMU makes the contributors neglect the BUG here. Should be fixed ASAP

[sry2]

Update: I believe another BUG is found also at the decompression of VideoMMMU:

VLMEvalKit/vlmeval/dataset/videommmu.py

Lines 504 to 505 in 2c25371

	with source, open(target, 'wb'):
	target.write(source.read())

here target is NOT a file pointer that presents in VideoMME.py:

VLMEvalKit/vlmeval/dataset/videomme.py

Lines 103 to 104 in 2c25371

	with source, target:
	target.write(source.read())

but a PosixPath object.
A possible fix might be:

with source, open(target, 'wb') as f:
    f.write(source.read())

Please check this carefully and update an official fix on it.