GCS 429 rateLimitExceeded when Curator keeps re-uploading the same requests_*.jsonl

[paras-genmo]

Running a Gemini batch with Curator 0.1.24:

INFO  Uploaded request batch file to gs://<bucket>/batch_requests/requests_0.jsonl
WARN  Request file …/requests_0.jsonl is being re-submitted.
ERROR Could not upload … :: reason 429 … object exceeded the rate limit for object mutation operations

Curator then loops, re-submits the same object name, and the job never progresses.

full traceback

TooManyRequests: 429 POST https://storage.googleapis.com/upload/storage/v1/b/<bucket>/o?uploadType=multipart: {
  "error": {
    "code": 429,
    "message": "The object <bucket>/batch_requests/requests_0.jsonl exceeded the rate limit for object mutation operations (create, update, and delete).",
    …

Google Cloud Storage enforces a per-object write quota (~1 mutation/s).
If the first multipart upload fails, Curator retries immediately with the same object path, hitting the limit and getting another 429 — forever.

[paras-genmo]

Curator should perform exponential backoff upon 429 errors, and ideally it should write batch requests to different objects since the per object rate limit is very strict.

[kartik4949]

@paras-genmo Hello, we are looking into the issue
Thanks

[kartik4949]

Seems like we need to resubmit the jsonl file with a resubmission tag incase of gemini.
let me see if it can be a tricky to do this.