Use varint length field for last_path encoding to support longer GCP object names

[r-uehara0219]

fix #71

Overview

This PR updates the last_path encoding mechanism in embulk-input-gcs. Instead of using a fixed 1-byte length field (limiting paths to 127 bytes), the new implementation uses a varint length field, which can span 1–2 bytes, to represent the length of the UTF-8 encoded string.

Why Is It Necessary?

GCP object names can be up to 1024 bytes in length. The previous implementation’s 1-byte length field restricted last_path to 128 characters, causing errors when handling longer object names. Switching to a varint length field removes this limitation and ensures that the plugin can support all valid GCP object names.

[hiroyuki-sato]

Hello, @r-uehara0219 Thank you for proposing this PR. We will check this PR. Please wait for a while.

By the way, We are Looking for long-term maintainers around the Embulk eco-system.

If you (and your team) are willing to raise your hand for the community on the total work of GCS plugins, you're welcome.

[dmikurube]

Thanks for your contribution, @r-uehara0219 !

This part seems to come from #47, which was to support Japanese characters, but the 128 limitation was a kind of tentative/ad-hoc one.

This pull-request fixes the ad-hoc one. It basically looks good to me! Then approved, but left a couple of minor comments. Could you address them?

[dmikurube]

It's understandable that this base64Encode method can handle 65536 bytes at most by itself (regardless of the limitation for task.getLastPath()), but it is a little bit confusing for code readers that it has a different limitation from task.getLastPath().

Can you align this with task.getLastPath() and leave a comment to explain why limited by that number?

[r-uehara0219]

Aligned byte limit to 1024 bytes and added a comment.
db7e7b9

[dmikurube]

I understand that this method is indirectly tested via base64Encode, but for sure and for readability, can you add some more tests directly for this encodeVarint method?

[r-uehara0219]

Tests were added. I use reflect to test private methods, but please point out if this should not be used as repository policy.
9f691e6

[dmikurube]

Thanks for your consideration! I think that the method can be just package-private (no visibility modifier) so that the method is visible from test classes.

[r-uehara0219]

Fixed to package-private. Thanks for the comment!
dea553b

[dmikurube]

Hmm, the tests are failing on GitHub Actions. Some of them might be for different reasons (ex. GCP credentials?), but at least testBase64 is failing for the new case. Can you take a look?

[r-uehara0219]

Sorry, only some tests were run in my local environment and the test results were not properly verified. Please wait while I fix the test.
(readme requires GCP_P12_KEYFILE to be specified, but what is actually needed was GCP_PRIVATE_KEYFILE😕 I will revise the readme and commit later.)

[r-uehara0219]

Hmm, the tests are failing on GitHub Actions. Some of them might be for different reasons (ex. GCP credentials?), but at least testBase64 is failing for the new case. Can you take a look?

The test EXPECTED was incorrect and has been corrected.
df160ff

[dmikurube]

Thanks for working on the fix!

Confirmed that testBase64 is fixed, but other failures are still there. I at first wondered if it could be a kind of expiration in access tokens (to GCP), but #73 was working good just yesterday. Then I guess that the cause of the failure is in this pull-request... Could you take a deeper look into it?

[r-uehara0219]

Most of the errors seem to occur when reading the P12 key in AuthUtils.java:98.

@dmikurube
Can you tell me when the GHA environment variable: GCP_PRIVATE_KEYFILE was last updated?

[dmikurube]

They're updated 3 years ago, but #74 passed just a couple of minutes ago... Hmm, could be a kind of permission issues by external contributors? Let me see.

[dmikurube]

Yeah, that's the cause. We should have used the GitHub pull_request_target event, instead of pull_request event, but it also had a security concern.

https://securitylab.github.com/resources/github-actions-preventing-pwn-requests/

We've already set this repo not to run any Workflow against a fork pull-req automatically without approval, then using pull_request_target should be fine.

Okay, let me merge this pull-req now, and let's see the master branch. On the other hand, we'll change the GitHub Actions event in another pull-req.

Thanks for your contribution!

[dmikurube]

Passed. :) https://github.com/embulk/embulk-input-gcs/actions/runs/13852924626/job/38763460455