feat(aws_s3): add local_file_path configuration for uploading files#3926
Open
juniorrhis1 wants to merge 3 commits intoredpanda-data:mainfrom
Open
feat(aws_s3): add local_file_path configuration for uploading files#3926juniorrhis1 wants to merge 3 commits intoredpanda-data:mainfrom
juniorrhis1 wants to merge 3 commits intoredpanda-data:mainfrom
Conversation
|
|
…connect into output-s3-local-file-stream
Contributor
|
Hi @juniorrhis1 👋 Could you please let us know why you'd prefer this approach over something like the config below? input:
file:
paths:
- test.txt
scanner:
to_the_end: {}
output:
aws_s3:
... |
Author
|
Hi @mihaitodor, to avoid load the entire file in memory. For small files or json files with one document per line (using lines scanner) this approach is prefered, but is not possible send large files without consumes a lot of memory. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
closes #3925
This pull request adds support for uploading files to AWS S3 from a specified local file path, in addition to the existing functionality of uploading message content. The changes include updates to both the documentation and the implementation to introduce a new configuration field,
local_file_path, which allows users to specify a local file to upload per message.Key changes:
Feature: Support for uploading from a local file path
local_file_pathto the S3 output component, allowing users to specify the path of a local file to upload instead of message content. This field supports interpolation functions for dynamic file path resolution. [1] [2] [3] [4] [5] [6]local_file_pathis set for each message; if so, it reads the specified file and uploads its contents, otherwise it uploads the message content as before. [1] [2]aws_s3.adocto describe the newlocal_file_pathfield, its type, default value, and usage examples.Implementation details
getUploadBodyto encapsulate the logic for choosing between local file content and message content.os,io).