Skip to content

S3 minimum part size #577

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
rvrangel wants to merge 3 commits into from
Closed

S3 minimum part size #577

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
ad51403
Migrate the S3 SDK from v1 to v2 (#16664)
frouioui Sep 17, 2024
eb62685
add S3 minimum part size defined by the user (#17171)
rvrangel Dec 26, 2024
ee28ac4
Merge branch 'slack-19.0' into s3-minimum-part-size
rvrangel Jan 14, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions examples/operator/operator.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -511,6 +511,9 @@ spec:
maxLength: 256
pattern: ^[^\r\n]*$
type: string
minPartSize:
format: int64
type: integer
region:
minLength: 1
type: string
Expand Down Expand Up @@ -1351,6 +1354,9 @@ spec:
maxLength: 256
pattern: ^[^\r\n]*$
type: string
minPartSize:
format: int64
type: integer
region:
minLength: 1
type: string
Expand Down Expand Up @@ -3898,6 +3904,9 @@ spec:
maxLength: 256
pattern: ^[^\r\n]*$
type: string
minPartSize:
format: int64
type: integer
region:
minLength: 1
type: string
Expand Down Expand Up @@ -5247,6 +5256,9 @@ spec:
maxLength: 256
pattern: ^[^\r\n]*$
type: string
minPartSize:
format: int64
type: integer
region:
minLength: 1
type: string
Expand Down
117 changes: 67 additions & 50 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,23 +3,22 @@ module vitess.io/vitess
go 1.22.8

require (
cloud.google.com/go/storage v1.39.0
cloud.google.com/go/storage v1.43.0
github.com/AdaLogics/go-fuzz-headers v0.0.0-20230811130428-ced1acdcaa24
github.com/Azure/azure-pipeline-go v0.2.3
github.com/Azure/azure-storage-blob-go v0.15.0
github.com/DataDog/datadog-go v4.8.3+incompatible
github.com/HdrHistogram/hdrhistogram-go v0.9.0 // indirect
github.com/aquarapid/vaultlib v0.5.1
github.com/armon/go-metrics v0.4.1 // indirect
github.com/aws/aws-sdk-go v1.50.32
github.com/buger/jsonparser v1.1.1
github.com/cespare/xxhash/v2 v2.2.0
github.com/cespare/xxhash/v2 v2.3.0
github.com/corpix/uarand v0.1.1 // indirect
github.com/dave/jennifer v1.7.0
github.com/evanphx/json-patch v5.9.0+incompatible
github.com/fsnotify/fsnotify v1.7.0
github.com/go-sql-driver/mysql v1.7.1
github.com/golang/glog v1.2.0
github.com/golang/glog v1.2.1
github.com/golang/protobuf v1.5.4
github.com/golang/snappy v0.0.4
github.com/google/go-cmp v0.6.0
Expand All @@ -33,8 +32,7 @@ require (
github.com/hashicorp/go-immutable-radix v1.3.1 // indirect
github.com/hashicorp/serf v0.10.1 // indirect
github.com/icrowley/fake v0.0.0-20180203215853-4178557ae428
github.com/jmespath/go-jmespath v0.4.0 // indirect
github.com/klauspost/compress v1.17.7
github.com/klauspost/compress v1.17.9
github.com/klauspost/pgzip v1.2.6
github.com/krishicks/yaml-patch v0.0.10
github.com/magiconair/properties v1.8.7 // indirect
Expand All @@ -49,9 +47,9 @@ require (
github.com/pires/go-proxyproto v0.7.0
github.com/pkg/errors v0.9.1
github.com/planetscale/pargzip v0.0.0-20201116224723-90c7fc03ea8a
github.com/planetscale/vtprotobuf v0.5.0
github.com/planetscale/vtprotobuf v0.6.1-0.20240319094008-0393e58bdf10
github.com/prometheus/client_golang v1.19.0
github.com/prometheus/common v0.49.0 // indirect
github.com/prometheus/common v0.50.0 // indirect
github.com/sjmudd/stopwatch v0.1.1
github.com/soheilhy/cmux v0.1.5
github.com/spf13/cobra v1.8.0
Expand All @@ -69,30 +67,36 @@ require (
go.etcd.io/etcd/client/pkg/v3 v3.5.12
go.etcd.io/etcd/client/v3 v3.5.12
go.uber.org/mock v0.2.0
golang.org/x/crypto v0.23.0 // indirect
golang.org/x/mod v0.16.0 // indirect
golang.org/x/net v0.25.0
golang.org/x/oauth2 v0.18.0
golang.org/x/sys v0.20.0
golang.org/x/term v0.20.0
golang.org/x/text v0.15.0 // indirect
golang.org/x/time v0.5.0
golang.org/x/tools v0.19.0
google.golang.org/api v0.168.0
google.golang.org/genproto v0.0.0-20240304212257-790db918fca8 // indirect
google.golang.org/grpc v1.62.1
golang.org/x/crypto v0.26.0 // indirect
golang.org/x/mod v0.17.0 // indirect
golang.org/x/net v0.28.0
golang.org/x/oauth2 v0.22.0
golang.org/x/sys v0.25.0
golang.org/x/term v0.23.0
golang.org/x/text v0.17.0 // indirect
golang.org/x/time v0.6.0
golang.org/x/tools v0.21.1-0.20240508182429-e35e4ccd0d2d
google.golang.org/api v0.193.0
google.golang.org/genproto v0.0.0-20240814211410-ddb44dafa142 // indirect
google.golang.org/grpc v1.66.0
google.golang.org/grpc/cmd/protoc-gen-go-grpc v1.3.0
google.golang.org/grpc/examples v0.0.0-20210430044426-28078834f35b
google.golang.org/protobuf v1.33.0
google.golang.org/protobuf v1.34.2
gopkg.in/DataDog/dd-trace-go.v1 v1.61.0
gopkg.in/asn1-ber.v1 v1.0.0-20181015200546-f715ec2f112d // indirect
gopkg.in/ldap.v2 v2.5.1
sigs.k8s.io/yaml v1.4.0
)

require (
github.com/Shopify/toxiproxy/v2 v2.8.0
github.com/Shopify/toxiproxy/v2 v2.9.0
github.com/aws/aws-sdk-go-v2 v1.30.4
github.com/aws/aws-sdk-go-v2/config v1.27.31
github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.17.15
github.com/aws/aws-sdk-go-v2/service/s3 v1.60.1
github.com/aws/smithy-go v1.20.4
github.com/bndr/gotabulate v1.1.2
github.com/dustin/go-humanize v1.0.1
github.com/gammazero/deque v0.2.1
github.com/google/safehtml v0.1.0
github.com/hashicorp/go-cleanhttp v0.5.2
Expand All @@ -109,41 +113,55 @@ require (
go.uber.org/goleak v1.3.0
go.uber.org/zap v1.27.0
golang.org/x/exp v0.0.0-20240222234643-814bf88cf225
golang.org/x/sync v0.6.0
golang.org/x/sync v0.8.0
gonum.org/v1/gonum v0.14.0
modernc.org/sqlite v1.29.2
)

require (
cloud.google.com/go v0.112.1 // indirect
cloud.google.com/go/compute v1.25.0 // indirect
cloud.google.com/go/compute/metadata v0.2.3 // indirect
cloud.google.com/go/iam v1.1.6 // indirect
github.com/DataDog/appsec-internal-go v1.5.0 // indirect
github.com/DataDog/datadog-agent/pkg/obfuscate v0.51.1 // indirect
github.com/DataDog/datadog-agent/pkg/remoteconfig/state v0.51.1 // indirect
cloud.google.com/go v0.115.1 // indirect
cloud.google.com/go/auth v0.9.2 // indirect
cloud.google.com/go/auth/oauth2adapt v0.2.4 // indirect
cloud.google.com/go/compute/metadata v0.5.0 // indirect
cloud.google.com/go/iam v1.2.0 // indirect
github.com/DataDog/appsec-internal-go v1.7.0 // indirect
github.com/DataDog/datadog-agent/pkg/obfuscate v0.56.1 // indirect
github.com/DataDog/datadog-agent/pkg/remoteconfig/state v0.56.1 // indirect
github.com/DataDog/datadog-go/v5 v5.5.0 // indirect
github.com/DataDog/go-libddwaf/v2 v2.3.2 // indirect
github.com/DataDog/go-sqllexer v0.0.11 // indirect
github.com/DataDog/go-tuf v1.0.4-0.5.2-debug // indirect
github.com/DataDog/sketches-go v1.4.4 // indirect
github.com/Microsoft/go-winio v0.6.1 // indirect
github.com/DataDog/go-sqllexer v0.0.14 // indirect
github.com/DataDog/go-tuf v1.1.0-0.5.2 // indirect
github.com/DataDog/sketches-go v1.4.6 // indirect
github.com/Microsoft/go-winio v0.6.2 // indirect
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.6.4 // indirect
github.com/aws/aws-sdk-go-v2/credentials v1.17.30 // indirect
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.16.12 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.16 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.16 // indirect
github.com/aws/aws-sdk-go-v2/internal/ini v1.8.1 // indirect
github.com/aws/aws-sdk-go-v2/internal/v4a v1.3.16 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.11.4 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.3.18 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.11.18 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.17.16 // indirect
github.com/aws/aws-sdk-go-v2/service/sso v1.22.5 // indirect
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.26.5 // indirect
github.com/aws/aws-sdk-go-v2/service/sts v1.30.5 // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/coreos/go-semver v0.3.1 // indirect
github.com/coreos/go-systemd/v22 v22.5.0 // indirect
github.com/cpuguy83/go-md2man/v2 v2.0.3 // indirect
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
github.com/dustin/go-humanize v1.0.1 // indirect
github.com/ebitengine/purego v0.6.1 // indirect
github.com/fatih/color v1.16.0 // indirect
github.com/ebitengine/purego v0.8.1 // indirect
github.com/fatih/color v1.18.0 // indirect
github.com/felixge/httpsnoop v1.0.4 // indirect
github.com/go-logr/logr v1.4.1 // indirect
github.com/go-logr/logr v1.4.2 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
github.com/google/s2a-go v0.1.7 // indirect
github.com/googleapis/enterprise-certificate-proxy v0.3.2 // indirect
github.com/googleapis/gax-go/v2 v2.12.2 // indirect
github.com/google/s2a-go v0.1.8 // indirect
github.com/googleapis/enterprise-certificate-proxy v0.3.3 // indirect
github.com/googleapis/gax-go/v2 v2.13.0 // indirect
github.com/hashicorp/errwrap v1.1.0 // indirect
github.com/hashicorp/go-hclog v1.6.2 // indirect
github.com/hashicorp/go-multierror v1.1.1 // indirect
Expand All @@ -164,7 +182,7 @@ require (
github.com/pelletier/go-toml/v2 v2.1.1 // indirect
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
github.com/prometheus/client_model v0.6.0 // indirect
github.com/prometheus/procfs v0.12.0 // indirect
github.com/prometheus/procfs v0.13.0 // indirect
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
github.com/rivo/uniseg v0.4.7 // indirect
github.com/rogpeppe/go-internal v1.12.0 // indirect
Expand All @@ -178,17 +196,16 @@ require (
github.com/tidwall/match v1.1.1 // indirect
github.com/tidwall/pretty v1.2.1 // indirect
go.opencensus.io v0.24.0 // indirect
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.49.0 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.49.0 // indirect
go.opentelemetry.io/otel v1.24.0 // indirect
go.opentelemetry.io/otel/metric v1.24.0 // indirect
go.opentelemetry.io/otel/trace v1.24.0 // indirect
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.52.0 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.53.0 // indirect
go.opentelemetry.io/otel v1.28.0 // indirect
go.opentelemetry.io/otel/metric v1.28.0 // indirect
go.opentelemetry.io/otel/trace v1.28.0 // indirect
go.uber.org/atomic v1.11.0 // indirect
go.uber.org/multierr v1.11.0 // indirect
golang.org/x/xerrors v0.0.0-20231012003039-104605ab7028 // indirect
google.golang.org/appengine v1.6.8 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20240304212257-790db918fca8 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20240304212257-790db918fca8 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20240814211410-ddb44dafa142 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20240827150818-7e3bb234dfed // indirect
gopkg.in/ini.v1 v1.67.0 // indirect
gopkg.in/yaml.v2 v2.4.0 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
Expand Down
Loading
Loading