VTGate VStream [Backport]: Ensure Reasonable Delivery Time for Reshard Journal Events#585
Merged
tanjinx merged 23 commits intoslack-19.0from Mar 24, 2025
Merged
Conversation
|
Thanks for the contribution! Before we can merge this, we need @mattlord to sign the Salesforce Inc. Contributor License Agreement. |
|
@makinje16 do we still need this? |
Author
Yes we do. Am planning to switch over to this since heartbeat backport is done. |
…vent (vitessio#16639) Signed-off-by: Malcolm Akinje <malcolm.akinje@gmail.com> Signed-off-by: Malcolm Akinje <makinje@slack-corp.com>
…nd variables in time related function calls (#590) * Fix accepting bind variables in time related function calls. (vitessio#14763) Signed-off-by: Manan Gupta <manan@planetscale.com> * fix test --------- Signed-off-by: Manan Gupta <manan@planetscale.com> Co-authored-by: Manan Gupta <35839558+GuptaManan100@users.noreply.github.com>
This upgrade allows us to control whether vtorc raises problems or not via an environment variable. Signed-off-by: Eduardo J. Ortega U. <5791035+ejortegau@users.noreply.github.com>
This is a backport of vitessio#17526 . Original PR description below: Description This is meant to make recovery actions more easily identified from the logs. See vitessio#17465 Signed-off-by: Eduardo J. Ortega U. <5791035+ejortegau@users.noreply.github.com>
* Ensure all topo read calls consider `--topo_read_concurrency` (vitessio#17276) Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * Revert "add keyrange support for vtorc clusters_to_watch (#457)" This reverts commit 45c2199. * [release-19.0] `vtorc`: require topo for `Healthy: true` in `/debug/health` (vitessio#17129) (vitessio#17351) Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> Signed-off-by: Manan Gupta <manan@planetscale.com> Co-authored-by: vitess-bot[bot] <108069721+vitess-bot[bot]@users.noreply.github.com> Co-authored-by: Tim Vaillancourt <tim@timvaillancourt.com> Co-authored-by: Manan Gupta <manan@planetscale.com> * `vtorc`: fetch all tablets from cells once + filter during refresh (vitessio#17388) Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * Support KeyRange in `--clusters_to_watch` flag (vitessio#17604) Signed-off-by: Manan Gupta <manan@planetscale.com> * missing func Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * Add api end point to print the current database state in VTOrc (vitessio#15485) Signed-off-by: Manan Gupta <manan@planetscale.com> --------- Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> Signed-off-by: Manan Gupta <manan@planetscale.com> Co-authored-by: vitess-bot[bot] <108069721+vitess-bot[bot]@users.noreply.github.com> Co-authored-by: Manan Gupta <manan@planetscale.com> Co-authored-by: Manan Gupta <35839558+GuptaManan100@users.noreply.github.com>
…599) * `vtorc`: improve handling of partial cell topo results Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * add unit test Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * improve test Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * add comments Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * move sort to test Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * goimports Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> --------- Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
* Add stats for shards watched by VTOrc Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * Use len() in make --------- Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
Signed-off-by: Manan Gupta <manan@planetscale.com> Co-authored-by: Manan Gupta <35839558+GuptaManan100@users.noreply.github.com>
* [release-19.0] smartconnpool: do not allow connections to starve (vitessio#17675) (vitessio#17683) Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com> Co-authored-by: vitess-bot[bot] <108069721+vitess-bot[bot]@users.noreply.github.com> * smartconnpool: Better handling for idle expiration (vitessio#17756) Signed-off-by: Vicent Marti <vmg@strn.cat> --------- Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com> Signed-off-by: Vicent Marti <vmg@strn.cat> Co-authored-by: vitess-bot[bot] <108069721+vitess-bot[bot]@users.noreply.github.com> Co-authored-by: Vicent Martí <42793+vmg@users.noreply.github.com> Co-authored-by: Tim Vaillancourt <tim@timvaillancourt.com>
Signed-off-by: Harshit Gangal <harshit@planetscale.com> Signed-off-by: Vicent Martí <42793+vmg@users.noreply.github.com> Co-authored-by: Harshit Gangal <harshit@planetscale.com> Co-authored-by: Vicent Martí <42793+vmg@users.noreply.github.com>
* `vtorc`: remove duplicate instance read from backend (vitessio#17834) Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * `vtorc`: add index for `inst.ReadInstanceClusterAttributes` table scan Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> --------- Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
…onsolidated query (vitessio#17244) (#614) Signed-off-by: Jun Wang <jun.wang@demonware.net> Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> Co-authored-by: jwang <121262788+jwangace@users.noreply.github.com> Co-authored-by: Jun Wang <jun.wang@demonware.net>
* Remove unused code in discovery queue creation (vitessio#17515) Signed-off-by: Manan Gupta <manan@planetscale.com> * vtorc: Cleanup unused code (vitessio#15508) Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com> * `vtorc`: cleanup discover queue, add concurrency flag (vitessio#17825) Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * `vtorc`: add tablets watched stats Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * fix missing merge conflict update Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * `vtorc`: skip unnecessary `inst.ReadTablet` in `logic.LockShard(...)` Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * `vtorc`: use `errgroup` in keyspace/shard discovery Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * fix import Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * fix ineffassign Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * missing import Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * `vtorc`: add stats for discovery workers Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * get count from backend Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * rm unused map Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> --------- Signed-off-by: Manan Gupta <manan@planetscale.com> Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com> Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> Co-authored-by: Manan Gupta <35839558+GuptaManan100@users.noreply.github.com> Co-authored-by: Dirkjan Bussink <d.bussink@gmail.com>
* VReplication: Improve error handling in VTGate VStreams (vitessio#17558) Signed-off-by: Tom Thornton <thomaswilliamthornton@gmail.com> * Backport vitessio#17858 --------- Signed-off-by: Tom Thornton <thomaswilliamthornton@gmail.com>
* fix bug in reverse `if` Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * simplify Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * add `ReadTabletCountsByShard` test Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * use map of map Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * capitalize Cell Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * gofmt lint Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * fix plural in names Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> --------- Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
…ssio#17000) (#623) Signed-off-by: Renan Rangel <rrangel@slack-corp.com>
…#16593) (#620) * VStream API: allow keyspace-level heartbeats to be streamed (vitessio#16593) Signed-off-by: Malcolm Akinje <makinje@slack-corp.com> * `slack-19.0` backport v22 `vtorc` optimizations + stats, part 3 (#618) * Remove unused code in discovery queue creation (vitessio#17515) Signed-off-by: Manan Gupta <manan@planetscale.com> * vtorc: Cleanup unused code (vitessio#15508) Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com> * `vtorc`: cleanup discover queue, add concurrency flag (vitessio#17825) Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * `vtorc`: add tablets watched stats Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * fix missing merge conflict update Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * `vtorc`: skip unnecessary `inst.ReadTablet` in `logic.LockShard(...)` Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * `vtorc`: use `errgroup` in keyspace/shard discovery Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * fix import Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * fix ineffassign Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * missing import Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * `vtorc`: add stats for discovery workers Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * get count from backend Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * rm unused map Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> --------- Signed-off-by: Manan Gupta <manan@planetscale.com> Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com> Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> Co-authored-by: Manan Gupta <35839558+GuptaManan100@users.noreply.github.com> Co-authored-by: Dirkjan Bussink <d.bussink@gmail.com> * Bp pr 17558 pr 17858.slack19.0 (#615) * VReplication: Improve error handling in VTGate VStreams (vitessio#17558) Signed-off-by: Tom Thornton <thomaswilliamthornton@gmail.com> * Backport vitessio#17858 --------- Signed-off-by: Tom Thornton <thomaswilliamthornton@gmail.com> * `slack-19.0`: re-backport tweaks from vitessio#17911 (#621) * fix bug in reverse `if` Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * simplify Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * add `ReadTabletCountsByShard` test Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * use map of map Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * capitalize Cell Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * gofmt lint Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * fix plural in names Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> --------- Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> --------- Signed-off-by: Malcolm Akinje <makinje@slack-corp.com> Signed-off-by: Manan Gupta <manan@planetscale.com> Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com> Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> Signed-off-by: Tom Thornton <thomaswilliamthornton@gmail.com> Signed-off-by: Malcolm Akinje <malcolm.akinje@gmail.com> Co-authored-by: Tim Vaillancourt <tim@timvaillancourt.com> Co-authored-by: Manan Gupta <35839558+GuptaManan100@users.noreply.github.com> Co-authored-by: Dirkjan Bussink <d.bussink@gmail.com> Co-authored-by: Tom Thornton <thomaswilliamthornton@gmail.com>
Signed-off-by: Manan Gupta <manan@planetscale.com> Signed-off-by: Malcolm Akinje <makinje@slack-corp.com> Co-authored-by: Manan Gupta <35839558+GuptaManan100@users.noreply.github.com>
…tessio#15393) (#627) Signed-off-by: Tanjin Xu <tanjin.xu@slack-corp.com> Co-authored-by: Matt Lord <mattalord@gmail.com>
makinje16
force-pushed
the
slack-19.0-vtgate-journal-events
branch
from
f1847f4 to
413f6cc
Compare
Signed-off-by: Malcolm Akinje <malcolm.akinje@gmail.com>
tanjinx
approved these changes
Mar 24, 2025
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This is a backport of vitessio#16639 needed by CDC to continue productionizing vstreams and related operations. Specifically, this fixes a race condition between sending a message and context cancellation at the grpc layer.
Related Issue(s)
Checklist
Deployment Notes