chore: Confirm that a public ConfigMap exists to check version #250

sameerdattav · 2026-01-30T12:19:19Z

This PR adds a non-blocking control-plane version metadata check to the Kubernetes backend.

The backend now verifies that the public Trainer ConfigMap exists and contains a kubeflow_trainer_api_version field, logging warnings when metadata is unavailable but never failing client initialization. Focused unit tests cover the expected behaviors.

Supersedes #223

Fixes #221

github-actions · 2026-01-30T12:19:30Z

🎉 Welcome to the Kubeflow SDK! 🎉

Thanks for opening your first PR! We're happy to have you as part of our community 🚀

Here's what happens next:

If you haven't already, please check out our Contributing Guide for repo-specific guidelines and the Kubeflow Contributor Guide for general community standards
Our team will review your PR soon! cc @kubeflow/kubeflow-sdk-team

Join the community:

Slack: Join our #kubeflow-ml-experience and #kubeflow-trainer Slack channels
Meetings: Attend the Kubeflow SDK and ML Experience bi-weekly meetings

Feel free to ask questions in the comments if you need any help or clarification!
Thanks again for contributing to Kubeflow! 🙏

Copilot

Pull request overview

This PR implements a non-blocking control-plane version metadata check for the Kubeflow Trainer Kubernetes backend. The check verifies that the kubeflow-trainer-public ConfigMap exists and contains a kubeflow_trainer_api_version field, logging warnings when metadata is unavailable but never failing client initialization. This represents a more lenient approach compared to the superseded PR #223, which enforced strict version compatibility.

Changes:

Added verify_backend() method to check for control-plane version metadata availability
Added comprehensive unit tests covering success and failure scenarios
Imported os module for environment variable access

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

File	Description
`kubeflow/trainer/backends/kubernetes/backend.py`	Implements non-blocking version check via `verify_backend()` method called during initialization; adds `os` import for namespace configuration
`kubeflow/trainer/backends/kubernetes/backend_test.py`	Adds test helper function and five test cases covering success, missing ConfigMap, missing data key, None data, and dev version scenarios; includes minor docstring formatting improvement

kubeflow/trainer/backends/kubernetes/backend_test.py

kubeflow/trainer/backends/kubernetes/backend.py

kubeflow/trainer/backends/kubernetes/backend_test.py

astefanutti · 2026-02-02T21:32:39Z

/ok-to-test

andreyvelich · 2026-02-02T21:38:29Z

/retest

andreyvelich · 2026-02-02T21:39:56Z

kubeflow/trainer/backends/kubernetes/backend.py

+        """Verify that the Trainer control plane exposes version metadata.
+
+        This check only ensures that the public control-plane ConfigMap exists
+        and contains a ``kubeflow_trainer_api_version`` field. It does not


The field name is kubeflow_trainer_version: https://github.com/kubeflow/trainer/blob/master/manifests/overlays/manager/kustomization.yaml#L31

andreyvelich · 2026-02-02T21:40:20Z

kubeflow/trainer/backends/kubernetes/backend.py

+        enforce version compatibility and never raises.
+        """
+
+        system_namespace = os.getenv("KUBEFLOW_SYSTEM_NAMESPACE", "kubeflow")


The default system namespace is:

Suggested change

system_namespace = os.getenv("KUBEFLOW_SYSTEM_NAMESPACE", "kubeflow")

system_namespace = os.getenv("KUBEFLOW_SYSTEM_NAMESPACE", "kubeflow-system")

andreyvelich · 2026-02-02T21:43:06Z

kubeflow/trainer/backends/kubernetes/backend.py

+        except Exception as e:
+            logger.warning(
+                "Trainer control-plane version info is not available: "
+                f"failed to read ConfigMap '{config_map_name}' in namespace "
+                f"'{system_namespace}', error: {e!r}."
+            )
+            return
+
+        data = getattr(config_map, "data", None)
+
+        if not isinstance(data, dict):
+            logger.warning(
+                "Trainer control-plane version info is not available: "
+                f"ConfigMap '{config_map_name}' in namespace "
+                f"'{system_namespace}' has no data dictionary."
+            )
+            return
+
+        server_version = data.get("kubeflow_trainer_api_version")
+
+        if not server_version:
+            logger.warning(
+                "Trainer control-plane version info is not available: "
+                f"ConfigMap '{config_map_name}' in namespace "
+                f"'{system_namespace}' is missing the "
+                "'kubeflow_trainer_api_version' data key."
+            )
+            return


Just put everything under the same try: condition, and log warning if something fails.
Additionally, you can log the Exeception in the message.
You should expect that configMap has the correct data there.

andreyvelich · 2026-02-02T21:43:21Z

kubeflow/trainer/backends/kubernetes/backend.py

+        if server_version == "dev":
+            return
+


you don't need this

Suggested change

if server_version == "dev":

return

andreyvelich · 2026-02-02T21:44:11Z

kubeflow/trainer/backends/kubernetes/backend_test.py

    )


+def test_version_check_success(caplog):


Please refactor this tests to use TestCases like described here: https://github.com/kubeflow/sdk/blob/aab449ac829ac02c804758e6d72f0b2652a2d783/AGENTS.md

coveralls · 2026-02-03T17:00:09Z

Pull Request Test Coverage Report for Build 21873621460

Details

53 of 53 (100.0%) changed or added relevant lines in 2 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.4%) to 67.958%

Totals
Change from base Build 21826703132:	0.4%
Covered Lines:	2825
Relevant Lines:	4157

💛 - Coveralls

sameerdattav · 2026-02-03T17:13:14Z

@andreyvelich , Thanks for the review, addressed all your comments
Kindly check again.

andreyvelich · 2026-02-03T17:36:25Z

kubeflow/trainer/backends/kubernetes/backend_test.py

+
+        return [record.getMessage() for record in handler.records]
+
+    def test_version_check_success(self) -> None:


Refactor these tests with a single test_verify_backend() function and TestCases, like described here: https://github.com/kubeflow/sdk/blob/main/AGENTS.md#3-testing-requirements

@andreyvelich, updated backend_test.py file, This matches the AGENTS.md pattern: a single pytest function, parametrized with the shared TestCase dataclass to cover multiple scenarios.

kubeflow/trainer/backends/kubernetes/backend.py

andreyvelich

Thanks for the update @sameerdattav!
/assign @astefanutti @akshaychitneni @kramaranya @szaher @Fiona-Waters

andreyvelich · 2026-02-09T18:48:57Z

kubeflow/trainer/backends/kubernetes/backend_test.py

+            },
+        ),
+        TestCase(
+            name="missing data key logs warning",


Can you update your test cases according to your updated implementation?
E.g. we don't check data key for now.

sameerdattav · 2026-02-09T19:13:52Z

@andreyvelich,
Removed checks for specific ConfigMap data keys and adjusted the tests so we only warn when the ConfigMap cannot be read. Empty or missing data is now treated as a valid, non-blocking case.

astefanutti · 2026-02-10T14:15:00Z

/retest

andreyvelich

/lgtm
/assign @kramaranya @astefanutti @kubeflow/kubeflow-sdk-team @Fiona-Waters

astefanutti

Thanks @sameerdattav!

/lgtm

Not sure why the docs workflow fails, it seems like a transient issue.

andreyvelich · 2026-02-10T15:44:08Z

@sameerdattav Can you try to rebase your PR to fix the CI?

Signed-off-by: Surya Sameer Datta Vaddadi <f20220373@goa.bits-pilani.ac.in>

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Surya Sameer Datta Vaddadi <137607947+sameerdattav@users.noreply.github.com>

Signed-off-by: Surya Sameer Datta Vaddadi <f20220373@goa.bits-pilani.ac.in>

Co-authored-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> Signed-off-by: Surya Sameer Datta Vaddadi <137607947+sameerdattav@users.noreply.github.com>

Signed-off-by: Surya Sameer Datta Vaddadi <f20220373@goa.bits-pilani.ac.in>

google-oss-prow · 2026-02-10T16:32:28Z

New changes are detected. LGTM label has been removed.

google-oss-prow · 2026-02-10T16:32:33Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from andreyvelich. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Signed-off-by: Surya Sameer Datta Vaddadi <f20220373@goa.bits-pilani.ac.in>

sameerdattav · 2026-02-10T16:51:56Z

@andreyvelich @astefanutti
i rebased and ran pre-commit locally to fix formatting issues.
One new file CLAUDE.md , had CRLF line ending, hence the pre-commit changed it to LF.
The ci checks seem fine now.

sameerdattav · 2026-02-10T16:59:00Z

@andreyvelich,
To prevent such line ending issues, there is a gitattributes fix, which is already in manifests.
I was planning to fix them across repos.
Related Slack thread : https://cloud-native.slack.com/archives/C0742LBR5BM/p1770067104325089
could you please provide your input on this?

andreyvelich · 2026-02-10T17:05:52Z

Sure, that sounds good, k8s also has the same setting: https://github.com/kubernetes/kubernetes/blob/master/.gitattributes#L2

Copilot AI review requested due to automatic review settings January 30, 2026 12:19

google-oss-prow bot requested review from andreyvelich, kramaranya and szaher January 30, 2026 12:19

google-oss-prow bot added the size/L label Jan 30, 2026

Copilot started reviewing on behalf of sameerdattav January 30, 2026 12:19 View session

sameerdattav mentioned this pull request Jan 30, 2026

chore: Enforce strict backend version compatibility checks #223

Closed

Copilot AI reviewed Jan 30, 2026

View reviewed changes

google-oss-prow bot added the ok-to-test label Feb 2, 2026

andreyvelich reviewed Feb 2, 2026

View reviewed changes

andreyvelich reviewed Feb 3, 2026

View reviewed changes

andreyvelich reviewed Feb 9, 2026

View reviewed changes

google-oss-prow bot assigned akshaychitneni, astefanutti, Fiona-Waters, kramaranya and szaher Feb 9, 2026

andreyvelich reviewed Feb 10, 2026

View reviewed changes

google-oss-prow bot assigned andreyvelich Feb 10, 2026

google-oss-prow bot added the lgtm label Feb 10, 2026

astefanutti reviewed Feb 10, 2026

View reviewed changes

Confirm that a public ConfigMap exists to check version

2572006

Signed-off-by: Surya Sameer Datta Vaddadi <f20220373@goa.bits-pilani.ac.in>

sameerdattav and others added 8 commits February 10, 2026 22:01

python 3.9 fix

ef8b31a

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Surya Sameer Datta Vaddadi <137607947+sameerdattav@users.noreply.github.com>

Exceptiom handling better

12fab9f

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Surya Sameer Datta Vaddadi <137607947+sameerdattav@users.noreply.github.com>

Addressing comments

215d8c7

Signed-off-by: Surya Sameer Datta Vaddadi <f20220373@goa.bits-pilani.ac.in>

Update kubeflow/trainer/backends/kubernetes/backend.py

3e42143

Co-authored-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> Signed-off-by: Surya Sameer Datta Vaddadi <137607947+sameerdattav@users.noreply.github.com>

Refactored tests into a single function and followed agents.md

c205a6d

Signed-off-by: Surya Sameer Datta Vaddadi <f20220373@goa.bits-pilani.ac.in>

CI friendly edit

ca87f08

Signed-off-by: Surya Sameer Datta Vaddadi <f20220373@goa.bits-pilani.ac.in>

pre-commit format checked

8b14bbc

Signed-off-by: Surya Sameer Datta Vaddadi <f20220373@goa.bits-pilani.ac.in>

Modified according to new updates

a17d830

Signed-off-by: Surya Sameer Datta Vaddadi <f20220373@goa.bits-pilani.ac.in>

sameerdattav force-pushed the trainer-backend-version-clean branch from 0db01b1 to a17d830 Compare February 10, 2026 16:32

google-oss-prow bot removed the lgtm label Feb 10, 2026

Ran pre-commit locally to fix formatting

5015d06

Signed-off-by: Surya Sameer Datta Vaddadi <f20220373@goa.bits-pilani.ac.in>

	system_namespace = os.getenv("KUBEFLOW_SYSTEM_NAMESPACE", "kubeflow")
	system_namespace = os.getenv("KUBEFLOW_SYSTEM_NAMESPACE", "kubeflow-system")


		return [record.getMessage() for record in handler.records]

		def test_version_check_success(self) -> None:

chore: Confirm that a public ConfigMap exists to check version #250

Are you sure you want to change the base?

chore: Confirm that a public ConfigMap exists to check version #250

Conversation

sameerdattav commented Jan 30, 2026

Uh oh!

github-actions bot commented Jan 30, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

astefanutti commented Feb 2, 2026

Uh oh!

andreyvelich commented Feb 2, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

coveralls commented Feb 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Test Coverage Report for Build 21873621460

Details

💛 - Coveralls

Uh oh!

sameerdattav commented Feb 3, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

andreyvelich left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sameerdattav commented Feb 9, 2026

Uh oh!

astefanutti commented Feb 10, 2026

Uh oh!

andreyvelich left a comment

Choose a reason for hiding this comment

Uh oh!

astefanutti left a comment

Choose a reason for hiding this comment

Uh oh!

andreyvelich commented Feb 10, 2026

Uh oh!

google-oss-prow bot commented Feb 10, 2026

Uh oh!

google-oss-prow bot commented Feb 10, 2026

Uh oh!

sameerdattav commented Feb 10, 2026

Uh oh!

sameerdattav commented Feb 10, 2026

Uh oh!

andreyvelich commented Feb 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

coveralls commented Feb 3, 2026 •

edited

Loading