Add Kubernetes Service support by wjhoward · Pull Request #4164 · Yelp/paasta

wjhoward · 2025-11-20T11:54:34Z

Ticket: HACK-4795

Add minimal Kubernetes Service creation support to PaaSTA.

This adds opt in Service creation via a new k8s_service config block, primarily for headless Services.

Changes:

SOA Service config schema
Service CRUD functions
Service sync orchestration logic
Unit tests

Current design decisions

Minimal config approach, k8s_service: {} is valid and would create a headless Service with default port 8888
Protocol is currently hardcoded to TCP, but can be extended
Service name set to the deployment name
Single port per service

Example config:

# Full config with all options
k8s_service:
  headless: true              # Optional, default: true
  port: 9000                  # Optional, default: 8888
  annotations:                # Optional, default: {}
    external-dns.alpha.kubernetes.io/hostname: my-app.example.com

# Minimal config (uses all defaults):
k8s_service: {}

itsgc · 2025-11-20T12:07:24Z

paasta_tools/kubernetes_tools.py

+            name="http", port=port, target_port=port, protocol="TCP"
+        )
+
+        service_spec = V1ServiceSpec(


super premature but this coudl be a great stepping stone to support service NLBs in paasta. According to https://kubernetes-sigs.github.io/aws-load-balancer-controller/latest/guide/service/nlb/ the only additions we would need are:

in the spec:

type should be settable to LoadBalancer

a new directive LoadBalancerClass would need to be set to service.k8s.aws/nlb

there is a third requirement which is to set an annotation of service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip but the current annotation support added in this PR would be enough

an alternative would otherwise to have a top level yelpsoa-config directive enable_nlb: True taht would set all of this in one go

I really like this idea, and it sounds like a small change to support it. I will wait to see what other reviewers think before making any changes.

yes, this is really great idea!

Rob-Johnson

this is really nice, thanks @wjhoward!

I think in the long term we might want both, but remember that right now, 99% of clients don't connect directly to a service/instance, but a smartstack namespace. there are cases where a user might want to talk to a specific service/instance, but we'll need to add support for creating services per namespace to serve most interactions.

I've written a few notes about that in the internal design doc I wrote. But this is a great start, thanks!

Rob-Johnson · 2025-11-20T12:17:25Z

paasta_tools/kubernetes_tools.py

+        port = k8s_service_config.get("port", 8888)  # Default to the PaaSTA port
+        annotations = k8s_service_config.get("annotations", {})
+        annotations[paasta_prefixed("managed")] = "true"
+        headless = k8s_service_config.get("headless", True)


is there ever a situation where we wouldn't want this to be true? I don't think we'd ever want a client to be connecting to a paasta service/instance using a k8s cluster ip - we'd only ever want to be using a loadbalancer/cloudmap/something else. I'd probably just remove this config option

Good point, I agree, I think we can remove this option for now. Thanks!

wjhoward · 2025-11-20T15:44:06Z

Thanks @Rob-Johnson , agreed - this is catering to a niche use case, specifically external DNS integration:

External Client → myapp.example.com → DNS provider (CloudMap/Route53) → Pod IPs

Where the external-dns controller reads the annotated headless Service to get Pod IPs and creates/updates DNS records. This bypasses Smartstack/mesh entirely for external access. Tagging @diwakar-dubey as they discovered the use case / requirement. To clarify a couple of points:

I think in the long term we might want both

Do you mean external DNS support and the NLB use case @itsgc mentioned, or namespace services?

but we'll need to add support for creating services per namespace to serve most interactions.

Yeah the intention here is specific to the external DNS use case, hence why I marked service config as opt in, are you thinking about expanding k8s Services to support namespace ingress also?

Rob-Johnson · 2025-11-20T15:51:25Z

Thanks @Rob-Johnson , agreed - this is catering to a niche use case, specifically external DNS integration:

External Client → myapp.example.com → DNS provider (CloudMap/Route53) → Pod IPs

Where the external-dns controller reads the annotated headless Service to get Pod IPs and creates/updates DNS records. This bypasses Smartstack/mesh entirely for external access. Tagging @diwakar-dubey as they discovered the use case / requirement. To clarify a couple of points:

even for external DNS, It'd be better if the DNS address was for a smartstack namespace than a specific paasta service/instance. given we're only using this for trivial redirects atm, we can get away with it because there is a 1:1 mapping from service/instance to namespace. but we can't guarantee that'll always be true.

I think in the long term we might want both

Do you mean external DNS support and the NLB use case @itsgc mentioned, or namespace services?

No, I mean having both a service that uses 'paasta_service' and 'paasta_instance' as a label selector (so goes to a specific instance), and one that uses a selector for registrations.

but we'll need to add support for creating services per namespace to serve most interactions.

Yeah the intention here is specific to the external DNS use case, hence why I marked service config as opt in, are you thinking about expanding k8s Services to support namespace ingress also?

nemacysts · 2025-11-20T16:17:20Z

paasta_tools/kubernetes_tools.py

+        if k8s_service_config is None:
+            return None
+
+        port = k8s_service_config.get("port", 8888)  # Default to the PaaSTA port


nit: could probably default to self.get_container_port() to handle the couple of services that don't set this to DEFAULT_CONTAINER_PORT (otherwise, probably worth replacing the hardcoded 8888 here with

paasta/paasta_tools/long_running_service_tools.py

Line 36 in cb5eca3

DEFAULT_CONTAINER_PORT = 8888

nemacysts · 2025-11-20T16:18:25Z

paasta_tools/kubernetes_tools.py


+    def get_kubernetes_service_config(self) -> Optional[KubernetesServiceConfigDict]:
+        """Returns the k8s_service config block, or None if not configured."""
+        return self.config_dict.get("k8s_service")


note: we'll also need to add k8s_service to https://github.com/Yelp/paasta/blob/cb5eca340f12e3d63807f4ce35bcccad0711abe2/paasta_tools/cli/schemas/eks_schema.json

nemacysts · 2025-11-20T16:21:16Z

tests/kubernetes/application/test_controller_wrapper.py

+
+def test_sync_service_creates_service_when_configured():
+    """Test sync_service creates a Service when k8s_service is configured."""
+    mock_client = mock.MagicMock()


i was gonna say that we should be using a mock that's autospec'd on the kube client...but it seems like this is a pre-existing pattern already in this file so nevermind :p

nemacysts · 2025-11-20T16:24:09Z

tests/test_kubernetes_tools.py

+
+        # Validate against the expected spec structure
+        # V1Service validation
+        assert isinstance(result, V1Service)


nit: mypy should already validate this - so we can probably drop the isinstance asserts from these tests

nemacysts · 2025-11-20T16:25:05Z

tests/test_kubernetes_tools.py

+        assert result.api_version == "v1"
+        assert result.kind == "Service"
+
+        # V1ObjectMeta
+        assert isinstance(result.metadata, V1ObjectMeta)
+        assert result.metadata.name == self.deployment.get_sanitised_deployment_name()
+        assert result.metadata.namespace == self.deployment.get_namespace()
+        # Annotations
+        assert (
+            result.metadata.annotations["external-dns.alpha.kubernetes.io/hostname"]
+            == "test.example.com"
+        )
+        assert result.metadata.annotations["paasta.yelp.com/managed"] == "true"
+        # Labels
+        assert result.metadata.labels["paasta.yelp.com/service"] == "kurupt"
+        assert result.metadata.labels["paasta.yelp.com/instance"] == "fm"
+        assert result.metadata.labels["paasta.yelp.com/managed"] == "true"
+
+        # V1ServiceSpec
+        assert isinstance(result.spec, V1ServiceSpec)
+        assert (
+            result.spec.cluster_ip == "None"
+        )  # K8s API expects string "None" for headless
+
+        # V1ServicePort
+        assert len(result.spec.ports) == 1
+        assert isinstance(result.spec.ports[0], V1ServicePort)
+        assert result.spec.ports[0].name == "http"
+        assert result.spec.ports[0].port == 8888
+        assert result.spec.ports[0].target_port == 8888
+        assert result.spec.ports[0].protocol == "TCP"
+
+        # Spec selector
+        assert isinstance(result.spec.selector, dict)
+        assert result.spec.selector == {
+            "paasta.yelp.com/service": self.deployment.get_service(),
+            "paasta.yelp.com/instance": self.deployment.get_instance(),
+        }


should we construct an expected_service and compare to that rather than comparing attribute-by-attribute manually?

Add Kubernetes Service support

8c44361

wjhoward requested a review from a team as a code owner November 20, 2025 11:54

itsgc reviewed Nov 20, 2025

View reviewed changes

wjhoward requested review from EvanKrall, Rob-Johnson, devkitx, diwakar-dubey, filipposc5 and nemacysts November 20, 2025 12:10

Merge branch 'master' into u/wjhoward/k8s-service-support

32f731d

Rob-Johnson reviewed Nov 20, 2025

View reviewed changes

nemacysts reviewed Nov 20, 2025

View reviewed changes

Rob-Johnson mentioned this pull request Nov 24, 2025

add setup_namespace_services #4167

Open

Conversation

wjhoward commented Nov 20, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Rob-Johnson left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wjhoward commented Nov 20, 2025

Uh oh!

Rob-Johnson commented Nov 20, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants