Enricher Pub/Sub consumer silently fails to pull messages despite correct IAM permissions and Workload Identity

[geoashis]

Environment
Cloud Provider: Google Kubernetes Engine (GKE) Autopilot

OpenSnowcat Collector: opensnowcat/opensnowcat-collector-pubsub:1.1.2

OpenSnowcat Enricher: opensnowcat/opensnowcat-enrich-pubsub:1.2.3 (used as a base image)

Authentication: GKE Workload Identity

Problem Description
The OpenSnowcat enricher pod starts up successfully without any errors in the logs. However, it does not consume any messages from the input Pub/Sub subscription (opensnowcat-raw-good-sub).

The collector is working correctly and publishing events to the raw-good topic.

The subscription shows a large and continuously growing number of unacked messages.

The enricher pod logs show that the application is running but idle, with metrics showing 0 for raw, good, and bad events.

Troubleshooting Steps Performed (Evidence)
We have conducted an extensive investigation and have confirmed that the underlying GCP infrastructure, networking, and IAM permissions are all configured correctly. The failure appears to be isolated to the enricher application's client.

1. Verified Pod Identity
   We confirmed that the enricher pod is correctly assuming the identity of its configured Google Service Account (GSA) via Workload Identity.

• Command run inside the pod: curl -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/email
• Result: The command successfully returned the correct GSA email: opensnowcat-collector-gsa@GCP_PROJECT.iam.gserviceaccount.com

2. Verified IAM Permissions & API Connectivity
We proved that the GSA has the correct permissions and that the pod has network connectivity to the Pub/Sub API. We did this by manually pulling a message from inside the same pod, which was successful.

Command run inside the pod: # First, get a token
TOKEN=$(curl -s -H "Metadata-Flavor: Google" "http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token" | grep -o '"access_token":"[^"]*' | cut -d'"' -f4)

Then, used the token to pull a message

curl -s -X POST "https://pubsub.googleapis.com/v1/projects/GCP_PROJECT/subscriptions/opensnowcat-raw-good-sub:pull" \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{"maxMessages": 1}'

Result: The command successfully returned a JSON payload with a message from the subscription, proving the identity and permissions are fully functional.

Conclusion
The underlying infrastructure is working as expected. The curl test proves that the pod's environment has the necessary identity and permissions to consume from the subscription.

The failure appears to be within the OpenSnowcat enricher application's internal Java-based Pub/Sub client, which is failing silently where a direct API call from the same environment succeeds.

Configuration Files

1. Dockerfile for the enricher:

FROM opensnowcat/opensnowcat-enrich-pubsub:1.2.3

COPY . /opensnowcat/config

2. enricher.conf:

{
  "auth": {
    "type": "Gcp"
  },
  "input": {
    "type": "PubSub",
    "subscription": "projects/GCP_PROJECT/subscriptions/opensnowcat-raw-good-sub"
  },
  "output": {
    "good": {
      "type": "PubSub",
      "topic": "projects/GCP_PROJECT/topics/opensnowcat-enriched-good"
    },
    "bad": {
      "type": "PubSub",
      "topic": "projects/GCP_PROJECT/topics/opensnowcat-enriched-bad"
    }
  },
  "monitoring": {
    "metrics": {
      "stdout": {
        "period": "10 seconds",
        "prefix": "osc.enrich.moitoring"
      }
    }
  }
}

3. resolver.json:

{
    "schema": "iglu:com.snowplowanalytics.iglu/resolver-config/jsonschema/1-0-0",
    "data": {
      "cacheSize": 500,
      "repositories": [
        {
          "name": "Iglu Repository",
          "priority": 10,
          "vendorPrefixes": [ "com.xyz" ],
          "connection": {
            "http": {
              "uri": "https://storage.googleapis.com/opensnowcat-iglu-schemas"
            }
          }
        },
        {
          "name": "Snowplow Analytics Iglu Repository",
          "priority": 10,
          "vendorPrefixes": [ "com.snowplowanalytics.snowplow" ],
          "connection": {
            "http": {
              "uri": "https://storage.googleapis.com/opensnowcat-iglu-schemas"
            }
          }
        },
        {
          "name": "IAB Spider and Robots Iglu Repository",
          "priority": 10,
          "vendorPrefixes": [ "com.iab.snowplow" ],
          "connection": {
            "http": {
              "uri": "https://storage.googleapis.com/opensnowcat-iglu-schemas"
            }
          }
        }
      ]
    }
}

[geoashis]

I want to sincerely thank you on behalf of the open source community for making this freely available. Your efforts in keeping this accessible and open are truly appreciated.
@joaolcorreia Any pointers would be really helpful.

[joaolcorreia]

@geoashis we tested and it works fine on our infra ... without access we can't help you much!
Maybe it is a permissions issue? Would you be willing to pay for support?