Incompatibility with OkHttp 4.x since 1.52.0

[blakeli0]

Describe the bug

Since v1.52.0, opentelemetry-exporter-sender-okhttp and opentelemetry-sdk-extension-jaeger-remote-sampler started to depend on okhttp 5.x. If my application still needs depend on okhttp 4.x, I will not be able to upgrade to newer versions of opentelemetry.

Furthermore, if any of my dependencies (e.g. google-cloud-storage) upgraded any opentelemetry artifacts beyond 1.52.0, I will not be able to upgrade them as well if I use opentelemetry-bom 1.51.0. This is because both aforementioned artifacts are in the bom.

Steps to reproduce
Given the following pom configuration

    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>io.opentelemetry</groupId>
                <artifactId>opentelemetry-bom</artifactId>
                <version>1.52.0</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>

    </dependencyManagement>

    <dependencies>
        <dependency>
            <groupId>io.opentelemetry</groupId>
            <artifactId>opentelemetry-exporter-sender-okhttp</artifactId>
        </dependency>
        <dependency>
            <groupId>com.squareup.okhttp3</groupId>
            <artifactId>okhttp</artifactId>
            <version>4.12.0</version>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-enforcer-plugin</artifactId>
                <version>3.6.2</version>
                <executions>
                    <execution>
                        <id>enforce</id>
                        <configuration>
                            <rules>
                                <requireUpperBoundDeps>
                                </requireUpperBoundDeps>
                            </rules>
                        </configuration>
                        <goals>
                            <goal>enforce</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

running mvn install would produce the following upper bound dependencies error

[ERROR] Require upper bound dependencies error for com.squareup.okhttp3:okhttp:4.12.0. Paths to dependency are:
[ERROR] +-org.example:testOtelOKHttp:1.0-SNAPSHOT
[ERROR]   +-com.squareup.okhttp3:okhttp:4.12.0
[ERROR] and
[ERROR] +-org.example:testOtelOKHttp:1.0-SNAPSHOT
[ERROR]   +-io.opentelemetry:opentelemetry-exporter-sender-okhttp:1.52.0
[ERROR]     +-com.squareup.okhttp3:okhttp:5.1.0 [runtime]

What did you expect to see?
I can continue to use okhttp 4.x and opentelemetry 1.52.0+ without being forced to upgrade my application.

What version and what artifacts are you using?
Artifacts: opentelemetry-exporter-sender-okhttp, opentelemetry-sdk-extension-jaeger-remote-sampler, opentelemetry-bom
Version: v1.52.0
How did you reference these artifacts? pom.xml

Environment
Compiler: Temurin-24.0.1
OS: MacOS 15.7.3

Workarounds

1. Ignoring the enforcer errors and force okhttp version to 4.12.0. This could potentially work if opentelemetry-exporter-sender-okhttp does not use any new features from 5.x, otherwise there could be runtime errors. However, even if it works, force downgrading major versions is risky and error prone. It would break if someone starts to use new features from 5.x accidentally.
2. Only downgrade opentelemetry-exporter-sender-okhttp to 1.51.0, still use opentelemetry-bom 1.52.0 to manage other opentelemetry artifacts such as opentelemetry-api. However, this requires more work like downgrading versions of transitive dependency kotlin-stdlib-jdk8. Managing different versions of opentelemetry artifacts is also costly and error prone in large corps with thousands of repos.

Possible solutions
Downgrade okhttp to 4.12.0 to maintain backward compatibility. Only upgrade it to the next major version if needed.

In addition, consider removing exporters such as opentelemetry-exporter-sender-okhttp and opentelemetry-sdk-extension-jaeger-remote-sampler from opentelemetry-bom. The problem of them being included in the bom is that, if a third party library is only dependent on opentelemetry-api and upgraded it to the latest, it would force the users of the third party library to bump opentelemetry-bom and all transitive dependencies included in it. IMO, the opentelemetry-bom should only manage core libraries with minimum transitive dependencies. The exporters should follow a different version strategy as well.

Separately, regarding the general upgrading strategy of transitive dependencies. I think it's OK to keep them up to date if there are no breaking changes, since people can always upgrade them to latest if needed. However, for major version bump (okhttp 4.x to 5.x) that contains a lot of breaking changes, upgrading it should be very careful since users of opentelemetry would be forced to upgrade their codebase. Depending on the scenario, upgrading major versions of transitive dependencies may warrant a major version bump.

[jack-berg]

If my application still needs depend on okhttp 4.x, I will not be able to upgrade to newer versions of opentelemetry.

You can safely use okhttp 4.x. We have encountered this type of thing before in #6026 and the resolution in #6045 was for us to verify compatibility with multiple versions of okhttp via additional test suites. Currently we test against 4.11 and latest, but can add additional versions if needed:

opentelemetry-java/exporters/otlp/all/build.gradle.kts

Lines 46 to 48 in 8e332fc

	"LATEST",
	"4.11.0"
	).forEach {

Ignoring the enforcer errors and force okhttp version to 4.12.0. This could potentially work if opentelemetry-exporter-sender-okhttp does not use any new features from 5.x

Do this - its our current answer and we mitigate the risks as discussed above.

In addition, consider removing exporters such as opentelemetry-exporter-sender-okhttp and opentelemetry-sdk-extension-jaeger-remote-sampler from opentelemetry-bom. The problem of them being included in the bom is that, if a third party library is only dependent on opentelemetry-api and upgraded it to the latest, it would force the users of the third party library to bump opentelemetry-bom and all transitive dependencies included in it. IMO, the opentelemetry-bom should only manage core libraries with minimum transitive dependencies.

The point of the bom is to align all the dependency versions that are meaningful to align. Something like, "you dont have to depend on all these things, but if you do, we recommend these versions". A third party library only dependent on opentelemetry-api has no need for the bom, and it should always be safe to upgrade to the latest.

However, for major version bump (okhttp 4.x to 5.x) that contains a lot of breaking changes, upgrading it should be very careful since users of opentelemetry would be forced to upgrade their codebase. [..] Depending on the scenario, upgrading major versions of transitive dependencies may warrant a major version bump.

Its possible that at some point okhttp does make a breaking change that makes supporting multiple major versions simultaneously impossible. In this case we would have some decision making to do, but i suspect we would start publishing multiple versions of our okhttp sender corresponding to the incompatible major versions.

[blakeli0]

Thanks Jack for the quick response!

Do this - its our current answer and we mitigate the risks as discussed above.

While this is doable in a small number of repos. For large enterprise companies with thousands of repos, depending on how the internal infra is set up, it is very hard or sometimes impossible to do it.

Something like, "you dont have to depend on all these things, but if you do, we recommend these versions".

I feel a little bit different about the BOM. I think it is more like "We recommend these versions for compatibility, you are at your own risk if you choose different versions". Hence most people are hesitate to use different versions if it is already managed in the BOM.

A third party library only dependent on opentelemetry-api has no need for the bom, and it should always be safe to upgrade to the latest.

This is true for the library itself, but for a user that depends on both older versions of opentelemetry-bom and newer versions of the third party library, they would run into similar enforcer errors. Considering the following pom that uses google-cloud-storage as an example:

    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>io.opentelemetry</groupId>
                <artifactId>opentelemetry-bom</artifactId>
                <version>1.51.0</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>

    </dependencyManagement>

    <dependencies>
        <dependency>
            <groupId>io.opentelemetry</groupId>
            <artifactId>opentelemetry-exporter-sender-okhttp</artifactId>
        </dependency>
        <dependency>
            <groupId>com.google.cloud</groupId>
            <artifactId>google-cloud-storage</artifactId>
            <!-- This version includes opentelemetry-api 1.52.0 -->
            <version>2.61.0</version>
        </dependency>
    </dependencies>

It would produce a similar enforcer error for opentelemetry-api

[ERROR] Require upper bound dependencies error for io.opentelemetry:opentelemetry-sdk-metrics:1.51.0. Paths to dependency are:
[ERROR] +-org.example:testOtelOKHttp:1.0-SNAPSHOT
[ERROR]   +-com.google.cloud:google-cloud-storage:2.61.0
[ERROR]     +-io.opentelemetry:opentelemetry-sdk-metrics:1.51.0 (managed) <-- io.opentelemetry:opentelemetry-sdk-metrics:1.52.0

While it is possible to force downgrade opentelemetry-api to 1.51.0, if the third party library (google-cloud-storage in this case) starts to use newer feature in 1.52.0, then there would be runtime errors.

i suspect we would start publishing multiple versions of our okhttp sender corresponding to the incompatible major versions

What if staying on the older version 4.12.0 for longer until it is really required to upgrade? For example, only if the exporter needs a new feature from 5.x, or 4.x is no longer maintained. Since the exporter is still compatible with 4.11, I assume it does not use any new features from 5.x and it is not a must-have to upgrade OkHttp to 5.x?

Overall, downgrading OkHttp to 4.x seems low risk for OpenTelemetry and offers high value for users. It would also demonstrate that OpenTelemetry takes compatibility seriously. I would really appreciate your consideration of this downgrade.

[jack-berg]

it is very hard or sometimes impossible to do it.

🤨 forcing resolution of transitive dependency conflicts is table stakes

While it is possible to force downgrade opentelemetry-api to 1.51.0, if the third party library (google-cloud-storage in this case) starts to use newer feature in 1.52.0, then there would be runtime errors.

Are you speaking specifically about opentelemetry-api or about dependencies in general? I'm only advocating for force downgrading the okhttp library. Within the realm of opentelemetry-java artifacts, our versioning guarantees mean that it should always be safe to use the later version of artifacts if a conflict occurs.

What if staying on the older version 4.12.0 for longer until it is really required to upgrade? For example, only if the exporter needs a new feature from 5.x, or 4.x is no longer maintained.

Does 4.x get security updates? I can't tell. Based on their docs, 4.x is supported. Yet I don't see any published minor / patch versions since 2023: https://mvnrepository.com/artifact/com.squareup.okhttp3/okhttp

That could mean they have had no issues, but seems hard to believe.

For every person who wants us to stay on 4.x for compatibility reasons, there will be another who want us on 5.x for compatibility reasons, and is equally nervous about forcing dependency resolution to 5.x when our bom says 4.x.

I think this is a tooling and information discovery problem:

• Information discovery: we support both 4.x and 5.x, and have thorough testing that ensures this. But its not obvious or easily discovered from users' perspective.
• Tooling: tooling like maven stumble on major version dependency conflicts, since its not obvious that a library like opentelemetry-java can support multiple major version simultaneously and there's no way for us to communicate that in a structured way (i.e. we can't state in our bom that either 4.x or 5.x is ok).

One practical thing we could do adopt a posture of publishing sender artifacts corresponding to each major version of okhttp, e.g. opentelemetry-exporter-sender-okhttp4, opentelemetry-exporter-sender-okhttp5, and in the future opentelemetry-exporter-sender-okhttp<n>. We would still have to pick a default to for opentelemetry-exporter-otlp to depend on, since while some applications will run into these types of conflicts, its the minority. And I would not want to impose the UX burden / cognitive load of having to choose a sender onto each and every user.

Since the exporter is still compatible with 4.11, I assume it does not use any new features from 5.x and it is not a must-have to upgrade OkHttp to 5.x?

Correct, besides the security / active maintenance argument. And since 5.x is actively developed, there may be perf benefits as well. Generally, its a hard sell to stay on old unmaintained dependencies. I do understand the desire to have comparability with old dependencies but to pin to them is just asking for a crisis.

[blakeli0]

🤨 forcing resolution of transitive dependency conflicts is table stakes

Forcing resolution usually comes with forcing transitive dependencies and disabling build rules. Also assuming that this solution needs to be applied to thousands of repos, people can use things like internal boms and/or parent poms to make it easier, but it can easily gets complicated if the internal dependency tree is too deep.

Are you speaking specifically about opentelemetry-api or about dependencies in general? I'm only advocating for force downgrading the okhttp library. Within the realm of opentelemetry-java artifacts, our versioning guarantees mean that it should always be safe to use the later version of artifacts if a conflict occurs.

I'm using opentelemetry-api as an example, but it is about any opentelemetry dependencies in general. In the above example, if google-cloud-storage already started using new features in opentelemetry-api 1.52.0, people have to use opentelemetry-api 1.52.0 and opentelemetry-exporter-sender-okhttp 1.51.0 to maintain backward compatibility with OkHttp 4.x. Which defeats the purpose of using opentelemetry-bom.

For every person who wants us to stay on 4.x for compatibility reasons, there will be another who want us on 5.x for compatibility reasons, and is equally nervous about forcing dependency resolution to 5.x when our bom says 4.x

I'm not sure I agree with this. I don't see why someone would want opentelemetry to be on 5.x for compatibility reasons, can you please provide a concrete example?

Force upgrading are usually much less risky than force downgrading. For example, force upgrading minor versions usually has no risk, but force downgrading a minor version could result in runtime errors if a new feature is being used. For major version bumps, I would prefer to stay on an older version (4.x) and test against newer version (5.x) instead of vice versa.

I think this is a tooling and information discovery problem

I agree tooling and information discovery definitely contributes to it. For example, maybe there could be compatibility matrix regarding what are the OkHttp versions that the exporter is compatible with. This could make people more comfortable downgrading it in their own repos.

On the other hand, it still requires people to downgrade it which may not be trivial as I mentioned. If we can stay on an older version, it does not require people to do things.

Generally, its a hard sell to stay on old unmaintained dependencies. I do understand the desire to have comparability with old dependencies but to pin to them is just asking for a crisis.

I agree its a hard sell to stay on old unmaintained dependencies, I would not advocate for it if OkHttp 4.x is already deprecated. But considering it is still officially supported (I know there are doubts if it is really supported), I think it is less risky.

[trask]

@blakeli0 I wonder if part of the problem is that google-cloud-storage has a dependency on the OpenTelemetry SDK instead of depending only on the OpenTelemetry API (see Libraries should only use the OpenTelemetry API)

[jack-berg]

I'm not sure I agree with this. I don't see why someone would want opentelemetry to be on 5.x for compatibility reasons, can you please provide a concrete example? Force upgrading are usually much less risky than force downgrading.

Suppose they are using okhttp 5.x in their application and depend on new features only available in 5.x. They add a dependency on opentelemetry-java where we've adjusted to have an okhttp 4.x dependency. When they need to resolve the okhttp dependency version, they worry that opentelemetry-java depends on 4.x features removed in 5.x. In this hypothetical, why is the worry about breaking changes from 4.x -> 5.x any less valid than worries about issues from 5.x -> 4.x downgrade due to using new features in 5.x? The point of major version bumps is to signal incompatible changes.

For the record, I don't want to nor does opentelemetry-java require anyone to upgrade or downgrade since we're compatible with both 4.x and 5.x.

But considering it is still officially supported (I know there are doubts if it is really supported), I think it is less risky.

Here's the okhttp 4.x and okhttp 5.x. 4.x's last update was 2023-10-16. A ctrl+f for the word "Fix:" yields 63 matches. Its hard to imagine that none of those matter. 🤷

---

Ok so we've had some back and forth and I think I understand your POV. Here are some (or most?) of the options for moving forward:

1. Do nothing. Keep the artifacts and dependencies the way they are, and update the docs to better indicate that we support both okhttp 4.x and 5.x, despite choosing 5.x.
2. Change opentelemetry-exporter-sender-okhttp to depend on okhttp 4.x. Update docs to indicate that we support okhttp 4.x and 5.x, despite choosing 4.x. When okhttp 4.x stops being supported, change the dependency to 5.x. If 4.x has unfixed security vulnerabilities reported against it, improvise 🤷.
3. Replace opentelemetry-exporter-sender-okhttp with opentelemetry-exporter-sender-okhttp4 and opentelemetryexporter-sender-okhttp5. Update opentelemetry-exporter-otlp to default to opentelemetry-exporter-sender-okhttp5.
4. Same as 3, but update opentelemetry-exporter-otlp to default to opentelemetry-exporter-sender-okhttp4.
5. Combine one of 1-4 with publishing new opentelemetry-exporter-sender-okhttp{major}-shaded artifacts. Probably interesting challenges here given the dependency on ktlin stdlib which is 1.7mb.

If I were to guess, you'd prefer to see 2 or 4, but maybe don't mind 3?

My POV is that 1 or 3 are the most realistic. Interested in what other @open-telemetry/java-approvers think.

[jkwatson]

1. Do nothing. Keep the artifacts and dependencies the way they are, and update the docs to better indicate that we support both okhttp 4.x and 5.x, despite choosing 5.x.
2. Change opentelemetry-exporter-sender-okhttp to depend on okhttp 4.x. Update docs to indicate that we support okhttp 4.x and 5.x, despite choosing 4.x. When okhttp 4.x stops being supported, change the dependency to 5.x. If 4.x has unfixed security vulnerabilities reported against it, improvise 🤷.
3. Replace opentelemetry-exporter-sender-okhttp with opentelemetry-exporter-sender-okhttp4 and opentelemetryexporter-sender-okhttp5. Update opentelemetry-exporter-otlp to default to opentelemetry-exporter-sender-okhttp5.
4. Same as 3, but update opentelemetry-exporter-otlp to default to opentelemetry-exporter-sender-okhttp4.
5. Combine one of 1-4 with publishing new opentelemetry-exporter-sender-okhttp{major}-shaded artifacts. Probably interesting challenges here given the dependency on ktlin stdlib which is 1.7mb.

Option 3 is certainly a common way of dealing with sort of thing, although would be a first for this project. I don't have a super strong opinion on this one, but am also not directly impacted by it much, whichever way we decide to go.

[blakeli0]

I wonder if part of the problem is that google-cloud-storage has a dependency on the OpenTelemetry SDK instead of depending only on the OpenTelemetry API

@trask, I should've used the error below which only includes opentelemtry-api, I don't think using opentelemetry-sdk is related in this case.

[ERROR] Require upper bound dependencies error for io.opentelemetry:opentelemetry-api:1.51.0. Paths to dependency are:
[ERROR] +-org.example:testOtelOKHttp:1.0-SNAPSHOT
[ERROR]   +-com.google.cloud:google-cloud-storage:2.61.0
[ERROR]     +-io.opentelemetry:opentelemetry-api:1.51.0 (managed) <-- io.opentelemetry:opentelemetry-api:1.52.0

Yeah I agree that a library should not use opentelemetry-sdk if not needed. In this case, google-cloud-storage has a built in exporter hence opentelemetry-sdk is needed.

In this hypothetical, why is the worry about breaking changes from 4.x -> 5.x any less valid than worries about issues from 5.x -> 4.x downgrade due to using new features in 5.x? The point of major version bumps is to signal incompatible changes.

In this case, customers would not notice it because there are no enforcer rules for lower boundary. There is a much stricter enforcer rule dependencyConvergence which requires all dependencies to be on the same version, but it is very rarely used. If they did notice it, upgrading a dependency version is usually considered less risky than downgrading, even for major versions. Because the next major version usually maintains some backward compatibility, which is true for OkHttp 5.x, but not vice versa. In practice, usually as long as all CI pass, then users would feel comfortable upgrading a dependency version.

If I were to guess, you'd prefer to see 2 or 4, but maybe don't mind 3?

I would prefer option 2, because it does not require users to change anything in their codebase.

Option 3 and 4 are definitely better than option 1, and better than requiring users to upgrade their codebase to be OkHttp 5.x compatible. But they may still require users to update their build files to reference opentelemetry-exporter-sender-okhttp4 now, so it may not be much better than requiring users to force downgrading OkHttp to 4.x.

[blakeli0]

@jack-berg Can you please confirm it is also OK to downgrade OkHttp to 4.x for opentelemetry-sdk-extension-jaeger-remote-sampler as well?
What you mentioned in previous comment confirms the compatibility for opentelemetry-exporter-sender-okhttp, but
opentelemetry-sdk-extension-jaeger-remote-sampler brings additional OkHttp dependency okhttp-jvm.

[jack-berg]

opentelemetry-sdk-extension-jaeger-remote-sampler is more difficult because it depends on okhttp directly, so we'd need to either split it out to have different version, or refactor it so its built on top of the sender abstraction and able to leverage whatever we end up doing for opentelemetry-exporter-sender-okhttp. That type of refactor is possible but won't be as quick.

Do you know of someone that has this okhttp 4 vs. 5 issue who is also depending on opentelemetry-sdk-extension-jaeger-remote-sampler? In my head, opentelemetry-sdk-extension-jaeger-remote-sampler is fairly niche.

[blakeli0]

Yes the customer uses both opentelemetry-sdk-extension-jaeger-remote-sampler and opentelemetry-exporter-otlp directly, and gets opentelemetry-exporter-sender-okhttp transitively.

I understand setting up a full CI might require refactoring, is it possible to do a one-off test to confirm the compatibility at this moment?