Gradle publishing overhaul

[Dwarfley]

This PR aims to overhaul the Gradle setup used for publishing LWJGL.

Goals

• Split the centralized build script into separate Gradle projects
• Publish Gradle module metadata by using software components
• Ensure existing published artifacts remain functionally identical
• Hide complex build logic to make module and BOM projects simple
• Make signing and repository credentials optional for local builds
• Extract all constants to gradle.properties for centralized access
• Ensure all build logic is compatible with the configuration cache
• Upgrade to Gradle 9.0.0 and update the build logic if required
• Make the setup more compatible with a full Gradle migration

Motivation

The main addition of this PR is the ability to publish Gradle module metadata. This is achieved by using software components instead of manually attaching artifacts to publications. The metadata allows Gradle users to simplify their build scripts by only requiring a single dependency for each binding. All necessary natives will be included automatically, provided the correct platform attributes have been set.

val lwjglVersion = "3.4.0"
val lwjglOs: String = "windows"
val lwjglArch: String = "x64"

configurations.matching(Configuration::isCanBeResolved).configureEach {
    attributes {
        attribute(OperatingSystemFamily.OPERATING_SYSTEM_ATTRIBUTE, objects.named(lwjglOs))
        attribute(MachineArchitecture.ARCHITECTURE_ATTRIBUTE, objects.named(lwjglArch))
    }
}

dependencies {
    
    implementation(platform("org.lwjgl:lwjgl-bom:$lwjglVersion"))
    
    implementation("org.lwjgl:lwjgl")
    implementation("org.lwjgl:lwjgl-glfw")
    implementation("org.lwjgl:lwjgl-stb")
    
}

To implement this, a significant reorganisation of the publishing logic was necessary. This eventually led to a full rewrite to improve the maintainability and ease of use. It did not make sense to keep the old code as most of it had to be changed in some way or another. Besides the improvements mentioned above, the reorganisation also paves the way for a full Gradle migration. Finally, the configuration cache was enabled to improve execution time.

Changes

Project structure

The Gradle project structure has now been separated to individual projects instead of a centralised one. These projects can be found in the publish folder. Here is the structure:

publish/
    build-logic/
        lwjgl-convention/
        lwjgl-plugins/
            ...
        lwjgl-utils/
    lwjgl-bom/
    lwjgl-modules/
        lwjgl-core/
        ...

Path	Use
publish/build-logic	Seperate build containing shared build logic
publish/build-logic/lwjgl-convention/	Precompiled script plugins (convention)
publish/build-logic/lwjgl-plugins/	Parent project for binary plugins (one project for each one)
publish/build-logic/lwjgl-utils/	Kotlin utils used by binary plugins
publish/lwjgl-modules/	Parent project for modules (one project for each one)
publish/lwjgl-bom/	The platform project

There is one project per binding/core and one for the platform (Maven BOM). Besides those projects, there is the build-logic build. It contains shared build logic and is used to create a composite Gradle build. Within are convention plugins (precompiled script), binary plugins and a Kotlin utilities project that all binary plugins can access.

Binary plugins

lwjgl-platforms

This plugin is used by the modules to register which platforms are supported. The list of supported platforms has therefore been moved to this plugin. An action can be registered that is executed for each supported platform.

lwjgl-publication

This plugin is used by modules and the platform to register publications. The publications consist of a title and a description that will be used to create a Maven publication. An action can be registered that is executed for each created Maven publication. The purpose of this plugin is to make the convention plugins be able to configure the Maven publication and to keep the build scripts simple.

lwjgl-adhoc

This plugin creates dependency configurations and a software component. It is used instead of the java-library plugin since no compilation is required. This keeps the setup clean from the large amount of unused features that the java-library provides. The plugin is used by modules to attach artifacts to the correct dependency configuration.

Convention plugins

lwjgl-publishing

This plugin configures publishing using Gradle properties. It is here that signing and repository configuration is done. The POM metadata is also constructed here and is applied to all Maven publications.

lwjgl-module

This is applied to all module projects. It contains build logic shared by all modules.

lwjgl-platform

This is applied to the platform project. It hides away most of the build logic so the build script can be simpler.

Kotlin utilities

For the moment, this only contains string utilities to convert between different cases.

Optional properties

Most of the properties have been made optional. This makes it easier to test the publishing locally without having to set all properties. The signing and repository credentials have been made optional for local publications. The repository urls have been made optional when not publishing to them. The POM metadata has also been made optional for local publications.

Module/Platform configuration

The configuration of the modules and platform have been moved to their respective projects. Here is the build script for the lwjgl-vulkan module as an example:

/*
 * Copyright LWJGL. All rights reserved.
 * License terms: https://www.lwjgl.org/license
 */
plugins {
    id("lwjgl-module")
}

lwjglPlatforms {
    addSupport(FREEBSD, NATIVE_OPTIONAL)
    addSupport(LINUX, NATIVE_OPTIONAL)
    addSupport(MACOS, NATIVE_REQUIRED)
    addSupport(WINDOWS, NATIVE_OPTIONAL)
}

dependencies {
    api(project(":lwjgl-modules:lwjgl"))
}

lwjglPublication.create {
    title("LWJGL - Vulkan bindings")
    description("A new generation graphics and compute API that provides high-efficiency, cross-platform access to modern GPUs used in a wide variety of devices from PCs and consoles to mobile phones and embedded platforms.")
}

The root project

The name of the root project has been changed to lwjgl to match the naming scheme used for the rest of the projects.

Platform automation

To let the platform know what native artifacts each module contains, each module produce a metadata artifact. This artifact is added to a metadata configuration. The platform then simply add all modules as dependencies to a similar metadata configuration. The artifacts are collected and each native artifact is added to the POM with XML post processing. This setup is the result of the following discussion. This might become a whole lot easier if there is any development in this issue.

Missing

I might have missed some changes as this became a quite large PR.

Notes

This is my first ever PR and it was also my first time using Kotlin and Intellij. This means that there could be things that I have missed, but I have done my best to make sure everything works. I have compared the old publications to the new one and I could not find any functional differences. The code has been formatted to the style configured by the project.

Also the .idea folder has been left untouched. It might require some changes to accommodate the new project structure.

This branch is a cleaned up version of my development branch. I ended up trying a few setups before settling on this one. If any part of the commit seem out of context, it is because of this.

@jjohannes Have shown interest in this PR but is unable to give feedback until he is back from vacation. I suggest we wait for him to return before deciding anything. He has much more experience when it comes to Gradle and the problem at hand.

If this get merged, I will start work on updating the relevant wiki texts.
In this case, the generated script on the website for Gradle should probably also be updated.

This PR is the result of the following discussions:

• Handling LWJGL natives with custom dependency configurations
• Handling LWJGL natives in modular Java projects using gradle

If this PR is too much of a change, then feel free to pick out the best parts. I would still wait for jjohannes before making any big decisions. In the mean while, feel free to have a look at the code. 🙂

[knokko]

In your example, you show how you can include the natives for windows-x64:

val lwjglVersion = "3.4.0"
val lwjglOs: String = "windows"
val lwjglArch: String = "x64"

configurations.matching(Configuration::isCanBeResolved).configureEach {
    attributes {
        attribute(OperatingSystemFamily.OPERATING_SYSTEM_ATTRIBUTE, objects.named(lwjglOs))
        attribute(MachineArchitecture.ARCHITECTURE_ATTRIBUTE, objects.named(lwjglArch))
    }
}

How would you include natives for multiple OS's and or arch's? For instance, including all of natives-windows, natives-windows-x86, and natives-windows-arm64? Or including every single native LWJGL has?

[Dwarfley]

@knokko In the current setup, there exists one variant for each platform (os + arch combination). This variant has the <binding>-platform capability which is the same for all platform variants. The normal variants then require the platform capability. This means that one of the platform variants will be included automatically alongside the normal variant.

The important part here is that only one platform variant can be included (the platform variant with the same platform attributes as specified in the example you showed). It is not possible to include more than one platform variant alongside the normal ones due to how capabilities work. When a variant has a capability, it signals that this variant is capable of something. It does not make sense to have two variants which add the same functionality. An example could be different logger implementations sharing a logging capability. Only one implementation should be used at any given time.

This also reflects the Java side of LWJGL. Each binding has native modules with the same name. Only one of them can be included at compile/runtime since no two modules can have the same name. So even if my Gradle setup would allow for multiple platforms, Java would complain that there exists more than one module with the same name.

The only reason for having multiple platforms at the same time that I could think of would be if you wanted to download them and use them in distributions. This, however, falls outside of normal compilation and execution and should probably be handled separately. For this I suggest artifact views. They allow you to redo the variant selection with different attributes. You would have to do one view for each platform unless we add platform group variants. They would still have the same platform capability, but would contain multiple or all natives. Here is an example of using artifact views to fetch a platform artifact:

val artifactView = configurations.runtimeClasspath.incoming.artifactView {
	withVariantReselection()
	attributes {
		attribute(OperatingSystemFamily.OPERATING_SYSTEM_ATTRIBUTE, objects.named("windows"))
		attribute(MachineArchitecture.ARCHITECTURE_ATTRIBUTE, objects.named("x64"))
	}
}

// use artifactView.files to get the files

Please note that the code above is more or less copied from the documentation that I linked and some tweaking could be necessary for real world applications. There could also be issues related to there being more files than just the native that would need to be addressed.

I am also open to suggestions if there is a better way. I designed it with the current LWJGL Java module design in mind. If that changes to allow multiple natives to be used at the same time, then my setup could change to reflect that.

[knokko]

This also reflects the Java side of LWJGL. Each binding has native modules with the same name. Only one of them can be included at compile/runtime since no two modules can have the same name.

It may be true that only one variant can be used at runtime (and using multiple doesn't make sense), but the existing LWJGL native system definitely allows us to ship multiple variants. For instance, I can happily do

dependencies {
	for (natives in ["natives-windows", "natives-windows-arm64", "natives-linux", "natives-linux-arm64", "natives-macos", "natives-macos-arm64"]) {
		runtimeOnly "org.lwjgl:lwjgl::$natives"
		runtimeOnly "org.lwjgl:lwjgl-glfw::$natives"
		runtimeOnly "org.lwjgl:lwjgl-sdl::$natives"
		runtimeOnly "org.lwjgl:lwjgl-vma::$natives"
		if (natives.contains("macos")) {
			runtimeOnly "org.lwjgl:lwjgl-vulkan::$natives"
		}
	}
}

If I use the com.gradleup.shadow plug-in with this setup, I can happily create a single JAR file containing all natives for 64-bit Windows, Linux, and MacOS. This same JAR file will work on all these platforms, and pick the right natives at runtime. (Don't ask me how this works though, I just consider it magic...)

Personally, I would appreciate it if doing this remains easy with your new setup.

For this I suggest artifact views. They allow you to redo the variant selection with different attributes.

I must admit my knowledge of Gradle is pretty much the bare minimum I need to compile my code and manage my dependencies. It is not immediately obvious to me how I would accomplish the same thing using artifact views. I think adding an example of doing this would be quite convenient.

In fact, I would say that LWJGL could really use a tutorial or example on how to publish games. The current LWJGL configurator just spits out a build.gradle that will work on the development machine, and occasionally surprises developers that their JARs fail on other machines. I think there are a couple of potentially interesting ways to publish, and ideally each of them should be easy with your new setup:

• generate a fat JAR that works on the developer OS/arch
• generate a fat JAR that works on all OS's and arch's supported by LWJGL
• generate a fat EXE that works on a specific OS/arch (for Linux and MacOS, this file would not have the .exe extension, but the concept is the same)
• generate a ZIP file that should be extracted before end users can run it, which may or may not support multiple OS's or arch's

This may seem quite out of scope for this PR, but please note that:

• adding an example is very useful documentation
• when you add an example, you find out whether the new setup supports this conveniently. if not, you could refine it.

That having said, I'm not the one who decides whether/when PR's get merged, so there is plenty of room for disagreement by you, or anyone else who happens to spectate this PR.

[Dwarfley]

@knokko Sorry for the delay in responding.

Personally, I would appreciate it if doing this remains easy with your new setup.

I agree and the intention of this PR is not to make anything more difficult. Rather it augments the current system by automatically selecting the correct native based on the specified platform attributes. I have based this design on the conversations I have had with people over at the Gradle forums. If the issue mentioned above leads to anything, the attributes might not even be needed. Gradle could set them based on the platform that is executing the build.

This all means that adding multiple natives as dependencies might be a bad idea. I do not know how a multiplatform build would work in this context. Since only one native module can be included, my guess is that the normal build should only include one native platform. Later when building a fat jar or some distribution, all natives are added separately. Artifact views could be used to fetch them, but I do not know how this would work with fat jar plugins like the one you mentioned. Another option could be to not use the same dependencies when running the build vs creating the fat jar. Regardless, I think the issue mentioned earlier will clear some of this up. I will add a comment so that multi platform releases are considered in the discussion.

This may seem quite out of scope for this PR, but please note that:

• adding an example is very useful documentation
• when you add an example, you find out whether the new setup supports this conveniently. if not, you could refine it.

That having said, I'm not the one who decides whether/when PR's get merged, so there is plenty of room for disagreement by you, or anyone else who happens to spectate this PR.

I do not disagree with you, I just do not have all the answered just yet. When the issue mentioned above concludes, I will hopefully have more answers. When a design is finally agreed upon, I could provide better examples and also update documentation to incorporate some of the things discussed. If the issue does not result in anything useful, I will have to investigate further how cross platform builds could be achieved.

Since the outcome of the issue is highly relevant, it would be best to wait and see what they come up with before making any further changes to my setup.

[knokko]

Another option could be to not use the same dependencies when running the build vs creating the fat jar.

I have no strong opinion about this. If using the same dependencies is complicated or dirty, we should find a better way. Building multi-platform distributions is after all something else than just running on the machine that builds it.

Since the outcome of the issue is highly relevant, it would be best to wait and see what they come up with before making any further changes to my setup.

Hm... I'm curious what this issue will lead to!

[jjohannes]

RE: Using multiple Jars in one build.

I think, there are two scenarios to distinguish:

1. Building multiple deliverables (exe, dmg, ...) for multiple operating systems in one build
   This scenario is what I mostly think about, if I want to deliver a complete application or installer (like a exe or a dmg). Which I can do nowadays, for example, with jpackage. Doing this should become easier with what this PR proposes. Here, you don't just have one runtimeClasspath but multiple (e.g. macosRuntimeClasspath, winRuntimeClasspath, ...) each with the OS/ARCH attributes set to different values. When one of the deliverables is built, the corresponding *RuntimeClasspath is used.
   This is how the java-module-packaging Gradle plugin operates. In this example you then have e.g. jpackageMac and jpackageWin which build different deliverables and each use a differently configured *RuntimeClasspath.
   This is very similar how Gradle is used in other contexts that have such "multi-variant"/"multi-deliverables" requirements such as Kotlin Multi-Platform or Android.
2. Building a single delivery (FatJar, "Fat" Zip, ...) that works on multiple systems
   This is what is described here: Gradle publishing overhaul #1081 (comment)

The scenarios look at the natives Jars differently:

1. Here the natives Jars are mutually exclusive. Which they are if you want to use them directly on the Module Path, because they all share the same Module Name. From this perspective, I cannot build one single deliverable, but only one per natives.
2. Here you basically say each natives Jar adds a feature (support for certain OS/ARCH) to my single deliverable. As described above, this is technically possible but you need to do some "tinkering" as the user to get a working result. E.g. ignore or throw away the duplicated module-info.class files.

This is only to re-iterate over what is discussed here from my point of view. With these different scenarios in mind, we should find a good compromise all scenarios users have.

I think the solution proposed here is good from what will get published. Building a "fat" deliverable would then have to be approached a bit differently. I think it is fine, but yes, it needs to be clearly documented. You basically "shift" the combination of different "Native Variants" from the dependency declaration (dependencies block) to the place where you build the "fat deliverable" and say there which variants you want to combine.
One advantage is that you do not need to mention each single Jar file in all combinations (the loop inside the dependencies block in this comment).

You would then no longer care about os/arch inside the dependencies block, but configure the Shadow plugin as follows if you have the *RuntimeClasspath configurations in place:

tasks.shadowJar {
  configurations = listOf(
    project.configurations.macRuntimeClasspath.get(),
    project.configurations.macArmRuntimeClasspath.get(),
    project.configurations.winRuntimeClasspath.get()
  )
}

(Here is a branch of the before mentioned example showing that in a working example)

[jjohannes]

Some initial feedback. The structure and direction looks good imo. But I wonder if this can be done with a bit less custom plugin code.

[jjohannes]

Which file should getFile("") refer to?

I had to change this to true to get it working on my machine.

[Dwarfley]

LWJGL uses Ant to build the artifacts and Gradle to publish them. The getFile(path: String) resolves a file relative to the module folder in the Ant output folder bin/RELEASE/. So getFile("") returns bin/RELEASE/lwjgl-glfw/ for the GLFW module. This folder contains all the artifacts for that module. The isPresent variable tells us if the module has been built by checking if that folder exists. If the module has not been built, then we skip its publication.

If you did not build the artifacts, then it would be expected behaviour to not see any publications. This could explain why you had to set isPresent to true.

This is based on this code in the original build script.

[jjohannes]

Okay thank you. That part of the build setup was something I did not yet fully understand. If I find some more time, I will see if I can test it fully by doing the Ant build first.

Maybe it would be helpful for future readers of the setup if there would be comment above the isPresent to explain what you just explained here. Maybe consider to rename getFile to something like getAntBuildResult. I thought getFile was part of a Gradle API because of the generic name. 😄

[jjohannes]

Have you tried using the java-library plugin and extending the java component instead of creating everything from scratch here? Was there an issue?

Otherwise it may save a lot of code that seems to repeat what the java-library plugin would already do for you.

[Dwarfley]

I did try a few different approaches before settling on this one. The reason for not using the java-library plugin was that it provides much more than was required. It also assumes that the project is a standard java library with source code that gets compiled into artifacts. This is not true in this case as there is no source code and only prebuilt artifacts. I would have to undo some of the things that the plugin has created in order to get it to work. For example the plugin configures artifacts that get built by tasks. Since there is no source, these artifacts would be missing. The project would also be cluttered with compilation stuff that is not needed.

I could start to undo some of the things that the plugin configured, but I decided that it was easier to start from the ground and build up than to start from the sky and destroy down. Especially since I do not have deep knowledge about what exactly needed to be undone.

I think we can leave it as it is until such a time that a full Gradle migration happens. In that case the lwjgl-adhoc plugin would of course be replaced by the java-library plugin. That being said, I could change it to use the java-library plugin if you think it would be better. In that case, I would probably need some guidance about what I need to undo in order for it to work.

[jjohannes]

Thanks for elaborating. I think attempting to use the java-library plugin and then "turning off" some things would have these advantages:

• Less custom code that repeats things that are already in the java-libary plugin and thus needs to be maintained to do the exactly same thing the plugin already does (Things like: defining api, runtimeOnly etc., setting the correct standard attribute values, ...)
• The setup is already following a "standard setup" as much as possible. If the build should move from Ant to Gradle in the future, one can do that by removing/replacing some of the "turned off" parts gradually. Instead of completely turning things around again.

From my experience, turning off basically means re-configuring the artifacts of published (consumable) variants. Then tasks like "compileJava" or "jar" will exist, but they will never run as part of the build as the artifacts they would produce are not used.

The main "trick" is doing this:

configurations.apiElements {    // published consumable configuration
    outgoing.artifacts.clear()  // Remove pre-configured JAR artifact, instead add you own artifacts afterwards
}

[jjohannes]

The term -platform is already used in Gradle (e.g. in attribute values) to refer to BOMs – as BOM is called platform in Gradle.
To avoid confusion, I would call this -natives. It is also good practice to use the same term that is used for the classifier of the underlying Jar(s) (which is -natives in this case).

[Dwarfley]

The setup uses the term platform since not all platforms contains natives. It could be confusing if a capability with the word native in the name does not actually contain natives. I understand that the term for a platform (os + arch) conflicts with the Gradle term for a platform (BOM). So I guess it comes down to which option is less bad. Do you suggest I change the platform capability only or the entire platform concept? Just changing the capability would be easy, but changing all mentions of platform (os + arch) could be trickier. It could potentially be confusing to add support for platforms if the term platform is replaced with natives:

lwjglNatives {
    addSupport(FREEBSD, OPTIONAL)
    addSupport(LINUX, OPTIONAL)
    addSupport(MACOS, REQUIRED)
    addSupport(WINDOWS, OPTIONAL)
}

Also do you mean that the -platform in the capability is actually conflicting (will produce errors) or just conflicting with the terminology? If it is actually conflicting, then there is no debate and I will of course change it.

[jjohannes]

Thanks for explaining. The term platform is really overused. 😞

I only talk about the String here. Not how you name it in general in the build setup. My thinking was that if the published Jar file has the classifier -natives, the capability that is linked to that through the metadata should also have -natives in the capability for consistency. May also easier for the users to comprehend if they need to use the capability directly in a dependency declaration in some special setups.
But tbh I have not yet the full overview of what will get published and may still miss something.

That being said, I don't think using -platform would lead to an error. It is used by Gradle, but in attribute values. Not in capabilities.

[Dwarfley]

@jjohannes I have now responded to your comments about the code and provided explanations for my decisions. Should you still have suggestions for changes, then I am open to implement them. I trust your judgement more than mine since I am not a Gradle/Kotlin expert. 🙂

I only ask that we are all in agreement before I spend any substantial amount of time on it.

[jjohannes]

@Dwarfley I added my thoughts to your comments. Maybe I should have mentioned that these are all just ideas/feedback. 😀 From my side, I don't think anything needs to be done totally different or something like that.

I only ask that we are all in agreement before I spend any substantial amount of time on it.

That makes sense. I will think about all of this some more and if I find the time, I will test this more thoroughly and share my thoughts then.