Bot fails to mark flaky tests as such in some cases

[yrodiere]

For instance this comment says that HbmXmlQuarkusConfigTest is failing.

But the corresponding build scan says HbmXmlQuarkusConfigTest is flaky.

This may just be a parsing/formatting problem when tests fail outside of a test method (e.g. in setup), since weirdly the GitHub comment adds a trailing dot to HbmXmlQuarkusConfigTest.

[gsmet]

FWIW, the report is the following:

<testcase name="" classname="io.quarkus.hibernate.orm.xml.hbm.HbmXmlQuarkusConfigTest" time="4.171">
<error message="java.lang.RuntimeException: Failed to start quarkus" type="java.lang.RuntimeException">
java.lang.RuntimeException: java.lang.RuntimeException: Failed to start quarkus at io.quarkus.test.QuarkusUnitTest.beforeAll(QuarkusUnitTest.java:708) at java.base/java.util.ArrayList.forEach(ArrayList.java:1596) Caused by: java.lang.RuntimeException: Failed to start quarkus at io.quarkus.runner.ApplicationImpl.doStart(Unknown Source) at io.quarkus.runtime.Application.start(Application.java:101) at java.base/java.lang.reflect.Method.invoke(Method.java:580) at io.quarkus.runner.bootstrap.StartupActionImpl.run(StartupActionImpl.java:285) at io.quarkus.test.QuarkusUnitTest.beforeAll(QuarkusUnitTest.java:661) ... 1 more Caused by: java.lang.RuntimeException: java.lang.OutOfMemoryError: Metaspace at io.quarkus.hibernate.orm.runtime.JPAConfig.startAll(JPAConfig.java:76) at io.quarkus.hibernate.orm.runtime.HibernateOrmRecorder.startAllPersistenceUnits(HibernateOrmRecorder.java:100) at io.quarkus.deployment.steps.HibernateOrmProcessor$startPersistenceUnits1868654632.deploy_0(Unknown Source) at io.quarkus.deployment.steps.HibernateOrmProcessor$startPersistenceUnits1868654632.deploy(Unknown Source) ... 6 more Caused by: java.lang.OutOfMemoryError: Metaspace at java.base/java.lang.ClassLoader.defineClass1(Native Method) at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1027) at io.quarkus.bootstrap.classloading.QuarkusClassLoader.loadClass(QuarkusClassLoader.java:516) at io.quarkus.bootstrap.classloading.QuarkusClassLoader.loadClass(QuarkusClassLoader.java:476) at org.hibernate.internal.SessionFactoryImpl.<init>(SessionFactoryImpl.java:292) at io.quarkus.hibernate.orm.runtime.boot.FastBootEntityManagerFactoryBuilder.build(FastBootEntityManagerFactoryBuilder.java:87) at io.quarkus.hibernate.orm.runtime.FastBootHibernatePersistenceProvider.createEntityManagerFactory(FastBootHibernatePersistenceProvider.java:72) at jakarta.persistence.Persistence.createEntityManagerFactory(Persistence.java:80) at jakarta.persistence.Persistence.createEntityManagerFactory(Persistence.java:55) at io.quarkus.hibernate.orm.runtime.JPAConfig$LazyPersistenceUnit.get(JPAConfig.java:154) at io.quarkus.hibernate.orm.runtime.JPAConfig$1.run(JPAConfig.java:61) at java.base/java.lang.Thread.runWith(Thread.java:1596) at java.base/java.lang.Thread.run(Thread.java:1583)
</error>
<system-out>
2024-06-04 19:10:11,997 WARN [org.hib.orm.deprecation] (build-20) HHH90000028: Support for `<hibernate-mappings/>` is deprecated [INPUT_STREAM : null]; migrate to orm.xml or mapping.xml, or enable `hibernate.transform_hbm_xml.enabled` for on the fly transformation
</system-out>
</testcase>
<testcase name="smokeTest" classname="io.quarkus.hibernate.orm.xml.hbm.HbmXmlQuarkusConfigTest" time="0.242"/>
<testcase name="ormXmlTakenIntoAccount" classname="io.quarkus.hibernate.orm.xml.hbm.HbmXmlQuarkusConfigTest" time="0.006"/>

So it's not exactly obvious that tests are flaky.

[yrodiere]

Right, no idea how easy it is to detect. But Develocity managed to detect it, so I guess there is a way.