nav2_costmap_2d reports test success despite internal failures

[tonynajjar]

25: [costmap_tests-4] [INFO 1770374255.630603724] [test_costmap.test_costmap]: Destroying
25: [costmap_tests-4] [WARN 1770374255.630609279] [rclcpp_lifecycle]: LifecycleNode is not shut down: Node still in state(1) in destructor
25: [costmap_tests-4] unknown file: Failure
25: [costmap_tests-4] C++ exception with description "parameter 'global_frame' cannot be set because it was not declared" thrown in the test body.
25: [costmap_tests-4] 
25: [costmap_tests-4] [  FAILED  ] TestNode.testDynParamsSet (2 ms)
25: [costmap_tests-4] [----------] 10 tests from TestNode (17 ms total)
25: [costmap_tests-4] 
25: [costmap_tests-4] [----------] Global test environment tear-down
25: [costmap_tests-4] [==========] 10 tests from 1 test suite ran. (17 ms total)
25: [costmap_tests-4] [  PASSED  ] 9 tests.
25: [costmap_tests-4] [  FAILED  ] 1 test, listed below:
25: [costmap_tests-4] [  FAILED  ] TestNode.testDynParamsSet
25: [costmap_tests-4] 
25: [costmap_tests-4]  1 FAILED TEST
25: [ERROR] [costmap_tests-4]: process has died [pid 1079878, exit code 1, cmd '/opt/auto_ws/build/nav2_costmap_2d/test/integration/inflation_tests_exec'].
25: [INFO] [launch]: process[costmap_tests-4] was required: shutting down launched system
25: [INFO] [lifecycle_manager-3]: sending signal 'SIGINT' to process[lifecycle_manager-3]
25: [INFO] [static_transform_publisher-2]: sending signal 'SIGINT' to process[static_transform_publisher-2]
25: [INFO] [map_server-1]: sending signal 'SIGINT' to process[map_server-1]
25: [lifecycle_manager-3] [INFO 1770374255.638471512] [rclcpp]: signal_handler(SIGINT/SIGTERM)
25: [lifecycle_manager-3] [INFO 1770374255.638548657] [lifecycle_manager]: Running Nav2 LifecycleManager rcl preshutdown (lifecycle_manager)
25: [lifecycle_manager-3] [INFO 1770374255.638589577] [lifecycle_manager]: Terminating bond timer...
25: [static_transform_publisher-2] [INFO 1770374255.638653190] [rclcpp]: signal_handler(SIGINT/SIGTERM)
25: [map_server-1] [INFO 1770374255.638755725] [rclcpp]: signal_handler(SIGINT/SIGTERM)
25: [map_server-1] [INFO 1770374255.638805524] [map_server]: Running Nav2 LifecycleNode rcl preshutdown (map_server)
25: [map_server-1] [INFO 1770374255.638846289] [map_server]: Deactivating
25: [map_server-1] [INFO 1770374255.638853997] [map_server]: Destroying bond (map_server) to lifecycle manager.
25: [INFO] [static_transform_publisher-2]: process has finished cleanly [pid 1079829]
25: [map_server-1] [INFO 1770374255.649101344] [map_server]: Cleaning up
25: [map_server-1] [INFO 1770374255.649155386] [map_server]: Destroying bond (map_server) to lifecycle manager.
25: [lifecycle_manager-3] [INFO 1770374255.649503660] [lifecycle_manager]: Destroying lifecycle_manager
25: [map_server-1] [INFO 1770374255.649706869] [map_server]: Destroying
25: [map_server-1] [WARN 1770374255.649732333] [rclcpp_lifecycle]: LifecycleNode is not shut down: Node still in state(1) in destructor
25: [INFO] [map_server-1]: process has finished cleanly [pid 1079828]
25: [INFO] [lifecycle_manager-3]: process has finished cleanly [pid 1079830]
25: -- run_test.py: return code 0
25: -- run_test.py: generate result file '/opt/auto_ws/build/nav2_costmap_2d/test_results/nav2_costmap_2d/inflation_tests.xml' with successful test
1/1 Test #25: inflation_tests ..................   Passed    1.39 sec

The following tests passed:
	inflation_tests

100% tests passed, 0 tests failed out of 1

Total Test time (real) =   1.39 sec
Finished <<< nav2_costmap_2d [1.45s]

[SteveMacenski]

@tonynajjar change this line to declare instead of set

navigation2/nav2_costmap_2d/test/integration/inflation_tests.cpp

Line 604 in 517ebf4

costmap->set_parameter(rclcpp::Parameter("global_frame", std::string("base_link")));

Does that fix the issue? That's likely part of recent refactors in parameter handling. If so, submit a PR and I can merge immediately.

[tonynajjar]

There are 2 issues:

1- A test is failing

2- colcon test reports success although a test is failing.

I'm talking about 2 here (and I'm worried it's not the only package affected)

[SteveMacenski]

It looks like we have a launch file working with that test

navigation2/nav2_costmap_2d/test/integration/CMakeLists.txt

Lines 96 to 103 in 517ebf4

	GENERATE_RESULT_FOR_RETURN_CODE_ZERO
	COMMAND "${CMAKE_CURRENT_SOURCE_DIR}/costmap_tests_launch.py"
	WORKING_DIRECTORY "${CMAKE_CURRENT_BINARY_DIR}"
	ENV
	TEST_MAP=${TEST_MAP_DIR}/TenByTen.yaml
	TEST_LAUNCH_DIR=${TEST_LAUNCH_DIR}
	TEST_EXECUTABLE=$<TARGET_FILE:inflation_tests_exec>
	)

.

The launch file returns success which is why the test passes https://github.com/ros-navigation/navigation2/blob/517ebf4594fe01e1cefd5a9611eff10d24e161e3/nav2_costmap_2d/test/integration/costmap_tests_launch.py

I think we need to fix that, which I don't have the immediate answer to off the top of my head. After we know the solution to this case though, its easy enough to grep for other launch files in test packages/directories and make the similar update. If you can model the fix here, I can update more globally. Luckily we don't use this launch pattern much outside of costmap2D + system test packages (and given we see system test failures, I think most of those are alright as they properly use a launch-testing framework that this does not)

[tonynajjar]

It's not affecting us, I summon the open-source good samaritans for this one. Help needed!

[MFarooq07]

@tonynajjar @SteveMacenski I would love to work on this one after I completely fix #5872

[tonynajjar]

@tonynajjar change this line to declare instead of set. Does that fix the issue?

Yes, I fix it in #5933.

2- colcon test reports success although a test is failing.

@MFarooq07 this is still all yours