[marker_radar_lidar_calibrator] Adding pair is not possible

[Sophie2216]

Hello!
Unfortunately, the marker radar lidar calibrator does not work for me. Hopefully, you can help me with the following problem:

I am trying to calibrate the lidar_top, which is a ouster_os2, with the lrr_front_ars548, which is a ARS548 3D radar. The initial TF tree is shown in the following screenshot:

For a better visualization, the TFs are located in the following way in 3D:

After the background extraction of the marker_radar_lidar_calibrator is finished, the corner reflector is placed into the environment and in the rviz2 visualization one can see the corner reflector as follows. Additionally, the blue dots are manually added to the marker_radar_lidar_calibrator view and showing the radar's detected objects according to the radar's topic.

A closer look is shown in the following two pictures, where it can be seen that the radars detected objects (blue) are not identical to the radar object_raw points (purple according your documentation of the marker_radar_lidar_calibrator). Several green marked lidar points (which are marker from the marker_radar_lidar_calibrator) also appear in the proximity of the corner reflector.


Since the marker_radar_lidar_calibrator finds lidar (green) and radar (purple) dots, the "Add lidar-radar pair" button was able to be pressed. Unfortunately, the pair is not established and the console has the following output:
[marker_radar_lidar_calibrator-2] [INFO] [1732291567.283530748] [marker_radar_lidar_calibrator]: lidarCallback [marker_radar_lidar_calibrator-2] [INFO] [1732291567.283549330] [marker_radar_lidar_calibrator]: There were no tracks [marker_radar_lidar_calibrator-2] [INFO] [1732291567.336945966] [marker_radar_lidar_calibrator]: lidarCallback [marker_radar_lidar_calibrator-2] [INFO] [1732291567.339657780] [marker_radar_lidar_calibrator]: Extracting foreground [marker_radar_lidar_calibrator-2] [INFO] [1732291567.339695228] [marker_radar_lidar_calibrator]: initial points: 131072 [marker_radar_lidar_calibrator-2] [INFO] [1732291567.340580046] [marker_radar_lidar_calibrator]: cropped points: 129459 [marker_radar_lidar_calibrator-2] [INFO] [1732291567.343981659] [marker_radar_lidar_calibrator]: voxel filtered points: 285 [marker_radar_lidar_calibrator-2] [INFO] [1732291567.344404879] [marker_radar_lidar_calibrator]: tree filtered points: 12 [marker_radar_lidar_calibrator-2] [INFO] [1732291567.480623977] [marker_radar_lidar_calibrator]: ransac filtered points: 12 [marker_radar_lidar_calibrator-2] [INFO] [1732291567.480723653] [marker_radar_lidar_calibrator]: Cluster extraction input size: 12 [marker_radar_lidar_calibrator-2] [INFO] [1732291567.480730139] [marker_radar_lidar_calibrator]: found cluster of size: 12 [marker_radar_lidar_calibrator-2] [INFO] [1732291567.480738445] [marker_radar_lidar_calibrator]: Extracting lidar reflectors from clusters [marker_radar_lidar_calibrator-2] [INFO] [1732291567.480745983] [marker_radar_lidar_calibrator]: Colored clusters size=12 [marker_radar_lidar_calibrator-2] [INFO] [1732291567.481355589] [marker_radar_lidar_calibrator]: Extracting foreground [marker_radar_lidar_calibrator-2] [INFO] [1732291567.481374676] [marker_radar_lidar_calibrator]: initial points: 200 [marker_radar_lidar_calibrator-2] [INFO] [1732291567.481387904] [marker_radar_lidar_calibrator]: cropped points: 135 [marker_radar_lidar_calibrator-2] [INFO] [1732291567.481406840] [marker_radar_lidar_calibrator]: voxel filtered points: 127 [marker_radar_lidar_calibrator-2] [INFO] [1732291567.481496107] [marker_radar_lidar_calibrator]: tree filtered points: 70 [marker_radar_lidar_calibrator-2] [INFO] [1732291567.481580595] [marker_radar_lidar_calibrator]: Cluster extraction input size: 70 [marker_radar_lidar_calibrator-2] [INFO] [1732291567.481588946] [marker_radar_lidar_calibrator]: found cluster of size: 4 [marker_radar_lidar_calibrator-2] [INFO] [1732291567.481593272] [marker_radar_lidar_calibrator]: found cluster of size: 4 [marker_radar_lidar_calibrator-2] [INFO] [1732291567.481597285] [marker_radar_lidar_calibrator]: found cluster of size: 4 [marker_radar_lidar_calibrator-2] [INFO] [1732291567.481600920] [marker_radar_lidar_calibrator]: found cluster of size: 4 [marker_radar_lidar_calibrator-2] [INFO] [1732291567.481604756] [marker_radar_lidar_calibrator]: found cluster of size: 4 [marker_radar_lidar_calibrator-2] [INFO] [1732291567.481608270] [marker_radar_lidar_calibrator]: found cluster of size: 4 [marker_radar_lidar_calibrator-2] [INFO] [1732291567.481611837] [marker_radar_lidar_calibrator]: found cluster of size: 4 [marker_radar_lidar_calibrator-2] [INFO] [1732291567.481615115] [marker_radar_lidar_calibrator]: found cluster of size: 4 [marker_radar_lidar_calibrator-2] [INFO] [1732291567.481618768] [marker_radar_lidar_calibrator]: found cluster of size: 4 [marker_radar_lidar_calibrator-2] [INFO] [1732291567.481622034] [marker_radar_lidar_calibrator]: found cluster of size: 4 [marker_radar_lidar_calibrator-2] [INFO] [1732291567.481625137] [marker_radar_lidar_calibrator]: found cluster of size: 4 [marker_radar_lidar_calibrator-2] [INFO] [1732291567.481628448] [marker_radar_lidar_calibrator]: found cluster of size: 4 [marker_radar_lidar_calibrator-2] [INFO] [1732291567.481631796] [marker_radar_lidar_calibrator]: found cluster of size: 4 [marker_radar_lidar_calibrator-2] [INFO] [1732291567.481634958] [marker_radar_lidar_calibrator]: found cluster of size: 4 [marker_radar_lidar_calibrator-2] [INFO] [1732291567.481638471] [marker_radar_lidar_calibrator]: found cluster of size: 4 [marker_radar_lidar_calibrator-2] [INFO] [1732291567.481642061] [marker_radar_lidar_calibrator]: found cluster of size: 4 [marker_radar_lidar_calibrator-2] [INFO] [1732291567.481645473] [marker_radar_lidar_calibrator]: found cluster of size: 4 [marker_radar_lidar_calibrator-2] [INFO] [1732291567.481649190] [marker_radar_lidar_calibrator]: found cluster of size: 1 [marker_radar_lidar_calibrator-2] [INFO] [1732291567.481652909] [marker_radar_lidar_calibrator]: found cluster of size: 1 [marker_radar_lidar_calibrator-2] [INFO] [1732291567.481659360] [marker_radar_lidar_calibrator]: Extracting radar reflectors from clusters [marker_radar_lidar_calibrator-2] [INFO] [1732291567.481663864] [marker_radar_lidar_calibrator]: Radar reflector id=0 size=4 center: x=48.80 y=31.76 z=1.38 [marker_radar_lidar_calibrator-2] [INFO] [1732291567.481670877] [marker_radar_lidar_calibrator]: Radar reflector id=1 size=4 center: x=46.23 y=-12.88 z=1.37 [marker_radar_lidar_calibrator-2] [INFO] [1732291567.481676019] [marker_radar_lidar_calibrator]: Radar reflector id=2 size=4 center: x=46.57 y=-0.98 z=0.97 [marker_radar_lidar_calibrator-2] [INFO] [1732291567.481694995] [marker_radar_lidar_calibrator]: Radar reflector id=3 size=4 center: x=43.78 y=0.54 z=0.74 [marker_radar_lidar_calibrator-2] [INFO] [1732291567.481700286] [marker_radar_lidar_calibrator]: Radar reflector id=4 size=4 center: x=43.71 y=31.40 z=1.52 [marker_radar_lidar_calibrator-2] [INFO] [1732291567.481705257] [marker_radar_lidar_calibrator]: Radar reflector id=5 size=4 center: x=38.28 y=-7.61 z=0.98 [marker_radar_lidar_calibrator-2] [INFO] [1732291567.481710326] [marker_radar_lidar_calibrator]: Radar reflector id=6 size=4 center: x=32.35 y=13.40 z=1.68 [marker_radar_lidar_calibrator-2] [INFO] [1732291567.481715132] [marker_radar_lidar_calibrator]: Radar reflector id=7 size=4 center: x=30.22 y=-4.73 z=0.95 [marker_radar_lidar_calibrator-2] [INFO] [1732291567.481719935] [marker_radar_lidar_calibrator]: Radar reflector id=8 size=4 center: x=24.12 y=-3.04 z=0.46 [marker_radar_lidar_calibrator-2] [INFO] [1732291567.481724275] [marker_radar_lidar_calibrator]: Radar reflector id=9 size=4 center: x=20.25 y=-3.39 z=0.05 [marker_radar_lidar_calibrator-2] [INFO] [1732291567.481728954] [marker_radar_lidar_calibrator]: Radar reflector id=10 size=4 center: x=13.09 y=-10.34 z=1.01 [marker_radar_lidar_calibrator-2] [INFO] [1732291567.481733534] [marker_radar_lidar_calibrator]: Radar reflector id=11 size=4 center: x=10.21 y=-0.30 z=-0.02 [marker_radar_lidar_calibrator-2] [INFO] [1732291567.481738467] [marker_radar_lidar_calibrator]: Radar reflector id=12 size=4 center: x=9.80 y=-2.76 z=0.47 [marker_radar_lidar_calibrator-2] [INFO] [1732291567.481743346] [marker_radar_lidar_calibrator]: Radar reflector id=13 size=4 center: x=10.57 y=-5.40 z=0.13 [marker_radar_lidar_calibrator-2] [INFO] [1732291567.481748367] [marker_radar_lidar_calibrator]: Radar reflector id=14 size=4 center: x=7.16 y=4.29 z=-0.23 [marker_radar_lidar_calibrator-2] [INFO] [1732291567.481752716] [marker_radar_lidar_calibrator]: Radar reflector id=15 size=4 center: x=4.87 y=-2.68 z=0.95 [marker_radar_lidar_calibrator-2] [INFO] [1732291567.481756941] [marker_radar_lidar_calibrator]: Radar reflector id=16 size=4 center: x=28.56 y=13.86 z=1.12 [marker_radar_lidar_calibrator-2] [INFO] [1732291567.481761633] [marker_radar_lidar_calibrator]: Radar reflector id=17 size=1 center: x=19.72 y=13.63 z=0.95 [marker_radar_lidar_calibrator-2] [INFO] [1732291567.481766340] [marker_radar_lidar_calibrator]: Radar reflector id=18 size=1 center: x=15.98 y=-15.70 z=0.37 [marker_radar_lidar_calibrator-2] [INFO] [1732291567.482007915] [marker_radar_lidar_calibrator]: Lidar detections: 0 Radar detections: 19 Matches: 0 Active tracks; 0 Converged tracks: 0
So it seems that radar detections are found but no lidar detections according to the console output although in the rviz2 visualization there are green marked lidar points.

So, why is it not possible to add the lidar-radar pair?

[knzo25]

@Sophie2216
Hi, thank for using our tools.

Regarding the ARS548, we support it in this branch. Are you using it?
In a little unrelated topic, are you using our driver for that device?

Until the related PR is merged it is difficult to give proper support. That being said, if you provide us some rosbag, we can at least have a look a it !

[Sophie2216]

@knzo25
Hi,
Currently, we are using the tier4/universe. So, thanks for redirecting us to the "feature/marker_radar_lidar_calibrator_support_radars_and_transformation_algorithms" branch. We will try this one.
Yes, we are using your driver for that device.

I would like to send you the rosbag via email. Would this be possible?

[knzo25]

Thanks for the confirmation, the before mentioned PR used our driver so I can not really give any sort of guarantees otherwise.
My mail is kenzo.lobos at tier4.jp

[knzo25]

@Sophie2216
Sorry, I have been away for an experiment. As soon as I finish catching up with work I will take a look

[Sophie2216]

@knzo25
No problem!

We have tried to do the lidar-radar calibration by using the "feature/marker_radar_lidar_calibrator_support_radars_and_transformation_algorithms" branch. Unfortunately, it didn't work. The default project was selected in combination with the marker_radar_lidar_calibrator and the following variables were inserted in the launcher configuration:

We thought that the radar's scan_raw topic may fit best from the following possible topics of the ARS548 radar:
/sensing/radar/lrr_front_ars548/detection_points
/sensing/radar/lrr_front_ars548/detections
/sensing/radar/lrr_front_ars548/marker_array
/sensing/radar/lrr_front_ars548/nebula_packets
/sensing/radar/lrr_front_ars548/object_points
/sensing/radar/lrr_front_ars548/objects
/sensing/radar/lrr_front_ars548/objects_raw
/sensing/radar/lrr_front_ars548/objects_raw_filtered
/sensing/radar/lrr_front_ars548/output/radar_detected_objects
/sensing/radar/lrr_front_ars548/output/radar_tracked_objects
/sensing/radar/lrr_front_ars548/scan_raw

The sensor_calibration_manager window, RViz, and the metrics plotter appeared, but the 'Calibrate' button was not available and in the terminal the following lines showed up:
{ade}-adm@innb0:~/WARE/calibration_tools_ws$ros2 run sensor_calibration_manager sensor_calibration_manager
QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-ware'
INFO:root:on_selected_calibrator: project_name=default_project calibrator_name=marker_radar_lidar_calibrator
INFO:root:Reading xml from: /home/ware/WARE/calibration_tools_ws/install/sensor_calibration_manager/share/sensor_calibration_manager/launch/default_project/marker_radar_lidar_calibrator.launch.xml
INFO:root:Closing calibrator selector view
^[[AINFO:root:Launching calibrator with the following command: ['ros2', 'launch', '/home/ware/WARE/calibration_tools_ws/install/sensor_calibration_manager/share/sensor_calibration_manager/launch/default_project/marker_radar_lidar_calibrator.launch.xml', 'radar_optimization_frame:=base_link', 'radar_frame:=lrr_front_ars548', 'lidar_frame:=lidar_top/os_lidar', 'msg_type:=radar_scan', 'transformation_type:=svd_3d', 'input_radar_msg:=/sensing/radar/lrr_front_ars548/scan_raw', 'input_lidar_pointcloud:=/sensing/lidar/top/points', 'rviz:=true']
[INFO] [launch]: All log files can be found below /home/ware/.ros/log/2024-12-13-14-15-05-821125-ware2-11976
[INFO] [launch]: Default logging verbosity is set to INFO
[INFO] [static_transform_publisher-1]: process started with pid [11998]
[INFO] [marker_radar_lidar_calibrator-2]: process started with pid [12000]
[INFO] [calibrator_ui_node.py-3]: process started with pid [12002]
[INFO] [metrics_plotter_node.py-4]: process started with pid [12004]
[INFO] [rviz2-5]: process started with pid [12006]
[static_transform_publisher-1] [WARN] [1734095706.152362657] []: Old-style arguments are deprecated; see --help for new-style arguments
[static_transform_publisher-1] [INFO] [1734095706.162631928] [lidar_broadcaster]: Spinning until stopped - publishing transform
[static_transform_publisher-1] translation: ('0.000000', '0.000000', '0.000000')
[static_transform_publisher-1] rotation: ('0.000000', '0.000000', '0.000000', '1.000000')
[static_transform_publisher-1] from 'lrr_front_ars548' to 'radar_frame'
[marker_radar_lidar_calibrator-2] ERROR: flag 'logtostderr' was defined more than once (in files './src/logging.cc' and '/home/ware/WARE/autoware/src/universe/external/glog/src/logging.cc').
[ERROR] [marker_radar_lidar_calibrator-2]: process has died [pid 12000, exit code 1, cmd '/home/ware/WARE/calibration_tools_ws/install/marker_radar_lidar_calibrator/lib/marker_radar_lidar_calibrator/marker_radar_lidar_calibrator --ros-args -r __node:=marker_radar_lidar_calibrator -r __ns:=/ --params-file /tmp/launch_params_b5w3go4o --params-file /tmp/launch_params_6kcf_vkz --params-file /tmp/launch_params_qb1f3nna --params-file /tmp/launch_params_nhoc1w32 --params-file /tmp/launch_params_d39u93di --params-file /tmp/launch_params_l3uild_r --params-file /tmp/launch_params_6mek7k99 --params-file /tmp/launch_params___den0ar --params-file /tmp/launch_params_edolvfxj --params-file /tmp/launch_params_imn0r8is --params-file /tmp/launch_params_u06dit3h --params-file /tmp/launch_params_hpoke5d2 --params-file /tmp/launch_params_o3fuzqbh --params-file /tmp/launch_params_g4_rq1wu --params-file /tmp/launch_params_5v02qdqq --params-file /tmp/launch_params_xirtwwkv --params-file /tmp/launch_params_s0dy57ls -r extrinsic_calibration:=calibrate_radar_lidar -r input_lidar_pointcloud:=/sensing/lidar/top/points -r input_radar_msg:=/sensing/radar/lrr_front_ars548/scan_raw'].

The following xml-file (I had to upload it as a txt-file since xml is not a supported file format for uploading) was used for launching the ARS548:
lrr_front_ars548.launch.txt

I hope that you can help with this amount of information to solve the issue such that we can calibrate with the marker_radar_lidar_calibrator.

Thanks for your help!

[Sophie2216]

Hi!
Do you have any idea why the marker_radar_lidar_calibrator from branch "feature/marker_radar_lidar_calibrator_support_radars_and_transformation_algorithms" is not working for us?

[Sophie2216]

Hello @knzo25 ,
Do you already have an idea why this calibration is failing for me and how this could possibly be resolved?
I would highly appreciate receiving an answer as soon as possible.

[knzo25]

@Sophie2216
I will try it now (finally got a breather while I crunch some data), but please remember that this repository is an OSS effort.

[knzo25]

I confirmed that the pipeline works well until cluster detection (using the tier4/main branch, i.e., using the old algorithm).
The problem seems to be that the reflector_max_height is set to an unsuited value. Depending on what frames you use, the value changes its meaning

On another node, for the ARS408 we used objects (the driver we used provided only those). For the ARS548 we recommend using detections, which come in the sensor frame, which is needed for optimization, unless you want to do the math by yourself afterwards.

I saw a lot of frames, which may have made me use incorrect ones, but please read the meaning of each frame we use in the documentation before deciding which ones to use

[Sophie2216]

@knzo25
Thanks for the quick reply! Very appreciated!

We added a frame lrr_front_ars548_footprint with xyz:=0 0 0 and rpy:=0 0 0, which has a parent the frame lrr_front_ars548.

First, we tried the "feature/marker_radar_lidar_calibrator_support_radars_and_transformation_algorithms" branch with the svd_2d algorithm. The recommended /sensing/radar/lrr_front_ars548/detections with the msg_type:=radar_cloud was used and led to the same error in the terminal als shown in my previous comment on Dec, 13th 2024.

Thus, we pulled the "tier4/universe" branch. When we used this branch, we wanted to use the detections as you suggested, but it was not possible since in the terminal the marker_radar_lidar_calibrator has written that the type RadarTracks is required. Unfortunately, /sensing/radar/lrr_front_ars548/detections has the type PointCloud. Thus, we tried it with /sensing/radar/lrr_front_ars548/objects_raw. With this topic it worked in the end such that we were able to register the radar-lidar pairs. BUT the /sensing/radar/lrr_front_ars548/objects_raw topic has as the header.frame_id the base_link. This does not calibrate the tf links from base_link to lrr_front_ars548.

Additionally, when we added the radar-lidar pairs and clicked "Send calibration", an error appeared in the sensor_calibration_manager window which said something like "Error. tf is not included" such that calibration was not able to be saved.

How can we use the detections topic for calibration?
Why does the feature/marker_radar_lidar_calibrator_support_radars_and_transformation_algorithms branch throw that error?

[knzo25]

I do not remember the specifics of the radar calibration development branch, for that you would need to reach its author.

radar_cloud should use the radar detections though (you can verify it through this and the driver repos)
Error. tf is not included seems to be an issue with the integratin of the sensor_calibration manager (either the launch or python counterpart)

[Sophie2216]

@vividf I have seen that you are an author of the branch 'feature/marker_radar_lidar_calibrator_support_radars_and_transformation_algorithms'. I need your expertise for solving my issue with the marker_radar_lidar_calibrator. I would highly appreciate receiving an answer as soon as possible. Please can you help me to find out what is the reason that the process dies as shown in my previous comment on Dec, 13th 2024 and how can I resolve the issue?

We would like to use this branch since it seems to be the only way to use the /sensing/radar/lrr_front_ars548/detections topic of the Radar ARS548, which has the type PointCloud. In the comment on January, 10th 2025, a better insight is given why we have to use this topic instead of using /sensing/radar/lrr_front_ars548/objects_raw, which would work already with the branch 'tier4/universe'.

[vividf]

@Sophie2216 Hi,

Could you please test using this branch? It contains the latest updates for supporting the ARS548.

For ARS548, please use either radar_cloud or radar_scan instead of radar_track. Based on the information I received earlier, the radar_track frame ID is set to base_link instead of radar_frame.

[Sophie2216]

@vividf Hi,
I was able to try the branch, which you recommended. The following parameters were used:
radar_optimization_frame = base_link (Since I saw that you used base_link here for ARS548, I used it too)
radar_frame = lrr_front_ars548
lidar_frame = lidar_top/os_lidar
msg_type = radar_cloud
transformation_type = svd_3d
input_radar_msg = /sensing/radar/lrr_front_ars548/detection_points
input_lidar_pointcloud = /sensing/lidar/top/points

Unfortunately, I was not able to add a lidar-raidar pair due to the flickering although I tried clicking multiple times the available button "Add lidar-radar pair". In the follwing video, the problem is shown by the projected foreground points of lidar and radar:

How can I fix this problem?