Deterministic startup of hardware interfaces

[firesurfer]

Is your feature request related to a problem? Please describe.

Currently the setup order of hardware interfaces is non deterministic in the way that when we have multiple hardware interfaces the order of: init, configure,activate is not staged.

By staged I mean that all hardware interfaces have to finish a specific stage before any hardware interface can go into the next stage

Example with two hardware interfaces

Interface A -> on_init
Interface B -> on_init

Interface A -> on_configure
Interface B -> on_configure

Interface A -> on_activate
Interface B -> on_activate

-> When all interfaces are done -> start read/write

This would help a lot in our case with sharing common hardware resources (ethercat bus master) between hardware interfaces.

Describe the solution you'd like

I suggest adding an additional flag to the configuration of the controller manager (e.g. deterministic_startup)
which enforces in the resource manager the staging procedure described above.

Describe alternatives you've considered

I currently see for our case only two alternatives:

a) Introduce a separate process which handles the setup and have the hardware interfaces communicate with that process -> But then I do not have to use ros2control at all
b) Basically introduce our own state machine in all related hardware interfaces which the startup procedure

Additional context

Questions:

1. Would a PR with such a feature be appreciated ?
2. Is there any big show stopper I am not aware of that prohibits implementing such a feature ?

[christophfroehlich]

This sounds reasonable, but I'd like to see the opinion of @saikishor @destogl or @bmagyar on this.

How do you share the resources across different hardware plugins? Can you provide a solution for #1956?

[firesurfer]

We implemented a singleton for our ethercat bus master which is shared between the different instances.
It is actually open source: https://github.com/Duatic/ethercat_sdk_master/blob/master/include/ethercat_sdk_master/EthercatMasterSingleton.hpp

Basically the idea is that every hardware interface registers the devices to the bus master instance and obtains a handle. When the hardware interface is in the configuration step we call markAsReady and if all handles have been marked as ready we start the bus.

In addition we currently need code which checks in read/write if the master has been started yet.

But in general this approach is hard to properly handle in detail especially because with ethercat we also need to worry about timing.

[saikishor]

Hello @firesurfer

We can discuss about that parameter, but let's brainstorm a bit because you might want n of the m components to start synchronized but the others not? Let's try to find a common ground?

What's the reason behind the Singleton EtherCAT master? Why not a System component directly?.

For now, you might be able to achieve the same with the hardware component services atleast

[firesurfer]

Hey @saikishor

We can discuss about that parameter, but let's brainstorm a bit because you might want n of the m components to start synchronized but the others not? Let's try to find a common ground?

Do I understand you correctly that you suggest only synchronizing selected components ?
Sound like a nice addition but makes implementation of course more difficult

What's the reason behind the Singleton EtherCAT master? Why not a System component directly?.

Its been quite some time since I actually did the initial implementation and also checked there for better options. As far as I am aware there is not really a better option for sharing common resources between hardware interfaces ?
Its basically the same issue as in #1956

[firesurfer]

@saikishor just as friendly ping - I would really like to hear your input before starting to work on this feature

[saikishor]

@firesurfer I already got a nice idea, I forgot to write it here. Sorry about that

The current hardware components XML support an argument call group, basically we use to make sure that if one of the group actuator is in error, we take down the whole group. You could use that information to be able to group the components together and then start them in the order as you wanted?. Will that work for you

[destogl]

@firesurefer if this is only about the startup, then I would consider external manual startup of components. This is what we are doing for the deterministic startup. Basically you say that all components should start in the unconfigured state, and then you manually go through the lifecycle.

[firesurfer]

@saikishor Sorry for the late answer. Somehow it is kinda random when I get notification emails from github.

You could use that information to be able to group the components together and then start them in the order as you wanted?. Will that work for you

In general this sounds like a good idea. It would have to support:

1. Define a group of hardware components
2. Declare this group as "synchronized"
3. Make sure they start as explained before:

Example with two hardware interfaces

Interface A -> on_init
Interface B -> on_init

Interface A -> on_configure
Interface B -> on_configure

Interface A -> on_activate
Interface B -> on_activate

-> When all interfaces are done -> start read/write

@destogl I wasn't even aware that there is the possibility to externally control the hardware component state.

Nevertheless I think adding a sychronization feature would help quite a lot as I ran in to the same/similar issue with multiple different pieces of hardware where a synchronized startup procedure would have helped quite a lot

[firesurfer]

@destogl I would like to test as an intermediate solution your suggestion, but I couldn't find any documentation or example on how to set the initial hardware interface state.

Perhaps you could give a small example on how that is done ?

[christophfroehlich]

Do we speak about hardware components (then see
hardware_components_initial_state and the hardware_spawner) or individual hardware interfaces? The latter don't have individual lifecycle states

[firesurfer]

I am talking about a class which inherits from hardware_interface::SystemInterface (https://github.com/Duatic/dynaarm_driver/blob/96d349aff821ec87926a6f126e7a57a8fcd3fad6/dynaarm_driver/dynaarm_hardware_interface_base/include/dynaarm_hardware_interface_base/dynaarm_hardware_interface_base.hpp#L52C45-L52C80)

which given that is has "on_init" up to "on_activate" I always thought to be a hardware component. Please correct me if I am wrong.

But I think this is exactly what I am looking for!

[christophfroehlich]

Yeah I know that we don't have a consistent naming here, as the package and namespace of the components (the plugins) also is hardware_interface. But we are at a point that renaming this is quite hard.

[nnarain]

Hello. Just jumping in to say we also use singletons to share data across hardware components. In our case it's for CAN bus and MODBUS devices, but the same idea.

Edit: missed the linked use issue, I'll comment there