Service orchestration architecture

[EclesioMeloJunior]

This discussion is related to the issue #2235

Description

Currently, in the gossamer codebase, we face the following problem: inner goroutines might fail without warning the outers services.

@timwu20 suggested the approach of an Orchestrator which will handler all the service's start and stop functions, and will be responsible for gracefully shutdown the node, waiting for running goroutines stop before exit completely.

All the services managed by the Orchestrator should implement the interface Service which has two methods: Start() (errCh chan error, err error) and Stop() error.

-Start method should return a error channel to the main caller, this channel will be responsible for notify the main parent Orchestrator that something went wrong inside the running service and we need to shutdown gracefully the node. The second return param error is used when the service failed to start.
-Stop method should shutdown the service gracefully, for example: stop running goroutines or saving the current state of the node. And the Stop method should close the returned error channel. The Stop method wil be called by the Orchestrator.

The Orchestrator will follow a pipeline pattern where we should initialize one service before other to guarantee the dependency chain. Some services might delay a considerable time, in this case we should use a readiness status were the Orchestrator only initialize the dependent service once the dependency is ready.

The Orchestrator should be aware of the problems inside the services so it need to collect and track the returned chan error from each service and listen from them in order to shutdown the node gracefully.

[EclesioMeloJunior]

Some examples:

• https://github.com/timwu20/service-example
• https://github.com/EclesioMeloJunior/goregistry