Multiple asyncapi yaml files

[alex-zywicki]

Hi,

Connexion offers the ability to mount multiple API yaml files onto a single server. Is this a capability that asynction has or that could be added in the future? I would like to use asynction for a product I am starting on that also uses connexion, and the ability to modularize features into plugins with separate yaml files is a must for me.

Is this something you would be willing to add or willing to merge if someone were to develop it?

[dedoussis]

Hi @alex-zywicki,

This is not something that Asynction (or Flask-SocketIO) supports atm.

Before going down the path of contemplating the addition of such a feature, I'd like to understand whether this multi-API setup would make sense in the Socket.IO world.

In Connexion, each API is registered (via the add_api method) under a separate base path. However, this cannot be the case for Socket.IO, as by design, a Socket.IO server only supports a single path.

As far as I can see, the only way to split a Socket.IO server into multiple API's is through namespaces. We could possibly have a multi spec setup, where each AsyncAPI file documents a unique subset of the namespaces of the server. IMHO, each AsyncAPI file should not depend on the rest, and should be sufficient to define a functional Socket.IO API, without needing to register any further spec. I would really like to avoid having incomplete specifications, and having to merge all specs into a mega spec.

To re-iterate, for the multi-API feature:

1. Each AsyncAPI file would document a subset of the namespaces of the server.
2. No 2 AsyncAPI's should document the same namespace
3. Each AsyncAPI file should be independent of the rest and capable of spinning up a stand-alone and functional Socket.IO server.
4. All AsyncAPI files would need to define the same server object (as there is a single path per server)

What do you think? Would the above proposal fit your product requirements? If yes, happy to think/talk about design and implementation.

[alex-zywicki]

Hi @dedoussis,

I have questions in regards to item 4.

From what I can gather in the current implementation, the server object in the spec is only used if you specify the server_name keyword argument of the from_spec function. It seems to me that item 4 would only need to hold true if you were actually passing that argument. Does that sound correct?

I agree completely with items, 1 and 3, but wonder if item 2 is strictly necessary in that a separate API file could be used to extend a namespace with new (non conflicting) messages. Since the message handler bindings are provided on a per message level I don't see how that would conflict. (But this is not a requirement on my end, only a theoretical use case)

Are you thinking that each yaml file would be loaded as a separate AsyncApiSpec instance and mounted on the same server?

I'm, also wondering if allowing the ref's to reference an external yaml partial would make sense as a means to facilitate writing more modular overall API's. That way if 2 namespaces wanted to reference a common message they could, or if all of your yaml files could references a servers.yaml partial that would allow for easy de-duplication of your server objects. (Once again, not strictly needed in my use case, but could be useful to me). This is not to say that facilitating standalone yaml files is not important, but rather to say that the ability to distribute your spec may be desired in some use cases.

[dedoussis]

@alex-zywicki

From what I can gather in the current implementation, the server object in the spec is only used if you specify the server_name keyword argument of the from_spec function. It seems to me that item 4 would only need to hold true if you were actually passing that argument. Does that sound correct?

This is correct, this problem arises only when the server_name argument is passed. Let's say you have multiple API files, each having its own unique set of servers. How would Asynction know which file to use in order to pick servers from? We could in theory merge all servers objects into one global servers object, but then we need to account for conflicts etc. I think merging of spec files is a recipe for disaster and I'm keen to avoid it.

wonder if item 2 is strictly necessary in that a separate API file could be used to extend a namespace with new (non conflicting) messages. Since the message handler bindings are provided on a per message level I don't see how that would conflict. (But this is not a requirement on my end, only a theoretical use case)

How are we going to handle the namespace handlers though, such as connect, disconnect and error (see x-handlers)? Consider the scenario of having a namespace that includes some authentication logic within the connection handler. Having another API that extends this namespace, means that one would need to re-define a connection handler for a namespace that already has one, leading to a conflict error being raised. The purpose of the Namespace concept in Socket.IO is to separate the concerns of an application. I feel that it's safe to stay at this level of separation rather attempting to go a level deeper and try to separate the concerns within a namespace.

Are you thinking that each yaml file would be loaded as a separate AsyncApiSpec instance and mounted on the same server?

Correct. Each YAML file should hold a valid and complete AsyncAPI specification, that can be deserialised into an AsyncApiSpec instance. An array of AsyncApiSpec instances will be stored as an attribute of the SocketIO instance. For each of the AsyncApiSpec instance of the array, the _register_handlers method is going to be called.

I'm, also wondering if allowing the ref's to reference an external yaml partial would make sense as a means to facilitate writing more modular overall API's. That way if 2 namespaces wanted to reference a common message they could, or if all of your yaml files could references a servers.yaml partial that would allow for easy de-duplication of your server objects. (Once again, not strictly needed in my use case, but could be useful to me). This is not to say that facilitating standalone yaml files is not important, but rather to say that the ability to distribute your spec may be desired in some use cases.

This will lead to incomplete spec files. I feel that we should stick to the principle of each API being complete and able to spin up a stand alone Socket.IO server by itself (point 3). Also, AFAIK connexion does not support such a feature. However, I do believe that Asynction should support modularity, but probably would be better to do it via Jinja2 rendering: https://github.com/zalando/connexion#dynamic-rendering-of-your-specification. See this comment from the relevant Connexion issue

[alex-zywicki]

@dedoussis Since the issue with the servers only arises when the server_name field is passed I think your original requirement of matching servers would be fine. I most likely wont be passing the server_name argument anyways.

With the additional factors regarding namespaces, I think your original point of having each file document a separate namespace makes sense.

As for the YAML file rendering/linkage, I think that for the sake of stating on topic on this issue that I will drop that topic for now and open another issue later if I need the feature.

All of that said, I believe that means your original summary stands.

Is this something you would be able to add? Or would you prefer a PR be provided?

[dedoussis]

@alex-zywicki Unfortunately, I won't be able to pick this up until late November. If that's bit late for you, a PR would be more than welcome!