Skip to content

Refactor distributed example to handle independent horizontal scaling #3

@amoeba

Description

@amoeba

While the main goal of the distributed example is to show off OpenTelemetry and how it applies to Flight, another goal is to show how to split the Flight server into two types of servers so each can be independently horizontally scaled to handle requests from many clients similar to what you could expect in a production Flight service. I think we want to show a fairly realistic example rather than something hacked together that requires lots caveats. Right we have the latter because:

  1. You can't horizontally scale the coordinator: When a data server starts up, it registers its flights with a single coordinator. It can't talk to two or more coordinators. The state of the world needs to live outside the coordinator server.
  2. If you only horizontally scaled the data server, the coordinator would break because I used an ordered map on the coordinator to map FlightInfo descriptors to Flights.

I think a good simplification would be to say that the state of the world is just what's on the filesystem and then to give the coordinator and data servers an identical view of it. And then you're left with the challenge of load balancing across replicas which I think could be handled one layer up the stack.

Metadata

Metadata

Assignees

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions