Feat/new projecting system dx

[dgafka]

Why is this change proposed?

This is next step into making new projecting system open to wide public.
New projecting system provides much more advanced features, and we need to come with Developer Experience, that will make it intent driven and easy to follow. Therefore this PR, introduces high level usage for new projecting system, that is based on declarative configuration using attributes.

Global tracking Projection (default)

#[ProjectionV2]
class CountTicketsProjections

Partitioned tracking Projection

#[Partitioned]
#[ProjectionV2]
class TicketsProjections

Polling Projection

#[Polling("tickets_projection")]
#[ProjectionV2]
class TicketsProjections

Streaming Projection

#[Streaming("streaming_channel")]
#[ProjectionV2]
class TicketsProjections

Pull Request Contribution Terms

• I have read and agree to the contribution terms outlined in CONTRIBUTING.

[dgafka]

That's for streaming channels, I may think how to extract that. For now it works as configurable option when message channel is passed here

[dgafka]

Main split Global vs PartitionedProjection. Users will be able to switch from old Projection to this attribute (as underyling projection is the same)

[dgafka]

@jlabedo looks good? :)

[dgafka]

@jlabedo I will also cover with tests that make sense partitioned project (current set of tests focus on globally tracked projection)

[jlabedo]

Here are my thoughts on the API: GlobalProjection vs PartitionedProjection makes partitioning the axis of the API.

1. It elevates an implementation detail

• Partitioning is a scaling / concurrency / correctness concern.
• The semantic concern is: what scope does this projection operate on?
• Making “partitioned” first-class in the name leaks infrastructure into the mental model.
• “Global” sounds like the default / normal case.
• “Partitioned” sounds like an advanced or optional mode.

2. It locks the API

• Today: global vs partitioned.
• Tomorrow: sharded, key-range, hash, tenant, aggregate, custom router?
• The API is now named after one specific strategy.

3. Could misguide users

• Users will ask: “Should I use Global or Partitioned?”
• That’s the wrong question.
• The real question is: “What is the projection’s consistency and isolation boundary?”.

Aternatives

- Isolation concept:

#[Projection(isolation: Isolation::SHARED)]
#[Projection(isolation: Isolation::PER_AGGREGATE)]
#[Projection(isolation: Isolation::PER_TENANT)]
#[Projection(isolation: Isolation::NONE)]

- Ordering requirement:

#[Projection(ordering: Ordering::GLOBAL)]
#[Projection(ordering: Ordering::PER_STREAM)]
#[Projection(ordering: Ordering::PER_AGGREGATE)]
#[Projection(ordering: Ordering::PER_TENANT)]
#[Projection(ordering: static function (array $headers): string {
  // requires php 8.5
  return $headers['a-custom-domain-key-where-ordering-should-be-preserved']; 
})]

Key thought : Partitioning is a capability, not a type

With these alternative apis, you state your absolute domain requirement, not configuring scalability. We can imagine that a future smarter system could decide which partitioning strategy to use based on isolation requirement cardinality (for instance, having a fixed amount of n partitions and computing a hash to give a partition to each event), or even dynamically (I think Axon is doing that)

Change naming vs old system

I agree that it would be better to change the attribute name vs the previous one, so the user better understands and to avoid wrong imports that would be hard to debug.
Here are some proposals:

• #[AdvancedProjection]: weird but at least honest that it is just another system
• #[ReadModelProjection]
• #[StateProjection]

[jlabedo]

Useless ai

[dgafka]

That will actually one of the difference between v1 and v2. Because with gap detection based on time in v1, we could do the events filters (even so this gap stategy not always worked :P).
But right now for global tracking, we can't really do such event filters, as those would be considered as gaps.

[dgafka]

3. Could misguide users
   Users will ask: “Should I use Global or Partitioned?”
   That’s the wrong question.
   The real question is: “What is the projection’s consistency and isolation boundary?”.

Hmm, ye global or partitioned are from solution space, not problem space.
Even so I agree, my concern here is different.

1. Assuming that people have not much knowledge, if that will be simple config change within same attribute, then it may create feeling that Ecotone's internals will take care of the switch.
   In reality that's not what going to happen, each one has separate tracking system, therefore projection will be considered fresh (and will start from the beginning).

2. What about streaming projection, the tracking there is no concern of Ecotone, as it will be part of the Streaming platform itself (partitions on Kafka streaming channel). It will also require passing the streaming channel name to the Projection (you can take a look now on Partitioned attribute as I've not extracted separate StreamingProjection attribute).
   So having single attribute creates design that leads to confusion, because we are exposing configs that are not part of given Projection type strategy (autoinitialization is not needed for partitioned, streaming channel is only needed for streaming projections). So even so we can guard that and disallow, that just shows that the design is simply wrong and now we need to compensate to avoid wrong set ups (with friendly design, wrong set ups are not even possible).

[jlabedo]

1. Assuming that people have not much knowledge, if that will be simple config change within same attribute, then it may create feeling that Ecotone's internals will take care of the switch.
   In reality that's not what going to happen, each one has separate tracking system, therefore projection will be considered fresh (and will start from the beginning).

That’s a strong argument, and I agree with it.
That said, we already have a similar situation today: changing partitionHeaderName in PartitionedProjection is also a semantic change that should reset the projection state. In that sense, some configuration changes are already lifecycle-breaking and require a full rebuild.

Conceptually, I see GlobalProjection as equivalent to a PartitionedProjection with a single, constant partition (for example, using the stream name as the partition key). From that perspective, they are two variants of the same execution model rather than fundamentally different projection types.

autoinitialization is not needed for partitioned

For aggregate-id–based partitioning, I agree that autoinitialization is mostly useless.
However, if the partition key is something like a tenant identifier, autoinitialization can still make sense.

2. What about streaming projection, the tracking there is no concern of Ecotone

I haven’t dug deeply into the streaming projection PR yet, but for that case I agree that a completely separate attribute makes sense. The execution and tracking responsibilities are clearly different there, and modeling it as a distinct projection type feels appropriate.

Don’t get me wrong, I fully agree that the current API is not in a good enough shape and should be improved before any public release — I just don’t have a clear alternative proposal yet.

[dgafka]

@jlabedo I've combined the approaches, so we do have one Projection attribute, however separate attributes for additional behaviours to be enabled/changed.
So it would be:

ProjectionV2 - Unified projection attribute (replaces both GlobalProjection and PartitionedProjection). In Ecotone 2.0 we will make Projection
Polling - Configures polling mode with endpointId parameter
Streaming - Marks projection as event-streaming based

Global tracking Projection (default)

#[ProjectionV2]
class CountTicketsProjections

Partitioned tracking Projection

#[Partitioned]
#[ProjectionV2]
class TicketsProjections

Polling Projection

#[Polling]
#[ProjectionV2]
class TicketsProjections

Streaming Projection

#[Streaming]
#[ProjectionV2]
class TicketsProjections

Autoinitialization

This is for now, I do think we will need to change this to something a bit different to ensure having "offline" partitioned projections. This is because right now, there is no way to "rebuild" with re-emitting events from projection

ProjectionConfiguration(automaticInitialization: true)
#[ProjectionV2]
class TicketsProjections

[jlabedo]

Seems very strong api 👌

[dgafka]

@jlabedo this also means that we won't be in possibility to use multiple Projection attributes on same class, but we should still be able to extend the class and do new attribute there

[jlabedo]

this also means that we won't be in possibility to use multiple Projection attributes on same class, but we should still be able to extend the class and do new attribute there

I am not sure allowing multiple attributes is a good idea ultimately: using a trait or extending a class is powerful enough.
With multiple attributes, you cannot declare one asynchronous and another synchronous projection without changing the current api.

[dgafka]

@jlabedo ye, I do not consider that as blocker anyhow. Just mentioning that this path, basically makes this impossible to do in future.