Support sending and receiving MSC4354 Sticky Event metadata. by reivilibre · Pull Request #19365 · element-hq/synapse

reivilibre · 2026-01-09T16:36:23Z

Part of: MSC4354 whose experimental feature tracking issue is #19409

Follows: #19340 (a necessary bugfix for /event/ to set this metadata)

Partially supersedes: #18968

This PR implements the first batch of work to support MSC4354 Sticky Events.

Sticky events are events that have been configured with a finite 'stickiness' duration,
capped to 1 hour per current MSC draft.

Whilst an event is sticky, we provide stronger delivery guarantees for the event, both to
our clients and to remote homeservers, essentially making it reliable delivery as long as we
have a functional connection to the client/server and until the stickiness expires.

This PR merely supports creating sticky events and receiving the sticky TTL metadata in clients.
It is not suitable for trialling sticky events since none of the other semantics are implemented.

The current plan is to follow this PR up with more PRs, roughly parcelled up as follows:

Implement the sliding sync extension specified in MSC4354, to proactively notify
Implement the oldschool sync support, specified in MSC4354, to do the same but for clients still using oldschool sync.
Notice new servers joining rooms and proactively tell them about sticky events.
Add special federation catch-up support for sticky events, so that (as long as we have a connection) they don't get 'dropped' in the gaps between federation /send requests.
Re-evaluate soft-failed sticky events when we think that might be possible

Add MSC4354 experimental feature flag
Expose MSC4354 enablement on /versions
Add constants for sticky events
Add sticky_events table
Add sticky events store and stream
EventBase: add the concept of sticky_duration
EventBuilder: allow building events with sticky event fields
store method: insert_sticky_events_txn
When persisting currently-sticky events, add to sticky event stream
Allow clients to send sticky events
Including delayed events
Add test helper for sending sticky events
Expose the sticky event TTL to clients
Add a test for sticky TTL calculation and exposure to clients

Including delayed events

Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>

synapse/api/constants.py

MadLittleMods · 2026-01-22T16:28:03Z

synapse/api/constants.py

+
+class StickyEvent:
+    QUERY_PARAM_NAME: Final = "org.matrix.msc4354.sticky_duration_ms"
+    FIELD_NAME: Final = "msc4354_sticky"


The separation between StickyEventField and StickyEvent.FIELD_NAME is slightly mind bending.

synapse/storage/databases/main/events.py

synapse/handlers/delayed_events.py

tests/rest/client/test_sticky_events.py

synapse/storage/databases/main/delayed_events.py

tests/rest/client/utils.py

Co-authored-by: Eric Eastwood <erice@element.io>

synapse/visibility.py

synapse/storage/databases/main/sticky_events.py

synapse/storage/databases/main/events_worker.py

MadLittleMods · 2026-01-28T03:02:34Z

synapse/storage/databases/main/sticky_events.py

+            expr_soft_failed = "COALESCE(((ej.internal_metadata::jsonb)->>'soft_failed')::boolean, FALSE)"
+        else:
+            expr_soft_failed = "COALESCE(ej.internal_metadata->>'soft_failed', FALSE)"


Ugh! Future stuff but we should consider making soft_failed a dedicated column of the events table or some table. Is this performant enough?

I can't say I'm entirely happy with it, but it feels like it shouldn't be a very common case and the table is altogether small.

I don't think it is super expensive to fish out of the small internal_metadata JSON field in the event_json table (other than doing that heap lookup).
(It would be nicer if we didn't have a freeform internal_metadata and instead had nullable fields IMO, but I'm not sure that would greatly improve much.)

At this point I think it's better than risking stale denormalised data.

tests/storage/test_sticky_events.py

synapse/storage/schema/main/delta/93/01_sticky_events.sql

synapse/storage/databases/main/sticky_events.py

tests/rest/client/test_sticky_events.py

synapse/storage/databases/main/delayed_events.py

Co-authored-by: Eric Eastwood <erice@element.io>

MadLittleMods · 2026-02-03T16:35:29Z

synapse/storage/databases/main/sticky_events.py

+            expr_soft_failed = "COALESCE(((ej.internal_metadata::jsonb)->>'soft_failed')::boolean, FALSE)"
+        else:
+            expr_soft_failed = "COALESCE(ej.internal_metadata->>'soft_failed', FALSE)"


trial-olddeps -> sqlite3.OperationalError: near ">>": syntax error

trial-olddeps CI failing ❌, https://github.com/element-hq/synapse/actions/runs/21635340214/job/62359543300?pr=19365#step:10:7282

tests.storage.test_sticky_events.StickyEventsTestCase.test_get_updated_sticky_events =============================================================================== [FAIL] Traceback (most recent call last): File "/home/runner/work/synapse/synapse/tests/storage/test_sticky_events.py", line 153, in test_get_updated_sticky_events_with_limit updates = self.get_success( File "/home/runner/work/synapse/synapse/tests/unittest.py", line 707, in get_success return self.successResultOf(deferred) File "/opt/hostedtoolcache/Python/3.10.19/x64/lib/python3.10/site-packages/twisted/trial/_synctest.py", line 706, in successResultOf self.fail( twisted.trial.unittest.FailTest: Success result expected on <Deferred at 0x7f36aca5efe0 current result: None>, found failure result instead: Traceback (most recent call last): File "/opt/hostedtoolcache/Python/3.10.19/x64/lib/python3.10/site-packages/twisted/internet/defer.py", line 517, in errback self._startRunCallbacks(fail) File "/opt/hostedtoolcache/Python/3.10.19/x64/lib/python3.10/site-packages/twisted/internet/defer.py", line 580, in _startRunCallbacks self._runCallbacks() File "/opt/hostedtoolcache/Python/3.10.19/x64/lib/python3.10/site-packages/twisted/internet/defer.py", line 662, in _runCallbacks current.result = callback(current.result, *args, **kw) File "/opt/hostedtoolcache/Python/3.10.19/x64/lib/python3.10/site-packages/twisted/internet/defer.py", line 1514, in gotResult current_context.run(_inlineCallbacks, r, g, status) --- <exception caught here> --- File "/opt/hostedtoolcache/Python/3.10.19/x64/lib/python3.10/site-packages/twisted/internet/defer.py", line 1443, in _inlineCallbacks result = current_context.run(result.throwExceptionIntoGenerator, g) File "/opt/hostedtoolcache/Python/3.10.19/x64/lib/python3.10/site-packages/twisted/python/failure.py", line 500, in throwExceptionIntoGenerator return g.throw(self.type, self.value, self.tb) File "/opt/hostedtoolcache/Python/3.10.19/x64/lib/python3.10/site-packages/synapse/storage/databases/main/sticky_events.py", line 144, in get_updated_sticky_events return await self.db_pool.runInteraction( File "/opt/hostedtoolcache/Python/3.10.19/x64/lib/python3.10/site-packages/synapse/storage/database.py", line 1015, in runInteraction return await delay_cancellation(_runInteraction()) File "/opt/hostedtoolcache/Python/3.10.19/x64/lib/python3.10/site-packages/twisted/internet/defer.py", line 1443, in _inlineCallbacks result = current_context.run(result.throwExceptionIntoGenerator, g) File "/opt/hostedtoolcache/Python/3.10.19/x64/lib/python3.10/site-packages/twisted/python/failure.py", line 500, in throwExceptionIntoGenerator return g.throw(self.type, self.value, self.tb) File "/opt/hostedtoolcache/Python/3.10.19/x64/lib/python3.10/site-packages/synapse/storage/database.py", line 981, in _runInteraction result: R = await self.runWithConnection( File "/opt/hostedtoolcache/Python/3.10.19/x64/lib/python3.10/site-packages/synapse/storage/database.py", line 1117, in runWithConnection return await make_deferred_yieldable( File "/opt/hostedtoolcache/Python/3.10.19/x64/lib/python3.10/site-packages/twisted/internet/defer.py", line 662, in _runCallbacks current.result = callback(current.result, *args, **kw) File "/home/runner/work/synapse/synapse/tests/server.py", line 814, in <lambda> d.addCallback(lambda x: function(*args, **kwargs)) File "/opt/hostedtoolcache/Python/3.10.19/x64/lib/python3.10/site-packages/twisted/enterprise/adbapi.py", line 293, in _runWithConnection compat.reraise(excValue, excTraceback) File "/opt/hostedtoolcache/Python/3.10.19/x64/lib/python3.10/site-packages/twisted/python/deprecate.py", line 298, in deprecatedFunction return function(*args, **kwargs) File "/opt/hostedtoolcache/Python/3.10.19/x64/lib/python3.10/site-packages/twisted/python/compat.py", line 403, in reraise raise exception.with_traceback(traceback) File "/opt/hostedtoolcache/Python/3.10.19/x64/lib/python3.10/site-packages/twisted/enterprise/adbapi.py", line 284, in _runWithConnection result = func(conn, *args, **kw) File "/opt/hostedtoolcache/Python/3.10.19/x64/lib/python3.10/site-packages/synapse/storage/database.py", line 1110, in inner_func return func(db_conn, *args, **kwargs) File "/opt/hostedtoolcache/Python/3.10.19/x64/lib/python3.10/site-packages/synapse/storage/database.py", line 819, in new_transaction r = func(cursor, *args, **kwargs) File "/opt/hostedtoolcache/Python/3.10.19/x64/lib/python3.10/site-packages/synapse/storage/databases/main/sticky_events.py", line 160, in _get_updated_sticky_events_txn txn.execute( File "/opt/hostedtoolcache/Python/3.10.19/x64/lib/python3.10/site-packages/synapse/storage/database.py", line 458, in execute self._do_execute(self.txn.execute, sql, parameters) File "/opt/hostedtoolcache/Python/3.10.19/x64/lib/python3.10/site-packages/synapse/storage/database.py", line 520, in _do_execute return func(sql, *args, **kwargs) sqlite3.OperationalError: near ">>": syntax error

Related discussion in #synapse-dev:matrix.org

MadLittleMods · 2026-02-03T16:44:22Z

synapse/storage/databases/main/events.py

+                    # The de-outliered event is sticky. Update the sticky events table to ensure
+                    # we deliver this down /sync.
+                    self.store.insert_sticky_events_txn(txn, [event])


Test for this scenario?

synapse/storage/databases/main/sticky_events.py

synapse/storage/schema/main/delta/93/01_sticky_events.sql

tests/rest/client/utils.py

tests/rest/client/test_sticky_events.py

tests/storage/test_sticky_events.py

MadLittleMods · 2026-02-03T16:56:27Z

synapse/events/builder.py


    content: JsonDict = attr.Factory(dict)
    unsigned: JsonDict = attr.Factory(dict)
+    sticky: StickyEventField | None = None


Probably fine as-is. We don't have any great options ⏩

reivilibre requested a review from a team as a code owner January 9, 2026 16:36

reivilibre changed the title ~~Support sending and receiving [MSC4354 Sticky Event](https://github.com/matrix-org/matrix-spec-proposals/pull/4354) metadata.~~ Support sending and receiving MSC4354 Sticky Event metadata. Jan 9, 2026

reivilibre force-pushed the rei/sticky_events1 branch from 2b83332 to 53d4486 Compare January 9, 2026 16:43

reivilibre added 3 commits January 12, 2026 12:09

Add MSC4354 experimental feature flag

aaa229f

Expose MSC4354 enablement on /versions

a25a5b5

Add constants for sticky events

7abcf6d

reivilibre force-pushed the rei/sticky_events1 branch 2 times, most recently from e49d4bd to 78ee6e8 Compare January 16, 2026 09:00

reivilibre added 11 commits January 16, 2026 09:07

Add sticky_events table

c2d4cf4

Add sticky events store and stream

ec5f2f8

EventBase: add the concept of sticky_duration

003fe1f

EventBuilder: allow building events with sticky event fields

c7075d6

store method: insert_sticky_events_txn

0a4f7e6

When persisting currently-sticky events, add to sticky event stream

53e4968

Allow clients to send sticky events

137c457

Including delayed events

Add test helper for sending sticky events

90e9539

Expose the sticky event TTL to clients

9a0b28e

Add a test for sticky TTL calculation and exposure to clients

b41ef41

Newsfile

fb67e9a

Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>

reivilibre force-pushed the rei/sticky_events1 branch from 78ee6e8 to fb67e9a Compare January 16, 2026 09:07

MadLittleMods reviewed Jan 22, 2026

View reviewed changes

reivilibre and others added 3 commits January 22, 2026 20:19

Use Duration

383d97c

Tweak comment on SQL column

47306d2

Update synapse/config/workers.py

beead8a

Co-authored-by: Eric Eastwood <erice@element.io>

reivilibre force-pushed the rei/sticky_events1 branch from 1c2a9ad to beead8a Compare January 22, 2026 20:20

reivilibre added 5 commits January 26, 2026 12:51

Comments on constants

1353ec3

FIELD_NAME -> EVENT_FIELD_NAME

a424310

Docstring explain stream writers for events and sticky_events

2522c38

Make explanation not double negative

450d5ee

Comment why 1 hour

34d23ef

Introduce StickyEventUpdate dataclass

33f953e

reivilibre mentioned this pull request Jan 26, 2026

MSC4354: Sticky Events matrix-org/matrix-spec-proposals#4354

Open

4 tasks

reivilibre added 9 commits January 26, 2026 13:25

Remove stale log

5f8f302

Docstring on delete_expired_sticky_events

015819e

Use StickyEventField

3c51231

Remove un-soft-failing related handling

11378ac

Add tracking issue

db9504f

Don't denormalise soft_failed as it could become mutable

ae5f1c5

Add tests for the sticky events store methods

6527373

Describe what types of events do and don't live in sticky_events table

d80d2c4

Note that events persisted before enablement won't be considered sticky

e3a8f19

reivilibre requested a review from MadLittleMods January 27, 2026 16:42

MadLittleMods approved these changes Jan 28, 2026

View reviewed changes

reivilibre and others added 4 commits January 29, 2026 14:16

Update synapse/storage/schema/main/delta/93/01_sticky_events.sql

f42af6a

Co-authored-by: Eric Eastwood <erice@element.io>

tests: just get the stream IDs between insertions

3be0e87

tests: Read raw table for checking expired sticky events are deleted

efc08d2

Comment why we min(origin_server_ts, now_ms)

5a5efc5

reivilibre force-pushed the rei/sticky_events1 branch from 89b20d3 to 5a5efc5 Compare February 3, 2026 13:42

reivilibre added 4 commits February 3, 2026 13:51

Explain skip rationale

d83656a

Expand comment on staggering

cf16293

bool subclass of int clarification

b0693b5

Add a test for sticky events not being sticky when feature disabled

c0c78dc

reivilibre requested a review from MadLittleMods February 3, 2026 13:59

reivilibre mentioned this pull request Feb 3, 2026

MSC4354: Sticky Events #18968

Closed

3 tasks

Apparently a ? in a SQL comment causes the migration to fail :D

2b822dd

MadLittleMods approved these changes Feb 3, 2026

View reviewed changes

reivilibre added 4 commits February 4, 2026 11:14

Suggested doc tweaks

601b1f3

Prevent starting up with feature when SQLite is too old

02d245f

Skip the test on old SQLite

737d482

Add test for de-outliering events

a535477

Conversation

reivilibre commented Jan 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

MadLittleMods Jan 22, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

MadLittleMods Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

reivilibre Jan 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

MadLittleMods Feb 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

trial-olddeps -> sqlite3.OperationalError: near ">>": syntax error

Uh oh!

MadLittleMods Feb 3, 2026

Choose a reason for hiding this comment

Uh oh!

MadLittleMods Feb 3, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

MadLittleMods Feb 3, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

reivilibre commented Jan 9, 2026 •

edited

Loading

reivilibre Jan 29, 2026 •

edited

Loading

MadLittleMods Feb 3, 2026 •

edited

Loading

`trial-olddeps` -> `sqlite3.OperationalError: near ">>": syntax error`