Hosting of Clickhouse for Aspects

[felipemontoya]

Following the interest there is in using Aspects I had a conversation with @Ian2012 who has worked on the aspects project for a while to understand what would be required to use Aspects in harmony projects.

Aspects builds on top of a series of different projects:

• superset
• clickhouse
• vector
• ralph
• dbt
• event routing backends

Now, in order to host an aspects install, the superset/vector/ralph/dbt and the lms with the event routing backends package are already well supported via a tutor plugin: https://github.com/openedx/tutor-contrib-aspects/.

The critical missing bit would be clickhouse. It is possible to install clickhouse inside of the k8s cluster by using the k8s template provided in the plugin. However so far we have found that clickhouse can be a heavy drain of resources and it caused instability at the complete cluster. I'm opening this ticket to track any work we do in supporting a more robust hosting of clickhouse as a workload or strategies for external hosting.

[bradenmacdonald]

Do superset and clickhouse support multi-tenancy?

[felipemontoya]

Clickhouse works as a database. You can have schemas and you can configure your superset/ralph/routing-backend to read/write to a particular schema.

Superset requires that you have at least one pod per namespace dedicated to superset.

You can configure therefore many instances of openedx, each in a namespace, each with its own superset pod. All of them using the same clickhouse installation.

I would not call that multi-tenancy in the way that I'm used to in the lms context, but multiple aspects installations are completely possible in a single k8s cluster sharing the clickhouse resource.

[Ian2012]

We plan to add the ClickHouse Operator helm chart as a dependency (disabled by default) with examples of CRD representing production-ready ClickHouse clusters compatible with Aspects.

[Ian2012]

Current WIP: #90