qa: Run QA directly from GitHub

[romac]

Description

Currently running QA experiments requires local setup and privileged access to DigitalOcean. To improve the QA process and make it easier to test the impact of code changes on performance and stability, we should add the ability to trigger QA runs directly from GitHub, either through workflow triggers or PR comments.

Proposed Solution

Add GitHub Actions workflow(s) that:

1. Can be triggered either:

   • Manually via a workflow_dispatch event (ie. a manual workflow trigger)
   • By commenting /run-qa on a PR
   • Those should be only available to some privileged users to prevent abuse

2. Runs the QA experiments on DigitalOcean

   • Create the necessary machines
   • Run QA
   • Tear down the machines
   • Alternatively, we could only allow a single QA run at a time and re-use the existing machines

3. Uploads results as GitHub artifacts

   • Store metrics (Prometheus DB), logs, and other relevant outputs
   • Generate a comment on the PR with:
     • Summary of key metrics
     • Link to download full artifacts
     • Any failed checks or warnings

Benefits

• Faster feedback loop for changes
• No need for local setup
• Consistent testing environment
• Easy access to results for reviewers
• Historical record of experiment results

Acceptance Criteria

• QA runs can be triggered from GitHub UI
• PR comments can trigger runs
• Results are automatically uploaded as artifacts
• Clear feedback provided in PR comments
• Proper error handling and reporting

Questions

• How long should we retain artifacts?

[adizere]

Two questions:

1. CSV

I agreed with Aaron we will provide to him the raw CSV data from each measurement. I think this is not too difficult but wanted to hear your thoughts.

To recall, we want two measurements: Latency and Throughput. Each measurement will have data points at every height (or timestamp, up to you) and every validator. So each CSV would have N series of data points -- 1 series per validator --, where each series is:

• for latency - a tuple X/Y s.t. X = latency in seconds, Y = data timestamp or height
• for throughput - a tuple X/Y s.t. X = tps as a number eg "40", Y = data timestamp or height

Aaron will create the plot to make it appropriate for website use -- see attached pictures for latency and throughput respectively.

2. History

Can we keep in the Malachite codebase the history of all CSVs we generate with each CI run?

[nenadmilosevic95]

For 1) I think that getting .csv should not be the problem. I guess we should query the Prometheus directly to get this data.

With respect to metrics, I think Prometheus is storing time stamps and not heights, so Y should be time stamp and not height.
Moreover, for latency we can use the consensus latency metric and we can generate a graph like the one above.
However, for throughput is a bit trickier, I am not sure what is the metric that is shown on this plot?

For 2) I guess this should not be the problem, but maybe we can store it elsewhere, for example on S3 bucket.

[adizere]

With respect to metrics, I think Prometheus is storing time stamps and not heights, so Y should be time stamp and not height.
Moreover, for latency we can use the consensus latency metric and we can generate a graph like the one above.

Perfect.

However, for throughput is a bit trickier, I am not sure what is the metric that is shown on this plot?

Good point! The original data is actually from CometBFT QA - here. I wonder if we can get something similar, ideally # of transactions per second.

For 2) I guess this should not be the problem, but maybe we can store it elsewhere, for example on S3 bucket.

Anything works for me as long as we can browse (or download) the historical data to inspect it. Pick whichever is easiest for you. For context, I would like to quote the history of performance measurements on the website, so that any reader can parse through it. The website will have a passage as follows:

History of performance measurements. The plots on this page are taken from weekly CI measurements. To see the history of prior measurements, see here [replace link to S3 or whatever repo we store data in].

[nenadmilosevic95]

In what format do we want to store this historical data, again .csv files?

This is the closest metric to the one above among those we currently have (Transactions per minute). Note here that it is different compared to the one above because we here have synthetic load, and basically the number of txs is defined by the block size that we configure when running the experiment, and it is not dictated by the load.

[adizere]

Can we do granularity per second (instead of per minute)?

it is different compared to the one above because we here have synthetic load, and basically the number of txs is defined by the block size that we configure when running the experiment, and it is not dictated by the load.

I think this is fine, because the system will be finalizing different amount of blocks/second, and therefore the # of transactions / second I'd expect to fluctuate slightly.

In what format do we want to store this historical data, again .csv files?

Yes!