Setup self-hosted runners

[BigLep]

We want to validate that foc-localnet continues to work via CI. We have been finding that github action runners aren't powerful enough. We want to get self-hosted runners going like we have with lotus in filecoin-project. This is a tracking issue for that work.

[BigLep]

@galargh : can you please help with this or point us on how to best do it while following the same setup we have with filecoin-project? (I assume there are some one-time costs for this because this project is in a different GitHub organization.)

[redpanda-f]

Unsure on what is missing here. There are OpenSSL troubles with self hosted runners? This does not seem to happen with Github Runners. See comparison runs here:

• GH Runner: https://github.com/FilOzone/foc-localnet/actions/runs/20809269255/job/60068261537?pr=16
• Self Hosted Runner: https://github.com/FilOzone/foc-localnet/actions/runs/20909505707/job/60069635925

cc @galargh . Is this a known issue?

[galargh]

  Could not find directory of OpenSSL installation, and this `-sys` crate cannot
  proceed without this knowledge. If OpenSSL is installed and this crate had
  trouble finding it,  you can set the `OPENSSL_DIR` environment variable for the
  compilation process.

  Make sure you also have the development packages of openssl installed.
  For example, `libssl-dev` on Ubuntu or `openssl-devel` on Fedora.

  If you're in a situation where you think the directory *should* be found
  automatically, please open a bug at https://github.com/rust-openssl/rust-openssl
  and include information about your system as well as this message.

  $HOST = x86_64-unknown-linux-gnu
  $TARGET = x86_64-unknown-linux-gnu
  openssl-sys = 0.9.111


  It looks like you're compiling on Linux and also targeting Linux. Currently this
  requires the `pkg-config` utility to find OpenSSL but unfortunately `pkg-config`
  could not be found. If you have OpenSSL installed you can likely fix this by
  installing `pkg-config`

Looks like you ned to install missing packages; similarly to https://github.com/filecoin-project/lotus/blob/df9e10ef0af35613c65178a5740fde37c03efc8e/.github/actions/install-system-dependencies/action.yml#L12

Our self-hosted runners are not guaranteed to have the same set of packages as hosted runners installed. Some we could definitely consider installing by default but it shouldn't be too much of an overhead for now to install it workflow-side, at least for now.

[redpanda-f]

Why foc-localnet PDP Tasks Keep Getting Failures in CI (Need more cores)

Hi Team, over last 2 to 3 weeks, I have been actively investigating why our CI was failing while the same configuration worked locally for me (and was replicated why @rvagg on his machine). First, it was github runners. Then, yesterday we shifted to self-hosted runner but CI still did not pass.

TL;DR

Our CI runners don't have enough CPU cores, even the self-hosted ones. PDP needs at least 6 cores to run properly, we probably have less. Curio literally denies starting services in constrained settings leading to PDP failures on CI.

What's Going On?

When you enable PDP in Curio, it spins up 7 different background task workers. Each one needs CPU resources, and Curio's harmonytask engine is pretty strict about not overcommitting - if there aren't enough CPUs available, tasks just sit there waiting.

The engine does simple math: Available CPUs = Total CPUs - CPUs already in use

When a task wants to run but there's no CPU left, we see errors like, which we already have seen before in CI (see attached image):

Did not accept PDPProve task: out of cpu: required 1 available 0

The 7 PDP Task Workers

When PDP kicks in, here's what spawns (feel free to correct me):

What It Does	CPU Needed	Notes
PDPProve	1 core	Generates the actual proofs - super important
PDPNotify	1 core	Sends notifications when uploads complete
PDPIndexing	1 core	Builds piece indexes
PDPIpni	1 core	Publishes to IPNI network
TerminateFWSS	1 core	Cleanup duty
DeleteDataSet	1 core	More cleanup
PDPProvingPeriod	0 cores	Lightweight period management
PDPInitPP	0 cores	Lightweight init stuff

So basically 6 workers need 1 CPU each, and 2 are lightweight enough to not need dedicated cores.

How Many CPUs Do We Actually Need?

Minimum to not break: 6 cores
You can technically run with this, but it'll be tight.

What you probably want: 8 cores
Gives some breathing room and lets things run concurrently without fighting over resources.

Production setup: 12+ cores
Especially if you're running other Curio stuff (sealing, WindowPoSt, etc.) alongside PDP.

The Fix

For CI: Make sure our self-hosted runners have at least 6-8 CPU cores. Right now they're configured in .github/workflows/ci.yml as [self-hosted, docker] runners and they present 2 CPU cores, 4 logical cores. This cannot really do CI at the moment. So yes, the ask is for "big" machines here. This is confirmed to work on large AWS machine, c5a.4xlarge (I set one up for testing today, and later stripped it down): https://github.com/FilOzone/foc-localnet/actions/runs/20945528772/job/60188435501#step:30:431 (Synapse E2E fails due to tangential issue of iso-ledger troubles)

For local testing: If you're testing PDP locally, make sure your dev machine should have enough cores already (I hope).

[BigLep]

@redpanda-f : can you confirm you are unblocked now? It looks like we're good per https://filecoinproject.slack.com/archives/C06CY12V83S/p1768325424281709