Physical restores fail due to database logs

[sfoulkes]

Report

I'm running version 1.21.1 of the operator. I have not been able to do a physical restore of the database as the process fails to delete the log directory and leaves the cluster in a broken state. From the pbm agent logs:

2026-02-05T17:36:31.000+0000 D [restore/2026-02-05T17:35:44.083981073Z] remove /data/db/index-77-9466172986792581070.wt
2026-02-05T17:36:31.000+0000 D [restore/2026-02-05T17:35:44.083981073Z] remove /data/db/index-79-9466172986792581070.wt
2026-02-05T17:36:31.000+0000 D [restore/2026-02-05T17:35:44.083981073Z] remove /data/db/index-80-9466172986792581070.wt
2026-02-05T17:36:31.000+0000 D [restore/2026-02-05T17:35:44.083981073Z] remove /data/db/index-9-9466172986792581070.wt
2026-02-05T17:36:31.000+0000 D [restore/2026-02-05T17:35:44.083981073Z] remove /data/db/journal
2026-02-05T17:36:31.000+0000 D [restore/2026-02-05T17:35:44.083981073Z] remove /data/db/key.db
2026-02-05T17:36:31.000+0000 E [restore/2026-02-05T17:35:44.083981073Z] mark error during restore: remove db path: remove 'logs': unlinkat /data/db/logs: directory not empty

➜  ~ k describe psmdb-restore restore-d-us-omm-db-01--replset-20260205-113307
Name:         restore-d-us-omm-db-01--replset-20260205-113307
Namespace:    system-percona-mongodb
Labels:       <none>
Annotations:  <none>
API Version:  psmdb.percona.com/v1
Kind:         PerconaServerMongoDBRestore
Metadata:
  Creation Timestamp:  2026-02-05T17:33:07Z
  Generation:          1
  Resource Version:    874693213
  UID:                 9bbb4dbf-4783-4645-b062-db692deb15ca
Spec:
  Backup Name:   cron-d-us-omm-db-01---20260205173000-sgfbv
  Cluster Name:  d-us-omm-db-01--replset
Status:
  Error:     remove db path: remove 'logs': unlinkat /data/db/logs: directory not empty
  Pbm Name:  2026-02-05T17:35:44.083981073Z
  State:     error
Events:      <none>

More about the problem

I'd expect the backup agent would be able to delete the logs and do the restore.

Steps to reproduce

1. Attempt to restore a physical backup

Versions

1. Kubernetes - 1.31.11
2. Operator - 1.21.1
3. Database - 8.0.12-4

Anything else?

No response

[egegunes]

pbm-agent should be able to delete all directories under db path. I tried with 1.21.2 and couldn't reproduce this issue.