SIDB primary unhealthy and refuses to start properly after worker node poweroff/on

[andbos]

We had one SIDB primary, one SIDB physical standby and Dataguardbroker configured in our lab before we shut down all the Openstack VMs on which the Kubernetes 1.33 nodes were running. After poweron a couple of weeks later the SIDB pods refused to start. Both SIDB were unhealthy and refused to start properly. The primary threw error ORA-01589: must use RESETLOGS or NORESETLOGS option for database open.

We deleted dataguardbroker and also the physical standby SIDB instance. Then deleted the primary SIDB pods. It didn't help. Is it possible to rescue the instance somehow?

kubectl -n oracle-database delete -f dataguardbroker.yaml  --grace-period=0 --force
Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
dataguardbroker.database.oracle.com "cnfdb-dgbroker" force deleted

kubectl -n oracle-database get dataguardbroker
No resources found in oracle-database namespace.

kubectl -n oracle-database delete -f singleinstancedatabase_standby.yaml

kubectl -n oracle-database get dataguardbroker
No resources found in oracle-database namespace.

kubectl -n oracle-database get singleinstancedatabase
NAME      EDITION      STATUS      ROLE      VERSION       CONNECT STR                    TCPS CONNECT STR   OEM EXPRESS URL
cnfdb11   Enterprise   Unhealthy   PRIMARY   19.28.0.0.0   10.62.207.172:1521/NCNFDB11   Unavailable        https://10.62.207.172:5500/em

kubectl  -n oracle-database get pods
NAME            READY   STATUS    RESTARTS      AGE
cnfdb11-v3d5o   0/1     Running   2 (21h ago)   44d

kubectl -n oracle-database delete pod cnfdb11-v3d5o
pod "cnfdb11-v3d5o" deleted

kubectl -n oracle-database get pods
NAME            READY   STATUS    RESTARTS   AGE
cnfdb11-ufvob   0/1     Running   0          5s

$ kubectl --kubeconfig -n oracle-database logs -f cnfdb11-ufvob
Defaulted container "cnfdb11" out of: cnfdb11, init-permissions (init), init-wallet (init)
[2026:01:09 07:33:50]: Acquiring lock .NCNFDB11.create_lck with heartbeat 30 secs
[2026:01:09 07:33:50]: Lock acquired
[2026:01:09 07:33:50]: Starting heartbeat
[2026:01:09 07:33:50]: Lock held .NCNFDB11.create_lck
ORACLE EDITION: ENTERPRISE

SQL*Plus: Release 19.0.0.0.0 - Production on Fri Jan 9 07:33:50 2026
Version 19.29.0.0.0

Copyright (c) 1982, 2025, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> ORACLE instance shut down.
SQL> ORACLE instance started.

Total System Global Area 5033162264 bytes
Fixed Size                  9188888 bytes
Variable Size            2617245696 bytes
Database Buffers         2399141888 bytes
Redo Buffers                7585792 bytes
SQL> Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.29.0.0.0
[2026:01:09 07:34:12]: Releasing lock .NCNFDB11.create_lck
[2026:01:09 07:34:12]: Lock released .NCNFDB11.create_lck
[2026:01:09 07:34:12]: Acquiring lock .NCNFDB11.exist_lck with heartbeat 30 secs
[2026:01:09 07:34:18]: Lock acquired
[2026:01:09 07:34:18]: Starting heartbeat
[2026:01:09 07:34:18]: Lock held .NCNFDB11.exist_lck

LSNRCTL for Linux: Version 19.0.0.0.0 - Production on 09-JAN-2026 07:34:18

Copyright (c) 1991, 2025, Oracle.  All rights reserved.

Starting /opt/oracle/product/19c/dbhome_1/bin/tnslsnr: please wait...

TNSLSNR for Linux: Version 19.0.0.0.0 - Production
System parameter file is /opt/oracle/product/19c/dbhome_1/network/admin/listener.ora
Log messages written to /opt/oracle/diag/tnslsnr/cnfdb11-ufvob/listener/alert/log.xml
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1)))
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=0.0.0.0)(PORT=1521)))

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC1)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 19.0.0.0.0 - Production
Start Date                09-JAN-2026 07:34:19
Uptime                    0 days 0 hr. 0 min. 0 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /opt/oracle/product/19c/dbhome_1/network/admin/listener.ora
Listener Log File         /opt/oracle/diag/tnslsnr/cnfdb11-ufvob/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=0.0.0.0)(PORT=1521)))
Services Summary...
Service "DATAGUARD" has 1 instance(s).
  Instance "NCNFDB11", status UNKNOWN, has 1 handler(s) for this service...
Service "NCNFDB11" has 1 instance(s).
  Instance "NCNFDB11", status UNKNOWN, has 1 handler(s) for this service...
Service "NCNFDB11_DGMGRL" has 1 instance(s).
  Instance "NCNFDB11", status UNKNOWN, has 1 handler(s) for this service...
The command completed successfully

SQL*Plus: Release 19.0.0.0.0 - Production on Fri Jan 9 07:34:19 2026
Version 19.29.0.0.0

Copyright (c) 1982, 2025, Oracle.  All rights reserved.


Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.29.0.0.0

SQL> SQL>
Database altered.

SQL>    alter database open
*
ERROR at line 1:
ORA-01589: must use RESETLOGS or NORESETLOGS option for database open


SQL>
System altered.

SQL> Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.29.0.0.0
The Oracle base remains unchanged with value /opt/oracle
Waiting for 1 sec(s) before restarting Oracle processes and opening the database

SQL*Plus: Release 19.0.0.0.0 - Production on Fri Jan 9 07:34:34 2026
Version 19.29.0.0.0

Copyright (c) 1982, 2025, Oracle.  All rights reserved.


Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.29.0.0.0

SQL> SQL>    alter database mount
*
ERROR at line 1:
ORA-01100: database already mounted


SQL>    alter database open
*
ERROR at line 1:
ORA-01589: must use RESETLOGS or NORESETLOGS option for database open



SQL> SQL>    alter database mount
*
ERROR at line 1:
ORA-01100: database already mounted


SQL>    alter database open
*
ERROR at line 1:
ORA-01589: must use RESETLOGS or NORESETLOGS option for database open


SQL>
System altered.

SQL> Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.29.0.0.0
The Oracle base remains unchanged with value /opt/oracle
Waiting for 10 sec(s) before restarting Oracle processes and opening the database
The Oracle base remains unchanged with value /opt/oracle
#####################################
########### E R R O R ###############
DATABASE SETUP WAS NOT SUCCESSFUL!
Please check output for further info!
########### E R R O R ###############
#####################################
The following output is now a tail of the alert.log:
Data Guard Broker initialization complete
Data Guard: verifying database primary role...
Data Guard: broker startup completed
Data Guard: primary database controlfile verified
Smart fusion block transfer is disabled:
  instance mounted in exclusive mode.
2026-01-09T07:35:20.713465+00:00
Errors in file /opt/oracle/diag/rdbms/ncnfdb11/NCNFDB11/trace/NCNFDB11_ora_496.trc:
ORA-01589: must use RESETLOGS or NORESETLOGS option for database open
ORA-1589 signalled during:    alter database open...
2026-01-09T07:49:15.166990+00:00
db_recovery_file_dest_size of 71680 MB is 36.39% used. This is a
user-specified limit on the amount of space that will be used by this
database for recovery-related files, and does not reflect the amount of
space available in the underlying filesystem or ASM diskgroup.

[andbos]

What we tried to do to rescue the SIDB instance:

Part 1: Initial Root Cause Analysis

The environment began in a crashed state because the Oracle Database Operator (running in Kubernetes) attempted to provision a PDB (ORCLPDB1), but the operation failed midway.

• CDB Failure: The Container Database (CDB) was stuck in a MOUNTED state because it couldn't reconcile the control file with the half-created PDB files.
• Operator Interference: Every time we tried to fix the database, the Kubernetes Operator would see the "Down" status and attempt to restart/re-provision the DB, essentially overwriting our manual fixes.
• Action Taken: We scaled the Operator to 0 (kubectl scale deployment ... --replicas=0) to gain manual control.

Part 2: Restoring the CDB

We successfully brought the Root Container (CDB$ROOT) back to life:

• Clearing the Obstacle: We dropped the corrupted ORCLPDB1 metadata that was preventing a clean boot.
• Instance Recovery: We performed a STARTUP FORCE, allowing Oracle to perform crash recovery on the Redo logs.
• Result: The CDB reached READ WRITE status, and the PDB$SEED reached READ ONLY.

Part 3: The "Ghost" Seed Problem

Once the CDB was up, we discovered why the PDB creation was failing.

• The Issue: The PDB$SEED (the system's master template) has a metadata corruption. Its data dictionary insists it has a Temporary Tablespace, but the actual file is missing or the internal pointer is "Empty."
• The Error: ORA-25153: Temporary Tablespace is Empty.
• The Logic: Because every new PDB is a physical clone of the Seed, they all inherit this "Empty Temp" error. Oracle validates the tempfile at the very end of the CREATE transaction; it fails, and rolls back.

Part 4: The "Inactive" Deadlock (The Final Wall)

This is where we spent the most effort. We entered a "Three-Way Deadlock":

• The Inactive State: Because the CREATE command failed at the finish line, Oracle left ORCLPDB1 in an INACTIVE state in the control file.
• The Command Lock: Oracle blocks almost all ALTER or OPEN commands on an INACTIVE PDB (ORA-65104).
• The Bypass Failure: We attempted to bypass the Seed using XML Manifests (DBMS_PDB) and Manual File Copies, but the Oracle 19c kernel still performed a mandatory "Temp Check" against the Seed's metadata before allowing the PDB to transition from INACTIVE to OPEN.

[andbos]

I suppose that all these ORA errors are not OraOperator specific but I'm pretty sure it would have got sorted had the database instances been running on VMs (no K8s). It seems like CNF Oracle protects itself against advanced rescue operations in a three layer defence:

1. The oracle-database-operator-controller-manager pods - sometimes needed to scale the deployment down to 0 replicas
2. The runOracle.sh script that restarts the DB - sometimes needed to stop it: pkill -STOP -f "runOracle.sh"
3. The Kubelet - kills the pod if the Oracle process disappears for even a few seconds, this makes life hard if you want to run for instance shutdown immediate, before you can write startup the Kubelet nukes the container...