Skip to content

SIDB standby stuck in Unhealthy #211

@andbos

Description

@andbos

My SIDB standby instance is stuck in Unhealthy even though the instance is running OK and Dataguard is running OK.
I tried /home/oracle/shutDown.sh immediate followed with /home/oracle/startUp.sh, worked as such but standby pod is never ready.

$ kubectl -n oracle-database get pods
NAME                   READY   STATUS    RESTARTS      AGE
cnfdb-dgbroker-h881r   1/1     Running   0             22h
cnfdb11-czmxg          1/1     Running   1 (28h ago)   28h
cnfdb12-7e3mo          0/1     Running   0             9m43s
$ kubectl -n oracle-database get sidb
NAME      EDITION      STATUS      ROLE               VERSION       CONNECT STR                    TCPS CONNECT STR   OEM EXPRESS URL
cnfdb11   Enterprise   Healthy     PRIMARY            19.29.0.0.0   10.240.222.19:1521/NCNFDB11    Unavailable        https://10.12.86.125:5500/em
cnfdb12   Enterprise   Unhealthy   PHYSICAL_STANDBY   19.29.0.0.0   10.12.86.125:1521/NCNFDB12     Unavailable        https://10.12.86.125:5500/em

$ kubectl -n oracle-database get sidb cnfdb12 -o json | jq '.status'
{
  "archiveLog": "true",
  "clusterConnectString": "cnfdb12-ext.oracle-database:1521/NCNFDB12",
  "conditions": [
    {
      "lastTransitionTime": "2026-01-15T09:45:27Z",
      "message": "command terminated with exit code 1",
      "observedGeneration": 1,
      "reason": "LastReconcileCycleFailed",
      "status": "True",
      "type": "ReconcileError"
    },
    {
      "lastTransitionTime": "2026-01-15T09:45:37Z",
      "message": "no reconcile errors",
      "observedGeneration": 1,
      "reason": "LastReconcileCycleCompleted",
      "status": "True",
      "type": "ReconcileComplete"
    },
    {
      "lastTransitionTime": "2026-01-27T13:17:21Z",
      "message": "no reconcile errors",
      "observedGeneration": 1,
      "reason": "LastReconcileCycleQueued",
      "status": "True",
      "type": "ReconcileQueued"
    }
  ],
  "connectString": "10.12.86.125:1521/NCNFDB12",
  "createdAs": "standby",
  "datafilesCreated": "true",
  "datafilesPatched": "false",
  "dgBroker": "cnfdb-dgbroker",
  "edition": "Enterprise",
  "flashBack": "true",
  "forceLog": "true",
  "initParams": {
    "cpuCount": 8,
    "pgaAggregateTarget": 1599,
    "processes": 640,
    "sgaTarget": 4800
  },
  "isTcpsEnabled": false,
  "oemExpressUrl": "https://10.12.86.125:5500/em",
  "pdbConnectString": "10.12.86.125:1521/ORCLPDB1",
  "pdbName": "ORCLPDB1",
  "persistence": {
    "accessMode": "ReadWriteOnce",
    "size": "300Gi",
    "storageClass": "csi-sc-cinderplugin"
  },
  "primaryDatabase": "cnfdb11",
  "releaseUpdate": "19.29.0.0.0",
  "replicas": 1,
  "role": "PHYSICAL_STANDBY",
  "sid": "NCNFDB12",
  "status": "Unhealthy",
  "tcpsConnectString": "Unavailable",
  "tcpsPdbConnectString": "Unavailable",
  "tcpsTlsSecret": ""
}
$ kubectl -n oracle-database get dgbroker
NAME             PRIMARY    STANDBYS   PROTECTION MODE   CONNECT STR                   STATUS    FSFO
cnfdb-dgbroker   NCNFDB11   NCNFDB12   MaxAvailability   10.12.86.110:1521/DATAGUARD   Healthy   true

$ kubectl -n oracle-database exec -it cnfdb11-czmxg -- /bin/bash
bash-5.1$ dgmgrl / as sysdba
DGMGRL for Linux: Release 19.0.0.0.0 - Production on Tue Jan 27 13:08:56 2026
Version 19.29.0.0.0

Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.

Welcome to DGMGRL, type "help" for information.
Connected to "NCNFDB11"
Connected as SYSDBA.
DGMGRL> show configuration

Configuration - dg_config

  Protection Mode: MaxAvailability
  Members:
  ncnfdb11 - Primary database
    ncnfdb12 - (*) Physical standby database

Fast-Start Failover: Enabled in Zero Data Loss Mode

Configuration Status:
SUCCESS   (status updated 40 seconds ago)

The standby database seems however to be 100% healthy:

$ kubectl -n oracle-database exec -it cnfdb12-7e3mo -- /bin/bash
bash-5.1$ dgmgrl / as sysdba
SQL> SELECT RECOVERY_MODE FROM V$ARCHIVE_DEST_STATUS WHERE DEST_ID=1;

RECOVERY_MODE
----------------------------------
MANAGED REAL TIME APPLY

SQL> SELECT name, open_mode, database_role, switchover_status FROM v$database;

NAME      OPEN_MODE            DATABASE_ROLE    SWITCHOVER_STATUS
--------- -------------------- ---------------- --------------------
NCNFDB11  READ ONLY WITH APPLY PHYSICAL STANDBY NOT ALLOWED

SQL> SELECT NAME, VALUE, DATUM_TIME FROM V$DATAGUARD_STATS WHERE NAME LIKE '%lag%';

NAME
--------------------------------
VALUE
----------------------------------------------------------------
DATUM_TIME
------------------------------
transport lag
+00 00:00:00
01/27/2026 13:06:03

apply lag
+00 00:00:00
01/27/2026 13:06:03

NAME
--------------------------------
VALUE
----------------------------------------------------------------
DATUM_TIME
------------------------------

SQL> SELECT PROCESS, STATUS, SEQUENCE# FROM V$MANAGED_STANDBY;

PROCESS   STATUS        SEQUENCE#
--------- ------------ ----------
ARCH      CONNECTED             0
DGRD      ALLOCATED             0
DGRD      ALLOCATED             0
ARCH      CLOSING             190
ARCH      CLOSING             192
ARCH      CLOSING             193
RFS       IDLE                  0
RFS       IDLE                  0
RFS       IDLE                194
MRP0      APPLYING_LOG        194
RFS       IDLE                  0

11 rows selected.

SQL> SELECT THREAD#, LOW_SEQUENCE#, HIGH_SEQUENCE# FROM V$ARCHIVE_GAP;

no rows selected

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions