Issue: Getting CRS-4013 , CRS-4000 error while starting up cluster services
One of the cluster node is reporting the CRS-4013 , CRS-4000 error while other node is up in 2 node RAC cluster
Solution:
CRS-4013: This command is not supported in a single-node configuration.
CRS-4000: Command Start failed, or completed with errors.
CRS-4000: Command Start failed, or completed with errors.
One of the cluster node is reporting the CRS-4013 , CRS-4000 error while other node is up in 2 node RAC cluster
Solution:
1. Error while starting up cluster on problematic node(mytestlab1283)
[ora12c@mytestlab1283 ~]$ ps -ef | grep pmon
ora12c 8572 8540 0 10:59 pts/0 00:00:00 grep --color=auto pmon
[ora12c@mytestlab1283 ~]$ . ./.bash_profile_asm
[ora12c@mytestlab1283 ~]$ which crsctl
/rdbms/app/12.1.0.2/grid/bin/crsctl
[ora12c@mytestlab1283 ~]$ sudo /rdbms/app/12.1.0.2/grid/bin/crsctl start crs
CRS-4013: This command is not supported in a single-node configuration.
CRS-4000: Command Start failed, or completed with errors.
2. Check resource status on working node (mytestlab12834)
[ora12c@mytestlab1284 ~]$ ps -ef | grep pmon
ora12c 2601 2550 0 11:09 pts/0 00:00:00 grep --color=auto pmon
ora12c 23415 1 0 May09 ? 00:10:50 asm_pmon_+ASM2
ora12c 23517 1 0 May09 ? 00:12:00 ora_pmon_oemdbmn2
[ora12c@mytestlab1283 ~]$ sudo /rdbms/app/12.1.0.2/grid/bin/crsctl stop crs -f
CRS-4013: This command is not supported in a single-node configuration.
CRS-4000: Command Stop failed, or completed with errors.
[ora12c@mytestlab1283 ~]$ crsctl check css
CRS-4639: Could not contact Oracle High Availability Services
CRS-4000: Command Check failed, or completed with errors.
[ora12c@mytestlab1284 ~]$ crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
[ora12c@mytestlab1284 ~]$ crsctl status res -t
------------------------------------------------------------
Name Target State Server State details Local Resources
-------------------------------------------------------------
ora.DATA.dg ONLINE ONLINE mytestlab1284 STABLE
ora.FRA.dg ONLINE ONLINE mytestlab1284 STABLE
ora.LISTENER.lsnr ONLINE ONLINE mytestlab1284 STABLE
ora.OCR_VOTING.dg ONLINE ONLINE mytestlab1284 STABLE
ora.asm ONLINE ONLINE mytestlab1284 Started,STABLE
ora.net1.network ONLINE ONLINE mytestlab1284 STABLE
ora.ons ONLINE ONLINE mytestlab1284 STABLE
--------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr 1 ONLINE ONLINE mytestlab1284 STABLE
ora.LISTENER_SCAN2.lsnr 1 ONLINE ONLINE mytestlab1284 STABLE
ora.LISTENER_SCAN3.lsnr 1 ONLINE ONLINE mytestlab1284 STABLE
ora.MGMTLSNR 1 OFFLINE OFFLINE STABLE
ora.cvu 1 ONLINE ONLINE mytestlab1284 STABLE
ora.mytestlab1283.vip 1 ONLINE INTERMEDIATE mytestlab1284 FAILED OVER,STABLE
ora.mytestlab1284.vip 1 ONLINE ONLINE mytestlab1284 STABLE
ora.oc4j 1 ONLINE ONLINE mytestlab1284 STABLE
ora.oemdbmn.db 1 ONLINE OFFLINE STABLE
2 ONLINE ONLINE mytestlab1284 Open,STABLE
ora.scan1.vip 1 ONLINE ONLINE mytestlab1284 STABLE
ora.scan2.vip 1 ONLINE ONLINE mytestlab1284 STABLE
ora.scan3.vip 1 ONLINE ONLINE mytestlab1284 STABLE
-------------------------------------------------------------------
[ora12c@mytestlab1284 trace]$ srvctl config database -d oemdbmn
Database unique name: oemdbmn
Database name: oemdbmn
Oracle home: /rdbms/app/oracle/12.1.0.2/db
Oracle user: ora12c
Spfile: +DATA/OEMDBMN/PARAMETERFILE/spfile.268.943049417
Password file: +DATA/OEMDBMN/PASSWORD/pwdoemdbmn.256.943049111
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools:
Disk Groups: DATA
Mount point paths:
Services:
Type: RAC
Start concurrency:
Stop concurrency:
OSDBA group: dba
OSOPER group: dba
Database instances: oemdbmn1,oemdbmn2
Configured nodes: mytestlab1283,mytestlab1284
Database is administrator managed
Database is administrator managed
3. Stop crs on working node (mytestlab12834)
[ora12c@mytestlab1284 ~]$ sudo /rdbms/app/12.1.0.2/grid/bin/crsctl stop crs
[sudo] password for ora12c:
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'mytestlab1284'
CRS-2673: Attempting to stop 'ora.crsd' on 'mytestlab1284'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'mytestlab1284'
CRS-2673: Attempting to stop 'ora.oemdbmn.db' on 'mytestlab1284'
CRS-2673: Attempting to stop 'ora.oc4j' on 'mytestlab1284'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'mytestlab1284'
.
.
.
.
.
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'mytestlab1284' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'mytestlab1284'
CRS-2677: Stop of 'ora.cssd' on 'mytestlab1284' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'mytestlab1284'
CRS-2677: Stop of 'ora.gipcd' on 'mytestlab1284' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'mytestlab1284' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[ora12c@mytestlab1284 ~]$ ps -ef | grep pmon
ora12c 6311 5970 0 11:29 pts/1 00:00:00 grep --color=auto pmon
4. Execute below on problematic node (mytestlab1283)
[ora12c@mytestlab1283 ~]$ sudo /rdbms/app/12.1.0.2/grid/bin/crsctl start crs
CRS-4013: This command is not supported in a single-node configuration.
CRS-4000: Command Start failed, or completed with errors.
sudo /rdbms/app/12.1.0.2/grid/crs/install/rootcrs.sh -deconfig -force
sudo /rdbms/app/12.1.0.2/grid/root.sh
5. Verify logs and crs process status on OS
[ora12c@mytestlab1283 ~]$ ps -ef | grep -i crs
root 25735 25646 0 12:55 pts/0 00:00:00 /bin/sh /rdbms/app/12.1.0.2/grid/crs/config/rootconfig.sh
root 25744 25735 0 12:55 pts/0 00:00:00 /rdbms/app/12.1.0.2/grid/perl/bin/perl -I/rdbms/app/12.1.0.2/grid/perl/lib -I/rdbms/app/12.1.0.2/grid/crs/install /rdbms/app/12.1.0.2/grid/crs/install/rootcrs.pl
root 25763 25744 0 12:55 pts/0 00:00:00 /bin/sh /rdbms/app/12.1.0.2/grid/crs/install/tfa_setup -silent -crshome /rdbms/app/12.1.0.2/grid
ora12c 26600 26372 0 12:57 pts/1 00:00:00 grep --color=auto -i crs
[ora12c@mytestlab1283 ~]$ ps -ef | grep -i crs
root 25735 25646 0 12:55 pts/0 00:00:00 /bin/sh /rdbms/app/12.1.0.2/grid/crs/config/rootconfig.sh
root 25744 25735 0 12:55 pts/0 00:00:00 /rdbms/app/12.1.0.2/grid/perl/bin/perl -I/rdbms/app/12.1.0.2/grid/perl/lib -I/rdbms/app/12.1.0.2/grid/crs/install /rdbms/app/12.1.0.2/grid/crs/install/rootcrs.pl
root 28153 25744 0 12:58 pts/0 00:00:00 sh -c /bin/su ora12c -c ' echo CLSRSC_START; /rdbms/app/12.1.0.2/grid/bin/cluutil -ckpt -oraclebase /rdbms/app/oracle -chkckpt -name ROOTCRS_INITRES ' 2>&1
root 28154 28153 0 12:58 pts/0 00:00:00 /bin/su ora12c -c echo CLSRSC_START; /rdbms/app/12.1.0.2/grid/bin/cluutil -ckpt -oraclebase /rdbms/app/oracle -chkckpt -name ROOTCRS_INITRES
ora12c 28155 28154 0 12:58 ? 00:00:00 bash -c echo CLSRSC_START; /rdbms/app/12.1.0.2/grid/bin/cluutil -ckpt -oraclebase /rdbms/app/oracle -chkckpt -name ROO CRS_INITRES
ora12c 28156 28155 0 12:58 ? 00:00:00 /bin/sh /rdbms/app/12.1.0.2/grid/bin/cluutil -ckpt -oraclebase /rdbms/app/oracle -chkckpt -name ROOTCRS_INITRES
ora12c 28159 28156 0 12:58 ? 00:00:00 /rdbms/app/12.1.0.2/grid/jdk/bin/java -DTRACING.ENABLED=true -DTRACING.LEVEL=2 -classpath /rdbms/app/12.1.0.2/grid/jlib/srvm.jar:/rdbms/app/12.1.0.2/grid/jlib/srvmhas.jar:/rdbms/app/12.1.0.2/grid/oui/jlib/OraCheckPoint.jar:/rdbms/app/12.1.0.2/grid/oui/jlib/OraInstaller.jar:/rdbms/app/12.1.0.2/grid/oui/jlib/xmlparserv2.jar oracle.ops.util.ClusterUtil -ckpt -oraclebase /rdbms/app/oracle -chkckpt -name ROOTCRS_INITRES
ora12c 28172 26372 0 12:58 pts/1 00:00:00 grep --color=auto -i crs
Copying coraenv to /usr/local/bin ...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /rdbms/app/12.1.0.2/grid/crs/install/crsconfig_params
2017/08/10 13:01:14 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.
2017/08/10 13:03:00 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
2017/08/10 13:03:01 CLSRSC-363: User ignored prerequisites during installation
2017/08/10 13:04:06 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.service'
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'mytestlab1283'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'mytestlab1283'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'mytestlab1283'
CRS-2673: Attempting to stop 'ora.gipcd' on 'mytestlab1283'
CRS-2673: Attempting to stop 'ora.evmd' on 'mytestlab1283'
CRS-2677: Stop of 'ora.gpnpd' on 'mytestlab1283' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'mytestlab1283' succeeded
CRS-2677: Stop of 'ora.evmd' on 'mytestlab1283' succeeded
CRS-2677: Stop of 'ora.gipcd' on 'mytestlab1283' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'mytestlab1283' has completed
.
.
.
.
CRS-2672: Attempting to start 'ora.LISTENER.lsnr' on 'mytestlab1283'
CRS-2676: Start of 'ora.LISTENER_SCAN3.lsnr' on 'mytestlab1283' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on 'mytestlab1283' succeeded
CRS-2676: Start of 'ora.LISTENER.lsnr' on 'mytestlab1283' succeeded
CRS-2672: Attempting to start 'ora.oemdbmn.db' on 'mytestlab1283'
CRS-2676: Start of 'ora.oc4j' on 'mytestlab1283' succeeded
CRS-2676: Start of 'ora.oemdbmn.db' on 'mytestlab1283' succeeded
CRS-6016: Resource auto-start has completed for server mytestlab1283
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2017/08/10 13:07:53 CLSRSC-343: Successfully started Oracle Clusterware stack
PRKO-2190 : VIP exists for node mytestlab1283, VIP name mytestlab1283-vip.vsnl.co.in
PRKZ-1072 : SCAN name "XPDBoemdbmn" is already registered on network 1
PRCS-1028 : Single Client Access Name (SCAN) listener resources already exist on network 1
PRCN-3004 : Listener MGMTLSNR already exists
PRCR-1086 : resource ora.cvu is already registered
PRCC-1014 : scan1 was already running
PRCR-1004 : Resource ora.scan1.vip is already running
PRCR-1079 : Failed to start resource ora.scan1.vip
CRS-5702: Resource 'ora.scan1.vip' is already running on 'mytestlab1283'
.
.
PRCC-1014 : cvu was already running
PRCR-1004 : Resource ora.cvu is already running
PRCR-1079 : Failed to start resource ora.cvu
CRS-5702: Resource 'ora.cvu' is already running on 'mytestlab1283'
Preparing packages...
cvuqdisk-1.0.9-1.x86_64
PRCN-3004 : Listener LISTENER already exists
PRCC-1015 : LISTENER was already running on mytestlab1283
PRCR-1004 : Resource ora.LISTENER.lsnr is already running
2017/08/10 13:08:36 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded
No comments:
Post a Comment