Remote Oracle Database Support : OCIException OCI error 1034

Error:

2013-04-26 22:43:38.154: [ora.cramadg.db][3523208960] {2:35163:2} [start] InstConnection::connectInt (2) Exception OCIException

2013-04-26 22:43:38.155: [ora.cramadg.db][3523208960] {2:35163:2} [start] InstConnection:connect:excp OCIException OCI error 1034

2013-04-26 22:43:38.155: [ora.cramadg.db][3523208960] {2:35163:2} [start] InstAgent::stop: connect1 errcode 1034

2013-04-26 22:43:38.155: [ora.cramadg.db][3523208960] {2:35163:2} [start] InstAgent::stop: connect2 oracleHome /oravl01/ora11g/112_GRID oracleSid CRAMADG2

2013-04-26 22:43:38.155: [ora.cramadg.db][3523208960] {2:35163:2} [start] InstConnection::connectInt: server not attached

2013-04-26 22:43:38.181: [ora.cramadg.db][3523208960] {2:35163:2} [start] InstConnection::shutdown mode 4

2013-04-26 22:43:38.181: [ora.cramadg.db][3523208960] {2:35163:2} [start] ORA-01031: insufficient privileges

2013-04-26 22:43:38.181: [ora.cramadg.db][3523208960] {2:35163:2} [start] InstAgent::stop: shutdown errcode 1031

2013-04-26 22:43:38.181: [ora.cramadg.db][3523208960] {2:35163:2} [start] InstAgent::stop: oci error try shutdown abort

2013-04-26 22:43:39.182: [ora.cramadg.db][3523208960] {2:35163:2} [start] InstConnection::connectInt: server not attached

2013-04-26 22:43:39.210: [ora.cramadg.db][3523208960] {2:35163:2} [start] ORA-01034: ORACLE not available

ORA-27101: shared memory realm does not exist

Linux-x86_64 Error: 2: No such file or directory

Process ID: 0

Session ID: 0 Serial number: 0

Background:

1. 2 Node RAC 11g R2 Linux Primary

2. 2 Node RAC 11gR2 Linux Standby/ADG

3. Error faced after PSU Patch on Standby , On Primary No issues faced

4. Cluster DB CRAMADG unable to start/stop through crsctl

$ sudo /oravl01/ora11g/112_GRID/bin/crsctl start cluster -all

CRS-4639: Could not contact Oracle High Availability Services

CRS-4000: Command Start failed, or completed with errors.

$ sudo /oravl01/ora11g/112_GRID/bin/crsctl start crs

CRS-4123: Oracle High Availability Services has been started.

$ crs_stat -t

CRS-0184: Cannot communicate with the CRS daemon.

$ sudo /oravl01/ora11g/112_GRID/bin/crsctl stop cluster

CRS-2796: The command may not proceed when Cluster Ready Services is not running

CRS-4000: Command Stop failed, or completed with errors.

$ sudo /oravl01/ora11g/112_GRID/bin/crsctl stop crs

CRS-2796: The command may not proceed when Cluster Ready Services is not running

CRS-4687: Shutdown command has completed with errors.

CRS-4000: Command Stop failed, or completed with errors.

5. ASM was coming Up through crsctl

$ ps -ef| grep -i pmon

ora11g 29812 1 0 14:21 ? 00:00:00 asm_pmon_+ASM1

ora11g 31393 3053 0 14:25 pts/0 00:00:00 grep -i pmon

6. Manually via sqlplus able to bring up database but not through crsctl

$ sqlplus "/ as sysdba"

SQL*Plus: Release 11.2.0.3.0 Production on Fri Apr 26 15:11:36 2013

Connected to an idle instance.

SQL> startup

ORA-32004: obsolete or deprecated parameter(s) specified for RDBMS instance

ORACLE instance started.

Total System Global Area 7733837824 bytes

Fixed Size 2243152 bytes

Variable Size 1946158512 bytes

Database Buffers 5670699008 bytes

Redo Buffers 114737152 bytes

Database mounted.

Database opened.

SQL> exit

7. GRID/ASM Home : /oravl01/ora11g/112_GRID

8. ORACLE Home : /oravl01/ora11g/112_RAC

Analysis:

1. crsd agent Log showing generic error ORA-27101 , ORA-01031 , ORA-01034 which occurs when environment variables like ORACLE_HOME & ORACLE_SID not pointing to correct home in single instance database.

Here, "connect2 oracleHome /oravl01/ora11g/112_GRID oracleSid CRAMADG2" crsctl stop was using Grid Home instead of oracle home for database resource

Log: /oravl01/ora11g/112_GRID/log/inpudicrmrdbprtrpt2/agent/crsd/oraagent_ora11g/ertinpudicrmrdbprtrpt2.log

2013-04-26 22:43:38.155: [ora.cramadg.db][3523208960] {2:35163:2} [start] InstAgent::stop: connect2 oracleHome /oravl01/ora11g/112_GRID oracleSid CRAMADG2

2013-04-26 22:43:38.181: [ora.cramadg.db][3523208960] {2:35163:2} [start] ORA-01031: insufficient privileges

2013-04-26 22:43:39.210: [ora.cramadg.db][3523208960] {2:35163:2} [start] ORA-01034: ORACLE not available

ORA-27101: shared memory realm does not exist

Linux-x86_64 Error: 2: No such file or directory

Process ID: 0

Session ID: 0 Serial number: 0

2. ORA-01031 when attempted to start CRAMADG database with ORACLE_HOME set to grid home /oravl01/ora11g/112_GRID , given hint that oracle home is not properly picked up by the database resource

$ sqlplus "/ as sysdba"

SQL*Plus: Release 11.2.0.3.0 Production on Fri Apr 26 14:35:28 2013

Connected to:

Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production

With the Real Application Clusters and Automatic Storage Management options

SQL> shut immediate;

ORA-01031: insufficient privileges

SQL> exit

3. Cluster resource state checked & found status as "Open,Readonly,Running from Old Oracle Home" for ora.cramadg.db which confirms that wrong ORACLE_HOME being picked up

$ crsctl stat res -t

--------------------------------------------------------------------------------

NAME TARGET STATE SERVER STATE_DETAILS

--------------------------------------------------------------------------------

Local Resources

--------------------------------------------------------------------------------

ora.DATA1.dg ONLINE ONLINE inpudicrmrdbprtrpt1