On PRKP-1033, CRS-0217, PRKP-1030 and CRS-0215 errors while using 'srvctl relocate service' command

Tweet

Today I had to relocate a service named MYPRD02 from one node to another, but it didn't work as usual and something new happened.
This production installation is an Oracle RAC environment formed by three nodes: it is based on Oracle Database Server 10gR2 patched to 10.2.0.4 for x86_64 Linux architecture.

All my clustered resources were online like the service I wanted to relocate.

[oracle@myrac01 ~]$ crsstat
HA Resource                                   Target     State             
-----------                                   ------     -----             
...
ora.MYRAC.MYPRD02.MYRAC1.srv                  ONLINE     ONLINE           
ora.MYRAC.MYPRD02.cs                          ONLINE     ONLINE      
...

I executed the following relocate service command using srvctl cli but I received the error I couldn't relocate that resource.

oracle@myrac01 ~]$ srvctl relocate service -d MYRAC -s MYPRD02 -i MYRAC1 -t MYRAC2
PRKP-1033 : Cannot relocate service MYPRD02 from instance MYRAC1 to instance MYRAC2.
CRS-0217: Could not relocate resource ''.

After the command was unsuccessfully completed my crsstat command showed the following new states for the same service:

[oracle@myrac01 ~]$ crsstat
HA Resource                                   Target     State             
-----------                                   ------     -----             
...
ora.MYRAC.MYPRD02.MYRAC1.srv                  ONLINE     OFFLINE           
ora.MYRAC.MYPRD02.cs                          OFFLINE     OFFLINE      
...

From srvctl cli interface my service was no more available...

[oracle@myrac01 ~]$ srvctl status database -d MYRAC -v
Instance MYRAC1 is running on node myrac01 with online services MYBI 
Instance MYRAC2 is running on node myrac02 with online services MYPRD01 
Instance MYRAC3 is running on node myrac03

...but querying my database I obtained:

SQL> select inst_id, name from gv$active_services  order by service_id ;

   INST_ID NAME
---------- ----------------------------------------------------------------
  1 SYS$BACKGROUND
  3 SYS$BACKGROUND
  2 SYS$BACKGROUND
  2 SYS$USERS
  1 SYS$USERS
  3 SYS$USERS
  3 MYRACXDB
  1 MYRACXDB
  2 MYRACXDB
  1 MYRAC
  3 MYRAC
  2 MYRAC
  2 MYPRD01
  1 MYPRD02
  1 MYBI

15 rows selected.

So from srvctl cli a new session trying to connect to MYPRD02 service shouldn't be able to estabilish a connection with the database, whereas looking at the GV$ACTIVE_SERVICES view the same service was up and active to process new incoming requests. Some errors were logged on crsd log located at $CRS_HOME/log//crsd/:

[oracle@myrac01 ~]$ cd $CRS_HOME/log/myrac01/crsd/
[oracle@myrac01 crsd]$ pwd
/opt/crs/oracle/product/10.2.0/crs/log/myrac01/crsd/
[oracle@myrac01 crsd]$ ls
crsd.log

[oracle@myrac01 crsd]$ vi crsd.log
...
2013-03-11 09:31:14.249: [  CRSRES][1494907200]0StopResource: setting CLI values
2013-03-11 09:31:14.321: [  CRSRES][1494907200]0Attempting to stop `ora.MYRAC.MYPRD02.MYRAC1.srv` on member `myrac01`
2013-03-11 09:31:15.350: [  CRSRES][1494907200]0Stop of `ora.MYRAC.MYPRD02.MYRAC1.srv` on member `myrac01` succeeded.
2013-03-11 09:31:15.396: [  CRSRES][1494907200]0Attempting to start `ora.MYRAC.MYPRD02.MYRAC1.srv` on member `myrac02`
2013-03-11 09:31:15.945: [  CRSRES][1494907200]0Start of `ora.MYRAC.MYPRD02.MYRAC1.srv` on member `myrac02` failed.
...

From that log my crs was able to stop the resource but then failed to relocate and start it on another node, in this case the second.
I tried to start it using the following command, trying first on second node and then on third node, but always obtaining the same errors:

[oracle@myrac01 ~]$ srvctl start service -d MYRAC -s MYPRD02 -i MYRAC2
PRKP-1030 : Failed to start the service MYPRD02.
CRS-0215: Could not start resource 'ora.MYRAC.MYPRD02.MYRAC1.srv'.

[oracle@myrac01 ~]$ srvctl status database -d MYRAC -v
Instance MYRAC1 is running on node myrac01 with online services MYBI 
Instance MYRAC2 is running on node myrac02 with online services MYPRD01 
Instance MYRAC3 is running on node myrac03

[oracle@myrac01 ~]$ srvctl start service -d MYRAC -s MYPRD02 -i MYRAC3
PRKP-1030 : Failed to start the service MYPRD02.
CRS-0215: Could not start resource 'ora.MYRAC.MYPRD02.MYRAC1.srv'.

Again on crsd log the following unsuccessfully attempts to start MYPRD02 service were logged:

2013-03-11 09:56:28.043: [  CRSRES][1494907200]0Attempting to start `ora.MYRAC.MYPRD02.MYRAC1.srv` on member `myrac02`
2013-03-11 09:56:28.319: [  CRSRES][1494907200]0Start of `ora.MYRAC.MYPRD02.MYRAC1.srv` on member `myrac02` failed.
2013-03-11 10:01:31.198: [  CRSRES][1494907200]0Attempting to start `ora.MYRAC.MYPRD02.MYRAC1.srv` on member `myrac03`
2013-03-11 10:01:31.454: [  CRSRES][1494907200]0Start of `ora.MYRAC.MYPRD02.MYRAC1.srv` on member `myrac03` failed.

I found an interesting log looking into the database log directory of the second node.
Into imon_MYRAC.log file located at /opt/app/oracle/product/10.2.0/db_1/log/myrac02/racg directory of the second node the following errors were logged, stating that the service MYPRD02 was already running. Those errors were logged at 09:56:28.173 that is few centiseconds later the srvtcl start service command failed to start from node one the service MYPRD02 on node two executed on 2013-03-11 at time 09:56:28.043.

2013-03-11 09:56:28.173: [    RACG][1246857536] [7383][1246857536][ora.MYRAC.MYPRD02.MYRAC1.srv]: CLSR-0002: Oracle error encountered while executing clsrcsnstartsrv : execute2

2013-03-11 09:56:28.173: [    RACG][1246857536] [7383][1246857536][ora.MYRAC.MYPRD02.MYRAC1.srv]: ORA-44305: service MYPRD02 is running
ORA-06512: at "SYS.DBMS_SYS_ERROR", line 86
ORA-06512: at "SYS.DBMS_SERVICE", line 444
ORA-06512: at "SYS.DBMS_SERVICE", line 365
ORA-06512: at line 1

The same errors were logged on the third node looking into imon_MYRAC.log file located at /opt/app/oracle/product/10.2.0/db_1/log/myrac03/racg directory. The srvctl start service command was executed at 10:01:31.198 from the first node and the errors were logged on the third node at 10:01:31.325:

2013-03-11 10:01:31.325: [    RACG][1281046848] [7439][1281046848][ora.MYRAC.MYPRD02.MYRAC1.srv]: CLSR-0002: Oracle error encountered while executing clsrcsnstartsrv : execute2
2013-03-11 10:01:31.325: [    RACG][1281046848] [7439][1281046848][ora.MYRAC.MYPRD02.MYRAC1.srv]: ORA-44305: service MYPRD02 is running
ORA-06512: at "SYS.DBMS_SYS_ERROR", line 86
ORA-06512: at "SYS.DBMS_SERVICE", line 444
ORA-06512: at "SYS.DBMS_SERVICE", line 365
ORA-06512: at line 1

I was able anyway to manually estabilish a connection to that service so the GV$ACTIVE_SERVICES view was reliable, while srvctl cli was misleading and, most important of all, no one was experiencing any kind of problems because it was performed during a planned operation.
On My Oracle Support I found two interesting notes that could fit on my specific errors:
10.2 Database Service Not Starting With Errors: CRS-1006, CRS-215, ORA-44305 [ID 752366.1] suggesting to apply a patch and 'Srvctl Start Service' Fails With PRKP-1030, CRS-0215 And ORA-1003, CLSR-0002 In Database Imon Logs [ID 818997.1] suggesting to kill racgimon process or reboot the instance.
So I first tried to kill racgimon process on first node:

[oracle@myrac01 crsd]$ ps -ef|grep racgimon
oracle    7405     1  0 Feb26 ?        00:03:17 /opt/app/oracle/product/10.2.0/db_1/bin/racgimon startd MYRAC
[oracle@myrac01 crsd]$ kill -9 7405

Then I executed again the srvctl start service command and finally I was able to start the service:

[oracle@myrac01 crsd]$ srvctl start service -d MYRAC -s MYPRD02 -i MYRAC2
[oracle@myrac01 ~]$ srvctl status database -d MYRAC -v
Instance MYRAC1 is running on node myrac01 with online services MYBI 
Instance MYRAC2 is running on node myrac02 with online services MYPRD01,MYPRD02 
Instance MYRAC3 is running on node myrac03

That's all.