由于测试环境打19c补丁失败，导致节点2  d2 crs无法启动！打补丁没有有效备份，亲人两行泪

先删再加节点。

一、删除节点

1.1 删除实例

参考

https://www.cnblogs.com/binliubiao/p/11996092.html

Deleting a Cluster Node on Linux and UNIX Systems

https://docs.oracle.com/en/database/oracle/oracle-database/19/cwadd/

adding-and-deleting-cluster-nodes.html#GUID-8ADA9667-EC27-4EF9-9F34-C8F65A757F2A

删除可以参考上述博客,添加节点建议参考MOS文档

Oracle E-Business Suite Release 12.2: Adding or Deleting 11gR2 or 12c1 Oracle RAC Nodes (Doc ID 1570554.1)

Oracle Database 19c Enterprise Edition Release 19.0.0.0.0

测试环境RAC2节点,打补丁2节点安装失败,因此对节点2进行删除后,再添加。

删除实例，1.1都在正常的节点1执行操作

【由于节点2集群crs无法启动,因此删除操作在节点1执行]srvctl stop instance -db db_unique_name -node node_name

srvctl relocate server -servers "server_name_list" -serverpool Free

srvctl config database -db db_unique_name

oracle$

srvctl config database -db orcl

dbca -silent -deleteInstance -nodeList d2 -gdbName orcl -instanceName orcl2 -sysDBAUserName sysdba -sysDBAPassword oracle

--删除实例需要本地实例已启动！

[FATAL] [DBT-11502] The instance (orcl1) is not running on the local node.

CAUSE: A locally running instance is required to perform the selected operation.

ACTION: Specify a locally running database, or execute DBCA on a node where the database instance is running.

$ srvctl start instance -d orcl -instance orcl1

$ dbca -silent -deleteInstance -nodeList d2 -gdbName orcl -instanceName orcl2Prepare for db operation

40% complete

Deleting instance

48% complete

52% complete

56% complete

60% complete

64% complete

68% complete

72% complete

76% complete

80% complete

Completing instance management.

100% complete

$ srvctl stop database -d orcl

本次测试环境3套DB

均为启动存活的节点1实例,进行删除实例操作。

SQL> alter system set enable_ddl_logging=true;

$ dbca -silent -deleteInstance -nodeList d2 -gdbName abc -instanceName abc2--观察DB Alert日志,并没有ddl操作,而是上次了实例2的redo and undo，并且RAC相关参数设置了一下！

2020-04-15T06:57:54.304305+08:00

Deleted Oracle managed file +DATA/ABC/ONLINELOG/group_3.309.1036713505

Deleted Oracle managed file +DATA/ABC/ONLINELOG/group_4.310.1036713537

Deleted Oracle managed file +DATA/ABC/DATAFILE/undotbs2.308.1036712911

2020-04-15T06:57:55.090943+08:00

ALTER SYSTEM RESET undo_tablespace SCOPE=SPFILE SID='abc2';

2020-04-15T06:57:55.098168+08:00

ALTER SYSTEM RESET instance_number SCOPE=SPFILE SID='abc2';

2020-04-15T06:57:55.107884+08:00

ALTER SYSTEM RESET thread SCOPE=SPFILE SID='abc2';

--

$ srvctl stop database -d abc

继续删除第三个实例

$ srvctl start instance -d nocdd -instance nocdd1

$ dbca -silent -deleteInstance -nodeList d2 -gdbName nocdd -instanceName nocdd2

1.2删除Oracle软件

检测交换分区,及产品软件清单目录 【节点2执行，删除的节点】

oracle$$ORACLE_HOME/oui/bin/runInstaller -updateNodeList ORACLE_HOME=/u03/app/oracle/product/19.0.0/db_1 "CLUSTER_NODES={d2}" -localStarting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB. Actual 3072 MB Passed

The inventory pointer is located at /etc/oraInst.loc

[oracle@d2:/home/oracle]$ cat /etc/oraInst.loc

inventory_loc=/u01/app/oraInventory

inst_group=oinstall

oracle$cd $ORACLE_HOME/deinstall

oracle$./deinstall -local

······

Specify the list of database names that are configured locally on this node for this Oracle home. Local

configurations of the discovered databases will be removed []: 回车，默认读取当前oracle_home目录······

The cluster node(s) on which the Oracle home deinstallation will be performed are:d2

Oracle Home selected for deinstall is: /u03/app/oracle/product/19.0.0/db_1

Inventory Location where the Oracle home registered is: /u01/app/oraInventory

Do you want tocontinue (y - yes, n - no)?[n]: y 继续,yes······

Successfully detached Oracle home '/u03/app/oracle/product/19.0.0/db_1' from the central inventory on the local node.

Successfully deleted directory '/u03/app/oracle/product/19.0.0/db_1' on the local node.

Successfully deleted directory '/u03/app/oracle' on the local node.

Oracle Universal Installer cleanup was successful.

Oracle deinstall tool successfully cleaned up temporary directories.

更新inventory

# 在保留任一节点执行命令

su- oracle

cd $ORACLE_HOME/oui/bin

./runInstaller -updateNodeList ORACLE_HOME=/u03/app/oracle/product/19.0.0/db_1 "CLUSTER_NODES={d1}"

1.3删除Grid软件

检查集群节点是否被锁定,只需要在删除的节点操作即可，是检测是否被锁定

[grid@d1:/home/grid]$ olsnodes -s -td1 Active Unpinned

[grid@d2:/home/grid]$ olsnodes -s -tPRCO-19: Failure retrieving list of nodes in the cluster

PRCO-2: Unable to communicate with the clusterware

# 如果节点被锁定，运行crsctl unpin css命令后继续下一步

crsctl unpin css -n delete_node

在不删除二进制文件的情况下取消配置Oracle Clusterwarehttps://docs.oracle.com/en/database/oracle/oracle-database/19/cwlin/unconfiguring-oracle-clusterware-without-removing-binaries.html#

GUID-41C5E8EE-68F9-4849-B4B8-BD82D4A17A57

在删除的节点上,以root用户执行

su - root

cd /u01/app/19.0.0/grid/crs/install

# /u01/app/19.0.0/grid/perl/bin/perl ./rootcrs.pl -deconfig -force

直接执行./rootcrs.pl提示./rootcrs.pl: line 88: =head1: command not found,perl无法执行脚本。

2020/04/15 17:03:56 CLSRSC-336: Successfully deconfigured Oracle Clusterware stack on this node

#若删除整个集群，在最后一个节点执行

#./rootcrs.pl -deconfig -force -lastnode
在删除的节点上执行删除软件命令操作d2，只删除本地节点2

su - grid

cd $ORACLE_HOME/deinstall

./deinstall -local

ASM configuration was not detectedin this Oracle home. Was ASM configured in this Oracle home (y|n) [n]: y

【asm 与grid安装在一起共享一个oracle_home]

ASM Diagnostic Destination : /u01/app/grid

ASM Diskgroups :

ASM diskstring : /dev/*_1*

Diskgroups will not be dropped

If you want to retain the existing diskgroups or if any of the information detected is incorrect, you can modify by entering 'y'.

Do you want to modify above information (y|n) [n]: n 【不修改上述对磁盘组的操作,不删除磁盘组]Oracle Grid Infrastructure Home is:

Oracle Home selected for deinstall is: /u01/app/19.0.0/grid

Inventory Location where the Oracle home registered is: /u01/app/oraInventory

Following Oracle Restart enabled listener(s) will be de-configured: ASMNET1LSNR_ASM,LISTENER,LISTENER_SCAN1

ASM instance will be de-configured from this Oracle home

Do you want to continue (y - yes, n - no)? [n]: y [确认清理上述资源】

####################### DEINSTALL CLEAN OPERATION SUMMARY #######################

Successfully detached Oracle home '/u01/app/19.0.0/grid' from the central inventory on the local node.

Failed to delete directory '/u01/app/19.0.0/grid' on the local node due to error : Either user has no permission

to delete or file is in use.

Review the permissions and manually delete '/u01/app/19.0.0/grid' on local node.

Failed to delete directory '/u01/app/oraInventory' on the local node due to error : Either user has no permission

to delete or file is in use.

Review the permissions and manually delete '/u01/app/oraInventory' on local node.

Oracle Universal Installer cleanup completed with errors.

Run 'rm -r /etc/oraInst.loc' as root on node(s) 'd2' at the end of the session.

Run 'rm -r /opt/ORCLfmap' as root on node(s) 'd2' at the end of the session.

Run 'rm -r /etc/oratab' as root on node(s) 'd2' at the end of the session.

Review the permissions and contents of '/u01/app/grid' on nodes(s) 'd2'.

If there are no Oracle home(s) associated with '/u01/app/grid', manually delete '/u01/app/grid' and its contents.

Oracle deinstall tool successfully cleaned up temporary directories.

root# 删除上述目录! 四个目录均递归删除！

删除CRS配置【存活的节点执行,已删除的crs都没了】

su - root

cd /u01/app/19.0.0/grid/bin

./crsctl delete node -n d2

[grid@d1:/u01/app/19.0.0/grid/crs/install]$ cluvfy stage -post nodedel -n 2 -verbose

Verifying Node Removal ...

Verifying CRS Integrity ...PASSED

Verifying Clusterware Version Consistency ...PASSED

Verifying Node Removal ...PASSED

Post-check for node removal was successful.

CVU operation performed: stage -post nodedel

Date: Apr 15, 2020 2:28:00 AM

CVU home: /u01/app/19.0.0/grid/

User: grid

发现存在一个vip 未被删除FAILED OVER,STABLE

# /u01/app/19.0.0/grid/bin/srvctl remove vip -vip d2 -force

二、添加节点

2.1添加Grid集群节点

前期准备工作，需要与安装RAC工作一样，本次环境都是好的【新增节点不需要再需要安装介质】

$ mkdir -p /u01/app/19.0.0/grid$ ssh d1 date;ssh d2 date; ssh d1-priv date;ssh d2-priv date;运行CVU程序

su-grid

cd ${ORACLE_HOME}/bin

cluvfy comp peer [-refnode ] -n node_list [-orainv orainventory_group] [-osdba osdba_group] [-verbose]

CVU用作检查与其他节点的兼容性的参考的节点。如果未指定此选项，则CVU报告节点列表中所有节点的值。

grid_node1$cluvfy comp peer-refnode d1 -n d2 -verbose

加节点(GI)

grid_node1$/u01/app/19.0.0/grid/addnode/addnode.sh -silent -ignoreSysPrereqs -ignorePrereqFailure "CLUSTER_NEW_NODES={d2}" "CLUSTER_NEW_PRIVATE_NODE_NAMES={d2-priv}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={d2-vip}"As a root user, execute the following script(s):1. /u01/app/oraInventory/orainstRoot.sh2. /u01/app/19.0.0/grid/root.sh

Execute/u01/app/oraInventory/orainstRoot.sh on the following nodes: [d2]

Execute/u01/app/19.0.0/grid/root.sh on the following nodes: [d2]

······

[root@d1~]# /u01/app/oraInventory/orainstRoot.sh

Changing permissions of/u01/app/oraInventory.

Adding read,write permissionsforgroup.

Removing read,write,execute permissionsforworld.

Changing groupname of/u01/app/oraInventory to oinstall.

The execution of the scriptiscomplete.

验证

$ cluvfy stage-post nodeadd -n d2

2.2添加Oracle软件

加节点(DB)

oracle_node1$/u03/app/oracle/product/19.0.0/db_1/addnode/addnode.sh -silent -ignoreSysPrereqs -ignorePrereqFailure "CLUSTER_NEW_NODES={d2}"As a root user, execute the following script(s):1. /u03/app/oracle/product/19.0.0/db_1/root.sh

Execute/u03/app/oracle/product/19.0.0/db_1/root.sh on the following nodes: [d2]

2.2添加DB

--Start the Instance on the New Node

The prerequisite steps to take prior to adding an instanceusing DBCA are asfollows:.

Ensure that all the existing instances have been startedusing an spfile fromthe shared location.

Verify that all the instances are listening on thedefaultgrid listener. The following command displays all the existing instances:

$ $CRS_HOME/bin/lsnrctl status

Run $ORACLE_HOME/bin/dbca fromany of the existing RAC instance Oracle homes.

Select"Oracle Real Application Clusters"database and click Next.

Select"Instance Management"and click Next.

Select"Add an Instance"and click Next.

Click Next to accept thedefault instance name (or change it, ifnecessary).

Check the summary window.

测试环境3套DB,启动单个实例后，静默方式添加节点2实例

$ srvctl start instance-d abc -instance abc1

$dbca-ignorePreReqs -ignorePrereqFailure -silent -addInstance -nodeName d2 -gdbName abc -instanceName abc2 -sysDBAUserName sys

-sysDBAPassword oracle

$ srvctl start database-d nocdd

$dbca-ignorePreReqs -ignorePrereqFailure -silent -addInstance -nodeName d2 -gdbName nocdd -instanceName nocdd2 -sysDBAUserName sys

-sysDBAPassword oracle

$ srvctl start database-d orcl

$dbca-ignorePreReqs -ignorePrereqFailure -silent -addInstance -nodeName d2 -gdbName orcl -instanceName orcl2 -sysDBAUserName sys

-sysDBAPassword oracle

DB_ALERT

2020-04-16T00:50:34.958060+08:00

CREATE SMALLFILE UNDO TABLESPACE "UNDOTBS2" DATAFILE SIZE 290M AUTOEXTEND ON NEXT 5120K MAXSIZE 32767M BLOCKSIZE 8192

2020-04-16T00:50:36.603592+08:00

Completed: CREATE SMALLFILE UNDO TABLESPACE "UNDOTBS2" DATAFILE SIZE 290M AUTOEXTEND ON NEXT 5120K MAXSIZE 32767M BLOCKSIZE 8192

ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 3 SIZE 200M,

GROUP 4 SIZE 200M

2020-04-16T00:50:39.079100+08:00

Completed: ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 3 SIZE 200M,

GROUP 4 SIZE 200M

ALTER DATABASE ENABLE PUBLIC THREAD 2

Completed: ALTER DATABASE ENABLE PUBLIC THREAD 2

2020-04-16T00:50:39.185364+08:00

ALTER SYSTEM SET instance_number=2 SCOPE=SPFILE SID='nocdd2';

2020-04-16T00:50:39.213198+08:00

ALTER SYSTEM SET thread=2 SCOPE=SPFILE SID='nocdd2';

2020-04-16T00:50:39.243485+08:00

ALTER SYSTEM SET undo_tablespace='UNDOTBS2' SCOPE=SPFILE SID='nocdd2';

2020-04-16T00:51:29.154327+08:00

补丁信息发现,添加节点补丁自动打上了！！！[grid@d1:/u01/app/19.0.0/grid/OPatch]$ ./opatch lsinventory

oracle ······

SQL> select ACTION,NAMESPACE,VERSION,BUNDLE_SERIES,COMMENTS FROM dba_registry_history;

ACTION NAMESPACE VERSION BUND COMMENTS

---------- ---------- -------------------- ---- ----------------------------------------

BOOTSTRAP DATAPATCH 19 RDBMS_19.5.0.0.0DBRUR_LINUX.X64_191213

RU_APPLY SERVER 19.0.0.0.0 Patch applied on 19.3.0.0.0: Release_Update - 190410122720