今天业务反馈有一台主机登入不上去，环境是RAC，只有一个节点重启了，另外一个节点正常。我同事比我先处理一段时间，导致了ocssd.log里面的日志没有拿下来(ohasd.log，ocssd.log这些日志产生的日志量比较大，不像DB的alter日志，当ocssd.log日志是会覆盖的，假设你9:00集群宕机的，当你10:00再去看日志会发现里面日志没有9:00日志了，都是最近的日志，这也是我为什么没有拿到ocssd.log里面发生故障点时间的日志)。同事确认过在主机发生重启的时候在ocsd.log里面没有因为网络问题导致节点驱逐导致脑裂的信息。

（1）先上主机，查看主机何时宕机重启的。

[root@zjhzbjwgzhzg01 cssd]# last reboot | head -1  --这条命令才看到主机重启的时间是08:52，这个时间是启动的时间，其实主机宕机的时间是比08:52早几分钟。
reboot   system boot  2.6.32-431.el6.x Wed Sep 12 08:52 - 10:30  (01:38)

（2）由于主机发生了重启，去查看操作系统让日志

操作系统日志：
ep 12 08:24:19 zjhzbjwgzhzg01 kernel: __ratelimit: 16 callbacks suppressed
Sep 12 08:24:19 zjhzbjwgzhzg01 kernel: [Hardware Error]: Machine check events logged  --这里可以看到在比主机重启的时间早一些的时候，radhat linux操作系统的主机已经开始报关于硬件方面的错误了。
Sep 12 08:24:23 zjhzbjwgzhzg01 kernel: [Hardware Error]: Machine check events logged
Sep 12 08:25:27 zjhzbjwgzhzg01 kernel: __ratelimit: 38 callbacks suppressed
Sep 12 08:25:27 zjhzbjwgzhzg01 kernel: [Hardware Error]: Machine check events logged
Sep 12 08:25:27 zjhzbjwgzhzg01 kernel: [Hardware Error]: Machine check events logged
Sep 12 08:26:00 zjhzbjwgzhzg01 mcelog: Corrected memory errors on page 595c01f000 exceed threshold 10 in 24h: 10 in 24h
Sep 12 08:26:00 zjhzbjwgzhzg01 mcelog: Location SOCKET:2 CHANNEL:0 DIMM:? []
Sep 12 08:26:00 zjhzbjwgzhzg01 mcelog: Offlining page 595c01f000
Sep 12 08:26:28 zjhzbjwgzhzg01 kernel: __ratelimit: 39 callbacks suppressed
Sep 12 08:26:28 zjhzbjwgzhzg01 kernel: [Hardware Error]: Machine check events logged
Sep 12 08:26:28 zjhzbjwgzhzg01 kernel: [Hardware Error]: Machine check events logged
Sep 12 08:27:31 zjhzbjwgzhzg01 kernel: __ratelimit: 88 callbacks suppressed
Sep 12 08:27:31 zjhzbjwgzhzg01 kernel: [Hardware Error]: Machine check events logged
Sep 12 08:27:31 zjhzbjwgzhzg01 kernel: [Hardware Error]: Machine check events logged
Sep 12 08:27:42 zjhzbjwgzhzg01 mcelog: Corrected memory errors on page 58dc5bf000 exceed threshold 10 in 24h: 10 in 24h
Sep 12 08:27:42 zjhzbjwgzhzg01 mcelog: Location SOCKET:2 CHANNEL:0 DIMM:? []
Sep 12 08:27:42 zjhzbjwgzhzg01 mcelog: Offlining page 58dc5bf000
Sep 12 08:27:42 zjhzbjwgzhzg01 mcelog: Corrected memory errors on page 591dd3f000 exceed threshold 10 in 24h: 10 in 24h
Sep 12 08:27:42 zjhzbjwgzhzg01 mcelog: Location SOCKET:2 CHANNEL:0 DIMM:? []
Sep 12 08:27:42 zjhzbjwgzhzg01 mcelog: Offlining page 591dd3f000
Sep 12 08:27:43 zjhzbjwgzhzg01 mcelog: Corrected memory errors on page 599c859000 exceed threshold 10 in 24h: 10 in 24h
Sep 12 08:27:43 zjhzbjwgzhzg01 mcelog: Location SOCKET:2 CHANNEL:0 DIMM:? []
Sep 12 08:27:43 zjhzbjwgzhzg01 mcelog: Offlining page 599c859000
Sep 12 08:27:43 zjhzbjwgzhzg01 mcelog: Corrected memory errors on page 599c858000 exceed threshold 10 in 24h: 10 in 24h
Sep 12 08:27:43 zjhzbjwgzhzg01 mcelog: Location SOCKET:2 CHANNEL:0 DIMM:? []
Sep 12 08:27:43 zjhzbjwgzhzg01 mcelog: Offlining page 599c858000
Sep 12 08:27:43 zjhzbjwgzhzg01 mcelog: Offlining page 599c858000 failed: Device or resource busy
Sep 12 08:27:43 zjhzbjwgzhzg01 kernel: soft offline: 0x599c858 page already poisoned
Sep 12 08:27:43 zjhzbjwgzhzg01 mcelog: Corrected memory errors on page 599c815000 exceed threshold 10 in 24h: 10 in 24h
Sep 12 08:27:43 zjhzbjwgzhzg01 mcelog: Location SOCKET:2 CHANNEL:0 DIMM:? []
Sep 12 08:27:43 zjhzbjwgzhzg01 mcelog: Offlining page 599c815000
Sep 12 08:27:43 zjhzbjwgzhzg01 mcelog: Offlining page 599c815000 failed: Device or resource busy
Sep 12 08:27:43 zjhzbjwgzhzg01 kernel: soft offline: 0x599c815 page already poisoned
Sep 12 08:27:44 zjhzbjwgzhzg01 mcelog: Corrected memory errors on page 599cc34000 exceed threshold 10 in 24h: 10 in 24h
Sep 12 08:27:44 zjhzbjwgzhzg01 mcelog: Location SOCKET:2 CHANNEL:0 DIMM:? []
Sep 12 08:27:44 zjhzbjwgzhzg01 mcelog: Offlining page 599cc34000
Sep 12 08:27:46 zjhzbjwgzhzg01 mcelog: Corrected memory errors on page 589d81d000 exceed threshold 10 in 24h: 10 in 24h
Sep 12 08:27:46 zjhzbjwgzhzg01 mcelog: Location SOCKET:2 CHANNEL:0 DIMM:? []
Sep 12 08:27:46 zjhzbjwgzhzg01 mcelog: Offlining page 589d81d000
Sep 12 08:27:53 zjhzbjwgzhzg01 mcelog: Corrected memory errors on page 599dd34000 exceed threshold 10 in 24h: 10 in 24h
Sep 12 08:27:53 zjhzbjwgzhzg01 mcelog: Location SOCKET:2 CHANNEL:0 DIMM:? []
--More--(4%)
Sep 12 08:35:06 zjhzbjwgzhzg01 mcelog: Location SOCKET:2 CHANNEL:0 DIMM:? []
Sep 12 08:35:06 zjhzbjwgzhzg01 mcelog: Offlining page 58ddd9c000
Sep 12 08:36:10 zjhzbjwgzhzg01 kernel: __ratelimit: 50 callbacks suppressed
Sep 12 08:36:10 zjhzbjwgzhzg01 kernel: [Hardware Error]: Machine check events lo
gged
Sep 12 08:36:11 zjhzbjwgzhzg01 kernel: [Hardware Error]: Machine check events lo
gged
Sep 12 08:36:45 zjhzbjwgzhzg01 mcelog: Corrected memory errors on page 591d45f00
0 exceed threshold 10 in 24h: 10 in 24h
Sep 12 08:36:45 zjhzbjwgzhzg01 mcelog: Location SOCKET:2 CHANNEL:0 DIMM:? []
Sep 12 08:36:45 zjhzbjwgzhzg01 mcelog: Offlining page 591d45f000
Sep 12 08:37:07 zjhzbjwgzhzg01 mcelog: Corrected memory errors on page 595c4df00
0 exceed threshold 10 in 24h: 10 in 24h
Sep 12 08:37:07 zjhzbjwgzhzg01 mcelog: Location SOCKET:2 CHANNEL:0 DIMM:? []
Sep 12 08:37:07 zjhzbjwgzhzg01 mcelog: Offlining page 595c4df000
Sep 12 08:37:14 zjhzbjwgzhzg01 kernel: __ratelimit: 61 callbacks suppressed
Sep 12 08:37:14 zjhzbjwgzhzg01 kernel: [Hardware Error]: Machine check events lo
gged
Sep 12 08:37:14 zjhzbjwgzhzg01 kernel: [Hardware Error]: Machine check events lo
gged
Sep 12 08:37:25 zjhzbjwgzhzg01 mcelog: Corrected memory errors on page 589d0ff00 --这里报错是内存报错
0 exceed threshold 10 in 24h: 10 in 24h
Sep 12 08:37:25 zjhzbjwgzhzg01 mcelog: Location SOCKET:2 CHANNEL:0 DIMM:? []
Sep 12 08:37:25 zjhzbjwgzhzg01 mcelog: Offlining page 589d0ff000
Sep 12 08:38:14 zjhzbjwgzhzg01 kernel: __ratelimit: 55 callbacks suppressed
Sep 12 08:38:14 zjhzbjwgzhzg01 kernel: [Hardware Error]: Machine check events lo
gged
Sep 12 08:38:14 zjhzbjwgzhzg01 kernel: [Hardware Error]: Machine check events lo
gged
Sep 12 08:39:15 zjhzbjwgzhzg01 kernel: __ratelimit: 72 callbacks suppressed
Sep 12 08:39:15 zjhzbjwgzhzg01 kernel: [Hardware Error]: Machine check events lo
gged
Sep 12 08:39:16 zjhzbjwgzhzg01 kernel: [Hardware Error]: Machine check events lo
gged
Sep 12 08:39:56 zjhzbjwgzhzg01 mcelog: Corrected memory errors on page 589c93f00
0 exceed threshold 10 in 24h: 10 in 24h
Sep 12 08:39:56 zjhzbjwgzhzg01 mcelog: Location SOCKET:2 CHANNEL:0 DIMM:? []
Sep 12 08:39:56 zjhzbjwgzhzg01 mcelog: Offlining page 589c93f000
Sep 12 08:40:19 zjhzbjwgzhzg01 kernel: __ratelimit: 56 callbacks suppressed
Sep 12 08:40:19 zjhzbjwgzhzg01 kernel: [Hardware Error]: Machine check events lo
gged
Sep 12 08:40:19 zjhzbjwgzhzg01 kernel: [Hardware Error]: Machine check events lo
gged
Sep 12 08:40:22 zjhzbjwgzhzg01 mcelog: Corrected memory errors on page 589ccff00
0 exceed threshold 10 in 24h: 10 in 24h
Sep 12 08:40:22 zjhzbjwgzhzg01 mcelog: Location SOCKET:2 CHANNEL:0 DIMM:? []
Sep 12 08:40:22 zjhzbjwgzhzg01 mcelog: Offlining page 589ccff000
Sep 12 08:40:25 zjhzbjwgzhzg01 mcelog: Corrected memory errors on page 595cd9f00
0 exceed threshold 10 in 24h: 10 in 24h
Sep 12 08:40:25 zjhzbjwgzhzg01 mcelog: Location SOCKET:2 CHANNEL:0 DIMM:? []
Sep 12 08:40:25 zjhzbjwgzhzg01 mcelog: Offlining page 595cd9f000
Sep 12 08:41:00 zjhzbjwgzhzg01 mcelog: Corrected memory errors on page 5c9cdb600
0 exceed threshold 10 in 24h: 10 in 24h
Sep 12 08:41:00 zjhzbjwgzhzg01 mcelog: Location SOCKET:2 CHANNEL:0 DIMM:? []
Sep 12 08:41:00 zjhzbjwgzhzg01 mcelog: Offlining page 5c9cdb6000   --这里可以看到日志断了，08:42-08:51分钟的日志看不到了,可以判断这段时间主机是宕机了。所以这里可以将故障的时间定为到08:40，如果要去查看ohasd.log,ocrsd.log,ocssd.log日志就应该将注意力放在08:40之前一段时间，而不是像无头苍蝇一样将大部分日志看一遍。
Sep 12 08:52:53 zjhzbjwgzhzg01 kernel: imklog 5.8.10, log source = /proc/kmsg st
arted.
Sep 12 08:52:53 zjhzbjwgzhzg01 rsyslogd: [origin software="rsyslogd" swVersion="
5.8.10" x-pid="7988" x-info="http://www.rsyslog.com"] start
Sep 12 08:52:53 zjhzbjwgzhzg01 kernel: Initializing cgroup subsys cpuset
Sep 12 08:52:53 zjhzbjwgzhzg01 kernel: Initializing cgroup subsys cpu
Sep 12 08:52:53 zjhzbjwgzhzg01 kernel: Linux version 2.6.32-431.el6.x86_64 (mock
build@x86-023.build.eng.bos.redhat.com) (gcc version 4.4.7 20120313 (Red Hat 4.4
.7-4) (GCC) ) #1 SMP Sun Nov 10 22:19:54 EST 2013
Sep 12 08:52:53 zjhzbjwgzhzg01 kernel: Command line: ro root=UUID=40b7f075-d922-
4995-aebe-de61630e7037 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=la
tarcyrheb-sun16 crashkernel=auto  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quie
t nohz=0ff intel_idle.max_cstate=0 processor.max_cstate=0 idle=poll transparent_
hugepage=never
Sep 12 08:52:53 zjhzbjwgzhzg01 kernel: KERNEL supported cpus:
Sep 12 08:52:53 zjhzbjwgzhzg01 kernel:  Intel GenuineIntel
Sep 12 08:52:53 zjhzbjwgzhzg01 kernel:  AMD AuthenticAMD
Sep 12 08:52:53 zjhzbjwgzhzg01 kernel:  Centaur CentaurHauls
Sep 12 08:52:53 zjhzbjwgzhzg01 kernel: BIOS-provided physical RAM map:
Sep 12 08:52:53 zjhzbjwgzhzg01 kernel: BIOS-e820: 0000000000000000 - 00000000000
9a000 (usable)
Sep 12 08:52:53 zjhzbjwgzhzg01 kernel: BIOS-e820: 000000000009a000 - 00000000000
a0000 (reserved)
Sep 12 08:52:53 zjhzbjwgzhzg01 kernel: BIOS-e820: 00000000000e0000 - 00000000001
00000 (reserved)
Sep 12 08:52:53 zjhzbjwgzhzg01 kernel: BIOS-e820: 0000000000100000 - 00000000755
07000 (usable)
Sep 12 08:52:53 zjhzbjwgzhzg01 kernel: BIOS-e820: 0000000075507000 - 0000000075c
bb000 (reserved)
Sep 12 08:52:53 zjhzbjwgzhzg01 kernel: BIOS-e820: 0000000075cbb000 - 0000000075d
bb000 (ACPI data)

（3）查看节点数据库日志（上面通过查看同事通过观察ocssd.log内容反馈不是网络导致的集群脑裂重启，可以判断网络这一块不是导致主机重启的原因，上面操作系统日志报错的是硬件和内存方面的错误，可能是主机重启的原因，但是也有可能是数据库层面的导致的，所以看看DB日志）
Errors in file /opt/oracle/oracle/diag/rdbms/sdh/sdh1/trace/sdh1_j003_91488.trc:
ORA-12012: error on auto execute of job 90926
ORA-01403: no data found
ORA-06512: at "IRM.PKG_DATACOMPARE_TASKEXECUTE", line 11
ORA-06512: at line 1
Errors in file /opt/oracle/oracle/diag/rdbms/sdh/sdh1/trace/sdh1_j002_91432.trc:
ORA-12012: error on auto execute of job 90927
ORA-01403: no data found
ORA-06512: at "IRM.PKG_DATACOMPARE_TASKEXECUTE", line 11
ORA-06512: at line 1
Wed Sep 12 08:31:39 2018
Errors in file /opt/oracle/oracle/diag/rdbms/sdh/sdh1/trace/sdh1_j004_92891.trc:
ORA-12012: error on auto execute of job 90929
ORA-01403: no data found
ORA-06512: at "IRM.PKG_DATACOMPARE_TASKEXECUTE", line 11
ORA-06512: at line 1
Wed Sep 12 08:38:45 2018
Thread 1 advanced to log sequence 338086 (LGWR switch)
  Current log# 1 seq# 338086 mem# 0: +SDH_SYS_DG/sdh/onlinelog/group_1.329.879523497
Wed Sep 12 08:38:45 2018
LNS: Standby redo logfile selected for thread 1 sequence 338086 for destination LOG_ARCHIVE_DEST_2
Wed Sep 12 08:38:50 2018  --可以看到在08:38-09:14之间的日志没有了，这段时间数据库是宕机的
Archived Log entry 1126324 added for thread 1 sequence 338085 ID 0xd26cc373 dest 1:
Wed Sep 12 09:14:56 2018
Starting ORACLE instance (normal) --同事手动将数据库拉起
************************ Large Pages Information *******************
Per process system memlock (soft) limit = UNLIMITED
 
Large page usage restricted to processor group "sys"
 
Total Shared Global Region in Large Pages = 200 GB (100%)
 
WARNING: --数据库启动的时候又报出来内存方面错误，之前主机也报出内存方面错误。
  The parameter _linux_prepage_large_pages is explicitly disabled. 
  Oracle strongly recommends setting the _linux_prepage_large_pages 
  parameter since the instance  is running in a Processor Group. If there is 
  insufficient large page memory, instance may encounter SIGBUS error 
  and may terminate abnormally.
 
Large Pages used by this instance: 102401 (200 GB)
Large Pages unused in Processor Group sys = 145081 (283 GB)
Large Pages configured in Processor Group sys = 153496 (300 GB)
Large Page size = 2048 KB
********************************************************************
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Initial number of CPU is 88
Number of processor cores in the system is 44
Number of processor sockets in the system is 4
Private Interface 'Bond1:1' configured from GPnP for use as a private interconnect.
  [name='Bond1:1', type=1, ip=169.254.185.193, mac=a0-00-01-00-fe-80-00-00-00-00-00-00-64-3e-00-00-00-00-00-00, net=169.254.0.0/16, mask=255.255.0.0, use=haip:cluster_interconnect/62]
Public Interface 'vlan304' configured from GPnP for use as a public interface.
  [name='vlan304', type=1, ip=10.212.252.84, mac=e0-97-96-06-e7-a5, net=10.212.252.80/28, mask=255.255.255.240, use=public/1]
Public Interface 'vlan304:1' configured from GPnP for use as a public interface.
  [name='vlan304:1', type=1, ip=10.212.252.87, mac=e0-97-96-06-e7-a5, net=10.212.252.80/28, mask=255.255.255.240, use=public/1]
Picked latch-free SCN scheme 3
Autotune of undo retention is turned off. 
LICENSE_MAX_USERS = 0
SYS auditing is disabled
NUMA system with 4 nodes detected
Starting up:  
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options.
ORACLE_HOME = /opt/oracle/oracle/product/11.2.0/db_1
System name:    Linux
Node name:      zjhzbjwgzhzg01
Release:        2.6.32-431.el6.x86_64
Version:        #1 SMP Sun Nov 10 22:19:54 EST 2013
Machine:        x86_64
Using parameter settings in server-side pfile /opt/oracle/oracle/product/11.2.0/db_1/dbs/initsdh1.ora
System parameters with non-default values:
  processes                = 3000
  sessions                 = 4576
  event                    = "10949 trace name context forever, level 1"
  sga_max_size             = 200G
  shared_pool_size         = 20G
  large_pool_size          = 3584M
  java_pool_size           = 3584M
  streams_pool_size        = 512M
  spfile                   = "+SDH_SYS_DG/sdh/spfilesdh.ora

（4）查看ASM日志，其实ASM磁盘有问题并不会导致集群宕了导致主机重启，这里只是为了看看

ASM日志

Wed Sep 12 08:56:59 2018
NOTE: No asm libraries found in the system   --开始重启ASM实例，可以看到在主机重启后，集群也自启动拉起了ASM。不出意外去看看crsd.log
MEMORY_TARGET defaulting to 1128267776.
* instance_number obtained from CSS = 1, checking for the existence of node 0... 
* node 0 does not exist. instance_number = 1 
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Initial number of CPU is 88
Number of processor cores in the system is 44
Number of processor sockets in the system is 4
Private Interface 'Bond1:1' configured from GPnP for use as a private interconnect.
  [name='Bond1:1', type=1, ip=169.254.185.193, mac=a0-00-01-00-fe-80-00-00-00-00-00-00-64-3e-00-00-00-00-00-00, net=169.254.0.0/16, mask=255.255.0.0, use=haip:cluster_interconnect/62]
Public Interface 'vlan304' configured from GPnP for use as a public interface.
  [name='vlan304', type=1, ip=10.212.252.84, mac=e0-97-96-06-e7-a5, net=10.212.252.80/28, mask=255.255.255.240, use=public/1]
Picked latch-free SCN scheme 3
Using LOG_ARCHIVE_DEST_1 parameter default value as /opt/oracle/grid/11.2.0/grid/dbs/arch
Autotune of undo retention is turned on. 
LICENSE_MAX_USERS = 0
SYS auditing is disabled
NOTE: Volume support  enabled
NUMA system with 4 nodes detected
Starting up:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options.
ORACLE_HOME = /opt/oracle/grid/11.2.0/grid
System name:    Linux
Node name:      zjhzbjwgzhzg01
Release:        2.6.32-431.el6.x86_64
Version:        #1 SMP Sun Nov 10 22:19:54 EST 2013
Machine:        x86_64
Using parameter settings in server-side spfile +OCR_VOTE/zjhzbjw-cluster/asmparameterfile/registry.253.876056333
System parameters with non-default values:
  large_pool_size          = 12M
  instance_type            = "asm"
  remote_login_passwordfile= "EXCLUSIVE"
  asm_diskstring           = "/dev/asmdisk/*"
  asm_diskgroups           = "OCR_VOTE"
  asm_diskgroups           = "SDH_SYS_DG"
  asm_diskgroups           = "ARCHIVE_DG"
  asm_diskgroups           = "WPS_SYS_DG"
  asm_power_limit          = 8
  diagnostic_dest          = "/opt/oracle/grid/grid"
Cluster communication is configured to use the following interface(s) for this instance
  169.254.185.193
cluster interconnect IPC version:Oracle UDP/IP (generic)

（5）查看CRS的日志

crsd日志：整个日志可以看到08:57的时候。CRS组件重启了

2018-09-12 08:37:37.898: [UiServer][1156998912]{1:47993:28042} Done for ctx=0x7fcfbc0088a0
2018-09-12 08:57:08.163: [ CRSMAIN][1714153248] First attempt: init CSS context succeeded.
[  clsdmt][1707702016]Listening to (ADDRESS=(PROTOCOL=ipc)(KEY=zjhzbjwgzhzg01DBG_CRSD))
2018-09-12 08:57:08.165: [  clsdmt][1707702016]PID for the Process [35434], connkey 1 
2018-09-12 08:57:08.165: [  clsdmt][1707702016]Creating PID [35434] file for home /opt/oracle/grid/11.2.0/grid host zjhzbjwgzhzg01 bin crs to /opt/oracle/gri
d/11.2.0/grid/crs/init/
2018-09-12 08:57:08.165: [  clsdmt][1707702016]Writing PID [35434] to the file [/opt/oracle/grid/11.2.0/grid/crs/init/zjhzbjwgzhzg01.pid] 
2018-09-12 08:57:08.607: [ CRSMAIN][1707702016] Policy Engine is not initialized yet!
2018-09-12 08:57:08.607: [ CRSMAIN][1714153248] CRS Daemon Starting
2018-09-12 08:57:08.608: [    CRSD][1714153248] Logging level for Module: allcomp  0
2018-09-12 08:57:08.608: [    CRSD][1714153248] Logging level for Module: default  0
2018-09-12 08:57:08.608: [    CRSD][1714153248] Logging level for Module: COMMCRS  0
2018-09-12 08:57:08.608: [    CRSD][1714153248] Logging level for Module: COMMNS  0
2018-09-12 08:57:08.608: [    CRSD][1714153248] Logging level for Module: CSSCLNT  0
2018-09-12 08:57:08.608: [    CRSD][1714153248] Logging level for Module: GIPCLIB  0
2018-09-12 08:57:08.608: [    CRSD][1714153248] Logging level for Module: GIPCXBAD  0
2018-09-12 08:57:08.608: [    CRSD][1714153248] Logging level for Module: GIPCLXPT  0
2018-09-12 08:57:08.608: [    CRSD][1714153248] Logging level for Module: GIPCUNDE  0
2018-09-12 08:57:08.608: [    CRSD][1714153248] Logging level for Module: GIPC  0
2018-09-12 08:57:08.608: [    CRSD][1714153248] Logging level for Module: GIPCGEN  0
2018-09-12 08:57:08.608: [    CRSD][1714153248] Logging level for Module: GIPCTRAC  0
2018-09-12 08:57:08.608: [    CRSD][1714153248] Logging level for Module: GIPCWAIT  0
2018-09-12 08:57:08.608: [    CRSD][1714153248] Logging level for Module: GIPCXCPT  0
2018-09-12 08:57:08.608: [    CRSD][1714153248] Logging level for Module: GIPCOSD  0
2018-09-12 08:57:08.608: [    CRSD][1714153248] Logging level for Module: GIPCBASE  0
2018-09-12 08:57:08.608: [    CRSD][1714153248] Logging level for Module: GIPCCLSA  0
2018-09-12 08:57:08.608: [    CRSD][1714153248] Logging level for Module: GIPCCLSC  0
2018-09-12 08:57:08.608: [    CRSD][1714153248] Logging level for Module: GIPCEXMP  0
2018-09-12 08:57:08.608: [    CRSD][1714153248] Logging level for Module: GIPCGMOD  0
2018-09-12 08:57:08.608: [    CRSD][1714153248] Logging level for Module: GIPCHEAD  0
2018-09-12 08:57:08.608: [    CRSD][1714153248] Logging level for Module: GIPCMUX  0
2018-09-12 08:57:08.608: [    CRSD][1714153248] Logging level for Module: GIPCNET  0
2018-09-12 08:57:08.608: [    CRSD][1714153248] Logging level for Module: GIPCNULL  0
2018-09-12 08:57:08.608: [    CRSD][1714153248] Logging level for Module: GIPCPKT  0
2018-09-12 08:57:08.608: [    CRSD][1714153248] Logging level for Module: GIPCSMEM  0
2018-09-12 08:57:08.608: [    CRSD][1714153248] Logging level for Module: GIPCHAUP  0
2018-09-12 08:57:08.608: [    CRSD][1714153248] Logging level for Module: GIPCHALO  0
2018-09-12 08:57:08.608: [    CRSD][1714153248] Logging level for Module: GIPCHTHR  0
2018-09-12 08:57:08.608: [    CRSD][1714153248] Logging level for Module: GIPCHGEN  0
2018-09-12 08:57:08.608: [    CRSD][1714153248] Logging level for Module: GIPCHLCK  0
2018-09-12 08:57:08.608: [    CRSD][1714153248] Logging level for Module: GIPCHDEM  0
2018-09-12 08:57:08.608: [    CRSD][1714153248] Logging level for Module: GIPCHWRK  0
2018-09-12 08:57:08.609: [    CRSD][1714153248] Logging level for Module: CRSMAIN  0
2018-09-12 08:57:08.609: [    CRSD][1714153248] Logging level for Module: clsdmt  0
2018-09-12 08:57:08.609: [    CRSD][1714153248] Logging level for Module: clsdms  0
2018-09-12 08:57:08.609: [    CRSD][1714153248] Logging level for Module: CRSUI  0
2018-09-12 08:57:08.609: [    CRSD][1714153248] Logging level for Module: CRSCOMM  0
2018-09-12 08:57:08.609: [    CRSD][1714153248] Logging level for Module: CRSRTI  0
2018-09-12 08:57:08.609: [    CRSD][1714153248] Logging level for Module: CRSPLACE  0
2018-09-12 08:57:08.609: [    CRSD][1714153248] Logging level for Module: CRSAPP  0
2018-09-12 08:57:08.609: [    CRSD][1714153248] Logging level for Module: CRSRES  0
2018-09-12 08:57:08.609: [    CRSD][1714153248] Logging level for Module: CRSTIMER  0
2018-09-12 08:57:08.609: [    CRSD][1714153248] Logging level for Module: CRSEVT  0
2018-09-12 08:57:08.609: [    CRSD][1714153248] Logging level for Module: CRSD  0
2018-09-12 08:57:08.609: [    CRSD][1714153248] Logging level for Module: CLUCLS  0
2018-09-12 08:57:08.609: [    CRSD][1714153248] Logging level for Module: CLSVER  0
2018-09-12 08:57:08.609: [    CRSD][1714153248] Logging level for Module: CLSFRAME  0
2018-09-12 08:57:08.609: [    CRSD][1714153248] Logging level for Module: CRSPE  0
2018-09-12 08:57:08.609: [    CRSD][1714153248] Logging level for Module: CRSSE  0
2018-09-12 08:57:08.609: [    CRSD][1714153248] Logging level for Module: CRSRPT  0
2018-09-12 08:57:08.609: [    CRSD][1714153248] Logging level for Module: CRSOCR  0
2018-09-12 08:57:08.609: [    CRSD][1714153248] Logging level for Module: UiServer  0
2018-09-12 08:57:08.609: [    CRSD][1714153248] Logging level for Module: AGFW  0
2018-09-12 08:57:08.609: [    CRSD][1714153248] Logging level for Module: SuiteTes  1
2018-09-12 08:57:08.609: [    CRSD][1714153248] Logging level for Module: CRSSHARE  1
2018-09-12 08:57:08.609: [    CRSD][1714153248] Logging level for Module: CRSSEC  0
2018-09-12 08:57:08.609: [    CRSD][1714153248] Logging level for Module: CRSCCL  0
2018-09-12 08:57:08.609: [ CRSMAIN][1707702016] Policy Engine is not initialized yet!
2018-09-12 08:57:08.609: [    CRSD][1714153248] Logging level for Module: CRSCEVT  0
2018-09-12 08:57:08.609: [    CRSD][1714153248] Logging level for Module: AGENT  1
2018-09-12 08:57:08.609: [    CRSD][1714153248] Logging level for Module: OCRAPI  1
2018-09-12 08:57:08.609: [    CRSD][1714153248] Logging level for Module: OCRCLI  1
2018-09-12 08:57:08.609: [    CRSD][1714153248] Logging level for Module: OCRSRV  1
2018-09-12 08:57:08.609: [    CRSD][1714153248] Logging level for Module: OCRMAS  1
2018-09-12 08:57:08.609: [    CRSD][1714153248] Logging level for Module: OCRMSG  1
2018-09-12 08:57:08.609: [    CRSD][1714153248] Logging level for Module: OCRCAC  1
2018-09-12 08:57:08.609: [    CRSD][1714153248] Logging level for Module: OCRRAW  1
2018-09-12 08:57:08.609: [    CRSD][1714153248] Logging level for Module: OCRUTL  1
2018-09-12 08:57:08.609: [    CRSD][1714153248] Logging level for Module: OCROSD  1
2018-09-12 08:57:08.609: [    CRSD][1714153248] Logging level for Module: OCRASM  1
2018-09-12 08:57:08.609: [ CRSMAIN][1714153248] Checking the OCR device
2018-09-12 08:57:08.610: [ CRSMAIN][1714153248] Sync-up with OCR
2018-09-12 08:57:08.610: [ CRSMAIN][1714153248] Connecting to the CSS Daemon
2018-09-12 08:57:08.610: [ CRSMAIN][1714153248] Getting local node number
2018-09-12 08:57:08.610: [ CRSMAIN][1714153248] Initializing OCR
[   CLWAL][1714153248]clsw_Initialize: OLR initlevel [70000]
2018-09-12 08:57:09.198: [  OCRRAW][1714153248]proprioo: for disk 0 (+OCR_VOTE), id match (1), total id sets, (2) need recover (0), my votes (2), total votes
 (2), commit_lsn (1469), lsn (1469)
2018-09-12 08:57:09.198: [  OCRRAW][1714153248]proprioo: my id set: (760227868, 1028247821, 0, 0, 0)
2018-09-12 08:57:09.198: [  OCRRAW][1714153248]proprioo: 1st set: (340549372, 760227868, 0, 0, 0)
2018-09-12 08:57:09.198: [  OCRRAW][1714153248]proprioo: 2nd set: (760227868, 1028247821, 0, 0, 0)
2018-09-12 08:57:09.207: [  OCRSRV][1714153248]th_init: Successfully retrieved CSS misscount [31].
2018-09-12 08:57:09.207: [  OCRSRV][1714153248]th_init: Successfully query CLSS mode [3].
2018-09-12 08:57:09.208: [  OCRSRV][1714153248]th_init:1: FROM PUBDATA Node num [2]Remote Listening Port [0] Cache invalidation port [0]
2018-09-12 08:57:09.208: [  OCRSRV][1714153248]th_init:1.1: FROM PUBDATA Node num [2]CLSC Private IP or GIPC connect string [gipcha<zjhzbjwgzhzg02><467b-167f
-bbd0-bd81><b6b7-5893-3f00-7370>]

通过上面可以判断出可能是由于内存问题和硬件问题导致的主机重启，和主机那边的人联系还真是内存条有问题导致整个主机的重启，同时导致集群的重启。

总结：当集群发生故障可以是硬件方面的，可能是等待事件导致CPU过高导致集群重启，也可能是掉盘导致的数据库挂了但是集群正常，这方面的内容要通过日志才可以判断，关键还是定位故障发生的时间，通过时间点日志定位故障。

RAC 主机内存条故障导致的一个节点重启相关推荐

1. oracle12c rac重启,Oracle12.2 RAC其中一个节点重启

   oracle12.2 RAC.Oracle Linux7.4 节点2自动重启,部分日志信息 2020-07-15 09:52:43.893 [OCTSSD(17627)]CRS-2411: The C ...

2. oracle重启一个节点集群,帮忙分析一例数据库两节点集群每隔几个月节点重启

   本帖最后由 鸣雏之叶1 于 2015-4-2 15:13 编辑 数据库版本:10204 操作系统版本:红帽4.7 64位 遇到一台双节点的rac,间隔几个月就会因为心跳原因导致某一个节点重启(随机), ...

3. 建立一个带头结点的线性链表,用以存放输入的二进制数,链表的每一个节点的data域存放一个二进制位。并在此链表上实现对二进制数加1的运算；

   1.题目:建立一个带头结点的线性链表,用以存放输入的二进制数,链表的每一个节点的data域存放一个二进制位.并在此链表上实现对二进制数加1的运算: 部分函数调用参考:https://blog.csdn ...

4. oracle双机切换 无法挂载,Linux下Oracle RAC一个节点宕机导致共享存储无法挂载的故障排除...

   环境:两台HP ML570 Linux AS4.5 Oracle 10g两台Server做了Oracle的RAC,通过SAN Switch连接到HP MSA1000故障现象:因为其中一个Ora 环境: ...

5. oracle rac节点重启,oracle RAC一个节点频繁重启解决

   oracle RAC一个节点频繁重启解决 类别:Oracle数据库   作者:码皇   来源:hijk139的专栏     点击: oracle RAC一个节点频繁重启解决故障现象:2011年的一次问 ...

6. Oracle显示表裂开,【案例】Oracle RAC脑裂导致节点重启原因分析

   天萃荷净 Oracle研究中心案例分析:运维DBA反映Oracle RAC重启,分析原因为脑裂导致,结合日志分析产生原因. 本站文章除注明转载外,均为本站原创: 转载自love wife & ...

7. Oracle RAC一节点宕机导致另一节点HANG的问题分析

   正所谓"福无双至,祸不单行",生产上有套2节点Oracle 11.2.0.4数据库,其中2节点因硬件故障宕机,1节点去HANG住了.我们一起来分析这起故障. 凌晨4点半,值班同时电 ...

8. oracle10g cssd日志,【案例】Oracle CSSD进程HANG导致RAC节点重启原因分析笔记

   [案例]Oracle CSSD进程HANG导致RAC节点重启原因分析笔记 时间:2016-11-04 19:20   来源:Oracle研究中心   作者:HTZ   点击: 次 天萃荷净 Oracl ...

9. 11gR2 RAC时间同异常导致节点down掉问题处理

   实验环境下11204的RAC环境,出现了一个节点DOWN掉的问题.检查日志信息后,在otcssd日志信息发现如下信息: 2016-01-17 23:15:20.564: [    CTSS][1175 ...

最新文章

1. 2014 WAP校园招聘笔试题
2. python自动化上传图片_接口自动化之Python3_Requests之上传头像
3. python 并行计算 multiprocessing、multiprocess模块 简介
4. 给Dev-Cpp开挂
5. [js] 解释下深度优先遍历和广度优先遍历的区别及如何实现
6. java基础—面向对象——变量的区别
7. 单机 mysql 复制_MySQL单机复制
8. C++中STL-stack的用法
9. HTML5新增input表单（HTML5）
10. Python Flask Web 第九课 —— flask 扩展小结
11. 功夫小子实践开发-基本工具类的分析和实现
12. MSP430+CC2500低功耗注意细节
13. intelliJ IDEA 2017 破解方法
14. matlab分数阶微分算子,数字图像处理以及数值运算中6种经典的分数阶微分掩模（分数阶微分算子）...
15. Python基础 | 快速实现label_to_index
16. HTML语言代码参考(中英文对照)
17. 上海“富爸爸” 神奇投资之路
18. HTML+CSS制作知乎登录页面
19. 1024. 视频拼接
20. 骂人的到底是些什么人

热门文章

1. 铜陵C语言培训,铜陵学院c语言程序设计报告答案
2. IP包流量分析(一)（python+pypcap+dkpt）
3. python匿名函数lambda_Python中什么情况下应该使用匿名函数lambda
4. python脱离pc自动化_Android手机脱离电脑直接运行UIAutomator2
5. Lattice ddr3教程全攻略之仿真篇
6. 【笔记】DeepIGeoS: A Deep Interactive Geodesic Framework for Medical Image Segmentation
7. 2021年下半年山东软考准考证打印时间
8. 如何命名计算机快捷键,重命名的快捷键,教您重命名的快捷键是什么
9. 深度神经网络及目标检测学习笔记
10. Appium环境搭建之Android SDK及模拟器的下载配置