Oracle NFS异常导致RAC集群宕机排查

本套RAC环境采用NFS做RAC共享存储.

dbca安装实例时,安装中途集群宕机,记录一下。

首先查看dbca安装日志:

[root@rac1 ~]# tail  -100f /oracle/app/cfgtoollogs/dbca/orcl/trace.log_2021-01-19_05-50-20PM
[progressPage.flowWorker] [ 2021-01-19 17:57:49.336 CST ] [InstanceStepOPS.executeImpl:1213]  create password file on FS
[progressPage.flowWorker] [ 2021-01-19 17:57:49.336 CST ] [InstanceStepOPS.executeImpl:1214]  Password file location prior to parsing +DATA/ORCL/orapworcl
[progressPage.flowWorker] [ 2021-01-19 17:57:49.338 CST ] [InstanceStepOPS.executeImpl:1216]  Password file location prior to parsing +DATA
[progressPage.flowWorker] [ 2021-01-19 17:57:49.339 CST ] [OracleHome.getVersion:1066]  OracleHome.getVersion called.  Current Version: null
[progressPage.flowWorker] [ 2021-01-19 17:57:49.341 CST ] [InventoryUtil.getOUIInvSession:349]  setting OUI READ level to ACCESSLEVEL_READ_LOCKLESS
[progressPage.flowWorker] [ 2021-01-19 17:57:49.342 CST ] [OracleHome.getVersion:1085]  Homeinfo /oracle/app/product/19.3.0/db_1,2
[progressPage.flowWorker] [ 2021-01-19 17:57:49.375 CST ] [OracleHome.getVersion:1113]  OracleHome.server.getVersion Version: 19.0.0.0.0
[progressPage.flowWorker] [ 2021-01-19 17:57:49.375 CST ] [OracleHome.getVersion:1134]  Current Version From Inventory: 19.0.0.0.0
[progressPage.flowWorker] [ 2021-01-19 17:57:49.375 CST ] [OracleHome.getVersion:1066]  OracleHome.getVersion called.  Current Version: 19.0.0.0.0
[progressPage.flowWorker] [ 2021-01-19 17:57:49.376 CST ] [CommonUtils.getPasswordFileCreateCmd:1467]  /oracle/app/product/19.3.0/db_1/bin/orapwd
[progressPage.flowWorker] [ 2021-01-19 17:57:49.376 CST ] [CommonUtils.getPasswordFileCreateCmd:1467]  file=+DATA
[progressPage.flowWorker] [ 2021-01-19 17:57:49.376 CST ] [CommonUtils.getPasswordFileCreateCmd:1467]  force=y
[progressPage.flowWorker] [ 2021-01-19 17:57:49.376 CST ] [CommonUtils.getPasswordFileCreateCmd:1467]  format=12
[progressPage.flowWorker] [ 2021-01-19 17:57:49.376 CST ] [CommonUtils.getPasswordFileCreateCmd:1467]  dbuniquename=orcl
[progressPage.flowWorker] [ 2021-01-19 17:57:49.376 CST ] [OsUtilsBase.execProg:2173]  beginning execProg with input array.
[progressPage.flowWorker] [ 2021-01-19 18:13:24.982 CST ] [OsUtilsBase.execProg:2210]  finished execProg with input array. Status:255
[progressPage.flowWorker] [ 2021-01-19 18:13:24.984 CST ] [OsUtilsBase.execProg:2213]  ExecProg /oracle/app/product/19.3.0/db_1/bin/orapwd failed. Std Err -  
 Std Out 

Enter password for SYS: 

OPW-00010: Could not create the password file.
ORA-03113: end-of-file on communication channel
Process ID: 18252
Session ID: 1223 Serial number: 12262

查看RAC集群日志:

2021-01-19 18:06:07.435 [OCSSD(4162)]CRS-1615: No I/O has completed after 50% of the maximum interval. If this persists, voting file /data/RAC1/ocrvote1 will be considered not funct
ional in 99320 milliseconds.
...
2021-01-19 18:12:31.550 [OCSSD(4162)]CRS-1614: No I/O has completed after 75% of the maximum interval. If this persists, voting file /data/RAC1/ocrvote1 will be considered not funct
ional in 49280 milliseconds.
...
2021-01-19 18:12:57.438 [OCSSD(4162)]CRS-1625: Node rac2, number 2, was shut down
2021-01-19 18:12:57.485 [OCSSD(4162)]CRS-1601: CSSD Reconfiguration complete. Active nodes are rac1 .
2021-01-19 18:12:57.497 [CRSD(4796)]CRS-5504: Node down event reported for node 'rac2'.
2021-01-19 18:12:57.928 [OCSSD(4162)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvDiskPingThread_0 not scheduled for 176100 msecs.
2021-01-19 18:13:01.550 [OCSSD(4162)]CRS-1613: No I/O has completed after 90% of the maximum interval. If this persists, voting file /data/RAC1/ocrvote1 will be considered not funct
ional in 19280 milliseconds.
2021-01-19 18:13:01.930 [OCSSD(4162)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvWorkerThread_0 not scheduled for 383910 msecs.
2021-01-19 18:13:05.932 [OCSSD(4162)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvDiskPingThread_0 not scheduled for 184100 msecs.
2021-01-19 18:13:09.933 [OCSSD(4162)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvWorkerThread_0 not scheduled for 391910 msecs.
2021-01-19 18:13:13.935 [OCSSD(4162)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvDiskPingThread_0 not scheduled for 192110 msecs.
2021-01-19 18:13:17.937 [OCSSD(4162)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvWorkerThread_0 not scheduled for 399920 msecs.
2021-01-19 18:13:21.558 [OCSSD(4162)]CRS-1604: CSSD voting file is offline: /data/RAC1/ocrvote1; details at (:CSSNM00058:) in /oracle/gridbase/diag/crs/rac1/crs/trace/ocssd.trc.
2021-01-19 18:13:21.560 [OCSSD(4162)]CRS-1606: The number of voting files available, 0, is less than the minimum number of voting files required, 1, resulting in CSSD termination to
 ensure data integrity; details at (:CSSNM00018:) in /oracle/gridbase/diag/crs/rac1/crs/trace/ocssd.trc
2021-01-19 18:13:21.566 [OCSSD(4162)]CRS-1656: The CSS daemon is terminating due to a fatal error; Details at (:CSSSC00012:) in /oracle/gridbase/diag/crs/rac1/crs/trace/ocssd.trc
2021-01-19 18:13:21.601 [OCSSD(4162)]CRS-1652: Starting clean up of CRSD resources.
2021-01-19 18:13:24.932 [OCSSD(4162)]CRS-1654: Clean up of CRSD resources finished successfully.
2021-01-19 18:13:24.935 [OCSSD(4162)]CRS-1655: CSSD on node rac1 detected a problem and started to shutdown.
2021-01-19 18:13:24.978 [ORAROOTAGENT(6329)]CRS-5822: Agent '/oracle/grid/crs_1/bin/orarootagent_root' disconnected from server. Details at (:CRSAGF00117:) {0:1:7} in /oracle/gridba
se/diag/crs/rac1/crs/trace/crsd_orarootagent_root.trc.
2021-01-19 18:13:25.283 [CRSD(26502)]CRS-8500: Oracle Clusterware CRSD process is starting with operating system process ID 26502
2021-01-19 18:13:27.326 [ORAAGENT(3935)]CRS-5011: Check of resource "ora.asm" failed: details at "(:CLSN00006:)" in "/oracle/gridbase/diag/crs/rac1/crs/trace/ohasd_oraagent_grid.trc
"
2021-01-19 18:13:27.464 [CSSDMONITOR(26560)]CRS-8500: Oracle Clusterware CSSDMONITOR process is starting with operating system process ID 26560
2021-01-19 18:13:31.320 [OCSSD(4162)]CRS-8503: Oracle Clusterware process OCSSD with operating system process ID 4162 experienced fatal signal or exception code 6.
2021-01-19T18:13:31.331013+08:00
Errors in file /oracle/gridbase/diag/crs/rac1/crs/trace/ocssd.trc  (incident=1):
CRS-8503 [] [] [] [] [] [] [] [] [] [] [] []
Incident details in: /oracle/gridbase/diag/crs/rac1/crs/incident/incdir_1/ocssd_i1.trc

2021-01-19 18:13:38.465 [CSSDAGENT(26712)]CRS-8500: Oracle Clusterware CSSDAGENT process is starting with operating system process ID 26712
2021-01-19 18:13:39.036 [OCSSD(26730)]CRS-8500: Oracle Clusterware OCSSD process is starting with operating system process ID 26730
2021-01-19 18:13:40.144 [OCSSD(26730)]CRS-1713: CSSD daemon is started in hub mode
2021-01-19 18:13:45.788 [OCSSD(26730)]CRS-1656: The CSS daemon is terminating due to a fatal error; Details at (:CSSSC00012:) in /oracle/gridbase/diag/crs/rac1/crs/trace/ocssd.trc
2021-01-19 18:13:45.849 [OCSSD(26730)]CRS-1652: Starting clean up of CRSD resources.
2021-01-19 18:13:45.856 [OCSSD(26730)]CRS-1653: The clean up of the CRSD resources failed.

可以看出磁盘心跳丢失,Voting disk离线,集群进程重启.

查看集群情况:

[grid@rac1 trace]$ crsctl check crs
CRS-4638: Oracle High Availability Services is online 
CRS-4535: Cannot communicate with Cluster Ready Services 
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon 
CRS-4534: Cannot communicate with Event Manager

尝试强制关闭.
[root@rac1 RAC4]# /oracle/grid/crs_1/bin/crsctl stop has
hang住
[root@rac1 RAC4]# /oracle/grid/crs_1/bin/crsctl stop crs -f
hang住

查看集群相关进程情况:

[root@rac1 ~]# ps -ef | grep d.bin
root      3725     1  1 00:48 ?        00:12:03 /oracle/grid/crs_1/bin/ohasd.bin reboot BLOCKING_STACK_LOCALE_OHAS=AMERICAN_AMERICA.AL32UTF8
grid      3958     1  0 00:48 ?        00:03:41 /oracle/grid/crs_1/bin/mdnsd.bin
grid      3998     1  0 00:48 ?        00:04:06 /oracle/grid/crs_1/bin/gpnpd.bin
grid      4069     1  0 00:48 ?        00:09:53 /oracle/grid/crs_1/bin/gipcd.bin
grid     24489     1  1 18:28 ?        00:00:00 /oracle/grid/crs_1/bin/ocssd.bin
root     27183     1  1 18:24 ?        00:00:02 /oracle/grid/crs_1/bin/osysmond.bin
grid     28244     1  0 18:17 ?        00:00:02 /oracle/grid/crs_1/bin/evmd.bin
root     29997  5602  0 18:29 pts/13   00:00:00 grep --color=auto d.bin

尝试kill进程前台仍然hang住,ohas进程无法关闭.

跟踪系统日志发现问题所在:

Jan 19 18:30:20 rac1 init.ohasd: OHASD starting
Jan 19 18:30:20 rac1 init.ohasd: Wrote 5327 to init.ohasd.
Jan 19 18:30:20 rac1 clsecho: /etc/init.d/init.ohasd: 5327 > /sys/fs/cgroup/system.slice/oracle-ohasd.service/tasks
Jan 19 18:30:20 rac1 init.ohasd: /etc/init.d/init.ohasd: 5327 > /sys/fs/cgroup/system.slice/oracle-ohasd.service/tasks
Jan 19 18:30:20 rac1 clsecho: /etc/init.d/init.ohasd: 5327 > /sys/fs/cgroup/cpu,cpuacct/tasks
Jan 19 18:30:20 rac1 init.ohasd: /etc/init.d/init.ohasd: 5327 > /sys/fs/cgroup/cpu,cpuacct/tasks
Jan 19 18:30:21 rac1 init.ohasd: Handshake with the watchdog's been completed
Jan 19 18:30:21 rac1 journal: Oracle Clusterware: 2021-01-19 18:30:21.663#012[(5327)]CRS-8500:Oracle Clusterware OHASD process is starting with operating system process ID 5327
Jan 19 18:30:41 rac1 kernel: nfs: server 172.16.10.1 OK
Jan 19 18:31:44 rac1 kernel: nfs: server 172.16.10.1 not responding, still trying
Jan 19 18:32:00 rac1 mount[9911]: mount to NFS server '172.16.20.1' failed: timed out, retrying
Jan 19 18:32:49 rac1 su: (to grid) root on pts/0
Jan 19 18:32:49 rac1 dbus[4785]: [system] Activating service name='org.freedesktop.problems' (using servicehelper)
Jan 19 18:32:49 rac1 dbus[4785]: [system] Successfully activated service 'org.freedesktop.problems'
Jan 19 18:33:41 rac1 kernel: nfs: server 172.16.10.1 not responding, still trying
Jan 19 18:34:10 rac1 mount[9911]: mount to NFS server '172.16.20.1' failed: timed out, retrying
Jan 19 18:35:40 rac1 kernel: nfs: server 172.16.10.1 OK
Jan 19 18:36:16 rac1 systemd: Starting Cleanup of Temporary Directories...
Jan 19 18:36:16 rac1 systemd: Started Cleanup of Temporary Directories.
Jan 19 18:36:20 rac1 mount[9911]: mount to NFS server '172.16.20.1' failed: timed out, retrying
Jan 19 18:36:26 rac1 kernel: nfs: server 172.16.10.1 not responding, still trying

尝试使用df -h命令查看挂载情况无法查询,系统hang住
手动进入挂载点目录,cd命令hang住。

联系NFS管理人员处理NFS异常问题,处理完成后查看:

[grid@rac1 RAC1]$ df -h
Filesystem                 Size  Used Avail Use% Mounted on
/dev/mapper/centos-root    110G   29G   81G  27% /
devtmpfs                   7.8G     0  7.8G   0% /dev
tmpfs                      7.8G  637M  7.2G   8% /dev/shm
tmpfs                      7.8G   20M  7.8G   1% /run
tmpfs                      7.8G     0  7.8G   0% /sys/fs/cgroup
/dev/sda1                  2.0G  167M  1.9G   9% /boot
tmpfs                      1.6G   12K  1.6G   1% /run/user/42
tmpfs                      1.6G   36K  1.6G   1% /run/user/0
/dev/loop0                 4.3G  4.3G     0 100% /mnt
172.16.10.1:/mnt/RAC/RAC1   10G  5.1G  5.0G  51% /data/RAC1
172.16.10.1:/mnt/RAC/RAC2   10G   96K   10G   1% /data/RAC2
172.16.10.1:/mnt/RAC/RAC3   10G   96K   10G   1% /data/RAC3
172.16.10.1:/mnt/RAC/RAC5  100G   96K  100G   1% /data/RAC5
tmpfs                      1.6G     0  1.6G   0% /run/user/1002
172.16.10.1:/mnt/RAC/RAC4  200G   91G  110G  46% /data/RAC4

解决后,集群重启:
查看集群情况:

[grid@rac1 trace]$ crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services 
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online

查看集群日志:

2021-01-19 18:44:55.829 [OHASD(19284)]CRS-0714: Oracle Clusterware Release 19.0.0.0.0.
2021-01-19 18:44:55.841 [OHASD(19284)]CRS-2112: The OLR service started on node rac1.
2021-01-19 18:44:56.129 [OHASD(19284)]CRS-1301: Oracle High Availability Service started on node rac1.
2021-01-19 18:44:56.129 [OHASD(19284)]CRS-8017: location: /etc/oracle/lastgasp has 2 reboot advisory log files, 0 were announced and 0 errors occurred
2021-01-19 18:44:57.166 [ORAAGENT(19949)]CRS-8500: Oracle Clusterware ORAAGENT process is starting with operating system process ID 19949
2021-01-19 18:44:57.172 [CSSDAGENT(19981)]CRS-8500: Oracle Clusterware CSSDAGENT process is starting with operating system process ID 19981
2021-01-19 18:44:57.201 [ORAROOTAGENT(19965)]CRS-8500: Oracle Clusterware ORAROOTAGENT process is starting with operating system process ID 19965
2021-01-19 18:44:57.208 [CSSDMONITOR(19992)]CRS-8500: Oracle Clusterware CSSDMONITOR process is starting with operating system process ID 19992
2021-01-19 18:44:57.772 [ORAAGENT(20117)]CRS-8500: Oracle Clusterware ORAAGENT process is starting with operating system process ID 20117
2021-01-19 18:44:58.124 [MDNSD(20216)]CRS-8500: Oracle Clusterware MDNSD process is starting with operating system process ID 20216
2021-01-19 18:44:58.118 [EVMD(20218)]CRS-8500: Oracle Clusterware EVMD process is starting with operating system process ID 20218
2021-01-19 18:44:59.173 [GPNPD(20426)]CRS-8500: Oracle Clusterware GPNPD process is starting with operating system process ID 20426
2021-01-19 18:44:59.998 [GPNPD(20426)]CRS-2328: GPNPD started on node rac1. 
2021-01-19 18:45:00.240 [GIPCD(20657)]CRS-8500: Oracle Clusterware GIPCD process is starting with operating system process ID 20657
2021-01-19 18:45:04.173 [OSYSMOND(21159)]CRS-8500: Oracle Clusterware OSYSMOND process is starting with operating system process ID 21159
2021-01-19 18:45:04.160 [CSSDMONITOR(21156)]CRS-8500: Oracle Clusterware CSSDMONITOR process is starting with operating system process ID 21156
2021-01-19 18:45:04.651 [CSSDAGENT(21231)]CRS-8500: Oracle Clusterware CSSDAGENT process is starting with operating system process ID 21231
2021-01-19 18:45:05.131 [OCSSD(21341)]CRS-8500: Oracle Clusterware OCSSD process is starting with operating system process ID 21341
2021-01-19 18:45:06.206 [OCSSD(21341)]CRS-1713: CSSD daemon is started in hub mode
2021-01-19 18:45:12.109 [OCSSD(21341)]CRS-1707: Lease acquisition for node rac1 number 1 completed
2021-01-19 18:45:13.201 [OCSSD(21341)]CRS-1621: The IPMI configuration data for this node stored in the Oracle registry is incomplete; details at (:CSSNK00002:) in /oracle/gridbase/
diag/crs/rac1/crs/trace/ocssd.trc
2021-01-19 18:45:13.202 [OCSSD(21341)]CRS-1617: The information required to do node kill for node rac1 is incomplete; details at (:CSSNM00004:) in /oracle/gridbase/diag/crs/rac1/crs
/trace/ocssd.trc
2021-01-19 18:45:13.561 [OCSSD(21341)]CRS-1605: CSSD voting file is online: /data/RAC1/ocrvote1; details in /oracle/gridbase/diag/crs/rac1/crs/trace/ocssd.trc.
2021-01-19 18:45:16.006 [OCSSD(21341)]CRS-1601: CSSD Reconfiguration complete. Active nodes are rac1 rac2 .
2021-01-19 18:45:17.139 [OCSSD(21341)]CRS-1720: Cluster Synchronization Services daemon (CSSD) is ready for operation.
2021-01-19 18:45:18.157 [OCTSSD(23190)]CRS-8500: Oracle Clusterware OCTSSD process is starting with operating system process ID 23190
2021-01-19 18:45:18.951 [OCTSSD(23190)]CRS-2403: The Cluster Time Synchronization Service on host rac1 is in observer mode.
2021-01-19 18:45:20.429 [OCTSSD(23190)]CRS-2407: The new Cluster Time Synchronization Service reference node is host rac2.
2021-01-19 18:45:20.430 [OCTSSD(23190)]CRS-2401: The Cluster Time Synchronization Service started on host rac1.
2021-01-19 18:45:24.935 [ORAROOTAGENT(19965)]CRS-5019: All OCR locations are on ASM disk groups [OCR], and none of these disk groups are mounted. Details are at "(:CLSN00140:)" in "/oracle/gridbase/diag/crs/rac1/crs/trace/ohasd_orarootagent_root.trc".
2021-01-19 18:49:25.591 [OCSSD(21341)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvDiskPingThread_0 not scheduled for 7660 msecs.
2021-01-19 18:49:25.592 [OCSSD(21341)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvWorkerThread_0 not scheduled for 8000 msecs.
2021-01-19 18:49:33.594 [OCSSD(21341)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvDiskPingThread_0 not scheduled for 15660 msecs.
2021-01-19 18:49:33.594 [OCSSD(21341)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvWorkerThread_0 not scheduled for 16000 msecs.
2021-01-19 18:49:41.595 [OCSSD(21341)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvDiskPingThread_0 not scheduled for 23670 msecs.
2021-01-19 18:49:41.596 [OCSSD(21341)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvWorkerThread_0 not scheduled for 24010 msecs.
2021-01-19 18:49:49.597 [OCSSD(21341)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvDiskPingThread_0 not scheduled for 31670 msecs.
2021-01-19 18:49:49.598 [OCSSD(21341)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvWorkerThread_0 not scheduled for 32010 msecs.
2021-01-19 18:49:57.599 [OCSSD(21341)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvDiskPingThread_0 not scheduled for 39670 msecs.

尝试手动启动ASM实例:

[grid@rac1 RAC1]$ sqlplus / as sysasm
SQL*Plus: Release 19.0.0.0.0 - Production on Tue Jan 19 18:53:26 2021
Version 19.3.0.0.0
Copyright (c) 1982, 2019, Oracle.  All rights reserved.
Connected to an idle instance.
SQL> startup;
ASM instance started
Total System Global Area 1137173320 bytes
Fixed Size                  8905544 bytes
Variable Size            1103101952 bytes
ASM Cache                  25165824 bytes
ASM diskgroups mounted

查询磁盘组挂载状态:

SQL> select NAME,state from v$asm_diskgroup;

NAME                           STATE
------------------------------ -----------
DATA                           MOUNTED
OCR                            MOUNTED

再次查询集群进程状态:

[grid@rac1 RAC1]$ crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online

然后DBCA进行装库,装库异常缓慢.
集群警告日志无限刷[OCSSD(21341)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvDiskPingThread_0 not scheduled for 7570 msecs.信息.

2021-01-19 19:22:31.830 [OCSSD(21341)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvDiskPingThread_0 not scheduled for 8410 msecs.
2021-01-19 19:22:41.831 [OCSSD(21341)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvWorkerThread_0 not scheduled for 8000 msecs.
2021-01-19 19:22:49.833 [OCSSD(21341)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvWorkerThread_0 not scheduled for 8000 msecs.
2021-01-19 19:22:56.834 [OCSSD(21341)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvDiskPingThread_0 not scheduled for 8460 msecs.
2021-01-19 19:23:05.836 [OCSSD(21341)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvWorkerThread_0 not scheduled for 8000 msecs.
2021-01-19 19:23:12.837 [OCSSD(21341)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvDiskPingThread_0 not scheduled for 7620 msecs.
2021-01-19 19:23:13.838 [OCSSD(21341)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvWorkerThread_0 not scheduled for 16000 msecs.
2021-01-19 19:23:25.839 [OCSSD(21341)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvWorkerThread_0 not scheduled for 8480 msecs.
2021-01-19 19:23:33.840 [OCSSD(21341)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvWorkerThread_0 not scheduled for 8000 msecs.
2021-01-19 19:23:41.841 [OCSSD(21341)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvWorkerThread_0 not scheduled for 8000 msecs.
2021-01-19 19:24:05.843 [OCSSD(21341)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvWorkerThread_0 not scheduled for 7990 msecs.
2021-01-19 19:24:13.844 [OCSSD(21341)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvWorkerThread_0 not scheduled for 8000 msecs.
2021-01-19 19:24:21.845 [OCSSD(21341)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvWorkerThread_0 not scheduled for 16010 msecs.

官方报错解释:

CRS-01719: Cluster Synchronization Service daemon (CSSD) string not scheduled for number msecs.
Cause: Excessive system load has prevented threads in the Cluster Synchronization Service daemon (CSSD) from being scheduled for execution for the time indicated in the message. This indicates the system is overloaded.

Action: Take steps to reduce the system load or increase system resources to handle the load.

由于性能太差,DBCA装库导致存储负载过高.DBCA结束后,CRS-01719问题消失。