起因：

最近研发反应JDBC和PL/SQL工具连接过慢，昨天事情多懒得去管

今早检查alter.log后发现报错

Fatal NI connect error

12637, connecting to:

(LOCAL=NO)

VERSION

INFORMATION:

TNS for Linux: Version 11.2.0.1.0 - Production

Oracle Bequeath NT Protocol Adapter for Linux: Version 11.2.0.1.0 -

Prod

uction

TCP/IP NT Protocol Adapter for Linux: Version 11.2.0.1.0 -

Production

Time: 23-APR-2013

11:05:10

Tracing not turned

on.

Tns error

struct:

ns main err code: 12637

TNS-12637: Packet receive

failed

ns secondary err code: 12532

nt main err code: 0

nt secondary err code: 0

nt OS err code: 0

opiodr aborting process

unknown ospid (24234) as a result of ORA-609

Tue Apr 23 11:05:15

2013

故障分析

检查 listener 日志确认没有异常，在部分机器 TNSPING XXX 60 等进行尝试结果正常，无链接问题

明显感觉，只要第一次 tnsping 链接正常，之后的 jdbc 链接就正常；如果第一次不正常，之后的 jdbc 链接就容易返回超时的错误。

尝试重启 lsnrctl stop/start

和重新加载配置文件lsnrctl

reload 重新注册数据库实例，仍然无法解决超时错误。

tnsping xxxx N 的时候明显观察到：

OK (56730 msec)OK (80 msec)

OK (80 msec)

OK (70 msec)

OK (70 msec)

OK (60 msec)

OK (47820 msec)

OK (70 msec)

同时 PING IP

的结果并没有出现延时，机器之前并未设置防火墙等，可以排除网络和防火墙问题。在连续

TNSPING

的情况下，出现明显链接延时。

解决方案

方案一：(有效，但是由于是测试环境，所以影响层面未评估)

注释掉 /etc/resolve.conf 配置文件中对 DNS 的解析解决[oracle@vcdog log]$ cat /etc/resolv.conf #nameserver 202.106.0.20

方案二：官方文档(效果不明显)

ORA-609 TNS-12537 and TNS-12547 in 11g Alert.log [ID

1116960.1]

Cause

The ORA-609 error is thrown when a client connection

of any kind failed to complete or aborted the connection process

before the server process was completely spawned.

Very often, this connection abort is due to a

timeout.Beginning with 10gR2, a default value for inbound

connect timeout has been set at 60

seconds.This time limit is often inadequate for the entire

connection process to

complete. We have also discovered that the ORA-609 occurs frequently in

installations where the database is monitored by DB Console and the

Enterprise Manager agent

(emagent).After the DB Console is started and as a matter of

routine, the emagent will repeatedly try to connect to the target

instances.We can see frequent emagent connections in the

listener.log without error.However, on occasion it may have failed to complete

the connection process at the database so an ORA-609 is

thrown.The emagent will simply retry the connection and may

be successful on the subsequent

try.(Provided there is no real fault occurring at the

listener or database).This temporary failure to connect will not be reported

back to DB Console and there will be no indication, except for the

ORA-609, that a fault occurred.

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Solution

It can be somewhat

challengingto determine the origin of the client that is causing

the error.

For that reason, we often recommend increasing the

values for INBOUND_CONNECT_TIMEOUT at both listener and server side

sqlnet.ora file as a preventive

measure.If the problemis due to connection timeouts, an increase in the

following parameters should eliminate or reduce the occurrence of

the ORA-609s.

e.g.Sqlnet.ora:

SQLNET.INBOUND_CONNECT_TIMEOUT=180Listener.ora:

INBOUND_CONNECT_TIMEOUT_listener_name=120

These settings are in seconds.Again, the default is 60.

If the issue persists and inbound connect does not

have any effect, the following steps are intended to help

locate the client that may be

causing the errors.

1)Suppress the TNS errors in the alert.log by setting

the following listener.ora file parameter: DIAG_ADR_ENABLED_listener_name=OFF

This will cause the TNS errors to be posted to the

ORACLE_HOME/network/log/sqlnet.log file that is local to the

database and may yield useful information about the client's

address.

For example, here's a snippet from a server side sqlnet.log where

client address info was posted:

Production Time: 15-FEB-2010

07:15:01Fatal NI connect error 12537, connecting

to:(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(Host=yourhost)(Port=1521))(CONNECT_DATA=(SID=PROD1DR)(CID=(PROGRAM=sqlplus)(HOST=client_host)(USER=client))))

Observe the PROGRAM and HOST fields on the last

line.This is where the connection originated.

Be sure to match timestamps in the sqlnet.log with the timestamps

of the alert.log errors.Once you've located the offending client, you can

enable client tracing to try and determine the cause:

TRACE_LEVEL_CLIENT=16

TRACE_DIRECTORY_CLIENT=

TRACE_TIMESTAMP_CLIENT=TRUE

DIAG_ADR_ENABLED=off<<<<<11g or newer client

requirement

If you need assistance with client or server tracing, please open

an SR with Global Customer Support.

2)Check the listener.log for client connections that

were logged at timestamps that match the ORA-609 timestamps as they

appear in the alert.log.The client information is recorded in each

listener.log entry.Since this error occurs AFTER the listener has handled

the connection, do not expect to see errors in the

listener.log.

Here's an example snippet of an incoming client connection that was

posted to the listener.log:

20-JAN-2009 17:08:45

(CONNECT_DATA=(SID=orcl)(CID=(PROGRAM=D:\oracle\product\10.1.0\Db_1\perl\5.6.1\bin\MSWin32-x86\perl.exe)(HOST=myclient)

Note that the exact timestamp, program name and client host will

often be recorded.Again, once you've located the offending client,

enable tracing (see above) to try to capture the connection

failure.3)Enable server side Oracle Net tracing and capture the

TNS error along with the incoming connection.

Match the PID that accompanies the ORA-609 to the server trace

label.e.g. ORA-609 : opiodr aborting process unknown ospid

(4799_1)*Note the PID

This PID would correspond to server trace

labeled:svr_4799.trc.Check the server trace for either TNS error (the 609

will not appear) and try to locate the originating client

address.If assistance is needed for this investigation, please

open an SR with Oracle Support.

See below for instuctions on enabling Oracle Net server

tracing.

The following details the discovery of the source of

an ORA-609 for a real case:

The alert.log reports the following messages

intermittently but frequently:

Mon Nov 16 22:39:22 2009ORA-609 : opiodr aborting process unknown ospid

(nnnn)

Enabled Oracle Net server tracing:

TRACE_LEVEL_SERVER=16

TRACE_DIRECTORY_SERVER=

TRACE_TIMESTAMP_SERVER=TRUE

DIAG_ADR_ENABLED=off

Reloaded listener and wait for error to appear again.:

ORA-609 : opiodr aborting process unknown

ospid(5233_1)

Note that the server trace file set that corresponded to this event

was namedsvr_5233*.trc.

Of course the timestamps of the alert.log event and the server

trace creation matched as well.

A review of the server trace showed only an EOF failure and

theTNS-12537 error:

Read unexpected EOF ERRORnserror: nsres: id=0, op=68, ns=12537

In this particular case, there was no information about the client

in the trace. This is atypical for a server trace.It may be that the client aborted before all the

client information was posted to the

file.However, there was post in the listener.log for an

emagent connection that was established at the same point in

time.

Here's an excerpt from a listener.log entry where an emagent

establishes a connection:

PROGRAM=D:\oracle\product\10.1.0\Db_1\bin\emagent.exe)

Checked the EM Agent traces and logs and discovered the following

entry:

Fatal NI connect error 12547, connecting

to:(LOCAL=NO)VERSION INFORMATION:TNS for Solaris: Version 11.1.0.7.0 -

ProductionOracle Bequeath NT Protocol Adapter for Solaris:

Version 11.1.0.7.0 - ProductionTCP/IP NT Protocol Adapter for Solaris: Version

11.1.0.7.0 - ProductionTime: 16-NOV-2009 22:39:22****Tracing to file:

/backup/sid_traces/sqlnetlog/svr_5233.trcTns error struct:ns main err code: 12547TNS-12547: TNS:lost contactns secondary err code: 12560nt main err code: 0nt secondary err code: 0nt OS err code: 0****Note the name of the server trace which contains

the PID:svr_5233.trcAlso, the timestamp of the agent event matches the

timestamp of the alert.log error.

Check the following locations for EM Agent traces. If

working with support on this issue and the EM Agent is suspected,

upload ALL files under:

$ORACLE_HOME/sysman/log/emagent.trc < Single node agent trace

location

$ORACLE_HOME/host/sysman/log/emagent.trc < RAC agent trace

location

It was determined that in this case, the emagent was aborting the

connection before it was complete and then simply reconnecting and

succeeding on the subsequent

try.No errors were reported in the listener log or

listener trace. No errors were returned to the DB

Console.There was no apparent outage of any

kind. No action was taken to

correct the ORA-609 in this

case.It was decided that the message was informational and

completely benign.

Please review the following documents for more information about

timeouts and tracing:Troubleshooting Guide TNS-12535 or ORA-12535 or

ORA-12170

ErrorsConnections that Used to Work in Oracle 10.1 Now

Intermittently Fail with ORA-3113,ORA-3106 or ORA-3136 from

10.2 OnwardsFiles Needed for Troubleshooting an EM 10G Service

Request

if an RDA is not AvailableHow to Enable Oracle SQLNet Client , Server , Listener

,

Kerberos and External procedure Tracing from Net ManagerUsing and Disabling the Automatic Diagnostic

Repository

(ADR) with Oracle Net for 11g