History Oracle Support - 04 March 2012 16:11:10 GMT+05:30 [ODM Solution/Action Plan] i) check the account status SQL>

select account_status from dba_users where username='<dbfs username>'; ii) Get the profile for dbfs user SQL> select username,profile from dba_users where username='<dbfs username>'; iii) Alter the user profile to unlimited SQL>ALTER PROFILE DEFAULT LIMIT PASSWORD_LIFE_TIME UNLIMITED; iv) change the password of dbfs user SQL>ALTER USER <dbfs username> IDENTIFIED BY <current DBFS password>'; v) now stop and start the dbfs resource and check whether it is mounting or not Oracle Support - 04 March 2012 16:08:18 GMT+05:30 [ODM Proposed Solution Justif] check SAP Oracle Support - 04 March 2012 16:08:02 GMT+05:30 [ODM Proposed Solution(s)] check SAP Oracle Support - 04 March 2012 16:07:49 GMT+05:30 [ODM Cause Justification] dbfs user account enters the GRACE period for the password and giving warning message. INFO : ERROR 28002 - ORA-28002: the password will expire within 6 days Oracle Support - 04 March 2012 16:06:59 GMT+05:30 [ODM Cause Determination] dbfs.out ~~~~~~~~~ [4dd27940 03/04/12 05:00:58.012007 .110. 1330855258 10139000 [41d68940 03/04/12 05:00:59.870164 ret:1 [41d68940 03/04/12 05:00:59.870204 [41d68940 03/04/12 05:00:59.870225 password will expire within 6 days

./LcdfThreadPool.h:275 ] DEBUG: Retrying after some time LcdfDBPool.cpp:188 ] ERROR: Failed to create session pool LcdfDBPool.cpp:381 ] INFO : OCI_SUCCESS_WITH_INFO LcdfDBPool.cpp:384 ] INFO : ERROR 28002 - ORA-28002: the

[41d68940 03/04/12 05:00:59.870241 LcdfDBPool.cpp:250 ] DEBUG: Clean up OCI session pool... [41d68940 03/04/12 05:00:59.873070 LcdfSfs.cpp:89 OS logs ~~~~~~~~ Mar 4 05:00:51 aodpdb02 snmpd[5538]: looks like a 64bit wrap, but prev!=new Mar 4 05:00:59 aodpdb02 dbfs_client[31368]: Fail to set up database connection.

Oracle Support - 04 March 2012 07:55:31 GMT+05:30 [ODM Issue Clarification] Unable to mount the dbfs mount point [oracle@aodpdb02 script]$ sh mount-dbfs.sh start mount-dbfs.sh mounting DBFS at /dbfs_direct from database dbm ORACLE_SID is dbm2 spawning dbfs_client command using SID dbm2 Start -- OFFLINE Oracle Support - 04 March 2012 16:04:49 GMT+05:30 [Call - Inbound] Call - Inbound ---------------------------Customer contact name (if not same as in SR header): Verify preferred method of contact: PHONE OR My Oracle Support customer portal Phone#: 9052409998 Email: pavan.bolla@clorox.com Comments for this update: Pavan called and told that the issue has been resolved after resetting the password of dbfs user.He has confirmed to close the SR ACTION PLAN -------------------Thank you. Prabu Oracle Global Customer Support Oracle Support - 04 March 2012 16:02:01 GMT+05:30 [Call - Inbound] Oracle Support - 04 March 2012 16:01:38 GMT+05:30 [ODM Action Plan] It looks that dbfs user account enters the GRACE period for the password and giving warning message. You need to alter the user password and password life to unlimited to avoid this issue in future ACTION PLAN:~~~~~~~~~~~~~~ i) check the account status SQL> select account_status from dba_users where username='<dbfs username>'; ii) Get the profile for dbfs user SQL> select username,profile from dba_users where username='<dbfs username>'; iii) Alter the user profile to unlimited SQL>ALTER PROFILE DEFAULT LIMIT PASSWORD_LIFE_TIME UNLIMITED;

iv) change the password of dbfs user SQL>ALTER USER <dbfs username> IDENTIFIED BY <current DBFS password>'; v) now stop and start the dbfs resource and check whether it is mounting or not Regards Prabu Oracle Support - 04 March 2012 16:00:23 GMT+05:30 [ODM Data Collection] Name -------=== ODM Data Collection === dbfs.out ~~~~~~~~~ [4dd27940 03/04/12 05:00:58.012007 .110. 1330855258 10139000 [41d68940 03/04/12 05:00:59.870164 ret:1 [41d68940 03/04/12 05:00:59.870204 [41d68940 03/04/12 05:00:59.870225 password will expire within 6 days

./LcdfThreadPool.h:275 ] DEBUG: Retrying after some time LcdfDBPool.cpp:188 ] ERROR: Failed to create session pool LcdfDBPool.cpp:381 ] INFO : OCI_SUCCESS_WITH_INFO LcdfDBPool.cpp:384 ] INFO : ERROR 28002 - ORA-28002: the

[41d68940 03/04/12 05:00:59.870241 LcdfDBPool.cpp:250 ] DEBUG: Clean up OCI session pool... [41d68940 03/04/12 05:00:59.873070 LcdfSfs.cpp:89 OS logs ~~~~~~~~ Mar 4 05:00:51 aodpdb02 snmpd[5538]: looks like a 64bit wrap, but prev!=new Mar 4 05:00:59 aodpdb02 dbfs_client[31368]: Fail to set up database connection. FileName ---------------FileComment ----------------------

PAVAN.BOLLA@CLOROX.COM - 04 March 2012 15:46:36 GMT+05:30 [Update from Customer] Upload to gtcr successful for the file messages.txt. PAVAN.BOLLA@CLOROX.COM - 04 March 2012 15:46:11 GMT+05:30 [Update from Customer] Upload to gtcr successful for the file dbfs.out. Oracle Support - 04 March 2012 15:24:58 GMT+05:30 [Notes] Generic Note ------------------------

Hi, I have reviewed the SR and uploaded logs. We need to get the tracing of dbfs_client mount I see that you are using wallet configuration for dbfs. Please see the below given example to mount the dbfs with tracing enabled su - oracle export ORACLE_SID=<dbfs db sid> export ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1 cd $ORACLE_HOME/bin dbfs_client <dbfs user>@ -otrace_file=/tmp/dbfs.out -otrace_level=1 -otrace_size=0 /<dbfs mount> after entering the password,it will just hang there itself. If its hangs there,please open another new terminal and check the below ++ ls -l /<dbfs mount> ++ cat /etc/mtab ++ upload /tmp/dbfs.out ++ latest OS logs -> /var/log/messages Regards Prabu Oracle Support - 04 March 2012 15:13:00 GMT+05:30 [Notes] Generic Note -----------------------Hi, The SR has been transferred to me.I am reviewing the SR and will update you as soon as possible. Regards Prabu Oracle Support - 04 March 2012 14:53:03 GMT+05:30 [Call - Inbound] Oracle Support - 04 March 2012 14:41:54 GMT+05:30 [Notes] Generic Note -----------------------UPDATE ======== Going through similar SRs with us . PAVAN.BOLLA@CLOROX.COM - 04 March 2012 14:36:25 GMT+05:30 [Update from

Customer] Hi Suppot, Please find the output below.. [oracle@aodpdb02 bin]$ ./crsctl stat res -t -------------------------------------------------------------------------------NAME TARGET STATE SERVER STATE_DETAILS -------------------------------------------------------------------------------Local Resources -------------------------------------------------------------------------------dbfs_mount ONLINE ONLINE aodpdb01 ONLINE OFFLINE aodpdb02 OFFLINE OFFLINE aodpdb03 OFFLINE OFFLINE aodpdb04 ora.DATA_AODP.dg ONLINE ONLINE aodpdb01 ONLINE ONLINE aodpdb02 ONLINE ONLINE aodpdb03 ONLINE ONLINE aodpdb04 ora.DBFS_DG.dg ONLINE ONLINE aodpdb01 ONLINE ONLINE aodpdb02 ONLINE ONLINE aodpdb03 ONLINE ONLINE aodpdb04 ora.LISTENER.lsnr ONLINE ONLINE aodpdb01 ONLINE ONLINE aodpdb02 ONLINE ONLINE aodpdb03 ONLINE ONLINE aodpdb04 ora.RECO_AODP.dg ONLINE ONLINE aodpdb01 ONLINE ONLINE aodpdb02 ONLINE ONLINE aodpdb03 ONLINE ONLINE aodpdb04 ora.asm ONLINE ONLINE aodpdb01 Started ONLINE ONLINE aodpdb02 ONLINE ONLINE aodpdb03 Started ONLINE ONLINE aodpdb04 Started ora.gsd OFFLINE OFFLINE aodpdb01 OFFLINE OFFLINE aodpdb02 OFFLINE OFFLINE aodpdb03 OFFLINE OFFLINE aodpdb04 ora.net1.network ONLINE ONLINE aodpdb01 ONLINE ONLINE aodpdb02 ONLINE ONLINE aodpdb03 ONLINE ONLINE aodpdb04 ora.ons ONLINE ONLINE aodpdb01 ONLINE ONLINE aodpdb02 ONLINE ONLINE aodpdb03 ONLINE ONLINE aodpdb04 ora.registry.acfs

ONLINE ONLINE aodpdb01 ONLINE ONLINE aodpdb02 ONLINE ONLINE aodpdb03 ONLINE ONLINE aodpdb04 -------------------------------------------------------------------------------Cluster Resources -------------------------------------------------------------------------------ora.LISTENER_SCAN1.lsnr 1 ONLINE ONLINE aodpdb03 ora.LISTENER_SCAN2.lsnr 1 ONLINE ONLINE aodpdb04 ora.LISTENER_SCAN3.lsnr 1 ONLINE ONLINE aodpdb01 ora.aodpdb01.vip 1 ONLINE ONLINE aodpdb01 ora.aodpdb02.vip 1 ONLINE ONLINE aodpdb02 ora.aodpdb03.vip 1 ONLINE ONLINE aodpdb03 ora.aodpdb04.vip 1 ONLINE ONLINE aodpdb04 ora.biprd.biprd1.svc 1 ONLINE ONLINE aodpdb01 ora.biprd.biprd2.svc 1 ONLINE ONLINE aodpdb02 ora.biprd.biprd3.svc 1 ONLINE ONLINE aodpdb03 ora.biprd.biprd4.svc 1 ONLINE ONLINE aodpdb04 ora.biprd.db 1 ONLINE ONLINE aodpdb01 Open 2 ONLINE ONLINE aodpdb02 Open 3 ONLINE ONLINE aodpdb03 Open 4 ONLINE ONLINE aodpdb04 Open ora.cvu 1 ONLINE ONLINE aodpdb03 ora.dbm.db 1 ONLINE ONLINE aodpdb01 Open 2 ONLINE ONLINE aodpdb02 Open 3 ONLINE ONLINE aodpdb03 Open 4 ONLINE ONLINE Oracle Support - 04 March 2012 14:33:11 GMT+05:30 [Notes] Generic Note -----------------------UPDATE ========= All the resource crsctl stat res -t output Oracle Support - 04 March 2012 14:30:03 GMT+05:30 [Notes] Generic Note -----------------------UPDATE ===========

Can you try mounting the /dbfs_direct mount /dbfs_direct ( as non-root user ) Oracle Support - 04 March 2012 14:21:12 GMT+05:30 [ODM Data Collection] Name -------=== ODM Data Collection === ld FileName ---------------FileComment ---------------------echo $LD_LIBRARY_PATH [oracle@aodpdb02 bin]$ echo $LD_LIBRARY_PATH /u01/app/oracle/product/11.2.0/dbhome_1/lib Oracle Support - 04 March 2012 14:20:04 GMT+05:30 [ODM Data Collection] Name -------=== ODM Data Collection === Doc File FileName ---------------FileComment ---------------------[oracle@aodpdb02 bin]$ ./crsctl stat resource dbfs_mount -t -------------------------------------------------------------------------------NAME TARGET STATE SERVER STATE_DETAILS -------------------------------------------------------------------------------Local Resources -------------------------------------------------------------------------------dbfs_mount ONLINE ONLINE aodpdb01 ONLINE OFFLINE aodpdb02 ONLINE OFFLINE aodpdb03 ONLINE OFFLINE aodpdb04 b) Please send the output of <GRID_HOME>/crs/script/mount-dbfs.sh check [oracle@aodpdb02 script]$ ./mount-dbfs.sh check Check -- OFFLINE ============================================================== ============================================================== ============================================================== ================================================

c) Can you start mounting again with <GRID_HOME>/crs/script/mount-dbfs.sh start [oracle@aodpdb02 script]$ sh mount-dbfs.sh start mount-dbfs.sh mounting DBFS at /dbfs_direct from database dbm ORACLE_SID is dbm2 spawning dbfs_client command using SID dbm2 Start -- OFFLINE ============================================================== ============================================================== ============================================================== ================================================ /var/log/messages -------------------Mar 4 02:22:05 aodpdb02 DBFS: /u01/app/11.2.0/grid/crs/script/mount-dbfs.sh mounting DBFS at /dbfs_direct from database dbm Mar 4 02:22:06 aodpdb02 ntpd[5630]: synchronized to LOCAL(0), stratum 10 Mar 4 02:22:07 aodpdb02 DBFS: ORACLE_SID is dbm2 Mar 4 02:22:07 aodpdb02 DBFS: spawning dbfs_client command using SID dbm2 Mar 4 02:22:07 aodpdb02 DBFS: Password: Mar 4 02:22:07 aodpdb02 dbfs_client[12064]: Fail to set up database connection. Mar 4 02:22:08 aodpdb02 DBFS: Start OFFLINE Mar 4 02:22:18 aodpdb02 DBFS: cleaning up DBFS using fusermount and kill on dbfs_client and mount.dbfs Mar 4 02:23:17 aodpdb02 ntpd[5630]: synchronized to 168.189.30.14, stratum 2 Mar 4 02:44:52 aodpdb02 dbfs_client[30552]: Fail to set up database connection. Mar 4 02:49:10 aodpdb02 sreport.sh: Starting new sos report at Sun Mar 4 02:49:10 EST 2012 Mar 4 02:49:12 aodpdb02 sreport.sh: Saving sos report /var/log/cellos/sosreports in /dev/mapper/VGExaDb-LVDbSys1 Mar 4 02:49:12 aodpdb02 sreport.sh: Report completed from starting date Sun Mar 4 02:49:10 EST 2012 Mar 4 02:52:07 aodpdb02 dbfs_client[3854]: Fail to set up database connection. Mar 4 03:00:52 aodpdb02 DBFS: /u01/app/11.2.0/grid/crs/script/mount-dbfs.sh mounting DBFS at /dbfs_direct from database dbm Mar 4 03:00:52 aodpdb02 DBFS: ORACLE_SID is dbm2 Mar 4 03:00:52 aodpdb02 DBFS: spawning dbfs_client command using SID dbm2 Mar 4 03:00:52 aodpdb02 DBFS: Password: Mar 4 03:00:53 aodpdb02 dbfs_client[10129]: Fail to set up database connection. Mar 4 03:00:53 aodpdb02 DBFS: Start OFFLINE Mar 4 03:01:04 aodpdb02 DBFS: cleaning up DBFS using fusermount and kill on dbfs_client and mount.dbfs Mar 4 03:06:46 aodpdb02 DBFS: mount-dbfs.sh mounting DBFS at /dbfs_direct from database dbm Mar 4 03:06:47 aodpdb02 DBFS: ORACLE_SID is dbm2 ============================================================== ============================================================== ============================================================== ================================================ [oracle@aodpdb02 script]$ sh mount-dbfs.sh stop mount-dbfs.sh: line 165: mount-dbfs.sh: command not found filesystem /dbfs_direct not currently mounted, no need to stop ============================================================== ============================================================== ============================================================== ================================================

]$ sh mount-dbfs.sh start mount-dbfs.sh mounting DBFS at /dbfs_direct from database dbm ORACLE_SID is dbm2 spawning dbfs_client command using SID dbm2 Start -- OFFLINE ============================================================== ============================================================== ============================================================== ================================================ crsctl start resource dbfs_mount [oracle@aodpdb02 bin]$ ./crsctl start resource dbfs_mount CRS-2672: Attempting to start 'dbfs_mount' on 'aodpdb04' CRS-2672: Attempting to start 'dbfs_mount' on 'aodpdb02' CRS-2672: Attempting to start 'dbfs_mount' on 'aodpdb03' CRS-2674: Start of 'dbfs_mount' on 'aodpdb02' failed CRS-2679: Attempting to clean 'dbfs_mount' on 'aodpdb02' CRS-2674: Start of 'dbfs_mount' on 'aodpdb04' failed CRS-2679: Attempting to clean 'dbfs_mount' on 'aodpdb04' CRS-2674: Start of 'dbfs_mount' on 'aodpdb03' failed CRS-2679: Attempting to clean 'dbfs_mount' on 'aodpdb03' CRS-2681: Clean of 'dbfs_mount' on 'aodpdb03' succeeded CRS-2681: Clean of 'dbfs_mount' on 'aodpdb02' succeeded CRS-2681: Clean of 'dbfs_mount' on 'aodpdb04' succeeded CRS-4000: Command Start failed, or completed with errors. ============================================================== ============================================================== ============================================================== ================================================ Oracle Support - 04 March 2012 14:06:26 GMT+05:30 [Notes] Generic Note -----------------------UPDATE ========== Going through all that. Please wait PAVAN.BOLLA@CLOROX.COM - 04 March 2012 14:03:32 GMT+05:30 [Update from Customer] Upload to gtcr successful for the file strace.out. PAVAN.BOLLA@CLOROX.COM - 04 March 2012 14:03:22 GMT+05:30 [Update from Customer] Upload to gtcr successful for the file SR_ODM_ACTION_PLAN.docx. PAVAN.BOLLA@CLOROX.COM - 04 March 2012 14:02:36 GMT+05:30 [Update from Customer] Hi Suport, Pleased find uploded files as per the ODM Action Plan.

Thanks, Pavan Oracle Support - 04 March 2012 13:33:12 GMT+05:30 [Notes] Generic Note -----------------------ACTION PLAN =============== a) Please send the output of crsctl stat resource dbfs_mount -t b) Please send the output of <GRID_HOME>/crs/script/mount-dbfs.sh check c) Can you start mounting again with <GRID_HOME>/crs/script/mount-dbfs.sh start d) Can you please upload /var/log/messages output during this time ? e) Please stop it again >/crs/script/mount-dbfs.sh stop f) Please start again using >/crs/script/mount-dbfs.sh start g) Can you also start using crsctl crsctl start resource dbfs_mount h) Can you do the tail of scriptagent_oracle.log ( please upload the output ) i) Can you try mounting the /dbfs_direct mount /dbfs_direct ( as non-root user ) Send the output please. j) Can you check the LD_LIBRARY_PATH is set or not on NODE 1 echo $LD_LIBRARY_PATH k) Can you make sure that the environment has : $ export LD_LIBRARY_PATH=$ORACLE_HOME/lib Then run the mount again with strace also : $ strace -fae -o /tmp/strace.out dbfs_client ... l ) Can we trace this using ? How to trace DBFS when any failure happens [ID 1320683.1] Please send the output of trace. Oracle Support - 04 March 2012 13:11:00 GMT+05:30 [ODM Research]

How to clean-up failed DBFS configuration (Doc ID 1312236.1) Accessing Dbfs Using Oracle Wallet Fails With Input/output error (Doc ID 1358194.1) How to trace DBFS when any failure happens [ID 1320683.1] Cluster Resource DBFS in not starting (Doc ID 1401303.1) Dbfs File System became unmounted due to mount-dbfs.sh timeout on Exadata Database Machine (Doc ID 1385927.1) Configuring DBFS on Oracle Database Machine (Doc ID 1054431.1) Errors when mounting a DBFS store using fstab due to missing libraries (Doc ID 1053008.1) Oracle Support - 04 March 2012 13:04:12 GMT+05:30 [Notes] Generic Note -----------------------He informed , even reboot has not resolved the issue. Oracle Support - 04 March 2012 13:03:41 GMT+05:30 [Notes] Generic Note -----------------------UPDATE =========== Called +91 9052409998 spoke to Pavan and informed that SR will be updated with action plan Oracle Support - 04 March 2012 13:01:43 GMT+05:30 [Call Back - Customer] Call Back Customer -----------------------------------Customer contact name (if not same as in SR header): Phone#: +91 9052409998 Reason for Call Back? The customer called requesting call back PAVAN.BOLLA@CLOROX.COM - 04 March 2012 13:01:35 GMT+05:30 [Update from Customer] Hi Support, As per the ODM action Plan we have rebooted the server, but the dbfs file system was not mounted.. Thanks, Pavan Oracle Support - 04 March 2012 12:57:54 GMT+05:30 [Call - Inbound] Oracle Support - 04 March 2012 12:37:49 GMT+05:30 [Notes] Generic Note -----------------------OWC Summary ============= Problem started after customer restarted the database. The issue is only on node 2 .

DBBS is mounting successfully on Node 1 but this node shows OFFLINE a) We checked TNS_ADMIN variable on both the nodes . It is not set b) We tried to start using crsctl start resource dbfs_mount It was ONLINE on OTHER nodes but not on this node it was showing OFFLINE c) We checked it using crsctl stat ( status ) It was again showing offline We tried to stop and start but was not successful d) We also tried to start using >/crs/script/mount-dbfs.sh But , it was still not mounting on Node 2 e) We checked listener and also the tnsping ( they are fine ) f) The difference is seen in following cat /etc/mtab dbfs-dbfs_user@:/ /dbfs_direct fuse rw,nosuid,nodev,max_read=1048576,default_permissions,allow_other,user=oracle 0 0 cat /etc/fstab /sbin/mount.dbfs#/@fsdb.local /dbfs_direct fuse rw,user,direct_io,allow_other,wallet,noauto 0 0 Node 2. txt cat /etc/mtab No entry for dbfs in mtab Where as cat /etc/fstab shows /sbin/mount.dbfs#/@fsdb.local /dbfs_direct fuse rw,user,direct_io,allow_other,wallet,noauto 0 0 g) /var/log/messages shows DBFS: Fail to connect to database server h) We also tried fusermount -u /dbfs_direct" and then /crs/script/mount-dbfs.sh start Was not successful

i) We checked scriptagent_oracle.log SUSPECTING password issue ( not sure now ) Because It asks for password and next line shows DBFS: Fail to connect to database server Presently waiting for customer to reboot and then proceed further Oracle Support - 04 March 2012 11:10:08 GMT+05:30 [Notes] Generic Note -----------------------UPDATE ============ The node1.txt ===================== Shows cat /etc/mtab dbfs-dbfs_user@:/ /dbfs_direct fuse rw,nosuid,nodev,max_read=1048576,default_permissions,allow_other,user=oracle 0 0 cat /etc/fstab /sbin/mount.dbfs#/@fsdb.local /dbfs_direct fuse rw,user,direct_io,allow_other,wallet,noauto 0 0 Node 2. txt cat /etc/mtab No entry for dbfs in mtab Where as cat /etc/fstab shows /sbin/mount.dbfs#/@fsdb.local /dbfs_direct fuse rw,user,direct_io,allow_other,wallet,noauto 0 0 ACTION PLAN =================== Please reboot this node and let us know the result We will resume OWC again steps to power down or reboot an Exadata storage cell without affecting ASM:1188080.1 PAVAN.BOLLA@CLOROX.COM - 04 March 2012 11:04:21 GMT+05:30 [Update from Customer] Upload to gtcr successful for the file node2.txt.

PAVAN.BOLLA@CLOROX.COM - 04 March 2012 11:04:10 GMT+05:30 [Update from Customer] Upload to gtcr successful for the file node1.txt. Oracle Support - 04 March 2012 09:27:01 GMT+05:30 [Notes] Generic Note -----------------------Use Internet Explorer . Make sure popup blocker is disabled . Visit strtc.oracle.com The conference id is 9726215 The conference key is 16882554 Oracle Support - 04 March 2012 09:24:05 GMT+05:30 [Call - Outbound] Call - Outbound -----------------------------Discussing with Pavan at +91 9052409998 ASTHA.PATEL@ORACLE.COM - 04 March 2012 09:19:38 GMT+05:30 [Call - Inbound] Call - Inbound ---------------------------Customer contact name (if not same as in SR header): Verify preferred method of contact: PHONE OR My Oracle Support customer portal Phone#: +91 9052409998 Email: pavan.bolla@clorox.com Comments for this update: The customer called in requesting for urgent call back on this SR. ACTION PLAN -------------------Contacting Engineer Thank you. Astha Oracle Global Customer Support Oracle Support - 04 March 2012 09:17:19 GMT+05:30 [Call - Inbound] Oracle Support - 04 March 2012 09:10:56 GMT+05:30 [Notes] Generic Note -----------------------UPDATE ========== a) Please provide the result of complete action plan b) /usr/local/lib/libclntsh.so.11.1 ( If this is there , the link between dbhome_1/lib/libclntsh.so.11.1 and /usr/lib seems missing ) c) Please compare the dbhome_1/lib/libclntsh.so.11.1 to other node ( ls -all ) Is there any link available on that node ?

PAVAN.BOLLA@CLOROX.COM - 04 March 2012 09:01:55 GMT+05:30 [Update from Customer] Hi Support, a)Previously its working fine, b)only on one node ./crsctl stat res -t dbfs_mount ONLINE ONLINE aodpdb01 ONLINE OFFLINE aodpdb02 ONLINE ONLINE aodpdb03 ONLINE ONLINE aodpdb04 ============================================================== ========= ls -all /u01/app/oracle/product/11.2.0/dbhome_1/lib/libclntsh.so.11.1 -rwxr-xr-x 1 oracle oinstall 52378028 Jan 27 2011 /u01/app/oracle/product/11.2.0/dbhome_1/lib/libclntsh.so.11.1 ============================================================== ================================ fstab /sbin/mount.dbfs#/@fsdb.local /dbfs_direct fuse rw,user,direct_io,allow_other,wallet,noauto 0 0 ============================================================== ==== /var/log/messages

Mar 3 21:22:36 aodpdb02 auditd[5101]: Audit daemon rotating log files Mar 3 21:52:48 aodpdb02 DBFS: /u01/app/11.2.0/grid/crs/script/mount-dbfs.sh mounting DBFS at /dbfs_direct from database dbm Mar 3 21:52:49 aodpdb02 DBFS: ORACLE_SID is dbm2 Mar 3 21:52:49 aodpdb02 DBFS: spawning dbfs_client command using SID dbm2 Mar 3 21:52:49 aodpdb02 DBFS: Password: Mar 3 21:52:49 aodpdb02 dbfs_client[5383]: Fail to set up database connection. Mar 3 21:52:50 aodpdb02 DBFS: Start OFFLINE Mar 3 21:53:00 aodpdb02 DBFS: cleaning up DBFS using fusermount and kill on dbfs_client and mount.dbfs Mar 3 22:07:37 aodpdb02 auditd[5101]: Audit daemon rotating log files Oracle Support - 04 March 2012 08:13:44 GMT+05:30 [Notes] Generic Note -----------------------UPDATE ========= The problem seems related to locating the library files here . SOME QUESTIONS ================

a) Was it working previously ? b) Is the problem only node specific and happening on all the nodes c) Can you mount the mount points using mount -a ( as root user ) d) Are there any entries in /etc/fstab related to DBFS ACTION PLAN ---------------------a) Do you have this file /usr/local/lib/libclntsh.so.11.1 ? ( To check if file is present ) b) Can you list the output of ls -all /u01/app/oracle/product/11.2.0/dbhome_1/lib/libclntsh.so.11.1 c) Are you seeing all services up and running using crs ? d) Can you follow the article and let us know the result Errors when mounting a DBFS store using fstab due to missing libraries (Doc ID 1053008.1) e) Can you try export LD_LIBRARY_PATH=$ORACLE_HOME/lib and then see if that helps ? We might take strace if above does not resolve the case. f) We might check the script output of Configuring DBFS on Oracle Database Machine (Doc ID 1054431.1) Oracle Support - 04 March 2012 08:05:30 GMT+05:30 [ODM Research] Configuring DBFS on Oracle Database Machine (Doc ID 1054431.1) Errors when mounting a DBFS store using fstab due to missing libraries (Doc ID 1053008.1) Bug 9309554: ERRORS WHEN MOUNTING A DBFS STORE USING FSTAB DUE TO MISSING LIBRARIES Bug 10337293: TEMPORARILY UNABLE TO MOUNT DBFS eventually, after just waiting some time the mounts were able to work. Oracle Support - 04 March 2012 07:55:57 GMT+05:30 [ODM Issue Verification] Verified the issue in the log file as noted in SR Itself Oracle Support - 04 March 2012 07:48:56 GMT+05:30 [ODM Research] Configuring DBFS on Oracle Database Machine (Doc ID 1054431.1) Errors when mounting a DBFS store using fstab due to missing libraries (Doc ID 1053008.1) Oracle Support - 04 March 2012 07:46:13 GMT+05:30 [Call - Outbound] Call - Outbound -----------------------------Called +91-9052409998 spoke to Pavan who informed this is Half Rack Exadata environment . He just restarted the database using shutdown immediate command and after it came up , this problem is seen now. Database is up and running . Informed Pavan , the problem will be analyzed and SR will be updated with action plan. PAVAN.BOLLA@CLOROX.COM - 04 March 2012 07:45:31 GMT+05:30 [Update from Customer] Database Servers: Exadata Software

Version & Patches Exadata Image - Version: 11.2.2.2 Database Servers: Grid Infrastructure Version & Patches Version 11.2.0.2.0 BP3 - 10387939 Database Servers: Oracle RDBMS Version & Patches Version 11.2.0.2.0 BP3 - 10387939 Storage Servers Exadata Software and Patches Exadata Image - Version: 11.2.2.2 PAVAN.BOLLA@CLOROX.COM - 04 March 2012 07:35:16 GMT+05:30 [Update from Customer] Hi Support, We getting the below error while mounting dbfs_client in production /u01/app/oracle/product/11.2.0/dbhome_1/bin/dbfs_client: error while loading shared libraries: libclntsh.so.11.1: cannot open shared object file: No such file or directory Thanks, Pavan +91-9052409998 ASTHA.PATEL@ORACLE.COM - 04 March 2012 07:34:07 GMT+05:30 [Administration] #### SEVERITY 1 REQUEST TEMPLATE ##### Can you easily recover from, bypass or work around the problem? Does your system or application continue normally after the problem occurs? Are the standard features of the system or application still available; is the loss of service minor? Is this a Data Corruption issue? Do you have a Complete Loss of Service? Is this a Test or Production Environment? Are you unable to proceed with implementation and the go live date is in less than 3 days? Please provide a description of the business impact: ALL PHONE NUMBERS SHOULD INCLUDE THE COUNTRY CODE AND AREA CODE. Customers Management name: Customer's Management 24x7 contact number: Primary Customer contact name Current Customer 24x7 voice phone number Current Customer Email address: Current Customer Pager/Fax number: Secondary Contact Name:

Secondary Contact Phone number: Secondary Contact Email: Comments for this update: ACTION PLAN -------------------ASTHA.PATEL@ORACLE.COM - 04 March 2012 07:33:20 GMT+05:30 [Customer Problem Description] Problem Description: The customer will update SR with Problem Description