Installing Oracle RAC DB on RHEL5 ?

Installing Oracle RAC DB on RHEL5 ?


Go thru the following,
1] http://www.oracle.com/technology/pub/articles/smiley_rac10g_install.html
2] Oracle metalink Note: 421308.1 - Requirements For Installing Oracle10gR2 On RHEL/OEL 5 (x86_64)
3] To map the devices using udev either singlepath or multipath.
NOTE:465001.1 - Configuring raw devices (singlepath) for Oracle Clusterware 10g Release 2 (10.2.0) on RHEL5/OEL5
NOTE:564580.1 - Configuring raw devices (multipath) for Oracle Clusterware 10g Release 2 (10.2.0) on RHEL5/OEL5


i have faced the following problems,


1] Configuring ssh for the first time works fine. If any changes done ( Eg: i changed the home path of oracle user),
then the ssh configuration will prompt for password no matter how many times you reconfigure the ssh.
i have created another OS user orclusr instead of oracle. If any one finds a solution, please reply back.
2] i have commented the localhost entry in the /etc/hosts for the xmanager to work.
3] use runisntaller -ignoreSysPrereqs
4] You will not face much problems in single patching, but in multi pathing while installing
crs (during root.sh), it will say the crs upgrade fails, it is an oracle bug Bug.4679769. apply the patch, it will copy new executable.
5] The vipca will fail because of the oracle bug refer NOTE:414163.1 , comment out or unset the LD_ASSUME_KERNEL in vipca executable.
6] During upgrade to 10.2.0.4, the local_listener error came and we have commented out that in the parameter file.

Error: libXp.so.6: cannot open shared object file while installing Oracle 10gR2

Error: libXp.so.6: cannot open shared object file
During installation of Oracle database 10g on RHEL 5.5, i have encountered the following error,

Exception java.lang.UnsatisfiedLinkError: /tmp/OraInstall2005-07-07_09-40-45AM/jre/1.4.2/lib/i386/libawt.so: libXp.so.6: cannot open shared object file: No such file or directory occurred.. java.lang.UnsatisfiedLinkError: /tmp/OraInstall2005-07-07_09-40-45AM/jre/1.4.2/lib/i386/libawt.so: libXp.so.6: cannot open shared object file: No such file or directory

Solution: Install libXp-1.0.0-8.1.el5.i386.rpm package.

\command was unexpected this time while starting Oracle weblogic 11.1.1.X.X

Due to scheduled power maintenance the server where Oracle 11G FM  11.1.1.X.X installed was rebooted and after reboot the Oracle Weblogic, WLS_FORMS and WLS_REPORTS didn't start when tried in the command line, 

Error Description - \command was unexpected this time while starting the Oracle Weblogic through command line.

Weblogic service and WLS_FORMS and WLS_REPORTS service didn't startup after double clicking it manually os starting through command line.         

The issue is due to the newly installed software in the Apps server which by default gets installed in  C:\program files and make an entry  in the PATH environment variable of the system. Go through the below support note for more details and action plan to overcome the error.

New Install Of ECM 11g On A 64 Bit Windows OS - Error Starting WebLogic [ID 1217713.1]

The problem occurs mainly in Windows platform. 

Failure of server APACHE bridge Error in Oracle Fusion Middle ware.

Even you found the Oracle weblogic, WLS_FORMS, WLS_REPORTS are up and in running state while accessing application you face “Failure of server APACHE bridge    

“Error

Oracle 11G FM  11.1.1.X.X          

Error Description - Failure of server APACHE bridge

  

Go through the Oracle document ID 1304095.1 "changed the hostname to the IP address in forms.conf and reports.conf file ($ORACLE_INSTANCE\config\OHS\ohs1\moduleconf\) as directed to overcome the issue,

The symptoms are most likely to occur Windows platform.

fatal error the application will terminate while installing/upgrading the Oracle weblogic release 10.3.X.X

We faced a strange issues while installing Oracle Web logic  10.3.3/10.3.4 in Windows 2008 R2 OS, the weblogic fails with fatal error the application will terminate. 

We appended the  - log & -log_priority options along to debug the issue. 

An example of command is given below: 

C:\Oracle\Softwares> java -jar <weblogic.jar> -log=weblog_inst.log -log_priority=debug

executing the above command will lead to the GUI, once the installation fails check the error logs created to resolve the issues.

java.lang.IllegalArgumentException: Negative time at java.io.File.setLastModified(File.java:1258) at com.bea.plateng.common.util.Touch.touchFile(Touch.java:99) at com.bea.plateng.common.util.Touch.touchFile(Touch.java:96) at com.bea.plateng.common.util.Touch.touch(Touch.java:79) at com.bea.plateng.wizard.silent.tasks.TouchTask.execute(TouchTask.java:268) at com.bea.plateng.wizard.silent.tasks.AbstractSilentTask.run(AbstractSilentTask.java:28) at java.lang.Thread.run(Thread.java:662)            Date was changed 2554 year to 2011(check OS sys date)

The possible reasons behind the errors are 

1) Date can be changed to a higher  value say 2554 year instead of 2011 (check OS system date)

2) Check the Date language settings which might be set to regional languages instead of English, here in my customer environment it's been set to THAI.

We change the Date settings to English to overcome the issue. 

EM console not working after upgrading the database to 10.2.0.4 or 10.2.0.5

After upgrading the Oracle database to 10.2.0.4 from 10.2.0.1, the EM console was not working and it throws errors when tried to start in command line. This error is generic to all platforms but we faced issues in Windows 2003 SP2 server after upgrading the Oracle database to 10.2.0.4.

Os Version – Windows 2008 SP2, Oracle 10g 10.2.0.4    

We tried to recreate the EM console with emca utility but it eventually failed with the below errors,

2011-07-01 17:21:44 Thread-6752 ERROR pingManager: nmepm_pingReposURL: Cannot connect to https://PGDB.sstc.local:1158/em/upload/: retStatus=-12011-07-01 17:21:45 Thread-8296 ERROR ssl: Open wallet failed, ret = 28750,2011-07-01 17:21:45 Thread-8296 ERROR http: 1568: Error initializing SSL connection for incoming request, aborting request. Ret=-1

These errors will be created under the ORACLE_HOME/sysman/logs

"2011-07-01 17:21:44 Thread-6752 ERROR pingManager: nmepm_pingReposURL: Cannot connect to https://PGDB.sstc.local:1158/em/upload/: retStatus=-12011-07-01 17:21:45 Thread-8296 ERROR ssl: Open wallet failed, ret = 28750,2011-07-01 17:21:45 Thread-8296 ERROR http: 1568: Error initializing SSL connection for incoming request, aborting request. Ret=-12011-07-01 17:21:45 Thread-1820 ERROR ssl: Open wallet failed, ret = 28750

2011-07-01 17:21:45 Thread-1820 ERROR ssl: nmehlenv_openWallet failed

Followed the Oracle support note ( 1222603.1) to overcome the issue Enterprise Manager Database Control Configuration - Recovering From Errors Due to CA Expiry on Oracle Database 10[1][1].2.0.4 or 10.2.0.5 [1222603.1]

Go-through the given metalink note and apply the patch 8350262 to resolve the issue. Enterprise Manager Database Control Configuration - Recovering From Errors Due to CA Expiry on Oracle Database 10[1][1].2.0.4 or 10.2.0.5 Video.

ORA-39006, ORA-39213: Metadata processing is not available while using data pump in Oracle 11g R2

While using Oracle datapump export/imporrt we faced a strange error given below 

ORA-39006: internal error ORA-39213: Metadata processing is not available, ORA-39006: internal error

Os  - Windows 2008 SP2         

Oracle 11gR2 11.2.0.1

Conn as sysdba and execute the below command,

EXECUTE SYS.DBMS_METADATA_UTIL.LOAD_STYLESHEETS;

Ref Metalink note: 394569.1 

ORA-00600: internal error code, arguments: [kzxcInitLoadLocal-7] in alert log file

We faced this ORA-00600 Internal issue 

Os version - Windows

Oracle Software - Oracle 11g 11.2.0.1           

ERROR at line 1: ORA-00600: internal error code, arguments: [kzxcInitLoadLocal-7], [64131], 

[ORA-64131: XMLIndex Metadata: failure during the looking up of the dictionary 

ORA-30966: error detected in the XML Index layer 

ORA-30966: error detected in the XML Index layer ], [], [], [], [], [], [], [], [], [] 

ORA-06512: at "XDB.DBMS_XDB", line 437 ORA-06512: at line 39

Recreated the XDB in order to overcome this error.

Connect as sysdba

spool xdb_reins.log

Shutdown Immediate;

Startup;

@?/rdbms/admin/catnoqm.sql

--IMPORTANT YOU HAVE TO SHUTDOWN AND STARTUP BETWEEN REMOVAL AND REINSTALL

Shutdown Immediate;

Startup;

@?/rdbms/admin/catqm.sql xdb xdb temp YES

spool off"           

It's better to do this activity after shutting down the listener. This action plan is taken from Oracle metalink, i will share the Ref doc ID asap.

Reports Servers Fail to Start and Stay In "Init" Status

This error commonly occurs only in Windows 2003 SP2 environment 

Oracle software details - Oracle 10g AS 10.1.2.3.0         

When we tried to start the report server from command line the report server didn't start and stayed in the INIT status for a long time. When we tried to start the report server through rwserver.exe we saw the report server starting and shutting down intermittently and then dies after the maximum retry operations

Places to check –

"-One of the following environment variables is pointing to a directory name that contains spaces (e.g:C:\DOCUMENTS AND SETTINGS\oracle\Local Settings\Temp ):

-TEMP

-TMP

-REPORTS_TMP

Or :

-The directory ""%ORACLE_HOME%\bin"" is not the first entry in the PATH environment variable (%ORACLE_HOME% in this case is the oracle home for the Middle Tier on which the report server exists).1)-Open the Windows registry and navigate to the following key:

HKEY_LOCAL_MACHINE\SOFTWARE\ORACLE\HOMEn

where n is the number of the ORACLE_HOME containing the installation.(You can find the ORACLE key name in the file <ORACLE_HOME>\bin\oracle.key)

-Then ,edit REPORTS_TMP and set the value to a valid directory without spaces for example: ""c:\temp"" .

Reference Doc - Reports Servers Fail to Start and Stay In "Init" Status [ID 579980.1]           

ORA-00600: internal error code, arguments: [kkzlcsl-1] while importing

Oracle 10g DB version 10.2.0.1          

Error - ORA-00600: [kkzlcsl-1] 

OS details - Windows 2003 SP 2

while importing. IMP-00017 IMP-00003 ORA-00942 we faced few issues while checking the alert log we fund the Oracle internal errors getting generated at the same time when the import is failing ORA-00600: internal error code, arguments: [kkzlcsl-1] 

Follow the metalink document, IMP-00017 IMP-00003 ORA-00942 USING FROMUSER & TOUSER ON SNAPSHOT LOG IMPORT ID 1017292.102. to overcome the import errors.

Enq: WF Contention tops in AWR report

This is one of the peculiar issue face in recent times, Enq: WF Contention topping the AWR report and the application users were unable to connect and work the database as the performance of the database was drastically going down. 

Enq: WF Contention    

Enq; WF Contention it's basically the MMON processes is periodically flush ASH data into AWR tables. Oracle introduced the WF enqueue which is used to serialize the flushing of snapshot. But if this is the top event (30% of the database time), then it is a really issue. The ADDM report suggests to investigate the wait in service "SYS$BACKGORUND".       

To overcome the Enq; WF Contention we need to Flush and re-create the AWR repository. 

Execute the below steps to re-create the AWR repository

Login as sysdba 

@?/rdbms/admin/catnoawr.sql;

@?/rdbms/admin/catawrtb.sql;

At the end run utlrp .sql to compile invalid objects.

Note: Instance reboot is mandatory after completing the above activity.

Go through this doc for more details ORA-13518 : 'Invalid Database Id' On Call to DBMS_WORKLOAD_REPOSITORY Package [ID 984447.1]

EXP-00003: no storage definition found for segment(%lu, %lu)

This export error is very generic, we faced this error in one of our customer site wr.to his Oracle and OS version


Oracle Software version - Oracle 10g 10.2.0.4                 

OS Version - HP-UX, 11.31  

Error while exporting a schema - EXP-00003: no storage definition found for segment(%lu, %lu)            

Oracle Bug Note ID [ID 154480.1] , Workaround is to use system/manager for the export E.g: exp system/system file=C:\ABC.dmp log=abc.log owner=ABC statisitcs=none

Intermittent FRM-92101 Error while accessing the application

Oracle 10g AS 10.1.2.0.2 to 10.1.2.3 

Intermittent FRM-92101 Error while accessing the application, this is due to the HTTP_Server OPMN process re-starts itself. 

Use the command opmnctl status -l to check the uptime of the Oracle OPMN processes, you can observe the time difference between the HTTP and other Oracle processes like DSA, OC4J. 

HTTP_Server ias-component restarts intermittently, displaying the following errors in the ipm.log found under $O_H/opmn/log folder

11/01/02 08:36:42 [4] Starting Process: HTTP_Server~HTTP_Server~HTTP_Server~1 (1165033639:0)

11/01/02 08:36:44 [4] Process Alive: HTTP_Server~HTTP_Server~HTTP_Server~1 (1165033639:9604)

11/01/02 09:24:24 [4] [libopmnohs] Process Ping Failed: HTTP_Server~HTTP_Server~HTTP_Server~1 (1165033639:9604) [The connection SSL receive timed out]

11/01/02 09:25:15 [4] [libopmnohs] Process Ping Failed: HTTP_Server~HTTP_Server~HTTP_Server~1 (1165033639:9604) [The connection SSL receive timed out]

11/01/02 09:26:06 [4] [libopmnohs] Process Ping Failed: HTTP_Server~HTTP_Server~HTTP_Server~1 (1165033639:9604) [The connection SSL receive timed out]

11/01/02 09:26:06 [4] [libopmnohs] Process Unreachable: HTTP_Server~HTTP_Server~HTTP_Server~1 (1165033639:9604)"   

Follow the metalink document OPMN Frequently Restarts HTTP_Server After Ping Failed [ID 553564.1] to overcome the frequent HTTP_server restart issue. 

File not found : F:\ORACLE_HOME\owb\*.ear while installing Oracle 11gR2( 11.2.0.1) Database.

This is a peculiar type of issue we faced while installing Oracle 11g R2 in the Windows 2008 server.

Os Details - Windows 2008 SP2 (x64)

 Software details  - Oracle Database 11g R2 version 11.2.0.1

While the installation was in progress in the GUI suddenly the installer screen got exited with this error, 

File not found : F:\ORACLE_HOME\owb\*.ear

First looking into the error we suspected the source files been corrupted, so we copied another source file and ended up with same error, turning to support.oracle documents we found the issue is not due to the installation file it's due to extracting the two zip files in two diff locations.

The solution to overcome this problem is to extract both the source zip files in same folder before proceeding with the installation of the Oracle 11g R2.

Registry component DBMS_SQLPA was found invalid after upgrading Oracle database from 10.2.0.1 to 10.2.0.4

After upgrading the ORACLE_HOME to 10.2.0.4 we checked the dba_registry and found the database components  DBMS_SQLPA was found invalid state.

Even after upgrading and running the utlrp.sql the component DBMS_SQLPA was found invalid.             

We searched the support documents and followed the below action plan and resolved the issues.

Connect to sqlplus as  sqlplus '/ as sysdba'

drop table plan_table;                                                               

@?/rdbms/admin/utlxplan

@?/rdbms/admin/prvtspao.plb                                            

@?/rdbms/admin/utlrp.sql                                      

select comp_name, status from dba_registry;        

Automating the Oracle Weblogic, WLS_FORMS & WLS_REPORTS in Windows OS - Oracle 11g Fusion Middleware.

Step: 1

Create a text file
%MIDDLEWARE_HOME%\user_projects\domains\servers\AdminServer\security\boot.properties

Add the following lines
username=webadmin
password=system01

Reference:Oracle support Note 980292.1 How to Start a WebLogic 10.3.x Domain AdminServer Without Interactively Supplying the Username / Password?

Step: 2

For troubleshooting / debugging purposes it is helpful to redirect standard out and error to a text file. backup and edit installSvc.cmd available in the path D:\Oracle\Middleware\wlserver_10.3\server\bin and change the line at the bottom of the file so it include the -log parameter e.g

rem *** Install the service
"%WL_HOME%\server\bin\beasvc" -install
-svcname:"%DOMAIN_NAME%_%SERVER_NAME%"
-javahome:"%JAVA_HOME%" -execdir:"%USERDOMAIN_HOME%"
-extrapath:"%WL_HOME%\server\bin" -password:"%WLS_PW%"
-cmdline:%CMDLINE%
-log:" D:\Oracle\Middleware\user_projects\domains\oracle11gfmserver\%SERVER_NAME%-stdout.txt"

Step: 3

Create a command script having the lines given below:

SETLOCAL
set DOMAIN_NAME=ClassicDomain
set USERDOMAIN_HOME=C:\middleware\FMW11g\user_projects\domains\ClassicDomain
set SERVER_NAME=AdminServer
set PRODUCTION_MODE=true
cd %USERDOMAIN_HOME%
call %USERDOMAIN_HOME%\bin\setDomainEnv.cmd
call "C:\middleware\FMW11g\wlserver_10.3\server\bin\installSvc.cmd"
ENDLOCAL

create a script for installAdmServer_Service.cmd

SETLOCAL
set DOMAIN_NAME=oracle11gfmserver
set USERDOMAIN_HOME=D:\Oracle\Middleware\user_projects\domains\oracle11gfmserverset SERVER_NAME=AdminServer
set PRODUCTION_MODE=true
cd %USERDOMAIN_HOME%
call %USERDOMAIN_HOME%\bin\setDomainEnv.cmd
call "D:\Oracle\Middleware\wlserver_10.3\server\bin\installSvc.cmd"
ENDLOCAL

Similarly create a script for installWLS_FORMS_Service.cmd

SETLOCAL
set DOMAIN_NAME=oracle11gfmserverset USERDOMAIN_HOME=D:\Oracle\Middleware\user_projects\domains\oracle11gfmserverset SERVER_NAME=WLS_FORMS
set PRODUCTION_MODE=true
set ADMIN_URL=http://oracle11gfmserver:7001
cd %USERDOMAIN_HOME%
call %USERDOMAIN_HOME%\bin\setDomainEnv.cmd
call "D:\Oracle\Middleware\wlserver_10.3\server\bin\installSvc.cmd"
ENDLOCAL

Similarly create a script for installWLS_REPORTS_Service.cmd

SETLOCAL
set DOMAIN_NAME=oracle11gfmserverset USERDOMAIN_HOME=D:\Oracle\Middleware\user_projects\domains\oracle11gfmserver

 set SERVER_NAME=WLS_REPORTS
set PRODUCTION_MODE=true
set ADMIN_URL=http://oracle11gfmserver:7001
cd %USERDOMAIN_HOME%
call %USERDOMAIN_HOME%\bin\setDomainEnv.cmd
call "D:\Oracle\Middleware\wlserver_10.3\server\bin\installSvc.cmd"
ENDLOCAL

Note: the highlighted part should be modified wr.to the environment.

Step: 4

So at the end of step 3 we must be having three bat files

installAdmServer_Service.cmd
installWLS_FORMS_Service.cmd
installWLS_REPORTS_Service.cmd

Executing one by one will create a service in the services.msc starting with the name beasvc ClassicDomain_WLS_FORMS. The Service "Startup Type" will be 'Automatic.

Step: 5

So now after every reboot the Oracle weblogic, forms and reports services will startup automatically. If services statuses are not up please check the log files (%stdout.txt) available under the path D:\Oracle\Middleware\user_projects\domains\ornrep for troubleshooting the issues during starup.

Reference Metalink note: How to Install MS Windows Services For FMW 11g WebLogic Domain Admin and Managed Servers? [ID 1060058.1]