We’ve just finished two successful Release 12.2.6 First Pass Upgrades in the last six weeks. We always encounter and resolve a few unique issues with each client. Here are two examples.

The first issue occurred while running Patch 10124646 (the 12.2.0 upgrade patch), and the second occurred after running that patch.

1. During the 10124646 patch (the upgrade to 12.2.0), with about 22000 jobs completed and about 10000 jobs remaining, we saw this error:

ATTENTION: All workers either have failed or are waiting:

FAILED: file CustMigrationTool.class on worker 1.
FAILED: file CustMigrationTool.class on worker 2.
FAILED: file CustMigrationTool.class on worker 3.
FAILED: file CustMigrationTool.class on worker 4.
FAILED: file EDRXDOMigration.class on worker 5.
FAILED: file CustMigrationTool.class on worker 6.
FAILED: file CustMigrationTool.class on worker 7.
FAILED: file CustMigrationTool.class on worker 8.
FAILED: file CustMigrationTool.class on worker 9.

According to the ad worker log: adworker001.log
Exception in thread “main” java.sql.SQLException: ORA-00604: error occurred at recursive SQL level 1
ORA-04063: package body “APPS.FND_TRACE” has errors
ORA-06508: PL/SQL: could not find program unit being called: “APPS.FND_TRACE”
ORA-06512: at line 5

at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:462)
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:397)
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:389)
at oracle.jdbc.driver.T4C7Ocommoncall.processError(T4C7Ocommoncall.java:100)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:481)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:205)
at oracle.jdbc.driver.T4C7Ocommoncall.doOLOGOFF(T4C7Ocommoncall.java:64)
at oracle.jdbc.driver.T4CConnection.logoff(T4CConnection.java:583)
at oracle.jdbc.driver.PhysicalConnection.close(PhysicalConnection.java:5222)
at oracle.apps.edr.util.EDRXDOMigration.main(EDRXDOMigration.java:466)

We found the partial solution in MOS Doc. ID: 1959572.1, CustMigrationTool.class Error Upgrading from EBS R12.1.3 to R12.2 as the Connection Is Not Being Made In the 60 Seconds Defined In sqlnet.ora, which says that the connection couldn’t be made in the 60 seconds timeout value defined in sqlnet.ora, so the value needs to be changed to 120 seconds. To resolve the issue:
1. Change sqlnet.ora, SQLNET.INBOUND_CONNECT_TIMEOUT=120
2. Restart the db server
3. Restart the patch

However, that was not the only solution that was required. It turns out that using Recommended Patch Collections (RPC) creates objects that have a higher version than objects delivered by the 12.2.0 upgrade. This means the 12.2.0 upgrade will not replace objects with a higher version number and the upgrade fails. We had to find the correct versions of the code that worked together and then drop a trigger in order to get the FND_TRACE package body and specification to compile. Very interesting.
_____________________________________________________________

2. When running the “Configure the Instance” step, we couldn’t run AutoConfig on the database tier. This step is from the “Finish the Upgrade” section of the 12.2 upgrade guide on page 3-8.

We found MOS Doc. ID: 2117174.1, 12.2 Adop Cleanup Fails with Error “ETCC not run in the Database Node” After Running ETCC on the Database Node.

This note suggests that either the /etc/host file is not correctly formatted, or the nodes don’t match in the TXK_TCC_RESULTS table or the FND_NODES table. While this note is not a direct “hit”, we did find the domain was appended to the node_name. We updated the node_name in the TXK_TCC_RESULTS table to fix the issue:

update APPLSYS.TXK_TCC_RESULTS set node_name=’data2′ where node_name =’data2.ca.local’;

Email sherri@trutek.com to discuss pricing or schedule a time to have us help you with a First Pass Upgrade. We have completed the First Pass Upgrade in as few as three weeks, but it can take longer if we experience new issues. Please see our webpage for more information about our First Pass Upgrades consulting engagement. We also provide training to the rest of your managers, DBAs and development staff to help them understand the best practices of Online Patching.