Prepare

Use adop phase=prepare sync_mode=delta. Sync_mode=delta is a new feature delivered in AD-TXK Delta 8. It is much faster than the previous sync_mode = patch. The new sync_mode=delta takes about 4-10 minutes on TruTek test servers. If the previous patch cycle was aborted, then FS_CLONE is run during the next prepare. The prepare phase can fail due to long running concurrent requests that are incompatible with the ADZDPATCH concurrent program.

FS_CLONE

Will copy all files including customizations and configurations – takes about 1 hour 40 minutes.Typiically, fs_clone only needs to be run after an abort. The usual sequence would be:

abort

cleanup (full)

fs_clone

Apply

We tested three apply modes: normal (default), hotpatch and downtime.

We found that downtime patching works, but the system must be down and the patch needs to be tested first in a test environment. Using hotpatch when the patch is not a hotpatch can result in a hung patch that can’t be aborted and can also cause an abandoned node. Neither downtime or hotpatch mode use an adop patch cycle and therefore cannot be aborted. Hotpatch mode requires the WebLogic Admin server to be running.

Finalize

Finalize prepares for the cutover by compiling invalid objects. This helps reduce the time cutover may need.

We found that if developers are applying hot custom code during finalize, that the compilation of objects can take a very long time (19 hours in one test)

Cutover

This is the most sensitive phase in ADOP for several reasons:

Users are not able to work during this phase, so the downtime needs to be very predictable

Any underlying code changes that affect tables, materialized views, and other objects may require the regeneration of logical views for tables, and regeneration of materialized views. The regeneration of materialized views from logical views is accomplished fairly quickly, however, repopulating the materialized view can take hours. I don’t think it is possible to prevent the regeneration of MVs, but we need to be able to predict the time it will take to regenerate MVs and other objects, so we can accurately advise the users/business of the downtime associated with each cutover.

The running concurrent requests need to be documented before we shutdown the concurrent managers. The ICM must also be shutdown. Otherwise, the cutover can take several hours to complete. This is because cutover waits for running concurrent requests to complete. It’s better to work with the business users and ask them to put long running requests on hold, before cutover starts.

We tested the cm_wait parameter (units are minutes). We found that no new requests were started, but the Internal Concurrent Manager is not shutdown/stopped. It took a long time for associated database sessions to be terminated.

Cleanup If a patch cycle has failed, it is possible to abort the patching cycle and return to normal runtime operation. Abort can be run for Prepare, Apply, Finalize, and Cutover phases. The abort command can only be used before successful completion of the cutover phase. After cutover, the system is running on the new edition, and abort is no longer possible for that patching cycle. An abort of prepare, apply or finalize, if complete on the master node and it fails on a slave node, will cause an abandoned node. There are two solutions: first, detach and re-attach the abandoned node. This can be very time consuming. If you have a “pairsfile” created ahead of time this can be done fairly quickly. There is also a  solution (undocumented on MOS) that has worked in all test cases, that requires shared disk on the apps tier. This solution is an update to the AD_ADOP_SESSIONS table, followed by an FS_CLONE.

Abort

This phase seems to be the easiest, and the patch session will determine if the cleanup needs to be standard or full. You can specify quick, but we’ve found it’s safest to let the cleanup determine what mode to use. If the proper mode is not used, then cleanup may be run during the prepare( followed by a fs_clone), extending the time for prepare.

Prepare

When the prepare phase is run, files are copied/synchronized from the Run filesystem to the Patch filesystem and code stubs are created in the patch edition of the database. These stubs are instantiated when they are called the first time or the actualize_all parameter is used.

The second file system contains a copy of all the components that make up an application tier file system, including:

• • APPL_TOP – Oracle E-Business Suite code
  • INST_TOP- Instance Configuration Home
  • FMW_HOME – Oracle Weblogic Server Home and Oracle E-Business Suite Domain
  • ORACLE_HOME – Oracle Application Server Home, Forms, Reports
  • IAS_ORACLE_HOME – Oracle OHS Home
  • COMMON_TOP – Oracle E-Business Suite Java code, third-party libraries

During the prepare phase, adop performs the following steps:

• Checks whether to perform a cleanup.
• Validates the system configuration to ensure that the system is ready to start an online patching cycle.
• Checks to see if the database is prepared for online patching:
• Checks if the database user is edition-enabled.
• Checks to see if the patch service has been created. adop requires that a special database service exists for the purpose of connecting to the patch edition. Its existence is validated when the prepare phase is run.
• Checks to see if logon trigger exists and is enabled.
• Checks the integrity of the database data dictionary, refer to MOS Note 1531121.1, Using the Online Patching Readiness Report in Oracle E-Business Suite Release 12.2.
• Checks that the E-Business Suite Technology Codelevel Checker (ETCC) has has been run, to verify that all required patches have been applied to the database.
• Checks system configuration on each application tier node to make sure that each application tier node is correctly registered, configured, and ready for patching.
• Checks for the existence of the “Online Patching In Progress” (ADZDPATCH) concurrent program. This program prevents certain predefined concurrent programs from being started, and as such needs to be active while a patching cycle is in progress (that is, while a database patch edition exists).

ADZDPATCH

If the ADZDPATCH program has not yet been requested to run, a request is submitted. If the request status is pending, it may be waiting for an incompatible program to finish. After incompatible programs finish, the ADZDPATCH status will change to running, and it will allow the prepare phase to proceed.

The next stage depends on whether the concurrent managers are running:

If the concurrent managers are all down, the prepare phase continues, with ADZDPATCH entering a status of pending (with the highest priority) until the managers are started.

If the concurrent managers are partially up, but there is no manager defined that can run ADZDPATCH, then the prepare phase will exit with an error.

If the concurrent managers are up, and there is one defined that can run ADZDPATCH, processing will loop until ADZDPATCH changes status from pending to running (that is to say, as noted in Step 2, no incompatible programs are found). The prepare phase then continues.

Note: ADZDPATCH is cancelled when the cutover phase is complete.

• Invokes the TXK script $AD_TOP/patch/115/bin/txkADOPPreparePhaseSynchronize.pl to synchronize the patches which have been applied to the run APPL_TOP, but not the patch APPL_TOP. The script depends on the adop repository for patches that have been applied on the run APPL_TOP but not the patch APPL_TOP.
• Checks the database for the existence of a patch edition, and creates one if it does not find one. An example from the log file that shows the prepare checking for the patch edition:

EVENT　 　 　 Prepare System

　700202704 09:00:10 00:00:00 ad.plsql.ad_zd.create_edition

STATEMENT　 SQL: create edition V_20170316_0900

　　  EVENT　 　 　 Patch edition not yet created by master node.

　 　 EVENT　 　 　 Will wait for another minute and retry.

• Calls the $AD_TOP/patch/115/bin/txkADOPPreparePhaseSanityCheck.pl script again to confirm that the database connection to the patch edition is working.
• Prepare does not run FS_CLONE. Prepare does run:

                    FsCloneStage

                    FsCloneApply

• Prepare has the old default sync_mode of “patch”. This can be thought of as the file sync mode, because it synchronizes the files of the Run edition to the Patch edition. The new value for sync_mode that is introduced with AD-TXK Delta 8, is “delta”. Delta mode can use rsync to significantly speed up the prepare phase.

There are two values for sync_mode:

sync_mode = patch, and

sync_mode = delta, (uses rsync) – this is new in AD-TXK delta 8

The benefit besides being much faster, is that sync_mode=deltat also copies all customizations and ignores the adop_sync.drv.

Much faster than the other two methods, this delta synchronization method uses your choice of third-party utility to synchronize the file systems by copying files as applicable from the source directory to the destination directory, optionally ignoring any files and directories you may decide to specify in an exclusion file.

 To use this method, specify the parameter/value pair sync_mode=delta on the adop command line:

$ adop phase=prepare sync_mode=delta

The delta_sync_drv.txt file includes examples for setting up synchronization using rsync on UNIX.