Overview of Online Patching Phases

Prepare

Use adop phase=prepare sync_mode=delta. Sync_mode=delta is a new feature delivered in AD-TXK Delta 8. It is much faster than the previous sync_mode = patch. The new sync_mode=delta takes about 4-10 minutes on TruTek test servers. If the previous patch cycle was aborted, then FS_CLONE is run during the next prepare. The prepare phase can fail due to long running concurrent requests.

FS_CLONE

Will copy all files including customizations and configurations – takes about 1 hour 40 minutes

Apply

We tested three apply modes: normal (default), hotpatch and downtime.

We found that downtime patching works, but the system must be down and the patch needs to be tested first in a test environment. Using hotpatch when the patch is not a hotpatch can result in a hung patch that can’t be aborted and can also cause an abandoned node

Finalize

We found that if developers are applying hot custom code during finalize, that the compilation of objects can take a very long time (19 hours in one test)

Cutover

This is the most sensitive phase in ADOP for several reasons:

Users are not able to work during this phase, so the downtime needs to be very predictable

Any underlying code changes that affect tables, materialized views, and other objects require the regeneration of logical views for tables, and regeneration of materialized views. The regeneration of materialized can take a long time, 1 hour plus. I don’t think it is possible to prevent the regeneration of MVs, but we need to be able to predict the time it will take to regenerate MVs and other objects, so we can accurately advise the users/business of the downtime associated with cutover.

The concurrent requests need to be documented before we shutdown the concurrent managers. The ICM must also be shutdown. Otherwise, the cutover can take several hours to complete.

We tested the cm_wait parameter (units are minutes). We found that no new requests were started, but the Internal Concurrent Manager is not shutdown/stopped.

Cleanup If a patch cycle has failed, it is possible to abort the patching cycle and return to normal runtime operation. Abort can be run for Prepare, Apply, Finalize, and Cutover phases The abort command can only be used before successful completion of the cutover phase. After cutover, the system is running on the new edition, and abort is no longer possible for that patching cycle. An abort of prepare, apply or finalize, if complete on the master node and it fails on a slave node, will cause an abandoned node. There are two solutions: first, detach and re-attach the abandoned node. This can be very time consuming. There is also a documented solution that has worked in all test cases, that requires shared disk on the apps tier. This solution is an update to the AD_ADOP_SESSIONS table, followed by an FS_CLONE.

Abort

This phase seems to be the easiest, and the patch session will determine if the cleanup needs to be standard or full. You can specify quick, but we’ve found it’s safest to let the cleanup determine what mode to use. If the proper mode is not used, then cleanup may be run during the prepare, extending the time for prepare.

Brief Comparison of R12.1.3 and R12.2.6 Upgrades

The table below shows the major comparable steps in the 11i to 12.1.3 upgrade and the 12.1.3 to 12.2.6. The highlighted sections in the 12.2.6 upgrade indicate the new steps relative to past upgrades. These new sections, while they don’t take very long to complete during the downtime window, require extensive effort to prepare the instance for the upgrade.

For example, the more customizations you have, the longer it will take to make sure your customizations are minimally compliant to the Release 12.2 development standards. The process of updating your customizations to match new table structures is still a major part of fixing your customizations, but now you must also make sure your customizations are at least minimally compliant.

The second new step highlighted below requires your database (DB) and middle tier (MT) to be patched to a level specified in the ETCC scripts, checkDBpatch.sh and checkMTpatch.sh. While these scripts are easy to run, it’s not always straightforward to resolve patch conflicts in the database or the middle tier. This step is complicated because the list of required patches is continuously changing.

It’s not so much that these new steps are difficult, but they are new and different from all previous upgrades and take some time to understand and implement.

Upgrade to 12.1.3                               Upgrade to 12.2.6

11i  -> 12.1.3                                       12.1.3  -> 12.2.6

Major Functional Changes                  Major Technical Changes

11i pre-upgrade patches                     12.1.3 pre-upgrade patches

Upgrade to 12.1.1 (16 hours)              Upgrade to 12.2.0 (8 hours)

1) Online Enablement Patch –

Requires minimal compliance

for customizations

2) DB and MT to have all patches

specified by ETCC

Upgrade to 12.1.3                                Upgrade to 12.2.6

 

Using ADOP in Release 12.2 – Prepare phase details

Prepare

When the prepare phase is run, files are copied/synchronized from the Run filesystem to the Patch filesystem and code stubs are created in the patch edition of the database. These stubs are instantiated when they are called the first time or the actualize_all parameter is used.

The second file system contains a copy of all the components that make up an application tier file system, including:

    • APPL_TOP – Oracle E-Business Suite code
    • INST_TOP- Instance Configuration Home
    • FMW_HOME – Oracle Weblogic Server Home and Oracle E-Business Suite Domain
    • ORACLE_HOME – Oracle Application Server Home, Forms, Reports
    • IAS_ORACLE_HOME – Oracle OHS Home
    • COMMON_TOP – Oracle E-Business Suite Java code, third-party libraries

During the prepare phase, adop performs the following steps:

  • Checks whether to perform a cleanup.
  • Validates the system configuration to ensure that the system is ready to start an online patching cycle.
  • Checks to see if the database is prepared for online patching:
  • Checks if the database user is edition-enabled.
  • Checks to see if the patch service has been created. adop requires that a special database service exists for the purpose of connecting to the patch edition. Its existence is validated when the prepare phase is run.
  • Checks to see if logon trigger exists and is enabled.
  • Checks the integrity of the database data dictionary, refer to MOS Note 1531121.1, Using the Online Patching Readiness Report in Oracle E-Business Suite Release 12.2.
  • Checks that the E-Business Suite Technology Codelevel Checker (ETCC) has has been run, to verify that all required patches have been applied to the database.
  • Checks system configuration on each application tier node to make sure that each application tier node is correctly registered, configured, and ready for patching.
  • Checks for the existence of the “Online Patching In Progress” (ADZDPATCH) concurrent program. This program prevents certain predefined concurrent programs from being started, and as such needs to be active while a patching cycle is in progress (that is, while a database patch edition exists).

ADZDPATCH

If the ADZDPATCH program has not yet been requested to run, a request is submitted. If the request status is pending, it may be waiting for an incompatible program to finish. After incompatible programs finish, the ADZDPATCH status will change to running, and it will allow the prepare phase to proceed.

The next stage depends on whether the concurrent managers are running:

If the concurrent managers are all down, the prepare phase continues, with ADZDPATCH entering a status of pending (with the highest priority) until the managers are started.

If the concurrent managers are partially up, but there is no manager defined that can run ADZDPATCH, then the prepare phase will exit with an error.

If the concurrent managers are up, and there is one defined that can run ADZDPATCH, processing will loop until ADZDPATCH changes status from pending to running (that is to say, as noted in Step 2, no incompatible programs are found). The prepare phase then continues.

Note: ADZDPATCH is cancelled when the cutover phase is complete.

  • Invokes the TXK script $AD_TOP/patch/115/bin/txkADOPPreparePhaseSynchronize.pl to synchronize the patches which have been applied to the run APPL_TOP, but not the patch APPL_TOP. The script depends on the adop repository for patches that have been applied on the run APPL_TOP but not the patch APPL_TOP.
  • Checks the database for the existence of a patch edition, and creates one if it does not find one. An example from the log file that shows the prepare checking for the patch edition:

EVENT      Prepare System

 700202704 09:00:10 00:00:00 ad.plsql.ad_zd.create_edition

STATEMENT  SQL: create edition V_20170316_0900

    EVENT      Patch edition not yet created by master node.

    EVENT      Will wait for another minute and retry.

  • Calls the $AD_TOP/patch/115/bin/txkADOPPreparePhaseSanityCheck.pl script again to confirm that the database connection to the patch edition is working.
  • Prepare does not run FS_CLONE. Prepare does run:

                    FsCloneStage

                    FsCloneApply

  • Prepare has the old default sync_mode of “patch”. This can be thought of as the file sync mode, because it synchronizes the files of the Run edition to the Patch edition. The new value for sync_mode that is introduced with AD-TXK Delta 8, is “delta”. Delta mode can use rsync to significantly speed up the prepare phase.

There are two values for sync_mode:

sync_mode = patch, and

sync_mode = delta, (uses rsync) – this is new in AD-TXK delta 8

The benefit besides being much faster, is that sync_mode=deltat also copies all customizations and ignores the adop_sync.drv.

Much faster than the other two methods, this delta synchronization method uses your choice of third-party utility to synchronize the file systems by copying files as applicable from the source directory to the destination directory, optionally ignoring any files and directories you may decide to specify in an exclusion file.

 To use this method, specify the parameter/value pair sync_mode=delta on the adop command line:

$ adop phase=prepare sync_mode=delta

The delta_sync_drv.txt file includes examples for setting up synchronization using rsync on UNIX.

How to Tell If a Patch Has been Applied in R12.2

You can check to see if a patch is really applied by using the AD_PATCH.IS_PATCH_APPLIED pl/sql function.

Usage:
select AD_PATCH.IS_PATCH_APPLIED(\’$release\’,\’$appltop_id\’,\’$patch_no\’,\’$language\’) from dual;

example sql:
SELECT adb.bug_number,ad_patch.is_patch_applied(‘122’, 1045, adb.bug_number)
FROM ad_bugs adb
WHERE adb.bug_number in (20034256);

or for single app tier installations:
select ad_patch.is_patch_applied(‘R12’,-1,20034256) from dual;

expected results:
EXPLICIT = applied
NOT APPLIED = not applied / aborted

Find TruTek at Collaborate 17!

If you’re considering upgrading to Oracle E-Business Suite Release 12.2, then take a look at all these ways to learn from TruTek about the latest information about the Applications while you’re at Collaborate 17:

I’m Teaching, Presenting, and Paneling at Collaborate 17

R12.2.6 One Day Workshop at Collaborate 17 SUNDAY

If you’re planning to attend Collaborate 17, don’t miss out on my E-Business Suite Release 12.2 Upgrade Workshop: The Whole Nine Yards Workshop. This all day Pre-Conference Workshop on Sunday, April 2nd, will include a free copy of my newest book, the little r12.2.6 upgrade essentials for managers and team members. The workshop is filling up fast, so be sure to register early.

OAUG Customization & Alternatives SIG Panel MONDAY

I’ll be participating on the OAUG Customization & Alternatives SIG panel from 2:45-3:45pm on Monday, April 3rd in Jasmine F.

The Release 12.2.6 Upgrade: Online Patching and Customizations TUESDAY

I’ll be presenting The Release 12.2.6 Upgrade: Online Patching and Customizations on Tuesday at 2:45pm.

Want to score a free copy of my latest book, the little 12.2.6 upgrade essentials for managers and team members? All you have to do is find me (while supplies last)! I’ll bring some copies to Collaborate 17, and my friends at eprentise will have copies at their Booth 1523 in the Exhibit Hall.

Not attending Collaborate 17? You can buy a copy of my new book at: http://www.lulu.com/content/paperback-book/the-little-r1226-upgrade-essentials/20707296

TruTek’s Newest R12.2.6 Book is Here: the little r12.2.6 upgrade essentials for managers and team members

Putting together the right team to tackle the upgrade, and understanding the issues that the team needs to consider to be successful, can be quite a challenge. the little r12.2.6 upgrade essentials for managers and team members describes the big picture of what you need to consider before tackling the Release 12.2.6 E-Business Suite upgrade. Based on TruTek’s popular R12.1 to R12.2 Technical Upgrade classes, this book describes what managers, functional, and technical team members need to know to prepare to upgrade from Release 12.1 to Release 12.2 of Oracle’s E-Business Suite of Applications.

**You can order Mike Swing’s newest book, and his other Release 12.1.3 and 12.2 books now.**

Fast Backups and Restores are Critical during EBS Upgrades, Testing and Development

The process of upgrading an EBS system is one requiring dozens or hundreds of steps. Think of it like climbing a wall. Each handhold or foothold is analogous to one of the steps in the upgrade.

What happens if you miss a handhold or foothold?

If you are close to the ground, then the fall isn’t dangerous, and you can get restarted easily.

But what happens if you’re 25 or 35 feet off the ground?

A fall from that height can result in serious injury, unless you are strapped to a climbing rope to catch your fall.

And that is role that backups play in any lengthy process with many discrete steps, such as an EBS upgrade. Performing a backup of the systems you’re upgrading every half-dozen or dozen steps allows you to restore to the most recent backup and resume work while having only lost a relatively small amount of work.

But think of the costs of taking a full backup of the systems you’re upgrading. Let’s assume that the EBS database is 1 TB in size, the dbTier is 4 GB, and the appsTier is another 4 GB. All told each backup takes up over a TB of backup media and takes 2 hours to perform, start to finish. So, for an upgrade process of 100 steps, and performing a backup every dozen steps, means performing 8-9 backups, at 2 hours apiece, totaling 16-18 hours alone! Not only do the backups represent a significant portion of the total number of steps to perform, but those steps also consume a significant portion of the total elapsed time of the entire upgrade process.

So, in the event of a mistake or mishap, it will take about 2 hours to restore to the previous backup, and then the intervening steps have to be replayed, which might take another couple of painstaking hours.

Because of this high cost of failure, believe it that each step will be double-checked and triple-checked, perhaps by a second or third person, before it is committed or the ENTER key is pressed.

No wonder upgrading EBS is so exhausting and expensive!

The fastest operation is the one you never perform

What if you never had to perform all those backups, but you could still recover your entire EBS systems stack to any point-in-time as if you had?

This is what it is like to use data virtualization technology from Delphix. You start by linking a set of “source” EBS components into Delphix, and these sources are stored in compressed format within the Delphix appliance. Following that, these sources can produce virtual copies taking up very little storage in a matter of minutes. A Delphix administrator can provision virtual copies of the entire EBS systems stack into a single Delphix JetStream container for use by an individual or a team. The components of the container, which include the EBS database, the EBS dbTier, and the EBS appsTier, become the environment upon which the upgrade team is working.

Delphix captures all of the changes made to the EBS database, the EBS dbTier, and the EBS appsTier, which is called a timeflow. Additionally, Delphix JetStream containers allow users to create bookmarks, or named points-in-time, within the timeflow to speed up the location of a significant point-in-time, and easily restore to it.

So each developer or team can have their own private copy of the entire EBS systems stack. As the developer is stepping through each step of the upgrade, they can create a bookmark, which takes only seconds, after every couple steps. If they make a mistake or a mishap occurs, then they can rewind the entire JetStream container, consisting of the entire EBS systems stack, back to the desired bookmark in minutes, and then resume from there.

Let’s step through this, using the following example EBS system stack already linked and ready to use within a Delphix appliance…

An example EBS system stack in Delphix

So here we see a full E-Business Suites R12.1 system stack linked into Delphix. In the screenshot below, you can see three dSource objects in the left-hand navigation bar, under the Sources database group or folder. These three dSource objects are named…

  1. EBS R12.1 appsTier
  2. EBS R12.1 dbTechStack
  3. EBSDB

The first two items are vFiles, containing the file system sub-directories representing the E-Business Suites R12.1 applications tier and database tier, respectively. The last item is a virtual database or VDB, containing the Oracle database for the E-Business Suites R12.1 system. Each are indicated as dSources by the circular black dSource icon with the letters “dS” within (indicated below by the red arrows)…

Delphix1

Registering EBS dSources as JetStream templates

The three dSources have been encapsulated within the Delphix JetStream user-interface within a templates, as shown by the JetStream icons showing a tiny hand holding out a tiny cylinder representing a datasource (indicated below by the red arrows) just to the right of the dSource icons…

Delphix2

Within the JetStream user-interface itself, we see that all three dSources are encapsulated within one JetStream template, with the three dSources comprising the three datasources for the template…

Delphix3 

Encapsulating EBS VDBs as JetStream containers

Once dSources are encapsulated as a JetStream template, then we can encapsulate VDBs and vFiles created from those dSources as JetStream containers. Here, we see the database group or folder called Targets circled in the left-hand navigation bar, and the blue icons for VDBs and vFiles with the letter “V” within are indicated by red arrow, while the darker icons for JetStream Containers are indicated by the green arrow…

Delphix4

Working with JetStream containers for EBS

With an entire EBS system stack under the control of Delphix JetStream, we can begin the lengthy process of upgrading EBS. Before we begin, we should create a JetStream bookmark called Upgrade Step 0 to indicate the starting point of the upgrade…

Delphix5

Then, after performing the first five steps of the upgrade in the EBS system stack, come back to the JetStream user-interface and create a JetStream bookmark named Upgrade Step 5, as shown below…

Delphix6

After every couple steps of the upgrade, or prior to any lengthy step, come back to the Delphix JetStream user-interface and create a bookmark named appropriate for the step in the upgrade process, as shown below…

Delphix7

Performing a rewind in a JetStream container for EBS

So we’re cruising along, performing productive steps in the EBS upgrade process, and not worrying about making backups.

WHOOPS! At upgrade step 59, we made a serious mistake!

If we had been making backups every 12 steps or so, then the most recent backup may have been Upgrade Step 48, which was 11 steps ago. Besides the multiple hours it may take to restore the backup in the database, the dbTier, and the appsTier, we are going to have to perform 11 steps over again.

In contrast, using Delphix JetStream, we have better choices. From the Delphix JetStream user-interface, we can restore to any point-in-time on the timeflow, whether it is one of the bookmarks we previously created at a known step in the upgrade process, or to any point-in-time location on the timeflow line, which may or may be located in the middle of one the upgrade steps, and thus not a viable point from which to restart.

So, because we want to restart the upgrade from a known and viable step in the process, we’re going to restore the entire EBS systems stack consistently back to the JetStream bookmark Upgrade Step 55

Delphix8

So, first let’s click on the JetStream bookmark named Upgrade Step 55 and then click on the icon for the Restore operation to start the rewind…

Delphix9

After being prompted Are You Sure and then confirming Yes I Am, then the Restore process starts to create a brand new timeflow for all three datasources in the container…

Delphix10

Please notice that the old timeflow is still visible and accessible to the left in the JetStream user-interface, even with the new timeflow being restored to the right.

Finally, the restore operation completes, and now all three parts of the EBS systems stack (i.e. appsTier, dbTier, and database) have been restored to Upgrade Step 55 in about 15 minutes elapsed time, a fraction of the 2 hours it would have taken to restore from backups.

Delphix11

Now we can resume the EBS upgrade process with barely any time lost!

Summary of benefits

So a few things to bear in mind from this example when using Delphix Jetstream…

  1. Once a JetStream container has been provisioned by the Delphix administrators (usually a role occupied by database administrators or DBAs), all the operations shown above are performed by JetStream data users, who are usually the developers performing the upgrade themselves
    • No need to coordinate backups or restores between teams == less time wasted
  2. Using Delphix removed the need to perform backups, because all changes across the timeflow of all three components of the EBS systems stack (i.e. appsTier, dbTechStack, and database) are recorded automatically within Delphix
    • No backups == less storage consumed == less time wasted
  3. Using Delphix removed the need to restore from backups, because the timeflow can be restored to any previous point-in-time in a fraction of the time it takes to perform a restore from backup
    • No restores == less time wasted

It is difficult to imagine not performing EBS upgrades using the methods we’ve used for decades, but the that time is here. Data virtualization, like server virtualization, is fast becoming the new norm, and it is changing everything for the better.

 

Attending Collaborate and Have an Oracle EBS Upgrade or Customization Project?

Delphix_Primary_CMYK

Check out Delphix at Booth #1613 and see how you can provision full-stake EBS environments in just minutes, automatically secure sensitive information, and lower TCO with space-efficient virtual data. You are invited to join Delphix for the following special events:

April 12th, Tuesday (at the Delphix Booth #1613)

  • 1:00 – 1:15        Delphix Demo
  • 1:15 – 2:00        How many instances do you need for an EBS Upgrade?Q&A with Mike Swing
  • 6:00 – 7:30        Happy Hour

April 13th, Wednesday (at the Delphix Booth #1613)

  • 2:00 – 2:45        Book signing with Mike Swing

Sessions you may be interested in:

Sunday, April 10:

  • Mike Swing’s Pre-Conference (added cost) 1 Day Workshop: E-Business Suite Release 12.2 Upgrade Workshop: The Whole Nine Yards, 9am-4pm, Breakers B
  • OAUG Archive and Purge SIG Meeting at 3:30pm

Monday, April 11:

  • Tim Gorman’s Linux/UNIX Tools for the Oracle DBA on April 11th at 9:15am, Palm A
  • Tim Gorman’s Accelerating DevOps Using Data Virtualization at 2:00pm, Jasmine B

Wednesday, April 13:

  • 11:45 –12:45     Oak Table World at Collaborate 16. Mandalay Bay Ballroom I
    • 11:45 – 11:55am – Internet of Things 101 – Alex Gorbachev, Pythian
    • 12:00 – 12:10pm – How Oracle Data Recovery Mechanisms Complicate Data Security, and What To Do About It, Tim Gorman, Delphix
    • 12:15: – 12:25pm – Some Useful Oracle Diagnostic Tricks – Tanel Poder, Gluent, Inc.
    • 12:30 – 12:40pm – Challenges and Solutions Masking Data – Kyle Hailey, Delphix
    • 12:45 – 12:55pm – Everything Exadata – Dan Norris, Oracle Corporation

TruTek’s Newest R12.2.5 Book is Ready: the little r12.2.5 upgrade essentials for managers and team members

littler1225upgradeessentialsguideFRONTcover 03-29-16wBorderPutting together the right team to tackle the upgrade, and understanding the issues that the team needs to consider to be successful, can be quite a challenge. _the little r12.2.5 upgrade essentials for managers and team members_ describes the big picture of what you need to consider before tackling the Release 12.2.5 E-Business Suite upgrade. Based on TruTek’s popular R12.1 to R12.2 Technical Upgrade classes, this book describes what managers, functional, and technical team members need to know to prepare to upgrade from Release 12.1 to Release 12.2 of Oracle’s E-Business Suite of Applications.

**You can order Mike Swing’s newest book, and his other Release 12.1.3 and 12.2 [books](http://www.lulu.com/spotlight/trutek) now.**

High Level Upgrade Plan to 12.2.5 with Database Migration to Linux

Aside

High Level Upgrade Plan to 12.2.5 with a Database Migration to Linux

        TASK NAME                                                             Duration     

  1. Procure Linux Hardware                                                     12d
  2. Install and configure OEL for EBS for DB                            2w
  3. Gather custom code                                                             1w
  4. Migrate Solaris database to Linux                                       1mo
  5. Develop Test Scripts                                                            1w
  6. Test Linux 64 bit DB server with Solaris Apps Tier              1w
  7. Freeze Patching of PROD                                                    0d
  8. Production Cutover to Linux Database servers                    2d
  9. Configure Linux Upgrade DB server – Install OEL                1w
  10. Clone Linux DB servers to Linux Upgrade server                 1w
  11. Configure Linux Apps Upgrade server- Install OEL              1w
  12. Install 12.2.0 on Linux 64 bit apps tier                                  1w
  13. Upgrade DB to 12.1.0.2 on Upgrade Server                        1w
  14. Test 12.1.0.2 DB with Solaris Apps Tier                               1w
  15. Upgrade PROD DB to 12.1.0.2                                             2d
  16. Clone PROD DB to Upgrade DB                                          1w
  17. Upgrade to 12.2.5                                                                 2w
  18. Develop and Apply Customizations to 12.2.5                       3w
  19. Test 12.2.5                                                                            3w
  20. Upgrade Pass #2 to 12.2.5                                                   2w
  21. Apply Customizations                                                           1w
  22. Test 12.2.5                                                                            1w
  23. Performance Upgrade to 12.2.5                                           1w
  24. Apply Customizations                                                           1w
  25. Test 12.2.5                                                                            1w
  26. Go Live                                                                                  4d