Using ADOP in Release 12.2 – Prepare phase details


When the prepare phase is run, files are copied/synchronized from the Run filesystem to the Patch filesystem and code stubs are created in the patch edition of the database. These stubs are instantiated when they are called the first time or the actualize_all parameter is used.

The second file system contains a copy of all the components that make up an application tier file system, including:

    • APPL_TOP – Oracle E-Business Suite code
    • INST_TOP- Instance Configuration Home
    • FMW_HOME – Oracle Weblogic Server Home and Oracle E-Business Suite Domain
    • ORACLE_HOME – Oracle Application Server Home, Forms, Reports
    • IAS_ORACLE_HOME – Oracle OHS Home
    • COMMON_TOP – Oracle E-Business Suite Java code, third-party libraries

During the prepare phase, adop performs the following steps:

  • Checks whether to perform a cleanup.
  • Validates the system configuration to ensure that the system is ready to start an online patching cycle.
  • Checks to see if the database is prepared for online patching:
  • Checks if the database user is edition-enabled.
  • Checks to see if the patch service has been created. adop requires that a special database service exists for the purpose of connecting to the patch edition. Its existence is validated when the prepare phase is run.
  • Checks to see if logon trigger exists and is enabled.
  • Checks the integrity of the database data dictionary, refer to MOS Note 1531121.1, Using the Online Patching Readiness Report in Oracle E-Business Suite Release 12.2.
  • Checks that the E-Business Suite Technology Codelevel Checker (ETCC) has has been run, to verify that all required patches have been applied to the database.
  • Checks system configuration on each application tier node to make sure that each application tier node is correctly registered, configured, and ready for patching.
  • Checks for the existence of the “Online Patching In Progress” (ADZDPATCH) concurrent program. This program prevents certain predefined concurrent programs from being started, and as such needs to be active while a patching cycle is in progress (that is, while a database patch edition exists).


If the ADZDPATCH program has not yet been requested to run, a request is submitted. If the request status is pending, it may be waiting for an incompatible program to finish. After incompatible programs finish, the ADZDPATCH status will change to running, and it will allow the prepare phase to proceed.

The next stage depends on whether the concurrent managers are running:

If the concurrent managers are all down, the prepare phase continues, with ADZDPATCH entering a status of pending (with the highest priority) until the managers are started.

If the concurrent managers are partially up, but there is no manager defined that can run ADZDPATCH, then the prepare phase will exit with an error.

If the concurrent managers are up, and there is one defined that can run ADZDPATCH, processing will loop until ADZDPATCH changes status from pending to running (that is to say, as noted in Step 2, no incompatible programs are found). The prepare phase then continues.

Note: ADZDPATCH is cancelled when the cutover phase is complete.

  • Invokes the TXK script $AD_TOP/patch/115/bin/ to synchronize the patches which have been applied to the run APPL_TOP, but not the patch APPL_TOP. The script depends on the adop repository for patches that have been applied on the run APPL_TOP but not the patch APPL_TOP.
  • Checks the database for the existence of a patch edition, and creates one if it does not find one. An example from the log file that shows the prepare checking for the patch edition:

EVENT      Prepare System

 700202704 09:00:10 00:00:00 ad.plsql.ad_zd.create_edition

STATEMENT  SQL: create edition V_20170316_0900

    EVENT      Patch edition not yet created by master node.

    EVENT      Will wait for another minute and retry.

  • Calls the $AD_TOP/patch/115/bin/ script again to confirm that the database connection to the patch edition is working.
  • Prepare does not run FS_CLONE. Prepare does run:



  • Prepare has the old default sync_mode of “patch”. This can be thought of as the file sync mode, because it synchronizes the files of the Run edition to the Patch edition. The new value for sync_mode that is introduced with AD-TXK Delta 8, is “delta”. Delta mode can use rsync to significantly speed up the prepare phase.

There are two values for sync_mode:

sync_mode = patch, and

sync_mode = delta, (uses rsync) – this is new in AD-TXK delta 8

The benefit besides being much faster, is that sync_mode=deltat also copies all customizations and ignores the adop_sync.drv.

Much faster than the other two methods, this delta synchronization method uses your choice of third-party utility to synchronize the file systems by copying files as applicable from the source directory to the destination directory, optionally ignoring any files and directories you may decide to specify in an exclusion file.

 To use this method, specify the parameter/value pair sync_mode=delta on the adop command line:

$ adop phase=prepare sync_mode=delta

The delta_sync_drv.txt file includes examples for setting up synchronization using rsync on UNIX.

How to Tell If a Patch Has been Applied in R12.2

You can check to see if a patch is really applied by using the AD_PATCH.IS_PATCH_APPLIED pl/sql function.

select AD_PATCH.IS_PATCH_APPLIED(\’$release\’,\’$appltop_id\’,\’$patch_no\’,\’$language\’) from dual;

example sql:
SELECT adb.bug_number,ad_patch.is_patch_applied(‘122’, 1045, adb.bug_number)
FROM ad_bugs adb
WHERE adb.bug_number in (20034256);

or for single app tier installations:
select ad_patch.is_patch_applied(‘R12’,-1,20034256) from dual;

expected results:
EXPLICIT = applied
NOT APPLIED = not applied / aborted

Find TruTek at Collaborate 17!

If you’re considering upgrading to Oracle E-Business Suite Release 12.2, then take a look at all these ways to learn from TruTek about the latest information about the Applications while you’re at Collaborate 17:

I’m Teaching, Presenting, and Paneling at Collaborate 17

R12.2.6 One Day Workshop at Collaborate 17 SUNDAY

If you’re planning to attend Collaborate 17, don’t miss out on my E-Business Suite Release 12.2 Upgrade Workshop: The Whole Nine Yards Workshop. This all day Pre-Conference Workshop on Sunday, April 2nd, will include a free copy of my newest book, the little r12.2.6 upgrade essentials for managers and team members. The workshop is filling up fast, so be sure to register early.

OAUG Customization & Alternatives SIG Panel MONDAY

I’ll be participating on the OAUG Customization & Alternatives SIG panel from 2:45-3:45pm on Monday, April 3rd in Jasmine F.

The Release 12.2.6 Upgrade: Online Patching and Customizations TUESDAY

I’ll be presenting The Release 12.2.6 Upgrade: Online Patching and Customizations on Tuesday at 2:45pm.

Want to score a free copy of my latest book, the little 12.2.6 upgrade essentials for managers and team members? All you have to do is find me (while supplies last)! I’ll bring some copies to Collaborate 17, and my friends at eprentise will have copies at their Booth 1523 in the Exhibit Hall.

Not attending Collaborate 17? You can buy a copy of my new book at:

TruTek’s Newest R12.2.6 Book is Here: the little r12.2.6 upgrade essentials for managers and team members

Putting together the right team to tackle the upgrade, and understanding the issues that the team needs to consider to be successful, can be quite a challenge. the little r12.2.6 upgrade essentials for managers and team members describes the big picture of what you need to consider before tackling the Release 12.2.6 E-Business Suite upgrade. Based on TruTek’s popular R12.1 to R12.2 Technical Upgrade classes, this book describes what managers, functional, and technical team members need to know to prepare to upgrade from Release 12.1 to Release 12.2 of Oracle’s E-Business Suite of Applications.

**You can order Mike Swing’s newest book, and his other Release 12.1.3 and 12.2 books now.**

Fast Backups and Restores are Critical during EBS Upgrades, Testing and Development

The process of upgrading an EBS system is one requiring dozens or hundreds of steps. Think of it like climbing a wall. Each handhold or foothold is analogous to one of the steps in the upgrade.

What happens if you miss a handhold or foothold?

If you are close to the ground, then the fall isn’t dangerous, and you can get restarted easily.

But what happens if you’re 25 or 35 feet off the ground?

A fall from that height can result in serious injury, unless you are strapped to a climbing rope to catch your fall.

And that is role that backups play in any lengthy process with many discrete steps, such as an EBS upgrade. Performing a backup of the systems you’re upgrading every half-dozen or dozen steps allows you to restore to the most recent backup and resume work while having only lost a relatively small amount of work.

But think of the costs of taking a full backup of the systems you’re upgrading. Let’s assume that the EBS database is 1 TB in size, the dbTier is 4 GB, and the appsTier is another 4 GB. All told each backup takes up over a TB of backup media and takes 2 hours to perform, start to finish. So, for an upgrade process of 100 steps, and performing a backup every dozen steps, means performing 8-9 backups, at 2 hours apiece, totaling 16-18 hours alone! Not only do the backups represent a significant portion of the total number of steps to perform, but those steps also consume a significant portion of the total elapsed time of the entire upgrade process.

So, in the event of a mistake or mishap, it will take about 2 hours to restore to the previous backup, and then the intervening steps have to be replayed, which might take another couple of painstaking hours.

Because of this high cost of failure, believe it that each step will be double-checked and triple-checked, perhaps by a second or third person, before it is committed or the ENTER key is pressed.

No wonder upgrading EBS is so exhausting and expensive!

The fastest operation is the one you never perform

What if you never had to perform all those backups, but you could still recover your entire EBS systems stack to any point-in-time as if you had?

This is what it is like to use data virtualization technology from Delphix. You start by linking a set of “source” EBS components into Delphix, and these sources are stored in compressed format within the Delphix appliance. Following that, these sources can produce virtual copies taking up very little storage in a matter of minutes. A Delphix administrator can provision virtual copies of the entire EBS systems stack into a single Delphix JetStream container for use by an individual or a team. The components of the container, which include the EBS database, the EBS dbTier, and the EBS appsTier, become the environment upon which the upgrade team is working.

Delphix captures all of the changes made to the EBS database, the EBS dbTier, and the EBS appsTier, which is called a timeflow. Additionally, Delphix JetStream containers allow users to create bookmarks, or named points-in-time, within the timeflow to speed up the location of a significant point-in-time, and easily restore to it.

So each developer or team can have their own private copy of the entire EBS systems stack. As the developer is stepping through each step of the upgrade, they can create a bookmark, which takes only seconds, after every couple steps. If they make a mistake or a mishap occurs, then they can rewind the entire JetStream container, consisting of the entire EBS systems stack, back to the desired bookmark in minutes, and then resume from there.

Let’s step through this, using the following example EBS system stack already linked and ready to use within a Delphix appliance…

An example EBS system stack in Delphix

So here we see a full E-Business Suites R12.1 system stack linked into Delphix. In the screenshot below, you can see three dSource objects in the left-hand navigation bar, under the Sources database group or folder. These three dSource objects are named…

  1. EBS R12.1 appsTier
  2. EBS R12.1 dbTechStack
  3. EBSDB

The first two items are vFiles, containing the file system sub-directories representing the E-Business Suites R12.1 applications tier and database tier, respectively. The last item is a virtual database or VDB, containing the Oracle database for the E-Business Suites R12.1 system. Each are indicated as dSources by the circular black dSource icon with the letters “dS” within (indicated below by the red arrows)…


Registering EBS dSources as JetStream templates

The three dSources have been encapsulated within the Delphix JetStream user-interface within a templates, as shown by the JetStream icons showing a tiny hand holding out a tiny cylinder representing a datasource (indicated below by the red arrows) just to the right of the dSource icons…


Within the JetStream user-interface itself, we see that all three dSources are encapsulated within one JetStream template, with the three dSources comprising the three datasources for the template…


Encapsulating EBS VDBs as JetStream containers

Once dSources are encapsulated as a JetStream template, then we can encapsulate VDBs and vFiles created from those dSources as JetStream containers. Here, we see the database group or folder called Targets circled in the left-hand navigation bar, and the blue icons for VDBs and vFiles with the letter “V” within are indicated by red arrow, while the darker icons for JetStream Containers are indicated by the green arrow…


Working with JetStream containers for EBS

With an entire EBS system stack under the control of Delphix JetStream, we can begin the lengthy process of upgrading EBS. Before we begin, we should create a JetStream bookmark called Upgrade Step 0 to indicate the starting point of the upgrade…


Then, after performing the first five steps of the upgrade in the EBS system stack, come back to the JetStream user-interface and create a JetStream bookmark named Upgrade Step 5, as shown below…


After every couple steps of the upgrade, or prior to any lengthy step, come back to the Delphix JetStream user-interface and create a bookmark named appropriate for the step in the upgrade process, as shown below…


Performing a rewind in a JetStream container for EBS

So we’re cruising along, performing productive steps in the EBS upgrade process, and not worrying about making backups.

WHOOPS! At upgrade step 59, we made a serious mistake!

If we had been making backups every 12 steps or so, then the most recent backup may have been Upgrade Step 48, which was 11 steps ago. Besides the multiple hours it may take to restore the backup in the database, the dbTier, and the appsTier, we are going to have to perform 11 steps over again.

In contrast, using Delphix JetStream, we have better choices. From the Delphix JetStream user-interface, we can restore to any point-in-time on the timeflow, whether it is one of the bookmarks we previously created at a known step in the upgrade process, or to any point-in-time location on the timeflow line, which may or may be located in the middle of one the upgrade steps, and thus not a viable point from which to restart.

So, because we want to restart the upgrade from a known and viable step in the process, we’re going to restore the entire EBS systems stack consistently back to the JetStream bookmark Upgrade Step 55


So, first let’s click on the JetStream bookmark named Upgrade Step 55 and then click on the icon for the Restore operation to start the rewind…


After being prompted Are You Sure and then confirming Yes I Am, then the Restore process starts to create a brand new timeflow for all three datasources in the container…


Please notice that the old timeflow is still visible and accessible to the left in the JetStream user-interface, even with the new timeflow being restored to the right.

Finally, the restore operation completes, and now all three parts of the EBS systems stack (i.e. appsTier, dbTier, and database) have been restored to Upgrade Step 55 in about 15 minutes elapsed time, a fraction of the 2 hours it would have taken to restore from backups.


Now we can resume the EBS upgrade process with barely any time lost!

Summary of benefits

So a few things to bear in mind from this example when using Delphix Jetstream…

  1. Once a JetStream container has been provisioned by the Delphix administrators (usually a role occupied by database administrators or DBAs), all the operations shown above are performed by JetStream data users, who are usually the developers performing the upgrade themselves
    • No need to coordinate backups or restores between teams == less time wasted
  2. Using Delphix removed the need to perform backups, because all changes across the timeflow of all three components of the EBS systems stack (i.e. appsTier, dbTechStack, and database) are recorded automatically within Delphix
    • No backups == less storage consumed == less time wasted
  3. Using Delphix removed the need to restore from backups, because the timeflow can be restored to any previous point-in-time in a fraction of the time it takes to perform a restore from backup
    • No restores == less time wasted

It is difficult to imagine not performing EBS upgrades using the methods we’ve used for decades, but the that time is here. Data virtualization, like server virtualization, is fast becoming the new norm, and it is changing everything for the better.


Attending Collaborate and Have an Oracle EBS Upgrade or Customization Project?


Check out Delphix at Booth #1613 and see how you can provision full-stake EBS environments in just minutes, automatically secure sensitive information, and lower TCO with space-efficient virtual data. You are invited to join Delphix for the following special events:

April 12th, Tuesday (at the Delphix Booth #1613)

  • 1:00 – 1:15        Delphix Demo
  • 1:15 – 2:00        How many instances do you need for an EBS Upgrade?Q&A with Mike Swing
  • 6:00 – 7:30        Happy Hour

April 13th, Wednesday (at the Delphix Booth #1613)

  • 2:00 – 2:45        Book signing with Mike Swing

Sessions you may be interested in:

Sunday, April 10:

  • Mike Swing’s Pre-Conference (added cost) 1 Day Workshop: E-Business Suite Release 12.2 Upgrade Workshop: The Whole Nine Yards, 9am-4pm, Breakers B
  • OAUG Archive and Purge SIG Meeting at 3:30pm

Monday, April 11:

  • Tim Gorman’s Linux/UNIX Tools for the Oracle DBA on April 11th at 9:15am, Palm A
  • Tim Gorman’s Accelerating DevOps Using Data Virtualization at 2:00pm, Jasmine B

Wednesday, April 13:

  • 11:45 –12:45     Oak Table World at Collaborate 16. Mandalay Bay Ballroom I
    • 11:45 – 11:55am – Internet of Things 101 – Alex Gorbachev, Pythian
    • 12:00 – 12:10pm – How Oracle Data Recovery Mechanisms Complicate Data Security, and What To Do About It, Tim Gorman, Delphix
    • 12:15: – 12:25pm – Some Useful Oracle Diagnostic Tricks – Tanel Poder, Gluent, Inc.
    • 12:30 – 12:40pm – Challenges and Solutions Masking Data – Kyle Hailey, Delphix
    • 12:45 – 12:55pm – Everything Exadata – Dan Norris, Oracle Corporation

TruTek’s Newest R12.2.5 Book is Ready: the little r12.2.5 upgrade essentials for managers and team members

littler1225upgradeessentialsguideFRONTcover 03-29-16wBorderPutting together the right team to tackle the upgrade, and understanding the issues that the team needs to consider to be successful, can be quite a challenge. _the little r12.2.5 upgrade essentials for managers and team members_ describes the big picture of what you need to consider before tackling the Release 12.2.5 E-Business Suite upgrade. Based on TruTek’s popular R12.1 to R12.2 Technical Upgrade classes, this book describes what managers, functional, and technical team members need to know to prepare to upgrade from Release 12.1 to Release 12.2 of Oracle’s E-Business Suite of Applications.

**You can order Mike Swing’s newest book, and his other Release 12.1.3 and 12.2 [books]( now.**

High Level Upgrade Plan to 12.2.5 with Database Migration to Linux


High Level Upgrade Plan to 12.2.5 with a Database Migration to Linux

        TASK NAME                                                             Duration     

  1. Procure Linux Hardware                                                     12d
  2. Install and configure OEL for EBS for DB                            2w
  3. Gather custom code                                                             1w
  4. Migrate Solaris database to Linux                                       1mo
  5. Develop Test Scripts                                                            1w
  6. Test Linux 64 bit DB server with Solaris Apps Tier              1w
  7. Freeze Patching of PROD                                                    0d
  8. Production Cutover to Linux Database servers                    2d
  9. Configure Linux Upgrade DB server – Install OEL                1w
  10. Clone Linux DB servers to Linux Upgrade server                 1w
  11. Configure Linux Apps Upgrade server- Install OEL              1w
  12. Install 12.2.0 on Linux 64 bit apps tier                                  1w
  13. Upgrade DB to on Upgrade Server                        1w
  14. Test DB with Solaris Apps Tier                               1w
  15. Upgrade PROD DB to                                             2d
  16. Clone PROD DB to Upgrade DB                                          1w
  17. Upgrade to 12.2.5                                                                 2w
  18. Develop and Apply Customizations to 12.2.5                       3w
  19. Test 12.2.5                                                                            3w
  20. Upgrade Pass #2 to 12.2.5                                                   2w
  21. Apply Customizations                                                           1w
  22. Test 12.2.5                                                                            1w
  23. Performance Upgrade to 12.2.5                                           1w
  24. Apply Customizations                                                           1w
  25. Test 12.2.5                                                                            1w
  26. Go Live                                                                                  4d


Installing startCD51 and Staging Release 12.2.5, Including of the Database

From Steven Chan’s blog,

“Instructions for downloading and applying this release update pack (RUP) for the EBS 12.2 codeline can be found here:

After going to


1225 download files1

Notice the Oracle Database files with most of the ETCC patches already applied:

1225 download files2

And, the Oracle Web Tier, software is already included. This makes updating to the latest Oracle Fusion Middleware much easier:

1225 download files3

After downloading all the files, running a md5sum to make sure they all downloaded correctly, I staged the files according to: R12.2: How To Create the Stage In Preparation For Installation (Doc ID 1596433.1)

I followed the notes that are included below, but had some additional fixes, in order for the install to work correctly:
1) build the stage according to note 1596433.1
2) chmod -R 755 /home/stage
3) xhost +
4) Clean out /etc/oraInventory/ContentsXML/inventory.xml
5) fix /etc/oraInst.loc follow note 1292347.1
6) to fix the database installer, rapidwiz didn’t show the actual error.
I went to /home/stage/TechInstallMedia/databas/database and ran
runInstaller to get the errors and to run the fixup script. Then
rapidwiz would work.

R12.2: How To Create the Stage In Preparation For Installation (Doc ID
MD5 Checksums for R12.2 Rapid Install Media (Doc ID 1505510.1)
[INS-10002] Inventory Initialization Failed or [INS-10102] Installer
Initialization Failed during Installation (Doc ID 1292347.1)

R12.2: How To Create the Stage In Preparation For Installation (Doc
ID 1596433.1)
MD5 Checksums for R12.2 Rapid Install Media (Doc ID 1505510.1)
[INS-10002] Inventory Initialization Failed or [INS-10102] Installer
Initialization Failed during Installation (Doc ID 1292347.1)

Patch Wizard Can Export its Patch Analysis Now!

Check out Patch 9703082, released in November 2015 for both Release 12.1 and 12.2. Oracle has added the ability to export your patch analysis from Patch Wizard. This works well for companies that want to document or research what objects will change if a patch is applied and send it to testers in an email. Until now, you either had to write your own script to pull the data out of the Patch Wizard tables, or point testers to the Patch Wizard screens. But there’s one caveat…

When you click on the Export button, it exports everything – a list of New Files, Changed Files, and Unchanged Files. We haven’t found a use for the Unchanged Files information, and the sheer quantity of records that this category produces can be overwhelming. Imagine if you analyzed RUP4 – your export file would contain thousands of Unchanged Files. Nonetheless, being able to send that export file via email is valuable, so here’s the workaround:

1. Export the file
2. Sort by Impact Type
3. Delete the Unchanged Files
4. Save and send to your tester