Fast Backups and Restores are Critical during EBS Upgrades, Testing and Development

The process of upgrading an EBS system is one requiring dozens or hundreds of steps. Think of it like climbing a wall. Each handhold or foothold is analogous to one of the steps in the upgrade.

What happens if you miss a handhold or foothold?

If you are close to the ground, then the fall isn’t dangerous, and you can get restarted easily.

But what happens if you’re 25 or 35 feet off the ground?

A fall from that height can result in serious injury, unless you are strapped to a climbing rope to catch your fall.

And that is role that backups play in any lengthy process with many discrete steps, such as an EBS upgrade. Performing a backup of the systems you’re upgrading every half-dozen or dozen steps allows you to restore to the most recent backup and resume work while having only lost a relatively small amount of work.

But think of the costs of taking a full backup of the systems you’re upgrading. Let’s assume that the EBS database is 1 TB in size, the dbTier is 4 GB, and the appsTier is another 4 GB. All told each backup takes up over a TB of backup media and takes 2 hours to perform, start to finish. So, for an upgrade process of 100 steps, and performing a backup every dozen steps, means performing 8-9 backups, at 2 hours apiece, totaling 16-18 hours alone! Not only do the backups represent a significant portion of the total number of steps to perform, but those steps also consume a significant portion of the total elapsed time of the entire upgrade process.

So, in the event of a mistake or mishap, it will take about 2 hours to restore to the previous backup, and then the intervening steps have to be replayed, which might take another couple of painstaking hours.

Because of this high cost of failure, believe it that each step will be double-checked and triple-checked, perhaps by a second or third person, before it is committed or the ENTER key is pressed.

No wonder upgrading EBS is so exhausting and expensive!

The fastest operation is the one you never perform

What if you never had to perform all those backups, but you could still recover your entire EBS systems stack to any point-in-time as if you had?

This is what it is like to use data virtualization technology from Delphix. You start by linking a set of “source” EBS components into Delphix, and these sources are stored in compressed format within the Delphix appliance. Following that, these sources can produce virtual copies taking up very little storage in a matter of minutes. A Delphix administrator can provision virtual copies of the entire EBS systems stack into a single Delphix JetStream container for use by an individual or a team. The components of the container, which include the EBS database, the EBS dbTier, and the EBS appsTier, become the environment upon which the upgrade team is working.

Delphix captures all of the changes made to the EBS database, the EBS dbTier, and the EBS appsTier, which is called a timeflow. Additionally, Delphix JetStream containers allow users to create bookmarks, or named points-in-time, within the timeflow to speed up the location of a significant point-in-time, and easily restore to it.

So each developer or team can have their own private copy of the entire EBS systems stack. As the developer is stepping through each step of the upgrade, they can create a bookmark, which takes only seconds, after every couple steps. If they make a mistake or a mishap occurs, then they can rewind the entire JetStream container, consisting of the entire EBS systems stack, back to the desired bookmark in minutes, and then resume from there.

Let’s step through this, using the following example EBS system stack already linked and ready to use within a Delphix appliance…

An example EBS system stack in Delphix

So here we see a full E-Business Suites R12.1 system stack linked into Delphix. In the screenshot below, you can see three dSource objects in the left-hand navigation bar, under the Sources database group or folder. These three dSource objects are named…

  1. EBS R12.1 appsTier
  2. EBS R12.1 dbTechStack
  3. EBSDB

The first two items are vFiles, containing the file system sub-directories representing the E-Business Suites R12.1 applications tier and database tier, respectively. The last item is a virtual database or VDB, containing the Oracle database for the E-Business Suites R12.1 system. Each are indicated as dSources by the circular black dSource icon with the letters “dS” within (indicated below by the red arrows)…


Registering EBS dSources as JetStream templates

The three dSources have been encapsulated within the Delphix JetStream user-interface within a templates, as shown by the JetStream icons showing a tiny hand holding out a tiny cylinder representing a datasource (indicated below by the red arrows) just to the right of the dSource icons…


Within the JetStream user-interface itself, we see that all three dSources are encapsulated within one JetStream template, with the three dSources comprising the three datasources for the template…


Encapsulating EBS VDBs as JetStream containers

Once dSources are encapsulated as a JetStream template, then we can encapsulate VDBs and vFiles created from those dSources as JetStream containers. Here, we see the database group or folder called Targets circled in the left-hand navigation bar, and the blue icons for VDBs and vFiles with the letter “V” within are indicated by red arrow, while the darker icons for JetStream Containers are indicated by the green arrow…


Working with JetStream containers for EBS

With an entire EBS system stack under the control of Delphix JetStream, we can begin the lengthy process of upgrading EBS. Before we begin, we should create a JetStream bookmark called Upgrade Step 0 to indicate the starting point of the upgrade…


Then, after performing the first five steps of the upgrade in the EBS system stack, come back to the JetStream user-interface and create a JetStream bookmark named Upgrade Step 5, as shown below…


After every couple steps of the upgrade, or prior to any lengthy step, come back to the Delphix JetStream user-interface and create a bookmark named appropriate for the step in the upgrade process, as shown below…


Performing a rewind in a JetStream container for EBS

So we’re cruising along, performing productive steps in the EBS upgrade process, and not worrying about making backups.

WHOOPS! At upgrade step 59, we made a serious mistake!

If we had been making backups every 12 steps or so, then the most recent backup may have been Upgrade Step 48, which was 11 steps ago. Besides the multiple hours it may take to restore the backup in the database, the dbTier, and the appsTier, we are going to have to perform 11 steps over again.

In contrast, using Delphix JetStream, we have better choices. From the Delphix JetStream user-interface, we can restore to any point-in-time on the timeflow, whether it is one of the bookmarks we previously created at a known step in the upgrade process, or to any point-in-time location on the timeflow line, which may or may be located in the middle of one the upgrade steps, and thus not a viable point from which to restart.

So, because we want to restart the upgrade from a known and viable step in the process, we’re going to restore the entire EBS systems stack consistently back to the JetStream bookmark Upgrade Step 55


So, first let’s click on the JetStream bookmark named Upgrade Step 55 and then click on the icon for the Restore operation to start the rewind…


After being prompted Are You Sure and then confirming Yes I Am, then the Restore process starts to create a brand new timeflow for all three datasources in the container…


Please notice that the old timeflow is still visible and accessible to the left in the JetStream user-interface, even with the new timeflow being restored to the right.

Finally, the restore operation completes, and now all three parts of the EBS systems stack (i.e. appsTier, dbTier, and database) have been restored to Upgrade Step 55 in about 15 minutes elapsed time, a fraction of the 2 hours it would have taken to restore from backups.


Now we can resume the EBS upgrade process with barely any time lost!

Summary of benefits

So a few things to bear in mind from this example when using Delphix Jetstream…

  1. Once a JetStream container has been provisioned by the Delphix administrators (usually a role occupied by database administrators or DBAs), all the operations shown above are performed by JetStream data users, who are usually the developers performing the upgrade themselves
    • No need to coordinate backups or restores between teams == less time wasted
  2. Using Delphix removed the need to perform backups, because all changes across the timeflow of all three components of the EBS systems stack (i.e. appsTier, dbTechStack, and database) are recorded automatically within Delphix
    • No backups == less storage consumed == less time wasted
  3. Using Delphix removed the need to restore from backups, because the timeflow can be restored to any previous point-in-time in a fraction of the time it takes to perform a restore from backup
    • No restores == less time wasted

It is difficult to imagine not performing EBS upgrades using the methods we’ve used for decades, but the that time is here. Data virtualization, like server virtualization, is fast becoming the new norm, and it is changing everything for the better.


Attending Collaborate and Have an Oracle EBS Upgrade or Customization Project?


Check out Delphix at Booth #1613 and see how you can provision full-stake EBS environments in just minutes, automatically secure sensitive information, and lower TCO with space-efficient virtual data. You are invited to join Delphix for the following special events:

April 12th, Tuesday (at the Delphix Booth #1613)

  • 1:00 – 1:15        Delphix Demo
  • 1:15 – 2:00        How many instances do you need for an EBS Upgrade?Q&A with Mike Swing
  • 6:00 – 7:30        Happy Hour

April 13th, Wednesday (at the Delphix Booth #1613)

  • 2:00 – 2:45        Book signing with Mike Swing

Sessions you may be interested in:

Sunday, April 10:

  • Mike Swing’s Pre-Conference (added cost) 1 Day Workshop: E-Business Suite Release 12.2 Upgrade Workshop: The Whole Nine Yards, 9am-4pm, Breakers B
  • OAUG Archive and Purge SIG Meeting at 3:30pm

Monday, April 11:

  • Tim Gorman’s Linux/UNIX Tools for the Oracle DBA on April 11th at 9:15am, Palm A
  • Tim Gorman’s Accelerating DevOps Using Data Virtualization at 2:00pm, Jasmine B

Wednesday, April 13:

  • 11:45 –12:45     Oak Table World at Collaborate 16. Mandalay Bay Ballroom I
    • 11:45 – 11:55am – Internet of Things 101 – Alex Gorbachev, Pythian
    • 12:00 – 12:10pm – How Oracle Data Recovery Mechanisms Complicate Data Security, and What To Do About It, Tim Gorman, Delphix
    • 12:15: – 12:25pm – Some Useful Oracle Diagnostic Tricks – Tanel Poder, Gluent, Inc.
    • 12:30 – 12:40pm – Challenges and Solutions Masking Data – Kyle Hailey, Delphix
    • 12:45 – 12:55pm – Everything Exadata – Dan Norris, Oracle Corporation

TruTek’s Newest R12.2.5 Book is Ready: the little r12.2.5 upgrade essentials for managers and team members

littler1225upgradeessentialsguideFRONTcover 03-29-16wBorderPutting together the right team to tackle the upgrade, and understanding the issues that the team needs to consider to be successful, can be quite a challenge. _the little r12.2.5 upgrade essentials for managers and team members_ describes the big picture of what you need to consider before tackling the Release 12.2.5 E-Business Suite upgrade. Based on TruTek’s popular R12.1 to R12.2 Technical Upgrade classes, this book describes what managers, functional, and technical team members need to know to prepare to upgrade from Release 12.1 to Release 12.2 of Oracle’s E-Business Suite of Applications.

**You can order Mike Swing’s newest book, and his other Release 12.1.3 and 12.2 [books]( now.**

High Level Upgrade Plan to 12.2.5 with Database Migration to Linux


High Level Upgrade Plan to 12.2.5 with a Database Migration to Linux

        TASK NAME                                                             Duration     

  1. Procure Linux Hardware                                                     12d
  2. Install and configure OEL for EBS for DB                            2w
  3. Gather custom code                                                             1w
  4. Migrate Solaris database to Linux                                       1mo
  5. Develop Test Scripts                                                            1w
  6. Test Linux 64 bit DB server with Solaris Apps Tier              1w
  7. Freeze Patching of PROD                                                    0d
  8. Production Cutover to Linux Database servers                    2d
  9. Configure Linux Upgrade DB server – Install OEL                1w
  10. Clone Linux DB servers to Linux Upgrade server                 1w
  11. Configure Linux Apps Upgrade server- Install OEL              1w
  12. Install 12.2.0 on Linux 64 bit apps tier                                  1w
  13. Upgrade DB to on Upgrade Server                        1w
  14. Test DB with Solaris Apps Tier                               1w
  15. Upgrade PROD DB to                                             2d
  16. Clone PROD DB to Upgrade DB                                          1w
  17. Upgrade to 12.2.5                                                                 2w
  18. Develop and Apply Customizations to 12.2.5                       3w
  19. Test 12.2.5                                                                            3w
  20. Upgrade Pass #2 to 12.2.5                                                   2w
  21. Apply Customizations                                                           1w
  22. Test 12.2.5                                                                            1w
  23. Performance Upgrade to 12.2.5                                           1w
  24. Apply Customizations                                                           1w
  25. Test 12.2.5                                                                            1w
  26. Go Live                                                                                  4d


Installing startCD51 and Staging Release 12.2.5, Including of the Database

From Steven Chan’s blog,

“Instructions for downloading and applying this release update pack (RUP) for the EBS 12.2 codeline can be found here:

After going to


1225 download files1

Notice the Oracle Database files with most of the ETCC patches already applied:

1225 download files2

And, the Oracle Web Tier, software is already included. This makes updating to the latest Oracle Fusion Middleware much easier:

1225 download files3

After downloading all the files, running a md5sum to make sure they all downloaded correctly, I staged the files according to: R12.2: How To Create the Stage In Preparation For Installation (Doc ID 1596433.1)

I followed the notes that are included below, but had some additional fixes, in order for the install to work correctly:
1) build the stage according to note 1596433.1
2) chmod -R 755 /home/stage
3) xhost +
4) Clean out /etc/oraInventory/ContentsXML/inventory.xml
5) fix /etc/oraInst.loc follow note 1292347.1
6) to fix the database installer, rapidwiz didn’t show the actual error.
I went to /home/stage/TechInstallMedia/databas/database and ran
runInstaller to get the errors and to run the fixup script. Then
rapidwiz would work.

R12.2: How To Create the Stage In Preparation For Installation (Doc ID
MD5 Checksums for R12.2 Rapid Install Media (Doc ID 1505510.1)
[INS-10002] Inventory Initialization Failed or [INS-10102] Installer
Initialization Failed during Installation (Doc ID 1292347.1)

R12.2: How To Create the Stage In Preparation For Installation (Doc
ID 1596433.1)
MD5 Checksums for R12.2 Rapid Install Media (Doc ID 1505510.1)
[INS-10002] Inventory Initialization Failed or [INS-10102] Installer
Initialization Failed during Installation (Doc ID 1292347.1)