Setting up automated Essbase Migrations in 11.1.1.3 with Life Cycle Management
What is LCM?
One of most attractive features of version 11 is Life Cycle Management. It is by far the biggest step that Oracle has taken in a true migration utility that has made migrating applications from DEV to TEST to PROD easier than ever before. But it does come with its share of nuances.
LCM actually existed in 9.3, but it was more of a command line tool that was integrated with Workspace, and it did not really offer a good solution to security and provisioning. LCM is now integrated (and installed) with Shared Services with a much easier to use GUI interface.
What it does
LCM migrates entire applications or individual “artifacts” between environments. If the environments share a common Shares Services, the migration can go directly from application to application. However, more commonly, the environments use separate Shared Services, so most migrations are done by exporting and importing to/from the file system.
The exports are stored in $HYPERION_HOME/common/import_export/<user>
Each user has directory here that stores their exported LCM artifacts. The exports are organized into a logical directory structure. Some use this structure to ZIP up entire applications and use them for version control and release management.
For Essbase, LCM will copy most everything that is needed for an Essbase Application, including
- Definitions of all databases within the applications
- Outlines
- Substitution Variables
- Rule Files
- Calculation Scripts
- Report Scripts
- Excel Files
- Location Aliases
- Security Filters
It does not do
- Data
- Versioning
- MXL files
- Triggers
- Partition definitions,
- custom defined macros/functions
Prerequisites
- In addition to product provisioning, users must be provisioned to the LCM Administrator Role. For Essbase, a user needs Server Access, Calc, and Database Manager privileges.
- Essbase must be in Shared Services Mode
- Environments must be same software release version
- LCM does not create applications, so the target app must exist, with the same name
- During export, the source applications need to be running, and likewise, on import, the target applications need to be running.
- When performing a migration, if the source artifacts have an earlier time stamp than the destination artifacts, the artifacts will not get migrated.
Using LCM
Here are the basic steps in migrating environments. I will go into more detail on each and share ideas on how to automate each.
- Create and execute a LCM Export migration definition file
- Copy the LCM Migration files to the target machine
- Change the hostname locations in the export files
- Create and execute a LCM import migration definition file
- Move data – if needed
LCM has a command line utility simply called utility.bat. Migrations are defined in migration definition files, which are XML files that contain all information about a migration (source, destination, artifacts for migrations, export and import options). There will be a migration definition file for the export process and import process.
1.) Create and execute a LCM Export migration definition file
You can certainly create the migration definition file manually, as it is just an XML file… however a much easier way is to use the GUI Shared Services Migration Wizard to do that.
Using the wizard is pretty straightforward. Within the Shared Services console, you simply check the artifacts/applications that you want to export.
For Essbase, I have found it a bit nicer to exclude Excel Files and Text Files. Your mileage may very, but the following brings over all the artifacts I normally care about.
For security, navigate under foundation and choose Shared Services. Check the Essbase server you want to export along with Users, Roles, and Groups.
Once you have selected what you want, click the Define Migration button to start the Migration Wizard. At the end of the wizard, just before you execute the migration, you have the ability to save the migration definition file.
This migration definition file has everything you need to repeat this exact export at will….you can run this migration export at any time either by launching a migration from the Shared Services Console, or using the LCM command line utility.
Automating
Execute Lifecycle Management Utility by entering the following at the command prompt:
Utility.bat c:/lcm/lcmdefinition.xml
Where c:/lcm/lcmdefinition.xml is the location of the migration definition file.
Additional command line arguments:
- [-estimate]: Estimates the Number of Artifacts in a Migration
- [-local]: Migrates to a Local File System Location
- [-hss url]: Uses the Shared Services URL
- [-d]: Defers Validation
- [-f]: Fully Executes Tasks
- [-b]: Specifies the Base Path
I suggest using the –f flag for best results.
It is important to note that the migration definition file excludes an ID and password by default. So when you run this from the command line, it will prompt for an ID and Password. To fully automate, just edit the file and put in the ID and Password . The connection info section in the XML file will have admin=”” and password=””. Just put in the correct values between the quotes.
2.) Copy the LCM Migration files to the target machine
After executing the migration, the exported Artifacts are stored in the import_export directory. Notice that each user has their own directory structure… here I did my migration as the admin user, and we see my export under that user’s directory.
This directory structure needs to be copied to the target Shared Services machine in the same exact manner. If the user directory does not exist on the target, you can simply create it using the same name.
Automating
I find the easiest way to copy export files between environments is just using a mounted drive from the target. It is much easier to script a simple copy from one drive to another than it is to script something like an automated FTP/SFTP. However, if a mounted drive will not work for you, consider setting up a trusted SFTP relationship with the two servers and using SFTP or scp to transfer files, that way passwords will not be hard coded in scripts.
You can actually set the path where LCM puts the artifacts. So, the location of the LCM Export files can be changed, say to a shared disk location between source and target by editing the migration.properties file.
Migration Properties File, C:/Hyperion/deployments/AppServNameAndVersion/SharedServices9/config/.
log.directory=../logs
log.config=../conf/log.xml
report.enabled=Y
report.folder_path=../reports
msr.queue.size=200
msr.queue.waittime=60
migration.scheduler.concurrency.count=5
#filesystem.artifact.path=C:/LCM/import_export
double-encoding=true
export.group.count=30
import.group.count=10000
3.) Change the hostname locations in the export files
When exporting some things, like Assigned Roles for Essbase, the .CSV exports can contain hostnames. These hostnames will have to be changed to the target hostnames. There are also some files like sourceinfo.xml and listing.xml that should also be changed. Be sure to look though the entire directory structure for files containing hostnames and ensure that all hostnames referring to the SOURCE system are change to the TARGET.
Automating
To Automate this, you will need to use some tool do a global search and replace of files to replace the hostnames.
The following is a list of free search and replace tools for windows:
http://www.thefreecountry.com/programming/searchandreplace.shtml
Unix, you can use sed… (or if you are really geeky, you can install sed for windows)
sed 's/FINDSTRING/REPLACESTRING/g' $fl.old > $fl
4.) Create and execute a LCM import migration definition file
So far, we have only exported artifacts and security and made it ready for import. We need to create another Migration Definition file from the import process, on the target machine.
Use the LCM Migration wizard to create the migration definition file, as you did for the export process. However, this time, your source will be the Filesystem. LCM sees the that this user has an artifact directory.
As you go through the import migration wizard, it will ask you which application to make the target.
For Import, there are a couple of different options, as the migration wizard needs to knowhow to handle possible conflicts.
The following list describes the Essbase import options:
- Overwrite Artifacts—Select to overwrite all artifacts in the destination location
- Restructure Cube—Choose from the following options:
- Retain cube data—Retains the data in the cube when restructuring the cube outline
- Discard cube data—Discards the data in the cube when restructuring the cube outline
- Keep only input data—Retains only the input data when restructuring the cube outline
- Keep only 0-level data—Retains only level 0 data when restructuring the cube outline
Finally at the end, you have the option to save the migration definition file, as you did for the import. This can then be used by the command line utility to automate the import as before.
Logs
There are many logs that LCM outputs. They are located in HYPERION/common/utilities/LCM/9.5.0.0/logs.
Work is divided into separate streams that produce their own logs. One export/import process can generate a handful of logs, so be sure to look at the time stamps of these files and look through them all for errors.
5.) Move Data – if needed
LCM is not designed to move data, only the artifacts. In many cases, people do not want to move Essbase data as part of a migration, as the data in the source and target are complexly different and come from different source systems.
However, if we want to move data, we have to do that ourselves through the traditional method of export/import. To automate this, we can use a MXL script to export data from the source and import it on the target. We can also consider just exporting level 0 data and calculating it on the target to save data copy time, depending on the size of the Essbase Cubes.
MXL level 0 Export command:
export database $APP.$DB
level0 data
to data_file `export.txt`;
MXL Import command:
import database $APP.$DB data
from data_file 'export.txt'
on error write to ‘export_errors.txt';
Putting it all together
Now that you have the process done and tested it is time to put on the finishing touches. Wrap the utility.bat commands, file copy, search/replace commands in overall batch scripts that can be used as scheduled jobs. Make soure you redirect the output to a place that can be reviewed and logged.
Troubleshooting/Issues
- For Lifecycle Management Utility migrations, set the Threshold value to DEBUG in the log.xml configuration file. The log.xml file is located in the LCM_HOME/conf directory, for example, HYPERION_HOME/common/utilities/LCM/9.5.0.0/conf.
- I have seen a few times that when attempting to export all Essbase artifacts, only the default calc scripts gets exported. Frustrating. This may need to be raised as a bug. I was able to get around this by creating another standalone migration definition that only migrates the calc scripts.
I have to say that OBIEE is pretty flexible and robust for high availability. There are many components to OBIEE, and each can be configured independently for load balancing as needed. The problem is that each MUST be configured independently. The Flexibility is great, but it can be a challenge to set up for the novice. And, I’m not sure how necessary it all is. Popular methodology for OBIEE is to install all components on every server in the cluster and load balance between them. It would be nice to have a more graphical, built in, approach to that…but in the mean time, we have loads of fun editing INI and XML files.
