Are you still manually starting services????

Starting and Stopping Services

One of the most confusing aspects of the Oracle-Hyperion Suite is the basic task of starting and stopping the services in the correct order.  Why is this so difficult?  Why is there no script? Why so many services?

It’s SOA

As with many enterprise class software packages, Oracle-Hyperion is based on a Service Oriented Architecture or SOA.  All components of the Hyperion Suite are glued together by a common foundation and protocols so all of the modules can interact with each other as needed.  What this means is that there is inherent flexibility in how one can design and architect an implementation.

If you wanted to (not that I would recommend it) you could install all of the modules and products on one machine. Or you can virtually install every component on separate servers in a mix-match of operating systems and configurations. This also allows some products to have multiple instances so that load balancing and scalability is possible. The point is that Oracle leaves it up to us to design an environment that meets our individual needs.

Unfortunately, that also means we have a lot of services and processes that we must maintain across many machines – all with dependencies to each other.  Visualizing the Hyperion foundation gives an insight into these dependencies so we can understand why the start order is so important – and it is.  I know that for many system admins, it is natural to simply set all services to automatically start at boot but, chances are, that is probably not going to work. Lets think of our foundation so we can start things in the correct order…

The relational database

The Relational database is the backbone of the Hyperion suite.  These are a set of databases (or schemas in Oracle) that hold metadata for the product. Shared Services, BI Plus, Essbase Administration Services, HFM, Planning, EPMA, ODI all have separate databases that are required to be up and running when the products are started, and should be started first.

Shared Services/OpenLDAP

Next is Shared Services.  When most products starts, they check in with Shared Services to get a handle on the location of vital components.  In some cases services will not even start if Shared Services is not running, and even if they do, they may not function.  Also remember, shared Services requires OpenLDAP. Be careful – Shared Services can take quite some time to fully come up…be sure to wait a few minutes to ensure it is all the way up before starting the other services.  Check by going to:  http://<HSS_HOST>:58080/interop

The BI reporting and Analysis foundation – BI Core,  Workspace

For BI, the CORE process is key.  BI Core is the common foundation for all of BI and acts as a traffic cop for session management, etc.

EPMA (if applicable)

If you are using EPMA, now is the time to start it.

Essbase Related Services

1. Essbase Server,
2. Essbase Administration Services,
3. Essbase Integration Services
4. Provider Services

The rest of BI – reporting and Analysis – in this order

1. Interactive Reporting
2. Financial Reporting Services
   1. FR Server
   2. FR RMI
   3. FR Scheduler
   4. FR Print Server
   5. FR Web Server
3. Web Analysis

The rest of the products (any order)

• Planning and RMI Service
• Hyperion Financial Management (HFM)
• Strategic Finance
• Performance Scorecard
• Data Relationship Management (DRM)
• Financial Quality Data Management(FDQM or FDM)
• Data Integration Management (DIM)
• Oracle Data Integrator
• OBIEE

See the Hyperion System 9 install_start_here guide for locations of the start scripts for each of these services.  The stop order is the reverse of the start order.

Automating Start/Stop through scripts

Because of SOA, there are seemingly an infinite number of ways any given Hyperion implementation can be designed.  As such, there is no way Oracle can bundle a single start-all or stop-all script for you.

Obviously there are many advantages to having a script start and stop everything, most notably for nightly backups and scheduled maintenance.  In large environments covering a handful of machines, manually logging into each server and starting/stopping the services can be quite time consuming.

Most Hyperion IT admins create a script to start/stop the services, as needed in order. For windows, the “sc” command is popular.  The basic syntax is:

Sc \\hostname start|stop “Service Name”

For example a start_all.bat script could begin with the following:

echo Starting OpenLDAP

sc \\HSS_HOST start “OpenLDAP-slapd”

timeout /t 5

echo Starting Shared Services

sc \\HSS_HOST start “HyS9SharedServices”

timeout /t 200

For unix, the easiest way is to create rsh relationships (with .rhosts) between the machines and use rsh to execute remote commands on each server.

For example to start OpenLDAP:

rsh HSS_HOST <HSS_HOME>/openLDAP/startOpenLDAP.sh

sleep 5

Tips:

• Create your scripts and play with the TIMEOUT/SLEEP values to optimize the speed but ensure all comes up correctly and functions.
• Launch start processes in parallel when you can (background in Unix).  If there are no dependencies, there is no reason to start them one-by-one and wait.

Hands Off Administration

What if you want to have the Finance department start/stop services as needed, but you don’t want to give them access to the servers????  Common problem.

One way is the a product called Network Services Manager.  See http://www.networkservicesmanager.com/

This tool is a GUI tool that will allow non-technical personnel quickly and easily start/stop services and monitor the status of services on a machine without the need to log into the servers.  It can take some time and effort to set up, but once set up, it can make the life of a system admin much easier for environments such as a development environment that requires frequent service restarts.  Please let me know if you would like to demo NSM for purchase, I can get you a full-feature evaluation of the software and a significant discount.  ☺

How Shared Services Works

What Shared Services Does

Shared Services is a component of the “foundation services”.  It is the mechanism that provides a common framework for user security and administration.  It is the first component that is installed in an installation.  Once up and running, all Hyperion products and modules “plug-in” to this base.  Shared Services provides a single interface to:

1. Define External Authentication providers (ie – corporate LDAP, MSAD, etc)
2. Provision Users and Groups
3. Life Cycle Management (promotion of artifacts between DEV and PROD)

Products communicate with Shared services though a common API which allows all the products to employ true single sign-on between the products.

Shared Services components

There are 4 major components of what we collectively call Shared Services:

1. Shared Services web server.  The Web Server for the Shared Services communication and interface.  Default URL and port is http://<server>:58080/interop
2. Native Directory.  A small file-based OpenLDAP directory that comes with Shared Services to store provisioning information
3. Relational Repository.  A small relational database (ie Oracle, SQL Server, etc) that stores location information
4. Corporate External Authentication (optional).  Your (already existing) corporate external authentication mechanism.

If you do not have a corporate external authentication provider you can use the provided Native Directory to create users and passwords to provision.  However, doing this puts you into the password and account maintenance business. If you do have one, you will want to use it, as that will be taken care of for you. Shared Services will not store passwords of externally authenticated users, it simply forwards on the ID/Password combination to the provider for a thumbs-up or thumbs-down. You can have multiple user directories configured and set a search order for them.

The Native directory holds user IDs and Passwords of Natively authenticated users, provisioning information for all users, and tracks user-group relationships.  Again, this is stored in the OpenLDAP repository that comes with Shared Services. The OpenLDAP that comes with HSS  is a simple file-based database that is in LDAP format which runs on port 58089.  You can actually use an LDAP browser and connect to OpenLDAP and browse around using a base DN of dc=css,dc=hyperion,dc=com.

The relational component is a separate database (or schema in Oracle) that holds registration information about the products in the environment.

The following diagram summarizes the components.  Again, the corporate LDAP authenticates the user (are they who they say they are), the relational holds product registration information (can they access this particular product), and the native directory handles authorization (do they have permission to do what they are requesting to do in this product).

Provisioning users and Groups

The User Management console using the URL http://<server>:58080/interop is used to provision users.  Provisioning is the process of granting access to users to certain products and services.  Here we browse through the users, and provision Henry for the BBB Essbase Application and the Finsrvs FDM application, and the TotPlan Planning Application.

Show all users

Right click on Henry

Provision

The Process of Authentication

Once a product is registered with Shared Services, it receives and stores location of the user directories.  So when a user logs into a product, the process is as follows:

1. The user enters the ID and Password into the product log-in screen
2. The product queries all the configured user directories to verify the credentials.  Upon success, the user is authenticated.
3. Once authenticated, the product contacts Shared Services to lookup the provisioning information of the user to see if the user has been given the access to the product and service.
4. SSO is enabled for this user now for the rest of the products they are provisioned for.

What This Means for our System Administrator Brethren

1. Shared Services must be the first to be installed and configured
2. Shared Services must be the first to be started (along with OpenLDAP).  Note:  it can take a while for Shared Services to come up.  Make sure you wait a while and check that HSS is all the way up by going to the URL before starting the other services)
3. Shared Services is a single point of failure for all Hyperion Products.
4. You must be diligent in backing up Shared Services, including
   • Shared Services relational database
   • OpenLDAP (see the backup/recovery guide ….there is a utility that backs up the OpenLDAP directory)

My Dedication to the Hyperion System Administrator

As I look online, browse white papers, attend user group meetings, and fight my way through 2-3 conferences a year, I see the common theme.

1.) What company/solution Oracle just purchased/announced

2.) What this new functionality can do for your business.

Then, inevitably, with the backing of Oracle name, corporations flock to the new solutions with promises of unsurpassed performance and ROI.  The focus is on functionality, delivery, and end results.  They show research on how companies can perform, analyze, and enjoy competitive advantages.  CEO’s, CIO’s, CFO’s eyes light up, drop a couple gazillion dollars and now own this great new enterprise application that will bring them freedom and successes.

Great… but what about IT?  Did anyone consider the poor system administrator?  The answer is probably not.  Chances are the CXO’s were so googly-eyed over the pretty charts and graphs they forgot that maybe we might need a hardware and software infrastructure to install, maintain, and support this application.

It happened to me.  Working at a major telecom company, I came in one morning with a stack of CDs on my chair that were labeled “Hyperion”.  Having no idea what it was, I tossed them aside.  Around three o’clock the same day, the director of fiance came to my desk and asked me if I had the software loaded yet. What????

“What magical vaporware are we going to install this on?”

“You want this WHEN???”

“What operating system does it support?”

“Does it line up with our Java/security/protocol standards?”

“How are we going to integrate this with our help desk?”

“Who is going to support this, HOW do we support this?”

If this has ever happened to you, you can understand my immediate desire to throw my badge in the trash and walk out forever.

The truth is that IT is always an afterthought in corporate applications…

…especially financial corporate systems.  Many times, software is ordered, purchased, and full project plans put together before ANY IT person is even in the loop.  Then, they have to play serious catch up on the solution, learn it overnight, and try to meet the aggressive schedule some accountant made up.

This often time leads to bad blood between Finance and IT.  I have seen it so bad that in many instances Corporate fiance has to create their own Corporate Fiance IT department that is outside the normal corporate-wide IT group.

That is where this blog comes in. This is my attempt to spread the message that proper IT involvement and design is just as important as the application functionality.

I call this 360 degree strategic Enterprise Planning…

360 degree Strategic Enterprise Planning

In essence, equal focus must be placed in good solid functional design (i.e. efficient report creation and application development) as well as solid IT design to deliver a complete delivered enterprise. One without the other is destined for failure.

Let’s remember… prior to System 9, the Hyperion suite of products was just that… a group of related, but separate applications that were duck-taped and bubble-gummed together to form a complete solution.  But really what this meant was that there was no common look and feel, users had to know where to go to use each product, and there was no reliable single sign-on across the enterprise. So the complexity rested on the end users to know where to go and what to do. the IT side was relatively simple, as each application could be installed, configured, and even upgraded separately, as needed.

Pre System 9

System 9

System 9 was totally re-written.  It has transformed from a collection of best-of-breed solutions to a complete enterprise.  As with other enterprise Applications such as SAP or People Soft, System 9 and 11 has a common foundation architecture.  The “separate” products such as Planning, HFM, etc can be though of as modules like the HR or AP/AR modules of SAP Bassis. What this gives us is a common portal and look-and-feel for end users to access the entire suite and true single sign-on.  However, it also moved the complexity from the end user IT.

Hyperion (Oracle EPM) is Enterprise class software

Just as laughable as it is for a CXO to hand a stack of SAP CDs to an unwitting IT tech and expect it to be installed overnight, so too is this. Just as there are dedicated IT System Admins that specialize in the administration of IT corporate databases, ERP, and CRM systems…we now have dedicated Hyperion System Admins. We now have a mature corporate mission critical offering that must be treated as such.  Concepts like help desk integration, backup/restore, disaster recovery, high availability, on-call support, monitoring and ticketing systems all naturally come in to play now.

This Blog is dedicated to the IT professionals that are blazing the trail in Hyperion system administration.  I am a consultant dedicated to IT infrastructure around the Oracle-Hyperion EPM and BI offerings:

http://www.EricHelmer.com

I have seen it all.  While you may or may not choose to hire our services, I will help as much as my time will allow to document periodic tips, tricks, and common issues that can help in the daily life of my System Administrator brethren.I am just getting started so please bear with me, but I promise I will do my best to keep this as active as I can.