Enabling Lights-Out Automation

One of the most common requests I receive (and my #1 out of the box feature) is to automate data processing through FDM.

To meet this need, FDM includes a powerful automation tool known as the Batch Loader. This functionality is not to be confused with windows/dos batch (i.e., .cmd, .bat) files. FDM batch loader functionality uses FDM scripts – similar to import scripts – to mimic the action of an end user performing the Workflow steps of Import, Validate & Export. 

As with an interactive session, batch loader can process data using the standard or multiload Workflow. For each workflow method, parallel or serial processing is available.

Standard Workflow

Multi-Load


'Execute a STANDARD batch
'Declare Local Variables
Dim lngProcessLevel
Dim strDelimiter
Dim blnAutoMapCorrect'Initialize Variables
lngProcessLevel = 50 'Import
strDelimiter = "~"
blnAutoMapCorrect = 0'Create the file collection
Set BATCHENG.PcolFiles = BATCHENG.fFileCollectionCreate(CStr(strDelimiter))'Execute a Standard Serial batch
BATCHENG.mFileCollectionProcess BATCHENG.PcolFiles, CLng(lngProcessLevel), , CBool(blnAutoMapCorrect)
  


'Execute a MULTI-LOAD batch
'Declare Local Variables
Dim lngProcessLevel
Dim blnAutoMapCorrect'Initialize Variables
lngProcessLevel = 50 'Import
blnAutoMapCorrect = 0'Create the file collection
Set BATCHENG.PcolFiles = BATCHENG.fFileCollectionCreateML'Execute a Multi-Load Serial batch
BATCHENG.mFileCollectionProcessML BATCHENG.PcolFiles, CLng(lngProcessLevel), CBool(blnAutoMapCorrect)
   


'Execute a STANDARD batch
'Declare Local Variables
Dim lngProcessLevel
Dim strDelimiter
Dim blnAutoMapCorrect
Dim lngParallelProcessCount
Dim strLoadBalanceServerName'Initialize Variables
lngProcessLevel = 50 'Import
strDelimiter = "~"
blnAutoMapCorrect = 0
lngParallelProcessCount = 2
strLoadBalanceServerName = "LoadBalanceServerName"'Create the file collection
Set BATCHENG.PcolFiles = BATCHENG.fFileCollectionCreate(CStr(strDelimiter))'Execute a Standard Parallel batch
BATCHENG.mFileCollectionProcessParallel BATCHENG.PcolFiles, CLng(lngProcessLevel), CLng(lngParallelProcessCount), CStr(strLoadBalanceServerName), , CBool(blnAutoMapCorrect)
  


'Execute a MULTI-LOAD batch
'Declare Local Variables
Dim lngProcessLevel
Dim blnAutoMapCorrect
Dim lngParallelProcessCount
Dim strLoadBalanceServerName'Initialize Variables
lngProcessLevel = 50 'Import
blnAutoMapCorrect = 0
lngParallelProcessCount = 2
strLoadBalanceServerName = "LoadBalancerServerName"'Create the file collection
Set BATCHENG.PcolFiles = BATCHENG.fFileCollectionCreateML'Execute a Multi-Load Parallel batch
BATCHENG.mFileCollectionProcessParallelML BATCHENG.PcolFiles, CLng(lngProcessLevel), CLng(lngParallelProcessCount), CStr(strLBServer), CBool(blnAutoMapCorrect)
  

Serial Processing:

Each file in the collection of files is processed sequentially.

Parallel Processing:

The application spawns multiple server processes to process groups/sets of files concurrently. The application is optimized to ensure that files that share a data segment table are assigned to the same process. This built-in intelligence ensures that the batch process runs as efficiently as possible.


Sample EmailAny of the above batch scripts can be easily implemented but the real value is the ability to layer in a process that provides real time status reporting.

This reporting is generally provided by email but most communication protocols can be supported. Click the image to the right to see an example email.

When preparing status reporting, you need to think about the various possible outcomes for each data set processed. For example:

    1. Are the unmapped items output to an excel file and attached to the email?
      Do you prefer FDM to load unmapped members to a suspense member?

    1. How do you transform the XML generated by the application to a format that can be attached to an email and viewed on any PC?
      Does a custom process need to be created to “override” the invalid intersections – thereby allowing the valid intersections to be loaded to HFM?

  • How do you output the Check report to PDF?
    1. If the batch includes multiple locations, how do you filter the status to the appropriate recipients?

    These are just a few of the questions and considerations when designing an end to end lights-out batch process. The inline image is an example of how I’ve addressed some of the most common questions/requirements.

    The out of the box functionality of the batch loader and the capability to layer in automated real time status reporting truly helps you realize an even greater return on your investment – not only for FDM but in your entire Oracle/Hyperion EPM product suite. The end to end data load automation process can be scheduled to run at set intervals. This reduces the workload for the various FDM data loaders while also eliminating the bottleneck that can be caused by time zone differences, resource availability, etc. The end result is a process that is more efficient and more dependable.

    I hope you are as excited about FDM’s automation capabilities as I am.  As you consider the use of FDM’s batch loader it may ease your trepidation to know that I have yet to (nor do I expect to) see FDM fail to address the various needs and requirements of an automated data load process.  I strongly encourage you to take the next step with your application and leverage the batch loader – you won’t regret the investment!    

    If you have question of comments, feel free to post them below – I will respond.

    This entry was posted in Automation and tagged , , , , , , . Bookmark the permalink.

    9 Responses to Enabling Lights-Out Automation

    1. Pingback: [BLOCKED BY STBV] Solution Highlight: Creating HFM Journals From Planning | The FDM Information Source

    2. Pingback: [BLOCKED BY STBV] Streamline Your Close | The FDM Information Source

    3. Pingback: [BLOCKED BY STBV] Suspense Mapping – You Can But Should You? | The FDM Information Source

    4. HyperionDJ says:

      Tony,

      Thanks for the great write up on Batch Loader. I was wondering if I could kick off the Batch Loader script via UPSShell? I want to use my Enterprise scheduler to start the UPSShell Script to load my FDM data files to their correct FDM Location.

      If there is another way that I can manually start the load process via script, can you point me in the right direction?

    5. Pingback: Set It & Forget It – Scheduling FDM Tasks | The FDM Information Source

    6. aknestel says:

      Can the batch loader script be called externally, e.g. by ODI? Thanks!

      • Tony Scalese says:

        Alexandra,

        Any custom FDM script can be executed from a batch process. I would suggest reviewing the Admin guide section regarding UPSShell. A link to the documentation can be found under my Resources page.

    7. Pingback: ERPi – The Future of Data Integration | The FDM Information Source

    Leave a Reply