Building a BI Developer's SuperView

Another limitation I find irritating when it comes to building BI templates is how basic the sample file generator is. My main beef is that it just takes the first 10 records in the Integration Object and spits them out. If you have a complicated IO with child ICs, it is possible, and even likely that those first ten records do not have the child detail records you need to test your report output. There are some ways around this, like hard coding a search spec on the BC against a thick client partial compile to generate a file with data you want, but that seems so inelegant. My other tick regarding this feature is that the report developer once again either needs to have a Siebel thick client or access to the Siebel Server file system to actually get the xml file produced. It seems like the whole point of all the BI Administration views is to avoid having to go to the file system. What to do...

Caveat Emptor. Configuration steps below are to give you an idea. I am posting this after I finished to highlight what I recall as the important pieces so not every step is included. You will need to create the new custom table CX_TMPL (or use another), create all links, applets, view objects, make BO/Screen changes and deploy them.

First I build a view with the same IO BC based applet as the vanilla view on top, and child applets for both attachments and a new object which is essentially a search spec. First the attachment bc. This is a new BC which you can copy from an existing attachment BC and change the names around. Here is mine, called 'Sample IO Attachment' based on S_FILE_ATT. Use the field name prefix 'Sample' instead of which ever prefix is used on the BC you are copying (Be sure to set the User Property DefaultPrefix to 'Sample' too):

Name	Join	Column	Force Active	Predefault Value	Text Length	Type
IO Id		PAR_ROW_ID	Y	Parent: 'Repository Integration Object.Id'	15	DTYPE_ID
Parent Key		X_PARENT_KEY		Parent: 'Repository Integration Object.Name'	100	DTYPE_TEXT

The Search Spec applet is based on a custom BC, 'Report IO Sample File Template', based on the new table, CX_TMPL (I use this table for other things too so I type spec each record):

Name	Join	Column	Force Active	Predefault Value	Text Length	Type
Name		NAME		Field: "Id"	100	DTYPE_TEXT
Parent Id	S_INT_OBJ	ROW_ID	Y	Parent: "Repository Integration Object.Id"	15	DTYPE_ID
Parent Name		PARENT_FLD		Parent: "Repository Integration Object.Name"	50	DTYPE_TEXT
Search Specification		CONSTRAINT	Y		250	DTYPE_TEXT
Type		TYPE		SAMPLE_IO_CONSTRAINT	30	DTYPE_TEXT
Number of Records		LN_NUM		10		DTYPE_INTEGER

The join, S_INT_OBJ, is based on the specification of Parent Name = NAME. Using name instead of Id allows the search specs to remain visible after repository moves.

You will also need the following Named Method User Property:

"GenerateConstrainedData", "INVOKESVC", "Report IO Sample File Template", "Workflow Process Manager", "RunProcess", "'ProcessName'", "'Export Sample IO To File'", "SearchConstraint", "[Search Specification]", "'IOName'", "[Parent Name]", "Path", "'..\XMLP\Data'", "Object Id", "[Parent Id]", "PageSize", "[Number of Records]"

This user property is to activate the button you will need to place on the applet based on this BC. On that applet (based on class CSSSWEFrameListBase), add a button which invokes the method 'GenerateConstrainedData'. No additional script should be needed there.

Create a Service Flow Workflow Process called 'Export Sample IO To File'


Here are the Process Properties:

Name	In/Out	Data Type
FileName	In	String
IOName	In	String
PageSize	In	String
Path	In	String
SearchConstraint	In	String
SiebelMessage	None	Hierarchy
ViewMode	In	String

The first 'Echo' step is a Business Service based on Workflow Utilities, Echo method. This step sets up all the variables used later in the process. Here are the arguments:

I/O	Argument	Type	Value/Property Name
Input	IOName	Process Property	IOName
Input	PageSize	Process Property	PageSize
Input	Path	Process Property	Path
Input	SearchConstraint	Process Property	SearchConstraint
Input	ViewMode	Process Property	ViewMode
Output	FileName	Expression	IIF([&FileName] is not null, [&FileName], [&IOName])

The next 'Export IO' step is a Business Service based on EAI Siebel Adapter, QueryPage method. This step queries the integration object. Here are the arguments:

I/O	Argument	Type	Value/Property Name
Input	OutputIntObjectName	Process Property	IOName
Input	PageSize	Process Property	PageSize
Input	SearchSpec	Process Property	SearchConstraint
Input	ViewMode	Process Property	ViewMode
Output	SiebelMessage	Output Argument	SiebelMessage

The next 'Write to File' step is the Business Service, EAI XML Write to File, WriteEAIMsg method. This step writes the property set out as an XML document to the file system. Here are the arguments:

I/O	Argument	Type	Value/Property Name
Input	FileName	Expression	[&Path]+"\"+[&Process Instance Id]+"_"+[&FileName]+".xml"
Input	SiebelMessage	Process Property	SiebelMessage

The final 'Attach' step is another Business Service, this one custom. The basic logic here is to add an Attachment to the file system which is first described in Oracle document 477534.1 (I have made some improvements which I will perhaps discuss another day). Here are the arguments:

I/O	Argument	Type	Value/Property Name
Input	AttBusinessComponent	Literal	Sample IO Attachment
Input	AttachmentFieldName	Literal	SampleFileName
Input	BusinessObject	Literal	Repository Integration Object
Input	File	Expression	[&Path]+"\"+[&Process Instance Id]+"_"+[&FileName]+".xml"
Input	ObjectId	Process Property	Object Id
Input	PrimaryBusinessComponent	Literal	Repository Integration Object

BI - Upload Limitation

I have recently been designated the BI technical resource on my project so am looking at the BI capabilities (on 7.8) for the first time. Despite a fairly complicated and mistake laden patch upgrade which I do not even want to get into, it is a pretty powerful tool, much better architected than Actuate. Anyway, there are also some pretty glaring limitations as well on how it is administered that require so little effort to fix, I decided to just go ahead and fix them.

My main beef is that the architecture requires your BI report developer to have access to both the BI file system and the Siebel Server file system. I suppose you could set this up in a way that minimizes security risk, but it just seems so unnecessary. Essentially, to upload a new BI Report Template, the developer creates a record in the BI Report Template administration view, attaches the two template files (an RTF and an XLF) and clicks the upload button. So far, so good. The problem is that these template files must also exist in a specific place in the Siebel Server file system as well to generate a report. But the code behind that button does not take the extra step to just copy the files to where they need to go. Also, there is an existing product defect where modifications to an existing report record require the developer to go into the BI File system and delete the entire directory containing that report template. So that is where I step in.

First I added two new System Parameters indicating the locations of the BI and Siebel home directories. There is a way to grab environment variables through script but I did not feel like investigating this so let's call that phase II. For example, here are my two:

Name	Value
BIHomeDirectory	E:\OraHome
SiebelHomeDirectory	E:\sea78\siebsrvr

Then, we need to trap the call to upload the templates file. This call is performed from 'Report Template BC' by the 'Upload' method. We need to always delete the directory before this upload is called. We also want to delete the existing template file from the Siebel server file system. Here is a script to place in the PreInvoke method to accomplish that (there are also some references to the Log and Frame objects):

switch (MethodName) {
  case "Upload":
    try {
      Log.StartStack("Business Component", this.Name()
        +".PreInvoke", MethodName, 1);
      this.WriteRecord();
      var sReturn, sCommand;
      var sSiebel = Frame.GetSysPref("SiebelHomeDirectory")
        +"\\XMLP\\TEMPLATES";
      var sPath = Frame.GetSysPref("BIHomeDirectory");
      var sFile = this.GetFieldValue("ReportTmplFileName")
        +"."+this.GetFieldValue("ReportTmplFileExt");

      sPath = sPath
        +"\\XMLP\\XMLP\\Reports\\SiebelCRMReports\\"
        +this.GetFieldValue("Report Name");
      Log.stepVars("BI Report Path", sPath, 3);

      sCommand = 'rmdir "'+sPath+'" /S /Q';
      sReturn = Clib.system(sCommand);
      Log.stepVars("Command",sCommand,"Success?",sReturn,3);

      sCommand = 'del "'+sSiebel+'\\'+sFile+'"';
      sReturn = Clib.system(sCommand);
      Log.stepVars("Command",sCommand,"Success?",sReturn,3);
    } catch(e) {
      Log.RaiseError(e);
    } finally {
      Log.Unstack("", 1);
    }
    break;
}
return (ContinueOperation);

Ok. That addresses the product defect for updates. Now the second part is to copy these template files to the Siebel server file system once the template files are uploaded. The following script can be added to the InvokeMethod event:

switch (MethodName) {
  case "Upload":
    try {
      Log.StartStack("Business Component", this.Name()
        +".Invoke", MethodName, 1);
      var sReturn, sCommand;
    
      var sSiebel = Frame.GetSysPref("SiebelHomeDirectory")+
        "\\XMLP\\TEMPLATES";
      var sPath = Frame.GetSysPref("BIHomeDirectory");
      var sFile = this.GetFieldValue("ReportTmplFileName")
        +"."+this.GetFieldValue("ReportTmplFileExt");

      sPath = sPath
        +"\\XMLP\\XMLP\\Reports\\SiebelCRMReports\\"
        +this.GetFieldValue("Report Name");
      Log.stepVars("Source Path",sPath,"Target Path", 
        sSiebel,"File to copy",sFile, 3);
      sCommand = 'copy "'+sPath+'\\'+sFile+'" "'+sSiebel
        +'\\'+sFile+'"';
      sReturn = Clib.system(sCommand); 
      Log.stepVars("Command",sCommand,"Success?",sReturn,3);
    } catch(e) {
      Log.RaiseError(e);
    } finally {
      Log.Unstack("", 1);
    }
    break;
}

And there you go.

ADM - List Of Values

There are plenty of posts on support web discussing the issues with migrating LOVs, but for my own sanity, I thought I would summarize all of the relevant issues in one place.

First, we need to address the defects. These are documented on support in Document 731411.1 but I will summarize here:

(1) Go to BC 'List Of Values Child (UDA)'
(2) Add a new field 'Parent Type' based on Join 'Parent LOV' and Column 'TYPE' with Text Length '30'.
(3) Expand the pickmap for the 'Parent' field. Replace pickmap field 'Type' with 'Parent Type' and uncheck Constrain flg.
(4) Go to the integration object 'UDA List Of Values'
(5) Find the Integration component 'List Of Values Child (UDA)'
(6) Add a new field to the integration component with Name = 'Parent Type'. Data Type = 'DTYPE_TEXT', Length = '30', Type = 'Data', External Name = 'Parent Type', External Data Type = 'DTYPE_TEXT', External Length = '30', External Sequence = '38', XML Tag = 'ParentType'
(8) Compile changes.

The SR then goes into some more detail on why after all that it still does not quite work. To understand, we need to see that the LOV ADM Integration Object is Hierarchical in one dimension. That is, there is the LOV_TYPE record and then there are the value records. But LOVs are frequently Hierarchical in two dimensions, by virtue of the Parent value. What I mean is that a given LOV value record will always have one 'parent' record, it's type or technical parent, and may have a second parent record, it's functional parent, if you will.

ADM loads the first, technical parent in the standard way, through the relationships of the Integration Object. To load the functional parent though, ADM must run in two passes, the first to create all the parent and child records, and the second to relate them. This is necessary because we cannot guarantee the sequence with which LOV value records will be placed in the extract file. If these value records do not exist in the target already, and the parent is alphabetically (or however else we chose to sort the records) after the child, then it would error if ADM did not take this approach. So how ADM takes two passes is by virtue of the ADM Data Type explorer. You will notice that the explorer does not actually specify the foreign key fields of an object to link them to each other. Its only purpose is to run ADM in multiple passes. But the twist is that ADM will actually process dependent data types setup in the explorer in reverse order, importing the children before the parent. I personally find this confusing from a terminology perspective. Perhaps a better way of naming these Data Types is to use 'LOV-2ndPass' instead of 'LOV-HierParent' and 'LOV-1stPass' instead of 'LOV-HierChild'. This way when we set up the search specifications for an ADM Export session, it is clear what we are trying to do.

OK, one more wrinkle to throw into the mix (just when you thought it was all making sense). There is actually a third parent relationship involved. That is the records that populate the S_LOV_REL table. I will be honest; I do not use the LOV explorer view that often and I don't really know what the point of this table is. In theory it can make LOVs M:M but I just don't think this is practical. Nevertheless, there are some vanilla uses of LOVs where these records are in fact used that way. The one that comes to mind is in payments, where the PAYMENT_TYPE_CODE values are children of the PAYMENT_METHOD_CODE and there are S_LOV_REL records created to store the relationships. The same issue applies when migrating these relationships. The related value must exist prior to the relationship being built.

One final note. I think the whole not deleting LOVs is well intended but more likely to cause confusion than solve anything. Here is why. Users can and will just change the Name/Value of a value record to something else in which case any sense of history is lost anyway. There are no foreign key relationships to LOVs so business data using these values is unaffected regardless. But others may disagree so this step is completely optional. I remove the no delete properties from the 'List Of Values Child (UDA)' BC and Integration Component. (I also allow deletes from the GUI but that is a separate issue). So my migration methodology is to synchronize values between environments for an initial release. You would take a different approach on a point release where values are likely to have been added directly to production and therefore may not exist in your DEV and TEST environments.

Anyway, what are we trying to do. Quite simply, we are trying to create all the Value records in pass 1, then we need to relate them to each other in Pass 2. I have already discussed how to group LOVs together for a release. This is where I diverge from Siebel's example because I am trying to think of real life scenarios where I am deploying releases, not just one LOV_TYPE. When creating the ADM Project/Session, here are the session items I use:

Data Type	Child Delete	Deployment Filter
LOV-2ndPass	Y	[Release] = '1.1'
LOV-1stPass	N	[Release] = '1.1' AND [List Of Values Relationship.Name] IS NULL

What this means is that the first pass includes all LOV_TYPE records that have been marked for this release and all LOV value records related to them. The second part of the expression basically just insures that no relationship records are included in the first pass. When ADM attempts to set the parent value on a child, it may not be able to find it so it will log a warning and move on. In the second pass, ADM will load all the Relationship records and set the Parent values that it missed on the first pass. I have also set child delete to true on the second pass so that this job effectively synchronizes the value records for the type records marked.

A Basic Interface - Integration Object User Props

I know this is meant to be a basic interface with so complexity, but let's be realistic about the requirements we are likely to get. Even a simple upsert of something as basic as a service request is likely to require a bit of digging into bookshelf so that the interface is able to mimic basic GUI functionality. I will discuss some of the most commonly used User Properties necessary to implement even an advanced interface. When in doubt about the syntax of any of these properties, take a look for an example in the Tools flat view.

PICKLIST

This is the most common Integration Component Field User Property you will see and it basically tells the EAI Siebel Adapter to validate the picklist in the interface. This property is generally created by the wizard so I bring it up only because validating the picklist here will allow for several different ways to interpret a picklist field value described by some of the user properties below.

PicklistUserKeys

In the GUI, when you type an account name in the Account field on another BC that has a picklist of Accounts, and there is more than one record matching that name (with different locations), a pick applet will pop open with the constrained list of accounts having that name. The GUI is letting a user decide which of the multiple records returned was meant to be picked. An interface does not have that luxury, so the PicklistUserKeys Integration Component Field User Property is provided to mimic this action. The value of this property should be a comma separated list of fields representing the logical key of the picklist record to look up. These fields must all be present in the integration component (though there values can be null). The 'PICKLIST' user property must also exist for the field where this property is used and its value must be 'Y'.

Ignore Bounded PickList

When a picklist is validated in the interface and the value passed is not found, the EAI Siebel Adapter stops processing and returns an error. If the data is expected to sometimes be missing though, you may want the foreign key to just be left blank. For instance, maybe the service request, in our example is tied to an order via a back office order number, but the order was never loaded. Add this user property with a value of 'Y' in combination with the PICKLIST user property with a value of 'Y'. The EAI Siebel Adapter will look up the record by the user key provided (can also be used in combination with PickListUserKeys) but if it is not found, will set the field to blank in the integration object before applying the data. Keep in mind that this property will only work as expected if the Picklist object the underlying BC uses to constrain the field is set to No Insert equals True, otherwise, the EAI Siebel Adapter will try to insert a record. Also note that in bookshelf there is a typo in that there should be spaces between the words of the property name.

FieldDependency

It is easy in the GUI to determine the order of the fields being picked, either by training or by sequencing the fields in a particular way during applet design. This may help set the fields that will be used to constrain the value of another field, frequently in a hierarchical picklist. In EAI, we achieve this result through this user property. It can be used multiple times with a sequence number, just like other BC and applet user properties. The value is a field integration component field name. Siebel claims that pickmapped constraints are automatically taken into account, and that may typically be the case, but I have seen times when it does not work, so this is a good fall back.

A Basic Interface - Web Service Workflow

Just about every interface consists of two basic components: the integration object(s) and the workflow or business service. I will demonstrate a workflow approach which will give you more opportunity to customize down the road.

It is here that we begin to differentiate the integration by the communication mechanism. Because I am designating this integration as a Web Service, that will drive the type of data this workflow will expect as an input and output. The workflow I build will eventually be exposed as a WSDL to be consumed by an external program. That WSDL should have the definition of the message it is expecting, in this case, the XSD, or definition of the Integration Object we just built. How we accomplish this is to set the Input Process Property to a Data Type of 'Integration Object' and to actually specify the integration object we built, in the Integration Object attribute of the process property.


You can also see my place holder for the SR Number that I want to return to the external system in the response message. The 'IncomingXML' property is already in the format needed to be passed to the EAI Siebel Adapter, so there is no conversion necessary. And we are assuming that the data being passed is exactly as it should be applied. You will create the following steps which I will explain (other than Start and End which are self explanatory):
The 'Upsert SR' is a Business Service calling 'EAI Siebel Adapter'. Now here is the another design decision to be made. Each of the available methods differentiate exactly how the data should be applied. But there are two broad determinations. If we were to use the Execute method, then the 'operation' element which exists in each component of the IO would be used to determine how the data should be applied. This gives more control the calling system (or a data map which I will discuss later). The other set of methods essentially comprise a One Size Fit All to applying all the data uniformly. I will use the latter approach here and set the method to 'Upsert'. There is only one component in my IO, so if it exists, it will be updated, otherwise it will be inserted. The input arguments for this step are the IncomingXML message from the external system and a parameter telling the EAI Siebel Adapter to create the Status Object.

There is one Output Argument. We no longer care about the input message at this point because it will have been applied so we just overwrite it with the return, which in this case will be the status key.
The last step in the WF is another Business Service step calling 'PRM ANI Utility Service', 'GetProperty' method. This business service has a plethora of useful methods for manipulating property sets. This particular method will extract the value of a field from an integration object instance. Here are the inputs:
The output is to set the process property 'SRNumber' with the Output Argument, 'Property Value'. When the return message is sent back to the calling system, this property will exist with the generated SR Number.

Simulating/Troubleshooting this WF from within tools is difficult as built so I sometimes add a bypass step off the start branch to read the integration object from a file. I may talk about this later but want to keep this post pretty straightforward. So for now, this workflow can just be deployed, checked in and activated.

A Basic Interface - Building the Integration Object

I am not sure how easy it will be to summarize EAI in a couple of blog posts as there are definately a lot of ifs and buts in the design process. Nevertheless, I think it wold be useful to show how to build a basic interface using a couple of different techniques. Frequently you client's enterprise architecture will drive which to use.

Integration generally takes one of three forms

• Query - Returns a data set of source data to be displayed in the target system
• Schema Update - Takes a hierarchical data structure and applies it to the target system
• Functional Action - Triggers a service to perform some set of business rules

There is perhaps some overlap here and any of these can be inbound or outbound to siebel, but this is a general way of categorizing your interfaces. And within each there are several different ways to implement more specific requirements.

Regardless of approach, the basic component of most interfaces is the structure of how data is viewed or applied. Let's say we need to Upsert a Service Request. A Schema Update assumes a hierarchical organization of data using the Integration Object data structure. Bookshelf provides extensive instruction on how these are built and configured to achieve certain goals so I will only touch on the highlights.

First, create an Integration Object in Siebel: from the Tools File Menu, New Object Wizard, EAI Tab, choose Integration Object. In the wizard, select the Project and choose 'EAI Siebel Wizard' from the second dropdown, and click Next. For the purpose of this example, we can just use the Service Request business object as the source object and the root BC will be Service Request. Enter a name of your choosing and click Next. In the next wizard page, deselect all child objects for which there are no fields to set. In this case that will be all of them except for the root as the more objects and fields in the message, the longer it will take the various architecture components to parse and translate the message. Click Next, then Finish on the next page.

Your Integration Object has been created. The next step is to verify the user keys. An integration object needs to have a valid user key in order to do an upsert. This basically specifies which key fields to use to find a record to update. In my example for Service Request, a key was not generated by the wizard so I will create one. Navigate to Integration Component Key in the explorer under the Service Request Integration Component. Create a new record, provide a name, set the sequence number to 1 and the key type to 'User key'. Create a child record in Integration Component Key Fields, provide a name and set the Field Name to 'Id'.

Another optional step we will use in this example is the Status Key. After creating a service request, I want to return the service request number to the external system as verification of success and so this SR can be referenced later by the customer. To do this we use the Status Key. This is basically a structure of the data set we wish to return from the EAI Siebel Adapter call and pass back to the calling system. A Status Key can be specified for each Integration Component so the final data set is the structure of all the keys combined hierarchically. In this case, navigate to Integration Component Key in the explorer under the Service Request Integration Component, create another new record, provide a name, 'StatusKey, set the sequence number to 1 and the key type to 'Status key'. Create a child record in Integration Component Key Fields, provide a name and set the Field Name to 'SR Number'.

Finally, while not absolutely necessary, you should inactivate all fields you are not using for each Integration Component. For an inbound upsert to Siebel, the calling system does not need to provide all the fields that are active in the IO schema, but if the field is active in the IO, then the external system Could send that data element which may have undesired affects depending on the interface. Make sure all fields used in the key are activated as well as all fields being passed from the external system. Unlike BC field lengths, the length property of an Integration Component Field is more important as when an XSD is generated and provided to the external system, this property will frequently be used by the web development tool to validate the data entered into that field. You can also change the XML Tag attribute to a label recognized by the external system (so long as spaces are removed).

One thing to keep in mind is that if an insert is desired, then the calling system should just pass a constant to the user key field, 'Id' so that Siebel will not find a record and a new one will be created. A value like '_New_Record_' is a safe value because the '_' will never be part of a generated row id.

Common (or not) eScript Syntax Errors

I would love to post a comprehensive list of gotchas, but then that would make them not gotchas if you know what I mean as I would know them all. So instead, I will mention what sidelined me for several hours last night and hope to spur some discussion about what other people have come across. If I think of others over time, I will try to update this post.

Space after the function name. I had copied and pasted some functions from somewhere else in my client's repository and the functions had no space between the name and the opening parenthesis of the passed variable declarations. I was not (and I guess still am not) aware of a limitation in this regard, but I saw all sorts of strange behavior afterward. Namely, the calls to these functions seemed to be ignored which took me a long time to realize. They seem to work fine in their original home elsewhere in the repository so this may be related to context, but suffice to say this is some thing to think about when troubleshooting.