Get CSF Key Inside SOA Composite on BPEL

We know that Oracle WSM (Web Services Manager) agents handles the Credential Store very well and is the best approach to calling secure services through client policies. For example, if we need to call a service that is secured by WS-Security Username Token, we can simply attach the “oracle/wss_username_token_client_policy” to the service reference, and then use the “csf-key” binding property to make a reference to some credential, and the OWSM agents will do the will do all the work.

However, many times we face the scenario where the service being called do not follow the standards or use some kind of home-made authentication, even inside the SOAP Body. Furthermore, there are many use cases where it can be useful to get access to a CSF Credential Key inside the BPEL flow, programmatically, to get username and password in hands to assign it to variables and do whatever you need.

It’s possible to do this trick using the “Credential Store Framework API” (CSF API), though a “Java Embedding” activity. There a few steps needed to accomplish this, which are described below:

1 – Prepare your IDE, adding the library “BC4J Security” in the project’s classpath on JDeveloper:
17_image1

17_image3

17_image2

2 – Create a “Java Embedding” activity to retrieve the username and password from the credential store:


try {  
  oracle.security.jps.JpsContextFactory jpsCtxFactory = oracle.security.jps.JpsContextFactory.getContextFactory();  
  oracle.security.jps.JpsContext jpsCtx = jpsCtxFactory.getContext();  
  oracle.security.jps.service.credstore.CredentialStore credStore = jpsCtx.getServiceInstance(oracle.security.jps.service.credstore.CredentialStore.class);  
  oracle.security.jps.service.credstore.PasswordCredential cred = (oracle.security.jps.service.credstore.PasswordCredential)credStore.getCredential("test-cred-map", "test-cred-key");  
  if (cred == null) {  
    System.out.println("Credential not found.");  
  } else {  
    String username = cred.getName();  
    String password = String.valueOf(cred.getPassword());  
    System.out.println("Credential username: " + username);  
    System.out.println("Credential password: " + password);  
  }  
} catch (Exception e) {  
  e.printStackTrace();  
  addAuditTrailEntry(e);  
}

If you want a cleaner code, declare the imports in the beginning of the BPEL file (just before “partnerLinks” tag) and use the simple class names – be careful about the differences in the import statement between BPEL 1.0 and 2.0: http://docs.oracle.com/cd/E23943_01/dev.1111/e10224/bp_java.htm#SOASE87123

3 – Prepare the SOA environment to accept your deployment, adding the “jps-manifest.jar” file in the BPEL compilation classpath “BpelcClasspath”:
17_image4

17_image5

17_image6

17_image7

Use the full qualified path here. The common path for this library is: $DOMAIN_HOME/oracle_common/modules/oracle.jps_11.1.1/jps-manifest.jar

4 – Create your credential for testing the code:

17_image8

17_image9

17_image10

17_image11

5 – Grant permission for your custom code to access your credential map and keys:

5.1 – Edit “system-jazn” file: $DOMAIN_HOME/config/fmwconfig/system-jazn-data.xml

5.2 – Add a grant permission, appending the following tag to the end of “jazn-data/system-policy/jazn-policy”:

<grant>
  <permissions>
    <permission>
      <class>oracle.security.jps.service.credstore.CredentialAccessPermission</class>
      <name>context=SYSTEM,mapName=test-cred-map,keyName=*</name>
      <actions>read</actions>
    </permission>
  </permissions>
</grant>

P.S.: omitting the “grantee” tag, any deployed code will get access to the credential map.

17_image12

6 – Restart both Admin and Managed Servers so “system-jazn” update can take effect;

7 – Test your code and see the magic:

17_image13

The approach shown in this article can be adapted to be used in other FMW technologies, like inside Service Bus (through a “Java Callout”) or some JavaEE application deployed on the WebLogic server.

Export Audit Trail from Database SOA 10g and 11g

For a while I thought that was impossible to query the SOA composite instance audit trail directly from database. Several references on internet say that only SOA Management APIs can understand the format in which this information is stored into SOA Dehydration Store. Based on this, I posted about a java tool which does the audit export based on these APIs: https://gibaholms.wordpress.com/2013/07/18/soa-11g-audit-trail-exporter-tool/

The “SOA 11g Audit Trail Exporter Tool” still very useful due to simple features like the easy-of-use, remote export, no need database access to export (only a valid EM user) and file output. However, I finally found a way to query the audit trail XML directly from Database. The audit trail extraction is very fast and simple, and can be done by anyone with access to the SOAINFRA tables.

Edit 03-12-2015: New package version 1.1 available, with a bug fix to work properly on SOA 10g (thanks FRIEDRICH OBER-HONGSERMEIER for the contribution).

To perform this extraction directly from database, I wrote a simple PL-SQL function that does the Audit Trail tricks. I encapsulated it into a package called SOA_UTIL which can be compiled into a controlled area with some governance:


CREATE OR REPLACE PACKAGE SOA_UTIL AS
  
   -------------------------------------------------------------------------------------------
   -- Written by   : Gilberto Holms (https://gibaholms.wordpress.com/)
   -- Last update  : 07/11/2014
   -- Version      : 1.1
   -- Description  : Utility code to handle Oracle SOA Suite tables
   -- Release notes:
   -- 	1.0: Initial release
   --	1.1: Bug fix to work properly on SOA 10g
   -------------------------------------------------------------------------------------------
  
  FUNCTION GET_AUDIT_TRAIL(P_SCHEMA_NAME VARCHAR2, P_CIKEY NUMBER) RETURN CLOB;
  
END SOA_UTIL;

/

CREATE OR REPLACE PACKAGE BODY SOA_UTIL AS
  
   -------------------------------------------------------------------------------------------
   -- Written by   : Gilberto Holms (https://gibaholms.wordpress.com/)
   -- Last update  : 07/11/2014
   -- Version      : 1.1
   -- Description  : Utility code to handle Oracle SOA Suite tables
   -- Release notes:
   -- 	1.0: Initial release
   --	1.1: Bug fix to work properly on SOA 10g
   -------------------------------------------------------------------------------------------
  
  FUNCTION GET_AUDIT_TRAIL(P_SCHEMA_NAME VARCHAR2, P_CIKEY NUMBER) RETURN CLOB
  AS
    V_AUDIT_BLOB BLOB;
    V_AUDIT_CLOB CLOB;

	V_CUR_AUDIT SYS_REFCURSOR;

	TYPE TP_AUDIT_RECORD IS RECORD(
      BLOCK NUMBER(38,0),
      LOG BLOB
	);
    TYPE TP_AUDIT_ARRAY IS TABLE OF TP_AUDIT_RECORD;
    V_AUDIT_ARRAY TP_AUDIT_ARRAY;
    V_AUDIT_COMPLETE BLOB;

	V_BUFFER_LENGTH PLS_INTEGER := 32767;
    V_BUFFER VARCHAR2(32767);
    V_READ_START PLS_INTEGER := 1;
BEGIN

	DBMS_LOB.CREATETEMPORARY(V_AUDIT_BLOB, TRUE);
	DBMS_LOB.CREATETEMPORARY(V_AUDIT_CLOB, TRUE);
	DBMS_LOB.CREATETEMPORARY(V_AUDIT_COMPLETE, TRUE);
	
	OPEN V_CUR_AUDIT FOR 'SELECT BLOCK, LOG FROM ' || P_SCHEMA_NAME || '.AUDIT_TRAIL WHERE CIKEY = :cikey ORDER BY COUNT_ID' USING P_CIKEY;
	FETCH V_CUR_AUDIT BULK COLLECT INTO V_AUDIT_ARRAY;
	CLOSE V_CUR_AUDIT;

	FOR j IN 1..V_AUDIT_ARRAY.COUNT LOOP
		IF j = 1 THEN
			DBMS_LOB.APPEND (V_AUDIT_BLOB, V_AUDIT_ARRAY(j).LOG);
		ELSE
			IF ( V_AUDIT_ARRAY(j).BLOCK = V_AUDIT_ARRAY(j-1).BLOCK ) THEN
				DBMS_LOB.APPEND (V_AUDIT_BLOB, V_AUDIT_ARRAY(j).LOG);
			ELSE
				DBMS_LOB.APPEND (V_AUDIT_COMPLETE, UTL_COMPRESS.LZ_UNCOMPRESS(V_AUDIT_BLOB));
				DBMS_LOB.CREATETEMPORARY(V_AUDIT_BLOB, TRUE);
				DBMS_LOB.APPEND (V_AUDIT_BLOB, V_AUDIT_ARRAY(j).LOG);
			END IF;
		END IF;
	END LOOP;

	DBMS_LOB.APPEND (V_AUDIT_COMPLETE, UTL_COMPRESS.LZ_UNCOMPRESS(V_AUDIT_BLOB));
	V_AUDIT_ARRAY.DELETE;

	FOR i IN 1..CEIL(DBMS_LOB.GETLENGTH(V_AUDIT_COMPLETE) / V_BUFFER_LENGTH) LOOP
		V_BUFFER := UTL_RAW.CAST_TO_VARCHAR2(DBMS_LOB.SUBSTR(V_AUDIT_COMPLETE, V_BUFFER_LENGTH, V_READ_START));
		DBMS_LOB.WRITEAPPEND(V_AUDIT_CLOB, LENGTH(V_BUFFER), V_BUFFER);
		V_READ_START := V_READ_START + V_BUFFER_LENGTH;
	END LOOP;
	
	RETURN V_AUDIT_CLOB;
	
END;
  
END SOA_UTIL;

/

Then you can use this function inside a main query which filters the desired instances. This function works for both SOA 10g and 11g.

Usage example for Oracle SOA Suite 11g:


SELECT 
  CI.COMPOSITE_NAME AS COMPOSITE_NAME,
  CI.CMPST_ID AS INSTANCE_ID,
  CI.CREATION_DATE AS CREATION_DATE,
  CI.STATUS AS STEP,
  SOA_UTIL.GET_AUDIT_TRAIL('FMW11116_SOAINFRA', CI.CIKEY) AS AUDIT_TRAIL_XML
FROM
  SOAINFRA.CUBE_INSTANCE CI
WHERE 
  CI.COMPOSITE_NAME = 'MyComposite'
  AND CI.CREATION_DATE BETWEEN (sysdate - 1) AND (sysdate)
  ORDER BY CI.CREATION_DATE DESC;

Usage example for Oracle SOA Suite 10g:


SELECT 
  CI.PROCESS_ID AS PROCESS_ID,
  CI.CIKEY AS INSTANCE_ID,
  CI.CREATION_DATE AS CREATION_DATE,
  CI.STATUS AS STEP,
  SOA_UTIL.GET_AUDIT_TRAIL('ORABPEL', CI.CIKEY) AS AUDIT_TRAIL_XML
FROM
  ORABPEL.CUBE_INSTANCE CI
WHERE 
  CI.PROCESS_ID = 'MyBPEL'
  AND CI.CREATION_DATE BETWEEN (sysdate - 1) AND (sysdate)
  ORDER BY CI.CREATION_DATE DESC;

Event Details

Edit 03-12-2015: Added Event Details topic.

The audit trail is a large XML containing an array of events. The event XML usually contains all the message and details, like show below:


<event ...>
	<message>...</message>
	<details><![CDATA[ ... ]]></details>
</event>

However, for some very large XML payload, these event details are stored outside this structure, causing the event XML looks like this:


<event ...>
	<message>...</message>
	<details id="0" />
</event>

As you can see above, there is no payload inside the details tag, but there is a detail “id”.

So, if you are parsing the audit trail and you are interested on the details payload, every time you face this situation for some specific instance, you must make a separate call to a table named “AUDIT_DETAILS”, like shown below:


SELECT UTL_COMPRESS.LZ_UNCOMPRESS(BIN) AS AUDIT_DETAILS_XML 
FROM SOAINFRA.AUDIT_DETAILS 
WHERE CIKEY = '162524' -- put the cikey here 
AND DETAIL_ID = 0; -- put the details id here 

I used this technique in the company I work for, and by using the concepts shown in this article, I was able to do a full extract and parsing from SOA 10g and 11g databases getting all the information I needed to my use case. I hope you find it useful too.

OSB Throw Exception in XQuery

In this post I will show how to throw exceptions from an Oracle Service Bus 11g XQuery and then handle it into the pipeline flow. This technique is very useful to identify business exceptions into transformations and do business validations against the message payload.

The function that throws the exception is very simple, and can be called from any part of an XQuery code:

error(xs:QName('MyBusinessFault'), 'My message for the fault')

To handle the exception in the pipeline flow, you must add a “Service Error Handler” and then add a “Stage” with the following pattern:

OSB Flow Pattern to handle the exception thrown from XQuery

1. Checks if the exception handled was thrown from a XQuery error function:

$fault/ctx:errorCode = 'BEA-382513'

2. Identify the specific business exception by its name:

contains(fn-bea:serialize($fault/ctx:reason), 'MyBusinessFault')

3. Create a SOAP Fault Body and assign to the “body” variable. Here comes a little trick to extract the message from the exception, you must do exactly as shown:

<soap-env:Body>
  <soap-env:Fault>
	 <faultcode>Server</faultcode>
	 <faultstring>{substring-after($fault/ctx:reason/text(), 'MyBusinessFault: ')}</faultstring>
	 <detail/>
  </soap-env:Fault>
</soap-env:Body>

4. Reply with failure.

The use case presented here is only an example, but you can adapt the code to do whatever you want.

SOA 11g Audit Trail Exporter Tool

This project is a simple audit trail exporter tool, made for Oracle SOA Suite 11g (tested on version 11.1.1.6). It works for both BPEL and BPM based SOA composites.

Edit 03-13-2015: Text reviewed.

The SOA “Dehydration Store” stores all instance info and audit trail on some SOAINFRA tables (like AUDIT_TRAIL, AUDIT_DETAILS, XML_DOCUMENT and so on). If you have read privileges to these tables, you can do the export directly from database, as I show in this article: Export Audit Trail from Database SOA 10g and 11g

However, if you don’t have direct access to database or want a simple and controlled way to get the audit information for a specific set of instances, you can do the export using the tool presented here.

In a production environment, as the number of instances increases, the Enterprise Manager interface stuck many times, and can be very slow to access composite instance audit trail. Furthermore, is impossible to find a single instance based on some request payload info.

To get programmatic access to SOA audit information and do much more cool things, Oracle provides the Oracle SOA Suite Infrastructure Management Java API:

http://docs.oracle.com/cd/E28280_01/admin.1111/e10226/soaadmin_apimanage.htm

This project embeds this API, and provides a command-line interface to export instances audit trail, which can be filtered by a date range or the proper instance id. The result is the full audit trail, in form of human readable XML, for every single auditable component of the instances that matches the search filters. Then you can use system tools like “grep” to find a specific instance based on payload content, and see its full execution audit trail.

You can download the SOA Audit Exporter Tool on the link below:

SOAAuditTrailExporter.zip

Usage example:


Full Export:

  java -jar SOAAuditTrailExporter.jar -host=10.2.2.35 -port=8000 -username=weblogic -password=welcome01 -outBaseFolder=/temp/SOAAuditTrailExporter -compositePartition=default -compositeName=MySOAComposite -compositeRevision=1.0

Export By Date:

  java -jar SOAAuditTrailExporter.jar -host=10.2.2.35 -port=8000 -username=weblogic -password=welcome01 -outBaseFolder=/temp/SOAAuditTrailExporter -compositePartition=default -compositeName=MySOAComposite -compositeRevision=1.0 -minCreationDate=07/01/2013 -maxCreationDate=07/03/2013

Export By Instance ID:

  java -jar SOAAuditTrailExporter.jar -host=10.2.2.35 -port=8000 -username=weblogic -password=welcome01 -outBaseFolder=/temp/SOAAuditTrailExporter -compositePartition=default -compositeName=MySOAComposite -compositeRevision=1.0 -instanceId=20158

Enjoy!

Siebel Connection Pool in Oracle Service Bus 11g

When a middleware is used to make a lot of Web Service calls to Siebel CRM, the usual login mechanism is too slow and expensive. In the usual login, the Siebel application makes one login / logoff to every Siebel layer involved (application, database, etc) to every single call. When making a lot of concurrent calls, contention may occur in any of the underlying layers and some requests can be delayed or rejected, returning an error to the caller in some cases.

To ease this problem and improve performance, the most commonly solution is to use a Stateless Session Management of Authorization, which means that username and password are provided only in the first call, and subsequent calls are made using a SessionToken that is returned every time for the next call.

Obviously it would be amazing to do some kind of Connection Pool, or more specifically, a Token Pool/Cache to open some stateless sessions on demand, and reuse them using the next tokens.

I did a search in Google to find some references and cases of this approach, and I found some articles about using JAX-WS SOAP Handlers. But I was disappointed because I can’t use SOAP Handlers in my Middleware, and I can’t propagate this pool solution along all my applications making a Point-To-Point integration to Siebel.

The SOA architecture that we use here takes advantage of the Oracle Service Bus 11g to encapsulate all Siebel calls acting as a Proxy, doing input validation, security centralization and enforcement, endpoint management, policy attachment, statistics monitoring, parameterized data enrichment, service lifecycle management, and so on. Basically, all clients take access to Siebel through OSB:

In this article I will show a solution to make the Siebel Connection Pool using SessionToken at the OSB level, in a SOA approach. The best thing about this solution is the fact that none of the Siebel service clients are impacted by this change because the connection pool is centralized at the mediation layer, in this case, OSB, and no interfaces are changed.

The Connection Pool Classes

To control the pool, I made some simple Java classes, the first one is a simple DTO do pack the Siebel username and password into a single object to act as the key to store the tokens (implementing equals and hashcode default generated by Eclipse):

package com.wordpress.gibaholms;

public class SiebelCredential {

	private String username;
	private String password;

	public SiebelCredential(String username, String password) {
		this.username = username;
		this.password = password;
	}

	@Override
	public String toString() {
		return "SiebelCredential [username=" + username + ", password=" + password + "]";
	}

	@Override
	public int hashCode() {
		final int prime = 31;
		int result = 1;
		result = prime * result + ((password == null) ? 0 : password.hashCode());
		result = prime * result + ((username == null) ? 0 : username.hashCode());
		return result;
	}

	@Override
	public boolean equals(Object obj) {
		if (this == obj)
			return true;
		if (obj == null)
			return false;
		if (getClass() != obj.getClass())
			return false;
		SiebelCredential other = (SiebelCredential) obj;
		if (password == null) {
			if (other.password != null)
				return false;
		} else if (!password.equals(other.password))
			return false;
		if (username == null) {
			if (other.username != null)
				return false;
		} else if (!username.equals(other.username))
			return false;
		return true;
	}

	public String getUsername() {
		return username;
	}

	public String getPassword() {
		return password;
	}

}

I created another simple class to hold a Siebel token with his creation date. This date will be used to avoid returning certainly expired tokens:

package com.wordpress.gibaholms;

import java.util.Date;

public class SiebelToken {

	private String token;
	private Date creationDate;

	public SiebelToken(String token, Date creationDate) {
		this.token = token;
		this.creationDate = creationDate;
	}
	
	public SiebelToken(String token) {
		this(token, new Date());
	}
	
	@Override
	public String toString() {
		return "SiebelToken [token=" + token + ", creationDate=" + creationDate + "]";
	}

	public String getToken() {
		return token;
	}

	public Date getCreationDate() {
		return creationDate;
	}

}

The most important class to control the pool is very simple, just making a static cache of a Queue of tokens mapped by credential (username and password pair). I also added some counters to be retrieved by one statistics service if necessary. As you can see, the methods to get/add tokens are synchronized to support concurrent requests from OSB. Obviously some short locks may occur, but the performance loss is insignificant compared to the login/logoff Siebel problem, because the methods are very simple and cheap:

package com.wordpress.gibaholms;


import java.util.HashMap;
import java.util.LinkedList;
import java.util.Map;
import java.util.Queue;

public class TokenPool {
	
	private static final int SIEBEL_TOKEN_EXPIRATION_IN_MINUTES = 15;
	
	private static final Map<SiebelCredential, Queue<SiebelToken>> tokensByCredential;
	private static long addCount;
	private static long pollCount;
	private static long hitCount;
	private static long missCount;
	
	static {
		tokensByCredential = new HashMap<SiebelCredential, Queue<SiebelToken>>();
	}
	
	public synchronized static String pollToken(String username, String password) {
		pollCount++;
		SiebelCredential credential = new SiebelCredential(username, password);
		if (tokensByCredential.containsKey(credential)) {
			Queue<SiebelToken> tokens = tokensByCredential.get(credential);
			while (tokens.size() > 0) {
				SiebelToken token = tokens.poll();
				long diffInMinutes = (System.currentTimeMillis() - token.getCreationDate().getTime()) / (60 * 1000);
				if (diffInMinutes < SIEBEL_TOKEN_EXPIRATION_IN_MINUTES) {
					hitCount++;
					return token.getToken();
				}
			}
		}
		missCount++;
		return null;
	}
	
	public synchronized static void addToken(String username, String password, String token) {
		addCount++;
		SiebelCredential credential = new SiebelCredential(username, password);
		if (!tokensByCredential.containsKey(credential)) {
			tokensByCredential.put(credential, new LinkedList<SiebelToken>());
		}
		tokensByCredential.get(credential).add(new SiebelToken(token));
	}

	public static void eraseCounters() {
		addCount = 0;
		pollCount = 0;
		hitCount = 0;
		missCount = 0;
	}
	
	public static long getAddCount() {
		return addCount;
	}

	public static long getPollCount() {
		return pollCount;
	}

	public static long getHitCount() {
		return hitCount;
	}

	public static long getMissCount() {
		return missCount;
	}
	
}

Notice also that we have a constant SIEBEL_TOKEN_EXPIRATION_IN_MINUTES to give us a hint to indicate if the token would be expired on Siebel. The token expiration time on Siebel is a system parameter named SessionTokenTimeout that can be adjusted according to your needs. The default value indicated in Siebel documentation is 15 minutes. More information can be found on the official docs here: http://docs.oracle.com/cd/B40099_02/books/EAI2/EAI2_WebServices32.html#wp178856

Using this simple date control we avoid having old tokens into pool and minimize the chances of waste Siebel calls to discover that the token is expired to try again with full header unnecessarily, improving the pool efficiency. However some expired tokens may still pass because sometimes may occur a milliseconds window between the token creation date on Siebel and the token creation date into our pool, so the expired token error still need to be handled.

Now just assemble these classes into a JAR file and we are ready to define the best Message Flow to get/put tokens into this cache in the appropriate moments.

The Service Bus Project

Our OSB project looks like this:

  • SiebelSWEBusiness.biz
    Is a generic business service for all Siebel Web Services. All SWE services use the same endpoint and are only differed by WSDL, so we can use a single BS of type “Any SOAP Service – SOAP 1.1”.

  • siebel-connection-pool-1.0.jar
    Is a simple JAR file containing the classes shown above in this article. We will use the “Java Callout” activity to get/add tokens from the pool using the static methods of the following class: “com.wordpress.gibaholms.TokenPool”.
  • SiebelAccount.sa
    Is a Service Account of type “Static” that holds the Siebel username and password. The OSB clients authenticate themselves through the Weblogic security realm using WS-Security, which centralizes the access policies to corporative resources.

  • AssignSiebelHeaderFull.xq
    Is a reusable XQuery which mounts the full Siebel header, with username and password:

  • AssignSiebelHeaderToken.xq
    Is a reusable XQuery which mounts the Siebel header with SessionToken tag only:

  • SiebelStatelessSession.proxy
    Is the main Proxy Service which contains all the logic to handle the requests and manage the Siebel tokens. It would be a bad idea to replicate all this logic along all Siebel Proxy Services, so I decided to encapsulate this logic into a single PS of type “Any SOAP Service” and protocol “local”. Then any Siebel service can use the connection pool, just routing the flow to this PS:

    The message flow shown above is the secret for the solution. To understand more deeply I suggest download the code and inspect the boxes, but I will try to explain the flow in a more high level:

    Stage

    Description

    AssignSiebelAccount Get the username and password from the SiebelAccount.sa for use later when mounting the headers
    GetTokenFromPool Try to get a token from the pool and assign to a variable. If the pool is empty, the variable will be null
    AssignSiebelHeader If exists token, the token is used in the header, otherwise, the full header is mounted
    BackupRequestBody This stage saves the original request body into a variable for the case of getting token expired error on the first attempt, because, in this case, it will be used for a second attempt
    DefineErrorType According to the return of the first attempt, we need to take some actions. The OSB Error Handler is a very bad place to put complex logic because of the product architecture, then the route handler only detects the type of error occurred and let the flow resume to the response pipeline, which takes the correct actions.
    HandleSiebelReturn In this place the flow detects four possibilities:

    TokenExpired: makes a second attempt using the full header, with username and password

    ErrorWithToken: put the response token into the pool and return the error

    ErrorWithoutToken: just return the error

    Success: put the response token into the pool and return the response

    For security reasons, the response header is removed from the message in all cases.

    HandleSecondAttemptError The flow will fall into this block only if the second attempt results in error too, so it just verify if the error response have a token, which is added into the pool, otherwise, the error is returned to the caller.
  • MySiebelService.wsdl
    This WSDL represents our Siebel regular SWE Web Service, generated by Siebel Tools. No secrets here.
  • MySiebelService.proxy
    This Proxy Service represents the access point to our Siebel service, which will be called by end consumers. Note that is very simple to reuse the “SiebelStatelessSession.proxy” by just creating an ordinary Proxy Service from Siebel service WSDL and adding a route, replying the original fault:

Supported Scenarios

  • Success on the first call – OK;
  • Generic application error on the first call, without returning token – OK;
  • Generic application error on the first call, returning token – OK;
  • Token expired error in the first call, and success on the retry call – OK;
  • Token expired error in the first call, and generic application error on the retry call, without retuning token – OK;
  • Token expired error in the first call, and generic application error on the retry call, retuning token – OK.

Benefits of the Solution

  • No changes to the service interfaces;
  • No changes to the service consumers;
  • No code replication;
  • No infrastructure logic leak to the service consumers;
  • Centralized solution;
  • Easy to manage and maintain;
  • Pluggable solution, easy to enable/disable to any service;
  • Incredible performance gain to the entire corporation, because all Siebel services can be pool enabled as fast as add a route in OSB.

Now we can easy enable the Connection Pool to any Siebel SWE service into the SOA mediation layer, with no impacts on actual service consumers and “zero” code replication, in a corporative way.

Attachments

(Updated 08/02/2012 – 08:47)

Source Code:
https://github.com/gibaholms/articles/tree/master/Siebel_Connection_Pool_in_OSB

XQuery TitleCase Function in Oracle Service Bus 11g

One of my clients made a requirement of transforming some customer data, returned by the legacy systems wrapped by OSB services, to the “Title Case” format, also known as “Pascal Case” and “Upper Camel Case”. That means that the text must be tokenized at the blank spaces and the first letter of every word must be in capitalized.

For example:

Input: hello world

Output: Hello World

The easiest and more convenient way to implement this requirement is by creating a function in pure Java and generating a Custom XPath or making a Java Call in OSB.

To my client, I made a pure Java Custom XPath, delivered with all the JUnit tests to guarantee the maintenance and evolution of the function. But, in parallel, I spent a little time making the “Title Case” function using only XQuery, just to exercise the language, which can be very complex at a first look.

Saying again, the only purpose of this article is to show the power of XQuery and demonstrate the kind of things that can be done using this powerful language, which supports variable creation, decision and flow control structures, like any other programming language.

Title Case XQuery test project structure:

Below is the “xq/TitleCase.xq” file, implemented only using XQuery:

(:: pragma type="xs:string" ::)

declare namespace xf = "http://tempuri.org/xq/TitleCase/";

declare function xf:TitleCase($str as xs:string) as xs:string {
	let $words := tokenize($str, '\s')
	return 
		let $result := 
			for $word in $words
			return concat(upper-case(substring($word, 1, 1)), substring($word, 2))
		return string-join($result, ' ')
};

declare variable $str as xs:string external;

xf:TitleCase($str)

To test the function, just create the XQuery above in the sbconsole and launch the test console:

The “Title Case” XQuery has only one string parameter, so input some text and execute it:

Then you must see the text “Title Cased”, like shown below:

To functions that will be widely and heavily used, making them using Java is the best approach of course. However, to do simplest and smaller things, or to avoid an xpath deploy and a server restart, you always can use the good XQuery language.

Attachments

Source Code:
https://github.com/gibaholms/articles/tree/master/XQuery_TitleCase_Function_in_OSB

Read XML Resource in Oracle Service Bus 11g

Some time ago I was thinking in how to read XML Resources available in OSB projects at runtime. It would be very useful to hold some configuration data to be read by the proxy services through XPath or Java Call. After many searches in Google, I did not find any documentation or example in how to read OSB resources content programmatically at runtime. The popular OSB APIs and MBeans do not provide any method to get the resources content, but only their references. The question was how to get the resources content through their references.

With the help of JShrink I spent some time decompiling the OSB native jar functions and making a reverse engineering to discover how to do this. Fortunately, after some good tries I found what I was looking for. Using some internal OSB built-in libraries we can access any project resource at runtime.

In this article I will show how to read a sample XML Resource programmatically at runtime. In this sample I will use a Custom XPath to read a XML resource that holds application configuration parameters. Obs.: the same API shown here can also be used into a Java Call.

The sample project structure will look like this:

File “xsd/Parameters.xsd”: a simple schema that we’ll use in this scenario to describe the XML structure that holds the parameters:

<xsd:complexType name="Parameters">
	<xsd:sequence>
		<xsd:element name="ParameterList" type="tns:ParameterList"/>
	</xsd:sequence>
</xsd:complexType>
<xsd:complexType name="ParameterList">
	<xsd:sequence>
		<xsd:element name="Parameter" type="tns:Parameter" minOccurs="0" maxOccurs="unbounded"/>
	</xsd:sequence>
</xsd:complexType>
<xsd:complexType name="Parameter">
	<xsd:sequence>
		<xsd:element name="Key" type="xsd:string"/>
		<xsd:element name="Value" type="xsd:string"/>
	</xsd:sequence>
</xsd:complexType>

File “xml/Parameters.xml”: the sample XML that will be read by our XPath function at runtime:

<ns0:Parameters xmlns:ns0="https://gibaholms.wordpress.com/samples/xsd/2012/02/parameters">
	<ns0:ParameterList>
		<ns0:Parameter>
			<ns0:Key>Color</ns0:Key>
			<ns0:Value>Blue</ns0:Value>
		</ns0:Parameter>
		<ns0:Parameter>
			<ns0:Key>Size</ns0:Key>
			<ns0:Value>500</ns0:Value>
		</ns0:Parameter>
	</ns0:ParameterList>
</ns0:Parameters>

File “xq/ReadXmlParameters.xq”: a sample test XQuery that calls our XPath function passing as argument the full reference path to the XML parameters file:

xquery version "1.0" encoding "Cp1252";
(:: pragma bea:schema-type-return type="ns0:Parameters" location="../xsd/Parameters.xsd" ::)

declare namespace xf = "http://tempuri.org/ReadXmlResource/xq/ReadXmlParameters/";
declare namespace ns0 = "https://gibaholms.wordpress.com/samples/xsd/2012/02/parameters";
declare namespace param = "https://gibaholms.wordpress.com/xpath/ReadXmlResource";

declare function xf:ReadXmlParameters() as element() {
    param:readXml("ReadXmlResource/xml/Parameters")
};

xf:ReadXmlParameters()

Now, let’s create the custom XPath that does the magic. Create a simple Java project and add the following compile dependencies:

  • <MIDDLEWARE_HOME>\Oracle_OSB1\modules\com.bea.common.configfwk_1.5.0.0.jar
  • <MIDDLEWARE_HOME>\Oracle_OSB1\modules\com.bea.core.xml.xmlbeans_2.1.0.0_2-5-1.jar
  • <MIDDLEWARE_HOME>\Oracle_OSB1\lib\modules\com.bea.alsb.resources.core.jar
  • <MIDDLEWARE_HOME>\Oracle_OSB1\lib\modules\com.bea.alsb.resources.xml.jar

The XPath project structure is shown below:

In the file “osb-readxmlresourcefunction.xml” we describe the XPath functions contract. This file is a requirement of OSB and must be copied to the functions directory with the generated jar file:

<xpf:xpathFunctions xmlns:xpf="http://www.bea.com/wli/sb/xpath/config">
	<xpf:category id="Custom Functions">
		<xpf:function>
			<xpf:name>readXml</xpf:name>
			<xpf:comment>This function reads a XML Resource file from OSB</xpf:comment>
			<xpf:namespaceURI>https://gibaholms.wordpress.com/xpath/ReadXmlResource</xpf:namespaceURI>
			<xpf:className>com.wordpress.gibaholms.xpath.ReadXmlResource</xpf:className>
			<xpf:method>org.apache.xmlbeans.XmlObject readXml(java.lang.String)</xpf:method>
			<xpf:isDeterministic>true</xpf:isDeterministic>
			<xpf:scope>Pipeline</xpf:scope>
			<xpf:scope>SplitJoin</xpf:scope>
		</xpf:function>
	</xpf:category>
</xpf:xpathFunctions>

The secret to access OSB resources at runtime is the various “*Repository” classes. The code of our “readXml” XPath function is shown below:

package com.wordpress.gibaholms.xpath;

import org.apache.xmlbeans.XmlException;
import org.apache.xmlbeans.XmlObject;

import com.bea.wli.config.Ref;
import com.bea.wli.config.component.NotFoundException;
import com.bea.wli.sb.resources.config.XmlEntryDocument;
import com.bea.wli.sb.resources.xml.XmlRepository;

public class ReadXmlResource {

	public static XmlObject readXml(String xmlRefPath) {
		Ref ref = new com.bea.wli.config.Ref("XML", Ref.getNames(xmlRefPath));
		XmlObject xmlObject = null;
		try {
			XmlEntryDocument xmlEntryDocument = XmlRepository.get().getEntry(ref);
			String xmlContent = xmlEntryDocument.getXmlEntry().getXmlContent();
			xmlObject = XmlObject.Factory.parse(xmlContent);
		} catch (NotFoundException e) {
			e.printStackTrace();
			throw new RuntimeException("XML Resource not found.");
		} catch (XmlException e) {
			e.printStackTrace();
			throw new RuntimeException("Error parsing XML content.", e);
		}
		return xmlObject;
	}
	
}

As we can see in the code, the class “com.bea.wli.sb.resources.xml.XmlRepository” does the magic of retrieve the XML resource content through it “com.bea.wli.config.Ref” reference. Following the same principle, we can use the other various “*Repository” classes to access other types of resources in projects at runtime, like Schemas, XSLTs, WSDLs and so on.

To test the function, just run the XQuery “xq/ReadXmlParameters.xq” in the sbconsole:

The test XQuery has no parameters, so just execute it:

Then you must see the parameters XML loaded at runtime, like shown below:

I’m glad to share this discovery with other ALSB / OSB developers and I hope help you to do more useful things with this knowledge.

If someone already has another solution to read OSB project resources at runtime or already use this API for some other purpose, please share the experience with us through comments. Thanks.

Attachments

Source Code:
https://github.com/gibaholms/articles/tree/master/Read_XML_Resource_in_OSB

Advanced Validation in Oracle Service Bus 11g

When implementing Proxy Services in Oracle Service Bus (old BEA AquaLogic Service Bus), it’s important to think about validation of request data.

The OSB provides a Validate action that is a good mechanism to validate when the incoming request is conformant to the XML Schema defined in the service contract. However, using only this action is a very poor mechanism to effectively validate a request because of the following reasons:

  • The Validate action accepts validation only against XML Schema (xsd), that has a limited syntax and cannot support more complex validations or rules;
  • XML Schema cannot support business oriented validations, only structural validations (and very limited);
  • A good choice for better support in validation would be Schematron, but Oracle Service Bus 11g does not provide Schematron support yet (SOA Suite Mediator already supports, but OSB not);
  • If your company uses Canonical Model, is very common to use on all your xsd entities elements with minOccurs=”0” to get full reusability on the model, however is hard to validate when a specific service operation needs that an element be mandatory, and it weakens the service contract.

In this article I will show a good pattern that I use to do more advanced, business oriented validation in Oracle Service Bus using XQuery. Note that deep XQuery knowledge is not in the scope of this article, for more in-depth knowledge about XQuery you need to google (a good start can be found at http://www.w3schools.com/xquery/default.asp). I will predict that the reader has basic XPath concepts to find some nodes in a XML instance (more at http://www.w3schools.com/xpath/default.asp).

In this sample I will focus on the Proxy Service, so I will ignore any business-proxy-business transformations and the sample will show a simple Business Service virtualization through data by-pass, but implementing some complex validation on the request input that is the main objective of this article. You can download the example artifacts in the links bellow this article.

Let’s take a look into the sample request input message that we’ll use in this scenario:

<xsd:element name="OrderRequest">
<xsd:complexType>
<xsd:sequence>
<xsd:element name="Customer">
<xsd:complexType>
<xsd:sequence>
<xsd:element name="FirstName" type="xsd:string" minOccurs="0" />
<xsd:element name="LastName" type="xsd:string" minOccurs="0" />
<xsd:element name="DocumentNumber" type="xsd:string" minOccurs="0" />
<xsd:element name="Email" type="xsd:string" minOccurs="0" />
<xsd:element name="Age" type="xsd:int" minOccurs="0"/>
<xsd:element name="Password" type="xsd:string" minOccurs="0" />
<xsd:element name="PasswordConfirmation" type="xsd:string" minOccurs="0" />
</xsd:sequence>
</xsd:complexType>
</xsd:element>
<xsd:element name="Product">
<xsd:complexType>
<xsd:sequence>
<xsd:element name="ProductID" type="xsd:string" minOccurs="0" />
<xsd:element name="ProductRestriction" minOccurs="0">
<xsd:simpleType>
<xsd:restriction base="xsd:string">
<xsd:enumeration value="EVERYONE"/>
<xsd:enumeration value="OVER18"/>
</xsd:restriction>
</xsd:simpleType>
</xsd:element>
<xsd:element name="Quantity" type="xsd:int" minOccurs="0" />
</xsd:sequence>
</xsd:complexType>
</xsd:element>
</xsd:sequence>
</xsd:complexType>
</xsd:element>

Observations:

  • Obviously this dummy contract is not real world and has no business relevance in an real order scenario;
  • The sample simulates the use of Canonical Model entities which sets minOccurs=”0” all over the place, but we know that some information is mandatory to our execute order operation.

The starting point of this sample is the structure shown below, that consists of:

  • One OSB Project named “AdvancedValidation” in the default configuration
  • The “OrderService-v1.wsdl” that represents the contract used in the sample
  • One Business Service named “OrderBusiness” created from the OrderService WSDL (I recommend you to generate a Mock Service in SoapUI tool to point this Business Service to and run your tests)
  • One Proxy Service named “OrderProxy” created from the OrderService WSDL, with a basic Message Flow that only routes the original request to the OrderBusiness in the execute operation.

By now the focus will be the Proxy Service message flow.

To start our robust validation let’s add the main pipeline pair and put the simple “Validate” action into the request branch, because the minimum expected is that the request message does comply to the XML Schema, and if don’t, the flow does not need to work anymore. In the XPath select the OrderRequest input element in the wizard, in variable body against its XML Schema definition present into the OrderService WSDL.

– XPath: ./ord:OrderRequest
– In Variable: body
– Against Resource: OrderService-v1.wsdl/OrderRequest
– Raise Error

Now comes the great part, lets prepare our advanced XQuery validation. Create a XSD named “validation.xsd” into xsd folder. This schema will hold the validation data though the flow. Use the following structure:

<xsd:complexType name="Validation">
<xsd:sequence>
<xsd:element name="Payload">
<xsd:complexType mixed="true">
<xsd:sequence>
<xsd:any/>
</xsd:sequence>
</xsd:complexType>
</xsd:element>
<xsd:element name="ValidationErrorList" type="tns:ValidationErrorList"/>
</xsd:sequence>
</xsd:complexType>
<xsd:complexType name="ValidationErrorList">
<xsd:sequence>
<xsd:element name="ValidationError" type="tns:ValidationError" minOccurs="0" maxOccurs="unbounded"/>
</xsd:sequence>
</xsd:complexType>
<xsd:complexType name="ValidationError">
<xsd:sequence>
<xsd:element name="code" type="xsd:int"/>
<xsd:element name="message" type="xsd:string"/>
</xsd:sequence>
</xsd:complexType>

The next step is create the XQuery transformation file to do the validation and fill the previous structure. The input of transformation is the OrderRequest schema and the output is the Validation schema.

– Source Types: OrderService-v1.wsdl/OrderRequest
– Target Types: validation.xsd/Validation

Now I suggest you to forget the visual editor. Drag-and-drop fields will not help you in more complex transformations (it’s not .NET rs) and will make a mess in the source code, so the better to do is reading some concepts of XQuery and then make your code by the hands. In this sample we’ll validate the following:
– The mandatory fields to the operation, that are: FirstName, DocumentNumber, Email, Age, ProductID and Quantity
– DocumentNumber: must contain only numbers
– Email: must comply to the regex (^[a-z0-9]+([\._-][a-z0-9]+)*@[a-z0-9_-]+(\.[a-z0-9]+){0,4}\.[a-z0-9]{1,4}$)
– Password and PasswordConfirmation: if filled, must be the same value
– ProductRestriction: if filled and value is “OVER18”, the customer must be 18+ years old
– Echo the full request message in the payload tag to use in the flow if necessary

(:: pragma bea:global-element-parameter parameter="$orderRequest" element="ns1:OrderRequest" location="../wsdl/OrderService-v1.wsdl" ::)
(:: pragma bea:schema-type-return type="ns0:Validation" location="../xsd/validation.xsd" ::)

declare namespace xf = "http://tempuri.org/AdvancedValidation/xq/RequestValidation/";
declare namespace ns1 = "https://gibaholms.wordpress.com/samples/wsdl/Order-v1.0";
declare namespace ns0 = "https://gibaholms.wordpress.com/samples/xsd/2011/11/validation";

declare function xf:RequestValidation($orderRequest as element(ns1:OrderRequest)) as element() {
<ns0:Validation>
<ns0:Payload>{$orderRequest/.}</ns0:Payload>
<ns0:ValidationErrorList>{
(: BEGIN - Required Field Validations 🙂
if (empty($orderRequest/ns1:Customer/ns1:FirstName/text())) then
<ns0:ValidationError>
<ns0:code>1</ns0:code>
<ns0:message>FirstName: Required Field</ns0:message>
</ns0:ValidationError>
else if (empty($orderRequest/ns1:Customer/ns1:DocumentNumber/text())) then
<ns0:ValidationError>
<ns0:code>2</ns0:code>
<ns0:message>DocumentNumber: Required Field</ns0:message>
</ns0:ValidationError>
else if (empty($orderRequest/ns1:Customer/ns1:Email/text())) then
<ns0:ValidationError>
<ns0:code>3</ns0:code>
<ns0:message>Email: Required Field</ns0:message>
</ns0:ValidationError>
else if (empty($orderRequest/ns1:Customer/ns1:Age/text())) then
<ns0:ValidationError>
<ns0:code>4</ns0:code>
<ns0:message>Age: Required Field</ns0:message>
</ns0:ValidationError>
else if (empty($orderRequest/ns1:Product/ns1:ProductID/text())) then
<ns0:ValidationError>
<ns0:code>5</ns0:code>
<ns0:message>ProductID: Required Field</ns0:message>
</ns0:ValidationError>
else if (empty($orderRequest/ns1:Product/ns1:Quantity/text())) then
<ns0:ValidationError>
<ns0:code>6</ns0:code>
<ns0:message>Quantity: Required Field</ns0:message>
</ns0:ValidationError>
(: END - Required Field Validations 🙂

else

(: BEGIN - Field Specific Validations 🙂
if (not(matches($orderRequest/ns1:Customer/ns1:DocumentNumber/text(), '^[0-9]+$'))) then
<ns0:ValidationError>
<ns0:code>7</ns0:code>
<ns0:message>DocumentNumber: Must Contain Only Numbers</ns0:message>
</ns0:ValidationError>
else if (not(matches($orderRequest/ns1:Customer/ns1:Email/text(), '^[a-z0-9]+([\._-][a-z0-9]+)*@[a-z0-9_-]+(\.[a-z0-9]+){0,4}\.[a-z0-9]{1,4}$'))) then
<ns0:ValidationError>
<ns0:code>8</ns0:code>
<ns0:message>Email: Not a Valid Email Format</ns0:message>
</ns0:ValidationError>
else if (not(empty($orderRequest/ns1:Customer/ns1:Password/text()))
and $orderRequest/ns1:Customer/ns1:Password/text() != $orderRequest/ns1:Customer/ns1:PasswordConfirmation/text()) then
<ns0:ValidationError>
<ns0:code>9</ns0:code>
<ns0:message>Password: Must Match PasswordConfirmation</ns0:message>
</ns0:ValidationError>
else if (not(empty($orderRequest/ns1:Product/ns1:ProductRestriction/text()))
and $orderRequest/ns1:Product/ns1:ProductRestriction/text() = 'OVER18'
and xs:int($orderRequest/ns1:Customer/ns1:Age/text()) < 18) then
<ns0:ValidationError>
<ns0:code>10</ns0:code>
<ns0:message>Customer Must Be OVER 18 Years</ns0:message>
</ns0:ValidationError>
(: END - Field Specific Validations 🙂

else ''
}</ns0:ValidationErrorList>
</ns0:Validation>
};

declare variable $orderRequest as element(ns1:OrderRequest) external;

xf:RequestValidation($orderRequest)

Now add an “Assign” activity to evaluate the validation XQuery against a newly created validation variable. We’ll use this variable in the error handler to gain access to the validation codes and messages.

– Expression: $body/ord:OrderRequest
– Variable: validation

We put now an “IF-ELSE” control to check if exists any validation errors. Don’t forget to add the validation xsd target namespace into “val” prefix.

– Add Namespace Definition: val – https://gibaholms.wordpress.com/samples/xsd/2011/11/validation
– Condition: count($validation/val:ValidationErrorList/val:ValidationError)

Then we raise a custom error with the code of our choice “BUS-1”. Note that this code will be used to capture the error in the handler, just remember that code.

– Code: BUS-1
– Message: Request Validation Error

Now the pipeline is finished. We now concentrate the fault treatment in a single error handler associated to the entire flow, adding a “IF-ELSE” action to act as a switch, verifying which of the code is present in the “fault” variable and manually throwing the typed faults declared in our service contract (OrderBusinessFault and OrderTechnicalFault). Obs.: the code “BEA-382505” refers to the internal OSB code that is used for “Validate” action errors.

– Condition: $fault/ctx:errorCode = ‘BEA-382505’


– Condition: $fault/ctx:errorCode = ‘BUS-1’

Now just add an Assign activity in the “if” branches that apply the typed soap faults. Just remember that the built-in $fault variable refers to the OSB infrastructure fault, and not to soap fault. So if you try to assign a soap fault directly to this variable, it just doesn’t work. To throw a soap fault you must assign the entire soap body. Don’t forget to add the “ord” prefix namespace pointing to “https://gibaholms.wordpress.com/samples/wsdl/Order-v1.0”. First let’s add the technical faults, that apply in case of schema validation failure (BEA-382505) and case else (any infrastructure error that we are not treating):

– Add Namespace Definition: ord – https://gibaholms.wordpress.com/samples/wsdl/Order-v1.0
– Variable: body
– Expression:

<soap-env:Body>
<soap-env:Fault>
<faultcode>soap-env:Server</faultcode>
<faultstring/>
<detail>
<ord:OrderTechnicalFault>
<ord:message>Schema Validation Failure</ord:message>
</ord:OrderTechnicalFault>
</detail>
</soap-env:Fault>
</soap-env:Body>

– Add Namespace Definition: ord – https://gibaholms.wordpress.com/samples/wsdl/Order-v1.0
– Variable: body
– Expression:

<soap-env:Body>
<soap-env:Fault>
<faultcode>soap-env:Server</faultcode>
<faultstring/>
<detail>
<ord:OrderTechnicalFault>
<ord:message>SOA Infrastructure Error</ord:message>
</ord:OrderTechnicalFault>
</detail>
</soap-env:Fault>
</soap-env:Body>

Now to add the treatment to our “BUS-1” fault, we can use the message applied by the validation XQuery (notice that any information that might be of interest can be propagated through the $validation variable). Don’t forget to declare the “val” namespace prefix to “https://gibaholms.wordpress.com/samples/xsd/2011/11/validation”:

– Add Namespace Definition: ord – https://gibaholms.wordpress.com/samples/wsdl/Order-v1.0
– Variable: body
– Expression:

<soap-env:Body>
<soap-env:Fault>
<faultcode>soap-env:Server</faultcode>
<faultstring/>
<detail>
<ord:OrderBusinessFault>
<ord:message>{$validation/val:ValidationErrorList/val:ValidationError[1]/val:message/text()}</ord:message>
</ord:OrderBusinessFault>
</detail>
</soap-env:Fault>
</soap-env:Body>

To finish our sample, just add the “Reply” activities to each “if” branch indicating that invocation returned “With Failure” (causes http 500 return code):

– Reply: With Failure

Well done! You can open the OSB console and do the tests in the Proxy Service. Input some data and observe the results. Don’t forget to start the Mock Service into SoapUI and point the Business Service endpoint to the mock address. An easy way to simulate a generic technical fault is stopping the mock service, crashing the business service like it was “unavailable”.Now I hope this article has helped you to doing more complex validations beyond the limits of xsd into Oracle Service Bus 11g.

Attachments

Source Code: https://github.com/gibaholms/articles/tree/master/Advanced_Validation_in_OSB