Oracle SOA 11g – Writing zip files using FTP’s Pipelines/Valves

Oracle SOA suite comes with variety of adapters. Among which, FTP & File adapters are widely used in larger enterprise while we had to work with files. Here, we are going to use the concept of valves/Pipeline for zipping files post processing. However, the usage is not just restricted to zipping. Think about some of the well-known use cases like encryption/decryption, compressing/decompressing, validating, transform, filter, etc, etc. Of-course, these can be achieved post & pre processing the data in FTP & File adapter.

Lets jump into it !

Use Case :   I am writing about the same use case which pushed me to explore valves. We live in a larger telecom company whose network generates Call Detail Record (CDR) for each customers in CSV format. Each file has CDR for multiple customers. Arguably, the billing system (Old) can process CSV file having single entry for CDR but in ZIP format :(. We have had used FTP adapter for connecting, processing & splitting the CSV and used the logic of Valves to ZIP the generated file.

Simply put, a valve is a customized java class attached with the FTP/File adapter before the data arrives to inbound adapter or as soon as the data leaves the outbound adapter. You can write your own valve which suites your requirement (like filtering, encrypting, transforming).  Two or more valves can be connected in sequence to form a Pipeline. So, you can perform validation in one valve, transformation in next , encryption in next & zipping at last valve of your outbound pipeline (or post processing). Below is the image given in Oracle for pre-processing a file :

pipeline

Although i am not going to explain a lot about how to set FTP adapter for polling and writing, below is how it make look like :

screenshot-from-2017-02-06-21-28-03

I archived zipping the files in two simple ways.

  1. create the needed valve for zipping & deploy it in server
  2. Attach the deployed valve to the FTP adapter to leverage it.

Step 1 : Create the needed valve : 

All valves should be extended from AbstractValve or StagedValve. The definition of these classes can be found in a jar located at $Middleware_home/Oracle_SOA1/soa/modules/oracle.soa.fabric_11.1.1/bpm-infra.jar. So while preparing your valve in any IDE, add the jar to your projects libraries. The class should have a method ‘execute’ getting & returning InputStreamContext object. I wrote something like this for our compress requirement:

public InputStreamContext execute(InputStreamContext inputStreamContext)
throws PipelineException, IOException {
String fileName = inputStreamContext.getMessageOriginReference();
System.out.println(“Zipping begins for ” + fileName);
InputStream realStream = inputStreamContext.getInputStream();
ByteArrayOutputStream fos = new ByteArrayOutputStream();
ZipOutputStream zip = new ZipOutputStream(fos);
zip.putNextEntry(new ZipEntry(fileName.split(“zip”)[0]+”csv”));
byte[] buffer = new byte[4096];
int length = 0;
while ((length = realStream.read(buffer)) >= 0) {
zip.write(buffer, 0, length);
}
zip.closeEntry();
zip.close();
System.out.println(“Done with the compression…”);
inputStreamContext.setInputStream(new ByteArrayInputStream(fos.toByteArray()));
return inputStreamContext;
}

Now, export the project as a jar and add it to your class Oracle SOA suite runtime environment $MIDDLEWARE_HOME/user_projects/domains/soa_domain/lib.

And, restart the server for SOA server to load that to the classpath.

Step 2 :Attach the deployed valve to the FTP adapter : 

Now, we have the valve which is ready to use it in our outbound FTP adapter. That can be done by adding the name of the file holding the list of valves in your JCA as one of the property. Something like this :

<property name=”PipelineFile” value=”mycompression.xml”/>

And, mycompression.xml which is added to my soa-project may look like below :

<?xml version=”1.0″ encoding=”UTF-8″ ?>
<pipeline xmlns=”http://www.oracle.com/adapter/pipeline”&gt;
<valves>
<valve>zipValves.Compress</valve>
</valves>
</pipeline>

This XML is special (called pipeline). The element valve can refer to one defined valve. In my case, Compress is my class name which is in zipValves package. Similarly, you can have N number of valves based on your requirement. Each valves are executed one after the other as referred in the pipeline xml.

Now, deploy the SOA project again and test it. Yahoo !  the files are zipped. For testing, before adding telecom.csv (having multiple CDR) to the read location, the folder looks like this :

screenshot-from-2017-02-06-21-58-27

Now, post processing the file, the directory will have files like this :

screenshot-from-2017-02-06-22-01-00

Below are some of the article that i have read before starts exploring it. It can be useful :

  1. Oracle
  2. AMIS
  3. javaoraclesoa

You can download the project from GIT.

Let me know your comments…. 🙂

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s