Exploring Mule..

ESB journey continues……

As mentioned in my previous blog, Enterprise service bus is a generic name for any solution that provides communication and translation between applications or between different processes within an application. ESB standardizes the notion of message bus. ESB is a topology, a way to organize components and their interactions.

In short, ESB is a way of doing things, “ a design pattern” to continue the evolution of standardizing the application integration.

The enterprise service bus is all about allowing different components or applications to communicate through a common messaging bus, usually implemented using JMS, MQ, or other messaging server. It allows traffic of all types to traverse it and has adaptors to allow servers of different message types to communicate seamlessly across the bus. For example, if you have an application that needs to raise an alarm based on some condition, and the action of that alarm is to send an email or an instant message to the administrator, the module that raises the alarm should not need to 'understand' SMTP or SIP (Session Initiation Protocol); it should just put the message on the bus and have the bus and the intended recipient handle it as they see fit

Mule is a lightweight messaging framework and an object broker that handles the interactions between your applications across JMS, e-mail, HTTP, and even XML-RPC.

Mule is in no way a replacement for existing application framework. In fact Mule leverages many open source projects such as Axis, Spring, ActiveMQ, Plexus and PicoContainer.

Mule fills a void in enterprise java development where an application requires complex interactions with a variety of systems on a variety of platforms. Mule makes light work of wiring these systems together in a robust decoupled environment and provides the necessary support to route, transport and transform data to and from these systems.

Mule provides a container that manages message-capable services known as Universal Message Objects (UMOs). UMOs can be plain old Java objects (POJOs). UMOs are made available to other UMOs and applications via message endpoints that allow messages to be routed across a wide array of protocols and technologies such as JMS, SMTP, JDBC, TCP, HTTP, file, intra-VM, etc.

Here are the principal ESB Mule components :

Mule Manager:- The Mule Manager is the primary component for each instance of a Mule server and is used to manage all Mule objects (connectors, endpoints, transformers, etc.) for each Mule server instance

Mule Model:- The Mule Model is the container in which user-supplied components are managed and executed. It controls message flow to and from user-supplied components and manages threading, lifecycle, and pooling of the components.

Universal Mule Objects (UMO) :- A UMO is a type of Java object that can
1. receive events "from anywhere"
2. send events
UMO Components are usually your business objects. They are components that execute business logic on an incoming event. UMO are standard JavaBeans (containers). Mule handles all routing and transformation of events to and from your objects based on the configuration of your component. Typically your POJO with translates into an UMO with the Mule configuration.

Connector:- A connector is the object that sends and receives messages on behalf of an endpoint. Connectors are bundled as part of specific transports or providers. For example, the FileConnector can read and write file system files. JDBC connector to connect to relational databases using jdbc. It supplies reads and writes to simple tables with acknowledgement of read rows.

Router:- A router is the object that do something with messages once they have been received by a connector, or prior to being sent out by the connector


Transports:- A transport or "provider", is a set of objects that add support to Mule to handle a specific kind of transport or protocol
Examples - the "Email Provider" enables Mule to send and receive messages via the SMTP, POP and IMAP protocols


Filter:- A filter optionally filters incoming or outgoing messages that are coming into or going out from a connector. Filters are used in conjunction with Routers
For example, the File Provider comes with a FilenameWildcardFilter that restricts which files are read by the connector based on file name patterns, like only files with the .xml extension can be routed

Transformers:- A transformer optionally changes incoming or outgoing messages in some way. This is usually done to make the message format usable by a downstream function
Examples - the ByteArrayToString transformer converts byte arrays into String objects

Endpoints:- Endpoints are used to connect to components in the server and external systems or to each other locally or over the network. An Endpoint is a specific channel through which two parties can communicate. Endpoints can be configured on various objects in Mule

• Inbound Routers - configured on Components and allow one or more endpoints to be registered on a component.


• Outbound Routers - configured on components and allow one or more outbound endpoints to be configured (with additional logic for controlling which endpoints are used for an event


• Catch all strategies - allow a single 'catch-all' endpoint to be configured for a router.


• Components - for convenience a single inbound and outbound endpoint can be set on the component directly, which is a little easier than configuring routers.


• Exception Strategies - A single endpoint can be configured on an exception strategy to all events to be sent to an error endpoint.
  
  Mule Event Flow
  
  

The nine stages of a mule event

Message Receiver Endpoint
Some Event triggers a message flow. For example

• A file being written into a folder


• A message arriving on a message queue


• A record in a database


• Data written to a socket

Inbound Router
The inbound router is the fist step in a message. Functions typically performed by an inbound router

• Filtering


• Remove duplicate messages


• Matching messages


• Aggregation (combining)


• Re-sequence data


• Forwarding

Interceptor
Used to intercept message flow into your service component or used to trigger monitor/events or interrupt the flow of the message
Example: an authorization interceptor could ensure that the current request has the correct credentials to invoke the service.


Inbound Transformer
If the inbound data is not in the correct format for the service it must be transformed at this point

Service Invocation
The actual service is performed. Service invocation can also be a "pass through"

Outbound Router
Dispatching the data to all the relevant endpoints

Outbound Transformer
Any transformations that needs to be done on the message after a service has been performed on the message can be executed before it is put into the endpoint

There are two code samples attached.

Sample 1 – This is the enhanced version of Stockquote webapp. This example fetches the Stock information the website and dumps into the DB using a JDBC connector.

Sample 2 (EmployeeSync) – This shows a typical data integration scenario using mule. The flow goes as follows.
The canonical Employee is called as the Global Generic Employee is also the source. To create or update an employee Information, we use the StdIn console. Once the data is entered, the control transfers to jBPM engine. jBPM was not a must here. The functionality could have been easily achieved by a custom filter. But I wanted to explore the jBPM integeration with Mule. The jBPM process engine identifies whether it is a Insert or an Update function and sets the appropriate endpoint. The Endpoint Selector appropriately then transfers the control to the corresponding message inbound flow.
If the employee is inserted, then the employee information is propogated to HRDB and payrollDB, which holds subset of cannonical Employee information. The new employee ids generated is stored back in the Global Generic Employee DB.


Conclusion:
Mule can be considered as “Spring for messaging”. In as non-invasive a manner as possible it connects, Java objects with messaging technologies. Mule provides a standard, generic interface to a wide variety of messaging systems. Mule is an enormous and very flexible framework that allows you to build such applications. If you are looking to build an agile business, and don't want to spend the heavy dollars that something like BizTalk, Websphere Message Broker would cost, it is well worth a look.

Next Steps:
This example should help you get started in configuring and extending the Mule enterprise service bus. Hopefully, it would have given you a greater understanding of the principle of how ESB works. Next I shall be posting on my experience with Mule- on a asynchronous long running transaction

Enterprise Service Bus - An Overview

Before we start talking about Enterprice Service Bus, lets us se what are the challenges of a distributed software system.

The challanges for a distributed software system

1. Interface or touch-point complexity between different pieces of the software components that are spread out across computers and networks, dependencies are built between those components, computers, and networks. If a software component is moved to reside on a different computer, or if a computer is moved onto a different network, or if the network architecture is modified in any way, there may be a rippling affect across the associated distributed applications.
2. Reduced reliability as more computers involved in processing a user request, the greater the chance a failure will occur. By far the most challenging of the reliability issues will be those that involve software communication. It's vital that when a component makes a request to another component, the response should arrive accurately and in a timely manner. The proper mechanisms must be in place to ensure that messages are re-sent when they are lost, and that they arrive in the same form as when they were sent, even if they are sent over foreign or unreliable networks.
3. There is a security challenge, when distributed components communicate over a network, as there is a potential for that communication to be tapped into, or otherwise spoofed. The potential for trouble increases when this communication spans networks, organizations, and the Internet. Malicious software, or people, may snoop network activity to retrieve sensitive data.

The Enterprise Service Bus Solution
There are patterns and tools that can be used to eliminate the three challanges just discussed. The three high level patterns or principles that provide for robust distributed software are

1. Reliable Communication - most important part of a distributed software system is its communication infrastructure
2. Service Oriented Architecture - Decomposition of software functonality into smaller pieces, that work with platform-neutral data structures and protocols. Examples of technologies in these areas are web services, the Java Message System (JMS), eXtensible Markup Language (XML), and the Simple Object Access Protocol (SOAP), respectively
3. Service Orchestration - Patterns and standards for the integration and proper orchestration of the services created in step 2

A tool is needed to enable these principals -- one that helps to reduce the complexity and costs associated with distributed software systems by implementing the three patterns. This tool is the Enterprise Service Bus (ESB). The ESB is both a tool and a software framework that helps you to create, deploy, and orchestrate (and communicate between) service components in a distributed system ESB solution for Interface or touch point complexity
No longer are the communication lines between disparate systems hard-coded. It helps to envision distributed application development and deployment differently. Adapters connect disparate software, enterprise messaging systems can be leveraged, and services can be integrated without writing a single line of associated code. This lets you focus on the construction of services and their associated business logic and not on the costly chores that go into building a robust distributed system -- the ESB eliminates them. That solves the complexity issue.

ESB solution for Reliability Challenge
ESB has built-in reliable messaging backbone. Because of this, reliability is virtually guaranteed to components built on the ESB. Additionally, the messages themselves, and the act of sending those messages, are normalized. This means that regardless of the initial form of the data you are sending between systems, and the form of communication a particular system is designed to use, the ESB transforms those messages and adapt various protocols to work together

ESB solution for Security Challenge
The ESB provides authentication and authorization services, as well as secure channels for component communication, thus solving most security issues. It has the added benefit of ensuring that security is implemented consistently across all services and components that use the ESB.

The ESB can best be described as having these characteristics

• Provides a reliable messaging infrastructure
• Enables SOA-based system development
• Is XML-based
• Supports web service standards (such as SOAP)
• Is platform independent
• Supports data transformation and routing services
• Enables service orchestration
• Supports transactions and security Integrates with existing standards, frameworks, and legacy systems

Overall, the foundation of the ESB includes enterprise messaging, SOA standards, message transformation, and message routing services, upon which you can efficiently build robust, distributed applications

Enterprise Messaging - Communication Infrastucture of ESB

Enterprise messaging systems provide message delivery guarantees, as well as transactional support, most often with two-phase commit. This provides the ESB, along with the applications built on top of them, a level of built-in reliability required for mission-critical application. For software components developed for an ESB, there is generally nothing special that needs to be done to tap into this service. It's provided transparently by the ESB and the enterprise messaging backbone it itself is built upon.
Enterprise messaging systems, and the ESB, provide the capability for software components to send, listen for, and process messages asynchronously. This helps to decouple components in a distributed software system, further enabling system reliability and scalability

The Java Messaging System (JMS) pecification describes an enterprise messaging system for Java that supports both topic-based and queue-based messaging paradigms. JMS also specifies many types of message objects with the goal of interoperating with other, non-Java/JMS, messaging systems. With enterprise messaging -- and in particular, JMS -- you can build distributed software systems that are truly platform and language-independent. JMS is a specification that describes the properties and behavior of an information pipe for Java software. These are two very important distinctions that must be understood before implementing JMS software. They are

1. JMS Provider - This is an enterprise messaging implementation that adheres to the JMS specification for message definition, delivery, storage, processing, and error handling. It is the implementation of the message pipe itself, not the software that uses it
2. JMS Application- Sometimes called a JMS client, this is the Java software that uses a JMS provider to send and receive messages over an information pipe. This software can be any application, small or large, that generates or processes messages

Standards are required to drive down the cost of integration. These standards need to account for platform and language independence. It also lets the developers focus on building software functionality and not infrastructure, thereby easing the pain of distributed software development. Three forms of messaging have become popular and standardized atleast in definition.

• Request-Response messaging
• Topic -based messaging [Publish and Subsribe ]
• Queue based messaging [ Store and forward ]

The ESB is built directly on top of the JMS information pipe, and leverages everything that JMS has to offer.

SOA Standards - Integration framework of ESB

SOA Standards helps to enable flexibility and reduce complexity

• Well defined interfaces with business level semantics
• Standardized communication protocols
• Flexible recombination of services to enhance software flexibility

SOA drives its technical strategy and vision from the basic concept of a service

• A service is an abstraction that encapsulates a software solution
• Developers build services, use services and develop solutions that aggregate services
• Composition of services into integrated solutions is a key activity

SOA Core Elements

• Service Assembly - technology and language-independent representation of the composition of services into business solutions
• Service Conponent - technology and language-independent representation of a service that can be composed with other services

SOA is a new paradigm for software paradigm. Organizations have discovered the benefits of using both internal services (those developed and deployed within the organization) as well as external ones (those built and deployed by other organizations).

As mentioned above SOA is a new pardigm, how to integrate those core legacy applications that don't support SOA standards into a newer distributed application. For this reason, you may need to use the data transformation functionality provided by the ESB, a built-in adapter, a custom adaptor or a combination. An adapter is a relatively simple piece of software that wraps a component's interface, or transforms data, to interface with a different system.

You can write this software yourself to, in effect, convert the software's interface into a web service or to be JMS-compliant. If the application uses enterprise Java or .NET, the tools used to build the system will more than likely be able to automate this process. For other cases where a data transformation or a built-in solution is not possible, you may need to build a more complex, custom adapter. The custom code you write may need to interface with older applications such as COBOL (see Figure 11) or to transform older, proprietary, data formats, to XML

The adapter pattern is straightforward in most cases, and will solve many of your ESB integration problems. For systems that comply with the ESB interface, yet whose messages do not match the standards for the rest of the system, the ESB message transformation capabilities may be used.

ESB Message Transformation - framework to decouple sender's and receiever's data view

Many services will accept messages in the form of XML, but they may need that XML to conform to a particular XML schema. As more receiver components are plugged into an ESB-based deployment, the more potential data transformations will need to take place. Developing custom code to transform the same message to conform to a different XML schema for each client is inefficient and costly

The ESB provides built-in message transformation services to take care of this for you. ESB allows each listener component plugged into the ESB to specify a format for which it requires the dataThe ESB will perform the transformation for the listening component, without any specialized code required on the part of the sender or the receiver. The net effect is that the receiver's view of the data is decoupled from the sender's view, the overall system's view, or that of any other receiver in the system. Message normalization and transformation is a feature that will also save you a lot of development, testing, and maintenance effort. Because the ESB decouples both the message sender and receiver from the message format (and hence one another), the development of either component will not impact the other.

Normalizing message formats doesn't just benefit message sender and receiver component developers; it enables the ESB to work with the messages; such as to route then effectively across the network and to the systems that require them

ESB Message Routing - framework to seperate service lookups from the components that require the services

In the classic cases of web services or JMS, message senders and receivers would have to find one another, and invoke each others services. The problem with this approach is that every component has some sense of awareness of the other. In most cases, this awareness is hard-coded within the component in a way that makes it difficult and expensive to change and redeploy

The ESB serves as an intermediary for communication and the routing of messages between components. The ESB separates service lookups from the components that require the services, by allowing to define how messages and events can be chained, resulting abstract message flows that components can plug into at runtime dynamically

Lets take a quick look of what a message flow is ?

A service endpoint is defined as a discrete operation devoid of its implementation. Service endpoints are abstract in the sense that you do not need to worry about how they're implemented, where they reside, or how they're plugged into the ESB. You simply know they exist based on their service definition, usually in the form of Service Component Definition Language (SCDL), which is an XML-based standard for defining business logic operations in a SOA. The definition of a message flow begins by chaining together various service endpoints (and hence the components that implement those services), applying any required message transformations, along with any rules for the overall workflow. The rules of the workflow can be defined in terms of the content within the messages, or they can be tied to a generalized workflow, also known as itinerary-based routing

• Content-based Routing - The act of routing messages based on content is generally implemented as script components deployed within the ESB. The script can be code in the form of JavaScript, or it can be in the form of Business Process Execution Language (BPEL) rules. The routing can be based on message validation to ensure that applicable messages contain the data they're supposed to, or that they conform to an expected message format
• Itinerary-based Routing - Itinerary-based message routing is based on pre-defined workflows, or step-by-step processing. This type of routing is similar to a Java Servlet filter chain, where the response from one Servlet becomes the input to the next Servlet in the chain, and so on. Whereas the Servlet chain is defined in a Java EE configuration file, the itinerary-based routing workflow is defined with routing rules in the ESB

BEPL - Framework for Orchestration and Business Process Management. BPEL is an XML-based language specifically designed to describe and orchestrate asynchronous business processes. A business process is really nothing more than a piece of business logic (code) that performs some data-related task, but which is meant to be part of a larger set of processing. In other words, a business process is a relatively small data-crunching task that is meant to be chained with other business processes to perform an overall larger task. These tasks (their methods, inputs, and outputs) can be combined in meaningful ways with BPEL. Plugging component's based on SCA ( Service Component Architecture) into the ESB and controlling them with BPEL allows you to build reliable, distributed, systems with central orchestration and monitoring capabilities.

Conclusion

This article set out to define what an ESB is, what it consists of, and what problems it is designed to solve. As a result you, should have and understanding of when to use an ESB, and what parts best apply to your own problem sets. An important lesson to learn is that, in practice, the ESB itself is not an application server, a messaging server, or a single piece of software that is installed and run on a piece of hardware. Instead ESB should be thought of as a toolkit, or a framework, from which you can choose the pieces to use or ignore as you design and build distributed software systems.

There are various Commercial and Open-source ESB. Next I shall be posting on my experience with Mule- a complete open-source ESB framework. Mule has started discussions for supporting theService Component Architecture specification. (http://mule.mulesource.org/display/SCA/Home)

Apche Tuscany - Infrastructure for building SOA application - Introduction

Link : http://incubator.apache.org/tuscany

Service Oriented Architecture (SOA) is basically an evolution of distributed computing. SOA provides a modularity of business logic, which can be presented as service for clients (client as in client-server architecture). These services are loosely coupled in nature. It is an architectural approach driven by the need to overcome the challanges of tightly coupled and department specific applications.

SOA Architecture is trying to solve the problem domain of how to construct and tie to-getaher, or assemble services.

A service is a unit of code that performs some function.

• May be addressed by a client locally or remotely
• Offers a contract

Services may be assembled in hetrogenous enviroment

• May be deployed across different runtimes e.g. J2EE Server, OGSi container, .Net Application Server
• May be written in diferent languages

SOA promises benefits such as

1. Improved business agility
2. Cost Reduction
3. Easy sharing of information in hetrogenous and distributed environments

What is Apache Tuscany ?

A open source project in incubation at Apache that provides infrastucture for building service-oriented applications. Tuscany is based on specifications defined by the Open SOA Collaboration. Tuscany is based on three independent technologies designed to work well to-geather. They are not dependent on one another and can be used independently. But these technologies togeather provide a full infrastructure for developing and running SOA based applications

• Service Component Architecture (SCA) for assembling service networks. It a simple, service-based model for construction, assembly and deployment of network of services (existing and new ones) that is language-neutral.
• Service Data Objects (SDO) for representing and tracking data as it flowsacross a service network. It provides a uniform interface for handling different forms of data, including XML documents, that can exist in a network of services and provides the mechanism for tracking changes
• Data Access Service (DAS) for declarative data access. It provides a simple SDO interface to relational databases