Enterprise Integration Patterns

Integration, like other design activities, can benefit from sharing ideas and proven strategies in the form of patterns. I highly recommend Gregor Hohpe and Bobby Woolf's Enterprise Integration Patterns website and book.

In my recent work I've discovered a few potential patterns of my own, which are documented here. The pictures use the pattern language developed in the Hohpe/Woolf book. If you want to be informed when I add new material, please subscribe to my RSS feed.

Suppose you have a legacy system with an EAI adaptor, or a similar source of messages, which reports on the current state of key business objects. The messages will typically tell you when something has changed, but not necessarily what has changed. It's common for such a message source just to report on the latest state of the objects. However, many changes may not be relevant to downstream systems - to properly filter the message stream they need to understand the nature of the change.

How can we understand the nature of each change, when the source system can't provide this information?

If the source system can't provide the information, then can we derive it? Yes. If we've seen a previous message describing the state of the same object, we can compare the state now with the state reported in that message.

Therefore the obvious solution is to add this change information using the EAI system itself. The solution will need a way of storing and retrieving messages, a way of comparing them with one another, and a way of adding the "change" information to the message. Fortunately these are all things we know how to do.

Use a combination of a message store, a message comparator and a content enricher to track changes between the latest message and the previous one, and add this change information to the outgoing message before it is published to other systems.

The message retrieval step needs to identify the contents of the message and retrieve the last  message for that business object from the message store. For example, if it is a "customer's details changed" message, this will need to extract the customer ID from the message, and find the previous message indexed by that customer ID. Ideally the required indexing information will be held in the message title or envelope, but otherwise this function will have to unpack and parse some of the message contents.

The form of message storage will depend on the infrastructure set up for the messaging system, and the number of business objects which must be tracked. If that number is small, it might be enough to store the messages in files named using the object ID. If larger, then a proper database may be appropriate. In either case we only need to store the latest message for each business object, so the required storage does not grow indefinitely.

The "changed items" information is added to the message by a "content enrichment" step. Assuming that the message is formatted in XML, this can take the form of a new element attached to the message header, which should be entirely transparent to message consumers not looking for it. If the messaging system uses a Common Message Model then the changed items node can be added to the common part of the structure as per the following model:

The change information can be as simple as a list of changed elements, or can be enhanced to include before and after values, and to identify moves, additions and deletions separately from simple value changes. If certain changes have specific business significance, they can be detected and highlighted at this step.

The message comparison step may be a bit of a challenge. Depending on the message format it may be as simple as a line by line comparison, or it may be more complex. For example, if two XML messages have the same content in the same element, but the position of the element has changed then this may not be significant. Fortunately the common platforms provide tools which make this task easier. For example in the Microsoft environment you can use the XMLDiff library which supports intelligent, fast, XML comparison where the programmer can easily choose which differences are significant and which should be ignored.

The message store can either be populated progressively by each change as it happens. This will result in one "everything's changed" message for each business object. Alternatively it can be primed by running some process in the legacy system to "touch" each appropriate business object in turn.

This design has several strengths:

• Transparency - the design has no impact on the legacy system and its adapter, or any existing message consumers,
• It can detect any required granularity of change down to individual elements and attributes,
• Incorporation of the changed items information into the message makes it easy to add appropriate filtering code into recipient message processors,
• It's easily extended to as many business objects as you want, without significant change to the core code, reducing the ongoing maintenance cost,
• It doesn't require you to know in advance what changes may be significant.