Author: Dave Hollander
Publication Date: January 15, 2003
© Copyright 2003 Contivo, Inc. All rights reserved. The information contained herein has been obtained from sources believed to be reliable. Contivo disclaims all warranties as to the accuracy, completeness or adequacy of such information. Contivo has no liability for errors, omissions, or inadequacies in this document.
Integration is now a competitive necessity for enterprises. Integration projects that span internal applications, link critical demand- and supply-chain partners or connect customers all face the problem of making disparate systems interoperate seamlessly by exchanging business information.
The integration imperative requires strategic solutions to accommodate the pervasive change and heterogeneity common in our IT environments. While many companies strive to reduce the number of applications, the effect on integration is minimal when compared to number of sources of change and variety. Sources include legacy systems, standards, applications and middleware that are introduced or revised, as well as organizations that restructure, merge, and move through the ongoing cycle of centralization and decentralization. Adding to this are new initiatives to gain competitive advantage often by deploying new applications and creating new business functions.
A successful integration strategy must provide an IT manager the ability to:
§ Accelerate development of global, networked business processes and visibility into distributed operations.
§ Increase flexibility by minimizing the risks associated with inevitable organizational and technical change and introducing reusability, accountability and collaboration to leverage today’s effort in future projects.
§ Reduce total costs when integrating legacy applications and web services even when scaled to enterprise boundaries and beyond.
To address the imperative of effective integration, companies explore a variety of integration strategies including deploying middleware, managing meta-data, and adopting business messaging standards. Unfortunately all too often, when faced with staffing changes and the need to run parallel, distributed, multi-department and inter-company integration projects, these strategies do not successfully scale and achieve the desired business results.
Integration using EAI middleware, Application Servers, B2B hubs or Service-Oriented-Architectures such as Web services deliver rapid connectivity but in doing so actually enlarge the problem. Meta-data management can improve the efficiencies of managing an application but seldom encompass the full breath of integration. And business standards implemented as messaging formats or “canonicals” are difficult to implement and keep current in the dynamic environment our businesses operate in.
While middleware, meta-data management, and messaging standards all have crucial roles in a successful enterprise integration strategy, deploying in a traditional approach they are insufficient to overcome the challenges presented by manual integration design. Traditional integration implementation methods are manual, labor intensive, and time-consuming processes. Traditionally integrated systems are expensive to maintain because a significant portion of the labor involved is dedicated to managing and mapping detailed relationships between the interchanged data. As the number of interconnected systems increases, manual integration methods result in implementation costs that can soar beyond five times the original investment in integration software.
“Manual integration is among the top problems facing major enterprises; there is simply no practical, cost-effective way to solve it manually.” – Jon Derome, Yankee Group
The goal of a successful integration strategy must be to reduce the incremental costs associated with every system that is integrated. As the number of systems interconnected goes up not only must costs and time-to-deploy go down, but it must result in increased flexibility to address heterogeneity and pervasive change. Traditional integration fails to meet these goals.
To meet and conquer the challenges of enterprise-scale integration, integration strategies must accommodate a variety of middleware, enable meta-data management capabilities that span multiple technologies and provide flexibility in applying business standards. Four key design factors in such a strategy are:
§ Dynamic –responsive to message iteration, application versioning, changes in your information chains, and your partners’ and your own organizational changes.
§ Semantic – focused on the meanings associated with data in documents, not representation.
§ Canonical – expresses business judgment of which differences in business information are important and which are not.
§ Model – an abstraction representation of an object, process, system, or information used to improve communications between all stakeholders, improve design quality, and reduce development and maintenance costs.
Strategies that successfully manage these four aspects of integration design will provide the IT manager with the ability to accelerate development, increase flexibility, and reduce total costs. In short, these are the keys to designing a strategy that encourages and engages business process owners in the quick and affordable development of networked processes.
Traditional integration design relies on creating static specifications of systems and interchanged information. Integration strategies that use messaging such as middleware, brokers, hubs or newer Service Oriented Architectures (SOA) such as Web Services rely on precise, rigid specifications of the interchanged messages. Interoperability is achieved by writing code based on these static specifications.
The challenge to message-based approaches is keeping up with the change that is ever-present in the business environment. To accommodate change, messaging architectures have historically introduced optional elements into their message formats. While “optionality” can provide some flexibility and improve acceptance, it raises implementation costs and reduces interoperability. Many messaging veterans, such as Health Level Seven’s (HL7) Modeling and Methodology Committee, find that optionality creates more problems than it solves.
“However, the wide uses of optionality, while helpful in gaining consensus and in writing a somewhat more compact specification, imposes significant costs on interface implementers. In fact, optionality, as a “feature” of HL7 Version 2.n, is associated with most of the issues that Version 3 is trying to solve. Optionality makes it harder to precisely define the semantics of a specific message and makes it virtually impossible to generate conformance claims for messaging.” – Message Development Framework, Version 3.3, HL7, p15
To accommodate the dual needs of keeping up with the dynamics while providing stable, static specifications, an integration solution should have the ability to separate message specifications into levels.
§ Abstract: Dynamically managed models
§ Concrete: Static interchange formats
§ Physical: Static code that executes
Abstract models describe the terms and concepts that must be exchanged to achieve a business result without the extra detail about how the data is stored in a message. Because they are less precise than the other two levels, abstract models are easier to change and can be kept up to date. Change should first be reflected in the abstract modeling layer to take advantage of quicker, lower cost processes that are focused on describing the impact of the change and assuring stakeholders consensus.
Concrete formats represent the full detail of the messages and system interfaces being integrated. Concrete formats describe contents and storage requirements all of the data to be exchanged. These formats are often expressed as COBOL Copybooks, database schemas, XML Schemas or Document Type Definitions (DTDs) or UML. These formats must be rigid and precise since they represent the specifications that programmers use to code behavior into their systems.
The physical layer in an integration environment is the actual runtime infrastructure. Runtime systems locate data in messages and initiate system behaviors based on the data. Traditionally, software programs must be written to locate the information as well as to initiate, perform, and monitor the appropriate behaviors. The trend toward meta-data driven systems, such as those implemented using XML Schema, provide some flexibility to this layer by enabling limited variations in the actual runtime messages and interfaces. However, the flexibility is limited to the ability of the meta-data standards to describe where data is located and the intended changes in system behavior.
Finally, to make the three-layer strategy work there must be a rigorous and repeatable refining and distilling process for moving a project through the three layers. Once abstract models are agreed upon, they must be able to be refined to add addition specification data about how the data is recorded in the message. Completed concrete formats then need to be mapped to identify how the data is to be transformed as it is exchanged and code that performs that transformation needs to be generated.
Augmenting traditional message design architectures with a third tier for abstract models can make the design environment dynamic enough to meet the strategic integration goals. To achieve this goal, it is not enough to just introduce a third tier—it must be designed to responsive to the intent of the exchanged information and it must enable meaningful and manageable business standards.
Meta-data management often focuses on the concrete formats for the data to be exchanged, stored, or processed. However, the meta-data (the data that describes the actual data) is seldom semantic—that is, it is seldom abstract enough to provide the scalability and adaptability needed to meet integration goals.
Traditional integration design seldom deals directly with semantics: the meaning of information. Semantics are the domain of human programmers and mappers who use their experience and specifications to infer the semantics of messages and interfaces. Unfortunately, while this traditional treatment leverages the strengths of humans to understand the nuances of semantics; it can not scale when faced with staffing changes nor does it meet the demands integration projects that span department or company boundaries.
The conventional attempt to overcome the semantic scaling problem is to initiate an effort to design or adopt a single, uniform dictionary of terms and messages. These projects become overwhelmed when they try to enforce these messages on all systems and partners involved or even try to promote the dictionary and messages as an industry level standard. Abstract models based on semantics can avoid this potentially devastating distraction.
A closer look at semantics will help illustrate how semantics benefit integration strategies. For example, consider a large multinational company’s policy that describes issuing, reclaiming and destroying keys. The policy may read something like: “Keys, notched and grooved, metal implements that are turned to open or close a lock, shall be…”.
The physical, not semantic, definition of “key” works fine until new technology is introduced. With the introduction of electronic keys, the policies need to be amended. “Keys, notched and grooved, metal implements that are turned or small devices that use a transmitter to open or close a lock.”
Policies generalized to use a semantic definition, “Devices used to open or close a lock”, do not have to be revised with every technology change. Concentration on intent enables innovation and insulates agreements about information from the influence of organizational change, technology and even cultural differences such as the differences between American and European conventions for representing number (1,000,000.00 vs. 1.000.000,00).
In message-based systems, semantics describe what information is interchanged, not how. The semantic of “ship-date” remains the same regardless if it is represented as 15-07-02 or Jul 15, 02. But determining whether “ship-date” means the date a shipment is 1) on the dock; 2) available to ship; or 3) loaded on the truck; is best addressed at the abstract model level. The model level can used to resolve what information is logically the same from a business perspective.
To manage semantics in the abstract model layer, an integration modeling solution would allow integration architects to identify semantic concepts like “ship-date” and group them into categories. Interfaces representing data messages are modeled to associated concepts to specific data fields. For example the data fields: Street, ADDRLINE, STRAS from xCBL, OAGIS and SAP IDocs respectively are all semantically equivalent and associated with the address1 concept in the address category and are grouped together.
To fully deliver on this abstract modeling process, the models must be able to be refined to add detail about how the data is represented. These details, used in the lower tiers, includes rules for transforming items that are logically the same but physically represented differently, and rules for locating the information in the message. To enable semantic decisions to automatically guide the deployment of physical runtime processes, rules must be defined such that they can be deployed on variety of platforms and systems as they are in the Contivo Enterprise Integration Modeling solution.
Semantic level agreements are essential for an integration strategy to successfully span the entire organization. If semantics are implemented differently at different points in a networked business process, then additional processes must be implemented to resolve differences. The development of a shared semantic model enables deeper understanding of the data within transactions and reduces the likelihood of not recognizing that two or more data items are related and represent the same business value.
The goal of achieving complex, global distributed business operations and visibility requires an effective strategy for managing semantics. Clear understanding of the intent of data within messages allows enterprises to develop networked business processes that have more direct impact on operations and with reduced duplicated data. Semantically enriched integration strategies assure faster implementation, less redundant data, and make it easier to keep current with changing business requirements.
In tradition integration strategies, business standards are sometimes implemented as static messaging formats commonly referred to as “canonicals”. Canonicals establish a shared, common view of business information by eliminating differences the business deems insignificant in enterprise data resources.
Differences exist because each disparate system and various standards have unique perspectives regarding what information is important and how to represent it. Dates, for example, not only vary in how they are represented (July 27, 2001 or 07 Jul 01) but also what they mean (ship-date or ready-to-ship-date). Canonicals represent both the preferred representation and the business judgment as to which data elements are unique and those that are equivalent. Eliminating differences can ease integrating systems and enable limited message and interface reusability.
Typically, canonicals are defined in the concrete tier where they describe all of the details about what information is to be included and how the information is to be structured. XML DTDs or Schemas are increasing being used to define canonical formats. Deployed successfully, canonical formats enable:
§ Reusable business events. Universally reusable messages exchanged between any set of applications.
§ Independence of action. Canonicals establish the technical requirements for achieving enterprise goals that project teams can use for planning and deploying integrations with a minimum of inter-team coordination.
While canonical formats are beneficial, for an integration strategy to be scalable it needs to be enhanced to overcome their drawbacks, which include:
Lots of models and formats. Because canonical formats are
static, they must be designed prior to integrating. However, once deployed,
canonicals are difficult to change because each and every system using the
canonical event will need to change. In practice, large-scale integration
environments use multiple versions and variations of canonicals at any given
Additionally, canonicals are just a few of the many formats used to describe and design interoperability between business systems. Successful canonical integration strategies must relate all of the models and formats and provide facilities to coordinate their development.
§ Performance. Few systems will use the canonical format as its native data format. To exchange data using canonicals, the source system will have to transform its data into the canonical and the receiving system will also have to transform the data. The performance implications of this additional transformation requirement can be too expensive. Business processes that must sustain very high throughput cannot afford the transformation overhead.
Traditionally canonicals do not use abstract semantic definitions enabling them to keep pace with our dynamic businesses environment. Canonicals, as commonly used today, are important but insufficient to assure that an integration strategy is scalable.
“Like other IT reuse strategies, interface and message reuse is hard to achieve” – R. Schulte; Tactical Guidelines, TG-15-1933; Gartner, Inc.
The solution lies in applying canonicals to the abstract model tier. Abstract, semantically defined canonical models establish a shared view of what information is necessary to be exchanged in a business process. Because abstract canonical models only deal with semantics and not formatting details they are faster and easier to develop and maintain than static, standard dictionaries and messages which must be implemented at both the concrete and physical level to achieve the desired results,
Abstract canonical model acts as an enterprise vocabulary from which multiple concrete formats can be created and maintained. When deployed in a system that manages the relationships between the abstract, concrete and physical layers, a single abstract semantic model can deliver reuse even with differing canonical formats and a variety of physical systems.
Modeling has been successfully applied to engineering design activities such as architecture, semiconductors, electronic circuits, and software. Yet modeling has not been broadly and successfully applied to integration.
Fundamentally, the success of modeling is due to its ability to identify significant issues and constraints and present these in multiple representations and at various levels of detail. This enables all stakeholders to collaborate in the design process and to reach consensus. It also makes it easier for designers to divide a problem in to smaller more manageable pieces.
For integration modeling to deliver the same benefits, it must be adapted to identify the issues and constraints that are important to integration. The most fundamental issue is semantic identity and equivalence and the representational details associated with the semantics.
To be scaleable across an enterprise and beyond, a successful integration design strategy must be able to use abstract semantic models as the organization responds to the forces of change around it. The modeling technique must also be able to refine the semantic model into static concrete formats that are used by system developers to implement system behavior and to further distill the formats into runtime code compatible with the physical systems it will execute on. Without processes for refining and distilling the models will go out of date and gather dust.
Integration design strategies that can leverage the benefits of modeling to semantically managing the integration data in multiple formats across multiple platforms will:
§ Encourage the development of networked processes and visibility with the flexibility to respond to change.
§ Involve all the stakeholders and encourages meaningful, informed communication and collaboration
§ Lead to integration projects that are implemented quickly and affordably.
In summary, dynamic, semantic, canonical models are keys to making your integration strategy deliver a competitive advantage.