3.6 - ESB Fundamentals

Editor's Notes

• #3: In this session we will be taking a look at the fundamentals of an ESB and hopefully provide you with a better understanding of not only what an ESB is, but also the problems that it solves for an organization, as well as the benefits that it provides over other integration strategies.
• #4: To facilitate our goals we will first define an Enterprise Service Bus, look at and distinguish between different Enterprise Integration Topologies, explore the various ways that an organization can use an ESB and leverage the features that one provides to help in the enterprise integration architecture.
• #5: This session will be broken down into four main topics, to help you better understand ESBs as a whole as well as the role that they play within an organization. What is an ESB? Why use an ESB? What problems does an ESB solver? What are the common features of an ESB?
• #6: An Enterprise Service Bus (ESB) is Enterprise software middleware that provides fundamental services in support of more complex application topologies. By placing a communications bus between Enterprise systems, and enabling those systems to talk to the bus rather than each other, an ESB decouples systems from each other. This in turn allows them to communicate without knowledge of, or dependency on, one another. An ESB is an alternative to point to point (P2P) integrations, which becomes brittle and hard to manage over time.
• #7: An ESB is an operating environment for services Provides the architecture needed for deep adoption of services Services need to be managed and cooperatively combined in order to be useful Centralized management of decentralized systems Separation of business logic from communication details Common Messaging Fabric Programs connect to a common messaging infrastructure Publish-subscribe messaging over distributed middleware Infrastructure services handle routing, security, management Most ESBs do not have the concept of an ‘event’ Neuron adds this important business concept to the bus and provides a centralized catalog of these events Enables: Microsoft developers to build EDA’s Simplifies Integration between BPM solutions Business people talk in events When an opportunity closes … do this, etc. Systems become more decoupled and there is a reduction in impedance mismatch with the business
• #8: Most organizations want to increase agility by reducing the time to market for new initiatives. By eliminating the need to integrate disparate systems, through a complex point-to-point architecture, allowing for the automation of process requirements which might incorporate various systems and users and by extending the life of legacy systems, while allowing for the incremental replacement and deprecation of those systems over time, an ESB helps promote this objective by allowing the organization to implement a simple, well defined and pluggable framework which scales to meet their needs.
• #9: There are a number of different Enterprise Applications and pieces of Infrastructure that organizations use to manage their data. In the beginning, when there was only one application that concerned itself with the data stored in these areas, point to point made sense. However as more and more applications get added over time, with out realizing it, organizations have created for themselves a spider web of point-to-point integrations that is incredibly complex and hard to maintain.
• #10: An ESB helps organizations resolve the challenges that we discussed earlier by providing them a framework for easily and quickly integrating systems together. By facilitating the distribution of information across enterprise systems, while allowing an organization to mask the differences between these systems (such as platforms, network protocols, data formats and software architectures), the ESB ensures delivery of information between the systems in an enterprise, even when they go offline. Along with this, an ESB also gives an organization with enhanced alerting and trouble shooting. By providing advanced logging to detect issues in the transmission of data, the ability to re-route information if an error occurs during delivery and the capability to enrich the information that is presented when an error occurs, all works towards making problem resolution fast and easy.
• #11: An ESB helps organizations solve all these previously discussed problems by providing a framework that facilitates communication between disparate systems within their enterprise architecture. Using a distributed hosting model for complex coding patterns, an ESB helps an organization reduce the time needed to develop and deploy new, or improvements to existing, functionality within the enterprise. This distributed hosting model also allows an organization to quickly and easily reuse coding patterns without having to port code across applications or platforms.
• #12: But don’t web services solve these issues? Web services do help in resolving some of these issues. They provide a way for systems to exchange information, and provide a consistent metadata story for that exchange. However they still suffer from a number of limitation and problems. An organization must still account for incompatible wire or data formats, as well as the different protocols each system might use. While there are standards and guidelines out there for the proper implementation of web services, there is still a very low adoption rate of these standards. Many organizations create what works right now, not what follows the guidelines and standards exactly. There is no true processing environment for web services before the data is sent out or once the data is received. Code has to be generated to handle the processing of the data sent across the service and is typically limited to one single application, so reusability is limited. While two .NET applications may be able to make us of the same code for processing data from a service, a Java application or even a database would need it’s own code to facilitate the handling of this data. Tight coupling still remains as each application sending data to the service, or receiving data from the service still has to have an awareness of the other end of the transaction.
• #13: While point to point is ok on a small scale, organizations with more complex architectures started to look for an alternative to make maintaining and managing integration much easier. Eventually the hub and spoke model, pioneered by Delta Airlines in 1955, came into play and made use of a central hub to facilitate the transmission of data throughout an enterprise. While this pattern is better than point-to-point communications, as it reduces the number of routes needed for information to reach the destination, it still had it’s own share of limitations and concerns. The ability to process information is limited by the resources available to the hub. The hub serves as a bottleneck as the ability to process and transmit data is entirely dependent on the hub’s capacity to process the information Changes at the hub can causes unexpected issues throughout the entire network Thus the concept of the ESB was born, to improve on what Hub and spoke attempted to accomplish.
• #14: As we discussed, point-to-point results in a number of problem for an organization. The complexity of maintain such a system, coupled with the lack of agility, scalability and extensibility make it a less than desirable architecture for an organization to use on a wide scale. When there are only 2 systems it seems like a manageable problem as that results in only 1 connection to be maintained Then you add another system Then another… And another… The exponential increase that occurs for each system added, means that just 10 systems results in 45 connections n(n-1) / 2
• #15: However with bus architecture you can alleviate the problems that point-to-point suffers from. Creating a loosely coupled enterprise makes maintenance much easier and reduces the development costs when re-engineering or introducing new systems into that enterprise. It also adds large gains when an organization looks at the future scalability and extensibility of their existing enterprise.
• #16: Every ESB comes with a range of features to help in the transmission of data across an organization. However not all ESBs have all these features. Some ESBs, like Neuron have evolved into full blown integration brokers
• #17: To get a better understanding of service orchestration, let’s think about an example such as a bank loan. A loan broker wants to make a loan request on behalf of a customer and uses an automated Loan Request Service. The broker accesses the Loan Request Service in the enterprise system to make the initial loan request, which is sent to an orchestrator (conductor) that then calls and invokes other services in the enterprise, partner systems and/or the cloud to process that request. The sub-services involved in the loan request might include A credit service to obtain credit scores for the customer from the various credit agencies A lenders service, which retrieves a list of available lenders for the loan A quote service to request quote from the lenders A service the process the quote with data from the application for the customer Together, the orchestrated services comprise the Loan Request Service, which then returns a list of quotes from potential lenders to the broker who made the original request.
• #18: Using our previous example of the bank loan. If the Loan Request service provides data in the form of an XML, but the Lenders service is REST based an accepts information only in the form of JSON, message transformation will be necessary in order to facilitate the communication as the data formats are not compatible with one another.
• #19: Using aspects of a message coming into the ESB, whether it be data pulled directly from the message, or aspects of the message such as headers or properties, we can route that message to different subscribers, or to all the subscribers, based on the business requirements for the organization.
• #20: Sometimes organizations need to incrementally deploy upgrades to a service. This can result in two versions of the same service being exposed, but only certain applications in the enterprise being able to make use of the new version of the service. Using service mediation an ESB enables the organization to make determinations of which client is routed to which service without having to address this concern from the client side.
• #21: Knowing what is going on within your organization is vital for ensuring the successful delivery and processing of data. If something goes wrong, an error occurs or a service goes down, being able to find that out as soon as it happens so that it can be addressed can make all the difference to both the organization and it’s customers. Real time monitoring of endpoints, as well as message tracking and failure reporting is essential in any well designed ESB.
• #22: An ESB should not only account for the obvious aspects of data transmission within an organization, but also account for those requirements not covered by a function specification. A well designed ESB will concern itself with those criteria one would use to judge the operation of a system (such as Performance, scalability, maintainability and usability) rather than just specific behaviors.
• #23: While many ESBs provide a way to perform processing on data being transmitted between endpoints, a workflow engine takes this concept one step farther. It provides the organization with a way to facilitate the flow of information between systems, while implementing persistence, tracking, fault tolerance and enhanced monitoring to the business process that are running, for the lifetime of the workflow. This allows for organizations with long running processes to track the process while it is running as well as resume the process from the last persistence point should an error occur.
• #24: While point to point integration is ok on a small scale, such as one application to another, larger scale enterprise architectures would benefit from using an ESB as it helps to facilitate the integration between the various systems in that enterprise. By providing an organization with a way to transmit data between systems, without having to tightly-couple the systems together, an ESB provides an organization with increased flexibility, maintainability and scalability across it’s enterprise architecture. Though hub and spoke is not the ideal way to handle the transmission of data between systems in an organization, a good ESB will understand the use for it in certain scenarios, and provide support for implementing in the right conditions. A good ESB will provide an organization with a full suite of features, not just one or two.