Service-oriented architecture (SOA) has become the single most important theme in software engineering. Clearly, the proliferation and unanimous acceptance of Web services, together with a new wave of case-like IDEs that support the development of SOA-based solutions, make SOA the preferred blueprint for building enterprise-wide distributed applications. At the same time, business process management (BPM) is making a strong comeback as a key enabler for modeling and operating the new agile enterprise. Infrastructure vendors have made BPM a key component of their product stack offerings, niche vendors provide vertical business solutions using proprietary BPM systems, and pure-play BPM vendors are gaining wider acceptance.

Although there are indications of the emergence of both trends, no clear and prevailing thought exists about the convergence of the two trends. Are they complementary notations, do they overlap, how do I use them together, and if so is there any additional benefit? Furthermore, why is the third wave of BPM poised to succeed when the BP reengineering of the late '80s failed?

In this three-article series, I will address these questions. First, I discuss how a best-practices architectural blueprint merges service-oriented architecture with a BPM framework in order to provide repeatable ways for modeling and building robust, enterprise-wide integration solutions. I describe why today, more than ever, any enterprise that uses technology to support its mission statement needs to have an architectural blueprint in place. And finally, I discuss what the challenges of doing business in real time are and how a BPM approach can be the key enabler for organizational agility, intelligent enterprise modeling, systems development, and customer-centric operational excellence.

In the second article, I will apply BPM techniques to model and architect a software solution that supports an "apply for car insurance" business scenario. I will cover two designs: a pure BPM and a hybrid. I will also cover some of the emerging modeling tools and standards, and discuss some of the challenges around modeling and various architectural choices and strategies.

In the third and final article, I will use BEA's 8.1 platform to build a running proof of concept. I will discuss the new visual programming paradigm that BEA's IDE introduces, its strengths and weaknesses, and some of the techniques required to build fully distributed, industrial-strength applications. I will also explain why the popular request/reply WEB protocol is at odds with event-based process modeling and its implications in making architectural decisions.

Architectural Patterns - Who Needs Them?
Is software engineering art or science? In science we have unambiguous definitions, theorems, and proofs. In the arts we have tools and techniques, trends, and best practices. In science, postulates are set forward and some become theorems, some are validated after centuries of research, and some may never be answered. In the arts, new technologies bring new trends, like the press and digital photography. If software engineering is a science, it should not be a difficult thing to define the terms we all use in our day-to-day business language, like service, Web service, service-oriented architecture, BPM, and BPM systems (BPMS). Surely I can prove with mathematical rigor the right execution plan in a database query algorithm. But can I answer in a crisp, concise, and universally accepted manner that B2B integration in J2EE is the right answer as opposed to an all .NET Web services solution? I understand that to some of us this is not even a question. Finally, a random survey of any of the usual trade publications points out that everybody is making their own definitions and some are even questioning the very essence of the existence of IT. I have to conclude that software engineering is still more of an art than science. That is exactly why we need best practices, frameworks, and repeatable processes in place.

Patterns encapsulate best practices, define a domain problem in a concise way, describe the forces that make the problem worth identifying, and propose a solution. Patterns do not solve unique problems. Practitioners combine patterns to solve more complex and some times unique problems. Christopher Alexander says, "the pattern is at the same time a thing, which happens in the world and the rule which tells us how to create that thing, and when we must create it. It is both a process and a thing." I recall my all-time favorite definition: an object is a data structure with an attitude or data and behavior. For now, Web services can be viewed as objects with one method. Conversational Web services, as implemented by the BEA WebLogic Platform 8.1, look more like real objects: instantiate it once and keep executing methods. In case you are still not sure that Web services are coarse-grained objects, consider this: (1) IBM, BEA, and Microsoft announced the WS-Eventing specification. It's just like the good old observer pattern for objects. (2) The Open Grid Services Architecture implements Web services' interface inheritance for the grid services protocol. Therefore, Web services provide data and behavior (the thing and the rule in Alexander's definition), and BPMS implements the process component of the pattern. SOA is an architectural pattern addressing enterprise integration and systems development.

SOA is not a revolution but rather an evolution of architectural trends we have seen developing for some time. It evolves around building distributed systems for the enterprise. Web services provide the underlying technology for solving the systems connectivity challenge, admittedly, in a universally accepted and unambiguous way. Perhaps for the first time Web services successfully solves the interoperability challenge, something that CORBA, COM, DCOM, and RPC could never have dreamed of. I am sure XML, acting as the neutral ground, has a lot to do with it. However, the inclusion of the BPMS framework within SOA is a new and revolutionary element. The third wave, described by Howard Smith and Peter Fingar, is a revolutionary new set of concepts, frameworks, and mainstream products. It is dramatically changing the way enterprises are transformed to agile managed and run global and collaborative e-commerce entities.

Business process management has been around for some time, and more so in industries that are not IT related. Concurrent engineering and six sigma were developed to address on-time collaboration in manufacturing and process improvement, and did have quite a success. However, in the late '80s business process reengineering management had rather limited success for many reasons. But the fundamental reason was that reengineering was a paper exercise. No software was around to support such a complex undertaking. BPM engineered the adaptive enterprise without regard to IT systems. As David Taylor writes:

"The demand for continuous process optimization requires a radical rethinking of how information systems are designed and constructed. It is no longer sufficient to produce fixed solutions to fixed problems."

Information systems, like the business models they support, must be adaptive in nature.

Taylor proposed convergent engineering, an OO-based development technique, as a way to develop the adaptive IT. However, OOP could not successfully address distributed computing and enterprise integration. In addition, business analysts, responsible for modeling the enterprise, did not speak OO.

BPMS establishes the process as the unifying construct for modeling, software design, and runtime execution. In the past, development trends have influenced the way we model the enterprise. Functional programming brought functional requirements techniques into vogue. Relational databases brought the proliferation of RDBS analysis and design. Object-oriented programming paved the way for OO analysis and use-case development. But in most cases, business analysts do not speak the development lingo, thus creating the need for another translation with the usual impact in requirements traceability.

BPM specifications are evolving rapidly into standards. Already, there are products in the market place that support process modeling, optimizations, and runtime execution. The BPMS process-centric approach to the systems development life cycle, as implemented by BEA's WebLogic Platform 8.1 and other BPMS products, eliminates the business requirements to runtime impedance mismatch.

Agile enterprise is about adaptive business and adaptive IT systems. If one challenge is new in building enterprise solutions, it is the rate at which requirements change. It is faster than ever. BPMS engines add a new layer in the traditional development stack (see Figure 1) and bring quality of services addressing fundamental challenges in enterprise integration. BPMS engines facilitate the soft wiring of the most volatile parts of the programming: the integration points. Soft wiring is explicitly described in a formal language and executed by BPMS engines (a.k.a., finite state machine engines). Business and IT resources together can review and modify processes in a visual intelligent IDE, as implemented by BEA WebLogic Integrator and other BPMS products. Deployment to runtime BPMS execution engines is one click away. Business simulation can run and performance engineering can be done before completion of the systems; it sounds just like case tools and in a way it is. SOA and BPMS tools bring real-time executive dashboards of the agile enterprise to the mainstream.

In the rest of this article, I'll describe the development of a typical financial services enterprise and propose a migration path to a BPMS-based SOA. The path is incremental but it does require strategic thinking and commitment to the future vision. In return, it will allow early return on investment and transform the legacy to a fully adaptive agile enterprise.

From Enterprise Vision to Organizational Silos
Enterprises start with a vision. CEOs and boards of directors take the vision and craft mission statements. C-level managers define policies and put processes in places to manage execution (see Figure 1). Functional roles and responsibilities are defined and organizational boundaries are created. Lines of business (LOB) can be horizontal or vertical in nature (see Figure 2). Vertical LOBs have the following characteristics:

• Stand-alone operational domains
• Own management and policies.
• Develop and maintain own IT - islands of automation
• Are large enough to create multiple lines of business; for example, mortgage-backed securities, munies, money markets, etc.)

• Provide business controls
• Regulatory governance and compliance
• Require access to data managed by vertical LOBs
• Manual processes and paper reports in place

There Are Processes Everywhere.
Can You See Them?
Processes can be customer facing or internal to the organization, or can be part of a larger process. We find internal processes within the same organization. Processes usually involve interaction among people and systems or just interaction among systems (see Figure 3). Trade processing is a good example of a large-scale process. A trader at the front office receives in his sales order system a trade execution order from a hedge fund manager, or he receives a fax or a phone call. The trader checks the inventory system for securities or funds and executes the trade with his counter-party. A paper ticket may be produced and a trading assistant may have to enter it in the downstream system.

Another internal process starts when a help desk analyst receives an exception report because one of the downstream systems made a wrong re-entry. He then looks up the data in one of the internal blotters (record of original entry), requests a fax from the back office (we assume the trade in question is past settlement date), and perhaps he repeats the same activity again because he hasn't received an answer in two days. The process eventually terminates when the analyst resolves the issue, unless of course he is transferred to another department or outside the company. Then consultants have to come in and trace the problems and processes, usually for a lot of money.

Monthly customer statements are a classical example of a periodic enterprise-wide process, usually owned by a horizontal LOB. In most cases, customers have accounts in products that are supported by different LOBs, for example, equities, U.S. bonds, and foreign currency. It would be rather confusing to send multiple statements at the end of the month. Legal and compliance issues also require cross-referencing multi-silo data. A major challenge of the Patriot and Sarbanes-Oxley Acts (a new business process, albeit not money making) is accessing data owned by a number of LOBs, sometimes halfway around the world. EAI techniques and messaging attempted to address these issues with the limitations explained earlier.

The Way to Agility: BPM-Centric SOA
Let's consider how a BPM-centric SOA with Web services can transform the existing legacy enterprise into an adaptive enterprise. Horizontal processes and exception management are perfect candidates for SOA enablement and can demonstrate a sizable and quick ROI. Without going through the rigors of a business process reengineering, we have to define a good process map. A process map is also the first major step in a business process redesign exercise. Using BPMS design tools (Proactivity, Intalio, Interfacing Technologies), you can associate metrics to processes and activities, for example, performance, cost, IT resources, FTEs, elapsed time, volumes, and so on. Many BPM design tools allow you to run simulations and continue with process optimization (run what-if scenarios) but that's not the focus of this article. For our focus, here are some characteristics of a good process map:

The next objective is to define small units of work by decomposing the activities. We call these Elementary Business Services (EBS). If you recall from multidimensional vector algebra, any point in space can be defined as a linear combination of the unit vectors. In our case, we define the universe of all EBS in a way that any process can be constructed by orchestrating a subset of EBS. As you might have guessed, we implement EBS as Web services. Identifying the right set of EBS and level of granularity is important. It is as important as it is designing objects. No surprise that the same rules and techniques apply: encapsulation, state dependency, cohesion, loose coupling, and refactoring.

The portfolio of EBS delivers a number of real benefits:

• It is the ultimate guide to reuse. EBS can be orchestrated in any imaginable way to form a new LOB.
• Continuous process improvement does not have to wait for IT to adapt to the new business model.
• EBS are available to the enterprise and business partners within the business ecosystem.
• Retiring a system does not have to be a disruptive process but an incremental one.
• Merger and acquisitions of IT can be achieved in a manageable cost-effective way.
• A new business process can be designed and executed in near real time.

Many financial services institutions have a horizontal line of business that manages high net worth private clients. In a BPMS SOA-enabled enterprise, developing the IT infrastructure to support such a new LOB can be completed literally in parallel while the business model is put in place (see Figure 5).

Consider the Amazon.com phenomenon, they did not invent any new EBS. All the EBS were in place in any other mail order catalog book store: order book(s), check inventory, charge credit card, print statements, prepare shipment, mail to customer. But it did invent a new process and challenged the establishment without even breaking a sweat.

As Howard Smith and Peter Fingar said: "In the third wave of BPM, stovepipe thinking and point-to-point technical integration give way to flexible, business process-based architectures." Furthermore, Gartner Group is now saying that companies that continue to hard-wire business logic into software or middleware or insist on manual steps will lose out to competitors that deploy process management architectures.

Doing Business in Real Time
It's hard to predict the future, to say the least, but we are very scientific about it and try to do it all the time, right or wrong. The science of statistics and forecasting is about predicting the future. Portfolio valuation and actuarial study is about projections. In reality, our predictions are based only on past performance and trends we've experienced. Doing business in real time requires anticipation of future business conditions. However, the fundamental business protocols and frameworks have to be in place. Today, technology innovation, BPMS, and SOA are the foundation that aligns business objectives with IT. Processes provide a new layer that encapsulates the change. In the same way that PowerBuilder and VB-like tools proliferated the development of client/server and relational database systems in the early '90s, BPMS engines will establish the future process-driven enterprise. As a matter of fact, I predict that in our life span, we will see a demand for runtime processes to change in real time or requirements for self-modifying processes. Clearly, humans want be able to be in charge of that change but BMPS engines will be able to facilitate it, using UDDI-š (š for process) to find the best possible service contract and make decisions using rules that describe domain expertise and market conditions. With the proliferation of BPMS, agility will give way to extreme adaptation.

Summary
In this article I presented a blueprint for merging SOA and BPM. Starting with a top-down process map of the enterprise we defined the portfolio of elementary business services. Vertical LOBs own and deploy the EBS. Web services implement and make them available to the enterprise. Using an instance of a BPMS engine, a new business process can be designed, developed, tested, and add business value in days, by combining existing EBS.

In my next article I will: (1) cover BPM techniques for modeling a real-world business insurance process, and propose a pure BPM and a hybrid solution; (2) design EBS and implement those using Web services and JMS connectivity; (3) propose a physical architecture using WebLogic Platform 8.1; and (4) discuss BPMS challenges and emerging patterns within a service-oriented architecture.

Until then: processes are everywhere. Can you see them?

References