A fiction to be automated & the introduction of an Orchestrator in the Business Process

Buna Ziua,

I could finally correct the problems with the animation in YouTube, then I can continue pouring the entries that were in the waiting queue. I promise I’ll update as quick as possible.

Having introduced the notion of orchestration in the last entry, we can get into deeper, fuzzier waters. Orchestration is as natural as vast, and many architects have created different solutions for the last 20 years. Nowadays there are standards, but so many and wrapped with so much marketing hype, that the subject is pervasive but remains dim, secluded. I illustrate here what I have succeeded to decode.

How to identify an orchestrator? it corresponds to the normal transfer of information that is present in the business process. This transfer is so fundamental that it is invisible in many organizations.

Let us imagine that your organization you has implemented paper-based processes . I know, I know. I don’t mean to abuse you but I am forced to create a horror scenario .

If this was the case, then customers (internal/external) should fill forms in order to ask for a service . Just imagine

For example, if you as employee want a new 24″ screen , you may be forced to fill a form to ask for that new screen . This paper-form should pass through the desk of your boss for a signature. Let us dream that your boss is happy with your request because your screen is more than 6 months old

Once she happily accepts, this form should go to some budget office. Let us imagine –then– that yours & some other requests are put together in a package that corresponds to the IT purchasings. To make things worse, assume that there is some person in the organization that should look at your request, ask for a few providers and examine their proposals, check them out once more with your boss or you, and then put it on the binder of the accepted & budgeted purchases. Ufff! Let us suppose that this binder arrives to the financial department, and that they will do the actual transaction.

In order to finish the BP, imagine that the provider receives and order (whose copy goes to your file binder) and that once the screen is sent, the bill will arrive to the accounting dpt. Someone over there will eventually pass it to treasury, that will do an –imaginary– payment to the provider.

Let us even imagine that your request has steadily grown to become a pile of paper forms with signatures from each person that has seen it, checked it up, approved or rejected it + photocopies of specifications/ bids/ offers/ contracts. Of course, in a theoretical scenario like this, one could yet imagine that copies of documents should be produced & sent to different sections of the organization

Of course, this is only fiction. Things like those cannot occur in real life

In order to avoid this Dantesque scenario, the wise management guys introduce an orchestrator. The orchestrator should replace the paper forms & binders. Not every single document, as most documents should be stored in a file for compliance requirements; such archive should be now performed in a digital format via a DMS (Document Management System — in french: GED) .

The orchestrator will require only the data that should be stored in order to satisfy the needs of future stages of the business process.

• For instance, the requester ID should be kept from the beginning because @ the end, the screen should be delivered to the requester (you ).
• Another example are the price and quantity of items that will be purchased, since this information will affect the financial information that is used for creating a budget, then to place the order, and finally to check the bill before issuing the corresponding payment.

Seen like this, any solution that can help us retransmit the informations to the services is good for the work. Be cautious with this kind of wild-, first-hand-, easily-sold, profusely-documented perspective of SOA. If you follow this simplistic analysis, then there is the danger to consider that workflow systems, ESB, Web-Services, CORBA, and any other EAI approach and technique are almost synonymous. They are very different things, specially when considering how the CRUD is split among the layers and the process’ stages, and when the persistence and the use of XML are analyzed in detail. Not to talk about BPM.

We will discuss those differences later on. Please try to think of what are the differences when:

• the orchestrator is implemented in such a way that all the information should pass through every process down the stream, against
• an orchestrator that sends to each process only the information that the specific process needs to perform its work

This reflection time will give you the first hints towards a classification. Please consider also the fact that most approaches mentioned above do not take into consideration the interactions of people during the process They assume that most information processing (transfers + services) is done automatically

This cannot be realistic in our scenario . Just give it a last thought. Then, try to sleep… if you can

Hasta luego!

The Business Process without orchestrator

In order to understand what you’re really obtaining from a BPMS, it is fundamental to see what the process looks like.

We illustrate here how an imaginary manufacturing company processes an order from a customer.

The company has 7 organizational units: Manufacturing (fabrication), Sales (ventes), Purchasings (achats), Marketing, Engine Certification, Certification of Technical Personnel.

Hey! I already started pouring some French here! Just as I had promised!

The company manufactures plane engines. Since this is a regulated industry, the engines must be tracked because of certification needs, and the personnel must also go through a detailed certification and tracking process.

• Everything starts when the order is received by the sales department. This is communicated the marketing deparment via the CRM.
• Once the deal is settled (price, quantity, discounts, etc.), the salesman checks whether everything that is required is already on stock.
• If it is not the case, the order goes to the manufacturing unit.
• First thing to do in the manufacturing unit: retrieve the Bill-Of-Materials and check that everything is ready.
• If some engine component is lacking, the purchasings unit should do what is required.
• Once all the components are there, the manufacturing can actually happen.
• The manufactured goods are packaged and put on the inventory.
• The sales department is notified that everything is ready.
• Shipping information is retrieved and confirmed with the customer
• For the transportation of goods, the purchasings department is called once more
• Once the negotiation with the provider of transportation is finished (weight, origin, destination, nature, price, delay, etc.) the goods are sent to the customer.
• After the goods are received by the customer, the Support Services (SAV-Service après vente) start. This service will feed the units of marketing with new statistics about problems, satisfaction, rate of failure, etc. This service also feeds the information required for the certification of each model of engine, and will help with all engine repairs (that can only be done by certified personnel).

This is illustrated in this illustration. It is an animation of the steps of the process.

It replaces and corrects this poor YouTube video. I will never put any other animation over there… it is quite un-intelligent to do what they did with my small example.

In a future entry, we will see what happens when this process is done via an orchestrator.

What is too often tacit and that should be know by both IT-guys and Biz-people

Konichiwa,

There are several aspects of the interface between the theory and practice of BPMS (and of the more general
workflow-systems) that are just never explained.

They are not necessarily present in the minds of non-specialists. I prefer to avoid the tacit understanding of such aspects. This often results in undeterministic and conflicting views of BPM, BPMN, and of the process descriptions that you create with it. It is also fundamental to grasp the essence of the architecture of BP-based systems (what is the place of BPMS and of workflows, of BPEL, BPEL4People, User Interaction, and others in the equation).

I won’t cover everything here but the main issues that help you understand the reasons that I consider most important to understand how to tackle a BP-project using Intalio and other BPMSs:

• The designer must clearly specify two logical sets:
  • the portions of the workflow (set of tasks) that can be performed automatically, and
  • those portions of workflow that require human intervention.
• The BP designer will generate an orchestrator that will be executed on a BPMS server. An orchestrator:
  • is an entity that guarantees that the execution of the business process follows the specification;
  • is external to the systems that will actually execute many of the actions, but it is the orchestrator who will command them to do those actions;
  • is not the owner of most of the information that is required for future actions, so it should grab and store that information for future use.
• The designer will then obtain two kinds of BP-pools (representing the coarse roles that participate in the process):
  • non-executable: each of the IT systems that the orchestrator must talk to, as well as every user interaction (web-based in most of cases)
  • executable: the orchestrator itself.
• The BPMS server:
  • provides an environment where to execute the orchestrator. Such environment is a virtual machine for BPEL and BPEL4People
  • provides a portal for managing the User-Interactions in synchrony with the BP-description
  • talks to security components, in order to authenticate and authorize BP-consumers

I hope this will make it easier to understand the structure of the resulting specification. More details on this subject will be discussed later.

Tchau!

Before you start creating the BP model in Intalio…

Bon giorno,

Most people find it difficult to start a business process modeling project with Intalio but when reusing examples. Consequently, I thought it would be useful to give a few hints of what it really takes to start from scratch. Of course, I’ll be as generic as possible and attempt to remain consistent with xPD, and we should address specific needs at some point in the future.

As I said before, the PD or BA is responsible for mapping the BP description by using Intalio Designer. Here, we follow a top-down approach where the specification comes from the business side; it may however remain still applicable for orchestration efforts coming from the IT side.

In order to model a Business Process, the BA should first have a model process, even a draft one. If it is not the case, she/he should create it, but this is a different issue we will talk about in a future entry. I will assume the BA has a list of the tasks that make up the process, and that YOU are the BA. Please, choose a simple process that is mostly sequential; we can add complexity later but the goal now is to understand.

The original model process is just a description. It does not posses several properties that are required to make it executable. So, we must transform this description (“AS-IS”) into the one you will obtain at the end of the modeling project (“TO-BE”) . What is the difference?

• Semantics (symbols used and grammar) of the description language are ad hoc.
• Data is not mapped: there is no way to communicate the context to the different tasks explicitly.
• Actors are often specified but in a physical/real environment, where paper & thick walls isolate roles, and where several specific IT systems/applications must be used.

We will write a new specification with more rigorous semantics by using BPMN. BPMN is a description language produced by consensus of a large community; it is a public standard, so people should be able to understand the BP-model 10 years from now, even in a different geographical area. That’s the good thing of this approach: it does not only facilitate reuse and maintenance, but also makes it possible to integrate the Biz-people onto the initiative.

Finally, before starting translating the description, please check first these points out:

• What is the event that can trigger this process? what makes the organization to launch a process like this?
• What is obtained as result at the end of the process?
• How many (coarse) roles/organizational units participate in the process?

PS: Because you should learn BPMN first; I propose you to download & read the Introduction to BPMN and to refer to websites such as BPM Initiative, BPMN.org, BPM Institute, and BPM Research. I won’t teach BPMN here, @ least not for the basics.

I will just tell you that BPMN diagrams are like flowcharts and that you must recognize only four shapes. Each shape represents a main element in the flow: event (circle) that indicate that something happened, activity (rounded rectangle) that mean some work is done (internally or by an external actor), selector/gateway (diamond) for representing control flow (branch/join), and connection(arrow) that indicate control flow and communication. In addition, the roles are represented by swimlanes (pools and lanes).

Hasta la vista!

The pre-conditions for making BPM and SOA possible

Namaste,

Of course I won’t talk here about every single condition. I will discuss, however, two critical issues regarding motivation, technology, and achievement.

Why? In the last few weeks, I’ve had very interesting discussions with some architects and senior developers. They are part of bigger architect and/or development teams and had addressed me as a may-be specialist that could assist them on finding the sense of their lives I summarize the several interactions in three logical groups. They are NOT physical groups of people, and most of them qualify at least in two logical groups.

Actually, they did not find the approach of their teams very correct however not wrong either. It was just that some doubts remained in their spirits, relentlessly distracting them from executing the tasks assigned to them. They could not only follow the same savanna corridor drawn by others without much thinking about where they were heading to.

The first of this groups presented me the following scenario: for several years, they had been developing a number of applications and wrappers for other applications in a big, corporate-wide effort to create a Service-Oriented Architecture. This SOA grows incrementally, and each new service (and the refactored ones) is integrated some time later in workflows and portal that were used by one or more parts of their organizations. So, what? many more years coming…

This is a really critical issue. Actually, they did not really understand what they were doing. When looking back, they felt that the effort done for so long by so many people did not really pay back. The project was late, more costly… the same history than before. So what? All this SOA stuff was just a ton of mumbo-jumbo, a lot of marketing.

This is something to avoid: projects should be shorter, incremental, ROI calculated rapidly, process owner satisfaction should become an upfront issue. Of course, this is an enormous change as for the projects in the past this was the thing to avoid. Please, this is a critical issue for SOA projects indeed. People should perceive the added value. Technical folks should be able to compare the final result.

The second logical group, a bit smaller, was constituted by people who either:

• wanted to know whether it was possible to create geo-spatial applications, or
• wanted to know whether it was really useful to create such applications for, for example, do inventory management

As you can see, there is a problem understanding the scope of a) the technical side of the architecture, as well as of b) the project. Why were they attracted into the geo-spatial applications? why YYYY-maps or GGGG-maps?

Hey! this obsession with mashups and Ajax make the technology the only issue. BPM is not about that. SOA enables this, but you only need a couple of web services for many cases and not a whole architecture. BPM and SOA are about the essentials of enterprise applications, not about the accidental issues such as information representation and interactivity of the user interface (but for applications whose goal is this one). BPM and SOA are intended to reduce complexity and decouple the architectural components. The process-based view of the world allows to map the business activities. If your solution requires visual geospatial components, just use them. If not, do not do it. Don’t mix what is not mixable and don’t compare what is not comparable.

The final, third group is about people that struggle with too many technical issues. Is this really necessary? is it useful? why so much complexity added? wasn’t it supposed to be the contrary? Hey, that hurts!!!

These guys pass their time solving issues about XML binding by hand, XML validation, doing the right messaging implementation for the routing, writing components for authorizing access, … the terms changed, but the fashion did not: the same well-known, vintage approach to architectural design and application development. They did not understand when I talked about performance metrics, about ROI, not even about BPMS like the ones provided by Intalio or PegaSystems (or BEA or Lombardi or MS). They have a daily battle with BPEL coding, SOAP coding, XML validation: the IT view only. They’ll never speedup.

BPM and SOA are more than that: they allow integrating the Biz- and the IT-worlds. They put the IT to the service of Biz, as it was in the old days. The tools are essential, but also understanding what really has to be measured. But… how to continue if we’re not sure all the exchanged XML docs are completely consistent? Well, I think you’re completely missing the point!!! As for everything else in the universe, there are a number of basic preconditions: your computer has the virtual machine installed and working correctly, the application server executes, it reads inputs and writes outputs,… and the computer is ON. You cannot suspect & validate everything!! Nevertheless, if you’re binding XML by hand everywhere and creating sockets by yourself instead of using the appropriate tools for parsing & validating XML and for creating the web services, and you also have to take care of the 13×5 XSL Transformations since people in the 5 independent development teams do not agree on what the XML schema is and, in addition, everybody wants to manage the round-trip exchange and persistence, first re-think your project very seriously: this is neither BPM nor “managed” SOA because you positively missed something in the process of selling it.

You know? I feel just like all these people. That’s why I already saw, experienced, studied, and read about many cases just like these. I know what it is and I want others to avoid it. I saw a bit too many organizations investing money on things they did not really understand. BPM is finally there, and I invite you to make the maximum profit from it.

We can conclude, then, that the pre-conditions for making BPM and SOA:

1. Communicate well: correctly and timely; do not overstate what can be done, do not let people go onto technical issues only; put the business process as central actors; show the roadmap. Integration requires agreement.
2. Focus: Do not let people wander in the woods, having crazy ideas of what can be done; do not let them concentrate a bit too much on technology. They will solve the wrong problem.

Aloha!

Evaluating the need for BPM and for SOA

Ni Hao,

Standard approaches define two sources for initiatives of projects on this domain: the Business(?) and the IT(?).

This is just a simplification but –unfortunately– people tend to take it as “the reference”. This becomes, then, a reductionist view of the world that does not allow to understand correctly the real diversity of scenarios. You should create a less constrained view of the enterprise, their products and their services, in the way that the Systemic Enterprise Architecture Methodology (SEAM) recommends.

As the terms business and IT are too fuzzy, let us first establish more precisely what standard approaches mean:
* Business:
- Operations
- Higher Management
- Internal Control
- …

* IT:
- Architecture
- Infrastructure
- B2B
- …

If we create the extended view of the enterprise, then another trigger becomes evident:
* Industry regulators:
- Compliance
- Traceability
- …

Of course, each of these organizational actors aims to improve a specific area of the organization, or to make explicit some factor as an indicator. We will discuss this subject further in an entry some time in the future. For the moment, I propose you to read the Beer Game; a short introduction can be found here.

Now, what can you obtain from an initiative of this type? well, the idea is to change the organization (only that ). I consider that there are at least 3 different types of result that can be obtained:

• Business Improvement ( on revenue or on operational performance)
• IT performance
• Process transparency

Revenue improvement is the fact that the systemic qualities of the solution allow you can now sell more to existing customers or to new customers, or that you can create new products (as a mix of existing ones); a more classical alternative is to increase the customer retention. A different beast here is when your organization is able to integrate a newly acquired organization quickly, in a transparent way. Most of the solutions on this area require what is called agility, a combined result of BPM and SOA.

The operational performance can be improved when the system is able to do things that it was unable to do before, when the flow of the processing is smoother because many things have been either done automatically by a built-in set-of-rules, or done quickly because all the required information “is just at your fingertips” (not spread through a diverse universe of IT systems and physical archives). This is a direct consequence of BPM.

The IT performance can be seen as the reduction of the cost of infrastructure, maintenance or reduction in the cost of new projects. How can this be achieved? because the infrastructure allows to interconnect/exchange information more easily, or is decoupled in such a way that changes become less and less critical to the environment of the associated application. More SOA than BPM.

Finally, the process transparency is the possibility to track the evolution of a request, and also of finding the hot spots that make its performance slower than expected. Also, the fact that the person that has been designated a specific task can be determined easily, in order to correct/add information for the process that can help in the decision process (contextual help for beginners, for example). Of course, this is BPM.

In a future entry, I will present some arguments that talk more to practitioners.

再见

Is this BP-project proposal for real?

Kalimera,

I have received some feedback about my proposal, and I will try to address the doubts about it. The word simplistic means that it is reductionist: not all the elements are there. It makes it more understandable and crispy. Not doing so will add much further detail very quickly making complexity appear in the process.

If you read carefully, the entries say “simplistic view” meaning that the view is reductionist, not the whole approach.

Why a reductionist view?

As I talk here about a BP-project, I have chosen to retain only three roles. This might be inconvenient for large projects, since the Information Architect, the platform specialists, process owners and other people should participate. Let me say that I focus first on the essential roles only, and that I will introduce other people as soon as the basic model will be described completely. Secondly, the 3 essential roles we cover are the ones that allow to create the Business Process Model. Other roles will allow us to integrate it with the SOA via the web services, know all the details of the information to be exchanged, add the security constraints and the specifics of the exchange protocol, etc. but this is not the core of what we aim to do.

In short, I know no other methodology that can be explained without a basic model. I expect no to create something more complex.

Is it formalized or peer-2-peer?

As many have already understood, this methodology is neither completely RUP-like, nor totally eXtreme-oriented. This attempts to be just a
good, effective combination of both; our goal is to remain eclectic, open to both improvements and customization.

Why RUP? Because it merges the best practices that have been compiled in many years and with experience accumulated in different domains, people know it, and even if the name is as generic as to mean nothing in particular, it also sells because it is as generic as to mean many things –including quality– to many people (It is just that I’m tired of meeting people that demand RUP, and later they show they know not much about it… but the word makes people calm… wow!)

Why eXtreme? Since we want models/code to be executed and validated in short loops, and to think about testing even before modeling, and to guarantee the quality during the execution… there is no other choice.

How many other roles, then?

Each role can be split onto 2-3 roles each, that may be often performed by 1-5 persons each. In really small projects you can find all three roles made by a single person. Of course I will talk about this in a future entry. Please be alert.

Arrivederci