State-full vs State-less

Appian is now more than BPM.  And even if it weren't, if it were simply BPM, I'd still use the best approach for my use case rather than the best approach for my tools.  I wouldn't care if I was maximizing the potential of Appian if I was doing it at the cost of a less performant system.

The processes can be long, they can be complex, they can have loads of exceptions, loads of different branching paths, and they can call multitudes of other processes.  That doesn't change.  All that changes with Stefan's approach, which I think virtually all us old-timers agree with, is how long the processes park in one spot doing nothing but taking up RAM.

Let's call it state-full for as long as it needs to be state-full, and then not.  Why not let the DB contribute something?  Especially if you get a better product as a result.

But.... how can one tell what is the best approach ? Going  state-full or going state-less ?  How do you weigh the pros and cons ?  The issue of idle processes, waiting long time for a trigger should not be a criteria, as the framework should be able to hibernate such processes with minimal overhead. .However going state-less requires mixed structure of state management as part of it is in the data layer and part of it is in active processes, so the art of breaking up your long lived process to shorter ones is artificial in nature and breaks the tooling of BPMN. So I guess Stefan's approach of keeping small short lived processes, is actually a shift towards the state-less approach, towards the data driven world, giving up all the goodies of BPMN.  I can totally agree to this approach if you need to scale horizontally and give up the state-full world, however if you are OK with vertical scaling then why do that and and still be fined with a premium cost of a BPM platform ? 

It sounds like you're saying "Why bother with Appian unless you can have long-lived processes?"  I've never heard anyone that actually liked the idea of long-lived processes before.  I don't see how it breaks the tooling of BPMN.  You're making really good processes and denoting them correctly.  Part of what makes them such good processes is that they stop in a timely manner.  How does using software to implement an efficient process ruin the functionality of the software?

And if you want to leave a process parked on a user input task for three days before someone completes the task, you can.  It's in there if you want to do it that way.  You can keep the goodies of BPMN.  But don't you lose more than you gain overall?

Plus, why Appian is also SAIL, mobile ready with no additional dev, countless integrations with other systems and vendors, process mining, RPA, IDP, and several other features; apps that can be built 10X faster than by traditional code.  I'm OK with not using BPMN to some theoretical ideal.

There's also folks out there who don't use the Process Modeler or process models for anything if they can help it.  They run ALL SAIL.  You can build Portals that way too; in fact you basically have to.

Not arguing all other goodies APPIAN has to offer, just questioning the BPM architecture. If you break your long lived process you have lost the view of your entire process (there is no a single canvas you can see your process end to end). You need to look at multiple PMs, and create a network of "chained" processes and/or cross boundaries via the data layer, your state is a collection of sub-states, some of them  live  in short-lived processes and some of them are parked in the database - quite a mess. Your short-lived process instances  are ''almost" equivalent of traditional web service calls (perhaps more capable, however simplified process models).

In that sense this is a "waste". If that all you need from a BPM platform then APPIAN/BPM is overkill. 

Thus, if your application is BPM one you should go APPIAN all the way and not break up your long lived process. Unless there are other factors which are strong cons, like hardware constrains, or other limitations imposed by the platform (like reporting, process upgrade concerns etc).

While I am a strong contender of thinking in processes and push very hard to put process control into Appian, I think reality is just not black or white. Long-lived processes have an inherent bigger problem with upgradeability, and a smaller problem with memory consumption.

Most of our apps are a hybrid of case management, where the user decides about the next action, and strict process where process logic defines the next actions. Both approaches require a different design and Appian is IMHO a fantastic platform to support both.

A BPM approach is the right way, then add other paradigms as required.

Ok, so I guess we share the same view, both fond of the BPM view of the solution however facing a reality where BPM at its full scale cannot be implemented due to platform limitations. I grew up as a developer on the data side and for some time have been wondering whether it is possible to switch to the BPM paradigm. Have been working with APPIAN in the last year doing some POC projects, however nothing  at scale. Until the last few months where I am overviewing a real life implementation executed by one of APIAN's big business partners.  To my surprise, they have created a mixed architecture, where state is managed both in the data layer and in longer lived processes. Finally, this project stumbled the upgradeability issue. So can you share you experience with that ? I can see there is a path of upgrading live processes, writing custom plug-ins, which seems to me a kill, giving I am not a JAVA guy, and have no experience and means of writing APPIAN plugins. The other way around I can think of is a process of killing the active processes, and restoring them back to the same state using SAIL. Is it applicable approach ? Is it possible to "milk" all the process variables from a running process, including all pending tasks and instantiate a new (upgraded) instance of the PM in the very same state ?

I think that a process that is active for more than 3 days is a design flaw.

A task should either be completed right away, or turned into a follow-up.

Think of the Getting-Things-Done methodology. My users have the chance to tell the application that they cannot complete that task right now but can plan their next activity. That specific process is then completed, but a follow-up, stored to the database, will remind that user at the right time. If necessary, a user can also restart that process model manually before the due date.

Then, I cut my processes into pieces at boundaries I identify by time of inactivity and isolation of any complex logic. A process that needs to wait a long time for a task to be completed, and then has to evaluate some complex logic will run into an upgrade problem. I try to put that logic at the beginning of a process model.

This is a high level description of my very successful approach that might miss some details, and also does not work for all scenarios out there. And it requires a change in how people work, which can become a bigger challenge than the actual implementation in Appian.

I am not sure I understand. Lets look at my case. We are dealing with a process of loan approval. It has multiple stages, involving different personas in each stage. I think it make sense to model this process with single high level process model that manages the state (stage) of the loan request. At various stages this loan request needs to collect data, do validations, get approvals of different personas and finally propagate to execution. It is a process that can be completed in matter of days or weeks. Should I give up a central per loan process which controls the flow of the loan request just because it takes a week rather then 3 days ? If I do then it means the state of the loan is managed in the database (not BPM), while segments (perhaps per stage) are managed by different process models. In this case, in order to provide reporting (which includes statuses regarding the inner state of the stages) must persist their states as well to the database - ending in data driven process. Alternatively, we could have a long lived process which has the full view of the loan and simplify the whole application. I can understand the need of breaking it to sub models due to limitation of the platform, however this is not a result of design flaw, but rather a flaw of the platform.

Let's start at the end. While I do not have experience with other BPM platforms, upgrading a running process to the new model seems to always a tricky problem. Now, the solution in the Appian space, is to implement a process as a series of process models, instead of using a overarching parent model. For me, the design flaw is more from a business process perspective than from an implementation perspective. I as a user want to be able to plan my work. And a endless list of tasks I cannot complete does not help. Follow-ups are a great way to do that.

In my experience, a real world application should try to put process control into Appian as much as possible. But there are reasons to put control into the hand of the user. With Appian you can easily do both.

You also mentioned reporting. I see at least two kinds of reports. Operations reporting is based on actual process performance and can easily be done using process reporting with some data added from the DB. The more strategic reporting requires more complex aggregations and time-line based data which you cannot get from process reporting. For this, you will need to persist data to a DB.

Nahum said:

Alternatively, we could have a long lived process which has the full view of the loan and simplify the whole application. I can understand the need of breaking it to sub models due to limitation of the platform, however this is not a result of design flaw, but rather a flaw of the platform.

I would not call this a flaw of the platform.  Appian will allow you to keep processes in memory with your over-arching parent process design for as long as you like, with reportability and no need to ever persist to any external database.  Then, it becomes a limitation of your hardware.  RAM, CPU, disk space.  Higher percentage of utilization will cause Appian to slow.  Thus, the recommendation shifts to shorter processes with the database holding state.

Shorter processes also increase upgradability.  Win-win.  Even software such as Microsoft Outlook requires a restart during or after an upgrade, to load a new instance of itself.  Why shouldn't an Appian process be allowed to do the same?  The longer one stays running, the more complex it will be to upgrade.

If you can teach me how I can reboot APPIAN and have all my running processes upgraded to their latest versions in the same state, I would be more then  happy. So this comparison to Outlook has no grounds.  Not sure why you say that the longer you run the more complex is to upgrade. The state of a process, is not function of the elapsed time since the process started. The state of the process, is a collection of process variable values plus a set of  its associated active tasks (and perhaps more entities). The fact that APPIAN cannot do it in a friendly and usable manner forces you to give up a full BPM design  and have all those hybrid solutions. 

If you can teach me how I can reboot APPIAN and have all my running processes upgraded to their latest versions in the same state, I would be more then  happy. So this comparison to Outlook has no grounds.  Not sure why you say that the longer you run the more complex is to upgrade. The state of a process, is not function of the elapsed time since the process started. The state of the process, is a collection of process variable values plus a set of  its associated active tasks (and perhaps more entities). The fact that APPIAN cannot do it in a friendly and usable manner forces you to give up a full BPM design  and have all those hybrid solutions. 

Nahum said:

If you can teach me how I can reboot APPIAN and have all my running processes upgraded to their latest versions in the same state, I would be more then  happy.

Sure, this occurs by using shorter processes.  When your next approval step in the process is a new sub process, it is rebooted to the new state essentially.

Nahum said:

Not sure why you say that the longer you run the more complex is to upgrade.

If you have a single process instance that lives for 6 months, and you upgrade monthly, living process instances will have to manage themselves across 6 different versions.  This is inherently more complex.

Nahum said:

The state of a process, is not function of the elapsed time since the process started.

It can be, consider processes with a status of "late" or "delayed".

Nahum said:

The fact that APPIAN cannot do it in a friendly and usable manner forces you to give up a full BPM design  and have all those hybrid solutions.

I would hardly consider using a database in conjunction with Appian processes per best practices, a "non-BPM" or "half-BPM" design.  Per Wikipedia, Gartner and others "BPM" is in fact a very general term.  I submit that if you are managing your business processes with Appian and a database, that is 100% BPM.

Business process management (BPM) is the discipline in which people use various methods to discover, model, analyze, measure, improve, optimize, and automate business processes.^[1][2] Any combination of methods used to manage a company's business processes is BPM.^[3] Processes can be structured and repeatable or unstructured and variable. Though not required, enabling technologies are often used with BPM.^[1]