Example for automating a time consuming manual process

My recent entries were quite technical and were meant to solve very specific problems or show how to overcome certain issues on the last mile that are very annoying in a per se well defined concept. Today I’d like to give a overview from a higher level to put my other posts into perspective.

The premise: You are a software/service provider

Imagine you have a program or software you want to offer to your customer but the customer, often technically inexperienced, should not have to deal with servers, providers, DNS entries, certificates, maintenance, the cloud and all the technical stuff that comes with this. He should just fill out a form and afterwards it “should just work”.

The start: A manual process

As soon as the order comes in a manual process takes place on the provider side:

  • Checking the provided data (are the VAT number and contact details correct, is it a real customer, etc.)
  • Create the virtual machine. Each customer gets one because problems on one machine should not affect the others.
  • Install and configure the software.
  • Setup DNS entries so that the page can be found by others (i.e. use “myshopname.com” instead of “172.16.47.11”)
  • Acquire certificates for state of the art secure communication. Under certain circumstances this has to be repeated every three month.
  • Register the page with tags in search engines.
  • Integrate the server into a monitoring environment so that you can call the customer that a problem has been solved instead of the customer calling you that he has a problem.

As you can see: There are quite a lot of steps that need to be done and all starts with a potential customer filling out a form. Now customers are volatile and if this takes longer than the “accepted amount of time” the customer will not use your service but maybe the competitors. I don’t have to go into detail what this means. The technical process on the other hand takes a certain amount of time and it has to be done by a person. While this person is doing it all new requests that come in during the process have to wait and get scheduled back which increases the duration even more. This means this is not very effective. Thus it should only be done during development of an automated process: Fist get something running, then make it better and more efficient. The next part will describe how such a process could look like.

The goal: Automation

The final goal is a process where the only thing indicating that a new customer has subscribed to the service is a payment at the end of the month but just sitting down and saying “I am going to do this!” is like standing in front of a mountain and trying to take it all in one step: Impossible. What is possible is automating and testing small parts on the way and putting them together to get a automated process.

First let’s analyze the mentioned manual steps:

As you can see there are several layers and “partners” where to get information from or where to send information to. Often these partners offer APIs (interfaces working with machine language that can be easily accessed from a program). So now I’d estimate where I could get which information from which leads to this annotated diagram:

Annotated process

Now I would have a basic understanding what needs to be done and where I could get my information from. This steps then translate directly into stages of two pipelines (automated and via trigger launchable processes). The first trigger (for the validation pipeline) would be the customer filling out the form and sending it and the second trigger (for the setup pipeline) would be the customer clicking on the confirmation link in the email he gets from the system.

Pipeline view of the processes

And in the end the customer would get a notification mail with generated access credentials and when he opens the page for the first time he would have to change the password to his own.

This is of course a massive oversimplification. In this process there are much more components involved: You need a application hosting the pipeline and listening to triggers, a server is required to provide software components, a monitoring server has to be set up and so on and so forth. Bringing everything into these diagrams would have been way too much for this example. What I wanted to show is that even very complex processes can be broken down to simple, manageable steps. Or as said in terms of programming: Divide and conquer

Thanks for reading and as always: I hope this helps somebody out there.

PS: If you want to modify the diagrams for yourself you can open the PNG files in diagrams.net just save them and open them on the page like this:

Just select the PNG and it should open as editable diagram.