Implementing the Correlation Identifier Pattern on Stateful Logic Apps using the Webhook Action

Introduction

In many business scenarios, there is the need to implement long-running processes which first send a message to a second process and then pause and wait for an asynchronous response before they continue. Being this an asynchronous communication, the challenge is to correlate the response to the original request. The Correlation Identifier enterprise integration pattern targets this scenario.

Azure Logic Apps provides a stateful workflow engine that allow us to implement robust integration workflows quite easily. One of the workflow actions in Logic Apps is the webhook action, which can be used to implement the Correlation Identifier pattern. One typical scenario in which this pattern can be used is when an approval step with a custom API (with a similar behaviour to the Send Approval Email connector) is required in a workflow.

In this post, I will show how to implement the Correlation Identifier enterprise integration pattern on Logic Apps leveraging the webhook action.

Some background information

The Correlation Identifier Pattern

Adapted from Enterprise Integration Patterns

The Correlation Identifier enterprise integration pattern proposes to add a unique id to the request message on the requestor end and return it as the correlation identifier in the asynchronous response message. This way, when the requestor receives the asynchronous response, it knows which request that response corresponds to. Depending on the functional and non-functional requirements, this pattern can be implemented in a stateless or stateful manner.

Understanding webhooks

A webhook is a service that will be triggered on a particular event and will result on an Http call to a RESTful subscriber. A much more comprehensive definition can be found here. You might be familiar with the configuration of webhooks with static subscribers. In a previous post, I showed how to trigger a Logic App by an SMS message with a Twilio webhook. This webhook will sends all events to the same Http endpoint, i.e. a static subscriber.

The Correlation Identifier pattern on Logic Apps

If you have used the Send Approval Email Logic App Connector, this implements the Correlation Identifier pattern out-of-the-box in a stateful manner. When this connector is used in a Logic App workflow, an email is sent, and the workflow instance waits for a response. Once the email recipient clicks on a button in the email, the particular workflow instance receives an asynchronous callback with a payload containing the user selection; and it continues to the next step. This approval email comes in very handy in many cases; however, a custom implementation of this pattern might be required in different business scenarios. The webhook action allow us to have a custom implementation of the Correlation Identifier pattern.

The Logic Apps Webhook Action

To implement the Correlation Identifier pattern, it’s important that you have a basic understanding of the Logic Apps webhook action. Justin wrote some handy notes about it here. The webhook action of Logic Apps works with an instance-based, i.e. dynamic webhook subscription. Once executed, the webhook action generates an instance-based callback URL for the dynamic subscription. This URL is to be used to send a correlated response to trigger the continuation of the corresponding workflow. This applies the Return Address integration pattern.

We can implement the Correlation Identifier pattern by building a Custom API Connector for Logic Apps following the webhook subscribe and unsubscribe pattern of Logic Apps. However, it’s also possible to implement this pattern without the need of writing a Custom API Connector, as I’ll show below.

Scenario

To illustrate the pattern, I’ll be using a fictitious company called FarmToTable. FarmToTable provides delivery of fresh produce by drone. Consumers subscribe to the delivery service by creating their personalised list of produce to be delivered on a weekly basis. FarmToTable requires to implement an SMS confirmation service so that an SMS message is sent to each consumer the day before the scheduled delivery date. After receiving the text message, the customer must confirm within 12 hours whether they want the delivery or not, so that the delivery is arranged.

The Solution Architecture

As mentioned above, the scenario requires sending an SMS text message and waiting for an SMS response. For sending and receiving the SMS, we will be using Twilio. More details on working with Logic Apps and Twilio on one of my previous posts. Twilio provides webhooks that are triggered when SMS messages are received. The Twilio webhooks only allow static subscriptions, i.e. calling one single Http endpoint. Nevertheless, the webhook action of Logic Apps requires the webhook subscribe and unsubscribe pattern, which works with an instance-based subscription. Thus, we need to implement a wrapper for the required subscribe/unsubscribe pattern.

The architecture of this pattern is shown in the figure below and explain after.

Components of the solution:

  1. Long-running stateful workflow. This is the Logic App that controls the main workflow, sends a request, pauses and waits for an asynchronous response. This is implememented by using the webhook action.
  2. Subscribe/Unsubscribe Webhook Wrapper. In our scenario, we are working with a third-party service (Twilio) that only supports webhooks with static subscriptions; thus, we need to create this wrapper. This wrapper is composed by 4 different parts.
  • Subscription store: A database to store the unique message Id and the instance-based callback URL provided by the webhook action. In my implementation, I’m using Azure Cosmos DB for this. Nevertheless, you can use any other suitable alternative. Because the only message id we can send to Twilio and get back is the phone number, I’m using this as my correlation identifier. We can assume that for this scenario the phone number is unique during the day.
  • Subscribe and Start Request Processing API: this is a RESTful API that is in charge of starting the processing of the request and storing the subscription. I’m implementing this API with a Logic App, but you can use an Azure Function, an API App or a Custom Api App connector for Logic App.
  • Unsubscribe and Cancel Request Processing API: this is another RESTful API that is only going to be called if the webhook action on the main workflow times out. This API is in charge of cancelling the processing and deleting the subscription from the store. The unsubscribe step has a similar purpose to the CancellationToken structure used in C# async programming. In our scenario, there is nothing to cancel though. Like the previous API, I’m implementing this with a Logic App, but you can use different technologies.
  • Instance-based webhook: this webhook is to be triggered by the third-party webhook with a static subscription. Once triggered, this Logic App is in charge of getting the instance-based callback URL from the store and invoking it. After making the call back to the main workflow instance, the subscription is to be deleted.

The actual solution

To implement this solution, I’m going to follow the steps described below:

1. Configure my Twilio account to be able to send and receive SMS messages. More details here.

2. Create a Service Bus Namespace and 2 queues. For my scenario, I’m using one inbound queue (ScheduledDeliveriesToConfirm) and one outbound queue (ConfirmedScheduledDeliveries). For your own scenarios, you can use other triggers and outbound protocols.

3. Create a Cosmos Db collection to store the instance-based webhook subscriptions. More details on how to work with Cosmos Db here.

  • Create Cosmos Db account (with the Document DB API).
  • Create database
  • Create collection.

4. Create the “Subscribe and Start Request Processing API”. I’m using a Logic App workflow to implement this API as shown below. I hope the steps with their comments are self-explanatory.

  • The workflow is Http triggered. It expects, as the request body, the scheduled delivery details and the instance-based callback URL of the calling webhook action.
  • The provided Http trigger URL is to be configured later in the webhook action subscribe Uri of the main Logic App.
  • It stores the correlation on Cosmos Db. More information on the Cosmos Db connector here.
  • It starts the request processing by calling the Twilio connector to send the SMS message.

The expected payload for this API is as the one below. This payload is to be sent by the webhook action subscribe call on the main Logic App:

{
    "callbackUrl": "https://prod-00.australiasoutheast.logic.azure.com/workflows/guid/runs/guid/actions/action/run?params",
    "scheduledDelivery": {
        "deliveryId": "2c5c8390-b6c8-4274-b785-33121b01e219",
        "customer": "Paco de la Cruz",
        "customerPreferredName": "Paco",
        "phone": "+61000000000",
        "orderName": "Seasonal leafy greens and fruits",
        "deliveryAddressName": "Home",
        "deliveryDate": "2017-07-20",
        "deliveryTime": "07:30",
        "createdDateTime": "2017-07-19T09:10:03.209"
    }
} 

You can have a look at the code behind here. Please use it just as a reference, as it hasn’t been refactored for deployment.

5. Create the “Unsubscribe and Cancel Request Processing API”. I used another Logic App workflow to implement this API. This API is only going to be called if the webhook action on the main workflow times out. The workflow is show below.

  • The workflow is Http triggered. It expects as the request body the message id so the corresponding subscription can be deleted.
  • The provided Http trigger URL is to be configured later in the webhook action unsubscribe Uri of the main Logic App.
  • It deletes the subscription from Cosmos Db. More information on the Cosmos Db connector here.

The expected payload for this API is quite simple, as the one shown below. This payload is to be sent by the webhook action unsubscribe call on the main Logic App:

{
    "id": "+61000000000"
}

The code behind is published here. Please use it just as a reference, as it hasn’t been refactored to be deployed.

6. Create the Instance-based Webhook. I’m using another Logic App to implement the instance-based webhook as shown below.

  • The workflow is Http triggered. It’s to be triggered by the Twilio webhook.
  • The provided Http trigger URL is to be configured later in the Twilio webhook.
  • It gets the message Id (phone number) from the Twilio message.
  • It then gets the instance-based subscription (callback URL) from Cosmos Db.
  • Then, it posts the received message to the corresponding instance of the main Logic App workflow by using the correlated callback URL.
  • After making the callback, it deletes the subscription from Cosmos Db.

The code behind for this workflow is here. Please use it just as a reference, as it is not ready to be deployed.

7. Configure the Twilio static webhook. Now, we have to configure the Twilio webhook to call the Logic App created above when an SMS message is received. Detailed instructions in my previous post.

8. Create the long-running stateful workflow. Once we have the implemented the subscribe/unsubscribe webhook wrapper required for the Logic App webhook action, we can start creating the long-running stateful workflow. This is shown below.

In order to trigger the Unsubscription API, the timeout property of the webhook action must be configured. This can be specified under the settings of the action. The Duration is to be configured the in ISO 8601 duration format. If you don’t want to resend the request after the time out, you should turn off the retry policy.

  • The workflow is triggered by messages on the ScheduledDeliveriesToConfirm Service Bus queue.
  • Then the webhook action:
    • Sends the scheduled delivery message and the corresponding instance-based callback URL to the Subscribe and Start Request Processing Logic App.
    • Waits for the callback from the Instance-based webhook. This would receive as an Http post the response send by the customer. If a response is received before the time out limit, the action will succeed and continue to the next action.
    • If the webhook action times out, it calls the Unsubscribe and Cancel Request Processing Logic App and sends the message id (phone number); and the action fails so the workflow does not continue. However, if required, you could continue the workflow by configuring the RunAfter property of the subsequent action.
  • If a response is received, the workflow continues assessing the response. If the response is ‘YES’, it sends the original message to the ConfirmedScheduledDeliveries queue.

The code behind of this workflow is available here. Please use it just as a reference only, as it hasn’t been refactored for deployment.

Now, we have finished implementing the whole solution! 🙂 You can have a look at all the Logic Apps JSON definitions in this repository.

Conclusion

In this post, I’ve shown how to implement the Correlation Identifier pattern using a stateful Logic App. To illustrate the pattern, I implemented an approval step in a Logic App workflow with a custom API. For this, I used Twilio, a third-party service, that offers a webhook with a static subscription; and created a wrapper to implement the subscribe/unsubscribe pattern, including an instance-based webhook to meet the Logic Apps webhook action requirements.

I hope you find this post useful whenever you have to add a custom approval step or correlate asynchronous messages using Logic Apps, or that I’ve given you an overview of how to enable the correlation of asynchronous messages in your own workflow or integration scenarios.

Feel free to add your comments or questions below, and happy clouding!

Cross-posted on Mexia Blog.
Follow me on @pacodelacruz

Microsoft Azure Integration Platform as a Service (iPaaS) – Logic Apps, Azure Functions, API Management and Service Bus working together

Originally posted on Mexia’s blog.

Introduction

If you work for an established organisation going through Digital Transformation or in a modern company born in the digital era, the chances are that IT is required to implement integration solutions more than ever before. Whether an organisation is an incumbent or a new entrant, they need to leverage the power of technology to lead the change or embrace it. Monolithic systems will not be capable of meeting the needs of digital organisations, creating the need to interconnect existing systems with best-of-breed cloud and distributed apps and to expose all these systems to other systems, business partners or consumer apps through easy to consume APIs.

Microsoft provides different technologies on Azure which enable us to build robust application, data and process integration solutions. One of the core offerings for Azure integration is Logic Apps – however there are other Azure PaaS offerings which when used with Logic Apps form the robust building blocks of first-class integration solutions. These technologies together provide a very rich and fully-managed Integration Platform as a Service (iPaaS). In this post, I will try to describe these Azure technologies that we can leverage to make our life easier, and perhaps even more fun, when implementing enterprise integration solutions.

Azure Services to build integration solutions

Out of the very large and growing list of Azure services, there are some that we can utilise to build our integration solutions, including Logic Apps, Azure Functions, API Management, and Service Bus Messaging. I will briefly describe each of them below.

Logic Apps

Logic Apps is a very robust and powerful platform to orchestrate and automate integration workflows. The visual designer together with all the available connectors, and deployment and management tools make it a great tool for this kind of scenarios. Logic Apps also provides the concept of serverless computing, in which you just focus on what you want to achieve, without worrying at all about servers, patching, scaling, etc.

Workflow and Orchestration Capabilities

Logic Apps workflows, which can be easily designed and implemented graphically (via the Azure Portal or Visual Studio), are based on the Workflow Definition Language. These workflows provide rich ways to process and manipulate data than can be obtained or pushed via different connectors. Below there is a figure which shows a simple Logic App workflow with different actions, a condition and a loop.

In Logic Apps workflows we can implement different actions, like calling an HTTP endpoint, calling Azure API Apps, calling WebHooks, waiting, calling a Logic App nested workflow, implementing conditions and loops, query arrays, terminating a workflow, or returning an HTTP Response.

Triggers, Input and Outputs

Logic Apps provides a growing list of connectors that allow us to trigger workflows and get data from and push data to many different protocols, SaaS applications, other Azure and Power Apps Services, and on-premises systems. Below there is a snapshot of the 100+ connectors available at the time of writing (few of them are still in preview).

Protocol Connectors

Azure Services and Power Apps Connectors

SaaS Connectors

B2B, XML, EDI and AS2 Connectors

Hybrid Connectors

As we can see, with all these connectors, Logic Apps allows us to very easily and quickly connect to many different protocols, apps, and also to other Azure Services. Additionally, the Enterprise Integration Pack brings some of the BizTalk Server functionality to Logic Apps and makes really easy to implement B2B integrations based on AS2 and EDI.

Azure Functions

Another Azure service that is very useful when building integration solutions is Azure Functions. Azure Functions are the evolution of Azure WebJobs, which allow us to create microservices or small pieces of code that can run synchronously or asynchronously as part of composite and distributed cloud solutions. Azure Functions are built on top of the WebJobs SDK but with some additional benefits and capabilities, including the option of being deployed on a serverless-style Dynamic Service Plan, in which you pay per consumption only, and also the capability of being triggered via HTTP. Additionally, with Functions you can deploy from very simple to quite complex pieces of code developed on different languages, such as bash (.sh), batch (.bat / .cmd), C#, F#, Node.Js, PHP, PowerShell, and Python.

Triggers, Inputs and Outputs

Azure Functions support different triggers, input and output bindings, which I summarise in the table below. These bindings make Functions very easy to be called and allow them to get data from and push data to other microservices, Logic Apps or other systems or services.

Type / Service Trigger Input Output
Schedule *
HTTP Call * * *
Azure Blob Storage * * *
Azure Event Hubs * * *
Azure Storage Queues * * *
Azure Service Bus Messaging * * *
Azure Storage Tables * *
Azure Mobile Apps Tables * *
Azure DocumentDB * *
Azure Notification Hubs *
Twilio SMS Message *
SendGrid Emails (Not fully documented yet) *
Files in Cloud File Storage SaaS, such as Box, DropBox, OneDrive, FTP and SFTP (Not fully documented yet) * * *

Processing

Integration Processes usually require custom data and message validation, enrichment, transformation or routing. These can simply be done via custom code on Azure Functions.

Service Bus Messaging

Azure Service Bus Messaging provides reliable message queuing and publish-subscribe messaging capabilities. Azure Service Bus Queues and Topics allow us to decouple in time upstream and downstream systems or different interrelated services. Service Bus Queues provide ordered message delivery to competing consumers, while Service Bus Topics enable publish-subscribe messaging scenarios where multiple consumers can process the same message.

The following Service Bus features make it a key component of most integration solutions on Azure:

  • temporal decoupling;
  • the ability to load balance message processing using competing consumers;
  • the capability of implementing Publish/Subscribe architecture via Service Bus Topics;
  • and the fact that Functions and Logic Apps can very easily read and write messages from and to Service Bus.

API Management

API Management is an Azure Service which functions as a API Gateway or Front-End for backend APIs in the cloud or on-prem. In addition, it provides a Developer portal which helps to speed up the adoption and use of the implemented APIs. Some of the benefits of API Management are:

Thanks to all these features, API Management can be leveraged on many integration solutions which require to expose RESTful APIs and require any kind of mediation.

When to use each of these technologies and how they complement each other

Now that I have described the Azure iPaaS offerings, it’s worth analysing when we could use each of these technologies and how they complement each other. I summarise this in the table below.

Technology When to use it Use together with
Logic Apps
  • To implement and orchestrate visually designed integration workflows.
  • To orchestrate distributed microservices.
  • To leverage the 100+ connectors to interact with different protocols, SaaS systems and services, other Azure services, and on-premises systems.
  • To implement cloud-based B2B integrations with AS2 and EDI.
  • Functions – so custom logic can be implemented in microservices to be orchestrated by LogicApps.
  • Service Bus
    to decouple in time different microservices and steps in the integration process.
  • API Management – for those HTTP triggered apps when some of the capabilities of API Management are required.
Azure Functions
  • To implement code-based microservices or processing.
  • Logic Apps
    so different microservices can be orchestrated.
  • Azure Service Bus Messaging – to decouple in time different microservices.
  • API Management – for those HTTP triggered functions when some of the capabilities of API Management are required.
Service Bus
  • To decouple in time upstream systems from downstream systems or different microservices.
  • To implement multiple consumers on a Publish/Subscribe pattern.
  • To allow load distribution with multi-instance competing consumers.
  • Functions and Logic Apps
    to decouple in time different microservices and steps in the integration process.
API Management
  • When any of the API Management features is required, for example: securing backend APIs, API response caching, request throttling, request routing, request transformation, API calls tracing or logging, usage and health analytics, or providing a rich portal for developers.
  • Functions and Logics Apps – that are
    triggered by an HTTP call and require some kind of mediation.

Summary

Azure provides a very robust Integration Platform as a Service (iPaaS), which is based on Logic Apps and can be complemented with Azure Functions, Service Bus Messaging and API Management

The breadth and capabilities of many different Azure technologies and how they complement each other is what differentiates Azure against other iPaaS vendors. We can leverage many different services to build first-class integration solutions. Logic Apps is the core engine to implement these. Logic Apps can connect to many different protocols, SaaS apps and Services, to other Azure and Power App Services, to on-premises systems and to B2B trading partners via a growing list of connectors. Logic Apps integration workflows can easily be extended with custom code implemented as microservices on Azure Functions. In order to decouple in time these integration processes, we can leverage Service Bus Messaging services. And in case we need to expose our integration services as RESTful APIs, we might want to make use of all the features and capabilities of API Management. Additionally, these integration solutions can be enhanced by other Azure services, such as Cognitive Services to, for example get the sentiment from social media feeds or Azure Active Directory for authenticating and authorising calls to our APIs. All of this with all the PaaS magic and the powerful DevOps capabilities of Azure.

Thanks for reading, and please share your thoughts or queries on this topic below J

Happy clouding!