Async Http APIs with Azure Durable Functions (and Polling Client)

070618_1006_AsyncHttpAP1.png

Introduction

Azure Durable Functions have support for different patterns, which enable us to build serverless and stateful applications without worrying about the state management implementation details. One of these useful patterns is the Asynchronous Http APIs. This pattern comes in handy when client applications need to trigger long running processes exposed as APIs and do something else after a status is reached. You might be thinking that the best way to implement an async API is by enabling the API to raise an event once it finishes so the client can react to it. For instance via a webhook (Http callback request), an Event Grid event, or even using SignalR. That is true, however, in many cases, client apps cannot be modified, or there are security or networking restrictions which make polling the best or the only feasible alternative. In this post, I’ll describe how to implement the Asynchronous Http API pattern on Durable Functions based on the polling client approach.

Before we get into the details, it’s worth noting that this pattern can be used to implement a custom trigger or action for Logic Apps with the polling pattern.

Scenario

To demonstrate how to implement this pattern, I’ll use the scenario of “Call for Speakers” in a conference. In this, potential speakers submit a topic through an app, and they are very keen to know as soon as possible if they have been selected to present. Of course, in a real scenario there will be timelines and speakers would be notified before the agenda is published, so they wouldn’t need to keep continuously asking for the status of their submission. But I believe you get the idea that this is being used for illustration purposes only 😉

Solution Overview

The solution is based on Azure Durable Functions’ building blocks, including orchestration clients, the orchestration function, and activity functions. I’ve used dummy activity functions, as the main purpose of this post is to demonstrate the capabilities to implement an Asynchronous Http API. In case you want to understand how to implement this further, you can have a look at my previous posts on how to implement approval workflows on Durable Functions.

The diagram below shows the different components of the solution. As you can see, there is a client app which submits the application by calling the Submit Azure function, and then can get the status of the application by calling a GET Status Azure function. The orchestration function controls the process and calls the activity functions.

070618_1006_AsyncHttpAP2.png

Solution Components

The main components of the solution are described below. The comments in the code should also help you to better understand each of them. You can find the full solution sample code here.

Submit Function

Http triggered function which implements the DurableOrchestrationClient. This function receives a “Call-for-Speakers” submission as a POST with a JSON payload as the request body. Then, it starts the submission processing orchestration. Finally, it returns to the client the URL to the endpoint to get the status using the location and retry-after http headers.

Process Submission Function

Orchestration function which implements the function chaining pattern for each of the stages of the submission approval process. It’s also in charge of updating the custom status of the instance as it progresses.

Get Status Function

This is an Http function that allows clients to get the status of an instance of a Durable Function orchestration. You can get more details of this implementation here.

In many of the samples, they explain you how to use the CreateCheckStatusResponse method to generate the response to the client. However, make sure that that you fully understand what this returns. Given that the returned payload includes management endpoints with their corresponding keys, by providing that, you would allow the clients not only to get the status of a running instance, but also to 1) get the activity execution history, 2) get the outputs of the already executed activity functions, 3) send external events to the orchestration instance, and 4) terminate that instance. If you don’t want to give those privileges to the clients, you need to expose an http function that returns only the minimum information required. In this wrapper function, you could also enrich the response if need be. If you only need to send the status back, I would recommend you to use the GetStatusAsync method instead. 

The sample function below makes use of the GetStatusAsync method of the orchestration client, and leverages the location and retry-after Http headers.

Wrapping Up

In this post, I’ve shown how to implement Asynchronous Http APIs using Durable Functions with the polling consumer pattern. I’ve also explain the differences between using CreateCheckStatusResponse and GetStatusAsync methods. The former prepares a response which expose different management endpoints with a key that allow clients to do much more than just getting the status, while the latter just returns the instance status. I hope you’ve enjoyed and found useful this post.

If you are interested in more posts about patterns on Durable Functions, you can also check out:

Happy clouding!

Cross-posted on Deloitte Platform Engineering Blog.
Follow me on @pacodelacruz.

Advertisements

Azure Durable Functions Pattern: Approval Workflow with SendGrid

Introduction

Durable Functions is a new (in preview at the time of writing) and very interesting extension of Azure Functions that allows you to build stateful and serverless code-based workflows. The Durable Functions extension abstracts all the state management, queueing, and checkpoint implementation commonly required for an orchestration engine. Thus, you just need to focus on your business logic without worrying much on the underlying complexities. Thanks to this extension, now you can:

  1. Implement long-running serverless code-based services beyond the current Azure Function limitation of 10 minutes (as long as you can break down your process into small nano-services which can be orchestrated);
  2. Chain Azure functions, i.e., call one function after the other and pass the output of the first one as an input to the next one (Function chaining pattern);
  3. Execute several functions asynchronously and then continue the workflow when any or all of the asynchronous tasks are completed (Fan-out and Fan-in pattern);
  4. Get the status of a long-running workflow from external clients (Async HTTP APIs Pattern);
  5. Implement the correlation identifier pattern to enable human interaction processes, such as an approval workflow (Human Interaction Pattern) and;
  6. Implement a flexible recurring process with lifetime management (Monitoring Pattern).

It’s worth noting that Azure Durable Functions is not the only way to implement stateful workflows in a serverless manner on Azure. Azure Logic Apps is another awesome platform, core component of the Microsoft Azure iPaaS, that allows you to build serverless and stateful workflows using a designer. In a previous post, I showed how to implement the approval workflow pattern on Logic Apps via SMS messages leveraging Twilio.

In this post, I will show how to implement the Human Interaction Pattern on Azure Durable Functions with SendGrid. You will see on the way that this implementation requires other Durable Functions patterns, such as, function chaining, fan-out and fan-in, and optionally the Async HTTP API Pattern.

Scenario

To illustrate this pattern on Durable Functions, I will be using a fictitious cat model agency called Furry Models Australia. Furry Models is running a campaign to attract the most glamorous, attractive, and captivating cats in Australia. They will be receiving photos of all aspiring cats and they need a streamlined approval process to accept or reject those applications. Furry Models want to implement this in an agile manner with a short time-to-market and with a very cost-effective solution. They know that serverless is the way to go!

11 Join Us

Pre-requisites

To build this solution, we will need:

  • SendGrid account. Given that Azure Functions provides an output binding for SendGrid to send emails, we will be relying on this service. In case you want to implement this solution, you would need a SendGrid account. Once you sign up, you need to get your API Key, which is required for the Azure binding. You can get more information about the SendGrid binding for Azure Functions and how to use it here.
  • An Azure Storage Account: The solution requires a Storage Account with 3 blob containers: requestsapproved, and rejected. The requests container should have public access level so blobs can be viewed without a SAS token. For your own solution, you might want to make this more secure.

Solution Overview

The picture below shows an overview of the approval workflow solution I’ve build based on Durable Functions.

Pictures of the aspiring cats are to be dropped in an Azure storage blob container called requests. At the end of the approval workflow, pictures should be moved to the approved or rejected blob containers accordingly.

20 Solution Overview

The steps of the process are described as follows:

  1. The process is being triggered by an Azure Function with the BlobTrigger input binding monitoring the requests blob container. This function also implements the DurableOrchestrationClient attribute to instantiate a Durable Function orchestration
  2. The DurableOrchestrationClient starts a new instance of the orchestration.
  3. Then, the Durable Function orchestration calls another function with the ActivityTrigger input binding, which is in charge of sending the approval request email using the SendGrid output binding.
  4. SendGrid sends the approval request email to the (cat) user.
  5. Then, in the orchestration, a timer is created so that the approval workflow does not run forever, and in case no approval is received before the timer finishes the request is rejected.
  6. The (cat) user receives the email, and decides whether the aspiring cat deserves to join Furry Models or not, by clicking the Approve or Reject button. Each button has a link to an HttpTrigger Azure Function which expects the selection and the orchestration instanceId as query params
  7. The HttpTrigger function receives the selection and the orchestration instanceId. The function checks the status of the orchestration instance, if it’s not running, it returns an error message to the user. If it’s running, it raises an event to the corresponding orchestration instance.
  8. The corresponding orchestration instance receives the external event.
  9. The workflow continues when the external event is received or when the timer finishes; whatever happens first. If the timer finishes before a selection is received, the application is automatically rejected.
  10. The orchestration calls another ActivityTrigger function to move the blob to the corresponding container (approved or rejected).
  11. The orchestration finishes.

A sample of the email implemented is shown below.

22b Sample Email

The Solution

The implemented solution code can be found in this GitHub repo. I’ve used the Azure Functions Runtime v2. I will highlight some relevant bits of the code below, and I hope that the code is self-explanatory 😉:

TriggerApprovalByBlob.cs

This BlobTrigger function is triggered when a blob is created in a blob container and starts the Durable Function ochestration (Step 1 above)

OrchestrateRequestApproval.cs

This is the Durable Function orchestration which handles the workflow and is started by the step 2 above.

SendApprovalRequestViaEmail.cs

ActivityTrigger function which sends the approval request via email with the SendGrid output binding (Step 3 above).

ProcessHttpGetApprovals.cs

HttpTrigger function that handles the Http Get request initiated by the user selection (click) on the email (Step 7 above).

MoveBlob.cs

ActivityTrigger function that moves the blob to the corresponding container (Step 10 above).

local.settings.json

These are the settings which configure the behaviour of the solution, including the storage account connection strings, the SendGrid API key, templates for the email, among others. You would need to implement these as app settings when deploying to Azure

Wrapping up

In this post, I’ve shown how to implement an Approval Workflow (Human Interaction pattern) on Azure Durable Functions with SendGrid. Whether you wanted to learn more about Durable Functions, to implement a serverless approval workflow or you run a cat model agency, I hope you have found it useful 🙂 Please feel free to ask any questions or add your comments below.

Happy clouding!

Follow me on @pacodelacruz

Cross-posted on Deloitte Platform Engineering Blog

Ten-point checklist when migrating from BizTalk to Azure iPaaS (Logic Apps)

Summary

Business are evolving increasingly fast, and IT can be an enabler or a deterrent of this evolution. IT changes should always bring business value and never compromise core business needs.

As part of becoming more agile, a common concern for many of our customers is the transition from on-premises integration platforms to a cloud or hybrid solution, in particular migrating their BizTalk Server environments to the Microsoft Azure Integration Platform-as-a-Service (iPaaS), which is based on Azure Logic Apps.

There are a number of reasons you might want to consider migrating your BizTalk solutions to the Microsoft Azure iPaaS, including:

  1. Enabling or supporting the digital transformation journey. Azure services, like Logic Apps, Azure Functions and API Management allow you to expose and consume modern APIs, which are key enablers of digital transformation.
  2. Reducing your OpEx. Significantly reduce your IT operation and licensing costs by leveraging PaaS and serverless components.
  3. Gaining Agility: Azure allows you to deliver business value in weeks, instead of months. Not only because of the tooling but also because of the capabilities of the platform and availability of hundreds of connectors.
  4. Unlocking new business solutions: With Azure, new business solutions are possible. From asynchronous messaging of Service Bus to eventing of Event Grid, to smart solutions with Stream Analytics, Cognitive Services and Machine Learning, to monetisation of APIs with API Management, to advanced monitoring and alerting with OMS and Log Analytics, to name a few.

With all of that said, supporting your digital transformation leveraging integration can sometimes be challenging. Therefore, it’s imperative you start early, plan thoroughly and implement well to avoid unnecessary complications.

To get the full article, download the document from my employers site with the ten-point checklist to get your BizTalk to Azure iPaaS transition in good shape.