Publishing Custom Queries of Logic Apps Execution Logs

In a previous post, I showed how to implement Business Activity Monitoring for Logic Apps. However, sometimes developers, ops, or business users want to query execution logs to get information about the processing of business messages. Whether for troubleshooting or auditing, there are some questions these personas might have, like:

  • When was a business document processed?
  • What was the content of a received document?
  • How was that message processed?

As we saw in that post, we can send diagnostic log information and custom tracked properties to Azure Log Analytics. We also saw how easy is to query those logs to get information about Logic Apps execution and messages processed. Now the question is, how can we publish those custom queries, so different users can make use of them? In this post, I’ll show one easy way to do that.

1. Tracking the relevant custom properties and sending data to Log Analytics.

The first thing to do is to track the relevant custom properties we need for our queries as tracked properties in our Logic App workflow. Then you need to configure the Logic App workflow to send diagnostics information to Azure Log Analytics. You can follow the instructions on my previous post to perform those steps.

2. Creating the queries to get the information our users need

Once the information is being logged on Log Analytics, we need to create the queries to give the users the information they need. For that, first we need to open the Azure Log Analytics Portal. To open the portal we need to

  • Go to the Log Analytics Resource on the Azure Portal
  • Go to Log Search
  • Click on Analytics

And now you are ready to create your own queries.

20 Open Log Analytics

Based on the tracked properties of the Logic App workflow shown in my previous post, I wrote this query to get all orders processed in the time range selected. This query returns, order number, total, date, channel, customer Id, the name of the Logic App workflow which processed this message, and the workflow run id. These last 2 columns would be quite handy for troubleshooting.

21a Query and Results

3. Saving the custom Azure Log Analytics query

Once we have the query ready, we can save it and export it, so later we can publish it. To do that, we follow the steps below

  • Click the Save button
  • Give a name and category to the query. The category is quite useful for searching among all saved queries.
  • Then we click the Export button
  • And select the Share a Link to Query option, so the link to the query is saved in the clipboard.

21 Save and Export Query

4. Publishing the custom Azure Log Analytics query to the users

After we have gotten the link to the query, we can publish it in the same Dashboard we created for our BAM charts described in my previous post. We need to:

  • Edit the shared Azure Dasboard
  • Add the Markdown tile.
  • Add the markdown text which contains the link to the query created above.

Now the users will have all the charts and custom queries they need in one single place!

22 Add Query to Dashboard

Making easier to users to get to the workflow run logs of a particular business message on Logic Apps.

Logic Apps provide a very detailed view of the execution of the workflows. However, I’ve been asked so many times to make easier to users to get the run details of a particular business message. Here is a tip on how to do it.

First, we need to create a query to get the runId of the workflow instance that processed the message. I created this query to get those details for the orders submitted by a particular user.

  • Once we have that query, we publish it to the same markdown tile in our dashboard.
  • We also add the link to the workflow Azure resource to the same dashboard tile.

Now users can query the orders submitted by a user, get the workflow run id, and get the workflow run details in very few clicks.

23 Query Logic App Instance by Business Id

Wrapping-up

In this post, we’ve seen how to create and publish custom queries of Logic Apps execution logs. We’ve also seen how to make easier to users to get the workflow run details of the processing of a particular business message. Now you should be ready to start creating and publishing your own custom queries and creating amazing monitoring and tracking dashboards for your Logic Apps solutions.

I hope you’ve got some useful tips from this post and you’ve enjoyed it. Feel free to leave your questions or comments below,

Follow me on @pacodelacruz

Cross-posted on Deloitte Platform Engineering Blog

Advertisements

Business Activity Monitoring on Azure Logic Apps with Azure Log Analytics

Introduction

Azure Logic Apps provide built-in monitoring tools that allow you to check the run history (including all inputs and outputs of triggers and actions), trigger history, status, performance, etc. Additionally, you can enable diagnostic logging on your Logic Apps and send all these runtime details and events to Azure Log Analytics. You can also install the Logic Apps Management Solution on OMS, which gives you a very rich aggregated view and charts of all your logic apps that are being monitored.

All these tools are great for developers or system administrators, who want to monitor and troubleshoot the execution of the workflows. However, sometimes we need to track and monitor more business-related information. Additionally, it’s quite common that business users want to monitor, with a business perspective, what’s happening at the integration layer.

In this post, I will show how to implement tracking capabilities for business-related information and how to create a Business Activity Monitoring (BAM) dashboard for Logic Apps.

Scenario

In a previous post, I introduced a fictitious company called “Farm to Table”, which provides fresh produce drone delivery. This company has been leveraging Logic Apps to implement their business processes, which integrate with multiple systems on the cloud. As part of their requirements they need to monitor the business activity flowing through this integration solution.

“Farm to Table” want to be able to monitor the orders they are receiving per channel. At the moment, customers can place orders via SMS, a web online store, and a mobile app. They also want to be able to track orders placed using customer number and order number.

Solution

Prerequisites

To be able to track and monitor business properties and activity on Logic Apps, there are some prerequisites:

  1. Azure Log Analytics workspace. We require an Azure Log Analytics workspace. We can use a workspace previously created or create a new one following the steps described here.
  2. Enable diagnostic logging to Azure Log Analytics on the Logic App which processes the messages we want to track, following the instructions detailed here.
  3. Install the Logic App Management solution for OMS on the Azure Log Analytics.

Adding Tracking Properties to the Logic App Workflow

Once enabled diagnostic logging, by default, Logic Apps tracks all workflow instances and actions on Azure Log Analytics. This tracking can very easily be extended using Tracked Properties on the workflow actions. In the tracked properties, we can include business related data; for example, in this scenario we could track customer Id, order number, order total, and channel.

“Farm To Table” has implemented an HTTP Triggered Logic App workflow that receives orders from different channels, validates the orders, maps it to a canonical model, enriches the message, and then puts them into a Service Bus Queue. The order canonical model processed by this workflow follows the schema of the instance below:

To track business properties of the orders, we will add the tracked properties to the action that sends the message to Service Bus. It’s noteworthy that when we use tracked properties within a Logic App action, we can only use the trigger input and the action’s inputs and outputs.

In the Logic App action that sends the message into a Service Bus queue, we will add the tracked properties to include the customer Id, order number, order Total, date, and channel. I’m also adding a flag to simplify my queries, but that’s optional. The code below shows the trackedProperties section added to our workflow action.

Once we have started tracking those properties and we have information already logged on Azure Log Analytics, we can start querying and creating charts for our own Business Activity Monitoring Dashboard.

Querying Azure Log Analytics

Let’s start querying what we are tracking from the Logic Apps on Log Analytics. Bear in mind that there is a delay between the execution of the Logic App and the log being available on Log Analytics. Based on my experience, it usually takes anywhere between 2 and 10 minutes. You can find detailed documentation on the Log Analytics query language here.

The query below returns the custom data I’m tracking on the Logic App workflow. When building your queries, it’s worth noting that

  • Workflow actions are being logged using the AzureDiagnostics Log Type and with the WorkflowRuntime category.
  • Logic Apps prepend the prefix “trackedProperties_” to each property and append a suffix to declare its type.

This query should return a result set as the one below:

Additionally, we can add filters to our queries, for instance, to get all orders by the CustomerId, we could use a query as follows:

Creating a Monitoring Dashboard.

Before, Microsoft suggested to create OMS custom views and queries for this purpose. The steps to create a custom OMS dashboard for Logic Apps are described here. However, after the upgrade of OMS to the new Log Analytics Query Language (previously known as Kusto query language), the recommended approach is now to use the new Azure Log Analytics portal, create queries and charts, and pin them to a shared Azure Dashboard. If have created both, custom OMS dahsboards and custom Log Analytics Azure Dashboards, you must agree that shared Azure Dashboards and Log Analytics charts are much more user friendly.

The steps to create and shared an Azure Dashboard to include Log Analytics data and charts are described here. We will follow these steps to create our own Business Activity Monitoring Dashboard as a shared Azure Dashboard. I won’t repeat what’s in Microsoft’s documentation, I’ll just show how I’m creating the charts to be added in the Dashboard.

Order Count by Channel

“Farm to Table” want to have a chart with the order count summarised by channel for the last 7 days in their BAM Dashboard. The query below returns those details.

Once we get the results, we need to select Chart and then the Doughnut option. After that, we are ready to pin our chart to the Azure shared dashboard.

15 Chart Count By Channel

Order Total by Date and Channel

The company also want to have a chart with the order total summarised by date and channel for the last 7 days in their Dashboard. The query below returns those details.

Once we get the results, we need to select Chart and then Stacked Columns. After that, we are ready to pin our chart to the Azure shared dashboard.

16 Chart Total By Date

Business Activity Monitoring Dashboard

Once we have pinned our charts, we would be able to see them in our Azure shared dashboard. These charts are very handy and allow us to dynamically visualise the data as shown below.

20 BAM Dashboard

Wrapping-up

In this post, we’ve seen how to easily track and monitor business information flowing through our Logic Apps, using Logic App native integration with OMS and Azure Log Analytics. Additionally, we’ve seen how friendly and cool the Log Analytics charts are. This gives Logic Apps another great competitive advantage as an enterprise-grade integration Platform as a Service (iPaaS) in the market.

I hope you’ve learned something useful and enjoyed this post! Feel free to post your comments or questions below

Happy clouding!

Paco

Follow me on @pacodelacruz

Cross-posted on Deloitte Platform Engineering Blog