Azure : Mixing LogicApps & Functions as a periodic data retriever

Introduction

There are various articles/blogs/etc that compare logicapps vs flow vs functions vs azure automation, etc… Though there was one use case where I often struggled what to use ;

What to use when I want to retrieve a file from X on a Y timed interval?

Azure Functions? Great abstraction, though the output files have random names. Sometimes / Often I want to be able to control that.

Flow? Doesn’t allow the customization I was looking for. More to integrate existing / popular services.

Azure Automation? Very good and gets the job done. The only downside, you need to code a lot of logic yourself.

Azure Logic Apps? Shows potential, but doesn’t let you include custom functions. Or does it…?!?

You can link Azure Functions to Logic Apps and create some the flow I was looking for.

 

The Flow

So what do I want to do?

  • On a daily basis
  • Retrieve content from an authenticated API
  • Save the content to a Blob storage

And afterwards I’ll use other services to process that data. 🙂

 

The Proof-of-Concept

What do I want to achieve? On a daily basis I want to retrieve data from a service provider that serves sports data. And if you are looking for such a thing, check out MySportsFeed! So back to our proof-of-concept; how will this look in Logic Apps?

2017-01-20-21_10_27-inputs-microsoft-azure

Let’s dissect this flow…

 

Timed Interval / Trigger

So I want this flow to run on a daily basis… We’ll be using “Recurrence” as first step in our Logic App.

 

2017-01-20-21_01_20-logic-apps-designer-microsoft-azure

 

Content Retrieval

For the retrieval of the content, I’ll be using an HTTP (GET) call (with “basic” authentication, as supported by the API)

2017-01-20-21_02_42-logic-apps-designer-microsoft-azure

You’ll immediately notice that the URI is the content of one of a function… That is correct! I’ll be using Azure Functions to generate the parameters I need.

 

Functions

How do I go about with the Functions? Add an action and select “Azure Functions” ;

2017-01-20-21_01_46-logic-apps-designer-microsoft-azure

Click “Choose an Azure Function”

2017-01-20-21_01_56-logic-apps-designer-microsoft-azure

And … you’ll never guess this one … choose your Azure function! 😉

First select the plan, I had two here, where I’ll be using the one of “kvaesnhl”.

2017-01-20-21_02_04-logic-apps-designer-microsoft-azure

Then I’ll select the function that I want, being the one that generates the url for today’s scoreboard.

2017-01-20-21_02_22-logic-apps-designer-microsoft-azure

And one more thing… Afterwards, be sure to enter an input. Otherwise you’ll get errors. I did this by using a “dummy”-input ;

2017-01-20-21_06_23-inputs-microsoft-azure

 

Function Details

For this flow, I used two functions ;

  • One to generate my daily url (with the data provider)
    2017-01-20-21_03_14-function-app-microsoft-azure
    2017-01-20-21_03_36-function-app-microsoft-azure
  • One to generate my daily file name (for the blob storage)
    2017-01-20-21_10_52-function-app-microsoft-azure
    2017-01-20-21_11_03-function-app-microsoft-azure

    And to be complete, I also used “kudu” to install the needed nodejs module “dateformat” ;

2017-01-20-21_03_56-diagnostic-console

 

 

Saving the content onto Blob Storage

And for the last step… I’ll save the content (body) of the HTTP call with the name I generated in my second function.

2017-01-20-21_09_55-inputs-microsoft-azure

The Result!

That went quite smooth! Let’s run this baby… And we’ll see all steps receive a green check. Once we go to the storage account, we’ll see that the file has been created!


2017-01-20-21_11_24-nvidia-geforce-overlay

And it contains the needed information…

2017-01-20-21_11_59-20170120-csv-excel

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.