Data Workflows in Azure : Taking an end-to-end look from ingest to reporting!

Introduction

There are a lot of scenario’s where organization are leveraging Azure to process their data at scale. In today’s post I’m going to go through the various pieces that can connect the puzzle for you in such a work flow. Starting from ingesting the data into Azure, and afterwards processing it in a scalable & sustainable manner.

 

High Level Architecture

As always, let’s start with a high level architecture to discuss what we’ll be discussing today ;

 

  • Ingest : The entire story starts here, where the data is being ingested into Azure. This can be done via an offline transfer (Azure DataBox), or online via (Azure DataBox Edge/Gateway, or using the REST API, AzCopy, …).
  • Staging Area : No matter what ingestation method you’re using, the data will end up in a storage location (which we’ll now dub “Staging Area”). From there one we’ll be able to transfer it to it’s “final destination”.
  • Processing Area : This is the “final destination” for the ingested content. Why does this differ from the staging area? Cause there are a variety of reasons to put data in another location. Ranging from business rules and the linked conventions (like naming, folder structure, etc), towards more technical reasons like proximity to other systems or spreading the data across different storage accounts/locations.
  • Azure Data Factory : This service provides a low/no-code way of modelling out your data workflow & having an awesome way of following up your jobs in operations. It’ll serve as the key orchestrator for all your workflows.
  • Azure Functions : Where there are already a good set of activities (“tasks”) available in ADF (Azure Data Factory), the ability to link functions into it extends the possibility for your organization even more. Now you can link your custom business logic right into the workflows.
  • Cosmos DB : As you probably want to keep some metadata on your data, we’ll be using Cosmos DB for that one. Where Functions will serve as the front-end API layer to connect to that data.
  • Azure BatchData Bricks : Both Batch & Data Bricks can be directly called upon from ADF, providing key processing power in your workflows!
  • Azure Key Vault : Having secrets lying around & possibly being exposed is never a good idea. Therefor it’s highly recommended to leverage the Key Vault integration for storing your secrets!
  • Azure DevOps : Next to the above, we’ll be relying on Azure DevOps as our core CI/CD pipeline and trusted code repository. We can use it to build & deploy our Azure Functions & Batch Applications, as for storing our ADF templates & Data Bricks notebooks.
  • Application Insights : Key to any successful application is collecting the much needed telemetry, where Application Insights is more than suited for this task.
  • Log Analytics : ADF provides native integration with Log Analytics. This will provide us with an awesome way to take a look at the status of our pipelines & activities.
  • PowerBI : In terms of reporting, we’ll be using PowerBI to collect the data that was pumped into Log Analytics and joining it with the metadata from Cosmos DB. Thus providing us with live data on the status of our workflow!

 

Now let’s take a look at that End-to-End flow!

Continue reading “Data Workflows in Azure : Taking an end-to-end look from ingest to reporting!”

Landscaping a Secure/Closed Loop Infrastructure in Azure with Terraform & Azure Devops

Introduction

Posts about security are always the ones that make everyone get really excited… Or maybe not everyone. 😉 Anyhow, what is typically the weakest link in any security design? Indeed, the human touch… The effects of this can range from having seen secrets to creating drift (unwanted changes vs de expected baseline). In today’s post, I’ll walk you through an example setup that aims to close some additional holes for you. How will we be doing this? By basically automating the entire infrastructure management with Azure Devops & Terraform. Now you’ll probably think, what does that have to do with security? Good response! We’re going to reduce the points to where human contact can interfere with our security measures. Though we want to do this without putting our agility at risk!

 

Blueprint

For this exercise, we’re going to leverage this blueprint ;

Continue reading “Landscaping a Secure/Closed Loop Infrastructure in Azure with Terraform & Azure Devops”

From Cloud Dev Station to Terraform landscaping in Azure

Introduction

A lot of people always keep telling me that they love Azure’s Cloud Shell. Oddly enough, I use it more occasionally and find my self using the WSL (Windows Subsystem for Linux) more. If I analyze it a bit, I recon it’s because I want to easily edit & use files with the Azure CLI (etc). Now, the Azure Cloud Shell has a way to persist files! Therefor I embarked on a small test to see what kind of workflow would work whilst working with Terraform and leveraging the Cloud Shell to apply the configurations.

 

Basic Workflow

So what did I come up with? As you know, I’m running my development workstation in the cloud. In addition, I’ve mounted the CloudDrive onto my workstation and cloned my GitHub repo to that location. Next up, I can author my files locally and afterwards push to my repository. As the local files are synced with the CloudDrive, they’ll immediately pop up in my Cloud Shell too. So I can apply them there…

Sounds great? Let’s take it for spin!

Continue reading “From Cloud Dev Station to Terraform landscaping in Azure”

Using Azure DevOps to deploy your static webpage (SPA) to Azure Storage

Introduction

To, without shame, grab the introduction of the “Static website hosting in Azure Storage” page ;

 

Azure Storage now offers static website hosting, enabling you to deploy cost-effective and scalable modern web applications on Azure. On a static website, webpages contain static content and JavaScript or other client-side code. By contrast, dynamic websites depend on server-side code, and can be hosted using Azure Web Apps.

As deployments shift toward elastic, cost-effective models, the ability to deliver web content without the need for server management is critical. The introduction of static website hosting in Azure Storage makes this possible, enabling rich backend capabilities with serverless architectures leveraging Azure Functions and other PaaS services.

 

Which, to me, sounds great! As one of my projects (VMchooser) is actually a static site (VueJS based Single Page App) that could just as well run on Azure Storage (thus reducing my cost footprint). So today we’re going to test that one out, and afterwards integrate it into our existing CI/CD pipeline (powered by Azure DevOps).

 

Continue reading “Using Azure DevOps to deploy your static webpage (SPA) to Azure Storage”

VSTS & Compiled Azure Functions – How to set up your basic CI/CD pipeline

Introduction

A question that pops up occasionally is how to setup your Azure Functions DevOps flow when you’re using C# underneath. Today’s post will be a brief one to run you through this process. If you should prefer a video on this… That exists too! Curtosiy of the app service product group.

 

Quick Howto

Let’s take a look at the build process. We have (at least, as this flow did not do any testing => “Shame on me!”) three steps in the build process ;

  • Restore the nuget packages
  • Build the solution (and create a single zip file)
  • Publish the artifact

 

So let’s take a look at one of my own builds… First I kick off with installing NuGet on my build agent (should it not already be present).

Continue reading “VSTS & Compiled Azure Functions – How to set up your basic CI/CD pipeline”

Using B-series for your Dev VM in Azure

Introduction

Earlier this week I tweeted my excitement of using an Azure B-series machine for my Dev VM in Azure. And Jan was curious to know what type I used…

Which got a response from Sven that I would probably blog on it…

So…

Continue reading “Using B-series for your Dev VM in Azure”

What does it look like to deploy a DevOps project into Azure for a Java Application?

Introduction

A few months ago “Azure Devops Projects” was released. As I haven’t had the time yet to test-drive it, I was still sceptic towards the service from a naming perspective. In full disclosure, for me DevOps is about three aspects ; People, Processes & Products (“Tools”). The last part is typically, and maybe surprisingly, the most easy part to do. That being said, as I tried this service, I must admit that this service reduces the friction to set up an end-to-end project. This is where the Azure Devops Project shines! It guides you in a step-by-step manner to set up the end-to-end project for a variety of languages and deployment methods.

 

A brief walk-through

As we all know, the proof of the pudding is in the eating. So let’s see what the flow looks like in reality?

Continue reading “What does it look like to deploy a DevOps project into Azure for a Java Application?”