Typically you notice that there are two dimensions / viewpoints when it comes to monitoring. On one side there is a team that wants to view everything related to the “infrastructure”, like for instance the kubernetes cluster. On the other hand, there is the typical application performance monitoring that starts from the application side. Sadly enough, in a lot of cases, those two are separated islands… 😦
As you might know, on the Azure front you can do Application Performance Monitoring with Application Insights and there is like a really awesome integration with Azure Monitor (“Log Analytics”) from the container space (kubernetes). Though I see you thinking it… Two separate solutions. Though, what a lot of people forget, is that they are actually using “Log Analytics” under the hood. And… That you can query across workspaces in Log Analytics! Which means that you can join the two and have an aggregated view to span both worlds.
Let’s take a look!
For this test, I’ve created a k8s cluster which is linked to a separate log analytics work-space. Where next to it, there is an application (Azure Function) inside of a docker container that is linked to Application Insights.
Continue reading “Unified monitoring view in Kubernetes : Linking infrastructure monitoring with application monitoring”
This week Ignite kicked off with a series of announcements (as always). One of those was “Visual Studio Online“…
In my eagerness I wanted to test drive it and check out what the developer experience would be, and if it could replace my development station in Azure. So let’s delve into it shall we?
Continue reading “Test driving the newly announced Visual Studio Online”
For today’s post we’re going to take a look at GitHub Connect … It’s the link between the On-Premises installation of GitHub Enterprise Server and the popular SaaS offering (as we all have come to love it) called GitHub. 😉
Installing GitHub Enterprise Server (on Azure)
So my journey for today started with registering for the GitHub Enterprise Trial, where I decided to install it on Azure… as my “On Premises” location.
Continue reading “Taking a look at Github Enterprise Server & Github Connect”
Earlier this week, a new capability called “Azure Private Link” (and also “Azure Private Endpoint”) went into public preview. As a nice copy & past from the documentation page ;
Azure Private Link enables you to access Azure PaaS Services (for example, Azure Storage and SQL Database) and Azure hosted customer/partner services over a Private Endpoint in your virtual network. Traffic between your virtual network and the service traverses over the Microsoft backbone network, eliminating exposure from the public Internet. You can also create your own Private Link Service in your virtual network (VNet) and deliver it privately to your customers. The setup and consumption experience using Azure Private Link is consistent across Azure PaaS, customer-owned, and shared partner services.
As always, we’ll take this one out for a spin! For this we’ll see if we can access a storage account privately (from a virtual machine) over the VNET.
Continue reading “Hardening your storage account with Private Link / Endpoint”
A few days ago the preview for the “User delegation SAS token” has seen the light. In today’s post, we’ll take a first glance on this new capability! Though why should we care about this feature? You can now create SAS tokens based on the scoped permissions of an AAD user, instead of linked towards the storage account key. From a security perspective this is REALLY awesome, cause you can harden the scope of a possible even more.
Continue reading “Taking the user delegation SAS tokens for a spin”
With the “Set Blob Tier” operation you can set the Access Tier of the blob object of a storage account. Now at times you know a certain object will go to tier that’s not your default access tier. Or you want to write immediately to archive. The cool thing is that AzCopy can assist you in this!
Take a look at the AzCopy flags… Here you’ll notice the “–block-blob-tier” flag. This is the one that’ll help you on writing directly to the access tier you want.
I’ve created a storage account in mint condition, where the default access tier is set to “Hot”.
Continue reading “Writing straight to the Azure Storage Access Tier you want (with AzCopy)”
There are a lot of scenario’s where organization are leveraging Azure to process their data at scale. In today’s post I’m going to go through the various pieces that can connect the puzzle for you in such a work flow. Starting from ingesting the data into Azure, and afterwards processing it in a scalable & sustainable manner.
High Level Architecture
As always, let’s start with a high level architecture to discuss what we’ll be discussing today ;
- Ingest : The entire story starts here, where the data is being ingested into Azure. This can be done via an offline transfer (Azure DataBox), or online via (Azure DataBox Edge/Gateway, or using the REST API, AzCopy, …).
- Staging Area : No matter what ingestation method you’re using, the data will end up in a storage location (which we’ll now dub “Staging Area”). From there one we’ll be able to transfer it to it’s “final destination”.
- Processing Area : This is the “final destination” for the ingested content. Why does this differ from the staging area? Cause there are a variety of reasons to put data in another location. Ranging from business rules and the linked conventions (like naming, folder structure, etc), towards more technical reasons like proximity to other systems or spreading the data across different storage accounts/locations.
- Azure Data Factory : This service provides a low/no-code way of modelling out your data workflow & having an awesome way of following up your jobs in operations. It’ll serve as the key orchestrator for all your workflows.
- Azure Functions : Where there are already a good set of activities (“tasks”) available in ADF (Azure Data Factory), the ability to link functions into it extends the possibility for your organization even more. Now you can link your custom business logic right into the workflows.
- Cosmos DB : As you probably want to keep some metadata on your data, we’ll be using Cosmos DB for that one. Where Functions will serve as the front-end API layer to connect to that data.
- Azure Batch & Data Bricks : Both Batch & Data Bricks can be directly called upon from ADF, providing key processing power in your workflows!
- Azure Key Vault : Having secrets lying around & possibly being exposed is never a good idea. Therefor it’s highly recommended to leverage the Key Vault integration for storing your secrets!
- Azure DevOps : Next to the above, we’ll be relying on Azure DevOps as our core CI/CD pipeline and trusted code repository. We can use it to build & deploy our Azure Functions & Batch Applications, as for storing our ADF templates & Data Bricks notebooks.
- Application Insights : Key to any successful application is collecting the much needed telemetry, where Application Insights is more than suited for this task.
- Log Analytics : ADF provides native integration with Log Analytics. This will provide us with an awesome way to take a look at the status of our pipelines & activities.
- PowerBI : In terms of reporting, we’ll be using PowerBI to collect the data that was pumped into Log Analytics and joining it with the metadata from Cosmos DB. Thus providing us with live data on the status of our workflow!
Now let’s take a look at that End-to-End flow!
Continue reading “Data Workflows in Azure : Taking an end-to-end look from ingest to reporting!”