Did you know almost every piece of software depends on OpenSource? Not sure… What libraries is your software using? Bingo! 😉
Now we all know that package management can be a true hell. Tracking everything and ensure you are up-to-date to achieve the needed security level is hard. Next to that, there is always the risk that your build will break to moving to a library version.
What if we could enhance that flow a bit? You guessed it… Today’s post will be around how we can leverage native GitHub features to help us in this area!
Let’s hit the slopes!
For this walk-through, we’ll use the following ;
- an existing code repository, where we’ve forked CoreUI’s VueJS repo
- GitHub’s actions to run a workflow on every pull request
- GitHub’s automated security feature that will send pull requests to us when it detects security issues
Want to test this one out or follow along? Browse to the following sample repository ; https://github.com/beluxappdev/CoreUI-VueJS-GitHubSecurityDemo
So let’s fork this sample repository!
Continue reading “Improving the security & compatibility aspects of package management with native GitHub features”
For today’s post we’re going to take a look at GitHub Connect … It’s the link between the On-Premises installation of GitHub Enterprise Server and the popular SaaS offering (as we all have come to love it) called GitHub. 😉
Installing GitHub Enterprise Server (on Azure)
So my journey for today started with registering for the GitHub Enterprise Trial, where I decided to install it on Azure… as my “On Premises” location.
Continue reading “Taking a look at Github Enterprise Server & Github Connect”
Earlier this week, a new capability called “Azure Private Link” (and also “Azure Private Endpoint”) went into public preview. As a nice copy & past from the documentation page ;
Azure Private Link enables you to access Azure PaaS Services (for example, Azure Storage and SQL Database) and Azure hosted customer/partner services over a Private Endpoint in your virtual network. Traffic between your virtual network and the service traverses over the Microsoft backbone network, eliminating exposure from the public Internet. You can also create your own Private Link Service in your virtual network (VNet) and deliver it privately to your customers. The setup and consumption experience using Azure Private Link is consistent across Azure PaaS, customer-owned, and shared partner services.
As always, we’ll take this one out for a spin! For this we’ll see if we can access a storage account privately (from a virtual machine) over the VNET.
Continue reading “Hardening your storage account with Private Link / Endpoint”
A few days ago the preview for the “User delegation SAS token” has seen the light. In today’s post, we’ll take a first glance on this new capability! Though why should we care about this feature? You can now create SAS tokens based on the scoped permissions of an AAD user, instead of linked towards the storage account key. From a security perspective this is REALLY awesome, cause you can harden the scope of a possible even more.
Continue reading “Taking the user delegation SAS tokens for a spin”
With the “Set Blob Tier” operation you can set the Access Tier of the blob object of a storage account. Now at times you know a certain object will go to tier that’s not your default access tier. Or you want to write immediately to archive. The cool thing is that AzCopy can assist you in this!
Take a look at the AzCopy flags… Here you’ll notice the “–block-blob-tier” flag. This is the one that’ll help you on writing directly to the access tier you want.
I’ve created a storage account in mint condition, where the default access tier is set to “Hot”.
Continue reading “Writing straight to the Azure Storage Access Tier you want (with AzCopy)”
In today’s post we’ll go through how you can setup an SPA (Single Page Application) to access the data pane of an Azure Storage account. For this I’ll be using NuxtJS (a Vue.js framework) for my boiler plating, and will rely on the its generic Oauth2 authentication library. The awesomeness here is that the bearer tokens will automatically be injected in your API calls (Axios). Though you can use this post as inspiration on how to get things working in another framework too!
Why this post? It has has been on my personal todo list for a while due to many reasons. One of which is to remove the barrier of using Azure Storage in this scenario. I find that it makes a lot of sense in way too many use cases. Though I must admit we’ve failed to reduce the learning curve in this scenario! You suddenly find yourself in flux between the complexities of AAD and the way how Azure Storage handles the presented JWT tokens. That being said, this is actually the only correct (secure) way of talking from an SPA to Azure Storage. DO NOT use storage keys or SAS tokens in your SPA. It’s like leaving your master keys out on the porch of your house. AAD is the correct way, and actually… With an SPA, the only correct authentication flow is actually Implicit Flow.
Sample Code Repository
If you are looking for the sample code used for this post ; https://github.com/kvaes/TasmanianTraders-NuxtJS-AzureStorageExample
Continue reading “How to talk to the Azure Storage APIs from a Single Page Webapplication (NuxtJS/VueJS) by using AAD (Oauth2 Implicit Flow)”
There are a lot of scenario’s where organization are leveraging Azure to process their data at scale. In today’s post I’m going to go through the various pieces that can connect the puzzle for you in such a work flow. Starting from ingesting the data into Azure, and afterwards processing it in a scalable & sustainable manner.
High Level Architecture
As always, let’s start with a high level architecture to discuss what we’ll be discussing today ;
- Ingest : The entire story starts here, where the data is being ingested into Azure. This can be done via an offline transfer (Azure DataBox), or online via (Azure DataBox Edge/Gateway, or using the REST API, AzCopy, …).
- Staging Area : No matter what ingestation method you’re using, the data will end up in a storage location (which we’ll now dub “Staging Area”). From there one we’ll be able to transfer it to it’s “final destination”.
- Processing Area : This is the “final destination” for the ingested content. Why does this differ from the staging area? Cause there are a variety of reasons to put data in another location. Ranging from business rules and the linked conventions (like naming, folder structure, etc), towards more technical reasons like proximity to other systems or spreading the data across different storage accounts/locations.
- Azure Data Factory : This service provides a low/no-code way of modelling out your data workflow & having an awesome way of following up your jobs in operations. It’ll serve as the key orchestrator for all your workflows.
- Azure Functions : Where there are already a good set of activities (“tasks”) available in ADF (Azure Data Factory), the ability to link functions into it extends the possibility for your organization even more. Now you can link your custom business logic right into the workflows.
- Cosmos DB : As you probably want to keep some metadata on your data, we’ll be using Cosmos DB for that one. Where Functions will serve as the front-end API layer to connect to that data.
- Azure Batch & Data Bricks : Both Batch & Data Bricks can be directly called upon from ADF, providing key processing power in your workflows!
- Azure Key Vault : Having secrets lying around & possibly being exposed is never a good idea. Therefor it’s highly recommended to leverage the Key Vault integration for storing your secrets!
- Azure DevOps : Next to the above, we’ll be relying on Azure DevOps as our core CI/CD pipeline and trusted code repository. We can use it to build & deploy our Azure Functions & Batch Applications, as for storing our ADF templates & Data Bricks notebooks.
- Application Insights : Key to any successful application is collecting the much needed telemetry, where Application Insights is more than suited for this task.
- Log Analytics : ADF provides native integration with Log Analytics. This will provide us with an awesome way to take a look at the status of our pipelines & activities.
- PowerBI : In terms of reporting, we’ll be using PowerBI to collect the data that was pumped into Log Analytics and joining it with the metadata from Cosmos DB. Thus providing us with live data on the status of our workflow!
Now let’s take a look at that End-to-End flow!
Continue reading “Data Workflows in Azure : Taking an end-to-end look from ingest to reporting!”
Last summer I posted about taking a look under the hood of the Azure Active Directory integration for a Linux Virtual Machine. For today, let’s take it a bit further… What if we would want to pre-provision a set of UIDs (User IDs) & GIDs (Group IDs) on a range of virtual machines for cross machine consistency. Let’s say, we would want to make use of an NFS drive and use the same UID/GID across all those boxes. Can we do that with the AAD extension? If so, how can we do it? Let’s hope we can… Otherwise it’ll become a rather short blog post.
This post is based upon my personal experience reverse engineering how this extension works. This is by no means a support statement. If you’re a technical nut (like myself) and want to know how you can tweak this at your own doing… Then this post is for you. 😉
Continue reading “Reverse engineering the “AADLoginForLinux” in order to tweak proactive user configuration”
Proxy servers are a very common thing in a lot of enterprises. They are used so that people cannot directly access the internet, and additional management capabilities to the flow (logging, authentication, …). Now that sounds very dandy, though what about those non-browser-based tools? How can we ensure that tools like Azure CLI, Azure Powershell & AzCopy work with our “beloved” enterprise proxy? That’ll be the topic for today!
What will we be doing today? I’ve setup a proxy server in my own lab… Basically deployed a Squid proxy by means of a container.
Next up, I’m going to use the three earlier mentioned tools on both Linux (WSL) & Windows, and see what needs to be done to get things working. In the following screenshots you’ll typically see a “split screen”, where left is a “tcpdump” on the box running the proxy server and right will be the commands on the box running the tools. If you see a lot of mumbo jumbo (network packets) on the left, that’ll mean that the proxy server was being used. Ready?!? Cool, let’s go!
Continue reading “What the proxy?!? How to use a proxy with the typical Azure tools…”
Earlier this week I received a two folded question ; “Does a service endpoint go over internet? As when I block the storage account tags with a NSG, my connection towards the storage account stops.” Let’s look at the following illustration ;
The first thing to mention here is that the storage account (at this time) always listens to a public IP address. The funky thing is, that in Azure, you’ll have a capability called “Service Endpoint“, which I already covered briefly in the past. For argument’s sake, I’ve made a distinction in the above illustration between the “Azure Backbone” and the “Azure SDN”. A more correct representation might have been to have said “internal” & “external” Azure Backbone in terms of the IP address space used. So see the “Azure Backbone” in the above drawing as the public IP address space. Here all public addresses reside. Where the “Azure SDN” is the one that covers the internal flows. Also be aware that an Azure VNET can only have address spaces as described by RFC1918. So why did I depict it like this? To indicate that there are different flows;
- Connections from outside of Azure (“internet”)
- Connections from within the Microsoft backbone (“Azure Backbone”)
- Connections by leveraging a service endpoint
So how does the service endpoint work?
So to answer the question stated above ;
- Q: Does a service endpoint go over “the internet”?
- A : Define “internet”…
- If you mean that it uses a public ip address instead of an internal one? Then yes.
- If you mean that it leaves the Microsoft backbone? Then no.
- If you mean that the service is accessible from the internet? Unless you open up the firewall, it won’t (by default, when having a service endpoint configured).
- Q: When I block the storage tag in my network security group (“NSG”), then the traffic stops. How come?
- A: The NSG is active on NIC level. The storage account, even when using a service endpoint, will still use the public IP. As this public IP is listed in the ranges that are configured in the service tag, you’ll be effectively blocking the service. This might be your objective… Though if you do this to “lock down the internet flow”, then you won’t achieve the requested you wanted. You should leverage the firewall functionality from the service for which the service endpoint was used.
As always, let’s do a deep dive to experience this flow! So I did set up the above drawing in my personal lab ;
Continue reading “Hardening your Azure Storage Account by using Service Endpoints”