Azure Data Lake Storage Zone Layout Automation

After laying out the structure for our zones, my client quickly asked, is there a way we can automatically stand up this structure each time we bring a new Tenant into our solution.  With a smile, I replied it was possible!  In the short term, we would use PowerShell to stand up these folders every time we add a tenant.   Later, we would automate this work through C# that would be called by an administrative web portal.  The fundamentals would remain the same, just the orchestration of these steps would change.

In this article, I’ll walk you through the basics of using PowerShell to stand up your zone folders.

Get the Azure PowerShell Modules

Before you can use PowerShell to automate anything, you’re going to need to install some modules. Get them here. Or you can open a new Powershell session and run the following code to get the one module we’ll need for setting up folder structures in Azure Data Lake Storage.

Save-Module -Name AzureRM.DataLakeStore -Path .

After the download is complete, run the following:

Install-Module -Name AzureRM.DataLakeStore -Scope CurrentUser -force

Now, we can get to work!

Authentication

For this demo, just use Login-AzureRmAccount to authenticate your PowerShell session to your Azure account.  For real automation, you’re going to want to save your credentials in windows Cred store, and pull them out whenever this script runs.

After logging in, we need to select our Data Lake Storage account.

Get-AzureRmDataLakeStoreAccount -Name $adls -ResourceGroupName $rg;

$adls stores the name of our storage account, and $rg is the name of the resource group that it’s in.  After selecting our storage account, we can start making folders there.

The Simple Secret

All we do in this script is check to see if the folder exists, and if it doesn’t, we create it!

$adls is still the name of our storage account.  $folder is the name of the folder we want to create.

if(!$(Test-AzureRmDataLakeStoreItem -Account $adls -Path $folder)) {
     New-AzureRmDataLakeStoreItem `
          -Account $adls `
          -Path $ParentFolder `
          -Folder;

}

So, if we want to create several folders for each tenant, you could simply store them all in a list, and call the above code in a loop for each item in your list.  Let it run, and then you’ll have all the folders in the structure you need!  Pretty easy, right?  Next time, I’ll answer the question: “Is it possible to apply Acess Control Lists to these new folders too?”

Leave a Reply

Your email address will not be published. Required fields are marked *