Skip to content

shannonlowder.com

Menu
  • About
  • Biml Interrogator Demo
  • Latest Posts
Menu

Author: slowder

Introduction to Alexa

Posted on January 16, 2018November 14, 2022 by slowder

When I received the first pre-order email for an Amazon Echo in late 2014, I was excited by this new technology. I pre-ordered right away. The Echo became the first of many devices that tied into the Alexa Voice Service (AVS).  This voice service encapsulates both speech recognition as well as Natural Language Understanding features. …

Continue reading

U-SQL: Automating Schema on Read

Posted on November 1, 2017November 14, 2022 by slowder

Moving to U-SQL for your ETL can feel like a step back from the drag and drop functionality we have in SSIS.  But there is one great thing about your ETL being defined in text rather than a UI: you can automate it!  Today I’m going to show you how you can automate the part…

Continue reading

U-SQL and ETL Processing–Part 2

Posted on October 31, 2017November 14, 2022 by slowder

We’ve been going through a simple U-SQL script to perform some ETL processing in Azure Data Lake. Last time, we started by covering some basic syntax like variables and expressions.  Now, we’ll pick up with some transformations. SSIS v U-SQL In traditional data warehouse ETL, we’ve been spoiled by the ease of drag and drop…

Continue reading

U-SQL and ETL Processing

Posted on October 26, 2017November 14, 2022 by slowder

When you get started with Azure Data Lake Analytics and U-SQL specifically, you may get a little confused. It looks like a mash-up of T-SQL and C#. Turns out, That’s exactly what it is!  You can find lots of information on MSDN, or GitHub, or StackOverflow. Let’s get started with some basics. Variables, DataTypes, and Case…

Continue reading

Azure Data Lake Storage ACL Automation

Posted on October 25, 2017November 14, 2022 by slowder

In my last blog entry, we covered how to layout folders in your Data Lake Storage account based on a logical design.  That’s only half the battle.  You also need to set up Access Control Lists (ACLs).  Setting up controls via the Azure portal is easy, but not something you can automate. Today, we’ll jump…

Continue reading

Azure Data Lake Storage Zone Layout Automation

Posted on October 24, 2017November 14, 2022 by slowder

After laying out the structure for our zones, my client quickly asked, is there a way we can automatically stand up this structure each time we bring a new Tenant into our solution.  With a smile, I replied it was possible!  In the short term, we would use PowerShell to stand up these folders every…

Continue reading

Azure Data Lake, Step by Step

Posted on October 18, 2017November 14, 2022 by slowder

Over the next few blog posts in this series, I’m going to share with you the story of how a Data Lake project comes together. As I tell this story, I’m going to keep pointing back to traditional ETL work and to Automation techniques.  Not all of these will include Biml.  I hope to help…

Continue reading

Data Warehouse Automation is for Data Lakes too!

Posted on October 17, 2017November 14, 2022 by slowder

Last week you might have seen a tweet about a day in the life of a Data Platform Consultant.  To say the least, my days are varied.This day, in particular, I was split between building out automated ETL tests using Biml and spinning up a new Azure Data Lake.  Up until recently, I would have…

Continue reading

The File Staging Loop

Posted on October 11, 2017December 20, 2022 by slowder

As I mentioned in the overview, the largest cost in terms of time is the requesting each web page from the web server and downloading that file to disc.  That’s why this file staging loop is a separate step from the parsing and transforming step.  This loop can be as simple as two steps: get…

Continue reading

Parsing and Extracting Web Data

Posted on October 4, 2017December 20, 2022 by slowder

In the last article, I laid out the architecture to dealing with this type of data source.  This time, we’re going to get in to the basics of parsing data. A Few New Technologies Before we dive into details, let’s cover the technologies this solution rests on.  First, there’s C#. As a Microsoft data professional,…

Continue reading
  • Previous
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • …
  • 46
  • Next
  • Career Development
  • Data Engineering
  • Data Science
  • Infrastructure
  • Microsoft SQL
  • Modern Data Estate
  • Personal
  • Random Technology
© 2023 shannonlowder.com | Powered by Minimalist Blog WordPress Theme