Skip to content

shannonlowder.com

Menu
  • About
  • Biml Interrogator Demo
  • Latest Posts
Menu

Author: slowder

Reporting and Dashboarding in Databricks

Posted on December 15, 2022December 19, 2022 by slowder

As a SQL Server user moving into Databricks, reporting and dashboarding can be a more manageable learning task. It all begins by changing to the SQL Persona in Databricks. Click the persona icon from the upper left menu, and choose SQL. Once in the SQL persona, you can begin querying any data your workspace has…

Continue reading

Data Engineering for Databricks

Posted on December 13, 2022December 20, 2022 by slowder

Since Databricks is a PaaS option for Spark and Spark is optimized to work on many small files, you might find it odd that you have to get your sources into a file format before you see Databricks shine. The good news is Databricks has partnered with several different data ingestion solutions to ease loading…

Continue reading

High-level Databricks Compute

Posted on December 5, 2022December 20, 2022 by slowder

Last time, I compared SQL Server storage to Databricks storage. This time, let’s compare SQL Server compute to Databricks compute. How SQL Server Processes a Query When you submit your workload to SQL Server, the engine will first parse it to ensure it’s syntactically correct. If it’s not, then it fails and returns right away….

Continue reading

High-level Databricks Storage

Posted on November 28, 2022November 28, 2022 by slowder

Understanding how Databricks works with storage can go a long way toward improving your use of Databricks. Storage In SQL Server, we store our data in two highly structured file types. The first type of file is a data file; these have an extension of MDF or NDF. You get one MDF per database, but…

Continue reading

Databricks for SQL Professionals

Posted on November 18, 2022November 19, 2022 by slowder

I’ve been a Microsoft Data professional for over 20 years. Most of that time I’ve spent in the SQL Server stack, the core query engine, SSIS, SSRS, and a little SSAS. But times changed, and the business problems grew more complex. As they did, I looked at other technologies to try and answer those questions….

Continue reading

Notebooks Explore Data

Posted on October 22, 2022November 14, 2022 by slowder

On a recent engagement, I was asked to provide best practices. I realized that many of the best practices hadn’t been collected here, so it’s time I fix that. The client was early in their journey of adopting Databricks as their data engine, and a lot of the development they were doing was free-form. They…

Continue reading

The Big Cost in Data Science

Posted on October 24, 2018November 13, 2022 by slowder

You hear time and time again how 50 to 80 Percent of Data science projects is spent on data wrangling munging and transformation of raw data into something usable.  For me personally. I’ve automated a lot of those steps. I built tools over the last 20 years that help me do more in less time….

Continue reading

NOAA Radar and Severe Weather Data Inventory

Posted on August 29, 2018November 13, 2022 by slowder

After I finished evaluating the Storm Events Database from NOAA, I was convinced we needed to look for machine recorded events. When you start poking around the NOAA site looking for radar data, you’ll find a lot of information about how they record this data in binary block format.  Within this data, you’ll find measurements…

Continue reading

Data Quaity Issues

Posted on August 22, 2018November 13, 2022 by slowder

This entry picks up the story behind my first data science project predicting hail damage to farms. In this article we identify data quality issues in our first data source. Property and Crop Damage In the NOAA documentation these two columns were recorded to say how much property and crop damage occurred in a given…

Continue reading

Data Science Project 1: Predicting Hail Damages

Posted on August 15, 2018November 13, 2022 by slowder

Early on in my new role I was asked to find out how risky it was to offer hail insurance for a given property.  If you haven’t worked with insurance before here’s the basics.  You’re placing a bet that says I’m betting something bad is going to happen.  The insurer is betting that it won’t…

Continue reading
  • Previous
  • 1
  • 2
  • 3
  • 4
  • 5
  • …
  • 46
  • Next
  • Career Development
  • Data Engineering
  • Data Science
  • Infrastructure
  • Microsoft SQL
  • Modern Data Estate
  • Personal
  • Random Technology
© 2023 shannonlowder.com | Powered by Minimalist Blog WordPress Theme