My T-SQL Tuesday contribution for this month is about the using Notebooks more with Azure Data Engineering services. Which is hosted this month by Steve Jones.
Steve has invited us all to talk about how we have used, or would like to use, a Jupyter Notebook. You can find out more about the invite by clicking here or on the image below.
For those of you who are not aware, Notebooks are basically documents that can contain code and text. With the idea being that you can run code in separate sections that are known as cells. Notebooks are very popular with Data Science users and are becoming more popular elsewhere.
Using Notebooks more with Azure Data Studio
Now, I have been a bit of a fan of Jupyter Notebooks and for a while. I was really excited when they were first introduced in Azure Data Studio. Which I think is a great multi-platform product that is full of potential.
In fact, I have covered how you can use Notebooks in Azure Data Studio in multiple posts.
For example, the post I did about Azure Data Studio Insiders build here.
Plus, I did a review of the SQL Server 2019 Lab that was created by Microsoft. Because the SQL Server 2019 Lab shows how useful Notebooks can be. You can read that post in detail here.
When Big Data Clusters were first announced I was really excited about it. I had been using the original scripts in SQL Server documentation to create them.
However, as soon as I found out that there was a new wizard in Azure Data Studio Insiders build, I started to use that to test deployments as well. In fact, I used it to test a lot of deployments for a session that I use to present.
In addition, there is also demo of how to deploy a Big Data Cluster using a Notebook in a video for the Azure Advent Calendar here. Obviously all the presenters were asked to wear Christmas jumpers for our contributions.
With this in mind, if you are new to Big Data Clusters and want to do an initial deployment, I highly recommend using the wizard in Azure Data Studio. Plus, I highly recommend looking at the other wizards as well. Like the ones relating to Azure Arc.
Viewing Notebooks in Azure DevOps
In fact, at one stage I also figured out how you could render Notebooks in Azure DevOps. So that you can view them in Azure DevOps itself. I shared it rather excitedly with my co-presenter.
Using Notebooks more with Azure Data Engineering services
Anyway, I’m still a big fan of using Notebooks in Azure Data Studio. However, I will admit that these days I am using Notebooks more with Azure Data Engineering services.
For example, the Notebooks you can use within Azure Databricks and Synapse Studio. Due to my change of role.
I shared my thoughts about the increase in demand for Azure Data Engineering roles in a previous post here. Anybody who’s using Azure Data Engineering services like these will tell you how popular the Notebooks within them are.
Final word about using Notebooks more
I hope my contribution this month about using Notebooks more with Azure Data Engineering services is useful. Plus, I hope it encourages some of you to look into these services more.
Of course, if you have any comments or queries about this post feel free to reach out to me. I am also very curious to hear from people who are using SQL Server 2019 Big Data Clusters in production at the moment. Since this post has reminded me about past tests.
In addition, feel free to mention your own experiences of using Notebooks with a comment.