Azure AI & Microsoft Foundry Courses and Certifications

A Lake House is a blend of the terms Data Lake and Data Warehouse. The idea is to get the best of both worlds, and you get that by using Delta tables. This means data is stored in files, but there is a transaction log attached to the files, so changes can be handled seamlessly.

The log also enables time travel, so you can easily restore earlier versions of a table. Databricks and Synapse Analytics are two examples of a modern Lake House.

In the summer of 2023, Microsoft launched an entire portfolio of products and services in a single platform called Fabric. The idea is to simplify access to data and working with it throughout the process, from raw data until it is visualized in Power BI.

Fabric includes a Data Factory for ETL/ELT processes. Data can then be stored directly as files in OneLake, or you can build a data warehouse or a lakehouse on top of it.

For many years, we have been used to automatically thinking of SQL when it comes to data analysis. The world has changed. It is no longer enough to rely on SQL alone.

The volume and type of data do not always make it possible to organize data in neat tables in a database. That is why there is growing interest in the new possibilities that Databricks and Synapse Analytics offer.

Both products can store data in delta tables, and you are free to work with data in notebooks using either Python or SQL. With Python, you have a full programming language at your disposal and can run complex analyses, machine learning models and algorithms on your data.

Azure offers several options when you need to choose a data warehouse solution. The simplest and least expensive solution is an Azure SQL database. 

For large volumes of data that need to be processed efficiently within a reasonable time, Synapse Analytics is the obvious choice. It is a parallel data warehouse where you can scale how much “compute” power you have available.

Synapse Analytics is available as a standalone resource in Azure, and it is also available in a new version through Microsoft Fabric.

Data Factory is "integration services" in the cloud - i.e., the replacement for SSIS (SQL Server Integration Services). Using pipelines, you can load, copy, transform and analyze data before sending it on to a data warehouse.

Data Factory is available as a standalone resource in Azure, and similar functionality is also available under the name pipelines in Synapse Analytics. Microsoft Fabric also includes a new, modern Data Factory.

Databricks is the leading product in data science for analytics and machine learning. It is an analytics platform based on Apache Spark.

Python code is written in a notebook and run on a Spark cluster.