Getting Data into Fabric

Introduction

This lesson opens the domain Prepare data (45–50% of the DP-600 skills measured, the heaviest domain of the exam). This course targets the English syllabus effective October 19, 2026. Microsoft does not publish the question count or the question-type mix for the exam, so OnLevelUp mock exams use a 50-question practice length and an unofficial 80% practice target; neither is equivalent to the official 700/1000 scaled passing score.

Every analytics solution starts with a decision: where does the data live, how does it get to OneLake, and which engine will serve it? The exam rarely asks for definitions. It describes a source system, a freshness requirement, a team skill set and a cost constraint, and asks which ingestion method and which Fabric item fit. After this lesson you will be able to create and manage data connections, discover data with the OneLake catalog and the Real-Time hub, choose between shortcuts, mirroring, pipelines, dataflows and notebooks, pick between a lakehouse, a warehouse and an eventhouse, and explain how OneLake integration exposes Eventhouse and semantic model data to other engines.

Learning objectives

• Create, share and secure data connections, and distinguish cloud connections from on-premises and virtual network gateways.
• Use the OneLake catalog and the Real-Time hub to find and reuse data.
• Choose among shortcuts, database mirroring, copy activity in pipelines, Dataflows Gen2, notebooks and eventstreams.
• Choose a data store (lakehouse, warehouse, eventhouse) from workload requirements.
• Explain OneLake availability for Eventhouse and OneLake integration for semantic models.

1. Data connections

A connection stores the endpoint, authentication method and privacy level used by a pipeline, a dataflow, a shortcut or a mirrored database. Connections are first-class objects that you manage from Manage connections and gateways.

Key facts:

• Cloud connections reach sources that are reachable from the Fabric service. On-premises data gateway connections reach sources inside a private network. A virtual network data gateway reaches sources behind a private endpoint or inside an Azure virtual network without installing software on a VM.
• Authentication kinds include organizational account (Microsoft Entra ID), service principal, workspace identity, basic, key and SAS. Prefer identities that do not expire with a person: service principal or workspace identity.
• A connection can be shared with users or groups so they can use it without learning the secret. Sharing grants use, not edit.
• The privacy level (Organizational, Private, Public) controls how mashups in Dataflows Gen2 may combine data. A wrong level can cause the "formula firewall" error or block query folding.
• Data parameters in connections let one pipeline run against Dev, Test and Prod sources.

Example. Contoso Retail keeps its ERP in an on-premises SQL Server 2019 instance. The BI team installs an on-premises data gateway cluster of two nodes on separate machines for availability, creates one connection erp-sql-prod with a service account, and shares it with the Data Engineering group. Pipelines and dataflows reuse the same connection, so a password rotation is done once.

2. Discovering data: OneLake catalog and Real-Time hub

The OneLake catalog is the place to browse, search and filter items across the tenant that you can access: lakehouses, warehouses, semantic models, eventhouses and more. You can filter by workspace, owner, item type and endorsement (Promoted, Certified, Master data). From the catalog you can open an item, see its lineage, and reach governance information. Using it, an analyst finds a Certified semantic model instead of building a duplicate.

The Real-Time hub is the tenant-wide catalogue of streaming data: data streams (eventstreams), Microsoft sources (such as Azure Event Hubs, IoT Hub, Fabric and Azure events) and Fabric events (workspace item events, OneLake events, job events). From the hub you can get events into Fabric by connecting a source, create an eventstream, and set alerts. Use it when the question mentions streaming sources, subscribing to events or discovering existing streams.

3. Ingesting or accessing data

The central distinction on the exam is access in place (no copy) versus ingest (copy into OneLake).

3.1 Shortcuts: access without copying

A OneLake shortcut is a pointer to data stored elsewhere. It appears as a folder or a table in a lakehouse (or a KQL database), and engines read it as if it were local.

• Targets: other OneLake locations (lakehouse, warehouse, KQL database, semantic model storage), Azure Data Lake Storage Gen2, Amazon S3 and S3-compatible storage, Google Cloud Storage, Dataverse, and other supported sources, including on-premises sources through a gateway.
• Deleting a shortcut deletes only the pointer. Deleting the target removes the data for every consumer.
• Shortcuts under Tables must point to Delta (Delta Lake) data; shortcuts under Files can point to any format.
• Authorisation: for OneLake targets, the shortcut can use the caller's identity (passthrough) or a delegated identity; for external targets a connection with a stored credential is used.
• Shortcuts can be cached for some external sources to reduce egress cost and latency.

3.2 Mirroring: a continuously replicated, queryable copy

Database mirroring replicates an operational database into OneLake as Delta tables, with near real-time latency, low-cost storage and no pipeline to maintain. Supported sources include Azure SQL Database, Azure SQL Managed Instance, SQL Server, Azure Cosmos DB, Snowflake, Azure Database for PostgreSQL and others, and the list grows. Mirroring also supports metadata mirroring for catalogs such as Azure Databricks Unity Catalog (it mirrors the metadata and creates shortcuts, so the data is accessed in place).

A mirrored database item gives you a SQL analytics endpoint and Delta tables you can reach from lakehouses with shortcuts, from notebooks, and from Direct Lake semantic models. Mirroring is the right answer when the requirement says near real-time replication of an operational database with minimal engineering effort.

3.3 Pipelines: orchestrated copy

A Data Factory pipeline orchestrates activities. The Copy activity moves large volumes at scale between many sources and sinks, supports incremental patterns (watermark column, change data capture) and runs on a schedule or a trigger. Other activities run notebooks, stored procedures, dataflows and other pipelines, with control flow (ForEach, If, Until) and parameters. A Copy job is a simpler, guided way to copy with built-in incremental loading and no pipeline authoring.

3.4 Dataflows Gen2: low-code transformation

Dataflows Gen2 use Power Query Online with more than 150 connectors. They suit analysts who need to transform while ingesting, with a destination (lakehouse, warehouse, KQL database, SQL database and others). They are not the best choice for multi-terabyte movement, where Copy activity or Spark is cheaper and faster.

3.5 Notebooks and eventstreams

Notebooks (Spark, PySpark, Scala, R, plus Python notebooks) give full code control for complex logic, APIs, and large transformations. Eventstreams ingest real-time events (Event Hubs, Kafka, IoT, change data capture from databases, and sample data), apply no-code transformations and route to an eventhouse, a lakehouse, a Reflex/Activator or a custom endpoint.

3.6 Choosing the method

Worked example 1: choosing the ingestion method. Statement. Fabrikam stores 8 TB of Parquet sales files in an Amazon S3 bucket. The BI team needs to query the files in a lakehouse today, but egress costs must be low and the data engineers do not want to maintain loads. The files are partitioned as Delta tables. Reasoning. (1) The requirement is access without maintenance, which points to a shortcut, not a pipeline. (2) The data is Delta, so the shortcut can sit under Tables and be queried with Spark and the SQL analytics endpoint. (3) Egress cost is a concern when queries repeatedly read the same remote data; enabling shortcut caching on the S3 shortcut reduces repeated egress. Solution. Create an S3 shortcut under Tables with a shared connection, and enable caching. Why the others are weaker. A Copy activity duplicates 8 TB and needs scheduling; a Dataflow Gen2 is not designed for terabyte movement; mirroring applies to databases, not file stores.

4. Lakehouse, warehouse or eventhouse?

Rules of thumb:

• Need to write with T-SQL (UPDATE, MERGE-style logic, multi-table transactions) and team knows SQL: warehouse.
• Need Spark, files, notebooks, ML or unstructured data: lakehouse. Its SQL analytics endpoint is read-only for tables.
• Need time-series, logs, telemetry, near real-time queries with KQL: eventhouse.
• Many solutions use more than one: ingest streaming into an eventhouse, expose it to a lakehouse by shortcut, serve curated tables from a warehouse.

5. OneLake integration for Eventhouse and semantic models

OneLake availability (Eventhouse). When you turn on OneLake availability for a KQL database or a table, Fabric maintains a Delta Lake copy of the data in OneLake. Any engine can then read it: Spark, T-SQL through a lakehouse shortcut, Power BI Direct Lake. Notes: data is logically available without a second ingestion pipeline; schema changes such as renaming a table or altering a column type may require turning the feature off, changing and turning it on again; use it when you want analysts to use SQL or Direct Lake on streaming data. The Eventhouse endpoint provides the opposite direction: it lets KQL query lakehouse and warehouse tables.

OneLake integration for semantic models. For import-mode semantic models, you can enable OneLake integration so the imported tables are exported to OneLake as Delta tables on refresh. Other engines can then query model data without a second extract. It requires a model on a capacity that supports it and a large-format storage setting, and the data is only as fresh as the last refresh.

6. Reading the question

Scenarios in this area typically include signals:

• "without copying", "keep data in the source": shortcut.
• "near real-time replication", "operational database": mirroring.
• "orchestrate", "incremental load", "terabytes": pipeline copy.
• "analysts", "low-code", "Power Query": dataflow.
• "events", "IoT", "telemetry": eventstream and eventhouse.
• "read-write T-SQL": warehouse. "Spark and files": lakehouse.

Common pitfalls

• Treating a shortcut as a copy: if the source is deleted or its permissions change, consumers lose access.
• Placing non-Delta data under Tables in a shortcut.
• Expecting to run INSERT or UPDATE through a lakehouse SQL analytics endpoint.
• Using Dataflows Gen2 for very large movement where Copy activity is cheaper.
• Binding a pipeline to a personal account: pipelines break when the person leaves.
• Forgetting a gateway for an on-premises source, or exposing a private source with a cloud connection.
• Confusing mirroring (replicates a database) with shortcut (points to storage).

Exam tip: when a scenario states "minimal development effort" or "no ETL to maintain", eliminate pipelines and notebooks first. When it states "must not duplicate data", eliminate everything but shortcuts (and metadata mirroring).

Mini case study

Tailspin Logistics runs three sources: an Azure SQL Database for orders, scanner telemetry from 4,000 trucks arriving through Azure Event Hubs, and historical GPS archives in an ADLS Gen2 account owned by another business unit that refuses to move them. Dispatchers need a dashboard refreshed within seconds for truck positions, finance needs daily order analytics, and the data science team needs the GPS archive with Spark.

Proposed solution.

1. Mirror the Azure SQL Database: orders are replicated near real-time with no pipelines; the SQL analytics endpoint serves finance.
2. Connect the Event Hubs source through the Real-Time hub and an eventstream into an eventhouse, which supports KQL queries on telemetry in seconds.
3. Enable OneLake availability on the telemetry tables so data scientists and finance can join them with orders using SQL or Spark.
4. Create an ADLS Gen2 shortcut in a lakehouse for the GPS archive (Delta under Tables, raw files under Files), using a shared connection with a service principal.
5. Certify the resulting gold semantic model so analysts find it in the OneLake catalog.

Commentary. Each source gets the lightest mechanism that meets its latency and ownership constraint: mirroring for the operational database, streaming for telemetry, access in place for the archive.

Key terms

• Connection: stored endpoint, credential and privacy level reused by items.
• OneLake catalog: tenant-wide discovery of Fabric items and their endorsement.
• Real-Time hub: catalogue of streaming data and events.
• Shortcut: pointer to data in another location, no copy.
• Mirroring: continuous replication of a database into Delta tables in OneLake.
• Copy activity / Copy job: scalable data movement.
• Dataflow Gen2: Power Query based low-code ingestion and transformation.
• Eventstream: no-code real-time ingestion and routing.
• OneLake availability: Delta copy of eventhouse data readable by other engines.

Summary checklist

• I can pick between cloud, on-premises gateway and virtual network gateway connections.
• I know when to use the OneLake catalog and when to use the Real-Time hub.
• I can choose between shortcut, mirroring, pipeline, dataflow, notebook and eventstream.
• I know that shortcuts under Tables need Delta data.
• I can choose between lakehouse, warehouse and eventhouse from workload signals.
• I can explain OneLake availability for Eventhouse and OneLake integration for semantic models.
• I remember that the practice length (50 questions) and 80% target are not official, and that the course targets the syllabus effective October 19, 2026.

Official sources

• Study guide for Exam DP-600: https://learn.microsoft.com/en-us/credentials/certifications/resources/study-guides/dp-600 (opens in a new tab)
• Data ingestion options in Microsoft Fabric: https://learn.microsoft.com/en-us/fabric/fundamentals/get-data (opens in a new tab)
• Unify data with OneLake shortcuts and mirroring: https://learn.microsoft.com/en-us/fabric/onelake/unify-data (opens in a new tab)
• Turn on OneLake availability for an eventhouse: https://learn.microsoft.com/en-us/fabric/real-time-intelligence/event-house-onelake-availability (opens in a new tab)