Getting Data into Microsoft Fabric: Shortcuts, Mirroring and Pipelines
Introduction
Getting data into Microsoft Fabric starts with deciding whether to read it in place or ingest a copy. This DP-600 lesson compares OneLake shortcuts, database mirroring, Data Factory pipelines, Dataflow Gen2, notebooks and eventstreams. It also explains how to choose a data store and make its data available to other Fabric engines.
Learning objectives
- Create, share and secure data connections, and distinguish cloud connections from on-premises and virtual network gateways.
- Use the OneLake catalog and the Real-Time hub to find and reuse data.
- Choose among shortcuts, database mirroring, copy activity in pipelines, Dataflows Gen2, notebooks and eventstreams.
- Choose a data store (lakehouse, warehouse, eventhouse) from workload requirements.
- Explain OneLake availability for Eventhouse and OneLake integration for semantic models.
1. Data connections
A connection describes how Fabric reaches a source or destination. It records details such as the endpoint and authentication method. Settings such as privacy level depend on the connector and workload. Connections are first-class objects that you manage from Manage connections and gateways.
Key facts:
- Cloud connections reach sources that are reachable from the Fabric service. On-premises data gateway connections reach sources inside a private network. A virtual network data gateway reaches sources behind a private endpoint or inside an Azure virtual network without installing software on a VM.
- Authentication kinds include organizational account (Microsoft Entra ID), service principal, workspace identity, basic, key and SAS. Where the connector and workload support it, use a service principal or workspace identity with narrowly scoped source permissions.
- Share a connection with the User role to grant use without exposing credentials or granting ownership. User with resharing can also grant access to others. Reserve Owner for people who manage the connection.
- The privacy level (Organizational, Private, Public) controls how mashups in Dataflows Gen2 may combine data. Set these classifications to reflect the sensitivity and trust boundaries of the sources. Power Query may isolate combinations or return a Formula Firewall error.
- Use environment-specific connections and parameterize supported pipeline properties. Check which fields and authentication methods the connector supports.
Example. Contoso Retail keeps its ERP in an on-premises SQL Server 2019 instance. The BI team installs an on-premises data gateway cluster of two nodes on separate machines for availability, creates one connection erp-sql-prod with a service account, and shares it with the Data Engineering group. Pipelines and dataflows reuse the same connection, so a password rotation is done once.
2. Discovering data: OneLake catalog and Real-Time hub
The OneLake catalog is the place to browse, search and filter items across the tenant that you can access: lakehouses, warehouses, semantic models, eventhouses and more. You can filter by workspace, owner, item type and endorsement (Promoted, Certified, Master data). From the catalog you can open an item, see its lineage, and reach governance information. Using it, an analyst finds a Certified semantic model instead of building a duplicate.
The Real-Time hub is the tenant-wide catalogue of streaming data: data streams (eventstreams), Microsoft sources (such as Azure Event Hubs, IoT Hub, Fabric and Azure events) and Fabric events (workspace item events, OneLake events, job events). From the hub you can get events into Fabric by connecting a source, create an eventstream, and set alerts. Use it for discovering existing streams, finding supported event sources and setting up routing or alerts.
| Need | Use |
|---|---|
| Find a certified lakehouse, warehouse or model | OneLake catalog |
| Browse or connect a streaming source | Real-Time hub |
| Inspect who owns an item and its lineage | OneLake catalog, lineage view |
| Subscribe to a Fabric or Azure event | Real-Time hub |
3. Ingesting or accessing data
Start by distinguishing access in place from ingestion. Then consider latency, source authentication, compute cost and who will operate the solution.
3.1 Shortcuts: access without copying
A OneLake shortcut is a pointer to data stored elsewhere. It appears as a folder or a table in a lakehouse (or a KQL database), and engines read it as if it were local.
- Targets: other OneLake locations (lakehouse, warehouse, KQL database, semantic model storage), Azure Data Lake Storage Gen2, Amazon S3 and S3-compatible storage, Google Cloud Storage, Dataverse, and other supported sources, including on-premises sources through a gateway.
- Deleting the shortcut object removes the pointer. Deleting files through a shortcut can delete the source files if that path allows writes. Removing the target itself affects all consumers.
- A shortcut under Tables can register supported table data, such as a Delta table. Supported Iceberg tables can also be presented as Delta through metadata virtualization. For raw files such as plain Parquet or CSV, use Files and read them with an engine that supports the format.
- Authorisation: for OneLake targets, check the caller’s effective access on both sides of the shortcut. Some engine paths use a delegated identity. External shortcuts use the configured connection. Review OneLake security and Fabric item permissions before granting access.
- Supported external shortcuts can cache files after reads to reduce repeated egress. Cache eligibility, file size and access patterns affect the savings. A shortcut still transfers data when read, and a cache can create local copies. Check residency constraints separately.
3.2 Mirroring: a continuously replicated, queryable copy
Database mirroring replicates an operational database into OneLake as Delta tables, without a custom ingestion pipeline. Measure actual replication lag and check source requirements, initial load and pricing before committing to a freshness or cost target. Examples include Azure SQL Database, SQL Server and Snowflake. Verify current support, versions and prerequisites for the specific source. Mirroring also supports metadata mirroring for catalogs such as Azure Databricks Unity Catalog (it mirrors the metadata and creates shortcuts, so the data is accessed in place).
A mirrored database item gives you a SQL analytics endpoint and Delta tables you can reach from lakehouses with shortcuts, from notebooks, and from Direct Lake semantic models. Mirroring is the right answer when the requirement says near real-time replication of an operational database with minimal engineering effort.
3.3 Pipelines: orchestrated copy
A Data Factory pipeline orchestrates activities. The Copy activity moves large volumes at scale between many sources and sinks, runs on a schedule or a trigger. With a watermark pattern, read changes since the last successful load and advance the watermark only after the sink write succeeds. Change data capture (CDC) requires source and connector support. Other activities run notebooks, stored procedures, dataflows and other pipelines, with control flow (ForEach, If, Until) and parameters. A Copy job is a guided way to copy without authoring a pipeline. Its incremental modes depend on the source and destination.
3.4 Dataflows Gen2: low-code transformation
Dataflow Gen2 uses Power Query Online to combine and transform data with a visual editor. Choose a supported destination, such as a lakehouse or warehouse. For multi-terabyte loads, compare throughput and compute cost with Copy activity and Spark. Where applicable, check whether source query folding can push work to the source.
3.5 Notebooks and eventstreams
Notebooks (for example, PySpark notebooks) give full code control for complex logic, APIs, and large transformations. Eventstreams ingest real-time events (Event Hubs, Kafka, IoT, change data capture from databases, and sample data), apply no-code transformations and route to an eventhouse, a lakehouse, Activator or a custom endpoint.
3.6 Choosing the method
| Requirement in the scenario | Best fit |
|---|---|
| Data already in ADLS Gen2 or S3, do not duplicate it | Shortcut |
| Operational Azure SQL DB, near real-time analytics, minimal effort | Mirroring |
| Nightly incremental load of 2 TB from on-premises SQL Server | Pipeline with Copy activity or Copy job with a supported gateway and connector |
| Analyst reshapes an Excel/SharePoint source with a visual interface | Dataflow Gen2 |
| Custom API pagination, complex Python logic | Notebook |
| IoT telemetry at thousands of events per second | Eventstream into an eventhouse |
| Data may be read remotely but must not be copied at rest into a separate Fabric store | Shortcut, after verifying cache, query results and residency constraints |
Worked example 1: choosing the ingestion method. Statement. Fabrikam stores 8 TB of Delta tables in Amazon S3. Each table has Parquet data files and a Delta transaction log. BI needs to read the tables from Fabric with minimal operational overhead and predictable egress.
Reasoning. First, check that the source is a supported S3 connection and grants read access. Then create the shortcut under Tables to register the Delta table for SQL and Spark. Measure egress for repeated queries before choosing a cache policy. The cache can reduce repeat reads of eligible files. It does not remove the first read or cover all file sizes.
Solution. Use an S3 shortcut under Tables. Enable caching only if the workspace policy permits local caching and measurements show that it helps. A scheduled copy adds a separate stored copy and a load to operate. A dataflow is useful for transformation. Database mirroring addresses a different source and replication requirement.
4. Lakehouse, warehouse or eventhouse?
| Criterion | Lakehouse | Warehouse | Eventhouse (KQL database) |
|---|---|---|---|
| Data types | Structured, semi-structured and unstructured files | Structured, relational | Streaming and time-series, log and telemetry data |
| Main engines | Spark, SQL analytics endpoint (read-only for table data) | T-SQL (supported DML and DDL) | KQL, with a subset of T-SQL |
| Write model | Spark, pipelines, dataflows | T-SQL INSERT/UPDATE/DELETE, multi-table transactions | Streaming ingestion, queued ingestion |
| Typical user | Data engineer, data scientist | SQL developer, DBA | Real-time analyst, observability |
| Strength | Open files, ML, medallion architecture | Enterprise SQL semantics, stored procedures | Interactive analytics on streaming and time-series data |
| Storage format | Delta in OneLake | Delta in OneLake | Proprietary indexed storage, optionally exposed as Delta |
Rules of thumb:
- Need to write with T-SQL (UPDATE, MERGE-style logic, multi-table transactions) and team knows SQL: warehouse.
- Need Spark, files, notebooks, ML or unstructured data: lakehouse. Its SQL analytics endpoint does not write table data, but supports SQL objects such as views and stored procedures.
- Need time-series, logs, telemetry, near real-time queries with KQL: eventhouse.
- Many solutions use more than one: ingest streaming into an eventhouse, expose it to a lakehouse by shortcut, serve curated tables from a warehouse.
5. OneLake integration for Eventhouse and semantic models
OneLake availability (Eventhouse). Enable it on a KQL database or table to expose a Delta copy through OneLake. Read the data with Spark, create a shortcut for SQL access, or use a supported Direct Lake path. Materialization has its own latency, so measure freshness and keep latency-sensitive dashboards on native KQL when needed. Check the documented schema change restrictions before altering a published table.
OneLake integration for semantic models. On a supported capacity, an Import model using large semantic model storage format can export imported tables to OneLake as Delta during refresh. Measures, DirectQuery tables and calculation groups are among the objects excluded from export. The files do not evaluate model RLS or OLS. When the model defines these roles, Fabric restricts access to the exported data to users with model Write permission. Govern any downstream copies separately.
6. Reading the question
Scenarios in this area typically include signals:
- "read existing files in place": shortcut. Check remote reads, cache and permissions.
- "near real-time replication", "operational database": mirroring.
- "orchestrate", "incremental load", "terabytes": pipeline copy.
- "analysts", "low-code", "Power Query": dataflow.
- "events", "IoT", "telemetry": eventstream and eventhouse.
- "read-write T-SQL": warehouse. "Spark and files": lakehouse.
Common pitfalls
- Treating a shortcut as a copy: if the source is deleted or its permissions change, consumers lose access.
- Expecting a folder of raw Parquet files to register as a SQL-queryable table under Tables.
- Expecting to run INSERT or UPDATE through a lakehouse SQL analytics endpoint.
- Choosing a bulk ingestion method without checking throughput, transformation requirements and cost.
- Binding a pipeline to a personal account: pipelines break when the person leaves.
- Forgetting a gateway for an on-premises source, or exposing a private source with a cloud connection.
- Confusing mirroring (replicates a database) with shortcut (points to storage).
Exam tip: when a scenario states "minimal development effort" or "no ETL to maintain", eliminate pipelines and notebooks first. When it states "must not duplicate data", eliminate everything but shortcuts (and metadata mirroring).
Mini case study
Tailspin Logistics runs three sources: an Azure SQL Database for orders, scanner telemetry from 4,000 trucks arriving through Azure Event Hubs, and historical GPS archives in an ADLS Gen2 account owned by another business unit that refuses to move them. Dispatchers need a dashboard refreshed within seconds for truck positions, finance needs daily order analytics, and the data science team needs the GPS archive with Spark.
Proposed solution.
- Mirror the Azure SQL Database: orders are replicated near real-time with no pipelines; the SQL analytics endpoint serves finance.
- Connect the Event Hubs source through the Real-Time hub and an eventstream into an eventhouse, which supports KQL queries on telemetry in seconds.
- Enable OneLake availability for telemetry used in downstream analysis. Create the required shortcuts and check data lag before joining it with orders. Keep the seconds-level dispatch view on the Eventhouse KQL path.
- Create an ADLS Gen2 shortcut in a lakehouse for the GPS archive (Delta under Tables, raw files under Files), using a shared connection with a service principal where supported. Use Spark for raw files and the SQL analytics endpoint for registered tables.
- Certify the resulting gold semantic model so analysts find it in the OneLake catalog.
Commentary. Mirroring serves operational data, Eventhouse serves live telemetry, and a shortcut serves the archive. When you promote those assets across environments, review Fabric Git and deployment pipelines to check which definitions and connections move with them.
Key terms
- Connection: stored endpoint, credential and privacy level reused by items.
- OneLake catalog: tenant-wide discovery of Fabric items and their endorsement.
- Real-Time hub: catalogue of streaming data and events.
- Shortcut: pointer to data in another location. Reads can use a supported cache.
- Mirroring: continuous replication of a database into Delta tables in OneLake.
- Copy activity / Copy job: scalable data movement.
- Dataflow Gen2: Power Query based low-code ingestion and transformation.
- Eventstream: no-code real-time ingestion and routing.
- OneLake availability: Delta copy of eventhouse data readable by other engines.
Summary checklist
- I can pick between cloud, on-premises gateway and virtual network gateway connections.
- I know when to use the OneLake catalog and when to use the Real-Time hub.
- I can choose between shortcut, mirroring, pipeline, dataflow, notebook and eventstream.
- I can distinguish a shortcut to a supported table format from a shortcut to raw files.
- I can choose between lakehouse, warehouse and eventhouse from workload signals.
- I can explain OneLake availability for Eventhouse and OneLake integration for semantic models.
Official sources
- Study guide for Exam DP-600: https://learn.microsoft.com/en-us/credentials/certifications/resources/study-guides/dp-600 (opens in a new tab)
- Data ingestion options in Microsoft Fabric: https://learn.microsoft.com/en-us/fabric/fundamentals/get-data (opens in a new tab)
- Unify data with OneLake shortcuts and mirroring: https://learn.microsoft.com/en-us/fabric/onelake/unify-data (opens in a new tab)
- OneLake shortcuts and caching: https://learn.microsoft.com/en-us/fabric/onelake/onelake-shortcuts (opens in a new tab)
- Iceberg table virtualization: https://learn.microsoft.com/en-us/fabric/onelake/onelake-iceberg-tables (opens in a new tab)
- OneLake integration for semantic models: https://learn.microsoft.com/en-us/fabric/enterprise/powerbi/onelake-integration-overview (opens in a new tab)
- Turn on OneLake availability for an eventhouse: https://learn.microsoft.com/en-us/fabric/real-time-intelligence/event-house-onelake-availability (opens in a new tab)
