Microsoft Fabric for Existing Synapse Users: Pipelines, Notebooks, and Lake Databases Without the Surprises

Scenario: The Synapse Workspace That “Just Works”

You’re running a Synapse workspace on Azure:

• Pipelines orchestrate nightly loads from Azure SQL DB and a couple of REST APIs
• Notebooks do data quality checks and write curated Parquet files to a data lake
• A serverless SQL pool exposes a lake database with external tables to Power BI

The plan is to:

• Move orchestration to Fabric Data Factory
• Move transformation notebooks into Fabric Data Engineering
• Replace the serverless SQL lake database with Fabric Lakehouse + SQL analytics endpoint

The goal: keep the behaviour as close as possible, avoid performance regressions, and not break the existing Power BI model.

We’ll use this scenario to anchor the comparisons and migration gotchas.

Fabric vs Synapse: Workspace and Engine Basics

Before diving into pipelines and notebooks, a few fundamentals that affect everything:

• Execution engine

  • Synapse: multiple engines (Spark pools, dedicated SQL, serverless SQL) configured per workspace
  • Fabric: multi-engine SaaS, capacity-backed; Spark and SQL endpoints are managed and auto-scaled within the Fabric capacity

• Storage

  • Synapse: you pick and manage the linked Azure Data Lake Storage Gen2 account
  • Fabric: OneLake is built-in; Lakehouses and Warehouses store data in OneLake with Delta/Parquet under the hood

• Security and governance

  • Synapse: Azure RBAC + Synapse RBAC + managed identities + your own storage ACLs
  • Fabric: Power BI-style workspace security + OneLake item-level permissions + capacity-level controls

This matters for migration because some behaviours you relied on (like direct access to the ADLS account, or serverless SQL external table definitions) don’t carry over directly.

Pipelines: Synapse vs Fabric Data Factory

What Carries Over Cleanly

Fabric Data Factory pipelines are conceptually close to Synapse pipelines:

• Activities like Copy, Notebook, Stored procedure, Web, Lookup exist in both
• Triggers (schedule, tumbling window, event-based) are available in Fabric, though configuration UX differs
• The orchestration model (activities, dependencies, failure paths) is similar enough that pattern-level migration is straightforward

For the scenario pipeline (SQL + REST + lake writes), the structure usually maps 1:1.

Key Differences That Matter in Practice

1. Linked services vs connections

   • Synapse uses linked services heavily for data sources
   • Fabric uses connections; some Synapse-linked-service types are not available in Fabric exactly as-is
   • Gotcha: you can’t just export/import linked services; you recreate connections and re-point activities

2. Integration runtime

   • Synapse pipelines use integration runtimes (auto-resolve or self-hosted)
   • Fabric Data Factory uses Fabric integration runtime (managed) plus self-hosted IR for on-prem or private network sources
   • If your Synapse pipeline depends on a specific IR configuration, you must replicate that using Fabric’s IR options

3. Activity support differences

   • Some Synapse activities (especially ones tied to dedicated SQL pools or certain Azure services) don’t have direct Fabric equivalents
   • Example: activities that rely on dedicated SQL pool maintenance or scale operations are not relevant in Fabric

For the scenario, the main migration steps are:

• Recreate data sources as Fabric connections (SQL DB, REST API)
• Rebuild the Copy activities to target a Lakehouse or Warehouse instead of raw ADLS
• Reconfigure triggers in Fabric Data Factory

Common Pipeline Migration Gotchas

1. Managed identities and authentication

   • In Synapse, you likely used the workspace managed identity for SQL DB and storage
   • In Fabric, authentication is typically via Microsoft Entra ID with service principals or user identities; Fabric items do not use a Synapse-style workspace managed identity
   • Fix: create a service principal with proper permissions on source systems, and configure connections to use it

2. Path semantics for lake writes

   • Synapse Copy to ADLS often used hierarchical folder structures
   • Fabric Copy to Lakehouse writes into tables (Delta) or files under /Files or /Tables
   • If downstream logic expects specific folder paths, you must align Lakehouse folder structure or refactor downstream queries

3. Retry and failure behaviour

   • Retry policies and failure handling exist in both, but defaults and limits can differ
   • Don’t assume Synapse pipeline retry counts or timeouts are preserved; explicitly set retries on critical activities

Notebooks: Spark in Fabric vs Synapse

Spark Environment Differences

On the surface, Fabric notebooks look like Synapse notebooks: cells, languages, Spark sessions. Underneath, there are important differences:

• Spark pool configuration

  • Synapse: you define Spark pools (node sizes, autoscale, versions)
  • Fabric: Spark is managed; configuration is capacity-based, you don’t create pools

• Storage access

  • Synapse: direct ADLS paths (abfss://) with workspace-linked storage
  • Fabric: OneLake paths (/LakehouseName/Tables/... or /LakehouseName/Files/...) plus shortcuts to external data

• Delta Lake support

  • Both support Delta Lake, but Fabric leans heavily on Delta for Lakehouse tables and SQL analytics endpoints

For the scenario, the notebooks doing data quality checks and writing Parquet need to be adjusted to work with Lakehouse and OneLake.

Typical Notebook Migration Steps

Assume a Synapse PySpark notebook that reads from ADLS and writes Parquet like this:

from pyspark.sql import functions as F
source_path = "abfss://raw@yourstorageaccount.dfs.core.windows.net/sales/"
df = spark.read.parquet(source_path)
clean_df = (
df
.filter(F.col("order_date").isNotNull())
.withColumn("load_timestamp", F.current_timestamp())
)
clean_df.write.mode("overwrite").parquet(
"abfss://curated@yourstorageaccount.dfs.core.windows.net/sales_clean/"
)

In Fabric, targeting a Lakehouse named SalesLakehouse and its Files area, the equivalent pattern is:

from pyspark.sql import functions as F
source_path = "/SalesLakehouse/Files/raw/sales/"
df = spark.read.parquet(source_path)
clean_df = (
df
.filter(F.col("order_date").isNotNull())
.withColumn("load_timestamp", F.current_timestamp())
)
clean_df.write.mode("overwrite").parquet(
"/SalesLakehouse/Files/curated/sales_clean/"
)

If you want the curated dataset to be queryable via the Lakehouse SQL analytics endpoint as a table, you typically write to /Tables and register it as a table in the Lakehouse.

Notebook Gotchas When Moving to Fabric

1. Path handling and OneLake semantics

   • Absolute ADLS URIs from Synapse won’t work as-is
   • Fix: use Lakehouse-relative paths (/LakehouseName/Files/... or /LakehouseName/Tables/...) or defined shortcuts

2. Spark configuration access

   • Synapse notebooks often tweak pool-level configs (e.g., executors, dynamic allocation)
   • Fabric notebooks have a more constrained configuration surface exposed; some pool-level tuning options are simply not available
   • Outcome: some performance tuning from Synapse can’t be replicated exactly; you rely more on capacity sizing and data layout

3. Library management

   • Synapse uses requirements files or pool-level libraries
   • Fabric provides mechanisms for managing libraries in the workspace, but the process and scope differ
   • Plan to review any custom library usage; don’t assume pool-level libraries carry over

Lake Databases vs Fabric Lakehouses and SQL Endpoints

This is where most Synapse users hit the first serious migration friction.

Synapse Lake Databases: How They Worked

In Synapse, a common pattern is:

• Define a lake database in the serverless SQL pool
• Create external tables on top of Parquet or CSV in ADLS
• Use T-SQL for reporting and Power BI import or DirectQuery

Example external table in Synapse serverless SQL:

CREATE EXTERNAL TABLE [dbo].[Sales]
WITH (
    LOCATION = 'sales/',
    DATA_SOURCE = [SalesDataSource],
    FILE_FORMAT = [ParquetFormat]
)
AS
SELECT * FROM OPENROWSET(
    BULK 'sales/*.parquet',
    DATA_SOURCE = 'SalesDataSource',
    FORMAT = 'PARQUET'
) AS [rows];

Fabric Lakehouse + SQL Analytics Endpoint

In Fabric, the equivalent pattern is built around Lakehouses and their SQL analytics endpoints:

• A Lakehouse stores data in OneLake, primarily as Delta tables under /Tables
• The Lakehouse exposes a SQL analytics endpoint that lets you query those tables with T-SQL
• Power BI can connect using Direct Lake (for supported scenarios) or import, without defining external tables manually

A typical Fabric SQL query over a Lakehouse table:

SELECT
    s.OrderId,
    s.OrderDate,
    s.CustomerId,
    s.TotalAmount
FROM
    dbo.Sales s
WHERE
    s.OrderDate >= DATEADD(day, -30, CAST(GETUTCDATE() AS date));

Here dbo.Sales is a Lakehouse table registered in the SQL analytics endpoint, backed by Delta files.

Key Differences vs Synapse Lake Databases

1. Table type: external vs managed Delta

   • Synapse lake databases: external tables over arbitrary files
   • Fabric Lakehouse: managed tables backed by Delta; external data is typically brought in via shortcuts or ingestion

2. Schema enforcement and evolution

   • External tables in Synapse can be loosely coupled to file schemas
   • Delta tables in Fabric enforce schema more strictly and support evolution; behaviour differs when files don’t match expected schema

3. Direct Lake and performance characteristics

   • Fabric introduces Direct Lake for Power BI over Lakehouse tables, which relies on Delta layout and specific conditions
   • Synapse serverless SQL doesn’t have an equivalent; performance characteristics differ significantly

For the scenario, moving from serverless SQL lake database to Fabric Lakehouse means:

• Recreating tables as Lakehouse tables (usually Delta)
• Refactoring external table definitions into ingestion or shortcut patterns
• Re-pointing Power BI datasets from Synapse serverless to Fabric Lakehouse / Warehouse

Lake Database Migration Gotchas

1. External table pattern doesn’t map 1:1

   • Fabric’s Lakehouse SQL endpoint doesn’t support the same external table pattern as Synapse serverless
   • Fix: either ingest the data into Lakehouse tables (Delta) or use shortcuts to external data and then register tables on top

2. File format and partitioning assumptions

   • Synapse external tables over Parquet may rely on specific folder partitioning (year=YYYY/month=MM/...)
   • Fabric Lakehouse can work with partitioned Delta, but you may need to adjust layout and ingestion to preserve partition benefits

3. Power BI connection mode changes

   • Power BI models that used Synapse serverless via DirectQuery won’t automatically behave the same when pointed at Fabric
   • You must choose between Direct Lake, import, or DirectQuery to Warehouse/Lakehouse SQL endpoint, each with different behaviour

Putting It Together: Migrating the Scenario Step by Step

Here’s a practical sequence that keeps risk under control.

1. Stand Up Fabric Workspace and Lakehouse

• Create a Fabric workspace with Data Engineering and Data Factory enabled
• Create a Lakehouse SalesLakehouse
• Define Lakehouse tables for key entities (e.g., Sales, Customers) using notebooks or Dataflows

2. Rebuild Ingestion Pipelines in Fabric Data Factory

• Create connections to Azure SQL DB and REST APIs
• Rebuild Copy activities to write into SalesLakehouse tables or /Files paths
• Configure triggers to match Synapse schedules

3. Port Notebooks for Data Quality and Transformation

• Copy notebook logic, adjust paths to OneLake/Lakehouse
• Ensure writes target Lakehouse /Tables for curated data
• Validate schema and data types, especially if moving from Parquet-only to Delta

4. Replace Lake Database with Lakehouse SQL Endpoint

• Register Lakehouse tables as needed
• Recreate key SQL views and logic in the Lakehouse SQL analytics endpoint

Example view creation:

CREATE VIEW dbo.v_SalesLast30Days
AS
SELECT
    s.OrderId,
    s.OrderDate,
    s.CustomerId,
    s.TotalAmount
FROM
    dbo.Sales s
WHERE
    s.OrderDate >= DATEADD(day, -30, CAST(GETUTCDATE() AS date));

• Test queries that previously ran in Synapse serverless against the Lakehouse endpoint

5. Repoint Power BI and Validate Behaviour

• Create a new Power BI dataset connected to the Lakehouse SQL endpoint or Direct Lake where appropriate
• Rebind existing reports to the new dataset
• Validate:
  • Row counts
  • Key measures
  • Refresh behaviour and duration

Practical Takeaway: Don’t Lift-and-Shift, Re-Pattern Intentionally

The core lesson for existing Synapse users is simple: Fabric is close enough to feel familiar, but different enough that a lift-and-shift mindset will hurt you.

For pipelines, notebooks, and lake databases, treat migration as a chance to re-pattern:

• Move from external-table thinking to Lakehouse/Delta thinking
• Align notebook paths and logic with OneLake and Lakehouse semantics
• Rebuild pipelines around Fabric connections and Lakehouse destinations instead of copying linked services

If you pick one area to get right first, make it the Lakehouse design. Once your storage and table patterns are solid, pipelines and notebooks tend to fall into place with fewer surprises.