Photo by Alexander Tsang on Unsplash
Migrating from Azure Data Factory to Microsoft Fabric Data Factory: Key Differences and Migration Strategies
Azure Data Factory (ADF) to Fabric Data Factory
Microsoft's Data Factory in Microsoft Fabric represents the evolution of Azure Data Factory (ADF), offering enhanced cloud-scale data movement and transformation services. This advancement aims to provide a more integrated, user-friendly, and enterprise-grade experience for handling complex Extract, Transform, Load (ETL) scenarios.
Migrating to Fabric capacities depends on each customer's needs. While Azure Data Factory and Synapse Gen2 remain supported, Microsoft is prioritizing investment in Fabric pipelines for enterprise data ingestion. Over time, Fabric capacities will offer increasing value, aligning with the Microsoft Fabric roadmap.
Key Differences Between ADF and Fabric Data Factory:
Integration Runtime:
- Configuration: In ADF, Integration Runtimes (IRs) are essential for data processing, requiring specific configurations for cloud or on-premises data movement. Fabric Data Factory simplifies this by eliminating the need for manual IR configuration, streamlining the setup process.
Data Transformation:
- Mapping Dataflows vs. Dataflow Gen2: ADF utilizes Mapping Dataflows for data transformations, whereas Fabric introduces Dataflow Gen2, offering a more intuitive and user-friendly interface for building transformations.
Activity Enhancements:
- New Integrations: Fabric Data Factory introduces new activities, such as Office 365 Outlook and Teams, enabling seamless communication and collaboration within data workflows.
Connectivity:
- Connections vs. Linked Services: Fabric replaces ADF's Linked Services with Connections, providing a more intuitive method to connect to data sources.
Triggers and Scheduling:
- Automation: Fabric utilizes schedulers and Reflex events to automate pipeline executions, with native support for file event triggers within pipelines.
Publishing and Integration Runtime:
- Simplified Workflow: In Fabric, there's no need to publish to save content; users can directly save or run pipelines, which automatically saves the content. Additionally, Fabric does not employ the concept of Integration Runtime as seen in ADF.
On-Premises Data Access:
- Gateway Usage: Access to on-premises data in Fabric is facilitated through the On-premises Data Gateway, streamlining connectivity for hybrid data scenarios.
Feature Comparison
| Feature | Azure Data Factory | Fabric Data Factory |
| Pipeline activities | ||
| Office 365 and Teams activities enable you to seamlessly send messages, facilitating efficient communication and collaboration across your organization | No | Yes |
| Create connections to your Power BI semantic model and Dataflow Gen2 to ensure your data is consistently refreshed and up-to-date | No | Yes |
| Validation in a pipeline to ensure the pipeline only continues execution once it validates the attached dataset reference exists, that it meets the specified criteria, or times out | Yes | Yes1 |
| Execute a SQL Server Integration Services (SSIS) package to perform data integration and transformation operations | Yes | Planned |
| Data transformation | ||
| Visually designed data transformations using Apache Spark clusters with Mapping Dataflows to create and manage data transformation processes through a graphical interface | Yes | No2 |
| Visually designed data transformations using the Fabric compute engine with the intuitive graphical interface of Power Query in Dataflow Gen2 | No | Yes |
| Connectivity | ||
| Support for all Data Factory data sources | Yes | In progress3 |
| Scalability | ||
| Ensure seamless execution of activities in a pipeline with scheduled runs | Yes | Yes |
| Schedule multiple runs for a single pipeline for flexible and efficient pipeline management | Yes | Planned |
| Utilize tumbling window triggers to schedule pipeline runs within distinct, nonoverlapping time windows | Yes | Planned |
| Event triggers to automate the execution of pipeline runs in response to specific or relevant event occurrences | Yes | Yes4 |
| Artificial intelligence | ||
| Copilot for Data Factory, which provides intelligent pipeline generation to ingest data with ease and explanations to help better understand complex pipelines or to provide suggestions for error messages | No | Yes |
| Content management | ||
| Data lineage view, which help users understand and assess pipeline dependencies | No | Yes |
| Deployment pipelines, which manage the lifecycle of content | No | Yes |
| Platform scalability and resiliency | ||
| Premium capacity architecture, which supports increased scale and performance | No | Yes |
| Multi-Geo support, which helps multinational customers address regional, industry-specific, or organizational data residency requirements | Yes | Yes |
| Security | ||
| Virtual network (virtual network) data gateway connectivity, which allows Fabric to work seamlessly in an organization's virtual network | No | Planned |
| On-premises data gateway connectivity, which allows for secure access of data between an organization's on-premises data sources and Fabric items | No | Yes |
| Azure service tags support, which is a defined group of IP addresses that is automatically managed to minimize the complexity of updates or changes to network security rules | Yes | Yes |
| Governance | ||
| Content endorsement, to promote or certify valuable, high-quality Fabric items | No | Yes |
| Microsoft Purview integration, which helps customers manage and govern Fabric items | Yes | Yes |
| Microsoft Information Protection (MIP) sensitivity labels and integration with Microsoft Defender for Cloud Apps for data loss prevention | No | Yes |
| Monitoring and diagnostic logging | ||
| Logging pipeline execution events into an event store to monitor, analyze, and troubleshoot pipeline performance | Yes | Planned |
| Monitoring hub, which provides monitoring capabilities for Fabric items | No | Yes |
| Microsoft Fabric Capacity Metrics app, which provides monitoring capabilities for Fabric capacities | No | Yes |
| Audit log, which tracks user activities across Fabric and Microsoft 365 | No | Yes |
Feature Mapping
| Azure Data Factory | Data Factory in Fabric | Description |
| Pipeline | Data pipeline | Data pipeline in Fabric is better integrated with the unified data platform including Lakehouse, Data warehouse, and more. |
| Mapping dataflow | Dataflow Gen2 | Dataflow Gen2 provides easier experience to build transformation. We are in progress of letting more functions of mapping dataflow supported in Dataflow Gen2 |
| Activities | Activities | We are in progress to make more activities of ADF supported in Data Factory in Fabric. Data Factory in Fabric also has some newly attracted activities like Office 365 Outlook activity. Details are in Activity overview. |
| Dataset | Not Applicable | Data Factory in Fabric doesn’t have dataset concepts. Connection is used for connecting each data source and pull data. |
| Linked Service | Connections | Connections have similar functionality as linked service, but connections in Fabric have more intuitive way to create. |
| Triggers | Schedule triggers and file event triggers | Fabric can use the scheduler and Reflex events to automatically run pipelines. File event triggers are supported natively in pipelines in Microsoft Fabric Data Factory. |
| Publish | Save, Run | For pipeline in Fabric, you don’t need to publish to save the content. Instead, you can use Save button to save the content directly. When you select Run button, it saves the content before running pipeline. |
| Autoresolve and Azure Integration runtime | Not Applicable | In Fabric, we don’t have the concept of Integration runtime. |
| Self-hosted integration runtimes | On-premises Data Gateway | On-premises Data Gateway enables access to on-premises data via the Fabric Data Factory. Details are in How to access on-premises data sources in Data Factory for Microsoft Fabric. |
| Azure-SSIS integration runtimes | To be determined | The capability in Fabric hasn’t confirmed the roadmap and design. |
| MVNet and Private End Point | To be determined | The capability in Fabric hasn’t confirmed the roadmap and design. |
| Expression language | Expression language | Expression language is similar in ADF and Fabric. |
| Authentication type in linked service | Authentication kind in connection | Authentication kind in Fabric pipeline already supported popular authentication types in ADF, and more authentication kinds are added. |
| CI/CD | CI/CD | CI/CD capability in Fabric Data Factory will be coming soon. |
| Export and Import ARM | Save as | Save as is available in Fabric pipeline to duplicate a pipeline. |
| Monitoring | Monitoring, Run history | The monitoring hub in Fabric has more advanced functions and modern experience like monitoring across different workspaces for better insights. |
Planning Your Migration:
Transitioning from ADF to Fabric Data Factory requires careful planning to ensure a seamless migration:
Assess Existing Artifacts:
- Inventory: Compile a comprehensive list of your ADF pipelines, dataflows, datasets, linked services, and triggers to understand the scope of migration.
Understand Feature Parity:
- Comparison: Identify which features and activities in ADF have direct equivalents in Fabric Data Factory and note any that may require reconfiguration or alternative approaches.
Recreate Necessary Components:
- Manual Recreation: Certain elements, such as datasets and linked services, may need to be manually recreated in Fabric Data Factory due to differences in architecture and functionality.
Testing and Validation:
- Thorough Testing: After migration, rigorously test all pipelines and dataflows to ensure they function as intended in the new environment.
Migration Strategies:
Depending on your organization's needs, consider the following migration approaches:
Phased Migration:
- Gradual Transition: Migrate components in stages, allowing for testing and validation at each step to minimize disruptions.
Parallel Run:
- Side-by-Side Operation: Run ADF and Fabric Data Factory concurrently, gradually shifting workloads once confidence in the new setup is established.
Big Bang Migration:
- Complete Switch: Move all components to Fabric Data Factory in a single effort, suitable for less complex environments or when downtime can be managed.
Conclusion:
Migrating to Data Factory in Microsoft Fabric offers numerous benefits, including enhanced integration, improved user experience, and advanced features. By carefully planning and executing your migration strategy, your organization can leverage these advantages to achieve more efficient and effective data integration and transformation processes.