Azure Data Factory: What It Does and When to Use It

Azure Data Factory is Microsoft's cloud service for moving and transforming data on a schedule. It connects to your sources, copies data to where it needs to go, runs transformations, and orchestrates the whole flow as pipelines you can monitor and rerun. If a warehouse is where analytical data lives, Azure Data Factory is one of the main tools that gets it there.

If you are building data pipelines in the Microsoft cloud, Azure Data Factory is the workhorse you will meet first. This guide explains what it does, the core concepts you need, how it handles both cloud and on-premises data, and how it now relates to Microsoft Fabric. I use it on client pipelines, so this is the practical tour rather than a feature dump.

What is Azure Data Factory?

Azure Data Factory, often shortened to ADF, is a managed data integration service. You do not run servers; you define pipelines that say where data comes from, what to do with it, and where it goes, and Azure runs them for you. It is built for both simple copying, moving files into a warehouse, and full transformation workflows.

Its job is orchestration and movement. It reaches into databases, files, SaaS apps, and APIs, brings data together, and drives it through the steps that turn raw source data into something a warehouse and reports can use. Think of it as the conductor of your data pipeline.

What Azure Data Factory does

At its core, ADF extracts data from sources, optionally transforms it, and loads it to destinations, the classic pipeline job. It handles both ETL and ELT styles: it can transform data as it moves, or simply land raw data and trigger transformations elsewhere. It also schedules these flows, retries failures, and gives you a monitoring view of every run.

Beyond movement, it coordinates other services. A pipeline can copy data, then run a stored procedure, then kick off a notebook, then refresh a dataset, stitching several steps into one governed flow with dependencies between them.

The core concepts

Pipelines and activities

A pipeline is a container of activities, and each activity is one step: copy data, run a data flow, execute a stored procedure, call a notebook. You chain activities with dependencies so they run in order, and only proceed when the previous step succeeds.

Datasets and linked services

A linked service is a connection: the credentials and address of a source or destination, like a SQL database or a storage account. A dataset points at the specific data within that connection, such as a table or a folder of files. Together they tell an activity exactly what to read or write.

Triggers

Triggers start pipelines. A schedule trigger runs on a clock, a tumbling window trigger processes fixed time slices, and an event trigger fires when a file lands in storage. Triggers are how a pipeline goes from something you run by hand to something that keeps your warehouse fed automatically.

Integration runtimes: cloud and on-premises

The integration runtime is the compute that actually does the work, and it is the concept that trips people up. The Azure integration runtime handles cloud-to-cloud data movement. The self-hosted integration runtime is a small agent you install inside your own network so ADF can reach on-premises databases securely, without exposing them to the internet.

If your data lives on-premises, the self-hosted integration runtime is what makes Azure Data Factory able to read it. Setting it up correctly is often the first real task on a hybrid pipeline, and it is worth getting right early.

Data flows versus pipeline copy

ADF offers two ways to transform. The copy activity moves data efficiently with light mapping. Mapping data flows are a visual, code-free way to build real transformations, joins, aggregations, derived columns, that run on managed Spark under the hood. You design the logic; Azure handles the cluster.

For simple movement, the copy activity is enough. For genuine transformation without writing Spark by hand, mapping data flows are the tool, though for heavy custom logic many teams still prefer notebooks. Choosing the lightest tool that does the job keeps pipelines cheap and maintainable.

Azure Data Factory and Microsoft Fabric

This is where things have shifted. Microsoft Fabric includes a Data Factory experience built on the same ideas, bringing pipelines and dataflows inside the unified platform alongside the lakehouse and warehouse. For new projects that live in Fabric, you often use Fabric's Data Factory rather than standalone ADF.

Standalone ADF is not going away; it remains the right choice for many Azure-centric pipelines, especially those integrating a wide range of services outside Fabric. But if your data platform is Fabric, expect to use the Data Factory that lives inside it. The concepts transfer directly either way.

Azure Data Factory versus SSIS

Teams coming from on-premises Microsoft data work know SSIS, SQL Server Integration Services. ADF is the cloud-era successor for orchestration and movement, and it can even run existing SSIS packages through the SSIS integration runtime, which eases migration. The mental model is similar; the difference is that ADF is managed, cloud-native, and scales without you provisioning servers.

Monitoring and reliability

A pipeline you cannot see is a pipeline you cannot trust. ADF gives you a monitoring view of every run, with status, duration, and the error on any failed activity, plus alerts you can route to email or Teams. Building in retries and sensible failure handling is what turns a pipeline from a script that sometimes breaks into something that runs unattended and tells you when it needs attention.

Security and networking

Moving data safely is half the job. The service authenticates to sources with managed identities where possible, so credentials are not scattered through pipelines, and it stores secrets in Azure Key Vault rather than in plain configuration. For private data, the self-hosted agent keeps traffic inside your network, and private endpoints let cloud movement avoid the public internet entirely.

Getting this right is not optional on real pipelines. A pipeline that works but leaks credentials or routes sensitive data over public paths is a problem waiting to happen, so I treat identity and networking as part of the build, not an afterthought bolted on later.

Pricing, in plain terms

Billing is usage-based, which is a strength and a trap. You pay for pipeline activity runs, for data movement, and for the compute behind data flows when they run. Light orchestration costs very little. Heavy visual data flows on managed Spark cost more, because you are renting a cluster for the duration.

The practical advice: use the cheap copy activity for plain movement, reserve data flows for genuine transformation, and turn off or consolidate anything running more often than the business actually needs. I have trimmed real bills just by right-sizing how frequently pipelines fire, since a job set to run hourly out of habit when daily would do wastes money every day.

Getting started with your first pipeline

A sensible first build is small: connect one source, copy a table into a lake or warehouse, add a schedule trigger, and watch it run in the monitor. Once that works end to end, you add transformation, dependencies, and alerting. Starting narrow and proving each piece beats designing an elaborate pipeline on paper and debugging the whole thing at once. The tool rewards an incremental approach.

A habit worth adopting early is parameterizing your pipelines. Instead of hardcoding a table name or a date, pass it in as a parameter, so one pipeline can serve many tables and environments. This is what separates a pile of near-duplicate pipelines from a small, reusable set that is easy to maintain. It takes a little more thought up front and pays back every time a new source arrives, because you extend the pattern rather than copying and editing yet another pipeline that then drifts out of step with the rest.

Common use cases

• Consolidating several source systems into a data warehouse on a schedule.
• Loading raw data into a lake, then triggering in-warehouse transforms.
• Migrating data between systems as a one-off or ongoing sync.
• Orchestrating a chain of steps across storage, databases, and compute.

When to use Azure Data Factory

Reach for Azure Data Factory when you need scheduled, monitored data movement in the Azure cloud, especially across many different sources and services. If your platform is Fabric, use its built-in Data Factory instead; if you live in the broader Azure ecosystem, standalone ADF is the dependable orchestrator. Either way, the same pipeline thinking applies, and the skills you build transfer cleanly between the two. What rarely transfers is a badly designed pipeline, so the effort you put into clean, parameterized, well-monitored flows pays off no matter which flavor of the tool you end up running.

If you want data pipelines built and monitored properly, that is core to my data engineering work, and you can tell me what you need moved. Microsoft's documentation lives on Microsoft Learn.