Top 10 Accenture Azure Data Factory & Azure Databricks Interview Questions (2026)

These Azure Data Engineer interview questions cover real-world scenarios involving Azure Data Factory (ADF), Azure Databricks, ADLS Gen2, Delta Lake, PySpark, Spark optimization, security, and production ETL pipelines. They are commonly asked in interviews at companies like Accenture, Capgemini, Cognizant, Deloitte, TCS, Infosys, Microsoft, and EY.

My project involved building a scalable ETL pipeline using Azure Data Factory and Azure Databricks. Data was collected from REST APIs, SQL databases, and millions of CSV files stored in ADLS Gen2. Azure Data Factory orchestrated data ingestion and workflow execution. Azure Databricks performed large-scale transformations using PySpark. The processed data was stored in Delta Lake following the Medallion Architecture (Bronze → Silver → Gold). Finally, Power BI consumed the Gold layer to build business dashboards and reports.

We reduced Azure storage and compute costs by converting CSV files into Parquet and Delta formats, implementing incremental data loads instead of full refreshes, enabling lifecycle management policies to move inactive data into Cool and Archive storage tiers, and configuring Databricks cluster auto-termination. Azure Cost Management reports showed approximately a 30% reduction in monthly storage costs.

Azure Data Factory Copy Activity can ingest files using wildcard paths. For large-scale processing, ADF triggers an Azure Databricks notebook where PySpark reads the files in parallel. Spark automatically distributes the workload across cluster nodes. After transformations, the data is written into Delta Lake for efficient querying.

Integration Runtime (IR) is the compute infrastructure used by Azure Data Factory to execute pipeline activities and move data. Azure Integration Runtime is used for cloud-based workloads, Self-hosted Integration Runtime connects to on-premises systems, and Azure SSIS Integration Runtime executes existing SSIS packages in Azure.

Parameters are read-only values passed to a pipeline when it starts execution and remain unchanged throughout the run. Variables are mutable values that can be updated during pipeline execution using activities like Set Variable or Append Variable.

I have worked with Copy Activity, Data Flow Activity, Lookup Activity, Get Metadata Activity, ForEach Activity, If Condition Activity, Until Activity, Web Activity, Databricks Notebook Activity, Execute Pipeline Activity, Set Variable Activity, Stored Procedure Activity, and Delete Activity.

ADF supports failure dependencies that can trigger Azure Logic Apps or Azure Functions. Logic Apps use Office 365 or SMTP connectors to send failure notifications automatically. Azure Monitor alerts can also notify operations teams whenever a pipeline execution fails.

Exception handling is implemented using On Success and On Failure dependencies. Retry policies handle transient failures, while Lookup and Validation activities verify input data before processing. Errors are logged into Azure Log Analytics or SQL tables, and alerts are sent for critical failures.

I first identify bottlenecks using ADF Monitor. Then I increase parallelism, optimize Integration Runtime, reduce unnecessary Data Flow transformations, enable partitioning, use Bulk Copy or PolyBase for SQL loading, and offload heavy transformations to Azure Databricks where Spark can process data in parallel.

Azure Blob Storage is general-purpose object storage, whereas ADLS Gen2 extends Blob Storage with a hierarchical namespace, POSIX-compliant ACLs, improved performance for analytics workloads, and better integration with Azure Databricks, Synapse Analytics, and Azure Data Factory.

The recommended production approach is to use a Service Principal or Managed Identity. The Storage Blob Data Contributor role is assigned through Azure IAM, credentials are stored securely using Databricks Secret Scopes, and Spark configurations authenticate using OAuth before accessing ADLS Gen2.

A Service Principal is an Azure Active Directory identity used by applications and services for secure, non-interactive authentication. It eliminates the need to use personal user credentials in production environments and enables role-based access control for Azure resources.

Databricks Runtime is the optimized execution environment provided by Databricks. It includes Apache Spark, Delta Lake, optimized libraries, performance enhancements, security updates, and machine learning frameworks required for large-scale distributed processing.

The Medallion Architecture organizes data into three layers. Bronze stores raw source data, Silver contains cleaned and validated data, and Gold stores aggregated, business-ready datasets optimized for reporting and analytics.

Delta Lake is an open-source storage layer built on top of Parquet that provides ACID transactions, schema enforcement, schema evolution, time travel, data versioning, scalable metadata handling, and optimized query performance. It solves problems such as data corruption, concurrent writes, and the small-file issue, making it the preferred storage format for enterprise data lakes.