Tiger Analytics Interview Questions – Azure Data Engineer

Q1- What is Lazy Evaluation in PySpark?

ANS- Lazy evaluation in PySpark refers to the deferred execution of transformations like map() and filter(). These operations are not computed immediately; instead, Spark waits until an action such as collect(), count(), or show() is called. This strategy allows Spark to optimize the execution plan, reduce unnecessary data scans, and improve overall performance by minimizing resource usage.

Q2- How Does Caching Improve Performance in PySpark?

ANS- In PySpark, caching is a performance booster. When you apply .cache() or .persist() to a DataFrame, Spark stores the data in memory (or disk if memory is insufficient). This avoids recomputation for repeated actions, making iterative algorithms and repeated queries significantly faster.

Q3- What’s the Difference Between Narrow and Wide Transformations in Spark?

• ANS- Narrow Transformations (e.g., map, filter) operate within a single partition. No data movement across nodes is required, making them fast and efficient.
• Wide Transformations (e.g., reduceByKey, join) involve shuffling data between partitions or nodes. These are more resource-intensive and require careful optimization.

Q4- How to Optimize Query Performance in Azure SQL Database?

ANS- To enhance query speed and efficiency in Azure SQL Database, follow these best practices:

• Create clustered and non-clustered indexes based on query patterns.
• Use query hints and analyze execution plans for bottlenecks.
• Avoid SELECT *; always specify required columns.
• Apply partitioning and regularly update statistics.
• Monitor performance using Query Performance Insight and Dynamic Management Views (DMVs).

Q5- How to Integrate Azure Data Factory (ADF) with Synapse Analytics?

ANS- To connect ADF with Azure Synapse Analytics, follow these integration steps:

• Set up Linked Services in ADF to authenticate and connect to Synapse.
• Build pipelines to ingest, transform, or move data.
• Use Copy Activity, Spark Notebooks, or Stored Procedures for data operations.
• Monitor pipeline execution via ADF’s Monitor tab.

Q6- How to Manage Schema Evolution in Azure Data Lake?

ANS- Schema evolution in Azure Data Lake is best handled using formats like Delta Lake or Apache Parquet, which support dynamic schema changes. Tools like Azure Data Factory or Databricks Auto Loader can be configured with the mergeSchema option to automatically adapt to evolving data structures during ingestion.

Q7- SQL Query to Find the Nth Highest Salary

Here’s a clean SQL query to retrieve the Nth highest salary from an employees table:

SELECT DISTINCT salary
FROM employees e1
WHERE N - 1 = (
  SELECT COUNT(DISTINCT salary)
  FROM employees e2
  WHERE e2.salary > e1.salary
);

Replace N with the desired rank (e.g., 3 for third highest salary).

Q8-  How do you implement CI/CD pipelines for deploying ADF and Databricks solutions?

ANS-   • Use Azure DevOps/GitHub for source control

            • Integrate ADF with Git repository

            • Use ARM templates for ADF deployment

             • Use Databricks Repos, notebook export/import, and databricks-cli

             • Use release pipelines for deployment automation 

Q9-  Write PySpark code to calculate the total sales for each product category. df.groupBy("category").agg(sum("sales").alias("total_sales")).show()

* Step 1: Import Required PySpark Modules

Start by importing the necessary PySpark functions and initializing your Spark session.

from pyspark.sql import SparkSession
from pyspark.sql.functions import sum

. Step 2: Create a Sample DataFrame

Let’s define a sample dataset with product categories and sales figures.

data = [
    ("Electronics", 1200),
    ("Clothing", 800),
    ("Electronics", 1500),
    ("Groceries", 300),
    ("Clothing", 700),
    ("Groceries", 400)
]

columns = ["category", "sales"]

spark = SparkSession.builder.appName("SalesByCategory").getOrCreate()
df = spark.createDataFrame(data, columns)

. Step 3: Group by Category and Aggregate Sales

Use groupBy() to group rows by category, and agg() with sum() to calculate total sales.

sales_by_category = df.groupBy("category").agg(
    sum("sales").alias("total_sales")
)

 Explanation:

• groupBy("category") clusters rows by product type.
• sum("sales") computes total sales per group.
• .alias("total_sales") renames the output column for clarity.

 Step 4: Display the Results

Use .show() to print the aggregated DataFrame.

sales_by_category.show()

Expected Output:

+-----------+-----------+
| category  |total_sales|
+-----------+-----------+
|Electronics|       2700|
|Clothing   |       1500|
|Groceries  |        700|
+-----------+-----------+

Q10- How Do Broadcast Joins Improve Performance in PySpark?

ANS- Broadcast joins in PySpark are a powerful optimization technique used when one of the datasets is small enough to fit in memory. Instead of shuffling large amounts of data across the cluster, Spark sends the smaller dataset to all worker nodes, enabling local joins and reducing network overhead.

from pyspark.sql.functions import broadcast

df.join(broadcast(small_df), "id")

📌 Use broadcast() when joining a large DataFrame with a small lookup table. 

Q11- What Is the Role of Driver and Executors in Spark Architecture?

ANS- In Apache Spark, the architecture is divided into two key components:

• Driver Program: Acts as the control node. It maintains the DAG (Directed Acyclic Graph), schedules tasks, and coordinates execution across the cluster.
• Executors: These are worker processes that run on cluster nodes. They execute tasks, perform computations, and return results to the driver.

Q12- How to Monitor and Optimize Azure Data Factory (ADF) Pipeline Performance?

ANS- To ensure smooth and efficient ADF pipeline execution, follow these monitoring and tuning strategies:

• Use the Monitor tab to track trigger runs and activity logs.
• Enable Log Analytics integration for deeper diagnostics.
• Analyze Activity Duration and Output metrics to identify bottlenecks.
• Configure retry policies, alerts, and timeouts for resilience.
• 
  

Q13- SQL Query: Find Employees Earning More Than Department Average?

Here’s a SQL query to identify employees whose salaries exceed the average salary of their department:

SELECT e.*
FROM employees e
JOIN (
  SELECT department_id, AVG(salary) AS avg_salary
  FROM employees
  GROUP BY department_id
) d ON e.department_id = d.department_id
WHERE e.salary > d.avg_salary;

Q14-  What Is Delta Lake and Why Is It Useful?

Delta Lake is an open-source storage layer built on top of Apache Parquet. It enhances data lakes with:

• ACID transactions for reliable data operations
• Time travel to access historical versions of data
• Schema evolution for flexible ingestion
• Concurrent writes for scalable data engineering

Q15- How to Handle Schema Drift in Azure Data Factory?

ANS- Schema drift occurs when incoming data changes structure over time. In ADF, you can manage this by:

• Enabling Auto Mapping in Copy Activity
• Checking Allow schema drift to accept dynamic columns
• Using dynamic column mapping to adapt to evolving schemas

Q16- Python Code: Check if a Number Is a Palindrome

ANS- Here’s a simple Python function to verify if a number reads the same forward and backward:

def is_palindrome(n):
    return str(n) == str(n)[::-1]

print(is_palindrome(121))  # Output: True

Q17-What Is Z-Ordering in Delta Lake and Why Is It Important?

ANS- Z-ordering in Delta Lake is a data layout optimization technique that clusters related records together based on one or more columns. This improves query performance by:

• Reducing disk I/O during filtering operations
• Enhancing data skipping for faster reads
• Benefiting time-series and range-based queries

Q18- How to Perform Incremental Data Loads in Databricks?

ANS- To implement incremental data ingestion in Databricks, follow these best practices:

• Use timestamp columns or watermarking to track changes
• Filter incoming records using conditions like last_updated > last_processed
• Apply merge (upsert) logic using Delta Lake’s MERGE INTO syntax to update or insert records efficiently

Q19- What Is Adaptive Query Execution (AQE) in Apache Spark?

ANS- Adaptive Query Execution (AQE) is a Spark feature that dynamically adjusts query plans based on runtime statistics. It enables:

• Dynamic join strategy switching (e.g., broadcast vs shuffle join)
• Skewed partition handling to balance workloads
• Shuffle partition coalescing to reduce overhead

Q20- How to Optimize Data Partitioning in Azure Data Lake Storage (ADLS)?

ANS- Efficient partitioning in ADLS improves query speed and reduces storage costs. Key strategies include:

• Partition by high-cardinality or frequently queried columns (e.g., date, region)
• Avoid excessive small files by tuning file size thresholds
• Use tools like Azure Data Explorer, Databricks, or Partition Discovery for insights

Q21- How to Build a Real-Time Analytics Pipeline in Azure?

ANS- To create a real-time data pipeline for analytics, follow this architecture:

1. Ingestion: Use Azure Event Hubs or IoT Hub to stream data
2. Processing: Apply Stream Analytics or Structured Streaming in Databricks
3. Storage: Write processed data to Delta Lake, Cosmos DB, or Synapse Analytics
4. Visualization: Connect to Power BI for real-time dashboards

Q22-  PySpark Code: Perform a Left Join Between Two DataFrames

ANS- Here’s a simple PySpark snippet to perform a left outer join:

df1.join(df2, on="id", how="left").show()

 This joins df1 with df2 on the id column, keeping all records from df1 and matching rows from df2. 🔍 SEO Keywords: PySpark left join example, DataFrame join PySpark, Spark SQL joins

Q23- What Are the Security Best Practices for Azure Data Lake?

ANS-To secure your Azure Data Lake Storage (ADLS) environment:

• Implement Role-Based Access Control (RBAC) and Access Control Lists (ACLs)
• Enable encryption at rest and in transit
• Use Managed Identities for secure service-to-service authentication
• Monitor threats using Azure Defender and Log Analytics

Q24- What Is Integration Runtime (IR) in Azure Data Factory?

ANS- Integration Runtime (IR) is the compute backbone of Azure Data Factory. It handles:

• Data movement across cloud and on-premises sources
• Data transformation using mapping data flows or SSIS packages
• Supports Azure-hosted, self-hosted, and SSIS-based runtimes for hybrid scenarios

Q25: How to Design a Fault-Tolerant Architecture for Big Data Workloads?

ANS- Building a fault-tolerant big data architecture is essential for ensuring reliability, scalability, and uninterrupted processing in distributed environments like Azure, Databricks, or Hadoop. Here's how to design one step by step:

 1. Implement Retry Logic and Checkpointing

Use automated retry mechanisms to handle transient failures in data ingestion or transformation. Combine this with checkpointing to save intermediate states, allowing recovery from the last successful step instead of restarting the entire pipeline.

2. Design for Idempotency

Ensure that your data processing logic is idempotent—meaning repeated executions produce the same result. This prevents duplication or corruption during retries or reprocessing.

 3. Use Delta Lake for ACID Transactions

Leverage Delta Lake as your storage layer to enforce ACID compliance, support concurrent writes, and enable time travel for rollback and auditability. This ensures data integrity even during system failures or schema changes.

 4. Monitor, Alert, and Recover Proactively

Set up robust monitoring and alerting using tools like Azure Monitor, Log Analytics, or Databricks Observability. Implement:

• Custom alerts for pipeline failures or latency spikes
• Disaster recovery plans with backup and restore strategies
• Audit logs for traceability and compliance

 Summary

A fault-tolerant big data architecture should be:

• Resilient to failures
• Consistent across retries
• Auditable and recoverable
• Optimized for performance and integrity