Our client is seeking a Data Engineer (Databricks) responsible for building, orchestrating, and optimizing Azure-based data pipelines and transformations to deliver reliable data into enterprise lakehouse and warehouse platforms.(Remote, USA)
Must Have skills
- Databricks (Data Engineering)
- DLT
- Azure Data Factory
- SQL
- PySpark
- Synapse (Dedicated SQL Pool)
- Azure DevOps
- Python
- Azure Function Apps
- Azure Logic Apps
Responsibilities
- Create and enhance data pipelines leveraging existing ingestion frameworks and tools.
- Orchestrate data pipelines using Azure Data Factory.
- Develop/enhance data transformations to parse, transform, and load data into Enterprise Data Lake, Delta Lake, and Enterprise DWH (Synapse Analytics).
- Perform unit testing and coordinate integration testing and UAT.
- Create pipeline documentation including HLD, DD, and runbooks.
- Configure compute, implement data quality rules, and manage pipeline maintenance.
- Conduct performance tuning and optimization.
- Provide production support and operational troubleshooting.
Other information
-
Primary platforms/tools: Azure Data Factory, Databricks (DLT), Synapse Dedicated SQL Pool, Azure DevOps, Python/PySpark, Azure Function Apps, Azure Logic Apps.
#LI-SB1
