Sr. Data Engineer-Azure Databricks

Job Title: Sr. Data Engineer
Location: Houston, TX
Length: 6-10 months
 
Job Summary
We are seeking a skilled Data Engineer with extensive, hands-on expertise in Azure Databricks to design, develop, and manage our enterprise data platform. In this architecture, all core transformations will be built directly within Databricks using Databricks Workflows and Jobs. The ideal candidate must have deep experience with Unity Catalog, DevOps (CI/CD), and the ability to drive Synapse migrations into code-driven Databricks environments.

Key Responsibilities
• Core Databricks Engineering: Design, build, and optimize end-to-end data pipelines entirely within Azure Databricks and Delta Lake using PySpark, SQL, or Scala.
• Orchestration & Automation: Build, schedule, and maintain complex production pipelines utilizing Databricks Workflows and Jobs as the primary orchestration engine.
• Data Legacy Migration: Lead and execute the migration of legacy data architectures into optimized Databricks solutions.
• Pipeline Conversion: Translate and refactor legacy business logic (such as converting historical Alteryx workflows) into clean, scalable Databricks notebook code.
• Data Governance: Implement and manage data access, security, schemas, and lineage across the platform using Databricks Unity Catalog.
• CI/CD & DevOps Automation: Develop, maintain, and automate Databricks workspace deployments and jobs using YAML and Classic CI/CD pipelines in Azure DevOps.

Certifications (Highly Prioritized)
• Active Professional Certifications are strongly preferred, such as:
o Databricks Certified Data Engineer Associate / Professional
o Microsoft Certified: Azure Data Engineer Associate (DP-203)
o Microsoft Azure Fundamentals (AZ-900) or equivalent foundational Azure knowledge is required at a minimum.'

• Alteryx Conversion: Familiarity with Alteryx workflows is a major plus, specifically to assist in reverse-engineering and converting legacy visual workflows into PySpark/SQL code. (Note: Candidates do not need to be Alteryx developers, but must be comfortable reading/migrating the ETL logic).