We are looking for a technically sound, hands-on Azure Data Factory (ADF) Developer with experience designing, implementing, and optimizing data integration and ETL processes.
This role requires a strong understanding of data flow, transformation, and orchestration in Azure Data Factory, with an emphasis on high-performance data pipelines and complex workflows.
The ideal candidate will have a keen eye for detail, a problem-solving mindset, and a proactive approach to optimizing ADF solutions for scale and efficiency.
Responsibilities:
- Design, develop, and deploy high-performance ETL pipelines using Azure Data Factory to support data integration needs across various business units.
- Create, monitor, and optimize complex data workflows, including data ingestion, transformation, and loading to and from various Azure data services (Azure SQL Database, Azure Data Lake Storage, etc.).
- Collaborate with cross-functional teams to gather data integration requirements and translate them into technical solutions using ADF.
- Develop and manage parameterized data flows, mapping data flows, and pipelines to enable reusable and flexible data processes.
- Implement robust error handling, logging, and monitoring mechanisms to ensure data quality, reliability, and traceability.
- Optimize existing ADF pipelines for performance, cost-effectiveness, and scalability, identifying areas for improvement and implementing best practices.
- Leverage other Azure services such as Azure Data Lake, and Azure SQL Database in combination with ADF for complete end-to-end data solutions.
- Implement data processing and ETL jobs with large datasets, ensuring that jobs run efficiently in high-volume data environments.
- Document workflows, processes, and procedures for knowledge sharing and operational support.
- Participate in code reviews, ensuring adherence to best practices and coding standards.
- Troubleshoot and resolve technical issues within data workflows and ETL jobs to minimize downtime and maximize performance.
Required Skills and Experience:
- 5-6 years of hands-on experience with Azure Data Factory in a production environment.
- Strong proficiency in designing and building complex ADF pipelines, including parameterized pipelines, mapping data flows, and orchestrating data workflows.
- Experience with various ADF integration components, including Linked Services, Datasets, Pipelines, Triggers, and Data Flows.
- Expertise in SQL, with experience in querying, transforming, and analyzing large datasets.
- Familiarity with other Azure services such as Azure Data Lake, and Azure SQL Database, with the ability to integrate them within ADF workflows.
- Experience in performance tuning of ADF pipelines and understanding of data partitioning and distribution for optimized data flows.
- Proficient in data modeling, data warehousing principles, and ETL best practices.
- Knowledge of Azure DevOps for CI/CD in ADF, including version control and pipeline deployments.
- Strong analytical and troubleshooting skills to resolve complex data integration and pipeline issues.
- Effective communication skills to collaborate with stakeholders, technical and non-technical alike.
Additional Information:
- Title: ADF (Azure Data Factory) Developer
- Location: Remote
- Experience: 5 – 6 years
- Shift Time: UK shift
Send Resumes To: raj@email.xtglobal.com