The Azure Data Engineer will be responsible for designing, developing, and maintaining data management and processing solutions on the Azure platform. They will work closely with Data Science and development teams to create scalable and high-performance data architectures.
Responsibilities:
Design and development of data pipelines: Design, develop, and maintain ETL (Extract, Transform, Load) data pipelines using Azure services such as Azure Data Factory, Azure Databricks, or Azure Synapse Analytics.
Data management: Integrate, transform, and manage data from various sources (databases, APIs, files, etc.) using Azure technologies.
Performance optimization: Optimize data processing workflows to ensure system performance, scalability, and security.
Data architecture implementation: Develop and maintain data architecture solutions using Azure services like Azure SQL Database, Azure Blob Storage, Azure Data Lake, and Azure Synapse Analytics.
Collaboration with teams: Work closely with Data Scientists, data analysts, and IT teams to design solutions that meet business needs.
Big Data management: Implement Big Data solutions using Azure services such as Azure HDInsight, Azure Databricks, or Azure Synapse.
System monitoring and maintenance: Monitor the performance of data pipelines and resolve any performance or data integrity issues promptly.
Required Skills:
Technical:
Expertise in using Microsoft Azure, including Azure Data Factory, Azure SQL Database, Azure Data Lake, Azure Databricks, Azure Synapse Analytics, and Azure Blob Storage.
Proficiency in programming languages such as SQL, Python, Scala, or Java for data processing.
Knowledge of ETL/ELT tools and data integration processes.
Experience in managing large-scale data and using Big Data services (Azure HDInsight, Spark).
Familiarity with version control tools and DevOps practices for continuous deployment of data solutions.
Database Skills:
Strong knowledge of relational (SQL) and non-relational (NoSQL) databases.
Experience with real-time data management systems like Kafka or Azure Event Hub.
Project Management and Communication:
Ability to work collaboratively with cross-functional teams in an agile environment.
Excellent communication skills to present technical results to non-specialists.
Data Security:
Knowledge of best practices for data security and access management on Azure.
Experience and Qualifications:
Master’s degree in Computer Science, Data Engineering, or a similar field.
3 to 5 years of experience as a Data Engineer, with strong expertise in the Azure ecosystem.
Microsoft Azure certifications (e.g., Microsoft Certified: Azure Data Engineer Associate) are a plus.
Experience with cloud data architectures and Big Data solutions.
