Apply via OneDayHire - Fastest Job Connect Starts Here!

Roles and Responsibilities

• Design and develop scalable and efficient data pipelines using Azure Data Factory, Azure Databricks, Kafka, and other relevant tools.
• Extract data from various applications using APIs, webhooks, or other methods.
• Ingest and process real-time data streams using Kafka.
• Transform and clean data to ensure data quality and consistency.
• Load data into ADLS Gen2 in optimized formats (e.g., Parquet, Delta Lake).
• Build and maintain data models and schemas.
• Collaborate with data analysts and business stakeholders to understand data requirements and deliver actionable insights.
• Optimize data pipelines for performance and cost-efficiency.
• Implement data governance and security best practices.
• Mentor junior data engineers and provide technical guidance.

Skills and Qualifications

• Strong proficiency in Python, Scala, or Java programming languages.
• Expertise in data engineering tools and technologies such as Azure Data Factory, Azure Databricks, ADLS Gen2, Kafka, and Delta Lake.
• Experience with cloud computing platforms, preferably Azure.
• Knowledge of data modelling, ETL/ELT processes, data warehousing, and real-time data processing concepts.
• Strong problem-solving and analytical skills.
• Ability to work independently and as part of a team.
• Excellent communication and interpersonal skills.
• Bachelor's degree in computer science, engineering, or a related field.

Preferred Qualifications

• Knowledge of machine learning and AI concepts.
• Certifications in Azure or related technologies.
• Experience with Node.js, Java or .NET Core for developing connector APIs.

Must-Have Technologies

• Python
• Scala or Java
• Azure Data Factory
• Azure Databricks
• ADLS Gen2
• Kafka
• Delta Lake
• SQL

This role offers a unique opportunity to work on challenging data engineering projects and contribute to the growth of our data-driven organization.