AZURE DATA ENGINEER – CAPE TOWN
Job ID 68949
Career Level Senior Professional
Experience 5 Years
Qualifications Bachelor's Degree
Intermediate – Senior, 5+ years experience
- Design and implement scalable, high-performance data pipelines to acquire, ingest, transform, and load data from various sources into data storage systems.
- Collaborate with data scientists and analysts to understand data requirements and ensure the availability of clean, reliable, and well-structured datasets.
- Develop and maintain ETL (Extract, Transform, Load) processes to move data between systems and prepare it for analysis.
- Ensure data quality, integrity, and consistency by implementing data validation, cleansing, and transformation techniques.
- Optimize and tune data pipelines for performance and efficiency, considering factors like data volume, processing speed, and resource utilization.
- Work with both structured and unstructured data, and integrate data from different sources, such as databases, APIs, logs, and external datasets.
- Implement data warehousing solutions to provide a unified view of data for reporting and analytics purposes.
- Collaborate with cross-functional teams to understand business needs and translate them into data solutions.
- Stay up to date with the latest data engineering technologies, tools, and best practices, and apply them to enhance data infrastructure.
- Monitor and troubleshoot data pipeline issues, performance bottlenecks, and data quality problems.
- Implement data security and privacy measures to ensure compliance with regulations and protect sensitive data.
- Document data engineering processes, pipelines, and solutions for knowledge sharing and future reference.
- Bachelor’s or masters degree in computer science, Engineering, Information Systems, or a Related field.
- Proficiency in programming languages commonly used in data engineering, such as Python, Java, Or Scala.
- Experience with data processing frameworks and tools, such as Apache Spark, Apache Kaia, or Apache Airflow.
- Strong SQL skills for data manipulation and querying in relational databases.
- Familiarity with cloud platforms (e.G., AWS, Azure, Google Cloud) and their data services. Knowledge of data modelling and database design principles.
- Understanding of data warehousing concepts and technologies.
- Experience with version control systems (e.G., Git) and collaborative development practices.
- Ability to work with large datasets and optimize data processing workflows.
- Problem-solving skills and attention to detail in identifying and resolving data-related issues.