AZURE DATA ENGINEER – CAPE TOWN
- Full time
- Cape Town, Western Cape, South Africa View on Map
- @Raw Ambition posted 2 days ago
- Posted : November 30, 2023 -Accepting applications
- Salary: R60,000.00 - R120,000.00 / Monthly
- View(s) 27
Job Detail
-
Job ID 68949
-
Career Level Senior Professional
-
Experience 5 Years
-
Qualifications Bachelor's Degree
Job Description
Intermediate – Senior, 5+ years experience
Job Description:
- Design and implement scalable, high-performance data pipelines to acquire, ingest, transform, and load data from various sources into data storage systems.
- Collaborate with data scientists and analysts to understand data requirements and ensure the availability of clean, reliable, and well-structured datasets.
- Develop and maintain ETL (Extract, Transform, Load) processes to move data between systems and prepare it for analysis.
- Ensure data quality, integrity, and consistency by implementing data validation, cleansing, and transformation techniques.
- Optimize and tune data pipelines for performance and efficiency, considering factors like data volume, processing speed, and resource utilization.
- Work with both structured and unstructured data, and integrate data from different sources, such as databases, APIs, logs, and external datasets.
- Implement data warehousing solutions to provide a unified view of data for reporting and analytics purposes.
- Collaborate with cross-functional teams to understand business needs and translate them into data solutions.
- Stay up to date with the latest data engineering technologies, tools, and best practices, and apply them to enhance data infrastructure.
- Monitor and troubleshoot data pipeline issues, performance bottlenecks, and data quality problems.
- Implement data security and privacy measures to ensure compliance with regulations and protect sensitive data.
- Document data engineering processes, pipelines, and solutions for knowledge sharing and future reference.
Requirements:
- Bachelor’s or masters degree in computer science, Engineering, Information Systems, or a Related field.
- Proficiency in programming languages commonly used in data engineering, such as Python, Java, Or Scala.
- Experience with data processing frameworks and tools, such as Apache Spark, Apache Kaia, or Apache Airflow.
- Strong SQL skills for data manipulation and querying in relational databases.
- Familiarity with cloud platforms (e.G., AWS, Azure, Google Cloud) and their data services. Knowledge of data modelling and database design principles.
- Understanding of data warehousing concepts and technologies.
- Experience with version control systems (e.G., Git) and collaborative development practices.
- Ability to work with large datasets and optimize data processing workflows.
- Problem-solving skills and attention to detail in identifying and resolving data-related issues.
Required skills
Related Jobs
-
November 30, 2023AZURE SOFTWARE DEVELOPER – CAPE TOWN Full time
Cape Town, Western Cape, South Africa -
November 30, 2023
AZURE SOFTWARE DEVELOPER – CAPE TOWN Full time
Cape Town, Western Cape, South Africa