Job Description
What we are looking for
Were looking for a candidate with a foundational understanding of data engineering concepts and handson experience with relevant technologies. This role provides the opportunity to grow into a senior data engineering position contributing to innovative cloudbased projects.
What does the job involve
- Data Pipeline Development: Assist in implementing and maintaining scalable data pipelines for ingesting processing and transforming large volumes of data from various sources using tools such as Databricks Python and PySpark.
- Data Modeling: Assist in designing data models and schemas for efficient storage retrieval and analysis of structured and unstructured data.
- ETL Processes : Develop and automate ETL workflows to extract data from diverse sources transform it into usable formats and load it into data warehouses data lakes or lakehouses.
- Big Data Technologies: Utilize big data technologies such as Spark Kafka and Flink for distributed data processing and analytics.
- Cloud Platforms: Collaborate with the team to deploy data solutions on cloud platforms such as AWS Azure or Google Cloud Platform (GCP) leveraging cloudnative services for data storage processing and analytics.
- Data Quality and Governance: Implement data quality checks validation processes and data governance policies to ensure accuracy consistency and compliance with regulations.
- Monitoring and Troubleshooting: Monitor data pipelines and assist in troubleshooting and resolving datarelated issues.
- DevOps: Commit code to version control.
- Collaboration: Collaborate with crossfunctional teams including data scientists analysts and software engineers to understand requirements and deliver datadriven solutions.
- Documentation: Create and maintain technical documentation including data architecture diagrams ETL workflows and system documentation to facilitate understanding and maintainability of data solutions.
- Best Practices : Continuously learn and apply best practices in data engineering and cloud computing.
Qualifications
- Programming Languages: Experience with Python Java Scala or SQL for data manipulation and scripting.
- Big Data Technologies: Experience with Databricks Spark Kafka and Flink.
- Version Control : Experience with systems like Git.
- Cloud Platforms: Familiarity with AWS Azure or GCP services.
- SQL: Proficiency in SQL for database management and querying.
- Apache Spark: Knowledge of Apache Spark for distributed data processing.
- Modern Data Architectures : Exposure to modern data architectures such as lakehouse.
- ETL Tools: Knowledge of ETL tools and technologies such as Apache Airflow Informatica or Talend.
- Cl oud Data Tools : Familiarity with cloud data services like AWS S3 EMR Redshift Glue Azure Data Factory Databricks BigQuery Dataflow and Dataproc.
Benefits
Competitive Salary
Location
Remote Egypt Kenya South Africa Nigeria
Job Tags
Full time, Remote job,