Data Engineer
We are looking for skilled Data Engineers to join our team. As a Data Engineer, you will be responsible for understanding business and technological challenges, developing data pipelines to tackle those challenges, and ensuring their smooth deployment. You will also be responsible for applying standard industry and within-the-company good practices and advancing our various patterns.
Responsibilities:
- Communicate with the Data Architect and your peers on the technical
solution you’re developing and communicate with the Project Manager
in charge of the project you’re working on.
- Write and communicate on new or updated interface contracts.
- Strong understanding of data warehousing concepts, data lakes,
ETL/ELT processes, and data modeling.
- Develop data pipelines based on the defined architecture.
- Ensure the regular good practices are applied.
- Deploy requested infrastructure, particularly using Terraform.
- Make peer reviews and ask your peers to review your code when
merging a new version of the codebase.
- Define tests with your project manager, based on the functional and
technical requirements of the pipeline you’re developing.
- Perform those tests and communicate regularly on the results.
- Regularly summarize the results of your tests in a dedicated document.
- Present to the Data Architect in charge of the architecture, and the Lead DataOps, the development performed through our Deployment.
- Track and communicate any potential errors in the entire period of
active monitoring following a deployment.
- Ensure diligent application of the deployment process, logging, and
monitoring strategy.
Requirements:
-4+ years of experience working as a Data Engineer.
- Proficiency with PySpark and Spark SQL for data processing.
- Experience with Databricks using Unit Catalog.
- Knowledge of Delta Live Tables (DLT) for automated ETL and workflow orchestration in Databricks.
- Familiarity with Azure Data Lake Storage.
- Experience with orchestration tools (e.g., Apache Airflow or similar) for building and scheduling ETL/ELT pipelines.
- Knowledge of data partitioning and data lifecycle management on cloud-based storage.
- Familiarity with implementing data security and data privacy practices in a cloud environment.
- Experience with Terraform and knowledge of good practices of GitOps.
If this sounds like you, send us your CV at the email below, and let’s talk!
talent.europe@99x.io