Salary 10% superannuation + banking package
Consultant Natasha Keith
Date posted 3 December 20192019-12-03 2020-02-01 it Wellington City Wellington NZ 6011 NZD 100000 1400000 1400000 YEAR Robert Walters https://www.robertwalters.co.nz https://www.robertwalters.co.nz/content/dam/robert-walters/global/images/logos/web-logos/square-logo.png
Work for a large financial private sector client who is seeking an expereinced Data Engineer.
We are seeking a Data Engineer to work on various projects using Kafka and Spark, as you will be using Kafka to source data from different sources and write it into topics and then use Spark as the transformation engine using Scala as the preferred language. The data is then stored in HDFS as parquet and Airflow is used as the scheduling tool.
Ideally you will have good knowledge in Scala and be able to write automated unit test cases for their transformations.
One project you could be working on is their real time platform, for example if you changed your address details, it gets ingested in their Data Warehouse and it is processed in overnight batch processing, so the information is loaded the next day and they have changed this to using a fast time data platform so they can upload everything in real time.
You will be using Spark and Scala to build their Business Transformations and ideally, you will have experience in building CI/CD pipelines from scratch, and previously designed and built solutions for data science and analytics.
You must possess Big Data experience helping the team work toward streaming data at scale enabling them to handle high velocity and high-volume data events. You must demonstrate a willingness to learn, adapt, and share knowledge while adding value to great existing team culture.
Finally, you will be working in a cross-functional teams in a fully Agile environment.
Apache data ecosystem: HDFS, Spark and Kafka
Knowledgeable in these concepts:
- Parallel and concurrent processing
- Distributed systems
- Virtualisation, especially AWS and VMWare
- Containerisation using Docker and Kubernetes
- Linux OS configuration and use, including shell scripting
- Java programming, JVM configuration and tuning
- Python programming and virtual environments
- Git source control with standard Git work-flows
- Network fundamentals and host-level routing
- Distributed system design and configuration
- Relational databases and SQL queries
- Security engineering practices and tools
- Streaming data at scale
- Handling high velocity, high volume data events
- Automated unit testing
- Agile and DevOps engineering practices and CI/CD tools
Experience in some of these are an advantage, equally important is a willingness to learn, and experience in similarly complex systems. Additional skills and competencies are desirable, including testing background, operations, networking and event-based architectures.
You care about DevOps, Agile, Open Source Software, Data, and participating in a great team culture.
Robert Walters endeavours to review all applications in a maximum of five working days. If you have not received correspondence within this timeframe please do not hesitate to contact Natasha Keith on 04 471 9729.