Data Engineer | International
|Position Type:||Full Time|
|Attachments:||No File Attached|
|Application Close Date:||05-Apr-2018|
- Multinational company – leader in its industry
- Exciting opportunity in a start-up function
- Asia Division, based in Singapore
IAG Firemark Lab is the innovation offshoot for IAG, a leading Australian general insurer on a mission to create a safer world for our customers and communities. We are a multi-disciplinary team building next generation products in collaboration with partners, from start-ups to universities and corporates.
About the role
The Data Engineer role ensures that data pipelines are scalable, repeatable and secure. It helps facilitate getting data from a variety of different sources, getting it in the right formats, adhering to data quality standards, and assuring that downstream users can get data quickly.
Additional responsibilities include developing prototypes and proof of concepts for the selected solutions, and implementing complex data projects with a focus on collecting, parsing, managing, analysing and visualising large sets of data using multiple platforms. Data Engineers should also have the capability to architect highly scalable distributed systems, using open source tools.
The Data Engineer role is a technical job that requires substantial expertise in a broad range of software development and programming skills. These professionals have knowledge of data analysis, end user requirements analysis, and business requirements analysis to develop a clear understanding of the business needs and to incorporate these needs into technical solutions. They have a solid understanding of physical database design principles, and the software development life cycle.
Skills and experience we're looking for
- Proficient understanding of distributed computing principles
- Extensive coding and programming in a range of languages such as Python, Scala, Java, R, etc.
- Proficiency with open source data storage platforms such as Hadoop
- Experience with building stream-processing systems, using solutions such as Storm or Spark-Streaming
- Experience with NoSQL databases, such as Cassandra
- Knowledge of various ELT techniques and frameworks, such as Flume
- Experience with various messaging systems, such as Kafka
- Good understanding of Lambda Architecture, along with its advantages and drawbacks
How to Apply
To submit your interest in this role, please click the ‘Apply' button below or write to email@example.com.