The individual filling the position will also define data requirements, develop mechanisms to gather and mine structured and unstructured data and will apply modeling and visualization techniques using related Big Data Programming Languages & Technologies such as EMR, Hadoop, Map Reduce, NoSQL, Azure, AWS Cloud Tool Stack, DataBricks, Snowflake, Apache Spark, Flume, Kafka, PySpark, Python, Java and other distributed environment technologies.
- Exercises technical leadership to define and drive the evolving architecture of the entire solution, and is accountable for the success of the customer solutions
- Works closely with the business and cross-functional architects to ensure that individual program and product strategies align with business needs
- Design and implement a highly scalable system for data pipeline and data management: -Data discovery: Define data requirements, explore and assess suitable data sources -Data migration: Create Extract Transform Load pipelines to collect and mine large scales of structured and unstructured data for data processing
- Data Cleansing: Perform initial data quality checks on raw & extracted data by running various data tools in the big data environment
- Data Transformation and Integration: Extract data from identified databases, process through ELT pipelines, and curate data to a structure that is relevant to the problem by selecting appropriate techniques
- Data Modelling: Conceptualize, design and develop logical and physical data models through analysis of complex data elements, systems, data flows, dependencies, and relationships -Develop and maintain data engineering best practices and contribute to insights on data analytics and visualization concepts, methods and techniques
- Present and review project technical objectives, status and learnings with senior leadership and develop mutually beneficial strategic alliances with customers
- Actively mentor junior members of engineering staff by openly sharing experience, perspective, and holding routine code review
The must-have skill sets
- 6+ years of experience related, relevant IT experience in the field of Big Data Distributed Technologies: EMR, Hadoop, Map Reduce, NoSQL, Azure, AWS Cloud Tool Stack, Event Hub, DataBricks, Snowflake
- Have strong technical knowledge in streaming technologies: Apache SPARK, Flume, Kafka
- Expert in programming languages: Python, Java, pySpark, Hive, Scalding, Cascalog, PyCascading
- Proficient in Azure /AWS cloud tool stack using Blob, ADLS Gen2, Lambda, QuickSight, Azure ML
- Have strong experience designing, implementing, and integrating REST or GraphQL APIs
- Have strong technical knowledge of 1 or more database technologies (MongoDB, Neo4j, PostgreSQL, etc) and maintain proficient knowledge of data structures.
- Have experience deploying applications using Azure or AWS cloud providers and understanding key application metrics (e.g.; performance, security, etc.)
- Have strong technical experience in data migrations, data cleansing, transformation, export, import
- Have Broad experience in data modeling concepts • Expert in Extract Load Transform (ELT) pipelines, data mining, data warehouse and data store concepts
- Able to apply principles of logical or scientific thinking to a wide range of intellectual and practical problems and work well in group problem solving situations
- Proficient in communicating technical findings to non-technical stakeholders
- Thrive in a fast-paced environment
- Have pragmatic problem-solving skills, which enable you to navigate ambiguous or loosely defined requirements
- Demonstrated an ability to drive adoption of technology recommendations which then have a direct measurable impact.
- Able to define the software implementation strategy and visualize, architect, and implement high performing software solutions to meet business goals
Equal employment opportunity
Rezilyens is an equal opportunity employer and is dedicated to fostering an inclusive and diverse environment for employees from all walks of life. We hire based on talent and we’re proud of our global perspective.