Senior/Leader Data Engineer (Database, MySQL, SQL)

We are looking for a Senior/Leader DataOps Engineer to join our team.

The Data Engineer is responsible for day-to-day technical development and delivery of data pipelines into StorAI data. You will ensure delivery of solutions based on the backbone of good architecture, best data engineering practises around operational efficiencies, security, reliability, performance and cost optimisation.

To be successful in this role, you have to be an experienced Data Engineer with demonstrated knowledge and experience within data delivery, ETL/ELT, data management, solution design and delivery. Critical to success will be strong communication skills, analytical mindset, excellent documentation skills and have demonstrable experience in working with a variety of business stakeholders.

Core Responsibilities

• Develop ETL/ELT solutions using Azure Bob Storage, Data Factory), Spark, and Python to load data across multiple sources into Azure Snowflake.
• Design, implement and maintain a robust, efficient and compliant data pipeline architecture.
• Create and maintain documentation on data pipelines
• Support various components of the data pipelines, including ingestion, validation, cleansing and curation.
• Manage and ensure the success of ongoing data pipeline routines.
• Help the team to maintain an overall position of strong data governance with great data observability (data monitoring & alerting, CI/CD pipelines).
• Build the infrastructure required for optimal acquisition, transformation, and provisioning of data from a variety of data sources.
• Design enhancements, updates, and programming changes for portions and subsystems of data pipelines, repositories, and models for structured/unstructured data.
• Engage with our software engineering team to ensure precise data points per application specifications
• Work with customer service, devops, and quality assurance teams to move software through the pipeline to the customer