Senior Data Engineer
StarHub
- Kuala Lumpur
- Permanent
- Full-time
- Data Architecture Design: Design and implement scalable and efficient data architecture, including data models, data warehouses, and data lakes. Understand the business requirements and design appropriate data models, data pipelines, and data warehouses.
- Data Integration: Integrate data from various sources such as databases and APIs into a unified format for analysis. Develop ETL (Extract, Transform, Load) processes and real-time data pipelines.
- Data Modeling: Design and implement dimensional and data models to support data warehouses, and analytical and reporting needs.
- Data Pipeline Development: Develop and maintain robust ETL processes and data pipelines for ingesting, processing, and transforming large volumes of data from various sources. Ensure data quality, reliability, and consistency throughout the pipelines.
- Performance Optimisation: Optimise the performance of data processing, visualization, and storage systems, including database tuning, query and ETL processes optimisation, and infrastructure scaling, to ensure timely and efficient data access.
- Data Governance and Security: Establish and enforce data governance policies and procedures to ensure data integrity, privacy, and compliance with regulations and internal policies. Manage access controls, encryption, and auditing of data.
- Tool and Technology Selection: Evaluate and select appropriate tools and technologies for data storage, processing, and visualisation.
- Collaboration and Communication: Collaborate with cross-functional teams such as data scientists, analysts, and business stakeholders to understand their requirements and deliver data solutions that meet their needs. Communicate technical concepts effectively to non-technical audiences.
- Documentation and Knowledge Sharing: Document data pipelines, processes, and best practices to facilitate knowledge sharing and ensure the maintainability of data solutions. Promote a culture of documentation and knowledge sharing within the team.
- Continuous Learning and Improvement: Stay updated with the latest trends, advancements, technologies, and best practices in data engineering through continuous learning and self-improvement.