Lead I - Data Engineerings

UST

  • Bayan Lepas, Pulau Pinang
  • Permanent
  • Full-time
  • 1 month ago
Job Description:Key Responsibilities:
  • Design, build, and maintain robust data pipelines using Azure Data Factory (ADF) to ingest data from diverse sources including APIs, Excel files, Azure Data Lake Storage Gen2 (ADLS Gen2), and SharePoint, ensuring secure access and efficient data flow.
  • Configure and manage SharePoint integration within ADF pipelines, handling authentication and access criteria effectively.
  • Develop complex data transformations using ADF Mapping Data Flows, applying best practices to create scalable, maintainable ETL/ELT workflows.
  • Implement and champion metadata-driven architecture for enhanced scalability, modularity, and reusability of data pipelines.
  • Perform rigorous data validation and testing to maintain high data quality and accuracy across ingestion and transformation stages.
  • Build dimensional models and apply data warehousing concepts to structure data optimally for analytics and reporting.
  • Leverage Azure Databricks for big data processing, data engineering, and advanced analytics use cases.
  • Use REST APIs effectively to source data, handle pagination, authentication, and error handling.
  • Design and optimize data storage and analytical solutions using Azure Synapse Analytics, balancing performance and cost.
  • Apply core data engineering concepts, including ETL/ELT design, pipeline orchestration, incremental loads, and monitoring.
  • Understand and contribute to Azure Fabric architecture for scalable distributed systems (advantageous).
  • Utilize Azure DevOps for CI/CD pipeline creation and management, enabling automated deployment and version control of data solutions (plus).
  • Develop serverless event-driven solutions using Azure Functions to support data workflows and processing tasks (plus).
  • Implement automated workflows and notifications using Azure Logic Apps integrated with data pipelines (plus).
  • Collaborate with cross-functional teams to understand business requirements and translate them into scalable technical data solutions.
  • Monitor and troubleshoot production pipelines to ensure data availability, reliability, and performance.
Technologies & Tools:
  • Azure Data Factory (ADF) — Pipeline orchestration, data ingestion, Mapping Data Flows
  • Azure Synapse Analytics — Data warehousing, big data analytics, serverless SQL pools
  • Azure Data Lake Storage Gen2 (ADLS Gen2) — Scalable data lake storage
  • Azure Databricks — Apache Spark-based big data processing and analytics
  • Azure Logic Apps — Workflow automation and integration (email notifications, s)
  • Azure Functions — Serverless compute for event-driven processing
  • Azure DevOps — CI/CD pipeline automation and source control
  • REST APIs — Data ingestion from external APIs including pagination and authentication
  • SharePoint Integration — Secure access and data ingestion from SharePoint sources
  • Metadata-driven Architecture — Scalable and reusable pipeline design methodology
  • Data Warehousing Concepts — Dimensional modeling, star schema, snowflake schema
  • Data Engineering Concepts — ETL/ELT, pipeline design, incremental load, data validation
  • Azure Fabric (plus) — Distributed systems and microservices platform
What we believe:We’re proud to embrace the same values that have shaped UST since the beginning. Since day one, we’ve been building enduring relationships and a culture of integrity. And today, it's those same values that are inspiring us to encourage innovation from everyone to champion diversity and inclusion, and to place people at the center of everything we do.Humility:We will listen, learn, be empathetic and help selflessly in our interactions with everyone.Humanity:Through business, we will better the lives of those less fortunate than ourselves.Integrity:We honor our commitments and act with responsibility in all our relationships.Equal Employment Opportunity StatementUST is an Equal Opportunity Employer.All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, or status as a protected veteran.UST reserves the right to periodically redefine your roles and responsibilities based on the requirements of the organization and/or your performance.Skills:Azure Data Factory,Azure Databricks,Azure SynapseAbout Company:UST is a global digital transformation solutions provider. For more than 20 years, UST has worked side by side with the world’s best companies to make a real impact through transformation. Powered by technology, inspired by people and led by purpose, UST partners with their clients from design to operation. With deep domain expertise and a future-proof philosophy, UST embeds innovation and agility into their clients’ organizations. With over 30,000 employees in 30 countries, UST builds for boundless impact—touching billions of lives in the process.

UST