Data Engineer

Position Accountability Statement:

This role requires a deep understanding of data engineering concepts, cloud platforms, and infrastructure automation.

Data Warehouse Architecture & Design:

• Design, implement, and manage the architecture and infrastructure of the data warehouse, ensuring that it meets business and technical requirements.
• Develop and maintain scalable data models, ensuring that the data warehouse is optimized for query performance and data retrieval.
• Create and manage data schemas, ensuring they align with industry best practices and business needs.
• Design the data warehouse for optimal scalability, ensuring it can handle large volumes of data and high query loads.
• Develop strategies for optimizing query performance and managing large datasets (e.g., partitioning, indexing, caching).
• Optimize the performance of the data warehouse by refining database structures, indexing strategies, partitioning, and optimizing queries.
• Troubleshoot performance bottlenecks, both in the data warehouse and within the ETL processes, and resolve issues to ensure smooth operations.

ETL Development & Data Integration:

• Design, implement, and optimize ETL (Extract, Transform, Load) pipelines to move and transform data from various sources into the data warehouse.
• Ensure data integrity, accuracy, and consistency during the ETL process
• Integrate data from multiple sources (databases, APIs, flat files, etc.) into the data warehouse.
• Develop processes to handle data from structured, semi-structured, and unstructured sources.
• Implement Continuous Integration (CI) and Continuous Deployment (CD) practices

• Ensure that data pipelines and processing systems are continuously monitored for failures, performance degradation, or anomalies.
• Implement and monitor data quality controls to ensure that data in the warehouse is accurate, consistent, and clean.
• Develop automated validation checks to ensure data integrity and perform regular audits on data accuracy.

Data Governance:

• Design and implement systems to control who has access to different datasets, enforcing security measures such as encryption and role-based access control (RBAC).
• Apply data masking techniques or anonymize data to comply with privacy regulations such as GDPR or HIPAA.
• Implement and enforce policies that classify data based on sensitivity or usage, such as personal data, financial data, or confidential business data.
• Track and document the data's journey through various systems, showing where it comes from, how it’s transformed, and where it’s stored.

Collaboration with Stakeholders:

• Work closely with data analysts, business intelligence teams, and data scientists to understand their data needs and provide high-quality data from the warehouse for reporting and analysis.
• Collaborate with IT teams to ensure that the data warehouse infrastructure aligns with overall company architecture and cloud platform

Data Analysis Support:

• Oversee and guide the analysis of complex datasets to extract meaningful insights for strategic decision-making.
• Review and validate analytical reports and outputs from the team to ensure high-quality deliverables.
• Analyze large and complex datasets to uncover trends, patterns, and business opportunities.

Qualifications:

• Bachelor’s degree in Computer Science, Engineering, Information Systems, or a related field.
• 3 to 5 years of experience in data engineering, data management, or similar roles with a focus on data warehousing.
• Strong experience with data engineering concepts and best practices.
• Hands-on experience with designing and implementing large-scale data systems, building and optimizing data pipelines, and working in cloud environments is highly valued.
• Strong proficiency in SQL, PL/SQL, T-SQL and similar technologies.
• Proficient in cloud platforms and related data technologies (Oracle Cloud Infrastructure would be a big plus).
• Hands-on experience with data warehousing solutions and ETL/ELT processes (Oracle

• Strong knowledge of data governance principles, including data quality, privacy, and compliance.
• Familiarity with Data Warehouses and data lakes is highly desirable (e.g., Redshift, Snowflake, Synapse, BigQuery, SAP Hana, Db2, s3, ADLS gen2, BigLake)
• Familiarity with DevOps practices, including CI/CD, infrastructure automation, Docker, and Kubernetes is a plus.
• Familiarity with Python, PySpark, Pandas and other relevant libraries is a plus.
• Ability to communicate technical concepts to non-technical stakeholders and document solutions clearly.
• Ability to work closely with data scientists, analysts, and business stakeholders to understand data requirements and optimize solutions.
• Strong analytical and troubleshooting skills to handle complex data issues and improve data flow efficiency.
• In-depth knowledge in MS EXCEL/Google Shee
• Strong experience with data visualization tools (e.g., Oracle Analytics, Sisense, Power BI).