Data Engineer – Orchestration & Ingestion (Apache Airflow / Apache Beam) - Sysgen RPO
Sysgen Makati Full-time
Data Engineer – Orchestration & Ingestion (Apache Airflow / Apache Beam)
Position Summary
We are seeking a highly skilled Data Engineer to lead orchestration and ingestion efforts for a modern data warehouse platform built on Google Cloud. This role requires expert-level proficiency in either Apache Airflow (via Cloud Composer) or Apache Beam (via Cloud Dataflow), with strong Python development skills.You will be responsible for designing, automating, and monitoring robust data pipelines—both batch and streaming—that ensure reliable, scalable, and observable data movement across the platform.
Key Responsibilities- Design, build, and manage orchestration frameworks for data pipelines using Apache Airflow or Apache Beam
- Author complex, dynamic, and maintainable DAGs or Beam pipelines in Python, implementing advanced dependency management, triggers, and scheduling logic
- Serve as a subject matter expert on orchestration best practices, including idempotency, modularity, backfilling, and performance tuning
- Architect scalable batch and streaming ingestion pipelines using Dataflow and Cloud Functions
- Ensure data integrity through exactly-once processing and robust validation logic
- Manage pipeline lifecycle including GCS staging, BigQuery loading, and GCP CLI automation
- Establish CI/CD workflows for deployment and testing using Git and Google Cloud Build
- Implement comprehensive monitoring and alerting strategies using Google Cloud’s operations suite
- Collaborate with ingestion and transformation teams to align workflows and dependencies
- Communicate pipeline health and status to stakeholders and proactively resolve operational issues
- Expert-level Python proficiency for orchestration and pipeline development
- Proven experience with Apache Airflow, including DAG authoring, dependency management, and SLA enforcement
- Hands-on experience with Cloud Composer environment setup, scaling, and security
- Strong background in Apache Beam and Dataflow for building efficient batch and streaming pipelines
- Experience with Google Cloud Functions for event-driven ingestion
- Familiarity with CI/CD tools and workflows, especially Git and Cloud Build
- Deep understanding of monitoring and alerting using Google Cloud’s operations suite
- Competency with GCP CLI and GCS best practices for automation and storage
- Strong experience with BigQuery API for programmatic data loading and schema management
- Demonstrated expertise in implementing idempotent logic and data validation frameworks
Artech Technology IncQuezon City, 11 km from Makati
Job Summary:
We are looking for a skilled Data Engineer – Data Platforms with strong experience in Python and modern data tools. The ideal candidate will work on building and managing data pipelines, integrating platforms, and supporting advanced...
Makati
Engineer (Automation) should have:
• A degree in Computer Science, Information Technology, or a related field.
• Proficiency in automation tools and programming languages relevant to cyber security and data security.
• Strong analytical skills to assess...
IBMQuezon City, 11 km from Makati
Location: Quezon City - UP Ayala Technohub
Schedule: Mid - Night Shift
Work Set-up: Hybrid (3x a week onsite)
Your role and responsibilities
A Data Engineer with expertise in Data Platforms specializes in developing applications on Big Data...