QA Automation Engineer | Hybrid | BGC, Taguig

Work Setup: Hybrid (2-3x onsite/week), Shifting schedules

Qualifications:

• At least 3 years of experience in Automation Testing using either PyTest/Selenium (Python)
• Has API testing experience (Postman or REST)
• Can start ASAP (May or Early June)

Job Description:

• End-to-End AI Pipeline Testing: Test entire GenAI and Agent Assist workflows, from input ingestion to model output. Validate correctness, safety, relevance, and performance of LLM responses (e.g., via Vertex AI).
• Prompt & Response QA (LLMs): Test prompt templates, ensure consistent outputs across scenarios. Create test matrices for hallucination detection, context drift, and grounding effectiveness.
• Speech-to-Text Accuracy Validation: Benchmark transcription quality (STT) under various audio conditions. Validate punctuation, speaker diarization, latency, and keyword recognition.
• Webhook and Dialogflow Flow Testing: Perform unit and integration tests for webhook logic (Node.js, Python). Test all Dialogflow CX states, transitions, fallbacks, slot-filling, and error recovery.
• Test Automation Frameworks: Build test scripts using PyTest, Selenium, Postman, REST Assured, or custom harnesses. Automate CI testing pipelines for webhooks, STT, and Agent Assist models.
• Data Validation & Quality Assurance: Test ETL pipelines for schema validation, null/missing values, type mismatches. Verify structured output consistency from GenAI responses, metadata tagging, and logging formats. Synthetic Test Case Generation: Use tools like Mockaroo, Faker, and custom scripts to generate test data (user prompts, transcripts, intents).
• Conversation Performance Testing: Measure latency and response times for Dialogflow CX, Agent Assist, and LLM calls. Use tools like JMeter or Locust for simulating concurrent conversation sessions.
• Model Regression Testing: Compare model outputs before/after updates using snapshot comparisons and semantic similarity checks.
• Stress Testing for Real-Time Systems: Test how Agent Assist behaves under high-concurrency scenarios (live calls, rapid context switches).
• GCP-Aware Testing: Run tests against cloud functions, Pub/Sub, and Vertex AI endpoints in GCP staging environments.
• CI/CD Test Integration: Integrate QA workflows into GitLab/GitHub Actions, Cloud Build pipelines for continuous deployment.
• Monitoring and Logging Validation: Validate audit logging, error tracking (Stackdriver), and alerts for fallbacks or system anomalies.
• Bug Lifecycle & Triage: Maintain traceable test cases (e.g., in TestRail, Zephyr), manage defect logs, perform RCA.
• UAT & Stakeholder Testing Support: Guide SMEs through UAT, gather structured feedback, and iterate test cases accordingly.
• Test Reporting & KPIs: Track test coverage, bug density, response accuracy rates, and resolution SLAs in weekly reports.