Data Engineer Cost Management

Job Description

We are looking for a skilled Data Engineer with strong expertise in Databricks and backend development to join our team. The ideal candidate will have hands-on experience in building scalable data pipelines, working with cloud-based databases, and supporting cost management initiatives through efficient data processing and optimization.

Key Responsibilities:

• Design, develop, and maintain scalable data pipelines using Databricks and PySpark
• Build and optimize backend services for data processing and integration
• Work with AWS Aurora and other databases to manage and process large datasets
• Collaborate with cross-functional teams to support cost management and analytics use cases
• Implement and manage data streaming solutions using Kafka
• Deploy and manage containerized applications using Kubernetes
• Develop serverless solutions using AWS Lambda
• Ensure data quality, performance tuning, and cost optimization of data workflows

Required Skills (Max 6):

• Strong experience with Databricks and PySpark
• Hands-on experience with AWS Aurora (or similar relational databases)
• Experience with Kafka for data streaming
• Working knowledge of Kubernetes (K8s)
• Experience with AWS Lambda (serverless architecture)
• Strong backend development skills

Nice to Have Skills:

• Experience with MongoDB, PostgreSQL, or advanced SQL
• Exposure to cloud cost optimization / cost management projects
• Familiarity with distributed systems and big data architectures