As a Backend Engineer, you will hold a crucial role in designing, developing, and maintaining our data platform, supporting critical data-driven decision-making processes across the organization. You will collaborate closely with cross-functional teams, including data scientists, analysts, and software engineers, ensuring the smooth flow of data and optimizing data pipelines. The ideal candidate will possess a strong background in backend engineering, with proven expertise in Java (including Spring Boot), Kafka, Data Pipelines, AWS, and ETL processes.
- Lead the design, development, and maintenance of efficient and scalable data pipelines, facilitating data collection, processing, and transformation from diverse sources.
- Implement real-time data streaming solutions using Kafka, ensuring timely data ingestion and availability.
- Utilize Java, including Spring Boot, to build robust and high-performance data processing applications within our data platform.
- Collaborate closely with cross-functional teams to comprehend data requirements, identify opportunities for data optimization, and support data-driven initiatives.
- Uphold data integrity, reliability, and availability by implementing effective ETL processes and conducting data quality checks.
- Leverage AWS services for data storage, processing, and analytics, adhering to security and performance best practices.
- Monitor and troubleshoot data pipeline performance, proactively identifying bottlenecks and implementing optimizations.
- Stay updated with industry trends and emerging technologies in data engineering, contributing to continuous improvement.