We're seeking a Data Engineer II who will own the near real-time data infrastructure powering our LLM-based insights platform for WW FBA. This role focuses on building high-performance streaming pipelines, optimizing embedding freshness, and implementing global latency strategies to ensure we deliver up-to-date, low-latency insights worldwide. You will play critical role in scaling AI-driven analytics across multiple regions while balancing performance and cost.Key job responsibilities
- Design and implement streaming data pipelines to process high-volume, near real-time data from multiple sources.
- Build and maintain the infrastructure supporting large language models, including embedding generation, vector storage, and retrieval systems.
- Develop and optimize a modern data lakehouse to support both batch and real-time analytics workloads.
- Implement caching strategies, query optimization, and multi-region deployment to achieve sub-second response times.
- Balance performance requirements with cost considerations through efficient resource utilization and workload optimization.
- Ensure data reliability, freshness, and compliance across the entire data pipeline.
- 3+ years of data engineering experience
- 4+ years of SQL experience
- Experience with data modeling, warehousing and building ETL pipelines
- 5+ years in data processing and AWS services (Kinesis, MSK, Lambda, Glue).
- Strong SQL, Python, and performance tuning expertise.
משרות נוספות שיכולות לעניין אותך