Design, deploy, and maintain Kafka clusters to support the organization’s real-time data streaming needs.
Ensure high availability, scalability, and fault-tolerance of Kafka environments.
Monitor and troubleshoot Kafka performance, latency, and throughput issues.
Implement security best practices for Kafka, including encryption, access controls, and auditing.
Leadership and Collaboration:
Lead a team of Kafka engineers, providing technical guidance and mentorship.
Collaborate with development, data engineering, and DevOps teams to integrate Kafka into various applications.
Communicate effectively with stakeholders to gather requirements and provide updates on Kafka-related initiatives.
Optimization and Innovation:
Optimize Kafka configurations for performance and resource utilization.
Stay updated on the latest Kafka features and best practices, incorporating them into the organization’s infrastructure.
Drive continuous improvement in messaging systems, exploring new tools and technologies to enhance capabilities.
Automation and Scripting:
Develop scripts and automation tools for routine Kafka tasks, such as scaling, monitoring, and deployment.
Integrate Kafka with CI/CD pipelines to streamline the deployment process.
Documentation and Compliance:
Create and maintain detailed documentation of Kafka architecture, configurations, and processes.
Ensure compliance with data governance and regulatory requirements related to messaging and data streaming.
Qualifications:
8+ years of experience with Apache Kafka, including designing and managing Kafka clusters.
5+ years experience and proficiency in Apache Kafka and related tools (e.g., Kafka Streams, Kafka Connect).
5+ years experience in Confluent Kafka Administration, Kafka Clustering concepts, Kafka Security, mTLS/RBAC
5+ years experience and strong scripting skills (Python, Shell, etc.) and experience with automation tools.
Knowledge of monitoring and logging tools (Prometheus, Grafana, ELK stack).
Familiarity with microservices architecture and event-driven systems.
Knowledge with containerization (Docker, Kubernetes) and cloud platforms (AWS, GCP, Azure).
Experience in a leadership role, managing a team of engineers or developers.
Strong background in distributed systems, real-time data processing, and messaging systems.
Experience in implementing projects
Experience in systems analysis and programming of software applications
Demonstrated Subject Matter Expert (SME) in area(s) of Applications Development
Demonstrated leadership, project management, and development skills
Strong ability to work in a highly focused technical environment together with other motivated and driven professionals, as well as the ability to work independently.
Excellent leadership and team management abilities.
Effective communication and collaboration skills.
Preferred Qualifications:
Experience with other messaging systems like RabbitMQ, ActiveMQ, Solace etc.
Certification in Apache Kafka or related technologies.
Experience with big data technologies like Hadoop, Spark, etc.
Experience with CFK (desirable)
Financial Industry experience is a plus
Education:
Bachelor’s in Computer Science, Information Technology, or a related field or equal experience
Master’s degree is a plus
Applications DevelopmentFull timeJersey City New Jersey United States$176,720.00 - $265,080.00