Senior Data Streaming Engineer

Midwest Tape | hoopla Full-time, Permanent Remote in the following states only: Ohio, Michigan, Indiana, Illinois, Tennessee, Texas, Florida, Colorado, Montana, Nevada, Rhode Island, Massachusetts. Must have flexibility to travel to headquarters periodically (est. 2-3 times a year) for onboarding and meetings. We are not sponsoring applicants for work visas for this role. For over 30 years, Midwest Tape has served libraries and their patrons with unparalleled access to the best content available in the media world. Our digital offering, hoopla Digital, combines the media and library expertise of Midwest Tape with the latest in web and mobile technology, delivering an all-in-one experience to the library world. As hoopla continues to grow on a global scale, we are expanding our team. At Midwest Tape, we know our people are the reason for our success. We value our employees and believe in empowering them to help them thrive. We offer competitive pay and benefits, a healthy work/life balance, and appreciation for a job well done. Position Summary Do you thrive on designing and building high-performance streaming solutions alongside other passionate engineers? We need your expertise As a Senior Data Streaming Engineer and collaborative architect, you'll be a key player in shaping our real-time data pipelines. Leverage your in-depth knowledge of Apache Kafka and Flink (Java 17) to co-create robust streaming architectures that follow microservices principles and domain-driven design (DDD) best practices. Your expertise won't just be on paper - you'll actively collaborate and participate with other skilled streaming developers to bring your vision to life. Success in this role also includes efficient and timely communication with technical and non-technical team members, proactive coordination, a love of new technologies, a strong desire to learn, a curious nature, and a passion for removing inefficiencies. Primary Duties and Responsibilities Lead the collaborative design and implementation of event-driven, real-time streaming architectures using Apache Kafka and Apache Flink, ensuring alignment with microservices principles and Domain-Driven Design (DDD) patterns. Foster a collaborative environment with fellow streaming developers, promoting knowledge sharing, mentorship, and continuous refinement of best practices in stream processing, fault tolerance, and scalability. Architect and implement production-grade, fault-tolerant Kafka pipelines and Flink applications using Java 17, leveraging Flink's DataStream, Table, or SQL APIs to process high-volume, low-latency data streams. Ensure compliance with company policies, data governance standards, and industry regulations in all aspects of streaming development and operations. Advocate and enforce best practices for stream processing, code quality (including code reviews), testing strategies, and maintainability to build a resilient and future-proof streaming infrastructure. Engage in solution architecture discussions, provide technical guidance, and conduct thorough code reviews to uphold high standards of software craftsmanship and system performance. Drive continuous improvement by identifying and proposing enhancements to operational workflows, technical stack, and development methodologies, focusing on efficiency, scalability, and cost-effectiveness. Collaborate with fellow engineers, DevOps and operations teams to proactively monitor, troubleshoot, and optimize streaming applications. Contribute to cross-functional initiatives, knowledge sharing sessions, and documentation efforts to elevate the team's expertise in Kafka, Flink, and event-driven architectures. Skills Proven expertise in designing, implementing, and optimizing event-driven, real-time streaming architectures using Confluent Kafka and Apache Flink. Advanced proficiency in Java 17 and extensive experience with Flink's DataStream, Table, and SQL APIs for developing complex stream processing applications. Deep understanding of microservices architecture, domain-driven design (DDD) principles, and event sourcing and CQRS patterns, with a track record of applying these concepts in stream processing systems. Extensive experience with Kafka ecosystem, including Kafka Connect for scalable and fault-tolerant data ingestion/egress, and familiarity with common connectors for databases, message queues, and cloud services. Demonstrated ability to provide technical leadership, mentor team members, and foster a collaborative environment that promotes knowledge sharing and continuous improvement. Strong SQL skills, including the ability to write, optimize, and review complex queries, especially in the context of stream-table joins and windowing operations in Flink SQL. Exceptional problem-solving, debugging, and performance tuning skills, with experience in root cause analysis of issues in distributed streaming systems. Proficiency with the Spring Framework, particularly Spring Boot and Spring Cloud, for building and deploying microservices-based streaming applications. Hands-on experience with in-memory data stores like Memcached and Redis for caching, state management, and enhancing the performance of streaming applications. Solid understanding of AWS cloud for deploying, scaling, and managing streaming workloads. Experience with DevOps practices, CI/CD pipelines, and containerization technologies (Docker, Kubernetes) to streamline the deployment and management of streaming applications. Proficiency in Agile/Scrum methodologies, with experience in sprint planning, daily stand-ups, and iterative development in a data-driven environment. Familiarity with collaboration tools such as JIRA and Confluence. Excellent interpersonal, written, and verbal communication skills, with the ability to articulate complex technical concepts to both technical and non-technical stakeholders. Experience with data governance, schema evolution strategies (Avro, and Json), and ensuring data quality and consistency in streaming pipelines. Knowledge of monitoring and observability tools (DataDog) for real-time insights into streaming application performance and data flows. Good interpersonal, written, and verbal communication skills. Preferred Education Requirements B.S./M.S., in software engineering/computer science/related area, or equivalent experience Certifications: Possessing a relevant certification (e.g., Confluent Certified Developer for Apache Kafka) demonstrates your commitment to professional development in this field. Preferred Experience 8 years developing scalable, fault-tolerant full-stack or backend systems in Java, with focus on event-driven and real-time applications. Deep expertise with Flink's SQL API for stream processing, including experience with complex event processing, temporal tables, UDFs to enrich streaming data. Advanced knowledge of Flink state management using RocksDB, Apache Ignite, and optimal checkpointing strategies. Experience with schema evolution strategies (Avro, Json) in Kafka and Flink to ensure data compatibility across pipeline upgrades. Experience with Docker for containerization and Kubernetes (preferably with Helm) for orchestrating streaming deployments. Strong DevOps skills including Git, CI/CD pipelines, and Infrastructure as Code (Terraform). Knowledge of stream processing performance tuning and benchmarking. Knowledge of data modeling techniques for streaming data, including temporal modeling and SCDs. Physical Requirements / Work Environment The work environment characteristics and physical demands described here are representative of those an employee encounters while performing the essential functions of this job. Reasonable accommodations may be made to enable individuals with disabilities to perform the essential functions. Ability to regularly sit for extended periods of time Physically able to attend meetings, presentations, and training sessions Perform repetitive motion type tasks Benefits: Medical, dental, & vision insurance 401k match Profit sharing Paid vacation and personal time Flex time 10 paid holidays Company performance bonus Holiday bonus Paid time to volunteer Training & career development opportunities All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, or status as a protected veteran. LI-Remote