Big Data Engineer - Streaming

Job Description

This role is for one of the Weekday's clients

Salary range: Rs 1900000 - Rs 2800000 (ie INR 19-28 LPA)

Min Experience: 4 years

Location: Remote (India)

JobType: full-time

We are looking for a highly skilled Big Data Engineer – Streaming to design, build, and maintain large-scale real-time data processing systems. This role is focused exclusively on streaming data pipelines and requires deep expertise in Java-based backend development along with hands-on experience in Apache Spark Streaming or Hadoop streaming ecosystems. The ideal candidate thrives in high-performance environments and has a strong passion for building scalable, low-latency data platforms.

Requirements

Key Responsibilities

• Design, develop, and optimize real-time streaming data pipelines for large-scale, distributed systems.
• Build and maintain robust data ingestion and real-time processing frameworks using Apache Spark Streaming and Hadoop-based streaming technologies.
• Develop high-performance, production-grade solutions using Java as the core programming language.
• Ensure streaming systems meet requirements for scalability, fault tolerance, reliability, and low latency.
• Integrate streaming pipelines with downstream applications, platforms, and analytics systems.
• Monitor, troubleshoot, and optimize performance bottlenecks across streaming data workflows.
• Collaborate closely with platform, infrastructure, and application teams to deliver end-to-end streaming solutions.
• Support CI/CD practices for continuous improvement and reliable deployment of data pipelines.

Technical Skills

• Strong expertise in real-time data processing and streaming architectures.
• Hands-on experience with Apache Spark Streaming and/or Hadoop streaming ecosystems (no batch processing).
• Advanced proficiency in Java for large-scale, distributed systems development.
• Experience with CI/CD pipelines and production-grade deployment practices.
• Familiarity with cloud platforms such as AWS, Microsoft Azure, or Google Cloud Platform.
• Solid understanding of system performance tuning, fault tolerance, and distributed computing concepts.

Skills

Big Data Engineering
Real-Time Streaming
Apache Spark Streaming
Hadoop Streaming
Java
Cloud Platforms (AWS / Azure / GCP)
CI/CD