Senior Streaming Architect/Engineer

Job Title: Senior Streaming Architect/Engineer

Location: Columbus, OH

Work Arrangement: Onsite

Employment Type: W2 Contract

Job Description:

Experience: 12+ years overall (8+ years for Senior Engineer level); 5+ years in event streaming

About the Role

We are looking for a Senior Streaming Architect / Engineer to design, build, and run our enterprise real-time data platform on Confluent Cloud and Apache Flink. You will lead the move from legacy batch and ETL workloads to a cloud-native, event-driven architecture on AWS and GCP. You will set technical direction, mentor engineers, and work with business and executive stakeholders.

Key Responsibilities:

• Architect multi-region, highly available Confluent Cloud environments with DR strategies (Cluster Linking, cross-region replication) supporting 200K+ events/sec.

• Design and build real-time pipelines in Apache Flink (DataStream, Table, and SQL APIs): event-time processing, watermarks, late-event handling, windowed aggregations (tumbling, hopping, session), and stream joins.

• Implement stateful, fault-tolerant processing with checkpointing, savepoints, and exactly-once semantics; tune parallelism, state backends, and operator chaining.

• Build and manage Kafka Connect source/sink connectors (CDC, RDBMS, S3, analytics platforms) with SMTs and DLQ/retry patterns.

• Own schema governance through Schema Registry (Avro/JSON/Protobuf) and compatibility rules.

• Develop event-driven microservices and Kafka Streams / ksqlDB applications using Java and Spring Boot.

• Lead migrations of on-prem middleware and Confluent Platform workloads to Confluent Cloud and AWS (EKS) with zero-downtime cutover.

• Automate infrastructure with Terraform (Confluent, AWS, GCP providers) and GitOps CI/CD (Harness, Jenkins).

• Implement secure connectivity (Private Service Connect, VPC, PrivateLink), RBAC, ACLs, SSL/SASL, and audit logging.

• Set up observability (Prometheus, Grafana, CloudWatch, Splunk); lead incident response, RCA, and performance tuning.

• Optimize partitioning, retention, and capacity to reduce infrastructure cost.

• Lead and mentor Kafka/Flink engineers, run training sessions, and communicate architecture trade-offs to leadership.

Required Qualifications

• 12+ years in software, integration, or data engineering, with 5+ years hands-on with Apache Kafka / Confluent in production.

• Strong hands-on experience with Apache Flink (DataStream API and Flink SQL), including state management and checkpoint tuning.

• Deep Kafka knowledge: partitioning, replication/ISR, retention, compaction, Kafka Connect, Kafka Streams, ksqlDB.

• Experience running Confluent Cloud in an enterprise setting on AWS and/or GCP.

• Advanced Java (11+) and Spring Boot; proficiency in Python and SQL.

• Experience with Terraform, Docker, and Kubernetes (EKS / GKE / OpenShift).

• Experience with CI/CD tools (Jenkins, Harness) and monitoring stacks (Prometheus, Grafana, Splunk).

• Proven cloud migration experience, from assessment and planning through cutover.

• Strong communication skills and experience with stakeholders in regulated industries.

Preferred Qualifications

• Confluent Certified Developer / Administrator for Apache Kafka; Confluent Certified Data Streaming Engineer (Flink).

• AWS Solutions Architect certification.

• Experience with Amazon MSK, Managed Service for Apache Flink, Kinesis, EMR, Glue, and MWAA.

• PySpark experience for batch and streaming ETL.

• Familiarity with Snowflake, BigQuery, PostgreSQL, Oracle, and DB2 as sources or sinks.

• Experience with CDC patterns and legacy integration platforms (IBM MQ, IIB, mainframe).

• Domain background in healthcare, insurance, banking/finance, or retail.

Tools & Technologies:

Confluent Cloud, Apache Kafka, Apache Flink, ksqlDB, Kafka Streams, Kafka Connect, Schema Registry, AWS (EKS, MSK, S3, IAM, VPC), GCP (GKE, Pub/Sub, PSC), Terraform, Kubernetes, Docker, Jenkins, Harness, Java / Spring Boot, Python, PySpark, Prometheus, Grafana, Splunk.

Work Authorization: Applicants must be legally authorized to work in the United States.

Equal Employment Opportunity

Themesoft Inc is an Equal Employment Opportunity employer. We consider qualified applicants without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, age, disability, veteran status, or any other status protected by applicable federal, state, or local law.

Thanks & Regards,

Vignesh

[email protected]