Pyspark Architect (Charlotte)

Pyspark Architect

Charlotte, NC-Onsite

Job Description

⦁ Participate in customer discussions to gather and analyze system requirements for performance, security and scalability.

⦁ Provide suggestions on solution architecture, contribute to tech framework development, support enterprise architecture definition, and drive design creation

⦁ Identify suitable technologies and tools available in the market, and perform a comparative analysis in line with client's business requirement, thereby identifying optimal architectural solution options

⦁ Participate in developing integration strategies based on validation of proof of concept of proposed architectural solutions

⦁ Facilitate a smooth transition by ensuring knowledge transfer and adherence to quality standards, while contributing to the build and deployment automation strategy

⦁ Collaborate with development teams to resolve complex technical issues using solution deployment and roll out methods, during build, deployment and support phases

⦁ Analyze system performance basis architectural solutions implemented and support in identifying potential risks in the system landscape

⦁ Create artifacts of architectural frameworks and technical solutions, thereby enriching team's knowledge repository

Your contribution to the team:

⦁ The ability to analyse, design, and develop scalable and secure architectures.

⦁ A knack for assessing multiple technologies, tools and frameworks to recommend optimal solutions.

⦁ Understanding of integration methodologies, proof-of-concept validation and deployment strategies.

⦁ The ability to document architectural frameworks and solutions.

⦁ Proactive communication with internal teams and clients, ensuring alignment with business objectives.

⦁ A knack to keep with up technical advancements to ensure relevant and effective recommendations.

⦁ The ability to maintain standards and ensure smooth transition through knowledge transfer

Required Skill and Experience

• Strong experience in technology consulting, enterprise and solutions architecture and architectural frameworks

• Strong experience in Cloudera Hadoop ecosystem, i.e. Hadoop, Hive, Scala, SPARK, Sqoop, Flume, Kafka, Python, NoSQL database technologies

• Good Understanding and knowledge of Hadoop architecture and various components such as high availability architecture, experience in workload management, scalability and distributed platform architecture

• Experience with design and implementation of ETL/ELT framework for complex warehouses/marts. Knowledge of large data sets and experience with performance tuning and troubleshooting

• Background in all aspects of software engineering with strong skills in parallel data processing, data flows, REST APIs, JSON, XML, and micro service architecture

Preferred Skill and Experience

• Experience in working in any hyperscalers, AWS, Azure, GCP, Databricks and Snowflake

• Experience in data virtualization/data federationfor creating data product (e.g. in Starburst) in heterogenous environment (Hadoop, RDBMS, NoSQL etc.)

• Hands-on development mentality, with a willingness to troubleshoot and solve complex problems