Principal Architect

Principal Architect | AWS & Databricks

Experience: 15+ years

Role Overview

We are seeking a Principal Architect with 15+ years of cloud architecture experience and strong hands-on expertise designing and implementing AWS and Databricks solutions. This person will lead the architecture of scalable, secure, and high-performance cloud data platforms while working across infrastructure, data engineering, governance, and DevOps.

Key Responsibilities

Design scalable, secure, and high-performance AWS and Databricks solutions using AWS Networking, Kinesis, ElastiCache, Redshift, EKS, Cognito, and other PaaS services.Build and maintain ETL/ELT pipelines using Python and PySpark within Databricks, including notebooks, jobs, Delta Lake, and Unity Catalog.Implement Bronze, Silver, and Gold medallion architecture and optimize Spark jobs for performance, scalability, reliability, and cost.Use Terraform to provision and manage Databricks workspaces, clusters, jobs, S3/ADLS storage, networking, IAM, and related infrastructure.Develop advanced SQL for data transformation, validation, analytics, and data quality.Build CI/CD pipelines using Jenkins and GitHub Actions.Integrate data from databases, APIs, streaming platforms, and file-based sources.Optimize cloud costs through autoscaling, spot instances, scheduling, and resource management.Monitor pipelines, troubleshoot failures, and implement observability and alerting.Required Skills

AWS Networking and VPC designAWS KinesisAWS ElastiCacheAWS RedshiftAWS Lambda, S3, and GlueAWS EKSAWS CognitoDatabricks, Delta Lake, Unity Catalog, and Delta Live TablesPython and PySparkAdvanced SQLTerraformJenkins and GitHub ActionsPreferred Skills

Azure Data Lake, Data Factory, Synapse, and Key VaultBig data and dimensional modelingLakehouse architectureSpark and Databricks performance tuningData governance, lineage, and complianceQualifications

Bachelor's degree in Computer Science, Information Technology, or related fieldAWS Certified Solutions Architect - Professional preferredDatabricks Certified Data Engineer Professional or Apache Spark certification preferred