Databricks Architect / Admin

Position Title: Databricks Architect / Admin Employment Type: Location: Hartford, CT (Hybrid – 3 Days Onsite / 2 Days Remote) Schedule: Monday–Friday, 8:00 AM – 5:00 PM Duration: 6-month Contract-to-Hire Team Size: 10–12 members (Onshore + Offshore) Work Environment: Agile, daily stand-up at 9:00 AM, ServiceNow workflow management Interview Process: 1–2 rounds (possible third round), panel with hiring manager and team members; technical + behavioral focus, no formal assessment. Travel: Minimal (commute only) A senior Databricks architect with deep platform administration experience who can function as a strategic advisor rather than a day-to-day operator. The ideal candidate combines strong Databricks architecture knowledge with AWS infrastructure expertise, cost optimization skills, Linux administration, and enterprise-scale data platform experience. Position Overview The Databricks Architect/Admin is a senior individual contributor responsible for architecting, governing, optimizing, and supporting the enterprise Databricks platform. This role is primarily consultative with selective hands-on responsibilities and serves as the subject matter expert for platform architecture, administration, governance, cost optimization, and enterprise integrations. The environment is already established; the selected candidate will provide strategic guidance, optimization recommendations, and ongoing support rather than building from scratch. This position is a backfill for a retiring resource and requires deep expertise to guide engineering teams and platform direction. The role requires a strong combination of: Databricks architecture expertise Advanced administration knowledge Cloud platform experience (AWS) Unix/Linux administration Cost management and optimization skills Data engineering and integration experience Strategic advisory capability Key Responsibilities Platform Architecture & Design Architect and govern enterprise Databricks environments including: Workspace topology Unity Catalog structures Access control frameworks Define and enforce: Cluster standards Runtime versions Instance pool strategies Auto-scaling policies Design scalable and performant pipelines using: Delta Live Tables (DLT) Databricks Workflows Structured Streaming Define architectural standards for: Delta Lake formats Partitioning Z-ordering OPTIMIZE/VACUUM strategies Lead integration architecture with upstream platforms: Fivetran Ab Initio Databricks Consulting & Advisory Serve as the primary Databricks consultant across teams Provide guidance on platform best practices Support technical decision-making Act as the escalation point for Databricks platform issues Mentor teams on platform usage and optimization Recommend new Databricks capabilities and roadmap enhancements Cost Management & Optimization A major focus area during the first 3–6 months. Responsibilities include: Own DBU consumption reporting and tracking Identify optimization opportunities across: Interactive clusters Jobs SQL Warehouses Detect inefficiencies such as: Excessive DBU consumption Looping jobs Misconfigured workloads Build cost attribution/chargeback models Support: Capacity planning Contract forecasting Vendor discussions Collaborate on monthly cost analyses Unix/Linux Infrastructure & Operations Administer and troubleshoot Linux environments supporting Databricks Manage: Init scripts Cluster lifecycle processes Shell scripting Build Bash and Python automation solutions Support: Monitoring Log aggregation Maintenance workflows Manage: File systems Permissions Data movement operations Perform EC2/VM diagnostics with infrastructure teams Governance, Security & Compliance Design governance frameworks in Unity Catalog Implement: Data lineage Tagging Access auditing Partner with cybersecurity teams Support: Secrets management Network isolation Encryption Maintain audit documentation and compliance artifacts Automation, AI & Machine Learning Implement automation for: Cluster lifecycle management Scheduling Alerting Self-healing workflows Leverage: AutoML MLflow Model Serving Integrate: Databricks Assistant GitHub Copilot Support scalable feature engineering and ML deployment pipelines Evaluate emerging: AI capabilities Generative AI tools LLM-integrated workflows Required Qualifications 7+ years in data engineering or platform engineering Minimum 4+ years of hands-on Databricks implementation experience Deep expertise in: Unity Catalog Delta Lake Databricks Workflows Delta Live Tables SQL Warehouses Strong Unix/Linux skills: Shell scripting Process management File systems Cron Python and PySpark expertise AWS or cloud platform experience: EC2 Storage Networking IAM/security SQL expertise for reporting and analysis Experience designing scalable and cost-efficient enterprise platforms Experience with automation frameworks Familiarity with AI/ML ecosystem Oracle database experience Git and CI/CD knowledge ServiceNow and Jira exposure Strong communication skills Preferred Q