Job Description
Job Responsibilities: • Install, configure, and deploy the Databricks Unified Analytics Platform in accordance with best practices and organizational requirements. Set up clusters, workspaces, notebooks, and other system objects as needed. • Manage user accounts, groups, permissions, and access controls within the Databricks platform. Define role-based access controls (RBAC). • Configure and manage Databricks clusters to optimize performance, scalability, and resource utilization. Monitor cluster health, adjust cluster configurations, and scale clusters up or down as needed to meet workload demands. • Monitor and optimize the performance of Databricks workloads, queries, and jobs. Identify and address performance bottlenecks, optimize Spark configurations, and fine-tune resource allocation for optimal performance. • Implement and enforce security measures to protect data and resources within the Databricks platform. • Provide support on network security, encryption, authentication, and auditing controls to ensure compliance with regulatory requirements and security policies. • Integrate Databricks with other data sources, data warehouses, and analytics tools as part of the data ecosystem. Automate routine tasks and workflows using Databricks APIs, command-line interfaces (CLIs), and scripting languages. • Monitor system health, resource utilization, and performance metrics using monitoring tools and dashboards. Set up alerts and notifications. • Develop and implement backup and recovery strategies to protect against data loss and ensure business continuity. Perform regular backups, validate backups, and test recovery procedures to minimize downtime. • Forecast future growth and resource requirements based on current usage patterns and business projections. Scale Databricks resources such as clusters and storage as necessary.
Responsibilities
Required Experience & Skills: • Specific experience in administering Databricks Unified Analytics Platform, minimum of 4 years of experience. • Strong understanding of big data technologies, distributed computing, and Apache Spark architecture. • Hands-on experience with cloud computing platforms such as AWS, Azure, or Google Cloud Platform. • Familiarity with infrastructure as code (IaC) tools such as Terraform or CloudFormation. • Excellent troubleshooting and problem-solving skills, with the ability to diagnose and resolve complex technical issues. Strong communication and collaboration skills, with the ability to work effectively in cross-functional teams.