Site Reliability Engineer (Infrastructure, Virtualization, & Compute)

SITE RELIABILITY ENGINEER (INFRASTRUCTURE, VIRTUALIZATION, & COMPUTE)

SpaceX is looking for an experienced engineer with deep working knowledge of virtualization, infrastructure and compute. This employee will be a member of the Information Technology Infrastructure team and will provide expertise in virtualization/compute, maintenance, scaling and optimization in support of critical business functions. The ideal candidate will be flexible and flourish in a fast paced and challenging environment. They should be a self-starter, self-motivator and possess ingenuity to excel at this position.

RESPONSIBILITIES:

• Design, install, operate, scale, and optimize enterprise compute, virtualization, and storage platforms, including hypervisor clusters, purpose-built servers, and high-performance storage (NAS, SAN, and object storage).
• Work with SpaceX engineers to gather requirements and to design, deploy, and support resilient, high-performance systems for engineering, production, and mission workloads.
• Perform capacity planning and performance engineering, and implement high-availability, backup, replication, and disaster recovery for virtual machines, datastores, and storage arrays.
• Install, configure, patch, and decommission servers, hypervisor hosts, and storage systems, and contribute to hardware and platform standards.
• Automate provisioning, patching, and day-to-day operations with scripting, templates, and infrastructure as code to reduce toil.
• Document standards and best practices, and partner with network, facilities, security, and application teams on connectivity, power and cooling, access, and workload placement.

BASIC QUALIFICATIONS:

• Bachelor’s degree in Computer Science or a STEM discipline and 1+ years of systems engineering experience; OR 3+ years of systems engineering experience in lieu of a degree.
• Experience deploying and supporting Linux servers in physical and virtualized environments (e.g. VMware via automation).
• Experience with the Linux shell as well as configuring and extending Linux instances (e.g. kernel modules, cgroups, pki, iptables, interfaces).

PREFERRED SKILLS AND EXPERIENCE:

• Experience in creating repeatable, reliable, scalable systems architectures, with high availability, fault tolerance, performance tuning, monitoring, and statistics/metrics collection.
• Experience operating production virtualization platforms such as VMware vSphere/vCenter (clustering, HA, DRS, vMotion, datastore design) and/or Hyper-V or KVM.
• Experience architecting and operating enterprise storage, including protocols and platforms such as NFS, SMB/CIFS, iSCSI, Fibre Channel, NVMe-oF, NetApp, Pure Storage, and object storage.
• Deep expertise generating server and host design specifications (CPU, memory, NIC, HBA, local vs. shared storage) to meet defined performance, availability, and density requirements.
• Experience with array- and hypervisor-based snapshots, replication, and backup integration for virtual machine and storage recovery.
• Proven ability to diagnose compute and storage performance issues (IOPS, latency, queue depth, CPU ready/co-stop, datastore contention, fabric/pathing).
• Experience implementing configuration management, provisioning, and workflow automation via infrastructure as code and CI/CD (e.g. Ansible, AWX/Tower, Puppet, cloud-init, Jenkins).
• Experience with programming and scripting languages such as Python, PowerShell, or Golang to automate day-to-day tasks and integrate with external systems through RESTful APIs.
• Experience implementing proactive alert and monitoring workflows and dashboards for Linux systems and virtualization platforms using Prometheus, Grafana, InfluxDB, or similar technologies.
• Experience supporting and scaling containerized applications in Linux environments.
• Experience using automation frameworks (e.g. Ansible, Terraform) to manage provisioning and post-provisioning lifecycles of infrastructure and Kubernetes installations.

ADDITIONAL REQUIREMENTS:

• Must be willing to work extended hours and weekends as needed.

ITAR REQUIREMENTS:

• To conform to U.S. Government export regulations, applicant must be a (i) U.S. citizen or national, (ii) U.S. lawful, permanent resident (aka green card holder), (iii) Refugee under 8 U.S.C. § 1157, or (iv) Asylee under 8 U.S.C. § 1158, or be eligible to obtain the required authorizations from the U.S. Department of State. Learn more about the ITAR here.