What you'll do...
You're right for the job if you are comfortable leading Major Incident Response with a technical team of engineer’s laser focused on restoring service across complex distributed systems. To successfully achieve this, you will draw upon your knowledge of the tech stack and tools to surface key data. You'll excel if you have enthusiasm for digging deep, and a flare for sharp technical communication, prioritization and organization. You will work directly with our SRE, Engineering and DevOps teams to support our next generation “always up” cloud-based technology platforms.
What you'll do:
The TDO will take command and control of Major Incidents focusing on restoration by identifying and coordinating with appropriate resources through all the phases of triage, restoration and validation. You will understand the technology stack and use this knowledge to ensure systems continue to meet production ready standards, Operational Excellence is key! Good judgement is crucial as you will own and deploy critical switches for command/control mitigation tooling. Your ability to continuously challenge yourself and develop a strong network with peers and stakeholders cross functionally will see you exceed in this role. Our goal is to protect the customer, merchant and associate experience and deliver outstanding levels of availability across Walmart Global Technology. To do so, you will need strong skills in the following areas:
- Control incident management processes and procedures.
- Calm under pressure when controlling major incident response.
- Excellent end to end technical understanding of core infrastructure, cloud services, platforms and micro-services.
- Ability to understand traffic flows and key dependencies between services.
- Ability to understand and capture key data from various sources, systems and people.
- Ability to effectively triage – be able to detect and determine symptom vs cause.
- Act as a technical leader and coach within the CCC.
- Analyze trends to pro-actively prevent incidents.
- Focus on leading immediate restoration vs root cause.
- Identify and drive continuous improvement efforts to reduce waste (eliminate, automate or streamline).
- Development of monitoring and alerting frameworks
- Absorb knowledge and understand complex distributed systems - ability to share and impart this knowledge into your peer group and beyond.
- Strong focus on collecting and inferring metrics.
- Influence the design of system architecture and tactical solutions.
- Provide data for and actively participate in root cause analysis partnering with the Problem Management function
- Design and implement technical solutions and process improvements to improve detection and resolution response that prevent repeat issues.
- Understanding and consideration of business strategies and priorities.
- Coach and help develop CCC engineers.
- Familiar with log centric tooling, ideally Splunk. Produce time series data and reusable dashboards for use both during and post event.
- Familiar with metrics creation and query languages (e.g., Graphite, TSDB, Prometheus) to produce dashboards and surface data for specific events.
- Define the standards and requirements for new service onboarding, ensuring they are fit of production.
- Drive standardization and service focused instrumentation.
What you'll bring:
- 6+ years in an infrastructure, systems, engineering or development environment delivering operational excellence to highly complex distributed systems.
- Bachelor's Degree in Computer Science or a related field, or relevant work experience.
- Strong and demonstrable incident management skills with relevant experience in an enterprise organization.
- Experience and exposure working in a 24/7 operations support environment.
- Methodical and systematic problem-solving approach, combined with a solid awareness of ownership, initiative and drive.
- Experience investigating, analyzing and troubleshooting large scale enterprise systems.
- Networking knowledge and understanding of network concepts, such as different protocols (TCP/IP, UDP, ICMP, etc.), MAC addresses, IP packets, DNS, OSI layers, and load balancing).
- Experience administering Unix/Linux in a production environment.
- Experience working with and developing enterprise monitoring/tooling/logging solutions like Grafana, Kibana, Splunk, Graphite, Nagios, New Relic, DynaTrace and Prometheus.
- Working knowledge of one or more cloud technologies such as AZURE, GCP, OpenStack.
- Experience with distributed version control like Git or similar
- Familiarity with continuous integration/deployment processes and tools such as Jenkins, Maven, Nexus, etc.
- Programming experience in one or more of the following languages: Go, Java, Python, Shell, etc.
- Experience in data science/machine learning would be advantageous.
Benefits: Beyond our great compensation package, you can receive incentive awards for your performance. Other great perks include 401(k) match, stock purchase plan, paid maternity and parental leave, PTO, multiple health plans, and much more.
The above information has been designed to indicate the general nature and level of work performed in the role. It is not designed to contain or be interpreted as a comprehensive inventory of all responsibilities and qualifications required of employees assigned to this job. The full Job Description can be made available as part of the hiring process.
You will also receive PTO and/or PPTO that can be used for vacation, sick leave, holidays, or other purposes. The amount you receive depends on your job classification and length of employment. It will meet or exceed the requirements of paid sick leave laws, where applicable.
For information about PTO, see
Eligibility requirements apply to some benefits and may depend on your job classification and length of employment. Benefits are subject to change and may be subject to a specific plan or program terms.
For information about benefits and eligibility, see
The annual salary range for this position is $90,000.00-$180,000.00 Additional compensation includes annual or quarterly performance bonuses. Additional compensation for certain positions may also include: - Stock
Minimum Qualifications... Outlined below are the required minimum qualifications for this position. If none are listed, there are no minimum qualifications.
Option 1: Bachelor's degree in computer science, computer engineering, computer information systems, software engineering, or related area and 3 years’ experience in software engineering or related area.Option 2: 5 years’ experience in software engineering or related area.
Preferred Qualifications... Outlined below are the optional preferred qualifications for this position. If none are listed, there are no preferred qualifications.
805 Se Moberly Ln, Bentonville, AR 72712, United States of America