The Compact Muon Solenoid (CMS) is one of the experiments recording high energy particle collisions at the Large Hadron Collider (LHC) at European Organization for Nuclear Research (CERN) in Geneva, Switzerland. CMS is operating a large and distributed computing infrastructure to handle CMS data and simulation at petabyte scales. CMS Computing Operations takes care of central workflows starting with processing data just recorded at CERN to processing data and producing Monte Carlo simulations at distributed computing centers.
We are seeking highly qualified Mid-Level Devops Engineer to work on CRAB, the CMS analysis tools for physics, which features a workflow and software implementation based on a client-server architecture with very thin clients communicating to a central server using RESTful http architecture.
CRAB server functionalities are currently splitted among a REST interface (a multi-component service which uses both a relational and a non-relational DataBase for communication and intermediate cache) and a currently single component server which is isolated from user but fetches work request from the DB and transform them into Direct Acyclic Graph (DAG) descriptions which are submitted to a pool of grid scheduler nodes. CRAB heavily relies on HTCondor DAGMAN for bookkeeping, state tracking and submission of jobs execution to remote nodes. The various components are currently deployed on CERN's OpenStack Cloud as individual VM's and managed via puppet. An evolution into Dockers and Kubernetes is in progress. Responsibility of the Devops Engineer includes to install and routinely upgrade the server code, to monitor server operation and address problems either by tuning system configuration or by requesting and coordinating changes with tool developers. The operator will also help in troubleshooting problems due to the system internals. The operator will contribute to and take responsibilities in the Open Stack, Puppet and Monitoring components, in particular the Monitoring part will require constant work and adaptation to changing conditions.
The successful candidate is expected to work on:
Provide backup support for Puppet in CMS Computing operations at large;
Understanding and improving the performance of CRAB
Contribute to the operation of the submission infrastructure of CMS, in particular those components directly related to CRAB submission;
Contribute to the configuration of the submission infrastructure with the distributed Grid and Cloud sites;
Provide backup support for Submission Infrastructure operations at large;
The candidate must have a Ph.D. degree or equivalent experience in Computer Science or other related areas.
The skills and qualifications we are looking for include, but not restricted to:
Experience with Linux service management and administration
Experience with python;
Experience in shell scripting
Experience with Puppet is not required, but highly desirable
Earlier experiences on the following subjects are considered a bonus:
Experience with batch systems and high throughput computing on GRID/CLOUD
Experience with HTCondor
Experience with Grafana and/or Kibana
The position is sponsored by the Compact Muon Solenoid experiment. The successful applicant will work at the European Organization for Nuclear Research in Geneva, Switzerland. This appointment is initially for 1 year and could be extended for up to an extra year.
The monthly income will be established accordingly to the academic qualification and previous experience of the candidate. In general, the income will be compatible to the equivalent level of a São Paulo Research Foundation (FAPESP) research internship abroad.
The deadline is 30 June 2019. We reserve the right not to make an appointment and continue searching after the closing date. Only shortlisted candidates will be contacted. Shortlisted candidates must be available for an interview.
The Advanced Institute for Artificial Intelligence (AI2) will be responsible for making a pre-selection of the candidates. The final decisions will be made by the Offline and Computing Group at CERN. AI2 is committed to the principles of equal opportunity, inclusivity, and diversity. Women remain underrepresented in STEM careers and we encourage their application for AI2 positions. We welcome applications from qualified candidates with any sexual orientation and members of minority groups.