workflow for job creation and submission

Overview: How a request is pull down to agent?

This quick reference page describes how the request is propagated through Global Queue, Local Queue and WMAgent

Base Information

There are two status mentioned in document. (request status is different from WorkQueue element status)

Request status (starts with lower case) is the collection of WQE (workqueue elements) status (starts with upper case).

DB stated here are all couchdbs (reqmgr_workload_cache, workqueue, workqueue_inbox).

Both workqueue (GQ, LQ) each contains two dbs (workqueue_inbox and workqueue)

Site threshold is getting from SSB. (Code is here)

Pending threshold is defined by percentage of available slots. (Code is here)

From ReqMgr to GQ

calling queueNewRequest()

1. GQ pulls "assigned" request from reqmgr db. (reqmgr_workload_cache couchdb)
2. Populate workqueue_inbox db with request. (WQE in workqueue_inbox is in "Negotiating" status)
3. Split requests by blocks from workqueue_inbox and insert to workqueue db (WQE is in "Available" status)
4. Update WQE in workqueue_inbox db to "Acquired" status and in turn change the request status in reqmgr db to "acquired"

There is no specific restiction for populating GQ. If GQ is not populated, most like team is not set for the request. After GQ is populated the status of request should change to "acquired" and GQE status is "Acquired" for workqueue_inbox and "Available" for workqueue

code reference

step 1-4 done by queueWork()

step 3 done by processInboundWork() - called in queueWork()

From GQ to LQ

calling pullWork() 1-4 / processWork() 5

1. Gets the resource information from wmbs resource db.
   • slots are counted as pending + running slots for each site (don't subtract running jobs)
   • drain site is excluded
2. Pulls available elements from GQ and match with the resources
   • randomly select the site from sites returned from step 1 above, then
   • check the site whitelist/blacklist with input block location and pile up data location (also xrootd option)
   • sort WQE by priority check whether given site has empty slots (by counting only jobs with higher or same priority as WQE) - This means higher priority WQE will gets matched first also if it has the same priority, request created first matches first.
3. If there is a match, updates GQE with LQ address and set GQE status to "Negotiating"
4. Replication is set to replicate the data from GQ. (GQE with LQ address will be replicated to LQ workqueue_inbox), This means all LQE in workqueue_inbox have exactly identical elements (including id) in GQ workqueue db. But # of GQE > # of LQE.
5. split work from LQ workqueue_inbox to workqueue (set WQE status in workqueue to "Available"). Elements in workqueue_inbox and workqueue are identical except the status

At this point, GQE status in workqueue_inbox and workqueue status changed to "Acquired" and WQE status in workqueue_inbox and workqueue are "Acquired" and "Available" respectfully

code reference

step 1-3 done by pullWork()

step 1 done by freeSlots()

step 2 done by avaiableWork()

step 4 is set by replication in AgentStatusWatcher

step 5 done by processInboundWork() - called in queueWork()

From LQ to WMBS

calling getWorks()

1. Gets the resource information from wmbs resource db.
   • slots are counted as pending + running slots for each site but subtract running jobs
   • include even for drain site
2. Pulls available elements from LQ and match with the resources
   • randomly select the site from site from 1 then
   • check the site whitelist/blacklist with input block location and pile up data location (also xrootd option)
   • sort WQE by priority check whether given site has empty slots (by counting only jobs with higher or same priority as WQE) - This means higher priority WQE will gets matched first also if it has the same priority, request created first matches first.
3. Creates wmbs subscription.

At this point LQE status in workqueue db set to "Running" and propagate to all the GQ db (workqueue and workqueue_inbox) - To do: couldn't find the code doing this yet.

code reference

step 1-3 done by getWork()

Input data (block) location update

Both GQE and LQE locations are updated periodically from cronjob and WorkQueueManager respectfully using PhEDEx block replica call. (Only "Available" WQE locations are updated) So if data is copied in new location after request is assigned it will still update locations for "Available" WQE.

Dataflow diagram