New PythonTask Interface in Work Queue | The Cooperative Computing Lab

The most recent version of Work Queue supports two different categories of tasks. Standard Task s describe a Unix command line and corresponding files, just as before. The new PythonTas k describes a Python function invocation and corresponding arguments:

def my_sum(x, y):
import math
return x+y
task = wq.PythonTask(mysum,10,20)
queue.submit(task)

When a task is returned, the function value is available as t.output :

task = queue.wait(5);

if task:

print("task {} completed with result {}".format(task.id,task.output))

Underneath, a PythonTask serializes the desired function and arguments, and turns it into a standard task which can be remotely executed, using all of the existing capabilities of Work Queue. And so, a task can be given a resource allocation, time limits, tags, and everything else needed to manage distributed execution:

task.specify_cores(4)

task.specify_gpus(1)

task.specify_tag("hydrazine")

Thanks for new CCL graduate student Barry Sly-Delgado for adding this new capability to Work Queue! See the full documentation.

Enjoy Reading This Article?