- Getting started
- The concepts
- First steps
- The prototypes
- The Tasks
- Composing Samplings
- The Samplings
- The Domains
- The Factors
- The Environments
- The Hooks
- The Sources
- The concepts
A big feature in OpenMOLE is the possibility of delegating the workload to a remote execution environment.
Delegation / grouping
OpenMOLE has been desinged so that the delegation a part of the workload to a remote environment is declarative. To acheive it first define the environment, then tells OpenMOLE witch task to delegate.
val env = BlaBlaEnvironment(....) val mole = task1 -- (task2 on env) -- task3
The use of a batch environment is generally not suited for short tasks (< 1 minute for a cluster to < 1 hour for a grid). In case your tasks are short you can group several executions. To group the execution by 100 in each job submited to the environment you should like that:
// requiered for grouping with the keyword 'by' import org.openmole.grouping.batch._ val env = BlaBlaEnvironment(....) val mole = task1 -- (task2 on env by 100) -- task3
By default OpenMOLE tasks run on a single thread. To use several cores of a coputer you can define a so called local environment and parameter the number of threads.
// For using 10 threads / cores val env = LocalEnvironment(10)
Delegation to the EGI grid
To delegate a task on EGI you need to register your certificate in OpenMOLE. In the console execute:
import org.openmole.plugin.environment.glite._ GliteAuthentication() = P12Certificate(encrypted, "/path/to/your/certificate.p12")
You need to execute this operation only once and for all. OpenMOLE persist this information.
Other certificate types are also available: PEMCertificate and ProxyFile.
//For pem GliteAuthentication() = PEMCertificate(encrypted, "/path/to/the/cert", "/path/to/the/key") //For proxy file GliteAuthentication() = ProxyFile("/path/to/the/proxy")
The sources for the glite authentication are available here.
Delegate a task on a virtual organization
Some virtual organization are predifined in OpenMOLE: biomed, francegrilles, complexsystems. To delegate a task on biomed for instance just specify in your workflow definition:
import org.openmole.plugin.domain.collection._ import org.openmole.plugin.domain.distribution._ import org.openmole.plugin.domain.modifier._ import org.openmole.plugin.sampling.combine._ import org.openmole.plugin.task.groovy._ import org.openmole.plugin.environment.glite._ // Declare the variable val i = Prototype[Int]("i") val j = Prototype[Int]("j") // Define the Hello task val hello = GroovyTask("hello", "println('Hello world! ' + i + ' ' + j)") hello addInput i hello addInput j //Define the exploration strategy val exploration = ExplorationTask( "exploration", Factor(i, 0 to 99 toDomain) x Factor(j, new UniformIntDistribution take 10) ) //Define the workflow val ex = exploration -< (hello on biomed) toExecution //Start the execution ex.start
Defining a virtual organisation
In addition to the predifined VO you can define yours quite easily. There is 2 reasons for defining your own VO: either it is no predifined in OpenMOLE or you want to specify requirements.
val env = GliteEnvironment("biomed")
Requirements could be specified for a given environment:
- cpuTime: the duration for the job
- memory: the memory for the job
- openMOLEMemory: the memory of attributed to OpenMOLE on the worker none, external tasks will run in a separated memory space and other task will run withi OpenMOLE memory
- achitecture: could be “x86_64″ for restricting your jobs to 64 bit systems
- jobType: should be set to “MPI” for MPI jobs
- cpuNumber: the number of CPU requiered for your job
- smpGranularity: the number of cores linked to a shared memory (same as cpuNumber for multi-threaded processes
val env = GliteEnvironment( "biomed", cpuTime = "PT4H", memory = 2000, architecture = "x64_86", jobType = "MPI", cpuNumber = 8, smpGranularity = 1, openMOLEMemory = 200 )
Delegation an OpenMOLE desktop grid
OpenMOLE provides a desktop grid. To use it with your workflow you might use the folowing environment definition:
import org.openmole.plugin.environment.desktopgrid._ val env = DesktopGridEnvironment(10000, "test", "test")
The first parameter is the port on which the desktop grid server is listening. The to othe parameters are a login and a password to connect to the desktop grid server.
Once your workflow has been started you can use the daemon (available for download in the download section) to execute job on other computers. Once the daemon achive extracted launch the folowing command:
./openmole-daemon -h email@example.com:port -p test -w nbWorker
For windows machines use the run.bat script with the same arguments.
For info the source code of the DesktopGridEnvironment is available here.
Delegation a SSH server
First you should initialize the authentication infomation to access the server.
To authenticate with login / password:
SSHAuthentication(0) = LoginPassword("login", encrypted, "machine.domain")
To authenticate with a private key:
SSHAuthentication(0) = PrivateKey( "/path/to/the/private/key", "login", encrypted, "machine.domain" )
The last part “machine.domain” can alse be only the “domain” if a same authentication is suitable for an entire domain.
Then you can use the SSH environment:
import org.openmole.plugin.environment.ssh._ val env = SSHEnvironment( "login", "machine.domain", 10 )
The number 10 correspond to the maximum number of concurrent process you want to launch on the remote server.
Delegation a PBS / Torque cluster
PBS is a venerable batch system for clusters. OpenMOLE enable to use the computationnal power provided by those kind of environment. For the authentication part, use the same method as ssh and configure your access to the master of the cluster.
Then you can use the PBS environment:
import org.openmole.plugin.environment.pbs._ val env = PBSEnvironment( "login", "machine.domain" )
Optionnaly you can define requirements cpuTime, memory and openMOLEMemory in the same way as the glite environment. You may also set the queue by using the attribute queue = “name of the queue”.