There are many cases where users or administrators would want to limit the usage of their on premise resources to allow only a limited jobs to run at the same time, or when you have a limit on your cloud resources but you still want to run your experiments.
cnvrg enables powerful and smart compute chooser where you can control on how many jobs to run on each compute.
For Cloud compute it could be simply the limit value for that instance, for On Premise compute it could be the number of jobs that can run in parallel (i.e. number of gpus, number of cpu per job, etc.)
Choosing more that one compute could be useful in cases like:
- Cloud compute: There is a limit on the number of instances you can have at the same time - if the first compute the user chose is not available, cnvrg will try to allocate all the options the user has chose and prioritized.
- On Premise Compute: Limited number of parallel jobs: When choosing a new compute resource a user will see how many free compute jobs are free on each compute and will be able to choose and prioritize its resources.
If none of the chosen compute are available, the job will get into the Queue, and will try again when one of the resources will free. This is very useful for very large size grid search, sharing a single gpu compute,etc.