docker
Differences
This shows you the differences between two versions of the page.
Both sides previous revisionPrevious revisionNext revision | Previous revision | ||
docker [2019/11/28 18:29] – [Proposed solution] mimbert | docker [2023/09/28 17:24] (current) – [Proposed solution] cmorin | ||
---|---|---|---|
Line 1: | Line 1: | ||
+ | ===== Docker ===== | ||
+ | |||
Since November 2019, a new way to conduct experiments is implemented in CorteXlab. | Since November 2019, a new way to conduct experiments is implemented in CorteXlab. | ||
- | ====== Why ====== | + | ==== Why ==== |
The legacy way of running experiments is to run one (or more) commands on each node of the experiment. These commands are run from the minus task. This has some drawbacks: | The legacy way of running experiments is to run one (or more) commands on each node of the experiment. These commands are run from the minus task. This has some drawbacks: | ||
Line 12: | Line 14: | ||
* fix issues | * fix issues | ||
* repeat the process | * repeat the process | ||
- | This workflow is painful, not interactive, | + | |
- | * The experimenter needs to pack in the task everything needed. This includes potentially big datasets, and if these datasets are different for each node, then the task will contain the union of all the datasets, which can ba huge. | + | * The experimenter needs to pack in the task everything needed. This includes potentially big datasets, and if these datasets are different for each node, then the task will contain the union of all the datasets, which can be huge. |
* The executable code has to be in the task. For simple scripts, it's ok, but for binaries, or as soon as there are some dependencies (libraries, which may also have dependencies of their own), building the task may become pretty difficult or impossible. See [[embedding_oot_modules_or_custom_libraries_binaries_in_minus_scenario]]). In particular, [[https:// | * The executable code has to be in the task. For simple scripts, it's ok, but for binaries, or as soon as there are some dependencies (libraries, which may also have dependencies of their own), building the task may become pretty difficult or impossible. See [[embedding_oot_modules_or_custom_libraries_binaries_in_minus_scenario]]). In particular, [[https:// | ||
* All the results are gathered with the task directory, compressed, and sent back to airlock. This means that the results may include huge unneeded things, such as experiment code, input datasets, etc. | * All the results are gathered with the task directory, compressed, and sent back to airlock. This means that the results may include huge unneeded things, such as experiment code, input datasets, etc. | ||
- | ====== Proposed solution | + | ==== Proposed solution ==== |
The proposed solution is to use [[https:// | The proposed solution is to use [[https:// | ||
- | * [[https:// | + | * [[https:// |
- | * When running | + | * Preparing |
+ | * Image preparation can be an interactive process, or it can be automated with a [[https:// | ||
* The exact same image can be used to test things on an experimenter' | * The exact same image can be used to test things on an experimenter' | ||
+ | * When running a task, images are instanciated to [[https:// | ||
* The experiment results are structured differently. For each node, there is one directory per container, containing the stdout/ | * The experiment results are structured differently. For each node, there is one directory per container, containing the stdout/ | ||
- | | + | * The homes of the users are NFS mounted on the nodes, and the home of the user is mounted in the containers |
docker.1574962173.txt.gz · Last modified: 2019/11/28 18:29 by mimbert