Dear all,
Am I correct that to run a restart using tgb one needs to specify a new output directory?
Indeed, after the first simulation finished I have just added following lines to the submit submit file:
TBG_restart="--restart"
...
TBG_programParams="!TBG_devices \
!TBG_gridSize \
!TBG_steps \
!TBG_movingWindow \
!TBG_restart \
!TBG_plugins"
After that I have launched the simulation with the command:
tbg -s bash -c $SUBMIT_DIR/0001gpus.cfg -t $SUBMIT_DIR/bash/bash_mpirun.tpl $RUN_DIR
where SUBMIT_DIR and RUN_DIR are kept exactly the same. The job failed with the message:
"job name already in use, can't create new folder"
However if for the restart I specify a different RUN_DIR (like $RUN_DIR_restart) and modify the restart option like TBG_restart="--restart --restart-directory <old_run_dir>/simOutput/checkpoints" then everything works just fine.
Could you please confirm that using tbg one needs to create a separate directory for the restart?
Best regards,
Danila.
|