Return-Path: <a.huebl@hzdr.de>
Received: from [149.220.60.110] (account huebl@hzdr.de [149.220.60.110] verified)
  by hzdr.de (CommuniGate Pro SMTP 6.1.12)
  with ESMTPSA id 16247810 for picongpu-users@hzdr.de; Mon, 23 Jan 2017 15:49:53 +0100
Subject: Re: Restart with TBG
To: picongpu-users@hzdr.de
References: <list-16246926@cg2.fz-rossendorf.de>
From: Axel Huebl <a.huebl@hzdr.de>
Organization: HZDR
Message-ID: <2db1ee8a-5856-d294-f8c0-65f6e9111680@hzdr.de>
Date: Mon, 23 Jan 2017 15:49:53 +0100
MIME-Version: 1.0
In-Reply-To: <list-16246926@cg2.fz-rossendorf.de>
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

Dear Danila,

> Could you please confirm that using tbg one needs to create a separate
> directory for the restart?

yes, with tbg, we usually restart in a separate folder, to avoid
accidentally overwriting things in the old one.

Otherwise, just apply those options in the tbg/submit.start file (or
copy and modify it) and then submit/execute it manually, so you stay in
the same dir.


Also, you might want to change the maximum time step (-s) for the restart.

Above information can be found in
  doc/TBG_macros.cfg [1]
and our wiki [2].


Cheers,
Axel

[1]
https://github.com/ComputationalRadiationPhysics/picongpu/blob/0.2.2/doc/TBG_macros.cfg
[2]
https://github.com/ComputationalRadiationPhysics/picongpu/wiki/Restarting-a-simulation

On 23.01.2017 14:23, Khikhlukha Danila wrote:
> Dear all,
> Am I correct that to run a restart using tgb one needs to specify a new
> output directory?
> Indeed, after the first simulation finished I have just added following
> lines to the submit submit file:
> TBG_restart="--restart"
> ...
> TBG_programParams="!TBG_devices     
> \                                                                                 
> 
>                    !TBG_gridSize    
> \                                                                                 
> 
>                    !TBG_steps       
> \                                                                                 
> 
>                    !TBG_movingWindow
> \                                                                                 
> 
>                    *!TBG_restart *    
> \                                                                                 
> 
>                    !TBG_plugins"    
> After that I have launched the simulation with the command:
> tbg -s bash -c $SUBMIT_DIR/0001gpus.cfg -t
> $SUBMIT_DIR/bash/bash_mpirun.tpl $RUN_DIR
> where SUBMIT_DIR and RUN_DIR are kept exactly the same. The job failed
> with the message:
> "job name already in use, can't create new folder"
> 
> However if for the restart I specify a different RUN_DIR (like
> $RUN_DIR_restart) and modify the restart option like
> TBG_restart="--restart --restart-directory
> <old_run_dir>/simOutput/checkpoints" then everything works just fine.
> 
> Could you please confirm that using tbg one needs to create a separate
> directory for the restart?
> 
> Best regards,
> Danila.

-- 

Axel Huebl
Phone +49 351 260 3582
https://www.hzdr.de/crp
Computational Radiation Physics
Laser Particle Acceleration Division
Helmholtz-Zentrum Dresden - Rossendorf e.V.

Bautzner Landstrasse 400, 01328 Dresden
POB 510119, D-01314 Dresden
Vorstand: Prof. Dr.Dr.h.c. R. Sauerbrey
          Prof. Dr.Dr.h.c. P. Joehnk
VR 1693 beim Amtsgericht Dresden