Return-Path: Received: from mx2.fz-rossendorf.de ([149.220.142.12] verified) by hzdr.de (CommuniGate Pro SMTP 6.1.12) with ESMTP id 15556317 for picongpu-users@cg.hzdr.de; Wed, 19 Oct 2016 18:00:15 +0200 Received: from localhost (localhost [127.0.0.1]) by mx2.fz-rossendorf.de (Postfix) with ESMTP id BBA9145977 for ; Wed, 19 Oct 2016 18:00:15 +0200 (CEST) X-Virus-Scanned: Debian amavisd-new at mx2.fz-rossendorf.de Received: from mx2.fz-rossendorf.de ([127.0.0.1]) by localhost (mx2.fz-rossendorf.de [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id N5cFOTMlWNjd for ; Wed, 19 Oct 2016 18:00:11 +0200 (CEST) Received-SPF: Pass (sender SPF authorized) identity=mailfrom; client-ip=147.231.234.10; helo=mailgw.eli-beams.eu; envelope-from=prvs=1100baf582=danila.khikhlukha@eli-beams.eu; receiver=picongpu-users@hzdr.de Received: from mailgw.eli-beams.eu (mailgw.eli-beams.eu [147.231.234.10]) by mx2.fz-rossendorf.de (Postfix) with ESMTPS id C974E40549 for ; Wed, 19 Oct 2016 18:00:11 +0200 (CEST) Received: from braun.eli-beams.eu ([10.1.5.17]) by mailgw.eli-beams.eu with ESMTP id u9JG0Bpk030144-u9JG0Bpm030144 (version=TLSv1.0 cipher=ECDHE-RSA-AES256-SHA bits=256 verify=NOT) for ; Wed, 19 Oct 2016 18:00:11 +0200 Received: from BRAUN.eli-beams.eu ([::1]) by braun.eli-beams.eu ([::1]) with mapi id 14.03.0301.000; Wed, 19 Oct 2016 18:00:11 +0200 From: Khikhlukha Danila To: "picongpu-users@hzdr.de" Subject: RE: [PIConGPU-Users] [PIConGPU-Users] [PIConGPU-Users] [PIConGPU-Users] Supercell concept Thread-Topic: [PIConGPU-Users] [PIConGPU-Users] [PIConGPU-Users] [PIConGPU-Users] Supercell concept Thread-Index: AQHSKhz3kl+S70weV0qoPpn0vQ4f26Cv6Gic Date: Wed, 19 Oct 2016 16:00:11 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US, cs-CZ Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.32.201.3] Content-Type: multipart/alternative; boundary="_000_BA7C853FEE430847B9C35FFCC6E5B2A52821399Cbraunelibeamseu_" MIME-Version: 1.0 --_000_BA7C853FEE430847B9C35FFCC6E5B2A52821399Cbraunelibeamseu_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Dear Ren=E9, thanks a lot, it is way more clear now. However I would like to ask you if you can recommend me some further readin= g to develop more intuition regarding this dependency on the stencil and ma= cro-particle shape? Unfortunately at the moment I can't understand where 32 is coming from for = the classical Yee scheme + CIC particle for instance. Thank you in advance, Danila ________________________________ From: picongpu-users@hzdr.de [picongpu-users@hzdr.de] on behalf of Ren=E9 W= idera [r.widera@hzdr.de] Sent: Wednesday, October 19, 2016 5:25 PM To: picongpu-users@hzdr.de Subject: Re: [PIConGPU-Users] [PIConGPU-Users] [PIConGPU-Users] [PIConGPU-U= sers] Supercell concept Dear Danila, the supercell size defines the number of worker threads and the shared memo= ry cache. 256 is a good value to utilize the most gpus. A supercell size for each directions needs to be greater or equal to the ne= eded neighbors of the stencil for the algorithms. This condition is checked= at compile time and depends on the selected solvers and the species shape. The size of the supercell per direction is independent. The volume x=D7y=D7= z of the supercell should be a multible of 32. best, Ren=E9 Am 19. Oktober 2016 17:14:28 MESZ, schrieb Khikhlukha Danila : Dear Ren=E9, thank you for the prompt replay. Indeed the problem is quite simple. So wra= p it up, the number of cell in each direction should give N % (N_gpu * N_su= percells) =3D=3D 0. Just for educational purpose: what is so special about the number 128 (I gu= ess it is a size of a cache)? Could in be for instance 256? Would it be pos= sible to specify the same number of super cells in X and Z direction? Thanks a lot, Danila. ________________________________ From: picongpu-users@hzdr.de [picongpu-users@hzdr.de] on behalf of Ren=E9 W= idera [r.widera@hzdr.de] Sent: Wednesday, October 19, 2016 4:59 PM To: picongpu-users@hzdr.de Subject: Re: [PIConGPU-Users] [PIConGPU-Users] Supercell concept Dear Danila, the volume per gpu needs to be a multiple of the superCell size. In your case 4176/8gpus=3D522. 522 is not dividable bei 8. 4160 cells in y direction should solve your problem. Please keep in mind if you change the superCell size to a value smaller tha= n 128cells the most simulation run slower. The default size with 8x8x4 shows the best result for the most cases. best, Ren=E9 Am 19. Oktober 2016 16:46:28 MESZ, schrieb Khikhlukha Danila : Dear all, I have some troubles trying to specify a computational grid with moving win= dows using picongpu v0.2.0. We were discussing this topic previously, howev= er I have the same problem again, so likely I misunderstand something from = the last time. So I'm trying to launch the simulation using 4 K80 cards: 8 GPU devices ove= rall. In the memory.param file I have specified the SuperCell layout as (2,= 8,2). I want to have one GPU in transversal direction and 8 in longitudinal= . So in cfg file I specified: TBG_gpu_x=3D1 TBG_gpu_y=3D8 TBG_gpu_z=3D1 Then I would like my real computational domain to have 256 x 3712 x 256 cel= l. Since the moving window reduces the real domain by 1 GPU in y direction,= I specified my grid as 256 x 4176 x 256. (4176 =3D 9/8*3712) TBG_gridSize=3D"-g 256 4176 256" However, trying to submit such a cfg file I'm receiving an assertion fail: void picongpu::MySimulation::checkGridConfiguration(PMacc::DataSpace,P= Macc::GridLayout) [with unsigned int DIM =3D 3u]: Assertion`gridSizeLo= cal[i] % MappingDesc::SuperCellSize::toRT()[i] =3D=3D 0' failed. However 4176 % 8 =3D=3D 0 and 256 % 2 =3D=3D 0. Could you please guide me how to solve this issue? Looks like I misundersta= nd the concept of the SuperCell. Thank you in advance, Danila. -- Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail gesendet= . --_000_BA7C853FEE430847B9C35FFCC6E5B2A52821399Cbraunelibeamseu_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable
Dear Ren=E9,
thanks a lot, it is  way more clear now.
However I would like to ask you if you can recommend me some further r= eading to develop more intuition regarding this dependency on the stencil a= nd macro-particle shape? 
Unfortunately at the moment I can't understand where 32 is coming from= for the classical Yee scheme + CIC particle for instance.

Thank you in advance,
Danila
From: picongpu-users@hzdr.de [picongpu-user= s@hzdr.de] on behalf of Ren=E9 Widera [r.widera@hzdr.de]
Sent: Wednesday, October 19, 2016 5:25 PM
To: picongpu-users@hzdr.de
Subject: Re: [PIConGPU-Users] [PIConGPU-Users] [PIConGPU-Users] [PIC= onGPU-Users] Supercell concept

Dear Danila,

the supercell size defines the number of worker threads and the shared memo= ry cache.
256 is a good value to utilize the most gpus.
A supercell size for each directions needs to be greater or equal to the ne= eded neighbors of the stencil for the algorithms. This condition is checked= at compile time and depends on the selected solvers and the species shape.=
The size of the supercell per direction is independent. The volume x=D7y=D7= z of the supercell should be a multible of 32.

best,
Ren=E9

Am 19. Oktober 2016 17:14:28 MESZ, schrieb Khikh= lukha Danila <Danila.Khikhlukha@eli-beams.eu>:
Dear Ren=E9,
thank you for the prompt replay. Indeed the problem is quite simple. S= o wrap it up, the number of cell in each direction should give N % (N_gpu *= N_supercells) =3D=3D 0.

Just for educational purpose: what is so special about the number 128 = (I guess it is a size of a cache)? Could in be for instance 256? Would it b= e possible to specify the same number of super cells in X and Z direction?<= /div>

Thanks a lot,
Danila.
From: picongpu-users@hzdr.de [picongpu-user= s@hzdr.de] on behalf of Ren=E9 Widera [r.widera@hzdr.de]
Sent: Wednesday, October 19, 2016 4:59 PM
To: picongpu-users@hzdr.de
Subject: Re: [PIConGPU-Users] [PIConGPU-Users] Supercell concept

Dear Danila,

the volume per gpu needs to be a multiple of the superCell size.

In your case 4176/8gpus=3D522. 522 is not dividable bei 8.
4160 cells in y direction should solve your problem.

Please keep in mind if you change the superCell size to a value smaller tha= n 128cells the most simulation run slower.
The default size with 8x8x4 shows the best result for the most cases.

best,

Ren=E9

Am 19. Oktober 2016 16:46:28 MESZ, schrieb Khikh= lukha Danila <Danila.Khikhlukha@eli-beams.eu>:
Dear all,
I have some troubles trying to specify a computational grid with movin= g windows using picongpu v0.2.0. We were discussing this topic previously, = however I have the same problem again, so likely I misunderstand something = from the last time.

So I'm trying to launch the simulation using 4 K80 cards: 8 GPU device= s overall. In the memory.param file I have specified the SuperCell layout a= s (2,8,2). I want to have one GPU in transversal direction and 8 in longitu= dinal. So in cfg file I specified:
TBG_gpu_x=3D= 1
TBG_gpu_y=3D8
TBG_gpu_z=3D1

Then I would lik= e my real computational domain to have 256 x 3712 x 256 cell. Since the mov= ing window reduces the real domain by 1 GPU in y direction, I specified my grid as 256 x 4176 x 256. (4176 =3D 9/8= *3712)
TBG_gridSize=3D&= quot;-g 256 4176 256"

However, trying = to submit such a cfg file I'm receiving an assertion fail:

void picongpu::MySimulation::checkGridConfiguration(PMacc::DataSpace<DIM= >,PMacc::GridLayout<DIM>) [with unsigned int DIM =3D 3u]: Assertio= n`gridSizeLocal[i] % MappingDesc::SuperCellSize::toRT()[i] =3D=3D 0' failed= .

However 4176 % 8= =3D=3D 0 and 256 % 2 =3D=3D 0. 

Could you please= guide me how to solve this issue? Looks like I misunderstand the concept o= f the SuperCell.

Thank you in adv= ance,
Danila.



--
Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail gesendet= .
--_000_BA7C853FEE430847B9C35FFCC6E5B2A52821399Cbraunelibeamseu_--