From: "Andrei Berceanu berceanu@runbox.com" <picongpu-users@hzdr.de>
Content-Type: text/plain; charset="utf-8"
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
To: "picongpu-users" <picongpu-users@hzdr.de>
CC: "picongpu-users" <picongpu-users@hzdr.de>
Subject: Re: [PIConGPU-Users] DGX-1
Date: Tue, 15 May 2018 17:44:04 +0200 (CEST)
In-Reply-To: <list-18512527@cg1.fz-rossendorf.de>
Message-Id: <E1fIc7Q-0006Ux-Pq@rmmprod05.runbox>

That is great, thank you guys for the detailed replies!

On Mon, 14 May 2018 14:04:37 +0200, "Axel Huebl a.huebl@hzdr.de" <picongpu-=
users@hzdr.de> wrote:

> Just to add to the arguments,
>=20
> the additional (non-tensor core) Flop/s in V100 vs. the 2 year-old P100
> satisfy the additional costs, that's why V100 are ok. Also, the V100
> 32GB are too my knowledge available for the same price as the "first"
> V100 (16 GB).
>=20
> So ideally, use the 32GB variants for larger problem sizes!
>=20
> Regarding NVlink & NVswitch in newer stations: we can enable RDMA over
> those via GPU direct in PIConGPU, although it's not yet mainline. In
> most settings outside of heavy strong-scaling tough, we are hiding
> latency well enough so that a smaller BW and longer latency won't slow
> down your simulation. (Read: the intra- and interconnet is not too
> important for PIConGPU since we assume the worst.)
>=20
> Anyway, if you plan things like in-node global FFTs, e.g. as an in situ
> plugin to get the envelope of a laser via a Hankel transform, NVlink and
> NVswitch will pay off.
>=20
>=20
> Cheers,
> Axel
>=20
> On 5/14/18 1:18 PM, Ren=C3=A9 Widera r.widera@hzdr.de wrote:
> > Dear Andrei,
> >=20
> >> My question is, would PIConGPU run on the DGX-1 and can it make use of
> > NVLink [2] v2.0?
> >=20
> > Currently we are not using NVLink. But it is planed to add support for
> > MPI GPU-Direct which should than use NVLink.
> >=20
> >> Also, I'm guessing it can't use the tensor cores in the V100 version
> > of DGX-1?
> >=20
> > Currently we are not using tensor cores. It is not fully clear if tensor
> > cores will give an advantage. ONe drawback of the tensor cores is that
> > the using fp16. In PIConGPU we use at least fp32 if you not activate
> > fp64 support.
> >=20
> > Never the less I think a DGX-1 with V100 is the right system.
> >=20
> > Ren=C3=A9 (psychocoderHPC)
> >=20
> > On 05/14/2018 12:36 PM, Andrei Berceanu berceanu@runbox.com wrote:
> >> Hi,
> >>
> >> First of all, let me provide some context: we are considering
> >> purchasing a DGX-1 system [1] from Nvidia for PIConGPU and are trying
> >> to decide between the P100 and V100 versions.
> >>
> >> My question is, would PIConGPU run on the DGX-1 and can it make use of
> >> NVLink [2] v2.0?
> >>
> >> Also, I'm guessing it can't use the tensor cores in the V100 version
> >> of DGX-1?
> >>
> >> Regards,
> >> Andrei
> >>
> >> [1] https://en.wikipedia.org/wiki/Nvidia_DGX-1
> >> [2] https://en.wikipedia.org/wiki/NVLink
> >> #############################################################
> >> This message is sent to you because you are subscribed to
> >> =C2=A0=C2=A0 the mailing list <picongpu-users@hzdr.de>.
> >> To unsubscribe, E-mail to: <picongpu-users-off@hzdr.de>
> >> To switch to the DIGEST mode, E-mail to <picongpu-users-digest@hzdr.de>
> >> To switch to the INDEX mode, E-mail to <picongpu-users-index@hzdr.de>
> >> Send administrative queries to=C2=A0 <picongpu-users-request@hzdr.de>
> >>
> >=20
>=20
> --=20
>=20
> Axel Huebl
> Phone: +49 351 260 3582
> Institute of Radiation Physics
> http://www.hzdr.de/crp
> Helmholtz-Zentrum Dresden - Rossendorf (HZDR)
> Bautzner Landstr. 400 | 01328 Dresden | Germany
> Board of Directors:
> Prof. Dr. Dr. h. c. Roland Sauerbrey, Dr. Ulrich Breuer
> Company Registration Number VR 1693, Amtsgericht Dresden
>=20
> #############################################################
> This message is sent to you because you are subscribed to
>   the mailing list <picongpu-users@hzdr.de>.
> To unsubscribe, E-mail to: <picongpu-users-off@hzdr.de>
> To switch to the DIGEST mode, E-mail to <picongpu-users-digest@hzdr.de>
> To switch to the INDEX mode, E-mail to <picongpu-users-index@hzdr.de>
> Send administrative queries to  <picongpu-users-request@hzdr.de>