Return-Path: Received: from [149.220.60.110] (account huebl@hzdr.de [149.220.60.110] verified) by hzdr.de (CommuniGate Pro SMTP 6.1.5) with ESMTPSA id 12122368; Wed, 30 Sep 2015 10:17:33 +0200 Message-ID: <560B9A9D.2000606@hzdr.de> Date: Wed, 30 Sep 2015 10:17:33 +0200 From: "Huebl, Axel" MIME-Version: 1.0 To: picongpu-users@hzdr.de CC: "Orban, Christopher" Subject: tbg: overlook scripts before start References: ,<1E88C655-2E00-474A-86B5-9CECA58C8D94@hzdr.de> ,<55F06FF3.3090604@hzdr.de> ,<55F0774F.20308@hzdr.de> In-Reply-To: Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha1; boundary="------------ms060802010001070105050909" This is a cryptographically signed message in MIME format. --------------ms060802010001070105050909 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi Chris, > It took me a while to figure out where to look for the PBS script that it produces. Sorry for that! tbg creates job files by default in the "destinationPath" (free parameter in tbg) and is then moved by our script src/picongpu/submit/submitAction.sh to the sub-directory "destinationPath/tbg/" The name is "submit.start", it lays alongside the "submit.cfg" and the "submit.tpl" that were used. > It would be preferable if there was an option to produce a PBS-compatible script but allow the user to do the final qsub after looking it over. Absolutely, just skip the option "-s", which stands for submit, See: $ tbg --help > TBG (template batch generator) > create a new folder for a batch job and copy in all important files > > usage: tbg -c [cfgFile] [-s [submitsystem]] [-t [templateFile]] [-p > project] [-o "VARNAME1=3D10 VARNAME2=3D5"] [-h] destinationPath > > -c | --cfg [file] - Configuration file to set up batch file. > Default: [cfgFile] via export TBG_CFGFILE > -s | --submit [command] - Submit command (qsub, "qsub -h", sbatch, > ...) > Default: [submitsystem] via > export TBG_SUBMIT > -t | --tpl [file] - Template to create a batch file from. > tbg will use stdin, if no file is specified. > Default: [templateFile] via > export TBG_TPLFILE > [...] > destinationPath - Directory for simulation output > when you run PIConGPU do you typically only use 1 MPI process per GPU? yes, one MPI host rank controls one GPU. So if you have 4 GPUs per node, you will need 4 MPI processes on it. Axel On 29.09.2015 22:00, Orban, Christopher wrote: > Hi Axel, > I finally got some more time to tinker with PIConGPU today a= nd it turned out that the binary I had compiled does successfully work on= the ruby cluster at the Ohio Supercomputer Center. > I must admidt that using tbg was more complicated than I ima= gined it to be. It took me a while to figure out where to look for the PB= S script that it produces. It would be preferable if there was an option = to produce a PBS-compatible script but allow the user to do the final qsu= b after looking it over. > Here is a quick question for you: when you run PIConGPU do y= ou typically only use 1 MPI process per GPU? I got confused the first tim= e I tried running a simulation on one node because I used mpiexec -n 20 o= n a cluster with 20 CPUs and 1 GPU. Single node operation only worked usi= ng 1 CPU (i.e. using mpiexec -n 1). Please let me know if having 1 MPI pr= ocess per GPU is the standard way of operating the code. > thanks, > Chris --=20 Axel Huebl Phone +49 351 260 3582 https://www.hzdr.de/crp Computational Radiation Physics Laser Particle Acceleration Division Helmholtz-Zentrum Dresden - Rossendorf e.V. Bautzner Landstrasse 400, 01328 Dresden POB 510119, D-01314 Dresden Vorstand: Prof. Dr.Dr.h.c. R. Sauerbrey Prof. Dr.Dr.h.c. P. Joehnk VR 1693 beim Amtsgericht Dresden --------------ms060802010001070105050909 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIPrDCC BNUwggO9oAMCAQICCFBOxvU9EbRkMA0GCSqGSIb3DQEBCwUAMHExCzAJBgNVBAYTAkRFMRww GgYDVQQKExNEZXV0c2NoZSBUZWxla29tIEFHMR8wHQYDVQQLExZULVRlbGVTZWMgVHJ1c3Qg Q2VudGVyMSMwIQYDVQQDExpEZXV0c2NoZSBUZWxla29tIFJvb3QgQ0EgMjAeFw0xNDA3MjIx MjA4MjZaFw0xOTA3MDkyMzU5MDBaMFoxCzAJBgNVBAYTAkRFMRMwEQYDVQQKEwpERk4tVmVy ZWluMRAwDgYDVQQLEwdERk4tUEtJMSQwIgYDVQQDExtERk4tVmVyZWluIFBDQSBHbG9iYWwg LSBHMDEwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDpm8NnhfkNrvWNVMOWUDU9 YuluTO2U1wBblSJ01CDrNI/W7MAxBAuZgeKmFNJSoCgjhIt0iQReW+DieMF4yxbLKDU5ey2Q RdDtoAB6fL9KDhsAw4bpXCsxEXsM84IkQ4wcOItqaACa7txPeKvSxhObdq3u3ibo7wGvdA/B CaL2a869080UME/15eOkyGKbghoDJzANAmVgTe3RCSMqljVYJ9N2xnG2kB3E7f81hn1vM7Pb D8URwoqDoZRdQWvY0hD1TP3KUazZve+Sg7va64sWVlZDz+HVEz2mHycwzUlU28kTNJpxdcVs 6qcLmPkhnSevPqM5OUhqjK3JmfvDEvK9AgMBAAGjggGGMIIBgjAOBgNVHQ8BAf8EBAMCAQYw HQYDVR0OBBYEFEm3xs/oPR9/6kR7Eyn38QpwPt5kMB8GA1UdIwQYMBaAFDHDeRu69VPXF+CJ ei0XbAqzK50zMBIGA1UdEwEB/wQIMAYBAf8CAQIwYgYDVR0gBFswWTARBg8rBgEEAYGtIYIs AQEEAgIwEQYPKwYBBAGBrSGCLAEBBAMAMBEGDysGAQQBga0hgiwBAQQDATAPBg0rBgEEAYGt IYIsAQEEMA0GCysGAQQBga0hgiweMD4GA1UdHwQ3MDUwM6AxoC+GLWh0dHA6Ly9wa2kwMzM2 LnRlbGVzZWMuZGUvcmwvRFRfUk9PVF9DQV8yLmNybDB4BggrBgEFBQcBAQRsMGowLAYIKwYB BQUHMAGGIGh0dHA6Ly9vY3NwMDMzNi50ZWxlc2VjLmRlL29jc3ByMDoGCCsGAQUFBzAChi5o dHRwOi8vcGtpMDMzNi50ZWxlc2VjLmRlL2NydC9EVF9ST09UX0NBXzIuY2VyMA0GCSqGSIb3 DQEBCwUAA4IBAQBjICj9nCGGcr45Rlk5MiW8qQGbDczKfUGchm0KbiyzE1l1sTOSG2EnFv/D stU1gvuEKgFJvWa7Zi+ywgZdbj9u4wFaW8pDY1yVtuExpx/VB19N5mWCTjL5w3x6S81NXHTu IfJ1AuxSPtLJatOQI25JZzW+f01WpOzML8+3oZeocj7JvEDWWqQIPda8gsO3tzKOsSyOam23 NQIZz/U5RFhjpyQAELC7/E6vbi84u6VXST/YblBvLJeW3B1GmmWJz67M8uXZn1OzPqEvkqnY C8aEHwTG6x7on321e6UC8STFJGMRNMxakyAqeYg6JUKQqWU7fIbTEhUjKfws2sw5W1QXMIIF YzCCBEugAwIBAgIHFj6RQyeZKDANBgkqhkiG9w0BAQUFADCBlDELMAkGA1UEBhMCREUxMjAw BgNVBAoTKUZvcnNjaHVuZ3N6ZW50cnVtIERyZXNkZW4tUm9zc2VuZG9yZiBlLlYuMSAwHgYD VQQLExdJbmZvcm1hdGlvbnN0ZWNobm9sb2dpZTEVMBMGA1UEAxMMRlpELUNBIC0gRzAyMRgw FgYJKoZIhvcNAQkBFglyYUBmemQuZGUwHhcNMTMwODI5MDkyNjExWhcNMTYwODI4MDkyNjEx WjBhMQswCQYDVQQGEwJERTE8MDoGA1UEChMzSGVsbWhvbHR6LVplbnRydW0gRHJlc2RlbiAt IFJvc3NlbmRvcmYgZS4gVi4gKEhaRFIpMRQwEgYDVQQDEwtIdWVibCwgQXhlbDCCASIwDQYJ KoZIhvcNAQEBBQADggEPADCCAQoCggEBANWIEuWannP8AfgdPc+sHhQjnwfs9bpNZXuptGRT 5iX4mwlPECrDOLBgfszBke+NGgXKJz1moIRZ8wvatJDQ8OTCbENoa1gmpCKZ4ryB+3XSxl+r BbM2eH7koCeagqbifypdoElI4wtc3QRLs8ZURhxfjWn+Vv3qs5od7HVypVka8WqLkqS6LgFL /LGyp+uqV0m778ExVaoWghXronhcDk10nUJSaWqHLCpHvWv/6fB8Tf08hNWRVB5ilUHHBWvu vCAQjhSbH4YFFoZDXB5PcFhvNOnT5cOIrTf5XMPMtubAvZ14S+wtkt9eDMZSOyT0KOkGw1cE 2X54vrdvkUHa/IMCAwEAAaOCAeowggHmMC8GA1UdIAQoMCYwEQYPKwYBBAGBrSGCLAEBBAMA MBEGDysGAQQBga0hgiwCAQQDADAJBgNVHRMEAjAAMAsGA1UdDwQEAwIF4DApBgNVHSUEIjAg BggrBgEFBQcDAgYIKwYBBQUHAwQGCisGAQQBgjcUAgIwHQYDVR0OBBYEFK228r82UkHXs2AB GJRtWlvbnP+IMB8GA1UdIwQYMBaAFKUpSPWVmRi1PjfIhgKKTr9C8B2MMBoGA1UdEQQTMBGB D2EuaHVlYmxAaHpkci5kZTB7BgNVHR8EdDByMDegNaAzhjFodHRwOi8vY2RwMS5wY2EuZGZu LmRlL2Z6ZC1jYS9wdWIvY3JsL2dfY2FjcmwuY3JsMDegNaAzhjFodHRwOi8vY2RwMi5wY2Eu ZGZuLmRlL2Z6ZC1jYS9wdWIvY3JsL2dfY2FjcmwuY3JsMIGWBggrBgEFBQcBAQSBiTCBhjBB BggrBgEFBQcwAoY1aHR0cDovL2NkcDEucGNhLmRmbi5kZS9memQtY2EvcHViL2NhY2VydC9n X2NhY2VydC5jcnQwQQYIKwYBBQUHMAKGNWh0dHA6Ly9jZHAyLnBjYS5kZm4uZGUvZnpkLWNh L3B1Yi9jYWNlcnQvZ19jYWNlcnQuY3J0MA0GCSqGSIb3DQEBBQUAA4IBAQA9Q7+cxRoFjWw8 oc1otv9P7yBdtY+JAVE1mmEGzeU/Tqqupe+/3N/e4euqPqzbTgcaw/H0e7K831cCe53ux+CB zJZH+kSkY+bqX+SxP8ndgRBDVFe6SvL+RiO49xOB4irg+a6otTWjuI8pUDTqUjGht82MD/rf 1gAUXRxPRU2nrt7BtyYtNKvP14xxnZrghvx/DZ6YsaVV14w1SpmtiiJ6q2WsbtiWKnoNCrWA 3Qr/9GK1tzGKQWHq21KDIm0bysjUt1hcwowM8OtPAQO66ywD72aR+NQB+LKTAnmPHBIgC7uW nLNpR9+CPubwTlq4z4f58uqnF5+JjiRgnfhJZGpvMIIFaDCCBFCgAwIBAgIHF5Bg2EcW5zAN BgkqhkiG9w0BAQsFADBaMQswCQYDVQQGEwJERTETMBEGA1UEChMKREZOLVZlcmVpbjEQMA4G A1UECxMHREZOLVBLSTEkMCIGA1UEAxMbREZOLVZlcmVpbiBQQ0EgR2xvYmFsIC0gRzAxMB4X DTE0MDUxMjE1MDU0NFoXDTE5MDcwOTIzNTkwMFowgZQxCzAJBgNVBAYTAkRFMTIwMAYDVQQK EylGb3JzY2h1bmdzemVudHJ1bSBEcmVzZGVuLVJvc3NlbmRvcmYgZS5WLjEgMB4GA1UECxMX SW5mb3JtYXRpb25zdGVjaG5vbG9naWUxFTATBgNVBAMTDEZaRC1DQSAtIEcwMjEYMBYGCSqG SIb3DQEJARYJcmFAZnpkLmRlMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA8m1G eCgBDpPgT8IUvZ2FQLJbvz6fpH1JA+DlZgNog4uFLue/6Lh9pT8EvcNbor8Qb2rt9rwRbk3p 3WqEc7AUZdxUY9ZGaYqcR9BUwHaFyaEcSYmKTfQ2scoxiazM+rmxu+UXtF/wBh8kHo9CSsY4 eWy1GSDkYksZlWhqOImpVkQmEWgoqu7brg60ug6YtbzE3n93MVQRQF4bFS2Kxvh+Rpq7u9eP 1UYIqG0+0G7T3v9CT/a7gw7qGQGkryoE+G3YoRq/1KM4+LZVWKgQzLGH3XMzgSbGlXzWfJ/a WsD88UkKyKjMiFeUisbO7yyM5FhYaZT2f6ZM25AuBvI+P+ZhOQIDAQABo4IB9jCCAfIwEgYD VR0TAQH/BAgwBgEB/wIBATAOBgNVHQ8BAf8EBAMCAQYwEQYDVR0gBAowCDAGBgRVHSAAMB0G A1UdDgQWBBSlKUj1lZkYtT43yIYCik6/QvAdjDAfBgNVHSMEGDAWgBRJt8bP6D0ff+pEexMp 9/EKcD7eZDAUBgNVHREEDTALgQlyYUBmemQuZGUwgYgGA1UdHwSBgDB+MD2gO6A5hjdodHRw Oi8vY2RwMS5wY2EuZGZuLmRlL2dsb2JhbC1yb290LWNhL3B1Yi9jcmwvY2FjcmwuY3JsMD2g O6A5hjdodHRwOi8vY2RwMi5wY2EuZGZuLmRlL2dsb2JhbC1yb290LWNhL3B1Yi9jcmwvY2Fj cmwuY3JsMIHXBggrBgEFBQcBAQSByjCBxzAzBggrBgEFBQcwAYYnaHR0cDovL29jc3AucGNh LmRmbi5kZS9PQ1NQLVNlcnZlci9PQ1NQMEcGCCsGAQUFBzAChjtodHRwOi8vY2RwMS5wY2Eu ZGZuLmRlL2dsb2JhbC1yb290LWNhL3B1Yi9jYWNlcnQvY2FjZXJ0LmNydDBHBggrBgEFBQcw AoY7aHR0cDovL2NkcDIucGNhLmRmbi5kZS9nbG9iYWwtcm9vdC1jYS9wdWIvY2FjZXJ0L2Nh Y2VydC5jcnQwDQYJKoZIhvcNAQELBQADggEBAC9Y+prXCAxJzhcGTqHbUWZN0BbctjNv4zor znGPFZ42NVfCSqR9gIRiwnDBYlBJ+Q+PppFZNE/97E1XCmk/iFWE89wWtEfTem5OPjKej3Ff nuVCl1e11o8re9j91KC2Sv4B5tXVwZ1C0tjJDFvA0c4g42pce38LNR5kpuGPeGXrDCscTF0R 1eTWzt2OLBPVjLl43Sf8RbIM+R0s1VSlb/YGnVLUAK8TcRuoNgDwIFa7uxtC7DP2c+WrQt4l ESADB2finpsWW9a+prAH6RWd8PuW3lPgXBd7vum8wpXfSVF3oXvPakBQaz57Dr9EC6GMMkKy 8PRxj3Ak0/XQAIKJWPoxggQBMIID/QIBATCBoDCBlDELMAkGA1UEBhMCREUxMjAwBgNVBAoT KUZvcnNjaHVuZ3N6ZW50cnVtIERyZXNkZW4tUm9zc2VuZG9yZiBlLlYuMSAwHgYDVQQLExdJ bmZvcm1hdGlvbnN0ZWNobm9sb2dpZTEVMBMGA1UEAxMMRlpELUNBIC0gRzAyMRgwFgYJKoZI hvcNAQkBFglyYUBmemQuZGUCBxY+kUMnmSgwCQYFKw4DAhoFAKCCAjUwGAYJKoZIhvcNAQkD MQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMTUwOTMwMDgxNzMzWjAjBgkqhkiG9w0B CQQxFgQUrx4gZ9DdBT/oYypuNSxVgeu5ei4wbAYJKoZIhvcNAQkPMV8wXTALBglghkgBZQME ASowCwYJYIZIAWUDBAECMAoGCCqGSIb3DQMHMA4GCCqGSIb3DQMCAgIAgDANBggqhkiG9w0D AgIBQDAHBgUrDgMCBzANBggqhkiG9w0DAgIBKDCBsQYJKwYBBAGCNxAEMYGjMIGgMIGUMQsw CQYDVQQGEwJERTEyMDAGA1UEChMpRm9yc2NodW5nc3plbnRydW0gRHJlc2Rlbi1Sb3NzZW5k b3JmIGUuVi4xIDAeBgNVBAsTF0luZm9ybWF0aW9uc3RlY2hub2xvZ2llMRUwEwYDVQQDEwxG WkQtQ0EgLSBHMDIxGDAWBgkqhkiG9w0BCQEWCXJhQGZ6ZC5kZQIHFj6RQyeZKDCBswYLKoZI hvcNAQkQAgsxgaOggaAwgZQxCzAJBgNVBAYTAkRFMTIwMAYDVQQKEylGb3JzY2h1bmdzemVu dHJ1bSBEcmVzZGVuLVJvc3NlbmRvcmYgZS5WLjEgMB4GA1UECxMXSW5mb3JtYXRpb25zdGVj aG5vbG9naWUxFTATBgNVBAMTDEZaRC1DQSAtIEcwMjEYMBYGCSqGSIb3DQEJARYJcmFAZnpk LmRlAgcWPpFDJ5koMA0GCSqGSIb3DQEBAQUABIIBADhGcVDpbnrJ0dBXAmI6hStLBbf6dueb OEmQSN2v8MsKQxbEv9/a0yrT5vqpaps7R5NFjopVKrJKSErEM7wW80vDbqO/cMoP/89h88EU nf+kafoxcrrUjphgEZ2IhTnKXFqmufn3JqHrRBoBbYGJ6qMA/epfZqf8IHm/grKN/8Z1X1lF jFEaz1FSYa3/R2nrkWhqHpjuK9p8a/FSWLitWGWWvwAgx4Lv9Lv4gjUFpTu/evYPQ0LIukTf Z6MLtG9sV2r9o2XB368LuJSdCoPSVv8CYhRIMR6jRHAiW0tD/17T43wDNO7iap4bc1H+3iAy 1olApKABKJpmyE82pD1qRZkAAAAAAAA= --------------ms060802010001070105050909--