NG Tech meeting, NG2018, Munich, Wednesday Wednesday: [Rolling out ARC 6] 1) the content of the first ARC 6 test release (going beyond hello-world scenario) job management: classic, emi-es, rest, local authorization: gridmap, gridmap-less (voms attributes only), authgroups (on queue level) nordugridmap fetching from voms data features: cache, data staging (in, out gridftp, xroot, https) LRMS world: slurm-python, fork, condor RTE, proxy RTE infosys: nordugrid & glue2, VO startup scripts (arex, gridftp, infosys, data delivery service), metapackages, NEW arc-ce start/stop metathing JURA (APEL, SGAS), manual record publishing server logfiles validator test that arc5 client works with ARC6 server test that condor client works against ARC6 server test that dirac system works with ARC6 server 2) The GOLD ARC CE setup: ARC CE for WLCG: gridftp+ldap+glue2+nordugrid; slurm, htcondor; centos 7, sessiondir cross-mounted on WNs and frontend (min 200 GB) jobs to use localdisk on the node cache (100-200TB for ATLAS) mounted on the WNs ARC CE for non-WLCG: EMI-ES with REST as experimental; slurm; centos 7 cross-mounted sessiondir 3) discussion: ARC 6 Testing, testplans, test infrastructure, test reports --a numbered list of well-defined test cases where test case is defined by: server-side arc.conf, deployment instructions, client-side commands to execute, expected output/behaviour. step 1: test-case definition template (Balazs): step 2: developers come with test cases (one wee, all) 40 tests defined type of tests: continous integration, manual tests, type of test resouces: Level 0: Gitlab CI autodeploy Level 1: Openstack persistent test machines w/log-in access to developers and changeable set-up and arc.conf Level 2: Testing on production sites? for the Continous Integration there is the coderefinery and we'll use a manual table for the manual tests no formal report will be written 4) ARC 6 in software repositories nordugrid repo: separeta repo for 5 and 5 distros: new releases of distros will get ARC 6 but we won't put ARC 6 into existing releases. 5) ARC6 documentation - ARC 6 configuration manual (reference) Balazs - ARC 6 installation instructions (modernizing the webpage http://www.nordugrid.org/documents/arc-server-install.html), Maiken - Tecnhical documents for new ARC6 stuff (coordinate): ARCHERY, event-driven, config parser, Maiken-plugin, RTE - ARC6 upgrade notes (server and client). Andrii - later we decide howto update sysadminguide.... [Code Development Echosystem, ARC Community issues] - Maiken: status report from the ARC release manager regarding GIT transition, pointers to most important entry points and info sources - Maiken: status of automatic build & testing - Bugzilla, GIT issues - howto contribute to ARC in git (community), short text to be written - Communication channels for developers. There exists a skyp chat, active developers are welcome to join - ISGC workshop, March 2019 - ARC developer days in the Autumn (October), retreat coupled to sysadmins outreach [Post ARC 6 topics, part 1] data management in ARC, ARC as data transfer service Presentation of the Rest interface of ACT [ARC 6 content] - Andrii: demo of ARCHERY - David: demo of arc.conf validator (what are the mandatory blocks) - Discussion: Backward incompatibility: ARC 5,6 client vs. ARC 5,6 servers There are no interface changes in ARC6 therefore v5 clients should work with v6 servers. (exception: check wallitme) Thursday ----------- [ARC 6 content part 2] - Balazs: new set of arc.conf variables proposed by Florido, Maiken, Alexandr - Florido/Martin/Andrii: the ARC6 config parser - David: what to do with internal config profiles: agreed remove arex from the generic profile - Balazs: defaults & mandatory blocks, block dependencies including default functionality & service blocks in ARC.CONF - Maiken & Mattias: demo of the "zero.conf" use case 1 i.e. apache-like out-of-box service start up right after package in - Andrii: demo of the new RTE handling in ARC6 - Andrii: non-trivial auhorization setups in ARC6 (LCAS, LCMAPS, VOs, authroups with queues, so on...) - Aleksandr: demo of the event-driven new AREX internal - Aleksandr: demo of the new REST interface - Discussion: the new REST interface vs infosys - Mattias: ARC6 build & runtime dependencies - Mattias: demo of the ARC6 startup scripts & packages, metapackages - discussion: deprecated code & features, implementation of all the code cleanup & removal decisions (CREAM, unicore, windows related, solaris, JAVA bindings, BES, EMIR, JDL, JSDL, ws-monitor, old uploaders/downloaders) - Balazs: defaults & mandatory blocks, block dependencies including default functionality & service blocks in ARC.CONF - Andrii: demo of ARCHERY - Mattias: ARC6 build & runtime dependencies - Mattias: demo of the ARC6 startup scripts & packages, metapackages - Aleksandr: demo of the event-driven new AREX internal - Aleksandr: demo of the new REST interface - Maiken & Mattias: demo of the "zero.conf" use case 1 i.e. apache-like out-of-box service start up right after package install - Andrii: non-trivial auhorization setups in ARC6 (LCAS, LCMAPS, VOs, authroups with queues, so on...) - David: demo of arc.conf validator (what are the mandatory blocks) - Florido/Martin/Andrii: the ARC6 config parser - David: what to do with internal config profiles - Discussion: Backward incompatibility: ARC 5,6 client vs. ARC 5,6 servers There are no interface changes in ARC6 therefore v5 clients should work with v6 servers. (exception: check wallitme) [Postponed ARC 6 topics] - Martin: client-side development, what would be nice to do if only there would be available manpower? do we have a vision for ARC 7 client? - discussion: What to do with GPGPUs and other unconventional hardware? - discussion: High availability CE, failover mode - discussion: contenarization with ARC, a "vision" on how containers could be launched on ARC CE and what a user/ARC is allowed/supposed to do in that workflow. - Balazs: Infosys for anonymous monitoring & GDPR - Balazs: How much we can simplify the infosys? - discussion: ARC python native clients - discussion: life after x509, ARC security roadmap - ARC 7 development roadmap