Technical Coordination Weekly: data transfer jobs

Tuesday 21 Oct 2025, 14:00 → 15:30 Europe/Stockholm

Zoom

Balazs Konya (Lunds universitet) , Balazs Konya (Lunds universitet)

ARC technical coordination group members and invited persons

21 October 2025, 14:00 - 15:22

Present: Balazs, Maiken, Mattias, Lars, Andrii, Andrej, Florido, Oxana

The meeting is dedicated to data transfer jobs

• Balazs summarises actions discussed in Kastrup in May, see https://docs.google.com/document/d/1i_4f1jKOPOBGLYT66wjhdYIz-AUe0g1uTUqQUlGd-oU/edit?tab=t.0
• Andrii reports on preliminary tests and experience. Some new attributes are needed, e.g. for accounting. New job types and states are also needed. Configuration for data-transfer-only servers should be simpler. An example use case could be pre-copying data to a cluster.
• Lars presents a proposal, see https://docs.google.com/document/d/1rdhZX3r0iRt96iaZJr12NwNNoPeuEySicPnOP1hO7ME/edit?usp=sharing
  • Some discussion on possibilities/feasibilities of using session directories as temporary storages that can be re-used, since we can access job directories via known URLs. URL-to-URL transfer might be an easier start, as this does not require designing a new SE. 
  • For streaming, session directory might not even be needed, and data can be placed directly in the destination
  • The proposal is good, a follow-up on developments is expected in 3-4 weeks

14:00 → 14:20 Introduction

Recap of the status after the Kastrup meeting

Speaker: Balazs Konya (Lunds universitet)

14:25 → 14:45 Proposal

Implementation proposal

Speaker: Lars T Hansen

14:50 → 15:20 Discussion