Dealing with Full SEs

Some SEs are shared with other VOs, therefore biomed users may not be responsible for an SE being full. To figure out if biomed is a heavy user of that SE or not, check the last VAPOR's space usage report, and pick up those with less than 10% AND less than 500 GB of free space left.

Then, before going further, check that the SEs do not fall into one of the following cases:

• a ticket is already open for that issue;
• the SE is in scheduled downtime, not in production, or not monitored (check faulty resources on VAPOR);
• the SE has been or will be decommissioned shortly (see the list of decommissioned resources).

If biomed uses more than 20% of the total used space, then apply the clean-up biomed files procedure, see next section.

Otherwise, apply the full SE warning procedure (end of this page).

1. Get the list of LFNs and DNs having files stored on this SE using LFCBrowseSE. Note that this may take a while (few hours): LFCBrowseSE <se.hostname> –vo biomed –lfn –dn –summary | tee se.hostname_files_dn
2. Submit a GGUS Team ticket, assign it to VOSupport, and put biomed-issues-followup [NOSPAM][AT] googlegroups.com in the CC field of the ticket. The concerned site should be notified of this ticket (field Notify site in GGUS).
3. Put a link or attach the files list in the ticket.
4. Notify the concerned users by email, and require them to move or delete their data:
   • Go to the last VAPOR catalog-based SE scan with minimum used space = 0, look for SE, and follow the link “Show email template”, then copy and paste the different fields (TO, SUBJECT and the body) into a new email. Set Reply-to=biomed-technical-support [NOSPAM][AT] googlegroups.com. You may also use the email template instead.
   • Attach the list of files to the email.
5. Send reminders to users every week until at least 90% (or at least 500 Go for big SEs) of the SE space is free. Note that you may need to contact the site admins in order to get the accurate value of the storage space for biomed (see https://ggus.eu/index.php?mode=ticket_info&ticket_id=58733).
6. The ticket is solved when at least 90% (or at least 500 Go for big SEs) of the SE space for biomed is free.

Submit a GGUS Team ticket, assign it to concerned site, and put biomed-issues-followup [NOSPAM][AT] googlegroups.com in the CC field of the ticket. Use the ticket template for admin notification.

The admin may answer that biomed uses a large part of the SE, in that case this may be due to zombie files (files that are not registered in the LFC). In such a case, we generally produce the list of entries from the LFC, send it to the administrator and ask to remove those files for us.

VAPOR automatic cleanup of dark data applies to DPM and dCache Storage Elements, using the GFAL API with GSIFTP access URLs.

StoRM SEs do not support GSIFTP as an access protocol, only SRM is available but it is very limited and does not allow long listing (limit to 2000 files by directory). As a result, the clean-up of a StoRM SE cannot be done from the user side only: it must involve an action on the administrator side, to get the dump of biomed files on the SE.

⇒ Use this ticket template to request a dump from the site admin.

This tool compares a dump provided by an admin with an LFC dump and removes dark data files on the SE. It is based on the experience we had with SE stormfe1.pi.infn.it (Pisa-style dump) and storm-01.roma3.infn.it (Roma-style dump), all example files are available from the repo. Before using it, make sure the format of the dump received from the administrator is the one expected by the tool. Contact me (Franck), if this is not the case.