[Lofarpwg] LTA Pulsar data - listings + Plan for deletion / re-processing

[Vlad Kondratiev]

Hi All,

I have put together preliminary plan to re-process the pulsar BF data in
the LTA to reduce its size together with the
deletion of pre-production data (pre-Cycle0) that we think can be deleted
(test, failed, obsolete, etc).
You can read it here:
https://docs.google.com/document/d/1_ma-geiG-IsdPOXZCReCwuMcrBIjaxeuIBVkSzSPpSE/edit?usp=sharing

The deletion part of pre-Cycle0 data is the 1st priority now, as we want to
clean LTA as much as possible. I guess some justification
has to be provided why we want to keep some data, but it should not be a
problem, I think. In my opinion most of the data
should be kept especially in Pulsars2 data. The largest part of the Pulsars
data is actually LOTAAS data (1.4 PB out of 1.9 PB)
that was ingested manually before RO took over. These data definitely will
stay but will need to be moved to corresponding
project folders on the LTA. The rest of the data we should check whether we
need it or not.

At the bottom of this email you can find links to 2 files, one is the
spreadsheet with ObsiDs/folders sizes for Pulsars2 and Pulsars project,
and the second is the Listing of files on the LTA for Pulsars project
excluding LOTAAS data with their sizes. Please take some time and
look over them, and also use the web-summary page for pre-Cycle0 data (
https://www.astron.nl/lofarpwg/psr-precycle0-time.html) that
has some diagnostic plots to have an opinion on what should stay/or be
deleted or shall we look at the data before making the decision.

I would like to have some discussion over this and re-processing plan at
the next PWG if possible.

Cheers,
Vlad
=====================================================================================================

Web-summary page with diagnostic plots for pre-Cycle0 data:
https://www.astron.nl/lofarpwg/psr-precycle0-time.html

Listing of all files in Pulsars project:
https://drive.google.com/file/d/1OU4eNZzZnZdsR40h9_ZxxPXqpCe-TD_t/view?usp=sharing

Spreadsheet with size info of the obsids/folders for Pulsars2 and Pulsars
(two different sheets):
https://drive.google.com/file/d/14hijNxmH1roDeYd6-qOEzF1mDA7Jns6m/view?usp=sharing
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.astron.nl/pipermail/lofarpwg/attachments/20200904/22b98707/attachment.html>