[Lofarpwg] Request for downloading LOFAR early-cycle Data (Cycle 0 till Cycle 6)

Thu Nov 30 10:04:04 UTC 2023

Dear All

As you know, the LOFAR ILT board announced that the LOFAR data collected in
cycles 0 to 6 (included) and contained in the LTA will be 'retired' (a
dystopic version of 'permanently deleted') next year.

Because of this, they asked to submit a form to request the possibility of
downloading data of interest (note that this is only a _request to
download_, not a direct access to the download).

I went through the list of projects, and selected those containing
timing-like data. As I think that these are relevant for many of you too, I
report my request below.

I do not know how the ILT will proceed with this, since they gave extremely
constraining timescales for the 'retirement' to happen -- and, as you can
see, the volume of data that I indicated is large. Note, though, that the
effective data volume will be much lower than 1 PB in the end, since LOFAR
tends to store both the folded and the search-mode data that the folded
archives were derived from (and we want to keep only the latter).

However, in case any of you can offer storage space to aid with the data
archiving, let me, Joris or Gemma know. Of course these observations will
then be accessible by anyone in our community.

Best

Caterina

[image: Google Forms]
Thanks for filling in Request for downloading LOFAR early-cycle Data (Cycle
0 till Cycle 6)
<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fforms%2Fd%2Fe%2F1FAIpQLSfwBT8OgALv_7vwDxFLCi7qHODCK9gIeKGTztYzXyOjzQ2XhQ%2Fviewform%3Fusp%3Dmail_form_link&data=05%7C01%7Clofarpwg%40astron.nl%7Cf4982dbb615d4b84149908dbf18bb68c%7C03e9259c32624694ad934dce7c9fa403%7C1%7C0%7C638369354601423669%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=d2qGZSWplDPxrNDJiV6mAA1JKDixHU28tuFj5boyEMg%3D&reserved=0>Here's
what was received.
Request for downloading LOFAR early-cycle Data (Cycle 0 till Cycle 6)

LOFAR is preparing the upgrade to the LOFAR2.0 system, which will offer
simultaneous low- and high-band observing, increased field-of-view, and
various other improvements to the sensitivity and operation of the
telescope. The new system will generate considerably more data than the
current one. The need for processing and storage of this data represents a
significant challenge, as it outstrips current affordable solutions. To
realize the goals of the new Observatory, the ILT board has approved a data
life cycle that optimises science data processing capacity and efficiency
with respect to affordable data storage. This includes data retirement. The
ILT anticipates that (final) science data products, generated through
standard pipelines (such as image cubes) can be kept available indefinitely
(under open access policies), but that raw/intermediate products may only
be retained for further processing within a restricted dwell time.

To prepare the ILT infrastructure for the LOFAR2.0 data challenges, a first
step needs to be taken now that involves the retirement of early Cycle data
from the LOFAR Long-Term Archive (LTA). After careful analysis, it has been
concluded that data from Cycle 0 up to and including Cycle 6 is either low
in metadata quality or already valorized and published (
https://science.astron.nl/telescopes/lofar/science-with-lofar/lofar-papers/)
or, based on the download statistics, of little demonstrable interest to
the current community.
*The ILT has therefore approved that mid-2024 this data will be deleted
from all LTA sites. *

Through this notification, we give the opportunity to any users (i.e. not
only the projects’ PI’s) who may need to retrieve data from these early
LOFAR Cycles, to signal this to the Observatory to coordinate the data
download. The requests will be evaluated by the Observatory for technical
feasibility and reviewed by the ILT-board for approval. We emphasize that
all data concerned is public data.

Please, refer to the overview distributed via e-mail to understand which
data is selected for retirement. To avoid excessive load of the data access
services at the LTA sites, the Observatory needs to be informed of your data
download plans and needs to evaluate them in relation to data access
requirements for other users and purposes. For this, we ask you to submit a
formal request for data download by using this Google form.

The deadline for submitting your request is Friday, 1 December, 12 UT
(noon).
Email *
caterina.tiburzi at inaf.it
What is your name and title?
*
Caterina Tiburzi, Dr
What is your affiliation?
*
INAF-OAC
What is your e-mail address
*
caterina.tiburzi at inaf.it
Which Cycle projects the data you wish to download belong to? Please
provide a comma separated list.
*
DDT5_003, LC0_008, LC0_011, LC0_022, LC1_003, LC1_020, LC1_027, LC1_035,
LC2_007, LC2_010, LC2_025, LC2_039, LC3_013, LC3_035, LC4_003, LC4_004,
LC4_009, LC4_019, LC4_021, LC4_032, LC5_003, LC5_018, LC6_028, LC6_030,
LT3_001, LT5_003
Which SaSID's per Cycle projects you wish to download? Please provide a
comma separated list.
*
All of the SasID of the projects listed above are required
What is the size (in TB) of your total download request for each of the 3
LTA sites?
*
1152.4 TB
What is the purpose of your data access request and what is your data
(re-)processing plan?
*
The purpose of the data access request is to ensure the storage of the
timing data, and the repeatability of the studies that have been already
published, as well as the studies that will be published based on these
data. In particular, repeatability is crucial for a fair use and exchange
of scientific information. Concerning the reprocessing plan, we plan to
fold the search-mode data present in the requested projects, in case that
the folded versions of the observation are not already present in the LTA.
In the latter case, we will remove almost all the search-mode data from the
observations, and only store the folded archives (that consist in a minimal
fraction of the total data volume)
What resources will you be using to store and further exploit the data you
request for download?
*
We plan to use local servers and computing centers.
Create your own Google Form
<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fforms%3Fusp%3Dmail_form_link&data=05%7C01%7Clofarpwg%40astron.nl%7Cf4982dbb615d4b84149908dbf18bb68c%7C03e9259c32624694ad934dce7c9fa403%7C1%7C0%7C638369354601423669%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=QGsfNn3kjo4opJLqpfgQP4QjvAqviOlfHm%2FS3IRTvnI%3D&reserved=0>
Report Abuse
<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fforms%2Fu%2F0%2Fd%2Fe%2F1FAIpQLSfwBT8OgALv_7vwDxFLCi7qHODCK9gIeKGTztYzXyOjzQ2XhQ%2Freportabuse%3Fsource%3Dhttps%3A%2F%2Fdocs.google.com%2Fforms%2Fd%2Fe%2F1FAIpQLSfwBT8OgALv_7vwDxFLCi7qHODCK9gIeKGTztYzXyOjzQ2XhQ%2Fviewform%26usp%3Dmail_receipt_abuse&data=05%7C01%7Clofarpwg%40astron.nl%7Cf4982dbb615d4b84149908dbf18bb68c%7C03e9259c32624694ad934dce7c9fa403%7C1%7C0%7C638369354601579992%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ALjnD0oCbDdmOEDRCaAVx1VQUHgZU2gFjYApdn8Sisc%3D&reserved=0>

-- 
Dr. Caterina Tiburzi
Researcher

INAF - Osservatorio Astronomico di Cagliari
Via della Scienza 5, 09047 Selargius (IT)

Email: caterina.tiburzi at inaf.it
Skype: caterina.tiburzi84
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.astron.nl/pipermail/lofarpwg/attachments/20231130/5b6a44ca/attachment-0001.html>