EISCAT-3D WP8 Deliverable D8.2

(1)

EISCAT-3D WP8

Deliverable D8.2

23 February 2007, v3.0, D.J.McKay, I.D. Finch, I.W.McCrea & [email protected]

1. Introduction... 3

1.1 Report objectives ... 3

2. Overview of the data and signal processing architecture... 3

3. Full-radar Receiver Beam ... 5

4. Interferometric Data... 7

5. Intermediate archive content... 8

6. Final archive content... 8

7. Remote sites and data distribution architecture ... 9

8. Data input and rates ... 10

9. Computing hardware... 12

9.1 Data storage ... 12

9.2 Data security ... 13

1. Data Robustness... 13

2. Data Permission ... 14

9.3 Enhanced-service archive ... 14

9.4 Data formatting and association... 14

9.5 Network solutions ... 15

10. Other Aspects not included at this stage ... 15

10.1 User Interaction with the EISCAT-3D Data System ... 15

10.2 Authentication and Data Security ... 15

10.3 Data Visualisation... 16

10.4 Complementary Data Handling ... 16

10.5 Post-processing software ... 16

11. Dependences on Other Work Packages ... 17

12. Summary and Conclusions ... 18

References... 18

Appendix A: Data Rates Explained ... 19

Summary ... 19

Conventional ISR Mode ... 20

3. Option 1 ... 20

4. Option 2 ... 22

5. Option 3 ... 24

6. Option 4 ... 26

Interferometry ... 28

Appendix B: EISCAT3D Data Storage Specification ... 29

Appendix B(1): Central Receiver Site - Permanent Archive (Single)... 29

Abstract ... 29

1. Introduction... 29

2. Aim of the Information-Gathering Exercise ... 29

3. Overall Requirements ... 30

4. Phased Deployment ... 31

5. Operating Modes... 31

6. Data Storage Specifications ... 32

(2)

8. Software Specifications ... 33

8.1. Operating System... 33

9. Total Cost of Ownership (TCO) ... 34

10. Physical Size and Requirements ... 34

11. Scalability and Migration... 34

12. Environmental Specifications ... 34

13. Error Recovery Requirements... 35

13.1. Overall Downtime... 35

14. Documentation... 36

15. Training... 36

16. Warranty ... 36

Appendix B(2): Remote Receiver Site - Remote Site Data Stores (x4) ... 37

Abstract ... 37

1. Introduction... 37

2. Aim of the Information-Gathering Exercise ... 37

3. Overall Requirements ... 38

4. Operating Modes... 38

5. Data Storage Specifications ... 39

6. Networking Specifications... 39

7. Software Specifications ... 40

8. Total Cost of Ownership (TCO) ... 40

9. Physical Size and Requirements ... 40

10. Environmental Specifications ... 41

11. Error Recovery Requirements... 41

12. Documentation... 42

13. Training... 42

(3)

1. Introduction

This document forms the second deliverable from Work Package 8 of the EISCAT-3D Design Study, which relates to the design of the data archive and distribution system for the new radar. The current document summarises the progress made on this area of the Design Study up to January 2007, and sets out the current understanding of the dependences between this Work Package and the other packages of the study. In particular, we concentrate on the design of the data system topology and possible hardware solutions for implementing it. Our principal conclusion is that, in spite of the very high data volumes which will be delivered by the proposed EISCAT-3D radar system, a data archiving solution capable of preserving all of the scientifically important data products is achievable using existing technology.

1.1 Report

objectives

This report addresses the following description of work, as listed in Annex 1 of the design study contract [1]

•

DoW-3

An initial low-level design relating to computing hardware, data storage, and network solutions for a secure archive will be undertaken, including evaluation of the relative merits of on-line and near on-line storage.

The same document defines the objective of the current deliverable as follows:

•

D8.2

Low-level design document: Networking and data storage requirements and favoured hardware solutions

It should be noted that many of the issues relevant to WP8 depend upon the outputs of the signal processing work package (WP9), the control and monitoring work package (WP7), the network and time infrastructure work package (WP12) and the expectations of the end users (the EISCAT-3D scientific case). Hence at this stage of the study, there is still a degree of limitation imposed on WP8 because definitive outputs have not yet been produced from the other packages. That said, the following report attempts to summarise all the information currently available for this work package and explains the rationale and options for the currently envisaged implementation of the EISCAT-3D data archiving and distribution system.

2. Overview of the data and signal processing

architecture

Figure 1 shows the current conceptual design for the signal processing functions to be carried out at the central site of the EISCAT-3D system. The central site (probably to be located somewhere near Tromsø), will be much larger than the other “receive-only” sites, but the topology for all sites should be basically similar. Although it belongs within the remit of WP9, the signal processing functionality has profound implications for the design of the data archive and distribution system, so we review it briefly in this section.

(4)

Figure 1: Signal processing architecture [2], showing where the data archiving fits in with other components of the EISCAT-3D signal processing. The “disk” icons mark the areas which are addressed by this work package.

(5)

The central (transmitter) site comprises a large number of antenna elements, forming a phased array for both transmission and reception. Because the individual elements are likely to be crossed Yagi antennas, it will be possible to measure signals with two linear polarisations. (We shall discuss polarisation issues later.) The receiver array at the central (transmitter) site will be much larger than those at the receive-only sites. The elements of each array are divided into modules. These modules are sub-arrays and may therefore be used as interferometric elements. The output signals from the elements within each module are beam-formed to generate the actual receiver beams. Since multiple beams will be formed, multiple copies of all the input signals from the elements will be required. The module crossbar accomplishes this task.

The output of the module crossbars are replicated sets of all the input signals from the elements. These will have already been amplified and digitised. These functions will probably be carried out close to the antennas themselves to prevent noise loss. The circuitry used as receiver protection (against the high power levels that are expected during the transmit phase of the radar duty cycle) is not relevant for this work package, since it does not modify the form of the signal, but merely time-slices it. However, it does not necessarily follow that the data rate is reduced, since it may be necessary to sample the transmitted waveform e.g. to obtain information on pulse shape for subsequent data analysis.

The beam-formers take a number of inputs along with configuration information about the relative locations of the antennas that provide the input signal within the entire phased array – not just the module. Thus each beam-former needs to be capable of full phase-correction and fractional sample addition, and must be capable of handling input signals from antennas over the full geometric extent of the array. This flexibility does not introduce additional complication, as it is simply a matter of scale. Furthermore, it means that any beam-former can be placed anywhere in the array; there will be no location-specific units, and this will ease manufacture, installation and maintenance. Combining modules for the final radar beam is then a matter of adding signals from the beam-formers, rather than having secondary beam-formers. It also means that in the interferometric case, the different interferometric elements have already had their delay/phase corrections introduced, and no further phase rotation or interferometric delay needs to be applied. A test array, consisting of a limited number of Yagi elements, is presently being constructed at Kiruna as part of WP9. It should be possible to build a fixed beam system for the test array very simply, thus creating a rapid prototyping environment for other parts of the system, including some aspects of WP8.

It is anticipated that the outputs from the beam-formers will be handled in two different ways. The first is that the beam-formed input for each beam from each module will be added with the beam-formed input of the same beam from all other modules. The second is that each beam-formed input from each module will be input into the interferometric analysis system. In the following sections, we discuss the data handling for each of these two pathways separately, exploring the implications for the data archiving and distribution systems in each case.

3. Full-radar Receiver Beam

The primary operating mode of the EISCAT-3D radar will be the formation of one or more full phased array beams, combining together the corresponding beams from all the modules of the array. This is simply a digital addition. Since the beam formers will be set up to accept inputs from the entire array, it is not necessary to apply additional corrections here.

From the perspective of the data acquisition system, the output of the full-adders (one per beam) will be split. One part will be put into a cyclic buffer and then streamed to disk. These data will be analogous to the sample level data currently taken in some existing EISCAT experiments. The other branch of the beam-formed output will be put into a digital receiver and then auto-correlated and integrated. These data will be the equivalent of the autocorrelated data taken in most experiments on the current EISCAT system.

(6)

The decision on which of these two data types (i.e. beam-formed or autocorrelated data) is chosen for storage at any particular time should be made in the context of the science requirements for the experiment being undertaken. In the current EISCAT system, the archive consists of autocorrelated data, which have the advantage that they can be time-integrated. However, there are certain scientific advantages to storing the beam-formed samples so that they can subsequently be correlated by off-line software. This potentially allows a more flexible approach to auto-correlation and time integration, though at the price of requiring more powerful computing facilities at the central site. In Section 8, we show example calculations of data rate and archive size, based on long-term storage of beam-formed sample streams and autocorrelated data. Both scenarios are challenging: beam-formed data rates are of order GB/s, which means that the consequent data volumes are unacceptably large for permanent storage. This would not preclude the short-term storage of such data in cyclic buffers prior to subsequent re-processing, but even these would need to be relatively large, unless the sample streams were heavily decimated.

Autocorrelated data are implicitly larger in size than sample level data, because of the need to store the full correlation lag matrix. However, this increase can be mitigated by integrating the data temporally and spatially, which is not possible for the sample level data stream. Our calculations in Section 8 show the effect of one such scheme, two more are detailed in Appendix A. This produces manageable data rates of as little as 2 MB/s, resulting in storage lifetimes on the order of 15 years for a Petabyte store. However, delivering autocorrelated data in this form will place stringent requirements on the signal processing work package (WP9) e.g. for decimation and correlation. It should also be remembered that storing autocorrelated data from both plasma lines as well as the ion line would increase the data rates and reduce the archive lifetimes by around a factor of 3 compared to the figures shown in Table 8.2. Because they can be integrated over range and time, autocorrelation functions probably offer the best solution for permanent storage. It is, however, also desirable to store data temporarily from at least some experiments as beam-formed, sample streams. Although of greater volume because they cannot be time integrated, such streams allow users to exploit the maximum flexibility in auto correlation and range gating. A scientific justification would be required to determine the autocorrelation and time integration strategies and to set the latency time of the cyclic buffer so that its duration can be varied according to the scientific importance of the data being taken. A strategy which combined short-term storage of sample streams with long-term storage of final autocorrelation functions would thus allow users to ensure that the contents of the permanent data archive represented the optimal measurement of the ionosphere which could be made from any given experiment.

Regardless of whether the final autocorrelated data are being calculated at the time of the experiment, some form of real-time auto-correlation is required, in order to generate visualisation data, which will allow monitoring of the radar performance and “first-look” views of the data, using a visualisation system similar to that implemented in the existing EISCAT system.

Aside from the issue of data volume, there are some other, less serious, problems associated with the storage of integrated data. For example, such data are often contaminated with cluttering signals from satellites and space debris. Such contamination is currently removed at the time of data analysis, but this is an inefficient method of clutter removal, since it generally takes place after the data have been pre-integrated. It should be possible to devise some intelligent software to eliminate such clutter in real-time, prior to data integration, but care would be needed to distinguish between genuinely unwanted clutter and unusual radar echoes which might have geophysical significance.

Note that in Figure 1, the disks which store the beam-formed samples and the auto-correlated outputs of the “normal” radar signal path from each radar beam are shown as being separate. This is because the data streams are actually independent, and it may not be necessary to store all of the potential data products in order to achieve the required science objective. The disk symbols indicate those points in the EISCAT 3D system where WP8 systems would need to take over from the signal processing systems of WP9 with regard to data handling.

(7)

4. Interferometric

Data

In addition to the normal beam-formed data generated by the phased array, it is also intended that the radar should have a second type of data product, namely a set of interferometric data with sub-beamwidth resolution. The availability of such data opens up an important area of new physics, not routinely accessible to the present EISCAT system.

Under normal conditions, the ionosphere can be considered as a beam-filling target which, within the size of the EISCAT beam (a circle of radius a few kilometres at F-region heights) is effectively homogeneous. This means that, within this volume, no strong structure exists within the ionosphere, so that the whole area of plasma illuminated by the radar has the same basic properties (density, temperature, composition, velocity etc). However, it is known from existing EISCAT studies that under some conditions these assumptions can break down. For example, during certain types of aurora, the EISCAT volume can be dominated by sub-beamwidth scatterers known as NEIALs (Naturally Enhanced Ion Acoustic Lines), whose size is assumed to be of order tens to hundreds of metres, and whose backscatter cross-sections can be two orders of magnitude above that of the surrounding ionosphere. There is an ongoing debate about what these phenomena represent, and the nature of the processes which lead to their formation. Hence it is important to study them further, and this cannot be done with conventional radars. However, NEIALs occur very infrequently (much less than 1% of observations display this behaviour) and unpredictably, making them very difficult to study by specialised experiments. A better solution would be to ensure that every observation undertaken by the EISCAT-3D radar, at least from the central site, was suitable for interferometric analysis, so that when a suitable event is detected, the interferometric content of the returned signal was analysed and the size and number of sub-beamwidth scale scatterers determined.

The methodology for making interferometric measurements on the EISCAT-3D system has already been outlined in the two reports produced by Work Package 5 [3], [4]. The reconstructed image is obtained by measuring the cross-phase and coherence of the signal measured by multiple pairs of sub-arrays within the overall antenna. For this reason, the central antenna array will probably not be in a neat geometrical form such as a square, rather the spatial distribution of the constituent modules will be chosen in a way which maximises the radar’s interferometric performance.

Because of the infrequency of sub-beamwidth scatterers and the high volume of interferometric data, it is not planned to store such data continuously. It should be obvious that interferometry data are only of interest when there is structure which needs to be resolved. At such times, the coherency of the signal between module pairs will rise to some finite value, while at other times it will be close to zero. Because of this, it should be sufficient to measure continuously the coherency between a few module pairs, with cross-phase and coherency data from all module pairs only being retained when the tested coherencies rose above a given threshold.

However, in order to catch the onset of sub-beamwidth scatterers, there is a requirement to record the full interferometry data in a short-term cyclic buffer so that, once the coherence threshold is exceeded, the interferometric data from the period immediately preceding the start of the event can be reconstructed. Once the coherence threshold is exceeded, the full interferometric data will flow to disk until the coherence drops below the threshold again. The durations of such events seldom appear to be more than a few minutes. In order to catch the final disappearance of the scatterers, it is planned that data should continue to be recorded to disk for a fixed time after the coherences from the tested module pairs fall below the threshold, unless the threshold is once again exceeded during this period.

This means that, as for the beam-formed data, it is proposed to use a buffered recording system for the interferometry data. The buffer is used for continuous recording in order to ensure data preservation while the coherence thresholds are tested. Normally, this buffer would be overwritten every few minutes, except when the coherences rise above the threshold value, at which point the buffered data will streamed to permanent storage, since they are assumed to correspond to the start of an interval of sub-beamwidth scattering.

(8)

5. Intermediate archive content

As discussed above, there are two types of “ring buffer” in the EISCAT-3D data storage system. These will have limited-duration storage, before the data are discarded. Within that time period, the scientist or operator must determine whether or not the data are worth keeping or processing into some other form, though most aspects of the decision-making process can probably be automated. The first of these buffers is for standard radar use (marked RDCB in Figure 1), and will store beam-formed data in the form of sample streams for a short period of time. Longer-term storage of such data might not be a given, but could be decided in real-time by automatic detection/analysis software. Similar to the interferometric coherence threshold detector (see above), such software would constantly monitor the incoming data and, using the delay provided by the ring buffer, pre-emptively intercept and store data deemed to be of scientific interest. If not, the data will be overwritten after a while, as the buffer cycles. It is anticipated that any automated algorithms which decide whether data are to be stored, and for how long, will need to be refined at the early operational stage of the project, as experience is gained in their use. By putting a connection point to the ring buffer in place during the initial deployment, later additions of such automatic detection and storage software can be added. In the interim, the connection would serve as a pick-off point, should particular experiments require the storage of data in an alternate form. However, such storage would initially need to be provided by the end-user, and the EISCAT-3D system would only provide the connection interface.

The interferometry component also has a cyclic buffer which, as described above, is used to buffer data which might be deemed useful for storing. The process of determining if the coherence threshold has been exceeded requires time for the correlation to be made, transforms to be performed, and the data integrated to a level at which signal can be detected for a decision to be made. Additionally, if the threshold limit is set high (to avoid spurious recordings of noise), then “back-tracking” the data somewhat is necessary to catch the rising edge of the event, which may be of scientific interest. This results in a cyclic buffer that needs to store data in the order of tens of seconds. Unlike the RDCB, it doesn’t make sense to talk about “picking off” data from this buffer, since the final interferometric storage would be designed to do this job.

Data storage in the RDCB will probably be in a low-level format (raw bit streams). If retrieved directly for experimental purposes, these data would need to be post-processed by the recovery system to convert them to a more usable format, and transferred to conventional media. If passed on directly to the next stage of the system (e.g. for autocorrelation), they would remain in the raw format, thus making them appear more or less as a transparent component of the system.

In addition to their potential uses as storage and raw-data pick-off points, the ring buffers would also allow the system to cope with anomalies in the regularity of the data flow. By the time the data reach this point, they will be on conventional network systems, and delays can be expected owing to packet collision and the quantised nature of the packet transfer protocol. As a result, the ring buffers will serve to regulate the flow of information, and would allow a catch-up grace period for data which are temporarily disrupted. Such disruptions, in addition to the “standard operation” disruptions listed above, would also include more major incidents, such as rebooting a down-stream computer. The RCDB would provide sufficient latency to recover, with time, the computer outage, as it would be possible to process the backlog of data faster than the incoming data rate. This would not apply in the case of the interferometric system.

6. Final archive content

As noted in Section 3, the contents of the permanent data archive, at least for the central site, will be data of two kinds, namely conventional ionospheric radar data and interferometric data. While storage of uncorrelated beam-formed sample streams is possible, an aggressive decimation strategy is needed to reduce the data input rates to values compatible with permanent storage. This will be achieved by storing time-integrated, atuocorrelated data as in the present EISCAT system, though this has the weakness that the integration and correlation strategies need to be preset. A short-term ring-buffer of

(9)

sample stream data however would allow an operator to vary the integration and correlation strategies post hoc while the data remained in the buffer.

It should also be noted that, because of the data rates involved, it is necessary to provide each beam-former with a direct connection to the archive, rather than attempting to multiplex several inputs into the same connection. This is why Figure 1 shows both correlated and uncorrelated data stores separately for each beam.

For the interferometry data, it only makes sense to store all interferometric channels in order to reconstruct the image-plane from all of the available visibility data. Furthermore, owing to the sizes of the intermediate interferometry products (such as cross-correlations), it does not make sense to store partially processed data. The storage problem for interferometry is thus equivalent to that for multiple streams of beam-formed data, in that it imposes a requirement to handle very high input data rates, but in this case only for relatively short time intervals. The full visibility data would be stored in the archive until they had been completely post-processed, at which point they could be deleted or moved elsewhere, and only the highest-level interferometry products would become part of the permanent archive (see Section 8).

7. Remote sites and data distribution architecture

Since the EISCAT-3D system is planned to be multi-static, with up to four remote sites, a key issue for WP8 is the question of how the data are moved from the various sites back into the central archive and from there on to the users. This is an area that WP8 has explored, within the limits imposed by other work packages. The limitations are those of WP7 (Control and Monitoring), WP9 (Signal Processing), WP5 (Interferometry) and the scientific case. Without the scientific priorities and the proposed data rates from the input work packages, it is not possible to finalise the intra-site data rates, and thus to fully define the limitations of the data distribution architecture. That said, the following is brief discussion of some of the options that are currently available.

It is assumed that the EISCAT-3D system will have some kind of master archive, ideally located at the central site, which will hold a copy of all the data which it was practicable to store centrally. The constraints on the size of the archive will include the large sizes of some of the data sets, especially for some of the interferometric data products, and the capacity of the network connections from the central archive to the remote sites of the EISCAT-3D system, some of which may be in remote locations. It is assumed that remote site data will be brought back to the central archive in the same form as the permanently stored data from the central site (see Section 3). In principle, the beam-formed remote site data are the same size as those from the central site. However, in practice it is intended that the radar will be run with an approximately 25% duty-cycle (i.e. the radar will only transmit for 25% of real-time with the central site receiving for the remaining 75% of the time). This means that the volume being observed by the remote sites is only illuminated for 25% of the time and data need only be recorded at those times. In addition, if transmitter sampling is not conducted at the transmitter site, then it will only produce 75% of the data compared with a site which is receiving for 100% of the time.

For the remote sites, it would be impossible for the full beam-formed data at 30 MHz bandwidth to be moved to the central archive by any standard public link. Transfer of such data could only be undertaken for short periods with data being accumulated at the remote sites and transferred to the central archive during later periods of operation with lower data rates. Longer term operations requiring full-bandwidth data could only be handled using either proprietary networks or physical transport of media. It is anticipated that bandwidth will always be more than sufficient to transfer any auto-correlated and integrated data products in real-time.

It is clear that some on-site storage will be needed at the remote sites, both because there needs to be some safeguard against data being lost in the event of a network outage and because data may need to be buffered on-site during times of high data rate operation (e.g. experiments with large numbers of

(10)

beams). There might be a requirement for such a buffer to have a latency time of a few weeks, given the possible logistical difficulties of repairing internet links in Arctic Scandinavia. Even in this case, the remote site archives would not require the Petabyte storage capabilities needed for a long-term archive at Tromsø, and a few tens of Terabytes would probably be sufficient, though the capacities for I/O of data to and from the remote archive should be the same as those for the central site.

Analysis capabilities would obviously need to be provided at the central site, but since this also acts as the central archive, there is not envisaged to be a requirement to move large amounts of raw data off the central site. A large computing facility for data analysis needs to be located at that point. Provision of this computing system is not covered by the existing design study.

A similar situation exists for interferometry, however this is more limited in scope, as all the interferometry data will be acquired from the central site (assuming no interferometry is done at the remotes). Data processing must be done on-site to keep the export data rates low, and experimenters who wish to take intermediate data away with them will probably have to do so using their own media. All other interferometry calculations will be done using local processing, provided as part of WP5, and only final results (figures, plots and animations) will be exported via conventional Internet links.

8. Data input and rates

Some indicative data rates for the output of the EISCAT-3D radar are shown below. These have been derived from more detailed spreadsheets which we have constructed to calculate the data volumes which the archive system should handle [6]. They should not be taken as a firm indication of what is planned, merely as a demonstration of the difficulties inherent in the storage of the various kinds of raw radar data.

Table 8.1 shows the total data rate for the whole central site, calculated by multiplying the sampling rate on each antenna element by the number of elements and the number of bits per sample. This represents the volume of data in the whole central site system, prior to beam forming. The table also shows the volume of sample-level data obtained after beam-forming, assuming that the data rate for each final beam is equivalent to the original data rate for each constituent antenna element.

Receiver bandwidth 30 MHz Polarisation channels 2

Bits per sample 16 b

Sampling rate 80 MHz

Data rate per element polarisation

1.28 Gb/s Data rate per element 2.56 Gb/s

Data rate from all elements 47.19 Tb/s

= 5.90 TB/s

Storage at site 1000.00 TB Storage duration for all

pre-beam formed data

169.5 s Assumed number of beams 1

Data rate per beam 2.56 Gb/s Beam-formed data rate 2.56 Gb/s

= 0.32 GB/s

Storage duration for all beam-formed data

3125000 s

= 36.2 days

Table 8.1: Data rates and storage durations for pre-beam-formed and beam-formed data at the EISCAT-3D central site.

(11)

It is clear that the permanent storage of data prior to beam-forming is not practical because of their huge volume. Even after beam-forming, however, the volume of sample-level data is too large for long-term storage unless the beam-formed data can be decimated. This will be done by autocorrelating and time integrating the autocorrelation functions, as is done in the present EISCAT radars.

Table 8.2 shows the anticipated data volumes which would occur assuming the storage of auto-correlation functions with a pre-integration time of 2 seconds. :

Required minimum range 50 km Required maximum range 1500 km Range increments 500

Number of lags per range 200 lags No. of data per matrix

profile

1.00E+05 data Data storage per datum 8 bytes Data per correlated matrix 0.80 MB Post-correlation integration 2 s Total data rate per beam 0.4 MB/s

= 1.4 GB/hr

Table 8.2: Data rates for auto-correlated data at the EISCAT-3D central site.

The assumed number of range gates (500) arises from a rough calculation assuming range increments as follows: 50-80 km: 100m, 80-100 km: 500m, 100-120 km: 1 km, 120-150 km: 3 km, 150-300 km: 5 km, 300-500 km: 10 km, 500-2000 km: 50 km. Each of these range increments is about 25% of the characteristic scale length of the ionosphere at the relevant altitude, as calculated from either the plasma scale height or the wavelength of characteristic wave and tidal oscillations. Autocorrelation functions are assumed to be complex with 100 lags in each of the real and imaginary parts.

Table 8.3 indicates the implications for the data archive of storing these auto-correlations, assuming decimated correlator output. It shows the expected lifetime of a buffer of 100 TB at the central site and for a permanent archive of 1000 TB.

Number of beams 5 beams

Total data rate per second 1.00 MB/s

= 3.6 GB/hr

= 30.8 TB/year

Central buffer duration 3.24 years Archive duration 32.48 years

Table 8.3: Archive durations for auto-correlated data at the EISCAT-3D central site.

The foregoing tables obviously only consider the data rate for the central site, and the overall data throughput will increase when remote sites are included. However, for these sites it should be possible to greatly reduce the volume of data because, for a pulsed radar, the range over which incoherent scatter return is received is limited to the subset of range gates in which the remote site beams intersect with those from the transmitter site. Although the precise ratio will depend upon the experiment geometry and the transmitter duty cycle, one might conservatively expect that the total amount of incoherent scatter data coming from all remote sites will be equivalent to, or less than, the volume of data coming from the central site. Hence it appears to be tractable to store the entire auto-correlated incoherent scatter data expected from the EISCAT-3D system in an archive of a size which is likely to be affordable at the time of project deployment, while still assuming 24 hour continuous operation. However, production of the autocorrelated data in this form may have significant implications for activities in the signal processing Work Package, WP9.

(12)

In the above tables, no consideration has been given to the interferometric data rate. We do not propose to make a separate provision for interferometric data storage, since this is assumed to use the same archiving resources as indicated for the standard incoherent scatter mode. We assume that users requiring the high volume intermediate products of the interferometry data, such as cross-correlation functions, will provide their own storage or that operational procedures will be altered in the allocation of storage resources if such products need to be stored. The most practical approach would be to limit the provision of permanent EISCAT archive storage for interferometry data to the highest-level derived products, such as images and movies, which are of relatively small volume, as discussed in Section 6. This means that the main WP8 requirement for interferometry is to allow temporary storage of the high bandwidth data from each module pair for subsequent post-processing, a problem equivalent to the short-term storage of decimated sample level data.

9. Computing

hardware

The computing hardware needed for the specification of WP8 can be divided into four main areas:

• Network infrastructure

• Data storage

• Data processing, and

• Visualisation facilities

These hardware elements are discussed in subsequent sections. Although the network infrastructure is the first aspect that the WP8 components present to external interfaces, the specification of the network infrastructure is actually defined by the need to transport data for storage, so we shall commence our review with the data storage component.

9.1 Data

storage

In the context of this discussion, data storage refers to the long-term archiving of data from the EISCAT-3D radar system. The data collected will represent a major financial investment, so their storage should be secure (discussed below) in terms of access, reliability and longevity. Large data stores of this nature are commonplace in the commercial world, although the EISCAT-3D project will be pushing the boundaries of size to a certain degree. However, the most important point is that the provision of such an archive is already possible with commercial data storage products. Three potential candidate archive systems have been identified so far.

One of these is the PetaBox™ system [7], supplied by Capricorn Technologies™. PetaBox claims to offer a “high density, highly reliable, highly scalable, and highly available” data storage. The company claims to deliver the lowest cost per terabyte of any storage platform for that capacity range. It also claims that the system is completely scalable, from individual terabyte nodes to petabyte clusters. A single 19-inch rack can support up to 120 Tbyte of raw disk space. Best quoted energy consumption is 27 watts per terabyte. This is an existing system, which can be purchased off-the-shelf.

The other two candidate storage systems are “future projects”. One of this is project BlackBox™ [8], by Sun Microsystems™. This system is a data store, housed in a standard shipping container. The advantage is that the entire unit can be moved, and that it is possible to simply add additional units. Furthermore, the environmental condition requirements will not be as rigorous as for a standard system. (The storage could, for example, be located in a warehouse, rather than air-conditioned computer building.)

The other system which we have explored is TotalStorage SAN File System™ [9] by IBM™. This is a storage system designed to work over a distributed, heterogeneous environment. It is supposedly scalable, using Storage Area Network (SAN) protocols, and allows data to be distributed, by only maintaining metadata on the central server, with clients accessing the raw data directly from the different distribution sub-servers. The total storage requirement could therefore be built from a series of

(13)

smaller storage sub-systems. The TotalStorage SAN File System is planned to be used as part of CERN’s next major data storage upgrade.

As can be seen from Section 8, the assumed data rates for the EISCAT-3D system place strict requirements on the I/O performance of the selected storage hardware, especially if sample-level data are being streamed directly into the data store. Using currently available technology, the manufacturer’s specifications discuss input data rates limited to around 4Gb/s. The data rate is therefore likely to be a critical factor in determining the way that the archive hardware is structured, imposing a requirement to configure the storage into a number of discrete units dictated by the maximum bandwidth available to each. For a multi-beam radar, it would be logical to imagine a structure based on the number of beam-formers, so that the archive system for a radar capable of 8 beams might be split into 16 discrete units, one for each beam polarisation.

While input data rates of Gb/s would not be required at all times, since it would be impossible to store a long-term data set taken at this rate, it is assumed that there will be some occasions where storage of data taken at such rates will be required, even if only for short intervals – intervals when the coherence threshold for interferometry is exceeded may be an example of this. Likewise, it is assumed that data access from the store will potentially require Gb/s rates, e.g. in order to supply users post-processing the products discussed above, or for the mirroring needed to ensure data robustness (see Section 9.2). Because of this, we have framed a specification test which has been sent to potential data store suppliers, asking them to verify that their data storage systems are capable of handling two simultaneous Gb/s input streams at the same time as two simultaneous Gb/s output streams. We have also suggested that we should participate in such a test to verify that the required specification is being achieved. Documents setting out the required specifications for the central archive and the remote site data stores are included as Appendix 1.

When discussing specific hardware solutions, it is necessary to inject a word of caution. The technology of data storage systems is a fast-changing field, with continuing rapid improvements in storage capacity and I/O speed. It is therefore almost certain that the technology in use today will have been succeeded by more capable systems by the time that the EISCAT-3D system begins to be deployed. The discussion of specific systems above is not so much aimed at identifying the actual system to be used in the final radar, rather to establish whether the required performance is feasible with current devices and to give some indicative costs for the data storage hardware, since it is clear that this will be an expensive component of the EISCAT-3D system.

9.2 Data

security

There are two aspects to data security:

• Ensuring that data are not lost – which will be referred to as “Data robustness”

• Ensuring that data may only be accessed by those permitted to access them – referred to here as “Data permission”

1. Data Robustness

There are two methods to ensure that data is not lost: replication and back-up. Replication is more convenient, but requires more maintenance. There is also the issue of concurrency – that is, keeping the copies synchronised. Machines that are physically close have the advantage of faster, more reliable interconnectivity, but the risk is that both might be damaged by the same catastrophe. Remote mirroring, on the other hand, may be constrained by interconnectivity limitations – the size of the “data pipe” between the two locations. However, modern transfer software (for example bbftp [10], GridFTP, etc.) is able to increase performance with multiple-connections [11][12]. This may also allow us to overcome problems such as the finite transfer limits for individual connections which might exist between the master archive and the chosen mirror sites. This also raises the question of the file system

(14)

structure and file-size quantisation (the current EISCAT system groups data into directories each of which contains one hour of data). Another problem with mirroring is how to ensure that the corruption of one mirror does not propagate to the others, which will require a degree of intelligent software. Given the large size and high data input rate of the data set being considered here, the optimum solution is probably for the mirror to be housed in a physically separated building on the same site as the master archive, so that the two can be connected by proprietary network links. The mirror archive does not necessarily need to employ the same hardware as the master. Although using common hardware and software would make maintenance easier, employing different solutions for the two systems might make the overall archive more resilient to faults. Since the mirror archive is less likely to be accessed for data output it could also include a large component of “off line” or “near-on-line” storage in which much of the data were held on storage media such as robotically accessed tape cartridges, rather than on spinning disks. The ability to transfer data to such media could also facilitate the transport of further data copies to off-site archives, which it might be impractical to update using public networks.

2. Data Permission

Controlling access to the archive is something that will require consideration, but this issue is not being addressed by the current phase of the design study. However, it is appropriate to mention it at this stage, so as to ensure it is kept in mind for future discussion. For example, we will need to consider whether the EISCAT-3D data architecture should implement a full authentication/accreditation system, or instead rely on users observing the so-called “rules of the road”.

9.3 Enhanced-service

9.4 Data formatting and association

In addition to the actual storage of the data, there is a degree of processing which needs to be performed. This processing is at a lower level than that required for functions such as data analysis or satellite-screening (see above). It is primarily, but not solely, the computing power required to perform two closely-related tasks:

(15)

• Data formatting

• Data association

The data formatting is the conversion of the data from raw bit-streams to a format suitable for storage on long-term media and subsequent retrieval. The data association task is that of combining the stream data coming from the primary radar output with the auxiliary data coming from other areas of the system, in particular the data produced by the Control and Monitoring Work Package (WP7), which will be required by the data reduction software.

9.5 Network

solutions

The network requirements of the overall project are generally being addressed under WP12. However, the data rates required by WP8 will be an important input to that work package. As indicated in Section 8, these data rates are extraordinarily high, and this will require a high degree of parallelisation for the various data streams. This is certainly possible, as the individual data streams are independent and may be channelled separately. For this, conventional fibre channel communications may be used. Initial input from WP12 suggests that preferred solutions may involve network protocols such as FPDP-II (Front-Panel-Data-Port) or Infiniband.

10. Other Aspects not included at this stage

For the current deliverable, (D8.2 in the project plan [1]) we have concentrated on specifying the topology of the EISCAT-3D data system, explaining the form that we expect the various data products to take and how these could be accommodated. We have also set out some options for hardware solutions which will make these requirements achievable. In addition to the various aspects of Work Package 8 which have been discussed above, there are a number of important issues which have not been considered in this report. This is either because the project plan calls for these topics to be addressed at a later stage of the design study, because the issues concerned are dependent on the resolution of questions relating to other work packages, or because the topics are not within the scope of the design study. Although we cannot discuss these issues in detail in the present report, it is worthwhile to list them here, and to indicate how they fit into the timeline of future WP8 activities.

10.1 User Interaction with the EISCAT-3D Data System

In addition to specifying the EISCAT-3D data system at the hardware level, it is also necessary to consider the types of interaction which the user might wish to have with the data system. In the original project plan [1] we envisaged a set of “access layer” software, which would provide the interactive interface between the user and the data holdings. Although the actual development of such software is not within the scope of the design study, the specification of the functionality required by the user, and suggestion of how this might be achieved in software is an important function of the design study. Some of these issues have already been alluded to in Section 9.3 above (the “Enhanced Service Archive”) and a more detailed outline of the required functionality will be the subject of the next deliverable (D8.3) due in July 2007.

10.2 Authentication and Data Security

This issue has been briefly discussed in Section 9.2 above, under the heading of “Data Security”. EISCAT data are, in principle, proprietary in the sense that Special Programme data are reserved for the proposing experimental team for the first year after the data are taken, normal Common Programme data are reserved for scientists in the EISCAT member countries, while only data from “World Day” Common Programme operations have unrestricted availability outside the member countries of the EISCAT association. Currently, access to EISCAT data is not handled by an authentication system, but relies on users respecting the “rules of the road”. However, a number of comparable projects have adopted more formal authentication facilities, based (for example) on the exchange of digital security

(16)

certificates. This issue could be considered as an aspect of user interaction, and will be discussed further over the coming months, with outlines of a possible system again being included in D8.3 (July 2007).

10.3 Data

Visualisation

Visualisation of the data (both system data and ionospheric measurements) being produced by the radar is a key aspect of user interaction, since the ability of a scientist or operator to monitor the ionospheric conditions or system performance will be key to controlling radar operations and influencing the choice of experimental modes. This part of Work Package 8 will assess the optimum methods for visualising the EISCAT data at various levels of processing, from sample level to derived data products. This will build on the heritage of EISCAT’s existing visualisation utilities, as well as facilitating the adequate representation of the types of user interaction defined by D8.3 (see above). Documents outlining the design of the visualisation systems for raw and analysed data will be produced as deliverables of WP8 (D8.4 and D8.5) due in February 2008.

10.4 Complementary Data Handling

The handling of “complementary data” is an important issue for the EISCAT-3D project, and can be split into three distinct categories. The first is metadata, which should be understood as data which describe the data themselves. This includes the ability of the data to be self-describing, such that software reading the data can use relatively generic interfaces, such as standard APIs, exploiting the fact that the data file contains descriptors specifying the size and type of the data which it contains. This is a well-known issue within the scientific community and a number of different approaches (e.g. CDF, HDF, XML) exist to address this problem.

The second issue concerns the type, volume and format of the data which describe the state of the radar system. The specification of such data largely falls within the remit of the Control and Monitoring Work Package (WP7), but this is a very significant issue for Work Package 8, because such system data need to be stored in a similar way to the ionospheric data, and need to be formally associated with them, because aspects of system configuration such as transmitter power, number of beams, beam pointing direction and signal processing status will be required by any subsequent data processing algorithms. The third issue regarding complementary data concerns inter-operability with other instruments, either those specifically taking data in conjunction with the EISCAT-3D radar, or those operating independently in a wider geophysical context. A number of geophysically important parameters, e.g. Joule heating rates and particle energy maps, can only be produced by combining EISCAT data with data from other instruments and would add significant value to the scientific data from the EISCAT-3D radar. The data formats and archive systems selected for such instruments therefore need to be considered with this inter-operability in mind, and consideration of all these issues is scheduled for the final phase of the design study, leading up to the last deliverable, D8.6, due in August 2009.

10.5 Post-processing

software

It is worthwhile to note in this section that the design of post-processing software (i.e. analysis programs equivalent to the present GUISDAP program currently used at EISCAT) it is not a specific part of the design study. However, there is a clear link between WP8 and such software. The external interface, and anticipated user-computing power and capacity will place limitations on what we can reasonably expect to export from the archive and these will be important factors in the eventual design of the data analysis software.

(17)

11. Dependences on Other Work Packages

As noted in several places above, the progress of WP8 depends upon the outputs of several other work packages of the design study. In this section, we have collected together these dependences to make clear exactly what inputs are anticipated from the various other work packages of the study.

Work Package 1 (Management of the design study) provides the overall organisational framework within which the project is carried out, consisting of financial and technical oversight, distribution of funds, and operation of the project web site.

Work Package 2 (Evaluation of design performance goals) has already delivered a baseline specification document describing some of the performance goals to be achieved [13].

Work Package 3 (Evaluation of options for the active element) concerns the transmitter hardware for the EISCAT-3D radar, and has no obvious implications for WP8.

Work Package 4 (Phased array receivers) concerns the evaluation of hardware options for the receiver side of the EISCAT-3D system, including the design of the arrays for the central and remote sites. Although it has no direct interaction with WP8, the details of the array design may have implications for the way in which the system will work, such as the number of possible simultaneous beams and the number of sub-arrays to be used in interferometry, which affect the size of the various data products. However, we would expect these implications to manifest themselves to WP8 through our interfaces with WP9 (signal processing) and WP5 (interferometry).

Work Package 5 (Interferometry) has close links to WP8, since it will define the number and size of any interferometry data products and the lengths of time for which these need to be stored. These two work packages have already been interacting very closely, and the design of the data system for interferometry, shown in Figure 1 and discussed in Section 4, reflects detailed discussions between the teams involved.

Work Package 6 (Active Element) is concerned with the detailed design of the transmitter side of the EISCAT-3D system, building on the choices made in WP3. It has no direct interfaces to WP8.

Work Package 7 (Distributed Control and Monitoring) is responsible for developing the control and monitoring system for the EISCAT-3D radar, mainly involving the identification of suitable software systems used elsewhere in the scientific community. We would expect this work package to have an important interface to WP8, and some effect on the data rate, since it will be responsible for establishing the amount and type of system status data to be collated. These data should then be archived by WP8 and the system status data need to be associated with the corresponding ionospheric data, since both will be required by subsequent data reduction software. The first requirement from WP7 is a list of the system status data to be provided and an indication of their size, format and frequency of provision.

Work Package 9 (Signal Processing) is probably the most important work package for WP8, since it will provide the specification of the standard ionospheric data products, such as sample series and/or auto-correlation matrices, together with important scaling factors such as the number of possible beams. We anticipate a close interaction between WP8 and WP9 over the coming months, as we refine the nature and volume of the products which the EISCAT-3D signal processors will deliver to the storage systems of WP8.

Work Package 10 (New Techniques) deals with the investigation of novel uses for incoherent scatter radar data. This package has no immediate interaction with WP8, though might possibly have some implications for the data rates if it resulted in radically new experimental concepts.

Work Package 11 (Implementation Blueprint) is the package under which the final detailed implementation plans will be prepared, using the output from other Work Packages. Although the bulk

(18)

of the effort is concentrated at the end of the project, some earlier work will be needed to develop and iterate the overall system design. This is likely to have implications for WP8, particularly in clarifying its interfaces to other Work Packages.

Work Package 12 (Networking and reference time and frequency) deals with issues of data transport within the EISCAT-3D system, at the level of network hardware and protocols, as well as the distribution of time and frequency standards. This clearly has implications for WP8, as it will determine the format and protocols used for streaming data to the archive systems.

12. Summary and Conclusions

This report has comprised deliverable D8.2 of the EISCAT-3D design study. In this document, we have summarised the status of WP8 at this stage of its development and identified a number of potential hardware solutions for the primary cost component – the data storage facility. At this stage, we have just begun to approach commercial suppliers with an indication of our performance requirements (see documents in Appendix 1), and it is still premature to identify a specific hardware solution. Rather, our aim has been to outline the relevant considerations for the hardware level of the EISCAT-3D data system, and to demonstrate that, even for the large data volumes under consideration, an archive solution is feasible using currently available technology.

References

[1] Annex-1, FP6-2003-Infrastructures-4

[2] DJM, “Interferometry and IS Signal Processing”, v1.1, 2006-12-13.

[3] Fundamentals of radar interferometry I: One baseline Stage 2 report, April 2006

https://e7.eiscat.se/groups/EISCAT_3D_info/WP5_Interferometry_Stage2_Report_April06.pdf [4] D5.1, EISCAT_3D Radar Imaging Arrays Configurations Report, submitted September 27, 2006

https://e7.eiscat.se/groups/EISCAT_3D_info/D5_1_EISCAT_3D_Radar_Imaging_Arrays_Configuration s_Report.pdf

[5] Lind et al., “Advanced techniques for incoherent scatter radar”, Session G3, URSI GA, Maastricht, 2002, http://www.phys.uit.no/~tom/publications/2002/ursi-0943.html

[6] DJM, Data rates analysis, 30-Aug-2006,

https://e7.eiscat.se/Members/dmckay/eiscat3d-pdf/data_rates_pdf/ [7] http://www.capricorn-tech.com/

[8] http://www.sun.com/emrkt/blackbox/

[9] http://www-306.ibm.com/software/tivoli/products/totalstorage-sfs/ [10] BBFTP Homepage, http://doc.in2p3.fr/bbftp/

[11] DJM, “RAL/ESC Network Performance Report #1”, 31-Aug-2006, http://www.eiscat.rl.ac.uk/~derek/badc/esc_report_1.pdf

[12] DJM, “RAL/DL Network Performance Report #1”, 10-Nov-2005, http://www.eiscat.rl.ac.uk/~derek/badc/dl_report_1.pdf

[13] D2.1, EISCAT_3D radar performance specification document, version 1.1 https://e7.eiscat.se/groups/EISCAT_3D_info/P_S_D_7.pdf

(19)

Appendix A: Data Rates Explained

Summary

All scenarios in this summary assume that transmitter sampling is conducted on a single

transmitter beam from the central site which is illuminating the sky for 25% of real time, and

that each of the four remote sites is sampling this transmitter beam with 10 receiver beams.

Raw data – no bandpass filtering (see Option 1)

1 Central Beam + 10 beams per remote site

3.52 GB/s 12.7 TB/hour

304 TB/day 8.5 PB/month 111 PB/year

Auto-correlated data (see Options 2-4)

High-resolution

(see Option 2)

Medium-resolution

(see Option 3)

Low-Resolution (see Option 4) 42.53 MB/s 21.27 MB/s 2.09 MB/s

153.12 GB/hour 76.56 GB/hour 7.52 GB/hour 3.67 TB/day 1.84 TB/day 0.18 TB/day 102.90 TB/month 51.45 TB/month 5.06 TB/month

1340 TB/year 671.12 TB/year 65.96 TB/year

High resolution data option has a temporal resolution of 1s; medium- and low-resolution data

options have a temporal resolution of 2s. High and medium resolution data options have a

range resolution of 300m; low-resolution data option has a range resolution of 3km.

Interferometry (average rate):

Central Site Only 0.64 MB/s 2.30 GB/hour 55.3 GB/day 1.55 TB/month 20.2 TB/year

(20)

Conventional ISR Mode

3. Option 1

This is essentially the option of storing the results coming from the beamformers without any

filtering. This is our worst case scenario since it results in the largest possible data rate,

excluding the possibility of recording data from the individual antenna elements.

The digital data rate from an individual antenna element is:

Receiver bandwidth 30 MHz

Polarisation channels 2

Bits per sample 16 b

Sampling rate 80 MHz

Data rate per element polarisation

1.28 Gb/s 0.16 GB/s

Data rate per element 2.56 Gb/s 0.32 GB/s

Since beam-formed data is created by combining per element data, the data rate of each beam

is the same as that for a single element.

a) Full 30MHz bandwidth beam-formed data without transmitter sampling, 25%

transmitting duty cycle (i.e. receiving for 75% of real-time):

Burst data rate at all sites. We give these in turns of bits and bytes per second, though the

average data rate in a second may be less than this.

1 Beam 5 Beams 10 Beams

2.56 Gb/s 12.8 Gb/s 25.6 Gb/s

0.32 GB/s 1.6 GB/s 3.2 GB/s

1.15 TB/hour 5.76 TB/hour 11.5 TB/hour

27.6 TB/day 138 TB/day 276 TB/day

Average data rate at the central site:

0.24 GB/s 1.2 GB/s 2.4 GB/s

Average data rate at the remote sites (sampling for 25% of real-time):

Average data rate at a single site:

0.08 GB/s 0.4 GB/s 0.8 GB/s

6.91 TB/day 34.6 TB/day 69.1 TB/day

Total average data rate at all (4) remote sites:

0.32 GB/s 1.6 GB/s 3.2 GB/s

(21)

Total average data rate for central site and remotes sites:

0.56 GB/s 2.6 GB/s 5.6 GB/s

297 TB/day

One hour of such operation, using 1 central beam and 10 beams at each remote site, would use

6% of the annual budgeted archive capacity of 200TB.

b) Full 30MHz bandwidth beam-formed data with transmitter sampling, 25%

transmitting duty cycle:

Average data rate for central site is the same as the burst data rate option 1(a), total average

data rate for remote sites remains unchanged from option 1(a).

Total average data rate for central site and remotes sites:

0.64 GB/s 3.2 GB/s 6.4 GB/s

304 TB/day

One hour of such operation, using 1 central beam and 10 beams at each remote site, would use

6% of the annual budgeted archive capacity of 200TB.

(22)

4. Option 2

Auto-correlated data as used in the current EISCAT system, with higher temporal and spatial

than current standard operations.

Required minimum range 50 km Required maximum range 1500 km Required range resolution 300 m

Range increments 4833

profile

4.83E+05 data Data storage per datum 4 bytes Number of Polarisations 2

Data per correlated matrix 3.87 MB Post-correlation integration 1 s Total data rate per beam 3.87 MB/s

a) Auto-correlated 1s-integrated beam-formed data without transmitter sampling, 25%

duty cycle:

Burst data rate at all sites:

30.93 Mb/s 154.67 Mb/s 30.93 Mb/s

3.87 MB/s 19.33 MB/s 38.67 MB/s

13.92 GB/hour 69.60 GB/hour 139.20 GB/hour 334.08 GB/day 1670.40 GB/day 3340.80 GB/day

9.35 TB/month 46.77 TB/month 93.54 TB/month 122.02 TB/year 610.11 TB/year 1220.23 TB/year

Average data rate at central site:

2.90 MB/s 14.50 MB/s 29.00 MB/s 10.44 GB/hour 52.20 GB/hour 104.40 GB/hour 250.56 GB/day 1252.80 GB/day 2505.60 GB/day

Average data rate at a single remote site:

0.97 MB/s 4.83 MB/s 9.67 MB/s

Total average data rate for all (4) remote sites:

(23)

Total average data rate for central and remote sites:

6.77 MB/s 33.83 MB/s 67.67 MB/s

41.57 MB/s 149.64 GB/hour

3.59 TB/day 100.56 TB/month

1.31 PB/year

One month of such operation, using 1 central beam and 10 beams at each remote site, would

use 50% of the annual budgeted archive capacity of 200TB.

b) Auto-correlated 1s-integrated beam-formed data with transmitter sampling, 25% duty

cycle:

Average data rate for central site is the same as the burst data rate option 2(a), total average

data rate for remote sites remains unchanged from option 2(a).

Total average data rate for central site and remotes sites:

7.73 MB/s 38.67 MB/s 77.33 MB/s

27.84 GB/hour 139.20 GB/hour 278.40 GB/hour 668.16 GB/day 3340.80 GB/day 6681.60 GB/day 18.71 TB/month 93.54 TB/month 187.08 TB/month 244.05 TB/year 1220.23 TB/year 2440.45 TB/year

42.53 MB/s 153.12 GB/hour

3.67 TB/day 102.90 TB/month

1.34 PB/year

One month of such operation, using 1 central beam and 10 beams at each remote site, would

use 51.5% of the annual budgeted archive capacity of 200TB.

(24)

5. Option 3

Auto-correlated data as used in the current EISCAT system, with higher temporal and spatial

than current standard operations.

Required minimum range 50 km Required maximum range 1500 km Required range resolution 300 m

Range increments 4833

profile

4.83E+05 data Data storage per datum 4 bytes Number of Polarisations 2

Data per correlated matrix 3.87 MB Post-correlation integration 2 s Total data rate per beam 1.93 MB/s

a) Auto-correlated 2s-integrated beam-formed data without transmitter sampling, 25%

duty cycle:

Burst data rate at central site:

15.2 Mb/s 121.6 Mb/s 152 Mb/s

1.93 MB/s 9.67 MB/s 19.33 MB/s

Average data rate at central site:

1.45 MB/s 7.25 MB/s 14.50 MB/s

Average data rate at a single remote site:

0.48 MB/s 2.42 MB/s 4.83 MB/s