Home > Data Products

Data Products

Direct Links

Data Access Policy [27KB PDF]

Daily Procedures


Raw data from all instruments are copied back to the ARI in Liverpool within minutes of being taken, through the night. Rough "quicklook" reductions are made using whatever calibration files (flats, fringe frames, etc) are available at that instant, so that users can perform sanity checks on their observations if they wish. The following morning, where a reduction pipeline is avilable, the data is automatically fully-reduced using the very latest calibration files, and loaded into the searchable Data Archive.

Data from all instruments (reduced if there is an automated pipeline and raw if there is not) are normally available for download that morning. An email is automatically sent to the PI notifying them of the new data (if you would also like an alert email sent to a Co-I please contact the LT Support Astronomer).

Where Data are Stored Online


There are three web-based interfaces to allow PIs and CoIs access to their data; Quicklook, Recent Data, and the Archive. The main distinguishing feature between these interfaces is the time-scale on which the data are made available. All of these data sources are password protected for as long as the data are still within their proprietary period, after which they become publicly available.

Available Until Data Processing Comments
Quicklook In real time, i.e. ~5 mins after observation Up to one week Partial Useful for monitoring in real time (or the following day) the quality of data acquired and which targets have been observed. Also provides access to time critical data such as GRBs.
Recent Next working day Up to two weeks Full A single place where users can look to see what data have been obtained for them in the past fortnight.
Archive Next working day Unlimited Full The primary data repository. Public and proprietary data.

In addition to the three main user interfaces discussed above, a small amount of data which for one reason or another are not available in the main searchable data archive, may be found in Unloaded Data. In most cases these are early comissioning data for particuilar instruments obtained when a reduction pipeline was not yet available or do not meet the quality control requirements for ingestion to the main archive. Most of the data included here are unreduced raw files direct from the instrument. Also included here are some very early (2004) LT data obtained before the robotic systems were comissioned. Most LT users will never want access to any of these files.

IN SUMMARY - the three main repositories for LT data are Quicklook, Recent Data, and the main LT Archive. For the majority of users, the most commonly used interface is Recent Data, though all three are listed below.

  • Quicklook

    • Reduced data made available almost immediately during the night.
    • In some cases observations may only be partially reduced (i.e. the pipeline may not always use the most up-to-date calibration files or may use reduction algorithms optimised for speed rather than accuracy). However, data products should certainly be good enough to check the overall quality of any data acquired.
    • For each night you will find:
      • JPEG images and reduced FITS files for all targets observed on that date.
      • An observing log.
    • Partially reduced FITS file names end in "_9.fits".
    • All proprietary data are Password protected.

  • Recent Data

    • The recent data archive contains exactly the same data as are loaded into the main searchable LT data archive. Data are simply held here for a couple of weeks for more convenient download. (Note that data uploaded to this Recent Data archive are also uploaded to the main LT Searchable Archive, described below, at the same time).
    • For new instruments without a working data reduction pipeline, the raw data may be held online here indefinately or until they can be reduced and loaded into the main data archive.
    • Data are stored in subdirectories named by the local civil date at the start of that night. A full night's data are stored together in the same subdirectory, i.e. the system does not create a new directory at midnight.
    • For each night you will find
      • JPEG images and reduced FITS files for all targets observed on that date.
      • A gzipped tar file containing all reduced data bundled together for easy download.
      • All raw data files (usually also contained in a gzipped tar file).
      • An observing log.
    • Reduced FITS file names end in "_1.fits" if they have undergone only basic instrumental reductions (bias subtraction, flat-fielding, etc.) or "_2.fits" if they include derived results such as object catalogues, extracted 1D spectra etc; unreduced FITS file names end in "_0.fits". Please see the individual instrument web pages for further details.
    • All proprietary data are Password protected.

  • Searchable Data Archive

    • This is the main repository for long term storage of ALL reduced LT data. There is currently no time limit on storage.
    • Contains all reduced LT data taken since September 2004. If you need to recover data obtained for you before that date, please look at the Unloaded Data or contact us.
    • Only contains those data which have been through the automated reduction pipelines. For instruments where no pipeline is currently in operation, alternative temporary measures may be in place to provide the data to users (see Recent Data above).
    • Reduced FITS file names end in "_1.fits" if they have undergone only basic instrumental reductions (bias subtraction, flat-fielding, etc.) or "_2.fits" if they include derived results such as object catalogues, extracted 1D spectra etc; unreduced FITS file names end in "_0.fits". Please see the individual instrument web pages for further details.
    • All proprietary data are Password protected.

up to top of page

Observating Logs & Error Codes


Observing logs are available on a per proposal basis so users have access to a full list of all observations taken on a particular night for a single proposal. These may be found in Recent Data.

If there have been any problems during the passage of data through the Data Pipeline, error codes are added to the observing log, and more detailed codes are written to custom FITS headers in the FITS data file (see below). These are used to alert users to more specific flags in the FITS header. The observing log codes are cumulative and are found in the log at the end of the line for each image. For example, if a problem is encountered in both overscan subtraction and bias frame subtraction, the flag would be set to -6. (i.e., -2 + -4 )


OBSERVING LOG CODES FOR THE CCD PIPELINE
Code #Error String
-2Overscan subtraction
-4Bias frame subtraction
-8Trimming off the overscan
-16Flat fielding
-32Dark frame subtraction
-64Fringe subtraction

up to top of page

Downloading Data & Proprietary Period


Data are available for immediate download over HTTP. Consult the Where Data are Stored section above to determine the access method best suited to you. Recent Data uses a static directory structure so that users always know where to look for their data. The Data Archive will automatically create hypertext links to all data that match submitted search criteria.

To access FITS data products, users will need the password that was provided to the PI in the notification email when his or her data were first loaded into the archive. The password is specific to the proposal ID (not the user ID) and any future or past data obtained for the proposal should be accessible using the same password. PIs may therefore want to circulate the proposal password with their CoIs so that they may access the data.

Data usually remain proprietary to the PI for one year after the end of the semester within which the data were acquired. Formal rules are defined by the LT's data access policy.

up to top of page

Calibrators and Standards


All standard calibration data (Photometric, polarimetric standards etc) taken with the LT are public and available for use by all observers.

As with all other data, once reduced these data can be accessed directly from the Searchable Data Archive or, in the short term, the Recent Data pages. Each set of calibrators has its own specific proposal ID. The table below lists the proposal ID users should interogate for each calibration standard.

See the instrument specific pages for details of what standards are observed, zero-points, colour transformations etc.


PROPOSAL IDs FOR INSTRUMENTAL CALIBRATION DATA
Proposal ID Instrument and Data Type
IOOStand IOO. Visible wavelength photometric standards
IOOFringe IOO. Very rarely updated because IO:O fringing is so weak de-fringing is not needed.
FrodoStand FRODOspec. We do not obtain any routine spectrophotometric standards.
FrodoFlat FRODOspec. Nightly tungsten lamps used for tracing fibre apertures.
RINGOStand
(Formerly "RINGO_STAND")		 RINGO3. Polarimetric standards. Both polarised and zero polarisation stars. The same proposal was previously used for standards observed with RINGO and RINGO2.
RATStand
(Formerly "RATCAM_STANDARDS")		 RATCam. Visible wavelength photometric standards
RATFringe
(Formerly "RATCAM_FRINGE")		 RATCam. Deep i' and z' band blank fields for creation of fringe frames
SupIRStand
(Formerly "SUPIRCAM_STANDARDS")		 SupIRCam. Near IR wavelength photometric standards. Instrument decommissioned.

up to top of page

Feedback Form


We welcome your thoughts and suggestions regarding your LT data and overall experience as a user. Please feel free to complete this Feedback Form at any time, or to contact either LT Staff or, if you prefer, a member of the LT User Group.