How to convert telescope-provided files to the standardized .s format#
This tutorial demonstrates how to convert files provided by the various telescopes into the standard ‘.s’ format used in SpecpolFlow.
The ‘.s’ format is essentially a text format with columns for wavelength, intensity, polarization, two polarimetric nulls, and uncertainties. This follow the format from LibreESPRIT.
In more detail, the ‘.s’ format has two lines of header, which are optional for SpecpolFlow. The first header line is a comment, the second has the number of pixels in the spectrum and the number of columns of spectrum (not counting the wavelength column). The rest of the file contains columns of wavelength (usually nm or A), Stokes I (continuum normalized or unnormalized), polarization (continuum normalized or unnormalized), null1, null2, and uncertainty.
The Stokes I, polarization, null1, and null2 data should all have the same normalization. That way the uncertainty column applies to all four spectrum data columns. If Stokes I is continuum normalized, the polarization should also be normalized by the continuum flux, not the full Stokes I. E.g. I/Ic, V/Ic, N1/Ic, and N2/Ic (not V/I and N/I!). Or if Stokes I is unnormalized, the polarization spectrum should also not be unnormalized (not even by Stokes I). E.g. I, V, N1, and N2 (again not V/I and N/I!).
The wavelengths will jump backwards in several places, since data from different spectral orders are concatenated together in the file. Within one order wavelengths will increase. Places where the wavelength decreases indicates the end of one order and the start of another. In places where the orders don’t overlap, order edges will show up as gaps in wavelength.
Note that ‘.s’ files from Narval (and the ESPaDOnS ‘.s’ files from Polarbase) are already in the correct format, and so do not need to be converted.
First, import SpecpolFlow:
Show code cell content
## Import necessary packages:
import specpolFlow as pol
ESPaDOnS files from UPENA#
For spectropolarimetric observations, UPENA provides a [name]p.fits file that contains columns for 4 versions of the spectrum:
Automatically normalized with radial velocity correction from the telluric lines
Automatically normalized without the radial velocity correction from the telluric lines
Unnormalized with radial velocity correction from the telluric lines
Unnormalized without the radial velocity correction from the telluric lines
When a more precise normalization is required, users may prefer to use the unnormalized spectra in order to perform the normalization with tools like normPlot. A tutorial for this is here.
However, the radial velocity correction from the telluric lines performed by UPENA on unnormalized files has often been found to be erroneous (probably because the code has been optimized to work with normalized spectra). The radial velocity correction from the telluric lines performed on normalized files is reliable.
Therefore, our ESPaDOnS converter will generate two ‘.s’ files:
n.s: The UPENA normalized spectrum, with automated radial velocity corrections from the telluric lines.u.s: The UPENA unnormalized spectrum using the velocity correction from the normalized spectrum. This is done starting from the unnormalized spectrum without the automated radial velocity correction, to which we have applied the radial velocity correction determined from the normalized spectrum..out: the content of the FITS header in a text format.
The flistout keyword is optional: if not provided (or None), then the .s files are saved in the same directory as the original .fits file. flistout can be set to the first part of a filename, possibly including directories, like path/root_filename_, then u.s n.s and .out is added at the end. So the files would be saved at that path with names like root_filename_n.s and root_filename_u.s. Finally, a list of input files can be provided, along with a list of output root filenames, to process several files at once.
In the example below, we convert the .fits file normalization_tutorialfiles/2378196p.fits to the same directory.
# using default output names
pol.converters.espadons(['ConvertToSFiles_tutorialfiles/2378196p.fits'])
# or specifying the output file name
pol.converters.espadons(['ConvertToSFiles_tutorialfiles/2378196p.fits'],
flistout=['ConvertToSFiles_tutorialfiles/star_name_'])
converting ConvertToSFiles_tutorialfiles/2378196p.fits
converting ConvertToSFiles_tutorialfiles/2378196p.fits
SPIRou files from APERO#
For spectropolarimetric observations with SPIRou, the APERO data reduction pipeline generates [name]p.fits containing the intensity and polarization spectra. For regular intensity spectroscopic observations, and each exposure in a polarimetric sequence (typically 4 exposures), the pipeline also generates [name]e.fits intensity spectra and [name]t.fits telluric corrected intensity spectra. The main function in SpecpolFlow for converting these to ‘.s’ files is converters.spirou.
The ‘.fits’ files from APERO contain nan values for pixels where the pipeline could not produce reliable values. Usually these are places where either the spectrum extraction failed (e.g. due to low flux at order edges), or the telluric correction failed (e.g. due to very strong telluric lines). The converters in SpecpolFlow provide options to: replace the nan values with 0 (and set the uncertainty to 100); remove the pixels with nan, and also remove small fragments of spectrum in regions with a lot of nans; or to just keep the nan values. Some details are in the API. Analyzing spectra containing nan values with SpecpolFlow will often lead to errors, so it is usually better to remove or replace those values.
The polarization spectrum in the p.fits files is normalized by the total Stokes I flux, not the continuum flux (e.g. V/I, not V/Ic). The SpecpolFlow tools, and the typical .s format, use either continuum normalized polarization and Stokes I (e.g. I/Ic and V/Ic), or both unnormalized polarization and Stokes I (e.g. just I and V, not V/I). The SpecpolFlow converter for SPIRou outputs the polarization and Stokes I spectra both normalized by the continuum, using the continuum normalization from APERO.
The converter generates both the ‘.s’ spectrum and a text file containing information from the .fits header, as [name].out. Generating this extra text file can be disabled using the saveFitsHeader=False option.
This first example converts a polarized p.fits spectrum (2305251p.fits) to a ‘.s’ file.
# the output name can be explicitly specified, in this case 2305251p.s
spec = pol.converters.spirou('ConvertToSFiles_tutorialfiles/2305251p.fits',
'ConvertToSFiles_tutorialfiles/2305251p.s',
ftype='p', nanTreatment='remove')
# or the output name can be generated from the input name,
# the converter will replace .fits with .s, so this will also generate 2305251p.s
spec = pol.converters.spirou('ConvertToSFiles_tutorialfiles/2305251p.fits',
ftype='p', nanTreatment='remove')
converting ConvertToSFiles_tutorialfiles/2305251p.fits to ConvertToSFiles_tutorialfiles/2305251p.s
converting ConvertToSFiles_tutorialfiles/2305251p.fits to ConvertToSFiles_tutorialfiles/2305251p.s
The intensity spectra from individual exposures in the ‘e.fits’ and ‘t.fits’ files are in the observer’s rest frame. The converter shifts them to the solar system barycentric rest frame, using the BERV calculated by APERO. The ‘p.fits’ files are already in the barycentric rest frame. The ‘e.fits’ and ‘t.fits’ files do not have formal uncertainties, so the uncertainties for each pixel are estimated from the square root of the flux. The flux in the ‘e.fits’ and ‘t.fits’ files is not blaze corrected, so the converter also applies a blaze correction. The spectra output for these files are not continuum normalized.
This example converts an e.fits spectrum (2305251e.fits) to a ‘.s’ file.
spec = pol.converters.spirou('ConvertToSFiles_tutorialfiles/2305251e.fits',
'ConvertToSFiles_tutorialfiles/2305251e.s',
ftype='e', nanTreatment='remove')
converting ConvertToSFiles_tutorialfiles/2305251e.fits to ConvertToSFiles_tutorialfiles/2305251e.s
applying barycentric radial velocity correction 19.2483 km/s, Barycentric Julian date 2458384.169601
The ‘t.fits’ files contain both the telluric corrected intensity spectrum, and an estimate of the telluric spectrum, which was removed from the original spectrum to generate the telluric corrected spectrum. The telluric spectrum can be a useful reference, for checking how strong telluric contamination was for a stellar line or region of interest.
The converter saves the corrected intensity spectrum and the telluric spectrum to two separate files.
This example converts a t.fits spectrum (2305251t.fits) to a ‘.s’ file.
# this will generate 2305251t.s and 2305251t.s.telluric
spec = pol.converters.spirou('ConvertToSFiles_tutorialfiles/2305251t.fits',
'ConvertToSFiles_tutorialfiles/2305251t.s',
ftype='t', nanTreatment='remove')
converting ConvertToSFiles_tutorialfiles/2305251t.fits to ConvertToSFiles_tutorialfiles/2305251t.s and ConvertToSFiles_tutorialfiles/2305251t.s.telluric
applying barycentric radial velocity correction 19.2483 km/s, Barycentric Julian date 2458384.169601
The main SPIRou converter can infer the type of file to convert if the input file name ends in p.fits e.fits or t.fits. Otherwise you can explicity specify the type of file with ftype='p', 'e' or 't'. There are also separate converter functions for the different file types: spirou_p, spirou_e, and spirou_t.
The main SPIRou converter can also process a list of files (the functions for individual file types only work with one in file at a time).
Here we combine both inferring the type of files and a processing a list of files, to re-process the previous examples.
# this will generate 2378196p.s, 2378196e.s, 2378196t.s and 2378196t.s.telluric
flist = ['ConvertToSFiles_tutorialfiles/2305251p.fits',
'ConvertToSFiles_tutorialfiles/2305251e.fits',
'ConvertToSFiles_tutorialfiles/2305251t.fits']
specList = pol.converters.spirou(flist, nanTreatment='remove')
converting ConvertToSFiles_tutorialfiles/2305251p.fits to ConvertToSFiles_tutorialfiles/2305251p.s
converting ConvertToSFiles_tutorialfiles/2305251e.fits to ConvertToSFiles_tutorialfiles/2305251e.s
applying barycentric radial velocity correction 19.2483 km/s, Barycentric Julian date 2458384.169601
converting ConvertToSFiles_tutorialfiles/2305251t.fits to ConvertToSFiles_tutorialfiles/2305251t.s and ConvertToSFiles_tutorialfiles/2305251t.s.telluric
applying barycentric radial velocity correction 19.2483 km/s, Barycentric Julian date 2458384.169601
Tip
It’s possible to use nanTreatment='replace' for parts of the analysis that care about identifying spectral orders (e.g. the continuum normalization). Then remove those pixels for parts of the analysis that don’t care about order edges and may be confused by points with a value of 0 (e.g. the LSD calculation).
For example:
spec = pol.converters.spirou('2305251p.fits', nanTreatment='replace')
# normalize the spectrum (with 0 in place of nan)
import normPlot
normPlot.normplot('2305251p.s')
# load the normalized spectrum, and only use pixels where the flux is above 0
# (the nan pixels were set to 0, and should still be 0 after normalizing)
# also only use pixels with reasonable errors
# (the nan pixels should still have extremely large errors after normalizing)
spec_norm = pol.read_spectrum('2305251p.s.norm')
spec_norm = spec_norm[(spec_norm.specI > 0.0) & (spec_norm.specSig < 0.1)]
spec_norm.save('2305251p-norm-trim.s')
# run the LSD code
pol.run_lsdpy('2305251p-norm-trim.s', 'mask-spirou.dat',
outLSDName='prof-2305251p.lsd', velPixel=2.3,
normDepth=0.2, normLande=1.2, normWave=1500.0)