Are clouds really grey ?

[Robin Leadbeater]

Yes this is a serious question

The AAVSO have just opened their spectroscopic database and published a manual for amateurs doing spectroscopy
https://www.aavso.org/apps/specdb/help

There is a lot wrong with this manual and their Fits header specification but a particulalry important issue is flux calibration

There they say that flux calibration (they do not distinguish between calibration in relative and absolute flux) can only be performed in photometric sky conditions and they recommend amateurs should normalise (rectify) their spectra before submitting them to their new database. This is of course different to the many thousands of amateur spectra already in the ARAS, BAA,BeSS databases.

It is true that accurate absolute flux calibration of spectra (in physical units) needs photometric conditions and a photometric slit (unless scaling using an indpendent magnitude measurement) but is this really true for relative flux calibration ? (ie the method normally used by amateurs, correcting for instrument response and atmospheric extinction using a reference star.)

I found some measurements of a reference star which I made under variable cloud conditions. The flux in the individual exposures varies by a factor of over 3x but after rescaling the spectra agree to better than +-10% so it would seem calibrating in relative flux is possible to reasonable accuracy even in extremely poor atmospheric conditions. (see the attached plots)

Robin

hd214994_20180905_073_Leadbeater.png (95.84 KiB) Viewed 5715 times

hd214994_20180905_073_Leadbeater_scaled.png (62.53 KiB) Viewed 5715 times

[HughAllen]

Thanks Robin, it's very interesting to learn of the new AAVSO spectroscopic database. Regarding the preference to submit rectified (normalised) spectra I am surprised by that. It can be very difficult reliably to draw the continuum especially for cooler stars with complex spectra. And what about the correction of emission line intensities?
Hugh

[Robin Leadbeater]

Hi Hugh,

I agree. While it might be useful to normalise spectra where the flux calibration is suspect or not possible. In my experience it is better for the user of the spectra to do it if necessary so that it is done consistently.

There are several other issues.

For example the reasons for flat correction and the recommended flat correction procedure are questionable and statements like
" ISIS does not have a straight forward way of performing this (Heliocentric) correction (if you could calculate it independently you could simply shift the
spectrum, but this calculation is non-trivial)"
shows that the author does not really understand the capabilities of ISIS (Provided the object is in SIMBAD this is just a matter of ticking a box in ISIS and the precision of the correction has been demonstrated to very high level by Christian's Exoplanet measurements)

The database Fits header requirements also fall below the BeSS standard and depend on the software writers to "do the right thing" and add the essential information. For example there is no way of knowing if the spectrum is in physical or relative units, has been normalised and if so how, has been heliocentric corrected, has had tellurics removed, has had cosmetic defects removed. All which are part of the BeSS standard

I get the feeling that the AAVSO and the main author of the document (a professional astronomer) are not really aware of the progress made and the work already done by amateurs in this area. AAVSO have been trying to get this off the ground for ten years now. I have been involved in it from time to time but they appeared not to be interested in what had already been achieved and preferred to go their own way and re-invent the wheel rather than driving the development of the field further.

Robin

[Ken Hudson]

Hi Robin,

You raise many issues in your original post and in your reply to Hugh. I have a few comments:

1. It seems to me that the author of the manual is referring to absolute flux calibration when he says that flux calibration requires photometric conditions. Here's the relevant quote from the manual:

"For some stars, high quality reference spectra exist that have been carefully corrected for instrumental response. Dividing your spectrum by the reference spectrum (for the same star) will allow determination of your instrumental response, which can then be used to produce spectra on an absolute rather than relative flux scale. These calibration spectra must be taken under photometric (completely cloud free) conditions."

Admittedly, it may be a bit ambiguous, but this section can be reworded to avoid any confusion in the future. The manual should also address instrumental response correction.

There is no stated requirement in the manual that the AAVSO Spectroscopy Database requires absolute flux calibration. This was just background information.

2. It appears that you and the manual are in agreement that absolute flux calibration needs photometric conditions.

3. Yes, ISIS does support Heliocentric correction. I think it is important to keep in mind this is the first edition of the AAVSO Spectroscopy Manual and clarifications/modifications will need to be made periodically so it is accurate and useful. I will take some of the blame here - I did proofread the manual and I missed that mistake.

4. With regard to FITS headers, I asked John Weaver (technical lead on the AAVSO Spectroscopy Database project) about the issues you raised. He said, "We realise that there are considerable differences between not only the exact procedures available between software, but also the documentation of these procedures within the FITS headers. Not only do we wish to avoid having users manually edit their headers to specify which procedures have been carried out, but we also realise that leveraging our extensive community to report potential issues is more effective than simply documenting a procedure in the header, as that does not guarantee that a procedure was carried out. We do, of course, encourage documentation within headers. In addition, we provide users with feedback in order to help them submit a calibration spectrum of a non-variable star before they submit real variable star spectra for the database."

Hugh raised a good point when he said, "It can be very difficult reliably to draw the continuum especially for cooler stars with complex spectra. And what about the correction of emission line intensities?" We should have caught that.

Finally, let me just say that the AAVSO Spectroscopy Database was developed in close cooperation with the BeSS team. Please check Stella Kafka's post in the astronomical_spectroscopy Yahoo Group for further information and a complete and thorough description of the AAVSO Spectroscopy Database.

Thank you for helping us identify some issues and opportunities for improvement.

Cheers,

Ken

[Andrew Smith]

Robin, according to the met office as the water droplets in cloud are much larger than the wavelength of visible light they scatter all visible wavelengths equally. Thus clouds are gray to first order.

Regards Andrew

[Robin Leadbeater]

Hi Ken,

1. It seems to me that the author of the manual is referring to absolute flux calibration when he says that flux calibration requires photometric conditions. Here's the relevant quote from the manual:

"For some stars, high quality reference spectra exist that have been carefully corrected for instrumental response. Dividing your spectrum by the reference spectrum (for the same star) will allow determination of your instrumental response, which can then be used to produce spectra on an absolute rather than relative flux scale. These calibration spectra must be taken under photometric (completely cloud free) conditions."

Yes it is possibly one of terminology but you would need to check with your lead author. Professionals often talk about flux calibration even in relative flux where we talk about IR correction. (This is normally includes extinction the way we commonly do it with a single ref star at similar air mass. Pros usually do things differently, separating out instrument response and atmospheric effects.) For example it says in the manual

"Flats should not be used to correct for the total instrument response (which depends on the CCD
sensitivity as a function of wavelength, the grating, etc.), because that assumes a uniform flux
across all wavelengths. That correction, called flux calibration, requires a different procedure to
be described later. It is not absolutely necessary to do a flux calibration, except for a few specific
kinds of science objects."

Which implies flux calibration, relative or absolute is not something routinely done.

The difference between normalising (ie rectifying spectra to make the continuum 1 everywhere), relative flux calibration (where the spectrum is corrected for instrument response and extinction and scaled to 1 at a point in the continuum) and absolute flux calibration (where the spectrum is calibrated in physical units) needs to be clarified somewhere.

Do you know how the recommendation to rectify (normalise to the continuum) spectra before submission came about? The guidelines for submission of the test spectrum clearly state that it should be normalised. In discussions I have had with some professionals in the past they had been of the view that amateurs would not be able to routinely produce (relative) flux calibrated spectra with useful accuracy. This proved not to be the case however

It might also be useful to mention the technique of absolute flux calibration using coincident photometric brightness measurements developed here

Cheers
Robin

[Robin Leadbeater]

Hi again Ken,

4. With regard to FITS headers, I asked John Weaver (technical lead on the AAVSO Spectroscopy Database project) about the issues you raised. He said, "We realise that there are considerable differences between not only the exact procedures available between software, but also the documentation of these procedures within the FITS headers. Not only do we wish to avoid having users manually edit their headers to specify which procedures have been carried out, but we also realise that leveraging our extensive community to report potential issues is more effective than simply documenting a procedure in the header, as that does not guarantee that a procedure was carried out. We do, of course, encourage documentation within headers. In addition, we provide users with feedback in order to help them submit a calibration spectrum of a non-variable star before they submit real variable star spectra for the database."

I think it is important to consider this from the perspective of the user of the spectra which might be sometime in the future. (This after all is the prime purpose of the database). If it is not specifically stated that all spectra in the database are rectified for example then this information is needed either in the header information or in meta data included with the spectrum when it is submitted. (which could then be inserted into the header as the BeSS database does). For a parallel example look at the kind of information included with brightness measurements in the AAVSO photometry database. Fortunately the solution is an easy one. Here we already have the BeSS standard which has already addressed these issues and is already used by popular amateur software (so that users can submit spectra to the existing BAA and BeSS databases) so why not apply this standard in its entirety to spectra submitted to AAVSO ?

Cheers
Robin