Use of CMOS detector for faint object spectrography

Design, construction, tuning of spectroscopes
Information and discussion about softwares (telescope remote, autoguiding, acquisition, spectral processing ...)
Christian Buil
Posts: 1431
Joined: Mon Sep 26, 2011 6:59 pm
Contact:

Re: Use of CMOS detector for faint object spectrography

Post by Christian Buil »

Robin, ATIK announces for its camera a noise of 0.6 electron. It appears that the detector is the same one of the the ASI1600 (Panasonic origin) for which the noise is 1.2 electron (at 20dB). By what mystery ATIK does better? Strange because this noise seems intrinsic to the sensor alone - we will see on production models...

Christian
Thilo Bauer
Posts: 37
Joined: Sun May 31, 2015 4:45 pm
Contact:

Re: Use of CMOS detector for faint object spectrography

Post by Thilo Bauer »

Just for amusement:

Do you know, what is really wrong with CMOS detectors and spectrographs?

Ok. Send out mail for order of new calibration lamps to France and also call for proposal to two resellers here in Germany for the ASI camera on the same day I ordered a new A/C compressor and fan clutch for my Lincoln at a seller in USA. My old Lincoln took the road to OHP and backwards, then the A/C compressor slowly desired to quit as my fan clutch did in a traffic jam on the way back near Lyon.

Know what? The spare parts from USA already arrived here in Germany, while waiting for the rest to reply. :D

Have a nice evening!

Thilo
Peter Somogyi
Posts: 420
Joined: Sun Jul 13, 2014 8:56 am

Re: Use of CMOS detector for faint object spectrography

Post by Peter Somogyi »

As for the ATIK link Robin posted before ( https://www.atik-cameras.com/news/diffe ... s-sensors/ ), I'm also interested in the statement of 'Variations in linearity and sensitivity between pixels' for CMOS.

At least if I'd get to a CMOS (again...), the very first I'd test was the linearity curve.
With variable gains (I do have an ASI for guiding and viewfinder), gain setting easily produce a nonlinear exposure-ADU curve.
Now the article suggests to me doing it per pixel...

The 2nd thing I'd test, is whether any filtration happens (e.g. shoot special shapes with a short focal length optics).

As for DSLRs, for me even the dark exposure time had to match with the light exposure (dark non-reproduceability). That's also something to be tested.
And, did hear that bias frames also change sometimes, however bias is not usable for DSLRs for the above reason. Question if it comes from the chip or camera.

So the following tests I'd put on the queue:
- linearity curves (at different chip locations!) and at different gains (= 3D curve for many pixels?)
- special shapes (any filtration by chip, like low pass?)
- can a dark can be reused for different exposures?
- and - of course - ripples

Remember, linearity is the key in spectro to have reliable measurement, even if by human eye everything looks fine or when get a visually looking meaningful spectrum.

- Peter

EDIT: also interesting ATIK mentions 'Pixel to pixel reproducibility' for CCDs. Quesion what it is (repeat of the linearity statement?)
Thilo Bauer
Posts: 37
Joined: Sun May 31, 2015 4:45 pm
Contact:

Re: Use of CMOS detector for faint object spectrography

Post by Thilo Bauer »

Hi Peter,
Peter Somogyi wrote:As for the ATIK link Robin posted before ( https://www.atik-cameras.com/news/diffe ... s-sensors/ ), I'm also interested in the statement of 'Variations in linearity and sensitivity between pixels' for CMOS.
Speaking for DSLR:

In general check for non-linearity once to clarify how the sensor behaves. Typical non-linearities are found in the highest intensities close to saturation. Here it is important. Within a certain range of intensities below this, you may ignore correction for non-linearity.

Check for non-linearities using dark calibrated flatfields of different illumination and perform noise analysis. Measured intensities and standard deviation shall follow a square-root relationship. If not, non-linearty is found and can be corrected. See Berry & Burnell (2005) for details about testing procedures and proper computation of noise properties. While intensities and noise shall follow a square-root relation, the major problem with this method is, that It is also sensitive for shutters not being linear. So trying to find a relation between shutter exposure time and noise will certainly yield a non-linear relation. Often this is not a problem of the sensors, but the mechanics of shutter. Use of a calibration curve from shutter non-linearities is a wrong decision, however. Use calibrated gray filters in the optical path or take flatfields with constant exposure, but at different illumination during twilight. I would recommend to analyze just intensities found and relate this to noise from the pixel intensities. Do NOT put this into relation with exposure time, shutter speed, or gain settings, but use calibration consistent with the settings of the camera. A dark frame for ISO 400 will not match an image of ISO 1600 and even temperature variation over night might be critical. A good decision will also be to keep maximum intensities below 50-60% of the saturation level. This is also safe for intensity variations from atmospheric image blur within imaging & spectroscopy.

Another test will be photometric observations of a field of stars from a well known reference. I did photometric observations for several star clusters a couple of times to find out how well a DSLR can be transformed to Johnson-Cousing BVR system. In short: Yes, we can. In contrast to the hypothesis of large non-linearties of a CMOS, photometry of star clusters yield stellar magnitudes with accuracy of much better than 0.01-0.03 mag. To be honest, I didn’t take non-linearities into account as longs as stars in the field have intensities below 70% of saturation. This is usually the case for 30 s exposure of a stellar fields having maximum brightness of 8(7) mag stars with a 8" telescope and focal reducer. Photometry is perfect even without correction for non-linearity. Having found a star cluster with no perfect match is a matter of quality of the observers and publication where data sets have been taken out. This is a different story and the valid assumption will be to have properly identified a poor publication record.
Peter Somogyi wrote:At least if I'd get to a CMOS (again...), the very first I'd test was the linearity curve.
With variable gains (I do have an ASI for guiding and viewfinder), gain setting easily produce a nonlinear exposure-ADU curve.
Again this is a test, if the gain settings will follow a linear configuration. We should be careful to find a wrong conclusion here of having found sensor non-linearity (see above discussion of proper testing method). This may widely differ between camera manufacturers, esp. true for the CCD imagers.
Peter Somogyi wrote:As for DSLRs, for me even the dark exposure time had to match with the light exposure (dark non-reproduceability). That's also something to be tested.
And, did hear that bias frames also change sometimes, however bias is not usable for DSLRs for the above reason. Question if it comes from the chip or camera.
Probably, we should align on the terminoiogy here: Distinguish between bias and offset. I would propose to use these definitions: Bias frame is obtained from any short or the shortest possible exposure time, while (constant?) offset means certain arbitrary electronic or digital offset to the measured signal. As dark signal and noise of CMOS evolve mildly compared to CCD, bias exposure is sufficient to be done with 1/100 s, while taking these as darks for the flats. Shorter shutter times may produce artifacts like unequal illumination due to inabilities of the shutter mechanics (seen as gradient in image).

Bias shall be measured. However, this is only required in case a proper noise analysis is required to calibrate the sensor characteristics. I have taken literally thousands of bias frames as dark calibration for my flatfield series over time from different Canon cameras over 10 years now. I would assume bias will not change much, but noise characteristics vary with time, temperature, or ISO (gain) settings.

The offset, which is found with several DSLR models is delivered as digital number within the EXIF header the raw image contents of DSLRs. This may be subtracted channel-wise during processing of the images (depending on which software you are using). There is also the term "black" or "black level" used for the offset. This offset may slightly vary, but is almost fixed and aligned around certain value of 2°N for Canon devices. Depending on how you stretch image intensity to fit into 16 bit numbers, these offset numbers differ, and vary with models like EOS 40D and 60D. The 60D has a remarkably high offset around value of 2048 compared to EOS 40D having a value around 1024. This kind of offset just limits the available dynamic range.

Canon cameras in general have their black level kept as a constant added to the raw data stored (providing the offset added in the image file header), while Nikon seems to subtract it before storing image to file. Therefore, with Nikon camera models (tested until today) you will find a fixed back level adjusted around zero point. This strategy to increase available dynamic range comes with the big disadvantage: Noise properties of the Nikons CANNOT be measured in a realistic way. This, because negative values are clipped in the raw image and the noise computation will yield wrong values from statistical computation of the raw image (many clipped zero values found). Noise computation for the Nikons in general yields values too low compared to what should be expected from Gaussian noise. Histograms clearly show this typical Nikon artifact as a single half of a Gaussian distribution, which should be symmetric, but is not.

Typically, CMOS sensors show fixed pattern noise. This is found as static patterns in the background when adding hundreds of dark images and can be corrected accordingly using averaged dark frames. The effect can also be seen as varying offset within the image lines and columns and may also vary with time and observing conditions (temperature). Earlier models, like the 400D, provided more dramatic effects compared to the newest models. I own one modified EOS 40D which produces an arbitrary static "glow" in a portion of the image off center using exposure times of >2 minutes. I never found out what is the driver of this sensor defect. It is not typical and it is no amplifier glow. As read earlier in this forum, use at least the same number of dark frames as recorded images from the object of interest. The same applies for the number of flats for series of images. Otherwise you will not eliminate static noise from the calibration.

You may find more details here: http://www.astroinformatics.de/index.ph ... &Itemid=71

There is also an older article published on a conference about scientific use of DSLR and noise properties. You will find it on my page as well.
Peter Somogyi wrote: - special shapes (any filtration by chip, like low pass?)
I would assume two different things here: (a) optical filters or blurring filters in front of the sensor or (b) „figures“ seen from readout smear or convolution with a defect function from readout electronics. The latter is a typical source of tricky non-linearities in case of the CCD, but not CMOS images with individually addressable pixels.

Regarding (a) I am astronomer: All cameras, that I use, are modified Canon cameras with the filters and glasses removed from the body. Image blur from the production camera bodies is just slightly (worse) blurred to adapt for the many optics, the manufacturer also produces. If the camera manufacturers wouldn’t introduce image blur by such filters, many lens optics would not pass independent testing from the various magazines… :-) The smoothing is intended to correct for undersampling of certain optics esp. with fast stops. Undersampled images introduce several nasty problems. So, this discussion is not related to non-linearity of the sensors as it deals with the optical path.

CMOS sensors are expected to show problems completely different from a CCD. Speaking about DSLR, I didn't find non-linearities and thus ignore it usually from the image calibration pipeline. This is not a general recommendation, but my experience with the EOS bodies. I also found advantages over CCD, like better handling because DSLR don't require large current for the cooling of the sensor and heating the front glass to prevent dew on the other hand. One of the biggest advantage of a DSLR is: No computer required to take the many pictures from a nightly session. Any long-term testing of cooled CMOS cameras for astronomy will be one of my next tasks in astronomy.

Analytics of the CCD was done somewhere done in the early 90ies. I already closed this file.

DSLRs are great for spectroscopy of the stars down to 10 mag with an 8" Cassegrain. But, I also realized, I need an additional cooled monochrome sensor for recording of the very faint object spectra and also narrow band filter images of H-II regions.

A major advantage of the modern CMOS sensors may be seen in mass production and integrated electronic circuits for the whole digitization. Therefore, and because of their special designs there are less problems with capabilities of camera manufacturers to create high quality and linear imagers. I don't expect many surprises from current CMOS sensors, but expect a modern high quality CMOS imager.

Today, I got someone by phone and placed my order...

Regards,

Thilo
etienne bertrand
Posts: 1040
Joined: Mon Aug 31, 2015 11:26 am

Re: Use of CMOS detector for faint object spectrography

Post by etienne bertrand »

Par rapport à ces caméras avec capteur CMOS qu'en est il pour un Alpy et un Lhires ? Je pense à la caméra ASI 1600.
- Alpy600 et fentes 23,19, 15µm : est-ce envisageable ?
- LhiresIII et fentes 35µm voir 23µm, puis 19 et 15µm possible ?

Comme je ne m'y connais pas du tout, je préfères demander. L'avantage est d'avoir un capteur bien plus grand, pour l'Alpy j'utilises une Atik414EX et pour le LhiresIII une Atik314L+
Olivier GARDE
Posts: 1243
Joined: Thu Sep 29, 2011 6:35 am
Location: Rhône Alpes FRANCE
Contact:

Re: Use of CMOS detector for faint object spectrography

Post by Olivier GARDE »

Etienne,

Les tests qu'a fait Christian avec l'ASI 1600 sont prométeurs, le seul problème que je trouve sur ces caméras CMOS, c'est la dynamique assez faible de la caméra 12 bits (4096 ADU maxi) au lieu de 16 bits (65536 ADU), ce qui veut dire que si tu fait une étoile ayant une raie en emission très grande par rapport au continuum de l'étoile, tu vas être obligé de limiter le temps de pose pour ne pas saturer la raie en emission, mais tu n'auras pratiquement plus de signal dans le bleu du spectre.

Certe ces nouvelles caméras ont un bruit très faible qui fait que l'on arrive tout de même à obtenir de bons résultats.

Il y a également les pixels "déviants" qui posent problème.

Le rêve serait des CMOS avec 16 bits de dynamiques
LHIRES III #5, LISA, e-Shel, C14, RC400 Astrosib, AP1600
http://o.garde.free.fr/astro/Spectro1/Bienvenue.html
etienne bertrand
Posts: 1040
Joined: Mon Aug 31, 2015 11:26 am

Re: Use of CMOS detector for faint object spectrography

Post by etienne bertrand »

Effectivement c'est assez délicat surtout pour les pixels déviants/chauds qui sont un gros problèmes.
L'ASI290, elle, a une profondeur de 14bits, ça irait peut être ? mais plutôt pour le Lhires ?
Mais le capteur est vraiment très petit : 1936 x 1096 avec des pixels de 2.9µm
Tu penses qu'ils arriveront à faire des 16 bits ?

Pour un Lhires ça serait intéressant d'avoir une caméra très très sensible à faible bruit vu le faible flux qu'il y a dans le rouge du spectre surtout quand on travaille sur un intervalle de 100A / Atik314L+
Pour ce spectro 12 ou 14 bits est il envisageable ? il y a des raies assez intenses comme sur des étoiles comme gamcas ou des très lumineuses, mais peut être on a pas besoin d'autant de dynamiques que comme quand on travaille sur tout le spectre ??
Ca serait interessant de voir jusqu'à quelle dynamique marche le Lhires parce que j'ai l'impression que le signal qui arrive sur le capteur est quand même très faible des que l'on passe les magnitudes 5/6 avec un C8.
Ca serait interessant d'en savoir plus..
Ca permettrait de travailler des étoiles mv = 6 à 10 avec des temps de pose plus courts qu'avec les CCD habituels ou j'ai l'impression que c'est exponentielle pour avoir du signal. CX Dra par exemple mv = 5.9 il me faut poser unitairement au moins 2000s pour avoir du signal.. c'est long surtout quand on est dehors :) .
Christian Buil
Posts: 1431
Joined: Mon Sep 26, 2011 6:59 pm
Contact:

Re: Use of CMOS detector for faint object spectrography

Post by Christian Buil »

Je cherche toujours un peu le point de fonctionnement optimal avec un CMOS comme celui qui équipe la ASI1600MM. Il y a un paramètre qui déroute un peu lorsqu'on vient du CCD, c'est le gain réglable du capteur. Faut ici trouver le mieux pour une application donnée...

Par exemple, à présent pour faire le tungstène et aussi les ThorAr (équivalent néon pour un LHIRES par exemple), j'adopte un gain de 10 dB.

En langage clair c'est un faible gain pour ce type de capteur. Du coup le bruit en pas codeur est vraiment bas à cause du faible gain et du bruit intrinsèquement faible aussi. Ainsi, la VRAI dynamique n'est pas si mauvaise que cela (la vrai dynamique : ratio entre 4096 et le niveau du bruit en pas codeur) - et on s'approche un peu du CCD.

Pour les spectres sur étoiles, j'oscille entre 20 et 30 dB, gain plus fort, mais bruit en électron un poil plus faible (vers 1,5 e- RMS pour la ASI1600MM). Ici la perte de dynamique ce fait sentir, c'est clair, on aimerait 14/16 bits, mais on arrive à observer, car autre différence avec le CCD, il vaut mieux privilégier des poses brèves, genre 30 à 120 secondes, même pour des objets faibles (j'ai souvent encore tendance à pousser à 300 secondes, mais c'est un mauvais réflexe qui vient du CCD - bon pour le CCD, moins pour le CMOS). C'est très paradoxal, mais c'est ainsi que l'on fait les meilleurs spectres avec CMOS. Bien sur il faut additionner 5 à 20 images de ce type suivant les cas.

Finalement, la dynamique ne m'a jamais vraiment posé de gros soucis dans ces essais, c'est plus le gros volume de données à traiter qui peut effrayer (vrai pour les utilisateurs de eShel ou toute l'image sert, avec un Alpy par exemple, on peut fenêtrer et la difficultés tombe).

Attention ASI1600MM sur Lhires III = non, car franges !!!!!
ASI1600MM sur Alpy ou LISA, oui c'est potentiellement bien et même très bien.

Christian
Michel Verlinden
Posts: 191
Joined: Tue Aug 16, 2016 12:11 am

Re: Use of CMOS detector for faint object spectrography

Post by Michel Verlinden »

Bonjour Christian,
Est-il indispensable d'enregistrer des images avec trames pleines (exception faite des flats, des darks et des offsets) en spectroscopie ?
Un fenêtrage est-il possible, lors du traitement, pour limiter le poids des fichiers de données ?
Cordialement,
Michel VERLINDEN
Christian Buil
Posts: 1431
Joined: Mon Sep 26, 2011 6:59 pm
Contact:

Re: Use of CMOS detector for faint object spectrography

Post by Christian Buil »

Comme je dits dans le message précèdent, oui bien sur, il est tout à fait possible de travailler dans une fenêtre (sauf avec un spectro échelle, où en général toute la surface du capteur est exploitée).

Avec un Lhires, un Alpy,... on peut définir une fenêtre d'une centaine de pixels de haut typiquement, ou un peu plus (200, 250, ...). Cette hauteur dépend de la fidélité avec laquelle on positionne l'étoile sur la fente i.e. au même endroit. Il faut aussi parfois tenir compte des flexions mécaniques. Et bien sur, il faut de la place de part et d'autre de la trace du spectre de l'étoile pour bien évaluer le fond de ciel.

Les dark, flat-field, peuvent être fait aussi en fenêtrage. Mais attention, il faut bien entendu que ce soit la même fenêtre que pour les images sciences. C'est là souvent que le bas blesse. il faut ce rappeler des coordonnées des coins qui définissent la fenêtre. Certains logiciel d'acquisition s'en souvienne, mais le comportement d'un logiciel comme PRISM me semble parfois capricieux (il est trop facile de modifier par accident les coordonnées de cette fenêtre). On peut aussi noter ces coordonnées sur papier... et ne pas perdre celui-ci (c'est pour cela que je fenêtre assez rarement, je perd ce genre de truc - mais ca ne concerne que moi;-). L'idée est de toujours utiliser la même fenêtre pour ne pas avoir à refaire les images d'étalonnage à chaque session d'observation.

Christian
Post Reply