Problem with median filtering

Design, construction, tuning of spectroscopes
Information and discussion about softwares (telescope remote, autoguiding, acquisition, spectral processing ...)
Post Reply
Tim Lester
Posts: 168
Joined: Fri Apr 04, 2014 6:31 pm
Location: Arnprior Ontario

Problem with median filtering

Post by Tim Lester »

After reading Christians excellent article on echelle optimization I started to use 3x3 median filtering in my data reduction pipeline as I am using an ASI1600m cmos sensor on my echelle.

However, I think I have uncovered a problem with median filtering that may be quite general and not just specific to my setup.
The following image is a portion of order 33 processed with median filtering applied after image calibration, before binning 2x2.
3x3 median filter.PNG
3x3 median filter.PNG (123.19 KiB) Viewed 2588 times
This shows ripple that looks like an interference effect, but I had not changed my instrument in any way and I did not have this before.
After processing again without the 3x3 median filter I got the following result - normal with no ripple
no median filter.PNG
no median filter.PNG (121.95 KiB) Viewed 2588 times
One can actually see the problem in the spectral image if you look closely. Here is a portion of the image after calibration but before median filtering.
cal.PNG
cal.PNG (89.24 KiB) Viewed 2588 times
and after a 3x3 median filter.
cal_3x3 median.PNG
cal_3x3 median.PNG (58.02 KiB) Viewed 2588 times
The ripple is hard to see but becomes more obvious with histogram stretching.
cal_3x3 median_stretch.PNG
cal_3x3 median_stretch.PNG (30.49 KiB) Viewed 2588 times
The ripple induced from applying a 5x5 median filter is obvious even without histogram stretching.
cal_5x5 median.PNG
cal_5x5 median.PNG (49.49 KiB) Viewed 2588 times
I used both MaximDL and PixInsight to apply the median filters. Both gave the same result.
I think the problem lies with the way a 3x3 median filter resamples a pixilated image, particularly curved and sloping lines. It appears not to be geometrically neutral.
3x3 or 5x5 average filtering does not produce this problem. Of course this is not useful for reducing the random cmos telegraph noise.
Here is the spectrum with 5x5 average filtering applied.
cal_5x5 average.PNG
cal_5x5 average.PNG (48.96 KiB) Viewed 2588 times
So , while median filtering is excellent for reducing cmos telegraph noise it can also lead to spurious artifacts.
How obvious these artifacts are will probably depend on how diffuse the lines on the spectral image are.
In my case they were unacceptable.
Peter Somogyi
Posts: 420
Joined: Sun Jul 13, 2014 8:56 am

Re: Problem with median filtering

Post by Peter Somogyi »

Median can be used without artifacts only when the _expected_ values are exactly the same - otherwise filtering out real signal as well...
The picture "cal_5x5 median.PNG" speaks clearly against.
My earlier experience from deep sky photography tells, simple average (+ sigma clipping for hotpixels - that can come from median) always work more naturally (but need to level pixels to the same common value, to make sigma clip work well).
Hope either not to see median in ISIS elsewhere than error detections (e.g. cosmetic), or have it well documented everywhere.
Peter
Christian Buil
Posts: 1431
Joined: Mon Sep 26, 2011 6:59 pm
Contact:

Re: Problem with median filtering

Post by Christian Buil »

Tim,

I haven't seen this type of ripple on my data during my eShel test optimization, but I fully understand that it can happen given the nature of the median filter.

The focal length of the collimator is 120mm in eShel. With an 85 mm camera lens (your situation), the spectrum width is therefore 50 microns x 85/120 = 35 microns, i.e. a width of 9.2 if you work in 1x1 binning during the acquisition (this is is it?). In my situation (135mm), the spectrum width is 15.0 pixels.

The difference is not very significant, so the difference in appearance is a surprise for me. Do you confirm this calculation?

Median filtering is probably too rudimentary, but sigma clipping is tricky because there can be only one frame, and threshold criteria are difficult to establish.

Christian
Tim Lester
Posts: 168
Joined: Fri Apr 04, 2014 6:31 pm
Location: Arnprior Ontario

Re: Problem with median filtering

Post by Tim Lester »

Hi Christian,

My collimator is 100mm but the main difference is that my echelle is not fiber fed. I use a short 75 um long slit.
The width of the lines is set by the seeing and is roughly Gaussian with a typical FWHM of 7 to 8 pixels (1x1 binning). So not very different from eShel.

I have been taking advantage of the very low cmos readout noise by dividing the total exposure into a least 6 sub exposures.
These I combine with sigma clipping in PixInsight. If seeing conditions change during the set I sometimes find that sigma clipping does not work well in which case I fall back to percentile clipping.
I use PixInsight because it produces pixel rejection maps that can be used to check the validity of the settings.
Unfortunately I find that this method starts to clip real signal well before all the cmos 1/f noise is eliminated. So the technique is only partially successful.

Tim
Post Reply