Skyhound Forums
Scheduler questions ... - Printable Version

+- Skyhound Forums (https://skyhound.com/forums)
+-- Forum: Imaging & Astrophotography (https://skyhound.com/forums/forumdisplay.php?fid=24)
+--- Forum: Deep Sky Astrophotography (https://skyhound.com/forums/forumdisplay.php?fid=16)
+--- Thread: Scheduler questions ... (/showthread.php?tid=324)

Pages: 1 2


Scheduler questions ... - choward94002 - 2018-09-26

I've been fiddling around with the scheduler, and am noticing something that's got me baffled ...

[attachment=178]


I've created a project as shown; 60 SNR, "Main extent/ arms" detail, IQ of A, multiple nights exposure ...

[attachment=179]

... when we look at the project in the Night Bar it looks OK; we have a blue line when it's possible to expose, a red line for the altitude, moon is tucked away, and we have two green bars; the top one we see an IQ B period from 2240 to 2300, an IQ A period from 2300 to 0050, then another IQ B band from 0050 to 0205, finally a "no way" band from 0205 to 0215.  In the lower bar we see a short period for setup in red, a period of time from 2305 to 0050 when it's good to expose and a yellow indent bar to indicate what region it's indicating.  That's all good ... I specified the minimum project detail as IQ A, multiple nights, cool ... and that's reflected in the top listbox ...

Notice on the bottom listbox it's giving me my filter time starting at 23:05 with 62x1 lights (it's occluded, but that listbox indicates it will be 2% complete toward my SNR 60 goal with this.  That makes sense, this is an SQM 19.1 site so I expect a short exposure time to help mitigate the skyfog, integration time of 62m ..

Look at the "Schedule" listbox though ... it shows at 23:02 I slew on over and change my filters and it's scheduling me for 62x1 lights, that's fine ...

If I bring up the "Exposure Calculator" to get a better idea of my signal and sky ADU values the Night Bar looks similar to the main one, and I get three "Exposure Opportunities" which correspond to the three green bands in the main bar; I've selected the middle one for IQ A, cool ...

... but here's where it gets strange; in the "Exposure Opportunities" for that IQ A row it's telling me a duration of 65min (fine, that's pretty close to 62min) but it's saying the ideal exp time is 1sec, when on the main dialog it's calling for 1min (one second is too short anyway unless I'm taking 65 * 60 lights for a total integration time of 65min).  Stranger still, in the "Calculate SNR for Exposure" it first set the "total time" edit for 115m and then in the group box it's telling me about 13 exp at 8min, which gives us a total integration time of 114min which matches the time of 115m it put into the edit box.

If I manually adjust the 115m that it put there to the 65min that the main dialog and the "Exposure Opportunities" listbox row call for then the text numbers get closer ... but am I doing something wrong that's causing these effects in the first place?

Thx!


RE: Scheduler questions ... - theskyhound - 2018-09-26

Hi,

In your Imaging Project is says "0 sec" for the Clear filter. I am updating the text here in the update, but what that means is that an SNR of 60 is not attainable for this object, likely because it doesn't get very high in the sky and the bright sky background. So by going ahead and scheduling, you are setting it up to fail, so to speak. So that's the first thing I see.

There are going to be (usllay small) differences between the Scheduler and the Exposure Calculator because their are subtle difference in how they waork, dur to their roles being defined in slightly different ways.

The "ideal" exposure time is often not the same as the exposure time that delivers the most SNR. The ideal exposure time is computed via a fairly well known formula. But this formula is limited by real world considerations that I does not include. What SkyTools does is to look at a fixed period of time. It tries different sub exposure times and takes the one that delivers the highest SNR, and this is often not the same as the ideal time. Also, you may have set up your Imaging System with a minimum exposure time of one minute. In that case, SkyTools will use 1 minute subs even if something less would increase the SNR.

There is a section in the Technical Reference of the Help about ideal sub exposure times, how they are calculated, and what the limitations are in real world applications.

Did I miss anything? If you have more questions, fire away.


RE: Scheduler questions ... - choward94002 - 2018-09-26

(2018-09-26, 08:37 PM)theskyhound Wrote: In your Imaging Project is says "0 sec" for the Clear filter. I am updating the text here in the update, but what that means is that an SNR of 60 is not attainable for this object, likely because it doesn't get very high in the sky and the bright sky background. So by going ahead and scheduling, you are setting it up to fail, so to speak. So that's the first thing I see.

Got it ... I assume that you're referring to the Exposure Calculator dialog "Calculate required exposure time", but I had assumed it was reporting that as the requested SNR there was 100 ...  since the main dialog is giving me a schedule of 62x1, should I always be checking the Exposure Calculator to make sure it's in agreement?

There are going to be (usllay small) differences between the Scheduler and the Exposure Calculator because their are subtle difference in how they waork, dur to their roles being defined in slightly different ways.

The "ideal" exposure time is often not the same as the exposure time that delivers the most SNR. The ideal exposure time is computed via a fairly well known formula. But this formula is limited by real world considerations that I does not include. What SkyTools does is to look at a fixed period of time. It tries different sub exposure times and takes the one that delivers the highest SNR, and this is often not the same as the ideal time. Also, you may have set up your Imaging System with a minimum exposure time of one minute. In that case, SkyTools will use 1 minute subs even if something less would increase the SNR.

There is a section in the Technical Reference of the Help about ideal sub exposure times, how they are calculated, and what the limitations are in real world applications.

Yep, I read the technical note and that makes sense ... but here my misunderstanding is, if I have different exposure times as reported in the main Scheduling dialog lower list box and then reported in the Exposure Calculator, assuming that I rejigger the values in the Exposure Calculator (SNR, total time, etc.) to match the project values, if they are different which one should I "trust" for the values to use?

Did I miss anything? If you have more questions, fire away.

Thx!


RE: Scheduler questions ... - choward94002 - 2018-09-27

The "ideal" exposure time is often not the same as the exposure time that delivers the most SNR. The ideal exposure time is computed via a fairly well known formula. But this formula is limited by real world considerations that I does not include. What SkyTools does is to look at a fixed period of time. It tries different sub exposure times and takes the one that delivers the highest SNR, and this is often not the same as the ideal time. Also, you may have set up your Imaging System with a minimum exposure time of one minute. In that case, SkyTools will use 1 minute subs even if something less would increase the SNR.

There is a section in the Technical Reference of the Help about ideal sub exposure times, how they are calculated, and what the limitations are in real world applications.

OK, doing more fiddling around ... the Exposure Calculator seems to be oriented around "what can I accomplish with the viewing session tonight", either "what's the highest SNR image I can take if I use the entire period for the IQ requested" or "how long will it take to get to the SNR requested given the period that the IQ is at the level requested" ... OK, got it ...

For multi-night it looks like a combination of Imaging Project dialog plus the scheduling window is what I need ... so from the attached project of M100, I can see ...

[attachment=181]

From the Imaging Project dialog, for an SNR of 20 with this object, IQ of A it will take me 22.1hr of imaging to get to where I want to be [yes?]

From the Scheduler, on the date that this is selected for the period selected ST4 is telling me that I need to take 51x1 lights of this object (91min total runtime, 51min total integration time) [yes?].  The lower listbox is telling me that this evening will complete 17% of my total integration time, and I will have an SNR for this evening of 3 for this light once it's been stacked [yes?]

Based on this information, I do a run that evening, and record the results the next morning in my observation log for the project ...

[attachment=182]

... note that even though I've set the exposure count and time in the "Edit Observation" dialog as 51x1, in the Imaging project dialog it's seeing this as 1 acceptable image with a total SNR of 0, completion 0% with a total time spent of 51min ... in the Observations section of the Imaging Project it reflects that as well, 51x1 exposures but no images ... is the program expecting some additional info?

I progress to the next day in the series, the 30th, but the schedule here doesn't recognize the exposures that I did on the 29th ... what should I do to cause it to recognize the previous observation period on the 29th and continue the series?

Also, let's assume that we figure out the progression and each day I do an additional 51x1 lights for another 17% ... that means that when we reach 100% we will have completed a total of 300min of exposures, or 5hr ... but that's much shorter than the Imaging Project dialog which tells me I have 22.1hr of exposures to do ... what could cause this?

Soldiering onward!  Undecided


RE: Scheduler questions ... - theskyhound - 2018-09-27

(2018-09-27, 01:59 AM)choward94002 Wrote: ... note that even though I've set the exposure count and time in the "Edit Observation" dialog as 51x1, in the Imaging project dialog it's seeing this as 1 acceptable image with a total SNR of 0, completion 0% with a total time spent of 51min ... in the Observations section of the Imaging Project it reflects that as well, 51x1 exposures but no images ... is the program expecting some additional info?

I progress to the next day in the series, the 30th, but the schedule here doesn't recognize the exposures that I did on the 29th ... what should I do to cause it to recognize the previous observation period on the 29th and continue the series?

Also, let's assume that we figure out the progression and each day I do an additional 51x1 lights for another 17% ... that means that when we reach 100% we will have completed a total of 300min of exposures, or 5hr ... but that's much shorter than the Imaging Project dialog which tells me I have 22.1hr of exposures to do ... what could cause this?

Soldiering onward!  Undecided

I would need to see a screen capture of the Observation tab of the Imaging Project to be able to better comment, but I suspect that root issue remains the same: this target object is very difficult using the Clear filter under your observing conditions. It doesn't think you recorded any SNR at all in the previous image stack. Did you? Or is this all pretend? 

Without a screen capture I am having trouble following the rest, but regarding the second part, SNR does not stack linearly. You are not going to add 17% each night.  You are going to add less and less SNR each night.


RE: Scheduler questions ... - choward94002 - 2018-09-27

(2018-09-27, 04:17 PM)theskyhound Wrote:
(2018-09-27, 01:59 AM)choward94002 Wrote: ... note that even though I've set the exposure count and time in the "Edit Observation" dialog as 51x1, in the Imaging project dialog it's seeing this as 1 acceptable image with a total SNR of 0, completion 0% with a total time spent of 51min ... in the Observations section of the Imaging Project it reflects that as well, 51x1 exposures but no images ... is the program expecting some additional info?

I progress to the next day in the series, the 30th, but the schedule here doesn't recognize the exposures that I did on the 29th ... what should I do to cause it to recognize the previous observation period on the 29th and continue the series?

Also, let's assume that we figure out the progression and each day I do an additional 51x1 lights for another 17% ... that means that when we reach 100% we will have completed a total of 300min of exposures, or 5hr ... but that's much shorter than the Imaging Project dialog which tells me I have 22.1hr of exposures to do ... what could cause this?

Soldiering onward!  Undecided

I would need to see a screen capture of the Observation tab of the Imaging Project to be able to better comment, but I suspect that root issue remains the same: this target object is very difficult using the Clear filter under your observing conditions. It doesn't think you recorded any SNR at all in the previous image stack. Did you? Or is this all pretend? 

Right now it's a full moon at my imaging site, so I'm shut down until Oct 2nd ... I'm planning on tasking three of the C11's to do some experiments with the numbers I'm getting from ST4, that's why I'm doing as much planning and whiteboarding now so that when I get some sky time I can make the best of it ...

Without a screen capture I am having trouble following the rest, but regarding the second part, SNR does not stack linearly. You are not going to add 17% each night.  You are going to add less and less SNR each night.

Hmm ... OK, then what's the value of having 'multi-night" exposures and tracking at all?  Obviously it's to take advantage of stacking adding signal at a higher rate than noise is added (and if we can keep the noise as low as possible, by controlling things like the SQM we expose at and the seeing of that night we can optimize that process).  I'll provide additional screen captures but my question about why the Imaging Project didn't let me make forward progress on a particular session remains ...

Doing more research into what makes the filter "register" or not (to prevent my "setting myself up for failure"), it appears that Exposure Calculator is single-session oriented ... the numbers on the left let me answer the question "what is the most SNR I can get for the time I've given it", and the numbers on the right answer the question "what is the most SNR I can get for the entire session available" ... so, if my IQ A time is 30min and I'm interested in getting a single SNR 20 picture for that night it will tell me "X exp at Y sec" which gives me 5min total (leaving 25min left in the IQ A session).  If I want to see what I'd get for the whole 30 min IQ A period it will tell me "Z exp at A sec will give an SNR of 45" (using the entire 30min IQ A session ...

Further research into what makes the "Clear" filter in the Imaging Project give me some number other than 0 seconds (which you said indicates it's not possible) and extrapolating backward with the Exposure Calculator yields that an Exposure Calculator session (left side numbers) SNR of 4+ makes the Imaging Project (and thus the Scheduler) happy ... until I can get the multi-night progress working ST4 will happily schedule each night for whatever I need to get to the 4+ SNR and I'm happy gathering those up over time ...

It's also unclear what an "SNR 4" light will look like; there are about as many ways to calculate signal SNR as there are stars in the sky, all I can do is take some current objects I've got and compare them.  One of my experiments I'm planning once the skies darken will be to take a series of SNR 4 lights of a galaxy and of a planetary nebula, stack them up and compare them to images I've already made of those in the past (with, of course, the same SQM).  I'll then use PixInsight's ImageIntegration tool to get an SNR value for various regions of those objects ... that will answer (for me) if a bunch of SNR 4 lights really can give me a usable picture with a sufficiently large number of stacks as well as tell me what "SNR 4" for ST4 equates to with PixInsigt's tool.

Onward!  Dodgy


RE: Scheduler questions ... - theskyhound - 2018-09-27

(2018-09-27, 04:59 PM)choward94002 Wrote:
(2018-09-27, 04:17 PM)theskyhound Wrote: Without a screen capture I am having trouble following the rest, but regarding the second part, SNR does not stack linearly. You are not going to add 17% each night.  You are going to add less and less SNR each night.

Hmm ... OK, then what's the value of having 'multi-night" exposures and tracking at all?  Obviously it's to take advantage of stacking adding signal at a higher rate than noise is added (and if we can keep the noise as low as possible, by controlling things like the SQM we expose at and the seeing of that night we can optimize that process). 

I'll provide additional screen captures but my question about why the Imaging Project didn't let me make forward progress on a particular session remains ...

I never said that there was no value in stacking images. In your question you made a calculation. Your calculation does not reflect the reality of how stacking images and SNR works. You can't expect twice the SNR every time you double the number of images in the stack. It just doesn't work that way. This is why we need complex software. Wink

I need the screen capture in order to answer your question.

"Doing more research into what makes the filter "register" or not (to prevent my "setting myself up for failure"), it appears that Exposure Calculator is single-session oriented ... the numbers on the left let me answer the question "what is the most SNR I can get for the time I've given it", and the numbers on the right answer the question "what is the most SNR I can get for the entire session available" ... so, if my IQ A time is 30min and I'm interested in getting a single SNR 20 picture for that night it will tell me "X exp at Y sec" which gives me 5min total (leaving 25min left in the IQ A session).  If I want to see what I'd get for the whole 30 min IQ A period it will tell me "Z exp at A sec will give an SNR of 45" (using the entire 30min IQ A session ..."

Yes, that is mostly correct. The SNR computation works for a single night. But you can set the Exposure time calculation to "Ideal Conditions" and it ignores the night and time. It just assumes ideal conditions into forever, just like the numbers in the table on the Exposure tab of the Imaging Project. If your interest is in knowing the *minimum* exposure time that will be required to reach the target SNR, then the table in the Exposure tab of the Imaging Project is what you want to look at. It is a minimum because it assumes ideal conditions.

"It's also unclear what an "SNR 4" light will look like; there are about as many ways to calculate signal SNR as there are stars in the sky, all I can do is take some current objects I've got and compare them.  One of my experiments I'm planning once the skies darken will be to take a series of SNR 4 lights of a galaxy and of a planetary nebula, stack them up and compare them to images I've already made of those in the past (with, of course, the same SQM).  I'll then use PixInsight's ImageIntegration tool to get an SNR value for various regions of those objects ... that will answer (for me) if a bunch of SNR 4 lights really can give me a usable picture with a sufficiently large number of stacks as well as tell me what "SNR 4" for ST4 equates to with PixInsigt's tool."

When it comes to extended objects there is only one way to calculate SNR. Stars are calculated differently, and there are two ways to calculate SNR for them. The tool you mention above uses the method for non-stars, so it should be similar to what SkyTools computes for extended objects. But you need to be consistent. Take a galaxy. You will measure a higher SNR in the bright nuclear region where there is more signal, and a much lower SNR way out on the extended halo. So when comparing SNR you need to compare apples to apples.

Also keep in mind that SkyTools is assuming that you did proper calibrations on all of your images and the SNR is computed for the final image stack. The calculated SNR ignores non-random noise sources such as cosmic rays and satellites.

Remember, untimely the point is to help you choose suitable targets and to obtain the highest quality images possible, not to reproduce the SNR exactly. What would that really mean for an extended object anyhow? The brightness varies across the image, so the SNR will do so as well. 

As far as what an SNR of 4 means to you, SkyTools is not meant to replace you or your experience. You will need to develop a feel for what an SNR of 4 means.



RE: Scheduler questions ... - choward94002 - 2018-09-27

[quote pid='1084' dateline='1538071347']
[quote pid='1082' dateline='1538067577']

As far as what an SNR of 4 means to you, SkyTools is not meant to replace you or your experience. You will need to develop a feel for what an SNR of 4 means.

[/quote]

[/quote]

Yep, and that's what makes ST4 such an intriguing tool to work with ... now that I have a reproducible way to generate the quantifiable "SNR 4" light I can get a feel for just what that means for galaxies, planetary nebulae, diffuse nebulae, all of the interesting stuff out there for a given SQM and seeing.  It might turn out that "SNR 8" at SQM 21.0 seeing 1.0" or "SNR 16" at SQM 18.2 seeing 1.5" is the minimum "SNR" for a galaxy to get the arms under various SQM's and seeing, maybe an "SNR 8" at SQM 20.5 seeing 1.0" or "SNR 20" at SQM 19.0 seeing 2.0" for a planetary, maybe "SNR 12" at SQM 20.1 seeing 1.2" or "SNR 25" at SQM 18.6 seeing 2.1" for an extended nebula works best for me ... 

And with ST4's scheduling, once I've gotten that "minimum SNR for SQM and seeing" I can now go back and based on what my night should be schedule up just the right objects to get my lights for them ... 

Big Grin


RE: Scheduler questions ... - theskyhound - 2018-09-27

Hello,

Another person has been telling me about how they use the SQM readings to plan their imaging. Now this is a little controversial to me, because I am not convinced that larger SQM values universally mean better quality images, and the mechanism of how it would do this isn't obvious to me.

Right now, SkyTools doesn't have a means of adjusting for sky "quality," for lack of a better term. Call it transparency, or SQM, or whatever. You can adjust the seeing, and the weather conditions determine how much extinction there is and affect the sky brightness to some extent, but a clear dark night is assumed. It is sort of a baseline. Crappy nights will get you less of everything. The sky brightness is part of the location because it wasn't meant to be changed.

I am starting to suspect that the way some people are treating their SQM readings, there would be some benefit to adding some sort of sky quality adjustment. I'm not really sure how to do this, however, because I suspect that in practice the higher SQM readings not only mean that there is less sky signal, but that there is more signal from the object. So maybe we are really talking about a link to the extinction with SQM? Obviously I need to look into this more.


RE: Scheduler questions ... - choward94002 - 2018-09-28

(2018-09-27, 11:36 PM)theskyhound Wrote: Hello,

Another person has been telling me about how they use the SQM readings to plan their imaging. Now this is a little controversial to me, because I am not convinced that larger SQM values universally mean better quality images, and the mechanism of how it would do this isn't obvious to me.

Right now, SkyTools doesn't have a means of adjusting for sky "quality," for lack of a better term. Call it transparency, or SQM, or whatever. You can adjust the seeing, and the weather conditions determine how much extinction there is and affect the sky brightness to some extent, but a clear dark night is assumed. It is sort of a baseline. Crappy nights will get you less of everything. The sky brightness is part of the location because it wasn't meant to be changed.

I am starting to suspect that the way some people are treating their SQM readings, there would be some benefit to adding some sort of sky quality adjustment. I'm not really sure how to do this, however, because I suspect that in practice the higher SQM readings not only mean that there is less sky signal, but that there is more signal from the object. So maybe we are really talking about a link to the extinction with SQM? Obviously I need to look into this more.

I had a "discussion" on CN recently on just this very topic of SQM and extinction [https://www.cloudynights.com/topic/631613-autoguiding-an-sct-8-inch-lx200-with-a-orion-mini-guide-scope-%E2%80%93-my-experience-in-detail/] and SQM very definitely has an impact on extinction ... if I'm imaging in skies where the limiting magnitude is 10 due to sky brightness and I'm shooting an object that is magnitude 12, I can be taking lights for the rest of my life and I'm not going to get anything ... but ST4 already takes this into account, by increasing the exposure time needed to catch something when SQM goes down, and decreasing it when SQM goes up ...

The way that I see it and have read it described, signal from all sources arrives at different rates.  Camera dark current is coming in at a measurable rate which is dependent upon temperature (which is why I'm surprised that you don't have a spot for the cooling temperature of the imaging camera, as that will make a big difference in calculated exposure times ...), object signal is coming in at a measurable rate (that's the object magnitude, and as you pointed out that will vary from spot to spot on the object) and skyfog signal is coming in at a measurable rate (that's the SQM/ sky brightness).  Each of these things is contributing electrons that will saturate the camera, shift the histogram to the left and the exposure is done.  The trick is, how do I limit the amount of "bad" signal coming in so that I give as much time as possible for the "good" signal to come in.  One source of "bad" signal is the camera dark current, I can control that with TEC cooling and postprocess it with darks.  Another source of "bad" signal is from the skyfog, I can control that by going to a place that has less skyfog (a dark site) or by using a filter (LPR or broadband) ... so what I'm left with is maximizing the time window for the "good" signal to accumulate before the sum of all three end my session.  To me, that's "SNR", how much "good" signal I have of the "bad" signal ... and empirically that makes sense, if I take shots with my uncooled DSLR, even though the CCD has much the same spec's as my cooled CCD, it saturates much, much faster than the same exposure with my CCD ... much less time for the constant signal to accumulate, much lower "SNR".  If I take a picture with my CCD here in 19.2 SQM Belmont, it saturates much faster than my 21.5 SQM Payson ... again, much less time for the constant signal to accumulate, much lower "SNR".  By cooling my camera, by imaging in Payson, I am able to let the CCD spend more time gathering the "good" signal and my "SNR" increases ...

Seeing, on the other hand, does not impact how much signal I can accumulate but rather the quality of that signal.  Imagine that I was in space, floating on the HST ... my SQM will be very high and my level of detail also very high.  Now, imagine that I have a fast rotating black hole between the object and me, and it's acting as a gravitational lense, bending the light from my object back and forth every few seconds ... even though my SQM is very high, because the light from the object is "wobbling" my level of detail, my "seeing", has gone down.  Back to Earth, suppose that I'm looking at an object like NGC 470, which is 2.9' by 1.6' in size ... and with my setup I've got a .33"/pixel resolving power, which means this object will fit into a very few pixels but enough that I can get *some* detail out of it.  "Seeing" to me means how much the light is going to wobble around due to atmospherics, so suppose that my "Seeing" is .8" .. that means that a photon from the center of the nucleus of NGC 470 is going to go +/- .8" from that center, which for an object that is 2.9' by 1.6' is probably going to stay in that central region ... high quality!  Suppose now that my seeing is bad, at 3.0" ... that means the photon from the center is going to go +/- 3.0" from the center which will likely put it into an arm region ... low quality!  Notice that this has nothing to do with how much of the signal I can accumulate ... that photon is still going to get collected by my CCD.  This only impacts WHERE on that CCD the photon is going to end up, in the center of the galaxy where it started or in the arms somewhere ...

So, "SQM" for me relates to how fast the CCD will fill up with the "bad" skyfog noise (just like cooling affects the "bad" dark current noise), "Seeing" for me relates to how much of the detail of the object is going to get "smeared".  Some objects like galaxies are more sensitive to "smearing" than others ... I will notice the effects of "Bad" seeing on NGC 470, I wouldn't notice a 5.0 seeing on my exposure of the North America nebula.  Some objects have very little signal (like NGC 470 at a 12.6 mag) so it's hard to get much signal from them before the CCD fills up from skyfog, other objects have a lot of signal (like M42 at a 4.0 mag) so I could probably image them from the center of Oakland ... so when I select objects to image I look to see how sensitive they are to skyfog ("SQM"), and how sensitive they are to detail ("Seeing"), put like with like and adjust my imaging sessions so when I've got crappy skies I image the binned and HyperStar objects, when I've got clear skies I image the F/10 and F/7 objects, when I've got crappy seeing I image the nebulae and when I've got clear skies I shoot for galaxies and planetary nebulae ...

Hope this makes some sense, I'm sure I've gotten most of these concepts and terms completely wrong but that's why I'm here to learn the right concepts and right terms!