dark.jpg is removed because mean ia above high treshold

[zanco]

Hi, took the pi from the roof to add some cooling. Also decided to to generate dark.jpg files to try if that improves the nightly images.

My dark.jpg's are not put in the supposed folder but removed:

2025-08-10T20:31:26.898336+02:00 rpi4-allsky allsky[8701]: Taking dark frames...
2025-08-10T20:31:26.901175+02:00 rpi4-allsky allsky[8701]: STARTING EXPOSURE at: 2025-08-10 20:31:26 @ 120.0 sec
2025-08-10T20:33:29.427662+02:00 rpi4-allsky allsky[8701]: > Saving dark image 'dark.jpg' 2025-08-10T20:33:29.433378+02:00 rpi4-allsky allsky[8701]: STARTING EXPOSURE at: 2025-08-10 20:33:29 @ 120.0 sec
2025-08-10T20:33:30.968479+02:00 rpi4-allsky allsky[8866]: removeBadImages.sh: File is bad: removed 'dark.jpg' (MEAN of 0.541334 is above high threshold of 0.01000)
2025-08-10T20:35:31.810156+02:00 rpi4-allsky allsky[8701]: > Saving dark image 'dark.jpg' 2025-08-10T20:35:31.816095+02:00 rpi4-allsky allsky[8701]: STARTING EXPOSURE at: 2025-08-10 20:35:31 @ 120.0 sec
2025-08-10T20:35:33.368785+02:00 rpi4-allsky allsky[9009]: removeBadImages.sh: File is bad: removed 'dark.jpg' (MEAN of 0.106028 is above high threshold of 0.01000)

But bot my low and high treshold are set to 0 which should mean disabled. I did try to set the high treshold to 1 which would be higher as the MEAN values reported in the log. Still the same message and no dark.jpg files are saved.
support-none-20250810204239.zip

[laheller]

@zanco @EricClaeys

I have the same problem. Already tried to adjust a lot of settings, but unfortunately none of them helped.

WebUI system message:

Multiple consecutive bad images.
Check the values of Remove Bad Images Threshold Low, Remove Bad Images Threshold High, and Max Auto-Exposure in the WebUI.

The content of my bad_images.txt file:

removed 'dark.png' (MEAN of 0.117188 is above high threshold of 0.10000)
removed 'dark.png' (MEAN of 0.125000 is above high threshold of 0.10000)
removed 'dark.png' (MEAN of 0.117188 is above high threshold of 0.10000)
removed 'dark.png' (MEAN of 0.117188 is above high threshold of 0.10000)
removed 'dark.png' (MEAN of 0.117188 is above high threshold of 0.10000)

WebUI - Post Capture Settings:
Remove Bad Images Threshold Low: 0
Remove Bad Images Threshold High: 0

Any idea?

BR,

Ladislav

[EricClaeys]

@laheller,
Try doing what I suggested @zanco do. A dark frame with a mean brightness of 0.117 is pretty bright and I suspect light may be leaking in.
If I am wrong and you just have really bright dark frames, change the number around line 97 in allsky/scripts/saveImage.sh.
And please attach one of your darks.

We recently increased the dark threshold from 0.01 (what @zanco's release has) to 0.1, and I'm willing to increase again if needed.

[EricClaeys]

Please double check that your camera lens is totally covered and NO light is getting in. Plastic lens caps let in light.
A mean brightness of 0.5 is like a daytime image.
Even your last image in the output had a mean of 0.106028 which is like a dark nighttime image.

Dark frames withe NO light hitting the sensor probably have a mean brightness of 0.005 or less.

When saving dark frames, no check is done for the low threshold and a high threshold of 0.01 is used (this was increased to 0.1 in the latest Allsky release.

To see what light is getting to the sensor, leave the lens covered and set exposure to Manual at 120 seconds with 0 for your low a d high thresholds. Then take a few normal pictures.

[laheller]

@EricClaeys

Please also check my answer above. I don't think any light is getting to my camera sensor, since the normal night image with the totally covered camera lens is like this:

I only see there the sensor noise coming from the ASI 120 MC-S camera.

BR,

Ladislav

[EricClaeys]

@laheller,
The ASI120 is very noisy and a lot of people switched to a diffetent camera.

Please attach one of your darks, don't paste a screenshot. I'd like to look at its characteristics.

[EricClaeys]

@zanco, @laheller,
I appreciate your support in helping determine what's going on. We will figure this out.

[laheller]

@EricClaeys

Attached you can check one of my fresh image with the totally covered camera lens:

BR,

Ladislav

[EricClaeys]

Can you attach the image instead of pasting a screenshot? You'll need to append .txt to the file name.

[EricClaeys]

@zanco,
Your log file has a ton of messages about the de421.bsp file not downloading. Take a look at this post for a fix to that. It's about half way down.

Also, can you upgrade to the latest Allsky release? It has better troubleshooting capabilities than the release you are on.
If you don't remember how to upgrade, see the Installing -> Allsky documentation page in the WebUI.

[zanco]

About the de421.bsp , after the upgrade to latest version by instructions I checked and the de421.bsp is there (now) and close to 17 MB size. Does that also solve the problem ?

I had one message after the upgrade about not being able to capture because of a "locale" problem which I solved by
sudo locale-gen en_US.UTF-8 and rconfiguring the locales. I might have made a / the wrong choice with answering the locales question during the upgrade. Now capturing images again :-)

[EricClaeys]

@zanco,
If the de421.bsp file isn't on the Pi, Allsky tries to download it after every image. A lot of people seem to have problems downloading it, possibly because the server it's on is flaky.

Your log file had a message about not being able to use locale en_US, which isn't a valid locale. Type locale --all to see the ones installed on your Pi. If the one you want isn't there, run raspi-config and go to the localization option and install it, then go into the WebUI and change the Locale to the newly-installed one.

All the locale does in Allsky is output some numbers using the decimal point and thousands separator of your local ("." and "," for the US). And Linux will display many error messages in the correct language. Allsky itself isn't internationalized.

[laheller]

@EricClaeys

OK, for me simply editing the removeBadImages.sh script & changing the line 97 to
HIGH=0.20000
did the trick: Darks are now created and saved.

Thanks for the tips & support!

BR,

Ladislav

[EricClaeys]

Please attach one of the darks so I can check it out. You'll need to append .txt to the file name.

[laheller]

Please attach one of the darks so I can check it out. You'll need to append .txt to the file name.

@EricClaeys

OK, here is one for 32 degrees Celsius with 30s and Gain = 50.
32.png.txt

[EricClaeys]

@laheller,
thanks. Would you please attach /var/log/allsky.log? I'd like to see what Allsky thought the mean was for 32.png.

[zanco]

dark.jpg.txt
Hi, thanks for you quick support.

I found this black padded foam box from a watch and decided to put the camera inside the box. I was quite sure that yesterday evening my camera was in the dark, covered with 8 layers of folded clothing, but you where right cause now I get dark.jpg images which are not removed :-)

I tailed the allsky.log cause with 120 seconds exposure time one has to realise it takes a bit for the first dark.jpg to arrive.

My log shows no "mean" value

tail -f /var/log/allsky.log
2025-08-11T16:47:01.480799+02:00 rpi4-allsky allsky[4161]: Taking Dark Frames: Yes
2025-08-11T16:47:01.480919+02:00 rpi4-allsky allsky[4161]: Debug Level: 1
2025-08-11T16:47:01.481017+02:00 rpi4-allsky allsky[4161]: On TTY: No
2025-08-11T16:47:01.481110+02:00 rpi4-allsky allsky[4161]: Tuning File (day): [none]
2025-08-11T16:47:01.481200+02:00 rpi4-allsky allsky[4161]: Tuning File (night): [none]
2025-08-11T16:47:01.481294+02:00 rpi4-allsky allsky[4161]: Overlay method: module
2025-08-11T16:47:01.481389+02:00 rpi4-allsky allsky[4161]: Temperature type: C
2025-08-11T16:47:01.481504+02:00 rpi4-allsky allsky[4161]: Allsky version: v2024.12.06_01
2025-08-11T16:47:01.482504+02:00 rpi4-allsky allsky[4161]: Taking dark frames...
2025-08-11T16:47:01.485112+02:00 rpi4-allsky allsky[4161]: STARTING EXPOSURE at: 2025-08-11 16:47:01 @ 120.0 sec
2025-08-11T16:49:03.795899+02:00 rpi4-allsky allsky[4161]: > Saving dark image 'dark.jpg'
2025-08-11T16:49:03.804358+02:00 rpi4-allsky allsky[4161]: STARTING EXPOSURE at: 2025-08-11 16:49:03 @ 120.0 sec
2025-08-11T16:51:06.419028+02:00 rpi4-allsky allsky[4161]: > Saving dark image 'dark.jpg'
2025-08-11T16:51:06.425213+02:00 rpi4-allsky allsky[4161]: STARTING EXPOSURE at: 2025-08-11 16:51:06 @ 120.0 sec
2025-08-11T16:53:08.990095+02:00 rpi4-allsky allsky[4161]: > Saving dark image 'dark.jpg'
2025-08-11T16:53:08.995944+02:00 rpi4-allsky allsky[4161]: STARTING EXPOSURE at: 2025-08-11 16:53:08 @ 120.0 sec
2025-08-11T16:55:11.501161+02:00 rpi4-allsky allsky[4161]: > Saving dark image 'dark.jpg'
2025-08-11T16:55:11.507564+02:00 rpi4-allsky allsky[4161]: STARTING EXPOSURE at: 2025-08-11 16:55:11 @ 120.0 sec
2025-08-11T16:57:14.055515+02:00 rpi4-allsky allsky[4161]: > Saving dark image 'dark.jpg'

Now preparing for updating Allsky version.

[EricClaeys]

The 32.png.txt file you attached yesterday looks like it was from an ASI120 camera - low resolution and full of hot pixels.
The dark.jpg.txt file you attached a couple hours ago is from an RPi HQ camera and doesn't have any obvious hot pixels, which is what I'd expect from that camera.

Was 32.png.txt taken in not-so-dark conditions? Do you have an ASI120 image taken in totally dark conditions?

The "mean" is only output to the log file at a higher debug level. It's probably not too important - the mean for the whole 32.png file was 0.0949717. I was curious what the mean was when the file was taken, since that mean only looks at the center of the image.

[zanco]

I think 32.png.txt was from user laheller ?

[EricClaeys]

@zanco, My bad. Sorry about that. @laheller did post 32.png.txt. Igore my post from 3 hours ago.
Your dark.jpg.txt looked very dark so the default dark threshold of 0.1 is good for you. I didn't check but your dark is probably 0.01 or darker.

[zanco]

Upgraded, now running, left the camera in day mode in the black box

tail -f /var/log/allsky.log
2025-08-11T17:22:13.753461+02:00 rpi4-allsky allsky[5356]: STARTING EXPOSURE at: 2025-08-11 17:22:13 @ 1.0 sec
2025-08-11T17:22:17.016788+02:00 rpi4-allsky allsky[5356]: > GOT IMAGE: shutter_us: 1.0 sec, gain: 2.000, mean: 0.000, target mean: 0.500, diff (target - mean): 0.500
2025-08-11T17:22:17.017262+02:00 rpi4-allsky allsky[5356]: STARTING EXPOSURE at: 2025-08-11 17:22:17 @ 1.0 sec
2025-08-11T17:22:20.300969+02:00 rpi4-allsky allsky[5356]: > GOT IMAGE: shutter_us: 1.0 sec, gain: 2.000, mean: 0.000, target mean: 0.500, diff (target - mean): 0.500
2025-08-11T17:22:20.301097+02:00 rpi4-allsky allsky[5356]: STARTING EXPOSURE at: 2025-08-11 17:22:20 @ 1.0 sec
2025-08-11T17:22:23.744371+02:00 rpi4-allsky allsky[5356]: > GOT IMAGE: shutter_us: 1.0 sec, gain: 2.000, mean: 0.000, target mean: 0.500, diff (target - mean): 0.500
2025-08-11T17:22:23.744870+02:00 rpi4-allsky allsky[5356]: > Saving DAY image 'image-20250811172220.jpg'
2025-08-11T17:26:23.756301+02:00 rpi4-allsky allsky[5356]: STARTING EXPOSURE at: 2025-08-11 17:26:23 @ 1.0 sec
2025-08-11T17:26:27.065077+02:00 rpi4-allsky allsky[5356]: > GOT IMAGE: shutter_us: 1.0 sec, gain: 2.000, mean: 0.000, target mean: 0.500, diff (target - mean): 0.500
2025-08-11T17:26:27.065292+02:00 rpi4-allsky allsky[5356]: > Saving DAY image 'image-20250811172623.jpg'

[EricClaeys]

It would be interesting to see the mean for 120 second exposure which is what your darks are.

[zanco]

Camera is outside again now, so no chance. Just in time for meteorite shower max.

[EricClaeys]

@zanco, don't bother. Things are working well for you now. Hope you get a lot of meteorite images!

[EricClaeys]

@laheller,
Would you please attach /var/log/allsky.log? I'd like to see what Allsky thought the mean was for 32.png.
When Allsky takes an image it calculates a mean brightness while looking only at the center of an image so it doesn't include any black border that many people have. Several of your darks were around 0.117 which is slightly over the threshold of 0.1.

I checked the mean of 32.png but it looked at the whole image, and it was 0.0949717 which is under the threshold.
I am curious what the mean was when 32.png was taken, so I can compare the "center of image only" mean to the "whole image" mean.

I will likely change Allsky - either increase the limit (probably to 0.15), or to leave at 0.1 but add a setting where users can override that. If you ugrade Allsky, the change you made to removeBadImages.sh will get wiped out.

[EricClaeys]

@laheller, Ladislav,
That's easy to do. My first set of darks were anything but dark. It took me a while to figure out how to cover the lens properly.

When you create the new darks, please run this before doing so:

sudo systemctl stop allsky
> /var/log/allsky.log
sudo systemctl start allsky

then attach /var/log/allsky.log when you're done with darks. Also, please attach your lowest-temperature dark and your highest-temperature dark.

The commands above will clear out the log file so when you upload it, it'll be easier for me to search for things.

When your darks were saved, the mean brightness was 0.035 at 23 C and around 0.08 at 29 C, so I suggest putting the threshold back to 0.1 so you catch problems with the darks.

---

By any chance did your Pi change times?
These entries in the log file say the image began at 08:00:32 and ended at 14:53:57:

2025-08-11T08:00:32.125863+02:00 allsky-ladsi allsky[885]: STARTING EXPOSURE at: 2025-08-11 08:00:32   @ 30.0 sec
2025-08-11T14:53:57.397509+02:00 allsky-ladsi allsky[885]:    *** WARNING: Time to take exposure (24805.3 sec) differs from requested exposure time (30.0 sec) by 24774.9 sec, threshold=6.0 sec
2025-08-11T14:53:57.399129+02:00 allsky-ladsi allsky[885]:   > GOT IMAGE @ mean 0.992, gain 50.

Thanks - Eric

[laheller]

That's easy to do. My first set of darks were anything but dark. It took me a while to figure out how to cover the lens properly.

@EricClaeys
Can you maybe give me some hints, how you cover the lens? I have a setup like this.

BR,

Ladislav

[EricClaeys]

My camera is on the top of a 5-meter pole so take it down when I do darks. I put it inside a dark room at night, then covered it with dark blankets.
Can you easily take yours off the tripod? People often put the Pi in the freezer for a while, then take it out and quickly cover it and start taking darks. This way they'll get darks at below 0 C and up to room temperature (maybe slightly higher if the Pi is in an enclosed box with no air circulation).

It's going to be hard to cover it when outside, but if you can take it off the tripod or lower it, perhaps you can put a thick box over it (if you have one that big). Then blankets on top of that.

If you are handy, you could try making a box out of wood or aluminum that has one side open and is just slightly larger than the dome. You could then put it on the case, covering the dome. Then something to keep light from getting in where the box meets the case - a dark blanket may be enough.

You can also try asking on the Allsky Facebook page - there are a lot of people there and I'm sure someone has a better answer than mine.

[laheller]

@EricClaeys
It's the same box as on the linked picture but not on a tripod. It's mounted on the wall of my garden house. But actually I can easily take it off. Probably I will put it INTO the garden house & cover with dark blanket.

Thanks for suggestions! Once done, I will come back with new the darks & allsky.log so you can analyze them.

BR,

Ladislav

[EricClaeys]

@laheller, good luck. I hope your darks make a noticeable improvement.

With a noisy ASI120 camera it will probably be necessary to have darks close to the same temperature as your images, including in the winter. That's why a lot of people put the camera in the freezer first.