This issue has been resolved and the small number of effected users have been informed via Email (Sunday 29th 2020)
This this the story of how a mutating DateTime object caused a bug in SnapShooter and caused backups for 42 hours to be incorrectly tagged.
It came to our attention last night (Saturday 28th) that daily backup tags have been incorrectly associated with backups. I have been at the computer since 4am this morning and can confirm the issue has been resolved.
You are receiving this email because you may have been affected.
Any backup after 2 pm UTC Friday the 26th may have been incorrectly tagged after creation, (a tag: daily, monthly, weekly, in this case only daily would have been affected).
You may find that a backup was taken at the right time, only for the scheduling system to delete it as it was not correctly tagged as daily. If you have backups more than once a day, you will find a different daily one tagged (3H later).
The cause:
On friday 2pm, a customer added a custom cron schedule, that was not UTC. (This is the first time someone has set cron but not used UTC)
The php library used to check if the cron expression should run, mutated the date.
This mutated date was passed to the tag assigning system
Backups from 2pm 26th to 8am 29th have been tagged incorrectly.
After the backup, the scheduling system would check how many daily backups you have and should have, and would have deleted the untagged backup.
The fix:
Cloning the time before passing it to the cron expression library. I have checked all over possible locations and can confirm fixed.
Next week I will be reinforcing testing around the scheduling system.
I am incredibly sorry. This bug has been in the system for months and has only just shown its head. If you have questions, please feel free to get in touch.
Regards
Simon