Tom Posted May 22, 2018 Posted May 22, 2018 We have written this post to provide an explanation of an issue some of you may have recently experienced. Due to a combination of circumstances for about 400 Tapestry accounts, some or all photos and videos added to observations made between May 16th and May 21st were deleted in error. We really apologise if you have been affected by this issue and we can assure you we are working hard on restoring this media. The cause of this was a mistake at our end, no data has been incorrectly shared. We believe that almost all the data is on our backup systems. We are working to recover the data from our backup systems in waves. Some of you may find your missing media has already been restored to your account, but it may take up to a week to return it all. Observations that have been changed since the media was added will not have the photos or videos re-uploaded. This is to avoid duplicating images that have been re-added to observation. It will also prevent the incorrect sharing of images if the observation was approved without the image being present. Once we have finished the restoration we will let you know. If you would then like us to try and restore media to observations that you have edited, please contact us by emailing customer.service@eyfs.info. Updates Wednesday 23 May 16:33. The restore from backups is progressing. About 600 observations still need their media restored. These should be complete by the end of the week. Thursday 24 May 14:30. About 150 observations still need their media restored. These should be complete by the end of tomorrow. We will have lists of observations where the media was not replaced because the observation was subsequently updated. We will restore those bits of media if you ask us to. Tuesday 29 May 16:43. Settings that have had all their media restored will have a banner telling them so. We are preparing the lists of observations that couldn't have their media restored because they had been edited. Wednesday 30 May 16:25. If you log in to the web version of Tapestry you should now see a banner that will tell you if we have not restored some media because the observation was edited or put in journal after the media may have gone missing. We can provide a list of those observations and/or restore the media if you email customer.service@eyfs.info. A technical explanation of what went wrong We have a deletion process that runs in the background, deleting media that it doesn't think is associated with an observation. This can happen when people start to upload an observation but then, for whatever reason, stop the upload before it completes. On Wednesday 16th of May we started to make a background change to our systems in preparation for some new features. Unfortunately, making this change started to use an unusually large amount of memory on our database servers, so we had to stop. This left our system in an inconsistent state, which we did not appreciate would cause a problem. The issue was that in some cases observations were being added without the new change, which the deletion process was now expecting to be present. This meant that, in those cases, the deletion process did not correctly link media with the observation it should be attached to, which meant it treated the media as unassociated with an observation and therefore deleted it. We have records of when the deletions occurred and we have a complete set of backups, so we believe that we should be able to restore almost all the media incorrectly deleted. Unfortunately, restoring from those backups can be a slow process, so it may take us some time to complete. Please do get in touch with us after a week though if any of you notice any photos and/or videos are still missing with the titles of the affected observations and when they were added and we can investigate for you. You can do this by emailing customer.service@eyfs.info.
Recommended Posts