Server Failure
Published: 2020-05-05 - By: gdm
Last updated on: 2020-05-05
Last updated on: 2020-05-05
visibility: Public - Headline
TroveStar crashed on May 1st. The error seems to be a corrupted filesystem. This was likely an error at the ISP, though we are not sure of the cause. The result was that the server could not access any of its files and hence every request to the TroveStar website was returning a "Forbidden" error message.
We invoked a ticket with the hosting provider, godaddy.com, early on Friday morning. They told us on the phone they would work on it an repair it by Monday (May 4th) at 9am. This seemed like a long time, but we had no choice if we wanted to use their professional support team. Furthermore they demanded a $79 payment to look at the issue before they even knew what caused it.
Monday morning rolled around and nothing had been done. We got on the phone once again with godaddy and they told us there was nothing they could do and we were out of luck. To be honest, I was shocked. The tier one support person spent two hours with me on the phone pleading with two different service desks to help me repair the server, but they said no, and even refunded me the $79 they had taken on Friday.
So we rolled up our sleeves and started a server rebuild from backups. We pay for a backup system which should have made this easy but it simply is not. When you 'rebuild' a server, all of your settings are lost. Also, it takes a long time to load all the backup data onto a new server, and long uploads often fail part way through.
Fortunately we had an offsite backup of the data from early morning on April 30th. The database backup has flaws and not 100% of the data restored, but there is some redundancy and in most cases we can retrieve by hand the little data that did not restore correctly through the automated script.
The image database is proving more troublesome as it is simply so large now. It partially restored last night (May 4th) but failed at a random point. Again, we have an offsite backup of all image data which can be installed manually if the automated restore fails a second time.
We apologize for any disruption this has caused and appreciate your patience and support. TroveStar Admin Team
We invoked a ticket with the hosting provider, godaddy.com, early on Friday morning. They told us on the phone they would work on it an repair it by Monday (May 4th) at 9am. This seemed like a long time, but we had no choice if we wanted to use their professional support team. Furthermore they demanded a $79 payment to look at the issue before they even knew what caused it.
Monday morning rolled around and nothing had been done. We got on the phone once again with godaddy and they told us there was nothing they could do and we were out of luck. To be honest, I was shocked. The tier one support person spent two hours with me on the phone pleading with two different service desks to help me repair the server, but they said no, and even refunded me the $79 they had taken on Friday.
So we rolled up our sleeves and started a server rebuild from backups. We pay for a backup system which should have made this easy but it simply is not. When you 'rebuild' a server, all of your settings are lost. Also, it takes a long time to load all the backup data onto a new server, and long uploads often fail part way through.
Fortunately we had an offsite backup of the data from early morning on April 30th. The database backup has flaws and not 100% of the data restored, but there is some redundancy and in most cases we can retrieve by hand the little data that did not restore correctly through the automated script.
The image database is proving more troublesome as it is simply so large now. It partially restored last night (May 4th) but failed at a random point. Again, we have an offsite backup of all image data which can be installed manually if the automated restore fails a second time.
We apologize for any disruption this has caused and appreciate your patience and support. TroveStar Admin Team
Click Here for all Articles on TroveStar News
The copyright for this article is held by the author.
The copyright for this article is held by the author.
rights were granted to: TroveStar LLC