Today, 01:17 PM #1
Xalin
[GOA] English Servers GM
Join Date: 2nd Nov 2004
[INFO] Prydwen down
--------------------------------------------------------------------------------
Update:
Quite possibly the least informative update since the creation of updates:
I don't have any new news. I just spoke to our techies and they're waiting for the Mythic techies to get into work. I might have some more news in an hour or two.
Last thing I heard was that we're still trying to clean up the database and that this will be a monster job.
As predicted I'll keep this thread locked so the updates don't drown.
Feel free to continue the discussion here
If you have any questions (don't take the piss with this) pm them to me. I will NOT respond to your PM's but I might put your question and answer in the next update.
__________________
English Servers GM (rr1l0, ml0 but at least I'm lvl 100 )
Dark Age of Camelot Community Website
How to contact us
Prydwen-down (Information thread)
From FH :
"Q: Have GoA/Mythic had technical personell working around the clock since server shutdown wednesday last week, trying to solve the Prydwen problem?
A: Not around the clock litterally, as in 24/7, but I can assure you we have worked on it as much as we could and it continues to be our biggest priority."
Such political language, means all and nothing
"Q: Have GoA/Mythic had technical personell working around the clock since server shutdown wednesday last week, trying to solve the Prydwen problem?
A: Not around the clock litterally, as in 24/7, but I can assure you we have worked on it as much as we could and it continues to be our biggest priority."
Such political language, means all and nothing
Gahn LoneWolf Celt Seraph Tiarna Prydwen
Na Fianna Dragun
Gahn Lonewolf Guardian [TDA] on Gunnar's Hold
Na Fianna Dragun
Gahn Lonewolf Guardian [TDA] on Gunnar's Hold
-
Ankh Morpork
I know some of this has already been discussed but I thought I might copy/paste it from FH anyway.
---------------------------------
Xalin
[GOA] English Servers GM
[INFO] Prydwen down
Update:
Quite possibly the least informative update since the creation of updates:
I don't have any new news. I just spoke to our techies and they're waiting for the Mythic techies to get into work. I might have some more news in an hour or two.
Last thing I heard was that we're still trying to clean up the database and that this will be a monster job.
As predicted I'll keep this thread locked so the updates don't drown.
Feel free to continue the discussion here
If you have any questions (don't take the piss with this) pm them to me. I will NOT respond to your PM's but I might put your question and answer in the next update.
Q: The day Prydwen was down I was supposed to pay my house. Will it still be there when the server comes back online?
A: When the server comes back up the game time will only have progressed several minutes or less. So it's like time stood still so your house will still be there. If your house was due to be repossessed 2 hours after it was shutdown it will now be repossessed 2 hours after the server comes back online.
Q: Is it a coincidence that this bug happened right after a 100 characters were banned?
A: Yes it is.
Q: Have GoA/Mythic had technical personell working around the clock since server shutdown wednesday last week, trying to solve the Prydwen problem?
A: Not around the clock litterally, as in 24/7, but I can assure you we have worked on it as much as we could and it continues to be our biggest priority.
Q: Will there be a "grace period" between Prydwen coming back up and clustering Prydwen and Excal
A: Yes. As it stands now we will be able to bring the database back almost completely but there might be some work left and untill we're sure Prydwen is good again we won't cluster.
Sorry for the questions I couldn't reply to. Most were "what if" questions and I simply can't afford to fuel rumors by replying to possible scenario's people come up with.
And sorry for not responding to the most asked question: "When will Prydwen be back up" ... trust me I'd reply to it if I knew but then again I'd post it as soon as I found out so you won't have to ask me again.
[GOA] English Servers GM
About the questions I've been getting via PM's:
Thank you for your questions and thank you for your support and continued patience. But...
Most of the questions are "what if" questions and most of those are about a rollback.
I will NOT reply to these questions as it's silly to reply to questions concerning a scenario that might or may not play out at all. You'll get pissed or happy about answers that might be completely unrelated to the reality of what's going to happen. I simply don't have time to go talk to our people about possible outcomes off this they are way to busy making sure we get a good outcome in the first place.
Right now it's unlikely we'll roll back more then a couple of hours (last backup before we brought the server down). And we will simply not decide our policies on returning items or compensating downtime untill we know exactly when the server will be back up and in what state.
Sorry to all those people that want answers to those particular questions. As soon as we have discussed this internally I will make it my priority to let you know exactly what our policies regarding return of items etc will be.
Feel free to keep sending me your PM's. If I have enough questions I can respond to I'll put up a new update.
---------------------------------
---------------------------------
Xalin
[GOA] English Servers GM
[INFO] Prydwen down
Update:
Quite possibly the least informative update since the creation of updates:
I don't have any new news. I just spoke to our techies and they're waiting for the Mythic techies to get into work. I might have some more news in an hour or two.
Last thing I heard was that we're still trying to clean up the database and that this will be a monster job.
As predicted I'll keep this thread locked so the updates don't drown.
Feel free to continue the discussion here
If you have any questions (don't take the piss with this) pm them to me. I will NOT respond to your PM's but I might put your question and answer in the next update.
Q: The day Prydwen was down I was supposed to pay my house. Will it still be there when the server comes back online?
A: When the server comes back up the game time will only have progressed several minutes or less. So it's like time stood still so your house will still be there. If your house was due to be repossessed 2 hours after it was shutdown it will now be repossessed 2 hours after the server comes back online.
Q: Is it a coincidence that this bug happened right after a 100 characters were banned?
A: Yes it is.
Q: Have GoA/Mythic had technical personell working around the clock since server shutdown wednesday last week, trying to solve the Prydwen problem?
A: Not around the clock litterally, as in 24/7, but I can assure you we have worked on it as much as we could and it continues to be our biggest priority.
Q: Will there be a "grace period" between Prydwen coming back up and clustering Prydwen and Excal
A: Yes. As it stands now we will be able to bring the database back almost completely but there might be some work left and untill we're sure Prydwen is good again we won't cluster.
Sorry for the questions I couldn't reply to. Most were "what if" questions and I simply can't afford to fuel rumors by replying to possible scenario's people come up with.
And sorry for not responding to the most asked question: "When will Prydwen be back up" ... trust me I'd reply to it if I knew but then again I'd post it as soon as I found out so you won't have to ask me again.
[GOA] English Servers GM
About the questions I've been getting via PM's:
Thank you for your questions and thank you for your support and continued patience. But...
Most of the questions are "what if" questions and most of those are about a rollback.
I will NOT reply to these questions as it's silly to reply to questions concerning a scenario that might or may not play out at all. You'll get pissed or happy about answers that might be completely unrelated to the reality of what's going to happen. I simply don't have time to go talk to our people about possible outcomes off this they are way to busy making sure we get a good outcome in the first place.
Right now it's unlikely we'll roll back more then a couple of hours (last backup before we brought the server down). And we will simply not decide our policies on returning items or compensating downtime untill we know exactly when the server will be back up and in what state.
Sorry to all those people that want answers to those particular questions. As soon as we have discussed this internally I will make it my priority to let you know exactly what our policies regarding return of items etc will be.
Feel free to keep sending me your PM's. If I have enough questions I can respond to I'll put up a new update.
---------------------------------
-
Ankh Morpork
Xalin
[GOA] English Servers GM
Update:
You might have noticed that Prydwen is no longer on the server list.
This does NOT mean Prydwen got (or will be) deleted or something. It's just a precaution. We'll be bringing the server up for internal testing soon and we wouldn't want people to accidently connect to it while it's open.
Some Questions and Answers:
Q: How much does GOA depend on Mythic to fix this problem ?
A: I don't know exactly how much Mythic is doing and how much we are doing but I do know that we have been in very close and almost constant contact with Mythic since the beginning of this problem.
Q: Will we get compensated in free days?
A: I can't answer this question simply because (like the other what if questions) we haven't decided this. However I deem it quite likely and if we do it will be to accounts that were open when the server went down.
Again I'd like to thank you for your patience and support. I expected to get a lot of "problematic" PM's when I asked you to send me your questions but most of your questions are very reasonable and justified. Keep sending them and I'll try to answer as many as possible in the next update.
We realise this is a very annoying situation and we realise that you are very frustrated by now. Trust us frustration is an emotion we are all very familiar with at the moment. Please remember that we are trying our best to solve this problem as quick and as good as possible.
-----------------------------
From Freddy's House
/Ankh
[GOA] English Servers GM
Update:
You might have noticed that Prydwen is no longer on the server list.
This does NOT mean Prydwen got (or will be) deleted or something. It's just a precaution. We'll be bringing the server up for internal testing soon and we wouldn't want people to accidently connect to it while it's open.
Some Questions and Answers:
Q: How much does GOA depend on Mythic to fix this problem ?
A: I don't know exactly how much Mythic is doing and how much we are doing but I do know that we have been in very close and almost constant contact with Mythic since the beginning of this problem.
Q: Will we get compensated in free days?
A: I can't answer this question simply because (like the other what if questions) we haven't decided this. However I deem it quite likely and if we do it will be to accounts that were open when the server went down.
Again I'd like to thank you for your patience and support. I expected to get a lot of "problematic" PM's when I asked you to send me your questions but most of your questions are very reasonable and justified. Keep sending them and I'll try to answer as many as possible in the next update.
We realise this is a very annoying situation and we realise that you are very frustrated by now. Trust us frustration is an emotion we are all very familiar with at the moment. Please remember that we are trying our best to solve this problem as quick and as good as possible.
-----------------------------
From Freddy's House
/Ankh
Okay, time for a few misconceptions to be cleared up.
Why didn't we work at the weekend?
Our server staff were working at the weekend. Unfortunately they are limited in what how much they can usefully do without input from Mythic devs who don't work weekends.
Is Prydwen going to be deleted?
No.
Why hasn't there been updates about this problem on the website.
Every single working day since the shutdown there has been news on the official site about this issue with the exception of yesterday. We aren't likely to be able to give you an update till the afternoons anyway as Mythic staff don't get to their office until 3pm our time so until we've had a chance to talk to them, we can't really give you any useful updates.
Will there be some form of compensation for lost time?
Yes. Exactly what it will be is yet to be decided however.
Why does it take so long to fix?
On Wednesday we could just have slapped the last known good database on the server and carried on with only a ~1hr rollback. However without finding the root cause and fixing that, there was no guarantee that we wouldn't encounter exactly the same problem again and possibly suffer more damage in the meantime. Our priorities were to limit the damage done and fix the issue as fast as possible. This is why Prydwen was taken offline and why we are concentrating all our resources on getting it back up in a healthy state.
I lost items/quests/etc before the server shutdown, what will happen to those?
If there's a rollback then I'm afraid you'll lose anything you did after the timestamp on the good database. If you lost stuff that you had before this point, then you should submit the details to RightNow if you haven't already and we'll look into it and restore what we can. As far as it is possible to fix things we will do so.
Is Prydwen a low priority server?
No. On a personal level, I and many of the English team have characters (in some cases our main characters) on Prydwen. We want the server back up as badly as you do.
Was the server hacked?
No. As far as we can tell it's a database corruption and not an attack or a duping hack. Duping is a side effect of the wonky database not the cause of it.
Who's fault is it?
Without knowing the exact cause of the problem (which I don't) it's not possible to say. After the first problems were reported on Tuesday evening, our techs ran thorough system checks to make sure the backup systems were working, there were no hardware issues, no disk-space problems etc. Everything we checked came back with a green light. We've been monitoring the other servers closely for any evidence of the problem occuring on them and so far it's solely a Prydwen problem. A database became corrupt and caused problems in related databases. Until we find out what caused the initial corruption however we really can't speculate on what is to blame.
Why didn't we work at the weekend?
Our server staff were working at the weekend. Unfortunately they are limited in what how much they can usefully do without input from Mythic devs who don't work weekends.
Is Prydwen going to be deleted?
No.
Why hasn't there been updates about this problem on the website.
Every single working day since the shutdown there has been news on the official site about this issue with the exception of yesterday. We aren't likely to be able to give you an update till the afternoons anyway as Mythic staff don't get to their office until 3pm our time so until we've had a chance to talk to them, we can't really give you any useful updates.
Will there be some form of compensation for lost time?
Yes. Exactly what it will be is yet to be decided however.
Why does it take so long to fix?
On Wednesday we could just have slapped the last known good database on the server and carried on with only a ~1hr rollback. However without finding the root cause and fixing that, there was no guarantee that we wouldn't encounter exactly the same problem again and possibly suffer more damage in the meantime. Our priorities were to limit the damage done and fix the issue as fast as possible. This is why Prydwen was taken offline and why we are concentrating all our resources on getting it back up in a healthy state.
I lost items/quests/etc before the server shutdown, what will happen to those?
If there's a rollback then I'm afraid you'll lose anything you did after the timestamp on the good database. If you lost stuff that you had before this point, then you should submit the details to RightNow if you haven't already and we'll look into it and restore what we can. As far as it is possible to fix things we will do so.
Is Prydwen a low priority server?
No. On a personal level, I and many of the English team have characters (in some cases our main characters) on Prydwen. We want the server back up as badly as you do.
Was the server hacked?
No. As far as we can tell it's a database corruption and not an attack or a duping hack. Duping is a side effect of the wonky database not the cause of it.
Who's fault is it?
Without knowing the exact cause of the problem (which I don't) it's not possible to say. After the first problems were reported on Tuesday evening, our techs ran thorough system checks to make sure the backup systems were working, there were no hardware issues, no disk-space problems etc. Everything we checked came back with a green light. We've been monitoring the other servers closely for any evidence of the problem occuring on them and so far it's solely a Prydwen problem. A database became corrupt and caused problems in related databases. Until we find out what caused the initial corruption however we really can't speculate on what is to blame.
-
Lieva
- Emerald Rider
- Posts: 5689
- Joined: Sat Feb 05, 2005 11:00 am
- Location: On the redundancy train to freedom :D
- Contact:
" wrote:http://forums.freddyshouse.com/showthread.php?t=170227[/url] time 4:58pm"]There will be news on the website today regarding Prydwen. The Mythic office only opened for the day a couple of hours ago and as you can imagine they need a little time today to get back to us with more info. Please don't expect official updates regarding this until late in the afternoon or early evening there simply isn't anything we can tell you until then.
Lievaordiea x Eldritch
Peonchants x Enchanter
Hibernia
Peonchants x Enchanter
Hibernia
-
Lieva
- Emerald Rider
- Posts: 5689
- Joined: Sat Feb 05, 2005 11:00 am
- Location: On the redundancy train to freedom :D
- Contact:
" wrote:[30-08-2005] Progress on Prydwen
The cause of the issues affecting Prydwen have now been identified. File corruption has been discovered in several hundred character inventories and we have been examining them one by one to purge the database, before beginning the tests and necessary checks preceding a server restart. This work explains the unusual period of server shutdown. There will naturally be compensation to Prydwen players for this unavailability and we are continuing to do everything that we can to get the server restarted as soon as possible. Thank you for your patience in the meantime.
Lievaordiea x Eldritch
Peonchants x Enchanter
Hibernia
Peonchants x Enchanter
Hibernia
-
Lieva
- Emerald Rider
- Posts: 5689
- Joined: Sat Feb 05, 2005 11:00 am
- Location: On the redundancy train to freedom :D
- Contact:
" wrote:Xalin:
------------------------
Update:
We're currently testing Prydwen internally and will continue to do this all evening. There will be news up on the official site later today as well.
Some Q&A:
I'm getting a lot of questions I already replied to in earlier Q&A's so if you sent in a question I'm not replying to have a look at the earlier posts it might already be answered.
Q: Once everything is sorted is there any chance of getting a detailed technical description of what went wrong?
A: I can imagine you'd like to hear this just out of curiosity but I don't think it's very likely we'll go into great detail. There is all kinds of agreements between Mythic and GOA about what we can talk about and what not and I think to play it safe we'll just won't speak of it much.
Q: Do you feel GoA is doing enough to inform people through the official channels?
A: Yes and no. I think we could do a lot better but I don't think it's nearly as bad as some people like to claim. I've already explained to you why information on the official site has been scarce and I stand by that explanation.
Q: Why is the broad playerbase not informed about GoA using FH as an official source of announcements?
A: Because FH isn't an official source of announcements. It's a forum a lot of our playerbase read and hence we've been using it to give you tidbits of information. Like I already said sometimes this information wasn't confirmed and ended up being plain wrong. We could never post information like that on our official site.
------------------------
" wrote:[31-08-2005] Maintenance on Thursday the 1st of September (morning)
A short maintenance operation will be carried out on Thursday morning to implement a patch. This patch aims to solve the problem of crashes occuring when consignment merchant listings were searched. The servers will be shut down at around 10.30 a.m. and should reopen before 12 noon.
although these are all quiotes by ankh he got the info from xalin thread on freddys house and from the goa web site" wrote:[31-08-2005] News about Prydwen
The database has been purged of the corrupted files that have been discovered and Prydwen was restarted today for internal testing. We will keep you informed of the results of these checks in the next news concerning the situation on the server. We apologise once more to all players who have been deprived of their characters since last week.
Lievaordiea x Eldritch
Peonchants x Enchanter
Hibernia
Peonchants x Enchanter
Hibernia
Prydwen news
News
Not the most informative post I know but we will have more definite information for you tomorrow. Expect to see an announcement regarding the reopening and compensation for players.News wrote:Tests are still continuing on Prydwen and we are busy double checking every database to make sure that the corruption we discovered has not affected other areas of the game. As you can imagine this is a very large job, but we hope you agree that the server should not be restarted until we can be absolutely sure of it's stability. We hope to be able to announce when the end of this process will be tomorrow. In the meantime we thank you for your continued patience.