Maybe we should send a message to GOA?

[Xest]

If this is true then I am amazed at this. Mythic clustered 21 servers when they first clustered, and they have since made further clustering changes to make some 4-server clusters. In all that work none of the server databases were corrupted (as far as I know) and there certainly hasn't been any down time beyond the few couple of hours for patches and changes.

If database changes for clustering just two servers is one of the reasons for this problem then why has it only happened on the euro servers this particular time, and not on any of the clustering done on the US servers?

Exactly, essentially it could be as simple as whoever was making the database changes making a slip and changing the wrong field or something similar. Again though it seems to come down to good practice, why were such devestating changes made to a live server? When Mythic were testing clustering they brought up a new test server - Hector to cluster it with Pendragon, the new server was an exact copy of one of the live servers, I beleive it was Palomides or Percival.

It would surely make sense to make a copy of one of the servers to emulate clustering with Gorre or another copied server and then if it works copy the fully tested and working version over instead of breaking the live server from the off. Did GOA do this? Emulate the changes internally on a copy of a live server? If so at least we can rule out lack of a proper system of testing - something which has plagued GOA heavily in the past.

Really just don't see how Mythic can be blamed when their servers run so flawlessly in comparison.

[Cryn]

Requiel said (here ), that the problem was likely due to added databases field for clustering. This was a big change to the database.

I hadn't seen that post. I don't get much time on forums these days so can't read the longer threads, even when I start them sometimes. Thanks for pointing it out.

As I've kinda said in another thread, this looks like a clustering-related balls up to me and as such I am guessing could have been avoided. I realise a lot of people will dismiss this as irrelevant blame-game stuff, but it does highlight that GOA need to improve.

As Briannon said, Mythic clustered a lot more servers with a lot less trouble. I think GOA should be looking at the differences between their work practices and Mythic's and seeing if there is anything in it that could lead them to better quality of service.

True, people can say "Oh well, clustering is done now and we don't have this risk in our future" but there's bound to be other work as DAoC continues to evolve and change and it would be folly of the worst kind to not come out of this last problem with some lessons learned.

[Lieva]

Addiction.

and btw requiel where does this "Sword of Pwn" drop I want one !!!

whos addicted?

i just dont feel safe playing on a server when i can spend 2yrs + perfecting it (like people have done) for it to get corrupted and whiped overnight without adiquate reasoning therefore my trust has been shaken completly.

[Requiel]

For the record I didn't say that the problem was likely due to the database changes for clustering, I said that there had been changes and they may have contributed to it. I don't know what the actual reason was as I said in that post.

[Xest]

For the record I didn't say that the problem was likely due to the database changes for clustering, I said that there had been changes and they may have contributed to it. I don't know what the actual reason was as I said in that post.

I think most people's worry is that when you don't know or at least can't tell people what the problem is they're worried that no one in GOA really knows what caused it and are hence worried that it could just happen all over again - to cut Sharkiths 5000 page posts short, what has GOA done to ensure this severe a problem wont happen again? Or to put it another way, why should players keep on paying and playing when they feel they could potentially just lose everything all over again?

[Succi]

why dont you just let mythic take care of the servers from now on?

=]

[Cyfr]

I think this is a good message. Crafted by my good friend Dan.

[Requiel]

I think most people's worry is that when you don't know or at least can't tell people what the problem is they're worried that no one in GOA really knows what caused it and are hence worried that it could just happen all over again - to cut Sharkiths 5000 page posts short, what has GOA done to ensure this severe a problem wont happen again? Or to put it another way, why should players keep on paying and playing when they feel they could potentially just lose everything all over again?

As I've said before, I don't know what caused the initial problem. What has happened since is that we've written our own backup system which runs in tandem with the Mythic routines. This will ensure that we won't ever be in the situation of having no viable backups again.
I don't think it's possible to guarantee 100% that you will never suffer any kind of catastrophic failure, software errors and hardware failures can always happen no matter how stringent your systems are, however there will never be a situation where we are having to talk people out of a complete database wipe again.

Exactly, essentially it could be as simple as whoever was making the database changes making a slip and changing the wrong field or something similar. Again though it seems to come down to good practice, why were such devestating changes made to a live server? When Mythic were testing clustering they brought up a new test server - Hector to cluster it with Pendragon, the new server was an exact copy of one of the live servers, I beleive it was Palomides or Percival.

It would surely make sense to make a copy of one of the servers to emulate clustering with Gorre or another copied server and then if it works copy the fully tested and working version over instead of breaking the live server from the off. Did GOA do this? Emulate the changes internally on a copy of a live server? If so at least we can rule out lack of a proper system of testing - something which has plagued GOA heavily in the past.

Really just don't see how Mythic can be blamed when their servers run so flawlessly in comparison.

The changes to the databases in preparation for clustering started over two months ago. A lot of the server downtime over June, July and August was to put in place many of the changes to the game binaries, the network and the databases in preparation for clustering. We had a test cluster running for about a month (we have four test servers in total) using various databases and in several languages. If any of you happen to play on the French servers, you'll know that the clustering of Orcanie and Carnac was very smooth indeed, certainly there was nothing to indicate problems with our implementation.

[Xest]

As I've said before, I don't know what caused the initial problem. What has happened since is that we've written our own backup system which runs in tandem with the Mythic routines. This will ensure that we won't ever be in the situation of having no viable backups again.

I think that's exactly the kind of comment everyone has been looking for to reassure them that their characters are at least somewhat safer now

[Ovi]

[quote="Xest"]I think that's exactly the kind of comment everyone has been looking for to reassure them that their characters are at least somewhat safer now ]


Indeed it does go a long way to feeling safer. However there is still the element of Mythic's backup wasn't good enough, does that mean US customers should be worried about their characters, was it specific to the Eu servers, do Mythic supliment their own backups on their servers?