PDA

View Full Version : Disaster Recovery and Risk Management



Spiny Norman
07-02-2005, 05:02 PM
I had a scenario arise at work today in the 1/3 of the organisation that is not under my direct control IT-wise. One of their main servers died ... turns out that the company that the IT had been outsourced to (who shall remain nameloess) and who were responsible for ensuring that the server's data was backed up, hadn't set up backups and they also have no "plan B".

Karthick, am wondering if you're able to discuss the Chess Chat situation vis-a-vis a disaster recovery situation. For example:

- what happens if a key server dies?

- what happens if something happens to you? is anyone in a position to take over in an administrative sense?

skip to my lou
07-02-2005, 05:34 PM
Karthick, am wondering if you're able to discuss the Chess Chat situation vis-a-vis a disaster recovery situation. For example:

- what happens if a key server dies?

The chesschat server has nightly backups to a secondary hard drive, and weekly FTP backups to a backup only server.

Raid 1 is common on the enterprise SCSI servers, and now even common with SATA servers.

The standard setup we do for most business servers are 3 drives SATA or SCSI, two on Raid 1 and one for nightly backup, along with 50GB of internal network backup space and external network backup space can be purchased at something like $1 per GB.

I lost track of the amount of times people have screwed their web hosting account beyond recognition and the only way to restore it is from a nightly backup.

The nightly backups also backup critical system config files.

Sometimes we get the extra mission critical servers that require load balanced & failover clusters, with Raid 5 on SCSI 15K RPM drives, and they are a pain to restore.


- what happens if something happens to you? is anyone in a position to take over in an administrative sense?

Not for the chesschat forum, no.

I have heard of at least one company that implemented Raid 1, but unfortunately they didn't know when the raid controller failed and it stopped writing to the second hard drive. When the first hard drive went bust they pretty much lost a few months of work.

Spiny Norman
07-02-2005, 06:17 PM
- what happens if something happens to you? is anyone in a position to take over in an administrative sense?

Not for the chesschat forum, no.

I read that as:

* if the server dies, someone else is in a position to recover it, even if you are unavailable for some reason; and

* if something happens to you (heaven forbid!) then there is nobody primed to pick up the admin role

Did I interpret that correctly? In terms of dealing with that last item, perhaps some key information could be documented (e.g. basic server and baord software concepts, key technical contacts for the servers and board software, IDs and passwords where appropriate, etc)? That information could be lodged somewhere trustworthy perhaps, just in case?

I just realised yesterday that I am spending an increasing amount of my life here (sad, isn't it?) as my committment to chess grows. It seemed therefore to be prudent that I consider investing some time and perhaps $$$ in helping ensure that this resource continues to prosper.

skip to my lou
07-02-2005, 07:16 PM
Yes you did interpret that correctly. I am not sure about a backup plan such as that. I have never really thought about it for chesschat.

At the moment the server costs me $279 USD / Month, and is still not fast enough to run CKN. What would be ideal is a cluster of Dual Xeons, with maybe one highly powerful Dual Opteron for the MySQL. When I run ChessLib w/ say few million games and creating a few simultaneous searches, the server comes down to a screeching halt. Perhaps it just needs more RAM and maybe Raid 5 but at the moment I can't spend much more.

I should probably add that ChessLib is designed to search and return results in less than 0.01 seconds, so it is not sequential like chessbase etc.

Spiny Norman
08-02-2005, 05:53 AM
Yes you did interpret that correctly. I am not sure about a backup plan such as that. I have never really thought about it for chesschat.

At the moment the server costs me $279 USD / Month, and is still not fast enough to run CKN. What would be ideal is a cluster of Dual Xeons, with maybe one highly powerful Dual Opteron for the MySQL. When I run ChessLib w/ say few million games and creating a few simultaneous searches, the server comes down to a screeching halt. Perhaps it just needs more RAM and maybe Raid 5 but at the moment I can't spend much more.

I should probably add that ChessLib is designed to search and return results in less than 0.01 seconds, so it is not sequential like chessbase etc.

Can you run the searches sequentially (serially) rather than in parallel? Drop the requests into a DB table, give the user a link to the soon-to-be-created results page, then have a daemon that picks up the requests and processes them one-by-one at a low-ish priority and builds the results page(s)? That would take a little longer to set up, but would save you lots of $$$ on the hardware front.

ursogr8
08-02-2005, 02:59 PM
<snip>
At the moment the server costs me $279 USD / Month, and is still not fast enough to run CKN. <snip>

We locally got a quote of $15.00/month at BBoardsR_US.xxx, but that was only for the bulletin board facility.
I guess your features like the chess playing library site in competition with ChessBase are occupying the extra resources, and incurring the cost increase from $15/month to $279/month.
And therefore obviously your product is much more attractive for eventual sale.

Do the potential buyers, that you mentioned, ask about the tone of posters? I would guess that no-one would want to buy into too much controversy.

regards
starter

skip to my lou
08-02-2005, 04:54 PM
We locally got a quote of $15.00/month at BBoardsR_US.xxx, but that was only for the bulletin board facility.

Even for the BB, good luck paying $15 / Month for a BB that has 50-90 people constantly online. This forum eats up bandwidth + cpu like anything in peak times.

Btw, it's probably on a low powered machine if it's in Australia. Mostly in Australia, web hosting companies use P4's, because P4's here cost as much as super high power dual processor servers in the U.S.

Difference between AU & US is a few hundred milliseconds.


Do the potential buyers, that you mentioned, ask about the tone of posters? I would guess that no-one would want to buy into too much controversy.

I'm not selling it, ever. I got about 3 offers over the last 4 years and I have declined all of them.

skip to my lou
08-02-2005, 05:01 PM
Can you run the searches sequentially (serially) rather than in parallel? Drop the requests into a DB table, give the user a link to the soon-to-be-created results page, then have a daemon that picks up the requests and processes them one-by-one at a low-ish priority and builds the results page(s)? That would take a little longer to set up, but would save you lots of $$$ on the hardware front.

Yes possible. Need to look into it though, I first need to learn a bit about MySQL and how to optimize it.