| Author |
Topic |
|
Kristen
Test
22859 Posts |
 Posted - 2004-10-22 : 09:44:10
|
| Client's machine filled its disks today. They haven't given any thought to administration of their server ... Ho!Hum!However, there are a couple of things I would like to give thought to.Note that I am aware that the client should not be using Maintenance Plans and they should have Alerts in place - but they have gotta buy that effort, and that hasn't happened yet ...... so here's what I did:I stopped SQL, and put up the standard "Doing Maintenance - please come back later" holding page for their web site. That prevents all [or so I thought] access to the DB from the Web site.Then I deleted all the Transaction backup files that SQL Maintenance Plan had failed to do, and generally made 20GB of space available ...Then I restarted SQL and set about running the Maintenance Tasks for Backup (which had failed overnight - but no alerts in place).... so, some questions:During the Backup Task which I ran it occurred to me that the Scheduled Tasks (exchange data with other systems etc.) could run. Also the Maintenance Plan Transaction backups could run (does this matter?). And as it turned out the "404 page handler" did run, and had a login to the database [outside the scope of the Holding Page I had put up] whilst I was trying to run the "intergrity checks" part of the maintenance plan - another 15 minutes to re-run that baby after setting the database to "DBO Only"; that didn't help, something still prevented it being set to ReadOnly (nothing showing in the Current Activity though)There's probably a whole stack of should-do things in a situation like this. Perhaps set all DBs to "dbo-only". Perhaps disable the scheduled tasks.I'd appreciate some thoughts on what you do/would do in such circumstances.ThanksKristen |
|
|
MuadDBA
628 Posts |
Posted - 2004-10-22 : 13:56:38
|
| script the loginsdrop them all, except for SArestart serverrun maintenance checks, backups, etcrecreate logins using script |
 |
|
|
tkizer
Almighty SQL Goddess
38200 Posts |
Posted - 2004-10-22 : 14:15:37
|
| Really that's what you do when you run out of disk space?I simply clean up the space by deleting files or backup log with no_log then DBCC CHECKDB to make sure disk problems didn't cause corruption. I don't put the database into single user or dbo only. I only do this if DBCC CHECKDB reports problems. I also don't disable scheduled tasks.Tara |
|
|
|
Kristen
Test
22859 Posts |
Posted - 2004-10-22 : 23:55:10
|
| I guess I was worried that I'd be in trouble if the scheduled tasks a) ran OK but ) I later discovered the database had t be restored from backup (i.e. was damaged in some way). The scheduled tasks cause updates to "senders" logs [i.e. on remote machine] that the data was transfered OK, so its a bit of a bear to have to sort out to retransmit data if a database is restored to an earlier state.And any client applications re-connecting before I'm confident the machine is in Good Order is a PiTA too ...I was long some of the tasks took actually - its a decent server, and the DBs are 5GB, 3GB and 1.5GB - so not all that big.Anyways, it took me 2.5 hours to sort out - the bill to the client should wake them up to put some proper processes in place.Kristen |
|
|
|
derrickleggett
Pointy Haired Yak DBA
4184 Posts |
Posted - 2004-10-23 : 13:58:06
|
| What was wrong that it had to be restored from backup Kristen?MeanOldDBAderrickleggett@hotmail.comWhen life gives you a lemon, fire the DBA. |
|
|
|
Kristen
Test
22859 Posts |
Posted - 2004-10-24 : 10:49:26
|
| Nothing, as it turned out [which is how it should be of course!], but if the database checks had suggested that the DB was broken I would have restored ... rather than try to repair - (site was live and web clients were waiting ...)Kristen |
|
|
|
eyechart
Master Smack Fu Yak Hacker
3575 Posts |
Posted - 2004-10-24 : 11:01:08
|
| You should check your SQL error log to make sure that you didn't run out of space in your database datafiles, barring that you should be fine.I'm with tara on this, just clear the free space and keep going.-ec |
|
|
|
Kristen
Test
22859 Posts |
Posted - 2004-10-24 : 11:13:07
|
| Need to be able to charge the client plenty of $$$ though - so that they put a proper managed solution in place, we predict disk space etc. properly, and then it a) doesn't happen and b) doesn't interrupt my already-heavily-interrupted-day. Or night. :-)Kristen |
|
|
|
eyechart
Master Smack Fu Yak Hacker
3575 Posts |
Posted - 2004-10-24 : 19:46:51
|
quote:
Originally posted by Kristen
Need to be able to charge the client plenty of $$$ though - so that they put a proper managed solution in place, we predict disk space etc. properly, and then it a) doesn't happen and b) doesn't interrupt my already-heavily-interrupted-day. Or night. :-)
I assume you are just kidding here. Saving the day isn't enough?-ec |
|
|
|
jen
Master Smack Fu Yak Hacker
4110 Posts |
Posted - 2004-10-24 : 22:05:49
|
i perform:1. directory inventory - have the client "archive" files to free up space, if not enough proceed to #2, else end of work.2. backup database and log3. truncate log file4. shrinkfile logthen you can charge the clients with your alerts, backup, maintenance plans.  i guess users would normally assume that if a server was in place and working fine at that time, it will be like that forever. they fail to recognize the importance of a good maintenance plan.-- even if we don't have external clients, our users have the same dilemma over the other servers (not handled by yours truly), then asks me to "fix" their problem. being proactive is the answer, not waiting for disaster to happen. and it's not being heroic either if you help them out, it's just to make them stop bothering you. LOL  --------------------keeping it simple... |
|
|
|
Kristen
Test
22859 Posts |
Posted - 2004-10-25 : 01:11:39
|
| "I assume you are just kidding here. Saving the day isn't enough?"We have alerted the client to the fact that they have no maintenance contract in place, and they have made no proper provision for robust backups or disaster planning.And yet they expect they can pick up the phone and say "our server is down" and get an immediate full-team-turnout to fix the problem. The box is taking $20,000 a day of web orders - its rediculous the lack of respect that they have for it ...Kristen |
|
|
|
Kristen
Test
22859 Posts |
Posted - 2004-10-25 : 01:14:34
|
| "then you can charge the clients with your alerts, backup, maintenance plans."I've told them that needs doing, so far nothing.I have already sent a snotty email about the distinction between Support and Maintenance. Looks like I'll now have to say that emergency support can only be provided if our own database maintenance procedures are in place. You know ... things that check for disk space once in a while, that sort of thing <sigh>Kristen |
|
|
|
MuadDBA
628 Posts |
Posted - 2004-10-25 : 11:06:20
|
quote:
Originally posted by tduggan
Really that's what you do when you run out of disk space?I simply clean up the space by deleting files or backup log with no_log then DBCC CHECKDB to make sure disk problems didn't cause corruption. I don't put the database into single user or dbo only. I only do this if DBCC CHECKDB reports problems. I also don't disable scheduled tasks.Tara
Well, um, no. But, in Kristen's case, it looked like one of the goals was to get all the users out of the DB.And the difference between support and maintenance is confused by just about everyone, often intentionally :)The client should have some sort of paper that tells them WHAT support is, if not, you're on the hook for whatever they think it is up until the point when you have to put the client relationship on the line. You're not the first company to discover this issue :) |
|
|
|
Kristen
Test
22859 Posts |
Posted - 2004-10-25 : 12:31:09
|
| We have "incident packs" and all that jazz. And we define maintenance as "bug fixes, upgrades to new versions". And we have user training, train-the-trainer-training and top-up-training. They get credited an incident if an incident turns out to be a bug - but they have to own an "incident" with which to pay for the investigation to find out!Otherwise its Time&Materials ...I reckon the difference is pretty clear, and the client knows it, but they haven't quite got around to worrying about it. Which I think is worrying<g>Kristen |
|
|
|
MuadDBA
628 Posts |
Posted - 2004-10-25 : 14:35:38
|
| What's funny is that they will pay for the incident (probably not cheap) but forego the maintenance which would prevent the incident.The next time you need to respond to this, you should do the bare minimum necessary to get the server up and runnign again (ie clean up 5MB of space so the DB runs for an hour or so) and then make them pay for another incident as soon as it goes belly-up. They'll get the message pretty quick that they need to set up a standard maintenance package :) |
|
|
|
Kristen
Test
22859 Posts |
Posted - 2004-10-26 : 01:03:02
|
| Trouble is it interrupts our [already busy] schedule.I think it is better to get the client to sort this out properly for their own good, so I'm going down the "get proper maintenance in place or get someone else out of bed next time" route.Kristen |
|
|
|
|
What to do following unscheduled failure?