At work on Christmas Eve

Post by **VW337** » Mon Dec 24, 2007 2:06 pm

So I am at work on Christmas eve 1 of 3 peeps.

I was told to go home at like 11am. Unfortunately I have to build out a new machine for a power user and this is my only chance, grr.

Damn machine is so jacked up in the bios it took 4 loads of XP to finally get the boot screen to be viewable, the driver kept tanking on every setting in the BIOS. I am so unfond of HP.

So how many others are enjoying a hair pulling last day at work before the holidays, and bored outta there minds?

Post by **Wakeup** » Mon Dec 24, 2007 2:25 pm

RAISES HAND AS HIGH as a 5'2 Asian can....

Post by **fuzzysnuggleduck** » Mon Dec 24, 2007 2:32 pm

I'm sitting around trying to get and entire network worth of services moved over to a high-availability and thus redundant architecture using Linux-HA. On top of that, I'm retiring some old services and migrating their data into new services, which can be tricky (NIS->LDAP).

Basically, I'm duplicating the majority of our network services onto two machines that can automatically "take over" the services should the other server (or component thereof) fail. Right now, we have one server that runs nearly all critical network services and when it fails, about 200+ people can't work (most of which, I don't even directly support... go figure)

Despite being a long process, I'm making progress.

The tough part will come when everything appears to be working well in the test environment and I start migrating the service over to the new systems. There'll be some tricky shit with inter-VLAN routing, DNS lookups, DHCP services for all subnets, etc.

However, I won't be here too much longer today. I've made good progress on the very core services such as DHCP and DNS, so now I'm building up the LDAP junk and going from there.

At the very least, all this work will mean a stable, centralized, standard platform from which to build and fix all the other fringe services.

rlockwood · Post by **rlockwood** » Mon Dec 24, 2007 3:13 pm

had to go in and do year end cleanup today. includes sucking the disgusting year old mess of coolant, tramp oil and bacteria out of 6 machines and refilling them with fresh. Spraying them out with fresh, then just sweeping/mopping and general cleanup. Wasnt terrible, and 3.5 hours of work for 8 paid... not bad. Its actually nice knowing I get to come in next week to a completely cleaned shop...

Post by **VW337** » Mon Dec 24, 2007 3:55 pm

fuzzysnuggleduck wrote: Basically, I'm duplicating the majority of our network services onto two machines that can automatically "take over" the services should the other server (or component thereof) fail.

one server? We have one per arterial service.........~24 servers.

I hope you have more than just one server for that many users.

Have fun.

Post by **VW337** » Mon Dec 24, 2007 3:56 pm

Wakeup wrote:RAISES HAND AS HIGH as a 5'2 Asian can....

Is there a step stool involved or are we just guessing how high you can get your hand? I know what you are going through there, I miss retail but I in no way shape or form miss the hours....

Post by **Wakeup** » Mon Dec 24, 2007 4:35 pm

hehe ya no step stool involved....

Me-262 · Post by **Me-262** » Mon Dec 24, 2007 7:36 pm

You'll probably hate me. I went in and had breakfast with my team this morning..... adn then we all went home. I can sympathize, I've worked late on Christmas eve myself before.

Post by **fuzzysnuggleduck** » Mon Dec 24, 2007 9:20 pm

VW337 wrote:
fuzzysnuggleduck wrote: Basically, I'm duplicating the majority of our network services onto two machines that can automatically "take over" the services should the other server (or component thereof) fail.
one server? We have one per arterial service.........~24 servers.

I hope you have more than just one server for that many users.

Have fun.

We do have multiple servers but there's one core machine. It runs the DNS server, the DHCP server, routes between VLANs and also performs backup services. The reliance on this one machine is a primary motive behind failover services.

If this one server goes down, Internet services go down for the most part. There are other servers for linux and windows terminal services, number crunching, a small cluster, backup server, windows login server, two file servers and a machine mastering this little side network for Sun machines.

On top of that, it's an IBM p505 meaning it has an IBM POWER5 processor, which is all fine and dandy but all the great features of this server only slow down the boot process enormously (10 minutes!) but also make rebooting it a pain. It has no video or mouse/keyboard ports. You need a IBM terminal or a serial terminal to see what's going on and actually provide a decent amount of input for it to boot. So, when it crashes, and it has twice, it's a pain in the ass to rush over and get it booting again while thin clients are hanging and many others loose Internet access.

</geek>

Mastiff · Post by **Mastiff** » Tue Dec 25, 2007 9:38 am

Post by **VW337** » Wed Dec 26, 2007 9:44 am

fuzzysnuggleduck wrote:
VW337 wrote:
fuzzysnuggleduck wrote: Basically, I'm duplicating the majority of our network services onto two machines that can automatically "take over" the services should the other server (or component thereof) fail.
one server? We have one per arterial service.........~24 servers.

I hope you have more than just one server for that many users.

Have fun.
We do have multiple servers but there's one core machine. It runs the DNS server, the DHCP server, routes between VLANs and also performs backup services. The reliance on this one machine is a primary motive behind failover services.

If this one server goes down, Internet services go down for the most part. There are other servers for linux and windows terminal services, number crunching, a small cluster, backup server, windows login server, two file servers and a machine mastering this little side network for Sun machines.

On top of that, it's an IBM p505 meaning it has an IBM POWER5 processor, which is all fine and dandy but all the great features of this server only slow down the boot process enormously (10 minutes!) but also make rebooting it a pain. It has no video or mouse/keyboard ports. You need a IBM terminal or a serial terminal to see what's going on and actually provide a decent amount of input for it to boot. So, when it crashes, and it has twice, it's a pain in the ass to rush over and get it booting again while thin clients are hanging and many others loose Internet access.

</geek>

I get it now, it is an over burdened DC.