Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[TheBoy]

Apparently an old wives tale, she tells me.


Woke up this morning to a barrage of emails that OOF is not displaying pages. Opps. Poxy primary DB server had run out of disk space, and shut itself down to protect itself. Quick fix - allocate ext4 reserved amount, start it up so it works, then in background allocate more physical disk space, and reset ext4 reserved back to 5%.

Just before midday, whilst on the phone, everything goes quiet. *Everything*. Silence. Its amazing how used you get to the fans. UPS went offline, knocking out all 3 physical servers, switches, router, modems, phones, the lot. Bloody Opps  . Pop the old (shagged) batteries back in, just to get UPS started, and slowing bring up the hypervisors and VMs. Unsurprisingly, the 2 OOF database servers disagreed, so had to break the sync and set it back up, which takes an age, as MariaDB needs a full dump of the master copy.

Get most of the stuff up, so now my email starts working, email from big bro (who is on holibobs), his website has died. FFS  .  Turns out to be *exactly* the same issue as what happened to OOF overnight, the primary DB server ran out of disk space. Whats the chances of that   


He's hoping the year improves.

[VXL V6]

Whats the chances of that   

If it can happen it will happen.

Fortunately power down's in DC's or just a single rack have never happened to me.  [Andy whistles to himself and leaves via the nearest fire door as the maglocks seem to have become de energised.....]

[TheBoy]

Whats the chances of that   

If it can happen it will happen.

Fortunately power down's in DC's or just a single rack have never happened to me.  [Andy whistles to himself and leaves via the nearest fire door as the maglocks seem to have become de energised.....]

Fortunately, at work, we have a lot of resilience, so when it happens, its impact is minimal when we lose a datacentre.

For OOF, and my bro's setup over at his premises, such resilience is a luxury too far.  We are forced to have too many single points of failure for cost and technical reasons.  A network switch crash the other day highlighted the impact, poxy Netgear shite!

[STEMO]

Put a bigger disk in the DB server. Next.

[VXL V6]

Whats the chances of that   

If it can happen it will happen.

Fortunately power down's in DC's or just a single rack have never happened to me.  [Andy whistles to himself and leaves via the nearest fire door as the maglocks seem to have become de energised.....]

Fortunately, at work, we have a lot of resilience, so when it happens, its impact is minimal when we lose a datacentre.

For OOF, and my bro's setup over at his premises, such resilience is a luxury too far.  We are forced to have too many single points of failure for cost and technical reasons.  A network switch crash the other day highlighted the impact, poxy Netgear shite!

Yes, similar at work, DC and all lower levels of resilience.... It's just that someone also installed Netbotz to capture those oops moments, 'wasn't me me guv, honest'

[TheBoy]

Put a bigger disk in the DB server. Next.

Feeling flush? That's very generous of you.  Remember, you'll also need to do similar to the secondary database that maintains a replica.


(More disk space already allocated).

[STEMO]

Put a bigger disk in the DB server. Next.

Feeling flush? That's very generous of you.  Remember, you'll also need to do similar to the secondary database that maintains a replica.


(More disk space already allocated).

Would that be the answer? Wow, lucky guess 

[STEMO]

Put a bigger disk in the DB server. Next.

Feeling flush? That's very generous of you.  Remember, you'll also need to do similar to the secondary database that maintains a replica.


(More disk space already allocated).

Would that be the answer? Wow, lucky guess 

I'd have tried a second hand dvd. 

[TheBoy]

Put a bigger disk in the DB server. Next.

Feeling flush? That's very generous of you.  Remember, you'll also need to do similar to the secondary database that maintains a replica.


(More disk space already allocated).

Would that be the answer? Wow, lucky guess 

STEMO = Computer God

[STEMO]

Put a bigger disk in the DB server. Next.

Feeling flush? That's very generous of you.  Remember, you'll also need to do similar to the secondary database that maintains a replica.


(More disk space already allocated).

Would that be the answer? Wow, lucky guess 

STEMO = Computer God

Are you being sarcastic  there? 

[T.m.f]

In reality the hamster died again,maybe upgrade it to a guinea pig so lasts longer 

[Gaffers]

In reality the hamster died again,maybe upgrade it to a guinea pig so lasts longer 

I think it finally succumbed to TB's Xmas indulgence in brussel sprouts