Omega Owners Forum

Please login or register.

Login with username, password and session length
Advanced search  

News:

Please check the Forum Guidelines at the top of the Newbie section

Pages: 1 [2] 3  All   Go Down

Author Topic: Who forgot to feed the hamster?  (Read 6714 times)

0 Members and 1 Guest are viewing this topic.

Doctor Gollum

  • Get A Life!!
  • *****
  • Offline Offline
  • Gender: Male
  • In a colds and darks puddleses
  • Posts: 30002
  • If you can't eat them, join them...
    • Feetses.
    • View Profile
Re: Who forgot to feed the hamster?
« Reply #15 on: 09 January 2017, 20:08:44 »

Need a new hamster wheel...

Thanks for the efforts getting it all sorted. It is genuinely appreciated  :-*
Logged
Onanists always think outside the box.

TheBoy

  • Administrator
  • *****
  • Offline Offline
  • Gender: Male
  • Brackley, Northants
  • Posts: 107026
  • I Like Lockdown
    • Whatever Starts
    • View Profile
Re: Who forgot to feed the hamster?
« Reply #16 on: 10 January 2017, 11:19:27 »

Sincere apologies from TB Towers about the 2 seperate outages yesterday.  Both outages were caused by both hypervisors losing site of their storage in fairly quick succession. Root cause why is as yet unknown, but a few days ago the hypervisors were going a bit mental trying to even out storage usage and latency - maybe this caused all the disks to go into meltdown.

Recovery times were extended, as a full storage and hypervisor restart takes several hours.


Automatic storage balancing currently disabled to see if we can get stability.
Logged
Grumpy old man

zirk

  • Omega Queen
  • *****
  • Offline Offline
  • Gender: Male
  • Epping Forest
  • Posts: 11443
  • 3.2 Manual Special Saloon ReMapped and LPG'd and
    • 3.2 Manual Special Estate
    • View Profile
Re: Who forgot to feed the hamster?
« Reply #17 on: 10 January 2017, 12:40:58 »

Sincere apologies from TB Towers about the 2 seperate outages yesterday.  Both outages were caused by both hypervisors losing site of their storage in fairly quick succession. Root cause why is as yet unknown, but a few days ago the hypervisors were going a bit mental trying to even out storage usage and latency - maybe this caused all the disks to go into meltdown.

Recovery times were extended, as a full storage and hypervisor restart takes several hours.


Automatic storage balancing currently disabled to see if we can get stability.

The Hyper Space Meltdown Unbalanced Stability thing sounds so much better than the TSSA Union and Eastern European Smiley Strike stuff that Mark came up with...  ;D

Either way, glad its sorted and much appreciated for your efforts again.  :y
Logged

STEMO

  • Guest
Re: Who forgot to feed the hamster?
« Reply #18 on: 10 January 2017, 13:26:02 »

Sincere apologies from TB Towers about the 2 seperate outages yesterday.  Both outages were caused by both hypervisors losing site of their storage in fairly quick succession. Root cause why is as yet unknown, but a few days ago the hypervisors were going a bit mental trying to even out storage usage and latency - maybe this caused all the disks to go into meltdown.

Recovery times were extended, as a full storage and hypervisor restart takes several hours.


Automatic storage balancing currently disabled to see if we can get stability.
https://youtu.be/Ac7G7xOG2Ag
Logged

TheBoy

  • Administrator
  • *****
  • Offline Offline
  • Gender: Male
  • Brackley, Northants
  • Posts: 107026
  • I Like Lockdown
    • Whatever Starts
    • View Profile
Re: Who forgot to feed the hamster?
« Reply #19 on: 10 January 2017, 13:52:04 »

Sincere apologies from TB Towers about the 2 seperate outages yesterday.  Both outages were caused by both hypervisors losing site of their storage in fairly quick succession. Root cause why is as yet unknown, but a few days ago the hypervisors were going a bit mental trying to even out storage usage and latency - maybe this caused all the disks to go into meltdown.

Recovery times were extended, as a full storage and hypervisor restart takes several hours.


Automatic storage balancing currently disabled to see if we can get stability.
https://youtu.be/Ac7G7xOG2Ag
Hypervisors = the clever hardware/software that allows multiple servers to run on a single physical server.  OOF is primarily made up of 5 (virtual) servers, not including all the infrastructure type stuff.  These 5 servers, along with several others, run across 2 hypervisors, which run on 2 HP Proliant physical servers.

Its called virtualisation/cloud, and its been all the rage for the past 5 years or so.

Downside is if a hypervisor starts to go wonky quite quickly, and doesn't have time to move its virtual machines to another, is a single failure can result in dozens of servers going offline at same time, which is what happened yesterday. Twice.  Although in the case yesterday, the 2 hypervisors that OOF is allowed to run on both died in realtively quick succession - so even if it did have time to move the virtual machines, it still would have died 20mins later anyway.


That better?
Logged
Grumpy old man

Andy B

  • Get A Life!!
  • *****
  • Offline Offline
  • Gender: Male
  • Bury Lancs
  • Posts: 39778
    • ML350 TDM SmartRoadster
    • View Profile
Re: Who forgot to feed the hamster?
« Reply #20 on: 10 January 2017, 14:00:18 »

....
That better?

Still no idea what you mean .......  ??? ??? ???  ;D
Logged

aaronjb

  • Guest
Re: Who forgot to feed the hamster?
« Reply #21 on: 10 January 2017, 14:06:45 »

Imagine it like passengers in a car.  The car is carrying the Finance Director, his deputy, the CEO and a sales girl.

If the sales girl dies they can all carry on.  If the FD dies, the deputy FD can take over.

If the car crashes, bursts into flames and kills them all, OOF goes down.  ;D
Logged

Doctor Gollum

  • Get A Life!!
  • *****
  • Offline Offline
  • Gender: Male
  • In a colds and darks puddleses
  • Posts: 30002
  • If you can't eat them, join them...
    • Feetses.
    • View Profile
Re: Who forgot to feed the hamster?
« Reply #22 on: 10 January 2017, 14:12:19 »

....
That better?

Still no idea what you mean .......  ??? ??? ???  ;D
There are two hamsters on shift at any one time. Halfway through the night shift, the diabetic hamster forgot to take its insulin and fell over, and his only shift partner noticed ten minutes later, and in a moment of panic, fell down the stairs, landing on his head.

The alarm was only raised when the dayshift arrived and couldn't get in, and in the ensuing chaos, they realised that management help was needed, (an admin password was required to reset the system), but unfortunately the boss was in Chamonix, and was unable to get back until late afternoon...

That any better?
Logged
Onanists always think outside the box.

Andy B

  • Get A Life!!
  • *****
  • Offline Offline
  • Gender: Male
  • Bury Lancs
  • Posts: 39778
    • ML350 TDM SmartRoadster
    • View Profile
Re: Who forgot to feed the hamster?
« Reply #23 on: 10 January 2017, 14:13:11 »

....
That better?

Still no idea what you mean .......  ??? ??? ???  ;D
There are two hamsters on shift at any one time. Halfway through the night shift, the diabetic hamster forgot to take its insulin and fell over, and his only shift partner noticed ten minutes later, and in a moment of panic, fell down the stairs, landing on his head.

The alarm was only raised when the dayshift arrived and couldn't get in, and in the ensuing chaos, they realised that management help was needed, (an admin password was required to reset the system), but unfortunately the boss was in Chamonix, and was unable to get back until late afternoon...

That any better?

 ;D ;D ;D ;D ;D much better!  :y
Logged

TheBoy

  • Administrator
  • *****
  • Offline Offline
  • Gender: Male
  • Brackley, Northants
  • Posts: 107026
  • I Like Lockdown
    • Whatever Starts
    • View Profile
Re: Who forgot to feed the hamster?
« Reply #24 on: 10 January 2017, 14:15:29 »

....
That better?

Still no idea what you mean .......  ??? ??? ???  ;D
There are two hamsters on shift at any one time. Halfway through the night shift, the diabetic hamster forgot to take its insulin and fell over, and his only shift partner noticed ten minutes later, and in a moment of panic, fell down the stairs, landing on his head.

The alarm was only raised when the dayshift arrived and couldn't get in, and in the ensuing chaos, they realised that management help was needed, (an admin password was required to reset the system), but unfortunately the boss was in Chamonix, and was unable to get back until late afternoon...

That any better?
aaronjb's anology was closer ;D
Logged
Grumpy old man

Doctor Gollum

  • Get A Life!!
  • *****
  • Offline Offline
  • Gender: Male
  • In a colds and darks puddleses
  • Posts: 30002
  • If you can't eat them, join them...
    • Feetses.
    • View Profile
Re: Who forgot to feed the hamster?
« Reply #25 on: 10 January 2017, 14:22:50 »

Given his profession, I should think so too :P

 ;D
Logged
Onanists always think outside the box.

TheBoy

  • Administrator
  • *****
  • Offline Offline
  • Gender: Male
  • Brackley, Northants
  • Posts: 107026
  • I Like Lockdown
    • Whatever Starts
    • View Profile
Re: Who forgot to feed the hamster?
« Reply #26 on: 10 January 2017, 14:23:22 »

(an admin password was required to reset the system), but unfortunately the boss was in Chamonix, and was unable to get back until late afternoon...
Joking aside, just to put all your minds at rest, should such a scenario happen, the Admins know a man who has full access in remotely to sort out restarts etc.  This is one (there are serveral) reason we use HP servers, to benefit from the iLO Lights Out management (other vendors also have their own technologies).

Said man was texting me early yesterday to say it had gone TITSUP*, so he's defo on the ball.

But it took until mid morning to get it back up, then it failed again early afternoon :(


* A TheRegister term, for any of our technology readers ;)
Logged
Grumpy old man

TheBoy

  • Administrator
  • *****
  • Offline Offline
  • Gender: Male
  • Brackley, Northants
  • Posts: 107026
  • I Like Lockdown
    • Whatever Starts
    • View Profile
Re: Who forgot to feed the hamster?
« Reply #27 on: 10 January 2017, 14:25:44 »

Given his profession, I should think so too :P

 ;D
I was cursing him last month, having to sit through a (very, very) condensed training course provided by his company...   ...and because it was so condensed, had to spend my own time going through the material (6 x A4 sized ring bound books, over an inch thick each).
Logged
Grumpy old man

Doctor Gollum

  • Get A Life!!
  • *****
  • Offline Offline
  • Gender: Male
  • In a colds and darks puddleses
  • Posts: 30002
  • If you can't eat them, join them...
    • Feetses.
    • View Profile
Re: Who forgot to feed the hamster?
« Reply #28 on: 10 January 2017, 14:27:20 »

Is that somewhere between OOPS and FUBAR? ;D
Logged
Onanists always think outside the box.

aaronjb

  • Guest
Re: Who forgot to feed the hamster?
« Reply #29 on: 10 January 2017, 14:27:51 »

I shall refrain from saying anything in case anyone from HQ is reading.. ;D
Logged
Pages: 1 [2] 3  All   Go Up
 

Page created in 0.011 seconds with 16 queries.