Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[tunnie]

This is a long shot, but any Sybase experts?

I'm booting at database, but it fails to find certain parts. They have just 'disappeared' - They have not been deleted or moved:

It loads a section ok:

kernel  Initializing virtual device 9, '/opt/sybdev/data/DEV_DB_011_DATA_08' with dsync 'on'.
00:00000:00001:2011/08/20 14:57:41.73 kernel  Virtual device 9 started using asynchronous i/o.

But then about 7 parts it fails to find (because quite rightly, they are not there):

nitializing virtual device 13, '/opt/sybdev/data/DEV_DB_011_DATA_12' with dsync 'on'.
00:00000:00001:2011/08/20 14:57:41.73 kernel  dopen: open '/opt/sybdev/data/DEV_DB_011_DATA_12', No such file or directory
00:00000:00001:2011/08/20 14:57:41.73 kernel  udactivate: error starting virtual disk 13

It previous managed to drop DATA_08, I recovered that, booted, and DB worked, for about an hour, then died again.

Booting it, now suggests _12 through to _20 are missing  :-?

Struggling to understand what would cause data from /opt/sybdev/data to go missing!  :-/

[aaronjb]

Struggling to understand what would cause data from /opt/sybdev/data to go missing!  :-/

fsck could, quite happily, have removed the files if the inode structure was corrupt..

Do you know if those are actual data files (in which case it's going to be hard to recover the DB!) or transactional log files?

Sadly I know squat about Sybase; can you ask me one on MySQL instead? 

[Martin_1962]

The company Sybase includes many products - which one is it?

There are SIX listed on their site.

One of their products I am an expert in - this sounds like it isn't that one!

[Jimbob]

recover from the backup prior to your reboot

[tunnie]

Struggling to understand what would cause data from /opt/sybdev/data to go missing!  :-/

fsck could, quite happily, have removed the files if the inode structure was corrupt..

Do you know if those are actual data files (in which case it's going to be hard to recover the DB!) or transactional log files?

Sadly I know squat about Sybase; can you ask me one on MySQL instead? 

How can I tell thats the case? 

recover from the backup prior to your reboot

Thats what I'm doing now, I used to be in Operations at this company. Now I'm doing anything technical related, i am certainly learning a lot!

Previously I copied a chunk and re-named it to fool the DB into booting, once booted I did a recovery using 7 stripes from an earlier backup, however the DB failed again. I think something is corrupted there, so now resorting to a later backup.

I just don't understand why it is removing the files it is  :-/

The company Sybase includes many products - which one is it?

There are SIX listed on their site.

One of their products I am an expert in - this sounds like it isn't that one!

Its Sybase ASE 

[aaronjb]

Struggling to understand what would cause data from /opt/sybdev/data to go missing!  :-/

fsck could, quite happily, have removed the files if the inode structure was corrupt..

Do you know if those are actual data files (in which case it's going to be hard to recover the DB!) or transactional log files?

Sadly I know squat about Sybase; can you ask me one on MySQL instead? 

How can I tell thats the case? 

There should be a log from fsck somewhere - location probably varies depending on Linux distribution (assuming it's Linux and not 'real' Unix).. /var/log/fsck?

If Sybase is removing those files then I'm guessing they're transactional logs - copying random ones around to new names is only going to screw the data up entirely, though..

[Martin_1962]

Struggling to understand what would cause data from /opt/sybdev/data to go missing!  :-/

fsck could, quite happily, have removed the files if the inode structure was corrupt..

Do you know if those are actual data files (in which case it's going to be hard to recover the DB!) or transactional log files?

Sadly I know squat about Sybase; can you ask me one on MySQL instead? 

How can I tell thats the case? 

recover from the backup prior to your reboot

Thats what I'm doing now, I used to be in Operations at this company. Now I'm doing anything technical related, i am certainly learning a lot!

Previously I copied a chunk and re-named it to fool the DB into booting, once booted I did a recovery using 7 stripes from an earlier backup, however the DB failed again. I think something is corrupted there, so now resorting to a later backup.

I just don't understand why it is removing the files it is  :-/

The company Sybase includes many products - which one is it?

There are SIX listed on their site.

One of their products I am an expert in - this sounds like it isn't that one!

Its Sybase ASE 

I would suggest trawling their web site and help files.

Not sure what there is for ASE but some of the servers have masses of help information.

BTW with ADS another of their products we have never had to do a full recovery but that uses discrete files.

All part of SAP now.

My thoughts are a full restore as well.

[tunnie]

I'm hoping this server was backed up to takes, as I need to do a restore about a month back now i think.

It still refuses too boot, apart from some test errors the log does not really indicate anything.

http://dl.dropbox.com/u/803897/Work/PP/Log1.rtf

*ignore the license error, they are too cheap to spend the £50k for it...

[Jimbob]

Remind us of the history...

This database was happily running till you accidently turned off the box?

when was it last rebooted

how often and what type of backups are taken?  full, hot backups?
in use files?

[tunnie]

Remind us of the history...

This database was happily running till you accidently turned off the box?

when was it last rebooted

how often and what type of backups are taken?  full, hot backups?
in use files?

It was bust long before I got my mits on it, I gets a call from my mate, said everything is down! They rebooted the server (not physically) Sadly we lost the logs from this...

It was decided we should bounce the box at the blade centre, the person I replaced said something similar had happend before. Hard reboot solved it.

Booting backup, it was missing _08, our thoughts it was corrupted/lost from the reboot.

But once we got it back, it was running for a short time, the DB failed again. Everytime I try & connect I get:

 Client Message
Layer 6, Origin 8, Severity 5, Number 3
ct_connect(): directory service layer: internal directory control layer error: Requested server name not found.



Ahh yes backups, as this company is failing, the end is near. To save money as the contact run out at Rackspace, everything was moved in-house, number of key services were not longer available, all redundancy was lost, BladeCentre was no longer RAID5, each blade ran on its own HD.


I'm investigating backups now, but it does not look good 

Shame, I only needed to keep this POS system running until November 

[tunnie]

Last physical reboot last Wednesday, soft reboot, yesterday. I can see Daily/Weekly/Monthly backups to tapes which are then outsourced to Iron Mountian, whats actually on those tapes I've no idea 

I used to be the Ops Analyst here, but now I'm doing the DB Architect's job, Windows Sys Admin, Linux Sys Admin, Billing Analyst, Ops job again, all rolled into one   

[Jimbob]

Ohh dear.

straw clutching time then....

Ahh well, if they wont spend, what do they expect....

that message wont go down well with management though.


Is the 'possibility' logged and monitored via a risk log?  if so at least its been highlighted, and reduces the trouble.

[tunnie]

Ohh dear.

straw clutching time then....

Ahh well, if they wont spend, what do they expect....

that message wont go down well with management though.


Is the 'possibility' logged and monitored via a risk log?  if so at least its been highlighted, and reduces the trouble.

Its such a shame, I joined this company fresh from Uni, it was 60 strong, going places. I had the chance of moving to New York for new product, then the Americans came and stuffed it all up.

They paid a considerable 7 figure some for the company, its now just 5 people left, 6 if you include me as part time.

I'm mates with my boss, so its all good for me.

To give you an idea of whats going on here, they want us (me) to drop the £15k SAN solution they bought 4 years ago, as part of cost cutting they want to use Dropbox as a General Share for all company documents     

 

[cem_devecioglu]

Tunnie,

for one thing I'm sure you are having trouble at the unix level before sybase..

which may require TheBoy to be hands on job

[tunnie]

Tunnie,

for one thing I'm sure you are having trouble at the unix level before sybase..

which may require TheBoy to be hands on job

When this all started failing, our first thoughts were hard drive failure. BladeCentre is 5 years old, the Sybase person who worked her before, also suggested that likely.

If no backups are around, we are fubared!