Log in

View Full Version : Any Redhat 4.0 ES Experts On here?


krhall
23-01-09, 01:10 PM
One of our webservers has just died and won't boot into it's OS.

I have managed to get most of the files and sites running from another server, but I need to try and recover the files for a MySQL 5 db and try to get them running on another machine.

I think the files I need are:

idbdata1
ib_logfile0
ib_logfile1

...and I think they reside in/var/lib/mysql

Just don't have a Danny on how I can get them off of the old box.

jamesterror
23-01-09, 01:15 PM
Hi,

Few Q's,

Do you have access to a KVM switch? Or something of the sort, or is server on site?

I run a hosting company and had a crash the other day which required a format, I got my provider to boot it with a difference Kernel and recover the entire system.

I presume your running Apache?

SoulKiss
23-01-09, 01:15 PM
One of our webservers has just died and won't boot into it's OS.

I have managed to get most of the files and sites running from another server, but I need to try and recover the files for a MySQL 5 db and try to get them running on another machine.

I think the files I need are:

idbdata1
ib_logfile0
ib_logfile1

...and I think they reside in/var/lib/mysql

Just don't have a Danny on how I can get them off of the old box.

Download an Ubuntu ISO - as well as being installers they are also LIVE systems that run from CD

Put in drive, select the Run rather than Install option (2nd one down I think) and let it boot.

Should detect your Network settings etc and allow you to copy the files you need off to another server.

Or if its an IDE/SATA drive in the old box, get a USB caddy and just mount it on the new box and copy the files off.

Oh and stop using Dead Rat :p

jamesterror
23-01-09, 01:17 PM
Download an Ubuntu ISO - as well as being installers they are also LIVE systems that run from CD

Put in drive, select the Run rather than Install option (2nd one down I think) and let it boot.

Should detect your Network settings etc and allow you to copy the files you need off to another server.

Or if its an IDE/SATA drive in the old box, get a USB caddy and just mount it on the new box and copy the files off.

Oh and stop using Dead Rat :p

I took the guess hes off site :)

krhall
23-01-09, 01:19 PM
I hate Linux all versions!!!!

Any ideas, where to get a decent ISO copy.

I'm on site and although not in front of the box atm I can be within minutes.

jamesterror
23-01-09, 01:21 PM
I use CentOS

http://www.centos.org/modules/tinycontent/index.php?id=15

Do you colocate yourself or have a provider? Usually you can get them to do it for you ;)

krhall
23-01-09, 01:24 PM
No everything is hosted internally, although I think I'll be revisiting after the week I have had so far.

Funny thing is all we were attempting to do was update the RALUS backupexec agent to 12.5...........

jamesterror
23-01-09, 01:28 PM
Strange.

Don't talk to me about the week I've had with my server, an outage last night due to memory crash and somebody attempted to hack my server last week causing it to crash.

**** take, but all fixed now. I'm learning linux step by step.

SoulKiss
23-01-09, 01:38 PM
I assume you are in The City - where abouts? I am on Chiswell St and have the 64bit ISO here:)

32bit version can be gotten here - http://ubuntu-releases.datahop.it/hardy/ubuntu-8.04.2-desktop-i386.iso

I have specified the Desktop version rather than Server as it may be easire to use the GUI tools if you are not that happy on the CLI


I hate Linux all versions!!!!

Any ideas, where to get a decent ISO copy.

I'm on site and although not in front of the box atm I can be within minutes.

SoulKiss
23-01-09, 01:39 PM
I use CentOS

You have my commiserations.....

Sudoxe
23-01-09, 01:40 PM
Hang on, hang on, hang on! DO NOT FORMAT THE SYSTEM!

The guy has a problem with mysql, and people are suggesting formatting and reinstalling the enitre machine? Do none of the previous repliers have a clue about Linux? (Hint: i know at least one does).

Right, now i've got that off my chest. I haven't used mysql in ~5 years, so i wont be much of use.

However, if you have a backup i.e. a recently mysql_dump then you have options to (re)move the files under /var/lib/mysql and import the files back into mysql. (This is likely the quickest way, if the data doesnt change much from the backup you have).

If the data changes, or you dont have a backup, a quick bit of googleing i found this (http://www.thegeekstuff.com/2008/09/how-to-repair-corrupted-mysql-tables-using-myisamchk/). It may be of help.

Dan

N.B FreeBSD on servers or debian if you must use linux. Ubuntu on the desktop.

SoulKiss
23-01-09, 01:42 PM
Hang on, hang on, hang on! DO NOT FORMAT THE SYSTEM!

The guy has a problem with mysql, and people are suggesting formatting and reinstalling the enitre machine? Do none of the previous repliers have a clue about Linux? (Hint: i know at least one does).

Right, now i've got that off my chest. I haven't used mysql in ~5 years, so i wont be much of use.

However, if you have a backup i.e. a recently mysql_dump then you have options to (re)move the files under /var/lib/mysql and import the files back into mysql. (This is likely the quickest way, if the data doesnt change much from the backup you have).

If the data changes, or you dont have a backup, a quick bit of googleing i found this (http://www.thegeekstuff.com/2008/09/how-to-repair-corrupted-mysql-tables-using-myisamchk/). It may be of help.

Dan

N.B FreeBSD on servers or debian if you must use linux. Ubuntu on the desktop.

My advice is to run a LIVE CD system so as not to touch the HD's

And Ubuntu on the Server is quite ok - but yes Debian would be better.

Sudoxe
23-01-09, 01:45 PM
My advice is to run a LIVE CD system so as not to touch the HD's

And Ubuntu on the Server is quite ok - but yes Debian would be better.

Yea, I missed the bit about it not booting, but its self explanatory to get a bootcd and get the FS' mounted, if the OP is a sysadmin. :confused:

Meh

jamesterror
23-01-09, 01:47 PM
Hang on, hang on, hang on! DO NOT FORMAT THE SYSTEM!

The guy has a problem with mysql, and people are suggesting formatting and reinstalling the enitre machine? Do none of the previous repliers have a clue about Linux? (Hint: i know at least one does).

Right, now i've got that off my chest. I haven't used mysql in ~5 years, so i wont be much of use.

However, if you have a backup i.e. a recently mysql_dump then you have options to (re)move the files under /var/lib/mysql and import the files back into mysql. (This is likely the quickest way, if the data doesnt change much from the backup you have).

If the data changes, or you dont have a backup, a quick bit of googleing i found this (http://www.thegeekstuff.com/2008/09/how-to-repair-corrupted-mysql-tables-using-myisamchk/). It may be of help.

Dan

N.B FreeBSD on servers or debian if you must use linux. Ubuntu on the desktop.

CentOS isn't that bad!

I didn't suggest, I stated what I did as it was a boot configuration error caused by an attack on server ;).

I'm pretty new to linux and trying all sorts of FreeBSD on home server and so far I found Debian is +1

SoulKiss
23-01-09, 01:47 PM
Yea, I missed the bit about it not booting, but its self explanatory to get a bootcd and get the FS' mounted, if the OP is a sysadmin. :confused:

Meh

I was going for the easiest non-technical options - the original post indicated thats what was needed.

SoulKiss
23-01-09, 01:48 PM
CentOS isn't that bad!

Centos ~= Dead Rat == Bad

Hmmm RPM-Hell anyone?

krhall
23-01-09, 01:51 PM
I haven't been really technical for a few years, but my Network Admin is off all week and unable to get him in and unforutately due to the credit crunch there is not a second member which we used to have.

So that leaves me.

I have had to remember how to create VLAN's and rebuild switches, this week too, so when the 2 linux boxes crashed I could've cried.........

I knew there was some geniuses on here hence the post.

sasquatch
23-01-09, 05:25 PM
The other option is to call Red Hat's technical support (01252 362700, it's one of the menu options off there). Assuming you have a valid support entitlement, of course (which you should if you're using RHEL). Sounds very much like the problem is with MySQL, but they should hopefully be able to help you - assuming you're using the RH supplied MySQL packages. I suspect you'll be told "reinstall from backup" - but they'll at least be able to help you through the process.

If you still get stuck, drop me a PM.

Oh, hello BTW - this is my first post to the .org :)

Baph
23-01-09, 05:44 PM
I haven't been really technical for a few years, but my Network Admin is off all week and unable to get him in and unforutately due to the credit crunch there is not a second member which we used to have.

So that leaves me.

I have had to remember how to create VLAN's and rebuild switches, this week too, so when the 2 linux boxes crashed I could've cried.........

I knew there was some geniuses on here hence the post.

Kev, as you're clearly in a bind at the mo, I won't get into the whole Linux debate that's going on above; how far have you got with the MySQL DB recovery?

Now, what I would do in this situation, is similar to SoulKiss, with a little tweak (as you're only interested in MySQL DB backup).

I personally, would boot from the Live CD of your choice (Ubuntu is as good as any), then setup a dummy MySQL server in the live environment. Shut that daemon down, and symlink the old /var/lib/mysql (the one on the hard disk), to the new one in the live environment. Then start the daemon & use mysql_dump to dump the databases.

Once that's done, you can then simply import the dumps into a new database on the new server, and away you run.

Also, a conversation for when the normal admin gets back, ask him about redundancy for this particular area of the setup. IMO, you should of just been able to rip the disks out & shove them in the next server (assuming the fault is BIOS/boot sequence related). If the problem is with OS changes stopping the boot, then what changed & when? Why was this change not approved (and tested) before being implimented in this situation?

timwilky
23-01-09, 08:09 PM
£1,500/day + expenses I will get it back for you. (eventually promise)

Grinch
23-01-09, 08:38 PM
Pay me some money and I'll come in next week and sort it out.. ;-)

krhall
24-01-09, 12:10 PM
Well I am currently working through the suggestions on here, I will let you know how I get on. Failing that see you Monday Grinch!

krhall
26-01-09, 09:27 AM
I have managed to get a mysql dump out of the box now, just not too sure what to do with it?

Sudoxe
26-01-09, 09:34 AM
I have managed to get a mysql dump out of the box now, just not too sure what to do with it?

You should be able to import it using something like

mysql dbname < your.dump.file
or
mysql < your.dump.file

However, you might need to specify the username/password, etc depending on the permissions on your setup.

Also, there maybe a mysqlimport or mysql_import command, but i really can't remember too much as it has been years since I used mysql.

ONE THING TO BE CAREFUL OF: You may have a full mysql dump, rather than a dump of just one database. This may modify other databases within the mysql setup. (You can have many databases within a mysql server).

Dan

Sudoxe
26-01-09, 09:37 AM
Just another thought.

What I would likely do is generate a mysql dump of the web database on the new machine (i.e. the one the data will be going on), then drop (delete) the database, create a new one with the same name and then import the dump file from the old server as above.

Dan

krhall
26-01-09, 11:39 AM
The dump file is useless, but I may have stumbled upon the 3 files I needed, but I cannot seem to get ubuntu to recognise the USB drive?

SoulKiss
26-01-09, 11:42 AM
If you have MSN, send me a message and I will talk you through it

David

krhall
26-01-09, 09:08 PM
Another one has gone now and is asking to runFSchk manually and when doing so is coming up with "too many illegal blocks"

Baph
27-01-09, 06:00 AM
Another one has gone now and is asking to runFSchk manually and when doing so is coming up with "too many illegal blocks"

Kev,

NB, the below assumes you're using ext2 or ext3.

Boot the system to the recovery mode (my RH knowledge is old, I believe it still uses a built in recovery mode - if not, boot to a live CD environment). Then run the following command:

sudo e2fsck -C0 -p -f -v /dev/xxx?


Where:
-C0 writes completion data back to the file descriptor (correcting errors basically).
-p automatically repair.
-f force checking.
-v verbose.
/dev/xxx? is the device you're having issues with. For example, /dev/sda2.

If it complains about the -p option, you can omit this, it'll just prompt you on every illegal block, which can be a pain. You can circumvent this, by using -y (yes to all questions), but as I'm blind to the system, I really can't advise doing that!! Patience is a virtue!

As for your MySQL issues, I assume you got my PM. :) Post the dumpfiles somewhere we can see them to see what's wrong with them.

EDIT: Did you manage to get Ubuntu to recognise the USB drive? `dmesg | grep usb` should reveal any issues there.

K
27-01-09, 07:05 AM
I have managed to get a mysql dump out of the box now, just not too sure what to do with it?
I suggest you flush it.:p

Tee hee hee - you said 'dump'...

... tee hee hee hee hee, you managed to get a dump out of the box...



... bad kitty.:mrgreen:



Sorry - I'm not helping am I. :oops: But it was funny.;)

krhall
27-01-09, 09:51 AM
Kev,

NB, the below assumes you're using ext2 or ext3.

Boot the system to the recovery mode (my RH knowledge is old, I believe it still uses a built in recovery mode - if not, boot to a live CD environment). Then run the following command:


Where:
-C0 writes completion data back to the file descriptor (correcting errors basically).
-p automatically repair.
-f force checking.
-v verbose.
/dev/xxx? is the device you're having issues with. For example, /dev/sda2.

If it complains about the -p option, you can omit this, it'll just prompt you on every illegal block, which can be a pain. You can circumvent this, by using -y (yes to all questions), but as I'm blind to the system, I really can't advise doing that!! Patience is a virtue!

As for your MySQL issues, I assume you got my PM. :) Post the dumpfiles somewhere we can see them to see what's wrong with them.

EDIT: Did you manage to get Ubuntu to recognise the USB drive? `dmesg | grep usb` should reveal any issues there.


I have whacked the ubuntu cd in this machine and am now making tar files of the data and ftping to another server just to be safe. Once I have done that I will run the FSCK command and see how that goes. It all appears so far to be caused by an updated RALUS agent for backup exec.

I am really in the s**t here and my job is looking decidely iffy atm.

Thanks all for the help.

SoulKiss
27-01-09, 09:54 AM
I have whacked the ubuntu cd in this machine and am now making tar files of the data and ftping to another server just to be safe. Once I have done that I will run the FSCK command and see how that goes. It all appears so far to be caused by an updated RALUS agent for backup exec.

I am really in the s**t here and my job is looking decidely iffy atm.

Thanks all for the help.

From what I understand of what you have posted here, its the person who decided that there was not enough budget to employ a 2nd support person for those times that the guy that usually does this is ill/on holiday.

Dont see why its YOUR job on the line.

krhall
27-01-09, 10:02 AM
True.....arghhh. Your ubuntu iso is coming up trumpts atm though for at least giving the option of copying the files off.

SoulKiss
27-01-09, 10:12 AM
True.....arghhh. Your ubuntu iso is coming up trumpts atm though for at least giving the option of copying the files off.

Its not what I would recommend running everything off of every day, but its the best at detecting and working with most hardware, and the GUI of the desktop version seemed like a good idea too (also that it automounts USB devices - when it can see them anyway)

Glad to be of help :)

krhall
27-01-09, 10:13 AM
I found a command to mount the usb drive in the end, a bit of a mission but seemed to work, ftp'd to another server instead anyway.

krhall
27-01-09, 10:58 AM
Kev,

NB, the below assumes you're using ext2 or ext3.

Boot the system to the recovery mode (my RH knowledge is old, I believe it still uses a built in recovery mode - if not, boot to a live CD environment). Then run the following command:


Where:
-C0 writes completion data back to the file descriptor (correcting errors basically).
-p automatically repair.
-f force checking.
-v verbose.
/dev/xxx? is the device you're having issues with. For example, /dev/sda2.

If it complains about the -p option, you can omit this, it'll just prompt you on every illegal block, which can be a pain. You can circumvent this, by using -y (yes to all questions), but as I'm blind to the system, I really can't advise doing that!! Patience is a virtue!

As for your MySQL issues, I assume you got my PM. :) Post the dumpfiles somewhere we can see them to see what's wrong with them.

EDIT: Did you manage to get Ubuntu to recognise the USB drive? `dmesg | grep usb` should reveal any issues there.

Gives me:

e2fsck: Bad magic number in super-block while trying to open /dev/VolGroup00/LogVol01

Grinch
27-01-09, 11:37 AM
Gives me:

e2fsck: Bad magic number in super-block while trying to open /dev/VolGroup00/LogVol01

You need to copy over a new super block, depending on the filesystem and the verson of Redhat you are using there should be copies at verious point in the FS. So you will need to fsck the filesystem and then went it asks you the superblock location you will need to tell it. If its the new version then if I remember correctly it knows where they are and just prompts for confirmation.

Something like this -

e2fsck -b newsuperblock filesystem


I can't remember the default superblock locations at creation as I don't have access to a linux box at the moment. Though when you create a new filesystem it normally tells you at creation.

Grinch
27-01-09, 11:39 AM
Try -

e2fsck -b 8193 <device>

Baph
27-01-09, 05:00 PM
Gives me:

e2fsck: Bad magic number in super-block while trying to open /dev/VolGroup00/LogVol01

Try -

e2fsck -b 8193 <device>

Grinch, that's basically what I was going to advise - if the original e2fsck failed. However, a little more detail (since I'm flying blind in this situation...)

OK, first, unmount the file system, then try the "sudo e2fsck -C0 -p -f -v /dev/xxx?" command again. The reason for this is that the superblock changes when the filesystem is mounted to prevent e2fsck from doing anything bad to mounted data. (Kev, this is the bit I was asking about in the email from my phone).

If that still fails for bad superblock, then you need to replace the superblock with the backup one. The location of the superblock is defined by the blocksize of the FS.

To find the blocksize, use the following:

tune2fs -l /dev/xxx? | grep "Block size"


If the blocksize is 1k, the backup superblock is at 8193. If it's 2k, then you want 16384, if it's 4 k, you want 32768.

Then run the command Grinch said, with the appropriate location for your blocksize.

After that's complete, with the filesystem unmounted, you should be able to run e2fsck properly again. Start with a basic look at the disc & repair any issues found by running it again. Like I said, patience is a virtue.

I don't have the free time I thought I had this evening, but drop me a mail after about 9pm & I should be around to pick it up. Alternatively I'll be in the office from 11pm too.

krhall
27-01-09, 08:57 PM
Chaps thanks for the info - I am still taking a full copy of the data which is taking forever, but as it was a backup agent which I believe caused the issue a full copy is what is required tar.gz files ftp to another server (windows thankfully).

Then I will attempt these, just playing the waiting game at the moment.

Tomorrow will be crunch day on this I think, if not later this evening.

Ed
27-01-09, 09:01 PM
Kev, I haven't a clue what you're talking about but I wish you all the best with this, and I'm thinking of you.

krhall
29-01-09, 06:58 AM
Cheers Ed.

Started running the check yesterday am and it is still running now.......

krhall
29-01-09, 04:40 PM
Just as I was racking one of the two new server, guess who decides to pop back? None other than the temporary front and back-end server and ready fresh from it's break to show a full website!!!

It's days are numbered though, so I am still intent on replacing.

I'm not out of the woods just yet, but there is light at the end of the tunnel. I really apprecaite the help you have given on this thread and if I make it to AR09 or Soho I will buy you a pint/coffee.