OpenStack/Nova rescue mode

The SysEleven Rescue Image

The rescue image was built and tested using Ubuntu 16.04 LTS as core system. It contains a variety of pre-installed tools to rescue your instance and supports most filesystems out of the box.

The rescue image is rebuilt regularly.
Hence it is advisable to check always for the current rescue image name or ID..

Goal

Prerequisites

Optional: temporary work environment

Show details

For this tutorial, we need a Linux environment and the OpenStack client. If you do not have that yet, you can create it with the following commands:

wget https://raw.githubusercontent.com/syseleven/heattemplates-examples/master/gettingStarted/sysElevenStackKickstart.yaml
...
openstack stack create -t sysElevenStackKickstart.yaml --parameter key_name=<ssh key name> <stack name> --wait
...

Now we need to connect to the created instance.

$ ssh syseleven@<server-ip>

The following commands need to be executed in the ssh session.

We also need the OpenStack credentials (openrc-file). You can download the file here.

$ source openrc

Launch Nova rescue mode

To be able to connect to the instance volume we need to launch nova rescue mode. For this we use a rescue image that SysEleven provides in OpenStack.

First we stop the instance

openstack server stop <server uuid>

Then we need to get the current ID of the rescue image

openstack image list --public | grep -i rescue

| ace507db-7acc-46dd-93e1-e561fb962f03 | Rescue-Ubuntu-16.04-sys11                              | active |
OpenStack client version 3.13.x is the minimum to be able to launch instances into the rescue mode.

Now we launch the instance into the rescue mode using the rescue image

# openstack server rescue --image < Image name / ID > <server name / uuid> 

openstack server rescue --image Rescue-Ubuntu-16.04-sys11 app001

Please wait a few seconds while the rescue image is started.

Login to nova rescue instance

Once you are logged in you can work with the rescue mode.

via SSH

If you used a SSH key while deploying the instance initially you can login via ssh

$ ssh ubuntu@<Instance floating ip>

via Console

The rescue image generated a random password with every start, which can be viewed on the console. The root user is only allowed to login via the console. Once logged in you can define a password for the systemuser "ubuntu" if no SSH key is available and login via SSH is preferred.

Get the root password:

root password

Login with the root password:

root password

Work with the nova rescue instance

Become root user and check for connected block devices. Then mount the device.

$ sudo -i
$ lsblk

NAME   MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda    253:0    0   5G  0 disk
└─vda1 253:1    0   5G  0 part /
vdb    253:16   0  50G  0 disk
└─vdb1 253:17   0  50G  0 part

$ mount /dev/vdb1 /mnt/

Now we can start the repairs.
We could check and repair the filesystem using fsck.

# Using the option -y, fsck will repair without asking.
$ sudo fsck -f /dev/vdb1
[...]

We could use testdisk and search for files in damaged filesystems.

$ sudo testdisk /dev/vdb1

TestDisk 7.0, Data Recovery Utility, April 2015
Christophe GRENIER <grenier@cgsecurity.org>
http://www.cgsecurity.org

  TestDisk is free software, and
comes with ABSOLUTELY NO WARRANTY.

Select a media (use Arrow keys, then press Enter):
>Disk /dev/vdb1 - 53 GB / 49 GiB


>[Proceed ]  [  Quit  ]

Note: Disk capacity must be correctly detected for a successful recovery.
If a disk listed above has incorrect size, check HD jumper settings, BIOS
detection, and install the latest OS patches and disk drivers.

[...]

Stop Nova rescue mode

First we "unrescue" the instance.

openstack server unrescue <server uuid>

Then we start the instance again.

openstack server start <server uuid>

Conclusion