Backup, re-install Ubuntu with full disk encryption, and restore all files and settings

When doing serious work like surfing the internet, writing, or programming, I like to do so from a single user interface regardless of whether I’m at work or home. Currently, this takes the form of a Linux laptop (Ubuntu) due to portability (laptop), power (Linux/Ubuntu), and the availability of a keyboard and touchpad (fast input).

I’ve always wanted to encrypt my laptop for privacy reasons. However, I dread the thought of a fresh OS re-install on my laptop because I would have to restore all the programs I use and the customizations I’ve configured. Sure, there are benefits to doing a fresh install like getting rid of unused programs and restoring only customizations that I truly use (I will definitely feel it if something I use is missing) to yield a less cluttered system. However, I after doing this a few times in the pass, I really don’t want to have to do it again since I don’t have any major issues with my current OS.

What I would like to do is backup my entire OS and files, re-install Ubuntu with full disk encryption, and restore the entire OS. That is, I would like the exact OS but with full disk encryption added. After some research, I found this post that describes how to perform a system backup and restore. I wondered whether the same procedure would work while following these instructions for full disk encryption during the OS installation phase. I asked on SuperUser and it appears to be fairly safe. Sources of possible complications might stem from /etc/fstab, /boot/, grub, and, as I’ll later find out, /etc/crypttab (fstab‘s equivalent for encrypted disks). I’ll now outline my attempt.

Backup

I backed up my entire system onto two external hard drives that were encrypted, just in case something wrong happened to one of the backup file.

cp /etc/fstab /media/MYUSBDRIVE/fstab.old
cp /etc/crypttab /media/MYUSBDRIVE/crypttab.old
sudo su
cd /
tar cvpzf /media/MYUSBDRIVE/boot.tgz /boot/
tar cvpzf /media/MYUSBDRIVE/dev.tgz /dev/
tar cvpzf /media/MYUSBDRIVE/backup.tgz --exclude=/proc --exclude=/lost+found --exclude=/backup.tgz --exclude=/mnt --exclude=/sys --exclude=media --exclude=/dev --exclude=/boot / ## I added media, dev, and boot


Kromey on SuperUser recommended that I also excluded /boot because I’m adding encryption. Hence, I should use the new /boot directory. Also, he mentioned I should also exclude /dev, which makes sense to me. However, the original post mentioned that there is debate about whether to include /dev or not. I opted to back up both /boot and /dev in separate files just in case I’ll need them later.

I backed up my system to two usb drive and set up encryption on a third disk simultaneously on a 2.2 GHz dual-core laptop. The backup of 350 GB of data took about 12 hours. This length of time might have stemmed from doing multiple backups at the same time and/or from compressing the data. If not constrained by space, I would recommend not compressing the tar file (removing the -z argument) to speed up the process.

To fail-safe my attempt and have a point where I could return to my old system if things did not work, I went ahead and made an image backup of the entire disk using dd. However, this HAS to be done while the disk is unmounted. I booted the Ubuntu 11.04 Installation Disk using a USB drive to “preview” Ubuntu. Once there, I did:

## unencrypt my usb drive
dd if=/dev/sda of=/media/MYUSBDRIVE/disk1.img


People discussing in the comments here recommended Clonezilla for the image backup to make sure things are fail-safe. I wanted to finish with this project fast so I didn’t use it. If I were to re-do this again somehow, I would probably ditch dd for Clonezilla.

I also backed up my list of packages and repositories just in case I can only restore /home (my files) and /etc (my configurations). This way, I will only use programs I compiled from source.

dpkg --get-selections | awk '!/deinstall|purge|hold/ {print $1}' > /media/MYUSBDRIVE/packages.list find /etc/apt/sources.list* -type f -name '*.list' -exec bash -c 'echo -e "\n##$1 ";grep -v -e \^# -e \^{1}' _ {} \; > /media/MYUSBDRIVE/sources.list.sav


Encryption

Followed these instructions for encryption while installing Ubuntu 11.04 Alternate. I did so from a USB boot disk created from unetbootin. Like before, I did not create a different volume for /home so it can be stored in /.

Boot up OS

When booting up, I get a blank screen with a blinking cursor. I think this is a known bug for Ubuntu 11.04 (possibly for 10.10 as well). It appears to be an issue with grub. I plugged in my USB drive to boot into Ubuntu preview and surprisingly, I get either a grub menu or a blank screen. I knew there were issues with encrypted LVM and Ubuntu 11.04 before. I tried Ctrl-Alt-F1 Ctrl-Alt-F7 and indeed, I saw the passphrase screen. I entered it and went back to TTY1 (Ctrl-Alt-F1) and logged into terminal console.

Restore

Now, I figured these issues out after having several things break. I’ll describe the solutions first and then describe how I debugged the issues.

First, backup the new /boot, /etc/fstab, and /etc/crypttab:

## unlock encrypted external usb drive and mount it using the command line
cp /etc/fstab /media/MYUSBDRIVE/fstab.new
cp /etc/crypttab /media/crypttab.new
sudo tar cvpzf /media/MYUSBDRIVE/boot.new.tgz /boot/


Next, restore my backup:

tar xvpfz backup.tgz -C /


This was a lot faster than the backup process. I believe it took about 4 hours.

Now, for some reason, I was not able to sudo in the current terminal. I pressed Ctrl-Alt-F2 to get to TTY2 and logged in. I did:

sudo cp /media/MYUSBDRIVE/crypttab.new /etc/crypttab ## my old file should be empty, new file should have content


For /etc/fstab, look at the /media/MYUSBDRIVE/fstab.new and copy the content into /etc/fstab, commenting out any content that is no longer relevant. For me, it looks something like:

# proc /proc proc nodev,noexec,nosuid 0 0
# /dev/sda1 / ext4 errors=remount-ro 0 1
# # swap was on /dev/sda5 during installation
# UUID=5e2279de-83a3-4d12-a5e7-cfbebff2f6c4 none swap sw 0 0
# /dev/scd0 /media/floppy0 auto rw,user,noauto,exec,utf8 0 0
proc /proc proc nodev,noexec,nosuid 0 0
/dev/mapper/vg01-vg01--vol02sys / ext4 errors=remount-ro 0 1
# /boot was on /dev/sdb1 during installation
UUID=a069371d-bfb2-4033-809d-d6fe6ee3c13d /boot ext4 defaults 0 2
/dev/mapper/vg01-vg01--vol01swap none swap sw 0 0
/dev/scd0 /media/floppy0 auto rw,user,noauto,exec,utf8 0 0
## remaining is my custom fstab from my old file


Now, if I restart with the USB boot disk plugged in, I should get a passphrase screen and be able to log in and use Ubuntu like normal. YAY!

Issues

Let me now describe some of my adventures with grub and initramfs. For grub, I tried to set NOMODESET in /etc/default/grub per this post:

sudo emacs -q -nw /etc/default/grub
## modify:
## GRUB_CMDLINE_LINUX_DEFAULT="quiet splash nomodeset"
sudo update-grub


This didn’t work and gave me a resolution that did not match the screen (image too big for screen; mouse down and up to see different parts of screen). I had to remove that option and updated grub.

I haven’t figured out how to fix grub. Some resources that I hope to lead me to the right solution: this and this.

Originally, I did not have to modify /etc/crypttab (copy the new one back) for the OS to boot. However, I wanted to make sure that everything is good in /boot (all the new init stuff, eg, encryption, and all the old init stuff, ie, what I restored) by running

sudo update-initramfs -u


(I did this because I know in the future, initramfs might be updated so I wanted to make sure I’m error free right now.)

After doing so, when booting with the USB stick plugged in, I was not asked for passphrase. The BusyBox shell appeared. Something was broken. To have a successful boot again, I had to restore /boot according to boot.new.tgz. I remember when I ran update-initramfs, I saw these messages:

update-initramfs: Generating /boot//initrd.img-2.6.38-11-generic
cryptsetup: WARNING: failed to detect canonical device of /dev/sda5
cryptsetup: WARNING: invalid line in /etc/crypttab -


I found this post that helped me investigate the initrd.img files. Using his initrd-extract.sh and initrd-create.sh scripts, I did:

cd /tmp
initrd-extract.sh /boot/initrd.img-2.6.38-11-generic /tmp/initrd.working
sudo update-initramfs -u -b /tmp ## this creates a new initrd, combining both old config and new config
## update-initramfs: Generating /tmp//initrd.img-2.6.38-11-generic
initrd-extract.sh /tmp/initrd.img-2.6.38-11-generic /tmp/initrd.update
## recursive diff: http://linux.devquickref.com/linux-recursive-diff.html
diff -u -r -B -N -s initrd.update initrd.working


After browsing the diff output, I noticed many files were identical, and many files were different. However, looking at those that were different, they don’t seem to be that important. I did notice /etc/crypttab, one being empty, and the other having something like

sdb5_crypt UUID=731a44c4-4655-4f2b-ae1a-2e3e6a14f2ef none luks


I copied the new crypttab file to /etc/crypttab.

Actually, I originally didn’t even backup my crypttab file. Thanks to the recursive diff, I was able to figure out what I needed to enter into the file (I used server’s /etc/crypttab as a reference and this to find out what needs to be inputted). After restoring the file’s content, I was able to see a screen asking for a passphrase again.

UPDATE: Fix Grub

The odd thing about my grub issue is that the system boots up when the original usb drive I used to install is plugged in and is booted. That is, it will go to grub but not the unetbootin menu that allows me to install ubuntu for preview, etc. I tried plugging in another USB boot disk and it did indeed give me the installation menu. It finally came to my mind that during my installation process, Ubuntu asked me to install Grub into the Master Boot Record (MBR) of the disk and I just accepted blindly. During that time, sda refers to the usb drive and sdb refers to my main disk. It might be the case that grub was not installed into the MBR of my disk.

The original backup post did mention about restoring Grub. I attempted these instructions but it did not work for me. It said something like /boot/grub/stage1 was not found. After perusing and trying different methods for reinstalling Grub or getting it installed on the MBR, the Boot-Repair finally worked for me. Boot into my Ubuntu system (with usb drive plugged in to successfully boot). Then remove USB drive. I then did:

sudo add-apt-repository ppa:yannubuntu/boot-repair
sudo apt-get update && sudo apt-get install -y boot-repair


Launch boot-repair. After the scan I chose “Advanced”. I re-installed Grub according to this:

• Re-install Grub
• Unhide Boot Menu for 10 seconds
• Create BootInfo file
• Separate /boot partition: sda1
• Force GRUB into sda

Then “Apply”. Afterwards, my system did boot successfully without the USB drive plugged in. If it didn’t, maybe try another run of Boot-Repair but now, “Restore MBR” (I did this prior to re-installing GRUB).

TO DO

1. Test suspend: DONE. This works.
2. Test hibernate:

Backup your gmail account in Linux/Unix or Mac OS X using getmail

Even though I have utmost trust in Google’s data centers (>= 99.9% uptime), I finally decided to backup my gmail account. I am pretty OCD when it comes to backing up my data (bi-weekly backups of my home directory into 3 external hard drives, one of which is in a water/fire-proof safe box; MOST important research-related files in the cloud via dropbox), but I’ve never backed up my mail archive which holds a lot of valuable information.

I googled it before but never got around to doing it. Finally, I’ve implemented my backup scheme, thanks to this, this, and this.

The main program that does the work is getmail. To install, issue the following in Debian (apt-get) or Mac OS X (Macports):

 <pre class="src src-sh"><span style="color: #ff4500;">## </span><span style="color: #ff4500;">Debian</span>


sudo apt-get install getmail4 ## Mac OS X via macports sudo port install getmail

mkdir ~/.getmail mkdir ~/Backup/gmail-archive emacs -q -nw ~/.getmail/getmail.gmail

Put the following in “~/.getmail/getmail.gmail” (use IMAP and save both an mbox and a maildir version of my gmail account):

 <pre class="src src-sh">[retriever]


type = SimpleIMAPSSLRetriever server = imap.gmail.com username = email.address password = my.password mailboxes = (“[Gmail]/All Mail”,) [destination] type = MultiDestination destinations = (‘[mboxrd-destination]’, ‘[maildir-destination]’)

[mboxrd-destination] type = Mboxrd path = ~/Backup/gmail-archive/gmail-backup.mbox

[maildir-destination] type = Maildir path = ~/Backup/gmail-archive/

[options] # print messages about each action (verbose = 2) # Other options: # 0 prints only warnings and errors # 1 prints messages about retrieving and deleting messages only verbose = 2 message_log = ~/.getmail/gmail.log # preserves your mail after backup delete = false # just get new mails read_all = false

Note the “read_all” option – without this, getmail downloads multiple versions of the mail and the hard drive will fill. This happened to me!

In the shell, type

<pre class="src src-sh">touch ~/Backup/gmail-archive/gmail-backup.mbox


mkdir ~/Backup/gmail-archive/tmp ~/Backup/gmail-archive/new ~/Backup/gmail-archive/cur getmail -r $HOME/.getmail/getmail.gmail To set a cron job for midnight, create the following Backup_gmail_emails.sh and place it in cron via “cron -e”: <pre class="src src-sh">Backup_gmail_emails.sh  #!/usr/bin/env bash # Note: -q means fetch quietly so that this program is silent /usr/bin/getmail -q -r$HOME/.getmail/getmail.gmail

chmod u+rx $HOME/Backup/Backup_gmail_emails.sh crontab -e: 00 00 * * *$HOME/Backup/Backup_gmail_emails.sh

Moved to WordPress for technical blogging

So I’ve decided to move my technical blog to wordpress, the .org version, not the .com version; that is, to host my own instance of wordpress. The move came from various reasons:

1. syntax highlighting for soure codes,
2. LaTeX support for my stat/math display,
3. post via email,
4. post via emacs,
5. text-to-html syntax for blogging (think org-mode),
6. easy to backup,
7. all these features must be permanent (ie, the services above cannot be changed and my posts no longer look “right”), and finally,
8. extensible

Points 4, 5, and 8 were never addressed with blogger, and point 2 was not consistent (contradicts point 7).

Posterous seemed very nice when i tried it out (especially point 3), but it didn’t seem too good for points 1 and 2 (although points 2 can be addressed by mathurl or Sitmo’s Equation Editor based on this post). I tried wordpress.com once, but thought it was inadequate. I knew it was possible to install your own wordpress version on a server but thought it was too much of a hassle. However, out of the blue I looked into the above features and found that wordpress had plugins (point 8) that really much fulfilled a lot of my needs.

I will outline how I installed wordpress on my school’s server and how I customized it to give me the features I want.

I pretty much followed the instructions from wordpress. My server already had apache on it, so I installed the following in addition:

<pre class="src src-sh">sudo apt-get update


sudo apt-get install php5 sudo apt-get install mysql-server sudo apt-get install php5-mysql ## mod_rewrite Apache module already installed wget http://wordpress.org/latest.tar.gz tar xvf latest.tar.gz mysql -u root -p ## enter password

In mysql, I have to set up the user and password:

<pre class="src src-sql">CREATE USER <span style="color: #ffa07a;">'ENTER MY USERNAME'</span>@<span style="color: #ffa07a;">'localhost'</span> IDENTIFIED BY <span style="color: #ffa07a;">'ENTER MY PASSWORD'</span>;


GRANT ALL PRIVILEGES ON . TO ‘ENTER MY USERNAME’@‘localhost’ WITH GRANT OPTION; EXIT

I also have to set up a database for the blog’s data by:

<pre class="src src-sh">mysql -u USERNAME -p


CREATE DATABASE wordpress;
GRANT ALL PRIVILEGES ON wordpress.* TO "ENTER MY USERNAME"@"localhost"
FLUSH PRIVILEGES;
EXIT


We also need to update the wp-config.php file:

<pre class="src src-sh">emacs -q -nw wp-config.php <span style="color: #ff4500;">## </span><span style="color: #ff4500;">follow instructions in above site, generate random key, and set table_prefix to "snc_" for supernerdycool</span>


sudo emacs -q -nw /etc/apache2/conf.d/virtual.conf sudo emacs -q -nw /etc/apache2/conf.d/virtual.conf ## make sure it looks like following: # # We’re running multiple virtual hosts. # NameVirtualHost * ## done

## make sure following is in /etc/apache2/httpd.conf: LoadModule rewrite_module /usr/lib/apache2/modules/mod_rewrite.so

sudo emacs -q -nw /etc/apache2/sites-available/blog.nguyenvq.com place following in it: # # Example.com (/etc/apache2/sites-available/adrc.nguyenvq.com) #

# Basic setup ServerAdmin EnterMyEmail ServerName blog.nguyenvq.com DocumentRoot /path/to/blog.nguyenvq.com

# HTML documents, with indexing.

## enable site sudo ln -s /etc/apache2/sites-available/blog.nguyenvq.com /etc/apache2/sites-enabled/blog.nguyenvq.com sudo /etc/init.d/apache2 restart

## go to http://blog.nguyenvq.com/wp-admin/install.php in web browser

I initially had some errors with mysql and php, but this showed me that I was missing mysql-php in the install (apt-get); restart apache again.

To install plugins, it is easiest to have an ftp server set up for the server. I installed it via “apt-get install proftpd ## select standalone.”

I addressed what I wanted above with:

1. SyntaxHighlighter Evolved,
2. wp-latex,
3. postie (require php5-imap for imap) or wordpress’s native email support (this didn’t work too well for me’) – email does not work well with wp-latex and syntaxhighlighter as this modifies the incoming text,
4. weblogger mode in emacs (turn on xml-rpc in wordpress),
5. Markdown for WordPress and bbPress
6. backup by using rsync (for wordpress files) and mysqldump for the database (data),
7. since these are installed on my own server, of course all these services are permanent (of course I can move them to another server too),
8. other plugins: Akismet, WP-reCAPTCHA, Acronyms, Search and Replace (very good for updating all posts), Search Unleashed(uncheck “Some themes don’t display any search result content. Enable this option to force the theme to display results” in options or when visiting post from google search result, syntax highlighting will get screwed up), ShareThis (share to twitter, facebook, …, all social media), and Simple Image Widget.

The SyntaxHighlighter Evolved and Markdown didn’t play nicely together. This post’s comments has the solution, by wrapping the source code chunk with the div tag and changing the add_filter line in markdown to 9 (see below). Since this is hard to display, I will describe the characters in text:

<pre class="src src-sh">LeftAngleBracket div RightAngleBracketDiv LeftSquareBracket bash RightSquareBracket


my code LeftSquareBracket FORWARD SLASH bash RightSquareBracket LeftAngleBracket FORWARD SLASH div RightAngleBracket

Also change markdown.php:

<pre class="src src-php">add_filter('the_content_rss', 'Markdown', 6);


syntax highlighter’s is less than 9

See this site for the lnaguages supported.

For LaTeX, include code by:

<pre class="src src-sh">DOLLARSIGN latex ENTER LATEX CODE DOLLARSIGN


to get $$latex y=x’\beta_0$$.

For Markdown syntax, look at this site for a quick reference. The main thing is to use two asterisks (\/\/) around the text for bold and one asterisk (\*) around the text for italics. For links, use

<pre class="src src-sh">LeftSquareBracket TEXT RightSquareBracket LeftParen URL RightParen


. There are a lot more, but these will be commonly used. I would like the syntax to be like that of org-mode, but I can’t have everything. I could hack the php file to modify the syntax like org-mode’s, but I’m too lazy now and don’t want to deal with the regexp’s.

I also like the notepad theme as it fits my technical blog quite nice.

backup wordpress site:

<pre class="src src-sh">rsync -av --modify-window=1 --delete username@myserver:location localLocation


## this backs up to my computer

backup database and some cron scripts (multiple scripts in the following):

<pre class="src src-sh"><span style="color: #ff4500;">### </span><span style="color: #ff4500;">http://codex.wordpress.org/Backing_Up_Your_Database</span>


mysqldump –add-drop-table -u MyUsername -p Databasename | bzip2 -c > blog.nguyenvq.com.bak.sql.bz2 ## wordpress is the database name, -p says to specify password after ## now enter password

#### restore database ### http://codex.wordpress.org/Restoring_Your_Database_From_Backup bzip2 -d blog.nguyenvq.com.bak.sql.bz2 mysql -u MyUserName -p Databasename < blog.nguyenvq.com.bak.sql ## enter password

chmod +x Backup_blog.nguyenvq.com_DB.sh chmod o-r Backup_blog.nguyenvq.com_DB.sh

crontab -e: 00 12 * * * /path/to/Backup_blog.nguyenvq.com_DB.sh 59 23 * * * /path/to/Backup_blog.nguyenvq.com_DB.sh ## need a carriage return at end of file

#### setup cron job on my computer ## make sure passwordless ssh is on Backup_blog.nguyenvq.com.sh: #! /usr/bin/env bash INCREMENT=date +%Y%m%d%H%M DIR=“$HOME/path/to/wherever” DIR=“$HOME/path/to/wherever/Wordpress_blog.nguyenvq.com” # -q for quiet in cron rsync -q -av –modify-window=1 –delete username@server:/path/to/blog $DIR0/ scp -q username@server:/path/to/nguyenvq.com.bak.sql.bz2$DIR/blog.nguyenvq.com.bak.sql.$INCREMENT.bz2 find$DIR -mtime +14 -exec rm -f {} \; ## delete files older than 14 days

crontab -e: 02 12 * * * /path/to/Backup_blog.nguyenvq.com.sh 02 00 * * * /path/to/Backup_blog.nguyenvq.com.sh

Backup my google calendars via shell and python scripts and scheduling them via cron

So recently my calendar got messed up a little (dupes) due my mobile phone (HTC Touch Pro running Windows Mobile 6.1) resetting [syncing again messed it up]. I realize (now???) I need to keep backups of my calendars and contacts.

I took the following script online (which requires python and the gdata module):

<pre class="src src-sh"><span style="color: #ff4500;">#</span><span style="color: #ff4500;">! /usr/bin/</span><span style="color: #00ffff;">python</span>


import gdata.contacts.service gd_client = gdata.contacts.service.ContactsService() gd_client.ClientLogin(‘uname@domain.com’, ‘password’) query = gdata.contacts.service.ContactsQuery() query.max_results = 2000 # change for max contacts returned feed = gd_client.GetContactsFeed(query.ToUri()) print feed

and adapted the following script to backup my calendars and contacts:

 <pre class="src src-sh"><span style="color: #ff4500;">#</span><span style="color: #ff4500;">! /bin/</span><span style="color: #00ffff;">bash</span>


## This script backs up my calendars — will use with Cron to backup daily. ## following is adapted from http://permanentinkdesign.com/articles/backing-up-a-google-calendar/ INCREMENT=date +%Y%m%d%H%M DIR=“$HOME/Documents/Backup_Google” USERNAME=“uname@domain.com” PASSWORD=“mypassword” ## Vinh’s calendar curl -s url-to-private-ics -o “$DIR/Vinh_$INCREMENT.ics” ## TNTTSP’s calendar’ curl -s url-to-private-ics -o “$DIR/TNTTSP_$INCREMENT.ics” ## Birthdays curl -s url-to-private-ics -o “$DIR/Birthdays_$INCREMENT.ics” ## Now backup Google Contacts ## http://www.joet3ch.com/2008/09/25/backup-google-contacts/$HOME/Documents/bin/Backup_Google_Contacts.py > ${DIR}/VinhContacts_$INCREMENT.xml

## Now backup Google Reader subscriptions ## http://www.clausconrad.com/blog/backup-google-reader-subscription-list-to-opml-automatically $HOME/Documents/bin/gr-opml-backup.py$USERNAME $PASSWORD > Vinh_GoogleReaderSubs_$INCREMENT.opml

find $DIR -mtime +14 -exec rm -f {} \; Every time the script runs, it backs up my 3 calendars and contacts. The very last line says delete if files are 14 days or older. Now, to set up my crontab to run this everyday at 10pm, I type “crontab -e” in the shell. I put <pre class="src src-sh">00 22 * * *$<span style="color: #eedd82;">HOME</span>/Documents/bin/Backup_Google.sh


when vi (or your default visual editor) opens up.