This site lists 52 things that people should know about cryptography, if you want to get into the cryptography world. I was also recommended Bruce Schneier’s book, Applied Cryptography: Protocols, Algorthms, and Source Code in C.

Backup, re-install Ubuntu with full disk encryption, and restore all files and settings

When doing serious work like surfing the internet, writing, or programming, I like to do so from a single user interface regardless of whether I’m at work or home. Currently, this takes the form of a Linux laptop (Ubuntu) due to portability (laptop), power (Linux/Ubuntu), and the availability of a keyboard and touchpad (fast input).

I’ve always wanted to encrypt my laptop for privacy reasons. However, I dread the thought of a fresh OS re-install on my laptop because I would have to restore all the programs I use and the customizations I’ve configured. Sure, there are benefits to doing a fresh install like getting rid of unused programs and restoring only customizations that I truly use (I will definitely feel it if something I use is missing) to yield a less cluttered system. However, I after doing this a few times in the pass, I really don’t want to have to do it again since I don’t have any major issues with my current OS.

What I would like to do is backup my entire OS and files, re-install Ubuntu with full disk encryption, and restore the entire OS. That is, I would like the exact OS but with full disk encryption added. After some research, I found this post that describes how to perform a system backup and restore. I wondered whether the same procedure would work while following these instructions for full disk encryption during the OS installation phase. I asked on SuperUser and it appears to be fairly safe. Sources of possible complications might stem from /etc/fstab, /boot/, grub, and, as I’ll later find out, /etc/crypttab (fstab‘s equivalent for encrypted disks). I’ll now outline my attempt.


I backed up my entire system onto two external hard drives that were encrypted, just in case something wrong happened to one of the backup file.

cp /etc/fstab /media/MYUSBDRIVE/fstab.old
cp /etc/crypttab /media/MYUSBDRIVE/crypttab.old
sudo su
cd /
tar cvpzf /media/MYUSBDRIVE/boot.tgz /boot/
tar cvpzf /media/MYUSBDRIVE/dev.tgz /dev/
tar cvpzf /media/MYUSBDRIVE/backup.tgz --exclude=/proc --exclude=/lost+found --exclude=/backup.tgz --exclude=/mnt --exclude=/sys --exclude=media --exclude=/dev --exclude=/boot / ## I added media, dev, and boot

Kromey on SuperUser recommended that I also excluded /boot because I’m adding encryption. Hence, I should use the new /boot directory. Also, he mentioned I should also exclude /dev, which makes sense to me. However, the original post mentioned that there is debate about whether to include /dev or not. I opted to back up both /boot and /dev in separate files just in case I’ll need them later.

I backed up my system to two usb drive and set up encryption on a third disk simultaneously on a 2.2 GHz dual-core laptop. The backup of 350 GB of data took about 12 hours. This length of time might have stemmed from doing multiple backups at the same time and/or from compressing the data. If not constrained by space, I would recommend not compressing the tar file (removing the -z argument) to speed up the process.

To fail-safe my attempt and have a point where I could return to my old system if things did not work, I went ahead and made an image backup of the entire disk using dd. However, this HAS to be done while the disk is unmounted. I booted the Ubuntu 11.04 Installation Disk using a USB drive to “preview” Ubuntu. Once there, I did:

## unencrypt my usb drive
dd if=/dev/sda of=/media/MYUSBDRIVE/disk1.img

This took about 6 hours.

People discussing in the comments here recommended Clonezilla for the image backup to make sure things are fail-safe. I wanted to finish with this project fast so I didn’t use it. If I were to re-do this again somehow, I would probably ditch dd for Clonezilla.

I also backed up my list of packages and repositories just in case I can only restore /home (my files) and /etc (my configurations). This way, I will only use programs I compiled from source.

dpkg --get-selections | awk '!/deinstall|purge|hold/ {print $1}' > /media/MYUSBDRIVE/packages.list
find /etc/apt/sources.list* -type f -name '*.list' -exec bash -c 'echo -e "\n## $1 ";grep -v -e \^# -e \^$ ${1}' _ {} \; > /media/MYUSBDRIVE/sources.list.sav


Followed these instructions for encryption while installing Ubuntu 11.04 Alternate. I did so from a USB boot disk created from unetbootin. Like before, I did not create a different volume for /home so it can be stored in /.

Boot up OS

When booting up, I get a blank screen with a blinking cursor. I think this is a known bug for Ubuntu 11.04 (possibly for 10.10 as well). It appears to be an issue with grub. I plugged in my USB drive to boot into Ubuntu preview and surprisingly, I get either a grub menu or a blank screen. I knew there were issues with encrypted LVM and Ubuntu 11.04 before. I tried Ctrl-Alt-F1 Ctrl-Alt-F7 and indeed, I saw the passphrase screen. I entered it and went back to TTY1 (Ctrl-Alt-F1) and logged into terminal console.


Now, I figured these issues out after having several things break. I’ll describe the solutions first and then describe how I debugged the issues.

First, backup the new /boot, /etc/fstab, and /etc/crypttab:

## unlock encrypted external usb drive and mount it using the command line
cp /etc/fstab /media/MYUSBDRIVE/
cp /etc/crypttab /media/
sudo tar cvpzf /media/MYUSBDRIVE/ /boot/

Next, restore my backup:

tar xvpfz backup.tgz -C /

This was a lot faster than the backup process. I believe it took about 4 hours.

Now, for some reason, I was not able to sudo in the current terminal. I pressed Ctrl-Alt-F2 to get to TTY2 and logged in. I did:

sudo cp /media/MYUSBDRIVE/ /etc/crypttab ## my old file should be empty, new file should have content

For /etc/fstab, look at the /media/MYUSBDRIVE/ and copy the content into /etc/fstab, commenting out any content that is no longer relevant. For me, it looks something like:

# proc /proc proc nodev,noexec,nosuid 0 0
# /dev/sda1 / ext4 errors=remount-ro 0 1
# # swap was on /dev/sda5 during installation
# UUID=5e2279de-83a3-4d12-a5e7-cfbebff2f6c4 none swap sw 0 0
# /dev/scd0 /media/floppy0 auto rw,user,noauto,exec,utf8 0 0
proc /proc proc nodev,noexec,nosuid 0 0
/dev/mapper/vg01-vg01--vol02sys / ext4 errors=remount-ro 0 1
# /boot was on /dev/sdb1 during installation
UUID=a069371d-bfb2-4033-809d-d6fe6ee3c13d /boot ext4 defaults 0 2
/dev/mapper/vg01-vg01--vol01swap none swap sw 0 0
/dev/scd0 /media/floppy0 auto rw,user,noauto,exec,utf8 0 0
## remaining is my custom fstab from my old file

Now, if I restart with the USB boot disk plugged in, I should get a passphrase screen and be able to log in and use Ubuntu like normal. YAY!


Let me now describe some of my adventures with grub and initramfs. For grub, I tried to set NOMODESET in /etc/default/grub per this post:

sudo emacs -q -nw /etc/default/grub
## modify:
## GRUB_CMDLINE_LINUX_DEFAULT="quiet splash nomodeset"
sudo update-grub

This didn’t work and gave me a resolution that did not match the screen (image too big for screen; mouse down and up to see different parts of screen). I had to remove that option and updated grub.

I haven’t figured out how to fix grub. Some resources that I hope to lead me to the right solution: this and this.

Originally, I did not have to modify /etc/crypttab (copy the new one back) for the OS to boot. However, I wanted to make sure that everything is good in /boot (all the new init stuff, eg, encryption, and all the old init stuff, ie, what I restored) by running

sudo update-initramfs -u

(I did this because I know in the future, initramfs might be updated so I wanted to make sure I’m error free right now.)

After doing so, when booting with the USB stick plugged in, I was not asked for passphrase. The BusyBox shell appeared. Something was broken. To have a successful boot again, I had to restore /boot according to I remember when I ran update-initramfs, I saw these messages:

update-initramfs: Generating /boot//initrd.img-2.6.38-11-generic
cryptsetup: WARNING: failed to detect canonical device of /dev/sda5
cryptsetup: WARNING: invalid line in /etc/crypttab -

I found this post that helped me investigate the initrd.img files. Using his and scripts, I did:

cd /tmp /boot/initrd.img-2.6.38-11-generic /tmp/initrd.working
sudo update-initramfs -u -b /tmp ## this creates a new initrd, combining both old config and new config
## update-initramfs: Generating /tmp//initrd.img-2.6.38-11-generic /tmp/initrd.img-2.6.38-11-generic /tmp/initrd.update
## recursive diff:
diff -u -r -B -N -s initrd.update initrd.working

After browsing the diff output, I noticed many files were identical, and many files were different. However, looking at those that were different, they don’t seem to be that important. I did notice /etc/crypttab, one being empty, and the other having something like

sdb5_crypt UUID=731a44c4-4655-4f2b-ae1a-2e3e6a14f2ef none luks

I copied the new crypttab file to /etc/crypttab.

Actually, I originally didn’t even backup my crypttab file. Thanks to the recursive diff, I was able to figure out what I needed to enter into the file (I used server’s /etc/crypttab as a reference and this to find out what needs to be inputted). After restoring the file’s content, I was able to see a screen asking for a passphrase again.

UPDATE: Fix Grub

The odd thing about my grub issue is that the system boots up when the original usb drive I used to install is plugged in and is booted. That is, it will go to grub but not the unetbootin menu that allows me to install ubuntu for preview, etc. I tried plugging in another USB boot disk and it did indeed give me the installation menu. It finally came to my mind that during my installation process, Ubuntu asked me to install Grub into the Master Boot Record (MBR) of the disk and I just accepted blindly. During that time, sda refers to the usb drive and sdb refers to my main disk. It might be the case that grub was not installed into the MBR of my disk.

The original backup post did mention about restoring Grub. I attempted these instructions but it did not work for me. It said something like /boot/grub/stage1 was not found. After perusing and trying different methods for reinstalling Grub or getting it installed on the MBR, the Boot-Repair finally worked for me. Boot into my Ubuntu system (with usb drive plugged in to successfully boot). Then remove USB drive. I then did:

sudo add-apt-repository ppa:yannubuntu/boot-repair
sudo apt-get update && sudo apt-get install -y boot-repair

Launch boot-repair. After the scan I chose “Advanced”. I re-installed Grub according to this:

  • Re-install Grub
  • Unhide Boot Menu for 10 seconds
  • Create BootInfo file
  • Separate /boot partition: sda1
  • Force GRUB into sda

Then “Apply”. Afterwards, my system did boot successfully without the USB drive plugged in. If it didn’t, maybe try another run of Boot-Repair but now, “Restore MBR” (I did this prior to re-installing GRUB).


  1. Test suspend: DONE. This works.
  2. Test hibernate:

Remote unlocking LUKS encrypted LVM using Dropbear SSH in Ubuntu

I recently performed a full disk encryption on my server using dm-crypt + LUKS. I did not address remote unlocking of the disk then because I did not know how. Remote unlocking is highly desirable I might not be physically near the server when a restart is necessary.

To remotely unlock the disk, one needs an ssh server running during startup (boot). Then, ssh into the server and unlock the disk with the passphrase. I originally was going to follow this post to perform remote unlocking via early-ssh. However, I couldn’t figure out how to do so. It appears early-ssh is no longer needed as the solution can be easily implemented with Dropbear SSH Server and Busybox in Ubuntu; see the documention at /usr/share/doc/cryptsetup/README.remote.gz.

It took me quite some time to figure out how to set things up. I first had issues with logging into the Dropbear server (normal user accounts won’t work); this post helped me figure out how to log in. Then I had a difficult time with how to unlock the disk once I’m in the server. The solution is elegantly described here and here.

Set up Dropbear SSH Server

sudo apt-get install dropbear busybox ## do not install early-ssh

There is an error in the dropbear hook script in initramfs-tools. To fix it, do

find /lib -name
## me:

At around line 30 in /usr/share/initramfs-toosl/hooks/dropbear, replace =cp lib/libnss_ “${DESTDIR}/lib/”= with =cp lib/x86_64-linux-gnu/libnss_ “${DESTDIR}/lib/”= (if early-ssh is installed, it will give further errors related to this).

Now, run:

update-initramfs -u

Enable the root account in Ubuntu as only the root user can login to Dropbear SSH Server during boot (entire disk is encrypted):

sudo passwd root
## enter root password
## to disable root account:
## sudo passwd -dl root

Now, in your laptop (not server), copy over the private key in order to login to Dropbear SSH Server:

scp user@remote.server:/etc/initramfs-tools/root/.ssh/id_rsa ~/.ssh/remote_dropbear_id_rsa

NOTE: It appears you HAVE to to use the generated private key in order to login. Login with password will not work. I also tried copying my laptop’s public key into the server’s /etc/initramfs-tools/root/.ssh/authorized_keys so that I can use my laptop’s key to login but that did not work. I might have to translate my laptop’s private key to dropbear’s formatin order for it to work. Since I have to use another file regardless, I’ll just use Dropbear’s private key.

Disable root login for OpenSSH as it is unsafe to login as root (we only allow root to login when Dropbear SSH server is running during startup and restrict root all other times):

## change in /etc/ssh/sshd_config
PermitRootLogin no

If I restart the server now, Dropbear SSH Server will run after some time when the system is waiting for the passphrase to unlock the disk. To SSH into the Dropbear server, do:

ssh -o "UserKnownHostsFile=~/.ssh/known_hosts.initramfs" -i "~/.ssh/remote_dropbear_id_rsa" root@my.server

Remote Unlocking

It appears the original method to unlock the disk does not work with Ubuntu 11.04:

ssh -o "UserKnownHostsFile=~/.ssh/known_hosts.initramfs" -i "~/.ssh/remote_dropbear_id_rsa" root@my.server "echo -ne "encryptionpassphrase" > /lib/cryptsetup/passfifo"

The error is due to Plymouth. Uninstalling or tinkering with Plymouth could cause other errors (like allowing remote unlocking to work but one loses the ability to unlock in at the server’s physical console). To get remote unlocking to work, follow the manual method described here:

## log into dropbear
## locate the process id (first column) for the /scripts/local-top/cryptroot script
kill -9 pid ## PID from previous
## look for a wait-for-root script and note the timeout on the command line; mine: 30
## wait 30 seconds
## enter passphrase
## locate process ID for /bin/sh -i
kill -9 PID

A more concise command is:

pid=`ps | grep "/scripts/local-top/cryptroot" | cut -d " " -f 3`; kill -9 $pid; sleep 35; /scripts/local-top/cryptroot; pid=`ps | grep "/bin/sh" | cut -d " " -f 3`; kill -9 $pid; exit

The disk should unlock and you can now ssh normally into the server (root not allowed!). YAY!

I’m sure one can automate this last portion using a script. Also, I would like to add a startup script that emails me when the server is waiting for a passphrase. This will be useful if the system restarts due to a power outtage without me knowing.

Strong (long) passwords


This xkcd comic brought forth a lot of discussion on the internet regarding how to choose your password. What I took away from it: it’s better to have really long passwords than to have short, complicated passwords. This post describes a reasonable system on how to set up your passwords. Basically, group your services into tiers that require different levels of security. Then set up long passwords that you can remember. Thanks to the comic and the aforementioined, I’ve come up with my own system of secure passwords. I won’t describe it due to security reasons =].

Full disk encryption on Ubuntu with dm-crypt + luks


In this post, I will outline my experience doing a full disk encryption on an Ubuntu computer. Note that this option is available through the installer only on the server edition or the alternate CD of ubuntu (not desktop).

Why would one want to encrypt their disk? A few scenarios:

  1. Suppose someone steals your laptop. Do you want them to have access to your files? With full disk encryption, they won’t even be able to boot up the laptop.
  2. Suppose you send in your disk for repair or exchange. Do you want your personal files to be freely accessible by others?
  3. Suppose the goverment wants to infringe on your right to privacy. Do you want them to easily access your files? Any access to my files will have to be consented by me.

To achieve full disk encryption, what we will do is set up an encrypted LVM. Before getting started, read this to know more about the benefits of an LVM. Then read this which explains the difference between a RAID setup and LVM. They are different things, and can be configured together. Then read this post which benchmarks the performance of the system with an encrypted and unencrypted disk. The difference in performance is negligible for the benefit of having secured data. Also look at this post which shows how one can gain access to an encrypted LVM drive; the bulk of the information came from here.

My setup: I have two 1.5TB disks set up using hardware RAID via the mobo’s BIOS. I then followed the instructions outlined here for setting up the encrypted LVM; the only difference is that I have a RAID1 configuration and did not a separate volume for “/home”. The setup is identical. Prior to trying this out, my concerns were addressed in the comments of that page and here. I ran into some issues as described in the comments of that page. Basically, I got a blank or unresponsive screen after the BIOS pages. I was not asked for my passphrase. Rebooting the computer yields the grub boot menu. Selecting recovery-mode, I was asked for my passphrase before the recovery menu appeared. I then selected boot normally and the server started. This was quite annoying because I did not want to do that many steps just to get a system booted each time. After a few hours of trying to find out what’s wrong and re-installing (thinking the culprit was the RAID setup), I found out that the passphrase is asked for in TTY7 (Control-alt-F7). I didn’t see it because I think TTY1 is Ubuntu’s Server default, hence I saw a blank or unresponsive screen. Now I know the installation process went well and it wasn’t because of RAID. However, I will have to go to TTY7, type in passphrase, and go back to TTY1 to log in. I guess this issue isn’t too problematic since it is a remote server, and don’t plan to be in front of it at each reboot. I plan to follow this post to set up early-ssh and dropbear to be able decrypt the drive via ssh. I haven’t figured out how to use it yet though because my username and password isn’t accepted by dropbear. I’ll update this post once I figure out how to login and submit the script to decrypt the drive.

In the future, I plan to add two more hard drives configured as RAID1. I guess I can just encrypt the drive like usual via dm-crypt and automount it by modifying crypttab/fstab.

UPDATE 9/9/2011 Changing passphrase by adding the new one and removing the old one

Changing a passphrase in dm-crypt was discussed here. Since I was on RAID1 and encrypted my entire LVM, I couldn’t operate on devices like /dev/sda5, etc. Actually, sda# and sdb# weren’t even in /dev/ even though they were listed in sudo fdisk -l. I tried cryptsetup luksDump on /dev/sda, /dev/sdb, and all in /dev/mapper/. The only one that was a valid LUKS device was pdc_dejidcjhg5. Thus, I did

sudo cryptsetup luksAddKey /dev/mapper/pdc_dejidcjhg5 ## added my new long passphrase
sudo cryptsetup luksRemoveKey /dev/mapper/pdc_dejidcjhg5 ## entered in passphrase I wanted removed
sudo cryptsetup luksDump /dev/mapper/pdc_dejidcjhg5 ## should show slot 0 is disabled, slot 1 is enabled

Encrypt hard drives and usb drives with dm-crypt and TrueCrypt


I recently explored encrypting usb drives and external hard drives (well, any hard drive really). I always wanted to lock all of my hard drives just to feel more secure about my files, but never got around to learning how to. In this post, I’ll describe my experience for usb drives and hard drives that aren’t used as the main disk for a computer to boot. I’ll describe full disk encryption for Ubuntu (where the OS resides) in another post.

I explored two methods. The first is using DM-Crypt with LUKS, which is the free and truly open-source solution. The second is using TrueCrypt, which is free (as in beer) and is open-source/proprietary (you can see source code, but I don’t know about modifying and redistributing). A more extensive list can be found here.

This post describes how one can go about encrypting the drive with dm-crypt. You can format the drive as FAT32, ext2, ext3, or anything really (this describes how to format a drive as FAT32 on the command line). On Linux, you just have to use cryptsetup to unencrypt the drive first. Then mount it for use. Ubuntu Desktop does this automatically (asks you for your passphrase) when the drive is plugged in. On a Windows machine, you have to install FreeOTFE in order to decrypt the drive. If the drive is FAT32, you will have access to it automatically after mounting the encrypted drive. If the drive is ext2 or ext3, you will have to install Ext2Fsd first in order to mount the drive. As of now, there isn’t a way to decrypt the drive on Mac OS X.

TrueCrypt is available on all three major platforms: Windows, Mac, and Linux. Thus, it is better than dm-crypt for usb drives in the sense that you can also use them on a Mac. You can use it to encrypt an entire disk or create an encrypted container file (pseudo partition?) to place files you want secured into. The latter approach is good for smaller files. For archives (big size), the FAT32 combination won’t work due to the 4.7GB file size limitation. Getting started with TrueCryt is quite easy as you just have to follow the GUI. To access the drive on any machine, you have to install TrueCrypt. If the drive is ext2 or ext3, again, you have to install Ext2Fsd on Windows. To mount an ext2 or ext3 drive on a Mac, you have to install ext2fuse, of which MacFuse is a pre-requisite.

What will I use? For a hard drive that I know I will only use with Linux, I’ll go the dm-crypt approach. For drives that I might bring around and plug into other platforms, I will either go with the dm-crypt + ext3 or TrueCrypt + ext3 approach. Why ext3? FAT32 is recognized on all three platforms, PS3, and most other devices that you can plug in the USB drive. However, if the disk is encrypted, the places I can plug the disk into is VERY limited (i.e. a computer). There’s no reason to prefer FAT32. Sure, I would have to install additional software on a Mac or Windows machine in order to read ext2/3. However, to unencrypt on those platforms, I would have needed to install TrueCrypt or FreeOTFE anyways, so this addtional install is trivial. Hmm, since I don’t really plug my drives into a Mac anymore, I think I’ll just stick with dm-crypt. The trick to all this is to have a spare unencrypted usb drive that you can use on a daily basis =].

Change passphrase by adding a new one and removing the old one

I thought it was impossible, but it’s quite easy. Just use cryptsetup luksAddKey /dev/MYDEVICE to add a new passphrase, and cryptsetup luksRemoveKey /dev/MYDEVICE to remove the old passphrase.

VPN service for an anonymous or untraceable internet presence

I recently considered the use a paid VPN service to connect myself to the internet. Why? The internet is a whole other world out there, and you just don’t know how much privacy you lose with all the connections you make on your computer. I see myself and the average user at a disadvantage when it comes to privacy because we aren’t savvy enough to know the underlying workings of the internet. The transmission of data packets from one device to another gives rise to the opportunity for a knowledgeable person (not me, of course ;) to decipher private information in that transmission. I’m not even referring to people stealing my password. It’s just scary to know people can find out what sites you’ve visited, what services you use, etc. Your privacy can be compromised without you even knowing it. People can figure out your daily habits and make judgment on the kind of person you are. This is quite scary. For example, your internet service provider (ISP) knows exactly what files (unencrypted) you are transporting on the internet: the source, the destination, the timing, duration, frequency, etc. However, if your connection between two devices are encrypted, for example, using SSL, then all the ISP (and other snoopers) see is a stream of data that have no idea what it is without the proper key.

When connected to a VPN, all the connections made between you and the internet is channeled through the VPN server. Thus, your footprint on the internet is that VPN server. What your ISP would see is a bunch of encrypted data that is passed from the VPN server and to your computer. I feel my personal freedom (privacy) is more guarded using such a service.

This site offers some reviews for the major services available. I ultimately tried HideMyAss due to their pricing (their yearly price ends up being like $6.55/month), the number of servers and ip addresses available, the location of these servers (30+ countries), the use of OpenVPN, their non-censoriship of connections (e.g., torrent), and the data they collect (the time you log on and the time you log off).

My original plan was to set the certificates and credentials working with an OpenVPN client on my Asus RT-N16 router running Tomato firmware. However, before getting there, I tried the service on my Ubuntu laptop using the OpenVPN and the provided scripts. I have to say, I was disappointed in the difference in speed. Without the VPN service, I download at 2+ Mbps. With it, I was downloading at 1.3 Mbps. I understand that speed loss is inevitable due to the encryption and data outing through one more server before it reaches my computer, but I was expecting 1.9 Mbps. I tried a few other servers but the speed didn’t improve. As a statistician, I should try it many, many more times. However, I didn’t have the time for it, and besides, I will only commit to the service if I get consistent speed that’s near my official bandwidth, and this obviously wasn’t the case. HideMyAss’s customer service suggested I use their “Speed Guide” functionality in their software to select the fastest server for me, but it wasn’t available for Linux. Moreover, if I were to want use the VPN service on my router, I would want to stick with a single server and forget about it, not “shop” around for the right server each time it got slow to get the best speed. Now if somehow the server selection was automatic or that they limit only a certain number of users per server to give the best speed to the users, then I think I would like the service more.

Not only did I want consistent speed, I also wanted no abrupt in service since I would run it on my router where all my internet-enabled devices depend on for the internet. I use VoIP for phone service, and I do not want to have an abrupt phone service due to the VPN service having issues. HideMyAss claims to have a 99.8% uptime rate. I assume this is really good because Google claims to have a 99.9% uptime rate.

I ultimately cancelled HideMyAss and got a refund. However, I’m still on the lookout for THE vpn service that has all the features of HideMyAss, but with negligible difference in speed. I’m sure as time goes by internet speed will only get faster, and maybe by then I wouldn’t mind not downloading at 3 Mbps if I can download at 2.5 Mbps and retain privacy.

If you have any suggestions for me, do let me know!

For now, I just have to stick with SSL-enabled sites for the exchange of private information (email, newsgroup, etc.). For the exchange of important data, I always use ssh anyways.

UPDATE: wanted to share this recent article on Lifehacker, and this useful comparison of vpn speed test that’s done periodically. I guess HideMyAss is the fastest out there, and what the speed I was observing is typical of the encryption overhead.

Real time file synchronization like Dropbox via Unison

Dropbox is a very nice tool for real time synchronization. It works very well to keep files from multiple devices (computers, phones, etc.) in sync. I use it mainly as a cloud-based backup for some of my files. However, it’s been on the headlines recently due to security and privacy concerns, leading to calls for encrypting your files prior to syncing with Dropbox.

I’ve always contemplated on running my own Dropbox-like service to have yet another safe backup of my files. Besides knowing where my data are stored exactly, I have (in theory) an unlimited amount of space. This post and this post outline solutions based on open source tools such as OpenSSH (for encrypted file transfer), lsyncd (for monitoring files), and Unison (rsync-like tool). I’ve attempted this setup, but failed to get things working with lsyncd (see the extensive discussion with the author via the comments).

I stumbled upon this post that outlines a solution based on the bleeding edge version of Unison, which includes the -repeat watch option, featuring the monitoring of files. However, the author outlined a solution for Mac OS X. I played around with the new Unison and arrived at a solution I am pretty satisfied with for my Ubuntu machines (easily extended to Mac and Windows, I’m sure). I will outline my setup in this post. Note that I have password-less ssh set up so that I can ssh into my server without typing in the password. Also, I am using Unison version 2.44.2, which I downloaded via svn around 7/16/2011.

Installing Unison

The same version of Unison must be installed on both the client and the server. Both my client and server runs Ubuntu (11.04 and 10.04 server). On the client, the folder I would like to sync is /home/vinh/Documents; the server’s destination is /home/vinh/Backup/Documents.

sudo apt-get install ocaml python-pyinotify
## install the .deb file from via `dpkg -i` if python-pyinotify is not in your repository
svn checkout
cd trunk
make NATIVE=true UISTYLE=text
## `make install` installs into $HOME/bin/
sudo cp src/unison /usr/local/bin/
sudo cp src/ /usr/local/bin/

Everything following is done on the client computer.



#! /bin/bash

## can't have extension in filename

# ssh username@server.ip -f -N -L 9922:server.ip:22 ## minimal
sudo -u local.username ssh username@server.ip -Y -C -f -N -L 9922:server.ip:22

## multiple instances can run in case of disconnect and reconnect

This script forwards my local port 9922 to the server’s port 22 via ssh. That way, I can ssh username@localhost -p 9922 if I wanted to connect to the server. I do this so that file synchronization can resume after a disconnect and reconnect (changed files does not get synced after a reconnect if I connect to the remote server directly).

Run sudo cp unisonNetworkOnPortForward /etc/network/if-up.d/ on Debian or Ubuntu. By doing this, the script will be executed whenever the computer is connected to a network (this will be different for non-debian-based distros). Note that multiple instances of this port forwarding will be present if the network is disconnected and reconnected multiple times. This makes things a little ugly, but I haven’t noticed any problems really. Also note that the script name cannot have a file extension or things will not work.

#! /bin/bash

## in /etc/rc.local, add:
## sudo -u local.username /path/to/ &

unison default ~/Documents ssh://username@localhost:9922//home/vinh/Backup/Documents -repeat watch -times -logfile /tmp/unison.log
# -times: sync timestamps
# -repeat watch: real-time synchronization via pyinotify

Add to /etc/rc.local before the last line:

sudo -u local.username /path/to/ &

This turns on unison sync at startup (unison will keep trying to connect to the server if it is disconnected). Again, this implementation is different for non-debian-based distros.

#! /bin/bash

unison -batch -times ~/Documents ssh://username@localhost:9922//home/vinh/Backup/Documents -logfile /tmp/unison.log

Run when you want to manually sync the two folders. I add the following line to cron (crontab -e) to have a manual sync everyday at 12:30pm:

30 12 * * * /path/to/

I set up this cron job because will only sync files that have changed while the unison process is running. This daily backup makes sure all my files are in sync at least once a day.

#! /bin/bash

ps aux | grep unison | awk '{print $2}' | xargs kill -9

I run this script on the client or server when I want to clean up unison processes. The one drawback about the monitor feature of unison currently is that the unison -server and process on the server is not killed when the unison process stops on the client side. After multiple connects, this will leave a lot of unison processes running on the server. Although I haven’t seen any issues with this, the script should make cleaning up the processes easier.

Start the service

Once these scripts are in their correct locations, first run to have the initial sync. Then restart the computer. You should see a unison and process by executing ps aux | grep unison on the client and server. Also, you should see an ssh process corresponding to the port forwarding by executing ps aux | grep ssh. Run touch foo.txt in the directory that you are watching and see if it appears on the server. Remove it and see if it gets deleted. Good luck!

What are some drawbacks with this setup compared to Dropbox? Well, I can’t revert back to files from a previous date, and I don’t have a dedicated Android app that I can access the files with. To solve the former, you can set up another cron job that syncs to a different location on your server every few days, giving you access to files that are a few days old. To solve the latter, I’m sure there are Android apps that allow you to access files via the sftp protocol.