ControlMaster in OpenSSH – speeding up editing files remotely with emacs + tramp

So I was googling around to find out how to change the shell in tramp for emacs, and I ran into this and this.

When editing remote files with emacs using tramp, opening and saving files can take a bit of time, due to re-logging in and authenticating. I discovered that you OpenSSH has a feature that allows one to re-use an existing connection to a remote host when opening new connections to that host. This is quite cool. Place the folling in ~/.ssh/config:

<pre class="src src-sh">Host *

ControlMaster auto ControlPath ~/.ssh/master-%r@%h:%p

Multiple Dropbox instances on UNIX systems (Linux, Mac OS X)

I have one main Dropbox that I use and try to grow by referrals. I use it mainly for files I feel are important (backup!) and I need accessible everywhere. Sometimes I need to collaborate with others or have additional files I need accessible everywhere (for a short period of time). I don’t want these files filling up my main account so I have side accounts (2gb). To get multiple instances on a Mac OS X I followed these instructions. For other Unix-like machines follow these instructions (includes a link for Windows too). NOTE: Don’t follow the Linux instructions on the first link.

UPDATE: For Linux, use the dropbox command from $HOME/.dropbox-dist/ (see this post). Thus, for my second Dropbox account, I do

<pre class="src src-sh">mkdir $<span style="color: #eedd82;">HOME</span>/.dropbox-alt

## content of dropbox2.sh: #!/bin/bash HOME=/home/username/.dropbox-alt /home/username/.dropbox-dist/dropbox start -i ## end of dropbox2.sh chmod 775 dropbox2.sh

Add /path/to/dropbox2.sh/ in System > Preferences > Startup Applications.

Accessing MS SQL Server from Mac OS X (or Linux): FreeTDS, unixODBC, RODBC, R

This post is a followup to a recent post I wrote. The issue at hand is that data are stored on a SQL server (Microsoft Server 2005 or version 9 of MS SQL Server to be exact) and that I would like to send queries from a command line interface, either from Mac OS X or a Linux system. The data queried from the server will most often be analyzed in R.

I will outline how to get all of this set up on a Mac OS X which should be similar to a Linux platform.

NOTE: Do not use the Macports version of the following packages – they are NOT needed and can be compiled from source without a problem.

Compile unixODBC (driver manager for different SQL servers)

First, Mac OS X comes with iODBC installed by default. However, FreeTDS is only tested on unixODBC so I’ll use this instead. I didn’t get any luck with iODBC but I didn’t give it nearly as much time as I did to get unixODBC working.

Install like usual: download, extract, ./configure, make, sudo make install. I used the default install directory /usr/local.

Compile FreeTDS (driver to communicate with MS SQL Server and Sybase databases)

First, I recommend getting the development version since all the recent updates are included. I had issues with locale Mac OS X and TDSVER=8.0 (anything higher than version 4.2 actually) specification using FreeTDS v0.82. To do so:

cvs -z3 -d:pserver:anonymous@freetds.cvs.sourceforge.net:/cvsroot/freetds checkout -P freetds
cd freetds
./autogen.sh ## generate the configure and make files
./configure --with-tdsver=8.0 --with-unixodbc=/usr/local
make
sudo make install

The configure line is taken from here. From that same documentation, we DON’T have to register the driver with the manager or add in the data source name for system wide access. I just dealt with local (home directory) files.

Recall from the previous post that I had issues with locale and TDSVER on Mac OS X. I emailed the mailing list and it was partially fixed (tsql and freebcp doesn’t work but bsqldb and isql works) in the development version of FreeTDS for TDSVER=8.0. My ~/.freetds.conf file:

[adrc]
host = IP.ADDRESS
port = 1433
tds version = 8.0
client charset = UTF-8 ##needed on MAC OS X

With this, I get query outputs in the command line:

$ bsqldb -S adrc -U USERNAME -P PASSWORD -h -t 't' -i katy.uds.sql | sed -n '2!p' > katy.uds.txt

To use isql (unixODBC), I have the following in my ~/.odbc.ini:

[adrcDSN]
Driver=/usr/local/lib/libtdsodbc.so
##Description=adrc
Trace=No
##servername=adrc
Server=IP.ADDRESS
##Address=IP.ADDRESS
Port=1433
TDS_Version=8.0
Database=IBADataDictionary
## can have above database or not

Now isql adrcDSN USERNAME PASSWORD works and looks similar to this.

Note, throughout my process of debugging, I usually precede the commands osql, isql, and tsql with environment variables such as `TDSDUMP=stderr TDSVER=8.0 isql …’ to see what’s going on.

RODBC – accessing the databases in R

The installed RODBC on my R in Mac OS X uses iODBC by default. I compiled from source to get it to work with unixODBC:

R CMD INSTALL -l ~/Rlib RODBC_1.3-1.tar.gz
## in R
library(RODBC, lib.loc="Rlib")
ch <- odbcConnect("adrcDSN", "USERNAME", "PASSWORD")

I’ve outlined how to access MS SQL Server from the command line and in R, both of what I want. Only thing missing is getting output in emacs…

Backup your gmail account in Linux/Unix or Mac OS X using getmail

Even though I have utmost trust in Google’s data centers (>= 99.9% uptime), I finally decided to backup my gmail account. I am pretty OCD when it comes to backing up my data (bi-weekly backups of my home directory into 3 external hard drives, one of which is in a water/fire-proof safe box; MOST important research-related files in the cloud via dropbox), but I’ve never backed up my mail archive which holds a lot of valuable information.

I googled it before but never got around to doing it. Finally, I’ve implemented my backup scheme, thanks to this, this, and this.

The main program that does the work is getmail. To install, issue the following in Debian (apt-get) or Mac OS X (Macports):

 <pre class="src src-sh"><span style="color: #ff4500;">## </span><span style="color: #ff4500;">Debian</span>

sudo apt-get install getmail4 ## Mac OS X via macports sudo port install getmail

mkdir ~/.getmail mkdir ~/Backup/gmail-archive emacs -q -nw ~/.getmail/getmail.gmail

Put the following in “~/.getmail/getmail.gmail” (use IMAP and save both an mbox and a maildir version of my gmail account):

 <pre class="src src-sh">[retriever]

type = SimpleIMAPSSLRetriever server = imap.gmail.com username = email.address password = my.password mailboxes = (“[Gmail]/All Mail”,) [destination] type = MultiDestination destinations = (‘[mboxrd-destination]’, ‘[maildir-destination]’)

[mboxrd-destination] type = Mboxrd path = ~/Backup/gmail-archive/gmail-backup.mbox

[maildir-destination] type = Maildir path = ~/Backup/gmail-archive/

[options] # print messages about each action (verbose = 2) # Other options: # 0 prints only warnings and errors # 1 prints messages about retrieving and deleting messages only verbose = 2 message_log = ~/.getmail/gmail.log # preserves your mail after backup delete = false # just get new mails read_all = false

Note the “read_all” option – without this, getmail downloads multiple versions of the mail and the hard drive will fill. This happened to me!

In the shell, type

<pre class="src src-sh">touch ~/Backup/gmail-archive/gmail-backup.mbox

mkdir ~/Backup/gmail-archive/tmp ~/Backup/gmail-archive/new ~/Backup/gmail-archive/cur getmail -r $HOME/.getmail/getmail.gmail

To set a cron job for midnight, create the following Backup_gmail_emails.sh and place it in cron via “cron -e”:

<pre class="src src-sh">Backup_gmail_emails.sh

#!/usr/bin/env bash # Note: -q means fetch quietly so that this program is silent /usr/bin/getmail -q -r $HOME/.getmail/getmail.gmail

chmod u+rx $HOME/Backup/Backup_gmail_emails.sh

crontab -e: 00 00 * * * $HOME/Backup/Backup_gmail_emails.sh

Accessing MS SQL Server from command line in Mac OS X and Linux/Unix

Unfortunately in my current line of work, I’ve been pulling data from MS SQL Server. My current workflow involves remotely logging into a windows machine, and from there, using the sql server software (MS SQL Server Management Studio) to issue queries and save data. This especially sucked in that I couldn’t write in the query file where I want my output results to go. I had to manually right-click the results, copy, and paste into a txt file.

Recently I started using Oracle’s SQLDeveloper with the 3rd party plugin jTDS to skip the remote login. SQLDeveloper allowed me to log directly into the SQL Server. This did not work well in that copying the results caused issues (memory?). Also, Query Analyzer was only available if the DB was Oracle’s. The added benefit was not great.

I finally found FreeTDS by googling “linux ms sql server command line.” To install it on Mac OS X, I needed to install it via Macports (sudo port install freetds). I had issues with it initially because it could not connect with the SQL Server of interest. The problem was that I needed to specify the correct TDS version; in my case, it was version 4.2 (see this linked from this thread). You can connect using two ways:

 <pre class="src src-sh"><span style="color: #eedd82;">TDSVER</span>=4.7 tsql -H myserverIP -p 1433 -U myusername -P mypassword

or

 <pre class="src src-sh">tsql -S myserverinlist -U myusername -P mypassword

with the following in “~/.freetds.conf”

 <pre class="src src-sh">[adrc]

host = myserverIP port = 1433 tds version = 4.2

See this for some examples. If you have connection issues, make sure you could telnet to the server using the right port first. See this. More information on the freetds.conf file can be found here.

For a query, remember to put go at the end of the query for it to execute (credit).

Be careful when routing input and output (“<” and “>”) as other messages from the tsql command will be present. Will post a solution when I find out.

Update 4/16/2010 – Batch output results from a query

I found out FreeTDS comes with the commands freebcp (bcp emulation/replication) and bsqldb. To get batch results, issue

bsqldb -U username -P password -S server.from.TDS.list -h -t '\t' -i query.sql -o query.out
## in 2nd line, has "-" characters (separate headers and data). to remove them:
bsqldb -U username -P password -S server.from.TDS.list -h -t '\t' -i query.sql | sed -n '2!p' > query.out

## or use freebcp
-- freebcp "$(cat query.sql)" queryout query.out -S server.from.TDS.list -U username -P password -c
## for freebcp, do not use "GO" in query

The above freebcp method does not work for me with the error Cannot bcp with TDSVER < 5.0.

Credit for for bsqldb goes to here. Removing the 2nd line is from here. This showed me how to get the correct freebcp command line syntax.

Next up is getting odbc (unixODBC to be exact) working!

Teaching with a graphic tablet and a projector: replacing a whiteboard or overhead projector

Many times when I teach, I often want to keep a copy of what I write on the board. Why? I can have a softcopy to refer to when students have follow-up questions, I can re-use these notes for the next discussion if there are only a few people, and I can re-use them in later quarters. In high school, I always liked the overhead projectors. Teachers can write and look get feedbacks from the students face to face. Also, you can just scroll your super long sheet up as you write and not worry about erasing like on the white board. Later in lecture, you can pull them back if you want need to refer to the previous notes. I always had an idea to use some kind of a tablet device with the projector, which all modern classrooms should have. I finally decided to buy a fairly inexpensive Genius graphic tablet from Walmart for $40. Wacom tablets are probably better, but they are very expensive, especially the larger ones. I guess they are expensive because they are meant to be used with photoshop by artists. In my case, I need a large tablet but it doesn’t have to be premium since all I’ll be doing is writing words and math symbols on it. My platform is a Mac OS X, and this link discusses some tablets and programs. This site discusses the tablet PC’s use in the classroom altogether. After some trials, the best program seems to be inkbook, which costs $40. However, I guess the Genius driver doesn’t play well with inkbook. To get it to work, I have to turn on “Handwriting Recognition” in System Preferences > Ink. In Inkbook, i can write in the ink mode. I see a yellowpad whenever I write. This SHOULD NOT be there, but it works for now. Wacom would play nicer. Update 3/27/2010: I returned the Genius tablet a few weeks ago as it does not play nice with inkbook. The author recommended a Wacom. However, it is too expensive for me (a decent sized one) and I won’t need one yet since I don’t need to TA next quarter. Hopefully, the Apple Ipad will have a pen with it so I can use it with this (main reason I would buy one). Otherwise I would have to consider getting a cheap netbook with a pen.

email yourself when a script or a job is done

i usually run long simulations in R on a remote server. i’ve read an article on linux journal about emailing yourself after something happens…i forgot. since my jobs are long, and i usually have to run multiple jobs sequentially, it’d be nice to look at my results immediately when they’re gone. good thing *nix based systems have the mail (or mailx) command:

<pre class="src src-sh"><span style="color: #b0c4de;">echo</span> <span style="color: #ffa07a;">""</span> | mail -s <span style="color: #ffa07a;">"Job Done"</span> address@mydomain.com

found the stuff here.

in R, i execute:

<pre class="src src-sh"><span style="color: #87cefa;">system</span>(<span style="color: #ffa07a;">'echo "" | mail -s "Job Done" address@mydomain.com'</span>)

of course, these are empty emails. u can use “<” or cat with pipe to get results emailed to you. cool!

file management: emacs dired to replace Finder in mac os x (and other OS)

I hate Finder in Mac OS X. I mean, it looks nice and all, but it is not customizable. I really liked the KDE window management back when I used Kubuntu because I can select/de-select files with the keyboard, cut/paste/copy files with the keyboard, have shortcuts to different locations, etc. Finder has some keyboard shortcuts, but not at all flexible. You can copy and paste files, but you can’t cut and paste files. Selecting/de-selecting files is not as flexible (eg, skipping a file). I can’t go to the sidebar window where all my bookmarked locations reside using the keyboard. Finder, and in general Mac OS X, is not so flexible. This bugs me a lot. I wanted to get stumpwm working in OS X so that things are more keyboard-oriented. However, it doesnt work with all the native mac apps. The fix is with dired mode in emacs. I spend most of my time in emacs already. With dired, I can even manage files inside emacs (I can do so in the shell, but this makes browsing and executing commands on files a breeze). Use “C-x f” and navigate to a location rather than a file. Learn all the keyboard shortcuts for delete, rename, execute shell command, etc. You can even cut/copy/paste with wuxch-dired-copy-paste. The functions below also allow me to open files the Mac way, with the “open” command. This is cool because files will be opened with the native Mac default application.

<pre class="src src-sh">;; http://www.emacswiki.org/emacs/DiredPlus

;; copy and paste files in dired ;; needed additional files, such as dired+, dired-details+, find-dired+, and find-dired- (require ‘wuxch-dired) (require ‘wuxch-dired-copy-paste) ;;http://stackoverflow.com/questions/1824696/function-to-call-same-shell-command-in-dired ;; (defun dired-do-shell-mac-open-vqn () ;; (interactive) ;; (dired-do-async-shell-command ;; “open” current-prefix-arg ;; (dired-get-marked-files t current-prefix-arg))) (defun dired-do-shell-mac-open-vqn () (interactive) (save-window-excursion (dired-do-async-shell-command “open” current-prefix-arg (dired-get-marked-files t current-prefix-arg))))

(define-key dired-mode-map (kbd “s-o”) ‘dired-do-shell-mac-open-vqn)

I can use Apple-o/Windows-o to open files with the default mac program for the marked files.I also use emacs’ bookmark system (“C-x r b” and “C-x r m”) to bookmark locations.Note that all this is not specific to Mac OS X. Dired’s been around forever – I just never picked it up really. Now with my “open” trick and the bookmark system, this should really replace my use of Finder in Mac OS X.

Update 11/10/2010 – On Linux

I’m back on Linux now. A few updates on launching files with the default application. Add the following to the emacs init file:

(defun dired-do-shell-launch-file-default ()
 (interactive)
 (save-window-excursion
 (dired-do-async-shell-command
 "$HOME/Documents/bin/open.sh" current-prefix-arg ;; linux;; multiple files
 ;; "nohup xdg-open" current-prefix-arg ;; linux can open multiple files, but one at a time
 ;; "see" current-prefix-arg ;; linux;; can open at most 1 file (being opened)
 ;; "open" current-prefix-arg ;; mac os x
 (dired-get-marked-files t current-prefix-arg))))
(define-key dired-mode-map (kbd "s-o") 'dired-do-shell-launch-file-default)

Need the script open.sh (thank you fledermaus from #emacs):

#! /bin/bash

for file in "$@"
do
nohup xdg-open "$file" &
done
sleep 1

Now, to unmount devices in emacs:

;; unmount disk in dired
;;http://loopkid.net/articles/2008/06/27/force-unmount-on-mac-os-x
(defun dired-do-shell-unmount-device ()
 (interactive)
 (save-window-excursion
 (dired-do-async-shell-command
 "umount" current-prefix-arg ;; linux
 ;; "diskutil unmount" current-prefix-arg ;; mac os x
 (dired-get-marked-files t current-prefix-arg))))
(define-key dired-mode-map (kbd "s-u") 'dired-do-shell-unmount-device)

If an error is returned, make sure the device is not being used anywhere, including opened in nautilus or in some terminal.

The ability to launch files with the default application (look into xdg-open, gnome-open, and see), unmnount devices, and being able to copy/paste files in emacs help makes dired the perfect file manager. I will use this from now on over Nautilus. It’ll compliment using stumpwm.

flags for compiling source code: pkg-config command

so i was trying to compile some C code on my server at school that involves the math and the GNU Scientific Library (gsl) libraries. that is, the source has the following lines:

 <pre class="src src-C"><span style="color: #b0c4de;">#include</span> <span style="color: #ffa07a;">&lt;stdio.h&gt;</span>

#include <math.h> #include <gsl/gsl_integration.h>

those link the header files. however, when compiling, gcc would complain that they can’t see functions such as sqrt, abs, etc, from the math library. it also complains about my call to a function from the gsl library. i googled the math library stuff and realized i need to pass the ‘-lm’ argument in gcc for the math library. similarly, i saw somewhere that refers to the ‘-gsl’ argument. however, this did not work for me. i tried many things. some more googling, i found this thread that shows the ‘pkg-config’ command that shows exactly what arguments u need for which library.

So, i get:

 <pre class="src src-sh">$ pkg-config --libs gsl

-lgsl -lgslcblas -lm

so i just need to execute:

 <pre class="src src-sh">gcc -lgsl -lgslcblas -lm myprogram.c

i really don’t understand how the GNU compilers work. on my mac, i don’t have to pass those arguments. on my debian server, i need to. at first i thought that including the header was enough. will need to learn more about this stuff.

PS I also remember using this command when i had a hard time compiling shell-fm on my mac os x.

UPDATE: so on my mac os x, after installing gsl, i need to use ‘gcc -lgsl myprog.c’ to use the gsl library. things are fine now. I also want to note that i could not compile GSL form source on my debian server…had to rely on apt-get.

home server + port forwarding

so i started running my own servers, one at school and one at home to test things such as a webserver. at school, no problem. got them to give me a hostname and to open certain ports (22 and 80).

at home, since i’m on a home network which has one public ip to my router, i have to use port forwarding for the outside world to connect to my home server. i had a lot of trouble with this as i couldn’t access my home server from inside, blaming that the problem was from my westell 9100em router, the one that came with verizon fios. i tried to bridge another router (as my main router) since i thought the router was the problem. however, the instructions were too damn complicated and the actiontec instructions did not match my westell. to fix it i even managed to switch from coax connection from the ONT box to ethernet and ran my own cable, and using my own router (trendnet). however, after setting up port forwarding, things still did not work.

long story short, i got it to work learning 2 things: 1. u can’t connect to your public ip from inside the network. u have to connect to the public ip from outside the network. to test this, i ssh to my server at school and ssh back home. 2. even though u set ur router to forward the port, ur computer may still be blocking outside connections. this was the case for mac os x. u have to set it to accept all incoming connections or to allow certain services/port; turning those servers on were not enough. in mac os x, u go to system preferences > security > firewall.

for servers in your home network, you should set them up to have a static ip, preferably outside the dhcp range. for ex, dhcp should give 100-255, and use 2-99 for static. i forward 22 -> 22 (ssh) and 80 -> 80 (http) for one of my server. to get my laptop going, i set something like 80000 -> 22 and 90000 -> 80. that way i can access both computers.

this took a lot of trial and error and learning. i have to say i took like 3 attempts, each with about 3 days of work to figure out. very inefficient i must say, but now i got things working so i can channel my energy to things that are more important to me, although these aren’t done in vain…i will make use of these servers for the things i’m about to do for school.

another note. i set the router to update a dyndns, and i forward my domain to this dyndns name. this way even if my ip refreshes i can still update it.

i like my cheap trendnet router so far. since the actiontec was not at fault, i could have gone back to coax. however, i ran a cat6 cable…this should be good. even if i get fios tv, i think i can go from my router (or any router i choose) to the actiontec and still have things working. i like the fios to be ethernet based so i am free to choose any routers i want, like my cheap wireless n trendnet router.