using R + ess-remote with screen in emacs

Dear list,

I brought up this issue before but a good solution never arised: being able to use screen on a remote server (so if something goes wrong on my side I can always resume that R session) inside of emacs in order to utilize ESS. The closest thing I found to a good work flow was to use ansi-term or multi-term and copying and pasting code in Emacs (ESS only worked nicely with shell-mode; reason at the end). Some would advise to use screen and open emacs inside of that screen, and voila, you have the luxuries of screen (attaching the session anywhere) and emacs+ESS (keybindings, etc). However, I prefer to use one main emacs session on my laptop/netbook (all my configurations are there), where I can have multiple projects and multiple R jobs opened at once.

I would like to share what I have working (for the time being), with the help of Michael Zeller in case others are interested.

  1. Place the following in .emacs:
  <pre class="src src-sh">;; used to send screen keybindings to shell<span style="color: #00ffff;"> in</span> emacs

(define-key shell-mode-map (kbd “C-l”) (lambda (seq) (interactive “k”) (process-send-string nil seq))) (define-key inferior-ess-mode-map (kbd “C-l”) (lambda (seq) (interactive “k”) (process-send-string nil seq)))

  1. In xterm (or the likes), ssh to the remote server and start screen. Detach it. (Need to do this first as starting the initial screen in emacs shell-mode becomes very ugly with the printing; resuming the same screen session also becomes messy in xterm)
  2. In emacs, M-x shell. Set:
  <pre class="src src-sh">$ <span style="color: #eedd82;">TERM</span>=xterm <span style="color: #ff4500;">## </span><span style="color: #ff4500;">give clearing capabilities to shell-mode</span>

  1. ssh to remote server. screen -r to resume the screen session. start R. M-x ess-remote.
  2. Send R code from R source files like before!
  3. To detach or do anything screen related, precede EACH keybinding with C-l. For example, C-a C-d to detach will now be C-l C-a C-l C-d. Yes this is cumbersome, but I don’t imagine screen keybinding to be used much at this stage since we are developing and debugging R code for say a simulation study =].

I would also like to note (for archival reasons) that ess-remote does not work with ansi-term and multi-term because of the inferior-ess-mode command, which stems from comint-mode and inferior-ess-send-input. If you remove this command in ess-remote, you don’t get an error but u can only send one line of code at a time from the R file.

Hope this helps someone out there.

Stream music to client via DAAP

Now that I have a NAS at home, I plan to place stuff that takes up a lot of space that doesn’t get used so often on there. However, with that comes the issue of file access. Since my NAS is linux based – ubuntu server edition to be exact – I can access files via ssh, sftp, ftp, sshfs (map the drive), samba, etc. I should probably write a blog post about sshfs and samba soon.

However, what if I want to listen to music in my music player, say itunes on Mac/Windows or rhythmbox (or amarok, bashee, etc) on Linux? The good thing about itunes and these bloated music players are that they have a database of all your music files, so finding the song is fairly quick (type in artist name or song title). Double clicking the files on the mounted drive is no problem, but having this “search and play” capabilities is quite useful. Luckily, there are many solutions to this, one of which is the DAAP method. Read here for an overview.

Game plan is as follow:

  1. Set up music server on NAS.
  2. Be able to access the songs in itunes or the likes on remote servers in the same network.
  3. Be able to access the songs in itunes or the likes on remote servers when on a different network, say at school (or at work).

The first point is fairly easy on Ubuntu: install Firefly Music Server (formerly known as mt-daapd). Following these instructions, I did

 <pre class="src src-sh">sudo apt-get install mt-daapd

sudo apt-get install libid3tag0 sudo apt-get install avahi-utils ## needed for remote access sudo adduser media su media mkdir /home/media/music exit

Place music in /home/media/music or create a symlink to the music folder.

 <pre class="src src-sh">sudo /etc/init.d/mt-daapd start

sudo update-rc.d mt-daapd defaults

Now, from that computer, go to http://ip.address:3689 (username and password is mt-daapd). On this site, we can set up the configuration. Set username/password, password to listen, and music folder. Then scan the files. (Might have to do sudo /etc/init.d/mt-daapd start).

From here, computers on the network should be able to see the music server in itunes or rhythmbox (point 2) and play music (after entering music password). NOTE: I initially thought that you had to “Open Audio Stream…” or the likes on the music players, but that is for internet radio or something; DAAP servers will broadcast the music server to the network.

To do point 3, the first thing to do is set up port forwarding on the router (3689 to 3689 on the music server). The remaining part is a bit tricky. First, you have to do ssh tunneling so that the remote music server appears to be on the same network as your computer/laptop. Then we have to forward the “broadcast” from the DAAP server (the server that itunes sees, for example).

For a Linux client, instructions can be found here and here:

 <pre class="src src-sh">ssh my@daap.server.no-ip.org -N -f -L local.ip:6689:localhost:3689

avahi-publish-address -v -H “name of your host”.local -s “My daap share” _daap._tcp. 6689 & ## find name of host from echo $HOSTNAME

Now, itunes or other clients on the remote server should see “My daap share” on the Shared list. NOTE: currently, Linux players such as rhythmbox and banshee can’t play music this way. I’m sure a fix should be coming soon. If a mac is also on the same network as this current linux client, it will see the shared daap server and connect and play music.

For a Mac client, instructions can be found here:

 <pre class="src src-sh">dns-sd -P <span style="color: #ffa07a;">"Home iTunes"</span> _daap._tcp local 3689 localhost 127.0.0.1 <span style="color: #ffa07a;">"Arbitrary text record"</span> &amp;

ssh -C -N -f -L 3689:localhost:3689 myusername@blahblahblah.dyndns.org

Itunes should be able to see the remote daap server.

For a Windows machine, I’m sure one would have to rely on RendezvousProxy, following these instructions. I will test this out soon (setting this up for my girlfriend).

ControlMaster in OpenSSH – speeding up editing files remotely with emacs + tramp

So I was googling around to find out how to change the shell in tramp for emacs, and I ran into this and this.

When editing remote files with emacs using tramp, opening and saving files can take a bit of time, due to re-logging in and authenticating. I discovered that you OpenSSH has a feature that allows one to re-use an existing connection to a remote host when opening new connections to that host. This is quite cool. Place the folling in ~/.ssh/config:

<pre class="src src-sh">Host *

ControlMaster auto ControlPath ~/.ssh/master-%r@%h:%p

home server + port forwarding

so i started running my own servers, one at school and one at home to test things such as a webserver. at school, no problem. got them to give me a hostname and to open certain ports (22 and 80).

at home, since i’m on a home network which has one public ip to my router, i have to use port forwarding for the outside world to connect to my home server. i had a lot of trouble with this as i couldn’t access my home server from inside, blaming that the problem was from my westell 9100em router, the one that came with verizon fios. i tried to bridge another router (as my main router) since i thought the router was the problem. however, the instructions were too damn complicated and the actiontec instructions did not match my westell. to fix it i even managed to switch from coax connection from the ONT box to ethernet and ran my own cable, and using my own router (trendnet). however, after setting up port forwarding, things still did not work.

long story short, i got it to work learning 2 things: 1. u can’t connect to your public ip from inside the network. u have to connect to the public ip from outside the network. to test this, i ssh to my server at school and ssh back home. 2. even though u set ur router to forward the port, ur computer may still be blocking outside connections. this was the case for mac os x. u have to set it to accept all incoming connections or to allow certain services/port; turning those servers on were not enough. in mac os x, u go to system preferences > security > firewall.

for servers in your home network, you should set them up to have a static ip, preferably outside the dhcp range. for ex, dhcp should give 100-255, and use 2-99 for static. i forward 22 -> 22 (ssh) and 80 -> 80 (http) for one of my server. to get my laptop going, i set something like 80000 -> 22 and 90000 -> 80. that way i can access both computers.

this took a lot of trial and error and learning. i have to say i took like 3 attempts, each with about 3 days of work to figure out. very inefficient i must say, but now i got things working so i can channel my energy to things that are more important to me, although these aren’t done in vain…i will make use of these servers for the things i’m about to do for school.

another note. i set the router to update a dyndns, and i forward my domain to this dyndns name. this way even if my ip refreshes i can still update it.

i like my cheap trendnet router so far. since the actiontec was not at fault, i could have gone back to coax. however, i ran a cat6 cable…this should be good. even if i get fios tv, i think i can go from my router (or any router i choose) to the actiontec and still have things working. i like the fios to be ethernet based so i am free to choose any routers i want, like my cheap wireless n trendnet router.

passwordless ssh

I can’t believe I never set this up. I ssh to a few of my school servers all the time. I always have to enter my password to authenticate. I just found out I could authenticate with a key file instead of entering my password. I found the information from here and here. To set up, execute:

# for no passphrase, use
ssh-keygen -b 1024 -N ""
# safer to use with passphrase, and make it long (24+ characters)!
ssh-keygen -t rsa -b 4096
ssh-copy-id login@server
# you'll have to enter your password one last time to get it there.

On my Mac OS X, I did not have ssh-copy-id, but from this link I found a shell script for it here.

edit files remotely: emacs + tramp

Suppose I want to edit a file remotely. I don’t want to download/ftp the file to my computer, edit, and send it back to the remote server. In emacs, I can edit it remotely using tramp via the ssh or rcp protocol. Put following in the .emacs file after installing tramp.

<pre class="src src-sh">;; tramp stuff

;; http://ubuntuforums.org/showthread.php?t=760038 (require ‘tramp) (setq tramp-default-method “ssh”)

Read a remote file by C-x C-f /user@your.host.com:/path/to/file. Note we Need that ‘/’ before username. This is a good reference for tramp.

Run a remote R session in emacs: emacs + ESS + R + ssh

I don’t know how, but somehow, I stumbled on how to run a remote R session in emacs.

Since Spring 2006 I’ve always used emacs with ESS to run R (did it on windows, switched to linux for years, and most recently, on my macbook). I liked this workflow because I get the same usual interface across multiple platforms. Plus, I use emacs for everything computery or scientific, like using coding Python or C.

Regarding the same interface across multiple platforms, I use, for example, emacs + ESS + R whenever I remotely log into the remote servers dedicated to computing. I just need to ssh into the server, fire up emacs and fire up R. However, I almost always write all of my code on the local computer, and when I’m ready to run the final code, I either run it as a batch script (utlizing nohup and &), through screen (to keep the session runnning after I log out, see my post on R with unix tools), or through emacs. These days, I’ve been doing it with screen mainly so disconnects to the server won’t interrupt my script.

I just found yet another way to do this: write code on my local computer and then send code to a remote R session in my local emacs. I ran into this by googling ‘emacs ess multiple R session’. Instructions are described here (section 3.3: ESS on remote computers). We need this file for things to work. This site clarified how to get graphics to work.

Instructions as follow:

  1. Download the ssh.el file.
  2. Install it the usual way or place it in ~/elisp or ~/.emacs.d. In your emacs init file, add:
 <pre class="src src-sh">;; add path to emacs

;; http://edward.oconnor.cx/2005/09/installing-elisp-files (add-to-list ‘load-path “~/elisp”)

;; load ssh.el file from elisp ;; this is to run ESS remotely on another computer in my own emacs, or just plai n old reading remote files ;; (require ‘ssh)

  1. Fire up emacs. Type ‘M-x ssh’. For the host settings, do something like ‘-X -C username@server.com’ (X is for X windows forwarding, C is for compression of graphics, so plots can be displayed faster). Type in password.
  2. You wil be logged into a shell session on your server. Fire up R by typing R then enter. R is now running in an emacs buffer. Type M-x ess-remote. For dialect, select r.
  3. Open up any .R file on your computer, and use the usual keyboard shortcuts to send code to the remote R session.

W can also achieve the same results without the ssh.el file. In emacs, type M-x shell. In the shell buffer, ssh into the server and then run R. Type M-x ess-remote and everything should still work.

Next thing to get working is to open remote files in my local emacs.