Remotely access files on your home Windows computer

Most of my home computers/servers run Linux, so accessing them remotely is quite easy via the ssh protocol. Even the Windows machines I own have an ssh server installed via Cygwin.

Now, for those not familiar with Linux, one could

  • Install freeSSHd and set up an ssh server on Windows.
  • Configure an account and have freeSSHd initiate at startup.
  • Forward port 22 on the home router to port 22 on the machine with freeSSHd (assign it a static ip from the router); or, use a different external port for safety reasons (eg, 1022 -> 22).
  • Use the portable executable of WinSCP to access your files on any other Windows computer with internet access.

Server Applications

I would like to compile a list of server applications that are available out there that I might use some day (or am using now). My preferences for software are GPL-based, lightweight, easy to setup up, easy to administer, easy to use, and widely-used (to get support). Most of these were found based on my experiences on the web or from searching.

For my web server, I have Ubuntu as the OS with Apache, MySQL, and /php, i.e., LAMP).

  1. UPDATE 1/8/2011 Web server: Apache is standard and most popular. nginx is supposedly a lot faster and more stable, especially for static content; it is also easy to configure. lighttpd and Mongoose are other alternatives. Here is a list and comparison.
  2. UPDATE 1/8/2011 Database: MySQL and PostgreSQL are popular open source programs. A list here.
  3. Content publishing/management system (CMS), e.g., blogs: WordPress, Joomla and Drupal (requires more knowledge of html and css). I use WordPress as my blogging platform. Most content websites I visit are using one of these platforms. Comparisons here. UPDATE 1/19/2011 For something simpler, consider PyBlosxom, or some solution that involves emacs and org-mode.
  4. Forum: phpBB. Comparisons here)
  5. Wiki: MediaWiki, MoinMoin, dokuwiki, mojomojo. Comparisons here.
  6. Audio and video streaming: GNUMP3d MPEG4IP, Ampache, Subsonic. I currently use Ampache, but might test out GNUMP3d some day. Note you can also use VLC to stream. A list here
  7. Images/pictures/photos: Gallery and phpGraphy. A comparison here and my testing of a few different applications here.
  8. Customer relationship management (CRM): vtiger, openCRX. A short list is here and a list of lists is here.
  9. Mailing list: Mailman is king; Majordomo. A short list can be found here.
  10. Single sign-on (SSO), Central Authentication Service (CAS): If the above services are related, consider integrating OpenID using a plugin (?). Some open source softwares are OpenAM, CoSign, jasig.
  11. Data collection, surveys, forms: LimeSurvey, phpESP, RedCAP (not open source; institutions only), Form Tools, and orbeon. This list of open source healthcare software might help.
  12. UPDATE 12/20/2010 Software Configuration Management (SCM) or Revision Control Software: of course, git or maybe Subversion. A comparison here.
  13. UPDATE 12/20/2010 Issue-tracking systems (bug tracking, help desk, service desk, tickets): Trac (interface with svn by default; can add git through a plugin) and Bugzilla for for source code (bugs), and OTRS, os Ticket, Request Tracker or eticketsupport for help desk. A comparison here; another here; stack overflow post.
  14. UPDATE 12/20/2010 Collaborative Software Development (Forge): FusionForge (descendant of the open source GForge). A list can be found here).
  15. UPDATE 12/20/2010 School-related: SchoolTool, Open Admin and Open School for administration (School Management Software). Check out Edubuntu and skolelinux as OS’s for students. Moodle is a Course Management Software (CMS) or Learning Management Software. Many more (different categories) can be found on SchoolForge. Loads of useful links and information regarding Linux and Education at this post.
  16. UPDATE 1/1/2011 Mail Server: See this and this. Looks like I would need Postfix (MTA) + many others, e.g., see this.
  17. UPDATE 1/8/2011 Enterprise Resource Planning (ERP) = CRM + HRMS, of which PeopleSoft is an example: Compiere and OpenERP; a list here and here.
  18. UPDATE 1/8/2011 Job portal/board: jobberBase; a list here.
  19. UPDATE 1/19/2011 Blog aggregators: Planet Venus, used by Planet Emacsen, or some wordpress plugins, used by R Bloggers (I think).
  20. UPDATE 8/29/2011 Question and Answer site like Stack Overflow and the Stack Exchange suite: OSQA and shapado; list here.

UPDATE 1/1/2011 See this list for applications of different types.

Migrating back to Linux (Ubuntu) from Mac OS X

After abour 1.5 year with Mac OS X on my beautiful black Macbook (v 4,1), I’ve decided to migrate back to Linux, namely Ubuntu 10.04 LTS. Why? Well, I was actually using Kubuntu (KDE version of Ubuntu) from about Spring 2006 to November 2008. At the time, I remember setting up Kubuntu on my Dell Inspiron 700m laptop was quite a task: getting the display to work at the right resolution, wireless, etc. Once everything was set up I loved the environment. Afew things bothered me then: no MS Office (OpenOffice doesn’t cut it sometimes when you have to collaborate), graphics environment (KDE) crashed from time to time (probably because laptop was too slow and I multi-tasked daily) and lack of support for third-party hardware (syncing Palm Centro and ipod, scanners, etc). I would have to do lots of research to figure out the third party issue, and often, a solution was not available (maybe arriving much later). I kind of got sick of this whole process so decided to switch to Mac OS X since it was Unix-based (all tools available), had MS Office, pretty, stable, nicely built (quality), and had third-party hardware support.

Once on the Macbook, I configured it to be my workstation with all the tools I need. Since Mac wasn’t truly Unix, compiling and getting certain software needed special attention despite the availability of MacPorts. Often times I would have to compile things from source manually (no apt-get!), maybe after having to tweak the source code a little.

Now I’m quite sick of not having apt-get or tweaking source code prior to compiling. So, I’m moving back. By now, the third-party issue isn’t too much of a concern since my ipod works with it and Android syncs to the cloud (and is Linux-based). Certain things will need to be resolved for new hardware specifications on newer Laptops, but this doesn’t bother me too much as I’m sure a solution will be available in a matter of weeks or months – plus, the community is very helpful. MS Office can run with WINE/Crossover Office (I did this before too, but I think things should be better now). I would also like to note that in the mist of all this I have servers and and a NAS that are debian- or ubuntu-based, and that I use the school’s Linux servers quite often. It’s only natural that my main workstation be Linux-based again, for ease of use and for efficiency and productivity.

Moved to WordPress for technical blogging

So I’ve decided to move my technical blog to wordpress, the .org version, not the .com version; that is, to host my own instance of wordpress. The move came from various reasons:

  1. syntax highlighting for soure codes,
  2. LaTeX support for my stat/math display,
  3. post via email,
  4. post via emacs,
  5. text-to-html syntax for blogging (think org-mode),
  6. easy to backup,
  7. all these features must be permanent (ie, the services above cannot be changed and my posts no longer look “right”), and finally,
  8. extensible

Points 4, 5, and 8 were never addressed with blogger, and point 2 was not consistent (contradicts point 7).

Posterous seemed very nice when i tried it out (especially point 3), but it didn’t seem too good for points 1 and 2 (although points 2 can be addressed by mathurl or Sitmo’s Equation Editor based on this post). I tried wordpress.com once, but thought it was inadequate. I knew it was possible to install your own wordpress version on a server but thought it was too much of a hassle. However, out of the blue I looked into the above features and found that wordpress had plugins (point 8) that really much fulfilled a lot of my needs.

I will outline how I installed wordpress on my school’s server and how I customized it to give me the features I want.

I pretty much followed the instructions from wordpress. My server already had apache on it, so I installed the following in addition:

<pre class="src src-sh">sudo apt-get update

sudo apt-get install php5 sudo apt-get install mysql-server sudo apt-get install php5-mysql ## mod_rewrite Apache module already installed wget http://wordpress.org/latest.tar.gz tar xvf latest.tar.gz mysql -u root -p ## enter password

In mysql, I have to set up the user and password:

<pre class="src src-sql">CREATE USER <span style="color: #ffa07a;">'ENTER MY USERNAME'</span>@<span style="color: #ffa07a;">'localhost'</span> IDENTIFIED BY <span style="color: #ffa07a;">'ENTER MY PASSWORD'</span>;

GRANT ALL PRIVILEGES ON . TO ‘ENTER MY USERNAME’@‘localhost’ WITH GRANT OPTION; EXIT

I also have to set up a database for the blog’s data by:

<pre class="src src-sh">mysql -u USERNAME -p

CREATE DATABASE wordpress;
GRANT ALL PRIVILEGES ON wordpress.* TO "ENTER MY USERNAME"@"localhost"
 IDENTIFIED BY "Enter WordPress Password";
FLUSH PRIVILEGES;
EXIT

We also need to update the wp-config.php file:

<pre class="src src-sh">emacs -q -nw wp-config.php <span style="color: #ff4500;">## </span><span style="color: #ff4500;">follow instructions in above site, generate random key, and set table_prefix to "snc_" for supernerdycool</span>

cd ~/ cd blog.nguyenvq.com cp -rf ~/Downloads/wordpress/* ./

sudo emacs -q -nw /etc/apache2/conf.d/virtual.conf sudo emacs -q -nw /etc/apache2/conf.d/virtual.conf ## make sure it looks like following: # # We’re running multiple virtual hosts. # NameVirtualHost * ## done

## make sure following is in /etc/apache2/httpd.conf: LoadModule rewrite_module /usr/lib/apache2/modules/mod_rewrite.so

sudo emacs -q -nw /etc/apache2/sites-available/blog.nguyenvq.com place following in it: # # Example.com (/etc/apache2/sites-available/adrc.nguyenvq.com) #

# Basic setup ServerAdmin EnterMyEmail ServerName blog.nguyenvq.com DocumentRoot /path/to/blog.nguyenvq.com

# HTML documents, with indexing.

Options Indexes FollowSymLinks MultiViews

## enable site sudo ln -s /etc/apache2/sites-available/blog.nguyenvq.com /etc/apache2/sites-enabled/blog.nguyenvq.com sudo /etc/init.d/apache2 restart

## go to http://blog.nguyenvq.com/wp-admin/install.php in web browser

I initially had some errors with mysql and php, but this showed me that I was missing mysql-php in the install (apt-get); restart apache again.

To install plugins, it is easiest to have an ftp server set up for the server. I installed it via “apt-get install proftpd ## select standalone.”

I addressed what I wanted above with:

  1. SyntaxHighlighter Evolved,
  2. wp-latex,
  3. postie (require php5-imap for imap) or wordpress’s native email support (this didn’t work too well for me’) – email does not work well with wp-latex and syntaxhighlighter as this modifies the incoming text,
  4. weblogger mode in emacs (turn on xml-rpc in wordpress),
  5. Markdown for WordPress and bbPress
  6. backup by using rsync (for wordpress files) and mysqldump for the database (data),
  7. since these are installed on my own server, of course all these services are permanent (of course I can move them to another server too),
  8. other plugins: Akismet, WP-reCAPTCHA, Acronyms, Search and Replace (very good for updating all posts), Search Unleashed(uncheck “Some themes don’t display any search result content. Enable this option to force the theme to display results” in options or when visiting post from google search result, syntax highlighting will get screwed up), ShareThis (share to twitter, facebook, …, all social media), and Simple Image Widget.

The SyntaxHighlighter Evolved and Markdown didn’t play nicely together. This post’s comments has the solution, by wrapping the source code chunk with the div tag and changing the add_filter line in markdown to 9 (see below). Since this is hard to display, I will describe the characters in text:

<pre class="src src-sh">LeftAngleBracket div RightAngleBracketDiv LeftSquareBracket bash RightSquareBracket

my code LeftSquareBracket FORWARD SLASH bash RightSquareBracket LeftAngleBracket FORWARD SLASH div RightAngleBracket

Also change markdown.php:

<pre class="src src-php">add_filter('the_content_rss', 'Markdown', 6);

to

add_filter(‘the_content_rss’, ‘Markdown’, 9);

syntax highlighter’s is less than 9

See this site for the lnaguages supported.

For LaTeX, include code by:

<pre class="src src-sh">DOLLARSIGN latex ENTER LATEX CODE DOLLARSIGN

to get \(latex y=x’\beta_0\).

For Markdown syntax, look at this site for a quick reference. The main thing is to use two asterisks (\/\/) around the text for bold and one asterisk (\*) around the text for italics. For links, use

<pre class="src src-sh">LeftSquareBracket TEXT RightSquareBracket LeftParen URL RightParen

. There are a lot more, but these will be commonly used. I would like the syntax to be like that of org-mode, but I can’t have everything. I could hack the php file to modify the syntax like org-mode’s, but I’m too lazy now and don’t want to deal with the regexp’s.

I also like the notepad theme as it fits my technical blog quite nice.

backup wordpress site:

<pre class="src src-sh">rsync -av --modify-window=1 --delete username@myserver:location localLocation

## this backs up to my computer

backup database and some cron scripts (multiple scripts in the following):

<pre class="src src-sh"><span style="color: #ff4500;">### </span><span style="color: #ff4500;">http://codex.wordpress.org/Backing_Up_Your_Database</span>

mysqldump –add-drop-table -u MyUsername -p Databasename | bzip2 -c > blog.nguyenvq.com.bak.sql.bz2 ## wordpress is the database name, -p says to specify password after ## now enter password

#### restore database ### http://codex.wordpress.org/Restoring_Your_Database_From_Backup bzip2 -d blog.nguyenvq.com.bak.sql.bz2 mysql -u MyUserName -p Databasename < blog.nguyenvq.com.bak.sql ## enter password

#### setup cron job on server Backup_blog.nguyenvq.com_DB.sh: #! /usr/bin/env bash mysqldump –add-drop-table -u MyUsername –password=MYPASSWORD Databasename | bzip2 -c > /path/to/blog.nguyenvq.com.bak.sql.bz2

chmod +x Backup_blog.nguyenvq.com_DB.sh chmod o-r Backup_blog.nguyenvq.com_DB.sh

crontab -e: 00 12 * * * /path/to/Backup_blog.nguyenvq.com_DB.sh 59 23 * * * /path/to/Backup_blog.nguyenvq.com_DB.sh ## need a carriage return at end of file

#### setup cron job on my computer ## make sure passwordless ssh is on Backup_blog.nguyenvq.com.sh: #! /usr/bin/env bash INCREMENT=date +%Y%m%d%H%M DIR=“$HOME/path/to/wherever” DIR=“$HOME/path/to/wherever/Wordpress_blog.nguyenvq.com” # -q for quiet in cron rsync -q -av –modify-window=1 –delete username@server:/path/to/blog $DIR0/ scp -q username@server:/path/to/nguyenvq.com.bak.sql.bz2 $DIR/blog.nguyenvq.com.bak.sql.$INCREMENT.bz2 find $DIR -mtime +14 -exec rm -f {} \; ## delete files older than 14 days

crontab -e: 02 12 * * * /path/to/Backup_blog.nguyenvq.com.sh 02 00 * * * /path/to/Backup_blog.nguyenvq.com.sh

home server + port forwarding

so i started running my own servers, one at school and one at home to test things such as a webserver. at school, no problem. got them to give me a hostname and to open certain ports (22 and 80).

at home, since i’m on a home network which has one public ip to my router, i have to use port forwarding for the outside world to connect to my home server. i had a lot of trouble with this as i couldn’t access my home server from inside, blaming that the problem was from my westell 9100em router, the one that came with verizon fios. i tried to bridge another router (as my main router) since i thought the router was the problem. however, the instructions were too damn complicated and the actiontec instructions did not match my westell. to fix it i even managed to switch from coax connection from the ONT box to ethernet and ran my own cable, and using my own router (trendnet). however, after setting up port forwarding, things still did not work.

long story short, i got it to work learning 2 things: 1. u can’t connect to your public ip from inside the network. u have to connect to the public ip from outside the network. to test this, i ssh to my server at school and ssh back home. 2. even though u set ur router to forward the port, ur computer may still be blocking outside connections. this was the case for mac os x. u have to set it to accept all incoming connections or to allow certain services/port; turning those servers on were not enough. in mac os x, u go to system preferences > security > firewall.

for servers in your home network, you should set them up to have a static ip, preferably outside the dhcp range. for ex, dhcp should give 100-255, and use 2-99 for static. i forward 22 -> 22 (ssh) and 80 -> 80 (http) for one of my server. to get my laptop going, i set something like 80000 -> 22 and 90000 -> 80. that way i can access both computers.

this took a lot of trial and error and learning. i have to say i took like 3 attempts, each with about 3 days of work to figure out. very inefficient i must say, but now i got things working so i can channel my energy to things that are more important to me, although these aren’t done in vain…i will make use of these servers for the things i’m about to do for school.

another note. i set the router to update a dyndns, and i forward my domain to this dyndns name. this way even if my ip refreshes i can still update it.

i like my cheap trendnet router so far. since the actiontec was not at fault, i could have gone back to coax. however, i ran a cat6 cable…this should be good. even if i get fios tv, i think i can go from my router (or any router i choose) to the actiontec and still have things working. i like the fios to be ethernet based so i am free to choose any routers i want, like my cheap wireless n trendnet router.