Find text or string in files of a certain type

One can use grep "mystring" myfile.ext to find the lines in myfile.ext containing mystring. One could also use grep "mystring" *.ext to find mystring in all files with extension ext. Similarly, one could use grep "mystring" /directory to search for mystring in all files in the directory. What if one wants to search for mystring in all *.ext files in a certain path /directory? Most posts online would suggest something along the line of

<pre class="src src-sh">find /directory -type -f -name <span style="color: #ffa07a;">"*.ext"</span> | xargs grep <span style="color: #ffa07a;">"mystring"</span>

However, the comments of this post shows how one could do it with grep:

<pre class="src src-sh">grep -r --include=*.ext <span style="color: #ffa07a;">"mystring"</span> /directory

Guide to accessing MS SQL Server and MySQL server on Mac OS X

Native GUI client access to MS-SQL and MySQL

We can use Oracle SQL Developer with the jTDS driver to access Microsoft SQL Server. Note: jTDS version 1.3.0 did not work for me; I had to use version 1.2.6. Detailed instructions can be found here.

We can use MySQL Workbench to access MySQL server. Setup is intuitively obvious.

Overview of ODBC on Mac OS X

Mac OS X has iODBC installed as it’s default ODBC manager. Most other Linux/UNIX system uses unixODBC to manage the ODBC drivers. This is the main reason why there’s so much confusion on getting ODBC to work on Mac OS X.

ODBC is kind of like an API for any software to access any DBMS easily, regardless of what DBMS it is and what OS it’s running on. Different software (e.g., R or Python) can utilize ODBC to access different DBMS through the following logic: Software -> ODBC Manager -> ODBC Driver for the DBMS -> DBMS Server (Software: R, Python, etc.; DBMS: MySQL, MS-SQL, etc.).

It doesn’t matter whether you use iODBC or unixODBC. Whichever one you use, just make sure the DBMS Driver and software you are using are configured/compiled to use with the same ODBC manager (usually set through the configure flags). For example, the R package RODBC and Python package pyodbc are compiled by default to use iODBC on Mac OS X. The DBMS drivers used must be compiled for use with iODBC. For iODBC, one could add data source names (DSN’s) at ~/Library/ODBC/odbc.ini. For unixODBC, one could add DSN’s at ~/.odbc.ini.

My current setup utilizes iODBC. I will outline the instructions for setting up MySQL and freeTDS (MS-SQL) drivers for use with RODBC and pyodbc through iODBC.

MySQL and FreeTDS with iODBC on Mac OS X

Install the MySQL Connector/ODBC driver. Driver should be at /usr/local/lib/libmyodbc5.so or /usr/local/lib/libmyodbc5w.so. Note: I’m unable to compile the driver from source on Mac OS X.

FreeTDS is an open source ODBC driver to access MS SQL Server. Install via Home Brew:

## install homebrew
ruby -e "$(curl -fsSL https://raw.github.com/mxcl/homebrew/go)"

## install freetds
brew install freetds

Driver should be at /usr/local/lib/libtdsodbc.so (symbolic linked).

Create ~/Library/ODBC/odbc.ini:

[sqlserver01]
Driver=/usr/local/lib/libtdsodbc.so
TDS_Version=4.2
Server=ip.address
Port = 1433
Trace = Yes
Description=my description
# Database=
# can't specify username and password for freetds

[mysql01]
Driver=/usr/local/lib/libmyodbc5.so
Server=hostname
Port=3306
charset=UTF8 
User=username
Password=password
# Database=
## can specify an actual database to each DSN

Install pyodbc via sudo pip install pyodbc. Test connections in python:

import pyodbc as p

con1 = p.connect("DSN=sqlserver01;UID=username;PWD=password")
con1.execute("select name from master..sysdatabases").fetchall()

con2 = p.connect("DSN=mysql01;UID=username;PWD=password")
con2.execute("show databases;").fetchall()

Install R using the installer. Install RODBC in the R interpreter via install.packages("RODBC"). Test connections in R:

library(RODBC)

ch1 <- odbcConnect(dsn="sqlserver01", uid="username", pwd="password")
odbcQuery(ch1, "select name from master..sysdatabases")
odbcFetchRows(ch1)

ch2 <- odbcConnect(dsn="mysql01", uid="username", pwd="password")
odbcQuery(ch2, "show databases;")
odbcFetchRows(ch2)

More on unixODBC on Mac OS X

If one wants to use unixODBC on Mac OS X instead, note the following:

  • First install unixODBC via Homebrew with brew install unixodbc.
  • Compile R from source to have it work with unixODBC (R binaries from the installer uses iODBC by default).
  • Can choose --with-odbc-manager=odbc when compiling RODBC.
  • When compiling freeTDS, include the argument with-unixodbc (pass to Homebrew or when compiling manually).
  • I’m unable to compile the MySQL Connector driver on Mac OS X from source (Homebrew or manually). Thus, it won’t work with unixODBC. I believe I tried unixODBC and MySQL Connector from macports, and those work.
  • pyodbc only works with iODBC on Mac OS X (inspect setup file). Currently I can’t get pyodbc to work with unixODBC on Mac OS X.

More differences between unixODBC and iODBC

unixODBC comes with the isql command to access different DBMS from the command line interpreter. iODBC comes with the iodbctest and iodbctestw commands. The command isql works for me on Mac OS X when I set freeTDS up to work with unixODBC (e.g., accessing MS SQL Server). I couldn’t access MySQL server because the MySQL Connector driver was compiled for use with iODBC.

If I use iODBC, I get the following for trying to access a MySQL server:

$ iodbctestw "DSN=sqlserver01;UID=username;PWD=password"
iODBC Unicode Demonstration program
This program shows an interactive SQL processor
Driver Manager: 03.52.0607.1008
1: SQLDriverConnectW = [MySQL][ODBC 5.1 Driver]Prompting is not supported on this platform. Please provide all required connect information. (0) SQLSTATE=HY000
1: ODBC_Connect = [MySQL][ODBC 5.1 Driver]Prompting is not supported on this platform. Please provide all required connect information. (0) SQLSTATE=HY000

When I try to access SQL Server, I get

$ iodbctestw "DSN=sqlserver01;UID=username;PWD=password"
iODBC Unicode Demonstration program
This program shows an interactive SQL processor
Driver Manager: 03.52.0607.1008
1: SQLDriverConnectW = [FreeTDS][SQL Server]Login failed for user 'username'. (18456) SQLSTATE=42000
2: SQLDriverConnectW = [FreeTDS][SQL Server]Unable to connect to data source (0) SQLSTATE=08001
1: ODBC_Connect = [FreeTDS][SQL Server]Login failed for user 'username'. (18456) SQLSTATE=42000
2: ODBC_Connect = [FreeTDS][SQL Server]Unable to connect to data source (0) SQLSTATE=08001

Don’t know why that is so. I guess it’s not too important to use an interactive interpreter. What matter is that the driver works with R and Python. Perhaps I should consider sqsh or do more searching.

Delete .DS_Store files

I have a bunch of .DS_Store files in many directories in my home folder because I transferred them from my old system, a Mac OS X. To remove them, do

 <pre class="src src-sh">find . -name .DS_Store -exec rm <span style="color: #ffa07a;">"{}"</span> <span style="color: #ffa07a;">';'</span>

in ~/. Got this in the comments of this post.

Compile R on Mac OS X

I did this before on my Macbook, but never documented. Wanted to document my attempt on the Mac virtual machine.

  1. Install Xcode (gcc/g++) and gfortran per this site.
  2. Download the R source file (2.12 in this example), and extract it.
  3. ./configure did not work (Fortran issue). Per this, this, and this, I used: ./configure --with-blas‘-framework vecLib’ –with-lapack –with-aqua –enable-R-framework F77=”gfortran -arch x86_64″= (the F77 is needed!).
  4. make then sudo make install.
  5. Add /Library/Frameworks/R.framework/Versions/Current/Resources/bin to PATH in ~/.bashrc.

Mac OS X in Linux via VirtualBox (as guest OS)

I recently installed a Windows virtual machine on my Linux laptop. I wanted to test out a virtualized Mac OS X machine via VirtualBox as well (just in case I ever needed to do something that can only be done on a Mac). My main references are this post and this post. Although they virtualize on a Windows host machine, it also worked on my Ubuntu 10.10 host laptop.

Things I needed:

  1. A Mac OS X 10.6 (snow leopard) installation dvd (RETAIL version, not the one that comes with the Apple machine). Buy a copy and rip it to iso.
  2. nawcom ModCD to be able to boot the installation DVD. The EFI that came with VirtualBox and the Empire EFI from the first post did not work.
  3. The latest myHack. I used version 1.1 (r123). Install this before the Apple update.
  4. Mac OS X 10.6.5 Update. Don’t do OS upgrade via the Automatic Updater, and don’t update without consulting the myHack website; things can break.

Some notes to myself:

  1. When creating the hard drive, use “dynamic” and just make it as big as possible. I originally did 20gigs and had to re-do to 100gigs. Since it is dynamic, you won’t take up space on your host system until you the space get filled up on the guest. Although there is a way to resize the hard drive on a Windows guest, I wasn’t able to do so on the Mac guest (different filesystem and there is that boot loader issue). Looking back in hindsight, I should’ve just made it 500gigs and move the virtualization on an external disk if I ever reach that (I highly doubt it!).
  2. To access files on the host, set up samba on the host and connect to it on the guest.
  3. I used dmg2img to convert dmg to iso (when needed).
  4. Follow the second post to change the resolution (modify /Extra/com.apple.Boot.plist and run =VBoxManage setextradata “MacOSX64″ “CustomVideoMode1″ “1366x768x32″= on the host). “1366x768x32″ is the resolution on my Toshiba Portege R705 running Ubuntu 10.10. Now I can go fullscreen.
  5. Sound most likely won’t work. I have no use for it anyways.

Good luck. I hope I won’t have to use it much, but it’s nice to do everything in one machine, running inside of Linux (as opposed to having multi-boot).

Virtualization with VirtualBox (running Windows inside Linux)

I recently re-installed Windows on an old laptop to use Adobe Acrobats and the likes. Some Windows programs just aren’t accessible on Linux via Wine (although MS Office works). I’ve decided to set up Windows as a virtual machine on my Linux laptop; let’s see if virtualization will take the separate Windows machine out of the picture. I went with VirtualBox since it is open source, free, and was mentioned on Linux Journal. My initial reaction is VERY favorable. It seems everything works flawlessly (internet, cd drive, accessing files from the host machine; usb supported only through PUEL proprietary version.). I’ll jot down what I would like to be able to do with VirtualBox but have not been able to yet later in the post.

For the remainder of this post, host machine refers to my Linux laptop, and guest machine refers to the virtual machine with Windows XP.

Installation

## open-source version
sudo apt-get install virtualbox-ose ## install on debian/ubuntu
## PUEL version -- needed for usb support
echo "deb http://download.virtualbox.org/virtualbox/debian maverick non-free" >> /etc/apt/sources.list
wget -q http://download.virtualbox.org/virtualbox/debian/oracle_vbox.asc -O- | sudo apt-key add -
sudo apt-get update
sudo apt-get install virtualbox-3.2 ## usb support

Start VirtualBox and create a new machine. I kept most of the default configs, such as dynamic hard drive space. Then pop in my Windows XP cd and install.

This post shows how to do certain things with VirtualBox on Ubuntu.

Access files from Host machine

Start the machine, click on the “Devices” menu -> “Install Guest Additions”. Download the iso and install it on the Guest machine (should be automatically mounted; if not, click on on it under “Devices”).

On the host:

mkdir ~/VirtualBoxShare
## shutdown virtual machine
VBoxManage sharedfolder add "WinXP" -name "vqnportege-share" -hostpath /home/vinh/ ## "WinXP" is the name of my virtual machine, I'm making my home directory accessible on the guest machine, calling this path "vqnportege-share"
## start virtual machine
## on guest, Run -> cmd -> net use x: \vboxsvrvqnportege-share
## x drive should be accessible on the guest machine now...now I can access files from my Linux home directory.

After installing Guest Additions, copy/paste for text between host and guest should work.

USB Support on PUEL version

Got the following here (for Ubuntu 10.10 Maverick):

sudo usermod -G vboxusers -a vinh

Restart host machine. Enable USB controller in for the windows machine (click settings in VirtualBox). After plugging usb device onto the host, click on the usb icon on the bottom of the virtual machine and select the device that you want to use in the guest machine. See this post for more details.

It’s that easy to set up a Windows virtual machine that is highly integrated with the host Linux machine.

Backup and Restore

According to this thread, you need to backup 3 files (“WinXP” is the name of the machine):

  1. ~/.VirtualBox/HardDisks/WinXP.dvi,
  2. ~/.VirtualBox/VirtualBox.xml, and
  3. ~/.VirtualBox/Machines/WinXP/WinXP.xml.

You can also “Take Snapshot” of the machine at a point in time. The file will be in ~/.VirtualBox/Machines/WinXP/Snapshots/{0001....}.vdi (this file analogous to the WinXP.dvi file).

To restore the virtual machine on a different system, place the files in the same path.

If you want to backup the machine and have it used in the same host machine (so two virtual machines that are identical), you will need to clone it to get a new id according to this post:

VBoxManage clonehd ~/.VirtualBox/HardDisks/WinXP.vdi ~/WinXP_Backup.vdi

Wish List

With VirtualBox, the user interface is a machine within a machine. I think with either VMWare and/or Parallels on Mac OS X (I saw on another person’s), each app window can be like a regular app’s window, ie, freely moving on the host’s desktop (not in a box). Also, you can access files between the two very seamlessly. The best part is file integration. If a file is set to open default by an app on the guest machine, then doubling clicking the file on the host will launch the virtual machine, open the app, and the file is opened. It is as if the the Windows apps were installed on the host machine. I don’t know if this is possible with VirtualBox or not yet. I’ll look more into this.

Create a video slideshow from images automatically

On a Mac, creating a video slideshow with some pictures is easy with iPhoto and iMovie. I haven’t found similar programs on the Linux side (yet). I mean, there are many photo manager apps such as F-Spot and Shotwell that can create a quick slideshow from the selected pictures, but they don’t export the slideshow into a video format that can be played anywhere (platform-indepedent). Also, they don’t have nice transitions between pictures. digiKam supposedly supports these features, but I think they’ve been taken out.

I found dvd-slideshow that lets me create a video slideshow with transitions from the command line. On ubuntu, install by

 <pre class="src src-sh">sudo apt-get install dvd-slideshow

Suppose I copy all the pictures I want in a directory. Run the following commands to create a vob (dvd) file:

 <pre class="src src-sh">dir2slideshow -t 5 -c 1 -n <span style="color: #ffa07a;">"Title"</span> -r /path/to/pics/ <span style="color: #ff4500;">## </span><span style="color: #ff4500;">title should be ascii only ## generate some files (txt file contains instructions)</span>

dvd-slideshow -n Name.Of.VOB.file -a /path/to/mp3/song Title.txt ## select name of out file, add audio to be played, and point to txt file with instructions ## for multiple audio files, add multiple -a /path/to/song; songs do not repeat, so add multiple -a if you want songs to repeat

## let’s convert to avi: mencoder out.vob -ofps 30 -ovc xvid -oac mp3lame -lameopts abr:br=128 -srate 48000 -vf scale -zoom -xy 720 -xvidencopts fixed_quant=4 -o out.avi

UPDATE 12/24/2010: dvd-slideshow-editor

My example above is very basic. You can actually modify the generated text file to modify the way the picture is presented (cropping and ken burn effects). You can also create a dvd-menu for your dvd. See this post for one person’s overview. See the wiki for a more complete overview.

Note that there are many GUI frontends to dvd-slideshow, the simplest of which is probably dvd-slideshow-editor. To install on Ubuntu,

sudo apt-get install python-wxgtk2.8 python-wxtools wx2.8-i18n
## now download the python script of dvd-slideshow-editor.

I can imagine using this program to add ken burn effects to my slideshow.

Migrating back to Linux (Ubuntu) from Mac OS X

After abour 1.5 year with Mac OS X on my beautiful black Macbook (v 4,1), I’ve decided to migrate back to Linux, namely Ubuntu 10.04 LTS. Why? Well, I was actually using Kubuntu (KDE version of Ubuntu) from about Spring 2006 to November 2008. At the time, I remember setting up Kubuntu on my Dell Inspiron 700m laptop was quite a task: getting the display to work at the right resolution, wireless, etc. Once everything was set up I loved the environment. Afew things bothered me then: no MS Office (OpenOffice doesn’t cut it sometimes when you have to collaborate), graphics environment (KDE) crashed from time to time (probably because laptop was too slow and I multi-tasked daily) and lack of support for third-party hardware (syncing Palm Centro and ipod, scanners, etc). I would have to do lots of research to figure out the third party issue, and often, a solution was not available (maybe arriving much later). I kind of got sick of this whole process so decided to switch to Mac OS X since it was Unix-based (all tools available), had MS Office, pretty, stable, nicely built (quality), and had third-party hardware support.

Once on the Macbook, I configured it to be my workstation with all the tools I need. Since Mac wasn’t truly Unix, compiling and getting certain software needed special attention despite the availability of MacPorts. Often times I would have to compile things from source manually (no apt-get!), maybe after having to tweak the source code a little.

Now I’m quite sick of not having apt-get or tweaking source code prior to compiling. So, I’m moving back. By now, the third-party issue isn’t too much of a concern since my ipod works with it and Android syncs to the cloud (and is Linux-based). Certain things will need to be resolved for new hardware specifications on newer Laptops, but this doesn’t bother me too much as I’m sure a solution will be available in a matter of weeks or months – plus, the community is very helpful. MS Office can run with WINE/Crossover Office (I did this before too, but I think things should be better now). I would also like to note that in the mist of all this I have servers and and a NAS that are debian- or ubuntu-based, and that I use the school’s Linux servers quite often. It’s only natural that my main workstation be Linux-based again, for ease of use and for efficiency and productivity.