optparse R package for creating command line scripts with arguments

Just discovered the optparse package in R that allows me to write a command line R script that could incorporate arguments (similar to Python’s argparse). Here’s an example:

#! /usr/bin/env Rscript

# http://cran.r-project.org/web/packages/optparse/vignettes/optparse.pdf
suppressPackageStartupMessages(library("optparse"))

option_list <- list(
    make_option(c('-d', '--date'), action='store', dest='date', type='character', default=Sys.Date(), metavar='"YYYY-MM-DD"', help='As of date to extract data from.  Defaults to today.')
)

opt <- parse_args(OptionParser(option_list=option_list))

# print(opt$date)
cat(sprintf('select * where contract_date > "%s"\n', opt$date)

Save this as my_scrypt.R, and do chmod +x my_script.R. Now check out ./my_script.R --help.

Add a printer on the command line – cups-pdf example

Suppose you need to add a printer to a computer on the command line, e.g., a server that you access via a terminal. This post outlines how to use some basic commands from the CUPS system. I will illustrate with commands for a network-connected printer and cups-pdf for printing to PDF files (~/PDF/):

 <pre class="src src-sh">sudo lpadmin -p cups-pdf -v cups-pdf:/ -E -P /usr/share/ppd/cups-pdf/CUPS-PDF.ppd

sudo lpadmin -p cups-pdf -v cups-pdf:/ -E -P /path/to/ppd ## can also use usb:/

The -E flag is crucial (Enables the printer). If a web browser is accessible from the system, you can also go to http://localhost:631 to add printers via the web browser. Almost as easy as adding it via the system-config-printer command on Ubuntu.

To check currently installed printers:

 <pre class="src src-sh">lpstat -t

LPR printing on command line

These days, with Linux distros such as Ubuntu, printing is quite easy. You can add and remove printers and print files just as easily (or even more so) compared to Windows or Mac. Sometimes I might just want to print from the command line. To do so, I make use of the lpr command.

To find out what printers are available, type:

 <pre class="src src-sh">lpstat -p -d <span style="color: #ff4500;">## </span><span style="color: #ff4500;">from CUPS manual</span>

This and this gives a good introduction to lpr.

 <pre class="src src-sh">lpr filename <span style="color: #ff4500;">## </span><span style="color: #ff4500;">print file using default printer</span>

lpr -P printername filename ## print file using the selected printer; see lpstat -p -d for list of printers lpr -# num filename ## to print num times

openoffice -p filename.doc ## for doc files

lpr should support txt, ps, and pdf files.

Split, cut, or sample a video file on the command line

There are many reasons to cut or split a video file. For example, one may want to cut a long video into multiple parts to upload to YouTube. I first ran into this and this, which suggests:

 <pre class="src src-sh">ffmpeg -ss 00:00:00 -t 00:01:00 -vcodec copy -acodec copy -i in.avi out.avi

## -ss: start position ## -t: end position ## can re-encode with other codecs

However, for some reason my out file is almost as large as my in file, even though I’m only sampling 1 minute out of the 2 hour segment. This lead me to the mencoder solution:

 <pre class="src src-sh">mencoder -ss 00:00:00 -endpos 00:00:01 -ovc copy -oac copy in.avi -o out.avi

The file size of my out file is more reasonable, and the speed is incredibly fast since I am not re-encoding.

Burn dvd from the command line

I recently had to back files from my Ubuntu-based NAS. I initially used xfburn, a medium-weight burning software. It does a pretty good job. However, I decided to burn via the command line to refrain from pointing and clicking. I found dvd+rw-tools. Some simple examples are found here; the full filename solution was found here. Suppose the dvd burner is /dev/sr0 (found from scanning through dmesg | less). Then I can burn files with:

 <pre class="src src-sh">growisofs -dvd-compat -input-charset=ISO-8859-1 -full-iso9660-filenames -Z /dev/sr0 -R -J -pad <span style="color: #ffa07a;">"/path/to/files"</span> <span style="color: #ff4500;">## </span><span style="color: #ff4500;">can use -dry-run</span>

eject /dev/sr0

(Batch) Amplify audio files from the command line

Audacity is great, but it’s a GUI and sometimes I just want to do basic things like amplify the sound of some mp3 files. Discovered mp3gain from this thread and wrote this script to batch amplify the files:

 <pre class="src src-sh"><span style="color: #ff4500;">#</span><span style="color: #ff4500;">! /bin/</span><span style="color: #00ffff;">bash</span>

## http://boards.straightdope.com/sdmb/archive/index.php/t-339352.html

gainval=10 for file in “$@” do mp3gain -g $gainval $file done

Note that mp3gain alters the original mp3 files directly. My default value is 10; feel free to alter it.

Downloading, cURL vs. wget

I do more and more downloading using the command line these days, mainly using wget and cURL. This is a good comparison of the two. Both are great at downloading. cURL supports more protocol (outside of http, https, ftp) and is bi-directional. wget can download files recursively (links on a webpage, and links that appear on pages linked from that website, and on).

how to set up sending mail from the command line

Often times I want to send emails to myself when a job is done or when something happens on the computer/server. Using the command line is the most portable and obvious as most software I run (eg, R) can execute from it.

This site shows how to use mail and mutt to send emails from the command line, but does not show how to set things up. Setting up actually took me a long time to figure out, as I wasn’t familiar with Mail User Agent (MUA), Mail Transfer Agent (MTA), etc. This site can help clear things up regarding those issue.

Now, down to business. We’ll need to install some software:

<pre class="src src-sh">sudo apt-get install mutt msmtp bsd-mailx <span style="color: #ff4500;">## </span><span style="color: #ff4500;">mailutils</span>

mailutils contain the mail command, which allows one to send mail but not attach files. mutt is an MUA that allows one to send mail from the command line and attach files. msmtp is an MTA that we can use with mail and mutt (mutt v1.5.x and above has it’s own built in smtp capabilities).

~/.msmtprc:

<pre class="src src-sh">account default

host smtp.gmail.com port 587 from my.email.address tls on #tls_starttls on tls_trust_file /usr/share/ca-certificates/mozilla/Equifax_Secure_CA.crt auth on user my.username (email address for gmail) password my.password logfile ~/.msmtp.log

Need to execute chmod 600 .msmrprc (user read/write only )or it won’t work.

~/.mailrc:

<pre class="src src-sh"><span style="color: #b0c4de;">set</span> <span style="color: #eedd82;">sendmail</span>=<span style="color: #ffa07a;">"/usr/bin/msmtp"</span>

~/.muttrc:

<pre class="src src-sh"><span style="color: #ff4500;">## </span><span style="color: #ff4500;">following if want to use msmtp to send mail</span>

set sendmail = “/usr/bin/msmtp” ## set realname = “vqn laptop” set from = my.email.address set use_from=yes set envelope_from=yes ## uncomment following and comment above sendmail line if want to use ## mutt’s built-in smtp ##set smtp_url=”smtp://username@smtp.gmail.com:587″ ##set smtp_pass=”my.password”

chmod 600 .muttrc since it contains password.

Note that to use TLS with msmpt, you need to set the certificate file. On a debian-basd system, you can find it in the /usr/share/ca-certificates/ directory, which has many. See below for references.

This site shows how to set up mutt with msmtp. This site shows how to set things up, along with the certificate stuff. More on certificate here. This will tell more on how to use the built-in smtp of mutt. The wiki of course has more information.

UPDATE 8/24/2011: From this post, use mail from bsd-mailx instead of mailutils to have the mail command working instead of the error msmtp: no recipients found.

UPDATE 11/13/2011: If you want to be able to email as the root user, make sure these instructions are set up for the root user’s home directory, /root/.

Accessing MS SQL Server from command line in Mac OS X and Linux/Unix

Unfortunately in my current line of work, I’ve been pulling data from MS SQL Server. My current workflow involves remotely logging into a windows machine, and from there, using the sql server software (MS SQL Server Management Studio) to issue queries and save data. This especially sucked in that I couldn’t write in the query file where I want my output results to go. I had to manually right-click the results, copy, and paste into a txt file.

Recently I started using Oracle’s SQLDeveloper with the 3rd party plugin jTDS to skip the remote login. SQLDeveloper allowed me to log directly into the SQL Server. This did not work well in that copying the results caused issues (memory?). Also, Query Analyzer was only available if the DB was Oracle’s. The added benefit was not great.

I finally found FreeTDS by googling “linux ms sql server command line.” To install it on Mac OS X, I needed to install it via Macports (sudo port install freetds). I had issues with it initially because it could not connect with the SQL Server of interest. The problem was that I needed to specify the correct TDS version; in my case, it was version 4.2 (see this linked from this thread). You can connect using two ways:

 <pre class="src src-sh"><span style="color: #eedd82;">TDSVER</span>=4.7 tsql -H myserverIP -p 1433 -U myusername -P mypassword

or

 <pre class="src src-sh">tsql -S myserverinlist -U myusername -P mypassword

with the following in “~/.freetds.conf”

 <pre class="src src-sh">[adrc]

host = myserverIP port = 1433 tds version = 4.2

See this for some examples. If you have connection issues, make sure you could telnet to the server using the right port first. See this. More information on the freetds.conf file can be found here.

For a query, remember to put go at the end of the query for it to execute (credit).

Be careful when routing input and output (“<” and “>”) as other messages from the tsql command will be present. Will post a solution when I find out.

Update 4/16/2010 – Batch output results from a query

I found out FreeTDS comes with the commands freebcp (bcp emulation/replication) and bsqldb. To get batch results, issue

bsqldb -U username -P password -S server.from.TDS.list -h -t '\t' -i query.sql -o query.out
## in 2nd line, has "-" characters (separate headers and data). to remove them:
bsqldb -U username -P password -S server.from.TDS.list -h -t '\t' -i query.sql | sed -n '2!p' > query.out

## or use freebcp
-- freebcp "$(cat query.sql)" queryout query.out -S server.from.TDS.list -U username -P password -c
## for freebcp, do not use "GO" in query

The above freebcp method does not work for me with the error Cannot bcp with TDSVER < 5.0.

Credit for for bsqldb goes to here. Removing the 2nd line is from here. This showed me how to get the correct freebcp command line syntax.

Next up is getting odbc (unixODBC to be exact) working!