Automatically specify line break options with termstr as CRLF or LF in SAS when importing data

It could be annoying when dealing with data from multiple platforms: Windows uses the carriage return (CR) and line feed (LF) to indicate a new line, UNIX uses LF, and Mac uses CR. Most companies have SAS running on a UNIX/Linux server, and it’s hard to tell which characters indicate a new line without going to a terminal to inspect the file.

Here’s a sas macro that creates a filename handle that could be used in PROC IMPORT or a DATA step. It will automatically detect CRLF and if not, default to LF. This assumes SAS is running on a UNIX server with access to the head and awk commands.

%macro handle_crlf(file, handle_name, other_filename_options=) ;
/* if there is a carriage return at the end, then return 1 (stored in macro variable SYSRC) */
%sysexec head -n 1 "&file" | awk '/\r$/ { exit(1) }' ;
%if &SYSRC=1 %then %let termstr=crlf ;
%else %let termstr=lf ;
filename &handle_name "&file" termstr=&termstr &other_filename_options ;
%mend ;
/* 
%handle_crlf(file=/path/to/file.txt, handle_name=fh) ;
proc import data=fh dbms=dlm replace outdata=d1 ;
   delimiter='|' ;
run ;
*/

First hackintosh with Windows dual boot using Intel NUC DC3217BY

wpid-2014-02-07-multibeast.png

My friend recently introduced me to the Intel NUC (DC3217BY). It’s basically a micro form factor barebone system that comes with Intel’s ULV i3 processor (powerful and low power consumpton). I decided to get one, slapped on 8 GB of ram, a 256 GB mSATA SSD, and a Broadcom based half-sized e-PCI network card, and Hackintosh’d it since the processor is similar to what’s in an Apple Macbook Air. Basic instructions for this particular hardware could be found here and here. A generic guide could be found here. For dual booting with Windows, this article and this post helped. This is what I recalled doing to set up:

  • Update the BIOS to the latest version
  • Create a bootable Windows 7 usb drive on Ubuntu using unetbootin (must be version 494) (drive must be formatted to NTFS)
  • Get Mac OS X 10.9.1 (Mavericks) from the Apple App store
  • Download Unibeast and Multibeast at tonymacx86
  • Download Chameleon Wizard
  • Download Kext Installer
  • On an existing machine with Mac OS X, /Applications/Utilities/Disk Utility and a (> 8 GB) usb flash drive to Mac OS Extended (Master Boot Record enabled).
  • Run Unibeast to load the Mac OS X installer on it
  • Copy Multibeast, Chameleon Wizard, and Kext Installer into this flash drive. Download DSDT.aml and the patched AppleIntelFramebufferCapri.kext here and place them on the flash drive as well (these are to get HDMI audio to work).
  • Boot up the flash drive, and boot the installer with the flags -x PCIRootUID=1 GraphicsEnabler=Yes per this post relevant to Mavericks
  • Once the Mac installer is booted, go to the Utilities Menu and launch Disk Utility. Format the hard drive into two partitions. The first should be called “Macintosh HD” and formatted to Mac OS Extended (Journaled) and the second should be called “BOOTCAMP” and formatted to MS-DOS (FAT).
  • Shutdown, insert in the Windows 7 usb, and install Windows on the second partition
  • Boot the Mac usb again, and install Mac OS X on the first partition
  • Boot the Mac usb again, and select to boot into the Mac OS X partition
  • Run Multibeast to do some post-configurations so that the hardware just works (options in image below)
  • Edit the org.chameleon.Boot via Chameleon Wizard per the image below
  • Copy DSDT.aml to /Extra/DSDT.aml
  • Install Kext Installer. Use it to install the patched AppleIntelFramebufferCapri.kext, then use it to rebuild permissions and kext cache. Restart the computer to have HDMI audio working.

Multibeast options: 2014-02-07-multibeast.png

Chameleon Wizard options for org.chameleon.Boot: 2014-02-07-chameleon-wizard.png

What would I do differently now? Consider getting a network card with bluetooth like the Dell DW1702 per this. I’m not sure if my monitor has speakers, so this would enable me to use wireless speakers. Update (8/22/2014): I ordered this wifi + bluetooth card (BCM943225 HMB/AzureWave AW-NB290) as it was cheap and that this guide shows it works well on a Mac after installing toledaARPT.kext from the repo. Now I have both wifi and bluetooth. The Windows driver for this card can be found here (direct link here).

Now, time to mount the small NUC to the back of my 27″ monitor.

Remotely access files on your home Windows computer

Most of my home computers/servers run Linux, so accessing them remotely is quite easy via the ssh protocol. Even the Windows machines I own have an ssh server installed via Cygwin.

Now, for those not familiar with Linux, one could

  • Install freeSSHd and set up an ssh server on Windows.
  • Configure an account and have freeSSHd initiate at startup.
  • Forward port 22 on the home router to port 22 on the machine with freeSSHd (assign it a static ip from the router); or, use a different external port for safety reasons (eg, 1022 -> 22).
  • Use the portable executable of WinSCP to access your files on any other Windows computer with internet access.

Make Windows like Linux

I’m back to a job that only allows Windows on our laptops and desktops. Here’s how I configured my workstation to be more Linux-like in order to increase my productivity:

  • Install Google Chrome and Firefox
  • Install an antivirus or security suite (Norton or Mcafee?); a free one is Avast
  • Map my caps lock key to control; if Admin access is not available, then use AutoHotKey by creating caps_to_control.ahk with Capslock::Control and creating a Startup shortcut
  • Download Cygwin and install the following: xinit (X server), python (2 and 3), gcc-*, openssh, screen, rsync, python-setuptools (easy_install)), git, subversion, xwinclipboard, procps (top command + others), aspell (I use flyspell in emacs), aspell-en, make, zip, unzip, patch, wget, perl, perl-dbi, libcrypt-devel (for perl DBD:ODBC), automake, autoconf, email (modify /etc/email/email.conf and enter correct server and credential information; sendmail is not needed for sending outbound emails, only needed to send internal emails)
  • Install the emacs binaries to C:\Documents and Settings\my_username\bin\emacs-ver_num (Windows XP) or C:\Users\my_username\bin\emacs-ver-num (Windows 7) and copy relevant image library dll files into the emacs binary directory in order for doc-view to work properly (eg, need libpng14-14.dll for emacs 24.3
    • See Vincent Goulet’s emacs distro to see what dll files are needed)
    • One could also use emacs w32 provided by cygwin, but ESS doesn’t seem to work because that version of emacs does not have the function w32-short-file-name compiled with it as needed by ESS; tentative solution can be found here, but it’s probably better to use the compiled emacs binaries available on GNU
  • Set the HOME environment variable to C:\Documents and Settings\my_username or C:\Users\my_username; set LC_ALL and LANG to en_US.utf-8 (for perl dbi error); set CYGWIN=nodosfilewarning
  • Edit environment variables by running the following in the command prompt: rundll32 sysdm.cpl,EditEnvironmentVariables. Add the following to the PATH environment variable: path_to_R;path_to_JRE;C:\Documents and Settings\my_username\My Documents\bin;C:\Documents and Settings\my_username\My Documents\bin\emacs-24.3\bin;C:\cygwin\bin. If we cannot edit System variables, then edit the user’s variables (eg, PATH to be path1;path2;%PATH%). If the settings aren’t saved for future sessions (eg, in Citrix), then create a symbolic link from /home/user_id to the desired home (eg, C:/Users/user_id), and add the following to ~/.bashrc: export PATH=/cygdrive/f/R/R-3.1.1/bin/x64:/cygdrive/f/bin/emacs-24.3/bin:/cygdrive/f/bin:/cygdrive/f/bin/cygwin64/bin:$PATH and export JAVA_HOME=F:/bin/jre7.
  • Add the following to ~/.bash_profile: . ~/.bashrc
  • Fix the carriage return issue (only for Windows XP)
  • Run touch ~/.startxwinrc to prevent xterm from launching whenever X server is started.
  • Install R using the Windows installer (works with Emacs ESS); install R studio
  • Get sshfs to work on Windows to mount my servers
  • Install Dropbox and symlink my .bashrc and .screenrc files
  • Get tramp in emacs to work properly in order to visit remote servers easily in emacs by first getting the latest copy of tramp, then configure and byte-compile the code (make) per the proper installation. Add plink, pageant, and all putty-related binaries into the PATH (~/bin). After creating an ssh key, use the putty kegen to convert id_rsa to id_rsa.ppk. Create a shortcut at startup that launches pageant /path/to/id_rsa.ppk. Then in emacs, one could access remote files using tramp via plink: /plink:username@host:/path/to/dir.
  • Copy X Windows shortcut to the Startup folder to automatically start it
  • Add export DISPLAY:0.0= to ~/.bashrc
  • Python packages: numpy, pandas, csvkit (easy_install works but pip install does not), jedi, epc, pyodbc, mysqldb, pymssql, psycopg; ddlgenerator (Python 3).
  • R packages: RODBC, RMySQL, RPostgreSQL, ggplot2, plyr, glmnet
  • Perl packages (for edbi in emacs): cpan RPC::EPC::Service, cpan YAML, cpan -i DBD::ODBC (gcc4 error; edit Makefile and change “CC=gcc4″ to “gcc”; “LD=g++”)
  • Set up ssh server via Cygwin and open up port 22 in Windows Firewall; freeSSHd is also an alternative
  • On Windows 8, the user might not be able to change group permissions (eg, can’t ssh using keys because the key is “too open”); fix by changing files/directories group to ‘User’
  • Use Autopatcher to install download all necessary updates and install them all at once
  • Have the following shortcut in the startup folder in order to have a a terminal open up at startup with screen initiated: C:\cygwin\bin\mintty.exe -e screen -s bash; in the shortcut, specify the home directory as the ‘Start In’ path
  • Install UniKey for typing in Vietnamese (place in ~/Documents/bin), 7-Zip for handling archive files, and Virtual CloneDrive for handling disk image files
  • Install CutePDF Writer (also download and install Ghostscript from CutePDF) for printing to PDF files
  • Install Java Runtime Environment (JRE); if admin privileges aren’t available, then extract the files manually into ~/Documents/bin/jre/
  • Other tools per Lifehacker: VLC, PDF-XChange, Foxit PDF Viewer
  • Configure sshd using openssh, make sure it starts at startup (Start > Run > Services; look for CYGWIN sshd), and allow /usr/sbin/sshd to pass through the Firewall; one could alternatively use freeSSHd
  • Bash shell in emacs via shell: add (setq shell-file-name "bash")

and (setq explicit-shell-file-name shell-file-name) to the emacs init file, and add the following to ~/.bashrc:

# http://stackoverflow.com/questions/9670209/cygwin-bash-does-not-display-correctly-in-emacs-shell
if [ "$EMACS" == "t" -a "$OSTYPE" == "cygwin" ]; then
    PS1='\u@\h \w\n$ '
fi

If Dropbox cannot be installed then symlink my ~/.emacs.d directory (need to use mklink in order for symlink to work properly).

If the computer is a dual-boot with Linux installed first, then one can change the order of the bootloader to Windows by following these instructions.

This is a good post to review.

Update system BIOS on a Linux machine with a Windows/DOS updater

I recently had to update my BIOS on my Toshiba Portege R705 laptop. Unfortunately, the updater can only be run on a Windows/DOS system. Luckily, I remembered reading this article on Linux Journal that described how one can boot into FreeDOS to update the system BIOS. I will outline what I did.

  1. Find a USB drive with a capacity < 2 GB.
  2. Using GParted, format the drive to be FAT16.
  3. Download fdbasecd.iso and install qemu (sudo apt-get install qemu).
  4. Boot up FreeDOS with qemu: qemu -hda /dev/sdb -cdrom /path/to/fdbasecd.iso -boot d (assuming usb drive is /dev/sdb). Continue booting from “cd-rom” (iso file) and install FreeDOS to “harddisk” (usb stick) with default values.
  5. Do not reboot system after FreeDOS is done installing. Type following in DOS prompt:
  <pre class="src src-sh">fdisk /mbr 1

cd edit fdconfig.sys ## for line that starts with “SHELLHIGH…”, change to “1234?SHELLHIGH…” ## save: Alt-f ## exit to Dos, exit qemu

  1. Test if USB drive will boot using qemu: qemu -hda /dev/sdb -boot c
  2. Copy .exe file into usb drive. When I tried executing it in FreeDOS, it says it can’t because it is a Win32 file. I extracted the EXE file and read the README. It contained some files I can run in DOS.
  3. Restart system and boot with USB drive. On my Toshiba laptop, I had to press ESC then F12 to select USB drive.
  4. Once in DOS, run the command to update BIOS. NOTE: I take no responsibilities for your actions. Update your BIOS at your own risk. You can brick the machine if you aren’t doing things correctly or use the wrong BIOS file.

Virtualization with VirtualBox (running Windows inside Linux)

I recently re-installed Windows on an old laptop to use Adobe Acrobats and the likes. Some Windows programs just aren’t accessible on Linux via Wine (although MS Office works). I’ve decided to set up Windows as a virtual machine on my Linux laptop; let’s see if virtualization will take the separate Windows machine out of the picture. I went with VirtualBox since it is open source, free, and was mentioned on Linux Journal. My initial reaction is VERY favorable. It seems everything works flawlessly (internet, cd drive, accessing files from the host machine; usb supported only through PUEL proprietary version.). I’ll jot down what I would like to be able to do with VirtualBox but have not been able to yet later in the post.

For the remainder of this post, host machine refers to my Linux laptop, and guest machine refers to the virtual machine with Windows XP.

Installation

## open-source version
sudo apt-get install virtualbox-ose ## install on debian/ubuntu
## PUEL version -- needed for usb support
echo "deb http://download.virtualbox.org/virtualbox/debian maverick non-free" >> /etc/apt/sources.list
wget -q http://download.virtualbox.org/virtualbox/debian/oracle_vbox.asc -O- | sudo apt-key add -
sudo apt-get update
sudo apt-get install virtualbox-3.2 ## usb support

Start VirtualBox and create a new machine. I kept most of the default configs, such as dynamic hard drive space. Then pop in my Windows XP cd and install.

This post shows how to do certain things with VirtualBox on Ubuntu.

Access files from Host machine

Start the machine, click on the “Devices” menu -> “Install Guest Additions”. Download the iso and install it on the Guest machine (should be automatically mounted; if not, click on on it under “Devices”).

On the host:

mkdir ~/VirtualBoxShare
## shutdown virtual machine
VBoxManage sharedfolder add "WinXP" -name "vqnportege-share" -hostpath /home/vinh/ ## "WinXP" is the name of my virtual machine, I'm making my home directory accessible on the guest machine, calling this path "vqnportege-share"
## start virtual machine
## on guest, Run -> cmd -> net use x: \vboxsvrvqnportege-share
## x drive should be accessible on the guest machine now...now I can access files from my Linux home directory.

After installing Guest Additions, copy/paste for text between host and guest should work.

USB Support on PUEL version

Got the following here (for Ubuntu 10.10 Maverick):

sudo usermod -G vboxusers -a vinh

Restart host machine. Enable USB controller in for the windows machine (click settings in VirtualBox). After plugging usb device onto the host, click on the usb icon on the bottom of the virtual machine and select the device that you want to use in the guest machine. See this post for more details.

It’s that easy to set up a Windows virtual machine that is highly integrated with the host Linux machine.

Backup and Restore

According to this thread, you need to backup 3 files (“WinXP” is the name of the machine):

  1. ~/.VirtualBox/HardDisks/WinXP.dvi,
  2. ~/.VirtualBox/VirtualBox.xml, and
  3. ~/.VirtualBox/Machines/WinXP/WinXP.xml.

You can also “Take Snapshot” of the machine at a point in time. The file will be in ~/.VirtualBox/Machines/WinXP/Snapshots/{0001....}.vdi (this file analogous to the WinXP.dvi file).

To restore the virtual machine on a different system, place the files in the same path.

If you want to backup the machine and have it used in the same host machine (so two virtual machines that are identical), you will need to clone it to get a new id according to this post:

VBoxManage clonehd ~/.VirtualBox/HardDisks/WinXP.vdi ~/WinXP_Backup.vdi

Wish List

With VirtualBox, the user interface is a machine within a machine. I think with either VMWare and/or Parallels on Mac OS X (I saw on another person’s), each app window can be like a regular app’s window, ie, freely moving on the host’s desktop (not in a box). Also, you can access files between the two very seamlessly. The best part is file integration. If a file is set to open default by an app on the guest machine, then doubling clicking the file on the host will launch the virtual machine, open the app, and the file is opened. It is as if the the Windows apps were installed on the host machine. I don’t know if this is possible with VirtualBox or not yet. I’ll look more into this.

Compiling and Cross-compiling R packages for Windows (win32)

so recently i’m learning how to compile and cross-compile R packages from source for windows machines, which means i have to create windows binaries. the first option is to build on a windows system, and the 2nd is to cross-build on an intel-based linux or mac system. i will outline my experience.

Send it off to a Build Farm If you have the source package ready for install on a linux based system, you can send it off to a server and have it automatically build. This is probably the easiest way.

Building R packages and binaries on a Windows system. References: http://cran.r-project.org/doc/manuals/R-admin.html#Building-from-source http://faculty.chicagobooth.edu/peter.rossi/research/bayes%20book/bayesm/Making%20R%20Packages%20Under%20Windows.pdf http://www.biostat.wisc.edu/~kbroman/Rintro/Rwinpack.html Requirements: Rtools and R source file

At the time of this writing, I am using R 2.9.2 and using Rtools29.

  1. First, install Rtools, which includes some basic unix utilities, compilers (MinGW), windows compatible perl, unicode, Tcl, and some bitmap stuff.

Install should be in C:\Rtools and C:\R. Go ahead and install everything, including the Tcl stuff. In the install, go ahead and click modify Path variable if you have admin access. if not, add the subfolders of Rtools that has bin into the user ‘Path’ variable (either for User or for System) from the Control Panel > System > something > Advanced tab > Environment variable button. The variable should be something like ‘H:\Rtools\bin;H:\Rtools\MinGW\bin\;H:\Rtools\perl\bin\’.

  1. go to start > run > cmd to open a command window. I placed my R source tar.gz file in H:\R. now, use tar zxvf to unarchive it.
  2. Move H:\R\Tcl into the R-version folder that we just unarchived. Move the folders ‘unicode’ and ‘bitmap’ from H:\R\src\gnuwin32 into R-version\src\gnuwin32. Main thing is we need the iconv.dll file.
  3. goto R-version\src\gnuwin32\. type make (or make all) to compile. this gave me a couple errors. if u don’t have Tcl in the right location, edit the R-version\src\gnu32\MkRules and change the path for Tcl. I also got an error where it couldn’t find the tmp folder, so i just created H:\tmp. R successfully compiled: see R-version\bin\.
  4. To make the windows binaries, I have a package folder prepared under mac/linux. I copied it to the windows machine, and i did something like in H:\R\ (where the package folder is in, say H:\R\MyPackage):

H:\R\R-version\bin\Rcmd.exe build –binary MyPackage The source should be compiled into a dll and i get a zip file where i could install on other windows machines.

Cross building package win32 (windows) binaries on an intel-based mac or linux Since i don’t have a windows system (i did it on my brother’s computer), it could be convenient to build this on my mac or linux system. The R core team used to support cross building, but stripped support as of v 2.9. I first outline my experience with R 2.8 on my Mac OS X. Look at the R-admin help file for cross-building in the 2.8. I also got help from this post on R-dev, and a few inquiries with the poster through email.

Requirements: R source code file MinGW for mac (I installed v.4.3.0 binaries and added the location to my PATH variable) R (mac/linux, your system) binaries (version must match that of source. i had R 2.8 currently installed at the time, so i cross-build this version). R win32 binaries (same version as well): u can install this from source using the cross-build stuff, but i kept getting errors. so, what i did, was used the windows installer and installed it using Wine on my Mac (~/.wine/drive_c/Program Files/R/R-version/ – the space in Program Files isn’t good, so i made a softlink via ln -s to ~/R)

In R-version (extracted somewher), go to /src/gnuwin32, edit MkRules, changed: BUILD=CROSS no CHM, this version picks this up by default with CROSS BINPREF=i386-mingw32-##i586-mingw32- R_EXE=/usr/bin/R###’I failed to read the instructions to set R_EXE in MkRules’

Now, suppose the name of my package is MyPackage and it is locted in Users/vinh/Downloads/Rwin (MyPackage folder is in here)

Instructions here are found in README.package of gnuwin32:

<pre class="src src-sh"><span style="color: #b0c4de;">cd</span> .../src/gnuwin32

make PKGDIR=/mysources RLIB=/R/win/library pkg-mypkg make PKGDIR=/mysources RLIB=/R/win/library lazyload-mypkg cd /R/win/library zip -r9X /dest/mypkg.zip mypkg

in my gnuwin32 folder, i did:

<pre class="src src-sh">make <span style="color: #eedd82;">PKGDIR</span>=/Users/vinh/Downloads/Rwin/ <span style="color: #eedd82;">RLIB</span>=/Users/vinh/R/library/ pkg-MyPackage

[it compiled my dll file] make PKGDIR=/Users/vinh/Downloads/Rwin/ RLIB=/Users/vinh/R/library/ kazyload-MyPackage cd ~/R/library zip -r9X MyPackage MyPackage

Now I have MyPackage.zip that can be installed on windows machines! Remember, ~/R is the softlinked version of where my windows version of R is.

Next, i’m going to attempt cross-compiling on R 2.9.2, where support is no longer supported, but as the post i referenced above described, this should still be do-able. I will update this.