Tag Archives: Simple solutions

CSV file: convert list to table/matrix

Update 2014-05-04: Added -N1/-N2 flags for sorting rows/columns numerically, and uploaded a windows binary.

I found myself having data in CSV-files with three columns; two dimensions and a value. It could look like:

20080102,AAPL,194.84
20090102,AAPL,90.75
20100104,AAPL,214.04
20110103,AAPL,329.57
20120103,AAPL,411.23
20080102,MSFT,35.22
20090102,MSFT,20.33
20100104,MSFT,30.95
20110103,MSFT,27.98
20120103,MSFT,26.765

Data typically looks like this because it is very easy to output transactions on this format. That is very nice if you want to load it into a database. But for other purposes (like plotting a graph using LibreOffice Calc or even Excel) it would be much nicer with a table/matrix-layout:

,AAPL,MSFT
20080102,194.84,35.22
20090102,90.75,20.33
20100104,214.04,30.95
20110103,329.57,27.98
20120103,411.23,26.765

I could not find a standard tool for this. I thought about different options, and finally decided it was quite easy to just write a little program. So I did. You use it like this:

$ ./csv-list2table -t < list.csv > table.csv

There are a few things to think about:

  • The switches -t or -T decides if column 1 or 2 will be rows
  • -N1 and -N2 can be used to treat/sort column 1 and 2 numerically
  • Rows and columns are outputed sorted
  • Holes/missing values outputed as ,,
  • Comma is the only accepted delimiter
  • Input must have exactly 3 columns
  • Pre/post-process data with sed and cut

As the last item mentions, sed can fix a file with other delimiters than comma, and cut can pick the columns you need from a list with more data than you need.

Finally, the code written in standard C: csv-list2table-1.1.c.
Old version: csv-list2table-1.0.c
Windows binary: csv-list2table-1.1.exe (should be no tricky dependencies)
Test file: csv-list2table-test.txt

Building should be trivial:

gcc -O2 -o csv-list2table csv-list2table.c

I dont think the code contains anything that should confuse any c-compiler on any reasonable platform.

Remote Desktop & Windows 7 Home Premium

Windows 7 Home Premium seems to be the right version of Windows for home usage, right? I mean for media, gaming, communication and casual work. And it is the version MS ships with a PC you can buy in a store. Well, Microsoft believes that Remote Desktop (the server part) is not a Home Premium feature, but a Professional feature. I think there are plenty of home-related uses for Remote Desktop (supporting a relative remotely, sitting in the sofa with your tablet while controlling your PC, access your computer from work or when traveling).

The upgrade to Professional is 180 Euros, and requires some work. This is simply Microsoft making their products suck for their paying customers! I’d say people who buy a PC in a store really don’t have the choice to pick Professional or Ultimate. The wares people are probably smart enough to not bother with anything less than Ultimate.

Luckily, there is a simple hack for Home Premium, and this one works today (well, yesterday), despite SP1 and everything.

Quickly and easily transfer files over network

You want to copy a file between two computers or Symbian phones, and the usual methods don’t feel that attractive? Network drives, scp and ftp requires configuring a server (that later might be a security risk). USB cables are never available when needed. DropBox and Bluetooth are too slow.

A while ago I described how to copy files with netcat, but that works best on *nix and is not so easy for people who do not like the command line. And, it does not work on mobile phones.

So, I wrote a little program that does what netcat does but is simple to use and has a GUI. And, I wrote it in QT, so it works in Mac OS X, Linux, Windows and Symbian. It is exactly the same on all platforms.

Do you want a simple way to copy files over the network? Download ParrotCopy and give it a try – instructions included:

ParrotCopy:

The Symbian version probably only works on Symbian^3 and has just been tested on Nokia N8. Let me know if you need a binary for an older Symbian device or a Maemo or Harmattan device.

Bugs
In server mode, the program tries to connect to www.google.com:80, to figure out its own IP. It is simply an ugly hack because I had problems with other methods. You may have problems if internet is not accessible. I will not release 1.0 until this is fixed.

Limitations
You have to manually name the file you receive, and you can transfer just one file at a time. These are not bugs, but future versions will probably do better. Netcat can be combined with tar, gzip etc. I hope to add at least tar in the future. For now, making a zipfile is a simple way to transfer many files.

Release 0.9.7
Now possible to copy contents of status field and file/folder fields (if you want to paste it into other application).

DOS flashing in 2011

Still BIOS updates and other firmware flash operations mostly need to be done from DOS. Seriously, DOS, in 2011?

I found a decent strategy. When you install your computer, make a small (256Mb) FAT partition with a FAT16 filesystem on it first on your harddrive (with some skills and luck you can use it also as linux /boot partition). Whenever you need to flash BIOS or anything else, put the DOS flash utility on this partition.

Make yourself a bootable FreeDOS CD. When you boot into freedos you just do:

  a:\> c:
  c:\> dir
  (dir output - to find the name of the update program)
  c:\> BIOS1234.exe

When everything is done you can just reset the computer with the reset-button, eject the CD, and boot your normal OS.

So, what is clever about this? It is not so easy to make FreeDOS read USB-keys or access the network, and the bootable CD is naturally read-only. But a little FAT16 filesystem first, on your first hard drive, is very easy to access. And you can easily put any upgrade files there, from any OS you like to run normally.

On a brand new computer (or hard drive), the first thing you do is boot FreeDOS (or your Linux live cd of choice) and create the little FAT partition. Then you can install your other OSs. If you got the computer with Windows already installed, and you intend to keep it, you probably dont need to flash anything anyway.

Youtube performance on PowerBook G4

My Apple PowerBook G4@866 Mhz can not handle youtube vidoes… when they are based on flash. However, go to the HTML 5 version of YouTube and performance is very reasonable.

Printing to Sharp AR-M450 in Ubuntu (using CUPS)

Printing to a Sharp AR-M450 using CUPS (the standard print system for most free *NIX and even Mac OS X) is possible, but took me some time to find the right settings. First, you need the printers IP address, in my case 10.2.0.98.

In CUPS you chose “Network Printer” and “lpd/lpr”. Then you set address=”10.2.0.98″ (replace with your own) and queue=”lpr”.

As printer driver, I chose “Generic PCL 6/PCL XL Printer”, using “CUPS+Gutenberg vX.X.X Simplified”.

This should get you running!

Simple method for copying files over network

Copying files between computers is, fundamental. However, sometimes it is not so easy. When struggling with NFS I have been thinking; how did Microsoft get this one more right than the other ones? But Windows file sharing is not so very simple either, and it is not exactly getting easier with new versions.

However, copying files between two computers (on the same network) IS easy, especially between *NIX-based systems. Here are two methods, and they may work on Windows too, with the right tools installed (cygwin).

The following examples presume you want to copy files from a client computer to a Mac OS X machine with IP 192.168.0.20.

SSHFS
sshfs allows you to mount any folder on the server, on any folder on the client. Normal permissions apply, and you need to have sshfs installed on the client, and sshd on the server (you have it if you can ssh to the server).

Lets assume I have a user zo0ok on the Mac OS X machine, and I want to access that users home directory. Then I do (on the client):

  > mkdir zo0ok-on-osx
  > sshfs zo0ok@192.168.0.20:/Users/zo0ok zo0ok-on-osx

I need to authenticate. Done! Now the contents of the remote home directory is available locally. Note; performance is not optimal, and things like random access to files might not work. But for many purposes it works perfectly.

Netcat (nc)
netcat (or just nc) is an insanely powerful, and simple, tool. You can pipeline things, not just between programs, but over the network. More people should know about it! The following instructions assume you are logged in and have a shell on both machines.

First just see that everything works. On the mac (server), listen to port 9999:

  macosx$ nc -l 9999

Second, on the client, send message:

  client$ echo Hello World | nc 192.168.0.20 9999

If everything is fine, the message was sent to the Mac and displayed on its prompt. nc should quit as it reaches end of file.

Copy a file (hello.txt – it must exist on the client):

  macosx$ nc -l 9999 > hello.txt
  client$ nc 192.168.0.20 9999 < hello.txt

If everything is fine, you have a file on the Mac, identical to the one on the client (use md5sum if in doubt).

Copy a large file:

  macosx$ nc -l 9999 | gunzip > ubuntu.iso
  client$ cat ubuntu.iso | gzip | nc 192.168.0.20 9999

If your CPU is faster than you network, file transfer will be faster when you compress the data.

Copy a folder:

  macosx$ nc -l 9999 | tar -x
  client$ tar -c mp3collection | nc 192.168.0.20 9999

Of course you can use the -z switch for tar to enable compression, but for you mp3-collection it is not a wise idea.

Only your imagination limits what you can do with nc!