Zeroflux

Migrating to Android

2009-06-11T15:06:00Z

I went to Google I/O in San Francisco this year, and lo and behold, they started off the conference by giving all 4000 of us free, unlocked Android phones! Not bad for a $300 conference.

I do like the iPhone (my current ride) a lot. The hardware is very sleek and the OS is pretty nice. But, their development philosophy seems very closed to a Linux geek like myself. First off, I have to live in the Apple world in order to build an iPhone app. I don't actually own a Mac, which is apparently a bit of a barrier right there. And if I actually want to geek out on my iPhone, I have to go around Apple's back and jailbreak it so I can run what I want.

Not so with the egalitarian Android. The SDK runs on Windows, Mac, or Linux. As soon as I installed the SDK, I had an easy shell from my workstation to the USB-connected phone. I'm not really a fan of Java, but this is already encouraging me to make some Android apps, a feeling I never really had from the iPhone.

In terms of general usability, here are some things I've noticed in my short time with Android so far:

The virtual keyboard is slightly laggy when responding to my finger taps (compared to iPhone). This can cause more typos that I don't register until I've already gone a few keys past, and so I find myself backspacing more often to fix mistakes. This is mitigated by typing in landscape mode, as the keyboard itself is bigger.
I love the Back button! IMHO, this functionality should really be on iPhones as well. If I open an email in Android, then click a URL inside, it will take me to the browser app. If I then click the Back button, it takes me back to my email message. With the iPhone, I have to click the Home button first and then load up the mail app again.
No two-finger pinch/zoom stuff in the default Android kernel, though I'm told this can be enabled with a custom kernel. Based on that, I'd guess that two-finger tricks will be arriving in an official capacity soon.
Google integration is really nice to have. If you already use Gmail et al. in your day-to-day life, then all this data is magically on your phone too. I'm sure MobileMe is like this too, but I think more people use Gmail than MobileMe. And Gmail is free.
There's no 1/8" headphone jack on this handset (HTC Magic), only a USB connection. Of course, they do provide headsets (two!) with the phone, but it'd still be nice to have a standard stereo jack.
In general, I find the app organization (or lack thereof) and both iPhone and Android to be annoying. Both choose to lay out all apps in a single panel, so it becomes time-consuming to find an app if you have a lot of them installed. I would prefer some sort of hierarchical system, so I could organize all my networking apps in one area, productivity apps in another, games in a third, etc.

So despite my general happiness with the iPhone as an end-user, I'm giving Android a solid chance. The first thing I had to do was get my contacts moved over to the new phone. This, surprisingly enough, proved to be non-trivial. You'd think it could be as simple as hitting an "Export to SIM" button on one phone and an "Import from SIM" on the other. The problem is that neither phone provides an "Export to SIM" function. They'll happily import contacts off a SIM, but I guess nobody has had a need to export yet. And any sort of third-party app was conspicuously absent from the App Store. Hmmm. It smells a bit like data lock-in, but I'd never say that out loud.

Eventually, I just SSH'ed into my iPhone and started poking around. Turns out the contacts are all centralized in a SQLite database! That made life a lot easier. I copied it over to my workstation and wrote a little Python script that massaged the contact data into CSV format. I imported the CSV file into my Gmail contacts and seconds later, they all showed up on my Android phone. Cinch.

If you're interested, the DB is in /var/mobile/Library/AddressBook/AddressBook.sqlitedb.

Here's the Python as well. It's a little rough around the edges -- I've been doing a lot of Javascript lately, and kept finding myself trying to do prototypal things in Python. That didn't work.

#!/usr/bin/python
import io, sys
import sqlite3

# get headers from CSV template
#   <http://theregoesdave.com/wp-content/uploads/2008/10/gmail.csv>
hdrs = io.open('gmail.csv', 'r').readline().strip().split(',')
print '"' + '","'.join(hdrs) + '"'

conn = sqlite3.connect('AddressBook.sqlitedb')
conn.row_factory = sqlite3.Row

cContact = conn.cursor();
cContact.execute("SELECT * FROM ABPerson ORDER BY Last,First")

# From the ABMultiValueLabel table
Labels = ("", "Mobile", "Work", "Home", "Other")

for contact in cContact:
	# get phone numbers
	cPhone = conn.cursor();
	cPhone.execute("SELECT value,label FROM ABMultiValue WHERE record_id=? AND property=3 ORDER BY label", (contact["ROWID"],))
	phones = {}
	for p in cPhone: phones[Labels[p["label"]]] = p["value"]
	if(len(phones) == 0): continue # skip people without phone numbers

	# get email addresses
	cEmail = conn.cursor();
	cEmail.execute("SELECT value,label FROM ABMultiValue WHERE record_id=? AND property=4 ORDER BY label", (contact["ROWID"],))
	emails = {}
	for p in cEmail: emails[Labels[p["label"]]] = p["value"]

	# Use "Section 1" for home/mobile
	# Use "Section 2" for work
	rec = {
		'Name': "%s %s" % (contact["First"], contact["Last"]),
		'E-mail': emails["Home"] if emails.has_key("Home") else emails["Work"] if emails.has_key("Work") else "",

		'Section 1 - Description': 'Home',
		'Section 1 - Email': emails["Home"] if emails.has_key("Home") else "",
		'Section 1 - Phone': phones["Home"] if phones.has_key("Home") else "",
		'Section 1 - Mobile': phones["Mobile"] if phones.has_key("Mobile") else "",

		'Section 2 - Description': 'Work',
		'Section 2 - Email': emails["Work"] if emails.has_key("Work") else "",
		'Section 2 - Phone': phones["Work"] if phones.has_key("Work") else "",
		'Section 2 - Company': contact["Organization"] if contact["Organization"] else ""
	}

	# prune empty fields and section descriptions
	def prune(dict, key):
		for k in dict.keys():
			if len(dict[k]) == 0: del dict[k]
		ct = 0
		for k in dict:
			if k.startswith(key): ct = ct + 1
		if ct == 1: del dict[key+' - Description']

	prune(rec, "Section 1")
	prune(rec, "Section 2")

	# write out as CSV
	row = []
	for h in hdrs:
		row.append(rec[h] if rec.has_key(h) else "")
	print '"' + '","'.join(row) + '"'

Hope it helps. Of course, you will need a jail-broken iPhone in order to SSH in and copy a raw file off it. If you use Windows, you can probably just sync your contacts to Outlook and then export from there.

Rocket Stick on Arch Linux

2009-06-09T07:06:00Z

Well, thanks to a kind employer and an ill-timed server crash, I now have a free Rocket Stick. It's an HSDPA USB modem from Rogers, a Canadian cellular carrier. Download speeds "up to" 7.2 Mbps in 3.5G. The downstream degrades steadily as I wander off into 3G and (gasp!) 2G.

Not a bad toy to have in the summer months. Now all I need is a really bright laptop screen and I can sit in parks all day, coding and yelling at hobos.

The device is a ZTE MF636 USB modem. Naturally, it's not supported under Linux. I don't even know if Rogers knows what Linux is. But the Internet came through, as it usually does, and within 45 minutes I had found all my answers on various forums and blogs. So here's a consolidated contribution that should work for most Linux distributions. I've tested this on Arch Linux, but the tools used are generic.

So the catch with this little device is that it has multiple personalities. When you first plug it into a Linux machine, the OS will see it as a generic CD-ROM device (eg, /dev/sr1). This is what the marketing boys call a "Zero CD" application install. The Windows/Mac drivers and software are in a small ROM on the USB device itself, so no extra CD required. If you're on a Windows machine, you simply plug the USB key into the computer and the computer will see it as a CD-ROM and fire up the Autorun executable. Voila.

What happens behind the scenes is this: The software runs off the CD-ROM and installs the drivers. When the installation is complete, the driver issues a command to the device that tells it to switch personalities. Now it's a USB modem, and has a new USB product ID associated with it.

This mode switch is the part that I used Windows for. I know, it sucks, but I had a Windows machine on hand so I took advantage.

1. Disable CD mode on the device.

Using a Windows machine, plug in the USB device and go through the short install wizard. Once done, close the Rogers app that starts up, then head into the Device Manager (Control Panel -> System -> Device Manager). Under the Ports section, find the COM port that's connected to the USB modem (ignore the Diagnostics one). Connect to that COM port through Hyperterminal, found in the Accessories area of the Start Menu. Connection parameters are

Bits per Second: 115200
Data bits: 8
Parity: None
Stop bits: 1
Flow Control: None

Once connected, type the following commands:

AT+ZOPRT=5
AT+ZCDRUN=8

This tells the modem not to use CD mode when it's first plugged into a computer. Now exit Hypterterminal and remove the USB modem. You're done with Windows.

2. Setup udev rules.

Create a file /etc/udev/rules.d/90-zte.conf that contains the following:

ACTION!="add", GOTO="ZTE_End"
SUBSYSTEM=="usb", SYSFS{idProduct}=="0031", SYSFS{idVendor}=="19d2", GOTO="ZTE_Modem"
GOTO="ZTE_End"

LABEL="ZTE_Modem"
RUN+="/sbin/modprobe usbserial vendor=0x19d2 product=0x0031", MODE="660", GROUP="network"
LABEL="ZTE_End"

3. Setup hal rules.

The current version of hal (that I have) doesn't know about the MF636, though it does acknowledge some earlier models. To inform it, create a file /etc/hal/fdi/information/10-modem.fdi with the following:

<?xml version="1.0" encoding="UTF-8"?> <!-- -*- xml -*- -->

<deviceinfo version="0.2">
  <device>
    <match key="info.category" string="serial">
        <!-- ZTE MF636 HSDPA USB dongle -->
        <match key="@info.parent:usb.product_id" int="0x0031">
          <match key="@info.parent:usb.interface.number" int="3">
            <append key="modem.command_sets" type="strlist">GSM-07.07</append>
            <append key="modem.command_sets" type="strlist">GSM-07.05</append>
          </match>
        </match>
    </match>
  </device>
</deviceinfo>

4. Create a wvdial configuration.

Wvdial is an easy-to-use frontend to PPPd. The configuration is fairly easy to comprehend. This one is probably longer than it needs to be, but I'll include it all. Make sure you replace the /dev/ttyUSB0 line with the node that your USB modem is connected to.

[Dialer Defaults]
Modem = /dev/ttyUSB0
ISDN = off
Modem Type = USB Modem
Baud = 7200000
Init = ATZ
Init2 = 
Init3 = 
Init4 = 
Init5 = 
Init6 = 
Init7 = 
Init8 = 
Init9 = 
Phone = *99#
Phone1 = 
Phone2 = 
Phone3 = 
Phone4 = 
Dial Prefix = 
Dial Attempts = 1
Dial Command = ATM1L3DT
Ask Password = off
Password = off
Username = na
Auto Reconnect = off
Abort on Busy = off
Carrier Check = off
Check Def Route = off
Abort on No Dialtone = off
Stupid Mode = on
Idle Seconds = 0
Auto DNS = on

5. Dial up to the interwebs.

Now just run wvdial to connect!

# wvdial -C /etc/wvdial.conf

If you see output reporting your PPP local and endpoint IP addresses, then you've won! Crack a beer and troll some forums.

6. Acknowledge the real heroes.

Thanks to the following webpages that gave me all this information:

Update: Fixed a bug in my udev rules.

Phatso

2009-04-29T10:04:00Z

A while back I wrote a tiny little micro framework for a contract. It's been sitting around on my hard drive for awhile, so I figured I should throw it up here in case anybody ever has need for such a thing.

Meet Phatso.

The framework itself is only about 100 lines of code. It handles basic dispatch and templating functionality, nothing more. There is an auxiliary file included for PDO-based database access as well.

Phatso is intended for small web applications that could still benefit from a bit of framework-y structure. A typical web app could probably reside in one or two PHP files. It's not designed to be a full-fledged framework, only to provide the basic stuff (clean URLs, templates) a little web app might like to have.

Clean Switch Blocks

2009-03-31T09:03:00Z

There's a time and a place for switch blocks in programming. Normally you switch on a variable, then provide a number of case blocks that provide a (hopefully succinct) chunk of code to be executed if the switch variable matches the case value.

<?php
switch($pizza) {
  case 'italian':
    $ingredients = array('ham','pepperoni','onions','peppers');
    break;
  case 'mediterranean':
    $ingredients = array('garlic','feta','tomatoes','spinach','olives');
    break;
  case 'hawaiian':
    $ingredients = array('ham','pineapple');
    break;
  default:
    // plain ol' cheese - borrrrrring
    $ingredients = array();
}
?>

Simple, right? But there are often times when you'll find yourself building a long, kinda ugly-looking if/elseif/else block simply because you need to evaluate something more complicated than the value of a variable.

<?php
if($a && $b) {
  func1();
} else if($a && $c) {
  func2();
} else if($b && $c) {
  func3();
} else if($c) {
  func4();
} else {
  func5();
}
?>

Those happen all the time and there's nothing wrong with them. Well, except for my contrived example. Forgiveness, please.

But you can often improve readability by putting blocks like this into a switch statement. The trick is to invert the placement of the switch variable and the case values. A switch operator can take any value statement and compare it to a number of other values (cases), so it follows that we can use a static value at the top.

Try this:

<?php
switch(true) {
  case $a && $b: func1(); break;
  case $a && $c: func2(); break;
  case $b && $c: func3(); break;
  case $c:       func4(); break;
  default:       func5();
}
?>

This does the same thing as the if/elseif/else block above, but it's more compact without sacrificing readability. But, like switch statements, there's a time and a place for them. If you have a lot of code that needs to go in each case block, then it's probably better to put the code in functions, or failing that, in a normal if/else block. Big bulky case statements can be tough to read, since they don't have the normal block delimiters.

Have any other useful programming idioms to save time or space?

Working Backwards: New Pronto Data Models

2009-03-25T06:03:00Z

For a long time, I've been wanting to redesign the data models in Pronto. They serve their purpose fairly well, but I've always found myself writing small methods in the model classes that don't really need to be written. They were simply enough that it was a waste of time to even have to write them.

A simple example:

<?php
  function get_active_users()
  {
    return $this->db->get_all("SELECT * FROM users WHERE status='active'");
  }
?>

My primary intent for doing this sort of stuff was to keep any SQL out of the controllers that might be requesting data like this. Of course, I ended up with a ton of little methods littering my data models. It wasn't ideal.

I also had a bunch of methods in the base model class that were more-or-less convenience methods to avoid some of these get_whatever() methods, stuff like Model::get_all(), Model::get_by(), etc. It was clear that there was some room for improvement here.

I had been struggling to come up with a very simple and robust API to replace this stuff, but I always got hung up on the details before I'd even get to how the API itself was going to work. I was trying to envision the internals and then move on to the actual API once I had the internals figured out.

So a couple nights ago I tried the reverse. I started with the API, ignoring (for now) how the whole thing would actually work under the hood. I started up vim and just wrote out a bunch of example method calls, just as I would want them to look. I ended up with a fairly small and fluent model API, and it actually proved to be pretty easy to work backwards into the internals.

The big change was adding an auxiliary model class, RecordSelector. This guy builds result sets of zero or more matching records. The matching rows can then be operated on with a few different primitive data manipulation calls (get, set, load, etc.).

Here's a few examples:

<?php
  // get first 3 users from Romania
  $users = $m->find("country='Romania'")->order('name')->limit(3)->load();

  // set column for one record
  $m->find('id=12')->set('country', 'United States');

  // delete people from Latvia (sorry Latvia)
  $m->find("country='Latvia'")->delete();
?>

So far I've been able to rip out a lot of app-level model code, while still keeping the code fairly concise and readable. Win.

I'm hoping to make a new Pronto release soon, but it still might be a few weeks. If you'd like to take a peek, you can grab the latest n' greatest through git:

$ git clone http://zeroflux.org/pronto.git

Small and Fun Project Seeks New Home

2008-10-12T06:10:48Z

Are there any C hackers out there who'd like to take on a new project?

I wrote knockd in my final year (2004) of university as a proof-of-concept. It was featured on Slashdot and generated a fair amount of interest, including some neat contributions from other port-knocker people out there.

I don't really use the program anymore, and so I'm probably not the best person to be maintaining it. If you are (or know of anyone who is) interested in adopting the project, please contact me. jvinet at zeroflux dot org.

HTPC For The Rest Of Us

2008-10-05T10:10:06Z

Bec and I have an old, beer-stained laptop that we use as our media player. It's a tank and the power button is broken, which can make things a bit tricky. But all in all, it works.

I also have an ATI Remote Wonder that we use with mplayer. So the only thing left is some sort of navigable menu system so I don't have to get off my lazy ass to pick a new show each time.

We used to use Geexbox, which is a great little home theatre PC (HTPC) setup. But Geexbox is a standalone system that boots off a CD into its own minimal flavour of Linux. Problem is that this tank of a laptop doesn't have built-in wireless, and the USB wifi adapter we're using requires a windows driver (via ndiswrapper) to work properly, and Geexbox doesn't have this, to the best of my knowledge.

So I installed Arch on the machine, got everything working, and started looking at other HTPC software. I tried Freevo and Elisa. They needed more setup and configuration than the plug-and-play Geexbox I was used to. I also tried a few smaller pieces of software that did the bare essential - just show a directory of files and let the user select one to watch. They all kinda worked, but not really. I think they were proof-of-concepts that never really got finished.

After the 19th segfault of one, I decided to write my own little guy. Hadn't used python in awhile, so it would be good for me.

So here it is, 188 lines with comments. It does only the basic file navigation stuff, using either the keyboard or an LIRC-compatible remote control. Once a file is selected, everything else is handled by mplayer.

If you want to use a remote, then you'll have to setup your LIRC configuration file to map your remote buttons to the correct functions in the HTPC menu. Here's an excerpt from my ~/.lircrc file:

begin
button = UP
prog = pyhtpc
config = UP
repeat = 5
end
begin
button = DOWN
prog = pyhtpc
config = DOWN
repeat = 5
end
begin
button = LEFT
prog = pyhtpc
config = BACK
end
begin
button = RIGHT
prog = pyhtpc
config = SELECT
end
begin
button = OK
prog = pyhtpc
config = SELECT
end
begin
button = POWER
prog = pyhtpc
config = QUIT
end

There will be bugs. If you find some, please fix them so I can take credit.

Bootstrap your i18n with Pronto

2008-09-29T06:09:02Z

I've been steadily working at rounding out some areas of Pronto lately, and one has been the i18n support. Pronto supported multiple languages and UTF-8 before now, but that functionality didn't extend to application modules, and my little hack to translate message files via Google Translate was only partially complete.

With the next release, however, this stuff will be actually usable. Google Translate supports all the major languages, so in a couple commands, you can now do this:

Scan your entire application codebase (including modules) and compile all strings into a messages file; and
Automatically translate that messages file into the following languages:

Arabic

Chinese Simplified

Chinese Traditional

Dutch

French

German

Greek

Italian

Japanese

Korean

Portuguese

Russian

Spanish

Of course, Google Translate is far from perfect, so this only really serves as a "good enough for now" solution. Once your application is production quality, so should your translations, and that means getting translations from real human beings.

Here's a quick excerpt showing how I build my initial messages file and then translate them:

[jvinet@neptune site]$ php pronto/bin/i18n_scan.php en English
Scanning app/bin
Scanning app/config
Scanning app/core
Scanning app/models
Scanning app/pages
Scanning app/plugins
Scanning app/profiles
Scanning app/templates
Scanning pronto/core
Scanning pronto/plugins
Scanning pronto/profiles

Successfully updated app/config/i18n/en/messages.php

[jvinet@neptune site]$ cd app/config/i18n

[jvinet@neptune i18n]$ php ../../../pronto/bin/translate_all.php
Translating English to Arabic...
Translating English to Chinese Simplified...
Translating English to Chinese Traditional...
Translating English to Dutch...
Translating English to French...
Translating English to German...
Translating English to Greek...
Translating English to Italian...
Translating English to Japanese...
Translating English to Korean...
Translating English to Portuguese...
Translating English to Russian...
Translating English to Spanish...

Yep, that's it. Stay tuned for the next release.

http://www.prontoproject.com

Wikipedia Strikes Again

2008-09-23T16:09:38Z

I had aspirations of getting a lot of work done today. But instead, I lost over two hours to Wikipedia.

It's happened before. I started out on the page for the Halifax Explosion, as we live near the memorial park for the event. The Explosion is touted as the largest manmade explosion until the Atomic Bomb.

So of course, I ended up following through and reading up on the A-Bomb, which then led me to the Little Boy and the Fat Man, and the gunshot-style design versus the implosion-style design. This led me to the properties of Uranium-235 and Plutonium-239, which then led me to the strong and weak nuclear forces, which then led me to ionizing radiation, which led me to the Curies and Becquerel, which led me to quarks and gluons, which led me to...

You get the picture. I'll never get that morning back.

I'm not even sure I actually learned anything. Stupid hyperlinks.

X-Canada: Halifax!

2008-09-06T07:09:34Z

We are now Haligonians.

Got here on Sunday, and we're finally starting to feel like we're getting set up here, at least a bit. It's been a long week, with many dollars spent, but I can now sit at a table with a computer, on the internet, with clean clothes, and even food. That's pretty good.

The trip across Canada was very cool. I'm actually in a bit of withdrawal about it, to tell the truth. I kinda liked the nomadic lifestyle of travelling 500km and camping on the shores of Lake Superior. Our only real concerns were gas, finding a campsite, and buying beer. We took a ton of pictures, and will post them up here or on FB once we sort through them all.

Bec and I are both feeling pretty homesick for Victoria, but that's not really surprising. We know nobody here, the road system is arguably worse than Victoria (streets intersect themselves and change names frequently), and the entire city seems fairly anti-parking in general. Also, the number of sushi joints is disconcertingly low. Speaking of disconcertingly-low, the microbrewery representation here is, while present, not comparable to BC.

We're currently living in a 100+ year-old house, one of the few in this area that survived the Halifax explosion back in 1917. It's not the worst, but we're definitely not looking forward to the heating bill this winter, so we may have to look for new digs in a couple months.

But there are good things too. Bec is really stoked about her new program, and the people here seem really friendly. Since I work from home, it's going to be a bit tricky to get out and make a social network here, so I've been considering the idea of a "real" job. Too soon to act on it, due to other obligations, but I might start looking in a month or two.

Also picking up the guitar daily so I can get back into a jamming circle. I've been missing that a lot, and apparently Halifax has a really rich music scene. It's Wintersleep's home city, so that's a good sign.

To friends and family back home - we miss you tons. Please visit us someday. Bring sushi.