Software Architecture

Broad overview

The basic "trick" getting a web and javascript based application to interface with local hardware that's not the keyboard. For good reason, we're not allowed to start messing with USB from within a webpage. This solution comes from the Ryo Katsuma of Tokyo Jogging that put a micro httpd server around his Wiimote code to drive Google StreetView.

There are three main chunks of code that required creation.

Arduino sketch to interface bike sensor triggers into serial strings
Perl based HTTP microserver to interface between browser and peripherals
Javascript to interface between microserver and Google Street View

Once the microserver has been started, the browser can connect and download all the required framework HTML and Javscript. When loading is complete, the browser issues AJAX requests every 100ms to the microserver for updates on the user's status. The microserver keeps track of each peripheral's state for immediate retrieval when requested by the browser. Javascript on the browser side will then handle all the calls to the Google Street View for updates on movement and point of view.

Arduino Sketch

The easiest code to write. In the main loop, watch pin 12 for changes from high to low and when that happens, send a "1\n" over the serial connection. Each "1" notifies the computer that a full revolution of the wheel has occured and we can calculate that the bike has travelled Π*wheel_diameter.

HTTP Microserver Architecture

The microserver runs three threads tasked with the following functions.

Monitor orientation sensors and handle the calculations for pitch and yaw
Monitor the serial ports for updates from the bicycle sensor and calculate total travel distance
Respond to requests from client brower for updates and pages

There is one shared memory structure between all the threads that holds peripheral state. The two threads for peripherals update this data structure as reports come in. When a request comes in from the browser, this data structure is serialized into JSON and sent to the browser.

Orientation Sensors Monitoring Thread

The Vuzix VR920 provides two sensors. A three-axis accelerometer and a three axis magnetometer (or eletronic compass). The data is recieved over the USB connection and on my Ubuntu 8.10 system, the usbhid driver takes care of dealing with the USB packets. It shows up at "/dev/usb/hiddevice0" and it's possible to read from the device like a file receiving a very verbose stream of updates.

After the binary stream, it yields 6 signed wored values. Acceleration horizontal, acceleration up, acceleration forward, magnetic horizontal, magnetic up, and magnetic forward. Some calibration is necessary so that the system has an idea what each field's range can be. Then it appears to be possible to treat the values as two vectors, that added together to a single vector can translate into single polar coodinate that holds the two key values for determining POV for Google Street View. See the function "calc_orientation" in server.pl for the equations.

I should note here that I didn't derive the proper equations as much as reimplement them. Trying to get things to work forced me to understand how they work.

There is A LOT of noise produced in the data, staying still causes lots of jitters to occur. It's a little better by using the average of every 40 reports but there's still quite of bit of wiggle.

Bike Wheel Rotation Monitoring Thread

This thread is easy. The FTDI breakout just shows up as a USB serial device so on my system, it's "/dev/ttyUSB0". Opening a blocking filehandle and every time a "1" is received, adding the circumfrence of the wheel to the total distance is all that's required.

HTTP Microserver Thread

This uses Gisle Aas' HTTP::Daemon module and works like a charm. It's very easy to setup special URLs that cause events like HMD view-point reset and peripheral status update requests. It also handles things like standard file serving. This was the second easiest part of the code (thanks Gisle!)

AJAX Browser Interface

I'll have to admit, this was the most challenging part. I used the awesome jQuery library with the timer plugin for event polling. Google's Street View API was surprisingly comprehensive which was great. However, the jitter from the headset data and the large number of asyncronous requests for every panorama point caused a headache.

In more depth, as the javascript polls the microserver and receives updates, it updates the point of view in the Street View panorama and checks to see if the user has travelled far enough to reach the next view point. When the user has travelled the distance to the next point, requests the system to update and sets up the new scene.

As the system is aware of which way the user is viewing, it's possible to handle intersections through the same interface. When an intersection is reached, the system will check if the user is looking down a turn and follow that. If not looking at anything in particular, just follow the same direction that the user was travelling previously and head through the intersection.

Calibration of the headset is required. There are three sets of data that need to be defined. Man there a lot of threes in this project. Anyways.

The range of values for each field determined by simply rotating the HMD and recording the minimum and maximum. Once these values are known, it's possible to calculate pitch (horizontal rotation) and yaw (vertical rotation)
By default, it seems that north is considered "forward". As it's much more pleasant to not have to crane the neck, this is a single value that sets the "0" reference point for the yaw.
Google Street View at 0 degrees thinks that I'm looking north. For the system to work, I need to set the initial point of view down the direction of a road, aka: heading.

Google, if you see this, would you consider adding a GLatLng object into your GStreetviewLink? When using GStreetViewClient, querying for a location yields a GStreetviewLink[] that requires a new query to Google for each link found to determine distance away. While I'm sure the servers can take it, it adds complexity to my code.

About

Aki is just another geek. He finds writing about himself in the third person awkward. He also works for a company called Gossamer-Threads that is filled with talented people - if you need custom code or hosting done, talk to us.