Reverse engineering a client USB driver with Nodejs

Its been a while since my last post and I got a little off track from my intended project #1(which I will be coming back to probably in the coming weeks) but I took a bit of an aside to plant a small garden and be outside a  bit. I did however take on a side project that tickled my fancy (remember how having fun is #1 on the list?): learning how this new-fangled USB works.

The idea started out of an upcoming potential project at work that got me learning nodejs (which is hella cool btw) and through the process of looking at how to use nodejs for something cooler than just a webserver or high-performance REST API, so I settled on a proof-of-concept attempt to access a USB device from node. Luckily a lot of the hard work had been done for me by others so it wasn’t actually necessary to develop my own nodejs USB module (which would have been cool in its own right, but not reinventing the wheel is another part of my list of “rules”).

The result of maybe a couple weeks of +- 1/hr per night (so maybe 10-15 hours total) is a rough prototype of a nodejs script that can read the temperature out of this little guy. The idea being to take some existing hardware, reverse engineer its protocol and write my own client “driver” to allow me to do more than I could do with the chincy, poor english standalone application that came with it on an archaic “CD-ROM” (whatever that is). As part of this project I tried to find other tutorials documenting the reverse-engineering process on a USB device and found several that got right up to the part where I was having problems and then stopped, so perhaps the internet can have 1 reasonably easy to find fairly-successful tutorial on reverse engineering a simple USB device!

This is going to be the overall structure of the rest of this post

  • USB Overview
  • USB Analysis Tools
  • Node and USB
  • Putting It All Together

USB Overview

A fair amount of time (maybe most?) was spent trying to become a quick expert in USB. Once I realized that wasn’t going to happen, I tried to learn as much about the USB bus as I could from a general hardware level and a low to mid level software. Despite reading and re-reading the same and different USB guides multiple times, it never seemed like I could totally absorb the protocol. It was interesting however to see how different USB is from other serial bus protocols that I have had experience with in lower level embedded system contexts (like I2C, SPI, RS232 etc). At the end of the day, give me a oscilloscope, mcu and a proprietary serial protocol any day. The USB protocol has to be so general to support the huge array of different devices that exist today. Not only that, but in many cases the devices wont have special drivers or software to come with them, so the result is a very abstract and challenging to understand system. Even after understanding the basic principles, you then have your OS of choice’s driver architecture to contend with (if you are so bold as to dare). The end result is that you mostly in abstractions, hopefully avoiding both OS specific driver architecture specifics AND hardware specifics, interacting mostly with basic USB concept abstractions.

I’m not going to try to go into depth on how the protocol works other than to impress the specific points I came away with. I found some pretty good resources that go into more and better detail than I could. In the next 3 resources, focus on understanding the process of USB transactions, and what endpoints, interfaces, and descriptors are. Note descriptions of descriptors for endpoints, interfaces, and configurations are because I had to refer to them many times in the nodejs code.

#1 This is a solid resource and despite the length (~30 pages), its worth reading through because it tries to cover everything. I skimmed most of the physical stuff and the really low level signal processing parts (despite that I like that kind of stuff, it wasn’t going to help me with what I set out to do).

#2 This is another resource that repeats some of the same content from #1, but goes into a little more detail. You can probably stick to page 1-5 (they are not short though) and skim a lot of the packet organization and signal-level stuff (unless you just are interested). The most useful things were about descriptors, interfaces, endpoints, and transactions (which are more applicable from the higher level client driver software level).

#3 This is a big, in depth review of the whole USB system but like the first two resources, a lot of it can be ignored. The link should start at chapter 3, when the discussion of descriptors, endpoints, and interfaces starts.

Takeaways

What you should take away from those resources is that a device is made up of interfaces which are then made up of endpoints. Endpoints can be IN or OUT for device-to-host and host-to-device respectively. Endpoint 0 has both an IN and OUT component for control transfers specifically and is present in all USB devices and is outside of an interface (it seems). Every USB device has at least 1 configuration and potentially more (my sensor had only 1). Configurations, interfaces, and endpoints all have descriptors that describe their settings and each has different fields. All of these attributes are general and don’t expose a lot of a devices internal workings (functionally), but it can help to understand what a devices capabilities are. In addition, in terms of the bus, it is important to remember that it is host centric, which means the host (your computer in most cases) is boss, and initiates ALL communication (control and data). All data transfer types are initiated from the host, even isochronous (specific intervals for streaming data), its just a matter of when, how and how often the host initiates a control or data transfer.

USB Analysis Tools

This was one of the most fun (god help me) parts of the whole process because it was the kind of byte-detective work that is sometimes fun. Anyone familiar with using wireshark for TCP/IP applications will be right at home, though I will say that I think USB traffic was harder to read and understand (the TCP/IP stack can be complex but it is also remarkably logical and intuitive once you understand some basic concepts).

A quick google search will show a nearly endless array of USB analyzers. These come in 2 main flavors: software and hardware. Since I’m not developing a physical peripheral, a hardware analyzer is overkill. For software analyzers there seems to be an equal array of command line and GUI analyzers. While command line tools can be powerful and more configurable sometimes, I really needed to visualize as much as possible and fit a lot of information onto a single screen, so I went for GUI programs. One promising choice was using USBPcap in conjunction with the wireshark gui (or without) since it is free and open source. I also tried two free pieces of software that had a decently long trial: USBTrace its seemingly hotter sister USBLyzer. USBLyzer seems to be an almost exact-but-nicer copy of USBTrace (or vice versa?) so I don’t know what is going on there, but they both showed mostly the same info. USBLyzer’s interface was better and it seemed to have more detail and was overall superior. Here is a report of my device: USB Temperature Sensor Report

Reverse Engineering Process

The process of reverse engineering involved trying to correlate actions to the actual bus transactions I could observe and then trying to find out what caused what. The cheesy Chinese (not there is anything wrong with that but the strange grammar gives it away) standalone program is below.

VB Anyone?

VB Anyone?

I spent quite a bit of time reading the analyses of transfers while reading more of the material from the USB overview above, but in retrospect I can make the process much more concise. There were really only 2 specific sets of operations I needed to do: capture the stream of USB packets when starting the software up and closing it, and then starting the capture with the software already up and clicking start and stop before stopping the capture.

These two interactions would have been enough to show me the process, which involve analyzing the control transfers and the data transfers. Note: for each single request there is an issue and a response. In the captures of the control transfers, the issue shows as “CLASS_INTERFACE” in the request column and the response is “Control Transfer”, but when using my nodejs script and a WinUSB driver they both say Control Transfer. Also, I had as many as 4 copies of the same request for each set of drivers installed (1 kernel, and 2 or 3 user for the different USB analysis tools installed), so that is why you see back to back duplicates.

class_interface_response

Control Transfers

There are 3 different control transfers involved, and I still do not know exactly what all 3 are for, but I know enough to reproduce them in the correct order to operate the hardware. Each control transfer had 2 useful pieces: the data and the report fields. Each of the reports had the same format, just with different data. The report is below, and it came important later in the node software. I can honestly say the only reason I noticed it was because the fields were the same as one of the node-usb API function parameters, not from some deep understanding of the process.

Startup transfer 1 (ST1) – data: 0x0182770100000000

This was always the first control transfer to take place.

Startup transfer 2 (ST2) – data: 0x0186FF0100000000

This transfer always took place after the first.

Data initiator (ST3) – data: 0x0180330100000000

This transfer always took place in-between data transfers.

Below is a summary of the control transfer turned out to be key in the node portion of the project. In the source code, look at the control transfers and read the API listing controlTransfer() function, the fields match up exactly. Initially I was trying to get the analyzer to print CLASS_INTERFACE for the ‘issue’ request. I never could, but using these values instead of trying to find a constant define for the bmRequestType in the usb module object saved me in the end.

setReport

Data Transfers

Data Transfers

The data transfer shows up above with filters applied to hide the control transfers. There are 3 distinctly different sets of data that come back, all in 8 byte sections (as defined in the endpoint descriptors). The first to be seen has the text “1.4Per1F” in ASCII. The second turns out to be the real data and reads 0x80 0x02 X X “er1F”, where the first byte always says 0x80, second 0x02, the next 2 change with temperature, and the final 4 are the same as the first request. The 3rd unique transfer just says “TEMPer1F”. There is obviously some pattern with each always having the last 4 bytes the same, and the 1.4 probably corresponds to the version as listed in the analyzer for the device descriptor, shown below.

After some detective work that involved taking captures with the software open and comparing the data packets from the capture from the temperature readings (was harder than it sounded like but in retrospect it should have been obvious), but it turned out that byte 2 (1 based) of the one changing reading was the whole number of the temperature (in hex, had to just convert). Byte 3 I knew was the decimal portion, but it took a little while to realize that to convert was to convert to base 10 and divide by 256 (which makes sense in retrospect).

deviceDescriptor

Clearly there was a lot of trial and error involved here. Actually the first ever data I got out of the thing was the “TEMPer1F” in node (which was unexpected at the time and I was totally stoked), because I only sent 1 control transfer or the wrong order or something, but there were few enough permutations of control transfers and input to figure it out. I’ve identified the following pattern that I missed until I finally got it working by trial and error:

Beginning of transfers

1. ST1

2. ST2

3. [ST1 and ST2 again in the case of opening up the software while capturing]

[Sometimes ST3]

Repeat

1. Data Transfer

2. ST3

End

1. [ST1 and ST2 in the case of clicking STOP in the software]

Node and USB

There is plenty of info out there on nodejs so just google it if you are unfamiliar. Coming into this project I knew that there would be a number of ways to access USB devices, and that they would probably be different between operating systems. I expected that it would mostly be done through C or C++, and since node was (for me) essential to the process, I looked into how to interact with C/C++ programs from node. Luckily, that is very doable. Being a total masochist, I half wanted to write an addon myself (and I might still down the road), but luckily for me others had already gotten there. There were initially several options, and I tried node-hid but the windows installation sucked and I could never get the addon to build correctly. The next choice was one simply called ‘usb‘, which turned out to work. In retrospect, the directions given for windows worked like a charm but it took me a while to get it right. For the most part you could just follow those, but I will point out a couple things that I had to do.

Zadig and libusb

The windows directions point to libusbx.org, but I went in circles for a while until I found libusb-win32 on SF (scroll down to download for the SF download link). Make sure to find the exe to make sure the dependent DLL’s get installed (other win32 ports of linux libraries).

Zadig is used to override the default windows drivers with the generic WinUSB driver that works with libusbx that is used by the usb nodejs addon. I had to run Zadig in compatibility mode for Windows XP (SP3) to be able to click Options>List All Devices (necessary to see the sensor). To identify your device (and not accidentally overwrite an important device with a WinUSB driver), use whatever analysis tool you settle on (like USBLyzer) to figure out the VIN/PIN of the device by trial and error and plugging/unplugging until you can identify which port you are plugging into, and then inspecting the device descriptor. The USB sensor has 2 interfaces, and I installed the WinUSB driver over both separately but since I only read from interface 1 that might not have been necessary.

When you do this, you will no longer be able to use the built in software, so hopefully you will have saved enough analysis data (that can be opened in USBLyzer or similar whenever), but if you need to go back it isn’t hard. Open the device manager and look under Universal Serial Bus Controllers for you device by trial and error and/or using the PORT# shown in your analyzer. All I had to do to get the stock windows HID driver installed again was to open the USB device in the device manager, click the driver tab and click uninstall. Pull the bad-boy out and plug it back in and then check in Zadig to see if it has changed back to HidUsb or whatever it was before.

At this point, installing the usb module and doing a quick test by finding my device using the vin/pin was pretty easy. I didn’t do this all in the order I am laying it out here, but if I were to re-do it this would be the way. I wasted a lot of time not knowing wtf I was doing. I will highly recommend one other module: node-inspector. If you like the chrome dev tools it is worth its weight in gold. It is a HUGE improvement on the gdb flashback-inducing built in node debugger. Now it is just a matter of finding a way to reproduce the process identified in your analysis within node.

I’m not going to try to go through my source code in depth, its reasonably commented. I will say that I did a fair amount of testing to make sure that there isn’t un-needed code in there (for instance, it is necessary to open the device, and the interface must be claimed before you can access the endpoints, etc).

So, take a look at the source code and there you have it. From a basic working knowledge of USB you will know enough terminology to know what you are doing (probably not necessary, but don’t just hack shit together like a noob, you may have to do it again some day), you can understand enough of whats going on in the analyzer to convert that into high-level nodejs code that can access your device. In my case, this adds a ton of value to my sensor because at this point anything can be done with the data. I can use socketio to stream it to other clients/servers, I can send it to my MQTT server, I can put it in database and make a custom graphing page in angular with the data. Before all I had was a janky standalone app, neat huh?

Source code and analysis files on github: https://github.com/thingsnsuch/Node-USB-Temp-Sensor

Advertisements

Server Stuff 2: REST api

This post should probably have come before MQTT because the work took place before I installed MQTT on my server, but MQTT got my interest and I wanted to write it up as I went along. My first foray into the server portion of this project was to write a very basic (and probably by competent node standards, bad) REST api.

The API is based off of this boilerplate. A boilerplate like this makes setup much easier. I’m going to go through the installation of the boilerplate, the configuration of Nginx, node, express and the whole rest of the plumbing stuff. After that, I’ll discuss the layout of the API itself and the mongo database.

TLDR: I don’t care about any of the plumbing!

Plumbing

Node

The first part of the plumbing is to install node. Node and NPM installation directions can be found for all kind of environments, but I decided to clone the github repositories and build them both, from here and here. Node must be built and installed first (NPM relies on it), after which NPM can be installed.

Nginx

Nginx was my choice for a webserver because I haven’t used it before and its one of the big top 3 or 4 webservers, and I already have experience with Apache (IIS is out of the question because I am on ubuntu). Running a webserver is not necessarily essential to a node server since node itself is already a webserver, but there are reasons to pair node with a general purpose webserver. One such reason is the same described here: it allows you to run multiple node services at one time, all on port 80. In addition, the webserver that is optimized for static content can be used for static html files while node can be used for REST api’s and dynamic content.

You can read here about how to install Nginx in ubuntu. The main thing that we’re wanting to do is set up what is known as a reverse proxy, whereby requests come into Nginx on port 80, and somehow Nginx knows which node instance to forward all the traffic to. The node instances can run on any local port, like 3000, but still be accessible on port 80 externally by a reverse proxy. In my case, when my domain is accessed with the prefix cm.XXXXcom/api/<endpoint>. I could also host any other number of subdomains for different projects, all node servers for different things.

To make the reverse proxy work, you have to configure it in Nginx with the config file located in /etc/nginx/conf.d/cm.<domain>.<tld>.conf to be something like below.

server { listen 80; server_name ~cm.<domain>.<tld>$; location / { proxy_pass http://localhost:<port&gt;; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection ‘upgrade’; proxy_set_header Host $host; proxy_cache_bypass $http_upgrade; }}

After writing the config file, try a >sudo nginx -t to make sure the config files are OK, and then >sudo service nginx start. At this point nginx should be good to go. Note: I snagged those settings so there may be ones I didn’t need or some useful ones I don’t have.

DNS

To get subdomains working (which you may or may not care about), you have to have a CNAME DNS record, along with your normal nameserver records. You can read about CNAME records here.

Boilerplate

The boilerplate can be cloned from here. After cloning the boilerplate, you can do an >npm install to get some of the dependencies and then bower install for the rest. You might want to install grunt-cli and bower globally (using npm for both, >npm install grunt-cli -g). Thats almost all it takes to get the boilerplate set up. The one thing you have to customize is the port in server.js and make sure this agrees with your nginx conf file.

API

After setting all of that up the API itself is pretty straightforward and easy. The goal of this first version was simply to make it possible to submit device readings, query for readings, query devices, and query for a devices configuration. This version of the API is as flexible as possible, making it easy to quickly add a new device (with a configuration) and submit readings.

The storage for devices, readings, and configurations is the nosql database Mongo, which allows readings to take a flexible form, so I am not locked into a specific schema.

API endpoints:

GET /api/devices
Returns: [deviceid1,deviceid2…]

GET /api/config/{KEY}
Returns: 1 record associated with that unique key

POST /api/search/{max_records}
Body: any JSON search object like {sensor: “temperature”} or {} for all readings
Returns: [{…},…,{…}]

POST /api/reading
Body: {key:XXXX, …}

There are definitely insecurities in this and it will surely be revised. The only “secure” portion of the configuration is the KEY. Posting readings requires a pre-defined key (the only necessary field in the reading object). This is not by any means fully secure, but it provides some security against basic junk requests or hijacking while making the development of a visualization page easy. The API can be secured eventually by enabling SSL encryption, which will protect the KEY from being sniffed over the air or wire.

This basic API should suffice to get started with the lower level parts of the system (like the microcontroller and the sensors), while having something to compare with the other serverside technologies I’m trying (like the MQTT server).

Server Stuff 1:Setting up MQTT on Ubuntu (pt 1)

So I mentioned earlier in a post that I was interested in the MQTT protocol. I really like the pub/sub architecture and the fact that each device can be a data producer and consumer. The most interesting thing though is the way you can use MQTT to interface external resources like twitter, cosm, or any other web service you might want to get set up as a subscriber to your data.

This evenings exercise was to set up an MQTT server and give it a test run. I am running an ubuntu 13.04 server on digital ocean. I chose the open source MQTT server mosquitto and so far so good. I’m going to lay out the steps I went through to get it working to a minimum level.

Step 1: Installing the server

I followed the directions here for Ubuntu but I had to add 1 other step. I did not have the command apt-add-repository, and as the downloads page said I used apt-get to install python-software-properties, but that page didn’t mention I also needed software-properties-common. After that, I used the commands sudo apt-add-repository ppa:mosquitto-dev/mosquitto-ppa and sudo apt-get update. On the downloads page they just seem to assume that we’ll figure out how to actually install the server itself, but here it is: sudo apt-get install mosquitto 

Step 2: Installing the clients

The downloads page does not list the client software, but it can be installed with the command sudo apt-get install mosquitto-clients. This gives you access to the commands mosquitto_pub and mosquitto_sub for publish and subscribe respectively. The man pages for each are here.

Step 2: Testing the install

As far as I’ve gotten on this first run is to set up the server, start it, and publish and subscribe one thing.

1. >mosquitto –daemon –verbose #stars the server as a daemon, verbose is optional.

2. >mosquitto_sub -t test -d #subscribe to the topic ‘test’ with debugging info.

3. Create a file, publish.txt with the word ‘hello’ in it.

4. >mosquitto_pub -t test -f publish.txt -d

You should see some text like the following in the window running mosquitto_sub:

Client mosqsub/31280-mynamehere sending CONNECT
Client mosqsub/31280-mynamehere received CONNACK
Client mosqsub/31280-mynamehere sending SUBSCRIBE (Mid: 1, Topic: test, QoS: 0)
Client mosqsub/31280-mynamehere received SUBACKSubscribed (mid: 1): 0
Client mosqsub/31280-mynamehere received PUBLISH (d0, q0, r0, m0, ‘test’, … (6 bytes))hello

There you go! You got MQTT up and running on your machine. The next step is to implement your own python publisher/subscriber using the mosquitto library described here. Now I didn’t do anything with the configuration file, I’m not using SSL/TLS or even a username and password but this is the most basic example, but now that this is working the sky is the limit! How about that, a short, sweet, to the point post. Might oughta make this a thing!

Project definition – Compost Monitor

Having discussed the project motivation, use case(s) etc, it is time to move on to the beginning of the technical work. This stage of project definition will encompass the main technical decisions that have to be made before actual implementation work can be done. These decisions are mostly high-level, even when pertaining to the specific hardware to be used. There wont be any discussion of actual code or nitty gritty details about protocols because at this stage, even if I have ideas about what I might have to do, I want to dig into the details at a time in which I can put them into action. The overall structure is going to be as follows:

  1. Communication Architecture, (high level)

  2. Protocols (a little lower level)

  3. Server-software (lower level still)

  4. Embedded hardware (lowest level)

Communication Architecture

The first part of this project from a development/technological position is to make some architectural choices. There are a number of different IoT architectures available for this kind of project, and each has a cascading effect on the rest of the system and requirements, so this is the kind of choice that needs to be made up front and stuck to. There are two main “categories” of architecture in my mind (citation definitely needed), the first I call the “sleepy system”, and the second being normally called “Real time” but I think that is a misnomer and confused with actual real time systems (which is a more somber, serious topic that deserves respect). Instead of real time, I will call this second category “on demand”.

Sleepy System

A sleepy system is one that is designed for low power operations that doesn’t place a lot of emphasis on current information. Knowing some measurement or observation that is a couple hours old is not a problem, and in many cases you might be more interested in the last few days or weeks worth of data more than what is happening right now. This well classifies the compost monitor, but day old graphs aren’t as sexy as minute by minute updates you know? Bi-directional communication is still possible in a sleepy system, but only when the embedded system initiates it.

On Demand Systems

There are some very cool developments in IoT techniques that are based on more asynchronous, bi-directional styles. The overall emphasis of the on demand architectural style is that data is requested from the device it is desired. This means the device may be polled from a server periodically to keep a dynamic web-page up to date, or the device may actually host the web page that displays the data. This type of system is open to bi-directional communication, the system could still operate as a sleepy system but then it wouldn’t be accessible if it were asleep, and it would cease to be available on demand. This is an attractive method, it would be cool to have the device posting data periodically but also being pollable such that an immediate value could be gathered. This would require some tricks to find a way to wake the device up on demand so that it could service these asynchronous requests.

To make this post shorter, check out the summary of IoT communication protocol architectures here. That post covers several of the specific implementations and protocols that could be used to implement a system like this. The details were muddling up the perfection that is this post so I moved it all out.

Protocol Conclusion

The overall category I’ve chosen is the “sleepy system” due to the lower complexity and cheaper components and the lack of necessity in accessing the system in real time. In the future I will attempt a more bi-directional but that can be V2.

In terms of protocols with which to implement the system, a REST interface is by far the easiest to implement which makes it an attractive option. I find the MQTT protocol very interesting as well and it would make a good addition as an implementation of a real-world IoT protocol. CoAP is also an interesting protocol that seems to be growing in popularity. As a result, I have decided to attempt to implement each protocol, starting with HTTP/REST since it is the easiest to get up and running, implementing MQTT next since it seems potentially easier than CoAP, and finally trying to implement CoAP.

This will require some forethought on the embedded system side by the use of a Strategy or Template pattern (depending on whether I wish to be able to switch between the methods at runtime or compile time. For the server components, it will be more of a matter of finding the appropriate server software to install, as there seems to be plenty of open source projects for both MQTT and CoAP. I’ll discuss how I am going to organize the implementation of multiple commnication protocols/architectures in the next post.

Serverside Software

The choice of server software to be used comes mostly from availability and what I want to learn as a result of the project. At work we use a .NET WebAPI based REST interface so to learn a popular upcoming REST api technique, I am going to use NodeJS based on the popular express framework. To store the data I am going to use MongoDB because it allows me to be less strictly structured up front and allows for more flexibility in the long run.

Embedded Software

My background in embedded systems involves mainly AVR microcontrollers (or, lower level VHDL digital prototyping). To choose a more modern system I had to chose between Arduino, Raspberry PI, or a TI development system (three of the more popular embedded technologies for hobbyists). I choose to use Arudino due to the fact that a) it is possible to access the lower-level architecture (which is an AVR microcontroller, which I have some experience with) and b) the way the AVR architecture is designed, it is often possible to minimize an arduino system to a smaller AVR microcontroller after the prototyping stage.

IoT communication protocols

There are a lot of protocols and communication methods available to IoT based projects and devices. They operate at different levels within the hierarchy and each have different pros and cons for a project like this. I wanted to briefly lay out some of the different options to give some comparison and background.

A lot of what is going on in IoT protocols takes place at the application layer of the protocol stack. The application layer is built on top of the transport layer. In TCP/IP this means either the TCP protocol or the UDP protocol. Different application layer protocols use different transport protocols. This is important to note because the transport protocol used can affect how the system performs, so taking this into consideration when choosing an application layer protocol is important. TCP is a connection based protocol, which means that it is reliable. Eventually, TCP will get your data to where it is going. It may have to retry a bunch of time, lose packets, timeout, and try again. This is good if your data is important, but it could incur very bad power efficiency if you have a spotty connection, and if your data isn’t so important that is has to go out right now, it may not be the best choice, especially if you know you are likely to have intermittent network issues. UDP on the other hand is unreliable, its the fire-and-forget protocol that plays it fast and loose. This means your data may never make it to its destination, but whatever your device will already be passed out on the couch again.

The following are some of the different application layer protocols that could be used to implement communication with an IoT device.

WebSockets

One method for bi-directional, on demand communication is websockets. WebSockets are part of the HTML5 specification and is largely used for real-time pushing of data from a server to a web-client. In traditional AJAX, most if not all communication begins on the client, requesting dynamic content from the server. This is used to update the UI dynamically without refreshing the page. This has drawbacks however, what if you are viewing “shared data” that can be updated by someone other than you, if this happens your data is out of date. Polling has been used for a long time to refresh that data, but WebSockets allow for a real time bidirectional connection between the server and the client, so the server can update any and all clients immediately when their data is out of date.

This is very cool from a embedded systems perspective, and I implemented a very rudimentary version of this for a independent study class (when I say very I mean very, it was just a physical LED and switch tied to a virtual LED and switch, the virtual switch turned on the physical LED and the physical switch turned on the virtual LED), but this was before I knew of WebSockets (and I was very inexperienced at javascript at the time) so I fell back on long polling  to get things done (which was pretty ugly tbh). Using HTML5 style websockets would require either a) implementing them myself in straight TCP sockets on the server and client (doable, probably in straight C and simplified), b) finding an embedded system library or a platform that already allows for this (better, more mature solution, but might limit my platform choices). This kind of real time async communication, while neat, doesn’t really lend itself to a battery powered remote system like mine, because I am more interested in trends rather than instantaneous values.

6LowPAN

Another option for server-originated communication is the 6lowpan technique, which since I first started reading about went from a “roll your own if you want it” option to now where you can actually buy 6lowpan hardware and development kits (that probably are somewhat proprietary and non-standard). The central idea here is that you have a 802.15.4 (*Bee) network in some star configuration with an “edge router” (I’ve seen it called that, not sure if that is the standard name or not) that serves as a translator between an IPv6 network and the *Bee network. In addition to the link/mac layer 802.15.4 traffic (which does not define a network or above layer, so no way to route data between networks), the 6lowpan spec defines a very minimized set of IP headers that make the *bee packets routable, which is very slick. The result is a system where you could have a big network of low-powered *bee based devices that have IPv6 addresses, these devices could host a web server/services (REST, SOAP [probably not]) that actually serves dynamic content on demand. The different here would be, instead of the device pushing the data to a remote database that then I would access when I wanted it, I would actually access the device directly and pull data from it. This means much more real-time access, in fact I could initiate a sensor reading right then and there to get some real fresh compost data (ew). However, this type of system requires more hardware, and the power management software on the device would have to be much more complicated to be able to wake up the device when a connection comes in and then put it back to sleep afterwards. I only have 1 device in mind right now which makes the hardware overhead pretty high, but this is cool as hell and I might try to implement it later.

The physical layer in this case is not that interesting, it is going to be RF for sure, which leaves either 802.15.4 or 802.11x or some proprietary low-frequency transceiver based dealy. As I may have mentioned before, I already have a WiFly unit and you can’t get cheaper than on-hand, right? This hardware selection picks a lot of levels of protocols for me actually, physical from 802.11b/g, link/mac (WiFi), transport (TCP or UDP) and optionally HTTP/FTP/etc on the application layer (which is optional with the WiFly).

I’ll try to be pretty brief in describing other choices here. If I went with a *Bee (802.15.4) wireless device here that would only handle the physical and link/mac layers for me (link is the lowest-level addressing layer, the ability to tell the difference between two network devices, while the MAC part of layer 2 describes how multiple networking devices share the same medium, ie the electromagnetic spectrum). The only real reason to go with *Bee would be to a) get a lower powered device or something that runs in the 900Mhz spectrum (longer ranges, better penetration of walls, obstacles) OR to run a 6LowPAN network like was discussed above. Now, if I wanted to go really cheap, I could go with a transceiver based setup where all I get is the physical layer and I have to write everything else in software (why is that kind of appealing?). This would be a great learning experience but in reality if my time is worth $0.01/hr, the $30 bucks for an *Bee or *Fi device will be well spent. Plus I do want to see those hot compost readings sometime soon, right? This is an area of optimization if you wanted to make a commercial product down the road, but I’m not thinking about that right now.

Ok that wasn’t that brief but if you are new to this stuff its helpful to have it broken down. Now to the interesting part of the protocol dicussion, the application layer. You might notice I skipped the network layer and transport layer, the reason for this is that a) the network layer for me is going to be IPv4 for now since I don’t really need any of the added features of IPv6 and I’m not even sure my WiFly supports it. Transport is either TCP or UDP, depending on which application layer+ protocols I choose.

HTTP+

What the + means here is that while the application layer is the highest layer of the TCP/IP model (not the OSI model though), there are other “protocols” that can be build on top of an application layer protocol (the most notable application layer protocol being HTTP). The HTTP protocol is by no means the only application layer protocol that could be used here, in fact there is a lot of work in the IoT space to develop more nimble, efficient protocols for IoT data. The reason for this is because the HTTP protocol has a lot of overhead since its main job has been for many years to serve up static content (web pages). It has been modified and adapted to serve up all kinds of other content though like dynamic HTML, XML, and most recently and popularly, JSON. Serving up and receiving JSON formatted data is the bread and butter of the REST web service architecture. RESTful web services use HTTP (on top of TCP) as the plumbing for writing cozy API’s that are nice to work with with AJAX. This is not a bad option for me because there are TCP and HTTP features on the WiFly to. In addition, a lot of my work involves REST API’s so the learning curve is much lower. There are however some alternatives that are more in the IoT realm that I am considering so as to get some experience in real IoT technology that might look good on a resume down the road.  Since HTTP is built on TCP, it is reliable and I can expect my readings to get recorded or my device will (literally) die trying.

MQTT

MQTT has been around for a while and began its life at IBM. I am not an expert in MQTT but I have been doing a lot of reading about it and trying to understand whether it would make a good fit for my project. Here is a good example of MQTT in action. My understanding of MQTT is that it is somewhat like the Observer design pattern with a few differences. In the observer pattern, any entity can implement the subject interface, which gives it the ability to be subscribed to by objects that implement the observer interface. This means when something changes on the subject, the subject calls the notify() method on all of the objects that are subscribed. An object can be both an observer and a subject, which means two objects could be interested in what each other does, so they are both subjects and both observers, passing updates back and forth. MQTT has a similar publish/subscribe paradigm where each node can be both a publisher and subscriber to different “feeds” (my term) that are loosely defined. The difference here seems to be that there is some kind of central system to which all of the nodes are connected, which is like a server. There seem to be a lot of MQTT servers out there, one of which at least is based off of IBM WebSphere. Each node that wants to publish and subscribe could be implemented in totally different ways, you could have a phone app, embedded system, desktop system, RSS Feed, Twitter bot etc that all have publish/subscribe abilities in the MQTT network, they just have to implement the interfaces that allow them to connect to the MQTT server. With the beauty of abstraction, the connections between the MQTT server and connected devices could be lots of different things. For instance, using MQTT I could have a web service that was subscribed to my compost monitor whose sole job was to make tweets when my compost monitor publishes a new datapoint. My compost monitor could subscribe to configuration settings like how often to sample. How often to sample could be published by my phone, my desktop, or even remote users on my twitter (I don’t actually twitter really its just an example). Another subscriber could be a storage system that uses MongoDB and a nodejs REST API to make the information available forever. From a power consumption standpoint, my embedded system would probably just be a publisher, else we run into the same situation with 6LowPAN and WebSockets, but it is a very elegant solution to how to organize a whole house full of instrumented things! MQTT is designed to use TCP by my understanding, but there are discussions on the webs about MQTT as an unreliable (UDP) protocol.

CoAP

CoAP is another effort to reduce overhead in the application layer for IoT type (constrained) devices. CoAP is cool because it is designed to be closely compatible with HTTP and REST, but rather than simply compressing HTTP headers, it is a totally seperarate protocol that maintains the same stateless interactions that HTTP does but with vastly lower overhead. This means that CoAP requests and responses can be easily mapped to full blow HTTP requests and responses, allowing the transition to the web to be a lot easier. My reading about CoAP has left me with the sense that it might be overly complex for my needs, but this presentation indicates some of the promise of using it. What it would entail would be much like MQTT, running a specialized UDP based server to be the endpoint for the CoAP traffic, converting CoAP requests to HTTP and converting the HTTP responses back to CoAP responses.

Project Description – Compost Monitor

As discussed in my initial post, I am going to begin a project with a project description post. This should help me to organize my thoughts and motivations for a project and help readers to understand why I am doing this, what I hope to learn and what the final product should look like. The shortest description of the project sufficient to go over the motivation is this: I want to be able to monitor the internal temperature of my outdoor compost system wirelessly so I can know that it is warm enough to sufficiently compost.

There are two main questions to get through before moving onto the technical parts of the project:

What is compost?

There are plenty of resources on what compost is, how to use it and how to do it in the city (where I am), so I’m not going to spend a lot of time on this topic. Feel free to ask any questions and after I get permission, I’ll add a link to a post on a shared blog that I contribute to that is more related to stuff like that. The most important compost-related info you need to know to understand my motivation is that organic materials in a pile-like structure needs to get to a certain temperature before it begins to form the nice stuff that plants crave (no not brawndo).

Why compost? 

There is a pretty good description in the blog I will link to soon of why to compost in general and why I care about composting but for extreme shorthand: I care about composting because it gives me a way to get rid of food scraps/waste without throwing it in the trash which is not good. I also grew up on a farm and I miss growing things, so as soon as the weather turns nice I plan on planting some stuff and I want some things to feed my plants. Ok, I probably spent a couple thousand words on my other blog just to say those couple sentences…anyways. The motivation for me picking this project as my first post-masters-thesis hardware/software IoT project is three fold:

1. It is damn simple compared to my thesis. It is very similar in terms of the types of technologies, so I have a decent starting point, but its a 1-3 sensor deal in my back yard and I already have 80% of the hardware, software, and tools needed to do it. This makes it <powerthirst_voice>EXTREMELY DOABLE</powerthirst_voice>.

2. The reasons in point 1 make this a good project to get back in the game, but it also will help me to develop a nice platform for monitoring other (potentially more complicated) stuff. This way I can explore all kinds of nifty technological stuff like protocols and hardware, best practices etc with a simple application that I can expect to finish.

3. It is a perfect example of how the IoT can combine very different interests and actually add value to non-technological hobbies. That doesn’t mean I wont get a lot of pleasure out of digging holes and getting dirty with simple analog shovels and mattocks that are in no way instrumented (….yet…..).

Back to the project description/use case

To make this somewhat of of a modern, agile process , I am going to start with a single epic story that will encompass what I want the system to do from a user perspective, which can then be used to define all the individual stories and tasks that, when implemented, will make up the system. Here goes….

As an enterprising young urban gardener on the go, I would like to be able to see what the current internal temperature of my compost whatever (tumbler, pile) is  from any web-enabled device and look at past temperatures so that I can know more quantitatively how my compost is progressing. 

Ok, that wasn’t too hard now was it. A lot will go into realizing that single story of course, but that locks down the big picture. One thing that makes this different than a product that I was making for someone else is that I am the customer in this case. That means I also get to decide HOW I want all this stuff to get done, how much I am willing to pay, how I want to access it etc. This can be good and bad because it can lead to mission-creep, but that is an expected problem for any personal project.

As the would-be user of this system I have some ideas about what this system will look like right now, but this could of course change. The first plan is very simple: A webpage somewhere that has a big CURRENT TEMPERATURE XX °C thing, and a graph that I can adjust the time period on that shows past temperatures from X to Y datetime. As a developer you would have mockups at some point to work off of, but I probably won’t get that rigorous about the process.

Alright, lets get to some hot TECHNICAL ACTION!