Its been a while since my last post and I got a little off track from my intended project #1(which I will be coming back to probably in the coming weeks) but I took a bit of an aside to plant a small garden and be outside a bit. I did however take on a side project that tickled my fancy (remember how having fun is #1 on the list?): learning how this new-fangled USB works.
The idea started out of an upcoming potential project at work that got me learning nodejs (which is hella cool btw) and through the process of looking at how to use nodejs for something cooler than just a webserver or high-performance REST API, so I settled on a proof-of-concept attempt to access a USB device from node. Luckily a lot of the hard work had been done for me by others so it wasn’t actually necessary to develop my own nodejs USB module (which would have been cool in its own right, but not reinventing the wheel is another part of my list of “rules”).
The result of maybe a couple weeks of +- 1/hr per night (so maybe 10-15 hours total) is a rough prototype of a nodejs script that can read the temperature out of this little guy. The idea being to take some existing hardware, reverse engineer its protocol and write my own client “driver” to allow me to do more than I could do with the chincy, poor english standalone application that came with it on an archaic “CD-ROM” (whatever that is). As part of this project I tried to find other tutorials documenting the reverse-engineering process on a USB device and found several that got right up to the part where I was having problems and then stopped, so perhaps the internet can have 1 reasonably easy to find fairly-successful tutorial on reverse engineering a simple USB device!
This is going to be the overall structure of the rest of this post
- USB Overview
- USB Analysis Tools
- Node and USB
- Putting It All Together
A fair amount of time (maybe most?) was spent trying to become a quick expert in USB. Once I realized that wasn’t going to happen, I tried to learn as much about the USB bus as I could from a general hardware level and a low to mid level software. Despite reading and re-reading the same and different USB guides multiple times, it never seemed like I could totally absorb the protocol. It was interesting however to see how different USB is from other serial bus protocols that I have had experience with in lower level embedded system contexts (like I2C, SPI, RS232 etc). At the end of the day, give me a oscilloscope, mcu and a proprietary serial protocol any day. The USB protocol has to be so general to support the huge array of different devices that exist today. Not only that, but in many cases the devices wont have special drivers or software to come with them, so the result is a very abstract and challenging to understand system. Even after understanding the basic principles, you then have your OS of choice’s driver architecture to contend with (if you are so bold as to dare). The end result is that you mostly in abstractions, hopefully avoiding both OS specific driver architecture specifics AND hardware specifics, interacting mostly with basic USB concept abstractions.
I’m not going to try to go into depth on how the protocol works other than to impress the specific points I came away with. I found some pretty good resources that go into more and better detail than I could. In the next 3 resources, focus on understanding the process of USB transactions, and what endpoints, interfaces, and descriptors are. Note descriptions of descriptors for endpoints, interfaces, and configurations are because I had to refer to them many times in the nodejs code.
#1 This is a solid resource and despite the length (~30 pages), its worth reading through because it tries to cover everything. I skimmed most of the physical stuff and the really low level signal processing parts (despite that I like that kind of stuff, it wasn’t going to help me with what I set out to do).
#2 This is another resource that repeats some of the same content from #1, but goes into a little more detail. You can probably stick to page 1-5 (they are not short though) and skim a lot of the packet organization and signal-level stuff (unless you just are interested). The most useful things were about descriptors, interfaces, endpoints, and transactions (which are more applicable from the higher level client driver software level).
#3 This is a big, in depth review of the whole USB system but like the first two resources, a lot of it can be ignored. The link should start at chapter 3, when the discussion of descriptors, endpoints, and interfaces starts.
What you should take away from those resources is that a device is made up of interfaces which are then made up of endpoints. Endpoints can be IN or OUT for device-to-host and host-to-device respectively. Endpoint 0 has both an IN and OUT component for control transfers specifically and is present in all USB devices and is outside of an interface (it seems). Every USB device has at least 1 configuration and potentially more (my sensor had only 1). Configurations, interfaces, and endpoints all have descriptors that describe their settings and each has different fields. All of these attributes are general and don’t expose a lot of a devices internal workings (functionally), but it can help to understand what a devices capabilities are. In addition, in terms of the bus, it is important to remember that it is host centric, which means the host (your computer in most cases) is boss, and initiates ALL communication (control and data). All data transfer types are initiated from the host, even isochronous (specific intervals for streaming data), its just a matter of when, how and how often the host initiates a control or data transfer.
USB Analysis Tools
This was one of the most fun (god help me) parts of the whole process because it was the kind of byte-detective work that is sometimes fun. Anyone familiar with using wireshark for TCP/IP applications will be right at home, though I will say that I think USB traffic was harder to read and understand (the TCP/IP stack can be complex but it is also remarkably logical and intuitive once you understand some basic concepts).
A quick google search will show a nearly endless array of USB analyzers. These come in 2 main flavors: software and hardware. Since I’m not developing a physical peripheral, a hardware analyzer is overkill. For software analyzers there seems to be an equal array of command line and GUI analyzers. While command line tools can be powerful and more configurable sometimes, I really needed to visualize as much as possible and fit a lot of information onto a single screen, so I went for GUI programs. One promising choice was using USBPcap in conjunction with the wireshark gui (or without) since it is free and open source. I also tried two free pieces of software that had a decently long trial: USBTrace its seemingly hotter sister USBLyzer. USBLyzer seems to be an almost exact-but-nicer copy of USBTrace (or vice versa?) so I don’t know what is going on there, but they both showed mostly the same info. USBLyzer’s interface was better and it seemed to have more detail and was overall superior. Here is a report of my device: USB Temperature Sensor Report
Reverse Engineering Process
The process of reverse engineering involved trying to correlate actions to the actual bus transactions I could observe and then trying to find out what caused what. The cheesy Chinese (not there is anything wrong with that but the strange grammar gives it away) standalone program is below.
I spent quite a bit of time reading the analyses of transfers while reading more of the material from the USB overview above, but in retrospect I can make the process much more concise. There were really only 2 specific sets of operations I needed to do: capture the stream of USB packets when starting the software up and closing it, and then starting the capture with the software already up and clicking start and stop before stopping the capture.
These two interactions would have been enough to show me the process, which involve analyzing the control transfers and the data transfers. Note: for each single request there is an issue and a response. In the captures of the control transfers, the issue shows as “CLASS_INTERFACE” in the request column and the response is “Control Transfer”, but when using my nodejs script and a WinUSB driver they both say Control Transfer. Also, I had as many as 4 copies of the same request for each set of drivers installed (1 kernel, and 2 or 3 user for the different USB analysis tools installed), so that is why you see back to back duplicates.
There are 3 different control transfers involved, and I still do not know exactly what all 3 are for, but I know enough to reproduce them in the correct order to operate the hardware. Each control transfer had 2 useful pieces: the data and the report fields. Each of the reports had the same format, just with different data. The report is below, and it came important later in the node software. I can honestly say the only reason I noticed it was because the fields were the same as one of the node-usb API function parameters, not from some deep understanding of the process.
Startup transfer 1 (ST1) – data: 0x0182770100000000
This was always the first control transfer to take place.
Startup transfer 2 (ST2) – data: 0x0186FF0100000000
This transfer always took place after the first.
Data initiator (ST3) – data: 0x0180330100000000
This transfer always took place in-between data transfers.
Below is a summary of the control transfer turned out to be key in the node portion of the project. In the source code, look at the control transfers and read the API listing controlTransfer() function, the fields match up exactly. Initially I was trying to get the analyzer to print CLASS_INTERFACE for the ‘issue’ request. I never could, but using these values instead of trying to find a constant define for the bmRequestType in the usb module object saved me in the end.
The data transfer shows up above with filters applied to hide the control transfers. There are 3 distinctly different sets of data that come back, all in 8 byte sections (as defined in the endpoint descriptors). The first to be seen has the text “1.4Per1F” in ASCII. The second turns out to be the real data and reads 0x80 0x02 X X “er1F”, where the first byte always says 0x80, second 0x02, the next 2 change with temperature, and the final 4 are the same as the first request. The 3rd unique transfer just says “TEMPer1F”. There is obviously some pattern with each always having the last 4 bytes the same, and the 1.4 probably corresponds to the version as listed in the analyzer for the device descriptor, shown below.
After some detective work that involved taking captures with the software open and comparing the data packets from the capture from the temperature readings (was harder than it sounded like but in retrospect it should have been obvious), but it turned out that byte 2 (1 based) of the one changing reading was the whole number of the temperature (in hex, had to just convert). Byte 3 I knew was the decimal portion, but it took a little while to realize that to convert was to convert to base 10 and divide by 256 (which makes sense in retrospect).
Clearly there was a lot of trial and error involved here. Actually the first ever data I got out of the thing was the “TEMPer1F” in node (which was unexpected at the time and I was totally stoked), because I only sent 1 control transfer or the wrong order or something, but there were few enough permutations of control transfers and input to figure it out. I’ve identified the following pattern that I missed until I finally got it working by trial and error:
Beginning of transfers
3. [ST1 and ST2 again in the case of opening up the software while capturing]
1. Data Transfer
1. [ST1 and ST2 in the case of clicking STOP in the software]
Node and USB
There is plenty of info out there on nodejs so just google it if you are unfamiliar. Coming into this project I knew that there would be a number of ways to access USB devices, and that they would probably be different between operating systems. I expected that it would mostly be done through C or C++, and since node was (for me) essential to the process, I looked into how to interact with C/C++ programs from node. Luckily, that is very doable. Being a total masochist, I half wanted to write an addon myself (and I might still down the road), but luckily for me others had already gotten there. There were initially several options, and I tried node-hid but the windows installation sucked and I could never get the addon to build correctly. The next choice was one simply called ‘usb‘, which turned out to work. In retrospect, the directions given for windows worked like a charm but it took me a while to get it right. For the most part you could just follow those, but I will point out a couple things that I had to do.
Zadig and libusb
The windows directions point to libusbx.org, but I went in circles for a while until I found libusb-win32 on SF (scroll down to download for the SF download link). Make sure to find the exe to make sure the dependent DLL’s get installed (other win32 ports of linux libraries).
Zadig is used to override the default windows drivers with the generic WinUSB driver that works with libusbx that is used by the usb nodejs addon. I had to run Zadig in compatibility mode for Windows XP (SP3) to be able to click Options>List All Devices (necessary to see the sensor). To identify your device (and not accidentally overwrite an important device with a WinUSB driver), use whatever analysis tool you settle on (like USBLyzer) to figure out the VIN/PIN of the device by trial and error and plugging/unplugging until you can identify which port you are plugging into, and then inspecting the device descriptor. The USB sensor has 2 interfaces, and I installed the WinUSB driver over both separately but since I only read from interface 1 that might not have been necessary.
When you do this, you will no longer be able to use the built in software, so hopefully you will have saved enough analysis data (that can be opened in USBLyzer or similar whenever), but if you need to go back it isn’t hard. Open the device manager and look under Universal Serial Bus Controllers for you device by trial and error and/or using the PORT# shown in your analyzer. All I had to do to get the stock windows HID driver installed again was to open the USB device in the device manager, click the driver tab and click uninstall. Pull the bad-boy out and plug it back in and then check in Zadig to see if it has changed back to HidUsb or whatever it was before.
At this point, installing the usb module and doing a quick test by finding my device using the vin/pin was pretty easy. I didn’t do this all in the order I am laying it out here, but if I were to re-do it this would be the way. I wasted a lot of time not knowing wtf I was doing. I will highly recommend one other module: node-inspector. If you like the chrome dev tools it is worth its weight in gold. It is a HUGE improvement on the gdb flashback-inducing built in node debugger. Now it is just a matter of finding a way to reproduce the process identified in your analysis within node.
I’m not going to try to go through my source code in depth, its reasonably commented. I will say that I did a fair amount of testing to make sure that there isn’t un-needed code in there (for instance, it is necessary to open the device, and the interface must be claimed before you can access the endpoints, etc).
So, take a look at the source code and there you have it. From a basic working knowledge of USB you will know enough terminology to know what you are doing (probably not necessary, but don’t just hack shit together like a noob, you may have to do it again some day), you can understand enough of whats going on in the analyzer to convert that into high-level nodejs code that can access your device. In my case, this adds a ton of value to my sensor because at this point anything can be done with the data. I can use socketio to stream it to other clients/servers, I can send it to my MQTT server, I can put it in database and make a custom graphing page in angular with the data. Before all I had was a janky standalone app, neat huh?
Source code and analysis files on github: https://github.com/thingsnsuch/Node-USB-Temp-Sensor