About

There is no text here

Tag cloud

(all)

Archives

01 Jul - 31 Jul 2006
01 Aug - 31 Aug 2006
01 Sep - 30 Sep 2006
01 Oct - 31 Oct 2006
01 Nov - 30 Nov 2006
01 Dec - 31 Dec 2006
01 Jan - 31 Jan 2007
01 Feb - 28 Feb 2007
01 Mar - 31 Mar 2007
01 Apr - 30 Apr 2007
01 May - 31 May 2007
01 Jun - 30 Jun 2007
01 Jul - 31 Jul 2007
01 Aug - 31 Aug 2007
01 Oct - 31 Oct 2007
01 Nov - 30 Nov 2007
01 Dec - 31 Dec 2007
01 Jan - 31 Jan 2008
01 Feb - 28 Feb 2008
01 Mar - 31 Mar 2008
01 Aug - 31 Aug 2009
01 Sep - 30 Sep 2009
01 Jan - 31 Jan 2010
01 May - 31 May 2010
01 Jun - 30 Jun 2010
01 Jul - 31 Jul 2010
01 Aug - 31 Aug 2010
01 Sep - 30 Sep 2010
01 Oct - 31 Oct 2010
01 Nov - 30 Nov 2010
01 Sep - 30 Sep 2011

Links

Search!

Last Comments

Loste (DJ-X11 hacking an…): Thanks for this!
coolgrey02 (Fun with FAT32): Once you discover quite a…
coolgrey02 (Fun with FAT32): You can never not work ri…
Tony-IZ0KVZ (DJ-X11 hacking an…): I’ve a standard serial (R…
oakley sunglasses… (GTX470 vs GTX275:…): Oakley Sunglasses start w…
Fake Oakleys Free… (GTX470 vs GTX275:…): Fake Oakleys Sunglasses …
Sneakers Isabel M… (Mumble and key re…): Welcome to our Sneakers I…
Ralph Lauren Outl… (Shortest lived pr…): We are committed Be Games…
hermes handbags (Shortest lived pr…): The Company has Hermes Ca…
Ralph Lauren Outl… (Parkzone Radian): When Moda Operandi and He…

Stuff

Powered by Pivot - 1.40.4: 'Dreadwind' 
XML: RSS Feed 
XML: Atom Feed 

DJ-X11 hacking and review

Thursday 22 September 2011 at 11:35 pm Recently I picked up a new toy: the Alinco DJ-X11. I intended to buy one a while ago but ran into difficulties obtaining one as the part of Japan that manufactures it was at the time underwater...


The scanner itself is very nice: here's a simple overview


  • 50khz to 1309.995MHZ
  • Second receiver that does 118-171Mhz and 336 to 470mhz
  • Inbuilt Lithium Ion battery
  • AM, FM, WFM, USB, LSB and CW receive modes
  • 20ch/sec memory scan, 100ch/sec sequential scan
  • Frequency counter / tune (tune to nearby strong signals)
  • Discriminator tap and IQ output with no hardware modding



    The last one's a bigger deal than it sounds. The discriminator output is the raw output of the signal before the amplifier, squelch or filters - vital for receiving any digital mode at any greater than about 1200 baud reliably. The IQ out (in phase / quadrature) encoded signal output means you can do SDR (where you process and demodulate the signal in software allowing you to do all sorts of fun things). The bandwidth of the output signal seemed to be about 80khz when it was pulled into a computer with 96khz sound in. I'll try and find one with 192k capability and see what I get, and a friend of mine is very interested and has access to some interesting and expensive gear, so you'll inevitably find the full specs and exactly what it's capable of soon. But as for the normal AF out, it's only on the left channel. It's capable of outputting on the right channel for the IQ out and I think the discriminator out is on the right as well. Come on, one more switch and you'd get right what every single other scanner gets wrong. Oh well, I'll just have to make an adaptor.


    The sensitivity on air band seems good - although the squelch could do with a bit of adjustment. You could open the squelch and reasonably clearly hear a signal that was rejected at "1".


    The user manual is reasonably good but it would be nicer if it wasn't half warnings and disclaimers. You *can* mention a feature without saying "Warning: we don't guarantee this works" and there is even a warning about poking yourself in the eye with the antenna when you have the scanner in your shirt pocket. There's also a warning saying not to put the plug pack into a power board with other devices - only a wall socket with nothing else. Why? Does it somehow draw 2400 watts? That, and *everything* voids the warranty - any "third party accessory". So I guess if you want to plug headphones in I hope nothing breaks... or you'd better be using Alinco's headphones. There's also two separate warnings not to open the device because you may get an "electric shock". From a device that runs off 4.8 volts and doesn't even have RF transmission hardware? Oh, and don't clean the unit with Benzene. It's forbidden (not to mention carcinogenic).


    Other features? The E version which I have apparently includes a voice inversion scramble decoder . All very great but besides it having one, it's not mentioned at all in the manual how or where to use it. Edit: found it: hit Fn then 5 for the CTCSS/DCS menu, then keep pressing 5 until it says SCR, turn the knob to select the inversion frequency between 2800 and 4200 in 50hz steps


    Overall, the UI makes reasonable sense. The only thing you might want to do is swap the main and sub band knobs in Set Mode as the "main" one is by default next to the antenna - and harder to get at than the "sub" one.


    Other tip: the sub band is actually capable of tuning down to 225mhz by entering a key sequence shown on page 33 of the manual. But you're on your own as to how well this works. What would be nice is the ability to do a similar thing to squeeze a few more Mhz out of the top of the sub band - 480 would be nice as UHF CB is on 477...


    The frequency counter tune works reasonably well - it found the frequency close to a transmitting radio nearby, only about 10khz off (it hit the start of where you could pick it up on FM).


    Scanning speed is very fast - and on the fastest settings doesn't seem to miss frequencies providing the signal is reasonable.


    Now for the bad points:


    The PC software.


    First, you need to buy a cable. And the cable is $80. There is a miniUSB socket on the back of the charge cradle. But It's not USB. The company I bought the radio from said that "only about one in five buy the cable, so four wouldn't need it and that would make the radio more expensive". I countered with "it costs $80, so they'd try to avoid doing so". Apparently it's normal to enter a hundred frequencies using the dial and keypad - not my choice of ways to spend a day, but some people are into strange things.


    There are two cables you can get: the ERW-7 and ERW-8. The ERW-7 plugs into the headphone socket and allows cloning only. I decided to look on eBay for knockoffs of both. According to the eBay seller I got my IC-R5 cable from, it works with the DJ-X11 too as an ERW-7. Except it doesn't. So I got a 3.3v FTDI breakout, connected TX and RX together and hooked it up to a 3.5mm plug. No luck there. Tried it on the R5 with CS-R5 and it worked.


    So I decided to have a go at the ERW-8 instead. Seeing as I couldn't buy one out of Hong Kong, I was going to have to do it myself. Alinco's site disclaimed any responsibility for driver issues or install questions (as they disclaim absolutely everything else, why not?), but provided driver install instructions. FTDI's PDF with a page of extra rubbish on the front. Ahh, so it's a FTDI USB to serial cable. I guess it's TTL serial on a miniUSB connector - that must make sense to someone somewhere... I opened the cradle up, and found where all the pins go. Multimetered the back of the radio and found that two of the contacts go straight to the battery. Guessed the third one was serial, and wrapped wires round the contacts, connecting the negative side of the battery to ground, and the other contact to TX and RX (TX and RX are typically connected to the one line - half duplex serial :D). It worked!


    I'm still not sure if the software doesn't recognise the radio through the headphone jack, because the two protocols are I believe different. Or maybe one of DTR, RTS, CTS etc is tied to ground/vcc in the Alinco cable to signal what it is.


    I decided that the cradle would be more useful with a real miniUSB on it, so I soldered the FTDI breakout in.


    Warning: one user has reported issues with his radio after using KG-SDR. The radio may not include protection circuitry inside it on the serial port. You probably want to wire a 10k resistor into the white wire below.





    That FTDI chip would cost under $5 in the quantities you would be buying them in (the breakout is $15 from Sparkfun, still a heap cheaper than the Alinco cable), would easily fit on that board and would be a ton more useful. There'd even be room to get it into the radio itself and provide a miniUSB socket on it. For what is a simple USB to serial adaptor, that is not OK. It should have been in the box with the FTDI chip in the cradle, not a $80 accessory. Rule #1: don't annoy people who can reverse engineer your product.


    I had to cut some plastic out of the bottom piece to make it all fit together though, so it's a little ugly on the back, but it works.


    With that out of the way, onto the PC software:








    The programming software is an utter joke. It's written in VB6, and I thought CS-R5 was bad before I used this. It's so much worse. The software assumes that you are running on Japanese Windows (and Alinco disclaims any issues that happen if you're not), and no, that table view is not directly editable. The UI is a total mess - a fixed height nightmare! I'm told that it's because I don't have the right version of the Tahoma font installed - installing fonts to make a program work?



    Oh, it does NOT end there.





    Looks like some eight year old's "My First Radio Scanner Controller". Overflowing text boxes, big ugly fonts, orange vs green colour scheme, improperly labelled buttons, badly aligned text, badly aligned UI elements, overflowing text... Oh, and if anything goes wrong, don't expect to be told why: all you get is "error". And finally, make sure you select which model you have in the not very well documented radio buttons up the top - if you don't have the X11E, you're just going to be told "Error" otherwise.


    It's even worse when you actually use it. I thought the Japanese were obsessed with quality and making things nice to use? I'm pretty sure even Feidaxian's programming software isn't this bad. Reading from the radio is slow when you want to read the whole lot of frequencies in (you thought reading in a R5 was slow when you didn't select the "high speed" option for serial?). It requests each memory channel from the radio sequentially. Very bad.


    Bank selection is even worse. Banks are in order in the list, and you say "the first 12 are in bank A, the next 6 are in bank B, the next 6 are in bank C". But if your bank allocations don't add up to 1200, it won't let you program (apparently just putting the rest in the last bank automatically is not OK). Oh, and if you want to add another frequency to bank A, you'll have to manually reprogram everything below it down a slot if you don't export as CSV and figure it out that way. Very very tedious.


    Expect a reverse-engineering of the software and possibly the ERW-8 cable to come soon. It'd be easier for me to write my own software than use that.
  • Projects

    Tuesday 09 November 2010 at 3:30 pm Stuff I still have to do:

    Networking

    • Rebuild home network, get it VPN'd to other sites
    • Move the modem to the front of the house, cable in ethernet.


    Hardware

    • Fix the ventilation issue in the server rack
    • Build the cluster of P4s
    • Get that MDD G4 working again
    • GPS enable the Radian
    • Convert the 35W HID to Ni-MH packs
    • Build a 12v car power / battery power transfer box


    Programming

    • Rebuild home network, get it VPN'd to other sites
    • Cross-site LDAP
    • Move the modem to the front of the house, cable in ethernet.
    • Rebuild stackunderflow.com and gm.stackunderflow.com websites
    • Rebuild Stack servers - perhaps it's time to retire the G3... or maybe it's the start of it's career doing something else :)
    • Get MCProxy to a release-able state
    • Get the Nearmap -> GEarth app working
    • Write a decent Kismet GUI in Python
    • Get the Realtek support on OS X Kismet off the ground
    • Learn mod_wsgi and Django
    • Finish other not-so-public things


    Misc

    • Do more LANning
    • Get WebCharts off the ground
    • Amateur Radio license
    • Add all the items to this list that I have inevitably forgotten

    The Code

    Tuesday 12 October 2010 at 9:12 pm I promised to upload the code ages ago. Here it is:
    http://svn.stackunderflow.com/svn/public/filerecover/

    I have it.

    Wednesday 01 September 2010 at 11:12 am First part
    Second part

    This is an ongoing saga, and if you don't know what this is part 3 of, the previous two parts are linked above.

    Right, so we know where the file starts, we know where it ends, and we know that the file should be pretty much linear as the card was empty. Screw the FAT, let's just read this thing from the clusters.

    #!/usr/bin/env python
    import sys
    numReservedSectors = 38
    bytesPerSector = 512
    sectorsPerCluster = 8
    numFATs = 2
    fat1length = 7771648

    def clusterToBytes(cluster):
        firstDataSector = (numReservedSectors * bytesPerSector) + (numFATs * fat1length)
        address = (((cluster - 2) * sectorsPerCluster) * bytesPerSector) + firstDataSector
        return address

    print clusterToBytes(0x0005CDF3)



    Bring in some values. Define a function to convert clusters to bytes.
    From the first bit, we know where it starts.


    File found: 4621 clusters (18 MB), start 0005CDF3



    So run the program and we get:
    1573685248 which is 0x5DCC8800

    From the second bit, we know that it is supposed to end at 0x05EED5800. Then we found that the data really ends at 0x0614B3600.


    start = 0x05DCC8800
    middle = 0x05EED5800
    end = 0x0614B3600

    size = end - start

    f = open(sys.argv[1],'r')
    f.seek(start)
    a = f.read(size)
    f.close()

    f = open("recovered.avi",'w')
    f.write(a)
    f.close()



    Running this results in many more frames and a bigger audio track.

    Here's the last frame:



    and here's the video:

    Recovered Crash Video final version from gm on Vimeo.

    Digging deeper

    Tuesday 31 August 2010 at 11:33 pm This is a continuation of yesterday's post "Fun with FAT32, and part 2 in the process of getting that video back.

    OK, so we have two promising looking files that don't play. Let's dive into the AVI file format.

    Microsoft have a published spec for it here.

    An AVI file is really a RIFF file. The basic structure of a RIFF file is a 4 byte identifier, followed by a 32-bit size. Then a length field, then a pad byte if the length isn't even. You then have a header with fields showing info like what codec etc etc. I'm not going to go into this in detail, because it's not actually important here (trust me). The one thing that is is that "DWORD", supposedly meaning "double word", a word being 32 bits, is not 64 bits. Microsoft compilers treat WORD and DWORD types as the same, or so I'm told. I'm just going to say "32 bit value" or "unsigned int", but even then, on some 64 bit platforms, ints are 64 bit, so be careful.

    The code will be available in the SVN. Also excuse the hardcodedness of a lot of things in this program - AVI files from the same camera don't tend to differ. Fat32 volumes formatted in many different things do.

    So I made a dodgy program to dump data from the headers and it would appear that our two mystery files are AVI files, but there's some stuff missing. The header stuff is in exactly the same position - my program just assumes this as I wasn't going to write a complete and proper RIFF implementation. So it likely won't work on AVI files not from this camera.



    Playable one on the left, unplayable one on the right.

    0x0, 0fps, 0 frames long. While the resolution is certainly known from the start, the number of frames quite reasonably isn't.

    I attempted to drop reasonable values into the file, but to no avail - it still didn't play. In MPlayer, Quicktime or VLC. VLC offered to repair the AVI index, and promptly crashed upon doing so. Oh well.

    The reason why previously I said that it didn't matter about the header is that I am just going to rip the data out of the AVI and deal with it elsewhere.

    import sys
    import block
    import struct
    import re

    f = open(sys.argv[1],'r')
    file = f.read()


    Yes we do read the file in. What do you mean you don't have 500MB of RAM?

    Now some calculations:
    The RIFF chunk with the actual data in it is named "movi".

    movicount = file.count("movi")
    print "movi block count: %i" % movicount
    moviloc = file.find("movi")
    print "movi block location: %i" % moviloc



    There should only be one instance of "movi" in the file. It's an outstanding coincidence if this happens in the video data. Or it's annoying if it happens in the header. We'll just hope it doesn't.


    print "regexing..."
    avidata = file[moviloc+4:]
    avichunks = re.split("(00dc|01wb)",avidata)[1:]

    chunktype = avichunks[0::2]
    chunkdata = avichunks[1::2]



    First, we get the entire file from 4 bytes after the "movi" chunk. We ignore it's size field - remember that this is a not properly finished file we are dealing with, and fields such as these may not be present yet.
    Now for some regex. We need to use re.split because the string split method doesn't support splitting on multiple things, and loses what it was split by.
    The way AVI works is that it interleaves audio and video data. This means that reading and writing can be done in one stream, greatly simplifying the recording and playback process. There are no seperate reads or seek operations to two seperate parts of the file (in the days AVI was new, we played fancy multimedia CD-ROMs. Seeking on CD-ROMs is horribly slow, but linear reads weren't so bad.). We have RIFF chunks with types indicating what they are.

    The first two digits identify the track number. dc means compressed video, and wb means audio data. There are four such codes - read the earlier linked spec to find out about them.

    You can have an overcomplicated regex here like "(\d\d)(dc|wb)(.*)" - I did originally, but it was actually matching movie data. There were no false matches with this one, and no tracks we are interested in besides the video and audio tracks.

    Anyway, this results in a list like so:
    ["","00dc","block of video data","00dc","more video data","01wb","audio data"]
    and so on.

    Next, if you don't know, in Python, array indexes can have up to 3 values.

    a[1] = item 1 of a (second item, arrays are indexed from 0)
    a[:] = all of A (same as "a")
    a[1:] = from item 1 onwards
    a[1:3] = from 1 to 3
    a[-1] = last value

    The third number is "every n".

    a[0::2] is every second value starting at 0, or every odd value.
    a[1::2] is every even value.

    a[::-1] is the array in reverse order.

    Anyway, we drop the first value of the array because it's an empty string.
    chunktype is the first element of every pair, chunkdata is the second.

    Now let's do something with them:


    streams = {}
    streams["00dc"] = []
    streams["01wb"] = []

    for i in xrange(len(chunktype)):
        stype = chunktype[i]
        sdata = chunkdata[i]
        streams[stype] += [sdata]



    We initialize the dictionary "streams" with two lists. Then we go through all the chunks, and place them at the end of the list they belong to.

    Now let's do something with this dictionary.


    for stream in streams.keys():
        f = open("streams/%s" % stream,'w')
        print "writing streams/%s" % stream
        for block in streams[stream]:
            f.write(block)
        f.close()



    Right, let's do this.



    Promising.

    Now we know that the video is motion JPEG, and that the audio is uncompressed. Let's look at one of the known good movies:



    I'll leave the video for later. Let's deal with the audio first. I'll use Audacity for this.



    And what do we have?


    It sounds like the radian... but what is that fast clicking in the background?
    There seems to be some extra bytes in here. Ahh... we forgot about the size bytes after all the audio blocks!

    Change the line where the blocks are written to this, and run again.


    f.write(block[4:])



    And no clicking. I can hear the Radian's motor running flat out and producing some very distorted sound, followed by it shutting off, and silence. Then it being run shortly... shouldn't there be some wind in here? I skip to the end, expecting the recording to end with a loud bang... it ends with me saying something after the motor winding down. This isn't the video - but why isn't it in the filesystem? I remember pulling the power too soon on a test run of my camera. That must be it. It must be the unidentified 18MB file, not the 70MB one. Or it could be neither of them. Hmm.

    Let's run it on the other file.

    It only produces 28 seconds of sound.


    The first few spikes are the canopy being put on, followed by the motor starting at 9 seconds in, and the throttle being maxed out at 18 seconds. The video ends 8 seconds later, still at full throttle. This doesn't sound like a test. I'm not saying anything. But I can't hear wind noise in it. And I could have sworn it took longer than 18 seconds from video start to crash the plane. I check another video - and it seems that at full throttle, there isn't wind noise - it's not audible above the motor.

    I guess there's one way to find out - let's mess with the video. VLC won't touch the 00dc file. MPlayer doesn't like it. Quicktime... wait what? A video!

    No. A frame. Damnit.



    But it's the video. I'm turning the camera on from the transmitter. I'm standing where I was standing when I launched the plane for it's disasterous flight. This is the video.

    Now let's take a look at the data.



    FF D8? That sounds oddly familiar. A quick Google search for "jpeg magic bytes" finds that it is the marker for the start of a JPEG file. FF D9 is supposed to be the end. I search for it. Not found. But FF D8 is in this file... a lot.

    A search for Motion JPEG format finds this page from the Planning for the Library of Congress Collections. It doesn't say much about the video format itself, but it does say:

    Avery Lee, writing in the rec.video.desktop newsgroup in 2001, commented that "MJPEG, or at least the MJPEG in AVIs having the MJPG fourcc, is restricted JPEG with a fixed -- and *omitted* -- Huffman table. The JPEG must be YCbCr colorspace, it must be 4:2:2, and it must use basic Huffman encoding, not arithmetic or progressive. . . . You can indeed extract the MJPEG frames and decode them with a regular JPEG decoder, but you have to prepend the DHT segment to them, or else the decoder won't have any idea how to decompress the data. The exact table necessary is given in the OpenDML spec."



    Interesting.

    I take a rough guess that I should split all the JPEGs out at FF D8.


    #!/usr/bin/env python
    import sys
    import block
    import struct
    import re

    f = open(sys.argv[1],'r')
    file = f.read()
    file = file.split("\xFF\xD8") # JPEG magic bytes

    index = 0
    for frame in file:
        f = open("frames/%04i.jpg" % index, 'w')
        f.write("\xFF\xD8" + frame)
        f.close()
        index += 1



    Not too complicated.



    Hmm, they seem to have icons. It's obvious why. Finder must be using the standard Apple libraries which are probably descended off Quicktime. Which when Quicktime notices that the JPEG has no DHT segment, uses the one from it's Motion JPEG decoder. Brilliant. Except that all these pictures can be displayed in Safari. Wait, that's an Apple app. Chromium? Still works. Imagemagick even displays it!

    I look at the Wikipedia article for JPEG. FF C4 is the marker for a Huffman table. And there's one there. I know what's going on here. The camera's chipset is feeding it JPEGs, and they're just bit-stuffing them into an AVI, at a rate of 30 of them every second. I guess none of the MJPEG decoders are picky.

    So, after deleting 0000.jpg (it's the bit of data from the start), I import the image sequence into QuickTime Pro at 30fps. It looks a little bit fast. I'm not sure why at this point, but I combine the audio and it doesn't match up. So I bring both into Final Cut Pro. I note the duration of the audio track, and slow the video track down to match the length. The video matches the audio perfectly. And I see for the first time, the video. However, it runs out of frames in the air, pointed upwards.



    Recovered crash video attempt 1 from gm on Vimeo.



    Did the camera not finish writing to the card? I remember from the firmware upgrade that it has 8MB of DRAM in it. It's probably not storing anywhere near 8MB of data in the RAM, most likely a few JPEG frames until it has enough to write a block into the AVI. Besides, 8MB would only be a few seconds. It felt like longer from that point to where the plane came down.

    I remember though, from the first step.


    Traversing files...
    Error: nextptr 00004000 is unallocated
    Error: nextptr 0005E000 is unallocated



    I wonder if it's writing the FAT one sector at a time. This would mean that the sectors are not actually marked as used, but that there is data there - after all it had to go somewhere. This was the last video of the day, and the card was not mounted prior to imaging, so if there is - it hasn't been overwritten. And since it's FAT, the allocation should be reasonably linear. Time to go back to stage 1 I guess. It will be obvious if the zeros are on a 512 byte boundary.

    Convert endianness, and 00E00500 is present in the FAT.



    It is. The next line is 0x00017CC00 = 1559552. 1559552 / 512 = 3046 exactly. That's a block boundary alright.
    (You are a hardcore geek if you know 512 = 0x200 and recognised it was a multiple from looking at the hex offset value).

    0x0005E000 = 385024.
    Remembering from the program, the address of a cluster is:
    firstDataSector = (numReservedSectors * bytesPerSector) + (numFATs * fat1length)
    address = (((cluster - 2) * sectorsPerCluster) * bytesPerSector) + firstDataSector

    So this gives us:
    ((385024 - 2) * 8 * 512) + (38 * 512) + (2 * 7771648)
    1577050112 + 19456 + 15543296
    = 1592612864
    = 0x5EED5800



    There's more data here, and this data is identical to the end of 33.avi.

    Not much further down, there's a 00dc (AVI image data block), 4 size bytes, and a FF D8 (JPEG start of image marker if you've forgotten)!



    The first lot of all zeros is at 0x0614B3600 = 1632318976

    1632318976 (what we thought was the end) - 1592612864 (where the 0 blocks start) = 39706112 (of extra data).
    39706112 / 1048576 = 37MB more, hopefully of the same file!

    Now that we know where it is, we can dig it up, and hope it's in order, because there's no FAT linked list to tell us that it is.

    Anyway, I'll continue this later, and hopefully find some results.

    Fun with FAT32

    Monday 30 August 2010 at 8:24 pm What's the worst part of crashing a RC glider? No, not the fact that I'll have to break out the epoxy... again.

    TLDR of the situation is that when the glider hit the orange fencing, it went through one of the holes up to the wings. The brushless motor and front section, being heavy, decided that they'd rather keep going forward, and so tore the front bit off. The battery decided it'd also like to come along for the ride... and so power was lost to the onboard camera.

    Which was recording video at the time. I'd like to see that video, and I know it's somewhere on the card. It has to be somewhere on the card. According to information that is part of the firmware upgrade, it has 8MB of DRAM, so it's definitely not staying in there. And it stops recording quickly.

    But there is no filesystem entry for the file. TestDisk doesn't see it. PhotoRec for some reason doesn't even see all the videos that *were* on the card normally - I guess it's designed for photos.

    So looks like I'm going to have to work out what's going on here, and write my own app to get the data back. Fat32 isn't that complicated. I imaged the card without mounting it, so I have a completely untouched copy of the card.

    This page is quite handy. So is this one. I'm going to assume you have at least some grasp of Python here.

    Let's dive right in:

    #!/usr/bin/env python
    import sys
    import block

    f = open(sys.argv[1],'r')
    volumeid = f.read(512)



    The "block" library I am using here is this one. It's not absolutely necessary but it does make things easier here.

    Now for some definitions:

    def getUInt8(data,offset):
        return block.Readview('B',block.Block(data[offset:offset+1]))[0]

    def getUInt16(data,offset):
        return block.Readview('H',block.Block(data[offset:offset+2]))[0]

    def getUInt32(data,offset):
        return block.Readview('I',block.Block(data[offset:offset+4]))[0]

    def syswrite(text):
        sys.stdout.write(text)
        sys.stdout.flush()



    Yes, we can do this without block, but it's easier this way. On a side note - I'm not sure what happens on a big-endian system.

    Now let's pull some values out of the first sector. Note that this is the first sector of the partition, not the first sector of the MBR. I imaged the partition, not the card. So we don't have to worry about the MBR or a partition table.

    OEMIdent = volumeid[0x03:0x0A]
    bytesPerSector = getUInt16(volumeid,0x0B)
    sectorsPerCluster = getUInt8(volumeid,0x0D)
    numReservedSectors = getUInt16(volumeid,0x0E)
    numFATs = getUInt8(volumeid,0x10)
    numDirEnt = getUInt16(volumeid,0x11)
    totalSectors16 = getUInt16(volumeid,0x13)
    totalSectors32 = getUInt32(volumeid,0x20)
    sectorsPerFAT = getUInt32(volumeid,0x24)
    rootDirFirstCluster = getUInt32(volumeid,0x2C)
    signature = getUInt16(volumeid,0x1FE)



    OEMIdent is a text string that identifies what was used to format the disk. In this case it is "mkdosfs".
    BytesPerSector is how many bytes/sector the disk media is. This is 512 on nearly every media - even 4k sector HDDs.
    SectorsPerCluster is how many sectors in a cluster. Fat32 doesn't address sectors individually, it addresses "clusters" of sectors. In our case it is 8, resulting in 4KiB clusters.
    numReservedSectors is essentially how many sectors into the disk the real fun starts. In our case it is 38. This is from the start of the disk and includes the first sector we are reading from now.
    numFATs is the number of FATs on the disk. I'll get to exactly what a FAT is later, but there are often (read: nearly always two) sequential copies of one on the disk to guard against filesystem corruption from a dodgy disk. In our case it is two.
    numDirEnt is the number of root directory entries.
    totalSectors16 is the 16-bit count of sectors. Only problem is, though, that like 640k of RAM supposedly being enough for anyone, a 16-bit value has a maximum of 65535, allowing a maximum of 32MB of sectors. Not quite enough for modern forms of storage. This is zero here. On a side note, I wonder if the people who originally wrote this thing are surprised that it's still in use today?
    TotalSectors32 is in the extended information area, an area previously unused by Fat12 and 16. This is a full 32-bit count of sectors, enough for 2TB of drive with 8k clusters, a limit that we have finally exceeded in a single drive. However, most people with drives this large are using ext3/4, HFS+ or NTFS on their HDDs by this stage.
    sectorsPerFAT is the size of one of the FATs in the filesystem, in sectors.
    rootDirFirstCluster is the where the root directory entry is on the disk. This is relative to the end of the FAT.
    Finally, we have a signature which should be 0xAA55.

    Now here's the fun stuff.
    Fat32 is really Fat28. Only the lower 28 bits of the address are actually used. Go figure.
    What's the address of the first cluster? 0? 1? No, it's 2. Clusters 0 and 1 don't exist. Why? You'll see later.

    So let's find the FAT:

    fat1addr = numReservedSectors*bytesPerSector
    fat1length = sectorsPerFAT*bytesPerSector

    f.seek(fat1addr)
    fat1 = f.read(fat1length)



    So what is the FAT? If the filesystem is named after it, it must be important right?
    The FAT is an array of 12, 16 or 32 bit values, one per cluster. For Fat32:

    • 0x00000000 means a free block.

    • 0x00000001 is "reserved" (read: they haven't thought of something dodgy to do with it yet).

    • 0x00000002 to 0x0FFFFFEF point to the next block in the file

    • 0x0FFFFFF0 to 0x0FFFFFF6 are reserved

    • 0x0FFFFFF7 means not to use this cluster as it has a bad sector in it

    • 0x0FFFFFF8 to 0x0FFFFFFF mean that this is the last sector in the file, or the only sector



    So essentially the FAT is a big pool of linked lists. And all kinds of fun stuff can happen, like:

    • Pointing to a free block - not too bad, reference is usually overwritten anyway

    • Two files pointing to the same block - cross-linked files - very bad. If this happens accidentally, the end of one file is lost, and replaced with the end of another. In addition, writing to the end of one will overwrite the other.

    • Pointing to an invalid block - likely will produce an error



    So let's read in the FAT.


    fatptr = 0
    usedclusters = {}

    print "Finding non-zero clusters"
    while (fatptr < fat1length/4): # 256 now
        nextlink = getUInt32(fat1,fatptr*4)
        if (nextlink > 0): # if this block does something
            usedclusters[fatptr] = nextlink
        fatptr += 1
        if ((fatptr % 1024) == 0):
            syswrite(".")

    print "\nFound %i used clusters = %iMB data\nTraversing files..." % (len(usedclusters),(len(usedclusters)*sectorsPerCluster*bytesPerSector)/1048576)



    We save all the non-zero FAT entries into a dictionary, keyed by their position. The value is the next link.
    Running this over my card image reveals that there is a different number of used sectors to the "used on disk" measurement... something's in the FAT but not linked to by the directory. Suspicious.

    Now let's do something with the data...

    files = []

    while True:
        keys = usedclusters.keys()
        if (len(keys) > 0):
            nextptr = usedclusters.keys()[0];
        else:
            break
        newfile = []
        while True:
            thisptr = nextptr
            if not (usedclusters.has_key(nextptr)):
                print "Error: nextptr %.8X is unallocated" % nextptr
            break
        nextptr = usedclusters[nextptr]
        del usedclusters[thisptr]
        newfile.append(thisptr)
        if (nextptr >= 0x0FFFFFF8):
            break
        files.append(newfile)

    print "%i files found" % len(files)

    for i in files:
        print "File found: %i clusters (%i MB), start %.8X" % (len(i),len(i)*sectorsPerCluster*bytesPerSector/1048576,i[0])



    Basically we pick the first item out of our set of keys. We are assuming the start of the file is before any part of it, but this may not be true. Then we retrieve the next sector from the FAT - essentially traversing the linked list until either the end, or until it links to a free sector. The second should never happen - but the camera did get it's power cut in the middle of recording.



    Traversing files...
    Error: nextptr 00004000 is unallocated
    Error: nextptr 0005E000 is unallocated
    34 files found
    File found: 1 clusters (0 MB), start 00000000
    File found: 1 clusters (0 MB), start 00000001
    File found: 1 clusters (0 MB), start 00000002
    File found: 1 clusters (0 MB), start 00000003
    File found: 1 clusters (0 MB), start 00000004
    File found: 1 clusters (0 MB), start 00000005
    File found: 1 clusters (0 MB), start 00000006
    File found: 1 clusters (0 MB), start 00000007
    File found: 1 clusters (0 MB), start 00000008
    File found: 1 clusters (0 MB), start 00000009
    File found: 1 clusters (0 MB), start 0000000A
    File found: 16546 clusters (64 MB), start 0000000B
    File found: 1 clusters (0 MB), start 00000108
    File found: 1 clusters (0 MB), start 000001DD
    File found: 1 clusters (0 MB), start 00000200
    File found: 1 clusters (0 MB), start 00000201
    File found: 1 clusters (0 MB), start 00000211
    File found: 1 clusters (0 MB), start 000002CB
    File found: 1 clusters (0 MB), start 0000034B
    File found: 1 clusters (0 MB), start 000003B5
    File found: 1 clusters (0 MB), start 000004B4
    File found: 1 clusters (0 MB), start 000004DC
    File found: 1 clusters (0 MB), start 0000060F
    File found: 1 clusters (0 MB), start 000013BF
    File found: 1 clusters (0 MB), start 0000175A
    File found: 1 clusters (0 MB), start 0000175B
    File found: 1 clusters (0 MB), start 0000175C
    File found: 20724 clusters (80 MB), start 000040BC
    File found: 1 clusters (0 MB), start 000053F3
    File found: 126452 clusters (493 MB), start 000091B1
    File found: 28972 clusters (113 MB), start 00027FA5
    File found: 97103 clusters (379 MB), start 0002F0D1
    File found: 90579 clusters (353 MB), start 00046C20
    File found: 4621 clusters (18 MB), start 0005CDF3



    The 1 cluster files are highly likely to be directory entries. The rest look promising - their sizes almost exactly match the videos that I can see on the mounted disk image of the card. However, there's a 18MB one, and an 64MB one that are unaccounted for. I think these are our files!

    Time for some more code:


    firstDataSector = (numReservedSectors * bytesPerSector) + (numFATs * fat1length)
    print "firstDataSector %i 0x%.8X" % (firstDataSector,firstDataSector)

    def getCluster(cluster):
        address = (((cluster - 2) * sectorsPerCluster) * bytesPerSector) + firstDataSector
        f.seek(address)
        return f.read(bytesPerSector * sectorsPerCluster)



    Work out where the data area starts: after the reserved section and FATs. Then work out how to read a cluster. Find it's address and length. Remember that sectors, bytes and clusters are not the same thing. A cluster is (in our case) 8 sectors. A sector is 512 bytes.

    Now do something with this:


    index = 0
    for file in files:
        g = open("recovered/%i" % index,'w')
        for cluster in file:
            g.write(getCluster(cluster))
        g.close()
        index += 1
        print "written %i" % (index)



    For each file, open a file and dump all the clusters into it. This will result in the files not having the exact same size they did - they will be padded to the nearest cluster. The actual filesize is in the directory - that we don't look at (yet).



    Well, they all play after changing their file extensions to AVI. Except 11 and 33. No go in Quicktime, VLC or mplayer.

    gm$ file *
    11.avi: RIFF (little-endian) data, AVI, 0 x 0, >30 fps, video: Motion JPEG, audio: uncompressed PCM (mono, 24000 Hz)
    27.avi: RIFF (little-endian) data, AVI, 720 x 480, 30.00 fps, video: Motion JPEG, audio: uncompressed PCM (mono, 24000 Hz)
    29.avi: RIFF (little-endian) data, AVI, 720 x 480, 30.00 fps, video: Motion JPEG, audio: uncompressed PCM (mono, 24000 Hz)
    30.avi: RIFF (little-endian) data, AVI, 720 x 480, 30.00 fps, video: Motion JPEG, audio: uncompressed PCM (mono, 24000 Hz)
    31.avi: RIFF (little-endian) data, AVI, 720 x 480, 30.00 fps, video: Motion JPEG, audio: uncompressed PCM (mono, 24000 Hz)
    32.avi: RIFF (little-endian) data, AVI, 720 x 480, 30.00 fps, video: Motion JPEG, audio: uncompressed PCM (mono, 24000 Hz)
    33.avi: RIFF (little-endian) data, AVI, 0 x 0, >30 fps, video: Motion JPEG, audio: uncompressed PCM (mono, 24000 Hz)


    Noticing a pattern here.


    Well that looks like the proper RIFF AVI header. Time to learn how AVIs work I guess... in a future blog entry.

    Parkzone Radian

    Sunday 29 August 2010 at 6:48 pm



    2m wingspan, foam construction, ~300w brushless outrunner motor.

    But of course I can't leave it stock.

    Parkzone Radian Second Flight and First Crash from gm on Vimeo.



    Well you quickly learn how to repair these things. 2-part epoxy, tape and hot glue are your friends here.

    But I'm standing down here flying the thing, I want to see what it's like up there!

    A bit of searching found a suitable camera - the FlyCamOne. It's small and light, and the cam module is separate from the electronics. I cut the camera loose from it's stand, and embedded it in the foam behind the canopy. The camera board is up the front with the speed controller. But the best feature is that it has a 3-pin servo connector on it - it draws power from the plane itself, thereby needing no batteries. As well as this, you can toggle the recording status from the controller - handy if you realise that you forgot to start the camera when the plane is in the air.

    So here's my flight yesterday:

    Radian Flight 2.0 from gm on Vimeo.


    Perfect weather.

    Today's weather wasn't so great... but this just makes the video more interesting for you :)

    Radian Flight 2.5 from gm on Vimeo.

    Mumble and key remapping on OS X

    Thursday 19 August 2010 at 11:30 pm If you haven't seen Mumble, you should probably check it out at http://mumble.sourceforge.net/. It's like TeamSpeak, Ventrilo or that kind of thing, but it's fully open source and completely configurable. There are servers for Linux and OS X (and probably Windows if you compile it). There are clients for Mac, Linux, FreeBSD, Windows, Maemo, Android and iPhone. And it's open source so you can code another one.

    There's also uMurmur, a minimalist Mumble server designed to run on very low resource devices (OpenWRT running routers etc).

    Anyway, the thing is, I want to use push-to-talk on it. But this needs an unused key. Initially I suggested Fn, but Fn-up and Fn-down are PgUp and PgDn (on a Mac laptop keyboard anyway). So every time you use them, you transmit. Then I realised I'd never really pushed the right option key. Perfect. But if I set it to option, it triggers on left option as well.

    Enter DoubleCommand. Install it and turn on the "Right Option acts as Enter" key. Now, go into the Mumble settings, and assign it as the PTT key. Then check the "Inhibit" checkbox - this captures the keystroke so it doesn't get passed onto apps (you don't type enter in the frontmost app).

    The only downside is if you use a keyboard with an enter key, you can't use it. I might write and submit a patch for DoubleCommand to remap right option to F13 (there's actually up to F24 as you can see in keycodes.h).

    But it works, and now I have a PTT trigger that's easy to reach and not used anywhere particularly important.

    The Godzilla Diaries

    Thursday 08 July 2010 at 2:22 pm My Dad and I will be doing a blog when we're over in Japan. We're leaving tomorrow, and you can follow it at: http://godzilladiaries.underflownetworks.com/. That is all.

    GTX470 vs GTX275: some benchmarks...

    Thursday 01 July 2010 at 9:25 pm GTX275 and E8500 at 3.3GHZ:
    3dMark Vantage: 13480 3dMarks, 34158 CPU, 11216 GPU
    3dMark 06: 14839 3DMarks, 6828 SM 2.0, 7545 SM 3.0, 2988 CPU
    FurryMark: 1920x1200 Fullscreen, 60000ms, MSAA off: 4037 points
    X-Plane 9: Extreme res, 3x1680x1050 multihead, 15nm visibility: ~20fps

    GTX470 and E8500 at 3518MHZ:
    3dMark Vantage: 16790 3dmarks, 38769 CPU, 14122 GPU
    3dMark 06: 16268 3dMarks, 6969 SM 2.0, 9103 SM 3.0, 3127 CPU
    FurMark: same as above: 4177 points
    X-Plane 9: ~25FPS, same settings, but I think it's actually achieving 15nm. This is without Nvidia eyefinity... i mean surround.
    Reboot test: All 3 screens work, every time :D

    Only annoyance is that I actually bought the GT220 because the fan in the card it's replacing is annoyingly loud, but MSI Afterburner allows you to adjust it down a bit.

    Linkdump