For the past three weeks or so, I’ve been writing C code again. It’s been both fun and frustrating, since it’s been probably twenty years or so since I’ve done any serious coding in that language. The application is for a startup, and I can’t reveal the specific task I’m doing, but it involves reading data from a video stream.
I could have used any video technology I wanted, but to avoid patent encumberances I went with the Ogg container format with the Theora video CODEC. Luckily, I don’t need to actually decompress the video stream, I just need to be able to manipulate it at the frame/page level. Shouldn’t be too hard, and for the most part it wasn’t. I started by looking at the reference implementations for Ogg and Theora (libogg and libtheora), but they were a little too full-featured for my application, so I started to roll my own.
Things were going pretty well, until I got into the Theora identification header. I’d read the relevant parts of the spec and did some poking around with a hex viewer on my sample Ogg files. Everything looked good, so I created a C structure definition that I thought matched the header:
typedef struct __attribute__ ((__packed__)) _THEORA_ID_HEADER {
THEORA_HEADER HDR; // Common header.
byte VMAJ; // The major version number.
byte VMIN; // The minor version number.
byte VREV; // The version revision number.
uint16 FMBW; // The width of the frame in macro blocks.
uint16 FMBH; // The height of the frame in macro blocks.
uint PICW:24; // The width of the picture region in pixels.
uint PICH:24; // The height of the picture region in pixels.
byte PICX; // The X offset of the picture region in pixels.
byte PICY; // The Y offset of the picture region in pixels.
uint32 FRN; // The frame-rate numerator.
uint32 FRD; // The frame-rate denominator.
uint PARN:24; // The pixel aspect-ratio numerator.
uint PARD:24; // The pixel aspect-ratio denominator.
byte CS; // The color space.
uint NOMBR:24; // The nominal bitrate of the stream, in bits per second.
ushort QUAL:6; // The quality hint.
ushort KFGSHIFT:5; // The amount to shift the key frame number by in the granule position
ushort PF:2; // Pixel format.
ushort RES:3; // Reserved.
} THEORA_ID_HEADER;
The only problem: the values I read from an actual Theora stream didn’t match the header. They almost matched. Except for the one field I actually cared about (of course). I need the KFGSHIFT field to interpret the granule positions of the Ogg stream, and even though it’s apparently hardcoded to 6 in every library out there, my OCD wouldn’t let me get by with that. I wanted the real value from the stream. So, I fired up a hex editor in Atom and started digging.
After counting bytes, and recounting bytes, I found the offending 16 bit value that contains the bitfields in the header: 00 c0. Or, bitwise:
msb lsb 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 +---------------+-------------- 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Which didn’t match the diagram in the Theora spec. Looking at that diagram, I was expecting: 80 01.
msb lsb 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 +---------------+-------------- 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0
WTF?! I expected QUAL, PF, and RES to all be 0. I expected KFGSHIFT to be 6. Why were the two most-significant bits set? What was going on? So I went back to the Theora spec. I read the code for the Theora reference implementation (libtheora). Finally, buried in section 5.2 of the spec I found the explanation of this craziness.
The decoder logically unpacks integers by first reading the MSb of a binary integer from the logical bitstream, followed by the next most significant bit, etc., until the required number of bits have been read. When unpacking the bytes into bits, the decoder begins by reading the MSb of the integer to be read from the most significant unread bit position of the source byte, followed by the next-most significant bit position of the destination integer, and so on up to the requested number of bits. Note that this differs from the Vorbis I codec, which begins decoding with the LSb of the source integer, reading it from the LSb of the source byte. When all the bits of the current source byte are read, decoding continues with the MSb of the next byte. Any unfilled bits in the last byte of the packet MUST be cleared to zero by the encoder.
So, logically bits are written to the Theora stream most-significant bit first. Okaaaay. A lot of people would look at this and say “well, they must have had a good reason for this”. I’ve been bit-twiddling since the early 80s, and I can unequivocally tell you: they don’t. This scheme is just confusing and crazy.
Sure, if you just write a bit streaming library and call it to read and write all of your values, you’ll never see this aspect. But just look at the structure in figure 6.2 above: only four values cannot be expressed in even byte quantities. A structure with bitfields is a fine way to deal with this problem, but instead of the bit fields in the header being arranged like this:
msb lsb 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 +---------------+-------------- R R R P P K K K K K Q Q Q Q Q Q Q = QUAL, K = KFGSHIFT, P = PF, R = RES
They’re actually arranged like this:
msb lsb 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 +---------------+-------------- K K K P P R R R Q Q Q Q Q Q K K Q = QUAL, K = KFGSHIFT, P = PF, R = RES
So, yes, that five-bit KFGSHIFT field is arranged so that it is virtually impossible to reference it as a single quantity, bit-wise. You can’t shift, you can’t mask. You have to access each piece separately and combine them. They also bear no resemblance to the diagram in the spec. So, my new, not-so-attractive structure looks like this (most fields omitted for brevity):
typedef struct __attribute__ ((__packed__)) _THEORA_ID_HEADER {
. . .
// !!!HACK!!! ok, per the Theora spec, bit fields are
// "read" and "written" to bytes most-significant bit first. This means
// that the KFGSHIFT field (which is, of course, the only field I'm interested
// in here) isn't stored contiguously in the file. The MSBs are in the first
// byte, the LSBs in the second. Crazy. http://www.theora.org/doc/Theora.pdf
ushort KFGSHIFT_MSB:2; // The amount to shift the key frame number by in the granule position
ushort QUAL:6; // The quality hint.
ushort RES:3; // Reserved.
ushort PF:2; // Pixel format.
ushort KFGSHIFT_LSB:3; // The amount to shift the key frame number by in the granule position
} THEORA_ID_HEADER;
#define THEORA_GET_KFGSHIFT(th) ((th).KFGSHIFT_MSB << 3 | (th).KFGSHIFT_LSB)
So, that structure along with the helper macro will give me the KFGSHIFT value I wanted, aka 6. Hopefully the next person implementing Theora will find this post and save themselves a head-desk moment.
Totally matches what I’ve found in using libraries – even Java: the edge cases are often not programmed for and end up taking me a few days to create work-arounds. I notice you often roll-your-own when doing UI layer work and have come around to that approach myself. The libraries are just too brittle both in js and in Android.
Nicely put. Appreciate it!
http://canadianpharmaciescubarx.com/
online pharmacy no prescription
canadian pharmacy online
canada drug prices
canadian drugs
Great advice. With thanks!
http://talahicc.com/
mexican pharmacies shipping to usa canadian pharmacies that ship to us canadian online pharmacies canadian pharmacy online
canadian drugstore
online canadian discount pharmacy
onlinecanadianpharmacy.com
canadian cialis
canadian rx pharmacy online
international pharmacy
Valuable postings. With thanks.
http://canadianpharmaciesnnm.com/
drug prices comparison trust pharmacy canada canadian drugstore top rated canadian pharmacies online
canada pharmacies prescription drugs
no prior prescription required pharmacy
pharmacy online mexico
canadian pharmacies online
no 1 canadian pharcharmy online
king pharmacy
Whoa all kinds of amazing facts.
http://canadianonlinepharmacynnm.com/
pain meds online without doctor prescription top rated canadian pharmacies online canada pharmacies online pharmacy canadian pharmacies without an rx
canadian pharmacies top best
prescription drug assistance
best canadian pharmacies
pharmacy prices compare
compare rx prices
no prescription pharmacy
Incredible all kinds of wonderful advice!
https://nicktambone.com/
canadian mail order pharmacies canadian cialis drug prices comparison northwest pharmacy
visit poster’s website
cheap drugs online
drugs online
pharmacy online store
mexican online pharmacies
best online pharmacy
Cheers! A lot of knowledge!
https://nicktambone.com/
my canadian pharmacy canada pharmaceutical online ordering cheap prescription drugs online pharmacies canada
safe canadian online pharmacies
canadian drug
canadian prescription drugstore
canadianpharmacy
online pharmacy without a prescription
pharmacy tech
This is nicely said! .
http://canadianpharmacysilo.com/
pain meds online without doctor prescription online pharmacies canada canada pharmacies without script canadian viagra
buy prescription drugs from canada
prescription pricing
prescription drug price comparison
canadian family pharmacy
safeway pharmacy
canada drug
You revealed this wonderfully.
http://canadianpharmacysilo.com/
canadian rx pharmacy online canada drug canadian drugstore canadian viagra
prescription drugs online without doctor
canadian drugstore online
us pharmacy no prior prescription
canadian pharmacies online prescriptions
canadian pharmacy no prescription
pharmacy online store
Effectively voiced certainly! !
https://nicktambone.com/
international pharmacy canadianpharmacy most reliable canadian pharmacies canadian viagra
canadian rx
visit poster’s website
mail order pharmacies
rx price comparison
canadian drugstore online
pharmacies near me
Terrific material. Many thanks!
http://www.canadianpharmacymsn.com/
drugs from canada canadian cialis mexican pharmacies canadian pharmacies online
canadian pharmacy no prescription
canada pharmacy online orders
pharmacy price comparison
prescription drugs canada
compare prescription prices
prescription cost comparison
Amazing stuff, Thanks!
http://www.canadianpharmacymsn.com/
canadian pharmacy reviews canada drugs online canadian pharmacy no prescription needed trust pharmacy canada
canadian cialis
canadian viagra
prescription cost comparison
discount pharmacy online
canadian pharmacy no prescription
drug costs
Reliable information. Appreciate it.
https://nicktambone.com/
canadapharmacyonline com canadianpharmacy canadian pharmacies online prescriptions london drugs canada
best online pharmacies canada
mexican pharmacies
buy prescription drugs online
online drugs
canada pharmacies online pharmacy
canadapharmacyonline.com
Nicely put. Cheers.
https://nicktambone.com/
canadian pharmacies global pharmacy canada pharmacy cost comparison canadian pharmacy online
drugstore online
web medical information
pharmacy prices
canadian online pharmacy
online canadian pharcharmy
buy prescription drugs without doctor
Seriously plenty of awesome info!
http://canadianpharmaciesnnm.com/
us pharmacy no prior prescription northwestpharmacy online canadian discount pharmacy global pharmacy canada
mexican online pharmacies
prescription drugs without prior prescription
canada drugs online pharmacy
compare prescription prices
legitimate canadian mail order pharmacies
best canadian prescription prices
Many thanks. I enjoy this.
https://nicktambone.com/
best online canadian pharcharmy canadian pharmacy prescription drug cost top rated canadian pharmacies online
mexican pharmacies
canada pharmacy
internet pharmacy
canada pharmacies online prescriptions
no prescription online pharmacy
highest rated canadian pharmacies
You actually suggested that very well.
http://canadianpharmaciesnnm.com/
mail order pharmacies online pharmacies of canada us pharmacy no prior prescription canadian pharmacies online
no prescription online pharmacy
canadian pharma companies
discount pharmacy
most reliable canadian pharmacies
canada drug pharmacy
canadian cialis
Helpful forum posts. Appreciate it.
http://canadianpharmacyntx.com/
online pharmacy no prescription needed mexican pharmacies shipping to usa buy prescription drugs from canada canada pharmacy
canada drugs pharmacy
online pharmacies canada
prescription cost
online pharmacy reviews
canadian pharma companies
online pharmacies no prescription
You have made your point quite well..
http://canadianpharmacysilo.com/
canadian pharmacy online canada london drugs canada online pharmacies no prescription online pharmacies canada
drug costs
prescription drugs online without doctor
online pharmacy reviews
discount prescription drugs online
pharmacy price comparison
canada online pharmacies
You’ve made your point.
http://canadianpharmacysilo.com/
best canadian prescription prices canadian pharmacies-24h northwest pharmacy canada canadian pharmacies that ship to us
canadian drug
compare rx prices
canada drug
drug prices comparison
rx pharmacy
canada drugs
You actually explained this really well!
http://canadianpharmacymim.com/
cialis online london drugs canada cheap prescription drugs canadian pharcharmy
online pharmacy usa
meds online
canadian rx pharmacy online
canada pharmacy online reviews
canadian online pharmacy reviews
online pharmacies
Regards! I value it!
http://canadianpharmacysilo.com/
pharmacy online canadian pharmacies that ship to us prescription drug assistance canadianpharmacy
24 hour pharmacy
canada pharmacies online
most reliable canadian pharmacies
canadian pharma companies
drug stores near me
buy prescription drugs without doctor
Thanks a lot. Loads of forum posts!
http://canadianpharmacyntx.com/
canadian pharmacy without prescription canada pharmacy online online pharmacy canada northwest pharmacy
canadian drug store
prescription drug cost
drugs for sale
canadian drug stores online
canadian pharmacy meds
rx online
Cheers. Ample facts!
http://canadianpharmaciescubarx.com/
drugstore online canada drug canadian discount pharmacy aarp approved canadian online pharmacies
canadian pharmacy no prescription
canadian pharmacies that are legit
best online international pharmacies
drugs without prescription
canadian pharmacies top best
canada drugs pharmacy
Wonderful posts. With thanks!
http://canadianpharmacymim.com/
buy online prescription drugs canadian pharmacies without an rx canadian pharmacy online canada drug
canada pharmacies prescription drugs
walgreens pharmacy
cheap prescription drugs
canadian drugs
canada drug pharmacy
canada pharmacies
Thank you, An abundance of write ups.
http://canadianonlinepharmacynnm.com/
buy prescription drugs from canada london drugs canada trust pharmacy canada canada drugs
my canadian pharmacy
most reliable canadian pharmacies
canadapharmacyonline.com
canadian pharmacy uk delivery
canada pharmacy online no script
canadian cialis
Very good tips. Thank you!
http://canadianpharmacymim.com/
canada pharmacy online reviews canadian cialis online pharmacies canada online pharmacies of canada
prescription pricing
pharmacy near me
buy prescription drugs online
online canadian pharcharmy
pharmacy without dr prescriptions
best canadian online pharmacy
Awesome facts. Thanks a lot!
http://www.canadianpharmacymsn.com/
canadian pharmacy online canadian online pharmacy online pharmacies canada canada pharmaceutical online ordering
canadian pharmacies
prescription drug cost
overseas pharmacies
buy prescription drugs online
drug price
best non prescription online pharmacies
Superb content. Many thanks.
http://canadianpharmaciescubarx.com/
no prescription pharmacy canada pharmacy online legitimate canadian mail order pharmacies northwestpharmacy
online pharmacy reviews
discount prescription drug
international pharmacies that ship to the usa
walgreens pharmacy
rx pharmacy
most reliable canadian pharmacies
Thank you. Ample forum posts!
http://canadianpharmaciescubarx.com/
canadian drugstore online canadian online pharmacies mexican pharmacies online canadianpharmacy
pharmacies near me
canadian pharmacies
drugs without prescription
discount pharmacy online
prescription drug cost
canadapharmacyonline com
This is nicely expressed! !
http://canadianpharmacyntx.com/
us pharmacy no prior prescription canada drugs online canada drugs pharmacy canadian pharmacy online
mail order pharmacies
global pharmacy canada
best online international pharmacies
no prescription pharmacies
drug price
prescription without a doctor’s prescription
Nicely put. Thanks.
http://canadianonlinepharmacynnm.com/
canadian pharcharmy online canada pharmacy canada pharmacy online top rated canadian pharmacies online
canadian prescription drugstore
legitimate canadian mail order pharmacies
no prescription pharmacies
mexican online pharmacies
drugs for sale
canada pharmacies online
With thanks! Excellent information!
http://canadianpharmaciesnnm.com/
canadian pharmacy no prescription needed canadian pharmacies online king pharmacy online pharmacies canada
prescription drugs online without
canada pharmacy online
safe canadian online pharmacies
drug prices comparison
canadian family pharmacy
canadian pharmacy no prescription
Kudos, Lots of information!
https://nicktambone.com/
canadian pharmacy online canadian pharmacy online canada drugs online pharmacy canada drug
global pharmacy canada
discount pharmacy online
prescription drugs prices
canada pharmacy online no script
the canadian pharmacy
canadian drugstore
Nicely put. Kudos.
http://canadianpharmacysilo.com/
approved canadian pharmacies online mexican pharmacies shipping to usa prescription drug price comparison north west pharmacy canada
canadian pharmacy meds
northwestpharmacy
no prescription pharmacy
pharmacy drug store
prescription cost
ed meds online
Incredible a good deal of amazing tips!
http://canadianpharmacysilo.com/
canada pharmacy no prescription mexican pharmacies shipping to usa canadian family pharmacy canadian pharmacies
northwestpharmacy
pharmacy cost comparison
canadian pharmacies shipping to usa
drug costs
canadian viagra
medical information online
Regards! I value it.
http://canadianpharmaciesnnm.com/
ed meds online north west pharmacy canada canadian discount pharmacy canadian pharmacies online
online pharmacies canada
canada drug prices
online prescription drugs
prescription drug assistance
canadian pharma companies
canada pharmacy
Cheers, An abundance of data.
http://canadianpharmacyntx.com/
canada meds canadian online pharmacies canadian pharmacy without prescription aarp approved canadian online pharmacies
canadian meds
prescription cost
canada pharmacy online no script
canadian rx pharmacy online
northwest pharmacy canada
prescription drug price comparison
Many thanks! Ample content!
http://talahicc.com/
buy online prescription drugs canadian pharmacies online canadian pharmacy meds top rated canadian pharmacies online
mexican pharmacy online
drugs without prescription
legitimate canadian mail order pharmacies
discount prescription drug
canada pharmacy online reviews
canadian drugstore online
Thanks, Numerous advice!
http://talahicc.com/
online pharmacy without a prescription trust pharmacy canada canada pharmacies prescription drugs canada drug
canada pharmacy online orders
safeway pharmacy
canada drugs online pharmacy
buy prescription drugs from canada
prescription drugs without prior prescription
safe canadian online pharmacies
You reported it terrifically!
http://canadianpharmacyntx.com/
canadadrugs canadian online pharmacy canadian drugstore north west pharmacy canada
prescription drugs online without
canada online pharmacy
cheap prescription drugs online
mexican pharmacies
canadian rx
king pharmacy
Appreciate it! Plenty of info.
http://talahicc.com/
best online pharmacy canadian pharmacies that ship to us discount drugs online pharmacy canadian pharmacy online
canadian pharmacy without prescription
prescription drug price comparison
onlinecanadianpharmacy.com
buy online prescription drugs
discount pharmacy
<a href="http://mirabiliatour.blogspot.com/ask?q=+Excellent+knowledge.+Thanks+a+lot.+http%3A%2F%2Fcanadianpharmaciesnnm.com%2F+canadian+pharmacy+reviews+canadian+pharcharmy+mexican+pharmacy+online+medications+%5Burl%3Dhttp%3A%2F%2Fcanadianpharmaciesnnm.com%2F%5Dnorthwest+pharmacy%5B%2Furl%5D+%0D%0Apharmacy+prices+%0D%0Acanada+drugs+online+%0D%0Aed+meds+online+%0D%0Acanada+online+pharmacy+%0D%0Aonline+pharmacies+canada+%0D%0Acanada+pharmacy&submit=ask”>canadian pharmacy online canada
Wow plenty of useful data.
http://canadianpharmacymim.com/
drug costs canada online pharmacies safeway pharmacy canadian viagra
overseas pharmacies
canadian viagra
canadian drugstore
best online pharmacy stores
mexican pharmacies
canadian prescription drugs
Cheers. I value this.
http://www.canadianpharmacymsn.com/
canadian meds canadian online pharmacy online canadian pharmacy canadian pharmacies that ship to us
pharmacy online store
canadian online pharmacy reviews
onlinecanadianpharmacy.com
best canadian prescription prices
discount pharmaceuticals
canadapharmacy
Excellent material. Appreciate it!
http://www.canadianpharmacymsn.com/
get prescription online top rated canadian pharmacies online canada pharmacies online prescriptions online pharmacies of canada
online pharmacy
canadian drugs without prescription
buy prescription drugs without doctor
drug costs
canadian pharmacies shipping to usa
pharmacies in canada
Cheers! Awesome stuff!
http://canadianpharmaciesnnm.com/
mexican pharmacy online medications canadian pharmacies-24h no 1 canadian pharcharmy online canadian online pharmacies
canadian online pharmacies prescription drugs
canadian discount pharmacy
online pharmacy usa
northwest pharmacy
cheap prescription drugs
canada pharmacies online prescriptions
Kudos! Very good information.
http://talahicc.com/
us pharmacy no prior prescription online pharmacies of canada buy cialis canadianpharmacy
prescription drug assistance
discount drugs online pharmacy
canadian pharmacy
prescription without a doctor’s prescription
online canadian pharcharmy
cheap canadian drugs
Reliable advice. Thanks!
http://canadianonlinepharmacynnm.com/
drug costs trust pharmacy canada online drugstore canada drug
drugs without prescription
mail order pharmacy
best online pharmacy stores
prescription price comparison
pharmacy online
canada pharmacies online pharmacy
You said that really well!
http://canadianpharmacyntx.com/
prescription drugs without prior prescription canadian pharmacies without an rx medical information online trust pharmacy canada
canadian pharmacy meds
prescription drug price comparison
highest rated canadian pharmacies
canadian meds
canada drug pharmacy
canada meds
You suggested it exceptionally well.
http://talahicc.com/
canada pharmacy online orders canadian online pharmacy prescription price comparison canadian pharmacies online
canadian pharmacy no prescription
canada drug prices
prescription drug cost
prescription drug
discount pharmacy
online pharmacy reviews
Superb material. Appreciate it.
http://canadianpharmacyntx.com/
prescription online canada drugs online prescription drug cost online pharmacies canada
the canadian pharmacy
canadian pharmacy no prescription
drug price
prescription drug
online discount pharmacy
pharmacies near me
You said it nicely..
http://canadianpharmacysilo.com/
canada pharmacy no prescription top rated canadian pharmacies online prescription drugs online without canada pharmaceutical online ordering
pharmacies in canada
mexican pharmacies online
drug costs
best canadian mail order pharmacies
buy prescription drugs canada
buy prescription drugs from canada
Helpful content. Thanks!
http://canadianonlinepharmacynnm.com/
online pharmacy reviews canadian viagra king pharmacy canadian pharmacy
canadian pharmacy reviews
buying prescription drugs canada
ed meds online
pharmacy prescription
canada pharmacy online
canadian prescription drugs
Nicely put. Appreciate it.
http://canadianpharmacysilo.com/
pharmacy near me canada drugs online online drugs online pharmacies canada
canadian pharmacies online
prescription drugs without prior prescription
pharmacy online
canadian online pharmacies
canadian drug stores online
canada drugs online pharmacy
Kudos, Very good information.
http://canadianpharmacymim.com/
buy cialis northwest pharmacy mexican pharmacies shipping to usa canadian pharcharmy
canada drug pharmacy
drugstore online
canada pharmacies
mexican pharmacies online
canadian pharmacy uk delivery
pharmacy online
Very good forum posts, Thank you!
http://canadianpharmacysilo.com/
canadian pharmacies online prescriptions canadian pharmacy canadian drugs without prescription canadian pharmacies-24h
best canadian prescription prices
buy prescription drugs without doctor
mexican online pharmacies
prescription cost comparison
discount pharmacy online
mexican pharmacy online