unsigned divide(unsigned x) { return x / 13; }

It gets better, I promise. Here’s the corresponding x64 assembly that gcc produces:

_divide:
        movl    $1321528399, %edx
        movl    %edi, %eax
        mull    %edx
        shrl    $2, %edx
        movl    %edx, %eax
        ret

Instead of division, there’s multiplication by a bizarre number: 1321528399. What wickedness is this?

1321528399 is an example of a “magic number:” a number that lets you substitute speedy multiplication for pokey division, as if by magic. In this post, I become a big fat spoilsport and ruin the trick for everyone.

Dividing by 13 is different than multiplying by 1321528399, so there must be more here than meets the eye. Compiling it as PowerPC reveals a bit more:

_divide:
        lis r0,0x4ec4
        ori r0,r0,60495
        mulhwu r3,r3,r0
        srwi r3,r3,2
        blr

mulhwu means “multiply high word unsigned.” It multiplies by 1321528399, but takes the high 32 bits of the 64 bit result, not the low 32 bits like we are used to. x86-64 does the same thing (notice the shift right instruction operates on %edx, which contains the high bits, not %eax which is the low bits). After multiplying, it shifts the result right by two. In C:

unsigned divide(unsigned x) {
   return (unsigned)(x*1321528399ULL >> 34);
}

So dividing by 13 is the same as multiplying by 1321528399 and then dividing by 2³⁴. That means that 2³⁴ divided by 13 would be 1321528399, right? In fact, it’s 1321528398.769 and change. Pretty close, but we’re not optimizing horseshoes, so how can we be sure this works all the time?

The answer to the question right above this line

It’s provable, and hopefully we can get some intuition about the whole thing. In the following proof, division is exact, not that integer-truncating stuff that C does; to round down I’ll use floor. Also, this only covers unsigned division: every number that appears is non-negative. (Signed division is broadly similar, though does differ in some details.) Lastly, we’re going to assume that the denominator d is not a power of 2, which is weirdly important. If d were a power of 2, we’d skip the multiplication and just use shifts, so we haven’t lost any generality.

We have some fixed positive denominator d and some variable nonegative numerator n, and we want to compute the quotient – no, wait, , since that’s what C does. We’ll multiply the top and bottom by 2^k, where k is some positive integer power – it represents the precision in a sense, and we’ll pick its value later:

We’re going to call that term m_exact, because it’s our “magic number” that lets us compute division through multiplication. So we have:

This equality looks promising, because we’ve hammered our expression into the shape we want; but we haven’t really done anything yet. The problem is that m_exact is a fraction, which is hard for computers to multiply. (We know it’s a fraction because d is not a power of 2). The tricky part is finding an integer approximation of m_exact that gives us the same result for any dividend up to the largest possible one (UINT_MAX in our case). Call this approximation m. So we have , where m is an integer.

When it comes to approximation, closer is better, so let’s start by just rounding m down: . Since m_exact cannot be an integer, m must be strictly less than m_exact. Does this m work, which is to say, does it behave the same as m_exact over the range we care about?

We’ll try it when the numerator equals the denominator: n = d. Of course then the quotient is supposed to be 1. This leads to:

That’s bad, because that last expression is supposed to be 1. The approximation is too small, no matter what k is.

Ok, let’s round up instead: . Does this value for m work?

Now, just means dividing 2^k by d and then rounding up. That’s the same as rounding up 2^k to the next multiple of d first, and then dividing exactly by d. (We know that 2^k is not itself a multiple of d because d is not a power of 2!) So we can write:

where e is the “thing to add to get to the next multiple of d,” which means that 0 < e < d. (Formally, e = d - 2^k mod d.) This value e is sort of a measure of how much “damage” the ceil does – that is, how bad the approximation m is.

So let’s plug this new m into our division expression, and then apply some algebra:

This last expression is really cool, because it shows the wholesome result we’re after, , and also an insidious “error term:” that , which represents how much our ceil is causing us to overestimate. It also shows that we can make the error term smaller by cranking up k: that’s the sense in which k is a precision.

K’s Choice

We could pick some huge precision k, get a very precise result, and be done. But we defined m to be , so if k is too big, then m will be too big to fit in a machine register and we can’t efficiently multiply by it. So instead let’s pick the smallest k that’s precise enough, and hope that fits into a register. Since there are several variables involved, our strategy will be to find upper bounds for them, replace the variables with their upper bounds, and simplify the expression until we can solve for k.

So we want the smallest k so that . This will be true if the error term is less than 1, because then the floor operation will wipe it out, right? Oops, not quite, because there’s also a fractional contribution from . We need to be sure that the error term, plus the fractional contribution of , is less than 1 for that equality to be true.

We’ll start by putting an upper bound on the fractional contribution. How big can it be? If you divide any integer by d, the fractional part of the result is no more than . Therefore the fractional contribution of is at most , and so if our error term is less than , it will be erased by the floor and our equality will be true.

So k will be big enough when   .

Next we’ll tackle . We know from up above that e is at most d-1, so we have . So we can ignore that factor: .

The term has a dependence on n, the numerator. How big can the numerator be? Let’s say we’re dividing 32 bit unsigned integers: n ≤ 2³². (The result is easily extended to other widths). We can make less than 1 by picking k = 32. To make it even smaller, less than , we must increase k by . That is,

Since we want k to be an integer, we have to round this up: . We know that value for k works.

What does that precision do to m, our magic number? Maybe this k makes m too big to fit into a 32 bit register! We defined ; plugging in k we have .

This is a gross expression, but we can put an upper bound on it. Note that , so we can write the following:

Nuts! Our approximation m just barely does not fit into a 32 bit word!

Since we rounded some stuff, you might think that we were too lazy and a more careful analysis would produce a tighter upper bound. But in fact our upper bound is exact: the magic number for an N-bit division really may need to be as large as N+1 bits. The good news is, there may be smaller numbers that fit in N bits for specific divisors. More on that later!

We are fruitful and multiply

In any case, we now have our magic number m. To perform the division, we need to compute as efficiently as possible. m is a 33 bit number, so how can a 32 bit processor efficiently multiply by that? One way would be to use a bignum library, but’s probably slow. A better approach is to multiply by the low 32 bits, and handle the 33rd bit specially, like this:

This looks very promising: m – 2³² is definitely a 32 bit number. Furthermore, we can compute that 2³²n term very easily: we don’t even need to shift, just add n to the high word of the product. Unfortunately, it’s not right: a 32 bit number times a 33 bit number is 65 bits, so the addition overflows.

But we can prevent the overflow by distributing one of the 2s in the denominator, through the following trick:

Here q is the magic number (minus that 33rd bit) times n, and n is just n, the numerator. (Everything should be multiplied by 2³², but that’s sort of implicit in the fact that we are working in the register containing the high product, so we can ignore it.)

Can the subtraction n – q underflow? No, because q is just n times the 32 bit magic number, and then divided by 2³². The magic number (now bereft of its 33rd bit) is less than 2³², so we have q < n, so n - q cannot underflow.

What about that +q? Can that overflow? No, because we can just throw out the floor to get an upper bound of which is a 32 bit number.

So we have a practical algorithm. Given a dividend n and a fixed divisor d, where 0 < d < 2³² and 0 ≤ n < 2³²:

1. Precompute
2. Precompute . This will be a 33 bit number, so keep only the low 32 bits.
3. Compute . Most processors can do this with one “high multiply” instruction.
4. Compute , which is an overflow-safe way of computing (n + q) >> 1. This extra addition corrects for dropping the 33rd bit of m.
5. Perform the remaining shift p:

That’s it! We know how to efficiently replace division with multiplication. We can see all these steps in action. Here is the assembly that Clang generates for division by 7, along with my commentary:

_divide_by_7:
 movl $613566757, %ecx  613566757 is the low 32 bits of (2**35)/7, rounded up
 movl %edi, %eax
 mull %ecx              Multiply the dividend by the magic number
 subl %edx, %edi        The dividend minus the high 32 bits of the product in %edx (%eax has the low)
 shrl %edi              Initial shift by 1 to prevent overflow
 addl %edx, %edi        Add in the high 32 bits again, correcting for the 33rd bit of the magic number
 movl %edi, %eax        Move the result into the return register
 shrl $2, %eax          Final shift right of floor(log_2(7))
 ret

Improving the algorithm for particular divisors

This algorithm is the best we can do for the general case, but we may be able to improve on this for certain divisors. The key is that e term: the value we add to get to the next multiple of d. We reasoned that e was less than d, so we used as an upper bound for e. But it may happen that a multiple of d is only slightly larger than a power of 2. In that case, e will be small, and if we're lucky, it will be small enough to push the entire "error term" under .

For example, let’s try it for the divisor 11. With the general algorithm, we would need . But what if we choose k = 35 instead? 2³⁵ + 1 is a multiple of 11, so we have e = 1, and we compute:

So k = 35 forces the error term to be less than , and therefore k = 35 is a good enough approximation. This is good news, because then , so our magic number fits in 32 bits, and we don't need to worry about overflow! We can avoid the subtraction, addition, and extra shifting in the general algorithm. Indeed, clang outputs:

_divide_by_11:
  movl  $-1171354717, %ecx
  movl  %edi, %eax
  mull  %ecx
  movl  %edx, %eax
  shrl  $3, %eax
  ret

The code for dividing by 11 is shorter than for dividing by 7, because a multiple of 11 happens to be very close to a power of 2. This shows the way to an improved algorithm: We can simply try successive powers of 2, from 32 up to , and see if any of them happen to be close enough to a multiple of d to force the error term below . If so, the corresponding magic number fits in 32 bits and we can generate very efficient code. If not, we can always fall back to the general algorithm.

The power 32 will work if , the power 33 will work if , the power 34 will work if , etc., up to , which will work if . By "chance", we'd expect this last power to work half the time, the previous power to work 25% of the time, etc. Since we only need one, most divisors actually should have an efficient, 32 bit or fewer magic number. The infinite series sums to 1, so our chances get better with increasing divisors.

It's interesting to note that if a divisor d has one of these more efficient magic numbers for a power , it also has one for all higher powers. This is easy to see: if 2^k + e is a multiple of d, then 2^k+1 + 2e is also a multiple of d.

This is good news. It means that we only have to check one case, (a 32 bit value) to see if there is a more efficient magic number, because if any smaller power works, that one works too. If that that power fails, we can go to the 33 bit number, which we know must work. This is useful information in case we are computing magic numbers at runtime.

Still, gcc and LLVM don't settle for just any magic number - both try to find the smallest magic number. Why? There's a certain aesthetic appeal in not using bigger numbers than necessary, and most likely the resulting smaller multiplier and smaller shifts are a bit faster. In fact, in a very few cases, the resulting shift may be zero! For example, the code for diving by 641:

_divide_by_641:
        movl    $6700417, %ecx
        movl    %edi, %eax
        mull    %ecx
        movl    %edx, %eax
        ret

No shifts at all! For this to happen, we must have , which of course means that e = 1, so 641 must evenly divide 2³² + 1. Indeed it does.

This inspires a way to find the other "super-efficient" shiftless divisors: compute the factors of 2³² + 1. Sadly, the only other factor is 6700417. I have yet to discover an occasion to divide by this factor, but if I do, I'll be ready.

So that's how it works

Did you skip ahead? It's OK. Here's the summary.

Every divisor has a magic number, and most have more than one! A magic number for d is nothing more than a precomputed quotient: a power of 2 divided by d and then rounded up. At runtime, we do the same thing, except backwards: multiply by this magic number and then divide by the power of 2, rounding down. The tricky part is finding a power big enough that the "rounding up" part doesn't hurt anything. If we are lucky, a multiple of d will happen to be only slightly larger than a power of 2, so rounding up doesn't change much and our magic number will fit in 32 bits. If we are unlucky, well, we can always fall back to a 33 bit number, which is almost as efficient.

I humbly acknowledge the legendary Guy Steele Henry Warren (must have been a while) and his book Hacker's Delight, which introduced me to this line of proof. (Except his version has, you know, rigor.)

If you immediately blurted out “x­is­an­array­of­three­const­pointers­to­functions­returning­pointers­to­arrays­of­five­pointers­to­functions­taking­int­returning­char“, and your last name doesn’t end with “itchie,” then chances are you used my new website:

www.cdecl.org

Yes, the venerable cdecl – the C gibberish to English translator – is now online. With AJAX! Every C declaration will be as an open book to you! Your coworkers’ scruffy beards and suspenders will be nigh useless!

Write C casts and declarations in plain English! Write plain English in the chicken scratchings and line noise we call C! The possibilities are twain!

(Click here to try it with that declaration!)

ANNNNND…it gets better! Did I mention that my version of cdecl supports blocks? That’s right, now you can totally nail that API that requires a block taking a pointer to a block taking a block taking a pointer to an int returning an int returning a pointer to a block taking void returning int!

This site:

• Converts readable English variable declarations or typecasts to C
• Converts C variable declarations or typecasts to English
• Supports Apple’s blocks extension to C
• Uses AJAX and has nifty effects
• Allows you to generate permalinks, so you can send hilarious declarations to your friends, or add them gratuitously to your blog

A note on licensing. The cdecl readme states:

You may well be wondering what the status of cdecl is. So am I. It was twice posted to comp.sources.unix, but neither edition carried any mention of copyright. This version is derived from the second edition. I have no reason to believe there are any limitations on its use, and strongly believe it to be in the Public Domain.

I hereby place my blocks changes to cdecl in the public domain. The cdecl source code, including my changes, is available for download on the site.

For those who like their punchlines first: std::string is designed to allow copy-on-write. But once operator[] is called, that string is ruined for COW forevermore. The reason is that it has to guard against a future write at any time, because you may have stashed away the internal pointer that operator[] returns – a feature that you surely know better than to use, but that costs you nevertheless.

The C++ standard string class is std::string. Here’s a string containing 10 megabytes of the letter ‘x’:

#include <string>
using std::string;
int main(void) {
    string the_base(1024 * 1024 * 10, 'x');
}

Big string. Now let’s make a hundred copies, one by one:

#include <string>
using std::string;
int main(void) {
    string the_base(1024 * 1024 * 10, 'x');
    for (int i = 0; i < 100; i++) {
        string the_copy = the_base;
    }
}

This runs in .02 seconds, which is very fast. Suspiciously fast! This old iMac can’t schlep a gigabyte of data in two hundredths of a second. It’s using copy-on-write.

So let’s write and measure the copy:

#include <string>
using std::string;
int main(void) {
    string the_base(1024 * 1024 * 10, 'x');
    for (int i = 0; i < 100; i++) {
        string the_copy = the_base;
        the_copy[0] = 'y';
    }
}

The program now runs in 2.5 seconds. These scant 100 writes resulted in a 100x slowdown. Not surprising: it’s exactly what we would expect from a COW string.

the_copy[0] returns a reference to a char. But maybe we aren’t going to write into it – perhaps we only want to read. How can these copy-on-write strings know when the write occurs, so they can copy? The answer is that they cannot – they must be pessimistic and assume you will write, even if you never do. We can trick it: don’t assign anything, and still see the performance hit:

#include <string>
using std::string;
int main(void) {
    string the_base(1024 * 1024 * 10, 'x');
    for (int i = 0; i < 100; i++) {
        string the_copy = the_base;
        the_copy[0];
    }
}

No writes! Just a read! But same performance as before. This isn’t copy on write – it’s copy on read!

Things get even worse. operator[] returns a reference: basically a sugary pointer. You can read from it or write to it. Or you can be a jerk and just hang onto it:

string original = "hello";
char & ref = original[0];
string clone = original;
ref = 'y';

In case that’s not clear, we made a string, stashed away a reference to the first character, copied the string, and then wrote to the string’s guts. The write is just a normal store to a memory location, so the poor string is caught completely unaware. If the copy-on-write were naive, then we would have modified both the original and the copy.

Oh man, is that evil. But the string’s got to be paranoid and guard against it anyways – in fact, the C++ standard explicitly calls out this case as something that has to work! In other words, when you get a reference to a char inside a string, the string is tainted. It can never participate in copy-on-write again, because you just might have stashed away the internal pointer it handed back, and write into the string as a completely unexpected time.

Of course you know better than to hang onto internal pointers. You probably don’t need this feature. But you pay for it anyways:

#include <string>
using std::string;
int main(void) {
    string the_base(1024 * 1024 * 10, 'x');
    the_base[0];
    for (int i = 0; i < 100; i++) {
        string the_copy = the_base;
    }
}

This takes the 2.5 seconds. Just one silly read wrecked all of the COW machinery for all the future copies. You’re paying for writes that never come!

Could the C++ standard have fixed this? Sure – the C++ standard goes into great detail about when iterators and references into strings are invalidated, and all it would have taken would be to add the copy constructor and operator=() to the list of reference-invalidating functions (section 23.1.5, for those who are following along in the standard). A second option would have been to declare that operator=() invalidates existing references for writing, but not for reading. But the standards committee preferred simplicity, convenience and safety to performance – uncharacteristically, if you’ll forgive a smidgen of editorializing.

If you call a reference-invalidating function, you can, in principle, get back in the COW fast lane. Although GNU STL does not take advantage of this in all cases, it does when resizing strings:

#include <string>
using std::string;
int main(void) {
    string the_base(1024 * 1024 * 10, 'x');
    the_base[0];
    the_base.push_back('x');
    for (int i = 0; i < 100; i++) {
        string the_copy = the_base;
    }
}

Allowing the string to modify itself allowed it to know that any previous references were invalidated. We're fast again! It's called a COW, and it's not about to blow it now.

The moral is that you should avoid using operator[], the at() function, iterators, etc. to read from a non-const string - especially a long string - if it may be copied in the future. If you do use these functions, you risk paying for a write that you don't make, and for hanging onto an internal pointer that you released. Unfortunately, there's no good function for just reading from a non-const string. You can do it by adding a crazy cast:

#include <string>
using std::string;
int main(void) {
	string the_base(1024 * 1024 * 10, 'x');
	const_cast<const string &>(the_base)[0];
	for (int i = 0; i < 100; i++) {
		string the_copy = the_base;
	}
}

This hits the fast path too.

Scene. I hope you enjoyed this post. As usual the point is not to rag on C++, but to explore aspects of its design that, for whatever reason, tilt a tradeoff away from performance.

Hooray, it’s Hex Fiend 2, a nearly complete rewrite of Hex Fiend that incorporates even better techniques for working with big files. Hex Fiend is my fast and clever hex editor for Mac OS X.

Click On Me To Get Hex Fiend

This app is about exploring the implementation of standard desktop UI features in the realm of files too large to fully read into main memory. Is it possible to do copy and paste, find and replace, undo and redo, on a document that may top a hundred gigabytes, and make it feel natural? Where do we run into trouble?

More on that later. For now, here’s what’s better about Hex Fiend 2:

• It’s embeddable. The most requested feature was "I want a hex view in my app." Now it’s really easy: Hex Fiend 2 is built as a relatively slim shell on top of a bundle-embeddable .framework. There’s a real API, sample code, and everything. See the Developer section of the Hex Fiend page.

• It’s faster. All around. Text rendering is speedier, and the backing data representation more efficient. It needs less I/O too.

  The save algorithm is especially improved. For example, inserting one byte at the beginning of a 340 MB file and hitting Save would take 52 seconds and 340 additional MB of temporary disk space with Hex Fiend. In Hex Fiend 2 it’s reduced to 22 seconds and requires no temporary disk space.

• Better UI. It supports discontiguous selection. Scroll-wheel scrolling no longer feels weird. You can group the bytes into blocks (credit to Blake), and you can hide and show different views (thanks to bbum). The big dorky line number view now shrinks to fit. The data inspector panel is inline. It has Safari-style inline find and replace and “pop out” selection highlighting.
• Long operations support progress reporting, cancellation, and don’t block the UI. For example, find and replace now has a progress bar and a cancel button, and you can keep using your document (or others) while it works.
• Longstanding bugfixes. Backwards searching is now optimized. Certain coalesced undo bugs have been addressed. There’s some basic cross-file dependency tracking.

I hope you find it useful.

So here’s the first episode of I Didn’t Order That, So Why Is It On My Bill, to be continued until I can’t think of any more.

Inline Functions

The C++ standard says this:

A static local variable in an extern inline function always refers to the same object.

In other words, static variables in inline functions work just like static variables in any function. That’s reasonable, because that’s probably what you want. That’s what statics are for, after all.

But wait, it also says this:

An inline function with external linkage shall have the same address in all translation units.

That’s borderline silly. Who cares what the address of the function is? When’s the last time you used the function pointer address for anything except calling it? Dollars to donuts says never. You aren’t using a function pointer as a hash table key. You aren’t comparing the same function via pointers from different files. And if you did, you’d probably have the good sense to ensure the functions are defined exactly once (i.e. not inline). You can easily live without this feature.

So hopefully I’ve established that you don’t use this feature. Now I have to show how you’re paying for it anyways. Well, this feature complicates linking. Ordinarily, there should be only one copy of a function, and if the linker finds more than one it gives you a multiple definition error. But with inline functions, the compiler is expected to generates multiple copies of the code and it’s up to the linker to sort it out. See vague linkage.

So the linker has more work to do, and link time is increased. Again, who cares? Linking occurs after compilation, and I promised you a runtime cost. But ah – link time is runtime when we’re using dynamic libraries!

Every time you start up a program, the linker has to make sure that your implementation of vector<int>::capacity() has the same address as the vector<int>::capacity() defined in libWhoGivesAHoot.dylib just in case you decide to go and compare their function pointers.

And it gets a little worse. You know how class templates have to live in header files? The function definition goes right there in the class declaration, and that makes them inline automatically. So nearly every function in a template gets this inline uniquing treatment. Every template function in a class that cannot or should not be inlined – because its address is taken, because it is recursive, because you’re optimizing for size, or simply because it’s too darn big and inlining is counterproductive – will incur a launch time cost.

The situation is dire enough that gcc added a "-fvisibility-inlines-hidden" flag. This is a minor violation of the C++ standard, but will improve launch time in cases with a lot of dynamic libraries. In other words, this flag says: I’m not using this feature, please take it off my bill.

C does not have this problem. Why not? C (in the C99 standard) has very different semantics for inline functions. To oversimplify a bit, a C99 inline function acts as sort of an alias for a real function, which must be defined once and only once. Furthermore, inline functions can’t have static variables, or have their addresses taken. This approach is more limiting, but doesn’t require the linker to bend over backwards, and so avoids the performance penalty of the C++ approach.

As any usability expert will tell you, the foundation of sound web page design is rounded corners. Their smooth curves lend an organic, almost sensual feel to a site, while other pages are un-navigable morasses of right angles.

Rounded corners are easy with tables and images! But tables make your page Web 2.0-incompatible, so most browsers cannot render it.

So when I decided to get some roundy corners for myself, I looked around, and the best I was able to find was Nifty Corners. There’s a lot to like about Nifty Corners, but look what happens if you try to make them overlap or put them on a, say, not-fixed-color background:

Click on the image to see the actual page.

Notice that the corners of the boxes suffer. The problem is that the area around the corner cannot be drawn transparently – it needs to be over a fixed color. Browser specific extensions would be perfect if they weren’t browser specific.

But I like crazy backgrounds and overlapping boxes. So in a half-hearted effort for geek cred, here’s how my roundy corners worked.

Hey! Flexible widths!

But but but…

• Isn’t that, like, a humongous flood of markup? Won’t the page take a long time to download?

Well, I worried about that too. But happily, all modern browsers support gzip compression – and boy does it work! Even though my last posting with comments is 382 KB in size, you only need to download 19.8 KB (Mac users can look in Safari’s Activity Viewer for the download size). So it ends up being smaller than, say, Yahoo!’s home page – and that’s just the HTML, not even counting all of its images. As a result, the page loads quite fast.

• Well, don’t all those divs make the page unmaintainable?

Yeah, I suppose so. I use a script to generate these boxes, so the source is readable to me – even if the generated HTML isn’t.

On another thread, I’ve fixed the Angband screensaver to remember the last character it played with. The user defaults had to be synchronized. Thanks to everyone who pointed this out.

Guest blogger: Gollum (Smeagol)

Where isst it? So it iss back – so fissh, the fish, raw and wriggling, he hasst brought its back!

Angband, the Hells of Iron, yest, the ancient ASCII roguelike, child spawn of Moria and VMS, it iss here once more. We wants it!

You hasst not seen it, Angband? But perhapss you have seen one like it, the filthy, filthy NetHack, yess, full of such stupids and jokeses, and no Smeagol! We hates it for ever! But we loves Angband, yess, Angband has Smeagol and the fat hobbitses and yes, it has my preciouss! We loves it, Angband!

We hass thought it lost. When OS X was a little child and hungry for software, we knows how it calls to fissh. We knows how fissh missed Angband, dearly missed Smeagol, yes. And we knows how fissh stoles it, and worked his tricksy little magic, so he could go down into a Carbonized Angband on OS X. And we knows fissh did, and it was easy, Carbon madess it simple.

But fissh hasst brought it to Cocoa, now, all fresh and it glitterses so subpixel pretty with Quartz, and resizesss so smooth, and animates with pretty graphics! So precious to fissh.

And ssuch love for Angband so fissh made a borg screensaveres! So now you can visits Smeagol and wring the filthy little neck of Saruman and murderes the fat Morgoth in your sleep! The screensaveres, yes!

Goes there now, fat hobbitses! The game and the screensaveres and the source codess! Angband need you, yess!

Angband for Mac OS X

Edit: fissh has fixed the problem with the screensaveres needing Angband to have been launched first. It should work no problems now.

"That’s a lot of cores. And while 80-core floating point monsters like that aren’t likely to show up in an iMac any time soon, multicore chips in multiprocessor computers are here today. Single chip machines are so 2004. Programmers better get crackin’. The megahertz free ride is over – and we have work to do."

There, that’s the noisome little pitch everyone’s been spreading like so much thermal paste. As if multiprocessing is something new! But of course it’s not – heck, I remember Apple shipping dualies more than ten years ago, as the Power Macintosh 9500. Multiprocessing is more accessible (read: cheaper) now, but it’s anything save new. It’s been around long enough that we should have it figured out by now.

So what’s my excuse? I admit it – I don’t get multiprocessing, not, you know, really get it, and that’s gone on long enough. It’s time to get to the bottom of it – or if not to the bottom, at least deep enough that my ears start popping.

Threadinology

Where to start, where to start…well, let’s define our terms. Ok, here’s the things I mean when I say the following, uh, things:

• Threads are just preemptively scheduled contexts of execution that share an address space. But you already know what threads are. Frankly, for my purposes, they’re all pretty much the same whether you’re using Objective-C or C++ or Java on Mac OS X or Linux or Windows…
• Threading means creating multiple threads. But you often create multiple threads for simpler control flow or to get around blocking system calls, not to improve performance through true simultaneous execution.
• Multithreading is the physically simultaneous execution of multiple threads for increased performance, which requires a dualie or more. Now things get hard.

Yeah, I know. "Multithreading is hard" is a cliché, and it bugs me, because it is not some truism describing a fundamental property of nature, but it’s something we did. We made multithreading hard because we optimized so heavily for the single threaded case.

What do I mean? Well, processor speeds outrun memory so much that we started guessing at what’s in memory so the processor doesn’t have to waste time checking. "Guessing" is a loaded term; a nicer phrase might be "make increasingly aggressive assumptions" about the state of memory. And by "we," I mean both the compiler and the processor – both make things hard, as we’ll see. We’ll figure this stuff out – but they’re going to try to confuse us. Oh well. Right or wrong, this is the bed we’ve made, and now we get to lie in it.

Blah blah. Let’s look at some code. We have two variables, variable1 and variable2, that both start out at 0. The writer thread will do this:

Writer thread

while (1) {
   variable1++;
   variable2++;
}

They both started out at zero, so therefore variable1, at every point in time, will always be the same as variable2, or larger – but never smaller. Right?

The reader thread will do this:

Reader thread

while (1) {
   local2 = variable2;
   local1 = variable1;
   if (local2 > local1) {
      print("Error!");
   }
}

That’s odd – why does the reader thread load the second variable before the first? That’s the opposite order from the writer thread! But it makes sense if you think about it.

See, it’s possible that variable1 and/or variable2 will be incremented by the writer thread between the loads from the reader thread. If variable2 gets incremented, that doesn’t matter – variable2 has already been read. If variable1 gets incremented, then that makes variable1 appear larger. So we conclude that variable2 should never be seen as larger than variable1, in the reader thread. If we loaded variable1 before variable2, then variable2 might be incremented after the load of variable1, and we would see variable 2 as larger.

Threadalogy

So we’ve got two threads operating on two variables, and we think we know what’ll happen. Let’s try it out, on my G5:

unsigned variable1 = 0;
unsigned variable2 = 0;

#define ITERATIONS 50000000

void *writer(void *unused) {
        for (;;) {
                variable1 = variable1 + 1;
                variable2 = variable2 + 1;
        }
}

void *reader(void *unused) {
        struct timeval start, end;
        gettimeofday(&start, NULL);
        unsigned i, failureCount = 0;
        for (i=0; i < ITERATIONS; i++) {
                unsigned v2 = variable2;
                unsigned v1 = variable1;
                if (v2 > v1) failureCount++;
        }
        gettimeofday(&end, NULL);
        double seconds = end.tv_sec + end.tv_usec / 1000000. - start.tv_sec - start.tv_usec / 1000000.;
        printf("%u failure%s (%2.1f percent of the time) in %2.1f seconds\n",
               failureCount, failureCount == 1 ? "" : "s",
               (100. * failureCount) / ITERATIONS, seconds);
        exit(0);
        return NULL;
}

int main(void) {
        pthread_t thread1, thread2;
        pthread_create(&thread1, NULL, writer, NULL);
        pthread_create(&thread2, NULL, reader, NULL);
        for (;;) sleep(1000000);
        return 0;
}

What do we get when we run this?

fish ) ./a.out
0 failures (0.0 percent of the time) in 0.1 seconds

So, we’re done?

Our expectations were confirmed! The writer thread ordered its writes so that the first variable would always be at least as big as the second, and the reader thread ordered its reads the opposite way to preserve that invariant, and everything worked as planned.

But we might just be getting lucky, right? I mean, if thread1 and thread2 were always scheduled on the same processor, then we wouldn’t see any failures – a processor is always self-consistent with how it appears to order reads and writes. In other words, a particular processor remembers where and what it pretends to have written, so if you read from that location with that same processor, you get what you expect. It’s only when you read with processor1 from the same address where processor2 wrote – or pretended to write – that you might get into trouble.

So let’s try to force thread1 and thread2 to run on separate processors. We can do that with the utilBindThreadToCPU() function, in the CHUD framework. That function should never go in a shipping app, but it’s useful for debugging. Here it is:

void *writer(void *unused) {
        utilBindThreadToCPU(0);
        for (;;) {
                variable1 = variable1 + 1;
                variable2 = variable2 + 1;
        }
}

void *reader(void *unused) {
        utilBindThreadToCPU(1);
        struct timeval start, end;
        gettimeofday(&start, NULL);
        ...

int main(void) {
        pthread_t thread1, thread2;
        chudInitialize();
        unsigned variable2 = 0;
        pthread_create(&thread1, NULL, writer, &variable2);
        pthread_create(&thread2, NULL, reader, &variable2);
        while (1) sleep(1000000);
        return 0;
}

To run it:

fish ) ./a.out
0 failures (0.0 percent of the time) in 0.1 seconds

NOW are we done?

Still no failures. Hmm… But wait – processors operate on cache lines, and variable1 and variable2 are right next to each other, so they probably share the same cache line – that is, they get brought in together and treated the same by each processor. What if we separate them? We’ll put one on the stack and leave the other where it is.

unsigned variable1 = 0;

#define ITERATIONS 50000000

void *writer(unsigned *variable2) {
        utilBindThreadToCPU(0);
        for (;;) {
                variable1 = variable1 + 1;
                *variable2 = *variable2 + 1;
        }
        return NULL;
}

void *reader(unsigned *variable2) {
        utilBindThreadToCPU(1);
        struct timeval start, end;
        gettimeofday(&start, NULL);
        unsigned i;
        unsigned failureCount = 0;
        for (i=0; i < ITERATIONS; i++) {
                unsigned v2 = *variable2;
                unsigned v1 = variable1;
                if (v2 > v1) failureCount++;
        }
        gettimeofday(&end, NULL);
        double seconds = end.tv_sec + end.tv_usec / 1000000. - start.tv_sec - start.tv_usec / 1000000.;
        printf("%u failure%s (%2.1f percent of the time) in %2.1f seconds\n", failureCount, failureCount == 1 ? "" : "s", (100. * failureCount) / ITERATIONS, seconds);
        exit(0);
        return NULL;
}

int main(void) {
        pthread_t thread1, thread2;
        unsigned variable2 = 0;
        chudInitialize();
        pthread_create(&thread1, NULL, writer, &variable2);
        pthread_create(&thread2, NULL, reader, &variable2);
        while (1) sleep(1000000);
        return 0;
}

So now, one variable is way high up on the stack and the other is way down low in the .data section. Does this change anything?

fish ) ./a.out
0 failures (0.0 percent of the time) in 0.1 seconds

Still nothing! I’m not going to have an article after all! Arrrghhh **BANG BANG BANG BANG**

fish ) ./a.out
0 failures (0.0 percent of the time) in 0.1 seconds
fish ) ./a.out
0 failures (0.0 percent of the time) in 0.1 seconds
fish ) ./a.out
0 failures (0.0 percent of the time) in 0.1 seconds
fish ) ./a.out
50000000 failures (100.0 percent of the time) in 0.1 seconds

Hey, there it is! Most of the time, every test passes, but that last time, every test failed.

Our Enemy the Compiler

The lesson here is something you already knew, but I’ll state it anyways: Multithreading bugs are very delicate. There is a real bug here, but it was masked by the fact that the kernel scheduled them on the same processor, and then by the fact that the variables were too close together in memory, and once those two issues were removed, (un)lucky timing usually masked the bug anyways. In fact, if I didn’t know there was a bug there, I’d never have found it – and I still have my doubts!

So first of all, why would every test pass or every test fail? If there’s a subtle timing bug, we’d expect most tests to pass, with a few failing – not all or nothing. Let’s look at what gcc is giving us for the reader function:

        lis r9,0x2fa
        ori r2,r9,61568
        mtctr r2
L8:
        bdnz L8
        lis r2,0x2fa
        ori r2,r2,61568
        mullw r2,r0,r2

Hey! The entire extent of that big long 50 million iteration loop has been hoisted out, leaving just the blue bits – essentially fifty million no-ops. Instead of adding one or zero each time through the loop, it calculates the one or zero once, and then multiplies it by 50 million.

gcc is loading from variable1 and variable2 exactly once, and comparing them exactly once, and assuming their values do not change throughout the function – which would be a fine assumption if there weren’t also other threads manipulating those variables.

This is an example of what I mentioned above, about the compiler making things difficult by optimizing so aggressively for the single threaded case.

Well, you know the punchline – to stop gcc from optimizing aggressively, you use the volatile keyword. So let’s do that:

volatile unsigned variable1 = 0;

#define ITERATIONS 50000000

void *writer(volatile unsigned *variable2) {
        utilBindThreadToCPU(0);
        for (;;) {
                variable1 = variable1 + 1;
                *variable2 = *variable2 + 1;
        }
        return NULL;
}

void *reader(volatile unsigned *variable2) {
        utilBindThreadToCPU(1);
        struct timeval start, end;
        ...

What does this change get us?

fish ) ./a.out
12462711 failures (24.9 percent of the time) in 3.7 seconds

It’s much slower (expected, since volatile defeats optimizations), but more importantly, it fails intermittently instead of all or nothing. Inspection of the assembly shows that gcc is generating the straightforward sequence of loads and stores that you’d expect.

Our Enemy the Processor

Is this really the cross-processor synchronization issues we’re trying to investigate? We can find out by binding both threads to the same CPU:

void *writer(unsigned *variable2) {
        utilBindThreadToCPU(0);
        ...

void *reader(unsigned *variable2) {
        utilBindThreadToCPU(0);
        ...

fish ) ./a.out
0 failures (0.0 percent of the time) in 0.4 seconds

The tests pass all the time – this really is a cross-processor issue.

So somehow variable2 is becoming larger than variable1 even though variable1 is always incremented first. How’s that possible? It’s possible that the writer thread, on processor 0, is writing in the wrong order – it’s writing variable2 before variable1 even though we explicitly say to write variable1 first. It’s also possible that the reader thread, on processor 1, is reading variable1 before variable 2, even though we tell it to do things in the opposite order. In other words, the processors could be reading and writing those variables in any order they feel like instead of the order we tell them to.

fish ) ./a.out
0 failures (0.0 percent of the time) in 479.5 seconds

Even the kitchen

Our problem is this: our processors are doing things in a different order than we tell them to, and not informing each other. Each processor is only keeping track of its own shenanigans! For shame! We know of two super-horrible ways to fix this: force both threads onto the same CPU, which is a very bad idea, or to use a lock, which is a very slow idea. So what’s the right way to make this work?

What we really want is a way to turn off the reordering for that particular sequence of loads and stores. They don’t call it "turn off reordering", of course, because that might imply that reordering is bad. So instead they call it just plain "ordering". We want to order the reads and writes. Ask and ye shall receive – the mechanism for that is called a "memory barrier".

And boy, does the PowerPC have them. I count at least three: sync, lwsync, and the hilariously named eieio. Here’s what they do:

• sync is the sledgehammer of the bunch – it orders all reads and writes, no matter what. It works, but it’s slow.
• lwsync (for "lightweight sync") is the newest addition. It’s limited to plain ol’ system memory, but it’s also faster than sync.
• eieio ("Enforce In-Order execution of I/O") is weird – it orders writes to "device" memory (like a memory mapped peripheral) and regular ol’ system memory, but each separately. We only care about system memory, and IBM says not to use eieio just for that. Nevertheless, it should still order our reads and writes like we want.

Because we’re not working with devices, lwsync is what we’re after. Processor 0 is writing variable2 after variable1, so we’ll insert a memory barrier to prevent that:

volatile unsigned variable1 = 0;

#define barrier() __asm__ volatile ("lwsync")

#define ITERATIONS 50000000

void *writer(volatile unsigned *variable2) {
        utilBindThreadToCPU(0);
        for (;;) {
                variable1 = variable1 + 1;
                barrier();
                *variable2 = *variable2 + 1;
        }
        return NULL;
}

So! Let’s run it!

fish ) ./a.out
260 failures (0.0 percent of the time) in 0.9 seconds

So we reduced the failure count from 12462711 to 260. Much better, but still not perfect. Why are we still failing at times? The answer, of course, is that just because processor 0 writes in the order we want is no guarantee that processor1 will read in the desired order. Processor 1 may issue the reads in the wrong order, and processor 0 would write in between those two reads. We need a memory barrier in the reader thread, to force the reads into the right order as well:

void *reader(volatile unsigned *variable2) {
        struct timeval start, end;
        utilBindThreadToCPU(1);
        gettimeofday(&start, NULL);
        unsigned i;
        unsigned failureCount = 0;
        for (i=0; i < ITERATIONS; i++) {
                unsigned v2 = *variable2;
                barrier();
                unsigned v1 = variable1;
                if (v2 > v1) failureCount++;
        }
        gettimeofday(&end, NULL);
        double seconds = end.tv_sec + end.tv_usec / 1000000. - start.tv_sec - start.tv_usec / 1000000.;
        printf("%u failure%s (%2.1f percent of the time) in %2.1f seconds\n",
               failureCount, failureCount == 1 ? "" : "s",
               (100. * failureCount) / ITERATIONS, seconds);
        exit(0);
        return NULL;
}

fish ) ./a.out
0 failures (0.0 percent of the time) in 4.2 seconds

That did it!

The lesson here is that if you care about the order of reads or writes by one thread, it’s because you care about the order of writes or reads by another thread. Both threads need a memory barrier. Memory barriers always come in pairs, or triplets or more. (Of course, if both threads are in the same function, there may only be one memory barrier that appears in your code – as long as both threads execute it.)

This should not come as a surprise: locks have the same behavior. If only one thread ever locks, it’s not a very useful lock.

31 Flavors

What’s that? You noticed that the PowerPC comes with three different kinds of memory barriers. Right – as reads and writes get scheduled increasingly out of order, the more expensive it becomes to order them – so the PowerPC allows you to request various less expensive partial orderings, for performance. Processors that schedule I/O out of order more aggressively offer even more barrier flavors. At the extreme end is the DEC Alpha, that sports read barriers with device memory ordering, read barriers without, write barriers with, write barriers without, page table barriers, and various birds in fruit trees. The Alpha’s memory model guarantees so little that it is said to define the Linux kernel memory model – that is, the set of available barriers in the kernel source match the Alpha’s instruction set. (Of course, many of them get compiled out when targetting a different processor.)

And on the other end, we have strongly ordered memory models that do very little reordering, like the – here it comes – x86. No matter how many times I run that code, even on a double-dualie Intel Mac Pro, I never saw any failures. Why not? My understanding (and here it grows fuzzy) is that early multiprocessing setups were strongly ordered because modern reordering tricks weren’t that useful – memory was still pretty fast, y’know, relatively speaking, so there wasn’t much win to be had. So developers blithely assumed the good times would never end, and we’ve been wearing the backwards compatibility shackles ever since.

But that doesn’t answer the question of why x86_64, y’know, the 64 bit x86 implementation in the Core 2s and all, isn’t more weakly ordered – or at least, reserve the right to be weaker. That’s what IA64 – remember Itanium? – did: early models were strongly ordered, but the official memory model was weak, for future proofing. Why didn’t AMD follow suit with x86_64? My only guess (emphasis on guess) is that it was a way of jockeying for position against Itanium, when the 64 bit future for the x86 was still fuzzy. AMD’s strongly ordered memory model means better compatibility and less hair-pulling when porting x86 software to 64 bit, and that made x86_64 more attractive compared to the Itanium. A pity, at least for Apple, since of course all of Apple’s software runs on the weak PowerPC – there’s no compatibility benefit to be had. So it goes. Is my theory right?

Makin’ a lock, checkin’ it twice

Ok! I think we’re ready to take on that perennial bugaboo of Java programmers – the double checked lock. How does it go again? Let’s see it in Objective-C:

+ getSharedObject {
    static id sharedObject;
    if (! sharedObject) {
        LOCK;
        if (! sharedObject) {
            sharedObject = [[self alloc] init];
        }
        UNLOCK;
    }
    return sharedObject;
}

What’s the theory? We want to create a single shared object, exactly once, while preventing conflict between multiple threads. The hope is that we can do a quick test to avoid taking the expensive lock. If the static variable is set, which it will be most of the time, we can return the object immediately, without taking the lock.

This sounds good, but of course you already know it’s not. Why not? Well, if you’re creating this object, you’re probably initializing it in some way – at the very least, you’re setting its isa (class) pointer. And then you’re turning around and writing it back to the sharedObject variable. But these can happen in any order, as seen from another processor – so when the getSharedObject method is called from some other processor, it can see the sharedObject variable as set, and happily return the object before its class pointer is even valid. Cripes.

But now you know we have the know-how to make this work, no? How? The problem is that we need to order the writes within the alloc and init methods relative to the write to the sharedObject variable – the alloc and init writes must come first, the write to sharedObject last. So we store the object into a temporary local variable, insert a memory barrier, and then copy from the temporary to the shared object. This time, I’ll use Apple’s portable memory barrier function:

+ getSharedObject {
    static id sharedObject;
    if (! sharedObject) {
        LOCK;
        if (! sharedObject) {
            id temp = [[self alloc] init];
            OSMemoryBarrier();
            sharedObject = temp;
        }
        UNLOCK;
    }
    return sharedObject;
}

There! Now we’re guaranteed that the initializing thread really will write to sharedObject after the object is fully initialized. All done.

Hmm? Oh, nuts! I forgot my rule – write barriers come in pairs. If thread A initializes the object, it goes through a memory barrier, but if thread B then comes along, it will see the object and return it without any barrier at all. Our rule tells us that something is wrong, but what? Why’s that bad?

Well, thread B’s going to do something with the shared object, like send it a message, and that requires at the very least accessing the isa class pointer. But we know the isa pointer really was written to memory first, before the sharedObject pointer, and thread B got ahold of the sharedObject pointer, so logically, the isa pointer should be written, right? The laws of physics require it! Isn’t that, like, you put an object in a box and hand it to me, and then I open the box to find that you haven’t put something into it yet! It’s a temporal paradox!

The answer is that, yes, amazingly, dependent reads like that can be performed seemingly out of order, but not on any processors that Apple ships. I’ve only heard of it happening in the – you guessed it – the Alpha. Crazy, huh?

So where should the memory barrier go? The goal is to order future reads – reads that occur after this sharedObject function returns – against the read from the sharedObject variable. So it’s gotta go here:

+ getSharedObject {
    static id sharedObject;
    if (! sharedObject) {
        LOCK;
        if (! sharedObject) {
            id temp = [[self alloc] init];
            OSMemoryBarrier();
            sharedObject = temp;
        }
        UNLOCK;
    }
    OSMemoryBarrier();
    return sharedObject;
}

Now, this differs slightly from the usual solution, which stores the static variable into a temporary in all cases. However, for the life of me I can’t figure out why that’s necessary – the placement of the second memory barrier above seems correct to me, assuming the compiler doesn’t hoist the final read of sharedObject above the memory barrier (which it shouldn’t). If I screwed it up, let me know how, please!

Do we want it?

That second memory barrier makes the double checked lock correct – but is it wise? As we discussed, it’s not technically necessary on any machine you or I are likely to encounter. And, after all, it does incur a real performance hit if we leave it in. What to do?

The Linux kernel defines a set of fine-grained barrier macros that get compiled in or out appropriately (we would want a "data dependency barrier" in that case). You could go that route, but my suggestion is to just leave a semi-standard comment to help you locate these places in the future. That will help future-proof your code, but more importantly, it forces you to reason carefully about the threading issues, and to record your thoughts. You’re more likely to get it right.

+ getSharedObject {
    static id sharedObject;
    if (! sharedObject) {
        LOCK;
        if (! sharedObject) {
            id temp = [[self alloc] init];
            OSMemoryBarrier();
            sharedObject = temp;
        }
        UNLOCK;
    }
    /* data dependency memory barrier here */
    return sharedObject;
}

Now are we done?

I think so, Mr. Subheading. Let’s see if we can summarize all this:

Locks are like tanks – powerful, slow, safe, expensive, and prone to getting you stuck.

• The compiler and the processor both conspire to defeat your threads by moving your code around! Be warned and wary! You will have to do battle with both.
• Even so, it is very easy to mask serious threading bugs. We had to work hard, even in highly contrived circumstances, to get our bug to poke its head out even occasionally.
• Ergo, testing probably won’t catch these types of bugs. That makes it more important to get it right the first time.
• Locks are the heavy tanks of threading tools – powerful, but slow and expensive, and if you’re not careful, you’ll get yourself stuck in a deadlock.
• Memory barriers are a faster, non-blocking, deadlock free alternative to locks. They take more thought, and aren’t always applicable, but your code’ll be faster and scale better.
• Memory barriers always come in logical pairs or more. Understanding where the second barrier has to go will help you reason about your code, even if that particular architecture doesn’t require a second barrier.

Further reading

Seriously? You want to know more? Ok – the best technical source I know of is actually a document called “memory-barriers.txt” that comes with the Linux kernel source. You can get it here. Thanks to my co-worker for finding it and directing me to it.

Things I wanna know

I’m still scratching my head about some things. Maybe you can help me out.

• Why is x86_64 strongly ordered? Is my theory about gaining a competitive edge over Itanium reasonable?
• Is my double checked lock right, even though it doesn’t use a temporary variable in the usual place?
• What’s up with the so-called "nontemporal" streaming instructions on x86?

Leave a comment if you know! Thanks!

td.ahleft { text-align: right; }

td.ahright { padding-top: 20px; font-size: 15pt; padding-left: 50px; line-height: 30px; text-align: left; }

tr.aheven { background-color: #DDEEFF; }

tr.ahodd { background-color: white; }

img.ahproduct { vertical-align: middle; }

	Hair care! Digital audio!
	Hair care! Digital audio!
	Hair care! Digital audio!
	Hair care! Digital audio!
	Hair care! Digital audio!
	Hair care! Digital audio!
	Hair care! Digital audio!
	Hair care! Digital audio!
	Hair care! Digital audio!
	Hair care! Digital audio!
	Hair care! Digital audio!
	Hair care! Digital audio!
	Hair care! Digital audio!
	Hair care! Digital audio!