So here’s the first episode of I Didn’t Order That, So Why Is It On My Bill, to be continued until I can’t think of any more.

Inline Functions

The C++ standard says this:

A static local variable in an extern inline function always refers to the same object.

In other words, static variables in inline functions work just like static variables in any function. That’s reasonable, because that’s probably what you want. That’s what statics are for, after all.

But wait, it also says this:

An inline function with external linkage shall have the same address in all translation units.

That’s borderline silly. Who cares what the address of the function is? When’s the last time you used the function pointer address for anything except calling it? Dollars to donuts says never. You aren’t using a function pointer as a hash table key. You aren’t comparing the same function via pointers from different files. And if you did, you’d probably have the good sense to ensure the functions are defined exactly once (i.e. not inline). You can easily live without this feature.

So hopefully I’ve established that you don’t use this feature. Now I have to show how you’re paying for it anyways. Well, this feature complicates linking. Ordinarily, there should be only one copy of a function, and if the linker finds more than one it gives you a multiple definition error. But with inline functions, the compiler is expected to generates multiple copies of the code and it’s up to the linker to sort it out. See vague linkage.

So the linker has more work to do, and link time is increased. Again, who cares? Linking occurs after compilation, and I promised you a runtime cost. But ah – link time is runtime when we’re using dynamic libraries!

Every time you start up a program, the linker has to make sure that your implementation of vector<int>::capacity() has the same address as the vector<int>::capacity() defined in libWhoGivesAHoot.dylib just in case you decide to go and compare their function pointers.

And it gets a little worse. You know how class templates have to live in header files? The function definition goes right there in the class declaration, and that makes them inline automatically. So nearly every function in a template gets this inline uniquing treatment. Every template function in a class that cannot or should not be inlined – because its address is taken, because it is recursive, because you’re optimizing for size, or simply because it’s too darn big and inlining is counterproductive – will incur a launch time cost.

The situation is dire enough that gcc added a "-fvisibility-inlines-hidden" flag. This is a minor violation of the C++ standard, but will improve launch time in cases with a lot of dynamic libraries. In other words, this flag says: I’m not using this feature, please take it off my bill.

C does not have this problem. Why not? C (in the C99 standard) has very different semantics for inline functions. To oversimplify a bit, a C99 inline function acts as sort of an alias for a real function, which must be defined once and only once. Furthermore, inline functions can’t have static variables, or have their addresses taken. This approach is more limiting, but doesn’t require the linker to bend over backwards, and so avoids the performance penalty of the C++ approach.

As any usability expert will tell you, the foundation of sound web page design is rounded corners. Their smooth curves lend an organic, almost sensual feel to a site, while other pages are un-navigable morasses of right angles.

Rounded corners are easy with tables and images! But tables make your page Web 2.0-incompatible, so most browsers cannot render it.

So when I decided to get some roundy corners for myself, I looked around, and the best I was able to find was Nifty Corners. There’s a lot to like about Nifty Corners, but look what happens if you try to make them overlap or put them on a, say, not-fixed-color background:

Click on the image to see the actual page.

Notice that the corners of the boxes suffer. The problem is that the area around the corner cannot be drawn transparently – it needs to be over a fixed color. Browser specific extensions would be perfect if they weren’t browser specific.

But I like crazy backgrounds and overlapping boxes. So in a half-hearted effort for geek cred, here’s how my roundy corners worked.

Hey! Flexible widths!

But but but…

• Isn’t that, like, a humongous flood of markup? Won’t the page take a long time to download?

Well, I worried about that too. But happily, all modern browsers support gzip compression – and boy does it work! Even though my last posting with comments is 382 KB in size, you only need to download 19.8 KB (Mac users can look in Safari’s Activity Viewer for the download size). So it ends up being smaller than, say, Yahoo!’s home page – and that’s just the HTML, not even counting all of its images. As a result, the page loads quite fast.

• Well, don’t all those divs make the page unmaintainable?

Yeah, I suppose so. I use a script to generate these boxes, so the source is readable to me – even if the generated HTML isn’t.

On another thread, I’ve fixed the Angband screensaver to remember the last character it played with. The user defaults had to be synchronized. Thanks to everyone who pointed this out.

Guest blogger: Gollum (Smeagol)

Where isst it? So it iss back – so fissh, the fish, raw and wriggling, he hasst brought its back!

Angband, the Hells of Iron, yest, the ancient ASCII roguelike, child spawn of Moria and VMS, it iss here once more. We wants it!

You hasst not seen it, Angband? But perhapss you have seen one like it, the filthy, filthy NetHack, yess, full of such stupids and jokeses, and no Smeagol! We hates it for ever! But we loves Angband, yess, Angband has Smeagol and the fat hobbitses and yes, it has my preciouss! We loves it, Angband!

We hass thought it lost. When OS X was a little child and hungry for software, we knows how it calls to fissh. We knows how fissh missed Angband, dearly missed Smeagol, yes. And we knows how fissh stoles it, and worked his tricksy little magic, so he could go down into a Carbonized Angband on OS X. And we knows fissh did, and it was easy, Carbon madess it simple.

But fissh hasst brought it to Cocoa, now, all fresh and it glitterses so subpixel pretty with Quartz, and resizesss so smooth, and animates with pretty graphics! So precious to fissh.

And ssuch love for Angband so fissh made a borg screensaveres! So now you can visits Smeagol and wring the filthy little neck of Saruman and murderes the fat Morgoth in your sleep! The screensaveres, yes!

Goes there now, fat hobbitses! The game and the screensaveres and the source codess! Angband need you, yess!

Angband for Mac OS X

Edit: fissh has fixed the problem with the screensaveres needing Angband to have been launched first. It should work no problems now.

"That’s a lot of cores. And while 80-core floating point monsters like that aren’t likely to show up in an iMac any time soon, multicore chips in multiprocessor computers are here today. Single chip machines are so 2004. Programmers better get crackin’. The megahertz free ride is over – and we have work to do."

There, that’s the noisome little pitch everyone’s been spreading like so much thermal paste. As if multiprocessing is something new! But of course it’s not – heck, I remember Apple shipping dualies more than ten years ago, as the Power Macintosh 9500. Multiprocessing is more accessible (read: cheaper) now, but it’s anything save new. It’s been around long enough that we should have it figured out by now.

So what’s my excuse? I admit it – I don’t get multiprocessing, not, you know, really get it, and that’s gone on long enough. It’s time to get to the bottom of it – or if not to the bottom, at least deep enough that my ears start popping.

Threadinology

Where to start, where to start…well, let’s define our terms. Ok, here’s the things I mean when I say the following, uh, things:

• Threads are just preemptively scheduled contexts of execution that share an address space. But you already know what threads are. Frankly, for my purposes, they’re all pretty much the same whether you’re using Objective-C or C++ or Java on Mac OS X or Linux or Windows…
• Threading means creating multiple threads. But you often create multiple threads for simpler control flow or to get around blocking system calls, not to improve performance through true simultaneous execution.
• Multithreading is the physically simultaneous execution of multiple threads for increased performance, which requires a dualie or more. Now things get hard.

Yeah, I know. "Multithreading is hard" is a cliché, and it bugs me, because it is not some truism describing a fundamental property of nature, but it’s something we did. We made multithreading hard because we optimized so heavily for the single threaded case.

What do I mean? Well, processor speeds outrun memory so much that we started guessing at what’s in memory so the processor doesn’t have to waste time checking. "Guessing" is a loaded term; a nicer phrase might be "make increasingly aggressive assumptions" about the state of memory. And by "we," I mean both the compiler and the processor – both make things hard, as we’ll see. We’ll figure this stuff out – but they’re going to try to confuse us. Oh well. Right or wrong, this is the bed we’ve made, and now we get to lie in it.

Blah blah. Let’s look at some code. We have two variables, variable1 and variable2, that both start out at 0. The writer thread will do this:

Writer thread

while (1) {
   variable1++;
   variable2++;
}

They both started out at zero, so therefore variable1, at every point in time, will always be the same as variable2, or larger – but never smaller. Right?

The reader thread will do this:

Reader thread

while (1) {
   local2 = variable2;
   local1 = variable1;
   if (local2 > local1) {
      print("Error!");
   }
}

That’s odd – why does the reader thread load the second variable before the first? That’s the opposite order from the writer thread! But it makes sense if you think about it.

See, it’s possible that variable1 and/or variable2 will be incremented by the writer thread between the loads from the reader thread. If variable2 gets incremented, that doesn’t matter – variable2 has already been read. If variable1 gets incremented, then that makes variable1 appear larger. So we conclude that variable2 should never be seen as larger than variable1, in the reader thread. If we loaded variable1 before variable2, then variable2 might be incremented after the load of variable1, and we would see variable 2 as larger.

Threadalogy

So we’ve got two threads operating on two variables, and we think we know what’ll happen. Let’s try it out, on my G5:

unsigned variable1 = 0;
unsigned variable2 = 0;

#define ITERATIONS 50000000

void *writer(void *unused) {
        for (;;) {
                variable1 = variable1 + 1;
                variable2 = variable2 + 1;
        }
}

void *reader(void *unused) {
        struct timeval start, end;
        gettimeofday(&start, NULL);
        unsigned i, failureCount = 0;
        for (i=0; i < ITERATIONS; i++) {
                unsigned v2 = variable2;
                unsigned v1 = variable1;
                if (v2 > v1) failureCount++;
        }
        gettimeofday(&end, NULL);
        double seconds = end.tv_sec + end.tv_usec / 1000000. - start.tv_sec - start.tv_usec / 1000000.;
        printf("%u failure%s (%2.1f percent of the time) in %2.1f seconds\n",
               failureCount, failureCount == 1 ? "" : "s",
               (100. * failureCount) / ITERATIONS, seconds);
        exit(0);
        return NULL;
}

int main(void) {
        pthread_t thread1, thread2;
        pthread_create(&thread1, NULL, writer, NULL);
        pthread_create(&thread2, NULL, reader, NULL);
        for (;;) sleep(1000000);
        return 0;
}

What do we get when we run this?

fish ) ./a.out
0 failures (0.0 percent of the time) in 0.1 seconds

So, we’re done?

Our expectations were confirmed! The writer thread ordered its writes so that the first variable would always be at least as big as the second, and the reader thread ordered its reads the opposite way to preserve that invariant, and everything worked as planned.

But we might just be getting lucky, right? I mean, if thread1 and thread2 were always scheduled on the same processor, then we wouldn’t see any failures – a processor is always self-consistent with how it appears to order reads and writes. In other words, a particular processor remembers where and what it pretends to have written, so if you read from that location with that same processor, you get what you expect. It’s only when you read with processor1 from the same address where processor2 wrote – or pretended to write – that you might get into trouble.

So let’s try to force thread1 and thread2 to run on separate processors. We can do that with the utilBindThreadToCPU() function, in the CHUD framework. That function should never go in a shipping app, but it’s useful for debugging. Here it is:

void *writer(void *unused) {
        utilBindThreadToCPU(0);
        for (;;) {
                variable1 = variable1 + 1;
                variable2 = variable2 + 1;
        }
}

void *reader(void *unused) {
        utilBindThreadToCPU(1);
        struct timeval start, end;
        gettimeofday(&start, NULL);
        ...

int main(void) {
        pthread_t thread1, thread2;
        chudInitialize();
        unsigned variable2 = 0;
        pthread_create(&thread1, NULL, writer, &variable2);
        pthread_create(&thread2, NULL, reader, &variable2);
        while (1) sleep(1000000);
        return 0;
}

To run it:

fish ) ./a.out
0 failures (0.0 percent of the time) in 0.1 seconds

NOW are we done?

Still no failures. Hmm… But wait – processors operate on cache lines, and variable1 and variable2 are right next to each other, so they probably share the same cache line – that is, they get brought in together and treated the same by each processor. What if we separate them? We’ll put one on the stack and leave the other where it is.

unsigned variable1 = 0;

#define ITERATIONS 50000000

void *writer(unsigned *variable2) {
        utilBindThreadToCPU(0);
        for (;;) {
                variable1 = variable1 + 1;
                *variable2 = *variable2 + 1;
        }
        return NULL;
}

void *reader(unsigned *variable2) {
        utilBindThreadToCPU(1);
        struct timeval start, end;
        gettimeofday(&start, NULL);
        unsigned i;
        unsigned failureCount = 0;
        for (i=0; i < ITERATIONS; i++) {
                unsigned v2 = *variable2;
                unsigned v1 = variable1;
                if (v2 > v1) failureCount++;
        }
        gettimeofday(&end, NULL);
        double seconds = end.tv_sec + end.tv_usec / 1000000. - start.tv_sec - start.tv_usec / 1000000.;
        printf("%u failure%s (%2.1f percent of the time) in %2.1f seconds\n", failureCount, failureCount == 1 ? "" : "s", (100. * failureCount) / ITERATIONS, seconds);
        exit(0);
        return NULL;
}

int main(void) {
        pthread_t thread1, thread2;
        unsigned variable2 = 0;
        chudInitialize();
        pthread_create(&thread1, NULL, writer, &variable2);
        pthread_create(&thread2, NULL, reader, &variable2);
        while (1) sleep(1000000);
        return 0;
}

So now, one variable is way high up on the stack and the other is way down low in the .data section. Does this change anything?

fish ) ./a.out
0 failures (0.0 percent of the time) in 0.1 seconds

Still nothing! I’m not going to have an article after all! Arrrghhh **BANG BANG BANG BANG**

fish ) ./a.out
0 failures (0.0 percent of the time) in 0.1 seconds
fish ) ./a.out
0 failures (0.0 percent of the time) in 0.1 seconds
fish ) ./a.out
0 failures (0.0 percent of the time) in 0.1 seconds
fish ) ./a.out
50000000 failures (100.0 percent of the time) in 0.1 seconds

Hey, there it is! Most of the time, every test passes, but that last time, every test failed.

Our Enemy the Compiler

The lesson here is something you already knew, but I’ll state it anyways: Multithreading bugs are very delicate. There is a real bug here, but it was masked by the fact that the kernel scheduled them on the same processor, and then by the fact that the variables were too close together in memory, and once those two issues were removed, (un)lucky timing usually masked the bug anyways. In fact, if I didn’t know there was a bug there, I’d never have found it – and I still have my doubts!

So first of all, why would every test pass or every test fail? If there’s a subtle timing bug, we’d expect most tests to pass, with a few failing – not all or nothing. Let’s look at what gcc is giving us for the reader function:

        lis r9,0x2fa
        ori r2,r9,61568
        mtctr r2
L8:
        bdnz L8
        lis r2,0x2fa
        ori r2,r2,61568
        mullw r2,r0,r2

Hey! The entire extent of that big long 50 million iteration loop has been hoisted out, leaving just the blue bits – essentially fifty million no-ops. Instead of adding one or zero each time through the loop, it calculates the one or zero once, and then multiplies it by 50 million.

gcc is loading from variable1 and variable2 exactly once, and comparing them exactly once, and assuming their values do not change throughout the function – which would be a fine assumption if there weren’t also other threads manipulating those variables.

This is an example of what I mentioned above, about the compiler making things difficult by optimizing so aggressively for the single threaded case.

Well, you know the punchline – to stop gcc from optimizing aggressively, you use the volatile keyword. So let’s do that:

volatile unsigned variable1 = 0;

#define ITERATIONS 50000000

void *writer(volatile unsigned *variable2) {
        utilBindThreadToCPU(0);
        for (;;) {
                variable1 = variable1 + 1;
                *variable2 = *variable2 + 1;
        }
        return NULL;
}

void *reader(volatile unsigned *variable2) {
        utilBindThreadToCPU(1);
        struct timeval start, end;
        ...

What does this change get us?

fish ) ./a.out
12462711 failures (24.9 percent of the time) in 3.7 seconds

It’s much slower (expected, since volatile defeats optimizations), but more importantly, it fails intermittently instead of all or nothing. Inspection of the assembly shows that gcc is generating the straightforward sequence of loads and stores that you’d expect.

Our Enemy the Processor

Is this really the cross-processor synchronization issues we’re trying to investigate? We can find out by binding both threads to the same CPU:

void *writer(unsigned *variable2) {
        utilBindThreadToCPU(0);
        ...

void *reader(unsigned *variable2) {
        utilBindThreadToCPU(0);
        ...

fish ) ./a.out
0 failures (0.0 percent of the time) in 0.4 seconds

The tests pass all the time – this really is a cross-processor issue.

So somehow variable2 is becoming larger than variable1 even though variable1 is always incremented first. How’s that possible? It’s possible that the writer thread, on processor 0, is writing in the wrong order – it’s writing variable2 before variable1 even though we explicitly say to write variable1 first. It’s also possible that the reader thread, on processor 1, is reading variable1 before variable 2, even though we tell it to do things in the opposite order. In other words, the processors could be reading and writing those variables in any order they feel like instead of the order we tell them to.

fish ) ./a.out
0 failures (0.0 percent of the time) in 479.5 seconds

Even the kitchen

Our problem is this: our processors are doing things in a different order than we tell them to, and not informing each other. Each processor is only keeping track of its own shenanigans! For shame! We know of two super-horrible ways to fix this: force both threads onto the same CPU, which is a very bad idea, or to use a lock, which is a very slow idea. So what’s the right way to make this work?

What we really want is a way to turn off the reordering for that particular sequence of loads and stores. They don’t call it "turn off reordering", of course, because that might imply that reordering is bad. So instead they call it just plain "ordering". We want to order the reads and writes. Ask and ye shall receive – the mechanism for that is called a "memory barrier".

And boy, does the PowerPC have them. I count at least three: sync, lwsync, and the hilariously named eieio. Here’s what they do:

• sync is the sledgehammer of the bunch – it orders all reads and writes, no matter what. It works, but it’s slow.
• lwsync (for "lightweight sync") is the newest addition. It’s limited to plain ol’ system memory, but it’s also faster than sync.
• eieio ("Enforce In-Order execution of I/O") is weird – it orders writes to "device" memory (like a memory mapped peripheral) and regular ol’ system memory, but each separately. We only care about system memory, and IBM says not to use eieio just for that. Nevertheless, it should still order our reads and writes like we want.

Because we’re not working with devices, lwsync is what we’re after. Processor 0 is writing variable2 after variable1, so we’ll insert a memory barrier to prevent that:

volatile unsigned variable1 = 0;

#define barrier() __asm__ volatile ("lwsync")

#define ITERATIONS 50000000

void *writer(volatile unsigned *variable2) {
        utilBindThreadToCPU(0);
        for (;;) {
                variable1 = variable1 + 1;
                barrier();
                *variable2 = *variable2 + 1;
        }
        return NULL;
}

So! Let’s run it!

fish ) ./a.out
260 failures (0.0 percent of the time) in 0.9 seconds

So we reduced the failure count from 12462711 to 260. Much better, but still not perfect. Why are we still failing at times? The answer, of course, is that just because processor 0 writes in the order we want is no guarantee that processor1 will read in the desired order. Processor 1 may issue the reads in the wrong order, and processor 0 would write in between those two reads. We need a memory barrier in the reader thread, to force the reads into the right order as well:

void *reader(volatile unsigned *variable2) {
        struct timeval start, end;
        utilBindThreadToCPU(1);
        gettimeofday(&start, NULL);
        unsigned i;
        unsigned failureCount = 0;
        for (i=0; i < ITERATIONS; i++) {
                unsigned v2 = *variable2;
                barrier();
                unsigned v1 = variable1;
                if (v2 > v1) failureCount++;
        }
        gettimeofday(&end, NULL);
        double seconds = end.tv_sec + end.tv_usec / 1000000. - start.tv_sec - start.tv_usec / 1000000.;
        printf("%u failure%s (%2.1f percent of the time) in %2.1f seconds\n",
               failureCount, failureCount == 1 ? "" : "s",
               (100. * failureCount) / ITERATIONS, seconds);
        exit(0);
        return NULL;
}

fish ) ./a.out
0 failures (0.0 percent of the time) in 4.2 seconds

That did it!

The lesson here is that if you care about the order of reads or writes by one thread, it’s because you care about the order of writes or reads by another thread. Both threads need a memory barrier. Memory barriers always come in pairs, or triplets or more. (Of course, if both threads are in the same function, there may only be one memory barrier that appears in your code – as long as both threads execute it.)

This should not come as a surprise: locks have the same behavior. If only one thread ever locks, it’s not a very useful lock.

31 Flavors

What’s that? You noticed that the PowerPC comes with three different kinds of memory barriers. Right – as reads and writes get scheduled increasingly out of order, the more expensive it becomes to order them – so the PowerPC allows you to request various less expensive partial orderings, for performance. Processors that schedule I/O out of order more aggressively offer even more barrier flavors. At the extreme end is the DEC Alpha, that sports read barriers with device memory ordering, read barriers without, write barriers with, write barriers without, page table barriers, and various birds in fruit trees. The Alpha’s memory model guarantees so little that it is said to define the Linux kernel memory model – that is, the set of available barriers in the kernel source match the Alpha’s instruction set. (Of course, many of them get compiled out when targetting a different processor.)

And on the other end, we have strongly ordered memory models that do very little reordering, like the – here it comes – x86. No matter how many times I run that code, even on a double-dualie Intel Mac Pro, I never saw any failures. Why not? My understanding (and here it grows fuzzy) is that early multiprocessing setups were strongly ordered because modern reordering tricks weren’t that useful – memory was still pretty fast, y’know, relatively speaking, so there wasn’t much win to be had. So developers blithely assumed the good times would never end, and we’ve been wearing the backwards compatibility shackles ever since.

But that doesn’t answer the question of why x86_64, y’know, the 64 bit x86 implementation in the Core 2s and all, isn’t more weakly ordered – or at least, reserve the right to be weaker. That’s what IA64 – remember Itanium? – did: early models were strongly ordered, but the official memory model was weak, for future proofing. Why didn’t AMD follow suit with x86_64? My only guess (emphasis on guess) is that it was a way of jockeying for position against Itanium, when the 64 bit future for the x86 was still fuzzy. AMD’s strongly ordered memory model means better compatibility and less hair-pulling when porting x86 software to 64 bit, and that made x86_64 more attractive compared to the Itanium. A pity, at least for Apple, since of course all of Apple’s software runs on the weak PowerPC – there’s no compatibility benefit to be had. So it goes. Is my theory right?

Makin’ a lock, checkin’ it twice

Ok! I think we’re ready to take on that perennial bugaboo of Java programmers – the double checked lock. How does it go again? Let’s see it in Objective-C:

+ getSharedObject {
    static id sharedObject;
    if (! sharedObject) {
        LOCK;
        if (! sharedObject) {
            sharedObject = [[self alloc] init];
        }
        UNLOCK;
    }
    return sharedObject;
}

What’s the theory? We want to create a single shared object, exactly once, while preventing conflict between multiple threads. The hope is that we can do a quick test to avoid taking the expensive lock. If the static variable is set, which it will be most of the time, we can return the object immediately, without taking the lock.

This sounds good, but of course you already know it’s not. Why not? Well, if you’re creating this object, you’re probably initializing it in some way – at the very least, you’re setting its isa (class) pointer. And then you’re turning around and writing it back to the sharedObject variable. But these can happen in any order, as seen from another processor – so when the getSharedObject method is called from some other processor, it can see the sharedObject variable as set, and happily return the object before its class pointer is even valid. Cripes.

But now you know we have the know-how to make this work, no? How? The problem is that we need to order the writes within the alloc and init methods relative to the write to the sharedObject variable – the alloc and init writes must come first, the write to sharedObject last. So we store the object into a temporary local variable, insert a memory barrier, and then copy from the temporary to the shared object. This time, I’ll use Apple’s portable memory barrier function:

+ getSharedObject {
    static id sharedObject;
    if (! sharedObject) {
        LOCK;
        if (! sharedObject) {
            id temp = [[self alloc] init];
            OSMemoryBarrier();
            sharedObject = temp;
        }
        UNLOCK;
    }
    return sharedObject;
}

There! Now we’re guaranteed that the initializing thread really will write to sharedObject after the object is fully initialized. All done.

Hmm? Oh, nuts! I forgot my rule – write barriers come in pairs. If thread A initializes the object, it goes through a memory barrier, but if thread B then comes along, it will see the object and return it without any barrier at all. Our rule tells us that something is wrong, but what? Why’s that bad?

Well, thread B’s going to do something with the shared object, like send it a message, and that requires at the very least accessing the isa class pointer. But we know the isa pointer really was written to memory first, before the sharedObject pointer, and thread B got ahold of the sharedObject pointer, so logically, the isa pointer should be written, right? The laws of physics require it! Isn’t that, like, you put an object in a box and hand it to me, and then I open the box to find that you haven’t put something into it yet! It’s a temporal paradox!

The answer is that, yes, amazingly, dependent reads like that can be performed seemingly out of order, but not on any processors that Apple ships. I’ve only heard of it happening in the – you guessed it – the Alpha. Crazy, huh?

So where should the memory barrier go? The goal is to order future reads – reads that occur after this sharedObject function returns – against the read from the sharedObject variable. So it’s gotta go here:

+ getSharedObject {
    static id sharedObject;
    if (! sharedObject) {
        LOCK;
        if (! sharedObject) {
            id temp = [[self alloc] init];
            OSMemoryBarrier();
            sharedObject = temp;
        }
        UNLOCK;
    }
    OSMemoryBarrier();
    return sharedObject;
}

Now, this differs slightly from the usual solution, which stores the static variable into a temporary in all cases. However, for the life of me I can’t figure out why that’s necessary – the placement of the second memory barrier above seems correct to me, assuming the compiler doesn’t hoist the final read of sharedObject above the memory barrier (which it shouldn’t). If I screwed it up, let me know how, please!

Do we want it?

That second memory barrier makes the double checked lock correct – but is it wise? As we discussed, it’s not technically necessary on any machine you or I are likely to encounter. And, after all, it does incur a real performance hit if we leave it in. What to do?

The Linux kernel defines a set of fine-grained barrier macros that get compiled in or out appropriately (we would want a "data dependency barrier" in that case). You could go that route, but my suggestion is to just leave a semi-standard comment to help you locate these places in the future. That will help future-proof your code, but more importantly, it forces you to reason carefully about the threading issues, and to record your thoughts. You’re more likely to get it right.

+ getSharedObject {
    static id sharedObject;
    if (! sharedObject) {
        LOCK;
        if (! sharedObject) {
            id temp = [[self alloc] init];
            OSMemoryBarrier();
            sharedObject = temp;
        }
        UNLOCK;
    }
    /* data dependency memory barrier here */
    return sharedObject;
}

Now are we done?

I think so, Mr. Subheading. Let’s see if we can summarize all this:

Locks are like tanks – powerful, slow, safe, expensive, and prone to getting you stuck.

• The compiler and the processor both conspire to defeat your threads by moving your code around! Be warned and wary! You will have to do battle with both.
• Even so, it is very easy to mask serious threading bugs. We had to work hard, even in highly contrived circumstances, to get our bug to poke its head out even occasionally.
• Ergo, testing probably won’t catch these types of bugs. That makes it more important to get it right the first time.
• Locks are the heavy tanks of threading tools – powerful, but slow and expensive, and if you’re not careful, you’ll get yourself stuck in a deadlock.
• Memory barriers are a faster, non-blocking, deadlock free alternative to locks. They take more thought, and aren’t always applicable, but your code’ll be faster and scale better.
• Memory barriers always come in logical pairs or more. Understanding where the second barrier has to go will help you reason about your code, even if that particular architecture doesn’t require a second barrier.

Further reading

Seriously? You want to know more? Ok – the best technical source I know of is actually a document called “memory-barriers.txt” that comes with the Linux kernel source. You can get it here. Thanks to my co-worker for finding it and directing me to it.

Things I wanna know

I’m still scratching my head about some things. Maybe you can help me out.

• Why is x86_64 strongly ordered? Is my theory about gaining a competitive edge over Itanium reasonable?
• Is my double checked lock right, even though it doesn’t use a temporary variable in the usual place?
• What’s up with the so-called "nontemporal" streaming instructions on x86?

Leave a comment if you know! Thanks!

td.ahleft { text-align: right; }

td.ahright { padding-top: 20px; font-size: 15pt; padding-left: 50px; line-height: 30px; text-align: left; }

tr.aheven { background-color: #DDEEFF; }

tr.ahodd { background-color: white; }

img.ahproduct { vertical-align: middle; }

	Hair care! Digital audio!
	Hair care! Digital audio!
	Hair care! Digital audio!
	Hair care! Digital audio!
	Hair care! Digital audio!
	Hair care! Digital audio!
	Hair care! Digital audio!
	Hair care! Digital audio!
	Hair care! Digital audio!
	Hair care! Digital audio!
	Hair care! Digital audio!
	Hair care! Digital audio!
	Hair care! Digital audio!
	Hair care! Digital audio!

I hope you find Hex Fiend useful for whatever purpose, but if you are interested in contributing changes on an ongoing basis, I’ll be happy to grant Subversion commit privileges to some interested developers who submit quality patches. There is a wiki aimed at developers accessible from the page, but daily builds, mailing lists, or discussion boards are also a possibility. You can contact me at the e-mail address at the bottom of the Hex Fiend page if you are interested in any of these.

Version 1.1.1 has some important bug fixes (see the release notes), so you should upgrade even if you are not interested in the source.

Mac OS 9 and NEXTSTEP were like two icy comets wandering aimlessly in space. Neither was really going anywhere in particular. And then, BANG! They collide, stick wetly together, spinning wildly! Thus was born Mac OS X – or so the legend goes.

How do you take these two comets, err, operating systems, and make a unified OS out of them? On the one hand, you have the procedural classic Macintosh Toolbox, and on the other you have object oriented OPENSTEP, as different as can be – and you’re tasked with integrating them, or at least getting some level of interoperability. What a headache!

You might start by finding common ground – but there isn’t much common ground, so you have to invent some, and you call it (well, part of it) CoreFoundation. Uh, let’s abbreviate CoreFoundation "CF" from now on. CoreFoundation will "sit below" both of these APIs, and provide functions for strings and dates and other fundamental stuff, and the shared use of CF will serve as a sort of least common demoninator, not only for these two APIs but also for future APIs. These two APIs will be able to talk to each other, and to future APIs, with CF types.

Ok, so the plan is to make these two APIs, the Mac Toolbox and OPENSTEP, use CF. Adding CF support to the Mac Toolbox is not that big a deal, because the Mac Toolbox APIs have to change anyways, to become Carbon. But the OPENSTEP APIs don’t have the same sort of problems, and shouldn’t have to change much to become Cocoa.

Like, for example, the Toolbox uses Pascal strings, and those have unfortunate length limitations and ignorance of Unicode, so we want to get rid of them – so we might as well use the interoperable replacement as the native string type in Carbon. But OPENSTEP’s NSString is already pretty nice. It would be a shame to have to make CFString replacements for all those Cocoa APIs that take and return NSStrings, just for interoperability with Carbon.

Rough Draft

So the solution is obvious, right? Just make NSString methods to convert to and from CFStrings.

@interface NSString (CFStringMethods)
- (CFStringRef)getCFString;
+ (NSString *)stringFromCFString:(CFStringRef)stringRef;
@end

So whenever you want to talk to Carbon, you get a CFStringRef from your NSString, and whenever you get a CFStringRef back from Carbon, you make an NSString out of it. Simple! But this isn’t what Apple did.

Second Revision

"Hey," you say. Some of you say. "I know what Apple did. I’m not so easily fooled! Check out this code:"

	#include <Foundation/NSString.h>

	int main(void) {
		NSLog(NSStringFromClass([@"Some String" class]));
		return 0;
	}

"What does that output? NSCFString. NSCFString. See? NSStrings must be really CFStrings under the hood! And you can do that because NSString is a class cluster – it’s an abstract interface. So that’s how you achieve interoperability: you implement NSStrings with CFStrings (but preserve the NSString API) and then all NSStrings really *are* CFStrings. There’s no conversion necessary because they’re the same thing.

"That’s how toll free bridging works!"

But hang on a minute. You just said yourself that NSString is an abstract interface – that means that some crazy developer can make his or her own own subclass of NSString, and implement its methods in whatever wacky way, and it’s supposed to just work. But then it wouldn’t be using CFStrings! It would be using some other crazy stuff. So when a Cocoa API gets a string and wants to do something CF-ish with it, the API would have no way of knowing if the string was toll-free bridged – that is, if it was really a CFString or a, y’know, FishsWackyString, without checking its class, and then it would have to convert it…blech!

Final Draft

So that’s a problem: Apple wants to toll free bridge – to be able to use NSStrings as CFStrings without conversion. But to do that, Apple also needs to support wacky NSString subclasses (that don’t use CFStrings at all) in the CFString API. That means making a C API that knows about Objective-C objects.

A C API that handles Objective-C objects? That’s some deep deep voodoo, man. But we have it and it works, right? We can just cast CFStringRefs to NSStrings, and vice versa, and for once in our lives we get to feel smug and superior, instead of stupid, when the compiler warns about mistmatched pointer types. "Look, gcc, I know it says CFStringRef, but just try it with that NSString. Trust me." It’s great! Right?

But how does it work? We could check, if only CoreFoundation were open source!

…

Oh, right. So let’s look at the CFStringGetLength() function and see what happens if you give it a weird string.

	CFIndex CFStringGetLength(CFStringRef str) {
	    CF_OBJC_FUNCDISPATCH0(__kCFStringTypeID, CFIndex, str, "length");

	    __CFAssertIsString(str);
	    return __CFStrLength(str);
	}

Any ideas where the Objective-C voodoo is happening here? ANYONE? You in the back? CF_OBJC_FUNCDISPATCH0 you say? I guess it’s worth a try.

CF_OBJC_FUNCDISPATCH0

So CF_OBJC_FUNCDISPATCH0 is the magic that supports Objective-C objects. Where’s CF_OBJC_FUNCDISPATCH0 defined? Here:

	// Invoke an ObjC method, return the result
	#define CF_OBJC_FUNCDISPATCH0(typeID, rettype, obj, sel) \
		if (__builtin_expect(CF_IS_OBJC(typeID, obj), 0)) \
		{rettype (*func)(const void *, SEL) = (void *)__CFSendObjCMsg; \
		static SEL s = NULL; if (!s) s = sel_registerName(sel); \
		return func((const void *)obj, s);}

Yikes! Let’s piece that apart:

	if (__builtin_expect(CF_IS_OBJC(typeID, obj), 0))

If we’re really an Objective-C object…

	rettype (*func)(const void *, SEL) = (void *)__CFSendObjCMsg;

…treat the function __CFSendObjCMsg as if it takes the same arguments as a parameterless Objective-C method (that is, just self and _cmd)…

	static SEL s = NULL; if (!s) s = sel_registerName(sel);

…look up the selector by name (and stash it in a static variable so we only have to do it once per selector)…

	return func((const void *)obj, s);

…and then call that __CFSendObjCMsg() function. What does __CFSendObjCMsg() do?

	#define __CFSendObjCMsg 0xfffeff00

0xfffeff00? What the heck? Oh, wait, that’s just the commpage address of objc_msgSend_rtp(). So __CFSendObjCMsg() is just good ol’ objc_msgSend().

CF_IS_OBJC

That leaves us with __builtin_expect(CF_IS_OBJC(typeID, obj), 0), the function that tries to figure out if we’re an Objective-C object or not. What does that do?

__builtin_expect() is just some gcc magic for branch prediction – here it means that we should expect CF_IS_OBJC to be false. That is, CF believes that most of its calls will be on CF objects instead of Objective-C objects. Ok, fair enough. But what does CF_IS_OBJC actually do? Take a look.

	CF_INLINE int CF_IS_OBJC(CFTypeID typeID, const void *obj) {
	    return (((CFRuntimeBase *)obj)->_isa != __CFISAForTypeID(typeID) && ((CFRuntimeBase *)obj)->_isa > (void *)0xFFF);
	}

(Keen observers might notice that this code is #ifdefed out in favor of:

	#define CF_IS_OBJC(typeID, obj) (false)

I believe this is for the benefit of people who want to use CF on Linux or other OSes, who aren’t interested in toll-free bridging and therefore don’t want to pay any performance penalty for it.)

Ok! There’s two parts to seeing if we’re an Objective-C object – we check (with a quick table lookup) whether our isa (class) pointer indicates that we "really are" a certain CF type, and if we’re not, we check to see if our class pointer is greater than 0xFFFF, and if it is, we’re an Objective-C object, and we call through to the Objective-C dispatch mechanism – in this case, we send the length message.

Summary

What are the consequences of all that? Well!

• CF objects, just like Objective-C objects, all have an isa pointer (except it’s called _isa in CF). It’s right there in struct __CFRuntimeBase.
• There are two toll-free bridging mechanisms! Some Objective-C objects "really are" CF objects – the memory layout between the Objective-C object and the corresponding CF object is identical (enabled in part by the presence of the _isa pointer above), and in that case the Objective-C methods are not invoked by the CF functions. For example, in this code:

  	CFStringGetLength([NSString stringWithCString:"Hello World!"]);
  

  There, -[NSString stringWithCString:] is returning an NSCFString (which you can verify by asking it for the name of its class), but -[NSCFString length] is never invoked – NO length method is invoked. You can verify that with gdb. Objects that "really are" their CF equivalents skip what’s usually thought of as the bridge, and "fall through" to the CF functions even when the CF functions are directly called on them. Obviously, this is an implementation detail, and you should not depend on this.

• That mechanism is also how bridging works the other way – how CF strings you get from, say, Carbon, can be passed around like Objective-C objects, because they really are Objective-C objects. The bridges are implemented entirely in CF and in the bridged classes – the Objective-C runtime is blissfully unaware.
• But! Plain ol’ Objective-C objects are sussed out by CF by checking to see if their class pointer is larger than 0xFFFF, and if so, ordinary Objective-C message dispatch is used from the CF functions. That’s the second toll-free bridging mechanism, and it must be present in every public CF function for a bridged object, except for features not supported Cocoa-side.
• Nowhere do we depend on the abstract class NSString at all – the bridge doesn’t check for it and Objective-C doesn’t care about it. That means that, in theory, CFStringGetLength() should "work" (invoke the length method) on any object, not just NSStrings. Does it? You can check it yourself. (Answer: yes!) Obviously, this is just an artifact of the implementation, and you should definitely not depend on this – only subclasses of NSString are supported by toll free bridging to CFString.
• Curiously, other "true" CF objects are not considered to be CF objects by this macro. For example,

  	CFStringGetLength([NSArray array]);
  

  will raise an exception because NSCFArray does not implement length. That is, CF_IS_OBJC is not asking "Are you a CF type?" but rather "Are you this specific CF type?" That should make you happy, because it raises a "selector not recognized" exception instead of crashing, which makes our code more debuggable. Thanks, CF!

• Why 0xFFFF? I’m glad you (I mean I) asked, since the answer (at least, what I think it is) has interesting connections to NULL. But that will have to wait until a future post.

Other approaches

My boss pointed out that there are other ways to achieve toll-free bridging, beyond what CF does. The simplest is to write your API with Objective-C and then wrap it with C:

	@implementation Array

	- (int)length {
	    return self->length;
	}

	@end

	int getLength(ArrayRef array) {
		return [(id)array length];
	}

You can even retrofit toll-free bridging onto an existing C API by wrapping it twice – first in Objective-C, then in C, and the "outer" C layer becomes the public C API. To wit:

	/* private length function that we want to wrap */
	static int privateGetLength(ArrayRef someArray) {
	   return someArray->length;
	}

	/* public ObjC API */
	@implementation Array

	- (int)length {
	   return privateGetLength(self->arrayRef);
	}

	@end

	/* public C API */
	int getLength(ArrayRef array) {
	   return [(id)array length];
	}

The point of that double feint, of course, is for the public C API to respect overrides of the length method by subclasses.

"Wrapping" up

So toll-free bridging is (one way) that Cocoa integrates with Carbon and even newer OS X APIs. It’s possible in large part because of Objective-C, but in this case, Apple gets as much mileage from the simple runtime implementation and C API as from its dynamic nature. You already knew that, I’ll bet – but hopefully you have a better idea of how it all works.

Now hands off! A coworker of mine makes the point that good developers distinguish between what they pretend to know and what they really know. The, uh, known knowns, and the known unknowns, as it were. The mechanism of toll-free bridging is not secret (it is open source, after all), but it is private, which means that you are encouraged to know about it but to refrain from depending on it. Use it for, say, debugging, but don’t ship apps that depend on it – because that prevents Apple from making OS X better. And nobody wants that! I mean the prevention part.

• Horizontal resizing
• Custom fonts
• Overwrite mode
• Hidden files
• Lots more goodies (see release notes)

Hex Fiend 1.1

May you find it useful!

"I’m going to beat grep by thirty percent!" I confidently crow to anyone who would listen, those foolish enough to enter my office. And my girlfriend too, who’s contractually obligated to pay attention to everything I say.

See, I was working on Hex Fiend, and searching was dog slow. But Hex Fiend is supposed to be fast, and I want blazingly quick search that leaves the bewildered competition coughing in trails of dust. And, as everyone knows, the best way to get amazing results is to set arbitrary goals without any basis for believing they can be reached. So I set out to search faster than grep by thirty percent.

The first step in any potentially impossible project is, of course, to announce that you are on the verge of succeeding.

I imagine the author of grep, Ultimate Unix Geek, squinting at vi; the glow of a dozen xterms is the only light to fall on his ample frame covered by overalls, cheese doodles, and a tangle of beard. Discarded crushed Mountain Dew cans litter the floor. I look straight into the back of his head, covered by a snarl of greasy locks, and reply with a snarl of my own: You’re mine. The aphorism at the top, like the ex girlfriend who first told it to me, is dim in my recollection.

String searching

Having exhausted all my trash-talking avenues, it’s time to get to work. Now, everyone knows that without some sort of preflighting, the fastest string search you can do still takes linear time. Since my program is supposed to work on dozens of gigabytes, preflighting is impossible – there’s no place to put all the data that preflighting generates, and nobody wants to sit around while I generate it. So I am resigned to the linear algorithms. The best known is Boyer-Moore (I won’t insult your intelligence with a Wikipedia link, but the article there gives a good overview).

Boyer-Moore works like this: you have some string you’re looking for, which we’ll call the needle, and some string you want to find it in, which we’ll call the haystack. Instead of starting the search at the beginning of needle, you start at the end. If your needle character doesn’t match the character you’re looking at in haystack, you can move needle forwards in haystack until haystack’s mismatched character lines up with the same character in needle. If haystack’s mismatch isn’t in needle at all, then you can skip ahead a whole needle’s length.

For example, if you’re searching for a string of 100 ‘a’s (needle), you look at the 100th character in haystack. If it’s an ‘x’, well, ‘x’ doesn’t appear anywhere in needle, so you can skip ahead all of needle and look at the 200th character in haystack. A single mismatch allowed us to skip 100 characters!

I get shot down

For performance, the number of characters you can skip on a mismatch is usually stored in an array indexed by the character value. So the first part of my Boyer-Moore string searching algorithm looked like this:

char haystack_char = haystack[haystack_index];
if (last_char_in_needle != haystack_char)
   haystack_index += jump_table[haystack_char];

So we look at the character in haystack and if it’s not what we’re looking for, we jump ahead by the right distance for that character, which is in jump_table.

"There," I sigh, finishing and sitting back. It may not be faster than grep, but it should be at least as fast, because this is the fastest algorithm known. This should be a good start. So I confidently ran my benchmark, for a 1 gigabyte file…

grep:	2.52 seconds
Hex Fiend:	3.86 seconds

Ouch. I’m slower, more than 50% slower. grep is leaving me sucking dust. Ultimate Unix Geek chuckles into his xterms.

Rollin', rollin', rollin'

My eyes darken, my vision tunnels. I break out the big guns. My efforts to vectorize are fruitless (I’m not clever enough to vectorize Boyer-Moore because it has very linear data dependencies.) Shark shows a lot of branching, suggesting I can do better by unrolling the loop. Indeed:

grep:	2.52 seconds
Hex Fiend (unrolled):	2.68 seconds

But I was still more than 6% slower, and that’s as fast as I got. Exhausted, stymied at every turn, I throw up my hands. grep has won.

grep's dark secret

"How do you do it, Ultimate Unix Geek? How is grep so fast?" I moan at last, crawling forwards into the pale light of his CRT.

"Hmmm," he mumbles. "I suppose you have earned a villian’s exposition. Behold!" A blaze of keyboard strokes later and grep’s source code is smeared in green-on-black across the screen.

while (tp < = ep)
	  {
	    d = d1[U(tp[-1])], tp += d;
	    d = d1[U(tp[-1])], tp += d;
	    if (d == 0)
	      goto found;
	    d = d1[U(tp[-1])], tp += d;
	    d = d1[U(tp[-1])], tp += d;
	    d = d1[U(tp[-1])], tp += d;
	    if (d == 0)
	      goto found;
	    d = d1[U(tp[-1])], tp += d;
	    d = d1[U(tp[-1])], tp += d;
	    d = d1[U(tp[-1])], tp += d;
	    if (d == 0)
	      goto found;
	    d = d1[U(tp[-1])], tp += d;
	    d = d1[U(tp[-1])], tp += d;
	  }

"You bastard!" I shriek, amazed at what I see. "You sold them out!"

See all those d = d1[U(tp[-1])], tp += d; lines? Well, d1 is the jump table, and it so happens that grep puts 0 in the jump table for the last character in needle. So when grep looks up the jump distance for the character, via haystack_index += jump_table[haystack_char], well, if haystack_char is the last character in needle (meaning we have a potential match), then jump_table[haystack_char] is 0, so that line doesn't actually increase haystack_index.

All that is fine and noble. But do not be fooled! If the characters match, the search location doesn't change - so grep assumes there is no match, up to three times in a row, before checking to see if it actually found a match.

Put another way, grep sells out its worst case (lots of partial matches) to make the best case (few partial matches) go faster. How treacherous! As this realization dawns on me, the room seemed to grow dim and slip sideways. I look up at the Ultimate Unix Geek, spinning slowly in his padded chair, and I hear his cackle "old age and treachery...", and in his flickering CRT there is a face reflected, but it's my ex girlfriend, and the last thing I see before I black out is a patch of yellow cheese powder inside her long tangled beard.

I take a page from grep

"Damn you," I mumble at last, rising from my prostrate position. Chagrined and humbled, I copy the technique.

grep:	2.52 seconds
Hex Fiend (treacherous):	2.46 seconds

What's the win?

Copying that trick brought me from six percent slower to two percent faster, but at what cost? What penalty has grep paid for this treachery? Let us check - we shall make a one gigabyte file with one thousand x's per line, and time grep searching for "yy" (a two character best case) and "yx" (a two character worst case). Then we'll send grep to Over-Optimizers Anonymous and compare how a reformed grep (one that checks for a match after every character) performs.

Innnnteresting. The treacherous optimization does indeed squeeze out almost 8% faster searching in the best case, at a cost of nearly 70% slower searching in the worst case. Worth it? You decide! Let me know what you think.

Resolved and refreshed, I plan my next entry. This isn't over, Ultimate Unix Geek.

Disclaimers

(Note: I have never met the authors or maintainers of grep. I'm sure they're all well balanced clean shaven beer and coffee drinkers.)

(Oh, and the released version of HexFiend will be slightly slower in this case, because of an overly large buffer that blows the cache. In other situations, the story is different, but more about those in a future post.)