On another thread, I’ve fixed the Angband screensaver to remember the last character it played with. The user defaults had to be synchronized. Thanks to everyone who pointed this out.

Where isst it? So it iss back - so fissh, the fish, raw and wriggling, he hasst brought its back!

Angband, the Hells of Iron, yest, the ancient ASCII roguelike, child spawn of Moria and VMS, it iss here once more. We wants it!

You hasst not seen it, Angband? But perhapss you have seen one like it, the filthy, filthy NetHack, yess, full of such stupids and jokeses, and no Smeagol! We hates it for ever! But we loves Angband, yess, Angband has Smeagol and the fat hobbitses and yes, it has my preciouss! We loves it, Angband!

We hass thought it lost. When OS X was a little child and hungry for software, we knows how it calls to fissh. We knows how fissh missed Angband, dearly missed Smeagol, yes. And we knows how fissh stoles it, and worked his tricksy little magic, so he could go down into a Carbonized Angband on OS X. And we knows fissh did, and it was easy, Carbon madess it simple.

But fissh hasst brought it to Cocoa, now, all fresh and it glitterses so subpixel pretty with Quartz, and resizesss so smooth, and animates with pretty graphics! So precious to fissh.

And ssuch love for Angband so fissh made a borg screensaveres! So now you can visits Smeagol and wring the filthy little neck of Saruman and murderes the fat Morgoth in your sleep! The screensaveres, yes!

Goes there now, fat hobbitses! The game and the screensaveres and the source codess! Angband need you, yess!

Angband for Mac OS X

Edit: fissh has fixed the problem with the screensaveres needing Angband to have been launched first. It should work no problems now.]]> http://ridiculousfish.com/blog/archives/2007/04/13/42/feed/ http://ridiculousfish.com/blog/archives/2007/02/17/barrier/ http://ridiculousfish.com/blog/archives/2007/02/17/barrier/#comments Sat, 17 Feb 2007 22:24:04 +0000 ridiculous_fish Programming http://ridiculousfish.com/blog/archives/2007/02/17/barrier/

"That’s a lot of cores. And while 80-core floating point monsters like that aren’t likely to show up in an iMac any time soon, multicore chips in multiprocessor computers are here today. Single chip machines are so 2004. Programmers better get crackin’. The megahertz free ride is over - and we have work to do."

There, that’s the noisome little pitch everyone’s been spreading like so much thermal paste. As if multiprocessing is something new! But of course it’s not - heck, I remember Apple shipping dualies more than ten years ago, as the Power Macintosh 9500. Multiprocessing is more accessible (read: cheaper) now, but it’s anything save new. It’s been around long enough that we should have it figured out by now.

So what’s my excuse? I admit it - I don’t get multiprocessing, not, you know, really get it, and that’s gone on long enough. It’s time to get to the bottom of it - or if not to the bottom, at least deep enough that my ears start popping.

Threadinology

Where to start, where to start...well, let’s define our terms. Ok, here’s the things I mean when I say the following, uh, things:

• Threads are just preemptively scheduled contexts of execution that share an address space. But you already know what threads are. Frankly, for my purposes, they’re all pretty much the same whether you’re using Objective-C or C++ or Java on Mac OS X or Linux or Windows...
• Threading means creating multiple threads. But you often create multiple threads for simpler control flow or to get around blocking system calls, not to improve performance through true simultaneous execution.
• Multithreading is the physically simultaneous execution of multiple threads for increased performance, which requires a dualie or more. Now things get hard.

Yeah, I know. "Multithreading is hard" is a cliché, and it bugs me, because it is not some truism describing a fundamental property of nature, but it’s something we did. We made multithreading hard because we optimized so heavily for the single threaded case.

What do I mean? Well, processor speeds outrun memory so much that we started guessing at what’s in memory so the processor doesn’t have to waste time checking. "Guessing" is a loaded term; a nicer phrase might be "make increasingly aggressive assumptions" about the state of memory. And by "we," I mean both the compiler and the processor - both make things hard, as we’ll see. We’ll figure this stuff out - but they’re going to try to confuse us. Oh well. Right or wrong, this is the bed we’ve made, and now we get to lie in it.

Blah blah. Let’s look at some code. We have two variables, variable1 and variable2, that both start out at 0. The writer thread will do this:

Writer thread

while (1) {
   variable1++;
   variable2++;
}

The reader thread will do this:

Reader thread

while (1) {
   local2 = variable2;
   local1 = variable1;
   if (local2 > local1) {
      print("Error!");
   }
}

That’s odd - why does the reader thread load the second variable before the first? That’s the opposite order from the writer thread! But it makes sense if you think about it.

See, it’s possible that variable1 and/or variable2 will be incremented by the writer thread between the loads from the reader thread. If variable2 gets incremented, that doesn’t matter - variable2 has already been read. If variable1 gets incremented, then that makes variable1 appear larger. So we conclude that variable2 should never be seen as larger than variable1, in the reader thread. If we loaded variable1 before variable2, then variable2 might be incremented after the load of variable1, and we would see variable 2 as larger.

Threadalogy

So we’ve got two threads operating on two variables, and we think we know what’ll happen. Let’s try it out, on my G5:

unsigned variable1 = 0;
unsigned variable2 = 0;

#define ITERATIONS 50000000

void *writer(void *unused) {
        for (;;) {
                variable1 = variable1 + 1;
                variable2 = variable2 + 1;
        }
}

void *reader(void *unused) {
        struct timeval start, end;
        gettimeofday(&start, NULL);
        unsigned i, failureCount = 0;
        for (i=0; i < ITERATIONS; i++) {
                unsigned v2 = variable2;
                unsigned v1 = variable1;
                if (v2 > v1) failureCount++;
        }
        gettimeofday(&end, NULL);
        double seconds = end.tv_sec + end.tv_usec / 1000000. - start.tv_sec - start.tv_usec / 1000000.;
        printf(”%u failure%s (%2.1f percent of the time) in %2.1f seconds\n”,
               failureCount, failureCount == 1 ? “” : “s”,
               (100. * failureCount) / ITERATIONS, seconds);
        exit(0);
        return NULL;
}

int main(void) {
        pthread_t thread1, thread2;
        pthread_create(&thread1, NULL, writer, NULL);
        pthread_create(&thread2, NULL, reader, NULL);
        for (;;) sleep(1000000);
        return 0;
}

fish ) ./a.out
0 failures (0.0 percent of the time) in 0.1 seconds

So, we’re done?

Our expectations were confirmed! The writer thread ordered its writes so that the first variable would always be at least as big as the second, and the reader thread ordered its reads the opposite way to preserve that invariant, and everything worked as planned.

But we might just be getting lucky, right? I mean, if thread1 and thread2 were always scheduled on the same processor, then we wouldn’t see any failures - a processor is always self-consistent with how it appears to order reads and writes. In other words, a particular processor remembers where and what it pretends to have written, so if you read from that location with that same processor, you get what you expect. It’s only when you read with processor1 from the same address where processor2 wrote - or pretended to write - that you might get into trouble.

So let’s try to force thread1 and thread2 to run on separate processors. We can do that with the utilBindThreadToCPU() function, in the CHUD framework. That function should never go in a shipping app, but it’s useful for debugging. Here it is:

void *writer(void *unused) {
        utilBindThreadToCPU(0);
        for (;;) {
                variable1 = variable1 + 1;
                variable2 = variable2 + 1;
        }
}

void *reader(void *unused) {
        utilBindThreadToCPU(1);
        struct timeval start, end;
        gettimeofday(&start, NULL);
        ...
        
int main(void) {
        pthread_t thread1, thread2;
        chudInitialize();
        unsigned variable2 = 0;
        pthread_create(&thread1, NULL, writer, &variable2);
        pthread_create(&thread2, NULL, reader, &variable2);
        while (1) sleep(1000000);
        return 0;
}

fish ) ./a.out
0 failures (0.0 percent of the time) in 0.1 seconds

NOW are we done?

Still no failures. Hmm... But wait - processors operate on cache lines, and variable1 and variable2 are right next to each other, so they probably share the same cache line - that is, they get brought in together and treated the same by each processor. What if we separate them? We’ll put one on the stack and leave the other where it is.

unsigned variable1 = 0;

#define ITERATIONS 50000000

void *writer(unsigned *variable2) {
        utilBindThreadToCPU(0);
        for (;;) {
                variable1 = variable1 + 1;
                *variable2 = *variable2 + 1;
        }
        return NULL;
}

void *reader(unsigned *variable2) {
        utilBindThreadToCPU(1);
        struct timeval start, end;
        gettimeofday(&start, NULL);
        unsigned i;
        unsigned failureCount = 0;
        for (i=0; i < ITERATIONS; i++) {
                unsigned v2 = *variable2;
                unsigned v1 = variable1;
                if (v2 > v1) failureCount++;
        }
        gettimeofday(&end, NULL);
        double seconds = end.tv_sec + end.tv_usec / 1000000. - start.tv_sec - start.tv_usec / 1000000.;
        printf(”%u failure%s (%2.1f percent of the time) in %2.1f seconds\n”, failureCount, failureCount == 1 ? “” : “s”, (100. * failureCount) / ITERATIONS, seconds);
        exit(0);
        return NULL;
}

int main(void) {
        pthread_t thread1, thread2;
        unsigned variable2 = 0;
        chudInitialize();
        pthread_create(&thread1, NULL, writer, &variable2);
        pthread_create(&thread2, NULL, reader, &variable2);
        while (1) sleep(1000000);
        return 0;
}

fish ) ./a.out
0 failures (0.0 percent of the time) in 0.1 seconds

fish ) ./a.out
0 failures (0.0 percent of the time) in 0.1 seconds
fish ) ./a.out
0 failures (0.0 percent of the time) in 0.1 seconds
fish ) ./a.out
0 failures (0.0 percent of the time) in 0.1 seconds
fish ) ./a.out
50000000 failures (100.0 percent of the time) in 0.1 seconds

Hey, there it is! Most of the time, every test passes, but that last time, every test failed.

Our Enemy the Compiler

The lesson here is something you already knew, but I’ll state it anyways: Multithreading bugs are very delicate. There is a real bug here, but it was masked by the fact that the kernel scheduled them on the same processor, and then by the fact that the variables were too close together in memory, and once those two issues were removed, (un)lucky timing usually masked the bug anyways. In fact, if I didn’t know there was a bug there, I’d never have found it - and I still have my doubts!

So first of all, why would every test pass or every test fail? If there’s a subtle timing bug, we’d expect most tests to pass, with a few failing - not all or nothing. Let’s look at what gcc is giving us for the reader function:

        lis r9,0x2fa
        ori r2,r9,61568
        mtctr r2
L8:
        bdnz L8
        lis r2,0x2fa
        ori r2,r2,61568
        mullw r2,r0,r2

Hey! The entire extent of that big long 50 million iteration loop has been hoisted out, leaving just the blue bits - essentially fifty million no-ops. Instead of adding one or zero each time through the loop, it calculates the one or zero once, and then multiplies it by 50 million.

gcc is loading from variable1 and variable2 exactly once, and comparing them exactly once, and assuming their values do not change throughout the function - which would be a fine assumption if there weren’t also other threads manipulating those variables.

This is an example of what I mentioned above, about the compiler making things difficult by optimizing so aggressively for the single threaded case.

Well, you know the punchline - to stop gcc from optimizing aggressively, you use the volatile keyword. So let’s do that:

volatile unsigned variable1 = 0;

#define ITERATIONS 50000000

void *writer(volatile unsigned *variable2) {
        utilBindThreadToCPU(0);
        for (;;) {
                variable1 = variable1 + 1;
                *variable2 = *variable2 + 1;
        }
        return NULL;
}

void *reader(volatile unsigned *variable2) {
        utilBindThreadToCPU(1);
        struct timeval start, end;
        ...

fish ) ./a.out
12462711 failures (24.9 percent of the time) in 3.7 seconds

It’s much slower (expected, since volatile defeats optimizations), but more importantly, it fails intermittently instead of all or nothing. Inspection of the assembly shows that gcc is generating the straightforward sequence of loads and stores that you’d expect.

Our Enemy the Processor

Is this really the cross-processor synchronization issues we’re trying to investigate? We can find out by binding both threads to the same CPU:

void *writer(unsigned *variable2) {
        utilBindThreadToCPU(0);
        ...

void *reader(unsigned *variable2) {
        utilBindThreadToCPU(0);
        ...

fish ) ./a.out
0 failures (0.0 percent of the time) in 0.4 seconds

The tests pass all the time - this really is a cross-processor issue.

So somehow variable2 is becoming larger than variable1 even though variable1 is always incremented first. How’s that possible? It’s possible that the writer thread, on processor 0, is writing in the wrong order - it’s writing variable2 before variable1 even though we explicitly say to write variable1 first. It’s also possible that the reader thread, on processor 1, is reading variable1 before variable 2, even though we tell it to do things in the opposite order. In other words, the processors could be reading and writing those variables in any order they feel like instead of the order we tell them to.

fish ) ./a.out
0 failures (0.0 percent of the time) in 479.5 seconds

Even the kitchen

Our problem is this: our processors are doing things in a different order than we tell them to, and not informing each other. Each processor is only keeping track of its own shenanigans! For shame! We know of two super-horrible ways to fix this: force both threads onto the same CPU, which is a very bad idea, or to use a lock, which is a very slow idea. So what’s the right way to make this work?

What we really want is a way to turn off the reordering for that particular sequence of loads and stores. They don’t call it "turn off reordering", of course, because that might imply that reordering is bad. So instead they call it just plain "ordering". We want to order the reads and writes. Ask and ye shall receive - the mechanism for that is called a "memory barrier".

And boy, does the PowerPC have them. I count at least three: sync, lwsync, and the hilariously named eieio. Here’s what they do:

• sync is the sledgehammer of the bunch - it orders all reads and writes, no matter what. It works, but it’s slow.
• lwsync (for "lightweight sync") is the newest addition. It’s limited to plain ol’ system memory, but it’s also faster than sync.
• eieio ("Enforce In-Order execution of I/O") is weird - it orders writes to "device" memory (like a memory mapped peripheral) and regular ol’ system memory, but each separately. We only care about system memory, and IBM says not to use eieio just for that. Nevertheless, it should still order our reads and writes like we want.

Because we’re not working with devices, lwsync is what we’re after. Processor 0 is writing variable2 after variable1, so we’ll insert a memory barrier to prevent that:

volatile unsigned variable1 = 0;

#define barrier() __asm__ volatile (”lwsync”)

#define ITERATIONS 50000000

void *writer(volatile unsigned *variable2) {
        utilBindThreadToCPU(0);
        for (;;) {
                variable1 = variable1 + 1;
                barrier();
                *variable2 = *variable2 + 1;
        }
        return NULL;
}

So! Let’s run it!

fish ) ./a.out
260 failures (0.0 percent of the time) in 0.9 seconds

So we reduced the failure count from 12462711 to 260. Much better, but still not perfect. Why are we still failing at times? The answer, of course, is that just because processor 0 writes in the order we want is no guarantee that processor1 will read in the desired order. Processor 1 may issue the reads in the wrong order, and processor 0 would write in between those two reads. We need a memory barrier in the reader thread, to force the reads into the right order as well:

void *reader(volatile unsigned *variable2) {
        struct timeval start, end;
        utilBindThreadToCPU(1);
        gettimeofday(&start, NULL);
        unsigned i;
        unsigned failureCount = 0;
        for (i=0; i < ITERATIONS; i++) {
                unsigned v2 = *variable2;
                barrier();
                unsigned v1 = variable1;
                if (v2 > v1) failureCount++;
        }
        gettimeofday(&end, NULL);
        double seconds = end.tv_sec + end.tv_usec / 1000000. - start.tv_sec - start.tv_usec / 1000000.;
        printf(”%u failure%s (%2.1f percent of the time) in %2.1f seconds\n”,
               failureCount, failureCount == 1 ? “” : “s”,
               (100. * failureCount) / ITERATIONS, seconds);
        exit(0);
        return NULL;
}

fish ) ./a.out
0 failures (0.0 percent of the time) in 4.2 seconds

That did it!

The lesson here is that if you care about the order of reads or writes by one thread, it’s because you care about the order of writes or reads by another thread. Both threads need a memory barrier. Memory barriers always come in pairs, or triplets or more. (Of course, if both threads are in the same function, there may only be one memory barrier that appears in your code - as long as both threads execute it.)

This should not come as a surprise: locks have the same behavior. If only one thread ever locks, it’s not a very useful lock.

31 Flavors

What’s that? You noticed that the PowerPC comes with three different kinds of memory barriers. Right - as reads and writes get scheduled increasingly out of order, the more expensive it becomes to order them - so the PowerPC allows you to request various less expensive partial orderings, for performance. Processors that schedule I/O out of order more aggressively offer even more barrier flavors. At the extreme end is the DEC Alpha, that sports read barriers with device memory ordering, read barriers without, write barriers with, write barriers without, page table barriers, and various birds in fruit trees. The Alpha’s memory model guarantees so little that it is said to define the Linux kernel memory model - that is, the set of available barriers in the kernel source match the Alpha’s instruction set. (Of course, many of them get compiled out when targetting a different processor.)

And on the other end, we have strongly ordered memory models that do very little reordering, like the - here it comes - x86. No matter how many times I run that code, even on a double-dualie Intel Mac Pro, I never saw any failures. Why not? My understanding (and here it grows fuzzy) is that early multiprocessing setups were strongly ordered because modern reordering tricks weren’t that useful - memory was still pretty fast, y’know, relatively speaking, so there wasn’t much win to be had. So developers blithely assumed the good times would never end, and we’ve been wearing the backwards compatibility shackles ever since.

But that doesn’t answer the question of why x86_64, y’know, the 64 bit x86 implementation in the Core 2s and all, isn’t more weakly ordered - or at least, reserve the right to be weaker. That’s what IA64 - remember Itanium? - did: early models were strongly ordered, but the official memory model was weak, for future proofing. Why didn’t AMD follow suit with x86_64? My only guess (emphasis on guess) is that it was a way of jockeying for position against Itanium, when the 64 bit future for the x86 was still fuzzy. AMD’s strongly ordered memory model means better compatibility and less hair-pulling when porting x86 software to 64 bit, and that made x86_64 more attractive compared to the Itanium. A pity, at least for Apple, since of course all of Apple’s software runs on the weak PowerPC - there’s no compatibility benefit to be had. So it goes. Is my theory right?

Makin’ a lock, checkin’ it twice

Ok! I think we’re ready to take on that perennial bugaboo of Java programmers - the double checked lock. How does it go again? Let’s see it in Objective-C:

+ getSharedObject {
    static id sharedObject;
    if (! sharedObject) {
        LOCK;
        if (! sharedObject) {
            sharedObject = [[self alloc] init];
        }
        UNLOCK;
    }
    return sharedObject;
}

What’s the theory? We want to create a single shared object, exactly once, while preventing conflict between multiple threads. The hope is that we can do a quick test to avoid taking the expensive lock. If the static variable is set, which it will be most of the time, we can return the object immediately, without taking the lock.

This sounds good, but of course you already know it’s not. Why not? Well, if you’re creating this object, you’re probably initializing it in some way - at the very least, you’re setting its isa (class) pointer. And then you’re turning around and writing it back to the sharedObject variable. But these can happen in any order, as seen from another processor - so when the getSharedObject method is called from some other processor, it can see the sharedObject variable as set, and happily return the object before its class pointer is even valid. Cripes.

But now you know we have the know-how to make this work, no? How? The problem is that we need to order the writes within the alloc and init methods relative to the write to the sharedObject variable - the alloc and init writes must come first, the write to sharedObject last. So we store the object into a temporary local variable, insert a memory barrier, and then copy from the temporary to the shared object. This time, I’ll use Apple’s portable memory barrier function:

+ getSharedObject {
    static id sharedObject;
    if (! sharedObject) {
        LOCK;
        if (! sharedObject) {
            id temp = [[self alloc] init];
            OSMemoryBarrier();
            sharedObject = temp;
        }
        UNLOCK;
    }
    return sharedObject;
}

There! Now we’re guaranteed that the initializing thread really will write to sharedObject after the object is fully initialized. All done.

Hmm? Oh, nuts! I forgot my rule - write barriers come in pairs. If thread A initializes the object, it goes through a memory barrier, but if thread B then comes along, it will see the object and return it without any barrier at all. Our rule tells us that something is wrong, but what? Why’s that bad?

Well, thread B’s going to do something with the shared object, like send it a message, and that requires at the very least accessing the isa class pointer. But we know the isa pointer really was written to memory first, before the sharedObject pointer, and thread B got ahold of the sharedObject pointer, so logically, the isa pointer should be written, right? The laws of physics require it! Isn’t that, like, you put an object in a box and hand it to me, and then I open the box to find that you haven’t put something into it yet! It’s a temporal paradox!

The answer is that, yes, amazingly, dependent reads like that can be performed seemingly out of order, but not on any processors that Apple ships. I’ve only heard of it happening in the - you guessed it - the Alpha. Crazy, huh?

So where should the memory barrier go? The goal is to order future reads - reads that occur after this sharedObject function returns - against the read from the sharedObject variable. So it’s gotta go here:

+ getSharedObject {
    static id sharedObject;
    if (! sharedObject) {
        LOCK;
        if (! sharedObject) {
            id temp = [[self alloc] init];
            OSMemoryBarrier();
            sharedObject = temp;
        }
        UNLOCK;
    }
    OSMemoryBarrier();
    return sharedObject;
}

Now, this differs slightly from the usual solution, which stores the static variable into a temporary in all cases. However, for the life of me I can’t figure out why that’s necessary - the placement of the second memory barrier above seems correct to me, assuming the compiler doesn’t hoist the final read of sharedObject above the memory barrier (which it shouldn’t). If I screwed it up, let me know how, please!

Do we want it?

That second memory barrier makes the double checked lock correct - but is it wise? As we discussed, it’s not technically necessary on any machine you or I are likely to encounter. And, after all, it does incur a real performance hit if we leave it in. What to do?

The Linux kernel defines a set of fine-grained barrier macros that get compiled in or out appropriately (we would want a "data dependency barrier" in that case). You could go that route, but my suggestion is to just leave a semi-standard comment to help you locate these places in the future. That will help future-proof your code, but more importantly, it forces you to reason carefully about the threading issues, and to record your thoughts. You’re more likely to get it right.

+ getSharedObject {
    static id sharedObject;
    if (! sharedObject) {
        LOCK;
        if (! sharedObject) {
            id temp = [[self alloc] init];
            OSMemoryBarrier();
            sharedObject = temp;
        }
        UNLOCK;
    }
    /* data dependency memory barrier here */
    return sharedObject;
}

Now are we done?

I think so, Mr. Subheading. Let’s see if we can summarize all this:

Locks are like tanks - powerful, slow, safe, expensive, and prone to getting you stuck.

• The compiler and the processor both conspire to defeat your threads by moving your code around! Be warned and wary! You will have to do battle with both.
• Even so, it is very easy to mask serious threading bugs. We had to work hard, even in highly contrived circumstances, to get our bug to poke its head out even occasionally.
• Ergo, testing probably won’t catch these types of bugs. That makes it more important to get it right the first time.
• Locks are the heavy tanks of threading tools - powerful, but slow and expensive, and if you’re not careful, you’ll get yourself stuck in a deadlock.
• Memory barriers are a faster, non-blocking, deadlock free alternative to locks. They take more thought, and aren’t always applicable, but your code’ll be faster and scale better.
• Memory barriers always come in logical pairs or more. Understanding where the second barrier has to go will help you reason about your code, even if that particular architecture doesn’t require a second barrier.

Further reading

Things I wanna know

• Why is x86_64 strongly ordered? Is my theory about gaining a competitive edge over Itanium reasonable?
• Is my double checked lock right, even though it doesn’t use a temporary variable in the usual place?
• What’s up with the so-called "nontemporal" streaming instructions on x86?

Mac OS 9 and NEXTSTEP were like two icy comets wandering aimlessly in space. Neither was really going anywhere in particular. And then, BANG! They collide, stick wetly together, spinning wildly! Thus was born Mac OS X - or so the legend goes.

How do you take these two comets, err, operating systems, and make a unified OS out of them? On the one hand, you have the procedural classic Macintosh Toolbox, and on the other you have object oriented OPENSTEP, as different as can be - and you’re tasked with integrating them, or at least getting some level of interoperability. What a headache!

You might start by finding common ground - but there isn’t much common ground, so you have to invent some, and you call it (well, part of it) CoreFoundation. Uh, let’s abbreviate CoreFoundation "CF" from now on. CoreFoundation will "sit below" both of these APIs, and provide functions for strings and dates and other fundamental stuff, and the shared use of CF will serve as a sort of least common demoninator, not only for these two APIs but also for future APIs. These two APIs will be able to talk to each other, and to future APIs, with CF types.

Ok, so the plan is to make these two APIs, the Mac Toolbox and OPENSTEP, use CF. Adding CF support to the Mac Toolbox is not that big a deal, because the Mac Toolbox APIs have to change anyways, to become Carbon. But the OPENSTEP APIs don’t have the same sort of problems, and shouldn’t have to change much to become Cocoa.

Like, for example, the Toolbox uses Pascal strings, and those have unfortunate length limitations and ignorance of Unicode, so we want to get rid of them - so we might as well use the interoperable replacement as the native string type in Carbon. But OPENSTEP’s NSString is already pretty nice. It would be a shame to have to make CFString replacements for all those Cocoa APIs that take and return NSStrings, just for interoperability with Carbon.

Rough Draft

So the solution is obvious, right? Just make NSString methods to convert to and from CFStrings.

@interface NSString (CFStringMethods)
- (CFStringRef)getCFString;
+ (NSString *)stringFromCFString:(CFStringRef)stringRef;
@end

So whenever you want to talk to Carbon, you get a CFStringRef from your NSString, and whenever you get a CFStringRef back from Carbon, you make an NSString out of it. Simple! But this isn’t what Apple did.

Second Revision

	#include <Foundation/NSString.h>
	
	int main(void) {
		NSLog(NSStringFromClass([@”Some String” class]));
		return 0;
	}

"That’s how toll free bridging works!"

But hang on a minute. You just said yourself that NSString is an abstract interface - that means that some crazy developer can make his or her own own subclass of NSString, and implement its methods in whatever wacky way, and it’s supposed to just work. But then it wouldn’t be using CFStrings! It would be using some other crazy stuff. So when a Cocoa API gets a string and wants to do something CF-ish with it, the API would have no way of knowing if the string was toll-free bridged - that is, if it was really a CFString or a, y’know, FishsWackyString, without checking its class, and then it would have to convert it...blech!

Final Draft

So that’s a problem: Apple wants to toll free bridge - to be able to use NSStrings as CFStrings without conversion. But to do that, Apple also needs to support wacky NSString subclasses (that don’t use CFStrings at all) in the CFString API. That means making a C API that knows about Objective-C objects.

A C API that handles Objective-C objects? That’s some deep deep voodoo, man. But we have it and it works, right? We can just cast CFStringRefs to NSStrings, and vice versa, and for once in our lives we get to feel smug and superior, instead of stupid, when the compiler warns about mistmatched pointer types. "Look, gcc, I know it says CFStringRef, but just try it with that NSString. Trust me." It’s great! Right?

But how does it work? We could check, if only CoreFoundation were open source!

...

Oh, right. So let’s look at the CFStringGetLength() function and see what happens if you give it a weird string.

	CFIndex CFStringGetLength(CFStringRef str) {
	    CF_OBJC_FUNCDISPATCH0(__kCFStringTypeID, CFIndex, str, “length”);
	
	    __CFAssertIsString(str);
	    return __CFStrLength(str);
	}

CF_OBJC_FUNCDISPATCH0

	// Invoke an ObjC method, return the result
	#define CF_OBJC_FUNCDISPATCH0(typeID, rettype, obj, sel) \
		if (__builtin_expect(CF_IS_OBJC(typeID, obj), 0)) \
		{rettype (*func)(const void *, SEL) = (void *)__CFSendObjCMsg; \
		static SEL s = NULL; if (!s) s = sel_registerName(sel); \
		return func((const void *)obj, s);}

Yikes! Let’s piece that apart:

	if (__builtin_expect(CF_IS_OBJC(typeID, obj), 0))

	rettype (*func)(const void *, SEL) = (void *)__CFSendObjCMsg;

	static SEL s = NULL; if (!s) s = sel_registerName(sel);

	return func((const void *)obj, s);

	#define __CFSendObjCMsg 0xfffeff00

CF_IS_OBJC

That leaves us with __builtin_expect(CF_IS_OBJC(typeID, obj), 0), the function that tries to figure out if we’re an Objective-C object or not. What does that do?

__builtin_expect() is just some gcc magic for branch prediction - here it means that we should expect CF_IS_OBJC to be false. That is, CF believes that most of its calls will be on CF objects instead of Objective-C objects. Ok, fair enough. But what does CF_IS_OBJC actually do? Take a look.

	CF_INLINE int CF_IS_OBJC(CFTypeID typeID, const void *obj) {
	    return (((CFRuntimeBase *)obj)->_isa != __CFISAForTypeID(typeID) && ((CFRuntimeBase *)obj)->_isa > (void *)0xFFF);
	}

	#define CF_IS_OBJC(typeID, obj) (false)

Ok! There’s two parts to seeing if we’re an Objective-C object - we check (with a quick table lookup) whether our isa (class) pointer indicates that we "really are" a certain CF type, and if we’re not, we check to see if our class pointer is greater than 0xFFFF, and if it is, we’re an Objective-C object, and we call through to the Objective-C dispatch mechanism - in this case, we send the length message.

Summary

What are the consequences of all that? Well!

• CF objects, just like Objective-C objects, all have an isa pointer (except it’s called _isa in CF). It’s right there in struct __CFRuntimeBase.
• There are two toll-free bridging mechanisms! Some Objective-C objects "really are" CF objects - the memory layout between the Objective-C object and the corresponding CF object is identical (enabled in part by the presence of the _isa pointer above), and in that case the Objective-C methods are not invoked by the CF functions. For example, in this code:

  	CFStringGetLength([NSString stringWithCString:“Hello World!”]);
  

  There, -[NSString stringWithCString:] is returning an NSCFString (which you can verify by asking it for the name of its class), but -[NSCFString length] is never invoked - NO length method is invoked. You can verify that with gdb. Objects that "really are" their CF equivalents skip what’s usually thought of as the bridge, and "fall through" to the CF functions even when the CF functions are directly called on them. Obviously, this is an implementation detail, and you should not depend on this.
• That mechanism is also how bridging works the other way - how CF strings you get from, say, Carbon, can be passed around like Objective-C objects, because they really are Objective-C objects. The bridges are implemented entirely in CF and in the bridged classes - the Objective-C runtime is blissfully unaware.
• But! Plain ol’ Objective-C objects are sussed out by CF by checking to see if their class pointer is larger than 0xFFFF, and if so, ordinary Objective-C message dispatch is used from the CF functions. That’s the second toll-free bridging mechanism, and it must be present in every public CF function for a bridged object, except for features not supported Cocoa-side.
• Nowhere do we depend on the abstract class NSString at all - the bridge doesn’t check for it and Objective-C doesn’t care about it. That means that, in theory, CFStringGetLength() should "work" (invoke the length method) on any object, not just NSStrings. Does it? You can check it yourself. (Answer: yes!) Obviously, this is just an artifact of the implementation, and you should definitely not depend on this - only subclasses of NSString are supported by toll free bridging to CFString.
• Curiously, other "true" CF objects are not considered to be CF objects by this macro. For example,

  	CFStringGetLength([NSArray array]);
  

  will raise an exception because NSCFArray does not implement length. That is, CF_IS_OBJC is not asking "Are you a CF type?" but rather "Are you this specific CF type?" That should make you happy, because it raises a "selector not recognized" exception instead of crashing, which makes our code more debuggable. Thanks, CF!
• Why 0xFFFF? I’m glad you (I mean I) asked, since the answer (at least, what I think it is) has interesting connections to NULL. But that will have to wait until a future post.

Other approaches

	@implementation Array
	
	- (int)length {
	    return self->length;
	}
	
	@end

	int getLength(ArrayRef array) {
		return [(id)array length];
	}

	/* private length function that we want to wrap */
	static int privateGetLength(ArrayRef someArray) {
	   return someArray->length;
	}
	
	/* public ObjC API */
	@implementation Array
	
	- (int)length {
	   return privateGetLength(self->arrayRef);
	}
	
	@end
	
	/* public C API */
	int getLength(ArrayRef array) {
	   return [(id)array length];
	}
	

"Wrapping" up

So toll-free bridging is (one way) that Cocoa integrates with Carbon and even newer OS X APIs. It’s possible in large part because of Objective-C, but in this case, Apple gets as much mileage from the simple runtime implementation and C API as from its dynamic nature. You already knew that, I’ll bet - but hopefully you have a better idea of how it all works.

Now hands off! A coworker of mine makes the point that good developers distinguish between what they pretend to know and what they really know. The, uh, known knowns, and the known unknowns, as it were. The mechanism of toll-free bridging is not secret (it is open source, after all), but it is private, which means that you are encouraged to know about it but to refrain from depending on it. Use it for, say, debugging, but don’t ship apps that depend on it - because that prevents Apple from making OS X better. And nobody wants that! I mean the prevention part.]]> http://ridiculousfish.com/blog/archives/2006/09/09/bridge/feed/ http://ridiculousfish.com/blog/archives/2006/08/24/hex-fiend-11/ http://ridiculousfish.com/blog/archives/2006/08/24/hex-fiend-11/#comments Thu, 24 Aug 2006 20:35:04 +0000 ridiculous_fish Mac OS X http://ridiculousfish.com/blog/archives/2006/08/24/hex-fiend-11/

Hex Fiend 1.1

"I’m going to beat grep by thirty percent!" I confidently crow to anyone who would listen, those foolish enough to enter my office. And my girlfriend too, who’s contractually obligated to pay attention to everything I say.

See, I was working on Hex Fiend, and searching was dog slow. But Hex Fiend is supposed to be fast, and I want blazingly quick search that leaves the bewildered competition coughing in trails of dust. And, as everyone knows, the best way to get amazing results is to set arbitrary goals without any basis for believing they can be reached. So I set out to search faster than grep by thirty percent.

The first step in any potentially impossible project is, of course, to announce that you are on the verge of succeeding.

I imagine the author of grep, Ultimate Unix Geek, squinting at vi; the glow of a dozen xterms is the only light to fall on his ample frame covered by overalls, cheese doodles, and a tangle of beard. Discarded crushed Mountain Dew cans litter the floor. I look straight into the back of his head, covered by a snarl of greasy locks, and reply with a snarl of my own: You’re mine. The aphorism at the top, like the ex girlfriend who first told it to me, is dim in my recollection.

String searching

Having exhausted all my trash-talking avenues, it’s time to get to work. Now, everyone knows that without some sort of preflighting, the fastest string search you can do still takes linear time. Since my program is supposed to work on dozens of gigabytes, preflighting is impossible - there’s no place to put all the data that preflighting generates, and nobody wants to sit around while I generate it. So I am resigned to the linear algorithms. The best known is Boyer-Moore (I won’t insult your intelligence with a Wikipedia link, but the article there gives a good overview).

Boyer-Moore works like this: you have some string you’re looking for, which we’ll call the needle, and some string you want to find it in, which we’ll call the haystack. Instead of starting the search at the beginning of needle, you start at the end. If your needle character doesn’t match the character you’re looking at in haystack, you can move needle forwards in haystack until haystack’s mismatched character lines up with the same character in needle. If haystack’s mismatch isn’t in needle at all, then you can skip ahead a whole needle’s length.

For example, if you’re searching for a string of 100 ‘a’s (needle), you look at the 100th character in haystack. If it’s an ‘x’, well, ‘x’ doesn’t appear anywhere in needle, so you can skip ahead all of needle and look at the 200th character in haystack. A single mismatch allowed us to skip 100 characters!

I get shot down

For performance, the number of characters you can skip on a mismatch is usually stored in an array indexed by the character value. So the first part of my Boyer-Moore string searching algorithm looked like this:

char haystack_char = haystack[haystack_index];
if (last_char_in_needle != haystack_char)
   haystack_index += jump_table[haystack_char];

So we look at the character in haystack and if it’s not what we’re looking for, we jump ahead by the right distance for that character, which is in jump_table.

"There," I sigh, finishing and sitting back. It may not be faster than grep, but it should be at least as fast, because this is the fastest algorithm known. This should be a good start. So I confidently ran my benchmark, for a 1 gigabyte file...

grep:	2.52 seconds
Hex Fiend:	3.86 seconds

Ouch. I’m slower, more than 50% slower. grep is leaving me sucking dust. Ultimate Unix Geek chuckles into his xterms.

Rollin', rollin', rollin'

My eyes darken, my vision tunnels. I break out the big guns. My efforts to vectorize are fruitless (I’m not clever enough to vectorize Boyer-Moore because it has very linear data dependencies.) Shark shows a lot of branching, suggesting I can do better by unrolling the loop. Indeed:

grep:	2.52 seconds
Hex Fiend (unrolled):	2.68 seconds

But I was still more than 6% slower, and that’s as fast as I got. Exhausted, stymied at every turn, I throw up my hands. grep has won.

grep's dark secret

"How do you do it, Ultimate Unix Geek? How is grep so fast?" I moan at last, crawling forwards into the pale light of his CRT.

"Hmmm," he mumbles. "I suppose you have earned a villian’s exposition. Behold!" A blaze of keyboard strokes later and grep’s source code is smeared in green-on-black across the screen.

while (tp < = ep)
	  {
	    d = d1[U(tp[-1])], tp += d;
	    d = d1[U(tp[-1])], tp += d;
	    if (d == 0)
	      goto found;
	    d = d1[U(tp[-1])], tp += d;
	    d = d1[U(tp[-1])], tp += d;
	    d = d1[U(tp[-1])], tp += d;
	    if (d == 0)
	      goto found;
	    d = d1[U(tp[-1])], tp += d;
	    d = d1[U(tp[-1])], tp += d;
	    d = d1[U(tp[-1])], tp += d;
	    if (d == 0)
	      goto found;
	    d = d1[U(tp[-1])], tp += d;
	    d = d1[U(tp[-1])], tp += d;
	  }

"You bastard!" I shriek, amazed at what I see. "You sold them out!"

See all those d = d1[U(tp[-1])], tp += d; lines? Well, d1 is the jump table, and it so happens that grep puts 0 in the jump table for the last character in needle. So when grep looks up the jump distance for the character, via haystack_index += jump_table[haystack_char], well, if haystack_char is the last character in needle (meaning we have a potential match), then jump_table[haystack_char] is 0, so that line doesn’t actually increase haystack_index.

All that is fine and noble. But do not be fooled! If the characters match, the search location doesn’t change - so grep assumes there is no match, up to three times in a row, before checking to see if it actually found a match.

Put another way, grep sells out its worst case (lots of partial matches) to make the best case (few partial matches) go faster. How treacherous! As this realization dawns on me, the room seemed to grow dim and slip sideways. I look up at the Ultimate Unix Geek, spinning slowly in his padded chair, and I hear his cackle "old age and treachery...", and in his flickering CRT there is a face reflected, but it’s my ex girlfriend, and the last thing I see before I black out is a patch of yellow cheese powder inside her long tangled beard.

I take a page from grep

"Damn you," I mumble at last, rising from my prostrate position. Chagrined and humbled, I copy the technique.

grep:	2.52 seconds
Hex Fiend (treacherous):	2.46 seconds

What's the win?

Copying that trick brought me from six percent slower to two percent faster, but at what cost? What penalty has grep paid for this treachery? Let us check - we shall make a one gigabyte file with one thousand x’s per line, and time grep searching for "yy" (a two character best case) and "yx" (a two character worst case). Then we’ll send grep to Over-Optimizers Anonymous and compare how a reformed grep (one that checks for a match after every character) performs.

	Best case	Worst case
Treacherous grep	2.57 seconds	4.89 seconds
Reformed grep	2.79 seconds	2.88 seconds

Innnnteresting. The treacherous optimization does indeed squeeze out almost 8% faster searching in the best case, at a cost of nearly 70% slower searching in the worst case. Worth it? You decide! Let me know what you think.

Resolved and refreshed, I plan my next entry. This isn’t over, Ultimate Unix Geek.

Disclaimers

(Note: I have never met the authors or maintainers of grep. I’m sure they’re all well balanced clean shaven beer and coffee drinkers.)

(Oh, and the released version of HexFiend will be slightly slower in this case, because of an overly large buffer that blows the cache. In other situations, the story is different, but more about those in a future post.)

Once again, this discussion is only my meager opinion. I do not speak for Apple, and none of what I have to write represents Apple’s official position.

The article is filled with claims such as "OS X is incredibly slow by design," and while the the BSD kernel is "excellent", the XNU kernel is "very inefficient and less stable" compared to Linux or BSD. However, without specifics, these assertions are meaningless; I will ignore them and concentrate on the technical aspects of what’s going on.

System calls

Sekhon does give one example of what he means. According to him,

This isn’t true, as anyone can verify from Apple’s public sources. For example, here is the assembly for the open function (which, of course, performs the open system call):

	mov	$0x5,%eax
	nop
	nop
	call	0x90110a70 <_sysenter_trap>
	jae	0x90001f4c 
	call	0x90001f43 
	pop	%edx
	mov	268455761(%edx),%edx
	jmp	*%edx
	ret

__sysenter_trap:
	popl %edx
	movl %esp, %ecx
	sysenter

	mov    $0x5,%eax
	int    $0x80
	jb     0xa8c71cc 
	ret

On to the benchmark

According to Sekhon, OS X performed poorly on his statistical software relative to Windows and Linux, and I was able to reproduce his results on my 2 GHz Core Duo iMac with Windows XP and Mac OS X (I do not have Linux installed, so I did not test it). So yes, it’s really happening - but why?

A Shark sample shows that Mac OS X is spending an inordinate amount of time in malloc. After instrumenting Sekhon’s code, I see that it is allocating 35 KB buffers, copying data into these buffers, and then immediately freeing them. This is happening a lot - for example, to multiply two matrices, Sekhon’s code will allocate a temporary buffer to hold the result, compute the result into it, allocate a new matrix, copy the buffer into that, free the buffer, allocate a third matrix, copy the result into that, destroy the second matrix, and then finally the result gets returned. That’s three large allocations per multiplication.

Shark showed that the other major component of the test is the matrix multiplication, which is mostly double precision floating point multiplications and additions, with some loads and stores. Because OS X performs these computations with SSE instructions (though they are not vectorized) and Linux and Windows use the ordinary x87 floating point stack, we might expect to see a performance difference. However, this turned out to not be the case; the SSE and x87 units performed similarly here.

Since the arithmetic component of the test is hardware bound, Sekhon’s test is essentially a microbenchmark of malloc() and free() for 35 KB blocks.

malloc

Now, when allocating memory, malloc can either manage the memory blocks on the application heap, or it can go to the kernel’s virtual memory system for fresh pages. The application heap is faster because it does not require a round trip to the kernel, but some allocation patterns can cause "holes" in the heap, which waste memory and ultimately hurt performance. If the allocation is performed by the kernel, then the kernel can defragment the pages and avoid wasting memory.

Because most programmers understand that large allocations are expensive, and larger allocations produce more fragmentation, Windows, Linux, and Mac OS X will all switch over from heap-managed allocations to VM-managed allocations at a certain size. That size is determined by the malloc implementation.

Linux uses ptmalloc, which is a thread-safe implemenation based on Doug Lea’s allocator (Sekhon’s test is single threaded, incidentally). R also uses the Lea allocator on Windows instead of the default Windows malloc. But on Mac OS X, it uses the default allocator.

It just so happens that Mac OS X’s default malloc does the "switch" at 15 KB (search for LARGE_THRESHOLD) whereas Lea’s allocator does it at 128 KB (search for DEFAULT_MMAP_THRESHOLD). Sekhon’s 35 KB allocations fall right in the middle.

So what this means is that on Mac OS X, every 35 KB allocation is causing a round trip to the kernel for fresh pages, whereas on Windows and Linux the allocations are serviced from the application heap, without talking to the kernel at all. Similarly, every free() causes another round trip on Mac OS X, but not on Linux or Windows. None of the defragmentation benefits of using fresh pages come into play because Sekhon frees these blocks immediately after allocating them, which is, shall we say, an unusual allocation pattern.

Like R on Windows, it’s a simple matter to compile and link against Lea’s malloc instead of the default one on Mac OS X. What happens if we do so?

Mac OS X (default allocator)	24 seconds
Mac OS X (Lea allocator)	10 seconds
Windows XP	10 seconds

These results could be further improved on every platform by avoiding all of the gratuitious allocations and copying, and by using an optimized matrix multiplication routine such as those R provides via ATLAS.

In short

To sum up the particulars of this test:

• Linux, Windows, and Mac OS X service small allocations from the application heap and large ones from the kernel’s VM system in recognition of the speed/fragmentation tradeoff.
• Mac OS X’s default malloc switches from the first to the second at an earlier point (smaller allocation size) than do the allocators used on Windows and Linux.
• Sekhon’s test boils down to a microbenchmark of malloc()ing and then immediately free()ing 35 KB chunks.
• 35 KB is after Mac OS X switches, but before Linux and Windows switch. Thus, Mac OS X will ask the kernel for the memory, while Linux and Windows will not; it is reasonable that OS X could be slower in this circumstance.
• If you use the same allocator on Mac OS X that R uses on Windows, the performance differences all but disappear.
• Most applications are careful to avoid unnecessary large allocations, and will enjoy decreased memory usage and better locality with an allocator that relies more heavily on the VM system (such as on Mac OS X). In that sense, this is a poor benchmark. Sekhon’s code could be improved on every platform by allocating only what it needs.

Writing this entry felt like arguing on IRC; please don’t make me do it again. In that spirit, the following are ideas that I want potential authors of “shootoffs” to keep in mind:

• Apple provides some truly excellent tools for analyzing the performance of your application. Since they’re free, there’s no excuse for not using them. You should be able to point very clearly at which operations are slower, and give a convincing explanation of why.
• Apple has made decisions that adversely impact OS X’s performance, but there are reasons for those decisions. Sometimes the tradeoff is to improve performance elsewhere, sometimes it’s to enable a feature, sometimes it’s for reliability, sometimes it’s a tragic nod to compatibility. And yes, sometimes it’s bugs, and sometimes Apple just hasn’t gotten around to optimizing that area yet. Any exhibition of benchmark results should give a discussion of the tradeoffs made to achieve (or cause) that performance.
• If you do provide benchmark results, try to do so without using the phrase "reality distortion field."

Click here to read more, see screenshots, or download Hex Fiend

Hex Fiend allows inserting and deleting as well as overwriting data. It supports 100+ GB files with ease. It provides a full undo stack, copy and paste, and other features you’ve come to expect from a Mac app. And it’s very fast, with a surprisingly small memory footprint that doesn’t depend on the size of the files you’re working with.

Hex Fiend was developed as an experiment in huge files. Specifically,

• How well can the Cocoa NSDocument system be made to work with very large files?
• How well can the Cocoa text system be extended to work with very large files?
• How well does Cocoa get along with 64 bit data in general?
• What are some techniques for representing more data than can fit in memory?

Check it out - it’s free, and it’s a Universal Binary. If you’ve got questions or comments about it or how it works, please leave a comment!

(Incidentally, the Hex Fiend main page was made with iWeb!)

Edit: I’ve discovered/been informed that drag and drop is busted. I will put out an update later tonight to fix this.

Edit 2: Hex Fiend 1.0.1 has been released to fix the drag and drop problems. Please redownload it by clicking on the icon above.