The following contains discussions of the endings of the movies Mississippi Burning, The Help, and Django Unchained. This maybe isn’t such a big deal since their endings are all eminently predictable if not a matter of historical record, but in fairness everything past the jump should be considered a spoiler.
The MapReduce paradigm was inspired by functional programming techniques, so why not take things full circle and rewrite a MapReduce job in a functional language? Here is a one-line Scala implementation of the classic “word-count” program.
List("to be or", "not to", "be").par.flatMap(_.split("""\s+""")).
foldLeft(Map[String, Int]())((m,s) => m + (s -> (m.getOrElse(s, 0)+1)))
// produces Map(to -> 2, be -> 2, or -> 1, not -> 1)
Okay so I used “flatMap” instead of “map” and “foldLeft” instead of “reduce”, but you get the idea. Note the par keyword, which invokes Scala’s parallel collections framework, transparently parallelizing the map step. Who needs a compute cluster when you’ve got the Scala REPL?
This is just a little stunt, of course, but if you’re serious about working with MapReduce and Scala the ScalaHadoop project looks like a good place to start.
That is how a picture is attached to reality; it reaches right out to it.
–Ludwig Wittgenstein
Here is Frank Sinatra performing Cole Porter’s “Night and Day” on his 1957 album A Swingin’ Affair.
Here is a solo guitar version of “Night and Day” I recorded a couple years ago.
Click to hear the recording, which was played on a guitar not unlike the one pictured here.
Here is a picture of the LP on which the Sinatra version of “Night and Day” appears.
Of course you can’t hear music by looking at a picture, but you could imagine putting this record on a turntable and hearing the same song as above. Specifically what would happen is a stylus would travel over a rotating sheet of vinyl. Variations in the height of the vinyl would be translated into vertical movements of the stylus, which would in turn be translated into an electrical signal that, when amplified and played through speakers, would produce a song identical to the first link above, give or take a few pops and skips. It would not be identical to my version, but it would nevertheless be the same song.
Here is the sheet music to “Night and Day” courtesy of Wikifonia, the web’s best source of free lead sheets.
This sheet music is the written form of both songs above because the markings on the page correspond to its notes. Well, not exactly. It’s written in the key of G, while both of the songs are played in D. It also doesn’t annotate the horn and bass lines of the Sinatra version, and it certainly doesn’t annotate the improvised riffs in my version of the song, but by longstanding convention you are considered to be playing a lead sheet so long as you stick more or less to the chord progression and at some point produce a recognizable version of the melody.
Of course I’m speaking loosely when I refer to the notes “on the page” because there is no page. There is an actual sheet of paper printed off from this very site sitting in a ring binder in my house, but you wouldn’t know that because you’re looking at an image on this website. Which strictly speaking is an image on some kind of screen in front of you. Nevertheless we’d say these are the same piece of sheet music, though they’re different from the paper one in my ring binder (it’s in the key of D) and also presumably different from whatever Sinatra’s session musicians played off of a little over fifty years ago.
When I talk about the songs above I of course don’t mean that a song is physically “above” the current line of text–that’s just a way of telling you how to go find the song on this page. The song itself is at the other end of the hyperlink. You click on the hyperlink, sending a request to a computer to fetch the song for you, which forwards the request to another computer and so on and so forth. Eventually the file containing the song is retrieved and sent back to you. There may be any number of intermediate copies of the song scattered among computers all over the world expediting the process of creating another copy which exists on your local machine. The song also temporarily exists as bursts of network packets intermingled with a mindbending number of other packets, where each packet is just a list of numbers which is in turn just a sequence of voltage variations passing down a piece of copper or fiber-optic cable. Unless we are network engineers, however, we conceive of these intermediary copies as a technicality, an artifact of the delivery system. (This is by analogy with a written note, which certainly does not become a series of identical notes when it is passed from hand to hand.)
The file containing the song actually resides in a data center, which is some giant humming room located in an unremarkable looking industrial park that, for all the amazing computational power it houses is primarily a masterpiece of the art of air conditioning. Either of the versions of “Night and Day” you just listened to exists on harddrives on some set of the servers in a room that looks something like this.
Were it necessary I suppose you could go to the data center and find a machine that contained the file and extract its harddrive. With the help of a hardware specialist you could even determine the approximate location on which the information is stored, either as a magnetic irregularities on a spinning drum or a set of transistors in a particular state. It would be possible to tap a small plastic box in two separate places with the head of a pin and correctly say that the song is here but not here, though outside of a harddrive manufacturing plant it’s not clear why you’d ever do such a thing. The song isn’t really a physical entity–it just happens to correspond to many physical entities of many disparate types.
And I’ve only scratched the surface of the correspondences. The song is also stored on cassette tapes, iPods, laptops, scratchy old 78s, 8 tracks, and Diamond Rios gathering dust in the bottoms of desk drawers. Oldies radio stations are broadcasting it in the form of radio waves all across the globe. At this moment the members of a pickup band are at a café, street corner, or living room somewhere, at a loss for what to play next until one of them says, “What about ‘Night and Day’?” The breath of a woman waiting at a bus stop in Oslo on her way to a graveyard shift at a 24-hour grocery store emerges in visible rhythmic puffs as she whistles “Night and Day” even though she doesn’t know the name of the song, or the lyrics, and in fact doesn’t even realize she’s whistling.
Why this orgy of isomorphism? Two reasons: communication and memory. If no one plays “Night and Day” no one will hear it, and if the song is never written down or recorded it will eventually be forgotten. But even memory provides no exit from the cycle of representation. Close your eyes and remember the song that you just heard a minute ago. That you can means there is now something about the configuration of your brain that bears a resemblance to radio waves and electrical variation in a length of fiber-optic cable and markings on a score and pressure fronts of air in the vicinity of a cabaret singer and once upon a time Cole Porter’s brain and his alone. The song is, we say, now part of you.
I’m purposefully making this all sound a bit mystical, but really there is no magic here. Unifying disparate phenomena under a single name is just something us humans do. And in this case the details of the mapping between any pair of phenomena is usually straightforward–there are plenty of accompanists who can turn the printed notes of a score into the correct key presses on a piano, plenty of audio engineers who understand the relationship between microphone diaphragm deflection and voltage levels in exhaustive detail. It’s just that there are so many mappings between so staggeringly many disparate kinds of things that it’s difficult to conceive of the proper level of abstraction to capture them all and is therefore tempting to say that the ability to be put into correspondence with each other is just what “Night and Day” is.
The above is mostly me doing an impersonation of Ludwig Wittgenstein’s Picture Theory of language from the Tractatus Logico-Philosophicus, which very roughly is the claim that a thought can represent reality by virtue of bearing a structural relationship to it. I say impersonation because I don’t fully understand the theory and because I can see where I am diverging from points I do understand, e.g. my concern with the physical groundings of these representations runs contrary to Wittgenstein’s insistence that the world is comprised of “facts” instead of “objects”. But a significant part of the Tractatus does hinge on the idea of structural similarity, and this is where there can be mutual illumination between the writings of a World War One era philosopher and the current explosion of information technology.
The story is that Wittgenstein’s inspiration came from a courtroom model of a traffic accident recreated on a table-top using toy cars. This is an isomorphism, sure, but I’ve always found it an unsatisfying example because the mapping is a little too clear. In both the real world and the courtroom we’re observing a spatial relationship between cars; it’s just in the latter case the cars are a lot smaller. There must be more to the Picture Theory than a shrink ray. The concept of a function from mathematics is closer to the mark, but it’s difficult to talk about functions without everyone too narrowly picturing numerical relationships drawn on graph paper. Likewise anyone who has given any thought to language, or musical scores, or photographs has wrestled with the idea of representation, but usually still on a very recondite philosophical level.
Then along came computers. The collapse of various forms of information into a single physical medium is a practical business we’ve all been witnessing in our daily lives. We ourselves have replaced our cassettes with CDs with an iTunes library and felt like we’ve done nothing, or contemplated how the purchase of a Kindle might allow us to recapture shelf space for knick-knacks. Photo developer kiosks and newspapers really are going out of business. Our daily life has given us a feel for the malleability of information, and about a century ago Wittgenstein gave it a name.
There is a night watchman whose job it is to keep an eye on an empty office building between the hours of midnight and 8:00 am. For most of his shift he sits at a desk near the front entrance watching a bank of surveillance monitors. Twice during his shift he makes rounds throughout the building. This takes about an hour. At 7:00 am he unlocks the front doors and spends the remainder of his shift sitting at the front desk greeting people as they come in.
At any given time you have a pretty good idea where in the building the night watchman will be. Between midnight and 1:00 am he’ll be at the desk. At 1:05 am he’ll be heading down the main hallway, starting his first patrol. You can guess almost to the minute when he’ll be on the third floor, or walking down the back stairs, or checking the door to the loading dock, and so forth. Of course these are just educated guesses–only the night watchman actually knows because he’s the only person in the building. Maybe some times instead of being at his desk he’s in the bathroom, or making himself a cup of coffee in the kitchenette down the hall. Maybe one night he lingers an unusually long time in a sixth floor conference room, mesmerized by a trickle of headlights on the highway below. In general though the watchman’s location is very predictable. This is a boring job.
You can imagine a video screen displaying the watchman’s probable locations as colors projected onto a floor plan of the building. Bight red represents where he is most likely to be, grading through orange and yellow for possible locations, heading down to cooler blues to represent unlikely locations, and finally becoming transparent for regions of the building where the night watchman almost certainly is not. There is a digital clock at the bottom of the screen that runs as you view an animation of the watchman’s whereabouts throughout the night. Between midnight and 1:00 am there is a tight red blob around the front desk. During his rounds the color spreads throughout the building, a red ball centered around where he’s likely to be, with a penumbra of orange that trails off into blue since the watchman doesn’t do exactly the same thing night after night. It is like watching a multicolored jellyfish ooze its way through the hallways and up the staircases, its edges flickering and luminescent. Even if you knew that this depiction represented a fairly abstract idea about probabilities, it would be easy to get into the habit of thinking that the office building was actually being guarded by an oozing shimmering jellyfish.
In order to make sure the night watchman is actually doing his rounds there are sensor boxes scattered throughout the building. He waves a keycard across one and it registers that he was there at a particular instant, allowing us to track his progress. He is next to the third floor women’s room at exactly 3:07 am, on the fourth floor stair landing at 3:25 am, outside the fifth floor conference room at 4:04 am and so forth. Now imagine how this would appear on our screen. We presume the watchman starts his rounds at 1:00 am, so we’d see the tight red ball detach itself from the front desk, spread out a bit, and begin oozing towards the stairs. It drifts like this for a while, its nucleus moving with purpose, but with blurred edges sniffing about, reflecting the fact that we can’t know exactly where the guy is. Then the watchman comes to the first sensor box and swipes his keycard. On the screen the diffuse blob instantly collapses into a single red dot right on top of the first sensor box because we know that’s where the night watchman is at that moment. Now as time continues to unfold, he will presumably resume his rounds and uncertainty will insinuate itself again. The single red point will swell back into an orangish haze of possibility, oozing its way through the building until the watchman reaches the next sensor box and swipes his card, at which point it will again collapse to a red dot.
If you only ever saw the screen this all might seem exceedingly odd. Not only is the office building being guarded by a shimmering jellyfish, but from time to time that jellyfish collapses to a single point, then reinflates and resumes its oozing. What’s more it certainly can’t be the electronic mechanism of the sensor box that causes the watchman to abruptly change his shape. No, it seems as if it’s our knowledge of the watchman’s location causes his dramatic collapse. The mere act of observing a thing changes it. What could be the deep forces at play here? Is it synchronicity? Are we tapping into the universal mind? Certainly we are at some boundary where science must cede authority to a mystical acknowledgement of how powerful and amazing our personal consciousness is!
There is nothing of the sort. There is just a night watchman. For a while we know where he is, and then we don’t.
Years ago when I was an undergraduate at my big-name second-tier Ivy league college I was sitting around with my fellow undergraduates over pizza talking about what makes life meaningful, like you do. There was a general consensus that creativity was an essential element. The best, of course, was to be a full-on bona fide artist like a novelist or painter, but we could imagine how non-artistic professions like scientist or entrepreneur might produce the same kind of satisfaction. Even if you were, say, a carpenter, there could be an artisanal quality to the work that would make it rewarding. The specifics didn’t matter, as long as life offered some avenue of personal expression.
One of us (a few years older, though looking back on it still quite young) disagreed. We were being narrow-minded, he said. The belief that creativity was the key to a happy life was peculiar to the sorts of people who attend big-name colleges. In fact, for most people on Earth the idea that everyone should aspire to be some kind of artist sounds as weird as the idea that everyone should aspire to be some kind of insurance salesman. Think of a 19th century peasant in Italy, he said. For that guy satisfaction came from providing food for his family and going to church. Personal creativity didn’t enter into it.
But, we objected, wasn’t the 19th century Italian peasant really a kind of artist? His medium was the soil, and his ability to grow food from seed was a creative act in the same way as transforming a block of marble into a statue. Sure it might seem like something else was going on, but underneath it all that guy had the same aspirations we did.
For some deeply religious people, the idea of decency is so wrapped up with the idea of a personal God that they find it difficult to speak about it otherwise. This can cause friction when they encounter secularists like myself. Each senses that the other shares the same values, but has such radically different ways of expressing them that communication becomes strained. Sometimes the religious person will try and work around this by suggesting that to recognize the decency of others and the goodness of the world is to experience God’s love, and therefore the secularist is actually religious in every way that matters, they just don’t realize it. Even if this is offered in a spirit of generosity it is an exasperating thing to hear. No, the secularist thinks, I’m not a child. I know what I believe, and it is different from what you believe.
That the world is populated by people who are different from us is scary. That the things dearest to our hearts may not be dear to them is so scary that it becomes hard to see. But life is more interesting when we see it anyway.
Was there any century worse than the nineteenth? Sure life was more brutish, poor, and short back in the Stone Age, but times were so tough then it seems churlish to do anything but marvel at the general aptitude for fashioning hand axes. The 1800s, on the other hand, weren’t that long ago. We still read the novels people wrote then, listen to the music they composed, and live in the houses they built. You don’t have to prepend too many “greats” to the word “grandparent” before you run into Victorian blood relatives. If the past is another country, the 19th century is like a town just down the road.
Yeah but lock your doors when you drive through, because those people were terrifying. See them staring dead-eyed out of daguerrotypes because smiling was apparently invented in 1921. The most generous, moral and open-minded among them was still a Ku Klux Klan Grand Wizard maundering on about women’s naturally childlike intellect while harpooning a whale. Hoop skirts, sexual repression, and hideous facial hair. But don’t you dare laugh, because that dork with the fussy string tie and the waxed moustache would gut you with a Bowie knife as soon as look at you. Violence was an omnipresent fact of life. The British fought the French fought the Dutch fought the Austrians for reasons that are now incomprehensible. If you were a young man you aspired to be a military officer, which meant you donned a giant plumed hat and tried to give your fellow Stratego characters gangreen with a musket.
Here in the United States we had slavery and the Indian Wars. Think Balkan ethnic cleansing from the 1990s, except happening across an entire contient. Conditions in much of the country were so primitive that there was no rule of law. People banded together in small paranoid groups and what order there was was maintained by men with pistols following brutal codes of honor. It’s the same situation that obtains in gang-ridden public housing projects today, but because back then the guns were being carried by white people the whole business is looked on with an inexplicable sense of nostalgia. People shivered in railroad cars, imagining what horrors awaited them in Denver. In crude clapboard brothels situated on the outskirts of flyblown frontier towns burly miners were serviced by prostitutes who looked like burly miners.
–I’m tired. I’m sick. I can’t remember the last time I had a hot shower or used a bathroom that was anything more than an excrement-filled ditch. Every person around me is a virulent racist, a religious fanatic, or both, and they reek. This is the worst camping trip ever. I want to go home. –Home? You are home.
And yet, amidst all this darkness, the 19th century is the era in which mathematics finally came together. Euler’s inspired insights coalesced into Riemann’s formal rigor. Maxwell captured the essential facts of electromagnetism in four vector calculus equations whose intricacy still delights. Epsilon-delta proofs moved Zeno’s two thousand year old paradoxes into the Solved column. Georg Cantor tamed infinity. This was no steady accumulation of knowledge or localized flurry of fresco-painting. No, it was a human project that had been brewing continuously since the days of classical Greece coming into sudden and breathtaking fruition. And it was pulled off by a handful of those same prim, bewhiskered, European zombies that in every other way inspire nothing but horror.
