Deadbeat Escapement

I have a new blog!

nathanial.hartman@gmail.com (Nathan Hartman) — Fri, 02 Oct 2009 05:40:00 +0000

My new blog is here. It has images and code, since I'm getting all serious about this stuff now. Maybe it will even have screencasts of me doing unnatural things with my emacs.

Objective-C, Javascript

nathanial.hartman@gmail.com (Nathan Hartman) — Tue, 11 Aug 2009 05:53:00 +0000

There are relatively few programmerly things that benefit from being done in a GUI. Most everything is better done as text or on a command line. But UI creation is a big exception to that. UIs are definitely best done in a builder. My experience with Cocoa's Interface Builder over the past month has been awesome. It's so awesome, I don't know how I could even ever go back to doing things by hand. That's how awesome it is.
What's more, it's turned me on to other GUI builders, like the one for WPF in Visual Studio; and it's totally obliterated any lingering desire I've had to write Swing or *name-that-framework* code anymore. Clearly, the way of the future is XML markup. And that's the key difference between something like Winforms and WPF or Cocoa; there's no generated code, just generated XML, and that XML is simply read in at runtime to set up your widgets and your event bindings. That's the way to do it -- it sortof has a web feel to it even.
Anyway, I can't ever go back to writing Swing GUIs by hand. It's too monstrous. And all the Swing GUI Builders just generate crappy Java code for you. I don't want code generators, I want markup generators. I want something like Glade for GTK, except for Swing. Though I'm not sure I'll ever get it since Sun has abandoned Swing and nobody likes Java Desktop Apps anyway.
But Objective-C is actually a pretty awesome language, and combined with Cocoa and all the Apple tools, OSX is a pretty awesome platform for rich application development. Everything is easy to use, the Objective-C runtime, though lacking garbage collection, is dynamic and has an excellent reference counting facility -- the Cocoa API is consistent and comprehensive... it's a pretty awesome package overall, and I can see why developers like developing applications for the Mac.
It does bother me though that I'm dependent on their tooling to be productive on Cocoa. You can't really write Cocoa interfaces without the Interface Builder, and all that stuff only runs on Mac and only is available for Mac. I might have missed it, but as far as I know there is no interface builder for GNUStep -- I don't even know if GNUStep has a UI library like Cocoa; maybe some sort of wrapper for GTK.
Anyway, I had an awesome time developing for the iPhone and for the OSX desktop environment, and I think Objective-C is a cool language that really intrinsically fits the MVC model and suites the task of creating desktop applications really well.

----

Javascript is really Scheme. It's Scheme with a half-assed, ugly varnish of C. But it's still Scheme to the core, and a lot of its querkyness can be explained in terms of it being Scheme; once you understand that it is Scheme, you understand why it behaves in certain ways. Like why variables declared in an if statement or a for-loop are available throughout the enclosing function scope. That's because, like in Scheme, environment is a factor of functions. Every time you create a new function, you instantiate a new environment to place your variables in. For loops and If-statements don't create new functions, so the environment your in, and hence all the variables you declare, are declared in the enclosing function scope, just like they would be with scheme "(define x 1)", but instead you have "var x = 1;".
Javascript could have been a lot more awesome if it had just been a little more straightforward. If they could have skipped the C-ishness, and just done straight Scheme with some Self, programmers coming from more typical languages wouldn't be so shocked by Javascript's schemeish and selfish behavior. If Javascript were more plane about what it was about and how it worked, there would be less confusion. The guy who wrote "Javascript: The Good Parts" said that he had an epiphany where he realized Javascript was just Scheme... he'd have had that epiphany sooner if Javascript wasn't perpetually travelling in Cstyle disguise.
But Javascript is pretty awesome, cause despite all its grossness, the first class functions and dynamic typing and prototype inheritance and object literals and array literals are all so awesome and so powerful that they make up for all the rest. And provided you avoid the nasty parts, you can program in Javascript just like you would Scheme, and you get something awesome and powerful like JQuery as a result. I think John Resig really understands or understood what Javascript was about when he created JQuery; he didn't try and force it to be like other languages and build some guady inheritance heirarchy that's classically fake -- he used lambdas and functional programming techniques to create a really awesome and expressive framework that brings the fun back into DOM manipulation and Ajax and such. I really love JQuery, and I'm starting to really love Javascript as I begin to understand more of how to write good Javascript. Indeed, the most important realization for any Web 2.0 developer to make is that you CAN write good Javascript. It is not impossible. Javascript isn't this terrible bomb waiting to go off. You can get it right, and it can empower your web pages.

Something interesting: Cappuccino is a web framework -- essentially a port of Cocoa to Javascript. What's really awesome about it (one of the things) is that you can use the Interface Builder to create your web guis. Because it's all xib files, they were able to write a converter that takes the xib xml and turns it into an equivalent for their framework. Pretty awesome.

Mac Development

nathanial.hartman@gmail.com (Nathan Hartman) — Fri, 12 Jun 2009 09:56:00 +0000

I'm not really doing any Mac development at the moment, though I have considered it. What surprises me about OSX is just that Apple hasn't modernized the development platform very much. They don't, for instance, have a virtual machine like the CLR. Why does Windows have .NET and OSX not have anything?
It seems somewhat stoneage to have the only language for Mac development be the venerable but crusty Objective-C. The standard variety of Objective-C does not have garbage collection, and it has all the unpleasant benefits of unmanaged C. I haven't done much development with Objective-C, so I can't say how often memory leaks arise; I do appreciate their auto-release pools and associated reference counting mechanisms, but I strikes me that in modern times, and for application development, there isn't any good excuse not to have automatic memory management in your application. I know that Objective-C 2.0 or whatever has a garbage collector, but it seems half-ass. And it doesn't work on the iphone apparently; so that screws most the Objective-C market.
I really like the idea of Macruby; the HotCocoa screencasts are very impressive. I'd absolutely love to be able to do Cocoa development without having to resort to Xcode and the interface builder. However, it is not clear how stable or production-ready MacRuby is. Their only at version 0.4 -- not that version numbers mean anything -- but I don't have any confidence that Apple is actually committed to the MacRuby project. I mean... where's their epic plan for future development on the Mac. Is MacRuby it? Is that all they've got? That's good enough I suppose, but it seems like their just diddling around with it: they haven't moved forward from Objective-C ever. No innovation, no development, except within the context of what's already there. I know they replaced Carbon or whatever with Cocoa, but I'm talking about building a real managed runtime like you see with .NET or the JVM. Something that can host multiple languages and provide the kinds of things developers have come to expect from modern platforms: the most important being (and I hate to harp on this, but it's true) -- memory management! There's no greater productivity booster for application development than not having to track down memory leaks all the time. And, like I said, maybe no one complains about that because it's not so much of an issue with Objective-C autorelease pools and strict naming conventions... but I still feel like this is some sort of medievilism on their part. Come on to the future why dontcha?
With respect to garbage collection, MacRuby solves that issue for Mac development. And it presents a nice dynamic (and non-compiled) language to add to the toolset. In the presentation I watched (perhaps I should post the link, but I am lazy), the dude mentioned that MacRuby would be set to become THE scripting language for the Mac platform. That makes sense of course, just because there isn't anything else. All the other scripting languages that run on Mac right now can only interact with Cocoa over a bridge; which sucks a lot. This would be the only instance of a scripting language running natively on the objective c runtime, with all the performance and easy accessiblity that entails.
Anyway... time to go. Bye.

Windows Development

nathanial.hartman@gmail.com (Nathan Hartman) — Sun, 31 May 2009 20:22:00 +0000

Until last week, I'd never done any Windows development. That's mostly because I think Microsoft is a little gross; and I don't like Windows. But it's also because I've mostly been doing Java or JVM related development, and so I haven't needed to specifically target anything Windowsy. Last week though, I got an assignment that involved coding a Windows service, and accessing a C API on Windows, so I had to learn.
It hasn't been as terrible as I thought it might be; though some parts have been terrible. One thing I hate is that, if you're going to be doing Windows development, you're almost obligated to use Visual Studio; especially if you're doing anything besides Java development. For me, cause I had to interact with some C, I needed access to the Microsoft compiler, and so I had to use Visual Studio. Plus I was exploring .NET, and I think Visual Studio is like the only .NET IDE. I would just as soon use emacs for everything, but then I'd have to use the command line c compiler, which wasn't written for humans. I don't mean GCC, I mean the microsoft c ompiler -- which was never meant to be used from the command line, and is completely unintuitive and alien (from my perspective, having tried to use it.)
On the plus side, installing Python, Perl, and Ruby (the most important scripting languages I can think of) was painless. They all come with msi installers that make everything a breeze. I imagine compiling them from source, which is easy to do on Linux, would be much more painful on Windows -- but, as they say, when in Rome, do as the Romans do, and therefore: when on Windows, do as Windows does -- meaning: use the fricken msi installer.
Because I cannot live without multiple desktops anymore -- osx and linux having addicted me to them -- I cannot live without an Ubuntu VM on my Windows partition. Since Windows doesn't offer multiple desktops, at least not on Windows XP, I've ended up using my Ubuntu VM for a lot of my development work, and just using Windows to compile and deploy stuff. I hop into the vm to do my work, I hop out to run the tests. Also, what irks me about XP (and I don't know if this will be an issue with Windows 7), is that you cannot create custom keybindings without making esoteric changes to the Registry. The evil and mysterious registry, that strikes terror into the mind of every sane developer, or even every sane IT person. In Ubuntu, or any Linux distro, it's as simple as editing xmodmap, or whatever the equivalent keybindings config file. But in Windows, nooooo. Not that simple. You have to do some black magic and possibly corrupt your whole machine. Vomit.
I think .NET is cool. The only thing I wish is that it wasn't tied to Windows. I know there's Mono, but Mono's implementation is kind-of sketchy. If Microsoft wasn't so self-possessed (and perhaps didn't own so much of the desktop market share) they'd make a proper port to Linux and to OSX, like Sun does for Java; though the Apple guys have taken the OSX Java port into the own hands recently and screwed everything up. Anyway, it'd be nice if I could use .NET, and especially nice if I could use .NET languages, just not on Windows. I'd like to program F# from the comfort of my own Ubuntu, in emacs, with a nice Linux runtime. Unfortunately, the dark gods of Microsoft software imperialism say no.
In conclusion, Windows development hasn't been the awful experience I thought it might be. It's been new and interesting. I've even experimented with the satanic Visual Basic .NET -- though just enough to learn the wretched syntax so I could follow examples. And for the record, I think C# has way too much syntactic sugar for its own good. It makes me lightheaded just looking at it. Finally, ctypes for Python is totally awesome. And that's all I have to say about that.

Clojurey

nathanial.hartman@gmail.com (Nathan Hartman) — Wed, 13 May 2009 16:23:00 +0000

Clojure went 1.0! w00t. And in the meantime, my blog languishes for lack of attention. What kind of blogger am I? to Leave it lying here for so long. Soaking in its own silly juices.But I'm back, for a moment. I must consolidate my web presence; fortify it; make it strong again. But it was never strong to start; yet it will be again! Imagine if James Joyce wrote a blog -- it wouldn't make any sense at all. It shocks me, no... flabergasts me, that one can get anything done in Clojure, cause afterall, it is a "functional" language; and aren't functional languages all universally useless? Don't functional languages just burn up cpu? It must be as the Great Maestro Hickey states: that Clojure is the most friendly face ever put to a functional language. Clojure is easier than Haskell because it doesn't have Monads. There was a guy... Ted Neward (I think) who said that the only way a functional language can ever take off outside academia is if it doesn't have monads. F# doesn't have monads, and look how well it is doing. And neither does Clojure, so it's sure to be a success. Instead of Monads, Clojure has mutable reference types; in that sense, it deviates from being a pure functional language. However, the semantics for mutating said reference types are strictly defined by the language -- it's not anarchy, not like Java. Clojure is much safer. The only way you can get yourself into trouble is my calling out to Java; if only Clojure had an effects system: then I would label all my Java code impure and be done with it. That'd be tricky though, being that Clojure is dynamic and everything. I've written several thousand lines of Clojure in the past few months, and its so dynamic: it could change any minute. In fact, as anyone who has ever worked with me will tell you, it does change every minute. and every half a minute. and not just little bits change. no, the whole thing changes. constant mutation! Most of it I've written over and over again in sundry different ways, since it took me a while to figure out what I was doing. But it's always that way with strange new languages. Some people are lulled into thinking that one programming language is much like another -- but it isn't so. C and Java and C++ and all those other gross static imperative languages may all seem pretty much the same, but beyond their provincial pale is a whole galaxy of strange and unnatural languigy artifice. I recall that Shakespeare line about living in a nutshell and thinking yourself king of infinite space. That's how people get. I'm probably that way. Clojure isn't nearly as hardcore as Haskell; I know, I've tried to write Haskell: and I've failed. You'll fail too! and if you don't fail... well.... then shame on you for being so smart. Remember, it wasn't me, it was Ted Neward who said: No Monads!

Language Libraries

nathanial.hartman@gmail.com (Nathan Hartman) — Sat, 07 Feb 2009 10:41:00 +0000

Libraries are a part of language's expressiveness. A language can be compressible and foldable, but if it doesn't have the right libraries, you'll write more code than you have to. Therefore, when you select a language for a project, emphasize libraries.

I don't think Python is a particularly expressive language. It has crappy broken lambdas, no syntax for arbitrary anonymous functions, no currying, no encapsulation, no overloading, no macros, and no regular expression syntax and lame string interpolation. As a scripting language I think Ruby is a whole lot better, and as an application language I think Scala trumps them both. But I'm using Python right because it has the libraries I need. Library beats language feature.
For large do-it-yourself projects it's the opposite: language beats library, because if you're going to be writing your own libraries, you don't want to pay a large language tax while you're doing it. Steve Yegge has talked about the enormous 1/2 Million line game he wrote, and he wishes he'd chosen a more expressive language than Java -- code size becomes a maintenance hazard; the less code you have the happier you'll be, and that is the foremost reason for picking an expressive language in large projects such as his: to avoid paying the language tax and to avoid producing a monstrously large code-base. Languages that support lots of higher order features like Scala or Ruby let you compress your code and factor out syntactic patterns (like for-each) which are intractable in languages with out higher order functions. In such languages, the control structures that come with the language are all that you're going to get, whereas in Scala or Ruby there's syntax for writing your own 'foreach', or you own 'do_times', or your own 'synchronized' etcetera.
Most of the projects I've worked on have not been more than a few thousand lines, ever; and that is because I've made extensive use of libraries and avoided writing thing myself. On that basis I will choose an anemic language like Python or (god forbid) Java if that's what it takes to get things done. Thankfully, I rarely have to choose Java anymore because there are so many other better languages on the JVM which can seamlessly interop with old Java code; but as in the case of Python, I still have to occasionally make the hard choice between a richer platform, or a richer language. And I choose a richer platform almost all of time: with the one exception of (as I said) large do-it-yourself projects for which you do not want to have to pay the language tax.

Streams

nathanial.hartman@gmail.com (Nathan Hartman) — Sat, 10 Jan 2009 03:23:00 +0000

So, to lead into this: Scala, despite being a quasi-functional language, doesn't have much laziness support. There is a lazy keyword, and you can have lazy arguments, but by default everything is eager including 99% of the collection functions. Why is this a problem? Because I've changed my mind and decided that instead of returning the first answer to my question, I want all possible answers; except I don't want to pay the price of computing all possible answers until I actually need them, ergo I want a lazy collection. But in order to get my lazy collection, I have to convert my entire API over to Streams, which is the one token lazy sequence in the entire Scala standard library. And it's a hassle!
Clojure, on the other hand, has almost everything set to lazy. Many was the time that I thought: "Hmmm... this would be so much more convenient if I were using Clojure." For whatever reason, I did not listen to myself.
But it's not too late. Perhaps I will make a hybrid project of Scala and Clojure. I'm not sure how difficult that would be, or even if it is possible at all. Maybe it'll be too ugly. Alternatively I could just convert entirely over to Clojure... but that seems like wasted effort; I don't want to keep reimplementing projects in different languages -- throws away too much work. Lastly I could just bite the bullet and use Scala's damn nasty Stream crap, and live with a duplicated API for streams and not-streams.
Something to ponder.

Constraints ReReVisited

nathanial.hartman@gmail.com (Nathan Hartman) — Thu, 08 Jan 2009 02:39:00 +0000

My wonderful and glorious, and long_windedly_titled Finite-Domain-Constraint-Satisfaction-Library, now solves Sudoku and sundry other Finite-Domain-Constraint-Satisfaction-Problems. Special thanks to Peter Norvig and the wonderful "AI: A Modern Approach" book. Also special thanks to the Cream constraint library (which was the original inspiration) and the Oz/Mozart language/library.
For those of you who haven't been following the progress of my library, it's written in Scala, and clocks in at just over 2000 loc; thats 1000 of library code plus 1000 lines of testing code -- which was essential because I refactored this thing so many times... if I hadn't had the tests I would have never gotten it working.
The future of the library is bright. There are more things to add and some improvements to make; however, its core parts seem to be feature complete. It uses the AC-3 algorithm to do the constraint propagation, and a simple backtracking search to solve the problems (if constraint propagation by itself doesn't do the trick). It maintains arc-consistency as it searches, and the next-variable-picker-algorithm, the constraint-propogation-algorithm, and the search-algorithm are all swappable. I'll be implementing more algorithms as time goes on, so the user may select the best strategy for solving their particular problem. Right now next-variables are selected using the Most-Restricted-Variable strategy, but for completeness I definitely need to add more options.
Oh, and special thanks to the Eular Project, which got me off on this constraints tangent in the first place.

Constraints Revisited

nathanial.hartman@gmail.com (Nathan Hartman) — Tue, 30 Dec 2008 09:05:00 +0000

In search problems, you often have a great many potential solutions in your space that never need visiting. Many of these solutions are obviously contradictory, and can be discarded early. By applying constraints and enforcing consistency of constraints on your solution space, you can prune off contradictory solutions before starting your search. Thus, can many problems with exponentially large numbers of potential solutions be reduced to much smaller, much more tractable problems of fewer solutions.
There. I said it.

Update: My constraint library is going well. I think it is approaching actual usefulness. Yesterday I fixed my domains so that they can contain arbitrary disjoint ranges of values; like unions of a bunch of different intervals. Today I'm implementing iterative depth first search, and the corresponding tests, as well as a couple new constraints like allDiff and notEq. Using operator overloading in Scala, constraint problems look like this:

val a = problem.newVar
val b = problem.newVar
a + b := 20
a - b := 10
problem.solve

and then you inspect the resulting values of 'a' and 'b' to find out what the solution is. Right now, since I don't have the search implemented and can only propagate constraints upto arc-consistency, problems like the above don't return a definite answer, just a range. But other problems that aren't so tricky do generate definite answers, and in any case, the current implementation correctly narrows the potential solutions for 'a' to "10 < x < 20" and for 'b' to "0 < x < 10". Since the naive alternative is an infinity of integers, you can undoubtedly see the utility.

Other Interesting Things:
I used TDD to begin the project, and I'm pleasantly surprised. It seems almost the natural way of doing it: stubbing out your project with client code (in test form) gets the ball rolling and the mental wheels turning. If you don't know where to start, pretend you already finished and write some tests to make certain you did a good job. They fail, you fix. No more staring at a blank page.
And unit tests are life! I think I would die if I had to do a project without them. I'm starting to find that the more unit tests I add, the better I feel. It's like a drug. Write the test, fix the bug, refactor rinse repeat. Soon I'll be one of those guys who has to have 100 percent code coverage. Nathan Hungry. Nathan Want Code Coverage.

Manic Programming

nathanial.hartman@gmail.com (Nathan Hartman) — Fri, 12 Dec 2008 12:10:00 +0000

Originally this post was going to be about Constraint Programming; but I've discovered, as I've attempted to write about it, that Constraint Programming is too technical. I just don't understand it well enough to explain it properly. There is, out there, a concise and correct explanation of Constraint Programming; I just can't produce it. So I'll talk about something else: something that is not technical, but totally subjective and therefore not bound by correctness or accuracy. I turn to my own topic: Manic Programming by Nathan.
Manic Programming is the phrase I've coined for myself to describe my own tendency to do a lot of different things without getting anything in particular done. I do a lot of hobby projects, and they all seem to have a characteristic short lifespan: they go on for awhile, and then I find a branch point somewhere and speed off on a tangent. I've heard this tendency described as "shaving the yack": to get lost in the depths of some portion of a problem. In my own case there remains so much I don't know (and may never know) about the computer field, that my curiosity gets the best of me and in lieu of finishing the project I'm working on, I invariably go off and try something else.
Now I'm not sure that there is anything wrong with not finishing hobby projects; they are, after all, just hobby projects. No one is paying me to do them. However, it would be nice to be like a photographer or an artist, and have a ready made pile of projects to show off to interested individuals. Instead I have a heap of partial works which can be enjoyed only by me.
So what is it that prevents me from finishing projects? I don't really know, but I can observe this general thing: that projects get boring. Part of it is that you become familiar with the concepts that spawned the project in the first place, and curiosity therefore dwindles. Another part of it is that finishing is tedious. In the Mythical Man Month, somewhere there's a graph which shows that the ratio of difficulty between a product and a do-it-for-yourself project is 9 to 1 which I think is basically true. Finishing really is hard. Polishing really is tedious, and in most cases it's just more fun to leave it and do something else.
And in defense of myself I'm not sure that Manic Programming is a bad thing! I do cover a lot of ground with my many orphan projects. It is my breadth first search of the knowledge tree. I do like to learn, but all at once and not one at a time. It probably hurts my marketability that I do not emphasize one particular thing, and people can't say "that guy, he is good at X". No I'm not good at X, but I'm mildly familiar with Y and Z.
I have neglected this blog for awhile, mostly because my enthusiasm ran out and I moved on to other things (the mania strikes again). However, I feel the need now to resume. Notice: perhaps the one thing that saves me from the mania, is my tendency to revisit old discarded projects and apply to them new methods. And behold, I revisit this blog after a period of neglect! And what new methods will I apply here? I wonder. Perhaps some code samples? A bit of technical jargon? Some inaccurate technical expositions on NP-Completeness? Stay tuned to find out.

Haskell, Lambert, and the Clarke Ellipsoid

nathanial.hartman@gmail.com (Nathan Hartman) — Mon, 27 Oct 2008 08:51:00 +0000

Math is best done in a functional language. I've browsed through the source of several map projection libraries, and none of them take a functional approach or use a functional language; some are simply imperative C, while others written in Java actually attempt to make map projections OO, and have super classes and inherited methods. Unnatural!! If math isn't functional, then nothing is. What I want to see in a projection library is the same thing I see in a map projection book: a clear, concise representation of the projection algorithm. Neither Proj4, nor its Java port, nor the Geotools library provides that clear representation. So I've started my own projection library, in Haskell.
Normally it's bad it start libraries that duplicate the functionality of other libraries; and in fact no one but myself is likely to ever use this Haskell port of Proj4, but I just can't help but thinking that map projection is a perfect application for Haskell. Everything is very mathy, which Haskell handles well, and 99% of everything is functional. Projection algorithms do not have side effects. Almost the entire library can be implemented in purely functional code. And no language is better, or more clear or more concise than Haskell for purely functional code. That's what Haskell is made for.
What will be interesting though, is after I have a few more projection algorithms implemented, seeing how I can hook up the library to the EPSG database, so I can pull projection parameters instead of having the user manually enter them all in. Proj4 has its own xml mapping from EPSG codes to projection parameters, and Geotools has its very own HSQL database of EPSG junk. I'll have to see what persistence options the oh so functional Haskell has to offer.

Library vs. Language

nathanial.hartman@gmail.com (Nathan Hartman) — Thu, 16 Oct 2008 19:25:00 +0000

One never wants to write Java in Ruby. But sometimes, you don't have a choice. Libraries are typically not language agnostic. A library reflects the language it's written in. Write a library in A, try to use it in B, and you'll see an aesthetic clash resulting from a mismatch of idioms. This mismatch is why it can be so unpleasant to use languages other than Java on the JVM. Most of the libraries on the JVM were written in Java, and therefore expect to be interacted with in a Java way. So when you fire up your fancy new dynamic language, with literal maps and lists, and blocks and procs and yields and dynamic this and vararg that, you discover none of those fancy features are applicable to your libraries, because they weren't written with those things in mind. This isn't so much of a problem for JRuby, because they do go out of their way to make Java look more like Ruby, but I find it's a major problem in a language like Clojure; which to begin with, doesn't resemble Java at all (i.e. isn't OO) and has very few in the way of native libraries. When I compare the user experience of writing Common Lisp code with Common Lisp libraries, it looks and feels so much better than writing Clojure code with Java libraries. All the fancy lisp in the world can't save you from the evil Java library monster.
I think one of the things the Clojure community has to do, besides come up with their own libraries, is to start writing native Clojure wrappers around Java libraries. I'm not sure if such a project is feasible, but I know that I'd rather write separate bindings than pollute my Clojure code with Javaness. Honestly I wonder if issues like these will get more attention now that virtual machines are becoming this great big multi-language platforms. Maybe someone will find a magic solution. Skinnable language agnostic libraries! With Java, Clojure, Ruby, and Python stylesheets, get yours now!

GAE

nathanial.hartman@gmail.com (Nathan Hartman) — Thu, 16 Oct 2008 06:21:00 +0000

Google App Engine is supposed to let you seamlessly scale your app on Google's hardware. But I'm afraid it isn't so. Not only does GAE not scale (they put caps on everything), but the non-relational backend they force you to use (for scalability) makes Django support horribly painful. It's a miracle that Django manages to work at all, and it's an even greater miracle how much of Django you can actually get working with something like app-engine-patch or appengine-helper. But though it works somewhat, it doesn't quite work enough.
ModelFormSets! I need them desparately, but for some reason they work not one bit. I try to use them and I get metaclass exceptions--presumably the result of messy leftovers from the app-engine-patch. So instead of using them I must not use them, and make do with regular FormSets, and the consequent data ingress/egress code which I must write manually. It is a recipe for spaghettization, which is what I'm experiencing. And however desparate I feel to stamp out this crufty code, without a fully operational Django it seems I cannot.
So that brings me to my point, which is that if GAE is to be useful at all, it can't be incompatible with the standard way of doing things. What's the point of hypothetically scaling, when none of the technology interoperates smoothly, and I (the humble developer) must spend ridiculous amounts of time puzzling out why some feature I desparately need simply won't work? And then after I'm done puzzling I have to out and spend more ridiculous amounts of time reimplementing the feature for myself; or worse, not implementing the feature and just letting my code overpopulate with redundant crud.
To hell with app engine. Give me back my relational database and real python.

Django Templates

nathanial.hartman@gmail.com (Nathan Hartman) — Fri, 10 Oct 2008 02:16:00 +0000

Django templates really blow. It was my understanding the the Python philosophy was not to prevent the programmer from doing things; even if they were considered dangerous. But then the excuse for these hands-tied-behind-your-back templates is nullified -- the only possible reason to create them and have them be so limited and restricted is to prevent people from getting into trouble, as in by over-coding the View or something. Now, I understand how we don't want our pretty Python views to turn into PHP, but come on. If I need it, I should be able to get it. The only thing I can say is thank god for Mako; it was surprisingly easy to replace the native Django craplets with the far superior and flexible Mako, even on the unnatural and unforgiving GAE.
I thought I really liked Django, but having to work with these evil templates has turned me 180 degrees. Erb from Rails didn't have this evilness, not even JSF had this evilness -- with JSF you could either slip back into JSP, or write custom tags (with relatively little hastle). I don't even want to touch Django's "filter" bastardizations. Ick.
On the magicness scale, Django is certainly a step down from Rails, but until now I had thought that was a good step down, a step down that let the programmer more easily see behind the curtain (as it were). But having to work extensively with these Django templates has altered my opinion totally. When the framework stops working for me, and starts actively opposing my actions, it's time for a change.
Hurray Mako, boo Django. Maybe I ought to switch over to Pylons or Turbogears for a bit, see what those are like.

Python vs Ruby

nathanial.hartman@gmail.com (Nathan Hartman) — Mon, 06 Oct 2008 14:05:00 +0000

I here it told that Ruby was, in part, an OO reaction to Python, i.e. that Matz didn't think Python was object oriented enough, and sought a better way. Python is certainly more functional (in the first class function sense) than Ruby -- indeed, methods just seem like syntacic sugar of functions over objects, with an implict self parameter instead of an explicit one. Objects themselves are little more than hashmaps.
But despite being a little hairier around the edges, I think Python's emphasis on first class functions over first class objects makes it more flexible. It's easier to manipulate methods in Python than it is in Ruby; in Ruby there's too much lambda/proc/block noise, and isn't so clear that methods on objects are simply functions with an implicit self parameter -- I'm not even sure that you can pull methods off objects in Ruby the way you can in Python, but then I haven't tried. Ruby encourages a more OOP perspective on things, which is not the case in Python.
It's also interesting to note that Django, perhaps in keeping with Python's relative hairyness, also doesn't bother with hiding too many nitty gritty details. In Django there just isn't as much magic. The request parameters get explicitly passed into the function, they don't magically appear. And it doesn't force you to do MVC. It's up to you. I heard one of the creators of Django say in a podcast, that he considers Django to be more like the factory for making the tools to make the product -- and a layer below Rails in abstraction.
Perhaps because I find it less magical and easier to understand, I prefer Django over Rails. It's not exactly a fair comparison, because my experience with Rails is limited to when I was young and foolish, but I would say that as a general principle, too much magic and you lose your way. I need to have my feet firmly on the ground; I want to know what's going on or else I feel out of control. I'm going to do another project in Rails; perhaps a website for my theorem prover, and in that sense I'll be giving it another chance -- but I think Django is the one for me.
It's also good to note that Python is considerably faster than Ruby.

Sundry Thoughts

nathanial.hartman@gmail.com (Nathan Hartman) — Mon, 22 Sep 2008 05:02:00 +0000

Git is the best version control system ever.
College is one of those things I have to do instead of programming, which is what I really want to do but can't cause I need a degree or else people won't pay me. It sucks especially cause I'm almost through the crappy general-ed part where I have to take writing classes and discussion classes, and almost onto the kick-ass degree only part; where I'll be programming like a madman. I'm not even sure anymore if I care about getting A's. I just want to pass the classes and get on with the good-stuff. My enthusiasm for school has been worse, but it's getting down to the ground right now. I was briefly interested in the new college experience; real college is a lot different than junior college, culture-wise; and that was interesting for awhile, but now I'm starting to feel closed in by homework and tests. It's not that the curriculum is so hard, as it is that I just can't keep myself together enough to push through things that aren't immediately gratifying. That's one of my bigger problems.
Other people get narcolepsy or epilepsy, I get avoid-olepsy. I avoid things I don't want to do, instead of doing them. It's not the most beneficial strategy I know, but it's obviously hardwired into my brain because I can't seem to stop! I need a self help book...
However, today was a good day by emotional standards. Woke up, felt good--didn't have a headache, unlike the past couple days. Now if only the neighborhood would regain power so that I might take my nightly walks without bumbling in the dark, everything would be great.

The problem with college teachers is that they repeatedly, and obviously erroneously, refer to their knowledge of the real world, and assert that in the real world, the stuff that they teach in the academic world actually matters. When of course, we all know that is not the case. What's astonishing (or perhaps only natural) is that teachers, who by vocation rarely step out into the "real world", unceasingly make reference to it. This may be an example of the Dunning-Kruger effect; so divorced from realness are they, that they paradoxically (and mistakenly) assume they know everything about it. But perhaps I give them too little credit. Certainly some of them occasionally breach the ivory tower--but I doubt it is as many as claim to.

A Question of Ajax

nathanial.hartman@gmail.com (Nathan Hartman) — Sat, 20 Sep 2008 06:56:00 +0000

So I've reached the point in Mipper, my Mips interpreter, that I'm constructing the website; indeed, I've reached the point in constructing the website where I have to decide just how to connect the server-side Mips interpreter, to my client-side UI. The obvious choice is Ajax... but just how much Ajax is the right amout of Ajax.
What I need to do is allow the Mips proggies to execute on the server side, where my pythonic interpreter lives (I'm unwilling to port it to javascript to run on the browser: I like it fine where it is), while somehow getting the output and register and memory changes updated to the browser. This task would be simple enough (just send the data over) if I had infinite bandwidth and a fiber optic cable, but I don't. I can only update the client side gui infrequently unless I want to overload my connection: imagine me sending data for every iteration of a tight loop. Not gonna happen!
What I decided I need to do, is exactly what debuggers like eclipse do: offer two modes, step and break. In break mode, the ui only stops and updates when you reach a breakpoint in the code. Step mode would be the interpreter executing one instruction at a time, per user request. Both these solutions would be relatively light on the bandwidth while at the same time giving the user the information that they need. And indeed, I think I'm going to implement these two options and have that be the way of it. I might in the future consider porting my pythonic interpreter to javascript, but for now it's just redundant time wasting. I have to get this project finished or else I'm never going to be able to concentrate on my school work.
Which brings me to my final point: I'm neglecting my school work for programming. Like an irresponsible pair of lovers, skipping class to make out in the dark, me and my project are neglecting our friends and our obligations. But once I've finished with it, oh what accomplishment! The website already looks great; I got a template off www.owsd.com called missunderstood that looks great.
Django has been a pleasure to work with. It has less magic than rails, but I think that is a good thing. Rails is often too behind-the-scenes; sometimes I want to be able to see what is going on. It gets to the point with rails, for me, that I don't even feel in control of the app anymore. But I really shouldn't say that now; I was less experienced when last I used it for anything. When next I come back to it, then I'll pass my final judgement.
For the client side behavior, I'm using a combination of JQuery (for DOM manipulation) and Ext for sexy user interface. Only the actual Run-the-Code part of the app relies on Ext. The rest is standard xhtml and css. Truly I don't like the ultra-scripty websites that're wannabe desktop apps; so called thick-rich-webapps. Discretion please. The web is supposed to look like the web.

Python Goodness

nathanial.hartman@gmail.com (Nathan Hartman) — Thu, 18 Sep 2008 12:47:00 +0000

It's possible that I like python more than ruby. Python seems to emphasize higher order functions more than ruby does; I mean, it's possible to do functional-ish programming in ruby, but not idiomatic. In python it might just be idiomatic. At least I have found it more natural to do functional-type-things in python than I did in ruby; and maybe that's just because I was using ruby 6 months ago, before I had touched Haskell or Clojure, and therefore didn't have a mind for composing functions, and making functions that return other functions, etc. I'll have to go back to ruby after this, and see if my new understandings make any difference.
Project Update: my MIPs interpreter, written in python, is working. It supports all the standard op codes (that I am aware of) along with the typical pseudo-instructions. The remaining issues with it are overflows and byte vs. word memory. Right now I'm just treating everything as words, no individual bytes, which is incorrect. And I'm not dealing with overflows correctly. In signed operations, I detect overflows and throw an exception, but in unsigned operations I don't do anything, just keep adding. That's a problem because, according to the bits, when I overflow an add I should go negative, but because python can handle arbitrary precision numbers, I have to build in that un-functionality myself. And it's a bummer man!
Instead of engineering my interpreter to maximum compliance, I'm starting the web portion of the project; wherein I create a web-gui in Django for the interpreter so I can put it up on Google App Engine, and show it off to everybody. Now, I don't know anything about Django; just that it's in Python and it is of the Rails Generation of web frameworks. Hopefully that means I'll be able to pick it up by analogy, but maybe not. I think I might do a significant portion of the UI in Javascript, and just serve up the interpretation functionality from the server-side. I've been wanting to give the EXT framework a serious go-round for awhile now, and this just might be my opportunity to do that. Though I do caution myself against trying to make a desktop app on the internet: I hear that is bad form, cause the web is supposed to look like the web not the desktop. If that is the case, then EXT is a bad influence: it's whole purpose seems to be bringing desktopishness to websites. Is it evil?
In any case, I'll just have to see what works. But you mark my words, I'm incorporating javascript somewhere into my application. Aside from the UI, the other big opportunity looks to be Ajax. Obviously the server-side interpreter code is going to have to communicate with the client somewhere, and why not in real-time, with no pesky click-refresh cycle to ruin the flow?
Anyway, hurricane Ike interrupted my school, so I have more to time waste on coding, hurray!

Mipper

nathanial.hartman@gmail.com (Nathan Hartman) — Mon, 08 Sep 2008 22:09:00 +0000

Since I'm a crazy person, and this semester I'm doing a computer architecture class, I've decided to write a MIPS emulator. Originally I was just going to do it in Java using ANTLR, and indeed that's what I spent Saturday on, but then I figured how cool would it be if I could throw the emulator up on Google App Engine once I was done. But App Engine only supports Python, so now I have to rewrite it in Python, which is what I spent Sunday on.
ANTLR has a python output target, but it's not fully supported and I'm afraid of it, so I decided to just roll the parser/lexer myself; normally I would say "Don't Do That!" but MIPS (or any asm) is pretty simple--that is, doesn't require complex parsing algorithms, so it can reasonably be done by hand without much of a fuss.
So now the lexer and parser are done, and I can merrily digest MIPS assembly code; what remains is to specify the behavior. I can do some of that now... but I still have some things to learn about MIPS behavior before I can complete my emulator. I don't think the emulator itself is so hard as figuring out how to shoe-horn it into app engine when I'm done, particularly since I have zero prior experience with Python or Django.
Some comments on Python: When I think Python, I think Haskell-Ruby. The indentation-matters approach is totally Haskell, and so are the tuples and list comprehensions, while the dynamic typing and scriptyness is Ruby. Lambdas are Haskell-Ruby, and the tuple-unwrapping is kindof like Haskell's pattern matching (though of a lesser magnitude).

Actors

nathanial.hartman@gmail.com (Nathan Hartman) — Fri, 05 Sep 2008 20:43:00 +0000

The Scala Actors library is gods gift to the JVM. In the absence of the mysterious and not yet fully realized STM, concurrency problems need Actors. Why do they need Actors? Because if they don't have Actors, then they have locks, and if they have locks, then they have deadlocks and unlocks and inconsistent shared global state spahgetti disasters. Actors solve this problem by keeping state local to the actors, and only allowing communication via message passing.
Loosely, each Actor belongs to its own thread of a execution; notice, you can pool them for efficiency. And each Actor communicates with other Actors via message passing: an asynchronous communication between the seperate objects not liable to be responded to, necessarily. As a consequence of the model, Actors tend to form loosely coupled asynchronous event-driven systems (without, necessarily, all the trappings of a traditional event-driven system).
Why do I care about Actors? Well I can't tell you. It might infringe on my old company's IP, and I wouldn't want to get sued now would I? However, let us say that I have experience with non actor based systems, and I have experience with my own personal actor based systems, and the latter is infinitely superior to the former.
Back in the day when I didn't know anything about anything, I tried to multithread a program by multithreading a program; which to me meant threading absolutely everything, and synchronizing absolutely everything. Of course, this was absolutely a disaster, and I will never do it again. If ever I am forced to uses locks again, it will be under protest, and I will cry myself to sleep; also I will emphasize, as everyone should, atomicity and resource pooling--for safety purposes in an inherently dangerous situation. It is unfortunate that I didn't discover the mysterious and wonderful language of Scala, and it's glorious Actor library sooner, cause it would have solved a lot of my problems.
As you may or may not know, Scala Actors are based on Erlang Actors, which in turn represent Erlang's answer to the concurrency question. As mentioned before, they operate by keeping state actor-local, and using asynchronous message passing for communication. What this produces is a loosely coupled asynchronous event-driven-esque system, which can be made extremely (emphasis extremely) fault tolerant through the use of a Supervisor-Worker pattern.
The Supervisor-Worker pattern (and I don't know if that's really the name, but in the name we have the two most important elements), is a relationship, like you might expect, between Actors designated as Supervisors which watch Actors designated as Workers, looking for signs of failure. If a worker fails, the Supervisor executes appropraite behavior to bring the worker back online, or else in some way handle the fault and keep the system from going down. This methodology is practiced and proven: evidence being the extraordinary uptime of some Erlang systems, nine nines or whatever.
What makes all this relevent to me is that if you ever were, hypothetically, to design a polling system for the collection and collation of sensor data, you might want to look at the Actor based approached, certainly in Scala and possibly in Erlang -- if you can stomach getting away from the Java platform. The reason being, as stated, you can produce very fault tolerant systems. And indeed, the systems just look nice anyway. An Actor is an easy abstraction to understand; the only drawback is that generally (unless you're willing to specify timeouts or risk deadlocks) the whole thing is asynchronous, so you have to adapt yourself to the asynchronous lifestyle as it were. I don't find this is too difficult for the work that I've done, but in more complicated software that has order-dependent logic in it (as in, things must be sequenced) you might have a greater issue with asynchronicity. However, I think this might be solvable using Monads, and I've implemented something like that; where you bind monadic Request objects together such that as one request is fulfilled, another is sent off, all in order. But I haven't used that in a large system so I can't tell you if it really works, especially if you are using a language that doesn't really support monadic types.
Actors rule!
Beware Locks!
What made me really think about Actors, besides actually using them, was taking the bus. I thought to myself, taking the bus is like a huge concurrency problem. You've got all these seperate threads (buses and people), that are supposed to meet at specified times (joins or walls or whatever). The problems arise when a bus runs late, and then you miss your transfer to the next bus, and all of a sudden your standing idle on the corner while you wait for the next. That's so much like a concurrency problem. Unfortunately, I think that the actor version of taking the bus is actually having your own car. And I'm way too green for that.

Started School

nathanial.hartman@gmail.com (Nathan Hartman) — Thu, 04 Sep 2008 12:57:00 +0000

I've started my first semester at U of H. It is nice being at an actual university for once. They have a very nice library with a substantial collection of CompSci material, which I am quite enjoying. I've picked up this book on "First Order Logic and Automated Theorem Proving", which is right up my alley. And you know, I've found that as I've gotten older and become more experienced with math, it's all totally understandable. Maybe it was experimenting with functional languages, especially Haskell which is very math-esque, but my ability to understand mathematical statements (algorithms and proofs in particular) has expanded noticably. Part of this is simply development. Up to a point (say your mid thirties) you do get more mentally competent. That's not to say I'm going to make very many personal breakthroughs in the upcoming 15 years, but I can at least count on getting better at things I practice at. After 35, of course, it's all downhill, so I'd better be good enough to go into something soft like management, else face the addlement of age in a technically demanding profession; which I can tell you is not ideal, having observed many over-ripe male college professors who are no longer capable of professing their profession!
Which brings me to a critique of my current classes. I'm only taking one course in CompSci, and that is Computer Architecture; and unfortunately it is yet another class taught by an old man whose many years of long life and toil have leeched away all enthusiasm from his head. His brain-cells cry out for sleep, and I would indulge them.
Thank goodness I don't require teachers to learn, because otherwise I'd be screwed at the prestigious University of Houston. I can tell why so many CompSci graduates have no idea about anything when they get out of school. If you don't have the ability and motivation to educate yourself, you're at the mercy of whatever institution you're at; and in my case, that would mean a mediocre education at best.
I watched a documentary recently about the declining standards of American colleges, i.e. grade inflation and drunken fratboys scating through their degrees. I don't know if it's really all that bad; but I can assert that any highschool teacher who uses the excuse when professing to her students "Learn this cause you'll need it in college" -- she is likely mistaken.
My calculus and PoliSci classes are really big, but they're both taught by experienced and competent teachers, so it's ok. My writing class is taught by a psychotic woman (who is fun, but crazy). My discussion class is taught by a middle-aged football player who has a sarcastic wit and no seriousness. What other class am I taking.... I can't remember. Anyway, I get to spend lots of time reading cool books in the library. I wish HCC had had a better compsci collection. It's a real constrast from U of H, and I bet someplace like Rice or another top university would have an even more impressive (and enviable) collection.

Libraries

nathanial.hartman@gmail.com (Nathan Hartman) — Sat, 09 Aug 2008 04:31:00 +0000

The most important piece of Unix philosophy for programs is "do one thing and do it well". If only more programs, and more libraries, followed that basic principle, and stopped bloating their code with runaway featuritis. As a real-life example of what to do, and what not to do, I take Proj4 and Geotools.
Proj4 is a C program/library that projects coordinates from one system to another. It doesn't do or include anything else -- i.e. have a gui or receive emails. The one thing that it does though, it does to exhaustion. It handles a laundry list of projections and conversions: indeed, the complete laundry list as far as I can tell (though I'm not expert). And it's easy to use from the command line or as an api.
Geotools, in contrast, is a bloated monster of a Java framework. It handles all things Geo. It can read shapefiles, csv files, gml files, mif and vpf (god knows what those are). It graphs, it visualizes, it projects and converts. And I venture to say it does no one thing particularly well.
I've only used it for projections, so perhaps I'm biased. You might point out that obviously Proj4 is going to be better at projections, cause that's all it does anyway, whereas Geotools has to a bunch of things. I'd argue that the Unix "one thing do it well" is a superior, and more composable, design than the "do it all at once" alternative. I'd rather have a bunch of cohesive and exhaustive libraries that I can compose together to get the functionality I need, than a massive behemoth that I must take or leave.
In all fairness, this post is motivated more by my frustration with Geotools weird plugin architecture than my concerns for their code-base. I spent a considerable amount of time trying to figure out how their crazy service-registry/factory/class-loader contraption worked, and it honestly left me exhausted. They have attempted to make their massive framework modular by (wait for it) dividing their code into modules... but the way they tie it all together is absolutely horrendous. They have a home-baked auto-wiring solution, like what you'd get in Spring (but would never use cause they advise against it and it's bad anyway). If you drop a jar on the classpath that contains an implementation class, their service-registry/factory/class-loader contraption attempts to auto-magically suck it up and make it available. Obviously this only works under the best of circumstances, and requires a deep understanding of the contraptions inner-workings of it is to be debugged or modified at all. There is a reason the Spring framework (and presumably other similiar frameworks) avoid auto-wiring and have configuration files. If for no other reason than a configuration file is more explicit: at least then you know which concrete classes are implementating your interfaces. In this case, I had to step through the initialization process with the debugger to find out what concrete classes I was getting. Ick!
Of course, I'm sure they had some mangled reason to do what they did. It's just frustrating coming in from the outside and not seeing the benefits of this madness, whatever they are--or if they are.

C version

nathanial.hartman@gmail.com (Nathan Hartman) — Sat, 09 Aug 2008 04:09:00 +0000

The C version of my Corpscon clone runs in about 3.9 seconds (for a 13mb shape file), versus the Java version which takes between 6 and 10 seconds. I'm surprised that I get such a performance boost; I'd thought that Java performance would be closer to the C performance. Of course, a 50% (or even 250%) increase in runtime may not seem very large--and it's indisputable that both implementations are fast enough, but I've read so many benchmarks that put Java right up there with C and C++. I guess that's only for certain things, and not for others. And that's only when you turn on the -server flag, and let the JVM eat your memory.
What's also interesting is that the lowest Java runtime, the 6 seconds, is based on an NIO implementation of my program which was far more complicated than the corresponding C implementation. Whereas C does all it's buffering/file-magic behind the scenes, and lets you fetch one char at a time penalty free, Java demands that you explicitly buffer everything, and that you call upon the complex NIO library to get C-level performance. And you don't even get it! NIO, you lied to me.
I'm certainly not an IO expert: I don't know the secrets of cache and optimal buffering. But in C I don't have to be... it appears to be done for me by the compiler. I'm sure I could extract even more performance if I knew what flags to set on GCC. I need to find a book on how caching works, and how I can utilize it to improve performance.
Another thing that might have influenced the C vs Java benchmark is that in C, I can use floats that are supported natively by the operating system, because C (as far as I know) does not guarantee consistent float behavior, unlike Java which does. Because C lets the hardware take care of the float stuff, it gets the best possible performance, whereas Java has to do software-level work to ensure consistency, and therefore looses performance. I need to get a profiler or something, to better understand the bottlenecks in my programs.

IO (revisited)

nathanial.hartman@gmail.com (Nathan Hartman) — Wed, 30 Jul 2008 14:14:00 +0000

Alas, my attempt at multithreading my corpscon clone (to speed up its conversion of files from one coordinate system to another) ended in no appreciable speed increase or decrease, but rather a drop in efficiency: it now uses twice as many cpu cycles, hurray!

IO

nathanial.hartman@gmail.com (Nathan Hartman) — Sat, 26 Jul 2008 10:30:00 +0000

Java IO seems pretty fast, assuming you properly buffer your input and output, but I keep wondering if C++ (or C for that matter) is much faster. I've seen conflicting benchmarks on both sides of that argument. What's clear is that at least all agree that Java is fast enough for general purposes. But to extract maximum performance, it may be necessary to go the native route... get down and dirty with gcc, as it were.
It's clear from experience (not mine, but other people's) that using a fast language does not make a fast program. More often than not, the theoretical performance of a low level language (like C) is lost in the complexity of the application. Because low level languages make programs harder to reason about, things don't get optimized to the point they could be. That's how you get the Ruby on Rails outperforming Java webapps phenomenon. Ruby the language is 50 or 100 times slower, but Rails the framework makes possible all the performance tuning you simply wouldn't have time for a Java app. The easier a program is to think about, the easier it is to optimize.
But what is simpler than reading a file and writing a file? The IO that my program is doing doesn't tax the brain. It must read in millions of points, compute some conversions, and write them out. In Out, IO!
Because I don't have to deal with a great deal of complexity, I feel the urge to use a low level language, just to see if I can extract more performance. It takes my Java version (after quite a bit of optimization using the low level NIO library) about 6 seconds to read a 15 megabyte file, compute the conversions, and write it all back out. That's pretty good, and possibly good enough, but I want to see what the limit is. I'm spending about 1.5 seconds reading, 1.5 calculating, and 3 seconds writing. I didn't use NIO for the output part of the program, and I imagine using it will bring me down to 4.5 seconds total. But I know the harddrive can read faster than 10/megabytes a second, so obviously there's some overhead that is slowing me down.
Part of the overhead is clear: each time I read a certain number of points, I stop to convert and write them to a file. So I keep repeating: Read, Convert, Write. What I should be doing is
Read Read Read
Convert Convert Convert
Write Write Write .... all in parallel. So I'm going to try executing the three pieces concurrently and seeing if that boosts performance. I think it will. Even if it does, it should be interesting to reimplement in C or C++, to see the absolute limit of performance.