Tuesday, April 22, 2003

The .NET Abstraction Pile

An abstraction is a boundary with two sides.  On the top side, the abstraction presents a simplified view.  Below, there is something more complex and more real.  The purpose of the abstraction is to obscure what is really going on.

The world hidden underneath an abstraction is quite likely to be yet another abstraction.  In fact, it is typical to have many abstractions stacked together, each one attempting to present an illusion which is even further from the truth.  If you stack them up vertically, the ones at the bottom are more real than the ones at the top.

This is what programmers do.  We build piles of abstractions.  We design our own abstractions and then pile them up on top of layers we got from somebody else.  Abstractions can be great.  We use them because they save us a lot of time.  But abstractions can also cause lots of problems.  They're never perfect, as Joel Spolsky explains in his excellent article on "The Law of Leaky Abstractions".

But you can't have the benefits of an abstraction without its risks.  We need to make wise decisions about our piles of abstractions.  I'll start by offering three rules to keep in mind:

Abstractions contain bugs.

Using somebody else's code can save a lot of time.  For example, a special GUI component allows you to program at a higher level of abstraction.  Why write your own?

You have to remember that you are accepting a tradeoff.  By using somebody else's code you are inheriting somebody else's bugs.  Often you are accepting risks that are not under your control.

Abstractions reduce performance.

Writing in Java is faster than writing in C.  But C code runs much faster than Java code.  It's a tradeoff that you cannot avoid.  All you can do is make the right choice.

Abstractions increase overall complexity.

The goal of each abstraction is to decrease complexity by presenting a simplified view of something else.  However, by the time you pile them all up, you've got a lot of complexity which you may have to deal with.  In fact, the more layers of abstraction you have, the more complexity you've got involved.

A Really Tall Pile

Let's work through an example.  Suppose that I am working from my home and I am ready to checkin some really important code changes to a source code file.  I'm using the SourceGear Vault client to connect to our server back at SourceGear's main office.  Below is a list of [almost] all the abstractions which are in play.

1. Vault.  The version control system itself is an abstraction.  It presents our users with concepts like Check Out, Check In, Label, Branch, Pin and Share.  This layer is obviously very important, since it's the only one we can charge money for.  :-)

Control Flow:

2. C#.  Vault is written entirely in C#, which is a very nice abstraction indeed.  From C# we get classes, objects, methods, strings, looping constructs, logical operators, and the ability to attach names to things.  Cool.
3. CLR.  C# runs on the Common Language Runtime, which is a huge abstraction.  In fact, if we all weren't so worried about comparing .NET to Java we would be calling the CLR a "virtual machine", which it is.
4. C++.  The CLR is written in lower level languages like C++ and C.
5. Assembly.  C++ is implemented by compiling it to x86 assembler code.  We've taken a big, big jump here.  Compared to C++, assembly language doesn't feel very abstract at all.
6. Microcode.  Did you think Assembly was the lowest level of programming?  Certainly not.  Each x86 assembler instruction is a little program written in an even lower level language called microcode.
7. Logical gates.  Microcode is implemented by circuits which provide logical gates, including NOT, AND, OR, and NAND.
8. Transistors. Logical gates are implemented by transistors, an electronic component with three wires sticking out of it.

Memory:

9. ArrayList.  The .NET framework gives us "collections" we can use to manage memory in aggregated ways.
10. Objects.  From the realm of OOP we get "objects", self-contained pieces of data which are bound to the operations which can be performed on that data.  Very handy.
11. GC.  This is a big one.  Because the .NET Common Language Runtime has a garbage collector, we can create objects and know that they will automatically be destroyed later when we are done with them.
12. Handles.  In reality, memory has to be explicitly requested and released from the operating system.  Each chunk of memory is identified by a handle. 
13. Virtual Memory.  This layer gives us another important illusion:  There is more memory available than we actually have.
14. RAM.  Random Access Memory is itself an abstraction.  Transistors don't really remember anything.  Furthermore, the notion of a bit doesn't really exist.    We simply assign conventions.  When a wire is at 5V, we call it a one.  When it has no voltage on it, we call it a zero.  Collect a few hundred million of these in one place and you've got a DIMM.  (Actually, it's 3.3V nowadays, right?)

The Check In Button:

15. Button.  The Check In dialog has a button on it.  When the user presses this button, the Check In operation will commence.  But the button itself is an abstraction.  It is designed to simulate the concept of a physical button like you might find on your microwave or TV.  No such button really exists.  Windows Forms provides this abstraction.
16. HWND.  Windows Forms is a layer of abstraction which is built on the Win32 API underneath.  The button is actually a window with its own WndProc.  .NET tries to hide this world, but it's definitely still there.  One of the glaring "leaks" in the Windows Forms abstraction is the absence of the Win32 ScrollWindow() call.
17. GDI.  The button is actually drawn using graphics primitives from GDI.  It doesn't just magically appear.  It needs to be drawn using things like DrawRect, fonts and colors.
18. Pixels.  GDI contains primitives like DrawLine, but these are implemented in terms of pixels.  Graphics primitives are actually not quite so primitive.  If you think line drawing is easy, look up Bresenham.
19. Video Card.  The pixels are actually an abstraction presented by a video card.
20. Monitor.  The monitor presents the illusion that all those pixels are organized into pictures and images.
21. Light.  I stop whenever I get to Physics or Chemistry.  For my purposes, light is real, not an abstraction.

Architecture of the Vault Client:

22. VaultClientPresentationLib.  We wrote this layer as part of Vault.  It contains all the windows and dialog boxes necessary to create the Vault GUI client.
23. VaultClientOperationsLib.  This layer is a big part of Vault.  It contains basic non-GUI primitives which are necessary to write a Vault client.  Create an instance of the ClientInstance class.  The methods on this class will communicate with the Vault server and simultaneously keep your local working folder updated.
24. VaultClientNetLib.  The previous layer actually calls VaultClientNetLib to communicate with the Vault server.  This layer is fairly thin.  It is mostly a wrapper around the Proxy Class.
25. Proxy Class.  This important layer is generated by Visual Studio .NET.  It presents the illusion that the XML Web Service on the Vault server is actually a C# class.
26. SOAP.  When a call is made through the proxy class, the parameters for that call are bundled up in SOAP format.  This format presents the concept of a method invocation message.
27. XML.  SOAP is built on XML, a syntax framework for representing data.
28. HTTP.  The SOAP message is transported to the server over HTTP, the networking protocol on which the Web is built.
29. DNS.  The Vault user types the name of the server, but that name isn't really useful.  It has to be converted to an IP address before real network communication can take place.  The Domain Name System is used to look behind the abstract name and get the actual machine address.
30. SSL.  The Secure Sockets Layer offers the illusion that communication over the Internet can be private.  This layer tries to look just like a regular socket, except all of the data is encrypted as it passes through to/from the socket itself.
31. Sockets.  This layer is a great abstraction.  Sockets present us with the illusion of connections and the ability to send and receive data between endpoints.
32. TCP.  The basic illusion of TCP is the idea that packets of data will arrive and in fact, will arrive in the order they were sent.
33. IP.  TCP is built on IP, which is even lower level network protocol.  At this layer, packets may or may not actually arrive, and they may arrive in a different order than how they were sent.
34. Ethernet.  The IP packets are carried on a cat5 wire sticking out the back of my computer.
35. Radio.  The Internet connection at my home is a wireless antenna pointed at the top of a grain elevator eight miles away.  So right now, the important code change I am trying to checkin is a bunch of radio signals which represent packets that may or may not arrive, but they are flying through the air, 25 feet above a corn field.

Architecture of the Vault Server:

36. VaultService.asmx.  The Vault server is an XML Web Service.  This allows us to think of our server as a collection of methods which will invoked in an "RPC-like" fashion.
37. ASP.NET.  The illusion of XML Web Services is actually provided by ASP.NET.
38. VaultServiceSQL.  This library provides a wrapper which insulates the rest of the server from having to know anything about SQL.
39. Stored Procs.  This layer is a collection of stored procedures running inside SQL Server.
40. SQL.  The SQL language is an enormous abstraction.  It presents concepts like tables, rows and indices, as well as atomic transactions.
41. IO calls.  Somewhere deep inside SQL Server 2000 is the place where data is actually written to the disk file.  They probably call the native Win32 IO calls.
42. NTFS.  The filesystem is a very important abstraction.  It presents the concept of files and folders, as well as permissions and attributes.
43. Partitions.  The filesystem exists on a "partition", which is a portion of the space on a hard disk.
44. RAID array.  The RAID controller presents the illusion of one hard disk when it is actually several.
45. Hard disk.  In practical terms, this was the goal of the checkin all along.  My bits are finally stored in my hard disk.  But the disk itself is actually an abstraction...
46. Platters.  The term "hard disk" sounds singular, but hard disks today usually have several platters inside.  These platters are the magnetic media where the data actually resides.

So there you have it -- 46 layers of abstraction which are all involved when I try to checkin my code.  That means there are 46 layers in which something might go wrong.

Actually the truth is that several of these abstractions are almost perfect.  For example, I've actually never had to worry about the layer between Assembly and Microcode.  As far as I am concerned, Assembly is an abstraction that always Just Works.

But it would be terribly wrong to ignore all those layers.  Yes, SourceGear's implementation of Vault required us to only write the code for a few of the layers above.  However, when it's time for QA and Tech Support, all 46 layers are fully in play.  Stuff Happens.  When a customer has a problem with Vault, the actual problem could be almost anywhere.  We have to figure out what's gone wrong, even if it's in one of the layers we didn't create.  Ask our tech support team how often layer 29 causes trouble. :-)

How to Kill Your Project

When you build software, you're going to end up making a lot of decisions about abstractions:

• Which abstractions do you want to build on?
• Where will you get the implementations of those abstractions (platforms, libraries, components)?
• How trustworthy are those implementations?

You have lots of alternatives.  For example, you can often make a tradeoff by choosing to work at a lower level of abstraction.  By doing so, your development process will move more slowly, but more of the risks will be under your control.  For example, if I had a really small magnet and really fine motor control skills, I could skip layers 1 through 45, drive to my office and modify those platters myself.  :-) 

The stakes are higher than you might think.  You can kill your project by making the wrong decisions about abstractions.  Do you remember the word processor called WriteNow?  This product was my favorite word processor back when I was a Macintosh fanatic.  WriteNow was really fast and had just the right mix of features.

Today, WriteNow is dead because somebody got burned by the decisions they made regarding abstractions.  You see, WriteNow was really fast because it was written in 68000 assembly language.  When Apple moved the Macintosh product line to the PowerPC, WriteNow had nowhere to go.

These choices are hard, and learning from your mistakes is an excellent (but painful) way to learn.  But over the years, I've gathered the following guidelines which help me make abstraction-related decisions:

Consider your context.

Developing a server operating system is different from developing an web-based HR application so employees can check their vacation days.  There is no formula which works well for all kinds of projects.  You need to understand what kinds of risks are appropriate for the kind of software you are trying to build.

As a general rule, developers of internal corporate applications tend to use more third party components than ISVs.  If you're writing code for the IT department of a company whose primary business is not software, then your salary is an expense, not an investment.  Your employer wants you to get the app done FAST, because it costs less to get it done that way.  Corporate IT developers want every decent abstraction they can get.

ISVs like SourceGear face a different set of problems.  If a third-party component brings even a minor loss of quality to the app, it can severely affect our sales as prospective customers look at our competitors.  But that same competition is tugging you in the other direction, reminding you that time-to-market can be critical.  Using third-party apps may be the only way to get your product to market within the window of opportunity, but the risks need to be studied closely.

The size of your company should affect your choices as well.  Nothing is more frustrating than being unable to ship a product because of a bug that you can't fix because it's in a third-party library.  For a very small company, the financial damage of a situation like this can be severe.  These are the times when you wish you had chosen to put more of your risks inside your own circle of control.

Place your trust carefully.

I recommend approaching third party code with a great deal of suspicion.  Never assume that an unknown component or platform will Just Work.  A little paranoia will probably pay off later.

When picking the pieces of your platform, as a general rule, "older is better".  You can walk with less worry on a path which has been well trodden by many people for years.  As an extremely obvious example, C is old and mature enough to be a platform which will yield very few surprises.

Evaluating newer technologies is harder.  Try to figure out who else is using the abstraction successfully.  Grab the technology and take it for a test drive.  In the end, you may not get enough evidence to lead to a completely confident decision.  If you really need the convenience offered by the abstraction, you may have to jump out with a little faith.

Learn to see through the abstractions. 

The most important point in this whole article is this:  You need to understand what's going on inside all your abstraction layers.  Each abstraction presents an illusion, but the best decisions happen when you can see through the illusion.

If you have a deep understanding of all the technology abstractions that are involved with your software, then you have two big advantages:

You can quickly isolate problems.

You can develop an intuition which will help you avoid those problems in the first place.

The first point is fairly obvious.  Troubleshooting goes much better when you know what's going on.  Have you ever watched someone try to solve a problem in the presence of several abstractions they didn't understand?  They feel helpless.  Usually, they start making wild guesses about where the problem could be.  I call this "stab in the dark debugging".  :-)

More importantly, if you can see through most abstractions then you can develop an intuition to make much better technology decisions.  Choosing the right libraries and components in your platform can prevent lots of problems before they ever happen.

Don't assume that this kind of deep technical knowledge becomes less valuable as you climb the management ladder.  Understanding this stuff can be a huge advantage in many kinds of decisions, right up to the most executive levels.  I believe the technical prowess of Bill Gates was a major reason why Microsoft beat every competitor in the eighties and nineties, even though Bill probably wrote no code, no specs, and no design documents.

Failures and Successes

It wouldn't be fair to only mention the mistakes of others when I've made so many excellent and instructive mistakes of my own.  :-) 

My most recent blunders in this area happened when we built SourceOffSite Collab on a pile of abstractions which was way too short.

• We built our own implementation of the "server pages" concept because ASP didn't meet our requirements perfectly.  Collab includes its own web server which processes pages we call "giglets".  In between the <% and %> we process JavaScript using the Mozilla engine which has been modified with special Collab-specific hooks.  In retrospect, we should have found a way to work around the limitations of Microsoft's standard dynamic page generation technology.
  
• We also implemented a complete system for XML-based procedure calls.  We rationalized this one because XML-RPC and SOAP just weren't quite perfect for our needs.  Hindsight now brings us to the same conclusion as above -- changing our requirements to fit the established platforms would have been the wiser choice.

Better decisions would have gotten SOS Collab to market sooner and we would have fewer code maintenance problems now.

Not all of our decisions went badly.  We made a great choice when we decided to build Vault using .NET.

From the beginning, I hoped that .NET was "Java done right".  I've used Java extensively, and I loved the productivity gains we got during the beginning and middle of the development cycle.  But things got ugly at the end.  All those layers of abstraction started contributing to our bug list.  I've been involved in a couple of projects which completely failed because Java was chosen.  (Yes, this is merely my opinion, and yes, there were other factors in the failure of those projects.)

Given our bad experiences with Java, our decision to use .NET took a fair amount of courage.  Early experiments looked promising, but we knew that we would have to wait for the endgame to really know if .NET could really be trusted all the way through.

As I write this, SourceGear Vault 1.0 has been shipping for over two months.  We have no regrets.  For an abstraction pile as large as the one described above, it's remarkable that this product works at all.  :-)

But the fact that it works well is nothing short of amazing.  We have test applications which continuously try to abuse Vault in ways that are abusive and profane.  If something goes wrong in any layer of abstraction, the whole test will come to a halt.  But we can let these tests run for days at a time without any problems whatsoever.

This success stands as a testimony to how incredible .NET really is.  We built a reasonably full-featured version control system in 14 months, and it works.  Sure, we had some trouble.  Layers 25, 37 and 40 didn't always behave like they should.  But layer 11 was problem-free, quite unlike its Java counterpart.  Considering the productivity gains we received, I never expected things to go so smoothly.

Note:  In response to the controversy generated by this article, I posted some followup remarks.