Skip navigation

Category Archives: computer graphics

This is part three; part one is OpenGL, part two is Early 3D hardware.

As the team continued to work on improving OpenGL’s capabilities, another graphics technology effort – DirectX – had also been moving forward in parallel. In this earlier era at Microsoft, it was not uncommon to have overlapping product efforts, or even efforts the competed directly against each other. The prevailing attitude about such cases was to let the market decide. DirectX was focused on enabling games on Microsoft’s Windows 95 consumer operating system. DirectX got its start as a 2D-based technology providing game developers direct access to the graphics card’s frame buffer. This was very useful functionality up to a certain point, but games like Doom and Quake made it clear that 3D graphics would play a dominant role in the future of interactive games.

Microsoft purchased one of the few available software-based 3D APIs – Rendermorphics’ Reality Lab – early in 1995. The goal was to jumpstart the addition of 3D graphics capabilities to DirectX by incorporating the Reality Lab technology. I was asked to do the technical due diligence of the Reality Lab stack.

Unlike OpenGL, Reality Lab was a scene graph API. This meant that the full geometric representation of a 3D environment was fed to the API. Based on the viewer’s position and other variables, the scene graph engine would compute the necessary details for rendering the objects in the 3D environment all the way down to the individual primitives rendered at the end of the pipeline. As a scene graph API, I thought Reality Lab was pretty good, but it seemed to me an odd match for the intended game developer audience.

Games like Doom and Quake built their own engines for rendering 3D environments, and these engines were highly optimized and specialized for the game’s specific characteristics. What most game developers were looking for was a lower-level immediate-mode 3D API that allowed them to plug their game engines at the stage of processing where primitives were ready to be rendered.

directx_logoDirectX hoped to satisfy this need by providing a mechanism for developers to pre-construct lists of low-level graphics primitives that could then be handed to the graphics card driver to execute. In theory, this allowed direct hardware execution of 3D primitives. In practice, the interface was cumbersome and difficult to use, provided no performance benefit for all the extra trouble, and had limited functionality compared to OpenGL.

What game programmers like John Carmack were discovering was that OpenGL already provided a clean, efficient immediate-mode 3D API that could be used very effectively for game development as well as non-game applications. Whereas DirectX required adhering to overhead-inducing COM-based protocols, OpenGL was a clean, straightforward, well-documented C API.

The game developer community tends to be data-driven and immune to market-speak and spin; their bullshit detectors can be exceptionally good. They were being told to use DirectX for 3D games, but they wanted to use something proven that already existed and just worked. John Carmack’s observations on the topic is still a good read.

More and more developers were coming to Microsoft and demanding that it fully support OpenGL on both Windows NT and Windows 95. The technical work for Windows 95 had already been done, and hardware vendors work working with a pre-release version of the OpenGL MCD driver development kit anticipating final release. Developers wanted the OpenGL driver update for Windows 95 released as soon as possible.

An influential group of game developers even banded together and signed an open letter to Microsoft asking that OpenGL be fully supported on both Windows NT and Windows 95.

This was not how it was supposed to work out. DirectX was supposed to be Microsoft’s consumer graphics technology, and OpenGL was supposed to be “just” for high-end CAD for non-consumer markets. But the reality was that visualization technology didn’t care about somewhat arbitrary market segmentations: a good 3D API was a good 3D API. Many developers felt that OpenGL was the best 3D API at the time and wanted to use it for their games. This has been described as a religious issue, but game developers simply wanted tools that would help them get their work done.

Microsoft was in a tough spot. It now had two competing efforts on its hands, and it couldn’t simply kill either. OpenGL was necessary for continuing to compete in the workstation market, and DirectX was its high-profile brand and technology for enabling PC games. And while OpenGL was an external standard, Microsoft owned DirectX and was free to shape it on its own. Even though Direct3D had gotten off to a rough start, Microsoft couldn’t afford to retrench from DirectX.

The open letter had pushed things to a crisis point, and shortly after it went public I had a meeting with the senior manager in Windows whose task had recently become to clean up the OpenGL/Direct3D mess. I remember the central point of the conversation was that I was being asked to take responsibility for both OpenGL and DirectX graphics and to focus on ensuring that DirectX improved as quickly as possible while continuing to support OpenGL for non-game applications.

This would mean a number of things for me. I would have to leave programming as anything other than an occasional hobby since I would have my hands more than full with providing engineering leadership. I would never see my MCD driver work available in the much larger Windows 95/98 market. And I would also have to disappoint a large contingent of game developers who had hoped for a different outcome.

Of course I could have gone and worked on something else, but that would have served no constructive purpose. Microsoft was sticking with DirectX graphics, and the best thing for developers, users, graphics vendors, and Microsoft was to make DirectX as good as possible. Combining the OpenGL and DirectX graphics development efforts was probably the best way to achieve that; the OpenGL team had at this point a deep bench of 3D graphics talent.

So I said yes, and began the hard work of integrating and re-balancing the teams and getting people on board with a slightly awkward new mission and new priorities.

Over the course of the next three years we steadily improved DirectX. I felt we had finally accomplished our goal of building a world-class 3D API when we released DirectX 8.

The experience was a formative one for me. I had to step outside the emotions surrounding the situation to find a way to make the greatest positive contribution possible. I had to earn the trust of a new team that had viewed my team and me as the enemy. And I had to take something that I felt was technologically inferior and make it great.

But great challenges can provide great opportunities…

My wife bas been pestering me to write a post or two about some of my early years at Microsoft. Thinking about what to write took me back to a time when my entire focus at work was writing the best code possible. In those days, I would even sometimes dream about code. This post talks about software implementation details and may leave some readers behind in a few spots. Bear with me; future posts will be less technical.

For many years, one of my main pursuits as a developer was software-based graphics acceleration. This meant using the CPU to render graphics on the screen as fast as possible by carefully tuning software and algorithms. The goal was to extract every last bit performance from the CPU. One of the key reasons I joined Microsoft in 1993 was for the opportunity to work for Mike Abrash who even then was renowned for mastery of CPU-based performance optimization, and he was aware of some of my work. He was running the Windows NT GDI (Graphics Device Interface) team at the time, and I had come on board to focus on optimizing GDI performance in the first version of Windows NT (somewhat confusingly named “Windows NT 3.1”).

GDI was a 2D graphics framework doing the important work of rendering fonts, lines, rectangles, and all of the primitives that rendered the Windows desktop UI and programs written for it. But being strictly 2D-based, GDI was never going to include 3D graphics, and 3D was as fascinating to me as VR is to some people today (including Mike, who is now Oculus’ chief scientist).

One of NT’s goals was to take a share of the workstation market, and Microsoft had licensed the OpenGL graphics API from Silicon Graphics (SGI). SGI’s business was based on selling expensive high-end hardware, and they had put little effort into the performance of their reference software implementation of the API. Without very expensive hardware, OpenGL was pretty useless.

I had fallen in love with 3D graphics in the late 80’s, and I joined the recently-formed OpenGL team soon after Windows NT shipped. Even though speeding up OpenGL wasn’t part of my job when I joined the team, I was excited to be part of effort that would broaden access to 3D graphics capabilities. My dream was to have 3D graphics be standard on every PC. My job was to help integrate OpenGL into the Windows operating system.

But official responsibilities aside, I couldn’t resist the temptation and challenge of speeding up OpenGL to make it useful without requiring very expensive hardware. I immediately started tinkering with the OpenGL stack when I had free time. This was the mid-90’s – the era of Intel’s new Pentium processor line. This family of CPUs allowed overlapping of floating-point and integer instructions – a primitive form of parallel processing. A floating-point instruction could be started, and then integer instructions could continue to be executed while the much lengthier floating point command was processed. This mixing of floating-point and integer operations was perfect for speeding up 3D rendering operations which could be broken into floating-point setup and fixed-point, integer-based scanline fragment processing.

A complicating factor in speeding up OpenGL was the complexity of its state machine. There were many possible combinations of rendering modes based on attributes such as color depth, z-buffering, shading model, transparency, texture-mapping, etc. An early (Windows 3.1 or Windows 95) solution for optimizing GDI rendering was to build the rasterizer on the fly on the stack based on the GDI state (I believe this was all or mostly Todd Laney’s handiwork). But I was working on Windows NT, and such clever hackery was not allowed in a next-generation operating system. After considering my options, I determined that my best bet was to pre-compile a set of renderers that represented the most common cases for rendering (for example, Gouraud-shaded, 16-bit color, 16-bit z-buffered triangles). I did this by building rendering functions that consisted of groupings of macro statements that themselves were chunks of hand-crafted inline assembly code. These macros could then be grouped together to perform unique sets of rendering operations. In order for this approach to work, I constructed a framework for the chunks of assembly code in the macro blocks to be able access variables and registers in a common and consistent way so that they could interoperate predictably and efficiently.

This was definitely not how the C language was intended to be used. It wasn’t pretty – I believe I even used “goto” statements out of necessity – but the code was highly effective. The approach also entailed risk because things could go horribly wrong with unanticipated edge cases. I remember one embarrassing bug that in certain cases failed to return the floating-point control register to its previous state which effected floating-point operations in the rest of the operating system. I quickly found and fixed the issue but it was a reminder that there was no safety net with what I was doing.

It’s interesting how thinking about past work jogs the mind. In the middle of writing this post, I had a dream about how I may have implemented dithering. Dithering is an old technique going back to the print business that allows gradations in tone and color to appear smoother by breaking up transitions using patterns of dots or pixels at different densities. For example, an area halfway between one color and another would have half the pixels in that area set to one color, and the remaining half set to the other color. Today, even our phones are capable of producing millions of colors eliminating the need for such techniques, but in the mid-90’s, the capabilities of most PCs were far more limited. In my dream, I implemented dithering look-up-tables (LUTs) for red, green, and blue values so that I could construct the right 15 or 16-bit RGB value using three highly-cached indexed memory operations and two OR instructions. I probably did something along those lines, but who knows…I’d love to have access to that old code (of no use to anyone at this point) just to remind myself how I did what I did.

My initial performance improvements were compelling enough to justify making the software acceleration work a full-time endeavor. I added functionality over time culminating with real-time texture-mapping using quadratic subdivision. Even though it was still part of a fully compliant general-purpose OpenGL implementation, textured rendering throughput got reasonably close to that of Doom, the texture-mapping speed champion at the time. In fact, my software-accelerated pipeline became competitive with high-end hardware graphics cards and beat them in some instances (anti-aliased lines comes to mind).

I had taken rendering performance from being measured in seconds per frame to dozens of frames per second; many operations were over 100 times faster than the original versions. Ah, the power of highly-tuned, efficient coding! I often worry that Moore’s Law is being buried under so many layers of frameworks, objects, interpreters, and interfaces that it’s hard to tell what the hardware is actually doing at the bottom of the pile multiple layers of indirection. Then again, even CPUs now hide some of their internal operation with sophisticated out-of-order instruction execution engines. Sadly, the last time I tried to out-optimize a modern C compiler with an integer-only routine, the results were a draw.

Pipes screen shotmaze2Our OpenGL implementation was incorporated into both Windows NT and Windows 9x code bases and was starting to get significant traction. Being part of the operating system and eliminating the need for hardware acceleration meant that an OpenGL application could now target a very wide audience. As a means of promoting our OpenGL’s capabilities, I wrote the first set of 3D screen savers (“Flying Objects”) which proved popular and inspired other people on the team to write their own as well (such as “Pipes” and “Maze” pictured here). For many years afterward, I would get a chuckle when I saw one of our screen savers running in the background of a TV or movie set. Our efforts had taken what had been an expensive workstation technology and made it readily available on millions of desktops.

Despite the effort’s success, the writing was already on the wall for the future of software-accelerated computer graphics. Hardware acceleration and the rise of the GPU were just around the corner.