Skip to content
How a 5-Person 'Performance Tool' Project Built Modern Virtual Machines
Fundamentals 28 min read

How a 5-Person 'Performance Tool' Project Built Modern Virtual Machines

In 1967, a five-person IBM Cambridge team disguised as a performance tool built CP-67/CMS, the first commercial virtual machine system, using a hardware address translator on a modified Model 40, later enabling nested virtualization to debug MVS, outperforming IBM's official TSS/360.

Radish, Keep Going!
Radish, Keep Going!
Radish, Keep Going!

The Crisis of TSS/360 and the Cambridge Alternative

In spring 1967, IBM's official time-sharing system TSS/360 on a System/360 Model 67 at MIT Lincoln Laboratory required 10 minutes to IPL from tape and crashed on average every 10 minutes (MTBF < 10 minutes). Meanwhile, Dick Bayles mounted an experimental tape from the Cambridge Scientific Center's five-person group — CP-67 — on the same Model 67. Within one minute, all IBM 2741 terminals were online. Center director Jack Arnow immediately demanded IBM replace TSS with CP-67, which entered daily production in April 1967.

CP-67 had been funded as a "performance measurement tool" for the Poughkeepsie plant, surviving on night-shift revenue from an MIT 7094. By stripping multi-user logic from the OS, the team created a 1:1 equivalent of the commercial instruction set — the modern virtual machine family — and six years later used five-level nested virtualization to rescue the very IBM flagship OS (MVS) that had sought to cancel it.

545 Technology Square: The Missing Dynamic Address Translation

The story begins in February 1964 when Norman Rasmussen established the IBM Cambridge Scientific Center (CSC) at 545 Technology Square, sharing the building with MIT's Project MAC (floors 5–9). In April 1964, IBM announced the System/360 family. To meet delivery schedules, management cut Gerry Blaauw's planned Dynamic Address Translation (DAT) hardware, leaving the first 360s without hardware virtual memory or dynamic relocation.

MIT, requiring hardware address translation for next-generation time-sharing, rejected IBM's "Blaauw Box" add-on proposal and chose the GE-645 for Multics development. Losing this flagship order prompted IBM to create the Flewellin House task force to build the Model 67 with native DAT and the official TSS/360.

Rasmussen, disagreeing with the monolithic-kernel TSS approach, hired Bob Creasy (co-author of the 1963 CTSS Programmer's Guide) and formed a five-person team: Les Comeau, Dick Bayles, Bob Adair, and John Harmon.

Instruction Set as Interface: The CP/CMS Bifurcation

Creasy observed CTSS's structural bottlenecks: kernel development required exclusive machine access during off-hours because the monolithic supervisor integrated hardware management, scheduling, isolation, and command interpretation. Any change risked system-wide instability.

In late 1964, Creasy and Comeau devised a bifurcation : split the multiprogramming environment into a physical hardware manager (Level 0, CP ) and a single-user execution layer (Level 1, CMS ). Multi-user isolation moved to the bottom layer; the upper layer became a pure single-user program.

CP/CMS's core design divides resource management and user support: CP provides each user a complete computing environment at the machine instruction level, solving multi-user sharing; CMS sheds sharing, allocation, and protection, offering only single-user service.

The interface between layers was not a new software protocol but the existing hardware specification: System/360 Principles of Operation .

Returning to fundamentals yielded the answer: the instruction set defined in the System/360 Principles of Operation is itself the time-sharing control program interface. Every user enjoys full System/360 capability; sharing and allocation happen beneath the instruction set via the control program.

CP manages real hardware and presents each user a standard System/360 virtual machine. CMS (Cambridge Monitor System, later Conversational Monitor System) strips multitasking and virtual memory logic; early versions could even IPL from a physical card reader. To a user at an IBM 2741 terminal, each CMS instance appears to start at physical address zero; system programmers can load test OS kernels in their virtual machines without monopolizing the physical mainframe.

256KB Bare Metal and the 64-Entry CAT Associative Box

Engineering faced dual constraints: the FAA had claimed all Model 50 capacity for air-traffic control, so CSC received a 256KB System/360 Model 40 — a second-line model lacking storage protection and DAT. Organizationally, IBM headquarters opposed a parallel time-sharing effort.

Rasmussen rebranded the project as a "performance measurement tool for paging characteristics" and funded it by renting out idle night time on MIT's 7094. With that budget, Comeau, Rex Seeber, and A. Bruce Lindquist physically modified the Model 40 bus, inserting a hardware box between CPU address generation and the memory bus: the CAT (Cambridge Address Translator) .

With 256 KiB physical memory and 4 KiB pages, there were 64 page frames. CAT contained 64 associative-memory slots, 1:1 mapped to frames. On a relocated access, the virtual machine number and virtual page number were fed in parallel; a hit output the 6-bit physical frame index (0–63); a miss triggered a page fault. The Model 40's memory cycle was long enough to overlap the associative lookup, adding no extra wait states.

IBM System/360 Model 40 control panel
IBM System/360 Model 40 control panel
IBM SLT ceramic module
IBM SLT ceramic module
Open-gate Model 40 cabinet
Open-gate Model 40 cabinet

Microcode also required changes. The Model 40 stored microcode in 4,096 × 60-bit words of Mylar punched-film TROS . Bayles, Comeau, and Fritz Giesin rewrote instruction-decode microcode so the CPU checked page presence before executing cross-page instructions, triggering a page fault if absent. Engineers debugged by pulling film cartridges, hand-punching corrections, and reinserting them.

By 1966, CP-40 and CMS ran on the modified Model 40, supporting 14 virtual machines each with independent 256KB virtual memory. The project supplied IBM's first real paging-system measurements. Comeau later noted the field had no success precedent: the only prior virtual-memory machine, Ferranti Atlas, performed poorly, and IBM decision-makers didn't know why.

On CP-40 the team observed thrashing — I/O saturation and zero CPU utilization from excessive paging. Comeau ran an 81-combination (4 factors × 3 levels) paging experiment consuming 63 system hours, quantifying the critical relationship between working-set size and replacement algorithms.

Privileged Instruction Trapping and Asynchronous I/O Channel Rewriting

Beyond address translation, virtualization required privileged instruction interception and async I/O handling. System/360 defines Supervisor State and Problem State. CP's principle: the physical CPU runs in physical supervisor state only while executing CP core code; when dispatching to a virtual machine, the physical PSW is set to Real Problem State . The virtual machine's register context and nominal execution level are maintained in CP's VMBLOK control block.

Normal compute and branch instructions execute directly on the physical CPU. When a guest attempts LPSW (Load PSW) or SSM (Set System Mask), a hardware protection interrupt traps to CP. CP checks VMBLOK: if the guest is nominally in problem state, the exception is reflected back; if in supervisor state, CP emulates the instruction by updating virtual registers in VMBLOK and resumes.

System/360 I/O channels are independent processors using direct physical addresses, bypassing CAT. On a guest SIO (Start I/O), CP traps, copies the entire Channel Command Word (CCW) chain, translating each virtual page address to a physical frame. For Minidisk (virtual disks carved from physical drives), CP also maps logical cylinder offsets to physical cylinder ranges.

IBM 2311 disk drive
IBM 2311 disk drive

During channel-command translation debugging, a pre-release OS/360 deadlocked on IPL under CP-40. Bayles traced it to the OS/360 IPL loader relying on undocumented hardware timing. A two-stage IPL patch accommodated this. Later, VM/370 introduced DIAGNOSE (and SVC 202) as a paravirtualized fast path, letting virtualization-aware guests call the monitor directly, an early hypercall precursor.

Michigan Model 67 with 2314 and 2540
Michigan Model 67 with 2314 and 2540

From 0.35 Throughput to Hardware SIE: The Performance Cost of Virtualization

When the DAT-equipped Model 67 shipped, Bayles' team had already emulated its instruction set on the modified Model 40, producing CP-67 , which took over production at Lincoln Lab.

More intriguing is that this "stepchild" of the time-sharing community is now more popular than its well-funded contemporaries — IBM's TSS and GE/Honeywell's Multics. Attributing success solely to clever early design is tempting, but the real key was the lack of spare hands and abundant compute to over-engineer.

Two distinct overhead phases emerged:

CAT hardware phase (1966) : Each associative cell cost ~35× a regular memory cell (still 8–10× by 1982), and fixed slots prevented sharing read-only code pages across VMs. Production systems moved to conventional two-level segment/page tables.

Software shadow-page-table phase : With System/370 and VM/370, guest OSes (OS/VS1, MVS) enabled their own paging. Hardware DAT supported only single-level translation, so VM/370 maintained composite two-level Shadow Translation Tables in software. Gum (1983) recorded each shadow entry resolution required at least 8 main-storage references .

Pure software shadow tables dropped paged-guest relative batch throughput (RBT) to 0.35 of native, extending elapsed time to 2.86× baseline.

This drove hardware assist evolution: IBM first added 13 VMA (Virtual-Machine Assist) microcode assists (RBT > 0.70), expanded to 35 in ECPS:VM , and finally in 1983's System/370-XA architecture consolidated over a hundred intercept and shadow-table operations into the hardware instruction SIE (Start Interpretive Execution) . This trajectory repeated two decades later with x86's move from software shadow tables to Intel VT-x / EPT.

Model 67 configuration console
Model 67 configuration console
Mike Alexander at Michigan Model 67
Mike Alexander at Michigan Model 67

Five-Level Nesting Saves MVS and the Birth of VM/370

Before hardware SIE, CP-67 became critical for IBM's mainstream OS delivery. CP-67 was released as unsupported Type-III software in June 1968. When early System/370 (June 1970) again deferred DAT, CSC retained 13 Boston field engineers to keep the team alive.

MVS development stalled: without DAT-equipped 370 prototypes, its paging kernel couldn't be tested on real hardware. French engineer Alain Auroux et al. built a five-level nested virtualization stack on a Model 67, simulating a DAT-equipped virtual System/370, enabling MVS testing before hardware arrival.

On August 2, 1972, IBM added DAT to all System/370 models and officially productized the Cambridge system as VM/370 . By Creasy's 1981 retrospective, it had run 15 years, with >2,500 installations across 400+ mainframes on five continents serving 50,000 users.

IBM System/370 Model 138
IBM System/370 Model 138

Design Legacy: Extensible Lightweight Core vs. Heavy Monolith

Comparing CP/CMS with TSS and Multics, the lesson is: adding features to an extensible high-performance system is far easier than improving performance of a large, all-inclusive heavy system.

This trade-off confined resource isolation and privilege protection within the instruction-set interface boundary, keeping the upper execution environment minimal. The same principle underpins today's minimal MicroVM isolation for serverless functions (e.g., AWS Firecracker ).

References

R. J. Creasy, The Origin of the VM/370 Time-sharing System , IBM J. Res. Dev., 25(5), Sep 1981, pp. 483–490.

L. W. Comeau, CP/40 — The Origin of VM/370 , Proc. SEAS AM82, Sep 1982, pp. 40–46.

L. H. Seawright and R. A. MacKinnon, VM/370 — a study of multiplicity and usefulness , IBM Syst. J., 18(1), 1979, pp. 4–17.

P. H. Gum, System/370 Extended Architecture: Facilities for Virtual Machines , IBM J. Res. Dev., 27(6), Nov 1983, pp. 530–544.

Melinda Variant, VM and the VM Community: Past, Present, and Future , SHARE 89, Sessions 9059–9061, Aug 1997.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

CMSvirtualizationnested virtualizationIBM System/360CP-67dynamic address translationshadow page tablesvirtual machine history
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.