It’s difficult to describe the process behind using AI to build a tool that you don’t quite know what it is, but now that you have the ability to do it, you’re compelled to continue and to find out.
These things sometimes take time to reveal themselves though, and I think I’m starting to understand what this one is and how it functions, out of the other bits and bobs I’ve managed to piece together of the “grand idea” (more of an intuition) that is fueling the whole thing.
For now, it’s called tape.deck.
Here’s a faux diagram that I got ChatGPT to make, and I think grounding it in a physicalization really helps for comprehension:
The idea in a nutshell is this:
Generative AI makes it possible to realize the same composition in many forms. This calls for tools that let authors shape a story’s underlying structure and guide how it becomes text, images, audio, or video.
After much back and forth, I managed to coax a pretty coherent simple specification for what this app/system/workflow/pipeline/etc. ought to be able to perform – at least for now. On-going usage and exploration would guide future development. Included below is GPT-6.1 Sol text:
tape.deck — simple app specification
Version 3 · October 8, 2026
What the app does
tape.deck supports medium-independent authoring and composition. The author arranges symbols, blocks, capsules, and instructions into a cassette that different renderers can realize as text, images, narration, or video. Audio and visual references can guide those realizations.
Author once, render many.
A cassette contains the underlying composition and the information needed to interpret it. Different treatments can change its expression while preserving its authored arrangement and constraints. Specific human-authored text can be included when exact wording matters.
The composition package
- Cassette — the whole package from which a medium-independent composition can be mixed and rendered.
- Head capsule — the first capsule; describes the cassette and holds shared instructions, mix settings, and rendering constraints.
- Registry slices capsule — contains background references, definitions, and allowed usages for the symbols included in the composition.
- Content capsules — flexible containers holding blocks or nested sub-capsules. Capsules can be chained together in sequences.
- Blocks — groups of compositional material that can be chained into sequences within a capsule.
- Frames — sit inside blocks and identify which symbols perform which actions upon which other symbols.
- Symbols, or symbol units — addressable elements drawn from the registry, including people, places, objects, groups, and ideas.
Setting and instructions can be established at cassette, capsule, or block level. Descendant material inherits that context, with explicit local refinements. Conflicting constraints remain visible for resolution.
Main functions
Story Space
Browse the fictional universe through its source material, symbols, and relationships. A searchable network opens the supporting passages and recorded connections. Multiple sourced versions can coexist. Later, the app can suggest promising paths or unexplored connections as starting points for new compositions.
Registry
Import books and scan them for meaningful symbols. Proposed entries, relationships, and merges go through human review. Approved entries retain their source passages, descriptions, connections, and usage rules. Audio, images, and other files can be attached as references. Review decisions persist.
Builder
Compose frames within blocks, arrange blocks within capsules, and sequence capsules into a cassette. Support nesting and occasional branches through a zoomable flow. Selecting an element opens its composition and instructions; relevant registry material appears contextually. Direct text entry supports guidance and exact inclusions.
Mixing
The head capsule holds mix settings: desired treatments, constraints, and allowable renderer choices. These can include tone, emphasis, viewpoint, permitted invention, language, medium, style, format, and genre.
Renderers apply those settings through their own capabilities and may offer further adjustments within the cassette’s constraints. A revised mix can produce another realization while preserving the composition and earlier results.
Preview
Inspect the assembled cassette before rendering. Preview is a focused, read-only document over a dimmed or blurred background, with collapsible navigation and a compact flow overview. It shows the package and effective instructions being sent to the renderer. Registry details open when selected.
Rendering and results
Run the cassette through different renderers and retain the resulting versions. Read text, play narration and video, and open images large with arrow-key navigation. Compare alternatives, approve or reject results, revise mixes, and export selected outputs.
Playback and distribution
The distributed object is the cassette. A network, channel, distributor, or other recipient can purchase or license it and produce its own realization for an audience or subscribers [or for personal use]. The underlying composition persists across those versions.
Playback is the experience-facing operation. It can present an existing render or generate a realization as it proceeds. A playback environment interprets the cassette’s authored arrangement, symbol references, allowed usages, and mix settings.
An interactive fiction experience could let audience choices influence the realization within permitted boundaries. A device could realize a cassette locally, or present a version produced elsewhere. These are possible applications of the same package and rendering model.
Manifest and review
Each rendering includes a manifest describing the basis of its content decisions and additions made by the renderer. Required human-authored wording can be pinned to persist verbatim in applicable outputs.
A separate review checks the realization against the cassette and flags omissions, undeclared additions, instruction drift, or unexpected symbol usages. Useful new usages can be approved and added to the registry. Uncertain findings remain available for human judgment.
The output can travel with a generated bill of materials, or GBOM, containing its input references, manifest, and review record.
Curation and reuse
Retain approvals, rejections, edits, and review decisions. Selected finished work can be scanned back into the registry, with new material proposed for approval. Over time, this record can support better suggestions and an experimental encoding of the author’s preferences.
Shared behavior
Work and decisions survive closing the app. Editing and browsing do not require model calls. Generation is an explicit action. Earlier compositions, mixes, and outputs remain available, and cassettes can be exported for use in other rendering or playback environments.
The main views are Story Space, Registry, Builder, Cassette, focused Preview, and Render Results. Mixing belongs within the Builder and cassette configuration. The visual style remains open.
First complete version
The first version supports importing source books, reviewing registry entries, composing and sequencing capsules and blocks, packaging and previewing cassettes, rendering text, reviewing results, and saving or exporting the work.
Images, narration, and video extend the same structure. Distributor playback, interactive experiences, suggested paths, taste encoding, and Content Credentials integration can develop from that foundation.
I’ll continue this in a separate post or this is going to get too long to parse all at once…
Leave a Reply
You must be logged in to post a comment.