Occupy AI is the 91st volume in the AI lore series by Lost Books, a Canadian publisher specializing in AI-assisted art books.
This volume uses ChatGPT v4 with Midjourney v5 to imagine what might happen if a radical direct democracy group occupied the headquarters of prominent AI labs, demanding transparent direct control of the technologies by those most impacted by them, ordinary people. The book grew out of recent blog posts exploring related issues.
It’s fun to take this exploration out of the realm of pure hypotheticals, and into a sort of fictional sandbox which can act as a test bed or proving grounds for some of these ideas and societal conflicts. Fiction is great for this because it’s so much faster than having to wait around for reality to catch up, and nobody gets hurt from the exercise.
Did a couple posts on this already and wanted to continue the train of thought on the idea of handheld controllers for navigating latent spaces of image diffusion models like Midjourney (though theoretically the same concepts should apply for navigating any high dimensional space).
When this idea first landed, I worked extensively with ChatGPT v4 to flesh the concept out in a bit more detail, but I never captured that anywhere public. So here it is…
Core design challenges
After sketching out the general idea, so we were both on the same page. I asked CGPT what are the core design challenges here. I won’t include every little piece of the response, but selected highlights are below.
The core design challenge in this project is indeed interpreting a high-dimensional latent space into a physically navigable form using a handheld controller with limited input options. Mapping the complexity of a multi-dimensional latent space onto a device with a restricted number of controls, while maintaining a smooth and intuitive user experience, can be difficult.
Some specific challenges include:
Dimensionality reduction: Latent spaces of image diffusion models can have hundreds of dimensions. The controller needs to facilitate navigation through these dimensions effectively while only using a few input controls (joysticks, dials, and buttons). Developing a suitable method for dimensionality reduction, such as PCA or t-SNE, that retains meaningful features and provides an intuitive navigation experience, is essential.
Interpolation and navigation: Designing a system that enables smooth transitions between images while navigating the latent space is challenging. Interpolation techniques, such as linear or spherical interpolation, must be carefully chosen and optimized for the best results.
I’m not actually sure what all that means, but following those trails lead to some interesting and useful discoveries. I won’t bother trying to explain it and will instead pass the baton to CGPT.
Dimensionality Reduction
ChatGPT says on this topic:
Dimensionality reduction is a technique used to reduce the number of variables or dimensions in a dataset while preserving the relationships and structures within the data. In the context of your project, dimensionality reduction would be used to simplify the high-dimensional latent space of image diffusion models into a lower-dimensional representation that can be effectively controlled and navigated using the handheld controller.
PCA (Principal Component Analysis) and t-SNE (t-Distributed Stochastic Neighbor Embedding) are two common dimensionality reduction techniques.
PCA is a linear technique that identifies the directions (principal components) in the high-dimensional space where the variance is maximized. By selecting the top principal components, you can project the high-dimensional data onto a lower-dimensional space while preserving most of the data’s structure.
t-SNE is a nonlinear technique that aims to maintain the local structure of the high-dimensional data in the lower-dimensional space. It’s particularly useful for visualizing high-dimensional data in a 2D or 3D space, as it tends to cluster similar data points together.
In your project, you would need to develop a suitable method for dimensionality reduction that:
Retains meaningful features: The chosen technique should preserve important relationships and structures in the latent space, so that the reduced representation remains meaningful and allows for intuitive navigation.
Provides an intuitive navigation experience: The reduced representation should be easy to navigate with the handheld controller. Users should be able to explore the latent space and understand the relationship between different points in this space.
Are there any existing controllers for high dimensional spaces?
Before spending a ton of time reinventing the wheel here, I asked if it knew of any existing controllers which could already handle this kind of high dimensional navigation. It gave these ideas:
While there isn’t a specific controller designed solely for navigating high-dimensional spaces, several existing devices can be adapted or customized for this purpose. Some examples include:
3Dconnexion SpaceMouse: This 3D controller is designed primarily for navigating 3D environments in CAD, modeling, and design software. It features a pressure-sensitive puck that can be pushed, pulled, twisted, or tilted to control movement in up to six degrees of freedom (X, Y, Z, pitch, roll, and yaw). Although it was not designed specifically for high-dimensional latent spaces, it could potentially be adapted for such a purpose with custom software.
MIDI controllers: MIDI controllers are used to control music production software and instruments, but they often feature a variety of knobs, sliders, and buttons that can be mapped to different parameters. These controllers could potentially be adapted for navigating high-dimensional spaces by assigning each control to a specific dimension or function.
The SpaceMouse actually looks pretty cool for 3D work like CAD:
And I saw elsewhere those buttons are all assignable to keyboard shortcuts, so perhaps there’s something there… I’ve also experimented with using small MIDI keyboards as custom macro controllers in the past (basically just for an assignable bank of buttons), and found that to be pretty useful too. So there might be something there, and I find this idea that music itself might just be a high dimensional space to be pretty intriguing.
User experience moving through High-D space
I then went in the direction of trying to nail down more specifics about what the UX might be like as you move from point to point within a latent or high dimensional space. Here’s excerpts around that:
Navigating through a latent space using a physical controller can be an immersive and engaging experience for the user. Here’s a step-by-step walkthrough of what the user experience might be like:
Initial setup: The user connects the handheld controller to the software interface running on their computer. The software interface presents a visual representation of the latent space, with the current position highlighted or centered.
Starting point: The user begins at a specific point in the latent space, which corresponds to a unique image generated by the image diffusion model.
Exploration: Using the handheld controller’s inputs (joysticks, dials, and buttons), the user navigates through the latent space by moving along different dimensions. As they move, the software interface updates the displayed image in real-time, providing a fluid and responsive experience.
Discovery: As the user navigates through the latent space, they will encounter images with varying degrees of similarity to their starting point. The user can find interesting, unique, or aesthetically pleasing images during their exploration.
Adjustments: The user can use the controller’s dials to adjust settings like interpolation speed, image resolution, or level of detail. These adjustments can help fine-tune the navigation experience, making it more enjoyable and tailored to the user’s preferences.
Saving and organization: If the user discovers an interesting image, they can press a dedicated button on the controller to save it to their image library. The software interface can include organizational tools, like folders or tags, to help users manage their saved images.
Model or category switching: The user can press a button on the controller to toggle between different image diffusion models or categories, allowing them to explore various aspects of the latent space.
I’ll come back to this notion of categories in a bit.
Mapping Latent Space
First, I proposed that there should be some kind of map that gets generated from explorations, so you can see and reproduce the track you took during a session (or across sessions).
Adding a map that visualizes the user’s path of travel and marks saved images as points of interest would further enhance the user experience and provide additional context for their exploration of the latent space. Here’s how the map feature could work:
Display: The software interface could include a separate panel or window displaying the map. The map can be a 2D or 3D representation of the reduced-dimensional latent space.
Path visualization: As the user navigates through the latent space, the map can show their path of travel as a continuous line or a series of connected points. This provides a visual reference for the user’s exploration and can help them understand the relationships between different areas of the latent space.
Saved points: When the user saves an image, the corresponding point in the latent space is marked on the map. This can be done using different symbols, colors, or icons to indicate the saved points. The user can hover over or click on these points to see a thumbnail of the saved image and access additional information or options (e.g., renaming, adding tags, or opening the full-resolution image).
Zoom and pan: The user can zoom in or out and pan around the map to get a better view of their path and the saved points. This allows them to easily navigate and manage their saved images in the context of their exploration.
History navigation: The map can also serve as a way to navigate through the user’s exploration history. By clicking on points along their path, the user can quickly return to previous locations in the latent space.
Exporting the map: The user can have the option to export the map as an image or a file format compatible with other mapping or visualization tools. This can help them share their exploration with others or analyze their path in more detail.
What the user sees in real time
Got it to nail down more clearly what the user sees in real time as they navigate along dimensions…
When the user navigates from one point to another in the latent space using the physical controller, they will see a continuous and smooth transition of images in real-time on the software interface. The images displayed during this transition will represent intermediate points in the latent space between the starting point and the destination.
The real-time visualization of the user’s movement through the latent space can be achieved using interpolation techniques, such as linear or spherical interpolation. Interpolation allows for a smooth blending of features between the starting and destination images, creating a seamless sequence of images that gradually change as the user moves through the latent space.
As the user navigates, they will see the images evolving, with features from the starting and destination points mixing and merging in different ways. This real-time visual feedback helps the user understand the relationships between different points in the latent space and discover new, interesting, or aesthetically pleasing images during their exploration.
Mode switching & dimension assignment
I’m flying by the seat of my pants here imagining this device, but having used image diffusion models a lot (I’m up to 92 AI art books that I published), I have a pretty strong if still intuitive sense of how it might all work. One thing I suggested was that we’d need a way to rapidly switch controller modes and/or assign dimensions to manipulate on the fly. This might be tricky given that, for instance, even a single pixel could be considered a dimension (depending on the model). Here’s what CGPT offered around this topic:
You’re correct that allowing users to rapidly flip between or reassign different dimensions to manipulate as they travel through the latent space is important for an intuitive and engaging experience. Here are a few potential solutions to achieve this:
Mode switching: Assign a button on the controller to switch between different modes, where each mode corresponds to a specific set of dimensions that the user can manipulate. In each mode, the controller’s inputs (joysticks, dials, etc.) can be reassigned to control the dimensions associated with that mode. The software interface can provide visual feedback indicating the current mode and the dimensions being controlled.
Dynamic dimension assignment: Include touch-sensitive controls or additional buttons on the controller that can be used to cycle through dimensions for each input control. For example, a user could tap a button to cycle the X-axis joystick control through different dimensions, while another button could be used to cycle the Y-axis control. Visual feedback on the software interface can display the currently assigned dimensions.
Custom presets: Allow users to create custom presets that define specific combinations of dimensions to be controlled by the inputs. These presets can be saved and loaded using the controller’s buttons or through the software interface. This enables users to quickly switch between their preferred dimension assignments during navigation.
On-screen interface: Implement an on-screen interface that users can interact with directly using a mouse, touch screen, or another pointing device. The on-screen interface can be used to change the dimensions assigned to the controller’s inputs, providing a visual way to manage dimension assignments.
It also offered voice & gesture controls as additional options, but I don’t want to include either of those in initial design scope. To me those are just secondary ways of accessing primary functions controlled in the app.
What the app UI might consist of
I even got it to output very basic SVG code of the software layout, but it’s not worth sharing. So here’s the text description of what it thinks the app window would contain:
Main window: The main window should be divided into two primary sections, the image preview area and the control panel.
Image preview area: This section occupies the majority of the main window and displays the current image generated based on the user’s position in the latent space. It should be large enough to comfortably view the image and observe the real-time changes as the user navigates through the latent space.
Control panel: The control panel can be located on the side or bottom of the main window and contain various interactive elements for controlling the software and adjusting settings. This panel can include: a. Dimension assignment: Include a list or grid of available dimensions, where users can click on a dimension to assign it to a specific controller input. The currently assigned dimensions can be highlighted or marked in some way. b. Model or category selection: Provide a dropdown menu or a list of available image diffusion models or categories for the user to choose from. c. Presets and modes: Include buttons or a dropdown menu for selecting custom presets or modes that define specific dimension assignments for the controller inputs. d. Interpolation settings: Provide sliders or input fields for adjusting interpolation speed, image resolution, or other relevant parameters. e. Navigation history: Display a visual timeline or a list of the user’s previous locations in the latent space, allowing them to click on a point to return to that location. f. Map view toggle: Include a button or checkbox to toggle the display of the map view, which shows the user’s path of travel and saved points in the latent space.
Map view (optional): If the map view is enabled, it can be displayed as a separate panel or window, either floating or docked within the main window. The map should show the user’s path of travel, marked saved points, and allow for zooming and panning.
Agile user stories
I wanted to see just how far I could push it to do the job of the product manager, so I had it write up some Agile user stories describing the software product only:
As a user, I want to see a real-time preview of the image generated based on my position in the latent space, so that I can observe the changes as I navigate.
As a user, I want to assign different dimensions to my controller inputs, so that I can easily switch between controlling various dimensions in the latent space.
As a user, I want to choose between different image diffusion models or categories, so that I can explore various aspects of the latent space.
As a user, I want to save interesting images I discover during my exploration, so that I can revisit or use them later.
As a user, I want to adjust interpolation settings like speed, image resolution, and level of detail, so that I can fine-tune my navigation experience.
As a user, I want to view a map of my path of travel through the latent space, so that I can better understand the relationships between different areas and locate saved images.
As a user, I want to access my navigation history, so that I can quickly return to previous locations in the latent space.
As a user, I want to create custom presets for dimension assignments, so that I can quickly switch between my preferred combinations during navigation.
There are obviously technical issues to work out here, but as a PM reading those, they actually feel like something that could probably be built with an engineering team who has suitable domain expertise.
Categories
Since it brought up ‘categories’ again, I finally inquired what it meant by that, and it’s very useful:
In the context of item 3, categories refer to different groups or subsets within the latent space that are associated with specific types of images, visual features, or content. These categories could be derived from the underlying image diffusion model, which may have been trained on various classes of images, or they could be based on user-defined groupings.
For example, if the image diffusion model was trained on a diverse dataset containing images of animals, landscapes, and buildings, the categories could be “Animals,” “Landscapes,” and “Buildings.” By selecting a category, the user would be able to focus their exploration within the latent space on images that correspond to that specific category.
Alternatively, users may create their own custom categories based on their interests or preferences. These custom categories could be defined by selecting and tagging images during the exploration, effectively creating a user-defined grouping within the latent space.
Allowing users to choose between different categories helps them to more efficiently explore the latent space, as they can focus on specific types of images or visual features that are of interest to them.
Visit adjacent neighborhoods
One thing I always want when I explore in Midjourney especially is, if I find a good area within the latent space, I want to be able to easily and effectively scan adjacent neighborhoods. Here’s what CGPT says on how to potentially do that:
Local exploration mode: When the user enables local exploration mode, they will be presented with multiple smaller image previews that represent the adjacent neighborhoods in the latent space. These previews can be arranged in a grid or circular pattern around the current position.
Neighborhood selection: The user can click on any of the smaller image previews to instantly move to that adjacent neighborhood in the latent space. This will update the main image preview to reflect the new position and generate a new set of adjacent neighborhood previews.
Adjusting proximity: Provide a slider or input field in the software interface that allows the user to adjust the “proximity” of the adjacent neighborhoods. A smaller proximity value will result in neighborhoods that are closer to the current position in the latent space, while a larger value will display neighborhoods that are farther away. This enables users to fine-tune the scope of their local exploration.
While I’m at it, the idea of actually physically mapping out a high dimensional space like this is not so crazy, it seems. I found this Google Arts & Culture experiment, which they’re calling a t-SNE map. It’s hard to do justice to this visualization in only 2D, but here’s a screenshot:
That’s a pretty zoomed out view of a landscape of visually similar images. Here’s a more zoomed in look:
The best thing is really to try it out on the website, but it seems like their sidebar items in the guided basically correspond to what CGPT describes as ‘categories.’
What’m envisioning here is even more fluid and for lack of a better word “VR,” in that not only could you see the specific images like you can above, but it would fluidly move between them with attributes (dimensions) receding or increasing as you travel.
Other modes of use
I asked about what other modes users might want in addition to local exploration, described earlier. It said:
Global exploration mode: This mode allows users to explore the entire latent space more freely, without being restricted to a specific category or local area. Users can navigate in any direction, discovering a wide variety of images and visual features along the way.
Category-focused mode: As mentioned earlier, this mode lets users focus on a specific category within the latent space, such as animals, landscapes, buildings, or user-defined categories. The navigation and exploration will be tailored to that specific category, making it easier for users to find images that match their interests.
Favorites mode: In this mode, users can revisit their saved images and explore the neighborhoods around those points. This can help users to discover new images that are similar to their favorites or to fine-tune their saved images by navigating the local areas around them.
Guided tour mode: This mode provides a pre-defined or user-generated sequence of points in the latent space that users can navigate through. The guided tour can be based on specific themes, popular locations, or user-created paths. This mode can be useful for users who are new to the latent space exploration or who want to follow a curated experience.
Random walk mode: In this mode, the software automatically generates a random path through the latent space, with the user being able to influence the direction or speed of the movement. This can create a more relaxed and serendipitous exploration experience, where users can discover unexpected images and visual features.
Search mode: If the underlying image diffusion model supports it, users can search for specific images or visual features within the latent space using keywords or example images. In this mode, the software would display search results as points in the latent space, which users can then navigate to and explore.
Collaborative mode: This mode allows multiple users to explore the latent space together, either by sharing a single exploration session or by viewing and interacting with each other’s paths and saved images. This can create a more social and engaging exploration experience, with users being able to share discoveries and collaborate on finding interesting images.
Referent-based mode
As I explained to CGPT:
i keep visualizing the ability to like define a group of referents, points or features or whatever in the latent space (like a blue ball, an elephant, a storm, etc) and then… something… explore themes around those but with different configurations or treatments. does that make sense?
And it replied, partial:
What you’re describing is a mode that allows users to define a group of referent points, features, or themes in the latent space and then explore variations and relationships between those referents. This can help users discover new and interesting images that share similarities with the chosen referents but have different configurations, styles, or treatments.
Here’s a potential approach to implementing this idea:
Referent selection: Provide a user interface element (e.g., a list or a grid) where users can add, remove, or modify referent points, features, or themes. Users can either manually select these referents by navigating the latent space or use a search functionality if the underlying image diffusion model supports it.
Theme exploration: Once the referents are defined, the software generates a set of variations or combinations based on the selected referents. These variations can be displayed as a grid of image previews or a more interactive visualization like a 2D or 3D scatter plot.
As it went on (and I clipped some less useful parts), I got to wondering, isn’t this just the same thing as simple prompt mode? How would this be different than just entering words for the things I want as my referents?
The concept you described earlier is different from a simple prompt mode. Both modes are aimed at guiding users to explore specific themes or concepts in the latent space, but they approach this goal in different ways.
In a simple prompt mode, users input a textual prompt (e.g., a keyword or a phrase), and the software generates a single image or a small set of images related to the prompt. The exploration in prompt mode is more linear, focusing on the specific concept described by the prompt. Users can adjust the prompt to refine the resulting images or explore other concepts, but the focus remains on a single theme at a time.
On the other hand, the referent-based mode you described earlier involves defining a group of referent points, features, or themes in the latent space and then exploring variations and relationships between those referents. The exploration in this mode is more dynamic and interactive, allowing users to discover new and interesting images that share similarities with multiple referents, while presenting different configurations, styles, or treatments.
In the referent-based mode, users have more control over the exploration process, as they can adjust parameters, add or remove referents, and interact with the generated variations in a more flexible way. This mode also promotes a deeper understanding of the relationships and connections between different areas of the latent space, as users can visually observe and manipulate the variations and combinations of referents.
So I think this distinction might be like… text prompts could be used to get me to a particular place in the latent space (or to modify what I’m seeing already). Whereas, a referent-mode might allow me to – as I’m exploring – say select certain entities that I see within the visualization… and maybe add them to a basket or something that I can take along with me as I go. And those referents wouuld show up or stick with me as I continue navigating the space, until I choose to release them. Something like that.
Short list of required app features
So, from there I had it go back through and distill all of this into a list of required features for the app:
Here is a short list of required features for the app, based on the discussions so far:
Real-time image preview: Display the generated image based on the user’s current position in the latent space, updating in real-time as the user navigates.
Dimension assignment: Allow users to assign different dimensions of the latent space to specific inputs or controls, enabling them to easily switch between controlling various dimensions.
Category selection: Provide the ability to choose between different categories or subsets within the latent space to focus the exploration on specific types of images or content.
Save images: Enable users to save interesting images they discover during exploration for future reference or use.
Interpolation settings: Allow users to adjust settings like interpolation speed, image resolution, and level of detail to fine-tune their navigation experience.
Exploration modes: a. Local exploration mode b. Global exploration mode c. Category-focused mode d. Favorites mode e. Guided tour mode f. Random walk mode g. Search mode (if supported by the image diffusion model) h. Collaborative mode i. Referent-based mode
Map view: Display a map of the user’s path of travel through the latent space, with the ability to show saved points, referent points, and other user-defined landmarks.
Navigation history: Provide access to the user’s navigation history, allowing them to quickly return to previous locations in the latent space.
Custom presets: Allow users to create and manage custom presets for dimension assignments, making it easy to switch between preferred combinations during navigation.
Adjustable proximity for local exploration: Provide a slider or input field to adjust the proximity of adjacent neighborhoods during local exploration.
Help and tutorials: Offer in-app help and tutorials to assist users in understanding and utilizing the various features and exploration modes.
Minimal feature set for a v1
So from that, I asked it to reduce those features for a stripped down v1 prototype/proof of concept, and it said:
Real-time image preview: Display the generated image based on the user’s current position in the latent space, updating in real-time as the user navigates.
Dimension assignment: Allow users to assign different dimensions of the latent space to specific inputs or controls, enabling them to easily switch between controlling various dimensions.
Save images: Enable users to save interesting images they discover during exploration for future reference or use.
Local exploration mode: Provide a focused exploration mode that allows users to navigate the immediate neighborhood around a specific point in the latent space.
Global exploration mode: Offer a more free-form exploration mode that enables users to explore the entire latent space without restrictions.
Adjustable proximity for local exploration: Provide a slider or input field to adjust the proximity of adjacent neighborhoods during local exploration.
Navigation history: Provide access to the user’s navigation history, allowing them to quickly return to previous locations in the latent space.
Wrap-Up
There’s a lot to digest in all of this of course. Apart from just the interesting aspects of the technical challenges described in this product design, the thing I’m most excited about is just how far you can take ChatGPT in a product design & planning session. The answer, at least in v4, is really really damn far.
Then, if it’s something that is amenable to the types of visual representations that Midjourney is capable of, this starts to become a pretty powerful set of tools. I suspect once they publicly release the image description tools for GPT-4, you’d also gain other possible vectors of insight into this design iteration process.
All told, this direction of a controller (hardware + software) specifically for Midjourney, but adaptable to any high-dimensional space or model, is a really compelling one for me as an artist making heavy use of the tools in the somewhat primitive state we see them today.
I’m not going to guarantee that I’ll be the one who builds it, but perhaps once Github Copilot X is released, I’ll be that much closer to actually being able to build some aspects of this down the road. If nothing else, I’m hoping these blog posts exploring these topics can inspire other innovations in the field. Exciting stuff!
This is something I came up with years ago living in a basement in Seattle (c. 2005?). I never had any name for the concept between “thinware” (software) and “thickware” (hardware), and the ability to sort of fluidly flip back and forth between the two. Seemed like a great science fiction concept, if nothing else. But how many sci fi conceptions have actually inspired real inventions? Tons! It’s almost like someone has to first imagine it, and then it can be built in our reality.
I used ChatGPT (v4) to help me come up with a name and better explanation for the whole idea package. Here it is.
What is the Holomorphic Continuum?
The Holomorphic Continuum is an innovative concept that reimagines the boundaries between the physical and digital worlds, challenging the traditional separation between hardware and software components in technology. This continuum represents a fluid spectrum, where hardware (thickware) and software (thinware) can coexist, interact, and transform into one another, effectively dissolving the barriers that have historically defined their roles and functions.
As technology advances, the need for more integrated and adaptive systems becomes increasingly apparent. The Holomorphic Continuum offers a framework for developing such systems by fostering a more unified and dynamic understanding of the relationship between hardware and software. By recognizing their potential for seamless interaction, this concept encourages the exploration of new possibilities in technology, extending beyond the limitations of current paradigms.
Within the context of the Holomorphic Continuum, an array of transformative technologies can be conceived, giving rise to novel applications and user experiences. One such example is the HoloMorphic System, a cutting-edge approach to the integration of hardware and software components, designed to facilitate greater adaptability, efficiency, and intuitive interaction.
Understanding Holomorphic Systems
A HoloMorphic System represents a novel approach to the interaction between hardware and software components in technology. The term “HoloMorphic” is derived from the words “holistic” and “morph,” suggesting a comprehensive and transformative relationship between the two domains. In this system, hardware and software exist along a fluid continuum, enabling seamless transitions between physical and digital states.
Key features of the HoloMorphic System include:
Dynamic State Transitions: Components within the system can transition between thickware (hardware) and thinware (software) states through processes of instantiation (thickening) and de-instantiation (thinning). This allows for flexible adaptation to different tasks and requirements.
Seamless Integration: The HoloMorphic System promotes a high degree of interoperability between hardware and software components, enabling more efficient communication and collaboration between different technological elements.
Adaptive Functionality: The fluid nature of the HoloMorphic System allows for the development of adaptive technologies that can automatically adjust their form and function to better suit specific tasks or contexts.
Enhanced User Experience: By blurring the lines between hardware and software, the HoloMorphic System has the potential to create more immersive and intuitive user experiences, allowing users to interact with technology in a more natural and seamless manner.
Possible applications for HoloMorphic Systems could range from advanced robotics, where physical components can morph to adapt to various tasks, to immersive virtual and augmented reality environments, where digital elements can be instantiated into physical forms to create more interactive experiences.
As a cutting-edge concept, the HoloMorphic System challenges traditional distinctions between hardware and software, opening the door to new possibilities in technological innovation and human-computer interaction.
Hyperrealities and Uncanny Valleys: Provoking Dialogue through AI-Generated Imagery
(a.k.a. “I dream in image generations”)
Written with help from ChatGPT (v4)
Having worked for years in content moderation, my artistic practice delves into issues around the realm of AI image and text generation: embarking on (mid)journeys that explore the boundaries of photorealistic and hyperrealistic depictions of unreal events. By exploring speculative alternative realities, I construct visual narratives that portray political and other figures in hypothetical scenarios that have never unfolded within our objective consensus reality. Through this process, I aim to evoke curiosity and critical thinking among viewers, encouraging them to closely examine and question the information presented to them.
Humor and absurdity intertwine with my work, questioning the dichotomy between good and bad content and challenging societal limits through satire and parody. I firmly believe that the power of image making extends beyond aesthetics, potentially serving as a catalyst for profound societal discussions and inspiring real political change. Thus, I advocate for inclusive access and development, as well as encouraging diverse ownership and direction of these tools.
Within the context of postmodernism, my work ventures into the realm of hyperreality, blurring the boundaries between the real and the imagined. Deliberately engaging with the concept of the uncanny valley, AI-generated images can elicit a sense of uncertainty and discomfort. This unease prompts us to question the authenticity and reliability of what we perceive, particularly the information presented by those in positions of authority. It encourages a deeper examination of the potential consequences and implications of AI-generated content.
As a society, we must recognize that the challenges posed by AI-generated content and potential misuses cannot be solved by a single technical fix, nor should we fall into the trap of relying on AI to tell us what is “real.” Instead, AI-generated content represents an ongoing condition that requires continuous adaptation and proactive efforts to address issues and improve. Rather than advocating for the cessation of these technologies, I believe that navigating this landscape requires continuous critical thinking, adaptation, and open dialogue. This approach fosters a more informed and discerning approach to the consumption and interpretation of AI-generated images, text, and other content.
My work aims to provoke thought, spark conversations, and contribute to the larger discourse surrounding the impact and implications of AI-generated content in our increasingly interconnected world. Everyone needs to have their say in this conversation so we can develop a more nuanced understanding of the role and influence of AI in our society.
The image concept of Trump loving pelicans started last week, and this was just a natural extension of the idea. I had to redo the images in --ar 16:9 to fit YouTube’s aspect ratio, but it seems to have worked out. (I tried re-using seeds, and also in remix mode adding the new aspect ratio, but none of those were fully reliable)
Only sad part is not all the images made it into the final cut. Here’s an expanded set of 41 images in 16:9 that includes everything in the video plus some extras. The full downloaded set was actually 168 items but I can’t get Imgur to accept them all currently.
I’ve been struggling for some time now to land on terms that adequately portray the “uncanny valley” not just between humans vs. AIs, but also between the so-called real and unreal, fiction, and non-fiction, etc. Though I’ve used somewhat for this the term “hyperreality,” it’s not a term everyone understands or agrees on the meaning of. Some might just think it refers to ultra-realistic paitings, though I use it in the original postmodernist sense of seamlessly combining fact and fiction. By and large though, I’ve found all the labels and attempts at labeling artifacts like those created with help from AI tools especially to be somewhat lacking in nuance and depth (which is why I’ve been trying to articulate a multi-dimensional style of analysis of them).
I took these questions recently into ChatGPT (v4), asking it to come up with ideas related to this, and to essentially “non-binary” but for relationship to reality, rather than to gender identity. That is, how can we talk about things that have a more complex relation to “real life” than simply real or fake, etc. Though I guess it’s a little obvious as a moniker, I let CGPT do a piece exploring this idea of “Reality-Fluid” art, and here it is.
In the midst of the digital revolution, we are witnessing an unprecedented convergence of technology, creativity, and reality. As artificial intelligence (AI) matures, it is not only transforming the ways we live and work, but also the ways we express and interpret our world. This revolutionary juncture has given birth to a new form of artistic expression—Reality-Fluid Art. This genre challenges the binary view of ‘real’ vs ‘fabricated’, instead inviting us into a fluid spectrum that blends these extremes.
What is Reality-Fluid Art?
Reality-Fluid Art, much like the term ‘gender-fluid’ referring to a gender identity that may change or move along the gender spectrum, refers to works of art that oscillate, blend, and blur the boundaries between the real and the fabricated. Using AI as a tool, artists create pieces that are not strictly tethered to an objective reality, nor entirely born of pure imagination. Instead, these works exist on a continuum that encompasses both these extremes and everything in between.
Why is it important?
The exploration of this new genre is vital for several reasons:
Reflects our Digital Reality
As we increasingly inhabit digital spaces, our perception of reality itself is being reshaped. We navigate through a world where the lines between the physical and the digital, the authentic and the manipulated, are continuously blurred. Reality-Fluid Art mirrors this evolving condition of our lived experience, providing a lens through which we can understand and articulate it.
Challenges the Binary Paradigm
Traditionally, art has been categorized into forms that either represent reality or depart from it—realism or abstraction, documentary or fiction. Reality-Fluid Art disrupts this binary, creating a space for works that embody both, or neither, simultaneously. This paradigm shift expands the possibilities for artistic expression and interpretation, offering a more nuanced understanding of reality and its many layers.
Encourages Ethical Discussions
As AI becomes more sophisticated in creating realistic depictions, ethical questions around authenticity, representation, and truth-telling become increasingly important. Reality-Fluid Art invites these conversations, encouraging us to grapple with the implications of AI in our perception of reality.
Fosters Innovation
The exploration of Reality-Fluid Art promotes technological and artistic innovation. It pushes the boundaries of what AI can achieve, challenging artists and technologists to develop new techniques, methods, and ways of thinking.
The era of Reality-Fluid Art is upon us, marking an exciting and challenging time in the evolution of artistic expression. As we engage with these works, we are not only audience members but also participants in a broader dialogue about the nature of reality, the role of technology, and the meaning of authenticity in our increasingly digital world.
There’s another downside: Once people grow accustomed to fake AI-generated videos, they’ll become even more hardened, cynical, and harder to reach and convince—for politicians, admakers, journalists, everyone.
With AI technology becoming more common in our lives, voter skepticism will only continue to grow.
::Clutches pearls::
Gasp! We wouldn’t want voters to be skeptical! Won’t someone think of the poor politicians and admakers who will need to work harder??
Not to grind an axe here, I’m just completely tapped out on hearing these arguments suggesting that skepticism and not being easily convinced are bad things, especially among voters. To me this is such absolute crazy talk, because of just how fucked up things are in the world right now; everyone should damn well better be skeptical af of basically just about everything. We should challenge it, we should question it, we should not accept being easily convinced. We should seek proof. This is the Way.
I’ve chronicled their creation here quite a bit without perhaps ever explaining in a straightforward way what I actually mean by AI lore books. Because I’d never quite articulated it to myself before, but ChatGPT helped me bring it home with the text below.
The AI Lore books are a collection of over 100 immersive volumes that meld the creative powers of artificial intelligence with pulp sci-fi, conspiracy fiction, and dystopian fantasy. These books are an entirely unique literary experience, featuring AI-generated images, text, and world-building that transport readers into an ever-evolving, thought-provoking narrative landscape, distributed across many volumes and universes.
The AI Lore books push the boundaries of storytelling, offering an unparalleled exploration of the human imagination and the future of fiction. As the lines between reality and fiction blur thanks to today’s hyperreal technologies, these books challenge our assumptions about the nature of creativity, the role of artificial intelligence in the arts, and the very definition of authorship. (/ChatGPT)
Genesis
Okay, back to me writing… First of all, I’ve always liked lore; I’m a nerd like that. When I wrote my first book, The Lost Direction, I was really into the lore. The book is more of a compendium of lore, set around a frame story – a bit like how the Arabian Nights is presented, or similar story-within-a-story novels. It’s a time-honored convention, though certainly not everyone’s cup of tea, which I respect.
During the pandemic, I had the good fortune to have that book reviewed by the Literary Review of Canada, which is a fairly prestigious print magazine up here. The reviewer was not into what they considered to be an over-reliance on lore. Understandable, but hey. Here are some choice quotes:
…[H]erein lies the challenge: how to balance lore with storytelling. Lore is not story. The task for authors is to take this raw material and fashion it into conflict, character arcs, and thematic exploration. As the video game writer and designer Matthew Colville says, “The work you did on your world, you did that work for you. And no one is going to care about it unless you can contextualize it dramatically. That’s the hard work.” The audience does not owe the creators interest in whatever rich backstory they have conjured for their settings. Readers want what they always want: narrative, character, conflict — the muscle and sinew of all storytelling. In that regard, fantasy is not unique, but this is where many of its authors go astray — they mistake lore for storytelling. Whether it’s from falling in love with their own mythos or from not trusting the reader to intuit backstory, a work fails when it collapses into exposition dumps that overwhelm the progression of the plot. Explanation is not engagement. The Force is an intriguing concept, but audiences fell in love with Star Wars because of Luke Skywalker and Darth Vader.
I’m not going to say any of that is anything but the truth. People like stories. But some people also like lore. It’s a niche market, and in my opinion the web today basically requires you to find a good niche market to make your splash in. I don’t think the reviewer saw it that way though:
So much time and effort is spent explaining the histories of these various civilizations and the tales of their conflicts and political milieus that the novel never settles into something resembling a compelling story. It reads instead like a textbook.
I’m someone who in my youth spent a great deal of time poring over the “textbooks” documenting the lore and legendarium of fictional worlds like those found in The Lord of the Rings (which my book is most certainly not equivalent to by any stretch of the imagination). While LOTR is a masterpiece, there’s a strong chance that I as a reader actually spent a great deal more time in the lore universe than in the actual book itself. But maybe that’s just me… though my sales seem to suggest otherwise.
Boucher is in gross violation of a classic, albeit overused and oversimplified, maxim: Show, don’t tell. In order to enchant, the author must engage, not explicate.
With the AI Lore books, in some cases I intentionally leaned into the exact inverse of this “age-old” wisdom. I tell. I tell a lot. Many, the majority, of the AI lore books are nothing more and nothing less than massive dumps of exposition, in both text and pictures. The overlap in a non-linear way. Many times there is not really any “story” and there are not any characters.
So if people want you to show not tell, why do so many of my readers come back and buy ten, twenty, or even – in one case more – than thirty volumes?
While I fully respect this reviewer’s opinion of that original novel – which was all “hand-written” (with no AI) – I’ll admit that this review acted as sort of the grain of sand that I pondered for a very long time, and which helped agitate to create the pearls that are the AI Lore books series. They turn completely on its head everything said in this review, and all the conventional wisdom of what a book is, and what even authorship is or is becoming thanks to today’s technologies.
I’ve noticed r/midjourney has a somewhat confusing rule around political content which reads:
4) No extraneous political discussion. Political images are okay, as long as they don’t fall under rules 2 and 3 (don’t be an edgelord or post anything agains MJ’s TOS). Discussion however will be moderated at our discretion.
Rule 3 regards NSFW content, which this is not. Rule 2 is also somewhat vague, like so many community-enforced rules on Reddit:
Don’t be a jerk. Be kind and respectful to each other. Don’t post inflammatory or edgelord posts– aka lets try and avoid toxicity. This will be enforced at moderators discretion.
I personally don’t think this post contains “toxicity” but perhaps it could be considered “edgelord.” Anyway, it wasn’t intended in a mean spirit; it was a genuine exploration of an alternative reality that is already being reported on in the news.
I get why people might not want to be constantly confronted with political content, but that is also the world we live in. Literally everything is political.
I’ve also seen extraordinarily vague “official” rules from the Midjourney bot in Discord, which include prohibitions against things such as:
Imagery that might be considered culturally insensitive
Sadly, Midjourney’s proposed “solution” to these extraordinarily complex speech issues is “let’s throw AI at it” – multiple layers of AI, without any human intervention. That is wrong.
Personally, I think political image-making is one of the most interesting and exciting uses of Midjourney, though I agree that image makers ought to be considerate and careful about what they produce and how it is distributed. But it’s not something which is going away, and we need to learn how to live with it.
I found recently an old interview with psychonaut Terence McKenna, part of which I thought neatly meshes with generative AI & what he claimed was a natural process of increasing complexity & novelty within the universe. I think his schtick on this starts at around 29:46 in the video, the timestamp is queued up below (but I recommend the whole video):
Via the transcript on YT:
what is man’s purpose to advance and preserve novelty. you know this is an ethical position it means you don’t replace rainforests with pastures you don’t censor books you don’t lean on people who make gender choices different from yours. no, the purpose of of being a human is to complexify reality. even more, to hand on a more diverse more complicated more multifaceted
He picks up the thread again later on, around 53:33:
as we go forward and the novelty continues to climax… well novelty is not necessarily good or nice novelty is complex that’s what it is and so I see really a concatenation of tendencies and forces. here at the end it’s only going to get weirder the level of contradiction is going to rise excruciatingly even beyond the excruciatingly present levels of contradiction.
so I think it’s just going to get weirder and weirder and weirder and finally it’s going to be so weird that people are going to have to talk about how weird it is. and at that point novelty Theory can come out of the woods because eventually people are going to say what the hell is going on it’s just too nuts it’s not enough to say it’s nuts you have to explain why…
Anyway, maybe this is all just apologia on my part for trolling, but I’m excited to re-discover this aspect of McKenna’s work, because it feels to some extent emotionally true, and is the first rationale I’ve landed on which actually offers some strange glimmer even of hope in this world of mixed up generative AI.
