Following up on my post about trying to use AI to design a Midjourney controller, I asked ChatGPT for help doing a generalized blog post introducing the issues here. Here it is with light edits from me…

Introduction

In recent years, the advancements in artificial intelligence and machine learning have led to the development of sophisticated generative models capable of producing stunning and realistic images. One of the most notable types of these models is image diffusion models, which can generate a wide variety of images based on their underlying latent spaces. However, navigating these high-dimensional latent spaces and understanding their structure can be a challenging task.

Currently, our exploration of latent spaces is often haphazard and sporadic, with no maps or guides to help us understand their complex topography. What if we could have, for lack of a better comparison, a “Google Street View” for exploring the latent space of image diffusion models? This blog posts introduces the idea of and problem space around a hardware-software controller which would bring us closer to being able to more intuitively navigate high-dimensional spaces.

Navigating High-Dimensional Latent Spaces

Latent spaces are high-dimensional mathematical spaces that encode the essential features and variations of generated images. The challenge lies in creating an intuitive method to explore these spaces and discover interesting or meaningful images. The proposed solution consists of a handheld controller, combined with a software interface, that can translate physical actions into navigation through the latent space.

The physical controller could include components such as joysticks, dials, or sliders, which allow the user to manipulate specific dimensions of the latent space. The software interface would display the generated image based on the user’s current position in the latent space and update in real-time as the user navigates. Additionally, the interface could provide various exploration modes, such as local and global exploration, to facilitate different types of exploration experiences.

Dimensionality Reduction and User Experience

One of the core challenges in navigating high-dimensional latent spaces is the need to reduce their dimensionality to a more manageable form, without losing meaningful features. Techniques such as PCA or t-SNE can be used to retain important characteristics while providing an intuitive navigation experience.

As users navigate the latent space using the controller and software, they would be able to view their path of travel, save points of interest, and explore adjacent neighborhoods. The software could also allow users to switch between different dimensions on the fly, providing a more dynamic and flexible exploration experience.

Possible Exploration Modes

In addition to local exploration, which focuses on the immediate neighborhood around a specific point in the latent space, other modes could be integrated into the software. For example, a prompt-based mode would enable users to input text prompts and generate images based on those themes. Another possibility is a referent-based mode, where users can define a set of referents, points, or features within the latent space (such as a blue ball, an elephant, or a storm). This mode would allow users to explore themes around these referents with different configurations or treatments, effectively enabling them to discover new and unique combinations of visual elements.

Conclusion

The prospect of an innovative hardware-software solution for navigating high-dimensional latent spaces opens up a world of creative possibilities, allowing users to delve into the intricate structures of generative models.

I kinda clipped it at the end, cause it always does corny conclusions, etc. It’s a little janky overall, but a good enough anchor on the topic to at least drop into the water for now.

Related writing & research