Flowering Spirit - An Exploration of AI Assisted Art

Everybody in the art world has been messing with AI recently, and I’m no exception. Personally, I find these tools really exciting and I’ve been experimenting quite a bit with how I can integrate them into my painting process. My recent painting, “Flowering Spirit”, represents the primary way I see myself working with AI going forward so I’m going to break down what I did and why. Please enjoy!

Not Mid

First off, let’s talk about the AI I’m not using. MidJourney is currently the hot topic, so a lot of comments I’m seeing about my work are asking if I’m using MidJourney specifically. I am using it, but not for this painting because for this process, I need to feed the AI a work-in-progress painting to start from rather than a simple written prompt.

Say Hello Mr. Disco

The AI I’m using for this painting is called Disco Diffusion. It’s publicly available and free to use (with a Google Colab subscription starting at $10 per month).

https://colab.research.google.com/github/alembics/disco-diffusion/blob/main/Disco_Diffusion.ipynb 

Disco is my first love in the AI art space and continues to be useful to me even as I explore other AI tools. Disco's default style is much more neutral than competing AIs and has greater control over its parameters. To me, Disco feels like an excellent starting point for anyone curious about this new field and is easy to recommend due to its open-source nature. Disco's one major drawback is speed.

While MidJourney typically takes about 60 seconds to complete an image, Disco takes about 20-40 minutes to complete depending on the parameters. That’s a big difference, but not a deal-breaker for me. Typically, I set Disco up in the morning to work throughout the day or before I go to bed so that I’ll have images to look at in the morning.

If you are interested in experimenting with it yourself, you can get up to speed very quickly using the tutorials from the YouTube channel Quick-Eyed-Sky: https://www.youtube.com/c/JPRobocat

Pretty Good Init

Disco Diffusion’s key feature that we’re going to be using today is called “init_image”. This allows us to feed an image to the AI and have it chew it up for us. We have control over all sorts of additional parameters about how much we want it to divert from the original, how hard we want it to work, and what we want it to achieve creatively.

In this case, I was working on an earlier version of “Flowering Spirit” that was painted exclusively in Photoshop. As it stood, the painting was... fine, but I wasn’t fully satisfied. The intent was to create something that looked less like a Magic: The Gathering illustration and more like fine art. I wanted to give it a more abstract and graphic quality than my usual work. As I said, the result of my manual painting process was…fine, but I found myself falling short of my own expectations. It’s just a little flat. In that way, it was the perfect fodder to experiment with adding a little AI fairy dust.

I uploaded the painting to my Google Drive and specified its location to Disco Diffusion as the Init_Image. I then gave a written description of the painting I was hoping to see it produce:

“A beautiful painting of a skeleton covered by flowers by James Jean”

While AI models can add elements of an artist’s flavor to an image, they cannot yet fully replicate a human’s style. So in this case, I’m using James Jean as a shorthand to describe the overall look and finish I’m hoping to achieve for this painting. The way it accomplishes this is not unlike the way I emulate other artist’s style when I build a mood board for my own paintings. It’s not really a copy, it’s more of a compilation of traits.

I asked for twenty variations along with three work-in-progress saves for each variation and set it to bake at the end of the day before I left work, so I would have them all waiting for me the next morning. When I arrived, all 20 pieces were done and waiting for me. Seeing them was like receiving a beautiful bouquet of flowers.

These images are falling much closer to my intended aesthetic than what I was able to achieve on my own, but the structure of the painting has almost totally fallen apart. As a collaborator, Mr. Disco is incredibly skilled at some things and a complete dunce at others. At least for now. The plan is to pull my favorite textures, ideas, and details from Mr. Disco’s work and combine it with my own back in Photoshop.

It’s Bashin’ Time

Photobashing is an art form that has existed primarily in the concept art world since its inception in the early days of digital artwork. It’s something I’ve never been much good at, so I’ve largely ignored it in favor of more direct methods of digital painting. 

More experienced photobashers are already taking the outputs from AI and build extraordinary images with them by weaving their pixels together in elaborate ways. I’m not good at that. Thankfully, the skill level required to combine multiple AI variations of a single image is very low. All I need to do in order to integrate Mr. Disco’s work with mine is to lay them on top of the original painting and use a layer mask to paint in the parts I want to use.

After only a few minutes of mushing different versions of this painting together, I’ve got a Frankenstein’s monster version that is making me very happy! It’s not perfect, but this process has brought energy and creativity back into my painting after a moment where I felt deeply lost and stuck. AI models have a naivete to them that is incredibly exciting to build on top of. Having the opportunity to build on top of these images feels like collaborating with another artist with whom I am sharing a deep bond.

Without losing control of my work, the contributions of the AI are adding things to it that I would never have imagined myself. The experience is like wearing an exoskeleton that enhances my creative powers.

Finishing Touches and Ruminations

The finishing touches to an AI-assisted painting are exactly the same as the finishing touches that I would apply to an unassisted painting. As a bit of a novice with this tool, I wonder if maybe I was a little too eager to leave the obvious tool marks of the AI in the painting as a matter of novelty. As I use this method more, I think I’m likely to spend more time working over the top of the AI’s work.

One of the important notes about this particular example is how long this process took. Once I had the images from the AI, the time it took for me to upscale them, photobash them, and finalize the painting was only a couple of hours. I was rushing to finish this piece so quickly partially as an experiment to see how fast I could make this work and partially due to a deadline. If I were to be working with the intent to create a high-resolution print of a painting I would have spent more time carefully hand placing more of my marks.

There is a lot of experimentation left to do and one of the things I’m left wondering is how fast it is possible to get a painting to a point where it’s ready to hand off to an AI for iteration and refinement. Disco Diffusion has some magical characteristics, but there are new AI models coming out all the time. It’s entirely possible that arcane uses of DALL-E 2’s toolset will allow a process like this that is both faster and more controllable than Disco. Beyond DALL-E 2, there are probably 100 more AIs yet to be developed.

Looking forward, it’s easy for me to imagine a full suite of AI tools that enable even more elaborate manipulation of images. What’s potentially possible for a single artist to create in a single day is beyond what anyone could have imagined a year ago. I, for one, am going to be enthusiastically using these tools to create paintings more quickly, more creatively, and at a higher level of polish. 

The largest question left lingering is what the implications of this technology will mean for the illustration industry, art education, or the larger creator economy. We’ll see! For now, I'm comfortable leaving speculation to more critical minds. My personal interest is in using these tools to achieve my own creative goals.

Thanks for joining me and thanks for your support here on Patreon.

Much love,

-Pete