Thoughts on AI art
Added 2023-06-01 06:13:47 +0000 UTCThe most recent video I published made some use of AI art. I wanted to write some reflections on that, more to clarify my own thoughts than anything else. Feel no obligation to read it all, I know you're subscribed here for math videos. If you have opinions on the topic and feel like adding a comment, I'd love to read it.
Long story short, I don't regret experimenting with using AI art, but for a mixture of reasons I don't plan to use these tools for future artwork needs in videos, at least not until a number of thorny questions are hammered out.
If you're an artist/animator looking for contract work, feel free to reach out to me with a portfolio.
Context
Several months ago I asked an artist I was working with to handle the visuals of the opening scene of this video.
This artist, Kurt Bruns, was someone I'd worked with before, adding the occasional hand-drawn digital art piece to new videos. Visuals on 3blue1brown are usually dominated by programmatically-generated mathematical animations, but I see a lot of room to broaden that stylistic scope when human stories add to a lesson, and one of my goals recently and moving forward is to do more of that.
When I sent Kurt the ask for this particular scene, I didn't have any constraints on how I wanted it illustrated, either in terms of the medium or the concept. Kurt has good taste and I knew I'd be happy with anything he felt satisfied with.
We had both talked about AI art before, having been part of the artist beta for DALL-E earlier that summer, and generally keeping an eye on the technology. Until this point any projects we had done together were hand-drawn digital artwork, but this time Kurt opted to experiment with using Midjourney to get the initial versions of the pieces we'd use.
It's worth highlighting that usage here does not look like typing one prompt, and using whatever pops out as-is. Initially, there's a lot of brainstorming, trying many ideas, and tossing what's not working well. The pieces themselves often have numerous problems that need cleaning up. Uncanny hands need to be repainted. Mangled background characters need to be fixed either to be more abstract so as to fade into the background or more detailed so as to not look weird. Most pertinently, getting multiple pieces to cover different parts of a scene in a way that looks stylistically consistent is a constant uphill battle.
This is all to say a lot of human labor and love went into the final result, but AI was most certainly a central part of the workflow.
But is it theft?
In tandem with the rise in the capability of these AI Image generation models, there's been a strong pushback from many artist communities regarding the ethics of these tools. The short version of the pushback is that artists did not consent to have their work used in training these large models, and this feels especially unjust when those tools threaten to replace many of those same artists. This is especially true when the names of those artists are used in prompts.
Some people will push back to this push back by saying it's analogous to what human artists do. You see lots of different pieces, you learn from them and take inspiration from different styles, and incorporate it into your own work.
To this, the "AI art is theft" community often replies that yeah, no, it's not at all the same as what human artists do. Processing a massive quantity of image data while gradient descending through a pile of weights that effectively compress all that data into a single model is nothing like the way human artists pass along their craft.
I'm paraphrasing, of course, but I've seen multiple back-and-forths like this repeated in online arguments.
So which is it? Are Midjourney/DALL-E/etc more analogous to human artists, learning and taking inspiration from all the freely visible (but potentially copyrighted) work that they see online? Or are they more like lossy compressions of massive image databases, whose generations are more like a souped-up copy-paste function?
To my taste, these tools feel sufficiently novel that it's unproductive to try reasoning by analogy. It's not really like either. It's a new kind of thing!
Some use cases feel like clear-cut jerk moves, like prompting a model to create something in the style of a living artist, then using that piece instead of licensing from or commissioning that artist. In a perfect world, I'd like to see such prompts result in royalties for the artist whose name is used.
In my case, I initially felt fine if an artist I hired chose to use it as a central part of his workflow, assuming it wasn't deliberately lifting any individual's particular style. Technology at its best augments it doesn't replace, and in this case, it freed an artist to rapidly iterate with many more ideas than he otherwise could have.
But should it have felt fine? In the eyes of some, these models are tainted regardless of how you use them. Even if the prompts weren't explicitly asking for the style of a particular artist, who's to say the model wasn't heavily "inspired" by a specific piece out there drawn by someone who wouldn't appreciate their style being regurgitated by a computer?
Legal concerns
The question is not a purely academic one. There are numerous pending court cases that concern the as-of-yet unanswered legal question of whether it's okay for these models to use online copyrighted work the way they have. Furthermore, what is the copyright status for images that they generate?
Maybe the outputs will be deemed public domain. Maybe there will emerge a mechanism to appropriately credit the copyright holder of the images which most heavily influenced an output. Maybe any model trained on images without the explicit consent of the copyright holders will be deemed illegitimate in a way that effectively renders their outputs unusable. We don't know yet!
If for no other reason, the ambiguity on this question alone is enough reason to hold off using the tools for new projects.
The fundamental quality problem
Even if all the ethical and legal concerns were somehow magically resolved, there's still the question of whether the outputs are actually good.
Certainly, many of these AI generations are stunning at first glance. In a shallow sense, many of them are much more beautiful and intricate than what most humans can produce.
There are some pragmatic issues making it work in videos. It's actually very time intensive to get a series of images that feel stylistically consistent, especially if there's a single character involved. That feels like a technology problem. I'm guessing there are either already tools, or there will be soon, that solve this.
There's a deeper issue, though, which is that the quality of a piece of art is not purely a function of the image. The story of where it came from matters.
For my intended use cases, where the whole goal of incorporating more artwork into videos is to add an element of humanity and character to the otherwise abstract and ruthlessly precise world of mathematics, there's something undermining if that artwork is machine-generated.
I don't know about you, but I often find myself feeling fonder towards artwork when I get a glimpse of how it was made, say by watching a timelapse of its creation, or hearing an artist describe why they made certain decisions. That fondness grows even more if I know a bit about the artist who made it.
AI artwork tends to have noticeable artifacts when you know to look for them. Artifacts in and of themselves are not bad, though. Brush strokes, lens flares, and film grain are all artifacts of specific media, but artists, photographers, and filmmakers often lean into these. Maybe they do so to appeal to a viewer's fondness for the medium as a whole. Or maybe it's to offer some small fingerprint of where the piece came from, a reminder that there is a story behind the work.
What I've noticed in myself as soon as I realize a piece is AI-generated, either by noticing the artifacts or by seeing the attribution, is that I'm of two minds.
One half of me feels the opposite of how I feel when learning more about how a human-made piece was created. Instead of fondness I'm left feeling flat. Even if the models improve to the point where no artifacts exist, the mere knowledge that something is AI-generated means there's a missing opportunity for a piece to be imbued with that bit of story for how it was crafted. Admittedly, I don't know the story behind how 99.9% of all images I see in the world were made, but somehow knowing whether or not there is a story changes things for me.
But there's another side of me, the technologist half, that delights in the fact that algorithms can create these images. It's the same part of me that delights in generative art and emergent patterns. It's the same part of me that resonates with Feynmann's tirade about how scientific knowledge about a flower only adds to its beauty.
If I had to guess, I'd speculate that as the years go on and more people do incredible things with AI tools, that second half will start to dominate. For me, at least. In the meantime, and I'm willing to guess this is true for enough other people out there that it matters, anything AI-generated artwork has an insurmountable hollowness.
Where does this leave things?
Until all these questions have clearer answers, I feel a lot better if future artwork is done by hand.
I posted multiple times last year about the Galois project. It'll still certainly happen, but it's been sitting on the backburner while I've been more focused on the usual videos. Kurt was also working with me on that one, and for a while, we pursued what it might look like to lean heavily into AI artwork, prompting Midjourney to create scenes from Galois's life in the style of contemporary artists like Delacroix.
A lot of the initial concepts look great, but due to a combination of the concerns I've mentioned here, and the fact that getting these to work coherently in a video is actually very challenging, I've swung back to a desire for all the artwork in that project to be human-made. Even setting aside the ethical and legal questions, I think this will result in a better final piece.
For the Summer of Math Exposition, a number of people have asked about our policy on using AI-generated material in entries. We do require that anyone submitting an entry has the rights to all assets used in their entry, and considering the fact that the question of who owns the copyright for AI generations is currently unresolved, we ask you to treat it as if someone else owned the copyright. That is, if it falls under fair use, go for it, but in general, it should be avoided. Also, it's just not clear to me that AI generations would actually make your piece any better.
If you've read this far, thank you for taking the time. I'd love to hear any thoughts you have!
Comments
makes sense to credit the original artist financially if there is profit being made from a generated piece that has taken influence from it. Too bad it is not possible to achieve this in a way that makes economic sense for the people who work on AI models. AI models do benefit Artists though I would say, the generated pieces can be used as inspiration.
2023-09-29 12:46:36 +0000 UTCThanks for this statement. No one can define what art means for everyone, but for me an important component is human intent. The artist is engaging in a kind of communication with the viewer. While the mere use of AI at all in the process of artistic creation doesn't by itself undermine this, the more if it there is, the less there is of the human element. Some people don't mind this, and it's not my place to tell they they're wrong. But I do mind it, and I'm heartened to see you (and others) publicly taking the position that how art is created matters.
Ben Prytherch
2023-08-16 22:35:34 +0000 UTCIn thinking about AI art and whether it’s theft, I was trying to think of an analogy and have been toying with one that I’ve started calling the “magic paintbrush” argument, and I find it helpful for pushing past the “it’s just like being inspired by normal art!” case. Imagine that it’s, say, the early 1900’s, and I’m a painter trying to make it as an artist. If I saw a few works by an up-and-coming artist named Picasso, his work getting cubism started in the Western canon could inspire me, and now I want to contribute to that movement. In order to make my paintings and contribute to the movement, I have to do various things like master perspective enough to confound it, maybe become proficient with turpentine painting if i’m used to linseed oil, basically i need to develop the brush techniques necessary to make a cubist oil painting. Let’s say I’m successful, and I paint what could be considered a cubist masterpiece, redolent of Picasso’s work but probably with my own flair because the work is related to my life experiences. Now, let’s go back to the time when I had just seen Picasso’s work, as soon as I’m taken by his style. But instead, in this moment a genie or fairy appears and bestows upon me a paintbrush with a special property—if the paintbrush has previously been brought near the works of any artist or multiple artists, it lets the user paint whatever they like in the style of any of the works it’s been shown. If I were to use that paintbrush and I paint what could be considered a cubist masterpiece, redolent of Picasso’s work but probably with my own flair because the work is related to my life experiences, how should this artwork be treated compared to the other universe where i developed the techniques? What if this happens instead later during the surrealist movement, but I don’t use Picassos and instead I use a collection of paintings from Rene Magritte, Leonora Carrington, Salvador Dali, Max Ernst, Meret Oppenheim, etc etc and produce a painting that could be considered a mix of their styles? What if it happens even later around the time of the pop art movement, where readymades, replications, and repetitions proliferate, but art is quickly becoming a commercial enterprise with design and animation etc etc becoming huge careers? I’ve found this helpful in recontextualizing AI art as a tool, although I am very on the fence about AI art. I make AI art all the time, but I’m a collage artist and like to include it in physical or digital collage so I usually transform it like I would any part of a collage piece. I also try not to let my mathematical and scientific background cloud my judgement at the excitement over new technologies, even if it’s a cool frontier.
Jarrod Sage
2023-06-03 04:41:09 +0000 UTC"Admittedly, I don't know the story behind how 99.9% of all images I see in the world were made, but somehow knowing whether or not there is a story changes things for me." This really resonated with me. A piece of art is so much more than just the pixels that make up its visual component. It has history, provenance, and a person or story behind how it was made, even if I'm not aware of the exact details (which is in itself sometimes interesting). To me, AI art has such a watered-down and shallow story that it ceases to be interesting and becomes just pixels on the screen. Even generative and algorithmic art still has clearly human decisions behind the design and refinement of the algorithm that leave their mark on it and make it beautiful, but generative AIs are so generalized that it feels to be in a different category. Not to say there isn't a human process to creating AI models, but it's so far removed from the end-product. Very much agree with your take on all this.
Daniel Beckwith
2023-06-02 21:14:29 +0000 UTCI have a different aesthetical experience with AI drawings - I think AI will become very powerful relatively soon, but I also think it will kill us before we get to the Dyson-sphere-like stuff. So the somewhat alien AI images make me think about the alien future that lies ahead and that I don't expect us to see. Using a different Feynman metaphor: I find this a bigger story on a bigger stage than the stories behind human art.
2023-06-02 10:14:34 +0000 UTCWhile I have not yet developed a strong opinion on AI art, I have to say that perverting Feynman's rant in service of a technology that *we don't understand* is a little gross. The beauty of scientific knowledge is a deeper understanding of the world, not an obviation of the need for such knowledge. The equivalent in art *is* the story that you may or may not know, of the cultural and historical context of the artist, of the style and technique choices, of the subject matter, etc. That's all there for you to learn if you want to dig in and find out more, just as the deeper processes of biology and chemistry and physics are there to learn if you want that extra layer of awe. That paragraph is the opposite sentiment of Feynman. And that's fine. Feynman wasn't always right, and perhaps he is not even *relevant* here. I don't think he is. And I think it's offensive for you to try to defend your position as if he'd automatically share it.
Michael Chui
2023-06-02 04:48:03 +0000 UTCI've been thinking about if I posted it to social media. I'm waiting until this is worked out enough for me to be okay. This cleared up some I thought about. And added some. I think I'm going check that I have posted some with giving the AI credit. As usual I will be late until joining in until I get clear legality and copyright. Thank you for your explanation of everything.
Bryan Williams
2023-06-01 21:57:44 +0000 UTCThanks for the thoughtful reply. I appreciate how you bring up inpainting. In general, it feels like the most significant implications for modern AI will be how it integrates into existing workflows, more so than how it replaces someone's job entirely.
3blue1brown
2023-06-01 18:00:37 +0000 UTCGrant, I'd like to thank you for taking a more even-handed approach to this problem than what I've seen elsewhere on the internet. I've observed a lot of different opinions about AI art, and they seem to break down along two main fault lines: 1) Whether AI art will ever be as good as (or better than) human art, and 2) whether that would be a good thing or a bad thing, if it happened. But (1) is more complicated than it sounds, because "good art" is subjective. We could easily end up with a model that produces *technically* great art, but has no understanding of composition, symbolism, continuity, etc., and requires supervision by a "real" (human) artist. And (2) is also difficult to answer, because one can argue that AI is merely accelerating trends that have been going on in the creative industries for decades (i.e. management squeezing all the money out of everything at the expense of artistic vision). Technology has long been regarded as value-neutral - if we don't want management to prioritize dollars over creativity, then we don't have a technology problem, we have a people problem (or maybe an antitrust problem). Sure, AI is may end up making the problem worse, but banning AI would just be treating symptoms, or so this argument goes. On the other hand, there are also plenty of people arguing that AI is somehow exceptional or different from other technologies, and should not be regarded as inherently neutral. Its tendency to incorporate biases from its training data is enough to give me pause, at least. There are many, many people who have little or no artistic ability. Some of them are physically or mentally disabled, and will never have much artistic ability. Some of them work long hours for little pay, and don't have time to learn to draw. And some of them, well, just don't want to put in the effort. This position is easy to characterize as "techbros calling artists privileged," but appealing to labels is unhelpful and primarily serves to inflame passions rather than clarify the situation. The point is that artists can do something that much of the world cannot. Some people, who (mostly) cannot do that thing, view this deficiency as a problem that ought to be solved. Some people, who (mostly) can do that thing, view things very differently. The problem that artists have here is that, in the long run, they are probably going to be outnumbered, because most people can't draw very well, and so most people are more likely to fall into the first camp than the second, if they develop an opinion at all. Regardless of which group is "right" in an abstract, Platonic sense, the position that benefits the smaller group is ultimately going to become less popular over time. That's not even getting into the professional artists, at least some of whom will regard AI more as a timesaving device and less as a threat to their jobs. Stable Diffusion offers inpainting, which can be used to make all sorts of changes to an existing image, from subtle artifact removal all the way to wholesale person or object insertion or removal. Sure, you can do that with Photoshop, but that takes more time, effort, and skill compared to doing it with inpainting, assuming you want a reasonably high-quality result. AI art is nearly always discussed in the context of txt2img generation, but inpainting is arguably the more important application, at least in the short term. For the time being, the ethics and law remain murky, so I support your decision to (effectively) ban AI from SoME3. But you may want to reevaluate that position when we get to SoME4 or 5. Things are moving far more rapidly than I think anyone would have anticipated a year or two ago.
Kevin
2023-06-01 07:39:46 +0000 UTC
