User Research in Building AI Products w Autodesk’s Tonya Custis

This is Cross Validated, a podcast where we speak to practitioners and builders who are making AI deployments a reality.

Today our guest is Tonya Custis, who is the Director of AI Research at Autodesk.  Tonya has over a decade of applied AI research experience at Autodesk, Thomson Reuters, eBay, and Honeywell. 

Listen and subscribe on Spotify and Apple.

Follow me on Twitter (@paulinebhyang)!

Transcription of our conversation:

Pauline: Welcome to Cross Validated, a podcast with practitioners and builders who are making AI in the enterprise a reality. I'm your host, Pauline Yang. I'm a partner at Altimeter Capital, a lifecycle technology investment firm based in Silicon Valley. Today, our guest is Tonya Custis, who is the director of AI research at Autodesk.

She leads AI research teams working on both foundational and applied research from science to product and has spent over 15 years in applied AI research at Autodesk. Thomson Reuters, eBay and Honeywell. Tonya, thank you so much for joining us today.

Tonya: Thanks for having me.

Pauline: I'm very excited to kick off and we can start with, introductions quick background on Autodesk, what the company's mission and then a little bit about your role.

Tonya: So at Autodesk, we make software for people who make things. So if you've ever ridden in a car or seen a skyscraper, used a smartphone, played a video game or even watched a movie, chances are you have experienced what millions of our customers are making with our software. So, some of the examples of our software are AutoCAD, like in architecture and engineering, construction, Inventor in design and manufacturing and Maya in media and entertainment.

My job is AI research director. I have a team of PhD AI research scientists and we collaborate with other industry labs and academic labs and product teams and we publish papers at top tier AI conferences and we work with the rest of the business to get what we've been working on into Autodesk products.

Pauline: Appreciate that background and on the company as well as yourself, we'd love to start with talking about what role does AI & ML play at auto desk. And can you talk about some of the needle moving use cases that you've seen.

Tonya: I think ML is just like now, it's a part of the modern software engineering toolbox.

So hopefully at modern software company is everywhere and it's really a table stakes. An undergrad software engineering student comes out knowing more about AI and ML than I learned, as like a grad student. So we use AI machine learning throughout the company, both like operation side and also in our products.

A lot of machine learning manifests itself in small ways that you might not even notice, like command recommendations or similarity search things that seem kind of boring, but are behind the scenes, or not boring, but you take for granted. And, I even argue that the best AI machine learning is the stuff you don't notice because it feels intuitive. It feels, like something that you would have done anyway, so we have a lot of machine learning in our products. Does things around segmentation and boundary detection for 2D and 3D geometry, floor plans or video game assets, for example.

So those are maybe things you don't notice so much, but I do think some of the most Needle moving use cases are the things with the biggest impact and I think those center around things like sustainability or the environment, like things that are making a difference and two of those that I can think, off the top of my head, Innovyze is a product that we have to model wastewater treatment plants to make them more compliant, more efficient and more effective. And we've also use AI in efforts around coral reef restoration and so that's uses robots and AI its super cool.

So I think those kinds of things make the biggest impact. But, I think we're kind of past the point where like an AI feature or an AI product is really even that exciting anymore because it's just how software works.

Pauline: No, it's funny because I think it was one of the Canva presentations that we saw this week, where their framing was no one wakes up wanting to use AI.

You want AI to empower the software and I love these sustainability products that you mentioned. I wouldn't have expected Autodesk to have those products. But certainly I think the product that comes to mind when I think of Autodesk is AutoCAD. And so we'd love to dive into that and talk a little bit about how does that software serve up AIML and ways that people who use it every day don't appreciate because AI is just running in the background versus is in your face. Hey, you're using AI.

Tonya: I mean, there's a line in there too like you want the customer to know if something's AI for ethical reasons, But there are a lot of little things like, recommending the next command, AutoCAD, I'm sure this number is not right, but it's a lot of commands, like 14,000 commands. It's really a lot. People go to grad school to learn how to use AutoCAD,

So, it's not the easiest to learn how to use. So one of the things that we do with AI is we, help users figure out like what, these sort of like Netflix use these commands. We recommend use these and that is done in a number of ways and sometimes it's just sort of like in user streams. Sometimes it's sort of like Hey, we've seen you never use this command Would you like to learn about it, So those are, some ways and I think, just piggybacking on what you said before, I think I like to say is how it's not a what and so it doesn't matter if you're using AI or not.

If it's not the best tool for that job and I think, we get so caught up in this, is it a feature is it this or that, There's an interesting line there, you want to be transparent with customers that something is AI, especially if it's offering suggestions or even making decisions for you. I think AutoCAD is fairly transparent about

Pauline: One of the areas that I've been thinking a lot about as to the progression of these AI technologies is how do you go from insights? So, in example AutoCAD, what command should I do next to actions? Doing it for you automatically and still with the human in the loop, you can say, yes absolutely, but certainly simplifying the workflow to say, maybe instead of 10 clicks or 10 commands, the human only executes a portion of that talk a little bit about that. What is the vision for AutoCAD? Just because that is such a well known product with an auto desk product.

Tonya: I can't speak to auto desk product. Roadmap or what exactly they're planning to do so in my team we kind of just view all like CAD software as a monolith to be honest. So like things that we look at and research are things that would apply to any of our products really. Overall, keeping the user in the loop is so important, especially for designers. It doesn't matter how you do something as long as it's the best way. So if I'm giving designers suggestions for things, they're almost never going to really like it because they're the designer, they like how they do.

Even if I had the best other designer behind a curtain and told them that like AI did it or something. They still wouldn't like it or no matter, so there is this idea where you have to let. The designer, the creative person is the driver and I think, they don't want a button that just is like complete my design and here you have a bridge or something. I think that's the scenario we talk about a lot or that people ask me about a lot. They're like, isn't that dangerous or what about this or what about that? I mean, that would be not great to have it, design a whole bridge and you wouldn't want to and the designer wouldn't want to.

So, I think exactly to your point, what is the granularity of where you're saving somebody time. And you're providing them something useful. So you could, and we have, and I've seen this at other jobs also where you automate this task. Let's say, it's maybe making some suggestions, but like it just takes a long to do it that by the time you get it back to the user, they're already six steps ahead of you because they're really fast and they're really good at their job or you give them a suggestion and then they have to just redo so much of it. Because they're like, I don't like this, I don't like that, so then, it doesn't save them time, So really it's about finding what the sweet spot is, where you're saving them time. Again, you're giving them either suggestions or whatever that intuitive to them, so they don't notice, oh, I would've done that myself. This is how, so I think its finding that sweet spot and it's a little bit different for every creative person. So that's fun.

And it's a little bit different for each workflow and I think one of the things even for our customers who are all using AutoCAD, let's say they all have pretty different workflows. So, it's really about how can we learn about those workflows and how can we learn what the right suggestions are written.  

Do you want to autocomplete this? Or would you be absolutely super irritated if we autocomplete this for you? I think generative AI gives us the possibilities more and more of, generating more design stuff for our customers. But, has been in the generative design, game for a long time.

We've had generative design products for quite a while and we've learned a lot from that too. For example, they don't like it when you just generate the whole thing for them. It isn't interesting to them and, making it an interactive.

They have a colleague sitting next to them that they can, bump ideas off of and see what makes sense or here are three possibilities. Like this one's good for this reason. This one's for that reason. Could we combine them?

If the computer can combine them for you, that would save you a lot of, and then, low level tasks like, doing layouts for office buildings takes a really long time and like, why? There's really no good reason. Is it a glamorous job that people like doing? No it is not. Finding those tasks that are things that take up a lot of time, but give relatively little value or maybe it's not value, maybe it's have little opportunity for creative expression, or being creative pushing that envelope. We want to make sure that we're giving people who are creative the most help. That they want and giving them the most room, staying out of the way, when they are doing what they do best.

Pauline: There's so many different vectors that I can ask because that was such a nuanced answer. Let me start by saying for the listeners who don't know at AutoCAD very well, can you give a few examples of like, what are the manual labor tasks that you think are low enough level and sort of can say, hey, this makes sense. There's a little bit more consensus across the designers that this should be automated.

Tonya: It totally depends on the workflow. One thing is like office, like putting all the cubes and an office furniture in an office layout, That's super tedious. One example from manufacturing is, if you have a part that you're building or an assembly you're building and there are 20 holes, all of the same size. You probably want to put the same bolt or whatever in all of them, but, often you have to manually place them all. So there's a lot of tasks that we would look at is analogs to copying pasting in text and in Word document and I think there's a lot of tasks like that.

It's like those tasks that really are sort of like copy and paste or you're even, like industrial designers or architects, a lot of what they do is the same from job to job, or they don't start from scratch. Sometimes they start from an old design and so, a lot of that can probably be made easier. By just understanding the intent of hey, I want to put 20 bolts in, not just one, or I want each wall in this room to be finished this way or, really sort of things almost that seem like common sense. To a lot of us, but how does the computer know? It doesn't know. It has no way to know.

Pauline: Super helpful to go through those examples. I want to come back to what you said about everyone's workflow is different. Everyone's maybe rate of wanting to adopt this is different and I've always had this framework that humans are going to be the biggest bottleneck to the proliferation of AI and so what does that mean for the user research perspective or workflow on your end as well as does it then make sense that may be the product of AI is deployed differently for different sets of customers and are you fine tuning for that Can you talk a little bit about that?

Tonya: I mean, one of if you look at like, search, which is the big first AI use case, the big first machine learning use case. It didn't take people too long to get to personalization and so I think personalization is a huge area of AI that, we talk about or we expect a lot and so, learning to personalize to different use cases or different people. We talked a lot about design intent. In my group, like we want to anticipate what the user is doing, if they, draw a cylinder, are they drawing an engine with like pistons and cylinders or are they drawing a water tower? So understanding your customer's sort of intention and how they're using the tool, I think is so important. And again, it's about giving the user the control, may be to invoke an AI feature if they want help, if they want us to generate six alternate designs of this, like they're asking for they would like it. Again, if it's, you don't want to just do stuff they don't want.

And offering them things that are intuitive and again save time. So I think, when we do, user testing, that's sort of a lot of what we're looking for like, does this save you time in this good task that you even care about like, so a lot of times. I am not a visual designer. So maybe we're creating algorithms to do things that like they don't even care to get help with. So it’s really important that you're making sure your domain experts and your AI experts are working together because, I could have the very smartest group of AI research scientists, which I do, and we know nothing about building a building or, designing manufacturing parts or, assemblies. And making sure again that you're what? It's correct that the problem you're trying to solve is the right problem and then figuring out how to solve it is AI the best way to solve it.

 If so, then yes, my team should help you. If not, if there is a better way, like if Physics or some other mathematical process is 100 percent going to work what you would not waste time using AI. But I think, it is about meeting the customer where they are, because, some people do not want all the AI and maybe aren't feeling good about it or don't want us to use their data or there's all sorts of things there and so, having giving the customer that agency, I think is a big deal the other thing is with user testing a lot of times what we find is that as soon as they see what it can do and that it does save them time or then they are really excited about it. But I think all AI has this creepy line. A few years ago, like Google is putting your airline flight in your calendar from your email. You're like, wow, hold on Google that is very uncomfortable. But now if they don't do that, I'm like, why isn't it in my calendar?

So that line, it really shifts and so the other thing when we do user testing, that's really important, is to sort of acknowledge those shifts. I think it's really easy, for companies and I, this has happened at not just Autodesk, other companies I've worked at where people would be like, but the customer thinks why, and that's like a, I don't know, an urban legend that you hear over and over you have to take into account that customers change and a lot of customers. Now have grown up using computers like their whole life and some things they're not going to an eye at that. Whereas like, say a, more elderly customer or more traditional customer is they won't like it as well or but maybe they will.

So that's the other thing like, there's a lot of this you really have to not only rely on user testing, but be flexible because that changes over time, even with the same user, what they're comfortable with, or what they like, even their workflows change. So, meeting them where they are is so important.

Pauline: It's really interesting that you bring up the line, the moving line, because certainly I think if you, November 2022 was such an important date in AI, given the release of ChatGPT and so as you mentioned Autodesk has been using generative AI in their products for a while. Certainly, I think generative AI has come into mainstream much more broadly in the last year.

And so I'm curious to hear what have been the mentality shifts that you're seeing this line shifting now that way more people are aware of this technology can try it in other products outside of what they're using for work and maybe expect more out of the tools for exactly that Google analogy for that exact reason.

Tonya: So, first of all, I want to say Autodesk has had generative design capabilities for years. So let's think about that is what a generating design. Generative AI gives us a new powerful tool as a how to also generate design. So in the past, important clarification. I appreciate that, so in the past we've used topological optimization and evolutionary algorithms, which arguably are AI. There's a lot of nuance and an argument to be had there by anybody who wants to have it, but it's how are you generating designs?

And so that's one way to do it and so historically we had the software that has been in this space for a long time because we did think, people want help generating these designs. Now, Generative AI gives us new ways to look at doing that and so, the tools that we had in the past were a little bit difficult to use.

You have to put in a ton of parameters, a ton of constraints, and then it's going to be like generate thousands of examples for you. With Dolly 2 or 3, Dolly 3, I like, I want a unicorn riding a bicycle on the moon like you just get it, so there's a lot of potential there of taking the technologies that we had before that we're doing this task of generating design with the new technology that we have in order to make it easier, make the interface different.

I mean I think a lot of what has been successful as generative AI has been, is that the interface is so easy. Just natural language is a very easy interface for people, but it's difficult for high precision tasks, if I wanted to generate a water bottle to do that very precisely in language that the computer would understand, would be not only difficult, but really boring and probably take a long time.

So different interfaces are good for different things. My team works a lot and has published quite a few papers on Generative AI in 3D CAD space. So focused on 3D representations that are used in CAD. So that's different than generating text.

It's different than generating images like in a 2D pixel space. There are a lot of   really cool opportunities there. But we're also, looking at different interfaces, like I said, language is very tedious for some tasks and my background is linguistics. So I'm not dissing language at all like that is what my actual PhD is in. I think, looking at things like, what about sketching to 3D or, I think for a lot of designers, that is a more efficient way to convey information. Then, I'm building a bench and it's going to be six feet long and four feet high and have like this many bar, I don't know, wood things in the back and, I don't even know the words, I'm sure there are words, so I mean those are really cool opportunities. I think, what Autodesk has done in the past with generative design has really helped us learn, what do customers want? They don't just want a button that you push and it gives you the answer,

Because the answer is never going to be the answer that a designer or creative person wants. They're going to want it their way, but how do we help them do it their way and get that opportunity for semantic input whether it be like... Language or sketch or whatever, and help them make it faster, help them get to what they have in their head into the design tool faster so they can make it faster.

And I think, language is really useful for things like, you already have a design and, you want to say, make it rounder or make it poofier, like things that are really descriptive to you. But it's often the degree of something is, it's a really useful sort of language thing in design, but it's not that first stage and a lot of also what we've seen, especially in architecture is people using mid journey and similar tools to make like super cool concepts, so that conceptual stage is really interesting, because if you're an architect and you're working with a client, you probably go through a lot of iterations of, is this what you want? And they'll say, no, that's awful or whatever. And so that's also a task that takes a lot of time traditionally.

Like an architect has to like make all these drawings and do all these things. They spent three days on this drawing and they bring it to a client and the client is like, that is not at all. Saving time in the different stages of the design process there’s a lot of different cool stuff that we're going to be able to do. We're starting to be able to do that. We haven't been able to do before.

Pauline: We talk about a few examples of what that looks like of what has been unlocked.

Tonya: Yes, I think that conceptual iteration for sure has already been unlocked. I think when we look at ways that Generative AI is used in the wild like various use cases in painting or style transfer, so we can think of, interpolation. I like this chair, but I want one that takes the back from this one and the seat from this one.

So, Generative AI is really good at translating things. So think of machine translation, but also think of it as text to whatever so those use cases are been unlocked to a big degree again like 2D to 3D sketch to 3D model, one 3D kind of model to another, like point clouds to meshes, we could think of things like even 3D to 2D, Or a picture of a picture of a building, Like the mid journey case to a 3D model of that building or to blueprints, like translating is a really good Generative AI use case. If you can, think through the implications of translating from one thing to another, you can probably turn it into a Generative AI use case.

Also like interpolation, like I said, so in text, we would think of that as like summarization. I like these two articles make me one. So you know, those kinds of tasks again, and it's ChatGPT is so useful because you can use almost always use an example of something that, large language models do, but you can always sort of think of it as an analog where we might apply it in different ways in like a 2D or 3D domain.

Other things that, large foundation models are good at taking a photo of something and translating into text, so you can think of use cases like, code violations, for construction or dangerous things, here's a picture. What's dangerous? Like the slatters, next to the wall like that might be dangerous for little kids or pool or somethings. So thinking of things like that, which again is, a translation task, but being able to frame all sorts of tasks that we do as humans, and a lot of them we do without even thinking about them or like again style transfer the great one like you show a customer a drawing and they're like well, I mean, sure, but I want it to be more like art deco just those kinds of things are becoming easier and easier with Generative AI. Not all of them are solved for sure.

I think, it's easy to see how we can solve them now, whereas I think in the past, they were just like magic, AI can do that and then the answer is always like, because that's magic. But, we're figuring it out now. And it's really exciting.

Pauline: I really appreciate all those different use cases particularly because, I can't say I personally am in AutoCAD every day, but all of those use cases you mentioned seem very much like that would benefit from Generative AI. So I appreciate you going through that. Last question before we move on to rapid fire is this debate of buy versus build and I think this has been a big hot topic for enterprises, particularly of the size and scale of Autodesk and so we'd love to start with just your framework of what should Autodesk be buying versus building in house?

Tonya: In my opinion, that normally should rest on the data. The AI research community is very open. There is no secret math and most of the time, if you look at something someone made, you can pretty much tell how they did it, AI is the how. What's interesting with, AI companies, AI startups, AI features, is more the what, so is the company addressing a problem that you don't solve like that seems like a good reason to acquire them. Do they have a market share that you don't have or clients you don't have but, are you solving a problem that you're not solving for your current customers like all of those are good? But a lot of times it's not because you couldn't build it in house. But, it's the data. So a lot of times, it's sort of some have a lot of data.

And so, a lot of early products, from startups they're a harness to collect the data. They have the idea of what's the workflow, what's the task, so they have the down and they're going to iterate on the how? The first version is maybe going to be rule based and really awful, or, maybe they're calling some guy in like Indiana to do it behind the scenes. Like maybe some kind of wizard of Oz situation. And then they get the data and eventually they can train a model and then they get the data and eventually they can train a model. And so, the best reason to buy a company. Would be because they have the data, it really has nothing to do with the technology the AI technology, there's no secret math, we all know how to do it. It's that we don't have the data to do it with most of the time and so to me, those would be the best reasons to buying, hiring AI talent right now is also difficult.

So like acquire kinds of situations, like we do want to build it but we literally do not have enough people. How do we get the people quickly who have that skill set? That's a great reason to acquire. Although, I'm not a business person but, I do think that a lot of times once you identify that, company has the talent that you would like to acquire, they also have joined attention themselves and they were very very expensive.

So there's a lot of, nuance and timing and stuff there but I think, the worst reason to buy an AI startup is for their tech is like a hundred percent of the worst reason.

Pauline: That I will say, I think that's a hot take.

Tonya: I totally stand behind it.

Pauline: How do you? Maybe outside of the acquisition? Framework and more of there's an external software vendor and would you buy software from them? What's your framework on in that perspective?

Tonya: That's a great question. I think it definitely depends but, there's no reason for you to make your own CRM system.

Like, some problems are just solved, there's established enterprise companies that are doing that, a lot of times it's easy especially an enterprise level to think about buying and that is also very true. Is like, there's no reason for any company really to build their own giant language model at this point should you may be fine tune it for your domain?

Like Autodesk has several like specialized domains. We probably want a media and entertainment one or an architecture one or something, but that is not a buy situation and I think anything that is sufficiently commoditized, and it's sufficiently mature. Generally, it makes more sense to buy, some companies don't they make their own. It's generally like, what do you have the special sauce or what could only you do and a lot of times that's what data does only your company have or do you have the people to do, but to do AI well you really need to understand the domain and the science, and have the data and have the computer and if you have all four of those things, then you probably should build it, but when you're missing any of those things you need to think about how you might get it, whether that's buying data using a third party tool for a well understood problem but that only you have the data for, like fine tuning the language model on a specific thing. But it really depends, but research is expensive.

Pauline: I think that's a fair assumption.

Tonya: I guess I don't really know, you want to point those researchers to the things that are going to give you the most competitive advantage.

Pauline: I love that framework and certainly very much aligned on how we think and much more clear cut. So I, appreciate that.

Tonya: I love how you call it a framework instead of just like me ranting random thoughts so.

Pauline: I love that and with that let's move on to rapid fire. First question.

What is your definition of AGI? And when do you think we'll get it?

Tonya: I have a no good answer for this. When do you think people will reach general intelligence? I'm not super worried I've been working on in this field for almost 20 years. Its super exciting, I love my job there's so much cool stuff happening, but at the end of the day, computers are really quite stupid and it's even harder to teach them general things than it is to teach them specific things. Will we get to the point where we’ve taught the computer to do a lot of very specific things and it makes it seem like it's doing a really good job at a high level task.

Sure. But those are going to still be very specific tasks, I guess I don't have an answer but I'm not worried. I don't think it's soon.

Pauline: Makes sense. Second rapid fire question. What is your AI regulatory mental framework?

Tonya: So, no tech is neutral. There is little evidence to me that AI requires a special regulatory framework that's a different than other technologies. We need to make sure that privacy, security and regulatory frameworks are in place the data we're using to train models, the data we're collecting from users, the data that even they are putting into our models like we need to make sure all of that is used responsibly is through like we need to make sure the data that's the data that's being exchanged between the user and the system is secure. We need to think about what the interface look like there's a big difference between you could use the same algorithm and have three different user interfaces, and one of them could be horribly unethical and dangerous and one could be very well thought out and have guardrails in place and be, used in such a way that it does promote safety so, we need to think of it as a system, for sure.

But, I do not believe however, that there really needs to be a separate regulatory system just for AI that might also be a hot take.

Pauline:  No, I think that makes sense. I think the devil will be in the details and certainly, I think that will be a very big topic that I think will also probably swing from two extreme on one side or another over time that's my prediction.

Tonya: I think put simply like you can regulate data and its use and make sure it's used well and privately and securely. You can't really regulate like how we use math.

Pauline: That’s right. I like that comparison. Third rapid fire question. What is the biggest challenge facing AI practitioners or researchers today?

Tonya: I just think keeping up. It's my job and I still can't even read all the papers I probably should be reading.

Pauline: It does feel like the volume has really exploded as again, this technology has entered into the public mainstream and so certainly, the volume is incredibly difficult.

Second to last rapid fire question. What is one thing that you believe strongly about the world of AI today that you think most people would disagree with you on?

Tonya: I think the idea that AI is unsafe in some way, or dangerous, is math unsafe? No, what people do with that math might be unsafe. What's unsafe are the people who aren't keeping your data safe is, what's unsafe is the biased data that some algorithms are trained on and how they're used. What's unsafe is using AI for bad use cases, or if you're using it for gatekeeping, keep people out of neighborhoods or something again, no tech is neutral, but also like tech doesn’t use it. The AI doesn't have any motives or motivation and so, a lot of times when people are afraid that like AI is going to do X or Y, it's saying more about the person saying it. Like what would they do if they were all powerful or omniscient or whatever they imagine AI is, it's more of a psychological thing, AI is not unsafe, but people there's always going to be bad actors. Unfortunately,

Pauline: Last rapid fire question. What is one of the biggest gaps that you see in building or deploying AI that you'd like to see either Autodesk or another company build?

Tonya: I don't think there's just one answer to that question, but I think the biggest gap. It's just data. We don't have, so often times when product managers are like, we want to use AI or we want to do this new thing and they're like, an AI can do it well, I mean, a lot of times they can't, but, a lot of times it's something that, people are bad at or the current product and also doesn't do well, like, those are the problems you want to solve and so because people are bad at it and because the current product doesn't also do it well, there is no data available that models that behavior or exists as examples to train from especially at scale. I think, the biggest gap is just data and thinking about creative ways. Like how do you get that data? How do you produce it? How do you collect it in an empty workflow that eventually you can, get people to generate a lot of it, like we're seeing that more and more and so I don't think there's one solution to it but, I do think that's the biggest gap.

Pauline: We have a saying in here that any AI problem is a data problem. So that's spot on with how we're thinking about it and with that, Tonya, thank you so much for joining us today.

It was a wonderful conversation and really appreciates you taking the time and coming on to speak with us.

Tonya: Thank you so much. It was super fun.