{
  "version": 3,
  "sources": ["ssg:https://framerusercontent.com/modules/mDkIT313K0Wt5oSoVDyx/eEJotbW2hN2CexkY5ppa/X7VlByTfx-15.js"],
  "sourcesContent": ["import{jsx as e,jsxs as t}from\"react/jsx-runtime\";import{Link as o}from\"framer\";import{motion as a}from\"framer-motion\";import*as n from\"react\";export const richText=/*#__PURE__*/t(n.Fragment,{children:[/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Patrick Debois:\"}),\"\\xa0[00:00:00] I think an LLM combined with tools and combined with agents, they give you massively better results. So I made a little thing that watches me on the camera. And when I did this, instead of typing, it was listening to me in voice mode. And so now I can switch between two modes and I'm just sitting on the sofa.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Simon Maple:\"}),\"\\xa0You're listening to the AI Native Dev brought to you by Tessl.\"]}),/*#__PURE__*/e(\"p\",{children:\"On today's episode, joining us once again, is Mr. Patrick Debois, the father of DevOps, but this time around, we're not going to be talking about DevOps, Patrick, it's going to be a nice, refreshing change for you, right? We're going to be talking about the journey that you had using coding assistants, learning to develop [00:01:00] using AI as part of your workflow.\"}),/*#__PURE__*/e(\"p\",{children:\"I've been watching from afar, sometimes quite close, seeing how you've been playing around with various, these editors tooling systems. It's been really fun to watch the passion that you have in, AI in general, but also how it can actually help you build software. So let's jump straight in.\"}),/*#__PURE__*/e(\"p\",{children:\"Patrick, father of DevOps and many other accolades. But today coder Patrick. Tell us a little bit about your background, first of all, briefly in the, why don't we do the coding space? Cause I think everyone knows you from the DevOps space. Coding, Patrick, what does that look like?\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Patrick Debois:\"}),\"\\xa0There's definitely a link between DevOps doing a lot of automation but particularly excited about how we can now automate more of our code generation.\"]}),/*#__PURE__*/e(\"p\",{children:\"And I see it also a way to standardize the way we work, make it more consistent. make it more secure. There's a loads of opportunities that machines are tireless helping us to do this. Where in the past, part of my job was VP engineering, scaling this out to all the engineers, the best practices.\"}),/*#__PURE__*/e(\"p\",{children:\"Now I have [00:02:00] AI that can help developers to do this all day and kind of get better for the good cause. So yeah. Yeah.\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Simon Maple:\"}),\"\\xa0Awesome. So let's go back to the early stages of the first time trying out AI as part of your coding style what did you use? Was it copilot?\"]}),/*#__PURE__*/e(\"p\",{children:\"Was it earlier than copilot?\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Patrick Debois:\"}),\"\\xa0I think at that time, ChatGPT came out. The first plugins were actually more like ChatGPT pilot so there were like VS codes. I know there was GitHub Copilot, but at the time I was maybe not doing too much of the coding, but there was a difference between doing it in line in your editor by commenting and making sure it was generating or auto completion thing.\"]}),/*#__PURE__*/e(\"p\",{children:\"And the next thing that came about a little bit with ChatGPT was the fact that we could ask it questions and to do things. So that kind of chat pane on the side helped to do things. We've come a long way. Initially it was like, okay, here's a block of code and it rarely ran. Now it's a lot more refined on immediately getting the code into your [00:03:00] editor and doing there.\"}),/*#__PURE__*/e(\"p\",{children:\"But that kind of was the switch, the inline editing was not always working too well at that time. But I'm talking like, early, a couple of years ago.\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Simon Maple:\"}),\"\\xa0Inline editing being changing existing code or?\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Patrick Debois:\"}),\"\\xa0No, more like you write a comment and then it auto completes to the comments. And then the auto completion was a little bit better, but we called a completion before.\"]}),/*#__PURE__*/e(\"p\",{children:\"So that was helpful, but not good enough at that time. Now it's tremendously better, but that was for me. It's nice, but it sometimes gets in my way. And then the chat came. And that was nice, but it also felt like, why am I copy and pasting this all the time? You felt like going out to stack overflow, coming back, it was not exactly what you needed, but it was a new way, a second way that we were Interacting with LLMs or AI models to generate code that we would do and generate in our code basis as well.\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Simon Maple:\"}),\"\\xa0It strikes me here and thinking back to a blog post, actually that Guypo wrote, which we [00:04:00] mentioned on the monthly a few weeks back. He wrote about this matrix of change of trust. And it's really interesting to see here. And we see this with a number of tools whereby various tools, let's take ChatGPT.\"]}),/*#__PURE__*/e(\"p\",{children:\"In fact, let's take something like Claude. With artifacts is a great example you can use it as an interface to ask questions to write to make changes to code and that's fine so long as you don't want to change code and the change piece there. The change in workflow is massive. Because we're no longer an IDE.\"}),/*#__PURE__*/e(\"p\",{children:\"We're not writing code. So the way in which we're interacting is a very big change. Then there are other tools like Cursor then allow for that conversational aspect was it, what did I call it in Cursor? It's compose composer. That's the composer.\"}),/*#__PURE__*/e(\"p\",{children:\"And it allows you to say I want to change this, or I want that, multi file change, whatever it is. But it pushes it straight back into the IDE. From the change perspective while it's considered an advancement because it reduces the copy pasting that we need to do.\"}),/*#__PURE__*/e(\"p\",{children:\"It [00:05:00] pushes us back into more the existing workflow rather than potentially what could be in the later down the road. Do you think we see adoption for various tools, maybe like more like the Cursors and things like that, because they're just far closer to where we are comfortable working and is that going to almost like trump various other tools like a, like whether it's from Anthropic or ChatGPT and things like that.\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Patrick Debois:\"}),\"\\xa0Interestingly enough, the chat interface, when the ChatGPT plugins came, they already had things like, oh, can you refactor this code? Can you explain me this code? All those kinds of functionality came extra, but they fell on the side. And as you say, now we can do this like with our code base and it's writing these things in the code base.\"]}),/*#__PURE__*/e(\"p\",{children:\"I think, whether you using Codium or Cody or like a lot of people have looked at it like this way of doing this together, I think what Cursor cracked was UX problem of us expressing what we wanted, the intent we [00:06:00] wanted, changing that code. That's something, we could have done with copy and paste, but they streamline the process, they optimize the process.\"}),/*#__PURE__*/e(\"p\",{children:\"And then the next part they did is when we're reviewing all the edited code, a brilliant thing they did is they're not just showing us the diff. They're not just showing us the inline thing. They're showing us a condensed version of what is changing. So they're helping us to validate the code that has been changing.\"}),/*#__PURE__*/e(\"p\",{children:\"And I think that's the biggest nice aspects that A it brought us into the flow Hey, we're writing and we're expressing, is it good? And then we were reviewing. So all these kinds of things got polished in the workflow.\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Simon Maple:\"}),\"\\xa0And I saw that and I think it was a LinkedIn tweet that, sorry, A LinkedIn tweet.\"]}),/*#__PURE__*/e(\"p\",{children:\"Yeah. LinkedIn post. A LinkedIn post. A LinkedIn face, Insta Post that you did? Yes. Excellent. And . Yeah. Where it was basically three things. You had the discussion, the diff so literally like a Git diff right. What was removed? What was added, and then a shorthand. These are the things that it's describing.\"}),/*#__PURE__*/e(\"p\",{children:\"It's done so effectively, almost like a code review on what it's done. In [00:07:00] case there are big changes. So it's the UX that kind of cracked that. And was that one of the big problems when you were looking at whether it's copilot or something in the early days? What was the killer feature?\"}),/*#__PURE__*/e(\"p\",{children:\"I guess you were missing in those early days to move to the next step. And what was that next step?\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Patrick Debois:\"}),\"\\xa0If you're coding and you're somehow in the flow, right? You want it to be snappy, right? So you want it to be, hey, I need information when I want it. I want to review and the first time that cycle with Cursor felt like, oh, this is, It keeps me into the flow.\"]}),/*#__PURE__*/e(\"p\",{children:\"The other things were like, okay, now I go somewhere on the side and in the past, it might've been a stack overflow. Then it was a chat, but now everything came into this is the problem I focus on. This is the next thing I need to think about. Give me the result. Oh, this is the next thing you need to think about.\"}),/*#__PURE__*/e(\"p\",{children:\"And kind of that amazing flow started to happen. It doesn't mean that the answers were perfect yet. But then the brilliant thing was if you're doing code completion you get one completion and you're like, ah, it's not what I wanted. And you're like, okay escape, because it's not what you want.\"}),/*#__PURE__*/e(\"p\",{children:\"Cursor [00:08:00] brought multiple line completions and multi completions as well. So they brought that into steroids. But then when you use compose, you just say I want a file, All that does it like this and, print out that and it gives you an answer. And when you're not happy with the answer, you just say change this.\"}),/*#__PURE__*/e(\"p\",{children:\"Yeah. And then you see, so you're having that conversational aspect, not on the chat, but immediately with your code and with the whole code base. So not just the one file, it understands everything of your code base. So it was aligning multiple things that were there before. In a kind of streamline the process that me as a developer, I found myself more and more just not typing the code, but just nudging the LLM or the tool to say this is what I want.\"}),/*#__PURE__*/e(\"p\",{children:\"No, a little bit more like that. Oh, that's good. That's not good. So that reviewing process got like insanely different.\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Simon Maple:\"}),\"\\xa0So let's talk about that in terms of an accuracy point of view.\"]}),/*#__PURE__*/e(\"p\",{children:\"It's funny when you talk, a lot of people who are still a little bit [00:09:00] skeptical about AI coding or just not what I'm trying to push it off a little bit too much.\"}),/*#__PURE__*/e(\"p\",{children:\"Very often you hear different levels of acceptance in terms of the, how accurate the suggestions are, whether it's what they want, whether it's even the style of implementation that they want, maybe they want to do it in a different way. Is it something that you care about enough in terms of the way?\"}),/*#__PURE__*/e(\"p\",{children:\"It's something's been implemented. Do you care about it? Like I wanted to use this type of set or collection or whatever versus this one. Do you care about that anymore or is it more the functionality that you want it to achieve that you're actually mostly discussing with the chat?\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Patrick Debois:\"}),\"\\xa0It's a dilemma. Do we still care about the code, how it looks like? Or do we just care?\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Simon Maple:\"}),\"\\xa0Is it your code anymore or is it your description of what the code should do?\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Patrick Debois:\"}),\"\\xa0I think my argument would be is that I still care in a way that when it fails as a human, I still want to understand how it works and why it works.\"]}),/*#__PURE__*/e(\"p\",{children:\"And navigating my code base could not just be a bunch of files naming a. py [00:10:00] and b. py. So there is a cognitive piece that I want to still understand what it is. What I see myself more and more do is I have a specific problem that I want to solve. I start off almost with a blank kind of project. And I iterate on that using Cursor and compose.\"}),/*#__PURE__*/e(\"p\",{children:\"So it's like a very well defined thing. And then I bring this into my bigger project. And then in that bigger project, I have more rules specified on how I want things. So one of the cool things of course is, besides taking the whole code base, having the compose was that you describe a little bit of a rules that the generation should adhere to.\"}),/*#__PURE__*/e(\"p\",{children:\"Like I'm coding in react and my front end. I always want to have an API test written in open API spec. I want all those things. So more and more, I could write my set of rules, like they call it Cursor rules and Cursor while generating takes that into as not just the [00:11:00] context of your code base, but it also takes that like your guidelines that it wants to take to generate the code that kind of sometimes gets in the way for a smaller project.\"}),/*#__PURE__*/e(\"p\",{children:\"Yeah. But for a bigger project. That's how you make it consistent across the whole code base.\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Simon Maple:\"}),\"\\xa0Yeah. And I presume, even with the smaller projects, there will be basic things that you will want to inherit from general development. Correct. And so we'll start with that.\"]}),/*#__PURE__*/e(\"p\",{children:\"So in terms of how we provide this type of context, there'll probably be some various system prompts that we'll always want to use. Are you adding these, are you creating these as like files, like supporting files that you include and reference in the composer? Or is there another way you include them?\"}),/*#__PURE__*/e(\"p\",{children:\"Cause you obviously you don't want to provide too much context or overwhelm it with huge number of instructions. So you want to be almost a little bit surgical and saying, I'm working on the backend here. I want you to use these backend requirements essentially, when you're creating these suggestions\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Patrick Debois:\"}),\"\\xa0So I have a base set of requirements I put in my project that I reuse across projects.\"]}),/*#__PURE__*/e(\"p\",{children:\"And then while doing [00:12:00] specific stuff, I add specific files. Let's say I'm doing front end stuff. Yeah. Then I'm specifically saying at front end specs, do this. So that kind of is a balance between. What I'm trying to achieve in the code and it's the same thing with adding documentation, depending on what I want to bring in is something that I decide on the moment.\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Simon Maple:\"}),\"\\xa0Yeah.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Patrick Debois:\"}),\"\\xa0So if you're specifying too much, it sometimes gets upset or it has conflicts and the code's not always that good. Yeah. So I, I do tweak this in small bits when I'm asking it to generate the code while I'm doing this.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Simon Maple:\"}),\"\\xa0And if you didn't add that kind of like context or like reference these various files, presumably you would still get to the same end point, but you would then have to iterate on its response more and more into the prompt.\"]}),/*#__PURE__*/e(\"p\",{children:\"Yeah. So you have more of a discussion going forward. So actually it's not quite how I would do that. I want to do it like this. It's interesting. There was a good quote from Des Traynor actually. He said something along the lines of the, our [00:13:00] willingness or our ability to want to write short prompts is robbing us from the need to be able to provide it with the detail it needs in order to give me the right answer. And it feels like this is effectively doing both. It provides us with the longer prompt the information it needs to say, this is what I want you. This is how I want you to provide me the good answers, but we're still being able to short prompt it.\"}),/*#__PURE__*/e(\"p\",{children:\"We're still being able to say, look, I want this, but we don't have to say exactly the detail of how we want. So you're almost like adding that little bit of background knowledge into the responses.\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Patrick Debois:\"}),\"\\xa0But it's like humans. When I say do a certain thing. And we have a lot of history and a lot of context are already, with two words. And so if I bring in a lot of context, in my editor and I provided with a set of rules, it can infer from the things that I want more information. A lot of the tools have like prompt compilers, they convert your simple prompt to a more complex prompt to get better results.\"]}),/*#__PURE__*/e(\"p\",{children:\"So this is definitely one of the ways they're getting, but the only way they can do this, if [00:14:00] you give them more context or specificity.\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Simon Maple:\"}),\"\\xa0Getting that detail out of your mind, actually out of your mind. Yeah. Yeah. Interesting. Another tool that you used was Aider, right? You use that before Cursor or, yeah, that was before Cursor, preCursor.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Patrick Debois:\"}),\"\\xa0So Aider is one of those open source tools to generate code. It's a little bit different from other tools in that it is more CLI based. So their approach was, you're coding, but at the same time you're running it like a demon in your CLI and you give it what you want. Similar to, a chat and it changed the code.\"]}),/*#__PURE__*/e(\"p\",{children:\"I think the quality was really good at what it was generating. But there were a few quirks that worked against me. One is it was auto committing codes that he thought was good. Which kind of ah of course you could disable it, but it was like, one of those things do we really want this?\"}),/*#__PURE__*/e(\"p\",{children:\"They had some cool things. Like I can talk with voice to it. And so I can have that iteration look more on a verbal thing, which is also something I tried with Cursor to make it like talk and have that conversation because I can just sit on the bench and talk [00:15:00] to it. Why are we still typing?\"}),/*#__PURE__*/e(\"p\",{children:\"And it's almost like that peer programming thing as well, right?\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Simon Maple:\"}),\"\\xa0We actually, when, a lot of the time when you just sat there at a keyboard, sometimes, you skip through various checks in your mind. Whereas actually, when you're trying to say something. You're thinking it through more and so as a result, you yeah, I guess you, you like fine tune exactly what you're asking for, which is interesting.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Patrick Debois:\"}),\"\\xa0But Aider has, what's nice is they blog a lot about how they do things. So they talk about how they do their internal diffing and sending small chunks to the LLM. So it's not overloaded with the whole code base. So they kind of work from there. This is also something Cursor does is they almost prioritize which part of the context are useful to solve your part of your problems.\"]}),/*#__PURE__*/e(\"p\",{children:\"So which is like optimizing for speed and context, because if you give them too much it's just a problem. Yeah. Something that I recently added is when the code is being generated, they have the notion of an architect reviewing your code. It's just the LLM taking another identity, but it critiques your code that's being generated.\"}),/*#__PURE__*/e(\"p\",{children:\"So you [00:16:00] don't have to do it yourself, but obviously latency plays into a big thing there because you want your code quick and you don't want to wait like for a minute. So there's a lot of caching optimization of the tools happening, but they're very open about it. And it's pretty cool. You can learn a lot about the source.\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Simon Maple:\"}),\"\\xa0Yeah. How good would you say those types of tools are truly understanding code? And do you feel, I see a lot of tools either mixing AI, so LLM versus a symbolic AI or something like that, or using static analysis of codes to better understand and model the architecture or the AST of your code, what's your experience been in those various approaches in truly providing good feedback versus just a very general feedback which can be hit and miss.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Patrick Debois:\"}),\"\\xa0It's one of those conceptions that people, they'll look at ChatGPT, they ask it to generate something and they don't like what they see. And they dismiss the whole option. I think an LLM combined with tools and combined with [00:17:00] agents, they give you massively better results, but what could go into some of the tools you mentioned that when it's cutting up your code, you don't want it to be cut in the middle of a sentence or a line of your code.\"]}),/*#__PURE__*/e(\"p\",{children:\"It needs to understand your code. That's why people are using AST. So the syntax tree, abstract syntax tree to parse things and chunk it up to the right pieces. So that's one layer of kind of improving.\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Simon Maple:\"}),\"\\xa0Which I was guessing when you were like talking about when the tools use various parts of the project or the code as context, because it knows those are relevant pieces.\"]}),/*#__PURE__*/e(\"p\",{children:\"I was wondering if it was using something like an AST to work that out.\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Patrick Debois:\"}),\"\\xa0Most of them are doing this to correctly. Divide the pieces in the right pieces. They need to look up or they need to change. So that's one thing then imagine the codes being generated, what you want in like a code generation pipeline, you want it to somehow specify and generate the tests that [00:18:00] it's doing. One of the tests could be like, I take the prompt and I convert it somehow into a functional test. Some of, I forgot the name, but some of the code editors actually are doing this first, the right to test first. And they help you write the tests and then they have the LLM write the code.\"]}),/*#__PURE__*/e(\"p\",{children:\"Because then you have your test harness and they have a better chance of success because the whole thing about TDD was actually refining what you wanted and how you wanted to see it. And then they give better hints to the, so that's another thing. There's loads of other tools. You can lint things when it's generated.\"}),/*#__PURE__*/e(\"p\",{children:\"Is it valid syntax? That's an easy one. So you can feed that back. Hey, there was an error or that you can run the code if it's a piece, but then you need sandboxing, there's all kinds of security issues. Can I run this? But in a pipeline, some of those parts you could do and you can see whether it executes and does not have like issues or it actually runs the tests as also an execute and give that back.\"}),/*#__PURE__*/e(\"p\",{children:\"There's [00:19:00] other things people have been putting in there, there's so many tools that, we use in our head, but these LLMs could use. Another one is code smells.\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Simon Maple:\"}),\"\\xa0Yeah.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Patrick Debois:\"}),\"\\xa0So if I'm editing the code and my code smells increase, I do not want to have that code. So they feed this back again, people from code scene, they do a lot of stuff kind of detecting because a lot of the refactoring is you do not want your code smells to go up.\"]}),/*#__PURE__*/e(\"p\",{children:\"And so feeding this back into the loop Hey, it's not okay. So I'll generate another. And that's under the hood. It's not like the end user see it, but obviously the longer the pipeline gets longer the latency, but if you look at coding agents that just run and run, so not specifically in your code editor, they could do this all day, right?\"}),/*#__PURE__*/e(\"p\",{children:\"So they can just run and see if it's optimized and go from there. There's other pieces of optimization that you have. Maybe you fine tune your model based on the code, based on your guidelines, deliberately inject errors [00:20:00] and see if they're caught and they're not good. So there's like a whole thing that people are doing this.\"}),/*#__PURE__*/e(\"p\",{children:\"And the other thing that people have learned is when they do changes is that do smaller changes and not like the big chunk with all the context, but it's almost like the humans. They need to understand almost like the, your code editor view plus and minus one. And that's the context they give it.\"}),/*#__PURE__*/e(\"p\",{children:\"And then they stuff a lot of context around it, but people are coming back from stuffing all the code into the LLM and see what happens.\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Simon Maple:\"}),\"\\xa0Too much context, basically overloading it with context. Let's talk about what you did with Cursor. I think that was super interesting beyond going beyond Cursor.\"]}),/*#__PURE__*/e(\"p\",{children:\"You made a couple of updates yourself built upon a Cursor. Tell us a little bit about some of the adjustments you made. First of all, what was the problem that caused you to do this? And and then how did you do that?\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Patrick Debois:\"}),\"\\xa0The first one is, so having looked a lot at the pipelines and, did my fair share of trying to make it work.\"]}),/*#__PURE__*/e(\"p\",{children:\"Sometimes you just want to use it. All right. And as Cursor was one of the better ones and like help that really give good [00:21:00] results. I thought it will be really nice to use that kind of whole logic inside of my own project. But Cursor is an IDE. It's not supposed to do any editing and so on. So what I did is I wrote a combination of a VS code plugin that allows you to type and an open CVE project that looks at your screen and see what our Cursor is open and then grabs the text and so on.\"}),/*#__PURE__*/e(\"p\",{children:\"So eventually it was able to wrap the IDE to an API. So I could just say run this, give me the change that you generated and go from there. And my whole idea was that I can now compare Cursor with Copilot, with all the stuff, and I just use the best tool, but most of the tools were closed in a way that they're in the IDE and you can't do anything about it.\"}),/*#__PURE__*/e(\"p\",{children:\"So that was my, I wanted to do this and then I could drive this code generation and add this as a tool to my agent.\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Simon Maple:\"}),\"\\xa0Yeah.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Patrick Debois:\"}),\"\\xa0So I got the best of all breeds. So\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Simon Maple:\"}),\"\\xa0I guess there's a couple of next steps on that. The first one is you could effectively automate it to say, [00:22:00] look, this is what I want.\"]}),/*#__PURE__*/e(\"p\",{children:\"Use five tools go and do it in the way you want to do it. Provide me with the results. I can then choose which result I want to use. And so you're effectively running a team of developers that are playing off each other. And you can say, actually, this is the one I want, and these are the reasons why I don't want the others.\"}),/*#__PURE__*/e(\"p\",{children:\"That's, one way. A second way could be to automate that and then say, is there a way I could actually take a tool that actually reviews that code and identifies the best of the options and suggests the best option to me and to why this was the best option around other codes. So there's like levels of you effectively being almost like a team lead of these AI developer agents.\"}),/*#__PURE__*/e(\"p\",{children:\"Yep. And you're managing that team effectively.\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Patrick Debois:\"}),\"\\xa0Yeah, indeed. And then you have the reviewing agents, the code writing agent and Yeah. So that was my motivation to play and make that into an API. But\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Simon Maple:\"}),\"\\xa0How successful was that in terms of then being able to get results back that you would be more likely to accept.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Patrick Debois:\"}),\"\\xa0It wasn't about being more likely to accept, but it was more like I can use it for my own tools.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Simon Maple:\"}),\"\\xa0Right?\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Patrick Debois:\"}),\"\\xa0Yes. So that [00:23:00] was the usefulness. No, I didn't have to write the whole pipeline. Somebody wrote it for me and I can just use it.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Simon Maple:\"}),\"\\xa0Consume that data however you want in your next step.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Patrick Debois:\"}),\"\\xa0And the second tool was about, we mentioned this already before, I find myself reviewing code more and then commenting on the code.\"]}),/*#__PURE__*/e(\"p\",{children:\"And I first tried, I think, a voice recording assistive tool, because then I wouldn't have to type anymore, right? So I was like, okay. But I found that even VSCode has a voice assistant. It is only typing into the code editor. It was not typing into the Cursor editor or kind of all the fields.\"}),/*#__PURE__*/e(\"p\",{children:\"Then I went to an accessibility tool with the voice. But then I figured I was doing this and my wife came in and started talking to me. I got like things on my screen. I do this? And sometimes I still needed that keyboard for the shortcut for activation. And I didn't want to do this.\"}),/*#__PURE__*/e(\"p\",{children:\"I actually came across an auto tool called Cursorless. There's no kind of connection between both, but they [00:24:00] had a fascinating view about using sounds like to activate certain things. And then they would say little rats putter, which means click on the next word with a red p in the editor.\"}),/*#__PURE__*/e(\"p\",{children:\"So they annotate the whole\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Simon Maple:\"}),\"\\xa0A little bit like what three words Yeah, so it's very complex.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Patrick Debois:\"}),\"\\xa0So I figured like, how can I do this? And what I ultimately ended up with was I thought when you want to have somebody listen to you, what do you do? You raise your finger, right? Okay.\"]}),/*#__PURE__*/e(\"p\",{children:\"So I made a little thing that watches me on the camera. And when I did this, instead of typing, it was listening to me in voice mode. But then there's a difference between like, when I say do this command is it now typing or do I want it to execute?\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Simon Maple:\"}),\"\\xa0Yeah.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Patrick Debois:\"}),\"\\xa0So I basically switch between this is typing and this is executing a command.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Simon Maple:\"}),\"\\xa0Yeah.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Patrick Debois:\"}),\"\\xa0And so now I can switch between two modes and I'm just sitting on the sofa saying, okay, this is the two fingers, compose, activate.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Simon Maple:\"}),\"\\xa0Yeah.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Patrick Debois:\"}),\"\\xa0It [00:25:00] brings up composer.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Simon Maple:\"}),\"\\xa0Yeah.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Patrick Debois:\"}),\"\\xa0Then I say, I'm typing what I want. And then I say, okay, so I'm just switching with my fingers.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Simon Maple:\"}),\"\\xa0And if you turn the fingers round, you get very angry at the idea.\"]}),/*#__PURE__*/e(\"p\",{children:\"If you choose to, yeah.\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Patrick Debois:\"}),\"\\xa0Anyway, I'm a big fan of the OpenAI real time interaction. A lot of people think of it as clunky, but it is able to talk conversationally back to you in a very good way. And the other thing they solved was I can interrupt the speech. So it's not okay. Damn. No, it's reading off my whole diff.\"]}),/*#__PURE__*/e(\"p\",{children:\"Yeah. Yeah. That's not what I want. Yeah. So I had some shortcuts by saying open compose and it was listening for the words open compose and kind of activating it on that. But then I went a step further and I thought but the LLM can just look at my texts and I hooked it up with function calling.\"}),/*#__PURE__*/e(\"p\",{children:\"And so if it thinks it should trigger the function, open compose, however, I say it or [00:26:00] whatever it determines that it's going to bring up open composer. So now I don't have to specifically say any more okay, this function is with that trigger word. If you don't know the trigger word, it's just listening on the conversation.\"}),/*#__PURE__*/e(\"p\",{children:\"And I just can keep adding more tools. Oh, no, I'm going into my browser. And when I go into my browser, you probably want to take a screenshot and you want to upload it into my Cursor. Because then you can look and generate me the front end code that I wanted. So this is just a conversational way that I brought up.\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Simon Maple:\"}),\"\\xa0That's brilliant. I love it. And I would love to almost see, if there was a whiteboard behind us, for example, and we're drawing out some architecture and it can start recognizing the architecture and almost using that as a format that we can describe and get code written in certain places, there's a ton of things where it can take more than just the text input and actually take visual cues, like you just mentioned, as well.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Patrick Debois:\"}),\"\\xa0Like you would ask what's on my screen?\"]}),/*#__PURE__*/e(\"p\",{children:\"Yeah. And it would tell you what's on your screen. Yeah. How do I improve it? Yeah. Here's two suggestions.\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Simon Maple:\"}),\"\\xa0Now [00:27:00] recently, certainly at the time of recording this, but maybe a few weeks ago when for people who are listening. There was some recent additions to Athropic's Claude whereby there is a beta that has been released or beta for our US friends, they effectively have some screen scraping technology that can identify what's in your screen.\"]}),/*#__PURE__*/e(\"p\",{children:\"And even interact. So you're effectively having a coder who can write into your IDE, whatever it is, even if there's no support for AI and things like that in that IDE, because Anthropic is now looking at your screen. How do you feel about that kind of thing?\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Patrick Debois:\"}),\"\\xa0First of all, I think it makes sense.\"]}),/*#__PURE__*/e(\"p\",{children:\"When I would use ChatGPT and I'm in a store, what do I do? I take a picture and say, okay, what do I need to know? And we've been in the IDE using this on the OS. It's very simple, similar to what I said. What's on my screen? How can you improve this? So now it's the next step. What they did is it's allowed to execute something locally.\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Simon Maple:\"}),\"\\xa0Yeah.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Patrick Debois:\"}),\"\\xa0But it is just another tool. But [00:28:00] now they run this locally. I think this was an evolution. Also, Microsoft has a way that they record everything that ever went on your screen, and you're able to recall, remember that spreadsheet, Simon, that we worked on last Friday or something. Boom.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Simon Maple:\"}),\"\\xa0Oh my God. Oh my God.\"]}),/*#__PURE__*/e(\"p\",{children:\"You didn't, you didn't see that spreadsheet. Yeah. Yeah.\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Patrick Debois:\"}),\"\\xa0No. So that kind of is in eminence in a way that the tools are just getting more on the local IDE and one of the things like with my tooling, I had to determine it's what's on my screen as well, because is it now Cursor or was it my browser?\"]}),/*#__PURE__*/e(\"p\",{children:\"So the machines were looking at the context. I was able to do this by looking at the process list and which was, when it was on my screen, but you could definitely look at other ways. The other thing I had to do was I had to look where a certain dial was and you could train a model, say, look at the screen, where's the certain dial, but then tomorrow there's something else.\"}),/*#__PURE__*/e(\"p\",{children:\"And then if you would have an if statement list of saying if a dial than that, it will not be very [00:29:00] efficient. But if I would train a model that just understands all the software tooling we're using today and recognizes what kind of screen, what are the actions you're supposed to be taking when you have this in front of your screen.\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Simon Maple:\"}),\"\\xa0Yeah.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Patrick Debois:\"}),\"\\xa0I know people are trying to make this even more self learning is they look at. Again, it's hooking it up with a tool. They look at it from the accessibility inspector, like they'll look at the buttons on the screen and they'll look like, are they clickable? Can we do something with that? Then they look at the hints of the screen, what they can do.\"]}),/*#__PURE__*/e(\"p\",{children:\"So this has been a little bit going on in a research project called OS world where they're already doing this in a virtual machine and you can spin this up and it gives you a desktop. And then they try all out all these scenarios, open my screen, open my browser, do this. And it's very similar to what I've described, like hooking up with the function calling.\"}),/*#__PURE__*/e(\"p\",{children:\"Once you know what's there, then you can take the next step on kind of the those things, would I trust it?\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Simon Maple:\"}),\"\\xa0Security was the next kind of thing. Cause it, once it basically [00:30:00] has access to whatever's on your screen, you don't know what's going to pop up in your screen next. And how do you control what it should and shouldn't have access to?\"]}),/*#__PURE__*/e(\"p\",{children:\"And I think there's a wider problem here ,in and around authorization and things like that. And we've talked with people, Tamar Yehoshua, who, gave some really interesting thoughts about how Glean actually have their authorization as a step before the LLM working out what its users actually do have access to and then providing the LLM with what the user should access versus have the LLM try and work that out and understand that because you have to assume the LLM is like a user in that respect.\"}),/*#__PURE__*/e(\"p\",{children:\"You don't have that control necessarily when you allow someone to completely look at your screen. So does that worry you at all from the security point of view, personal data, anything like that?\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Patrick Debois:\"}),\"\\xa0I think it does in the way that I would run this on a virtual desktop and be sure what I give access to it.\"]}),/*#__PURE__*/e(\"p\",{children:\"Obviously not everybody, is supposed to be tech savvy. At a certain point you [00:31:00] understand what is allowed and what isn't allowed. I think there is an inherent problem with the LLM in that there is hallucination. Yes, we're trying to reduce it, but we are never sure what it's going to do.\"}),/*#__PURE__*/e(\"p\",{children:\"So we might restrict what it's able to do. That's our best thing. But then obviously if you want it to change document files. Where do you stop and then you get a tedious thing. Like when it's going to ask you every time, can I change this doc? And you're just going to say, okay, whatever. So that compliance is worrisome in a way that we know the technology inherently is not always repeatable, that it's always giving the same result.\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Simon Maple:\"}),\"\\xa0But it's incredible to be able to share context across applications. That's going to be so powerful, I think. And the intent as well to be able to say, Oh, I'm in this app. This is the level that the user requires this detail on, but now I'm in an IDE. I need to create actual implementation and go a much, much lower level to provide value.\"]}),/*#__PURE__*/e(\"p\",{children:\"Amazing. Before we close, I'd love to talk a little bit of more breadth in terms of when you are coding, what [00:32:00] other assistant tools have you played with that has really added value, most value to your coding experience, thinking here, whether it's test support, whether it's documentation support, whether it's anything else like that.\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Patrick Debois:\"}),\"\\xa0The one that I quite often use is V0, which is for a front end generation. Yeah,\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Simon Maple:\"}),\"\\xa0yeah.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Patrick Debois:\"}),\"\\xa0So that's helps you. And I think it's nice that you specify the website that you want. They generate a code and you can iterate on that.\"]}),/*#__PURE__*/e(\"p\",{children:\"And now you can sync this back into your repo as well. So it's an additional tool. The Cursors and the like are not really focused on that, but it helps you with the front end kind of stuff. I've been exploring a tool called runme.dev, which is more of a tool on writing notebooks for DevOps commands.\"}),/*#__PURE__*/e(\"p\",{children:\"Another way of not writing all the scripts, but keeping mixture of documentation in there. The documentation I usually do in line, like one of my specification is like whenever you write something, add documentation. So there's a lot more tooling around that. And the [00:33:00] other tool is now I have to think it was e2b if I'm not mistaking for sandboxing, I always mess the letters up.\"}),/*#__PURE__*/e(\"p\",{children:\"So that was three, but for a sandbox execution of functions. So I get like a fast execution solution.\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Simon Maple:\"}),\"\\xa0All a little bit similar to a kind of like a replit style, would you say, or slightly different?\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Patrick Debois:\"}),\"\\xa0The replit, it is similar, but this one is very focused on sandboxing and just gotcha.\"]}),/*#__PURE__*/e(\"p\",{children:\"So it's easy to integrate in your code if you want something to happen. Yeah. And the other tool, gosh, I use all the tools that don't remember the name, but I think the other one was AI inspects and it allows you to do much like evaluations. But in your VS code, it's like a, somewhere a government project.\"}),/*#__PURE__*/e(\"p\",{children:\"But what's really nice. And most of the evaluations, they take a long time. They're run somewhere else, like as Langsmith and so on. But this plugin allows me to keep it into the VS code and have the runs also across multiple [00:34:00] LLMs and see which one was the best for this kind of aspect. Nice. So those are a few add ons to the tools.\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Simon Maple:\"}),\"\\xa0Yeah. Amazing. And for people who are just thinking about starting off and wanting to try something what kind of, of all the tools that you've mentioned, is Cursor the one that you would start off with or something else?\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Patrick Debois:\"}),\"\\xa0I guess if you're very much into front end, you go to the V0.\"]}),/*#__PURE__*/e(\"p\",{children:\"Yeah. If you're a little bit more into the generate whole applications, there's another one called Bolt. new. And then Cursor, if you're day to day code writing back end front end. That's the one I would do. And then there's the obvious ones like Anthropic artifacts that help you as well for generating some of this stuff.\"}),/*#__PURE__*/e(\"p\",{children:\"So those are a typical, tool base. But I think what's happening is that all the tools. They're learning from each other and hopefully, you have everything somewhere close into one tool which is interesting. For example, one of the next GitHub Copilots they will look at your test coverage and then as you just tests [00:35:00] also something you can use.\"}),/*#__PURE__*/e(\"p\",{children:\"Everybody's exploring and maybe that was another reason why to build an API around it, because then I can use whatever tool for the right job.\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Simon Maple:\"}),\"\\xa0Amazing, Patrick, it's been a blast as always to have a chat and thank you for so much like very practical, hands on tips and tooling suggestions for people.\"]}),/*#__PURE__*/e(\"p\",{children:\"So No, no excuses now go out and have a play and see what you think. Let us know what you're using, what you're enjoying. Thanks again, Patrick, for joining us twice.\"}),/*#__PURE__*/e(\"p\",{children:\"The first person twice on the, on the AI Native Dev.\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Patrick Debois:\"}),\"\\xa0My pleasure. In a couple of months, we'll automate it ourselves. Absolutely. During the podcast, like notebook LLM.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Simon Maple:\"}),\"\\xa0Yeah, there we go. There we go. And we can just relax on a beach somewhere, watching our own jobs being done better than we could. Amazing. Thanks very much and join see you on the next episode.\"]}),/*#__PURE__*/e(\"p\",{children:\"Thanks for tuning in. Join us next time on the AI Native Dev brought [00:36:00] to you by Tessl.\"})]});export const richText1=/*#__PURE__*/t(n.Fragment,{children:[/*#__PURE__*/e(\"h4\",{children:/*#__PURE__*/e(\"strong\",{children:\"Des Traynor's Perspective on AI Product Development\"})}),/*#__PURE__*/e(\"p\",{children:'Des Traynor, founder of Intercom, provides an in-depth look into the complexities of integrating AI into traditional product development. According to Des, the integration of generative AI introduces a paradigm shift in how products are developed. He emphasizes that while developers love to ship code, the real challenge with AI models lies in understanding whether they work effectively in production environments. Des mentions the concept of \"torture tests,\" which are designed to simulate the most demanding scenarios AI models might face in real-world usage. This rigorous testing is crucial to ascertain the performance and reliability of AI models in production.'}),/*#__PURE__*/e(\"p\",{children:'Des also highlights the non-deterministic nature of AI, which requires ongoing evaluation and adaptation. In the episode, he states, \"You have to do so much, whereas in typical boring bread and butter B2B SaaS\u2026 you\\'re assuming that the mathematics worked or whatever.\" This underscores the need for continuous monitoring and testing to ensure AI systems perform as expected and adapt to changing user inputs and environments. He further elaborates on the need to shift from a deterministic mindset to one that embraces the spectrum of possibilities AI presents, which is a significant departure from traditional software development paradigms.'}),/*#__PURE__*/e(\"p\",{children:\"Moreover, Des discusses the importance of understanding the full lifecycle of AI products, from conception to deployment, and the subsequent need for iterative testing and refinement. He underscores the complexity of building AI products that can evolve over time, reflecting real-world data and user interactions. This dynamic approach requires a deep understanding of both the technical capabilities of AI and the practical implications of deploying these technologies at scale.\"}),/*#__PURE__*/e(\"h4\",{children:/*#__PURE__*/e(\"strong\",{children:\"Guy Podjarny's Insights on Building LLM-Based Products\"})}),/*#__PURE__*/e(\"p\",{children:'Guy Podjarny shares his experiences from Tessl and Snyk on developing LLM-based products. He discusses the inherent difficulties in evaluating AI products, particularly the challenges posed by their non-deterministic behavior. Guy emphasizes the importance of adapting CI/CD processes to accommodate the unique requirements of AI development. He notes, \"Some of the tools are quite immature,\" highlighting the need for innovation and improvement in the tools and methodologies used for AI product development.'}),/*#__PURE__*/e(\"p\",{children:\"Guy also stresses the necessity of empowering developers to work in an AI-first environment. He believes that developers must embrace the ambiguity that comes with AI and learn to navigate it effectively. This involves a shift in mindset from traditional deterministic programming to understanding and managing the probabilities and uncertainties inherent in AI systems. Additionally, Guy highlights the role of continuous integration and continuous deployment (CI/CD) frameworks in streamlining AI development, emphasizing the need for robust testing environments that can handle the unpredictability of AI outputs.\"}),/*#__PURE__*/e(\"p\",{children:\"Furthermore, Guy discusses the challenges of maintaining consistency and reliability in AI products, advocating for a holistic approach that combines both automated testing and human oversight. He stresses the importance of fostering a culture of experimentation and learning within development teams, where developers are encouraged to explore new techniques and methodologies to optimize AI performance and usability.\"}),/*#__PURE__*/e(\"h4\",{children:/*#__PURE__*/e(\"strong\",{children:\"Rishabh Mehrotra on the Importance of Evaluation\"})}),/*#__PURE__*/e(\"p\",{children:'Rishabh Mehrotra, Head of AI at SourceGraph, delves into the critical role of evaluation in AI development. He introduces the \"zero to one\" metric for testing models, which focuses on the effectiveness of evaluation datasets in determining model performance. Rishabh argues that \"writing a good evaluation is more important than training a good model,\" emphasizing that without robust evaluation metrics, developers may not accurately assess the impact of model improvements.'}),/*#__PURE__*/e(\"p\",{children:\"Rishabh also discusses the importance of creating feature-aware evaluation datasets, which are tailored to specific use cases and environments. He points out that industry benchmarks may not always reflect real-world usage, and developers need to develop evaluations that align with actual user experiences and expectations. This approach ensures that AI models are tested against scenarios that truly represent the complexities and nuances of their intended applications.\"}),/*#__PURE__*/e(\"p\",{children:\"Additionally, Rishabh highlights the significance of iterative evaluation processes, where continuous feedback and data-driven insights are used to refine and enhance AI models. He advocates for a dynamic approach to evaluation, where metrics are continuously updated and aligned with evolving user needs and technological advancements. This ensures that AI models remain relevant and effective in addressing the challenges of modern software development.\"}),/*#__PURE__*/e(\"h4\",{children:/*#__PURE__*/e(\"strong\",{children:\"Tamar Yehoshua on Enterprise Search Challenges\"})}),/*#__PURE__*/e(\"p\",{children:\"Tamar Yehoshua, President of Products and Technology at Glean, explains how Glean manages enterprise search across sensitive data sources. She discusses the challenges of using LLMs as a judge and jury for evaluating AI responses, particularly in environments where data sensitivity is paramount. Tamar highlights the difficulty of ensuring non-determinism in enterprise environments, where users expect consistent and reliable outputs.\"}),/*#__PURE__*/e(\"p\",{children:\"Glean addresses these challenges by employing suggestive prompts and structured templates to guide users, thus managing expectations and improving user experience. Tamar shares that Glean has dedicated teams for evaluation and uses LLMs to judge the completeness, groundedness, and factualness of AI responses, providing a nuanced approach to handling LLM outputs. This strategy ensures that AI-generated responses are not only accurate but also contextually relevant and aligned with user expectations.\"}),/*#__PURE__*/e(\"p\",{children:\"Moreover, Tamar discusses the importance of transparency and user empowerment in AI-driven enterprise search solutions. She emphasizes the need for clear communication and guidance, enabling users to understand and trust the outputs generated by AI systems. By leveraging a combination of human expertise and automated evaluation tools, Glean ensures that its AI solutions are both robust and user-friendly, catering to the diverse needs of modern enterprises.\"}),/*#__PURE__*/e(\"h4\",{children:/*#__PURE__*/e(\"strong\",{children:\"Simon Last on Continuous Improvement at Notion\"})}),/*#__PURE__*/e(\"p\",{children:\"Simon Last, Co-Founder and CTO of Notion, shares Notion's approach to logging failures and creating reproducible test cases. He describes an iterative loop of collecting failures, adjusting prompts, and validating fixes, which ensures continuous improvement of AI capabilities. Simon emphasizes the importance of privacy and user consent in data sharing for evaluation purposes.\"}),/*#__PURE__*/e(\"p\",{children:'Simon highlights the necessity of a repeatable system for managing AI failures and improvements. He states, \"You need to make sure that those work and they don\\'t regress,\" underscoring the importance of maintaining a robust evaluation framework that ensures AI models continue to meet user expectations and adapt to new challenges. This approach enables Notion to deliver reliable and effective AI-driven solutions that enhance user productivity and collaboration.'}),/*#__PURE__*/e(\"p\",{children:\"Additionally, Simon discusses the value of transparency and collaboration in AI development, advocating for open communication and feedback loops within development teams. By fostering a culture of continuous learning and improvement, Notion is able to refine its AI capabilities and deliver innovative solutions that meet the evolving needs of its users.\"}),/*#__PURE__*/e(\"h4\",{children:/*#__PURE__*/e(\"strong\",{children:\"Evaluation and Testing Strategies Across Industries\"})}),/*#__PURE__*/e(\"p\",{children:\"The episode brings together common themes and strategies from all guests regarding AI evaluation and testing. A key takeaway is the balance between synthetic tests and real-world scenarios in ensuring AI product reliability. The discussions emphasize the role of human judgment and automation in refining AI models and outputs, highlighting the need for a comprehensive approach to evaluation and testing.\"}),/*#__PURE__*/e(\"p\",{children:\"Furthermore, the guests underscore the importance of adaptability and resilience in AI development, encouraging developers to embrace new technologies and methodologies to optimize their AI solutions. By fostering a culture of innovation and experimentation, organizations can harness the full potential of AI to drive business success and deliver exceptional user experiences.\"})]});export const richText2=/*#__PURE__*/t(n.Fragment,{children:[/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(o,{href:\"https://www.intercom.com/\",motionChild:!0,nodeId:\"X7VlByTfx\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(a.a,{children:\"Intercom\"})})}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(o,{href:\"https://sourcegraph.com/\",motionChild:!0,nodeId:\"X7VlByTfx\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(a.a,{children:\"SourceGraph\"})})}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(o,{href:\"https://www.glean.com/\",motionChild:!0,nodeId:\"X7VlByTfx\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(a.a,{children:\"Glean\"})})}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(o,{href:\"https://www.notion.so/\",motionChild:!0,nodeId:\"X7VlByTfx\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(a.a,{children:\"Notion\"})})})]});export const richText3=/*#__PURE__*/e(n.Fragment,{children:/*#__PURE__*/t(\"p\",{children:[\"[00:00:00] Introduction by Simon Maple\",/*#__PURE__*/e(\"br\",{}),/*#__PURE__*/e(\"br\",{}),\"[00:02:00] Des Traynor on AI Product Development\",/*#__PURE__*/e(\"br\",{}),/*#__PURE__*/e(\"br\",{}),\"[00:13:00] Rishabh Mehrotra on the Importance of Evaluation\",/*#__PURE__*/e(\"br\",{}),/*#__PURE__*/e(\"br\",{}),\"[00:21:00] Tamar Yehoshua on Enterprise Search Challenges\",/*#__PURE__*/e(\"br\",{}),/*#__PURE__*/e(\"br\",{}),\"[00:34:00] Simon Last on Continuous Improvement at Notion\",/*#__PURE__*/e(\"br\",{}),/*#__PURE__*/e(\"br\",{}),\"[00:49:00] Summary and Closing Remarks\"]})});export const richText4=/*#__PURE__*/t(n.Fragment,{children:[/*#__PURE__*/e(\"p\",{children:\"**Simon Maple:** [00:00:00] You're listening to the AI Native Dev brought to you by Tessl\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"Hello and welcome to another episode of the AI Native Dev. And this is going to be a bit of a mash up episode where we're going to take a listen to a number of excerpts from previous episodes where we talk about whether we as organizations are ready to be able to say this product works better after these changes to my model and this really takes account of various things, including eval testing, evaluations, regression testing and so forth. This is gonna be a mashup between Des Traynor, intercom founder as well as Rishabh Mehrotra [00:01:00] who's the Head of AI at SourceGraph. Tamar Yehoshua, who is the president of products and technology at Glean. And lastly, but by no means last, Simon Last, the Notion co founder and CTO. We're going to start off with Des Traynor's take and one of the things that I love about Des Traynor is, of course he recognizes devs love to ship code and one of the things that is really important when we're using LLMs in our applications is we don't truly understand, we don't truly know if something really works until it's in production using real production data, real data that the users are throwing at it.\"}),/*#__PURE__*/e(\"p\",{children:\"And one of the lovely quotes that Des mentioned in his section is that with sufficiently simple tests, Des could be as smart as Einstein. Because the tests are so simple, they could both obviously do so so well at it. In this section of the episode Des talks about this wonderful thing called torture test, which is in Des's words all the psychotic scenarios [00:02:00] that Fin, the intercom tool will get put into in the real world. So it's this set of torture tests that can really ascertain whether changes to the model changes to prompts, et cetera really do have an effect in production. So over to Guy and Des..\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"**Guy Podjarny:** We have the pleasure, dubious pleasure of building LLM based products at Tesla. I've had some of that at Snyk. It's different. They're annoying. They are hard to evaluate whether you're doing your job correctly or not. Sometimes costs are different. CICD is different. Some of the tools are quite immature.\"}),/*#__PURE__*/e(\"p\",{children:\"And we're talking probably a year before that experience that I'm referring to here. How hard was it for a developer who previously didn't work in that surrounding to adapt it? How did you help the dev team? Not just, Hey, I reprioritize this feature or am I overblowing the change?\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"**Des Traynor:** The change is big.\"}),/*#__PURE__*/e(\"p\",{children:\"It's actually been a subject of a lot of internal debate. I'll give the umbrella answer and [00:03:00] then we can get to specifics in a little bit. But generally speaking, everything about how we build software products changes when you bring in generative AI, probability based systems, or whatever you want to call it.\"}),/*#__PURE__*/e(\"p\",{children:\"There's a classic product development systems like double diamond. It's like research, make decisions, build, like ship, and then see if it works, whatever. You have like extra diamonds all over the place when it's like AI, because It's, you don't start with here's the user problem. Let's solve it. You tend to start with what's technically possible to begin with.\"}),/*#__PURE__*/e(\"p\",{children:\"And then you say, Hey, does this new technical capability, let's say, Hey, we can read PDFs in a second. Okay. Does that map onto any problems that we have to solve for our users? And then you find one of those. And then, so then you've worked out this thing can be built and it seems like it solves this problem.\"}),/*#__PURE__*/e(\"p\",{children:\"Even then, everyone's DNA and software is, so build it and ship it and move on to the next feature. What's the big deal? And I think the problem is once these things go live, you still don't know if it works or not. So you have to do so much, whereas in typical boring bread and butter B2B SaaS, someone's, oh, I [00:04:00] wonder, can we merge expenses?\"}),/*#__PURE__*/e(\"p\",{children:\"Okay. So you get two receipts, you have to combine them. Oh, okay. If you ship that feature and people use it, you're like, great. You're not sitting here going, Ooh, I wonder if it works. You're assuming that they're like the mathematics worked or whatever.\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"**Guy Podjarny:** Yeah.\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"**Des Traynor:** Whereas in all these systems, once it goes live, it goes through a whole other, like almost like product evolution, which is it actually working the way we thought it did.\"}),/*#__PURE__*/e(\"p\",{children:\"And obviously we do our best to not just test these systems on what we would call happy paths. A happy path is like, how do I reset my password? And unhappy path is, hey, my account is stored in my husband's name and it's my credit card. And I need to move it to his credit card, but also we need to change address at the same time.\"}),/*#__PURE__*/e(\"p\",{children:\"That's where like these systems struggle massively. They're all brilliant at like, how do I reset my password? In the same way that if me and Einstein sit down and take a maths test, and it's just, Basic arithmetic. I look as a, I'm a smart, yeah. So you have to have this higher pass filter to get through.\"}),/*#__PURE__*/e(\"p\",{children:\"But nothing like reality bats last in AI it's the [00:05:00] last person to step up and reality tells you like, hey, Fin works great in all these circumstances, but here's a pocket that it doesn't work in. Here's an area that it that I can't reach or whatever. Yeah. So I think the first thing I'd say to the question of enabling devs, empowering devs to work in this LLM first world is like getting outta your head the idea that things just simply do or don't work because it's all a spectrum now getting out of your head that your core job is to just clarify or remove ambiguity. Ambiguity is all over the place. It's on your job is it's such a. There's like an art and science to like threading all the needles, defining the useful solutions.\"}),/*#__PURE__*/e(\"p\",{children:\"And that's before we've actually sat down in front of a tech center and started banging out code. That's is in working out what problem we could solve, why we think it's solvable, like most of your prototypes in this era are like written by just prompting an LLM and seeing what it does with certain prompts and seeing what it does in response to certain asks.\"}),/*#__PURE__*/e(\"p\",{children:\"And basically if you can get to a consistent, reliable set of behaviors, And then you can start to think about [00:06:00] how would we productize this? What are the API calls? And then what's the UI look like? And then how would we describe it? All that, but you have so much like pre work to do all that said, there is obviously around any given like product, if we just, I'll take a Fin because that's the one I know best. There is still a lot of bread and butter B2B SaaS shit you have to build. Someone has to import docs and somebody has to have permissions and it has to be charged for it. So there's a pricing feature. Like it's not all, it's not like you're an LLM engineer or bust, it's the secondary features that are fall into the classic.\"}),/*#__PURE__*/e(\"p\",{children:\"Like our reporting engine still draws graphs, but that's not new. But.\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"**Guy Podjarny:** The just like any like engine, if you built any sort of proper product at the end of the day, the engine is 10, 20 percent of the surface of the product.\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"**Des Traynor:** Exactly that. And I think if there's a whole thin wrapper, thick wrapper debate that happens a lot in the industry.\"}),/*#__PURE__*/e(\"p\",{children:\"And I think like Fin is very much like a thick wrapper. There's a lot around Fin that needs to be there to make it work really well, such that the actual, the pieces are like the little, like the resolution engine that sits inside of it. That is a part of the puzzle. It's a really important [00:07:00] high impact part.\"}),/*#__PURE__*/e(\"p\",{children:\"But it alone isn't sufficient, it still needs the 16 other things, the vector search, the content import, and all that sort of stuff.\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"**Guy Podjarny:** So it sounds like a massive change. Sounds like the, a lot of the changes is in the sum, the total of the product. There is a good chunk of people that are still working on things that don't require them to change their, way of working still a good portion that did maybe in the total.\"}),/*#__PURE__*/e(\"p\",{children:\"So I guess, are there some key takeaways? Like I'm sure you've tried things and some work, you had a little bit of a headstart cause they didn't have some AI capabilities you had.\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"**Des Traynor:** But we had an AI product live since I think 2017, 2018. So like we definitely had prior experience here and that was one of the reasons we can move fast because we are able to reuse a lot of our existing features.\"}),/*#__PURE__*/e(\"p\",{children:\"For example, like the vector search engine, we'd already built one for a previous product, et cetera. So that's one of the reasons why we were like out of the gates so fast. If I was to really say like the biggest lessons I've observed, like speaking of what I'm skipping over here is I'm sure there are libraries and APIs and all sorts of stuff you need [00:08:00] to read and get good at, et cetera.\"}),/*#__PURE__*/e(\"p\",{children:\"I think to actually be a performance engineer in this era requires some amount of ability to interrogate data or perform data science to measure outcomes. To give you a very simple example. When we make a change, we've run probably like 50 different AB tests on Fin's performance. And it's very tempting.\"}),/*#__PURE__*/e(\"p\",{children:\"Everyone comes in who might join the group and say something like, Oh, I have a cool feature. I could add, how about we make it so that blah, like, how about we make it so that it'll always ask this question at the end of the thing and what you don't realize is like all of these things are different iterations of a resolution engine, whose job is ultimately to given a question, answer it.\"}),/*#__PURE__*/e(\"p\",{children:\"And if you break that, it doesn't matter how cool your little addition was. Now, the thing is that's not a binary yes, no condition. It's like today Fin's performance is about 50%. So 50 percent of inbound support volume will be resolved by Fin. And if somebody rolls out a cool feature that brings that 50 down to 40, that's a problem.\"}),/*#__PURE__*/e(\"p\",{children:\"**Guy Podjarny:** Yeah.\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"**Des Traynor:** [00:09:00] If they do that and don't even know to check or to realize, or to look at its performance over a trailing 30 days or whatever, that's an even bigger problem. So there's a very much, you need to work out. You have to understand the trick to working with these systems is to bottle up a probabilistic outcome and productize it.\"}),/*#__PURE__*/e(\"p\",{children:\"And you have to work out who is allowed play with the probabilities of that. And understand that it's not like two different worlds. It's very easy for somebody in our messenger team to damage the performance of Fin, even though they don't write to that code base, but if they frame it wrong and they introduce Finn in the wrong way, it'll change what people write and if it changes what people write, then it'll change how Fin performs.\"}),/*#__PURE__*/e(\"p\",{children:\"So it's really the extra skill required here is the ability to assess. Does this thing do the job it's supposed to do at a really high performance level? And you have to have some way to measure that, some way to test it, have bake offs of different engines against each other, some way to have an acceptance framework for when, all right, that's been live for 30 [00:10:00] days.\"}),/*#__PURE__*/e(\"p\",{children:\"No one's complained. It looks like we've got a 4 percent increase in resolutions, ship it. It's like that, none of those skills really existed before in, I'd say in most companies, but definitely not in Intercom.\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"**Guy Podjarny:** Yeah, no, it sounds fascinating. I guess the good news is there's some analogy between what you're describing and the state of the art in terms of continuous deployment, right?\"}),/*#__PURE__*/e(\"p\",{children:\"If you're working with flags, if you're in Facebook and you're shipping, I forget the name of their sort of feature system there, it's some gradual, a number of people, then there's some similarity of it, but this is much more en masse, much more predictable and pretty much does it kill almost substantially the value of a sort of CICD assessment for anything that isn't the more bread and butter SaaS piece?\"}),/*#__PURE__*/e(\"p\",{children:\"How much having run with us now for a bit, how trustworthy or how revealing are the results of the synthetic tests or the CICDs type, the pre deployment tests that you have.\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"**Des Traynor:** We have two tests. We have one test we perform internally, which is expensive to run, which is like our torture test. So let's just say [00:11:00] it's all the psychotic scenarios that Fin will get put into in the real world, but we were, we want to make sure it doesn't like barf.\"}),/*#__PURE__*/e(\"p\",{children:\"So we don't do this unless we have good reason, but let's just say like GPT 4.0 drops or something like that. 4.0, the tempting thing to do. And you'll see a lot of our competitors do this. It's just, Holy shit, new model, plug it in. We're good. And I think what that says to me is it speaks to the maturity of their rollout.\"}),/*#__PURE__*/e(\"p\",{children:\"Like we also know the same API end points to ping. If you want 4.0 I think it's not, we weren't sitting here going, Oh, I wonder how do you do this? It says our approach to such a scenario is. Okay, let's, first of all, run this through the torture test. And our torture test is there to deliberately give us hard situations where we have a very clear outcome.\"}),/*#__PURE__*/e(\"p\",{children:\"So for example, let's say Tessl is using Fin. Would you want Tessl bot to recommend competitors over Tessl? Would you want it to answer a tier three question? If it didn't know the answer, how would you want it to behave? Would you want it to make up an answer? A long variety, like genuinely hundreds of scenarios where most people agree on what the right [00:12:00] outcome is?\"}),/*#__PURE__*/e(\"p\",{children:\"But like GPT 4. 0 might not agree with that. So this is where like you have to control and contain the behavior. So once it clears the torture test and we look at its performance, then we'll move to look, let's put this live. Let's see how it works in actual reality, which oftentimes is just\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"**Guy Podjarny:** In a gradual fashion, like you wouldn't turn it on for everybody.\"}),/*#__PURE__*/e(\"p\",{children:\"You would graduate. You would. Yes, absolutely.\"}),/*#__PURE__*/e(\"p\",{children:\"**Simon Maple:** That's a great segment from Des, and one of the things that I love at the end that he said was, it's super easy just to, drop in a new model into your application, but you really don't know what that means for users without that pre testing and the evaluation testing. How does that new model behave?\"}),/*#__PURE__*/e(\"p\",{children:\"How is it going to change its answers to the same user question if you change that model? Really important to understand. That changing is easy, but it's not the smart thing necessarily to do without that pretests. So thank you very much to Des. Okay, next up, we have Rishabh, the Head of AI at SourceGraph.\"}),/*#__PURE__*/e(\"p\",{children:\"Now, Rishabh is very passionate [00:13:00] about evaluations. Because he has a very deep history in machine learning. And one of the things that he talks about quite a bit in his piece here is about how you can test models with the zero to one metric. And one thing actually really strong takeaway that he leaves us with is how writing a good evaluation is actually more important than training a good model.\"}),/*#__PURE__*/e(\"p\",{children:\"Over to Rishabh.\"}),/*#__PURE__*/e(\"p\",{children:\"And testing is going to be very similar as well, right? Yeah. if someone's writing code in the, in their ID line by line, and that's maybe using a code generation or as well, like Cody, they're going to likely want to be able to have test step, step in sync. So as I write code, you're automatically generating tests that are effectively providing me with that assurance that the automatically generated code is working as I want to.\"}),/*#__PURE__*/e(\"p\",{children:\"**Rishabh Mehrotra:** Yeah. That's a great point. I think that this is more like errors multiply. Yeah. If I'm evaluating something. After long writing it, then it's worse off, right? Because the errors, I could have stopped the errors earlier [00:14:00] on and then debugged it and fix it locally and then moved on. Yeah. So especially, so.\"}),/*#__PURE__*/e(\"p\",{children:\"Taking a step back. Look, I love evaluation. I really, in machine learning, I started my PhD thinking that, hey, maths and like fancy graphical models are the way to have impact using machine learning. Right? And you spend one year in the industry realized, nah, it's not about the fancy models. It's about, do you have an evaluation?\"}),/*#__PURE__*/e(\"p\",{children:\"You have these metrics. Do you know when something is working better? Yeah. So I think getting the zero to one on evaluation on these data sets, That is really key for any machine learning problem.\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"**Simon Maple:** Yep.\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"**Rishabh Mehrotra:** Now, especially when\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"**Simon Maple:** What do you mean by the zero to one, though? Yeah,\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"**Rishabh Mehrotra:** zero to one is, look at, like, whenever a new language model gets launched, right?\"}),/*#__PURE__*/e(\"p\",{children:\"People are saying that, hey, for coding, LLM's Llama 3 does well on coding. Why? Because, oh, we have this human eval dataset and a pass at one metric. Let's unpack that. Human eval dataset is a dataset of 164 questions. Hey, write me a binary search in this code, right? So, essentially, it's like, you get a text and you write a function.\"}),/*#__PURE__*/e(\"p\",{children:\"And then you're like, Hey, does this function run correctly? So they have a unit test for that. And if it passes, then you get plus one, [00:15:00] right? So now this is great. It's a great start. But is it really how people are using Cody and a bunch of other coding tools? No, they are like, if I'm an enterprise developer, if let's say I'm in a big bank, then I have 20, 000 other peers and there are like 30, 000 repositories, right?\"}),/*#__PURE__*/e(\"p\",{children:\"I am not writing binary search independent of everything else, right? I'm working in a massive code base, which has been edited across the last 10 years. And there's some dependency by some team in Beijing. And there's a function which I haven't even read, right? And maybe it's in a language that I don't even care about.\"}),/*#__PURE__*/e(\"p\",{children:\"I understand. My point is, the evaluation which we need for these real world products is different than the benchmarks which we have in the industry, right? Now, the 0 to 1 for evaluation is that, hey, sure, let's use Passit 1 and Humeval at the start on day 0, but then we see that you improve it by 10%, we have results when we actually did improve Passit 1 by 10 15%, we tried it online on Cody users, and the metrics dropped.\"}),/*#__PURE__*/e(\"p\",{children:\"**Simon Maple:** Yeah.\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"**Rishabh Mehrotra:** And we're writing a blog post about it at offline online correlation. Yeah. Because if you trust your offline metric, pass it one, you improve it. You hope [00:16:00] that, Hey, amazing users are going to love it. Yeah.\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"**Simon Maple:** It wasn't true. The context is so different.\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"**Rishabh Mehrotra:** Yeah. The context is so different. Now this is, this means that I got to develop an evaluation for my feature.\"}),/*#__PURE__*/e(\"p\",{children:\"And I got my evaluation should represent how my actual users using this feature feel about it. Just because it's better on a metric, which is an industry benchmark doesn't mean that improving it will improve actual user experience.\"}),/*#__PURE__*/e(\"p\",{children:\"**Simon Maple:** And can that change from user to user as well? So you mentioned a bank there.\"}),/*#__PURE__*/e(\"p\",{children:\"If five other banks, is it going to be the same for them? If it's something not in the FinTech space, is it going to be different for them?\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"**Rishabh Mehrotra:** That's a great point. I think the nuance you're trying to say is that, hey, one, are you even feature aware in your evaluation? Because Parset 1 is not feature aware, right?\"}),/*#__PURE__*/e(\"p\",{children:\"**Simon Maple:** Yeah.\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"**Rishabh Mehrotra:** Parset 1 doesn't care about autocomplete or unitized generation or code fixings. I don't care what the end use case or application is. This is just the evaluation. So I think the first jump is. Have an evaluation data set, which is about your feature, right? The evaluation data set for unit test generation is going to be different than code completion.\"}),/*#__PURE__*/e(\"p\",{children:\"It's going to be different than code edits. It's going to be different than chat. So I think the zero to one we were talking about five minutes earlier, you got to do [00:17:00] zero to ones for each of these features. Yeah. And that's not easy because evaluation doesn't come naturally. Yeah. And once you have it, then the question becomes that, hey, okay, once I have it for my feature, then, hey, can I reuse it across industries?\"}),/*#__PURE__*/e(\"p\",{children:\"**Simon Maple:** Is it testing now? Is it guardrails that become the most important thing? And almost, I would say, more important than code, or is still code the thing that we need to care about the most?\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"**Rishabh Mehrotra:** Yeah. I think like, if I view this extreme of like, again, if I put my evaluation hat right, I think I want to be one of the most prominent proponents, vocal proponent of evaluation in the industry, not just in code AI, in the machine learning industry, we should do more evaluation.\"}),/*#__PURE__*/e(\"p\",{children:\"So there, I would say that writing a good evaluation is more important than writing a good model. Yeah. Writing a good evaluation is more important than writing a better context source because you don't know what's a better context source if you don't have a way to evaluate it, right? So I think for me, evaluation precedes any feature development.\"}),/*#__PURE__*/e(\"p\",{children:\"Yeah. If you don't have a way to evaluate it, then you don't, you're just shooting darts in the dark room, right? Some are going to land by luck. Now [00:18:00] there, in that world, right? I have to ensure that unit tests and evaluation is like ahead in terms of importance and just like code. That said.\"}),/*#__PURE__*/e(\"p\",{children:\"I think overall, right? What's more important is like task success, right? Which is again, what is our success? You're, you're not just looking at unit test as an evaluation. You're looking at evaluation of the overall goal, which is, Hey, do I do this task right? And then I think if that's fair as an orchestrator, if I start treating these AI agents could be Cody, autocomplete or like, like any specific standalone agent powered by SourceGraph as well, probably.\"}),/*#__PURE__*/e(\"p\",{children:\"So in those words, evaluation of that task, because you are the domain expert. Assume AGI exists today. Assume the foundation models are going to get smarter, smarter, like billions of dollars, trillions of dollars eventually into it. We train these fans, again, the smartest models and they can do everything.\"}),/*#__PURE__*/e(\"p\",{children:\"But you are best placed to understand your domain on what the goal is right now. So you are the only person who can develop that evaluation of like, how do I know that you're correct? How do I know we're 90 percent correct, 92 percent correct? And again, right, the marginal gain on 92 to 94 is going to be a lot more harder than going from 80 [00:19:00] to 90, right?\"}),/*#__PURE__*/e(\"p\",{children:\"It always gets safer. Like, I mean, there's going to be like an exponential hardness increase over there. So essentially the point then becomes purely on evaluation, purely on unit test, right? What makes us, what are the nuances of this problem, of this domain, which, the model needs to get right. And are you, are we able to articulate those and I'll be able to generate those unit tests or generate those guardrails and evaluations so that I can judge how the models are getting better on that topic.\"}),/*#__PURE__*/e(\"p\",{children:\"Right. So the models are going to be far intelligent, great. But then what is success? You as a domain expert get to define that. Yeah. And this is a great thing, not just about coding, but also like any. domain expert using machine learning or these tools across domains, you know what you're using it for, right?\"}),/*#__PURE__*/e(\"p\",{children:\"The other AGI tools are just tools to help you do that job. So I think the onus is on you to write good evaluation or even, I mean, maybe tomorrow LLM as a judge and like people are developing foundation models just for evaluation, right? So there are going to be other tools to help you do that as well.\"}),/*#__PURE__*/e(\"p\",{children:\"Code foundation models for like unit tests, maybe that's the thing in six months from now, right? the point then becomes, what should it focus on? That's the role you're playing. [00:20:00] Orchestrating, but like orchestrating on the evaluation. Oh, did you get that corner piece right? Oh, you know what? This is a criticality of the system, right?\"}),/*#__PURE__*/e(\"p\",{children:\"Again, right? The payment gateway link and the authentication link, some of these get screwed up, then massive bad things happen, right? So you know that. So I think that's where like the human in the loop and your input to the system starts getting crossed.\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"Great stuff from Rishabh. Next up we're going to be hearing from Tamar Yehoshua, who's the president of products and technology at Glean. And Glean's a really interesting company. Effectively does enterprise search across a vast number of sources, many of which have very sensitive data, including HR systems and various things within that enterprise.\"}),/*#__PURE__*/e(\"p\",{children:\"So Glean as a product allow multiple models to be used and the user can select which one they choose to use. Glean, interestingly, obviously don't have access directly into that data because that data is very sensitive to the customer. So when it comes to thinking about how Glean would test and run evaluations [00:21:00] across multiple models with data that they don't have access to, it becomes a bit of a challenge and makes evaluation testing very hard.\"}),/*#__PURE__*/e(\"p\",{children:\"Now, one of the really interesting things that was talked about in this episode was how Glean use LLMs as a judge and as a jury and it's a really interesting concept of not necessarily having employees from Glean look at the prompts and the queries that are being created and try to validate them.\"}),/*#__PURE__*/e(\"p\",{children:\"But that's actually about having an LLM do that validation. So the LLM is determining whether those queries are correct. And of course, that determinism. is very hard as well to have in that jury perhaps lessons that non determinism effect of an LLM. So this is a really another great episode.\"}),/*#__PURE__*/e(\"p\",{children:\"And so over to Guy and Tamar\"}),/*#__PURE__*/e(\"p\",{children:\"**Guy Podjarny:** So I guess let's talk, indeed, keep promising, talking about the sausage making, let's do that now. I guess, what is it like building a product that works this way? The results of the product vary by the [00:22:00] data that comes in, the success rates are very different, and that's on top of the fact that the LLMs move at such a kind of lightning pace, and so you get new models all the time, and if I understand correctly, you actually even dont pick the model.\"}),/*#__PURE__*/e(\"p\",{children:\"You have to work with models that the customer does. So maybe tell us a little bit about how do you interact with the models or just to say that, but what I'm really interested is you talked about the LLM as a judge. You had a post about this. Just how do you know that your product works?\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"**Tamar Yehoshua:** It's very difficult.\"}),/*#__PURE__*/e(\"p\",{children:\"No, it's the non deterministic aspect is the most interesting and challenging. My first week at Glean, I was talking to the head of the assistant quality team, just to learn, what do you do and how does it work? And realize that a lot of their time was spent talking to customers. Who did not have the right expectations of what the, what it could do, or we're complaining that it was non deterministic.\"}),/*#__PURE__*/e(\"p\",{children:\"I did this query once I did it again, and it didn't get the same thing. And we're starting to get used to ChatGPT and this, the [00:23:00] concept of non deterministic, but in an enterprise, you're a CIO, you buy software, you pay a lot of money for it. You expect it to have the same answer every time. And so one is getting our customers comfortable with what LLMs can do and what the boundary conditions are is part of what we have to do in our product.\"}),/*#__PURE__*/e(\"p\",{children:\"And it can't just be in the marketing and in the enablement, you have to have some way in the product to take care of this as well, to understand what are the challenges going to be for people. So I explained the RAG architecture. So we have our way of evaluating the search, which. Most of the team came from Google search ranking built eval tools, just like Google search had.\"}),/*#__PURE__*/e(\"p\",{children:\"So that kind of that's like bread and butter. Like, how do you eval search? We have a whole process for eval.\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"**Guy Podjarny:** This is not the user side. This is internal. This is to evaluate whether your search is working correctly. With information, the engineers, your engineers,\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"**Tamar Yehoshua:** this is our ranking, the ranking algorithms with a lot of [00:24:00] information on exactly what was triggered and what wasn't.\"}),/*#__PURE__*/e(\"p\",{children:\"And what scores it had. It is a little bit more difficult because at Google, you could use third party raters to say, here's a change we're making. Is this change good or bad evaluated? Because this is enterprise data, we can't use third party raters. We can only use our tools and engineers to look at the data.\"}),/*#__PURE__*/e(\"p\",{children:\"So that's a, uh, another wrinkle on top for, for enterprise. But, but going back to your question of what's different is understanding the user mindset when they're using this product, how do you help them through that to give them guardrails so they can better understand what the product can do, and then how do you evaluate it and how do you make sure that it's working as intended?\"}),/*#__PURE__*/e(\"p\",{children:\"With this new technology that nobody fully understands why it's giving the answer that it does.\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"**Guy Podjarny:** Yeah. What are some examples of things that you would do in the product to, to help, to help people?\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"**Tamar Yehoshua:** A big one is suggest one of the things about Glean is that we understand your [00:25:00] organization. So in the connectors that we do, we also have a Workday connector, Active Directory, so we understand who you are, who's, who are your peers, who's on your team, so we can suggest prompts that people in your team has been, you have been using, Oh, you're a PM. Here's a prompt from another PM that the things that they've been doing that you might want to try. And so we can suggest or generic suggestions. We've been experimenting with all different ones, but that can help guide people into areas of this is the right, this is ways that you can get value.\"}),/*#__PURE__*/e(\"p\",{children:\"And that's going to be really important. And we want to do a lot more of that. And the Glean apps was a big way of doing that as well. Here's some more structured prompts and triggers. This is where, if you have an IT question, you'll go here. If you have a, want to build a customer brief, here's a Glean app for building a customer brief.\"}),/*#__PURE__*/e(\"p\",{children:\"So that helps people go to things that somebody has curated. Somebody like the 5 percent of people in the company really [00:26:00] understand how to work with prompts and LLMs. They're going to do that and they're going to help. And we're going to be doing more and more of that. And I hope that is a way that's whole nother angle of how we are doing that.\"}),/*#__PURE__*/e(\"p\",{children:\"And then we also have just teams dedicated to eval. And understanding what changes need to be made, what don't, how to evaluate new models. Um, as you mentioned, we, customers can decide what model they want to use. We validate the model. We certify, I should say, if Gemini 1. 5 Pro comes out. So we will certify it for our customers before we enable them to use it.\"}),/*#__PURE__*/e(\"p\",{children:\"But we have let our customers pick OpenAI, Anthropic or Gemini for the LLM aspect of the work. And so that's another thing that's tricky also is working with the different models,\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"**Guy Podjarny:** but how, so I, I understand it like the suggest notion or the idea of disseminating actually probably a useful practice for any product, not a LLM specific [00:27:00] one, which is take the forerunners and provide an easy way to disseminate their sample uses to the rest of the organization. But on the other side in terms of what happens if it fails, what happens if hallucinations are a thing in this world? So ranges from it didn't okay. Maybe you're good on the search side. So it finds the relevant data.\"}),/*#__PURE__*/e(\"p\",{children:\"You feel pretty good, but didn't understand it. Did it process it correctly? Did it present it correctly? How do you evaluate it when you certify what types of? tools are at your disposal to know when you're using a new model or even just evolving your software that it got better.\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"**Tamar Yehoshua:** So first of all, in the product, customers can do a thumbs up, thumbs down.\"}),/*#__PURE__*/e(\"p\",{children:\"Obviously we get more thumbs down than thumbs up because that's just the nature of people. But we, but that's helpful until all those queries go come back to us so that we know here's the setup. of bad queries. We also evaluate things like in search, it's easy. Did somebody click on it? Did they find what they were looking for in the system?\"}),/*#__PURE__*/e(\"p\",{children:\"It's trickier because they might have gotten the answer or not gotten the answer. But for example, if they [00:28:00] try a couple of queries in the system and go to search afterwards and then find the document that they needed, we know that the assistant didn't give them the answer. So we have some, we have a metric.\"}),/*#__PURE__*/e(\"p\",{children:\"They index the satisfaction, right? Index from search and the assistant. So we look at that and we measure that very heavily of how many bad queries did we get? How many thumbs down did we get? So that's one.\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"**Guy Podjarny:** Yeah. And that's posted. Those are all things that, yes, that's the refer people using the product.\"}),/*#__PURE__*/e(\"p\",{children:\"**Tamar Yehoshua:** Right. That's the proxy for how well are we doing? And if we're doing hill climbing to improve, is it going up or going down? And then evaluating is super tricky. So we have, as you mentioned, we've started using LLM as a judge, and there are many ways that the LLM could go wrong, or the whole assistant could go wrong.\"}),/*#__PURE__*/e(\"p\",{children:\"It could pick the wrong query to send to the retrieval engine. The retrieval could not find the document, it could find the wrong one, or it could miss some something. And then in the generative, not be a complete answer. It could [00:29:00] not be grounded in the facts. It might pull in public data instead of the data that you had.\"}),/*#__PURE__*/e(\"p\",{children:\"And then, so you've got the completeness. The groundedness and the factualness. So we've been using LLMs to judge our answers for the assistant in these different areas. So completeness is one that we get the most thumbs down. If it's not a complete answer, we'll get thumbs down for it. And then we've correlated the thumbs down with LLM as a judge and the completeness.\"}),/*#__PURE__*/e(\"p\",{children:\"So that's the most easiest for an LLM to then evaluate the results of the LLM.\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"**Guy Podjarny:** Yeah.\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"**Tamar Yehoshua:** And so we have completeness. We have grounded. Did it tell you which, which context it came from? And then the factualness is the hardest and the factualness. What you need is a golden set, essentially\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"**Guy Podjarny:** being the, you did not hallucinate.\"}),/*#__PURE__*/e(\"p\",{children:\"**Tamar Yehoshua:** Grounded is more of the hallucinations. Okay. We don't have a big problem with hallucinations at gleam because of the RAG based architecture and because we do the citations. So it, the [00:30:00] groundedness is the most aligned to, um, to hallucinations, but sometimes it's not grounded in an enterprise stock because it might be like the stock price that you asked for, for a company, and it might just be public knowledge.\"}),/*#__PURE__*/e(\"p\",{children:\"The factfulness is, was it, LLMs are very confident. So they'll say with great confidence that something is correct. And then a user will not thumbs down those because they'll just assume it's correct. And those are the most dangerous. And that what we're doing is trying to actually have a golden set.\"}),/*#__PURE__*/e(\"p\",{children:\"We're having an LLM extract queries from documents and then measure the effectiveness of are we finding them. So we get the golden set. And then we're so this 1 actually, we're still working on the fact how to best measure it. But the best part that we've done now is now we have a repeatable process for LLM is judge.\"}),/*#__PURE__*/e(\"p\",{children:\"We have an eval framework of how we use, and we can just turn the crank when a new model comes out and we can get an evaluation across [00:31:00] these metrics for new models. And for changes as we're making changes in the code, we can evaluate them more easily. Is it perfect? No, but it's a lot better than, you know, engineers manually going and looking at every query and\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"**Guy Podjarny:** that golden set needs to be created per customer.\"}),/*#__PURE__*/e(\"p\",{children:\"**Tamar Yehoshua:** So we use a lot of times we use our data for some of these golden sets to make sure, but we do have in our eval, we actually run queries in our customer environments in their deployments because we can't look at their content. But we can run things and evaluate them and get the results of the evaluation.\"}),/*#__PURE__*/e(\"p\",{children:\"**Guy Podjarny:** So that's interesting. And that's a good learning. So you can't look at their data. You have to make sure that your check did not cause problems with their data or see, at least try to assess if that's the case. So what you're agreeing with them is that you'd be able to run some processes that are not actually functional or run a bunch of these tests on their platform with their data, but you won't access the data. It would just get the results of like thumbs up, thumbs down, [00:32:00] different version of the thumbs up, thumbs down to say, yeah, it feels good for you to, to deploy this, uh, this new version or to upgrade to this new model.\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"**Tamar Yehoshua:** We're very cautious of making sure we have very strict agreements of how we're handling customer data, but we absolutely run regressions and which it was a, it's an interesting process that we've gone through.\"}),/*#__PURE__*/e(\"p\",{children:\"**Guy Podjarny:** Yeah. And so we'll talk about LLM as a judge, which is the, the, the LLM looks at the answer and says the result I see in the golden set seems sufficiently familiar, sufficiently similar to the result of the product, the live product gave right now. And then you, I found interesting in the blog post that you're, you're You wrote about this, there's this notion of LLM jury, which has the risk of taking the analogy a bit too far, but I guess is that, do you want to say a couple of words about that?\"}),/*#__PURE__*/e(\"p\",{children:\"**Tamar Yehoshua:** Uh, it's exactly what it sounds like. Just multiple, you want to assess not just one voice, but multiple voices to make sure that you're aligning.\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"**Guy Podjarny:** And I guess partly a means to deal [00:33:00] with the fact that LLMs themselves, like the evaluator itself is a random or is a non deterministic entity. So oddly actually quite aligned to the reason there is a judge and jury in the actual human judicial system.\"}),/*#__PURE__*/e(\"p\",{children:\"**Rishabh Mehrotra:** And last up we have Simon Last, Notion co founder and CTO. Now Notion AI is, of course, one of the most impressive running real world applications of AI. And in the snippets that we're going to be sharing with you today, one of the interesting things here is about how Simon details how Notion effectively log failures that are occurring with exact reproduction capabilities and what this allows them to do is effectively build up a data set of failures with regressions.\"}),/*#__PURE__*/e(\"p\",{children:\"Now there's an iterative improvement loop here whereby one can collect the failures, adjust prompts, rerun the evaluations, and then validate the fixes against those evaluations. So it's a really interesting loop [00:34:00] of how there's a continuous improvement with Validation from the evals as well and of course one really important thing is there needs to be an opt in from a privacy point of view.\"}),/*#__PURE__*/e(\"p\",{children:\"It needs to be that opt in for data sharing for evaluation purposes only that allow and ensure rather that privacy and segregation from test to production data. So here's Simon, co founder of Notion and CTO\"}),/*#__PURE__*/e(\"p\",{children:\"**Guy Podjarny:** Yeah, I'm very curious to dive in because indeed some things are the same and you just want to tap into them and some things are quite substantially different. But I guess still on the organization, do you think about the team that is originally building it as a platform team? How did you consider that division now that you've been building this for a couple of years and you have, it sounds like multiple teams working it, how do you think about the shared infrastructure, if you will, between those or the shared learnings or evaluation methods as we'll touch on in a sec?\"}),/*#__PURE__*/e(\"p\",{children:\"How do you think about, I guess, enabling these other teams to adopt AI more easily?\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"**Simon Last:** Yeah, that's a good question. Yeah. Initially we didn't think of it that [00:35:00] way. It's more just like a end to end product team. The only goal is just to\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"**Guy Podjarny:** just ship them.\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"**Simon Last:** Yeah. Just to ship useful stuff. And then we're learning along the way how to do evals.\"}),/*#__PURE__*/e(\"p\",{children:\"And then our goal is to make it more of a platform team. So we're trying to take what we build and expose it as something reproducible. I would say it's actually pretty tricky because in many ways we don't really do any. Training of models. It's all about taking the best models out there and packaging them into the product and making everything work well, which involves doing logging in the right way, like setting up your prompt, doing logging in the right way, doing evals, all that stuff.\"}),/*#__PURE__*/e(\"p\",{children:\"And it's interesting. Most of it is actually, there's not that much technically challenging. I would say about the platform aspect. A lot of it is more about the best practice of how you do it and the knowledge of. What steps you take and even how to think about it. Yeah, it's a tricky thing to get other people to work on without context.\"}),/*#__PURE__*/e(\"p\",{children:\"What we've had most success with actually is when someone new wants to work on AI, we've had a lot of success with just having them join the AI team temporarily. And then they just join our standups and [00:36:00] get in the weeds with us every day. And then quickly pick up all the context around like how to do evils, how to.\"}),/*#__PURE__*/e(\"p\",{children:\"Yeah, we found it challenging the other way where we just like give them like the code pointers to all the different layers of the stack.\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"Yeah, here's a thing that would run the inference or whatever it is, but\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"Yeah, and the code isn't that complicated. I would say the complexity is actually more in like the best practice of how you do things and the mental model, I don't even think about it.\"}),/*#__PURE__*/e(\"p\",{children:\"**Guy Podjarny:** I think that makes perfect sense. I actually had Patrick Dubois was a big kind of DevOps luminary on the podcast. And we talked about how the analogies work from the DevOps era. And a lot of it is indeed about platform teams and reuse, but also about embedding and about someone's walking a mile in your shoes a little bit and learning both for empathy, as you might get annoyed later on when the product doesn't work quite as you want, but for skill sharing.\"}),/*#__PURE__*/e(\"p\",{children:\"And in security in DevSecOps, we use the same approach. I think we're running out of letters. We can add into the DevSecOps kind of combo. I think we need to find a new strategy there, but I think the approaches themselves are work quite well. I love that approach and it makes perfect sense to me. Maybe let's, indeed, [00:37:00] I want to talk a bit more about the skills and those gaps, but maybe let's first describe.\"}),/*#__PURE__*/e(\"p\",{children:\"These are what has been repeated oftentimes in this probably said most succinctly by Dez at Intercom in an early episode here is that it's really hard to know whether your product works when you're helping on AI. So I guess what has your experience been when you're building these capabilities? How do you think about evaluation, about testing, about knowing whether It hits a threshold but also knowing whether it hasn't regressed.\"}),/*#__PURE__*/e(\"p\",{children:\"**Simon Last:** Yeah, it's super hard. It's a really different experience than the pre AI world. Most of my experience is in building Notion, building products and it's been honestly really painful. It's very exciting and fun and like I'm pretty obsessed with the new technology but it's very painful and I've missed the old world.\"}),/*#__PURE__*/e(\"p\",{children:\"In the old world I could have an idea. And it might take me longer than I thought, but I could definitely ship it. But in the AI world, I can often have an idea and then be surprised that it didn't work in some way I didn't expect. I would say, I guess the way I think about it is, it's a two fold challenge of, [00:38:00] One is, for the situations that you test, you need to make sure that those work and they don't regress.\"}),/*#__PURE__*/e(\"p\",{children:\"And then there's this nebulous space of things that you haven't even tested. And it can be arbitrary, and then filling enough of that space so that you're confident that it's gonna match the distribution of what users are gonna request enough that you're reasonably confident, you can never like, like, like fully match it.\"}),/*#__PURE__*/e(\"p\",{children:\"Yeah. I guess the way I think about that is, yeah, you need like a repeatable system or engine around this. And the key pieces are something like you need, you need to set up good logging, such that interactions can be called up again. If it fails in some way, you can go look at that. And it's extremely important to be able to exactly reproduce.\"}),/*#__PURE__*/e(\"p\",{children:\"This is the failure situation, the log has to be an exact reproduction of the error. And then you should be able to just rerun that inference with the exact same input. That's really important. If you can't do that, you're totally screwed.\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"**Guy Podjarny:** And that creates probably some challenges. Like [00:39:00] I fully relate to the full logging, even more so than in regular instrumented systems.\"}),/*#__PURE__*/e(\"p\",{children:\"How do you work around the kind of the privacy concerns or some of the user data concerns around it?\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"**Simon Last:** Yeah. So, so we take that really seriously. What we do is we have an opt in early access program. Okay. So when you use Notion AI, we show you this little pop up and the default is no, but you can opt into sharing data with us and we don't use it for training.\"}),/*#__PURE__*/e(\"p\",{children:\"It's just for evaluation. So basically that allows us to see the logs. Like if you thumbs down, we can actually see the log with the input and then we can add it to an eval data set and we segregate out the prod data to make sure that it's not contaminated with other sources. But yeah, that's been extremely helpful.\"}),/*#__PURE__*/e(\"p\",{children:\"Let people opt in and our user base is large enough that even if a pretty small percentage opt in, it's still quite a lot of data.\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"**Guy Podjarny:** So logging is one.\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"**Simon Last:** Yeah, so logging is one. So yeah, I would say the next step is around collecting the failures into data sets. That's pretty important. So just like organizing them in some reasonable way such that for the given task, you have some data set or data sets that are [00:40:00] like, these are all the cases that we care about and all the known previous failures.\"}),/*#__PURE__*/e(\"p\",{children:\"And then the next bit is being able to evaluate that a given failure is successful. still happening or not. And that can be really hard. There's many ways to do evals. I think about you can have evals. Definitely best use deterministic when you can, and then you can use like a model graded eval when not.\"}),/*#__PURE__*/e(\"p\",{children:\"For model graded evals, they work best when it's as concrete and specific as possible. I've seen people fail A lot doing modigrade evals, but when it's like too generic or the task is too difficult, I think I said this in the last one, but if your modigrade eval is not like robustly easy for the model, then you have to have an eval for your eval and you just created like a new stack of problems,\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"**Guy Podjarny:** an infinite loop.\"}),/*#__PURE__*/e(\"p\",{children:\"So it relates to the, the data piece. So the data sounded like. Going further back, you have to create a set of test cases for the things you want to work. And then a set of cases for problematic ones. I guess that's synthetic. That's at the beginning. Why is your Delta feature? And then you need to have [00:41:00] access to real world scenarios.\"}),/*#__PURE__*/e(\"p\",{children:\"And you curate out of that data set of failure cases with the proper user permission, like amidst the users that opted in to help improve the system. I guess, do you accumulate over there good cases as well? Does the data set mostly grow in the negative sense of don't do this or Is it the same one? Is it basically, you're like, here's a scenario, here's the correct and here's the incorrect answer?\"}),/*#__PURE__*/e(\"p\",{children:\"**Simon Last:** I tend not to care about good cases that much. I think the, the main focus is on, cause what's the actionable value of that? Maybe it's nice to have a few of them just as a sanity check, but, but to me, the main goal is to collect concrete regressions and then fix them and then make sure that they stay fixed.\"}),/*#__PURE__*/e(\"p\",{children:\"And then over time you're ratcheting up. All these things that used to not work are now working, and then you're growing a set of things that continue to work, and that's ideal. You'll often find issues that you just can't really fix, and so, that's the, that's part of the\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"**Guy Podjarny:** Yeah, we'll come back to that.\"}),/*#__PURE__*/e(\"p\",{children:\"That's really interesting, because, you know, a product works at this percentage of Yeah,\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"**Simon Last:** models have limits. There's many issues that we can [00:42:00] fix, and we're not a foundation model company. It's like, alright, we'll just wait until the next model comes out, and hopefully it'll fix this one. So you want some kind of It could be just like a, like a human grade eval, although that can, that'll quickly get out of control.\"}),/*#__PURE__*/e(\"p\",{children:\"Then the really key next thing is you need to enter this loop where you find some new regression, you add it to your dataset. Then you change the prompt in some way. Then you rerun the example to the dataset. And then you need to be able to decide whether it's, like, better or worse. And ideally that's as automated as possible.\"}),/*#__PURE__*/e(\"p\",{children:\"And then ship that change. And then there's just this constant loop of new example, improve the prompt.\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"**Guy Podjarny:** And I think on the eval side, Notion is pretty open or flexible in what you can do with it. There are a lot of degrees of freedom of it. It's easy to think about non deterministic evaluations that you can have.\"}),/*#__PURE__*/e(\"p\",{children:\"Notion is, you know, you're looking at text and unstructured data and the likes. What's an example of a structured evaluation that you can do in a Notion AI capability?\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"**Simon Last:** Yeah, a lot of them, like, anything to do with formatting the output in some way. [00:43:00] Maybe it's XML, maybe it's JSON. That's ideal. That's ideal.\"}),/*#__PURE__*/e(\"p\",{children:\"I think about if you're having some issue, the best way to solve it is by solving it with a validator. Make it so that the invalid output is impossible by structuring the output in some way where you can deterministically say that the bad output is incorrect. That's like the best way to solve any problem.\"}),/*#__PURE__*/e(\"p\",{children:\"And it's the easiest way to evaluate it. Another. Deterministic eval that we use all the time is, I love using many classifiers as, as part of the flow. Classifiers are really great. I, by classifier, I just mean some inference that outputs like an enum, like, like set of possible values. Classifiers are really great.\"}),/*#__PURE__*/e(\"p\",{children:\"Cause they're really easy to eval. Uh, because the output. You\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"**Guy Podjarny:** classify the result of the, of the product's activity. Like you classify the output from the LLM.\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"**Simon Last:** No, actually as part of the flow. So. For example, one thing we need to do as part of the chat experience is decide whether to search or not, just as an example.\"}),/*#__PURE__*/e(\"p\",{children:\"So it's, should I search or not search? Like, yes or no, basically. These are the classifiers that have more than two possible values, but yeah, those are really great [00:44:00] in large part because they're so easy to evaluate. Because you just have this grown truth output, and then you just compare the actual versus expected.\"}),/*#__PURE__*/e(\"p\",{children:\"It's really great for that reason. You can collect a big data set. And you can get a score.\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"**Guy Podjarny:** And I guess what has been the driver behind the choice not to fine tune, or actually, you know what, I might be jumping to conclusions here. You said you're not training foundation models of it. Do you find yourself fine tuning the model?\"}),/*#__PURE__*/e(\"p\",{children:\"**Simon Last:** Yeah, so we've tried fine tuning extensively. I've personally banged my head on it a lot. I think I need to ask OpenAI, but I feel like I'm, if not the highest, maybe like 99th percentile, like number of fine tunes created. Yeah. We try fine tuning quite a lot. I would say, yeah, the big issue with fine tuning is that you're making your job like a hundred times harder at least because I need to collect this whole data set, the actual fine tuning process can be difficult.\"}),/*#__PURE__*/e(\"p\",{children:\"Especially if you're using a lot slower, you have to wait for things to train. And then, it's really hard to debug issues. You [00:45:00] basically, like, create this black box where any of your new examples, as you do successive fine tuning runs, any new examples can poison the entire dataset. You can mess up the model.\"}),/*#__PURE__*/e(\"p\",{children:\"So, it really requires you to have extremely good evaluations, and it's really hard to debug issues. So, I've had issues with fine tuning where, you know, I literally would spend weeks trying to figure out What the problem was and just extremely hard. Yeah, I would say I'm honestly not that bullish on Companies outside of the foundation model companies doing fine tuning and if I like meet a startup and they say they're doing fine tuning I actually now think of that as like a negative update It's a lack of\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"**Guy Podjarny:** experience you buy into the promise, but it doesn't actually Yeah, it sounds cool.\"}),/*#__PURE__*/e(\"p\",{children:\"**Simon Last:** And I got bit by this bug too. Like, it sounds so cool. And like, I think, as engineers, we want to have more control. We want to do a technically cool thing. Um, you know, the technically, like, powerful thing. And I was totally susceptible to that. And it was really fun to do it. Um, but, I don't know. I'm a huge fan of in context learning.\"}),/*#__PURE__*/e(\"p\",{children:\"[00:46:00] Like, it's gonna get better. Yeah, there's a lot of reasons to do it. Another big reason not to fine tune is, if you're not a foundation model company, you really want to just be using the best model. And the progress is so fast. And so if something doesn't work now, there's a good chance it's going to work in the near future.\"}),/*#__PURE__*/e(\"p\",{children:\"And if you're fine tuning, you're really just locking yourself in to this process. Slow, complicated process that makes it hard to update. And context learning is amazing because if it's a new model, I can switch the next day.\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"**Guy Podjarny:** And I guess in that case, that was my next question is like, how often do you try different models and how rigorous a testing process do you feel is necessary to make that switch?\"}),/*#__PURE__*/e(\"p\",{children:\"Because those are probably new features or new features. You control the timeline. A new model comes along. I guess you still control the timeline of how quickly you, you roll it out. How do you think about the trade off between wanting to tap into the new hotness versus, I don't know, potentially slow, or how confident do you feel about, Hey, like the next one came out.\"}),/*#__PURE__*/e(\"p\",{children:\"Can I sort of slot that in and get it out within the week?\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"**Simon Last:** Yeah, that's a good question. I would say the speed at which you can do it is really [00:47:00] dependent on the quality of your evals. How good do your evals, like, how good are they at telling you confidently that this change will improve the overall experience?\"}),/*#__PURE__*/e(\"p\",{children:\"Yeah. I would say that's one of the reasons why. Evals are so important. Probably changing prompts is more important than changing models. That's going to happen much more often. Uh, but the same evals work for both. Right. Um, but yeah, we try to, yeah, we've definitely invested a lot in really making sure the tools are right.\"}),/*#__PURE__*/e(\"p\",{children:\"And, and the evals are, are, are really good, but also I would say as a products company, I think it's pretty important not to be just like jerked around constantly by new models coming out. I think it's important to like, think about like. Really understanding the task that you uniquely really care about a product and your alpha should be really deeply understand the task and what it means to produce a good result and the product experience around that and Like watching new capabilities and releasing them as they [00:48:00] come out.\"}),/*#__PURE__*/e(\"p\",{children:\"I think it's really about you and your product and like thinking about the models as like a means to an end to, to enable that. And very often a new model, like just, it's like shiny and cool, but like maybe it just doesn't really get you much benefit for the tasks that you care about.\"}),/*#__PURE__*/e(\"p\",{children:\"**Guy Podjarny:** Yeah. I'm curious now. What's your favorite example? Is it just that? Oh, it's an app. Yeah. So it's a. It's a pretty simple app, but it touches all the pieces that, uh, that I think are interesting. So it's a calorie tracking app. All it is, it's a table with the food and the amount of calories, and then there's an input.\"}),/*#__PURE__*/e(\"p\",{children:\"And you can just type whatever you want, and then it's supposed to extract out each food item and the calories added to the table. So it involves a database, a backend, a frontend, and then it also has to write a prompt and call into a language model to do the extraction. And last time I tried, it actually did get the initial version.\"}),/*#__PURE__*/e(\"p\",{children:\"And then I asked for just one small feature change and then it didn't work from there. They've made huge improvements and it's like super impressive. But yeah, I feel like we're not quite there yet still.\"}),/*#__PURE__*/e(\"p\",{children:\"**Simon Maple:** And a massive thank you to Simon there. And of course, [00:49:00] Tamar, Rishabh. and Des at the very start of the episode and, of course my co host Guypo for helping host many of those sessions this has been a very interesting episode for me personally, just because it's been really nice going through some of the past episodes and also, picking out some of those learnings that are consistent and common from many leaders who have kindly given the time to share their thoughts and learnings with us.\"}),/*#__PURE__*/e(\"p\",{children:\"Yeah let us know what you think. If there's other topics that you feel are common across episodes, let us know and we'll create some further mashups of them till then. Thanks very much for listening again and tune into the next episode.\"}),/*#__PURE__*/e(\"p\",{children:\"**Simon Maple:** Thanks for tuning in. Join us next time on the AI Native Dev brought to you by Tessl.\"})]});export const richText5=/*#__PURE__*/t(n.Fragment,{children:[/*#__PURE__*/e(\"h4\",{children:/*#__PURE__*/e(\"strong\",{children:\"The Role of Generative AI in DevOps\"})}),/*#__PURE__*/e(\"p\",{children:'Generative AI has become a game-changer in the DevOps landscape, accelerating code generation and enhancing development workflows. As Roxane Fischer mentions, \"Gen AI is great and aids all of our lives like 10x, 100x faster.\" However, the rapid creation of code also brings potential pitfalls, such as increased legacy code and the need for precise code reviews. In the podcast, Roxane emphasizes the importance of providing precise instructions and context to AI systems: \"Garbage in, garbage out. What do you put into your prompt?\" This highlights the necessity for developers to carefully craft their inputs to maximize AI\\'s effectiveness.'}),/*#__PURE__*/e(\"p\",{children:\"Developers must be aware that while generative AI can significantly speed up the coding process, it doesn't replace the need for critical thinking and context understanding. The AI can produce code snippets or even entire modules quickly, but without the right context, it can introduce errors or inefficiencies. For instance, a developer might use AI to generate a function for handling user authentication, but if the prompt doesn't specify certain security protocols or business logic, the resulting code might be insecure or inappropriate for the application.\"}),/*#__PURE__*/e(\"p\",{children:\"Moreover, the automation of code generation raises questions about code ownership and accountability. With AI generating large portions of code, developers might struggle to maintain a sense of ownership over their work. This can lead to challenges in debugging and maintaining code, as developers may not fully understand the AI-generated portions. It's crucial for teams to establish processes for reviewing and validating AI-generated code to ensure it meets their quality standards and business requirements.\"}),/*#__PURE__*/e(\"h4\",{children:/*#__PURE__*/e(\"strong\",{children:\"Challenges with AI in Infrastructure as Code\"})}),/*#__PURE__*/e(\"p\",{children:\"The integration of AI in infrastructure as code (IaC) presents unique challenges. Roxane points out the exponential increase in code, which can lead to faster creation of legacy code and a reduced sense of code ownership among developers. She explains that infrastructure code testing is more complex than application code testing, requiring multiple layers of testing. Roxane warns that without proper context, AI-generated code can lead to issues such as hard-coded values and inconsistent metadata, which may cause future outages.\"}),/*#__PURE__*/e(\"p\",{children:\"In the context of IaC, the complexity arises because infrastructure configurations are often less visible than application code, making it harder to spot errors. For example, an AI model might generate a Terraform script to set up a network configuration. If the script includes hard-coded IP addresses or lacks necessary tags for resource management, it could lead to misconfigurations that are difficult to diagnose and correct.\"}),/*#__PURE__*/e(\"p\",{children:\"To mitigate such risks, organizations should implement robust testing frameworks for IaC, similar to those used for application code. This includes automated testing for syntax errors, policy violations, and security issues. Additionally, incorporating AI-driven tools that specialize in IaC validation can help identify potential problems early in the development cycle, ensuring that the infrastructure remains secure and reliable.\"}),/*#__PURE__*/e(\"h4\",{children:/*#__PURE__*/e(\"strong\",{children:\"Synthesis AI vs. Generative AI\"})}),/*#__PURE__*/e(\"p\",{children:'Roxane introduces the concept of Synthesis AI, which focuses on analyzing logs and metadata to provide insights. Unlike generative AI, which creates content, Synthesis AI is more mature and framed. Roxane explains, \"Synthesis AI is something that we believe is more mature in terms of taking a lot of information and finding the patterns.\" This distinction underscores the complementary roles of Synthesis and generative AI in DevOps workflows.'}),/*#__PURE__*/e(\"p\",{children:\"Synthesis AI excels in environments where massive amounts of data need to be analyzed to discern patterns or gain insights. For instance, in a DevOps setting, Synthesis AI could be used to parse logs from various systems to identify trends, anomalies, or potential issues before they escalate. This capability is particularly valuable for root cause analysis, enabling teams to quickly pinpoint the source of problems in complex systems.\"}),/*#__PURE__*/e(\"p\",{children:\"By combining Synthesis AI with generative AI, DevOps teams can create a more holistic approach to infrastructure management. Synthesis AI can identify areas of concern or inefficiency, while generative AI can propose solutions or optimizations. This dual approach allows organizations to not only detect issues but also implement improvements proactively, enhancing overall system performance and reliability.\"}),/*#__PURE__*/e(\"h4\",{children:/*#__PURE__*/e(\"strong\",{children:\"Large Language Models and Data Requirements\"})}),/*#__PURE__*/e(\"p\",{children:'Large Language Models (LLMs) are trained with extensive datasets, but this poses challenges for IaC. Roxane highlights the reluctance to share infrastructure code openly due to its sensitivity, resulting in data scarcity that affects AI performance. She notes, \"One of the issues is that nobody really wants to put the infra on clear on GitHub. It\\'s too sensitive.\" This scarcity limits the effectiveness of AI models in generating accurate infrastructure code.'}),/*#__PURE__*/e(\"p\",{children:\"The sensitivity of infrastructure code stems from the fact that it often contains details about network configurations, security settings, and other critical information. This makes organizations hesitant to share such code publicly, limiting the data available for training AI models. As a result, AI models may not be as effective in generating or optimizing IaC compared to other types of code, such as application logic.\"}),/*#__PURE__*/e(\"p\",{children:\"To address this challenge, organizations can explore secure data-sharing arrangements that allow them to contribute anonymized or synthetic data to AI training efforts. Additionally, leveraging AI techniques that require less data or can learn from synthetic data can help bridge the gap, enabling more effective AI-driven IaC solutions.\"}),/*#__PURE__*/e(\"h4\",{children:/*#__PURE__*/e(\"strong\",{children:\"Determinism in AI and DevOps\"})}),/*#__PURE__*/e(\"p\",{children:'The concept of determinism in AI outputs is crucial for infrastructure management. Roxane explains that deterministic graphs represent cloud resources and their connections, providing a stable context for AI models. She states, \"Your infrastructure is a graph...you need to have this context, this deterministic context, about your own infra.\" The balance between deterministic data and probabilistic AI models is essential for generating reliable content.'}),/*#__PURE__*/e(\"p\",{children:\"Deterministic approaches in AI help ensure consistent and predictable outcomes, which is vital for managing complex infrastructure systems. By representing infrastructure as a graph, organizations can map out dependencies and connections between resources, creating a clear picture of their environment. This deterministic model serves as a foundation for AI-driven analysis and optimization, allowing teams to understand the impact of changes and make informed decisions.\"}),/*#__PURE__*/e(\"p\",{children:\"Incorporating determinism into AI models also helps mitigate the risks associated with probabilistic outputs. By grounding AI-generated insights in a deterministic framework, organizations can ensure that the recommendations and actions proposed by AI are relevant and accurate, reducing the likelihood of errors or unexpected outcomes.\"}),/*#__PURE__*/e(\"h4\",{children:/*#__PURE__*/e(\"strong\",{children:\"Real-World Applications and Tools\"})}),/*#__PURE__*/e(\"p\",{children:\"AI's application in root cause analysis and log triage is transforming DevOps practices. Roxane mentions tools like cleric.io and Datadog, which leverage AI for efficient log analysis. Integrating AI into existing DevOps pipelines enhances efficiency and accuracy, as AI models can quickly sift through extensive datasets to identify issues and correlations.\"}),/*#__PURE__*/e(\"p\",{children:\"These tools use AI to automate the process of analyzing logs and identifying patterns that might indicate issues or inefficiencies. For instance, Datadog uses machine learning to detect anomalies in log data, alerting teams to potential problems before they impact system performance. Similarly, cleric.io uses AI to correlate log data from different sources, providing a comprehensive view of system health and helping teams quickly identify the root cause of issues.\"}),/*#__PURE__*/e(\"p\",{children:\"By incorporating these tools into their workflows, DevOps teams can reduce the time and effort required for manual log analysis, allowing them to focus on higher-value tasks such as optimizing performance and enhancing user experience.\"}),/*#__PURE__*/e(\"h4\",{children:/*#__PURE__*/e(\"strong\",{children:\"Future Trends and the Role of Anyshift\"})}),/*#__PURE__*/e(\"p\",{children:'Looking ahead, Roxane envisions a future where deterministic and AI-driven processes are seamlessly integrated. Anyshift\\'s mission is to enhance visibility for SRE teams through a digital twin of infrastructure. Roxane explains that Anyshift aims to \"give back some visibility to SRE teams to answer actually key questions.\" AI plays an educational role in explaining complex infrastructure changes, bridging the gap between development and operations.'}),/*#__PURE__*/e(\"p\",{children:\"The concept of a digital twin involves creating a virtual representation of an organization's infrastructure, complete with all its dependencies and configurations. This digital twin serves as a dynamic model that can be analyzed and optimized using AI, allowing teams to simulate changes and assess their impact before implementing them in the real world.\"}),/*#__PURE__*/e(\"p\",{children:\"By leveraging the digital twin approach, Anyshift aims to provide SRE teams with the insights they need to manage infrastructure more effectively, reducing the risk of outages and improving overall system performance. The integration of AI into this model enhances its capabilities, allowing teams to make data-driven decisions and implement best practices with greater confidence.\"})]});export const richText6=/*#__PURE__*/e(n.Fragment,{children:/*#__PURE__*/t(\"p\",{children:[\"Roxane's \",/*#__PURE__*/e(o,{href:\"https://www.linkedin.com/in/roxane-fischer-92a52414b/\",motionChild:!0,nodeId:\"X7VlByTfx\",openInNewTab:!0,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(a.a,{children:\"Linkedin\"})})]})});export const richText7=/*#__PURE__*/t(n.Fragment,{children:[/*#__PURE__*/e(\"p\",{children:\"[00:00:00] Introduction\"}),/*#__PURE__*/e(\"p\",{children:\"[00:01:00] Roxane's Background\"}),/*#__PURE__*/e(\"p\",{children:\"[00:03:00] Role of Generative AI in DevOps\"}),/*#__PURE__*/e(\"p\",{children:\"[00:06:00] Infrastructure as Code Challenges\"}),/*#__PURE__*/e(\"p\",{children:\"[00:09:00] Synthesis AI vs. Generative AI\"}),/*#__PURE__*/e(\"p\",{children:\"[00:12:00] Large Language Models and Data Scarcity\"}),/*#__PURE__*/e(\"p\",{children:\"[00:15:00] Determinism and AI\"}),/*#__PURE__*/e(\"p\",{children:\"[00:18:00] Real-World Applications and Tools\"}),/*#__PURE__*/e(\"p\",{children:\"[00:20:00] Future Trends and Anyshift's Role\"}),/*#__PURE__*/e(\"p\",{children:\"[00:23:00] Conclusion and Call to Action\"})]});\nexport const __FramerMetadata__ = {\"exports\":{\"richText\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"richText2\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"richText6\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"richText3\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"richText7\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"richText5\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"richText4\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"richText1\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"__FramerMetadata__\":{\"type\":\"variable\"}}}"],
  "mappings": "iOAAsJ,IAAMA,EAAsBC,EAAIC,EAAS,CAAC,SAAS,CAAcD,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,iBAAiB,CAAC,EAAE,uUAAuU,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,cAAc,CAAC,EAAE,oEAAoE,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mXAAmX,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qSAAqS,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6RAA6R,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,iBAAiB,CAAC,EAAE,4JAA4J,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2SAA2S,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gIAAgI,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,cAAc,CAAC,EAAE,kJAAkJ,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8BAA8B,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,iBAAiB,CAAC,EAAE,8WAA8W,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8XAA8X,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uJAAuJ,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,cAAc,CAAC,EAAE,qDAAqD,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,iBAAiB,CAAC,EAAE,4KAA4K,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+fAA+f,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,cAAc,CAAC,EAAE,8TAA8T,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uTAAuT,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wPAAwP,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0QAA0Q,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gbAAgb,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,iBAAiB,CAAC,EAAE,6VAA6V,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kXAAkX,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+TAA+T,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4NAA4N,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,cAAc,CAAC,EAAE,uFAAuF,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0TAA0T,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4SAA4S,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qGAAqG,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,iBAAiB,CAAC,EAAE,0QAA0Q,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6TAA6T,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wSAAwS,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mUAAmU,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0cAA0c,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2HAA2H,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,cAAc,CAAC,EAAE,qEAAqE,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6KAA6K,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+SAA+S,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4RAA4R,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,iBAAiB,CAAC,EAAE,6FAA6F,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,cAAc,CAAC,EAAE,mFAAmF,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,iBAAiB,CAAC,EAAE,yJAAyJ,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oWAAoW,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6VAA6V,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wcAAwc,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+FAA+F,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,cAAc,CAAC,EAAE,oLAAoL,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iTAAiT,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+SAA+S,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,iBAAiB,CAAC,EAAE,4FAA4F,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2XAA2X,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,cAAc,CAAC,EAAE,WAAW,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,iBAAiB,CAAC,EAAE,gOAAgO,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,cAAc,CAAC,EAAE,oOAAoO,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qqBAAqqB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wMAAwM,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,iBAAiB,CAAC,EAAE,2ZAA2Z,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oJAAoJ,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,cAAc,CAAC,EAAE,oNAAoN,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,iBAAiB,CAAC,EAAE,8TAA8T,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gSAAgS,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gTAAgT,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kEAAkE,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,cAAc,CAAC,EAAE,qVAAqV,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,iBAAiB,CAAC,EAAE,iYAAiY,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8UAA8U,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gVAAgV,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,cAAc,CAAC,EAAE,ocAAoc,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,iBAAiB,CAAC,EAAE,6cAA6c,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4MAA4M,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,cAAc,CAAC,EAAE,+KAA+K,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yEAAyE,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,iBAAiB,CAAC,EAAE,omBAAomB,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gUAAgU,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yZAAyZ,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0KAA0K,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,cAAc,CAAC,EAAE,WAAW,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,iBAAiB,CAAC,EAAE,4QAA4Q,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uVAAuV,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mVAAmV,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2SAA2S,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0IAA0I,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,cAAc,CAAC,EAAE,wKAAwK,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0NAA0N,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,iBAAiB,CAAC,EAAE,iHAAiH,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wfAAwf,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wWAAwW,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qHAAqH,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,cAAc,CAAC,EAAE,WAAW,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,iBAAiB,CAAC,EAAE,yCAAyC,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,cAAc,CAAC,EAAE,sJAAsJ,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wUAAwU,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2XAA2X,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iDAAiD,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,iBAAiB,CAAC,EAAE,6JAA6J,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,cAAc,CAAC,EAAE,sHAAsH,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,iBAAiB,CAAC,EAAE,sGAAsG,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,cAAc,CAAC,EAAE,YAAY,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,iBAAiB,CAAC,EAAE,gJAAgJ,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,cAAc,CAAC,EAAE,2DAA2D,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,iBAAiB,CAAC,EAAE,yIAAyI,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,ySAAyS,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8RAA8R,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8SAA8S,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4BAA4B,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,cAAc,CAAC,EAAE,oEAAoE,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,iBAAiB,CAAC,EAAE,+LAA+L,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2PAA2P,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,cAAc,CAAC,EAAE,WAAW,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,iBAAiB,CAAC,EAAE,mFAAmF,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,cAAc,CAAC,EAAE,WAAW,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,iBAAiB,CAAC,EAAE,0IAA0I,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,cAAc,CAAC,EAAE,WAAW,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,iBAAiB,CAAC,EAAE,uCAAuC,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,cAAc,CAAC,EAAE,WAAW,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,iBAAiB,CAAC,EAAE,sGAAsG,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,cAAc,CAAC,EAAE,wEAAwE,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yBAAyB,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,iBAAiB,CAAC,EAAE,2SAA2S,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0SAA0S,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kVAAkV,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+TAA+T,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,cAAc,CAAC,EAAE,kbAAkb,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,iBAAiB,CAAC,EAAE,6CAA6C,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6GAA6G,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,cAAc,CAAC,EAAE,mWAAmW,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qQAAqQ,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,iBAAiB,CAAC,EAAE,2CAA2C,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oVAAoV,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,cAAc,CAAC,EAAE,WAAW,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,iBAAiB,CAAC,EAAE,+SAA+S,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,cAAc,CAAC,EAAE,2BAA2B,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0DAA0D,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,iBAAiB,CAAC,EAAE,uPAAuP,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0XAA0X,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0VAA0V,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,cAAc,CAAC,EAAE,WAAW,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,iBAAiB,CAAC,EAAE,mWAAmW,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2WAA2W,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4GAA4G,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,cAAc,CAAC,EAAE,yPAAyP,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,ofAAof,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qMAAqM,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,iBAAiB,CAAC,EAAE,iHAAiH,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4SAA4S,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,ubAAub,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,cAAc,CAAC,EAAE,2VAA2V,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4VAA4V,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,iBAAiB,CAAC,EAAE,sFAAsF,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,cAAc,CAAC,EAAE,WAAW,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,iBAAiB,CAAC,EAAE,8IAA8I,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+SAA+S,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wYAAwY,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uGAAuG,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,cAAc,CAAC,EAAE,sGAAsG,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,iBAAiB,CAAC,EAAE,4FAA4F,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sTAAsT,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0VAA0V,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,cAAc,CAAC,EAAE,kOAAkO,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,iBAAiB,CAAC,EAAE,mEAAmE,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qUAAqU,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oWAAoW,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gJAAgJ,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,cAAc,CAAC,EAAE,mKAAmK,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wKAAwK,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sDAAsD,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,iBAAiB,CAAC,EAAE,yHAAyH,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,cAAc,CAAC,EAAE,wMAAwM,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kGAAkG,CAAC,CAAC,CAAC,CAAC,EAAeC,EAAuBH,EAAIC,EAAS,CAAC,SAAS,CAAcC,EAAE,KAAK,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,qDAAqD,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+pBAA+pB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0oBAAsoB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,keAAke,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,wDAAwD,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+fAA+f,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0mBAA0mB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qaAAqa,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,kDAAkD,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6dAA6d,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0dAA0d,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,ycAAyc,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,gDAAgD,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sbAAsb,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yfAAyf,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8cAA8c,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,gDAAgD,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4XAA4X,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kdAAmd,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qWAAqW,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,qDAAqD,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uZAAuZ,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2XAA2X,CAAC,CAAC,CAAC,CAAC,EAAeE,EAAuBJ,EAAIC,EAAS,CAAC,SAAS,CAAcC,EAAE,IAAI,CAAC,SAAsBA,EAAEG,EAAE,CAAC,KAAK,4BAA4B,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBH,EAAEI,EAAE,EAAE,CAAC,SAAS,UAAU,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeJ,EAAE,IAAI,CAAC,SAAsBA,EAAEG,EAAE,CAAC,KAAK,2BAA2B,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBH,EAAEI,EAAE,EAAE,CAAC,SAAS,aAAa,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeJ,EAAE,IAAI,CAAC,SAAsBA,EAAEG,EAAE,CAAC,KAAK,yBAAyB,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBH,EAAEI,EAAE,EAAE,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeJ,EAAE,IAAI,CAAC,SAAsBA,EAAEG,EAAE,CAAC,KAAK,yBAAyB,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBH,EAAEI,EAAE,EAAE,CAAC,SAAS,QAAQ,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeC,EAAuBL,EAAID,EAAS,CAAC,SAAsBD,EAAE,IAAI,CAAC,SAAS,CAAC,yCAAsDE,EAAE,KAAK,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,CAAC,EAAE,mDAAgEA,EAAE,KAAK,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,CAAC,EAAE,8DAA2EA,EAAE,KAAK,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,CAAC,EAAE,4DAAyEA,EAAE,KAAK,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,CAAC,EAAE,4DAAyEA,EAAE,KAAK,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,CAAC,EAAE,wCAAwC,CAAC,CAAC,CAAC,CAAC,EAAeM,EAAuBR,EAAIC,EAAS,CAAC,SAAS,CAAcC,EAAE,IAAI,CAAC,SAAS,+FAA+F,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,goCAAgoC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gnBAAgnB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sUAAsU,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gSAAgS,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qCAAqC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mUAAmU,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+WAA+W,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2TAA2T,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oWAAoW,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wQAAwQ,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6BAA6B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8LAA8L,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8UAA8U,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qTAAqT,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2uBAA2uB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0WAA0W,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gvBAAgvB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4EAA4E,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iLAAiL,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sIAAsI,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kUAAkU,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2IAA2I,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iTAAiT,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yLAAyL,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wOAAwO,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mZAAmZ,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kTAAkT,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yYAAyY,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kVAAkV,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6BAA6B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mWAAmW,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,ubAAub,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8XAA8X,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0NAA0N,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oMAAoM,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2ZAA2Z,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mLAAmL,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4SAA4S,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wUAAwU,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0WAA0W,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6XAA6X,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0SAA0S,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qFAAqF,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iDAAiD,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+TAA+T,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sTAAsT,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0ZAA0Z,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kBAAkB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mbAAmb,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uVAAuV,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gVAAgV,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2MAA2M,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2BAA2B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gDAAgD,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yEAAyE,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0GAA0G,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sVAAsV,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,ubAAub,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oUAAoU,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+aAA+a,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4BAA4B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uPAAuP,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mEAAmE,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sIAAsI,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yOAAyO,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gGAAgG,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iJAAiJ,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qMAAqM,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4BAA4B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0XAA0X,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,obAAob,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mNAAmN,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mTAAmT,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+VAA+V,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mTAAmT,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,odAAod,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wTAAwT,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kXAAkX,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0fAA0f,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4TAA4T,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kTAAkT,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gWAAgW,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wQAAwQ,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iWAAiW,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6cAA6c,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2SAA2S,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uSAAuS,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8BAA8B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qeAAqe,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uSAAuS,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0CAA0C,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qZAAqZ,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0cAA0c,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4XAA4X,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mHAAmH,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mLAAmL,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qJAAqJ,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0TAA0T,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mYAAmY,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qGAAqG,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4HAA4H,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0qBAA0qB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sVAAsV,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uWAAuW,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iXAAiX,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yLAAyL,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,ihBAAihB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+RAA+R,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iGAAiG,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yVAAyV,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iUAAiU,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sNAAsN,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uHAAuH,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sVAAsV,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gVAAgV,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qXAAqX,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oFAAoF,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6BAA6B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mOAAmO,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uDAAuD,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+bAA+b,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gTAAgT,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gUAAgU,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6XAA6X,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qEAAqE,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uUAAuU,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iqBAAiqB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uOAAuO,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,ygBAAygB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4KAA4K,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0RAA0R,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+eAA+e,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sZAAsZ,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gNAAgN,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,olBAAolB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0FAA0F,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sLAAsL,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uCAAuC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yGAAyG,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gfAAgf,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sVAAsV,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yUAAyU,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+IAA+I,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8EAA8E,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0LAA0L,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+dAA+d,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uaAAua,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kbAAkb,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6UAA6U,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4aAA4a,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qUAAqU,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6VAA6V,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qPAAqP,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qKAAqK,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0GAA0G,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sRAAsR,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+TAA+T,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wIAAwI,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0CAA0C,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wXAAwX,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mTAAmT,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oZAAoZ,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qCAAqC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8VAA8V,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kZAAkZ,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wUAAwU,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uRAAuR,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kDAAkD,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+FAA+F,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uWAAuW,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2UAA2U,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6GAA6G,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sOAAsO,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8KAA8K,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2KAA2K,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oTAAoT,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iUAAiU,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oEAAoE,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yHAAyH,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mLAAmL,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4UAA4U,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iGAAiG,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kQAAkQ,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6eAA6e,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oUAAoU,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,ogBAAogB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sGAAsG,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0WAA0W,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qVAAqV,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wOAAwO,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uMAAuM,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uXAAuX,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gEAAgE,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0RAA0R,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2UAA2U,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8hBAA8hB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gSAAgS,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wVAAwV,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kVAAkV,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8MAA8M,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,ygBAAygB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8OAA8O,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wGAAwG,CAAC,CAAC,CAAC,CAAC,EAAeO,EAAuBT,EAAIC,EAAS,CAAC,SAAS,CAAcC,EAAE,KAAK,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,qCAAqC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,ooBAAqoB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qjBAAqjB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kgBAAkgB,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,8CAA8C,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uhBAAuhB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gbAAgb,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mbAAmb,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,gCAAgC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8bAA8b,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,ubAAub,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2ZAA2Z,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,6CAA6C,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+cAAgd,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0aAA0a,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mVAAmV,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,8BAA8B,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0cAA0c,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0dAA0d,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kVAAkV,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,mCAAmC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wWAAwW,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sdAAsd,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6OAA6O,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,wCAAwC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,scAAuc,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sWAAsW,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+XAA+X,CAAC,CAAC,CAAC,CAAC,EAAeQ,EAAuBR,EAAID,EAAS,CAAC,SAAsBD,EAAE,IAAI,CAAC,SAAS,CAAC,YAAyBE,EAAEG,EAAE,CAAC,KAAK,wDAAwD,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBH,EAAEI,EAAE,EAAE,CAAC,SAAS,UAAU,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeK,EAAuBX,EAAIC,EAAS,CAAC,SAAS,CAAcC,EAAE,IAAI,CAAC,SAAS,yBAAyB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gCAAgC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4CAA4C,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8CAA8C,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2CAA2C,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oDAAoD,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+BAA+B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8CAA8C,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8CAA8C,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0CAA0C,CAAC,CAAC,CAAC,CAAC,EACp2nIU,EAAqB,CAAC,QAAU,CAAC,SAAW,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,UAAY,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,UAAY,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,UAAY,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,UAAY,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,UAAY,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,UAAY,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,UAAY,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,mBAAqB,CAAC,KAAO,UAAU,CAAC,CAAC",
  "names": ["richText", "u", "x", "p", "richText1", "richText2", "Link", "motion", "richText3", "richText4", "richText5", "richText6", "richText7", "__FramerMetadata__"]
}