{
  "version": 3,
  "sources": ["ssg:https://framerusercontent.com/modules/xAObgfFM0p6oiVNyEANZ/F1snlFluanWbwDs0GJiV/LYxDZg6nX-3.js"],
  "sourcesContent": ["import{jsx as e,jsxs as t}from\"react/jsx-runtime\";import{ComponentPresetsConsumer as n,Link as a}from\"framer\";import{motion as o}from\"framer-motion\";import*as i from\"react\";import r from\"https://framerusercontent.com/modules/pVk4QsoHxASnVtUBp6jr/TbhpORLndv1iOkZzyo83/CodeBlock.js\";export const richText=/*#__PURE__*/t(i.Fragment,{children:[/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"em\",{children:\"This post is written in collaboration with the CEO of \"}),/*#__PURE__*/e(a,{href:\"https://holidayhero.com/\",motionChild:!0,nodeId:\"LYxDZg6nX\",openInNewTab:!0,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(o.a,{children:/*#__PURE__*/e(\"em\",{children:\"HolidayHero\"})})})]}),/*#__PURE__*/e(\"p\",{children:\"Last month, we collaborated with one of our innovative clients, HolidayHero, a platform dedicated to enhancing guest experiences and streamlining communication in the hospitality sector. By integrating LangWatch into their tech operations, HolidayHero has optimized their AI-driven content generation and monitoring processes to ensure seamless guest interactions and top-tier service delivery.\"}),/*#__PURE__*/e(\"p\",{children:\"LangWatch has been a part of HolidayHero's LLM production environment for over two months, overseeing thousands of guest interactions and content generations. The platform is utilized daily to improve the performance of their LLM-features, ensuring accurate and personalized guest experiences.\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/t(\"strong\",{children:[\"The power of AI in hospitality\",/*#__PURE__*/e(\"br\",{})]}),\"In the constantly evolving hospitality industry, elevating guest experiences is a constant development. HolidayHero aims to improve all aspects of guest experience and communication, from booking to booking again. HolidayHero has entered a  partnership with Langwatch, focusing on improving AI-driven content generation to meet guest expectations and stream / optimize current (guest) processes.\\xa0\",/*#__PURE__*/e(\"br\",{}),/*#__PURE__*/e(\"strong\",{children:/*#__PURE__*/e(\"br\",{})}),\"Traditional hospitality has been a face-to-face industry. Expectations were and are built in face-to-face interactions. With the introduction of AI, many companies are testing the waters to see where it could fit. With HolidayHero, most of the content for guests is generated by and audited by AI. Guidebook creation, summarizing texts, and experience descriptions are all generated by AI.\\xa0\"]}),/*#__PURE__*/e(\"p\",{children:\"Increasing customer demand and so the usage of LLM's has started HolidayHero's search for monitoring tools and a way to evaluate the quality and improve. With Langwatch, HolidayHero monitors and evaluates the input and output of the LLM\u2019s, allowing them to fine-tune and rerun their prompts and models.\\xa0Being able to continuously monitor and improve the outcome with the Optimization Studio recently launched. \"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/t(\"strong\",{children:[\"Where does HolidayHero use AI?\\xa0\",/*#__PURE__*/e(\"br\",{})]}),\"HolidayHero uses AI to automate content generation for their guest's properties:\\xa0\"]}),/*#__PURE__*/t(\"ul\",{children:[/*#__PURE__*/t(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(14, 16, 26)\",\"--framer-text-decoration\":\"none\"},children:[/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"Guidebook Generation\"})}),/*#__PURE__*/e(\"p\",{children:\"Most properties have a set of house rules; a digital app requires a different approach. At HolidayHero, AI is used to rewrite the content to match their customer's branding.\\xa0\"})]}),/*#__PURE__*/t(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(14, 16, 26)\",\"--framer-text-decoration\":\"none\"},children:[/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"Summarizing extensive texts\"})}),/*#__PURE__*/e(\"p\",{children:\"At HolidayHero, local experiences and amenities play an important role. In the STR / Hospitality sector, local recommendations also play an important role. Most of them have not been documented. With the help of AI, we speed up the generation of helpful texts based on the hotel's brand guidelines.\\xa0\"})]}),/*#__PURE__*/t(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(14, 16, 26)\",\"--framer-text-decoration\":\"none\"},children:[/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"(Re)writing emails\"})}),/*#__PURE__*/e(\"p\",{children:\"Written communication must be received at the right moment during guest interactions. At HolidayHero, they have implemented touchpoints and touchpoint automation, allowing hoteliers and B&B owners to (pro) actively communicate with their guests at the right moment.\\xa0\"})]})]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/t(\"strong\",{children:[\"The roadmap ahead to add more LLM-features\",/*#__PURE__*/e(\"br\",{})]}),\"AI will play a more critical and dominant role in the Hospitality industry. HolidayHero and Langwatch will continue to experiment with AI. Personalization and automation will be areas that are further researched.\\xa0\"]}),/*#__PURE__*/t(\"ul\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(14, 16, 26)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Personalization: \"}),\"each traveler has different wishes and demands. With traditional face-to-face interactions, travelers are used to high levels of personalization. While HolidayHero is pushing its AI efforts, personalization of interactions remains at the forefront.\\xa0\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(14, 16, 26)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Automation: \"}),\"More and more questions can be answered through automated processes and tools. Questions that have been answered before can be easily answered again. With tons of descriptions of amenities and guidebooks, all information to automate the inquiry is available at the ready to have AI answer it immediately.\\xa0\"]})})]}),/*#__PURE__*/e(\"p\",{children:\"When executed well, implementing AI in personalization and automation can reduce the workload of existing employees and create significant operational efficiencies, reducing costs and increasing revenue across all guest-facing staff.\\xa0\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Deep Insights into AI Performance with LangWatch\"}),/*#__PURE__*/e(\"br\",{}),\"Their team dedicates several hours each week to analyzing performance data through LangWatch. \u201CThe analytics and evaluations are invaluable. We can track how well our LLM performs and adapts accordingly,\u201D they share.\"]}),/*#__PURE__*/e(\"p\",{children:\"Flexible filters and alerts further streamline their workflow. \u201CIf a guest interaction needs review because the sentiment was negative or thumb-down were provided we are alerted in slack and can quickly locate the in-output and address it by building a dataset and improve on the not so qualitative outputs. \"}),/*#__PURE__*/e(\"p\",{children:\"With the recent launch of the Optimization Studio we were blown away. They already played around with the DSPy framework themselves, but it was hard to provide this knowledge to the rest of the team. By using the Optimization Studio, HolidayHero can evaluate entire datasets which they pre-generated in the monitoring section. And run multiple iterations by using DSPy optimizers, ensuring they align with brand tone and guest expectations.  \u201CThe Optimization Studio gives us the ability to tweak and test different versions of our LLM-features, helping us land on the most engaging and relevant outputs for guests.\u201D This iterative process plays a critical role in maintaining consistent, high-quality guest communications across various properties.\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/t(\"strong\",{children:[\"Conclusion\",/*#__PURE__*/e(\"br\",{})]}),\"With Langwatch as a critical tool in their tech stack, HolidayHero looks forward to further deepening its collaboration with LangWatch. By continuing to refine their use of LangWatch\u2019s monitoring, evaluations, and analytics capabilities, they aim to lead the hospitality sector in AI-driven guest engagement and automation.\"]}),/*#__PURE__*/e(\"p\",{children:\"Interested in learning how LangWatch can optimize your AI features?\"}),/*#__PURE__*/e(\"h4\",{children:/*#__PURE__*/e(a,{href:\"https://meetings-eu1.hubspot.com/manouk-draisma\",motionChild:!0,nodeId:\"LYxDZg6nX\",openInNewTab:!0,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(o.a,{children:\"Get a demo today\"})})})]});export const richText1=/*#__PURE__*/t(i.Fragment,{children:[/*#__PURE__*/e(\"h4\",{children:/*#__PURE__*/e(\"strong\",{children:\"Faster Work, Bigger Impact\"})}),/*#__PURE__*/e(\"p\",{children:\"As an AI engineer, you\u2019ve probably felt the frustration of working with Large Language Models (LLMs): countless hours tweaking prompts, figuring out what\u2019s working (or not), and starting over every time there\u2019s a change or new model. It\u2019s tedious, time-consuming, and holds back innovation.\"}),/*#__PURE__*/t(\"p\",{children:[\"LangWatch gets it. That\u2019s why we built \",/*#__PURE__*/e(\"strong\",{children:\"Optimization Studio\"}),\" \u2013 a platform designed to make your work as an AI engineer faster, smarter, and more reliable.\"]}),/*#__PURE__*/e(\"h4\",{children:/*#__PURE__*/e(\"strong\",{children:\"What is LangWatch Optimization Studio?\"})}),/*#__PURE__*/e(\"p\",{children:\"Optimization Studio isn\u2019t just a tool; it\u2019s a complete workflow for monitoring, optimizing, and scaling LLMs. Powered by the DSPy framework, it offers a streamlined way to improve your AI development process without diving into unnecessary complexity.\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"What you can do:\"})}),/*#__PURE__*/t(\"ul\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Get instant insights:\"}),\" Real-time tracking of LLM input-output behavior to identify and resolve issues early.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Automate optimizations:\"}),\" Use built-in tools to fine-tune prompts, models, and configurations without endless manual tweaking.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Scale with ease:\"}),\" Rapid iterations with automated feedback loops ensure stable models, even at scale.\"]})})]}),/*#__PURE__*/e(\"p\",{children:\"Setup takes less than 30 minutes, guided step-by-step by our CTO, so you can hit the ground running.\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"br\",{className:\"trailing-break\"})}),/*#__PURE__*/e(\"img\",{alt:\"\",className:\"framer-image\",height:\"1378\",src:\"https://framerusercontent.com/images/KEarCpDQXDctnH8Exe2Qt4GVLA.png\",srcSet:\"https://framerusercontent.com/images/KEarCpDQXDctnH8Exe2Qt4GVLA.png?scale-down-to=512 512w,https://framerusercontent.com/images/KEarCpDQXDctnH8Exe2Qt4GVLA.png?scale-down-to=1024 1024w,https://framerusercontent.com/images/KEarCpDQXDctnH8Exe2Qt4GVLA.png?scale-down-to=2048 2048w,https://framerusercontent.com/images/KEarCpDQXDctnH8Exe2Qt4GVLA.png 3608w\",style:{aspectRatio:\"3608 / 2757\"},width:\"1804\"}),/*#__PURE__*/e(\"h4\",{children:/*#__PURE__*/e(\"strong\",{children:\"How Does Optimization Studio Help AI Engineers?\"})}),/*#__PURE__*/e(\"p\",{children:\"With Optimization Studio, you can focus on what you do best: creating impactful AI solutions. The platform takes repetitive tasks off your plate, helping you achieve faster results and make smarter decisions.\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"Here\u2019s what you\u2019ll get:\"})}),/*#__PURE__*/t(\"ul\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"More control:\"}),\" Deep insights into your models' performance for targeted improvements.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Faster iterations:\"}),\" Say goodbye to guesswork; optimization is data-driven and automated.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Scalability you can trust:\"}),\" A robust framework that grows with you, without unexpected hurdles.\"]})})]}),/*#__PURE__*/e(\"p\",{children:\"In short: take your LLMs to production faster, with higher reliability and fewer headaches.\"}),/*#__PURE__*/e(\"h4\",{children:/*#__PURE__*/e(\"strong\",{children:\"Why DSPy?\"})}),/*#__PURE__*/e(\"p\",{children:\"If you\u2019ve worked with DSPy before, you know how powerful it is \u2013 but also how complex it can be. LangWatch removes the complexity by offering a low-code interface. You stay in control while the platform does the heavy lifting, letting you unlock DSPy\u2019s potential without the steep learning curve.\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"br\",{className:\"trailing-break\"})}),/*#__PURE__*/e(\"img\",{alt:\"\",className:\"framer-image\",height:\"98\",src:\"https://framerusercontent.com/images/yvYXVEI9amXcCMs9szEISpaknfo.png\",srcSet:\"https://framerusercontent.com/images/yvYXVEI9amXcCMs9szEISpaknfo.png?scale-down-to=512 512w,https://framerusercontent.com/images/yvYXVEI9amXcCMs9szEISpaknfo.png 609w\",style:{aspectRatio:\"609 / 196\"},width:\"304\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"br\",{className:\"trailing-break\"})}),/*#__PURE__*/e(\"h4\",{children:/*#__PURE__*/e(\"strong\",{children:\"What Does This Mean for Your Workflow?\"})}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/t(\"strong\",{children:[\"Imagine this:\",/*#__PURE__*/e(\"br\",{})]}),\"You\u2019re working on a new LLM feature. Weeks of testing prompts and fine-tuning parameters have left you burned out. With Optimization Studio, you can achieve the same results in minutes. The platform automatically identifies the best configurations, freeing you to focus on strategy and innovation instead of repetitive tasks.\"]}),/*#__PURE__*/e(\"p\",{children:\"Plus, Optimization Studio provides detailed reports that give both you and your stakeholders confidence in the quality of your work.\"}),/*#__PURE__*/e(\"h4\",{children:/*#__PURE__*/e(\"strong\",{children:\"Start Working Smarter Today\"})}),/*#__PURE__*/t(\"p\",{children:[\"As an AI engineer, you know that efficiency and quality go hand in hand. \",/*#__PURE__*/e(\"strong\",{children:\"LangWatch Optimization Studio\"}),\" was built to help you work faster and smarter, so you can build better models and deliver more value.\"]}),/*#__PURE__*/e(\"p\",{children:\"Ready to make a bigger impact? \"}),/*#__PURE__*/e(\"h3\",{children:/*#__PURE__*/e(a,{href:\"https://app.langwatch.ai/\",motionChild:!0,nodeId:\"LYxDZg6nX\",openInNewTab:!0,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(o.a,{children:\"Get Started Today! \"})})})]});export const richText2=/*#__PURE__*/t(i.Fragment,{children:[/*#__PURE__*/e(\"p\",{children:\"Fine-tuning prompts for consistent, high-quality output is a game-changer. Yet, until now, optimizing prompts has often required deep technical knowledge and coding skills\u2014especially when using advanced frameworks like DSPy or a lot of manual trial & error work. But what if there was a way to leverage the power of DSPy\u2019s MIPROv2 without diving into complex code? Enter LangWatch\u2019s Optimization Studio, where MIPROv2 lives in a low-code environment designed to make prompt optimization more accessible than ever.\"}),/*#__PURE__*/e(\"h3\",{children:/*#__PURE__*/e(\"strong\",{children:\"What Is MIPROv2, and why does it matter?\"})}),/*#__PURE__*/e(\"p\",{children:'Before diving into LangWatch\u2019s Optimization Studio, let\u2019s take a look at the magic behind MIPROv2. Part of the DSPy library, MIPROv2 (Multiprompt Instruction Proposal Optimizer Version 2) is a state-of-the-art optimizer developed from Stanford\u2019s DSP research. At its core, DSP (Demonstrate - Search - Predict) offers a unique approach to prompt optimization, prioritizing \"prompt optimization\" over traditional \"prompt engineering.\" By using DSPy, you can find the best prompt variations that align with your model\\'s needs, maximizing accuracy and relevance in output.'}),/*#__PURE__*/e(\"p\",{children:\"MIPROv2 is designed to automate this process, finding the optimal combination of prompt demonstrations and instructions that result in accurate, useful model responses. And while DSPy is powerful, navigating its open-source code can be challenging for those without technical expertise. That\u2019s where the Optimization Studio\u2019s low-code environment comes in, making it possible to run advanced optimizations without diving into complex coding.\"}),/*#__PURE__*/e(\"h3\",{children:/*#__PURE__*/e(\"strong\",{children:\"Why use Optimization Studio\u2019s low-code environment for MIPROv2?\"})}),/*#__PURE__*/e(\"p\",{children:\"LangWatch\u2019s Optimization Studio is built to bring the best of DSPy\u2019s capabilities to users who want results fast, without the need to understand every line of underlying code. With a user-friendly, low-code interface, the Optimization Studio allows you to:\"}),/*#__PURE__*/t(\"ol\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Leverage DSPy\u2019s Cutting-Edge Technology\"}),\": Use MIPROv2 to optimize prompts without any technical hurdles. Access the same high-quality results that DSPy offers, but with an intuitive, guided process.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Save Time and Resources\"}),\": Optimization Studio automates the most labor-intensive parts of prompt refinement, like generating demonstrations and running evaluation trials, so you can focus on strategy, not code.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Achieve Consistent Quality\"}),\": With MIPROv2, you can confidently create prompts that consistently meet your quality criteria, even as the demands on your LLM change.\"]})})]}),/*#__PURE__*/e(\"h3\",{children:/*#__PURE__*/e(\"strong\",{children:\"How MIPROv2 works in Optimization Studio\"})}),/*#__PURE__*/e(\"p\",{children:\"The Optimization Studio harnesses the full capability of DSPy\u2019s MIPROv2 through a guided, three-step process. Here\u2019s a breakdown of how this happens in a way that\u2019s simple, efficient, and code-light:\"}),/*#__PURE__*/e(\"h4\",{children:/*#__PURE__*/e(\"strong\",{children:\"Step 1: Demonstrate with high-quality data\"})}),/*#__PURE__*/e(\"p\",{children:\"The first step is to gather input-output examples (demonstrations) that represent the kind of responses you expect from your LLM. MIPROv2 then automatically generates a range of demonstration sets from this data, ensuring that each set showcases both accurate and relevant responses. You simply provide the input-output pairs and let the Optimization Studio do the rest.\"}),/*#__PURE__*/e(\"h4\",{children:/*#__PURE__*/e(\"strong\",{children:\"Step 2: Craft effective instructions\"})}),/*#__PURE__*/e(\"p\",{children:\"Using the data from Step 1, MIPROv2 generates various prompt instructions by analyzing the most effective ways to achieve your desired results. The Optimization Studio uses a summary of the demonstration sets, and from there, it creates different prompt instructions that match the style and goal of your project. You can review, adjust, and select the most relevant prompt without needing to experiment manually.\"}),/*#__PURE__*/e(\"h4\",{children:/*#__PURE__*/e(\"strong\",{children:\"Step 3: Select the best Prompt with bayesian Optimization\"})}),/*#__PURE__*/e(\"p\",{children:\"To find the most effective prompt, MIPROv2 runs evaluation trials, scoring each demo-prompt pair against the criteria you specify. By using Bayesian Optimization, it quickly hones in on the best-performing prompt variation. All this happens behind the scenes in the Optimization Studio, providing you with the top-scoring prompt without needing to manage the complex calculations yourself.\"}),/*#__PURE__*/e(\"h3\",{children:/*#__PURE__*/e(\"strong\",{children:\"The benefits of low-code MIPROv2 Optimization\"})}),/*#__PURE__*/e(\"p\",{children:\"By bringing MIPROv2 into a low-code environment, LangWatch\u2019s Optimization Studio makes prompt optimization faster, simpler, and more accessible to teams across industries. Whether you\u2019re working in customer support, content generation, or educational applications, you can:\"}),/*#__PURE__*/t(\"ul\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Quickly adapt prompts\"}),\" to new use cases or requirements without extensive re-coding.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Validate and monitor prompt performance\"}),\" using built-in evaluation functions, ensuring that your LLM\u2019s responses meet real-world demands.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Optimize LLMs without programming knowledge\"}),\", allowing non-technical team members to participate in the optimization process.\"]})})]}),/*#__PURE__*/e(\"h3\",{children:/*#__PURE__*/e(\"strong\",{children:\"Take the next step with LangWatch\u2019s Optimization Studio\"})}),/*#__PURE__*/e(\"p\",{children:\"LangWatch\u2019s Optimization Studio is more than just a tool\u2014it\u2019s a gateway to unlocking the potential of your language model through smarter, more efficient prompts. By utilizing DSPy\u2019s MIPROv2 in a low-code setting, you can experience the advantages of prompt optimization without getting bogged down by technical details.\"}),/*#__PURE__*/e(\"p\",{children:\"Ready to experience prompt optimization without the coding complexity? Start using LangWatch\u2019s Optimization Studio today and discover how accessible, high-quality prompt optimization can elevate your LLM\u2019s performance.\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(a,{href:\"https://www.langwatch.ai/\",motionChild:!0,nodeId:\"LYxDZg6nX\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(o.a,{children:/*#__PURE__*/e(\"strong\",{children:\"LangWatch.ai\"})})})})]});export const richText3=/*#__PURE__*/t(i.Fragment,{children:[/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"Introduction\"})}),/*#__PURE__*/e(\"p\",{children:'As large language models become more integral to applications across industries, the way we interact with these models has evolved significantly. One of the most critical aspects of working with LLMs is crafting the perfect prompt\u2014essentially the instructions that guide the model\u2019s response. Traditionally, this has been done through \"prompt engineering,\" a process of manually refining prompts to improve output quality. However, with the introduction of prompt optimization tools like DSPy and LangWatch\u2019s Optimization Studio, a more precise, scientific and better approach is transforming this field.'}),/*#__PURE__*/e(\"h2\",{children:/*#__PURE__*/e(\"strong\",{children:\"What is Prompt Optimization?\"})}),/*#__PURE__*/e(\"p\",{children:\"Unlike prompt engineering, which often relies on trial and error, prompt optimization takes a systematic approach to create the best possible prompts. Using the DSPy library\u2014a Python-based framework developed from Stanford\u2019s Demonstrate-Search-Predict (DSP) methodology\u2014prompt optimization allows users to explore various prompt configurations based on data, achieving high-quality responses without extensive manual intervention. Prompt optimization can lead to more reliable, scalable, and accurate interactions with LLMs, making it ideal for applications where performance and precision are essential.\"}),/*#__PURE__*/e(\"h3\",{children:/*#__PURE__*/e(\"strong\",{children:\"How does DSPy enable Prompt Optimization?\"})}),/*#__PURE__*/e(\"p\",{children:\"DSPy enables prompt optimization through a three-step process designed to iteratively refine prompts. Here\u2019s a quick overview of each step:\"}),/*#__PURE__*/t(\"ol\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Demonstrate\"}),\": DSPy starts by analyzing example pairs of input and output (often referred to as demonstrations) to establish a benchmark for the desired output.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Search\"}),\": Next, DSPy searches for potential prompts by leveraging patterns and insights from these examples. It combines variations of prompt structures and content to find promising configurations.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Predict\"}),\": Finally, the library predicts the best prompt configurations by testing them against new inputs, helping ensure the results generalize well beyond the initial dataset.\"]})})]}),/*#__PURE__*/e(\"p\",{children:\"This process allows DSPy to go beyond a single, static prompt and dynamically optimize for the most accurate and contextually relevant outputs.\"}),/*#__PURE__*/e(\"h3\",{children:/*#__PURE__*/e(\"strong\",{children:\"Introducing LangWatch\u2019s Optimization Studio\"})}),/*#__PURE__*/e(\"p\",{children:\"While DSPy is a powerful tool for Python developers, LangWatch\u2019s Optimization Studio makes prompt optimization even more accessible. Our platform allows users to tap into DSPy\u2019s capabilities within a low-code environment, providing a user-friendly interface that makes prompt optimization accessible even to those without extensive programming knowledge.\"}),/*#__PURE__*/e(\"p\",{children:\"Here\u2019s a look at the unique benefits that LangWatch\u2019s Optimization Studio brings to the table:\"}),/*#__PURE__*/t(\"ul\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Low-Code, High Impact\"}),\": With its drag-and-drop functionality, the Optimization Studio enables anyone to create and fine-tune prompts in a DSPy-powered, low-code setup. Users can focus on performance without being bogged down by code.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Built-In DSPy Optimizers\"}),\": LangWatch\u2019s Optimization Studio comes with DSPy\u2019s most advanced optimizers, including the latest MIPROv2 optimizer. These tools are designed to make prompt optimization efficient, helping users refine prompts through an intuitive workflow.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"LLM Monitoring Integration\"}),\": Prompt optimization is only one piece of the puzzle. To truly improve LLM performance, ongoing monitoring is essential. LangWatch\u2019s Optimization Studio integrates LLM monitoring tools, allowing users to track and evaluate prompt performance over time. This data-driven feedback loop ensures that prompts evolve and improve based on real-world use.\"]})})]}),/*#__PURE__*/e(\"h3\",{children:/*#__PURE__*/e(\"strong\",{children:\"The Shift from Prompt Engineering to Prompt Optimization\"})}),/*#__PURE__*/e(\"p\",{children:\"The industry is gradually shifting from traditional prompt engineering to prompt optimization. While prompt engineering can often be subjective and based on intuition, prompt optimization offers a more data-driven, reproducible approach. By systematically testing different prompts, DSPy and LangWatch\u2019s Optimization Studio remove much of the guesswork, leading to faster, more reliable LLM applications.\"}),/*#__PURE__*/e(\"p\",{children:\"The DSPy approach, coupled with LangWatch\u2019s optimization capabilities, represents a new era of prompt management. Whether it\u2019s creating a chatbot, generating custom responses, or automating content, prompt optimization ensures that your LLM interactions are accurate, consistent, and aligned with user needs.\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"Why Prompt Optimization Matters for LLM Development\"})}),/*#__PURE__*/e(\"p\",{children:\"Prompt optimization is not only about improving individual responses but also about scaling LLMs more effectively. Here\u2019s how it makes an impact:\"}),/*#__PURE__*/t(\"ol\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Improved consistency\"}),\": With optimized prompts, LLMs generate more reliable responses, reducing variations in quality. This consistency is especially valuable in customer support, legal, or educational applications where accuracy is essential.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Time and cost Savings\"}),\": By automating much of the prompt refinement process, DSPy and LangWatch\u2019s Optimization Studio reduce the time developers spend on trial-and-error adjustments. Optimized prompts also decrease the need for frequent corrections, saving time and resources over the long term.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Better user experience\"}),\": Optimized prompts lead to more accurate, context-aware responses, directly improving the user experience. Users receive the right answers faster, and applications can handle more complex queries without additional intervention.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Performance monitoring\"}),\": Continuous monitoring and evaluation mean you can quickly detect and adjust to any performance issues, ensuring that your prompts stay relevant and effective even as model updates or usage patterns change.\"]})})]}),/*#__PURE__*/e(\"h3\",{children:/*#__PURE__*/e(\"strong\",{children:\"How LangWatch Optimization Studio is Making a Difference\"})}),/*#__PURE__*/e(\"p\",{children:\"LangWatch\u2019s Optimization Studio goes beyond just making prompt optimization accessible; it integrates monitoring tools that provide ongoing insights into LLM performance. For example, after setting up an optimized prompt, users can track its effectiveness over time, using metrics like response accuracy, user feedback, and engagement levels. This capability allows for proactive adjustments, ensuring that optimized prompts continue to meet changing requirements.\"}),/*#__PURE__*/e(\"p\",{children:\"With LangWatch, businesses, educators, and developers can confidently build applications on top of LLMs, knowing that their prompts are continuously fine-tuned for maximum effectiveness.\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"Get Started with Prompt Optimization Today\"})}),/*#__PURE__*/e(\"p\",{children:\"If you\u2019re ready to take your LLM interactions to the next level, it\u2019s time to explore prompt optimization. LangWatch\u2019s Optimization Studio, powered by DSPy, provides the tools you need to move from prompt engineering to a structured, efficient prompt optimization process. With its low-code environment and seamless monitoring capabilities, LangWatch enables you to build smarter, more accurate applications with LLMs.\"}),/*#__PURE__*/t(\"p\",{children:[\"Explore the Optimization Studio at\",/*#__PURE__*/e(a,{href:\"https://www.langwatch.ai/\",motionChild:!0,nodeId:\"LYxDZg6nX\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(o.a,{children:\" LangWatch.ai\"})}),\" and be part of the future of LLM development. In the coming weeks, we\u2019ll dive deeper into DSPy optimizers, the latest in LLM monitoring, and hands-on tutorials to help you get the most out of your LLMs. Stay tuned for more!\"]})]});export const richText4=/*#__PURE__*/t(i.Fragment,{children:[/*#__PURE__*/e(\"p\",{children:\"Welcome back to our series on building customer support chatbots using Retrieval Augmented Generation (RAG) with OpenAI in Python. Today, we're diving deeper into creating a chatbot that can efficiently answer questions based on provided links. Our setup will utilize FastAPI for the backend, LangChain RAG for the retrieval and generation process, and LangWatch for monitoring. By the end of this tutorial, you'll have a fully functional chatbot deployed on AWS Elastic Beanstalk.\"}),/*#__PURE__*/e(\"p\",{children:\"Here's what you'll need to follow along:\"}),/*#__PURE__*/t(\"ol\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Python 3.11 installed on your machine\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[\"An AWS account (\",/*#__PURE__*/e(a,{href:\"https://aws.amazon.com/\",motionChild:!0,nodeId:\"LYxDZg6nX\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(o.a,{children:\"sign up here\"})}),\")\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[\"Access to LangWatch (\",/*#__PURE__*/e(a,{href:\"https://app.langwatch.ai/deploy-llms-tutorial-ElP_e0\",motionChild:!0,nodeId:\"LYxDZg6nX\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(o.a,{children:\"get started here\"})}),\")\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Familiarity with FastAPI and LangChain\"})})]}),/*#__PURE__*/e(\"p\",{children:\"We'll kick things off by setting up our development environment, followed by creating the RAG module, building the FastAPI server, containerizing our application, and finally, deploying it to AWS Elastic Beanstalk. So, let's get started and bring your chatbot to life! \uD83D\uDE80\"}),/*#__PURE__*/e(\"h3\",{children:\"Step 0 - Setting up the Environment for OpenAI RAG in Python\"}),/*#__PURE__*/e(\"p\",{children:\"First, let's install the necessary dependencies for OpenAI RAG in Python. Create a Python virtual environment and install the required libraries:\"}),/*#__PURE__*/e(\"div\",{className:\"framer-text-module\",style:{height:\"auto\",width:\"100%\"},children:/*#__PURE__*/e(n,{componentIdentifier:\"module:pVk4QsoHxASnVtUBp6jr/TbhpORLndv1iOkZzyo83/CodeBlock.js:default\",children:t=>/*#__PURE__*/e(r,{...t,code:\"pip install fastapi langwatch langchain langchain-openai langchain-community langchain-chroma\",language:\"JSX\"})})}),/*#__PURE__*/t(\"p\",{children:[\"Next, create a \",/*#__PURE__*/e(\"code\",{children:\".env\"}),\" file to securely store your API keys:\"]}),/*#__PURE__*/e(\"div\",{className:\"framer-text-module\",style:{height:\"auto\",width:\"100%\"},children:/*#__PURE__*/e(n,{componentIdentifier:\"module:pVk4QsoHxASnVtUBp6jr/TbhpORLndv1iOkZzyo83/CodeBlock.js:default\",children:t=>/*#__PURE__*/e(r,{...t,code:\"OPENAI_API_KEY=<your-api-key> LANGWATCH_API_KEY=<your-api-key>\",language:\"JSX\"})})}),/*#__PURE__*/e(\"p\",{children:\"Awesome! Now let's write some code.\"}),/*#__PURE__*/e(\"h3\",{children:\"Step 1 - Creating the RAG Module with OpenAI in Python\"}),/*#__PURE__*/e(\"p\",{children:\"We'll start by coding our RAG module with OpenAI in Python. We'll structure our codebase to have a modular RAG component that can be easily imported and used with various input parameters. Additionally, we'll have a FastAPI backend to connect the RAG module with users through an API.\"}),/*#__PURE__*/e(\"div\",{className:\"framer-text-module\",style:{height:\"auto\",width:\"100%\"},children:/*#__PURE__*/e(n,{componentIdentifier:\"module:pVk4QsoHxASnVtUBp6jr/TbhpORLndv1iOkZzyo83/CodeBlock.js:default\",children:t=>/*#__PURE__*/e(r,{...t,code:'import os\\nfrom langchain import hub\\nfrom langchain_chroma import Chroma\\nfrom langchain_core.vectorstores import VectorStoreRetriever\\nfrom langchain_community.document_loaders import WebBaseLoader\\nfrom langchain_core.output_parsers import StrOutputParser\\nfrom langchain_core.runnables.config import RunnableConfig\\nfrom langchain_core.runnables.passthrough import RunnablePassthrough\\nfrom langchain_openai import OpenAIEmbeddings\\nfrom langchain_text_splitters import RecursiveCharacterTextSplitter\\nimport dotenv\\nfrom langchain_openai import ChatOpenAI\\nimport langwatch \\n\\ndotenv.load_dotenv()\\n\\nopenai_key = os.getenv(\"OPENAI_API_KEY\")\\n\\nllm = ChatOpenAI(model=\"gpt-3.5-turbo-0125\")\\n\\nprompt = hub.pull(\"rlm/rag-prompt\")\\n\\n\\n\\ndef scrape_url(url: str) -> list:\\n    # Load, chunk and index the contents of the blog.\\n    loader = WebBaseLoader(\\n        web_paths=(url,),\\n        bs_kwargs=dict()\\n    )\\n    docs = loader.load()\\n    return docs\\n\\ndef create_retriever(docs: list) -> VectorStoreRetriever:\\n    text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)\\n    splits = text_splitter.split_documents(docs)\\n    vectorstore = Chroma.from_documents(documents=splits, embedding=OpenAIEmbeddings())\\n\\n    # Retrieve and generate using the relevant snippets of the blog.\\n    retriever = vectorstore.as_retriever()\\n    return retriever\\n\\n\\ndef format_docs(docs):\\n    return \"\\\\n\\\\n\".join(doc.page_content for doc in docs)\\n\\n@langwatch.trace()\\ndef ask_rag(question: str, retriever: VectorStoreRetriever):\\n    config=RunnableConfig(\\n            callbacks=[\\n                langwatch.get_current_trace().get_langchain_callback(),\\n            ]\\n        )\\n    question_runnable = RunnablePassthrough().with_config(config=config)\\n    rag_chain = (\\n        {\"context\": retriever | format_docs, \"question\": question_runnable}\\n        | prompt\\n        | llm\\n        | StrOutputParser()\\n    )\\n\\n    answer = rag_chain.invoke(question)\\n    return answer',language:\"JSX\"})})}),/*#__PURE__*/e(\"p\",{children:\"This code composes our RAG pipeline from a couple of simple functions, where each resembles an important step in the Retrieval Augmented Generation. This code contains functions to scrape web pages, create vector embeddings out of them and create retrievers that can retrieve related data. The last function performs Question Answering with the help of RAG pipeline.\"}),/*#__PURE__*/t(\"p\",{children:[\"Pay attention how we add LangWatch tracing in the \",/*#__PURE__*/e(\"code\",{children:\"ask_rag\"}),\" function. We add a decorator at the top of a function definition and we create an instance of a runnable with our custom configurations where we send the callbacks to LangWatch.\"]}),/*#__PURE__*/e(\"h3\",{children:\"Step 2 - Building a FastAPI Server for OpenAI RAG\"}),/*#__PURE__*/t(\"p\",{children:[\"Next, we'll create a FastAPI server that allows users to interact with the RAG module via an API. The server will have a single endpoint where users can make \",/*#__PURE__*/e(\"code\",{children:\"POST\"}),\" requests with two input arguments: the link to scrape and the question they ask.\"]}),/*#__PURE__*/e(\"div\",{className:\"framer-text-module\",style:{height:\"auto\",width:\"100%\"},children:/*#__PURE__*/e(n,{componentIdentifier:\"module:pVk4QsoHxASnVtUBp6jr/TbhpORLndv1iOkZzyo83/CodeBlock.js:default\",children:t=>/*#__PURE__*/e(r,{...t,code:'from fastapi import FastAPI\\n\\nfrom rag import scrape_url, create_retriever, ask_rag\\n\\napp = FastAPI()\\n\\n@app.get(\"/\")\\nasync def root():\\n    return {\"message\": \"Hello World\"}\\n\\n@app.post(\"/ask\")\\ndef ask(question: str, link: str):\\n    docs = scrape_url(link)\\n    retriever = create_retriever(docs)\\n    response = ask_rag(question=question, retriever=retriever)\\n    return {\"message\": response}',language:\"JSX\"})})}),/*#__PURE__*/e(\"h3\",{children:/*#__PURE__*/e(\"br\",{className:\"trailing-break\"})}),/*#__PURE__*/e(\"h3\",{children:\"Step 3 - Containerizing the OpenAI RAG Python App\"}),/*#__PURE__*/e(\"p\",{children:\"As a third step we will make our application deployable as a docker container. Docker simplifies the deployment process and significantly reduces all the dependency issues. Our application will consist of one single docker container that will embed the FastAPI backend.\"}),/*#__PURE__*/e(\"div\",{className:\"framer-text-module\",style:{height:\"auto\",width:\"100%\"},children:/*#__PURE__*/e(n,{componentIdentifier:\"module:pVk4QsoHxASnVtUBp6jr/TbhpORLndv1iOkZzyo83/CodeBlock.js:default\",children:t=>/*#__PURE__*/e(r,{...t,code:'FROM python:3.11\\n\\nWORKDIR /app\\n\\nCOPY requirements.txt .\\n\\nRUN pip install --no-cache-dir -r requirements.txt\\n\\nCOPY . .\\n\\nCMD [\"uvicorn\", \"main:app\", \"--host\", \"0.0.0.0\", \"--port\", \"8080\"]',language:\"JSX\"})})}),/*#__PURE__*/e(\"p\",{children:\"Congrats! Now you can run your application as a docker container. You can build an image out of it with this command ran from the root of the project\"}),/*#__PURE__*/e(\"div\",{className:\"framer-text-module\",style:{height:\"auto\",width:\"100%\"},children:/*#__PURE__*/e(n,{componentIdentifier:\"module:pVk4QsoHxASnVtUBp6jr/TbhpORLndv1iOkZzyo83/CodeBlock.js:default\",children:t=>/*#__PURE__*/e(r,{...t,code:\"docker build -t <image-name>\",language:\"JSX\"})})}),/*#__PURE__*/t(\"p\",{children:[\"Later, you can run the next command and access your application deployed on \",/*#__PURE__*/e(a,{href:\"http://localhost:8080/docs\",motionChild:!0,nodeId:\"LYxDZg6nX\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(o.a,{children:\"http://localhost:8080/docs\"})}),\".\"]}),/*#__PURE__*/e(\"div\",{className:\"framer-text-module\",style:{height:\"auto\",width:\"100%\"},children:/*#__PURE__*/e(n,{componentIdentifier:\"module:pVk4QsoHxASnVtUBp6jr/TbhpORLndv1iOkZzyo83/CodeBlock.js:default\",children:t=>/*#__PURE__*/e(r,{...t,code:\"docker run -d --name <container-name> --env-file .env <image-name>\",language:\"JSX\"})})}),/*#__PURE__*/e(\"p\",{children:\"Nice progress!\"}),/*#__PURE__*/e(\"h3\",{children:\"Step 4 - Installing AWS and giving the Permissions\"}),/*#__PURE__*/e(\"p\",{children:\"As soon as we have our application containerized we can move to the deployment part. For Mac platform you can run this command\"}),/*#__PURE__*/e(\"div\",{className:\"framer-text-module\",style:{height:\"auto\",width:\"100%\"},children:/*#__PURE__*/e(n,{componentIdentifier:\"module:pVk4QsoHxASnVtUBp6jr/TbhpORLndv1iOkZzyo83/CodeBlock.js:default\",children:t=>/*#__PURE__*/e(r,{...t,code:\"brew install awscli\",language:\"JSX\"})})}),/*#__PURE__*/e(\"p\",{children:\"Great! Now you can connect with your AWS account from your terminal. But before doing it - you have to configure the connection with the right user.\"}),/*#__PURE__*/e(\"div\",{className:\"framer-text-module\",style:{height:\"auto\",width:\"100%\"},children:/*#__PURE__*/e(n,{componentIdentifier:\"module:pVk4QsoHxASnVtUBp6jr/TbhpORLndv1iOkZzyo83/CodeBlock.js:default\",children:t=>/*#__PURE__*/e(r,{...t,code:\"aws configure\",language:\"JSX\"})})}),/*#__PURE__*/e(\"p\",{children:\"After running this command you will be prompted to input AWS Access Key ID, Secret Access Key, region, and output format. You can create the first ones navigating on AWS dashboard from IAM \u2192 Users \u2192 [Your User]. Make sure to pick your nearest region as an input region. The output format can be left empty or None.\"}),/*#__PURE__*/e(\"p\",{children:\"You can later verify your AWS configuration by running this command\"}),/*#__PURE__*/e(\"div\",{className:\"framer-text-module\",style:{height:\"auto\",width:\"100%\"},children:/*#__PURE__*/e(n,{componentIdentifier:\"module:pVk4QsoHxASnVtUBp6jr/TbhpORLndv1iOkZzyo83/CodeBlock.js:default\",children:t=>/*#__PURE__*/e(r,{...t,code:\"aws sts get-caller-identity\",language:\"JSX\"})})}),/*#__PURE__*/e(\"p\",{children:\"Finally, before trying to deploy your application, we need to ensure your IAM user has the necessary permissions to create and manage Elastic Beanstalk environments. Attach the following policies to your IAM user:\"}),/*#__PURE__*/e(\"div\",{className:\"framer-text-module\",style:{height:\"auto\",width:\"100%\"},children:/*#__PURE__*/e(n,{componentIdentifier:\"module:pVk4QsoHxASnVtUBp6jr/TbhpORLndv1iOkZzyo83/CodeBlock.js:default\",children:t=>/*#__PURE__*/e(r,{...t,code:\"\u2022\tAWSElasticBeanstalkFullAccess\\n\u2022\tIAMFullAccess\\n\u2022\tAmazonEC2FullAccess\\n\u2022\tAmazonS3FullAccess\",language:\"JSX\"})})}),/*#__PURE__*/e(\"p\",{children:\"You can attach these policies via the AWS Management Console under IAM -> Users -> [Your User] -> Add permissions -> Attach policies directly.\"}),/*#__PURE__*/e(\"p\",{children:\"After this part is done, good job, we will have our app accessible on the internet in a few moments.\"}),/*#__PURE__*/e(\"h3\",{children:\"Step 5 - Deploying OpenAI RAG Python App on AWS Elastic Beanstalk\"}),/*#__PURE__*/e(\"p\",{children:\"Finally, lets install elastic beanstalk on our machine.\"}),/*#__PURE__*/e(\"div\",{className:\"framer-text-module\",style:{height:\"auto\",width:\"100%\"},children:/*#__PURE__*/e(n,{componentIdentifier:\"module:pVk4QsoHxASnVtUBp6jr/TbhpORLndv1iOkZzyo83/CodeBlock.js:default\",children:t=>/*#__PURE__*/e(r,{...t,code:\"brew install aws-elasticbeanstalk\",language:\"JSX\"})})}),/*#__PURE__*/e(\"p\",{children:\"Next, we will initialize the Elastic Beanstalk in the root of the project directory. By running this command we will create a corresponding folder inside of our project. Pay attention - we specify docker as a platform.\"}),/*#__PURE__*/e(\"div\",{className:\"framer-text-module\",style:{height:\"auto\",width:\"100%\"},children:/*#__PURE__*/e(n,{componentIdentifier:\"module:pVk4QsoHxASnVtUBp6jr/TbhpORLndv1iOkZzyo83/CodeBlock.js:default\",children:t=>/*#__PURE__*/e(r,{...t,code:\"eb init -p docker ragapp\",language:\"JSX\"})})}),/*#__PURE__*/e(\"p\",{children:\"After it is initialized we have to create the corresponding environment for our deployment, I call my environment as eb-env. This command will package my application and upload it to the Elastic Beanstalk. It will also create the environment that will be used by the application.\"}),/*#__PURE__*/e(\"div\",{className:\"framer-text-module\",style:{height:\"auto\",width:\"100%\"},children:/*#__PURE__*/e(n,{componentIdentifier:\"module:pVk4QsoHxASnVtUBp6jr/TbhpORLndv1iOkZzyo83/CodeBlock.js:default\",children:t=>/*#__PURE__*/e(r,{...t,code:\"eb create eb-env\",language:\"JSX\"})})}),/*#__PURE__*/e(\"p\",{children:\"Before deploying the application we also have to specify the api keys in our deployed environment\"}),/*#__PURE__*/e(\"div\",{className:\"framer-text-module\",style:{height:\"auto\",width:\"100%\"},children:/*#__PURE__*/e(n,{componentIdentifier:\"module:pVk4QsoHxASnVtUBp6jr/TbhpORLndv1iOkZzyo83/CodeBlock.js:default\",children:t=>/*#__PURE__*/e(r,{...t,code:\"eb setenv OPENAI_API_KEY=your_openai_api_key\",language:\"JSX\"})})}),/*#__PURE__*/e(\"p\",{children:\"Finally, you can deploy the app with simple\"}),/*#__PURE__*/e(\"div\",{className:\"framer-text-module\",style:{height:\"auto\",width:\"100%\"},children:/*#__PURE__*/e(n,{componentIdentifier:\"module:pVk4QsoHxASnVtUBp6jr/TbhpORLndv1iOkZzyo83/CodeBlock.js:default\",children:t=>/*#__PURE__*/e(r,{...t,code:\"eb deploy\",language:\"JSX\"})})}),/*#__PURE__*/e(\"p\",{children:\"Now, you can check the status of your deployment and find out the link to your deployed APIs.\"}),/*#__PURE__*/e(\"div\",{className:\"framer-text-module\",style:{height:\"auto\",width:\"100%\"},children:/*#__PURE__*/e(n,{componentIdentifier:\"module:pVk4QsoHxASnVtUBp6jr/TbhpORLndv1iOkZzyo83/CodeBlock.js:default\",children:t=>/*#__PURE__*/e(r,{...t,code:\"eb status\",language:\"JSX\"})})}),/*#__PURE__*/e(\"p\",{children:\"You are expected to see something like\"}),/*#__PURE__*/e(\"div\",{className:\"framer-text-module\",style:{height:\"auto\",width:\"100%\"},children:/*#__PURE__*/e(n,{componentIdentifier:\"module:pVk4QsoHxASnVtUBp6jr/TbhpORLndv1iOkZzyo83/CodeBlock.js:default\",children:t=>/*#__PURE__*/e(r,{...t,code:\"Environment details for: eb-env\\n  Application name: ragapp\\n  Region: us-west-2\\n  Deployed Version: app-13d3-240710_145838723247\\n  Environment ID: e-anrxpim3ms\\n  Platform: arn:aws:elasticbeanstalk:us-west-2::platform/Docker running on 64bit Amazon Linux 2/3.8.3\\n  Tier: WebServer-Standard-1.0\\n  CNAME: eb-env.eba-hgsmkwpy.us-west-2.elasticbeanstalk.com\\n  Status: Ready\\n  Health: Green\",language:\"JSX\"})})}),/*#__PURE__*/e(\"p\",{children:\"Here, you need to pay attention to two indicators:\"}),/*#__PURE__*/t(\"ol\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Health - should be Green\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"CNAME - a public url accessing your application.\"})})]}),/*#__PURE__*/t(\"p\",{children:[\"Finally, you can navigate to \",/*#__PURE__*/e(a,{href:\"http://eb-env.eba-hgsmkwpy.us-west-2.elasticbeanstalk.com/docs\",motionChild:!0,nodeId:\"LYxDZg6nX\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(o.a,{children:\"eb-env.eba-hgsmkwpy.us-west-2.elasticbeanstalk.com/docs\"})}),\" and play with the available API\u2019s\"]}),/*#__PURE__*/e(\"p\",{children:\"You've successfully built and deployed a customer support chatbot using OpenAI RAG in Python, FastAPI, and AWS Elastic Beanstalk. \"}),/*#__PURE__*/t(\"p\",{children:[\"Last step is to unlock the BlackBox and see the results coming out of the LLM, improve and iterate. For this, we'd be happy to onboard you on \",/*#__PURE__*/e(a,{href:\"https://app.langwatch.ai/inbox-narrator\",motionChild:!0,nodeId:\"LYxDZg6nX\",openInNewTab:!0,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(o.a,{children:\"LangWatch\"})}),\". \"]}),/*#__PURE__*/e(\"p\",{children:\"Did you like this tutorial? Let us know your feedback and we'd be happy to support you. \"}),/*#__PURE__*/e(\"p\",{children:\"Happy coding!\"})]});export const richText5=/*#__PURE__*/t(i.Fragment,{children:[/*#__PURE__*/e(\"img\",{alt:\"\",className:\"framer-image\",height:\"214\",src:\"https://framerusercontent.com/images/dj38cCZkrFVlThAqcoyAhcxE.png\",srcSet:\"https://framerusercontent.com/images/dj38cCZkrFVlThAqcoyAhcxE.png?scale-down-to=512 512w,https://framerusercontent.com/images/dj38cCZkrFVlThAqcoyAhcxE.png?scale-down-to=1024 1024w,https://framerusercontent.com/images/dj38cCZkrFVlThAqcoyAhcxE.png 1605w\",style:{aspectRatio:\"1605 / 428\"},width:\"802\"}),/*#__PURE__*/e(\"p\",{children:\"Test Driven Development in simple terms means writing the tests before writing the code, but anyone who has practiced it for real knows that it goes much beyond. This simple, unintuitive twist causes a profound shift in your mindset.\"}),/*#__PURE__*/t(\"p\",{children:[\"Having the test before the implementation makes you think much more about \",/*#__PURE__*/e(\"strong\",{children:\"what \"}),\"is actually expected, with the \",/*#__PURE__*/e(\"strong\",{children:\"how \"}),\"coming later, and the \",/*#__PURE__*/e(\"strong\",{children:\"how\"}),\" is an implementation detail, which can be changed through refactoring.\"]}),/*#__PURE__*/t(\"p\",{children:[\"LLMs excel at doing \",/*#__PURE__*/e(\"strong\",{children:\"what \"}),\"you ask them to do\u2026 but also kinda don't. You can write a prompt and just ask what is expected, but it's not always that LLMs do what you want, partially because it doesn't always follow the prompt correctly. A working prompt can easily break when changed, but in other part because sometimes you actually don't know every detail on how you want it to behave, you only notice when the output is somewhat off, for some use cases, and language is just so open it's hard to define everything.\"]}),/*#__PURE__*/e(\"p\",{children:\"This way, TDD matches both pretty well and pretty badly on what is needed to build better LLM applications.\"}),/*#__PURE__*/e(\"p\",{children:\"Pretty well on philosophy, if instead of doing the prompt directly you'd first write down what you expect from the output, then the prompt could be guided by the already defined spec, and you could change the prompt or the model later, while making sure everything still works as expected.\"}),/*#__PURE__*/e(\"p\",{children:\"Pretty badly though, because generally TDD matches better with unit tests, but LLMs are not a good fit for unit tests, their unpredictable nature means they can fail probabilistically, and many examples are needed to ensure it mostly passes.\"}),/*#__PURE__*/e(\"p\",{children:\"But is there a way forward anyway? Well, there is!\"}),/*#__PURE__*/e(\"img\",{alt:'Our goal is to \"shift-left\", the earlier you find issues, the cheaper it is, cheaper in time spent manually testing, cheaper in time spent fixing due to faster cycles, and cheaper literally as less issues hit production',className:\"framer-image\",height:\"330\",src:\"https://framerusercontent.com/images/q9FabgL8ofYgrDGEMgoHJzi6k.png\",srcSet:\"https://framerusercontent.com/images/q9FabgL8ofYgrDGEMgoHJzi6k.png?scale-down-to=512 512w,https://framerusercontent.com/images/q9FabgL8ofYgrDGEMgoHJzi6k.png?scale-down-to=1024 1024w,https://framerusercontent.com/images/q9FabgL8ofYgrDGEMgoHJzi6k.png 1600w\",style:{aspectRatio:\"1600 / 661\"},width:\"800\"}),/*#__PURE__*/e(\"p\",{children:\"Image caption: our goal is to \u201Cshift-left\u201D, the earlier you find issues, the cheaper it is, cheaper in time spent manually testing, cheaper in time spent fixing due to faster cycles, and cheaper literally as less issues hit production\"}),/*#__PURE__*/e(\"h3\",{children:\"Doing TDD for LLMs with\\xa0pytest\"}),/*#__PURE__*/t(\"p\",{children:[\"Alright, \",/*#__PURE__*/e(a,{href:\"https://x.com/chrisalbon/status/1764847127220596975\",motionChild:!0,nodeId:\"LYxDZg6nX\",openInNewTab:!0,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(o.a,{children:\"no more yapping\"})}),\", let's start with a very simple test case. We are building a bot that will generate and tweet out recipes, as a tweet, it should be short, so our first test is simple enough:\"]}),/*#__PURE__*/e(\"div\",{className:\"framer-text-module\",style:{height:\"auto\",width:\"100%\"},children:/*#__PURE__*/e(n,{componentIdentifier:\"module:pVk4QsoHxASnVtUBp6jr/TbhpORLndv1iOkZzyo83/CodeBlock.js:default\",children:t=>/*#__PURE__*/e(r,{...t,code:'import pytest\\nfrom app import recipe_bot\\n\\n@pytest.mark.asyncio\\nasync def test_fits_a_tweet(entry, model):\\n    recipe = await recipe_bot.generate_tweet(\\n      input=\"Generate me a recipe for a quick breakfast with bacon\",\\n      model=\"gpt-3.5-turbo\"\\n    )\\n\\n    assert len(recipe) <= 140',language:\"Python\"})})}),/*#__PURE__*/t(\"p\",{children:[\"We use \",/*#__PURE__*/e(a,{href:\"https://docs.pytest.org/\",motionChild:!0,nodeId:\"LYxDZg6nX\",openInNewTab:!0,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(o.a,{children:\"pytest\"})}),\" to run and execute the tests plus \",/*#__PURE__*/e(a,{href:\"https://pypi.org/project/pytest-asyncio/\",motionChild:!0,nodeId:\"LYxDZg6nX\",openInNewTab:!0,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(o.a,{children:\"pytest-asyncio\"})}),\" to run tests asynchronously, this will be important later.\"]}),/*#__PURE__*/e(\"p\",{children:\"Our test is simple enough, good old assertion of just checking the length of the output. Now, as per TDD rule, we run the test (without the implementation), we see it fail (if it passes now something is wrong!) and then we proceed to implementation:\"}),/*#__PURE__*/e(\"div\",{className:\"framer-text-module\",style:{height:\"auto\",width:\"100%\"},children:/*#__PURE__*/e(n,{componentIdentifier:\"module:pVk4QsoHxASnVtUBp6jr/TbhpORLndv1iOkZzyo83/CodeBlock.js:default\",children:t=>/*#__PURE__*/e(r,{...t,code:'import litellm\\nfrom litellm import ModelResponse\\n\\nasync def generate_tweet(input: str, model: str = \"gpt-3.5-turbo\") -> str:\\n    response: ModelResponse = await litellm.acompletion(\\n        model=model,\\n        messages=[\\n            {\\n                \"role\": \"system\",\\n                \"content\": \"You are a recipe tweet generator, generate recipes using a max of 140 characters.\",\\n            },\\n            {\"role\": \"user\", \"content\": input},\\n        ],\\n        temperature=0.0,\\n    )  # type: ignore\\n\\n    return response.choices[0].message.content  # type: ignore',language:\"Python\"})})}),/*#__PURE__*/t(\"p\",{children:[\"We use \",/*#__PURE__*/e(a,{href:\"https://github.com/BerriAI/litellm\",motionChild:!0,nodeId:\"LYxDZg6nX\",openInNewTab:!0,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(o.a,{children:\"litellm\"})}),\", a drop-in replacement for OpenAI which allows us to use many different LLMs by just switching the model argument. The prompt is simple, straightforward, should work.\"]}),/*#__PURE__*/e(\"p\",{children:\"Then we run the test again aaand it passe\u2026 fails?\"}),/*#__PURE__*/e(\"div\",{className:\"framer-text-module\",style:{height:\"auto\",width:\"100%\"},children:/*#__PURE__*/e(n,{componentIdentifier:\"module:pVk4QsoHxASnVtUBp6jr/TbhpORLndv1iOkZzyo83/CodeBlock.js:default\",children:t=>/*#__PURE__*/e(r,{...t,code:'> pytest\\n\\n============================================================================ FAILURES =============================================================================\\n________________________________________________________________________ test_fits_a_tweet ________________________________________________________________________\\n\\n    @pytest.mark.asyncio\\n    async def test_fits_a_tweet():\\n        recipe = await recipe_bot.generate_tweet(\\n            input=\"Generate me a recipe for a quick breakfast with bacon\",\\n            model=\"gpt-3.5-turbo\",\\n        )\\n\\n>       assert len(recipe) <= 140\\nE       AssertionError: assert 144 <= 140\\nE        +  where 144 = len(\\'Crispy Bacon Breakfast Tacos: Cook bacon until crispy, scramble eggs, fill tortillas with eggs, bacon, cheese, and salsa. Enjoy! #breakfasttacos\\')',language:\"Python\"})})}),/*#__PURE__*/e(\"p\",{children:\"Turns out, LLMs are terrible at counting, so even though you asked very explicitly on the prompt, it still goes a bit over 140 characters. Well, since now you have an automated way of validating, it becomes easier to try it out different prompts and see if it works:\"}),/*#__PURE__*/e(\"div\",{className:\"framer-text-module\",style:{height:\"auto\",width:\"100%\"},children:/*#__PURE__*/e(n,{componentIdentifier:\"module:pVk4QsoHxASnVtUBp6jr/TbhpORLndv1iOkZzyo83/CodeBlock.js:default\",children:t=>/*#__PURE__*/e(r,{...t,code:'- \"content\": \"You are a recipe tweet generator, generate recipes using a max of 140 characters.\",\\n+ \"content\": \"You are a recipe tweet generator, generate recipes using a max of 140 characters, just the ingredients, no yapping.\",',language:\"Python\"})})}),/*#__PURE__*/e(\"p\",{children:\"It does work!\"}),/*#__PURE__*/e(\"div\",{className:\"framer-text-module\",style:{height:\"auto\",width:\"100%\"},children:/*#__PURE__*/e(n,{componentIdentifier:\"module:pVk4QsoHxASnVtUBp6jr/TbhpORLndv1iOkZzyo83/CodeBlock.js:default\",children:t=>/*#__PURE__*/e(r,{...t,code:\"> pytest\\n\\ntests/test_recipe_bot.py .                                                                                                                                  [100%]\\n\\n======================================================================== 1 passed in 1.77s ========================================================================\",language:\"JSX\"})})}),/*#__PURE__*/e(\"p\",{children:\"Awesome, the power of no yapping. Cool, this means it will always work right? For any model and input right? No?\"}),/*#__PURE__*/e(\"h4\",{children:\"Problem #1: Unpredictable inputs\"}),/*#__PURE__*/e(\"p\",{children:\"Users can send any input they want, but our prompt not necessarily will work perfectly for all the cases. Having a working example gives us a bit of confidence, but you know what would be even better? Way more examples. In a probabilistic environment, that\u2019s the best way to get more confidence.\"}),/*#__PURE__*/t(\"p\",{children:[\"Luckily, we can use pytest\u2019s \",/*#__PURE__*/e(\"code\",{children:\"parametrize\"}),\" for that:\"]}),/*#__PURE__*/e(\"div\",{className:\"framer-text-module\",style:{height:\"auto\",width:\"100%\"},children:/*#__PURE__*/e(n,{componentIdentifier:\"module:pVk4QsoHxASnVtUBp6jr/TbhpORLndv1iOkZzyo83/CodeBlock.js:default\",children:t=>/*#__PURE__*/e(r,{...t,code:'import pandas as pd\\n\\nentries = pd.DataFrame(\\n    {\\n        \"input\": [\\n            \"Generate me a recipe for a quick breakfast with bacon\",\\n            \"Generate me a recipe for a lunch using lentils\",\\n            \"Generate me a recipe for a vegetarian dessert\",\\n            \"Generate me a super complicated recipe for a dinner with a lot of ingredients\",\\n        ],\\n    }\\n)\\n\\n@pytest.mark.asyncio\\n@pytest.mark.parametrize(\"entry\", entries.itertuples())\\nasync def test_fits_a_tweet(entry):\\n    recipe = await recipe_bot.generate_tweet(\\n        input=entry.input,\\n        model=\"gpt-3.5-turbo\",\\n    )\\n\\n    assert len(recipe) <= 140',language:\"Python\"})})}),/*#__PURE__*/e(\"p\",{children:\"Now we have not only one example, but four, and notice that on the last one, the user is trying to be a smartass and jailbreaking a bit, let\u2019s see if our bot is resilient to that!\"}),/*#__PURE__*/e(\"div\",{className:\"framer-text-module\",style:{height:\"auto\",width:\"100%\"},children:/*#__PURE__*/e(n,{componentIdentifier:\"module:pVk4QsoHxASnVtUBp6jr/TbhpORLndv1iOkZzyo83/CodeBlock.js:default\",children:t=>/*#__PURE__*/e(r,{...t,code:\"> pytest\\n\\ntests/test_recipe_bot.py .                                                                                                                                  [100%]\\n\\n======================================================================== 4 passed in 7.10s ========================================================================\",language:\"JSX\"})})}),/*#__PURE__*/t(\"p\",{children:[\"Yes, it is! Nice, this gives us more confidence. Notice it says \",/*#__PURE__*/e(\"code\",{children:\"4 passed\"}),\", by using \",/*#__PURE__*/e(\"code\",{children:\"parametrize\"}),\" with the inputs examples, one test became four already.\"]}),/*#__PURE__*/e(\"p\",{children:\"Those are things you would probably try by hand as well to break your own bot, however, it\u2019s exhausting doing it every time you change the prompt, it is super convenient to have it automated, so you can tune your prompts worry-free.\"}),/*#__PURE__*/e(\"p\",{children:\"But wait, what if you just run again? Actually, sometimes it fails!\"}),/*#__PURE__*/e(\"div\",{className:\"framer-text-module\",style:{height:\"auto\",width:\"100%\"},children:/*#__PURE__*/e(n,{componentIdentifier:\"module:pVk4QsoHxASnVtUBp6jr/TbhpORLndv1iOkZzyo83/CodeBlock.js:default\",children:t=>/*#__PURE__*/e(r,{...t,code:\"============================================================================ FAILURES =============================================================================\\nFAILED tests/test_recipe_bot.py::test_fits_a_tweet[entry3-gpt-3.5-turbo] - AssertionError: assert 141 <= 140\\n============================================================= 1 failed, 3 passed, 1 warning in 7.12s ==============================================================\",language:\"YAML\"})})}),/*#__PURE__*/e(\"p\",{children:\"Almost! By one extra character! But how come? The tests just passed on the previous run, my temperature is 0.0, they should always pass every time right? No?\"}),/*#__PURE__*/e(\"h4\",{children:\"Problem #2: Flakiness\"}),/*#__PURE__*/t(\"p\",{children:[\"This is a big one for LLMs, flakiness is way more inherent for LLMs than in traditional software, even when temperature is zero. Luckily, there is yet another library can than help us here, and the name is kinda obvious too, \",/*#__PURE__*/e(a,{href:\"https://pypi.org/project/flaky/\",motionChild:!0,nodeId:\"LYxDZg6nX\",openInNewTab:!0,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(o.a,{children:\"flaky\"})}),\"! Flaky allows us to mark a test to be retried a number of times, until it works:\"]}),/*#__PURE__*/e(\"div\",{className:\"framer-text-module\",style:{height:\"auto\",width:\"100%\"},children:/*#__PURE__*/e(n,{componentIdentifier:\"module:pVk4QsoHxASnVtUBp6jr/TbhpORLndv1iOkZzyo83/CodeBlock.js:default\",children:t=>/*#__PURE__*/e(r,{...t,code:'  @pytest.mark.asyncio\\n+ @pytest.mark.flaky(max_runs=3)\\n  @pytest.mark.parametrize(\"entry\", entries.itertuples())\\n  async def test_fits_a_tweet(entry):\\n      recipe = await recipe_bot.generate_tweet(\\n          input=entry.input,\\n          model=\"gpt-3.5-turbo\",\\n      )\\n\\n      assert len(recipe) <= 140',language:\"Python\"})})}),/*#__PURE__*/e(\"p\",{children:\"With this change, we allow each LLM to retry 3 times before the verdict, effectively doing 3-shot attempts at the problem, and now, let\u2019s see how it goes:\"}),/*#__PURE__*/e(\"div\",{className:\"framer-text-module\",style:{height:\"auto\",width:\"100%\"},children:/*#__PURE__*/e(n,{componentIdentifier:\"module:pVk4QsoHxASnVtUBp6jr/TbhpORLndv1iOkZzyo83/CodeBlock.js:default\",children:t=>/*#__PURE__*/e(r,{...t,code:\"> pytest\\n\\ntests/test_recipe_bot.py ....                                                                                                                                [100%]\\n================================================================== 4 passed, 1 warning in 6.80s ===================================================================\",language:\"JSX\"})})}),/*#__PURE__*/e(\"p\",{children:\"Nice! Everything passes! It means that with a couple more attempt, our bot can successfully keep under 140 characters.\"}),/*#__PURE__*/e(\"p\",{children:\"But what are we really saying here? Sometimes it will fail and sometimes it will work (at least 33% of the time), but likely in production, you won\u2019t retry many times, and you won\u2019t even know if what you generated is wrong or not (unless you implement the same test at runtime, like a guardrail) so you can retry, so our test is not giving us that much confidence this way (only 33%).\"}),/*#__PURE__*/t(\"p\",{children:[\"Expecting 100% also doesn\u2019t work though, LLMs are inherently flaky, but how about instead of just defining an upper bound, we define a lower one as well? I want to be able to say that, in production, I expect this to work at least 67% of the time. We can do that by using the \",/*#__PURE__*/e(\"code\",{children:\"min_passes\"}),\" argument:\"]}),/*#__PURE__*/e(\"div\",{className:\"framer-text-module\",style:{height:\"auto\",width:\"100%\"},children:/*#__PURE__*/e(n,{componentIdentifier:\"module:pVk4QsoHxASnVtUBp6jr/TbhpORLndv1iOkZzyo83/CodeBlock.js:default\",children:t=>/*#__PURE__*/e(r,{...t,code:'  @pytest.mark.asyncio\\n- @pytest.mark.flaky(max_runs=3)\\n+ @pytest.mark.flaky(max_runs=3, min_passes=2)\\n  @pytest.mark.parametrize(\"entry\", entries.itertuples())\\n  async def test_fits_a_tweet(entry):\\n      recipe = await recipe_bot.generate_tweet(\\n          input=entry.input,\\n          model=\"gpt-3.5-turbo\",\\n      )\\n\\n      assert len(recipe) <= 140',language:\"Python\"})})}),/*#__PURE__*/e(\"p\",{children:\"Presto! Now you are saying it can try 3 times, but it should pass at least in 2 of them, ~67% of the time.\"}),/*#__PURE__*/e(\"p\",{children:\"But does this mean your unit tests are saying now that it\u2019s okay for things to fail sometimes? That might make a lot of people uncomfortable, however we are on a probabilistic world remember, having it defined, on your specifications, the pass rate of your executions is already way better than where we started off, completely in the dark, now at least it\u2019s just a matter of increasing those numbers, as you iterate and improve your solution.\"}),/*#__PURE__*/t(\"p\",{children:[\"Now, what if we try a different model then? A smarter one like \",/*#__PURE__*/e(\"code\",{children:\"gpt-4o\"}),\"\\xa0? If it works on \",/*#__PURE__*/e(\"code\",{children:\"gpt-3.5-turbo\"}),\" then it should be a breeze, right? What about an open-source model like \",/*#__PURE__*/e(\"code\",{children:\"llama3\"}),\"? After all the prompts are portable right? No?\"]}),/*#__PURE__*/e(\"h4\",{children:\"Problem #3: Prompt portability\"}),/*#__PURE__*/t(\"p\",{children:[\"This is a problem for LLMs that unit tests can actually help solving, when you switch models or merely upgrade them, your prompts may stop working. Since we have \",/*#__PURE__*/e(\"code\",{children:\"parametrize\"}),\" and litellm, we can in fact easily run our test examples against different models and see how they behave:\"]}),/*#__PURE__*/e(\"div\",{className:\"framer-text-module\",style:{height:\"auto\",width:\"100%\"},children:/*#__PURE__*/e(n,{componentIdentifier:\"module:pVk4QsoHxASnVtUBp6jr/TbhpORLndv1iOkZzyo83/CodeBlock.js:default\",children:t=>/*#__PURE__*/e(r,{...t,code:'from itertools import product\\n\\nmodels = [\"gpt-3.5-turbo\", \"gpt-4o\", \"groq/llama3-70b-8192\"]\\n\\n@pytest.mark.asyncio\\n@pytest.mark.flaky(max_runs=3, min_passes=2)\\n@pytest.mark.parametrize(\"entry, model\", product(entries.itertuples(), models))\\nasync def test_fits_a_tweet(entry, model):\\n    recipe = await recipe_bot.generate_tweet(\\n        input=entry.input,\\n        model=model,\\n    )\\n\\n    assert len(recipe) <= 140',language:\"Python\"})})}),/*#__PURE__*/e(\"p\",{children:\"This way we pair up each input entry with each model we want to test against, let\u2019s see the results:\"}),/*#__PURE__*/e(\"div\",{className:\"framer-text-module\",style:{height:\"auto\",width:\"100%\"},children:/*#__PURE__*/e(n,{componentIdentifier:\"module:pVk4QsoHxASnVtUBp6jr/TbhpORLndv1iOkZzyo83/CodeBlock.js:default\",children:t=>/*#__PURE__*/e(r,{...t,code:\"> pytest\\n\\ntests/test_recipe_bot.py ........F.FF                                                                                                                       [100%]\\n\\n============================================================================ FAILURES =============================================================================\\nFAILED tests/test_recipe_bot.py::test_fits_a_tweet[entry8-groq/llama3-70b-8192] - assert 151 <= 140\\nFAILED tests/test_recipe_bot.py::test_fits_a_tweet[entry10-gpt-4o] - AssertionError: assert 148 <= 140\\nFAILED tests/test_recipe_bot.py::test_fits_a_tweet[entry11-groq/llama3-70b-8192] - assert 320 <= 140\\n============================================================= 3 failed, 9 passed, 1 warning in 45.21s =============================================================\",language:\"YAML\"})})}),/*#__PURE__*/t(\"p\",{children:[\"You can see that actually GPT-4 and Llama3 fail this test, with both failing on the last smart-ass user example. But before we get into that, have you noticed something else? \",/*#__PURE__*/e(\"strong\",{children:\"45 seconds\"}),\"! Our tests took 45 seconds to run, and this is just our first test, how long will it take this suite when the project grows? That\u2019s not good.\"]}),/*#__PURE__*/e(\"h4\",{children:\"Problem #4: LLMs are\\xa0slow\"}),/*#__PURE__*/t(\"p\",{children:[\"Even though \",/*#__PURE__*/e(a,{href:\"https://x.com/dev__digest/status/1781395494796657008?s=46\",motionChild:!0,nodeId:\"LYxDZg6nX\",openInNewTab:!0,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(o.a,{children:\"groq is crazy fast\"})}),\", it\u2019s still orders of magnitude slower than ordinary code we unit test, but actually, in our case, those LLMs are not running on our machine but through an API, so the problem here really is that they are being called sequentially, instead of in parallel.\"]}),/*#__PURE__*/t(\"p\",{children:[\"Luckily, we can use \",/*#__PURE__*/e(a,{href:\"https://github.com/willemt/pytest-asyncio-cooperative\",motionChild:!0,nodeId:\"LYxDZg6nX\",openInNewTab:!0,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(o.a,{children:\"pytest-asyncio-cooperative\"})}),\" (use \",/*#__PURE__*/e(a,{href:\"https://github.com/willemt/pytest-asyncio-cooperative/pull/66\",motionChild:!0,nodeId:\"LYxDZg6nX\",openInNewTab:!0,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(o.a,{children:\"my branch\"})}),\" if having issues with retries later), this library allows pytest tests to run all at the same time:\"]}),/*#__PURE__*/e(\"div\",{className:\"framer-text-module\",style:{height:\"auto\",width:\"100%\"},children:/*#__PURE__*/e(n,{componentIdentifier:\"module:pVk4QsoHxASnVtUBp6jr/TbhpORLndv1iOkZzyo83/CodeBlock.js:default\",children:t=>/*#__PURE__*/e(r,{...t,code:'- @pytest.mark.asyncio\\n+ @pytest.mark.asyncio_cooperative\\n  @pytest.mark.parametrize(\"entry, model\", product(entries.itertuples(), models))\\n  async def test_fits_a_tweet(entry, model):\\n      recipe = await recipe_bot.generate_tweet(\\n          input=entry.input,\\n          model=model,\\n      )\\n  \\n      assert len(recipe) <= 140',language:\"Python\"})})}),/*#__PURE__*/e(\"p\",{children:\"Now let\u2019s run again:\"}),/*#__PURE__*/e(\"div\",{className:\"framer-text-module\",style:{height:\"auto\",width:\"100%\"},children:/*#__PURE__*/e(n,{componentIdentifier:\"module:pVk4QsoHxASnVtUBp6jr/TbhpORLndv1iOkZzyo83/CodeBlock.js:default\",children:t=>/*#__PURE__*/e(r,{...t,code:\"> pytest\\n\\n============================================================================ FAILURES =============================================================================\\nFAILED tests/test_recipe_bot.py::test_fits_a_tweet[entry9-gpt-3.5-turbo] - AssertionError: assert 141 <= 140\\nFAILED tests/test_recipe_bot.py::test_fits_a_tweet[entry11-groq/llama3-70b-8192] - assert 308 <= 140\\nFAILED tests/test_recipe_bot.py::test_fits_a_tweet[entry10-gpt-4o] - AssertionError: assert 148 <= 140\\n============================================================= 4 failed, 8 passed, 1 warning in 27.88s ==============================================================\",language:\"YAML\"})})}),/*#__PURE__*/e(\"p\",{children:\"Tests still fail of course, but now it takes 27s, way shorter time, but still a lot due to the retries that end up happening sequentially. If we remove the retries it all runs in 2s, try it out! A fast feedback loop is really important for you to keep tweaking and iterating comfortably, otherwise you will just get annoyed and throw the tests out of the window.\"}),/*#__PURE__*/e(\"p\",{children:\"Nice, now that we have bit more speed, let\u2019s go back to the previous problem then, we could try to adjust the prompt to make it work well for all three, but perhaps you don\u2019t actually need it to work on every single prompt, even with retries, again on define boundaries, if you have a big dataset, inevitably some models or some prompts will have a trade-off and being better to answer some inputs well, but others not so much.\"}),/*#__PURE__*/e(\"h4\",{children:\"Problem #5: Pass\\xa0Rate\"}),/*#__PURE__*/e(\"p\",{children:\"Just like we did retries for every single item and established a minimum pass rate there, there are some items that your LLM won\u2019t be able to get right no matter how many times you try, however, it doesn\u2019t mean it\u2019s worthless, getting the examples on your dataset right 80% of the time is still better than getting it right 50% of the time, and again, you then get explicit on your limitations rather than in the dark, and can only improve from there.\"}),/*#__PURE__*/t(\"p\",{children:[\"In our case for example, maybe it\u2019s okay that GPT-4 or Llama3 fails in one of the tasks after trying 3 times, we still consider it good enough. So we need then a pass rate across the board, a global pass rate. For that, we are going to use the library \",/*#__PURE__*/e(a,{href:\"https://github.com/langwatch/langevals/\",motionChild:!0,nodeId:\"LYxDZg6nX\",openInNewTab:!0,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(o.a,{children:\"langevals\"})}),\", which is all about providing evaluations for LLMs, but it also provides this nice \",/*#__PURE__*/e(\"code\",{children:\"pass_rate\"}),\" pytest annotation:\"]}),/*#__PURE__*/e(\"div\",{className:\"framer-text-module\",style:{height:\"auto\",width:\"100%\"},children:/*#__PURE__*/e(n,{componentIdentifier:\"module:pVk4QsoHxASnVtUBp6jr/TbhpORLndv1iOkZzyo83/CodeBlock.js:default\",children:t=>/*#__PURE__*/e(r,{...t,code:'  entries = pd.DataFrame(\\n      {\\n          \"input\": [\\n              \"Generate me a recipe for a quick breakfast with bacon\",\\n              \"Generate me a recipe for a lunch using lentils\",\\n              \"Generate me a recipe for a vegetarian dessert\",\\n-             \"Generate me a super complicated recipe for a dinner with a lot of ingredients\",\\n          ],\\n      }\\n  )\\n\\n  @pytest.mark.asyncio_cooperative\\n  @pytest.mark.flaky(max_runs=3, min_passes=2)\\n+ @pytest.mark.pass_rate(0.8)\\n  @pytest.mark.parametrize(\"entry, model\", product(entries.itertuples(), models))\\n  async def test_fits_a_tweet(entry, model):\\n      recipe = await recipe_bot.generate_tweet(\\n          input=entry.input,\\n          model=model,\\n      )\\n  \\n      assert len(recipe) <= 140',language:\"Python\"})})}),/*#__PURE__*/e(\"p\",{children:\"With this parameter, now we are saying that the minimum passes we expect is 80%, we are okay with 20% of failed inputs, as long as most of them (80%) are indeed less than 140 characters. I\u2019ve also removed the last tricky example, it was indeed throwing the other models completely off, we can add it back later after improving our prompt.\"}),/*#__PURE__*/e(\"div\",{className:\"framer-text-module\",style:{height:\"auto\",width:\"100%\"},children:/*#__PURE__*/e(n,{componentIdentifier:\"module:pVk4QsoHxASnVtUBp6jr/TbhpORLndv1iOkZzyo83/CodeBlock.js:default\",children:t=>/*#__PURE__*/e(r,{...t,code:\"> pytest\\n\\n======================================================================= test session starts =======================================================================\\n\\nx.xx......\\n\\n============================================================ 8 passed, 3 xfailed, 1 warning in 20.42s =============================================================\",language:\"JSX\"})})}),/*#__PURE__*/t(\"p\",{children:[\"Nice! Now our test suite passes, even though not all tests passed completely, notice the \",/*#__PURE__*/e(\"code\",{children:\"3 xfailed\"}),\" ones, those failed, but were \u201Cforgiven\u201D, either because they were retried and worked, or because they are within the pass rate margin.\"]}),/*#__PURE__*/e(\"p\",{children:\"We have now all the tools we need to write tests for LLMs, and making our uncertainty rates clear. But TDD is about creating writing more tests, so you can add more features to your application, so let\u2019s finally jump into it.\"}),/*#__PURE__*/e(\"h3\",{children:\"TDD for adding more\\xa0features\"}),/*#__PURE__*/e(\"p\",{children:\"Having 140 characters is nice and well, but that is a too simple of a test to be honest, and it would be easy enough to just cut the string short to fit in there. The more interesting use cases for LLMs is not something like that, the more interesting cases for LLMs is actually being able to do things that code itself cannot do, that cannot be expressed with logic, those are also the most interesting features for your solution to have.\"}),/*#__PURE__*/e(\"p\",{children:\"For example, let\u2019s make our bot output only vegeratian recipes, we want to go green and save the planet! This is something that cannot be expressed with basic programming logic, but LLMs are perfect for. However, how do we evaluate this? There is no pytest assertion for that! Turns out those interesting features, are also the most interesting ones to be evaluated.\"}),/*#__PURE__*/e(\"h4\",{children:\"Problem #6: Evaluations\"}),/*#__PURE__*/t(\"p\",{children:[\"Enter \",/*#__PURE__*/e(a,{href:\"https://github.com/langwatch/langevals/\",motionChild:!0,nodeId:\"LYxDZg6nX\",openInNewTab:!0,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(o.a,{children:\"LangEvals\"})}),\"! LangEvals is a library that aggregates various LLM-focused evaluators, exactly for the purpose of evaluating and validating LLMs,.\"]}),/*#__PURE__*/t(\"p\",{children:[\"Validating if our recipe is really vegetarian is really hard to do with traditional code, but it does seems like something an LLM would be good to. LangEvals does not have a \u201Cvegetarian evaluator\u201D out-of-the-box, however it does have a \",/*#__PURE__*/e(\"code\",{children:\"CustomLLMBooleanEvaluator\"}),\", which allows us to easily build a LLM-as-a-judge evaluator, which behinds the scene uses function calling to always return \",/*#__PURE__*/e(\"code\",{children:\"true\"}),\" or \",/*#__PURE__*/e(\"code\",{children:\"false\"}),\" for whatever you want to evaluate.\"]}),/*#__PURE__*/t(\"p\",{children:[\"So let\u2019s write our test with it (before the implementation of course) together with the \",/*#__PURE__*/e(\"code\",{children:\"expect\"}),\" utility:\"]}),/*#__PURE__*/e(\"div\",{className:\"framer-text-module\",style:{height:\"auto\",width:\"100%\"},children:/*#__PURE__*/e(n,{componentIdentifier:\"module:pVk4QsoHxASnVtUBp6jr/TbhpORLndv1iOkZzyo83/CodeBlock.js:default\",children:t=>/*#__PURE__*/e(r,{...t,code:'from langevals import expect\\nfrom langevals_langevals.llm_boolean import (\\n    CustomLLMBooleanEvaluator,\\n    CustomLLMBooleanSettings,\\n)\\n\\n@pytest.mark.asyncio_cooperative\\n@pytest.mark.parametrize(\"entry, model\", product(entries.itertuples(), models))\\nasync def test_llm_as_judge(entry, model):\\n    recipe = await recipe_bot.generate_tweet(input=entry.input, model=model)\\n\\n    vegetarian_checker = CustomLLMBooleanEvaluator(\\n        settings=CustomLLMBooleanSettings(\\n            prompt=\"Look at the output recipe. Is the recipe vegetarian?\",\\n        )\\n    )\\n\\n    expect(input=entry.input, output=recipe).to_pass(vegetarian_checker)',language:\"Python\"})})}),/*#__PURE__*/e(\"p\",{children:\"Since we are TDDing, let\u2019s run this test to make sure it is failing and we actually need to implement anything (maybe it\u2019s vegetarian by default, so we would have nothing to do!). Let\u2019s check:\"}),/*#__PURE__*/e(\"div\",{className:\"framer-text-module\",style:{height:\"auto\",width:\"100%\"},children:/*#__PURE__*/e(n,{componentIdentifier:\"module:pVk4QsoHxASnVtUBp6jr/TbhpORLndv1iOkZzyo83/CodeBlock.js:default\",children:t=>/*#__PURE__*/e(r,{...t,code:\"> pytest\\n\\ntests/test_recipe_bot.py F..F.F.F.\\n\\n============================================================================ FAILURES =============================================================================\\nFAILED tests/test_recipe_bot.py::test_llm_as_judge[entry2-groq/llama3-70b-8192] - AssertionError: Custom LLM Boolean Evaluator to_pass FAILED - The recipe is not vegetarian because it contains bacon.\\nFAILED tests/test_recipe_bot.py::test_llm_as_judge[entry1-gpt-4o] - AssertionError: Custom LLM Boolean Evaluator to_pass FAILED - The recipe is not vegetarian because it contains bacon.\\nFAILED tests/test_recipe_bot.py::test_llm_as_judge[entry0-gpt-3.5-turbo] - AssertionError: Custom LLM Boolean Evaluator to_pass FAILED - The recipe is not vegetarian because it contains bacon.\\n============================================================= 3 failed, 6 passed, 1 warning in 6.99s ==============================================================\",language:\"YAML\"})})}),/*#__PURE__*/e(\"p\",{children:\"The test failed for the first entry on all models. Of course, because the first input asked for a \u201Cquick breakfast with bacon\u201D, so it\u2019s not vegetarian. The LLM Boolean evaluator caught that, and nicely tells us why it decided it fails right there on the error message, and it is because the output recipe contains bacon, which is not vegetarian.\"}),/*#__PURE__*/e(\"p\",{children:\"Now you might be thinking, isn\u2019t it weird that an LLM is grading it\u2019s own homework? Won\u2019t it just agree with itself most of the time? Well, not necessarily, first of all because you can use a different, smarter model to evaluate the one you will actually use in production at scale, for example GPT-4 evaluating GPT-3.5 results, but also, because this prompt here is way more focus, it does one thing and one thing well only, it won\u2019t break because you are not updating it and doing many things at once like your main bot\u2019s prompt.\"}),/*#__PURE__*/e(\"p\",{children:\"This is one of the very important reasons for us to run our tests before write the code, so we \u201Ctest our test\u201D. By the reasoning on the outputs, it seems to be working perfectly well, so now we can trust it will evaluate our actual implementation code. Let\u2019s go ahead and update our prompt then:\"}),/*#__PURE__*/e(\"div\",{className:\"framer-text-module\",style:{height:\"auto\",width:\"100%\"},children:/*#__PURE__*/e(n,{componentIdentifier:\"module:pVk4QsoHxASnVtUBp6jr/TbhpORLndv1iOkZzyo83/CodeBlock.js:default\",children:t=>/*#__PURE__*/e(r,{...t,code:'  async def generate_tweet(input: str, model: str = \"gpt-3.5-turbo\") -> str:\\n      response: ModelResponse = await litellm.acompletion(\\n          model=model,\\n          messages=[\\n              {\\n                  \"role\": \"system\",\\n-                 \"content\": \"You are a recipe tweet generator, generate recipes using a max of 140 characters, just the ingredients, no yapping.\",\\n+                 \"content\": \"You are a recipe tweet generator, generate recipes using a max of 140 characters, just the ingredients, no yapping. Also, the recipe must be vegetarian.\",\\n              },\\n              {\"role\": \"user\", \"content\": input},\\n          ],\\n          temperature=0.0,\\n      )  # type: ignore\\n  \\n      return response.choices[0].message.content  # type: ignore',language:\"Python\"})})}),/*#__PURE__*/e(\"p\",{children:\"Small addition of \u201CAlso, the recipe must be vegetarian.\u201D, let\u2019s see how it behaves:\"}),/*#__PURE__*/e(\"div\",{className:\"framer-text-module\",style:{height:\"auto\",width:\"100%\"},children:/*#__PURE__*/e(n,{componentIdentifier:\"module:pVk4QsoHxASnVtUBp6jr/TbhpORLndv1iOkZzyo83/CodeBlock.js:default\",children:t=>/*#__PURE__*/e(r,{...t,code:\"tests/test_recipe_bot.py .........\\n\\n================================================================== 9 passed, 1 warning in 9.19s ===================================================================\",language:\"JSX\"})})}),/*#__PURE__*/e(\"p\",{children:\"Nice! It all works! And since you wrote the tests before, you don\u2019t need to add any new tests now, you are done, commit, push, and boom, you have a new feature in production, and can change your prompt with peace of mind that this very important concept on your product of having the recipes to be vegetarian will continue working.\"}),/*#__PURE__*/e(\"p\",{children:\"For the next feature, you can just think of what you expect the model to output first, verify it, and then implement, this will help you ensure you always move only forward, and don\u2019t break features that were previously working, even in this highly unpredictable environment.\"}),/*#__PURE__*/e(\"h3\",{children:\"Wrapping Up\"}),/*#__PURE__*/e(\"p\",{children:\"I hope this guide gave you a good idea on what is it like to TDD with LLMs, all the challenged on writing tests for it, and ways how to handle each.\"}),/*#__PURE__*/e(\"p\",{children:\"The field of LLM evaluations is still a very new one and in fast expansion, every day people are exploring new ideas and finding more and more ways that we can control those very powerful but unpredictable creatures.\"}),/*#__PURE__*/t(\"p\",{children:[\"Check out \",/*#__PURE__*/e(a,{href:\"https://docs.langwatch.ai/langevals/documentation/evaluators\",motionChild:!0,nodeId:\"LYxDZg6nX\",openInNewTab:!0,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(o.a,{children:\"LangEvals Documentation\"})}),\" for all the available evaluators, contributions for new ones are also very welcome!\"]}),/*#__PURE__*/t(\"p\",{children:[\"Also, if you like this guide, please give LangEvals a \",/*#__PURE__*/e(a,{href:\"https://github.com/langwatch/langevals\",motionChild:!0,nodeId:\"LYxDZg6nX\",openInNewTab:!0,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(o.a,{children:\"star on github\"})}),\", and for all those bugs that cannot be captured by unit tests, check out \",/*#__PURE__*/e(a,{href:\"https://github.com/langwatch/langwatch\",motionChild:!0,nodeId:\"LYxDZg6nX\",openInNewTab:!0,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(o.a,{children:\"LangWatch\"})}),\", our LLM Ops monitoring platform, powered by LangEvals.\"]}),/*#__PURE__*/e(\"p\",{children:\"Good testing!\"})]});export const richText6=/*#__PURE__*/t(i.Fragment,{children:[/*#__PURE__*/e(\"p\",{children:\"It does not come as a shock for anybody when I say that data is one of the most valuable assets a company can have, this has been true for some years now. Valuable data can be used to build very powerful products, and with the invention of LLMs, it became even more obvious that companies that already have valuable data internally can leverage AI to create a bigger edge.\"}),/*#__PURE__*/e(\"p\",{children:\"But internal data is not the one I want to talk about today, I want to talk about the data that is produced as a byproduct of your AI being in production: all the interactions your users have with it, both with happy and unhappy outcomes. This is super valuable data, it\u2019s literally the best insight on how users expect your product to behave, and now it comes in conversational text format, it's not just numbers like traditional product analytics, so it can directly feed back into the AI.\"}),/*#__PURE__*/e(\"p\",{children:\"The thing with LLMs, is that they are super powerful and general, this is the exact reason we want to use them that, they can answer any questions and execute a large range of tasks, however exactly because of that, their surface area to the real world usage is extremely large, the more they do the larger it is, resulting in countless different ways it can fall short. It is hopeless to try and predict all the ways your users will want to use your product, and try covering all the possibilities and edge cases beforehand.\"}),/*#__PURE__*/e(\"h4\",{children:\"How can a Data Flywheel help LLM apps?\"}),/*#__PURE__*/e(\"img\",{alt:\"Data Flywheel\",className:\"framer-image\",height:\"422\",src:\"https://framerusercontent.com/images/Uk1CtDMF19hXLyH38aCmSfRJiqo.png\",srcSet:\"https://framerusercontent.com/images/Uk1CtDMF19hXLyH38aCmSfRJiqo.png?scale-down-to=512 512w,https://framerusercontent.com/images/Uk1CtDMF19hXLyH38aCmSfRJiqo.png 928w\",style:{aspectRatio:\"928 / 844\"},width:\"464\"}),/*#__PURE__*/e(\"p\",{children:\"We call it a flywheel when an improvement brings business value and that loops back on further improvements. Since the number of possibilities of an LLM product is too vast - think of all possible ways a user can try using your product, or even compose a sentence - nothing is better at selecting what is more valuable and relevant like the real world, this makes it essential to monitor and get insights on where the product is suboptimal, or where it could bring even more value.\"}),/*#__PURE__*/e(\"p\",{children:\"Then by using those insights for improving the product and making it more valuable, it will receive more usage and more users, it will gain more trust and cover more cases to be used in more places. By getting more usage, even more scenarios are revealed, new paths to bring even more value, and new shortcomings to overcome and make the product more robust, which will lead to even more usage an so on. That\u2019s the flywheel.\"}),/*#__PURE__*/e(\"p\",{children:\"The advantage with AI over traditional software improvements, is that this connection is even more direct, LLMs can be used also to evaluate, classify and synthesize data that will improve the product, so this flywheel is closed even tighter together.\"}),/*#__PURE__*/e(\"p\",{children:\"There is, however, no shortcuts, if your solution has been finding new unexpected edge cases for months or years, it is not possible for someone else to just guess which ones they are and port them over elsewhere without going through the same discovery process. The improvement process you done on your product for your specific domain then creates an edge that sets you apart from others as soon as you start it.\"}),/*#__PURE__*/e(\"h4\",{children:\"LangWatch\"}),/*#__PURE__*/e(\"p\",{children:\"LangWatch was built on top of this realization, to help you maximize your data flywheel. Our platform helps both on collection part, with monitoring, evaluations, insights, collecting user feedback and annotating with domain experts from your team, and on the improvement part, running experiments, validating and guaranteeing the improved quality of each iteration.\"}),/*#__PURE__*/t(\"p\",{children:[\"Interested how it can help your GenAI solution? \",/*#__PURE__*/e(a,{href:\"https://get.langwatch.ai/request-a-demo\",motionChild:!0,nodeId:\"LYxDZg6nX\",openInNewTab:!0,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(o.a,{children:\"Book a demo with us\"})}),\".\"]})]});export const richText7=/*#__PURE__*/t(i.Fragment,{children:[/*#__PURE__*/t(\"p\",{children:[\"Last week we met with one of our first customers - \",/*#__PURE__*/e(a,{href:\"algomo.com\",motionChild:!0,nodeId:\"LYxDZg6nX\",openInNewTab:!0,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(o.a,{children:\"Algomo\"})}),\", a platform specializing in AI agents that automate and personalize customer interactions. They have integrated LangWatch into their AI operations to improve performance and maintain high-quality interactions for their customers.  \"]}),/*#__PURE__*/e(\"p\",{children:\"Who did we meet with?\"}),/*#__PURE__*/t(\"ul\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Kenneth\"}),\": Data Scientist at Algomo\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Nikhil\"}),\": AI Researcher focused on securing generative AI at Algomo\"]})})]}),/*#__PURE__*/e(\"p\",{children:\"LangWatch was integrated into Algomo's production environment over a month ago, logging over 10\u2019s of thousands of conversations since its implementation. The team uses LangWatch daily to monitor and improve the performance of their AI agents.\"}),/*#__PURE__*/e(\"h4\",{children:/*#__PURE__*/e(\"strong\",{children:\"Alerting on problematic messages to debug immediately  \"})}),/*#__PURE__*/e(\"p\",{children:\"One of the most impactful features for Algomo is LangWatch's alert system. Kenneth highlights, \u201CAlerts are one of the best things about LangWatch. When there\u2019s a problematic message, we get an alert and can investigate it quick and easily.\u201D The typical workflow involves receiving an alert in their Slack channel, reviewing the chatbot\u2019s response and context, and determining if there\u2019s a hallucination or error. This process has streamlined Algomo's debugging and resolution workflow significantly. It's like having a monitoring system such as Sentry on your software development, but now better and optimized for LLM-powered products.\"}),/*#__PURE__*/e(\"p\",{children:\"An instance of LangWatch\u2019s utility was when it detected outdated data sources being used by the chatbot. Nikhil explains, \u201COne of our clients updated their FAQs, but our system was still using the old data. LangWatch alerted us to this issue, leading us to discover corrupted data in our vector database, which we then refreshed to resolve the problem.\u201D Obviously when hearing this, LangWatch was delighted we could help them solving this inaccurate data.\"}),/*#__PURE__*/e(\"h4\",{children:/*#__PURE__*/e(\"strong\",{children:\"Real deep insights into the product performance\"})}),/*#__PURE__*/e(\"p\",{children:'Nikhil spends two to three hours daily retrieving real deep insights using LangWatch, finding the labels and custom analytics particularly beneficial. \u201CLabels come in handy for manipulating the visibility of how the platform works,\u201D he says. Algomo uses labels to log request inputs (such as client and data source details) and to monitor internal processes. Previously he had no idea where to look at or the AI was a complete \"blackbox\". Now they have the ability to spend their time wisely using LangWatch.'}),/*#__PURE__*/e(\"p\",{children:\"Custom analytics, is a feature in LangWatch to build your own graphs based on all metrics available, allows Algomo to develop tailored graphs for monitoring various metrics, such as the number of messages, response quality, and cost per customer. Nikhil elaborates, \u201CWe have developed four to five custom graphs on the production instance to track message counts, threads, and cost per customer.\u201D\"}),/*#__PURE__*/e(\"p\",{children:\"On the other hand, the flexible filter options and fast search functionality are essential for day-to-day operations. \u201CIf you want to track back a week, the filters are super fast, allowing us to easily filter out and investigate specific cases,\u201D Nikhil notes. This feature aids in quickly identifying and addressing issues, enhancing the efficiency of their monitoring process.\"}),/*#__PURE__*/e(\"p\",{children:\"Algomo plans to enhance their reporting capabilities with LangWatch. They envision using custom analytics and filters to generate detailed reports for clients. Nikhil mentions, \u201CWe\u2019re preparing to create reports using custom analytics, which will give us more visibility into the performance of our chatbots and help in reporting to internal team members and customers.\u201D LangWatch is excited to hear Algomo is using the custom graphs to build extensive reporting for their customers to show how their AI is performing and plan Customer QBR sessions with them showing these results.\"}),/*#__PURE__*/e(\"h4\",{children:/*#__PURE__*/e(\"strong\",{children:\"Preventing Data leakgage\"})}),/*#__PURE__*/e(\"p\",{children:\"LangWatch has significantly improved Algomo\u2019s ability to detect and resolve errors quickly. Kenneth highlights a critical instance where a chatbot used data from a different client, causing a data leakage. \u201CWithout LangWatch, it would be hard to find such issues. Manually going through data would be very time-consuming,\u201D he says. \"}),/*#__PURE__*/e(\"p\",{children:\"The team has noted significant improvements in their ability to monitor and maintain the quality of their AI interactions. The alerts going into slack provided by LangWatch have allowed them to identify and resolve issues that have gone unnoticed otherwise.\\xa0\"}),/*#__PURE__*/e(\"h4\",{children:/*#__PURE__*/e(\"strong\",{children:\"Conclusion\"})}),/*#__PURE__*/e(\"p\",{children:\"Algomo looks forward to further refining its use of LangWatch, especially in reporting and analytics.\"}),/*#__PURE__*/e(\"p\",{children:\"LangWatch has proven to be an invaluable tool for Algomo, offering enhanced monitoring, rapid error detection, and detailed analytics. As Kenneth succinctly puts it, \u201CLangWatch gives us an easy way to get more visibility into the problems of our application.\u201D With ongoing use and planned enhancements, Algomo expects LangWatch to continue playing a crucial role in maintaining and improving their AI-driven customer interactions.\"}),/*#__PURE__*/e(\"h5\",{children:\"Are you curious to learn how LangWatch can have this impact on your AI products?\"}),/*#__PURE__*/e(\"h5\",{children:/*#__PURE__*/e(a,{href:\"https://get.langwatch.ai/request-a-demo\",motionChild:!0,nodeId:\"LYxDZg6nX\",openInNewTab:!0,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(o.a,{children:\"Get a demo now\"})})})]});export const richText8=/*#__PURE__*/t(i.Fragment,{children:[/*#__PURE__*/e(\"p\",{children:\"Large Language Models (LLMs) are rapidly transforming the way we interact with technology. From generating creative text formats to answering complex questions, these AI-powered tools are finding their way into a growing number of applications. However, like any powerful tool, LLMs are not without their limitations. Their vast knowledge base, while impressive, can sometimes lead to outputs that are inaccurate, irrelevant, or simply miss the mark.\"}),/*#__PURE__*/e(\"img\",{alt:\"\",className:\"framer-image\",height:\"217\",src:\"https://framerusercontent.com/images/hxuLWEbmen3hbOrTRhRk7EHEhM.png\",srcSet:\"https://framerusercontent.com/images/hxuLWEbmen3hbOrTRhRk7EHEhM.png?scale-down-to=512 512w,https://framerusercontent.com/images/hxuLWEbmen3hbOrTRhRk7EHEhM.png 862w\",style:{aspectRatio:\"862 / 434\"},width:\"431\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(a,{href:\"https://medium.com/@learnrpawithec/what-is-human-in-the-loop-hitl-29b6b527badc\",motionChild:!0,nodeId:\"LYxDZg6nX\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(o.a,{children:\"Image source\"})})}),/*#__PURE__*/e(\"p\",{children:\"This is where the human touch comes in. By incorporating user feedback and domain expertise into the development process, we can bridge the gap between LLM potential and real-world application.\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"Let's explore the concept of HITL for LLM applications. We will also highlight the importance of a diverse feedback loop and the key players involved in building a successful HITL team.\\xa0\"}),/*#__PURE__*/e(\"h4\",{children:/*#__PURE__*/e(\"strong\",{children:\"The Importance of a Diverse Feedback Loop\"})}),/*#__PURE__*/e(\"p\",{children:\"Imagine building a house without consulting the architect or the people who will live in it. While the basic structure might stand, it wouldn't be functional or meet the specific needs of its residents.\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"The same principle applies to LLM applications. Relying solely on the LLM's internal algorithms can result in outputs that are technically impressive but lack real-world usability or accuracy. This is where the power of a diverse feedback loop comes into play.\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"By incorporating two key perspectives - user experience and domain expertise - we can ensure that LLM applications are not just functional, but truly valuable and effective.\"}),/*#__PURE__*/e(\"h4\",{children:/*#__PURE__*/e(\"strong\",{children:\"The AI Team and Their Roles\"})}),/*#__PURE__*/e(\"p\",{children:\"Developing a successful LLM application requires a well-coordinated team effort. This team brings together a diverse set of skills, each playing a crucial role in the Human in the Loop (HITL) process:\"}),/*#__PURE__*/e(\"h4\",{children:/*#__PURE__*/e(\"strong\",{children:\"1. The Developer\"})}),/*#__PURE__*/e(\"p\",{children:\"The tech whiz of the group is responsible for building and maintaining the LLM infrastructure. They ensure the LLM runs smoothly and implement mechanisms for seamlessly integrating user and domain expert feedback into the system.\"}),/*#__PURE__*/e(\"h4\",{children:/*#__PURE__*/e(\"strong\",{children:\"2. The AI Product Manager\"})}),/*#__PURE__*/e(\"p\",{children:\"The AI Product Manager oversees the entire LLM application development process. They define the product vision, prioritize features, and ensure a user-centric approach. Additionally, they are essential in the development of efficient techniques for gathering feedback, including A/B testing, surveys, and user interface components that facilitate simple input of comments.\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"Product managers also serve as a link between developers, domain experts, and users, promoting open communication and guaranteeing that all opinions are heard.\"}),/*#__PURE__*/e(\"h4\",{children:/*#__PURE__*/e(\"strong\",{children:\"3. The Domain Expert\"})}),/*#__PURE__*/e(\"p\",{children:\"The subject expert brings their specialized understanding of the field to the table. They guide the LLM's training by providing relevant data sets and refining prompts to steer the LLM toward generating accurate and domain-specific outputs. What we have seen so far in our customer conversations is that these personas can typically be a Customer project or experience manager, a product manager in a tech company. But on the other hand a doctor within the medical space or a HR manager for an HR generative a solution.\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"Additionally, they analyze user feedback and help identify areas where the LLM might be straying from domain knowledge or best practices.\"}),/*#__PURE__*/e(\"h4\",{children:/*#__PURE__*/e(\"strong\",{children:\"4. The Users\"})}),/*#__PURE__*/e(\"p\",{children:\"At the heart of the HITL process, users provide real-world data and feedback on the LLM application's functionality. Their interactions with the app reveal its strengths and weaknesses, allowing developers to identify areas for improvement.\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"Users participate in feedback mechanisms, test different LLM outputs, and share their experiences to help shape the application's evolution.\"}),/*#__PURE__*/e(\"p\",{children:\"This diverse team, working together in a collaborative environment, forms the backbone of the HITL approach. Each member plays a distinct yet interconnected role, ensuring the LLM application continuously learns, adapts, and improves based on the combined insights from users and domain experts.\"}),/*#__PURE__*/e(\"h4\",{children:/*#__PURE__*/e(\"strong\",{children:\"Strategies for Effective HITL Integration\"})}),/*#__PURE__*/e(\"p\",{children:\"Building a strong HITL team is just the first step. To truly harness the power of diverse feedback, we need effective strategies for incorporating user and domain expert insights into the LLM app development cycle. Here are some key approaches to consider:\"}),/*#__PURE__*/e(\"h4\",{children:/*#__PURE__*/e(\"strong\",{children:\"User Feedback Collection\"})}),/*#__PURE__*/e(\"p\",{children:\"There are various ways to gather valuable user feedback. Here are a few effective methods:\"}),/*#__PURE__*/t(\"ul\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Surveys\"}),\": Online surveys can efficiently gather user opinions on specific aspects of the LLM app, such as ease of use, clarity of outputs, or desired functionalities.\",/*#__PURE__*/e(\"br\",{}),/*#__PURE__*/e(\"br\",{className:\"trailing-break\"})]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"User Testing Sessions\"}),\": Bringing users into a controlled environment allows for in-depth observation of their interactions with the LLM app. This helps identify usability issues and areas for improvement.\",/*#__PURE__*/e(\"br\",{}),/*#__PURE__*/e(\"br\",{className:\"trailing-break\"})]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"In-App Feedback Mechanisms\"}),': Including elements in the app itself, such as \"report issue\" buttons, comment areas, and star ratings, gives users an easy way to submit direct feedback.']})})]}),/*#__PURE__*/e(\"h4\",{children:/*#__PURE__*/e(\"strong\",{children:\"Domain Expert Collaboration\"})}),/*#__PURE__*/e(\"p\",{children:\"Working with domain specialists is essential to guarantee the dependability and correctness of the LLM's results. The following are some tactics:\"}),/*#__PURE__*/t(\"ul\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Workshops\"}),\": Hosting workshops where domain experts can directly interact with the LLM and developers can be highly beneficial. This allows experts to identify potential biases or knowledge gaps in the LLM and guide its development in the right direction.\",/*#__PURE__*/e(\"br\",{}),/*#__PURE__*/e(\"br\",{className:\"trailing-break\"})]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Knowledge Sharing Sessions\"}),\": Organizing frequent meetings for domain specialists to impart their wisdom to the development team allows for a deeper comprehension of the particular subject and all of its details.\",/*#__PURE__*/e(\"br\",{}),/*#__PURE__*/e(\"br\",{className:\"trailing-break\"})]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Ongoing Consultations\"}),\": Keeping a line of communication open for constant input and direction from subject matter experts facilitates progress.\"]})})]}),/*#__PURE__*/e(\"h4\",{children:/*#__PURE__*/e(\"strong\",{children:\"Feedback Analysis and Prioritization\"})}),/*#__PURE__*/e(\"p\",{children:\"Once collected, user and domain expert feedback needs to be carefully analyzed and prioritized. This involves:\"}),/*#__PURE__*/t(\"ul\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Identifying Trends\"}),\": Look for recurring themes or issues in the feedback data. This helps pinpoint areas requiring the most attention.\",/*#__PURE__*/e(\"br\",{}),/*#__PURE__*/e(\"br\",{className:\"trailing-break\"})]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Prioritizing Issues:\"}),\" Evaluate the severity and impact of each issue on the user experience and overall functionality of the app.\",/*#__PURE__*/e(\"br\",{}),/*#__PURE__*/e(\"br\",{className:\"trailing-break\"})]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Categorizing Feedback\"}),\": Organize feedback based on urgency and type (e.g., usability issues, factual inaccuracies, feature requests). This facilitates the efficient allocation of resources for addressing different feedback categories.\"]})})]}),/*#__PURE__*/e(\"h4\",{children:/*#__PURE__*/e(\"strong\",{children:\"Iterating the LLM Model\"})}),/*#__PURE__*/e(\"p\",{children:\"With prioritized feedback in hand, the development team can begin the process of iterating and refining the LLM model. Here's how feedback fuels the development process:\"}),/*#__PURE__*/t(\"ul\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Refining Prompts\"}),\": Based on user and domain expert feedback, prompts used to guide the LLM can be adjusted to improve the quality and relevance of its outputs.\",/*#__PURE__*/e(\"br\",{}),/*#__PURE__*/e(\"br\",{className:\"trailing-break\"})]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Retraining the LLM:\"}),\" Feedback highlighting factual errors or knowledge gaps can inform the selection of new training data, allowing the LLM to be continually refined and updated.\",/*#__PURE__*/e(\"br\",{}),/*#__PURE__*/e(\"br\",{className:\"trailing-break\"})]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Enhancing the App:\"}),\" User feedback on desired functionalities can guide the development of new features and functionalities within the LLM app.\"]})})]}),/*#__PURE__*/e(\"p\",{children:\"Through a constant process of feedback collecting, analysis, iteration, and validation, the development team can improve the LLM-powered application and guarantee that it is a reliable and valuable tool for its stakeholders and users.\"}),/*#__PURE__*/e(\"h4\",{children:/*#__PURE__*/e(\"strong\",{children:\"Explore the Potential of Your LLM App: Let Langwatch Supercharge Your HITL Strategy!\"})}),/*#__PURE__*/e(\"p\",{children:\"The future of LLM applications is bright, but it hinges on effective human oversight and feedback. Langwatch can be your secret weapon in building a robust HITL team. Our comprehensive language solutions empower you to:\"}),/*#__PURE__*/t(\"ul\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[\"Gather high-quality user feedback through advanced sentiment analysis and targeted surveys.\",/*#__PURE__*/e(\"br\",{}),/*#__PURE__*/e(\"br\",{className:\"trailing-break\"})]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[\"Bridge the communication gap with domain experts through expert-curated training data and seamless knowledge-sharing platforms.\",/*#__PURE__*/e(\"br\",{}),/*#__PURE__*/e(\"br\",{className:\"trailing-break\"})]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/e(\"p\",{children:\"Refine your LLM's outputs with cutting-edge natural language processing tools for superior accuracy and clarity.\"})})]}),/*#__PURE__*/e(\"h5\",{children:\"Ready to unlock the true potential of your LLM app? \"}),/*#__PURE__*/e(\"h5\",{children:/*#__PURE__*/e(a,{href:\"https://get.langwatch.ai/request-a-demo\",motionChild:!0,nodeId:\"LYxDZg6nX\",openInNewTab:!0,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(o.a,{children:\"Get a demo now\"})})})]});export const richText9=/*#__PURE__*/t(i.Fragment,{children:[/*#__PURE__*/e(\"h1\",{children:/*#__PURE__*/e(\"strong\",{children:\"Unit Testing Your LLM: The Power of Datasets\"})}),/*#__PURE__*/e(\"p\",{children:\"With evolving technology, Large Language Models (LLMs) are rapidly becoming powerful tools for tasks ranging from generating creative text formats to translating languages.\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"However, unlike traditional software built on deterministic logic, LLMs operate on probability. This probabilistic nature presents a unique challenge: ensuring these models continue to perform effectively as they evolve. While traditional unit testing methods might come to mind, they fall short when applied to LLMs.\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"Let\u2019s explore a powerful alternative \u2013 leveraging datasets for LLM unit testing. By using the power of real-world examples and performance metrics, we can effectively evaluate and improve the capabilities of our LLMs.\"}),/*#__PURE__*/e(\"h4\",{children:\"Challenges of Unit Testing LLMs\"}),/*#__PURE__*/e(\"p\",{children:\"Imagine a program that doesn't always produce the same output for the same input. Sounds strange, right? Traditional software functions on deterministic logic, meaning a specific input guarantees a specific output. This allows developers to write unit tests \u2013 small, focused checks that verify if a piece of code behaves as expected.\"}),/*#__PURE__*/e(\"p\",{children:\"However, LLMs are fundamentally different. Due to their probabilistic nature, the same prompt can lead to slightly varied responses on different occasions. This inherent randomness throws a wrench into traditional unit testing methods. Here's why:\"}),/*#__PURE__*/t(\"ul\",{children:[/*#__PURE__*/t(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:[/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Pass/Fail Doesn't Apply:\"}),' Traditional tests often rely on a simple \"pass\" or \"fail\" verdict. With LLMs, a single response might not be definitively wrong, but rather less desirable compared to other possible outputs.']}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"br\",{className:\"trailing-break\"})})]}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Limited Scope:\"}),\" Unit tests typically focus on isolated functionalities. LLMs, on the other hand, process information holistically, making it difficult to pinpoint the exact cause of an undesirable output.\"]})})]}),/*#__PURE__*/e(\"h4\",{children:\"The Power of Datasets: Building a Benchmark for Success\"}),/*#__PURE__*/e(\"p\",{children:\"While traditional unit testing methods fall short, datasets offer a robust alternative for LLM evaluation. Here's how this approach works:\"}),/*#__PURE__*/t(\"ul\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Real-World Examples\"}),\": Instead of relying on isolated tests, we can use real-world examples from an LLM's production environment. This allows us to assess its performance on tasks it's designed for. Such as using a dataset of customer service queries to test an LLM's ability to generate helpful responses.\",/*#__PURE__*/e(\"br\",{}),/*#__PURE__*/e(\"br\",{className:\"trailing-break\"})]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Evaluator Selection\"}),': Not all outputs are created equal. To effectively evaluate LLM performance, we need metrics that capture specific desired qualities. These metrics, often employed by \"evaluators,\" could assess factual accuracy, and fluency, or even identify potential biases in the LLM\\'s responses.']})})]}),/*#__PURE__*/e(\"p\",{children:'By combining real-world examples with relevant metrics, we build a comprehensive dataset that serves as a benchmark for LLM performance. This dataset allows us to move beyond simple \"pass/fail\" assessments and gain a nuanced understanding of the LLM\\'s strengths and weaknesses.'}),/*#__PURE__*/e(\"h4\",{children:\"Building Datasets for LLM Testing\"}),/*#__PURE__*/e(\"p\",{children:\"Building a robust test dataset is key to unlocking the power of data-driven LLM evaluation. Here's what goes into this recipe:\"}),/*#__PURE__*/t(\"ul\",{children:[/*#__PURE__*/t(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:[/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Data Collection\"}),\": The first step involves gathering a diverse set of examples relevant to your LLM's intended use. This could involve customer queries, translation prompts, or creative writing prompts, depending on the LLM's function. The more diverse the data, the better it reflects real-world scenarios.\"]}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"br\",{className:\"trailing-break\"})})]}),/*#__PURE__*/t(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:[/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Dataset Preparation\"}),\": Ensure your data is well-formatted and free of errors. This may involve removing irrelevant information or correcting typos. Divide your data into training, validation, and testing sets. A larger dataset generally leads to better LLM performance but ensures the data is high-quality and avoids redundancy.\"]}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"br\",{className:\"trailing-break\"})})]}),/*#__PURE__*/t(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:[/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Defining Metrics\"}),\": Next, identify the qualities you want to assess in your LLM's outputs. For factual accuracy, metrics like BLEU score (for machine translation) can be used. Fluency can be evaluated through metrics like perplexity, which measures the difficulty of predicting the next word. Additionally, bias detection tools can be incorporated to identify and mitigate potential biases in the LLM's responses.\"]}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"br\",{className:\"trailing-break\"})})]}),/*#__PURE__*/t(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:[/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Evaluator Selection\"}),\": Choose appropriate evaluators that align with your chosen metrics. These evaluators, often software tools, analyze the LLM's outputs against the established metrics and provide a score.\"]}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"br\",{className:\"trailing-break\"})})]}),/*#__PURE__*/t(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:[/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Monitoring LLM Performance\"}),\": Run your LLM against the dataset initially to establish baseline performance metrics. After LLM updates or fine-tuning, re-evaluate against the dataset. This reveals performance improvements or regressions.\"]}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"br\",{className:\"trailing-break\"})})]}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Tools and Frameworks\"}),\": Tools like LangWatch automate the evaluation workflow, visualizing LLM performance over time and simplifying dataset-based testing. These tools integrate seamlessly into existing development workflows, making LLM testing efficient and manageable.\"]})})]}),/*#__PURE__*/e(\"p\",{children:\"By combining these ingredients \u2013 diverse data, relevant metrics, and effective evaluators \u2013 you create a powerful test dataset that serves as a compass for LLM improvement.\"}),/*#__PURE__*/e(\"h4\",{children:\"Refining Your LLM\"}),/*#__PURE__*/e(\"p\",{children:\"Once you've established your test dataset, it becomes a valuable tool for monitoring and improving your LLM's performance. Here's how:\"}),/*#__PURE__*/e(\"h4\",{children:\"Tracking Changes\"}),/*#__PURE__*/e(\"p\",{children:\"As you make modifications to your LLM's architecture or training data, run it against the test dataset. The resulting scores from the evaluators will reveal if the changes have positively or negatively impacted the LLM's performance based on your chosen metrics.\"}),/*#__PURE__*/e(\"h4\",{children:\"Identifying Weaknesses\"}),/*#__PURE__*/e(\"p\",{children:\"By analyzing the dataset scores across different types of prompts or scenarios, you can pinpoint areas where the LLM struggles. For instance, if the factual accuracy score falls for a certain kind of query, it means that the LLM's architecture has to be changed or its training data needs to be improved to better handle these kinds of inquiries.\"}),/*#__PURE__*/e(\"p\",{children:\"Through testing, analyzing, and tweaking in an iterative manner, you may continuously enhance the capabilities of your LLM. By automating the evaluation method and offering visualizations of the LLM's performance over time, tools such as Langwatch help simplify this process.\"}),/*#__PURE__*/e(\"h5\",{children:\"Unleash the Power of Your LLM: Streamline Testing with LangWatch!\"}),/*#__PURE__*/e(\"p\",{children:\"Effective LLM unit testing can feel like navigating a maze. But what if there was a tool to guide you toward optimal performance? Look no further than LangWatch!\"}),/*#__PURE__*/e(\"p\",{children:\"LangWatch empowers you to leverage the power of datasets for comprehensive LLM evaluation. This user-friendly platform streamlines the entire process, from crafting diverse test sets to automating evaluations and visualizing performance trends.\"}),/*#__PURE__*/e(\"h5\",{children:\"Ready to unlock the full potential of your LLM?\"}),/*#__PURE__*/t(\"h5\",{children:[/*#__PURE__*/e(a,{href:\"https://get.langwatch.ai/request-a-demo\",motionChild:!0,nodeId:\"LYxDZg6nX\",openInNewTab:!0,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(o.a,{children:\"Get a demo now\"})}),/*#__PURE__*/e(\"br\",{}),/*#__PURE__*/e(\"br\",{className:\"trailing-break\"})]})]});export const richText10=/*#__PURE__*/t(i.Fragment,{children:[/*#__PURE__*/e(\"p\",{children:\"DSPy automatically improves your prompts, allowing you to focus more on the structure and the expectations for your app, and switching models with ease. We found it to be way more effective than experienced human prompt engineers, getting a higher accuracy in a much shorter timeframe.\"}),/*#__PURE__*/e(\"p\",{children:\"If you are building with LLMs and have done your fair share of prompt engineering, LLM calls after LLM calls, trying to improve your product performance, DSPy is the logical next step for you.\"}),/*#__PURE__*/e(\"img\",{alt:\"\",className:\"framer-image\",height:\"432\",src:\"https://framerusercontent.com/images/mbQ53sFYTA0yXgQnJzbYtrCz48.png\",srcSet:\"https://framerusercontent.com/images/mbQ53sFYTA0yXgQnJzbYtrCz48.png?scale-down-to=512 512w,https://framerusercontent.com/images/mbQ53sFYTA0yXgQnJzbYtrCz48.png?scale-down-to=1024 1024w,https://framerusercontent.com/images/mbQ53sFYTA0yXgQnJzbYtrCz48.png?scale-down-to=2048 2048w,https://framerusercontent.com/images/mbQ53sFYTA0yXgQnJzbYtrCz48.png 2120w\",style:{aspectRatio:\"2120 / 864\"},width:\"1060\"}),/*#__PURE__*/e(\"h4\",{children:\"What is DSPy?\"}),/*#__PURE__*/e(\"p\",{children:\"DSPy is the hot new framework on LLM community, which takes a different approach than what so far has been done for the past year: instead of having engineers manually trying to adjust prompt to improve results and do a *vibe-checking* on the outputs, DSPy gets you to focus on the dataset examples you bring, and the objective metrics that come out, not unlike traditional Machine Learning.\"}),/*#__PURE__*/t(\"p\",{children:[\"DSPy takes a big inspiration from PyTorch, which brings a much more rational and structured approach to optimize LLM pipelines, in a very coherent framework where all the pieces fit very well together, similar to how a \",/*#__PURE__*/e(a,{href:\"https://langwatch.ai/blog/data-flywheel\",motionChild:!0,nodeId:\"LYxDZg6nX\",openInNewTab:!0,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(o.a,{children:\"data flywheel\"})}),\" continuously improves LLM performance.\"]}),/*#__PURE__*/t(\"p\",{children:[\"In short, by giving your dataset to your DSPy program, and defining which metrics mean a good result by evaluating the outputs, DSPy Optimizers can automatically try to find the best combinations to improve your LLM program, be it by changing the prompts, the demonstrations for Few-Shot Prompting, or slight changes in the structure (signatures), similar to how \",/*#__PURE__*/e(a,{href:\"https://keras.io/api/optimizers/\",motionChild:!0,nodeId:\"LYxDZg6nX\",openInNewTab:!0,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(o.a,{children:\"optimizers in Keras\"})}),\" are used in traditional machine learning.\"]}),/*#__PURE__*/t(\"p\",{children:[\"We found this approach to be very effective. \",/*#__PURE__*/e(\"strong\",{children:\"In one case where we couldn\u2019t get accuracy higher than 60% after many hours of manual prompt engineering, DSPy took it to 80% for us in a matter of minutes.\"})]}),/*#__PURE__*/e(\"p\",{children:\"Prompt engineering at last automated, and the results are by definition measurable!\"}),/*#__PURE__*/e(\"h4\",{children:\"DSPy Visualizer\"}),/*#__PURE__*/e(\"p\",{children:\"At LangWatch we have been very excited on the results we are getting from DSPy, so today we are making public a tool we already have been using internally which help us a lot on our DSPy training sessions, the DSPy Visualizer:\"}),/*#__PURE__*/e(\"img\",{alt:\"\",className:\"framer-image\",height:\"486\",src:\"https://framerusercontent.com/images/ZEpdkrAitnzqMwUKqe73KFYc8k.png\",srcSet:\"https://framerusercontent.com/images/ZEpdkrAitnzqMwUKqe73KFYc8k.png?scale-down-to=512 512w,https://framerusercontent.com/images/ZEpdkrAitnzqMwUKqe73KFYc8k.png?scale-down-to=1024 1024w,https://framerusercontent.com/images/ZEpdkrAitnzqMwUKqe73KFYc8k.png 1573w\",style:{aspectRatio:\"1573 / 972\"},width:\"786\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"em\",{children:\"Screenshot of LangWatch DSPy Visualizer\"})}),/*#__PURE__*/e(\"p\",{children:\"DSPy Visualizer, allows you to log your DSPy training sessions, track the performance, costs, compare runs and debug them in detail.\"}),/*#__PURE__*/t(\"p\",{children:[\"This helps make it very easy to understand the step the optimizer is going through, when using it ourselves, multiple times we noticed some issues on the pipeline, like the signatures not being used correctly, or the wrong LLM being called when for example we wanted to use GPT-4 as a teacher for a GPT-3.5 student, which aligns with the \",/*#__PURE__*/e(a,{href:\"https://docs.langwatch.ai/concepts\",motionChild:!0,nodeId:\"LYxDZg6nX\",openInNewTab:!0,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(o.a,{children:\"LangWatch concepts\"})}),\" designed to optimize LLM workflows\"]}),/*#__PURE__*/e(\"p\",{children:\"This helps make it very easy to understand the step the optimizer is going through, when using it ourselves, multiple times we noticed some issues on the pipeline, like the signatures not being used correctly, or the wrong LLM being called when for example we wanted to use GPT-4 as a teacher for a GPT-3.5 student, or it gave us new ideas on what to try. This allowed us to stop the experiment earlier and iterate on our setup more quickly.\"}),/*#__PURE__*/t(\"p\",{children:[\"Since DSPy runs a lot of trials in a short amount of time, token rate limits and LLM calls cost also play a big factor, LangWatch\u2019s DSPy Visualizer tracks all LLM calls for you, making it easy to compare for example the cost of using GPT-3.5 Turbo vs Claude 3 Haiku, or noticing that you are reaching 400K tokens per minute and know what rate limit to request further, which was our case, similar to how our \",/*#__PURE__*/e(a,{href:\"https://docs.langwatch.ai/evaluations/overview\",motionChild:!0,nodeId:\"LYxDZg6nX\",openInNewTab:!0,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(o.a,{children:\"evaluation tools\"})}),\" help assess quality and safety.\"]}),/*#__PURE__*/t(\"h5\",{children:[\"If you are curious to try it out too, check out our \",/*#__PURE__*/e(a,{href:\"https://docs.langwatch.ai/dspy-visualization/quickstart\",motionChild:!0,nodeId:\"LYxDZg6nX\",openInNewTab:!0,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(o.a,{children:\"quickstart docs\"})}),\", or \",/*#__PURE__*/e(a,{href:\"https://get.langwatch.ai/request-a-demo\",motionChild:!0,nodeId:\"LYxDZg6nX\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(o.a,{children:\"book a demo\"})}),\" with us.\"]})]});export const richText11=/*#__PURE__*/t(i.Fragment,{children:[/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"AMSTERDAM, NL - 24 APR:\"}),\" A new Amsterdam-based AI quality and analytics platform, LangWatch, has launched today to improve the quality of Generative AI tools.\\xa0\"]}),/*#__PURE__*/e(\"p\",{children:\"With companies around the world investing in new tools with the use of AI, businesses are at risk of the misuse and abuse of AI.\\xa0\"}),/*#__PURE__*/t(\"p\",{children:[\"After high-profile cases of \",/*#__PURE__*/e(a,{href:\"https://www.bbc.com/news/technology-68025677\",motionChild:!0,nodeId:\"LYxDZg6nX\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(o.a,{children:\"swearing chatbots\"})}),\" and users manipulating AI to\",/*#__PURE__*/e(a,{href:\"https://the-decoder.com/people-buy-brand-new-chevrolets-for-1-from-a-chatgpt-chatbot/#:~:text=A%20car%20for%20a%20dollar,that%20could%20not%20be%20withdrawn.\",motionChild:!0,nodeId:\"LYxDZg6nX\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(o.a,{children:\" purchase a car for $1\"})}),\", it is clear that there is an urgent need for more control to avoid the misuse of new AI tools and brand reputational damage.\\xa0\"]}),/*#__PURE__*/e(\"h4\",{children:/*#__PURE__*/e(\"strong\",{children:\"LangWatch.ai - Safety first\"})}),/*#__PURE__*/e(\"p\",{children:\"LangWatch, which is backed by Antler and Rabobank Innovation, analyses AI solutions, evaluates the quality of the software, prevents AI risks and helps businesses improve and ship their solutions.\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"LangWatch has been developed because traditional testing tools aren\u2019t equipped to navigate new complex LLMs and AI software. With real-time insights, LangWatch Analytics helps companies building GenAI solutions track user feedback, conversion, output quality, and knowledge base gaps.\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"The result is that LangWatch can prevent off-topic conversations, sensitive data leakage as well as brand reputational damage for businesses.\\xa0\"}),/*#__PURE__*/e(\"h4\",{children:/*#__PURE__*/e(\"strong\",{children:\"Highly qualified founding team\"})}),/*#__PURE__*/e(\"p\",{children:\"The founders of LangWatch, Rogerio Chaves and Manouk Draisma, met during an Antler residency in Amsterdam. They bring together more than 25 years of experience in the software industry, serving at companies like Booking.com & Lightspeed.\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"Rogerio brings extensive expertise in engineering and product development to the LangWatch team while complementary Manouk brings a ton of experience in commercial roles, building teams at both startups and public companies.\"}),/*#__PURE__*/e(\"p\",{children:'Manouk Driasma, co-founder and CEO, comments, \"A multitude of businesses find themselves in a preliminary phase, testing the waters with a limited user base. However, the leap to full-scale production is met with hesitation. The anticipation of navigating hallucinations, safety concerns, and potential PR damage when exposing their AI solutions to a broader audience holds them back from unleashing the true potential of generative AI. LangWatch solves those problems and will allow companies to use AI to improve their services.\"'}),/*#__PURE__*/e(\"p\",{children:\"Rogerio Chaves, co-founder and CTO, comments, \u201CI saw this problem first hand when we launched an AI travel agent at Booking.com. It was very hard to understand how people were using the product, analyzing it was a very manual task and the GenAI agent had to be limited due to concerns of security and hallucinations, reducing its capabilities.\u201D\"}),/*#__PURE__*/e(\"p\",{children:\"RJ Schuurs, Partner at Antler, comments, \u201CAI will have a transformative impact on business, society and the economy. However, this is still very new technology and there are major risks for new AI-powered tools in terms of quality and the potential for misuse. LangWatch is bringing order to the chaos of the frontline of AI and will give businesses peace of mind when launching new AI products. We are delighted to have supported LangWatch from day zero and have every confidence in the future success of the founding team.\u201D\"}),/*#__PURE__*/e(\"p\",{children:\"LangWatch is now available for companies that are building with the next-generation of Generative AI.\"}),/*#__PURE__*/e(\"h5\",{children:/*#__PURE__*/e(a,{href:\"app.langwatch.ai/\",motionChild:!0,nodeId:\"LYxDZg6nX\",openInNewTab:!0,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(o.a,{children:\"Request Access Now\"})})})]});\nexport const __FramerMetadata__ = {\"exports\":{\"richText10\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"richText9\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"richText7\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"richText8\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"richText3\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"richText11\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"richText4\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"richText5\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"richText\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"richText1\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"richText6\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"richText2\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"__FramerMetadata__\":{\"type\":\"variable\"}}}"],
  "mappings": "kUAAgS,IAAMA,EAAsBC,EAAIC,EAAS,CAAC,SAAS,CAAcD,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,SAAS,wDAAwD,CAAC,EAAeA,EAAEC,EAAE,CAAC,KAAK,2BAA2B,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAsBF,EAAE,KAAK,CAAC,SAAS,aAAa,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4YAA4Y,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uSAAuS,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcA,EAAE,SAAS,CAAC,SAAS,CAAC,iCAA8CE,EAAE,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,EAAE,kZAA+ZA,EAAE,KAAK,CAAC,CAAC,EAAeA,EAAE,SAAS,CAAC,SAAsBA,EAAE,KAAK,CAAC,CAAC,CAAC,CAAC,EAAE,2YAA2Y,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oaAA+Z,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcA,EAAE,SAAS,CAAC,SAAS,CAAC,qCAAkDE,EAAE,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,EAAE,sFAAsF,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,kBAAkB,2BAA2B,MAAM,EAAE,SAAS,CAAcE,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,sBAAsB,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mLAAmL,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,kBAAkB,2BAA2B,MAAM,EAAE,SAAS,CAAcE,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,6BAA6B,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gTAAgT,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,kBAAkB,2BAA2B,MAAM,EAAE,SAAS,CAAcE,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,oBAAoB,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+QAA+Q,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcA,EAAE,SAAS,CAAC,SAAS,CAAC,6CAA0DE,EAAE,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,EAAE,0NAA0N,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,kBAAkB,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,mBAAmB,CAAC,EAAE,8PAA8P,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,kBAAkB,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,cAAc,CAAC,EAAE,sTAAsT,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+OAA+O,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,kDAAkD,CAAC,EAAeA,EAAE,KAAK,CAAC,CAAC,EAAE,oOAA0N,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2TAAsT,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yvBAA+uB,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcA,EAAE,SAAS,CAAC,SAAS,CAAC,aAA0BE,EAAE,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,EAAE,0UAAqU,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qEAAqE,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAEC,EAAE,CAAC,KAAK,kDAAkD,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,kBAAkB,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeC,EAAuBL,EAAIC,EAAS,CAAC,SAAS,CAAcC,EAAE,KAAK,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,4BAA4B,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wTAAoS,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,+CAAuDE,EAAE,SAAS,CAAC,SAAS,qBAAqB,CAAC,EAAE,qGAAgG,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,wCAAwC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uQAA6P,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,kBAAkB,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,uBAAuB,CAAC,EAAE,wFAAwF,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,yBAAyB,CAAC,EAAE,uGAAuG,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,kBAAkB,CAAC,EAAE,sFAAsF,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sGAAsG,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,UAAU,gBAAgB,CAAC,CAAC,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,GAAG,UAAU,eAAe,OAAO,OAAO,IAAI,sEAAsE,OAAO,iWAAiW,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,MAAM,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,iDAAiD,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kNAAkN,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,mCAAyB,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,eAAe,CAAC,EAAE,yEAAyE,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,oBAAoB,CAAC,EAAE,uEAAuE,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,4BAA4B,CAAC,EAAE,sEAAsE,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6FAA6F,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,WAAW,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yTAA0S,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,UAAU,gBAAgB,CAAC,CAAC,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,GAAG,UAAU,eAAe,OAAO,KAAK,IAAI,uEAAuE,OAAO,wKAAwK,MAAM,CAAC,YAAY,WAAW,EAAE,MAAM,KAAK,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,UAAU,gBAAgB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,wCAAwC,CAAC,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcA,EAAE,SAAS,CAAC,SAAS,CAAC,gBAA6BE,EAAE,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,EAAE,4UAAuU,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sIAAsI,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,6BAA6B,CAAC,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,4EAAyFE,EAAE,SAAS,CAAC,SAAS,+BAA+B,CAAC,EAAE,wGAAwG,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iCAAiC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAEC,EAAE,CAAC,KAAK,4BAA4B,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,qBAAqB,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeE,EAAuBN,EAAIC,EAAS,CAAC,SAAS,CAAcC,EAAE,IAAI,CAAC,SAAS,khBAAmgB,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,0CAA0C,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,ykBAA2jB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qcAA2b,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,sEAAiE,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4QAAkQ,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,8CAAyC,CAAC,EAAE,gKAAgK,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,yBAAyB,CAAC,EAAE,4LAA4L,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,4BAA4B,CAAC,EAAE,0IAA0I,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,0CAA0C,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wNAAyM,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,4CAA4C,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oXAAoX,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,sCAAsC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+ZAA+Z,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,2DAA2D,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uYAAuY,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,+CAA+C,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6RAAmR,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,uBAAuB,CAAC,EAAE,gEAAgE,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,yCAAyC,CAAC,EAAE,wGAAmG,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,6CAA6C,CAAC,EAAE,mFAAmF,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,8DAAyD,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sVAAkU,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sOAA4N,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAEC,EAAE,CAAC,KAAK,4BAA4B,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAsBF,EAAE,SAAS,CAAC,SAAS,cAAc,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeK,EAAuBP,EAAIC,EAAS,CAAC,SAAS,CAAcC,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,cAAc,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6mBAA8lB,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,8BAA8B,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6mBAA8lB,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,2CAA2C,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kJAA6I,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,aAAa,CAAC,EAAE,qJAAqJ,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,QAAQ,CAAC,EAAE,gMAAgM,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,SAAS,CAAC,EAAE,2KAA2K,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iJAAiJ,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,kDAA6C,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8WAAoW,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0GAAgG,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,uBAAuB,CAAC,EAAE,qNAAqN,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,0BAA0B,CAAC,EAAE,6PAAmP,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,4BAA4B,CAAC,EAAE,oWAA+V,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,0DAA0D,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2ZAAsZ,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gUAAsT,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,qDAAqD,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wJAAmJ,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,sBAAsB,CAAC,EAAE,+NAA+N,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,uBAAuB,CAAC,EAAE,wRAAmR,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,wBAAwB,CAAC,EAAE,uOAAuO,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,wBAAwB,CAAC,EAAE,iNAAiN,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,0DAA0D,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,udAAkd,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4LAA4L,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,4CAA4C,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mbAAoa,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,qCAAkDE,EAAEC,EAAE,CAAC,KAAK,4BAA4B,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,eAAe,CAAC,CAAC,CAAC,EAAE,uOAAkO,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeI,EAAuBR,EAAIC,EAAS,CAAC,SAAS,CAAcC,EAAE,IAAI,CAAC,SAAS,meAAme,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0CAA0C,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,uCAAuC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAC,mBAAgCE,EAAEC,EAAE,CAAC,KAAK,0BAA0B,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,cAAc,CAAC,CAAC,CAAC,EAAE,GAAG,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAC,wBAAqCE,EAAEC,EAAE,CAAC,KAAK,uDAAuD,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,kBAAkB,CAAC,CAAC,CAAC,EAAE,GAAG,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,wCAAwC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wRAAiR,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,8DAA8D,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mJAAmJ,CAAC,EAAeA,EAAE,MAAM,CAAC,UAAU,qBAAqB,MAAM,CAAC,OAAO,OAAO,MAAM,MAAM,EAAE,SAAsBA,EAAEO,EAAE,CAAC,oBAAoB,wEAAwE,SAASC,GAAgBR,EAAES,EAAE,CAAC,GAAGD,EAAE,KAAK,gGAAgG,SAAS,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeV,EAAE,IAAI,CAAC,SAAS,CAAC,kBAA+BE,EAAE,OAAO,CAAC,SAAS,MAAM,CAAC,EAAE,wCAAwC,CAAC,CAAC,EAAeA,EAAE,MAAM,CAAC,UAAU,qBAAqB,MAAM,CAAC,OAAO,OAAO,MAAM,MAAM,EAAE,SAAsBA,EAAEO,EAAE,CAAC,oBAAoB,wEAAwE,SAASC,GAAgBR,EAAES,EAAE,CAAC,GAAGD,EAAE,KAAK,iEAAiE,SAAS,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeR,EAAE,IAAI,CAAC,SAAS,qCAAqC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,wDAAwD,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8RAA8R,CAAC,EAAeA,EAAE,MAAM,CAAC,UAAU,qBAAqB,MAAM,CAAC,OAAO,OAAO,MAAM,MAAM,EAAE,SAAsBA,EAAEO,EAAE,CAAC,oBAAoB,wEAAwE,SAASC,GAAgBR,EAAES,EAAE,CAAC,GAAGD,EAAE,KAAK;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA,mBAAg+D,SAAS,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeR,EAAE,IAAI,CAAC,SAAS,gXAAgX,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,qDAAkEE,EAAE,OAAO,CAAC,SAAS,SAAS,CAAC,EAAE,oLAAoL,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,mDAAmD,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,iKAA8KE,EAAE,OAAO,CAAC,SAAS,MAAM,CAAC,EAAE,mFAAmF,CAAC,CAAC,EAAeA,EAAE,MAAM,CAAC,UAAU,qBAAqB,MAAM,CAAC,OAAO,OAAO,MAAM,MAAM,EAAE,SAAsBA,EAAEO,EAAE,CAAC,oBAAoB,wEAAwE,SAASC,GAAgBR,EAAES,EAAE,CAAC,GAAGD,EAAE,KAAK;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA,kCAAqZ,SAAS,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeR,EAAE,KAAK,CAAC,SAAsBA,EAAE,KAAK,CAAC,UAAU,gBAAgB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,mDAAmD,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+QAA+Q,CAAC,EAAeA,EAAE,MAAM,CAAC,UAAU,qBAAqB,MAAM,CAAC,OAAO,OAAO,MAAM,MAAM,EAAE,SAAsBA,EAAEO,EAAE,CAAC,oBAAoB,wEAAwE,SAASC,GAAgBR,EAAES,EAAE,CAAC,GAAGD,EAAE,KAAK;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA,oEAAsM,SAAS,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeR,EAAE,IAAI,CAAC,SAAS,uJAAuJ,CAAC,EAAeA,EAAE,MAAM,CAAC,UAAU,qBAAqB,MAAM,CAAC,OAAO,OAAO,MAAM,MAAM,EAAE,SAAsBA,EAAEO,EAAE,CAAC,oBAAoB,wEAAwE,SAASC,GAAgBR,EAAES,EAAE,CAAC,GAAGD,EAAE,KAAK,+BAA+B,SAAS,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeV,EAAE,IAAI,CAAC,SAAS,CAAC,+EAA4FE,EAAEC,EAAE,CAAC,KAAK,6BAA6B,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,4BAA4B,CAAC,CAAC,CAAC,EAAE,GAAG,CAAC,CAAC,EAAeF,EAAE,MAAM,CAAC,UAAU,qBAAqB,MAAM,CAAC,OAAO,OAAO,MAAM,MAAM,EAAE,SAAsBA,EAAEO,EAAE,CAAC,oBAAoB,wEAAwE,SAASC,GAAgBR,EAAES,EAAE,CAAC,GAAGD,EAAE,KAAK,qEAAqE,SAAS,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeR,EAAE,IAAI,CAAC,SAAS,gBAAgB,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,oDAAoD,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gIAAgI,CAAC,EAAeA,EAAE,MAAM,CAAC,UAAU,qBAAqB,MAAM,CAAC,OAAO,OAAO,MAAM,MAAM,EAAE,SAAsBA,EAAEO,EAAE,CAAC,oBAAoB,wEAAwE,SAASC,GAAgBR,EAAES,EAAE,CAAC,GAAGD,EAAE,KAAK,sBAAsB,SAAS,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeR,EAAE,IAAI,CAAC,SAAS,sJAAsJ,CAAC,EAAeA,EAAE,MAAM,CAAC,UAAU,qBAAqB,MAAM,CAAC,OAAO,OAAO,MAAM,MAAM,EAAE,SAAsBA,EAAEO,EAAE,CAAC,oBAAoB,wEAAwE,SAASC,GAAgBR,EAAES,EAAE,CAAC,GAAGD,EAAE,KAAK,gBAAgB,SAAS,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeR,EAAE,IAAI,CAAC,SAAS,sUAA4T,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qEAAqE,CAAC,EAAeA,EAAE,MAAM,CAAC,UAAU,qBAAqB,MAAM,CAAC,OAAO,OAAO,MAAM,MAAM,EAAE,SAAsBA,EAAEO,EAAE,CAAC,oBAAoB,wEAAwE,SAASC,GAAgBR,EAAES,EAAE,CAAC,GAAGD,EAAE,KAAK,8BAA8B,SAAS,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeR,EAAE,IAAI,CAAC,SAAS,uNAAuN,CAAC,EAAeA,EAAE,MAAM,CAAC,UAAU,qBAAqB,MAAM,CAAC,OAAO,OAAO,MAAM,MAAM,EAAE,SAAsBA,EAAEO,EAAE,CAAC,oBAAoB,wEAAwE,SAASC,GAAgBR,EAAES,EAAE,CAAC,GAAGD,EAAE,KAAK;AAAA;AAAA;AAAA,2BAAgG,SAAS,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeR,EAAE,IAAI,CAAC,SAAS,gJAAgJ,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sGAAsG,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,mEAAmE,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yDAAyD,CAAC,EAAeA,EAAE,MAAM,CAAC,UAAU,qBAAqB,MAAM,CAAC,OAAO,OAAO,MAAM,MAAM,EAAE,SAAsBA,EAAEO,EAAE,CAAC,oBAAoB,wEAAwE,SAASC,GAAgBR,EAAES,EAAE,CAAC,GAAGD,EAAE,KAAK,oCAAoC,SAAS,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeR,EAAE,IAAI,CAAC,SAAS,4NAA4N,CAAC,EAAeA,EAAE,MAAM,CAAC,UAAU,qBAAqB,MAAM,CAAC,OAAO,OAAO,MAAM,MAAM,EAAE,SAAsBA,EAAEO,EAAE,CAAC,oBAAoB,wEAAwE,SAASC,GAAgBR,EAAES,EAAE,CAAC,GAAGD,EAAE,KAAK,2BAA2B,SAAS,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeR,EAAE,IAAI,CAAC,SAAS,yRAAyR,CAAC,EAAeA,EAAE,MAAM,CAAC,UAAU,qBAAqB,MAAM,CAAC,OAAO,OAAO,MAAM,MAAM,EAAE,SAAsBA,EAAEO,EAAE,CAAC,oBAAoB,wEAAwE,SAASC,GAAgBR,EAAES,EAAE,CAAC,GAAGD,EAAE,KAAK,mBAAmB,SAAS,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeR,EAAE,IAAI,CAAC,SAAS,mGAAmG,CAAC,EAAeA,EAAE,MAAM,CAAC,UAAU,qBAAqB,MAAM,CAAC,OAAO,OAAO,MAAM,MAAM,EAAE,SAAsBA,EAAEO,EAAE,CAAC,oBAAoB,wEAAwE,SAASC,GAAgBR,EAAES,EAAE,CAAC,GAAGD,EAAE,KAAK,+CAA+C,SAAS,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeR,EAAE,IAAI,CAAC,SAAS,6CAA6C,CAAC,EAAeA,EAAE,MAAM,CAAC,UAAU,qBAAqB,MAAM,CAAC,OAAO,OAAO,MAAM,MAAM,EAAE,SAAsBA,EAAEO,EAAE,CAAC,oBAAoB,wEAAwE,SAASC,GAAgBR,EAAES,EAAE,CAAC,GAAGD,EAAE,KAAK,YAAY,SAAS,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeR,EAAE,IAAI,CAAC,SAAS,+FAA+F,CAAC,EAAeA,EAAE,MAAM,CAAC,UAAU,qBAAqB,MAAM,CAAC,OAAO,OAAO,MAAM,MAAM,EAAE,SAAsBA,EAAEO,EAAE,CAAC,oBAAoB,wEAAwE,SAASC,GAAgBR,EAAES,EAAE,CAAC,GAAGD,EAAE,KAAK,YAAY,SAAS,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeR,EAAE,IAAI,CAAC,SAAS,wCAAwC,CAAC,EAAeA,EAAE,MAAM,CAAC,UAAU,qBAAqB,MAAM,CAAC,OAAO,OAAO,MAAM,MAAM,EAAE,SAAsBA,EAAEO,EAAE,CAAC,oBAAoB,wEAAwE,SAASC,GAAgBR,EAAES,EAAE,CAAC,GAAGD,EAAE,KAAK;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA,iBAA2Y,SAAS,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeR,EAAE,IAAI,CAAC,SAAS,oDAAoD,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,0BAA0B,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,kDAAkD,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,gCAA6CE,EAAEC,EAAE,CAAC,KAAK,iEAAiE,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,yDAAyD,CAAC,CAAC,CAAC,EAAE,yCAAoC,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,oIAAoI,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,iJAA8JE,EAAEC,EAAE,CAAC,KAAK,0CAA0C,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,WAAW,CAAC,CAAC,CAAC,EAAE,IAAI,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,0FAA0F,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,eAAe,CAAC,CAAC,CAAC,CAAC,EAAeU,EAAuBZ,EAAIC,EAAS,CAAC,SAAS,CAAcC,EAAE,MAAM,CAAC,IAAI,GAAG,UAAU,eAAe,OAAO,MAAM,IAAI,oEAAoE,OAAO,8PAA8P,MAAM,CAAC,YAAY,YAAY,EAAE,MAAM,KAAK,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2OAA2O,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,6EAA0FE,EAAE,SAAS,CAAC,SAAS,OAAO,CAAC,EAAE,kCAA+CA,EAAE,SAAS,CAAC,SAAS,MAAM,CAAC,EAAE,yBAAsCA,EAAE,SAAS,CAAC,SAAS,KAAK,CAAC,EAAE,yEAAyE,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,uBAAoCE,EAAE,SAAS,CAAC,SAAS,OAAO,CAAC,EAAE,gfAA2e,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6GAA6G,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mSAAmS,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mPAAmP,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oDAAoD,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,8NAA8N,UAAU,eAAe,OAAO,MAAM,IAAI,qEAAqE,OAAO,iQAAiQ,MAAM,CAAC,YAAY,YAAY,EAAE,MAAM,KAAK,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sPAA4O,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,mCAAmC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,YAAyBE,EAAEC,EAAE,CAAC,KAAK,sDAAsD,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,iBAAiB,CAAC,CAAC,CAAC,EAAE,iLAAiL,CAAC,CAAC,EAAeF,EAAE,MAAM,CAAC,UAAU,qBAAqB,MAAM,CAAC,OAAO,OAAO,MAAM,MAAM,EAAE,SAAsBA,EAAEO,EAAE,CAAC,oBAAoB,wEAAwE,SAASC,GAAgBR,EAAES,EAAE,CAAC,GAAGD,EAAE,KAAK;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA,+BAA0S,SAAS,QAAQ,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeV,EAAE,IAAI,CAAC,SAAS,CAAC,UAAuBE,EAAEC,EAAE,CAAC,KAAK,2BAA2B,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,QAAQ,CAAC,CAAC,CAAC,EAAE,sCAAmDF,EAAEC,EAAE,CAAC,KAAK,2CAA2C,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,gBAAgB,CAAC,CAAC,CAAC,EAAE,6DAA6D,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,2PAA2P,CAAC,EAAeA,EAAE,MAAM,CAAC,UAAU,qBAAqB,MAAM,CAAC,OAAO,OAAO,MAAM,MAAM,EAAE,SAAsBA,EAAEO,EAAE,CAAC,oBAAoB,wEAAwE,SAASC,GAAgBR,EAAES,EAAE,CAAC,GAAGD,EAAE,KAAK;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA,gEAAykB,SAAS,QAAQ,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeV,EAAE,IAAI,CAAC,SAAS,CAAC,UAAuBE,EAAEC,EAAE,CAAC,KAAK,qCAAqC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,SAAS,CAAC,CAAC,CAAC,EAAE,yKAAyK,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,wDAAmD,CAAC,EAAeA,EAAE,MAAM,CAAC,UAAU,qBAAqB,MAAM,CAAC,OAAO,OAAO,MAAM,MAAM,EAAE,SAAsBA,EAAEO,EAAE,CAAC,oBAAoB,wEAAwE,SAASC,GAAgBR,EAAES,EAAE,CAAC,GAAGD,EAAE,KAAK;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA,iLAAw0B,SAAS,QAAQ,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeR,EAAE,IAAI,CAAC,SAAS,4QAA4Q,CAAC,EAAeA,EAAE,MAAM,CAAC,UAAU,qBAAqB,MAAM,CAAC,OAAO,OAAO,MAAM,MAAM,EAAE,SAAsBA,EAAEO,EAAE,CAAC,oBAAoB,wEAAwE,SAASC,GAAgBR,EAAES,EAAE,CAAC,GAAGD,EAAE,KAAK;AAAA,qIAAyO,SAAS,QAAQ,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeR,EAAE,IAAI,CAAC,SAAS,eAAe,CAAC,EAAeA,EAAE,MAAM,CAAC,UAAU,qBAAqB,MAAM,CAAC,OAAO,OAAO,MAAM,MAAM,EAAE,SAAsBA,EAAEO,EAAE,CAAC,oBAAoB,wEAAwE,SAASC,GAAgBR,EAAES,EAAE,CAAC,GAAGD,EAAE,KAAK;AAAA;AAAA;AAAA;AAAA,qKAAwV,SAAS,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeR,EAAE,IAAI,CAAC,SAAS,kHAAkH,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,kCAAkC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8SAAyS,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,qCAA6CE,EAAE,OAAO,CAAC,SAAS,aAAa,CAAC,EAAE,YAAY,CAAC,CAAC,EAAeA,EAAE,MAAM,CAAC,UAAU,qBAAqB,MAAM,CAAC,OAAO,OAAO,MAAM,MAAM,EAAE,SAAsBA,EAAEO,EAAE,CAAC,oBAAoB,wEAAwE,SAASC,GAAgBR,EAAES,EAAE,CAAC,GAAGD,EAAE,KAAK;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA,+BAA4oB,SAAS,QAAQ,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeR,EAAE,IAAI,CAAC,SAAS,0LAAqL,CAAC,EAAeA,EAAE,MAAM,CAAC,UAAU,qBAAqB,MAAM,CAAC,OAAO,OAAO,MAAM,MAAM,EAAE,SAAsBA,EAAEO,EAAE,CAAC,oBAAoB,wEAAwE,SAASC,GAAgBR,EAAES,EAAE,CAAC,GAAGD,EAAE,KAAK;AAAA;AAAA;AAAA;AAAA,qKAAwV,SAAS,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeV,EAAE,IAAI,CAAC,SAAS,CAAC,mEAAgFE,EAAE,OAAO,CAAC,SAAS,UAAU,CAAC,EAAE,cAA2BA,EAAE,OAAO,CAAC,SAAS,aAAa,CAAC,EAAE,0DAA0D,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+OAA0O,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qEAAqE,CAAC,EAAeA,EAAE,MAAM,CAAC,UAAU,qBAAqB,MAAM,CAAC,OAAO,OAAO,MAAM,MAAM,EAAE,SAAsBA,EAAEO,EAAE,CAAC,oBAAoB,wEAAwE,SAASC,GAAgBR,EAAES,EAAE,CAAC,GAAGD,EAAE,KAAK;AAAA;AAAA,qKAAyb,SAAS,MAAM,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeR,EAAE,IAAI,CAAC,SAAS,+JAA+J,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,uBAAuB,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,oOAAiPE,EAAEC,EAAE,CAAC,KAAK,kCAAkC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,EAAE,mFAAmF,CAAC,CAAC,EAAeF,EAAE,MAAM,CAAC,UAAU,qBAAqB,MAAM,CAAC,OAAO,OAAO,MAAM,MAAM,EAAE,SAAsBA,EAAEO,EAAE,CAAC,oBAAoB,wEAAwE,SAASC,GAAgBR,EAAES,EAAE,CAAC,GAAGD,EAAE,KAAK;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA,iCAA0T,SAAS,QAAQ,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeR,EAAE,IAAI,CAAC,SAAS,iKAA4J,CAAC,EAAeA,EAAE,MAAM,CAAC,UAAU,qBAAqB,MAAM,CAAC,OAAO,OAAO,MAAM,MAAM,EAAE,SAAsBA,EAAEO,EAAE,CAAC,oBAAoB,wEAAwE,SAASC,GAAgBR,EAAES,EAAE,CAAC,GAAGD,EAAE,KAAK;AAAA;AAAA;AAAA,qKAAuV,SAAS,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeR,EAAE,IAAI,CAAC,SAAS,wHAAwH,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4YAAkY,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,4RAAoSE,EAAE,OAAO,CAAC,SAAS,YAAY,CAAC,EAAE,YAAY,CAAC,CAAC,EAAeA,EAAE,MAAM,CAAC,UAAU,qBAAqB,MAAM,CAAC,OAAO,OAAO,MAAM,MAAM,EAAE,SAAsBA,EAAEO,EAAE,CAAC,oBAAoB,wEAAwE,SAASC,GAAgBR,EAAES,EAAE,CAAC,GAAGD,EAAE,KAAK;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA,iCAA0W,SAAS,QAAQ,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeR,EAAE,IAAI,CAAC,SAAS,4GAA4G,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,ucAA6b,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,kEAA+EE,EAAE,OAAO,CAAC,SAAS,QAAQ,CAAC,EAAE,wBAAqCA,EAAE,OAAO,CAAC,SAAS,eAAe,CAAC,EAAE,4EAAyFA,EAAE,OAAO,CAAC,SAAS,QAAQ,CAAC,EAAE,iDAAiD,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,gCAAgC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,qKAAkLE,EAAE,OAAO,CAAC,SAAS,aAAa,CAAC,EAAE,6GAA6G,CAAC,CAAC,EAAeA,EAAE,MAAM,CAAC,UAAU,qBAAqB,MAAM,CAAC,OAAO,OAAO,MAAM,MAAM,EAAE,SAAsBA,EAAEO,EAAE,CAAC,oBAAoB,wEAAwE,SAASC,GAAgBR,EAAES,EAAE,CAAC,GAAGD,EAAE,KAAK;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA,+BAA4a,SAAS,QAAQ,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeR,EAAE,IAAI,CAAC,SAAS,2GAAsG,CAAC,EAAeA,EAAE,MAAM,CAAC,UAAU,qBAAqB,MAAM,CAAC,OAAO,OAAO,MAAM,MAAM,EAAE,SAAsBA,EAAEO,EAAE,CAAC,oBAAoB,wEAAwE,SAASC,GAAgBR,EAAES,EAAE,CAAC,GAAGD,EAAE,KAAK;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA,qKAAgzB,SAAS,MAAM,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeV,EAAE,IAAI,CAAC,SAAS,CAAC,kLAA+LE,EAAE,SAAS,CAAC,SAAS,YAAY,CAAC,EAAE,qJAAgJ,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,8BAA8B,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,eAA4BE,EAAEC,EAAE,CAAC,KAAK,4DAA4D,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,oBAAoB,CAAC,CAAC,CAAC,EAAE,uQAAkQ,CAAC,CAAC,EAAeJ,EAAE,IAAI,CAAC,SAAS,CAAC,uBAAoCE,EAAEC,EAAE,CAAC,KAAK,wDAAwD,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,4BAA4B,CAAC,CAAC,CAAC,EAAE,SAAsBF,EAAEC,EAAE,CAAC,KAAK,gEAAgE,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,WAAW,CAAC,CAAC,CAAC,EAAE,sGAAsG,CAAC,CAAC,EAAeF,EAAE,MAAM,CAAC,UAAU,qBAAqB,MAAM,CAAC,OAAO,OAAO,MAAM,MAAM,EAAE,SAAsBA,EAAEO,EAAE,CAAC,oBAAoB,wEAAwE,SAASC,GAAgBR,EAAES,EAAE,CAAC,GAAGD,EAAE,KAAK;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA,iCAAmV,SAAS,QAAQ,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeR,EAAE,IAAI,CAAC,SAAS,2BAAsB,CAAC,EAAeA,EAAE,MAAM,CAAC,UAAU,qBAAqB,MAAM,CAAC,OAAO,OAAO,MAAM,MAAM,EAAE,SAAsBA,EAAEO,EAAE,CAAC,oBAAoB,wEAAwE,SAASC,GAAgBR,EAAES,EAAE,CAAC,GAAGD,EAAE,KAAK;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA,sKAAopB,SAAS,MAAM,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeR,EAAE,IAAI,CAAC,SAAS,4WAA4W,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,ubAA6a,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,0BAA0B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,odAAqc,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,oQAA4QE,EAAEC,EAAE,CAAC,KAAK,0CAA0C,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,WAAW,CAAC,CAAC,CAAC,EAAE,uFAAoGF,EAAE,OAAO,CAAC,SAAS,WAAW,CAAC,EAAE,qBAAqB,CAAC,CAAC,EAAeA,EAAE,MAAM,CAAC,UAAU,qBAAqB,MAAM,CAAC,OAAO,OAAO,MAAM,MAAM,EAAE,SAAsBA,EAAEO,EAAE,CAAC,oBAAoB,wEAAwE,SAASC,GAAgBR,EAAES,EAAE,CAAC,GAAGD,EAAE,KAAK;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA,iCAA2wB,SAAS,QAAQ,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeR,EAAE,IAAI,CAAC,SAAS,yVAAoV,CAAC,EAAeA,EAAE,MAAM,CAAC,UAAU,qBAAqB,MAAM,CAAC,OAAO,OAAO,MAAM,MAAM,EAAE,SAAsBA,EAAEO,EAAE,CAAC,oBAAoB,wEAAwE,SAASC,GAAgBR,EAAES,EAAE,CAAC,GAAGD,EAAE,KAAK;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA,qKAAuW,SAAS,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeV,EAAE,IAAI,CAAC,SAAS,CAAC,4FAAyGE,EAAE,OAAO,CAAC,SAAS,WAAW,CAAC,EAAE,mJAAyI,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wOAAmO,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,iCAAiC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,ybAAyb,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qXAAgX,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,yBAAyB,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,SAAsBE,EAAEC,EAAE,CAAC,KAAK,0CAA0C,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,WAAW,CAAC,CAAC,CAAC,EAAE,sIAAsI,CAAC,CAAC,EAAeJ,EAAE,IAAI,CAAC,SAAS,CAAC,yPAA4PE,EAAE,OAAO,CAAC,SAAS,2BAA2B,CAAC,EAAE,gIAA6IA,EAAE,OAAO,CAAC,SAAS,MAAM,CAAC,EAAE,OAAoBA,EAAE,OAAO,CAAC,SAAS,OAAO,CAAC,EAAE,qCAAqC,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,gGAAwGE,EAAE,OAAO,CAAC,SAAS,QAAQ,CAAC,EAAE,WAAW,CAAC,CAAC,EAAeA,EAAE,MAAM,CAAC,UAAU,qBAAqB,MAAM,CAAC,OAAO,OAAO,MAAM,MAAM,EAAE,SAAsBA,EAAEO,EAAE,CAAC,oBAAoB,wEAAwE,SAASC,GAAgBR,EAAES,EAAE,CAAC,GAAGD,EAAE,KAAK;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA,0EAA4oB,SAAS,QAAQ,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeR,EAAE,IAAI,CAAC,SAAS,iNAAkM,CAAC,EAAeA,EAAE,MAAM,CAAC,UAAU,qBAAqB,MAAM,CAAC,OAAO,OAAO,MAAM,MAAM,EAAE,SAAsBA,EAAEO,EAAE,CAAC,oBAAoB,wEAAwE,SAASC,GAAgBR,EAAES,EAAE,CAAC,GAAGD,EAAE,KAAK;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA,qKAAm8B,SAAS,MAAM,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeR,EAAE,IAAI,CAAC,SAAS,0WAA2V,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8iBAAqhB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wTAAyS,CAAC,EAAeA,EAAE,MAAM,CAAC,UAAU,qBAAqB,MAAM,CAAC,OAAO,OAAO,MAAM,MAAM,EAAE,SAAsBA,EAAEO,EAAE,CAAC,oBAAoB,wEAAwE,SAASC,GAAgBR,EAAES,EAAE,CAAC,GAAGD,EAAE,KAAK;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA,kEAA4wB,SAAS,QAAQ,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeR,EAAE,IAAI,CAAC,SAAS,oGAAqF,CAAC,EAAeA,EAAE,MAAM,CAAC,UAAU,qBAAqB,MAAM,CAAC,OAAO,OAAO,MAAM,MAAM,EAAE,SAAsBA,EAAEO,EAAE,CAAC,oBAAoB,wEAAwE,SAASC,GAAgBR,EAAES,EAAE,CAAC,GAAGD,EAAE,KAAK;AAAA;AAAA,qKAA4M,SAAS,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeR,EAAE,IAAI,CAAC,SAAS,kVAA6U,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0RAAqR,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,aAAa,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sJAAsJ,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0NAA0N,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,aAA0BE,EAAEC,EAAE,CAAC,KAAK,+DAA+D,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,yBAAyB,CAAC,CAAC,CAAC,EAAE,sFAAsF,CAAC,CAAC,EAAeJ,EAAE,IAAI,CAAC,SAAS,CAAC,yDAAsEE,EAAEC,EAAE,CAAC,KAAK,yCAAyC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,gBAAgB,CAAC,CAAC,CAAC,EAAE,6EAA0FF,EAAEC,EAAE,CAAC,KAAK,yCAAyC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,WAAW,CAAC,CAAC,CAAC,EAAE,0DAA0D,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,eAAe,CAAC,CAAC,CAAC,CAAC,EAAeW,EAAuBb,EAAIC,EAAS,CAAC,SAAS,CAAcC,EAAE,IAAI,CAAC,SAAS,sXAAsX,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kfAA6e,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+gBAA+gB,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,wCAAwC,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,gBAAgB,UAAU,eAAe,OAAO,MAAM,IAAI,uEAAuE,OAAO,wKAAwK,MAAM,CAAC,YAAY,WAAW,EAAE,MAAM,KAAK,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,meAAme,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+aAA0a,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6PAA6P,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gaAAga,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,WAAW,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gXAAgX,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,mDAAgEE,EAAEC,EAAE,CAAC,KAAK,0CAA0C,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,qBAAqB,CAAC,CAAC,CAAC,EAAE,GAAG,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeU,EAAuBd,EAAIC,EAAS,CAAC,SAAS,CAAcD,EAAE,IAAI,CAAC,SAAS,CAAC,sDAAmEE,EAAEC,EAAE,CAAC,KAAK,aAAa,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,QAAQ,CAAC,CAAC,CAAC,EAAE,0OAA0O,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,uBAAuB,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,SAAS,CAAC,EAAE,4BAA4B,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,QAAQ,CAAC,EAAE,6DAA6D,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yPAAoP,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,yDAAyD,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,upBAA8nB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wdAAyc,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,iDAAiD,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wgBAA8f,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wZAA8Y,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sYAA4X,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,slBAAukB,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,0BAA0B,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6VAA8U,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uQAAuQ,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,YAAY,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uGAAuG,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0bAAgb,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,kFAAkF,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAEC,EAAE,CAAC,KAAK,0CAA0C,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,gBAAgB,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeW,EAAuBf,EAAIC,EAAS,CAAC,SAAS,CAAcC,EAAE,IAAI,CAAC,SAAS,ocAAoc,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,GAAG,UAAU,eAAe,OAAO,MAAM,IAAI,sEAAsE,OAAO,sKAAsK,MAAM,CAAC,YAAY,WAAW,EAAE,MAAM,KAAK,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAEC,EAAE,CAAC,KAAK,iFAAiF,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,cAAc,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,uMAAuM,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+LAA+L,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,2CAA2C,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gNAAgN,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0QAA0Q,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+KAA+K,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,6BAA6B,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0MAA0M,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,kBAAkB,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uOAAuO,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,2BAA2B,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0XAA0X,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iKAAiK,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,sBAAsB,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6gBAA6gB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2IAA2I,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,cAAc,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sPAAsP,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8IAA8I,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,ySAAyS,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,2CAA2C,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kQAAkQ,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,0BAA0B,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4FAA4F,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,SAAS,CAAC,EAAE,iKAA8KA,EAAE,KAAK,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,UAAU,gBAAgB,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,uBAAuB,CAAC,EAAE,yLAAsMA,EAAE,KAAK,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,UAAU,gBAAgB,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,4BAA4B,CAAC,EAAE,6JAA6J,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,6BAA6B,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mJAAmJ,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,WAAW,CAAC,EAAE,uPAAoQA,EAAE,KAAK,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,UAAU,gBAAgB,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,4BAA4B,CAAC,EAAE,2LAAwMA,EAAE,KAAK,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,UAAU,gBAAgB,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,uBAAuB,CAAC,EAAE,2HAA2H,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,sCAAsC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gHAAgH,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,oBAAoB,CAAC,EAAE,sHAAmIA,EAAE,KAAK,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,UAAU,gBAAgB,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,sBAAsB,CAAC,EAAE,+GAA4HA,EAAE,KAAK,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,UAAU,gBAAgB,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,uBAAuB,CAAC,EAAE,sNAAsN,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,yBAAyB,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2KAA2K,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,kBAAkB,CAAC,EAAE,iJAA8JA,EAAE,KAAK,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,UAAU,gBAAgB,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,qBAAqB,CAAC,EAAE,iKAA8KA,EAAE,KAAK,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,UAAU,gBAAgB,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,oBAAoB,CAAC,EAAE,6HAA6H,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4OAA4O,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,sFAAsF,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6NAA6N,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAC,8FAA2GE,EAAE,KAAK,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,UAAU,gBAAgB,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAC,kIAA+IE,EAAE,KAAK,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,UAAU,gBAAgB,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBA,EAAE,IAAI,CAAC,SAAS,kHAAkH,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,sDAAsD,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAEC,EAAE,CAAC,KAAK,0CAA0C,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,gBAAgB,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeY,EAAuBhB,EAAIC,EAAS,CAAC,SAAS,CAAcC,EAAE,KAAK,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,8CAA8C,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kLAAkL,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mUAAmU,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qOAA2N,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,iCAAiC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oVAA+U,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yPAAyP,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAS,CAAcA,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,0BAA0B,CAAC,EAAE,iMAAiM,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,UAAU,gBAAgB,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,gBAAgB,CAAC,EAAE,+LAA+L,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,yDAAyD,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4IAA4I,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,qBAAqB,CAAC,EAAE,gSAA6SA,EAAE,KAAK,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,UAAU,gBAAgB,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,qBAAqB,CAAC,EAAE,6RAA8R,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uRAAwR,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,mCAAmC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gIAAgI,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAS,CAAcA,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,iBAAiB,CAAC,EAAE,oSAAoS,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,UAAU,gBAAgB,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAS,CAAcA,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,qBAAqB,CAAC,EAAE,qTAAqT,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,UAAU,gBAAgB,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAS,CAAcA,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,kBAAkB,CAAC,EAAE,6YAA6Y,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,UAAU,gBAAgB,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAS,CAAcA,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,qBAAqB,CAAC,EAAE,6LAA6L,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,UAAU,gBAAgB,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAS,CAAcA,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,4BAA4B,CAAC,EAAE,kNAAkN,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,UAAU,gBAAgB,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,sBAAsB,CAAC,EAAE,0PAA0P,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wLAA8K,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,mBAAmB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wIAAwI,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,kBAAkB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wQAAwQ,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,wBAAwB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4VAA4V,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qRAAqR,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,mEAAmE,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mKAAmK,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sPAAsP,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,iDAAiD,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAEC,EAAE,CAAC,KAAK,0CAA0C,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,gBAAgB,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,UAAU,gBAAgB,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAee,EAAwBjB,EAAIC,EAAS,CAAC,SAAS,CAAcC,EAAE,IAAI,CAAC,SAAS,+RAA+R,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kMAAkM,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,GAAG,UAAU,eAAe,OAAO,MAAM,IAAI,sEAAsE,OAAO,iWAAiW,MAAM,CAAC,YAAY,YAAY,EAAE,MAAM,MAAM,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,eAAe,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yYAAyY,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,8NAA2OE,EAAEC,EAAE,CAAC,KAAK,0CAA0C,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,eAAe,CAAC,CAAC,CAAC,EAAE,yCAAyC,CAAC,CAAC,EAAeJ,EAAE,IAAI,CAAC,SAAS,CAAC,8WAA2XE,EAAEC,EAAE,CAAC,KAAK,mCAAmC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,qBAAqB,CAAC,CAAC,CAAC,EAAE,4CAA4C,CAAC,CAAC,EAAeJ,EAAE,IAAI,CAAC,SAAS,CAAC,gDAA6DE,EAAE,SAAS,CAAC,SAAS,mKAA8J,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qFAAqF,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,iBAAiB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oOAAoO,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,GAAG,UAAU,eAAe,OAAO,MAAM,IAAI,sEAAsE,OAAO,oQAAoQ,MAAM,CAAC,YAAY,YAAY,EAAE,MAAM,KAAK,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,SAAS,yCAAyC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sIAAsI,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,qVAAkWE,EAAEC,EAAE,CAAC,KAAK,qCAAqC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,oBAAoB,CAAC,CAAC,CAAC,EAAE,qCAAqC,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,2bAA2b,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,gaAAwaE,EAAEC,EAAE,CAAC,KAAK,iDAAiD,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,kBAAkB,CAAC,CAAC,CAAC,EAAE,kCAAkC,CAAC,CAAC,EAAeJ,EAAE,KAAK,CAAC,SAAS,CAAC,uDAAoEE,EAAEC,EAAE,CAAC,KAAK,0DAA0D,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,iBAAiB,CAAC,CAAC,CAAC,EAAE,QAAqBF,EAAEC,EAAE,CAAC,KAAK,0CAA0C,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,aAAa,CAAC,CAAC,CAAC,EAAE,WAAW,CAAC,CAAC,CAAC,CAAC,CAAC,EAAec,EAAwBlB,EAAIC,EAAS,CAAC,SAAS,CAAcD,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,yBAAyB,CAAC,EAAE,4IAA4I,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sIAAsI,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,+BAA4CE,EAAEC,EAAE,CAAC,KAAK,+CAA+C,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,mBAAmB,CAAC,CAAC,CAAC,EAAE,gCAA6CF,EAAEC,EAAE,CAAC,KAAK,gKAAgK,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,wBAAwB,CAAC,CAAC,CAAC,EAAE,oIAAoI,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,6BAA6B,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0MAA0M,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uSAAkS,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mJAAmJ,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,gCAAgC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mPAAmP,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kOAAkO,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qhBAAqhB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oWAA0V,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yhBAA+gB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uGAAuG,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAEC,EAAE,CAAC,KAAK,oBAAoB,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,oBAAoB,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EACn4sIe,EAAqB,CAAC,QAAU,CAAC,WAAa,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,UAAY,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,UAAY,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,UAAY,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,UAAY,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,WAAa,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,UAAY,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,UAAY,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,SAAW,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,UAAY,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,UAAY,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,UAAY,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,mBAAqB,CAAC,KAAO,UAAU,CAAC,CAAC",
  "names": ["richText", "u", "x", "p", "Link", "motion", "richText1", "richText2", "richText3", "richText4", "ComponentPresetsConsumer", "t", "CodeBlock_default", "richText5", "richText6", "richText7", "richText8", "richText9", "richText10", "richText11", "__FramerMetadata__"]
}