{
  "version": 3,
  "sources": ["ssg:https://framerusercontent.com/modules/96onCPwMGox8XB0AjCSW/6CZpdvlva43VI6KCZrQZ/wi5ius45d-6.js"],
  "sourcesContent": ["import{jsx as e,jsxs as t}from\"react/jsx-runtime\";import{ComponentPresetsConsumer as n,Link as o}from\"framer\";import{motion as a}from\"framer-motion\";import*as i from\"react\";import{Youtube as r}from\"https://framerusercontent.com/modules/NEd4VmDdsxM3StIUbddO/1de6WpgIbCrKkRcPfQcW/YouTube.js\";import s from\"https://framerusercontent.com/modules/pVk4QsoHxASnVtUBp6jr/TbhpORLndv1iOkZzyo83/CodeBlock.js\";export const richText=/*#__PURE__*/t(i.Fragment,{children:[/*#__PURE__*/e(\"p\",{children:\"In-context learning is a fast and easy way to start improving prompt results. Let's learn how to use in-context learning to quickly improve a simple LLM response for a common business use case: customer service queries on an e-commerce platform. The goal is to generate responses that are not only concise and accurate but also appropriate in tone and context. \"}),/*#__PURE__*/e(\"h2\",{children:\"What is In-Context Learning\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"em\",{children:\"In-Context Learning\"}),' is a fancy way of saying \"provide examples.\" It is useful to think of LLMs as brilliant but inexperienced assistants. Without experience, anyones (much less an LLMs) response to a question will be unpredictable.  By adding examples with explanations, the LLM gains experience and \"learns\" how you want it to behave. But you can\\'t provide an example for every user input. This is where scenarios and evaluation come in. By creating a metric baseline for the models initial behavior, you can measurably add \"just enough\" in-context learning to improve efficiently. Let\\'s improve a simple prompt for WebBizz to see how this works.   ']}),/*#__PURE__*/e(\"h2\",{children:\"Setting up the model and prompt\"}),/*#__PURE__*/e(\"p\",{children:\"Using Okareo, the evaluation process begins with setting up a scenario for evaluation. Scenarios are the inputs that will go to the language model. The LLM then generates a response, which Okareo analyzes.\"}),/*#__PURE__*/t(\"p\",{children:[\"For those who want to follow along or see the complete code, you can find the full notebook \",/*#__PURE__*/e(o,{href:\"https://github.com/okareo-ai/okareo-python-sdk/blob/main/examples/qa_eval_custom_check.ipynb\",motionChild:!0,nodeId:\"wi5ius45d\",openInNewTab:!0,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(a.a,{children:\"here\"})}),\".\"]}),/*#__PURE__*/e(\"p\",{children:'Using a simple customer service example, we first set up the model with our simple prompt, \"Answer the question\". The results generated by the model are then evaluated by Okareo as shown below.'}),/*#__PURE__*/e(\"img\",{alt:\"\",className:\"framer-image\",height:\"725\",src:\"https://framerusercontent.com/images/8dwoj5HdDRvrJje2JhWC4xDL2VU.png\",srcSet:\"https://framerusercontent.com/images/8dwoj5HdDRvrJje2JhWC4xDL2VU.png?scale-down-to=512 512w,https://framerusercontent.com/images/8dwoj5HdDRvrJje2JhWC4xDL2VU.png?scale-down-to=1024 1024w,https://framerusercontent.com/images/8dwoj5HdDRvrJje2JhWC4xDL2VU.png?scale-down-to=2048 2048w,https://framerusercontent.com/images/8dwoj5HdDRvrJje2JhWC4xDL2VU.png 2790w\",style:{aspectRatio:\"2790 / 1450\"},width:\"1395\"}),/*#__PURE__*/e(\"p\",{children:\"This prompt and model got a score of 4.04 on conciseness and 4.26 on relevance. Let's see how we can improve this. \"}),/*#__PURE__*/e(\"h2\",{children:\"Improving the prompt\"}),/*#__PURE__*/e(\"p\",{children:\"We can improve the prompt by using in-context learning to teach the model the right way to answer a question. We can do this by providing examples of good and bad responses to the model. For this example, we will be adding an example of a good response to the prompt. See below for the code that updates the prompt.\"}),/*#__PURE__*/e(\"div\",{className:\"framer-text-module\",style:{height:\"auto\",width:\"100%\"},children:/*#__PURE__*/e(n,{componentIdentifier:\"module:pVk4QsoHxASnVtUBp6jr/TbhpORLndv1iOkZzyo83/CodeBlock.js:default\",children:t=>/*#__PURE__*/e(s,{...t,code:'# Define a template to prompt the model to provide an answer\\n# based on the context \\nANSWER_GIVEN_CONTEXT_TEMPLATE = \"\"\" \\n  You will be given a question and a context. You should \\n  provide an answer to the question based on the context. \\n  Here is an example on how to answer the question based \\n  on the context: Question: What are some ways WebBizz \\n  uses technology to improve customer experience? Context: \\n  WebBizz has implemented a chatbot on their website to \\n  provide instant support to customers. They have also \\n  introduced a loyalty program that rewards customers for \\n  repeat purchases. WebBizz allows for the creation of \\n  wishlists, which can be shared with friends and family. \\n  They also offer personalized recommendations based on \\n  past purchases. WebBizz uses technology in the warehouse \\n  to optimize inventory management and ensure timely \\n  delivery of orders. Answer: WebBizz uses technology to \\n  improve customer experience by implementing a chatbot on \\n  their website, introducing a loyalty program, allowing \\n  for the creation of wishlists, and offering personalized \\n  recommendations. This is an ideal example of how you \\n  should answer the question based on the context provided. \\n  Note how the non-relevant information is omitted, and \\n  the focus is on the key points related to the question. \\n\"\"\" \\n# Register the model to use in a test run \\nmodel_under_test = okareo.register_model(\\n  name=\"OpenAI Answering Model\",\\n  model=OpenAIModel(\\n    model_id=\"gpt-3.5-turbo\",\\n    temperature=0,\\n    system_prompt_template=ANSWER_GIVEN_CONTEXT_TEMPLATE,\\n    user_prompt_template=USER_PROMPT_TEMPLATE,\\n  ),\\n  update=True\\n) ',language:\"Python\"})})}),/*#__PURE__*/e(\"p\",{children:\"Now, we can run the evaluation again to see if the changes have improved the model's performance.\"}),/*#__PURE__*/e(\"img\",{alt:\"\",className:\"framer-image\",height:\"725\",src:\"https://framerusercontent.com/images/6gKfMnscASk8e5GJoenFcCDj54s.png\",srcSet:\"https://framerusercontent.com/images/6gKfMnscASk8e5GJoenFcCDj54s.png?scale-down-to=512 512w,https://framerusercontent.com/images/6gKfMnscASk8e5GJoenFcCDj54s.png?scale-down-to=1024 1024w,https://framerusercontent.com/images/6gKfMnscASk8e5GJoenFcCDj54s.png?scale-down-to=2048 2048w,https://framerusercontent.com/images/6gKfMnscASk8e5GJoenFcCDj54s.png 2790w\",style:{aspectRatio:\"2790 / 1450\"},width:\"1395\"}),/*#__PURE__*/e(\"p\",{children:\"As we can see from the second picture, our in-context learning approach has improved both the conciseness and relevance of the response. The answer now better addresses the customer's specific questions without any extraneous information.\"}),/*#__PURE__*/e(\"h2\",{children:\"Conclusion\"}),/*#__PURE__*/e(\"p\",{children:\"This improvement in response quality can have significant real-world impacts:\"}),/*#__PURE__*/t(\"ul\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Increased customer satisfaction due to clear, direct answers\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Reduced time spent by customers reading responses\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Potential reduction in follow-up queries, easing the load on customer service\"})})]}),/*#__PURE__*/e(\"p\",{children:\"By iteratively using Okareo to evaluate and refine our prompts, we can continuously improve our LLM's performance for specific tasks. This process of evaluation, refinement, and re-evaluation becomes a powerful tool in developing more effective AI-driven solutions.\"}),/*#__PURE__*/e(\"p\",{children:\"Remember, the key to effective LLM optimization is continuous evaluation and refinement. Regular use of evaluation tools, combined with thoughtful prompt engineering, can lead to significant improvements in your generated responses.\"}),/*#__PURE__*/e(\"h2\",{children:\"Okareo automates the process of LLM evaluation\"}),/*#__PURE__*/e(\"p\",{children:\"Start evaluating LLM applications in CI with Okareo and ensure that your LLM-powered functionality works as expected while also saving you countless hours of manual testing. Okareo is free to use for smaller projects (of up to 5k model data points, 1k evaluated rows and 50k scenario tokens).\"}),/*#__PURE__*/t(\"p\",{children:[\"You can \",/*#__PURE__*/e(o,{href:\"https://app.okareo.com/account/sign-up\",motionChild:!0,nodeId:\"wi5ius45d\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(a.a,{children:\"get started\"})}),\" with Okareo immediately for free. The example evaluation in this tutorial will help you get started, as well as our \",/*#__PURE__*/e(o,{href:\"https://github.com/okareo-ai/okareo-cookbook/\",motionChild:!0,nodeId:\"wi5ius45d\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(a.a,{children:\"cookbook\"})}),\" project which contains many more working examples.\"]}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"br\",{className:\"trailing-break\"})})]});export const richText1=/*#__PURE__*/t(i.Fragment,{children:[/*#__PURE__*/e(\"div\",{className:\"framer-text-module\",style:{\"--aspect-ratio\":\"560 / 315\",aspectRatio:\"560 / 315\",height:\"auto\",width:\"100%\"},children:/*#__PURE__*/e(n,{componentIdentifier:\"module:NEd4VmDdsxM3StIUbddO/1de6WpgIbCrKkRcPfQcW/YouTube.js:Youtube\",children:t=>/*#__PURE__*/e(r,{...t,play:\"Off\",shouldMute:!0,thumbnail:\"High Quality\",url:\"https://youtu.be/KHuko_o-wdY?si=5jgf0k5D1381AAIK\"})})}),/*#__PURE__*/t(\"p\",{children:[\"This short video demo is a companion to my recent \",/*#__PURE__*/e(o,{href:\"https://okareo.com/blog/posts/bootstrap-finetuning\",motionChild:!0,nodeId:\"wi5ius45d\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(a.a,{children:\"blog post on bootstraping and improving LLM fine-tuning\"})}),\".\"]}),/*#__PURE__*/e(\"p\",{children:\"The video starts with a quick overview of RAG architectures, which we frame as starting with classification followed by retrieval followed by generation. Then, I show how to use Okareo to help evaluate the classification piece of a RAG agent, and I show how Okareo lets you:\"}),/*#__PURE__*/t(\"ul\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Create synthetic fine-tuning scenarios\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Visualize classification metrics on model score cards and evaluations\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Perform failure row scenario extraction to improve your fine-tuning set\"})})]}),/*#__PURE__*/e(\"p\",{children:\"Want to dive deeper with Okareo? Then you can:\"}),/*#__PURE__*/t(\"ul\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(o,{href:\"https://okareo.com/docs/getting-started/overview\",motionChild:!0,nodeId:\"wi5ius45d\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(a.a,{children:\"Read our documentation\"})}),\" to learn more about evaluation and synthetic data concepts.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(o,{href:\"https://app.okareo.com/account/sign-up\",motionChild:!0,nodeId:\"wi5ius45d\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(a.a,{children:\"Sign-up\"})}),\" with Okareo for free.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[\"Try the notebooks used in this demo for yourself (\",/*#__PURE__*/e(o,{href:\"https://github.com/okareo-ai/okareo-python-sdk/blob/main/examples/classification_finetuning_eval_part1.ipynb\",motionChild:!0,nodeId:\"wi5ius45d\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(a.a,{children:\"Part 1\"})}),\" and \",/*#__PURE__*/e(o,{href:\"https://github.com/okareo-ai/okareo-python-sdk/blob/main/examples/classification_finetuning_eval_part2.ipynb\",motionChild:!0,nodeId:\"wi5ius45d\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(a.a,{children:\"Part 2\"})}),\"). \"]})})]}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"br\",{className:\"trailing-break\"})})]});export const richText2=/*#__PURE__*/t(i.Fragment,{children:[/*#__PURE__*/t(\"p\",{children:[\"Language models behave in unpredictable ways. This can be charming if you\u2019re messing around with ChatGPT, but in high stakes situations, a large language model (LLM) that goes off script can be \",/*#__PURE__*/e(o,{href:\"https://www.bbc.com/news/articles/cd11gzejgz4o\",motionChild:!0,nodeId:\"wi5ius45d\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(a.a,{children:\"reputationally\"})}),\" or \",/*#__PURE__*/e(o,{href:\"https://www.upworthy.com/prankster-tricks-a-gm-dealership-chatbot-to-sell-him-a-76000-chevy-tahoe-for-1-rp\",motionChild:!0,nodeId:\"wi5ius45d\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(a.a,{children:\"materially\"})}),\" harmful. To avoid such gaffes, we want to ensure that models deployed in production behave predictably and factually. In this post, we show you how to use instruction fine-tuning to improve your LLM's performance and how Okareo can accelerate your fine-tuning workflow. \"]}),/*#__PURE__*/e(\"img\",{alt:\"\",className:\"framer-image\",height:\"541\",src:\"https://framerusercontent.com/images/F5usCFRj9VPfdUUS0CYSw7CtGU.png\",srcSet:\"https://framerusercontent.com/images/F5usCFRj9VPfdUUS0CYSw7CtGU.png?scale-down-to=512 512w,https://framerusercontent.com/images/F5usCFRj9VPfdUUS0CYSw7CtGU.png?scale-down-to=1024 1024w,https://framerusercontent.com/images/F5usCFRj9VPfdUUS0CYSw7CtGU.png 1602w\",style:{aspectRatio:\"1602 / 1082\"},width:\"801\"}),/*#__PURE__*/e(\"h2\",{children:\"Why fine-tune when you can prompt?\"}),/*#__PURE__*/e(\"p\",{children:\"The first approach to improving predictability and accuracy of a model is prompt engineering, which is done by iterating between tweaking the model\u2019s system prompt and assessing the LLM\u2019s performance on a fixed, baseline scenario. Prompt engineering in this fashion is often sufficient, but this can often result in large, cumbersome prompts consisting of numerous tokens, eating up the LLM\u2019s context window.\"}),/*#__PURE__*/e(\"p\",{children:\"Enter fine-tuning \u2013 a form of supervised training which uses well-labeled examples to tune an LLM to achieve specific behaviors. The key idea here is that instead of exhaustively describing the desired behavior the LLM should follow (\\xe0 la prompt engineering), we are teaching the LLM how to behave by example. In addition to better performance, this method can yield in much smaller system prompts that consume a relatively small portion of the context window.\"}),/*#__PURE__*/e(\"p\",{children:\"As a form of supervised learning, fine-tuning relies on the availability of an instruction set, which consists of curated input/output pairs preceded by a task description.  In practice, getting access to such data can be time-consuming and expensive (e.g., hiring human annotators to sift through production data). Instead of waiting on production data and labelers to start fine-tuning, we can leverage synthetic data for our initial instruction set to bootstrap the data collection process.\"}),/*#__PURE__*/e(\"p\",{children:\"In this blog post, we will demonstrate how you can use Okareo at all stages of fine-tuning, from bootstrapping your instruction set, to evaluating your fine-tuned model\u2019s performance, and finally to augmenting your instruction set in a systematic way to improve your fine-tuned model\u2019s performance.\"}),/*#__PURE__*/e(\"h2\",{children:\"Case Study: Fine-tuning Phi-3 for Intent Detection\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Key Ideas\"}),\": Okareo synthetic generation and evaluations; Train/test splits; Parameter-efficient fine-tuning; Data augmentation\"]}),/*#__PURE__*/e(\"blockquote\",{children:/*#__PURE__*/t(\"p\",{children:[\"The guide is an outline of our \",/*#__PURE__*/e(\"strong\",{children:\"fine-tuning example notebooks (\"}),/*#__PURE__*/e(o,{href:\"https://github.com/okareo-ai/okareo-python-sdk/blob/main/examples/classification_finetuning_eval_part1.ipynb\",motionChild:!0,nodeId:\"wi5ius45d\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(a.a,{children:/*#__PURE__*/e(\"strong\",{children:\"Part 1\"})})}),/*#__PURE__*/e(\"strong\",{children:\" and \"}),/*#__PURE__*/e(o,{href:\"https://github.com/okareo-ai/okareo-python-sdk/blob/main/examples/classification_finetuning_eval_part2.ipynb\",motionChild:!0,nodeId:\"wi5ius45d\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(a.a,{children:/*#__PURE__*/e(\"strong\",{children:\"Part 2\"})})}),/*#__PURE__*/e(\"strong\",{children:\")\"}),\". To follow along, \",/*#__PURE__*/e(o,{href:\"https://app.okareo.com/account/sign-up\",motionChild:!0,nodeId:\"wi5ius45d\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(a.a,{children:\"sign up for Okareo\"})}),\" and get your API token now!\"]})}),/*#__PURE__*/e(\"p\",{children:\"To frame our fine-tuning workflow, let\u2019s consider a RAG architecture for answering questions about a hypothetical online retailer, WebBizz. To help route user questions to the proper database, we will fine-tune an intent detection model to assign the right label to the question.\"}),/*#__PURE__*/e(\"p\",{children:\"To fine-tune our intent detection LLM, we take the following approach:\"}),/*#__PURE__*/t(\"ol\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[\"Generate user questions based on WebBizz articles using the \",/*#__PURE__*/e(o,{href:\"https://okareo.com/docs/getting-started/concepts/scenarios#reverse-questions\",motionChild:!0,nodeId:\"wi5ius45d\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(a.a,{children:\"Okareo reverse question generator\"})})]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Split the synthetic questions into train/test splits\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Format the train split as an instruction set\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Finetune Phi-3 on the train split\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[\"Evaluate the fine-tuned LLM on the train/test splits in an \",/*#__PURE__*/e(o,{href:\"https://okareo.com/docs/guides/classification_overview\",motionChild:!0,nodeId:\"wi5ius45d\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(a.a,{children:\"Okareo classification evaluation\"})})]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[\"Construct an augmented train split by using the \",/*#__PURE__*/e(o,{href:\"https://okareo.com/docs/guides/classification_overview\",motionChild:!0,nodeId:\"wi5ius45d\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(a.a,{children:\"Okareo rephrasing generator\"})}),\" on misclassified rows\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Finetune Phi-3 on the augmented train split\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Compare the augmented fine-tuned LLM\u2019s train/test performance to the original fine-tuned model\"})})]}),/*#__PURE__*/e(\"h3\",{children:\"1. Generate User Questions\"}),/*#__PURE__*/e(\"p\",{children:\"Our RAG will have a database of WebBizz articles to use as context for generated answers. Before making fine-tuning instructions, we need representative questions that a WebBizz customer might ask.\"}),/*#__PURE__*/e(\"p\",{children:\"To get a set of such questions, we will use the reverse question generator in Okareo. The basic idea is that given a set of articles, the generator produces one or more relevant questions that are answered by the article.\"}),/*#__PURE__*/e(\"img\",{alt:\"\",className:\"framer-image\",height:\"567\",src:\"https://framerusercontent.com/images/R2XDZcJlzW6y1Lyhws7m7KrLNQg.png\",srcSet:\"https://framerusercontent.com/images/R2XDZcJlzW6y1Lyhws7m7KrLNQg.png?scale-down-to=512 512w,https://framerusercontent.com/images/R2XDZcJlzW6y1Lyhws7m7KrLNQg.png?scale-down-to=1024 1024w,https://framerusercontent.com/images/R2XDZcJlzW6y1Lyhws7m7KrLNQg.png?scale-down-to=2048 2048w,https://framerusercontent.com/images/R2XDZcJlzW6y1Lyhws7m7KrLNQg.png 2790w\",style:{aspectRatio:\"2790 / 1134\"},width:\"1395\"}),/*#__PURE__*/e(\"p\",{children:\"Example of synthetic questions generated on a sustainability-related WebBizz article.\"}),/*#__PURE__*/e(\"h3\",{children:\"2. Make Train/Test Splits\"}),/*#__PURE__*/e(\"p\",{children:\"In conventional ML development, we have the notion of a train split and a test split. The train split is used to calculate the model\u2019s loss, and this loss is used during backpropagation to update the weights of the model. The test split is used to evaluate the model\u2019s loss on data that the model has not \u201Cseen\u201D during training. Holding out a test split allows us to have more confidence in the model\u2019s performance on new data that the trained model has not seen (i.e., production data).\"}),/*#__PURE__*/t(\"p\",{children:[\"We can apply the same principle of train/test splits to LLM fine-tuning. In practice, we use scikit-learn\u2019s \",/*#__PURE__*/e(o,{href:\"https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.StratifiedShuffleSplit.html\",motionChild:!0,nodeId:\"wi5ius45d\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(a.a,{children:\"StratifiedShuffleSplit\"})}),\" to ensure that the intent class distribution is consistent between the two splits.\"]}),/*#__PURE__*/e(\"img\",{alt:\"\",className:\"framer-image\",height:\"542\",src:\"https://framerusercontent.com/images/dfWRsgQU5y1lFLh8nkDB8wEoZm8.png\",srcSet:\"https://framerusercontent.com/images/dfWRsgQU5y1lFLh8nkDB8wEoZm8.png?scale-down-to=512 512w,https://framerusercontent.com/images/dfWRsgQU5y1lFLh8nkDB8wEoZm8.png?scale-down-to=1024 1024w,https://framerusercontent.com/images/dfWRsgQU5y1lFLh8nkDB8wEoZm8.png 1824w\",style:{aspectRatio:\"1824 / 1084\"},width:\"912\"}),/*#__PURE__*/e(\"p\",{children:\"Example of splitting synthetic questions into train and test.\"}),/*#__PURE__*/e(\"h3\",{children:\"3. Format the train split as an instruction split\"}),/*#__PURE__*/e(\"p\",{children:\"Now that we have our splits, we can format our questions as fine-tuning instructions. Generally speaking, instructions should be formatted with the following three fields:\"}),/*#__PURE__*/t(\"ul\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Instruction: Description of the task, input/output format, etc. (i.e., intent detection)\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Input: Text used to prompt the LLM (i.e., user question)\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Output: Expected response from the LLM (i.e., predicted intent)\"})})]}),/*#__PURE__*/e(\"p\",{children:\"For our intent detection task, we adopt the instruction template pictured below.\"}),/*#__PURE__*/e(\"img\",{alt:\"\",className:\"framer-image\",height:\"421\",src:\"https://framerusercontent.com/images/RlwAkEiK0IjSgm5qPf1CLYgu60.png\",srcSet:\"https://framerusercontent.com/images/RlwAkEiK0IjSgm5qPf1CLYgu60.png?scale-down-to=512 512w,https://framerusercontent.com/images/RlwAkEiK0IjSgm5qPf1CLYgu60.png?scale-down-to=1024 1024w,https://framerusercontent.com/images/RlwAkEiK0IjSgm5qPf1CLYgu60.png?scale-down-to=2048 2048w,https://framerusercontent.com/images/RlwAkEiK0IjSgm5qPf1CLYgu60.png 2650w\",style:{aspectRatio:\"2650 / 842\"},width:\"1325\"}),/*#__PURE__*/e(\"h3\",{children:\"4. Fine-tune Phi-3 on the formatted train split\"}),/*#__PURE__*/t(\"p\",{children:[\"We will fine-tune \",/*#__PURE__*/e(o,{href:\"https://huggingface.co/microsoft/Phi-3-mini-4k-instruct\",motionChild:!0,nodeId:\"wi5ius45d\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(a.a,{children:\"Phi-3-mini-4k-instruct\"})}),\", a 3.8B parameter open source LLM from Microsoft, to perform intent detection on our user questions. To perform the fine-tuning, we provisioned a GCP VM with a single \",/*#__PURE__*/e(o,{href:\"https://www.nvidia.com/en-us/data-center/l4/\",motionChild:!0,nodeId:\"wi5ius45d\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(a.a,{children:\"Nvidia L4 GPU\"})}),\".\"]}),/*#__PURE__*/t(\"p\",{children:[\"To fine-tune the model in Python, we started with the \",/*#__PURE__*/e(o,{href:\"https://www.philschmid.de/instruction-tune-llama-2\",motionChild:!0,nodeId:\"wi5ius45d\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(a.a,{children:\"excellent tutorial by Phillip Schmid of Hugging Face\"})}),\". He demonstrates how to fine-tune Llama2 using several parameter-efficient fine-tuning techniques, including \",/*#__PURE__*/e(o,{href:\"https://arxiv.org/pdf/2106.09685\",motionChild:!0,nodeId:\"wi5ius45d\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(a.a,{children:\"LoRA\"})}),\", \",/*#__PURE__*/e(o,{href:\"https://huggingface.co/blog/4bit-transformers-bitsandbytes\",motionChild:!0,nodeId:\"wi5ius45d\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(a.a,{children:\"4-bit quantization\"})}),\", and \",/*#__PURE__*/e(o,{href:\"https://arxiv.org/abs/2205.14135\",motionChild:!0,nodeId:\"wi5ius45d\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(a.a,{children:\"flash attention\"})}),\", which all help the LLM fit in GPU memory.\"]}),/*#__PURE__*/e(\"h3\",{children:\"5. Evaluate the fine-tuned LLM in Okareo\"}),/*#__PURE__*/e(\"p\",{children:\"After fine-tuning Phi-3, we used Okareo to perform a classification evaluation. In brief, this process involved:\"}),/*#__PURE__*/t(\"ul\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Uploading the train/test splits as scenarios\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Uploading the fine-tuned Phi-3 as a CustomModel\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Starting test runs of the CustomModel on each scenario\"})})]}),/*#__PURE__*/e(\"p\",{children:\"On the \u201CPublished\u201D tab of Okareo, we can compare different evaluation runs against each other. The figure below shows key classification metrics for the fine-tuned model as reported by Okareo.\"}),/*#__PURE__*/e(\"img\",{alt:\"\",className:\"framer-image\",height:\"834\",src:\"https://framerusercontent.com/images/Esa0ugZORh7EaWz51TAkCjTS6G0.png\",srcSet:\"https://framerusercontent.com/images/Esa0ugZORh7EaWz51TAkCjTS6G0.png?scale-down-to=512 512w,https://framerusercontent.com/images/Esa0ugZORh7EaWz51TAkCjTS6G0.png?scale-down-to=1024 1024w,https://framerusercontent.com/images/Esa0ugZORh7EaWz51TAkCjTS6G0.png?scale-down-to=2048 2048w,https://framerusercontent.com/images/Esa0ugZORh7EaWz51TAkCjTS6G0.png 2088w\",style:{aspectRatio:\"2088 / 1668\"},width:\"1044\"}),/*#__PURE__*/e(\"p\",{children:\"Okareo Model Cards showing fine-tuned Phi-3 performance on the WebBizz train/test splits.\"}),/*#__PURE__*/e(\"p\",{children:\"We can also click into one of these evaluations to get more granular information about the evaluation run, like the confusion matrix.\"}),/*#__PURE__*/e(\"img\",{alt:\"\",className:\"framer-image\",height:\"871\",src:\"https://framerusercontent.com/images/FXMLPJY9FQvtgEtzbhFWrrWBk.png\",srcSet:\"https://framerusercontent.com/images/FXMLPJY9FQvtgEtzbhFWrrWBk.png?scale-down-to=512 512w,https://framerusercontent.com/images/FXMLPJY9FQvtgEtzbhFWrrWBk.png?scale-down-to=1024 1024w,https://framerusercontent.com/images/FXMLPJY9FQvtgEtzbhFWrrWBk.png 1756w\",style:{aspectRatio:\"1756 / 1742\"},width:\"878\"}),/*#__PURE__*/e(\"p\",{children:\"Okareo Evaluation showing topline metrics and the confusion matrix of Phi-3 on the test split.\"}),/*#__PURE__*/e(\"h3\",{children:\"6. Augment the train split with misclassified rows\"}),/*#__PURE__*/e(\"p\",{children:\"Given our fixed train split, how can we quickly marshall more data to improve our fine-tuned model\u2019s performance? In this case, we will use the results of our classification evaluation to help us guide this process. More specifically, we will do the following:\"}),/*#__PURE__*/t(\"ol\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Filter the train split based on incorrectly classified questions\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Use the rephrasing generator on these questions\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Make an \u201Caugmented\u201D train split with the rephrased questions\"})})]}),/*#__PURE__*/e(\"img\",{alt:\"\",className:\"framer-image\",height:\"621\",src:\"https://framerusercontent.com/images/duXHUr20vIhH3K2phNuBIkeGiI.png\",srcSet:\"https://framerusercontent.com/images/duXHUr20vIhH3K2phNuBIkeGiI.png?scale-down-to=512 512w,https://framerusercontent.com/images/duXHUr20vIhH3K2phNuBIkeGiI.png?scale-down-to=1024 1024w,https://framerusercontent.com/images/duXHUr20vIhH3K2phNuBIkeGiI.png?scale-down-to=2048 2048w,https://framerusercontent.com/images/duXHUr20vIhH3K2phNuBIkeGiI.png 2426w\",style:{aspectRatio:\"2426 / 1242\"},width:\"1213\"}),/*#__PURE__*/e(\"p\",{children:\"Strategy for augmenting training data for our fine-tuning workflow\"}),/*#__PURE__*/e(\"h3\",{children:\"7. Repeat fine-tuning with the augmented split\"}),/*#__PURE__*/e(\"p\",{children:\"This step is identical to step #3, but this time we use the augmented train split to craft our instruction set.\"}),/*#__PURE__*/e(\"h3\",{children:\"8. Compare augmented LLM performance\"}),/*#__PURE__*/e(\"p\",{children:\"After uploading the augmented Phi-3 model to Okareo, we can compare its classification metrics against the original fine-tuned model. We observe that the augmented model\u2019s performance improves across all metrics for both splits!\"}),/*#__PURE__*/e(\"img\",{alt:\"\",className:\"framer-image\",height:\"1575\",src:\"https://framerusercontent.com/images/uFiK9jndOLURNYcxzXup7zKJE.png\",srcSet:\"https://framerusercontent.com/images/uFiK9jndOLURNYcxzXup7zKJE.png?scale-down-to=1024 612w,https://framerusercontent.com/images/uFiK9jndOLURNYcxzXup7zKJE.png?scale-down-to=2048 1224w,https://framerusercontent.com/images/uFiK9jndOLURNYcxzXup7zKJE.png 1884w\",style:{aspectRatio:\"1884 / 3150\"},width:\"942\"}),/*#__PURE__*/e(\"h2\",{children:\"Conclusion\"}),/*#__PURE__*/e(\"p\",{children:\"In this post, we started by comparing prompt engineering and fine-tuning. Then we demonstrated how we can use Okareo\u2019s synthetic data generators and classification evaluations to accelerate fine-tuning of an intent detection model for a RAG system. In a future post, we will showcase how Okareo can be used to help you fine-tune and evaluate other stages of your RAG pipeline, including retrieval and generation.\"})]});export const richText3=/*#__PURE__*/t(i.Fragment,{children:[/*#__PURE__*/e(\"p\",{children:\"Okareo is a large language model (LLM) evaluation tool that allows you to integrate LLM evaluation into your CI workflow. We show how to get Okareo working in your CI.\"}),/*#__PURE__*/e(\"p\",{children:\"When building a product that uses third-party LLMs (such as those available in OpenAI), it's important to regularly evaluate that these LLMs still work as expected, especially when you update a model or other parts of your software. However, testing LLMs isn't as straightforward as testing regular application code.\"}),/*#__PURE__*/e(\"p\",{children:\"Application code testing can be easily automated, by creating unit tests and running them as part of your continuous integration (CI) pipelines, so that every time your code gets updated, the tests can be re-run by the CI system against the new code. But as LLMs have non-deterministic outputs, it's impossible to write unit tests for the parts of your app that interact with LLMs, as the output will change each time you run the test. Until now, manual testing has been the only real way to ensure the accuracy of LLMs as they continue to develop.\"}),/*#__PURE__*/e(\"p\",{children:\"Okareo allows you to skip tedious work on manual tests by offering a way to automate tests for non-deterministic systems like LLMs and other AI models. Using Okareo it's now possible to integrate your automated LLM evaluations into your CI workflow.\"}),/*#__PURE__*/e(\"h2\",{children:\"What is LLM evaluation?\"}),/*#__PURE__*/e(\"p\",{children:\"LLM evaluation is a way to measure how good an LLM is at performing certain tasks \u2014 such as text completion, summarization, or question answering. It assesses the performance of an LLM by comparing the model's output against some expected results to check things like accuracy and relevancy.\"}),/*#__PURE__*/e(\"p\",{children:\"Without LLM evaluation, you won't be able to check that your model can be generalized beyond the training data you've given it. As LLMs are now being integrated into app development, it's important to have confidence in your model at all times \u2013 hence the need to add an LLM evaluation step to your CI workflow.\"}),/*#__PURE__*/e(\"h2\",{children:\"What is Okareo?\"}),/*#__PURE__*/e(\"p\",{children:\"Okareo is a tool for developers that can evaluate the output of the LLMs that power your AI apps. You can write tests that evaluate your LLM in either TypeScript or Python, and it has a CLI interface for running all your LLM evaluations, meaning it can be easily integrated into your CI workflow, no matter which CI provider you use.\"}),/*#__PURE__*/t(\"p\",{children:[\"Okareo's LLM evaluation process involves comparing a \",/*#__PURE__*/e(o,{href:\"https://okareo.com/docs/getting-started/concepts/scenarios\",motionChild:!0,nodeId:\"wi5ius45d\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(a.a,{children:/*#__PURE__*/e(\"em\",{children:\"scenario\"})})}),\" (consisting of inputs to a model along with their corresponding expected results) with the actual results of an LLM given the same inputs.\"]}),/*#__PURE__*/t(\"p\",{children:[\"Okareo treats the LLM as a black box \u2014 it's simply interested in whether the output data of the LLM conforms to certain rules or standards, known as \",/*#__PURE__*/e(o,{href:\"https://okareo.com/docs/getting-started/concepts/scenarios\",motionChild:!0,nodeId:\"wi5ius45d\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(a.a,{children:/*#__PURE__*/e(\"em\",{children:\"checks\"})})}),\". Checks are a way to compare the similarity of the expected and actual results according to specific metrics like \",/*#__PURE__*/e(\"em\",{children:\"consistency\"}),\" or \",/*#__PURE__*/e(\"em\",{children:\"relevance\"}),\".\"]}),/*#__PURE__*/t(\"p\",{children:[\"You can either use Okareo's pre-defined checks or create your own custom ones. Some checks are measured as a simple pass or fail, and others are scored with a range (such as 1-5). For those with a range, you can decide the threshold at which it passes or fails. For example, you might choose to set the minimum pass threshold of relevance to 4 out of 5. By default, all checks must pass in order for your LLM evaluation CI pipeline to pass overall, although this can be overridden with a property called \",/*#__PURE__*/e(\"code\",{children:\"[error_max](https://okareo.com/docs/sdk/okareo_typescript)\"}),\", which specifies how many errors you\u2019re willing to tolerate and still pass.\"]}),/*#__PURE__*/e(\"img\",{alt:\"Flow diagram showing the processes that happen in Okareo as part of LLM evaluation.\",className:\"framer-image\",height:\"600\",src:\"https://framerusercontent.com/images/Koxwj53hhGYWNT2z9raSkGSoBA0.png\",srcSet:\"https://framerusercontent.com/images/Koxwj53hhGYWNT2z9raSkGSoBA0.png?scale-down-to=512 512w,https://framerusercontent.com/images/Koxwj53hhGYWNT2z9raSkGSoBA0.png?scale-down-to=1024 1024w,https://framerusercontent.com/images/Koxwj53hhGYWNT2z9raSkGSoBA0.png?scale-down-to=2048 2048w,https://framerusercontent.com/images/Koxwj53hhGYWNT2z9raSkGSoBA0.png 2160w\",style:{aspectRatio:\"2160 / 1200\"},width:\"1080\"}),/*#__PURE__*/e(\"h2\",{children:\"How to use Okareo to evaluate an LLM-powered application or feature in CI\"}),/*#__PURE__*/t(\"p\",{children:[\"The main steps involved are to create an LLM, write evaluation tests for the LLM using Okareo's TypeScript or Python libraries, and finally run those tests as part of a CI workflow using the Okareo CLI. To follow along with this tutorial, there is no need to create your own LLM as we'll be evaluating an existing one (\",/*#__PURE__*/e(\"code\",{children:\"gpt-3.5-turbo\"}),\" on OpenAI).\"]}),/*#__PURE__*/t(\"p\",{children:[\"First, you'll need to sign up to \",/*#__PURE__*/e(o,{href:\"https://app.okareo.com/account/sign-up\",motionChild:!0,nodeId:\"wi5ius45d\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(a.a,{children:\"Okareo\"})}),\" and \",/*#__PURE__*/e(o,{href:\"https://platform.openai.com/signup\",motionChild:!0,nodeId:\"wi5ius45d\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(a.a,{children:\"OpenAI\"})}),\" and get API keys for each account.\"]}),/*#__PURE__*/t(\"p\",{children:[\"This example uses the Okareo \",/*#__PURE__*/e(o,{href:\"https://okareo.com/docs/sdk/okareo_typescript\",motionChild:!0,nodeId:\"wi5ius45d\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(a.a,{children:\"TypeScript\"})}),\" SDK, but \",/*#__PURE__*/e(o,{href:\"https://okareo.com/docs/sdk/okareo_python\",motionChild:!0,nodeId:\"wi5ius45d\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(a.a,{children:\"Python\"})}),\" is also available.\"]}),/*#__PURE__*/e(\"h3\",{children:\"Running an Okareo flow locally\"}),/*#__PURE__*/t(\"p\",{children:[\"An Okareo flow is a TypeScript or Python script that calls different parts of the Okareo API to complete an Okareo evaluation. Below are the instructions you'll need to get a flow running locally. We'll be using a text summarization model for our example, and we\u2019ve shared the \",/*#__PURE__*/e(o,{href:\"https://github.com/okareo-ai/okareo-cookbook/tree/main/tutorials/llm-evaluation-in-ci\",motionChild:!0,nodeId:\"wi5ius45d\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(a.a,{children:\"entire project on GitHub\"})}),\" with you to make it easier to follow along.\"]}),/*#__PURE__*/t(\"ol\",{children:[/*#__PURE__*/t(\"li\",{\"data-preset-tag\":\"p\",children:[/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Install the Okareo CLI\"}),\": Install this first on your local machine so you can test your Okareo evaluations locally before integrating them into your CI. The \",/*#__PURE__*/e(o,{href:\"https://okareo.com/docs/sdk/cli\",motionChild:!0,nodeId:\"wi5ius45d\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(a.a,{children:\"installation instructions \"})}),\"describe how to set Okareo's language to TypeScript and create a file structure for your Okareo flows. Once you've finished following these instructions you should have a directory structure as follows:\"]}),/*#__PURE__*/e(\"div\",{className:\"framer-text-module\",style:{height:\"auto\",width:\"100%\"},children:/*#__PURE__*/e(n,{componentIdentifier:\"module:pVk4QsoHxASnVtUBp6jr/TbhpORLndv1iOkZzyo83/CodeBlock.js:default\",children:t=>/*#__PURE__*/e(s,{...t,code:\"[Project]   \\n  .okareo\\n    config.yml\\n    flows\\n      <your_flow_script>.ts \",language:\"JSX\"})})})]}),/*#__PURE__*/t(\"li\",{\"data-preset-tag\":\"p\",children:[/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Install dependencies for Okareo and OpenAI:\"}),\" You can install the required packages with \"]}),/*#__PURE__*/e(\"div\",{className:\"framer-text-module\",style:{height:\"auto\",width:\"100%\"},children:/*#__PURE__*/e(n,{componentIdentifier:\"module:pVk4QsoHxASnVtUBp6jr/TbhpORLndv1iOkZzyo83/CodeBlock.js:default\",children:t=>/*#__PURE__*/e(s,{...t,code:\"npm install okareo-ts-sdk openai\",language:\"JSX\"})})}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"br\",{className:\"trailing-break\"})})]}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[\"Next, set your API keys and Okareo project ID environment variables so they can be used in \",/*#__PURE__*/e(\"code\",{children:\"<your_flow_script>.ts\"}),\" and \",/*#__PURE__*/e(\"code\",{children:\"config.yml\"}),\".\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Register your model with Okareo:\"}),\" In this case, we're using Open AI's \",/*#__PURE__*/e(\"code\",{children:\"gpt-3.5-turbo\"}),\" model. For this example, we're asking the model to summarize text that is sent to it into a single sentence.\"]})})]}),/*#__PURE__*/e(\"div\",{className:\"framer-text-module\",style:{height:\"auto\",width:\"100%\"},children:/*#__PURE__*/e(n,{componentIdentifier:\"module:pVk4QsoHxASnVtUBp6jr/TbhpORLndv1iOkZzyo83/CodeBlock.js:default\",children:t=>/*#__PURE__*/e(s,{...t,code:'// Define the system prompt to be sent to the OpenAI model \\nconst SUMMARIZATION_CONTEXT_TEMPLATE = \"You will be provided with text. Summarize the text in 1 simple sentence.\"\\n// Define the user prompt to be sent to the OpenAI model \\n// \"{input}\" is a placeholder variable that will later be replaced with \\n// the input from each item in your scenario  \\nconst USER_PROMPT_TEMPLATE = \"{input}\"  \\n// Use a UNIQUE_BUILD_ID which can be used to ensure that the names or tags  \\n// of anything you create are unique to every CI run  \\nconst UNIQUE_BUILD_ID = (process.env.DEMO_BUILD_ID || `local.${(Math.random() + 1).toString(36).substring(7)}`);  \\n// Register your model with Okareo  \\nconst model = await okareo.register_model({\\n  name: MODEL_NAME,\\n  tags: [`Build:${UNIQUE_BUILD_ID}`],\\n  project_id: project_id, \\n  models: {\\n    type: \"openai\",\\n    model_id:\"gpt-3.5-turbo\",\\n    temperature:0.5,\\n    system_prompt_template:SUMMARIZATION_CONTEXT_TEMPLATE,\\n    user_prompt_template:USER_PROMPT_TEMPLATE,\\n  } as OpenAIModel,\\n  update: true,\\n}); ',language:\"JSX\"})})}),/*#__PURE__*/e(\"ol\",{start:\"4\",children:/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Create your scenario:\"}),\" Your scenario is a set of inputs and expected results. Scenarios can either be manually created (known as a \",/*#__PURE__*/e(o,{href:\"https://okareo.com/docs/getting-started/concepts/scenarios#seed-scenarios\",motionChild:!0,nodeId:\"wi5ius45d\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(a.a,{children:\"seed scenario\"})}),\") or generated from previous scenarios. In this example, each result is an ID of a particular document in the \",/*#__PURE__*/e(\"code\",{children:\"gpt-3.5-turbo\"}),\" vector database. The documents have already been manually verified as being acceptable results for the input data.\"]})})}),/*#__PURE__*/e(\"div\",{className:\"framer-text-module\",style:{height:\"auto\",width:\"100%\"},children:/*#__PURE__*/e(n,{componentIdentifier:\"module:pVk4QsoHxASnVtUBp6jr/TbhpORLndv1iOkZzyo83/CodeBlock.js:default\",children:t=>/*#__PURE__*/e(s,{...t,code:'const TEST_SEED_DATA = [\\n  SeedData({\\n    input:\"WebBizz is dedicated to providing our customers with a seamless online shopping experience. Our platform is designed with user-friendly interfaces to help you browse and select the best products suitable for your needs. We offer a wide range of products from top brands and new entrants, ensuring diversity and quality in our offerings. Our 24/7 customer support is ready to assist you with any queries, from product details, shipping timelines, to payment methods. We also have a dedicated FAQ section addressing common concerns. Always ensure you are logged in to enjoy personalized product recommendations and faster checkout processes.\", \\n    result:\"75eaa363-dfcc-499f-b2af-1407b43cb133\"\\n  }),\\n  SeedData({\\n    input:\"Safety and security of your data is our top priority at WebBizz. Our platform employs state-of-the-art encryption methods ensuring your personal and financial information remains confidential. Our two-factor authentication at checkout provides an added layer of security. We understand the importance of timely deliveries, hence we\\'ve partnered with reliable logistics partners ensuring your products reach you in pristine condition. In case of any delays or issues, our tracking tool can provide real-time updates on your product\\'s location. We believe in transparency and guarantee no hidden fees or charges during your purchase journey.\",\\n    result:\"ac0d464c-f673-44b8-8195-60c965e47525\"\\n  }),\\n  SeedData({\\n    input:\"WebBizz places immense value on its dedicated clientele, recognizing their loyalty through the exclusive \\'Premium Club\\' membership. This special program is designed to enrich the shopping experience, providing a suite of benefits tailored to our valued members. Among the advantages, members enjoy complimentary shipping, granting them a seamless and cost-effective way to receive their purchases. Additionally, the \\'Premium Club\\' offers early access to sales, allowing members to avail themselves of promotional offers before they are opened to the general public.\",\\n    result:\"aacf7a34-9d3a-4e2a-9a5c-91f2a0e8a12d\"\\n  }) \\n]; \\n// Get the ID of your Okareo project (which is needed to create a scenario \\n// for your particular project). You can find your project name in the top \\n// right of the Okareo app. \\nconst PROJECT_NAME = \"Global\"; \\nconst project: any[] = await okareo.getProjects(); \\nconst project_id = project.find(p => p.name === PROJECT_NAME)?.id; \\n// Create the scenario \\nconst scenario: any = await okareo.create_scenario_set({\\n  name: `${SCENARIO_SET_NAME} Scenario Set - ${UNIQUE_BUILD_ID}`,\\n  project_id: project_id,\\n  seed_data: TEST_SEED_DATA\\n}); \\n',language:\"JSX\"})})}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Tip: Alternatively, you can upload a scenario set from a file using upload\"}),/*#__PURE__*/e(\"em\",{children:/*#__PURE__*/e(\"strong\",{children:\"scenario\"})}),/*#__PURE__*/e(\"strong\",{children:\"set().\"})]}),/*#__PURE__*/e(\"ol\",{start:\"5\",children:/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Run the LLM evaluation:\"}),\" Call Okareo's \",/*#__PURE__*/e(\"code\",{children:\"run_test()\"}),\" function, remembering to set the type to \",/*#__PURE__*/e(\"code\",{children:\"NL_GENERATION\"}),\" as this is a natural language model that you're testing and Okareo also does testing for other types of models. Also pass in any checks that you want to be done at this stage.\"]})})}),/*#__PURE__*/e(\"div\",{className:\"framer-text-module\",style:{height:\"auto\",width:\"100%\"},children:/*#__PURE__*/e(n,{componentIdentifier:\"module:pVk4QsoHxASnVtUBp6jr/TbhpORLndv1iOkZzyo83/CodeBlock.js:default\",children:t=>/*#__PURE__*/e(s,{...t,code:'const eval_run: components[\"schemas\"][\"TestRunItem\"] =\\n  await model.run_test({\\n    model_api_key: OPENAI_API_KEY,\\n    name: `${MODEL_NAME} Eval ${UNIQUE_BUILD_ID}`,\\n    tags: [`Build:${UNIQUE_BUILD_ID}`],\\n    project_id: project_id,\\n    scenario: scenario,\\n    calculate_metrics: true,\\n    type: TestRunType.NL_GENERATION,\\n    checks: [\\n      \"coherence_summary\",\\n      \"consistency_summary\",\\n      \"fluency_summary\",\\n      \"relevance_summary\"\\n    ]\\n  } as RunTestProps);',language:\"JSX\"})})}),/*#__PURE__*/e(\"ol\",{start:\"6\",children:/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Set the thresholds for your checks:\"}),\" These will be needed for reporting purposes, to help determine whether your evaluation was a success. A common way to do this is to set a minimum threshold that each metric must meet (on average) in order to pass or fail.\"]})})}),/*#__PURE__*/e(\"div\",{className:\"framer-text-module\",style:{height:\"auto\",width:\"100%\"},children:/*#__PURE__*/e(n,{componentIdentifier:\"module:pVk4QsoHxASnVtUBp6jr/TbhpORLndv1iOkZzyo83/CodeBlock.js:default\",children:t=>/*#__PURE__*/e(s,{...t,code:'const report_definition = {\\n  metrics_min: {\\n    \"coherence\": 4.0,\\n    \"consistency\": 4.0,\\n    \"fluency\": 4.0,\\n    \"relevance\": 4.0,\\n  }\\n}; ',language:\"JSX\"})})}),/*#__PURE__*/e(\"ol\",{start:\"7\",children:/*#__PURE__*/t(\"li\",{\"data-preset-tag\":\"p\",children:[/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Set up reporting:\"}),\" Okareo's GenerationReporter gives you statistics on how well your evaluation passed each check metric and whether it passed overall.\"]}),/*#__PURE__*/t(\"p\",{children:[\"Using\",/*#__PURE__*/e(\"code\",{children:\"reporter.log()\"}),\"will log the details of your success or failure, and print a link to a detailed online report, but you can also report whether the evaluation was a success or failure by checking if \",/*#__PURE__*/e(\"code\",{children:\"reporter.pass\"}),\" is true.\"]})]})}),/*#__PURE__*/e(\"div\",{className:\"framer-text-module\",style:{height:\"auto\",width:\"100%\"},children:/*#__PURE__*/e(n,{componentIdentifier:\"module:pVk4QsoHxASnVtUBp6jr/TbhpORLndv1iOkZzyo83/CodeBlock.js:default\",children:t=>/*#__PURE__*/e(s,{...t,code:\"const reporter = new GenerationReporter({\\n  eval_run :eval_run,\\n  ...report_definition,\\n});\\nreporter.log();\",language:\"JSX\"})})}),/*#__PURE__*/t(\"ol\",{start:\"8\",children:[/*#__PURE__*/t(\"li\",{\"data-preset-tag\":\"p\",children:[/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Handle evaluation failures and other errors\"}),\": There are two types of problem that you need to make sure your code can handle:\"]}),/*#__PURE__*/t(\"ol\",{children:[/*#__PURE__*/t(\"li\",{\"data-preset-tag\":\"p\",children:[/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Okareo reports that an evaluation did not pass\"}),\": As we will later be running this code in GitHub Actions, we can use the GitHub Actions core library to handle failures, which will work locally and in GitHub Actions.\"]}),/*#__PURE__*/e(\"div\",{className:\"framer-text-module\",style:{height:\"auto\",width:\"100%\"},children:/*#__PURE__*/e(n,{componentIdentifier:\"module:pVk4QsoHxASnVtUBp6jr/TbhpORLndv1iOkZzyo83/CodeBlock.js:default\",children:t=>/*#__PURE__*/e(s,{...t,code:'import * as core from \"@actions/core\";\\nif (!reporter.pass) {\\n  core.setFailed(\"CI failed because the Okareo reporter failed.\");\\n} ',language:\"JSX\"})})})]}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"TypeScript runtime errors:\"}),\" It's standard practice to handle runtime errors by wrapping your code in a try/catch block.  You can continue to use the GitHub Actions core library to  report the error.\"]})})]}),/*#__PURE__*/e(\"div\",{className:\"framer-text-module\",style:{height:\"auto\",width:\"100%\"},children:/*#__PURE__*/e(n,{componentIdentifier:\"module:pVk4QsoHxASnVtUBp6jr/TbhpORLndv1iOkZzyo83/CodeBlock.js:default\",children:t=>/*#__PURE__*/e(s,{...t,code:'try {     \\n  // TypeScript code that calls Okareo \\n} catch (error) {\\n  core.setFailed(\"CI failed because of an error calling Okareo: \"+error.message);\\n}',language:\"JSX\"})})})]}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Run your Okareo flow script:\"}),\" On your local machine, run the command below.\"]})})]}),/*#__PURE__*/e(\"div\",{className:\"framer-text-module\",style:{height:\"auto\",width:\"100%\"},children:/*#__PURE__*/e(n,{componentIdentifier:\"module:pVk4QsoHxASnVtUBp6jr/TbhpORLndv1iOkZzyo83/CodeBlock.js:default\",children:t=>/*#__PURE__*/e(s,{...t,code:\"okareo run -f <your_flow_script> \",language:\"Shell\"})})}),/*#__PURE__*/t(\"p\",{children:[\"This will output reporting data to the CLI from the \",/*#__PURE__*/e(\"code\",{children:\"reporter.log()\"}),\" function, indicating whether the evaluation has passed or failed, and printing a link that takes you to a reporting page in Okareo where you can visualize your results more easily.\"]}),/*#__PURE__*/e(\"img\",{alt:'Screenshot of command line output for a successful \"okareo run\" command.',className:\"framer-image\",height:\"121\",src:\"https://framerusercontent.com/images/c4DerxtNNGVmne8pdlKHvUdVVAM.png\",srcSet:\"https://framerusercontent.com/images/c4DerxtNNGVmne8pdlKHvUdVVAM.png?scale-down-to=512 512w,https://framerusercontent.com/images/c4DerxtNNGVmne8pdlKHvUdVVAM.png?scale-down-to=1024 1024w,https://framerusercontent.com/images/c4DerxtNNGVmne8pdlKHvUdVVAM.png 1815w\",style:{aspectRatio:\"1815 / 243\"},width:\"907\"}),/*#__PURE__*/e(\"img\",{alt:'Screenshot of command line output for a failed \"okareo run\" command.',className:\"framer-image\",height:\"313\",src:\"https://framerusercontent.com/images/r0iniYZRPC8EdGw99egDtdllPCc.png\",srcSet:\"https://framerusercontent.com/images/r0iniYZRPC8EdGw99egDtdllPCc.png?scale-down-to=512 512w,https://framerusercontent.com/images/r0iniYZRPC8EdGw99egDtdllPCc.png?scale-down-to=1024 1024w,https://framerusercontent.com/images/r0iniYZRPC8EdGw99egDtdllPCc.png 1815w\",style:{aspectRatio:\"1815 / 627\"},width:\"907\"}),/*#__PURE__*/e(\"img\",{alt:\"Screenshot of the Okareo reporting page for a text summarization LLM evaluation.\",className:\"framer-image\",height:\"365\",src:\"https://framerusercontent.com/images/ptS6urbs4IcH8ESZ8HFfw5sCvs.png\",style:{aspectRatio:\"983 / 731\"},width:\"491\"}),/*#__PURE__*/e(\"h3\",{children:\"Integrating Okareo into your CI workflow\"}),/*#__PURE__*/t(\"p\",{children:[\"Now that you've got your Okareo flow running locally, you can add it to your CI workflow. To run Okareo in CI, you will need to install the Okareo CLI on your CI server, set up your API keys as CI environment variables, and add your \",/*#__PURE__*/e(\"code\",{children:\"okareo run\"}),\" command to your CI workflow.\"]}),/*#__PURE__*/e(\"p\",{children:\"Here, we show how to do this in GitHub Actions, but the principles can be easily generalized to any other CI provider, like CircleCI, BitBucket Pipelines, or GitLab CI/CD.\"}),/*#__PURE__*/t(\"p\",{children:[\"Make sure you already have a GitHub repo with your Okareo project in it (the \",/*#__PURE__*/e(\"code\",{children:\".okareo\"}),\" directory should be at the top level of the project).\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Add secrets as environment variables:\"}),\" On your repository's main page, click \",/*#__PURE__*/e(\"em\",{children:\"Settings\"}),\" > \",/*#__PURE__*/e(\"em\",{children:\"Secrets and variables\"}),\" > \",/*#__PURE__*/e(\"em\",{children:\"Actions\"}),\", and then \",/*#__PURE__*/e(\"em\",{children:\"New repository secret\"}),\". Add secrets for \",/*#__PURE__*/e(\"code\",{children:\"OKAREO_API_KEY, OKAREO_PROJECT_ID\"}),\"  and \",/*#__PURE__*/e(\"code\",{children:\"OPENAI_API_KEY.\"})]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Setup workflow file:\"}),\" Inside your repo, click the \",/*#__PURE__*/e(\"em\",{children:\"Actions\"}),\" tab, then on \",/*#__PURE__*/e(\"em\",{children:\"Set up a workflow as yourself\"}),\". This will create a GitHub Actions workflow config file at \",/*#__PURE__*/e(\"code\",{children:\".github/workflows/main.yml\"}),\".\"]}),/*#__PURE__*/t(\"p\",{children:[\"Add the following code to your \",/*#__PURE__*/e(\"code\",{children:\"main.yml\"}),\" file. The Okareo action installs the Okareo CLI before Okareo tries to run your evaluation.\"]}),/*#__PURE__*/e(\"div\",{className:\"framer-text-module\",style:{height:\"auto\",width:\"100%\"},children:/*#__PURE__*/e(n,{componentIdentifier:\"module:pVk4QsoHxASnVtUBp6jr/TbhpORLndv1iOkZzyo83/CodeBlock.js:default\",children:t=>/*#__PURE__*/e(s,{...t,code:\"name: Text summarization Okareo flow\\nenv:\\n  DEMO_BUILD_ID: ${{ github.run_number }}\\n  OKAREO_API_KEY: ${{ secrets.OKAREO_API_KEY }}\\n  OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}\\n  OKAREO_PROJECT_ID: ${{ secrets.OKAREO_PROJECT_ID }}\\n\\non:\\n  push:\\n    branches: [ \\\"main\\\" ]\\n  pull_request:\\n    branches: [ \\\"main\\\" ]\\n\\njobs:   \\n  text-summarization:\\n    runs-on: ubuntu-latest\\n    defaults:\\n    run:\\n    working-directory: .\\n    permissions:\\n    contents: 'read'\\n    id-token: 'write'\\n    steps:\\n      - name: Checkout\\n      uses: actions/checkout@v4\\n      \\n      - name: Okareo Action\\n      uses: okareo-ai/okareo-action@v2.5\\n      \\n      - name: Text Summarization Evaluation\\n      \\n      run: |\\n        okareo -v\\n        okareo run -f text_summarization \",language:\"YAML\"})})}),/*#__PURE__*/e(\"p\",{children:\"Now save your workflow file. This will trigger your GitHub Actions workflow to run (as will any push or pull request to your main branch).\"}),/*#__PURE__*/e(\"img\",{alt:\"Screenshot of the LLM evaluation running in a CI workflow (GitHub Actions), with each step checked off to show a successful run.\",className:\"framer-image\",height:\"363\",src:\"https://framerusercontent.com/images/asNT0RJrdNQs0ePpG88OTbFB5g.png\",srcSet:\"https://framerusercontent.com/images/asNT0RJrdNQs0ePpG88OTbFB5g.png?scale-down-to=512 512w,https://framerusercontent.com/images/asNT0RJrdNQs0ePpG88OTbFB5g.png 1003w\",style:{aspectRatio:\"1003 / 727\"},width:\"501\"}),/*#__PURE__*/e(\"h2\",{children:\"Okareo automates the process of LLM evaluation\"}),/*#__PURE__*/e(\"p\",{children:\"Start evaluating LLM applications in CI with Okareo and ensure that your LLM-powered functionality works as expected while also saving you countless hours of manual testing. Okareo is free to use for smaller projects (of up to 5k model data points, 1k evaluated rows and 50k scenario tokens).\"}),/*#__PURE__*/t(\"p\",{children:[\"You can \",/*#__PURE__*/e(o,{href:\"https://app.okareo.com/account/sign-up\",motionChild:!0,nodeId:\"wi5ius45d\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(a.a,{children:\"get started\"})}),\" with Okareo immediately by signing up and installing the software. The example evaluation in this tutorial can help you get started, as well as our \",/*#__PURE__*/e(o,{href:\"https://github.com/okareo-ai/okareo-cookbook/\",motionChild:!0,nodeId:\"wi5ius45d\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(a.a,{children:\"cookbook\"})}),\" project which contains many working examples.\"]})]});export const richText4=/*#__PURE__*/t(i.Fragment,{children:[/*#__PURE__*/e(\"p\",{children:\"Evaluating the output of our LLM components makes intuitive sense. But what to evaluate and when to do it is not as clear. From the world of functional testing, we know about the test pyramid that spans unit testing to integration testing and everything between. But model evaluation does not conform nicely to this structure. Data Scientists build models intentionally holding data back for validation and testing. So, what is the right approach to LLM evaluation?\"}),/*#__PURE__*/e(\"img\",{alt:\"Screenshot of evaluation results showing 7 key metrics including Consistency, Relevance, Fluency, Demo.Actions.Length, and Demo.Summary.JSON\",className:\"framer-image\",height:\"391\",src:\"https://framerusercontent.com/images/l37rIJZYGbltWknFLIU7feCbEU8.png\",srcSet:\"https://framerusercontent.com/images/l37rIJZYGbltWknFLIU7feCbEU8.png?scale-down-to=512 512w,https://framerusercontent.com/images/l37rIJZYGbltWknFLIU7feCbEU8.png?scale-down-to=1024 1024w,https://framerusercontent.com/images/l37rIJZYGbltWknFLIU7feCbEU8.png?scale-down-to=2048 2048w,https://framerusercontent.com/images/l37rIJZYGbltWknFLIU7feCbEU8.png 2584w\",style:{aspectRatio:\"2584 / 782\"},width:\"1292\"}),/*#__PURE__*/t(\"h2\",{children:[\"What are evaluations\",/*#__PURE__*/e(o,{href:\"https://blog.okareo.com/blog/posts/intro_checks#what-are-evaluations\",motionChild:!0,nodeId:\"wi5ius45d\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(a.a,{children:\"\u200B\"})})]}),/*#__PURE__*/e(\"p\",{children:\"As developers working with LLM endpoints, we need to bridge our best practices for external API integration testing with learnings from Data Science where behavioral analysis during development is key. But before we jump in, let's agree on some common concepts.\"}),/*#__PURE__*/t(\"ol\",{style:{\"--framer-font-size\":\"16px\",\"--framer-text-alignment\":\"start\",\"--framer-text-color\":\"rgb(28, 30, 33)\",\"--framer-text-stroke-width\":\"0px\",\"--framer-text-transform\":\"none\"},children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Behavior-based development. The key to effective use of LLMs is to embrace the lack of determinism and think in terms of behaviors.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:['Transactional interaction. Most LLMs are called through standard API endpoints + payloads. This may seem counter-intuitive to call out. \"',/*#__PURE__*/e(\"em\",{children:\"Of course they are!\"}),'\" you say. Well true. It also allows us to use our well-understood API delivery habits. Remember, AI != LLM. AI can be accessed in a variety of fashions. For example a classifier may augment records in a datastore with meta-data, clustering labels, workflow steps, and more. But nearly all LLMs are called via API.']})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Evaluation is more than testing. There seems to be a growing use of the term evaluation to mean test. This is far from correct. Because of behavior and transactional-ity, LLM evaluation is critical to development, validation, testing, and production feedback.\"})})]}),/*#__PURE__*/e(\"p\",{children:\"Evaluation should be thought of as a collection of distinict assertions on the output of a model. To drive evaluation, you must have a representative range of inputs from which to determine the distribution of assertion results. Two evaluation runs of an identical model and system should not be assumed identical. If this is the desire, non-stochastic approaches should be considered instead.\"}),/*#__PURE__*/t(\"h2\",{children:[\"Parts of Evaluation\",/*#__PURE__*/e(o,{href:\"https://blog.okareo.com/blog/posts/intro_checks#parts-of-evaluation\",motionChild:!0,nodeId:\"wi5ius45d\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(a.a,{children:\"\u200B\"})})]}),/*#__PURE__*/e(\"p\",{children:'There are five phases of evaluation for LLM output. Each phase has a unique set of needs. Although all of it is called \"evaluation\" there are useful differences for each phase. The phases of evaluation are during:'}),/*#__PURE__*/t(\"ul\",{style:{\"--framer-font-size\":\"16px\",\"--framer-text-alignment\":\"start\",\"--framer-text-color\":\"rgb(28, 30, 33)\",\"--framer-text-stroke-width\":\"0px\",\"--framer-text-transform\":\"none\"},children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Prompt Development\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Build/Unit Validation\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Integration/Deployment\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Feedback (Production Analysis)\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Model Finetuning\"})})]}),/*#__PURE__*/t(\"p\",{children:[\"At Okareo we have deconstructed evaluation into smaller units that we call \",/*#__PURE__*/e(o,{href:\"https://blog.okareo.com/docs/getting-started/concepts/checks\",motionChild:!0,nodeId:\"wi5ius45d\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(a.a,{children:\"Checks\"})}),\". This allows us to address the needs of each phase dynamically, drawing from a collection of checks that may overlap but are always relevant for each phase.\"]}),/*#__PURE__*/t(\"h3\",{children:[\"Scenarios\",/*#__PURE__*/e(o,{href:\"https://blog.okareo.com/blog/posts/intro_checks#scenarios\",motionChild:!0,nodeId:\"wi5ius45d\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(a.a,{children:\"\u200B\"})})]}),/*#__PURE__*/e(\"p\",{children:\"At this point, it is important to spend a moment on scenarios. Scenarios are inentional collections of inputs and ouputs along a specific variable dimension that is used to shine a light on the behavior of a model. If a LLM prompt seeks to generate code, then a scenario set should constitute dozens, hundreds or thousands of input/output pairs that can be used to explore (e.g. evaluate) dependency constraints set by the prompt. It is reasonable to assume that for each set of behaviors expected from an LLM interaction, there should be a corollary set of scenarios used to stretch and shine a light on the behavior.\"}),/*#__PURE__*/t(\"h3\",{children:[\"Checks\",/*#__PURE__*/e(o,{href:\"https://blog.okareo.com/blog/posts/intro_checks#checks\",motionChild:!0,nodeId:\"wi5ius45d\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(a.a,{children:\"\u200B\"})})]}),/*#__PURE__*/t(\"p\",{children:[\"Fundamentally, \",/*#__PURE__*/e(o,{href:\"https://blog.okareo.com/docs/getting-started/concepts/checks\",motionChild:!0,nodeId:\"wi5ius45d\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(a.a,{children:\"checks\"})}),' are metrics. The output of a check is either binary (pass/fail) or a numeric (1-n). Each check included in an evaluation is passed a scenario row and can use any combination of prompt, model input, model output, and model expected result to determine the appropriate score for the row. Binary checks are used to assert passing or failing. Numeric checks provide distributions. For example, a summary that must be less than 256 characters should be pass/fail. But a summary that should be \"short\" may just returns the length so we can review the distribution of \"short\" answers. Maybe the prompt needs to be \"hyper short\" or \"no more than 50 words\".']}),/*#__PURE__*/e(\"p\",{children:\"Okareo Checks come in three flavors:\"}),/*#__PURE__*/t(\"ul\",{style:{\"--framer-font-size\":\"16px\",\"--framer-text-alignment\":\"start\",\"--framer-text-color\":\"rgb(28, 30, 33)\",\"--framer-text-stroke-width\":\"0px\",\"--framer-text-transform\":\"none\"},children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Native Okareo Checks\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Custom Deterministic\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Custom Judge/Jury\"})})]}),/*#__PURE__*/t(\"p\",{children:[\"Okareo comes with access to a growing list of \",/*#__PURE__*/e(\"strong\",{children:\"native\"}),\" checks. Checks in this category are typically very generic and come largely from academic research. The list currently includes Levenshtein Distance, Bleue Score, Fluency, Consistency, and many more.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Custom deterministic\"}),\" checks are written in natural language, python or typescript (ts coming soon). Like all checks, these have access to the full range of context on the row they are evaluating. They are intended to be very fast and allow you to cycle through large numbers of scenarios very quickly. Deterministic checks will often make heavy use of language semantics and regular expressions to assert specific expectations on model output. For example, a check could pass/fail if the ouput is in markdown or has source references.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Judge/Jury\"}),\" checks are very powerful but need to be used carefully. Checks from this category use a model, often an LLM, to analyze output of another model. This process can be fast. But it isn't hard to unintentionally create a very expensive (time and money) judge. For example, the decomposition pattern where each sentence of an output is decomposed into a unique jury judgement and the judge re-assembles the completed list for a final score may mean 100s of calls per scenario. There are good reasons to do this. But for now, it won't be fast. So, use the judge/jury pattern thoughtfully.\"]}),/*#__PURE__*/t(\"h3\",{children:[\"Phases of Evaluation\",/*#__PURE__*/e(o,{href:\"https://blog.okareo.com/blog/posts/intro_checks#phases-of-evaluation\",motionChild:!0,nodeId:\"wi5ius45d\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(a.a,{children:\"\u200B\"})})]}),/*#__PURE__*/e(\"p\",{children:\"Checks are a powerful tool for development, not just evaluation. The world of just build it and figure out testing later misses the power of evaluation during development.\"}),/*#__PURE__*/e(\"p\",{children:\"If I was building a user interface, I would make a set of changes then look to see the unique change in the UI. If I was building a service, I would make changes and then run my unit test to see the unique change in the result. So, why wouldn't I do the same with prompts? Enter manual validation with a playground. Most playgrounds enable me to poke a model with a prompt and see the result. But where my UI or API change is discrete and deterministic. My prompt change is not. So, as useful as manual validation is for rapid feedback, it is ultimately a poor and slow approach to multi-behavior evaluation. The solution is chunked up evaluation where each chunk is phase appropriate. After all, I did not become more patient or more interested in QA just because I sterted using LLMs. But the range of opportunities to stub my toe, get barked at by the business, or slow my co-workers with regressions and unexpected behaviors have all increased if I don't adapt.\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Prompt Development: Deterministic Distributions\"}),' During development, use deterministic Checks that provide scores. This provides you with an understanding of response disributions. At this point \"failure\" is not important - possibly not even defined yet. You want to see the possible behaviors your model is returning and find the peaks, valleys, and steps your prompt is driving.']}),/*#__PURE__*/t(\"ul\",{style:{\"--framer-font-size\":\"16px\",\"--framer-text-alignment\":\"start\",\"--framer-text-color\":\"rgb(28, 30, 33)\",\"--framer-text-stroke-width\":\"0px\",\"--framer-text-transform\":\"none\"},children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Establish 3-6 deterministic checks that you run during development\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Each check should provide a metric that you can use to see the distribution of results\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"If pass/fail is appropriate, use it. But the goal of this stage is to understand results\"})}),/*#__PURE__*/t(\"li\",{\"data-preset-tag\":\"p\",children:[/*#__PURE__*/e(\"p\",{children:\"Create enough metrics that you can use them to define the assertions for pass/fail regression\"}),/*#__PURE__*/e(\"img\",{alt:\"\",className:\"framer-image\",height:\"192\",src:\"https://framerusercontent.com/images/QQqQigAIbuKH6OVyNtVpDiVzkE0.png\",srcSet:\"https://framerusercontent.com/images/QQqQigAIbuKH6OVyNtVpDiVzkE0.png?scale-down-to=512 512w,https://framerusercontent.com/images/QQqQigAIbuKH6OVyNtVpDiVzkE0.png?scale-down-to=1024 1024w,https://framerusercontent.com/images/QQqQigAIbuKH6OVyNtVpDiVzkE0.png 1124w\",style:{aspectRatio:\"1124 / 384\"},width:\"562\"})]})]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Build/Unit Validation: Deterministic Pass/Fail\"}),\" Unit testing is the gatekeeper to the check-in and PR. These should be deterministic checks that pass or fail. Although you may choose to maintain some number of distributional checks, the goal of this phase is rapid clarity on readiness. These are the checks you should use in CI whenever a prompt changes and prior to any merge. Pass/Fail checks should have paired with distributional checks that are either run at the same time or can be run if the model fails. The Okareo SDKSs allow you to make this determination lazily as part of the evaluation flow.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Integration/Deployment: Judge/Jury\"}),\" Now comes the domain of the judge and jury. Before deployment during integration validation, you should take the few additional minutes to validate that the system behaves as expected. LLMs are really good at this. The same non-determinism that makes it hard to pin down precise results is able to interpret interaction and provide visibility into expected vs actual results. Like checks oriented towards pass/fail and distributions, the judge/jury checks should use a combination of both approaches. However in this case, don't divide them. Use both at the same time. Since the process is relatively slow and expensive by definition, don't require it to be run twice. Do the full analysis and then determine readiness for deployment. Okareo has a variety or reporters for use in CI to set thresholds and determine readiness for deploy. Using checks at this stage allows you to lean into Continuous Deployment for LLM prompts.\"]}),/*#__PURE__*/e(\"p\",{children:'Aren\\'t LLMs poor at scoring? True, for scoring but not assessment. There are several techniques for addressing this difference. Judges can be limited to subsets of the total content and provide pass/fail judgement for each. For example, assess each sentence. The aggregate of the assessments can become the score. On the other hand a clear rubrik for scoring gives the stochastic system a way to categorize and find best fit to a score category. Simply saying \"... and score from 1-100\" is not going to be sufficient. Be specific about scores. Consider using a smaller scale like 1-5 but be specific about each position in the score domain.'}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Feedback (Production Analysis): Deterministic + Scenarios\"}),' The production phase is one of the most varied from existing patterns. Although techniques like \"canary\" or \"A/B\" validation are fine for deployment stability and health, they have little to do with the proper functioning of the prompt or model. It is in production when all the unusual interactions start. Existing tools for observability and error reporting will indicate systemic stability. But behavior is different. The simplified thumbs-up, thumbs-down concept is useful but has problems. We will dedicate an entire blog to that in the future. The key to this phase is collecting feedback from users and from existing checks to alert behaviors that are well outside the expected norms. This phase also requires reputational and jail breaking checks to ensure that the model behaves within guidelines - not just operates. When interactions are outside of expected norms based on a subset of checks from above, it is useful to gather the input/output pairs and generate a synthetic scenario set that you can use to improve the prompt, data pipeline or model itself. This is one of the many entry points into fine tuning. We strongly suggest that interactions from this stage should lead to additional guardrail checks and expansions/modifications of your regression scenarios/checks. After all, the best regression suites are best on real-world interaction.']}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Model Fine Tuning: Expanded Scenarios\"}),\" This stage is listed last but really should be happening throughout the development lifecycle. Here the evaluations should focus on the gap between the original model and the fine-tuned model. Were enough fine-tuning examples provided? The deterministic and judgment checks should be used as part of the fine-tuning evaluation process to determine success. When complete, like Feedback, elements from the new scenarios and checks should be incorporated into the Build and Integration phases to protect future drift and regressions.\"]}),/*#__PURE__*/t(\"h2\",{children:[\"All Together Now\",/*#__PURE__*/e(o,{href:\"https://blog.okareo.com/blog/posts/intro_checks#all-together-now\",motionChild:!0,nodeId:\"wi5ius45d\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(a.a,{children:\"\u200B\"})})]}),/*#__PURE__*/e(\"p\",{children:\"Okareo has a rich set of capabilities to help you evaluate complex model output. In this case, we looked at checks, an element of evaluation to drive metrics. Checks should be used at each phase of development. The most efficient and teams that we see use these methods throughout their development lifecycle. They don't leave it to the end as an afterthought as has been the historical norm with functional quality.\"}),/*#__PURE__*/e(\"p\",{children:\"Assembling checks into useful evaluations that are time and cost efficient requires some thought. By thinking in terms of LLM development phase, we are able to limit time spent and maximize the signals of readiness.\"})]});export const richText5=/*#__PURE__*/t(i.Fragment,{children:[/*#__PURE__*/t(\"p\",{children:[\"Summarization has become a mainstay of LLM usage. In this blog we'll use a meeting transcript summarization example. You can get to a solution in an AI playground fairly quickly. Start adding the inevitable need to pull out meta-data, active speakers, decisions and you rapidly start down the path of language-coding with LLMs. This gets challenging, quickly. Let's explore a few techniques to improve development and regression of transcript summarization prompts.\",/*#__PURE__*/e(\"br\",{}),/*#__PURE__*/e(\"br\",{className:\"trailing-break\"})]}),/*#__PURE__*/e(\"img\",{alt:\"\",className:\"framer-image\",height:\"453\",src:\"https://framerusercontent.com/images/DWJYGfjTAjFrS397C2xa7Q07TQ.png\",srcSet:\"https://framerusercontent.com/images/DWJYGfjTAjFrS397C2xa7Q07TQ.png?scale-down-to=512 512w,https://framerusercontent.com/images/DWJYGfjTAjFrS397C2xa7Q07TQ.png?scale-down-to=1024 1024w,https://framerusercontent.com/images/DWJYGfjTAjFrS397C2xa7Q07TQ.png?scale-down-to=2048 2048w,https://framerusercontent.com/images/DWJYGfjTAjFrS397C2xa7Q07TQ.png 2590w\",style:{aspectRatio:\"2590 / 906\"},width:\"1295\"}),/*#__PURE__*/e(\"p\",{children:\"Transcripts brings a unique set of challenges. Where written documents have an author and have structure, the same is not true of transcripts.  Most transcripts have multiple speakers, often have side-chatter, and have few if any clear topical sign-posts. This is not to say that document summarization as part of a RAG doesn't come with challenges. But we will leave RAG summarization to a future blog. \"}),/*#__PURE__*/e(\"h2\",{children:\"Transcript Summarization\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/t(\"code\",{children:['INFO: In this blog we will invent the evaluation loop for a project summarizing city council meetings. The source content, we will use is from the public domain \"meeting bank\" of city council transcripts hosted by ',/*#__PURE__*/e(o,{href:\"https://zenodo.org/records/7989108\",motionChild:!0,nodeId:\"wi5ius45d\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(a.a,{children:\"Zenodo\"})}),\".\"]})}),/*#__PURE__*/e(\"p\",{children:\"Most transcript summarization pipelines seek to:\"}),/*#__PURE__*/t(\"ul\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Create structure from the transcript (Tasks, Agenda, Decisions, etc)\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Identify key events and timing\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Identify the participants\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Produce per-topic summaries\"})})]}),/*#__PURE__*/e(\"p\",{children:\"Using a small excerpt from a Seattle City Council meeting. We can easily see multiple people speaking, decisions being made and a naturally wandering structure.  Thankfully LLMs do a reasonably good job of recognizing what was intended vs said.\"}),/*#__PURE__*/e(\"blockquote\",{children:/*#__PURE__*/e(\"p\",{children:\"\u2026report with Councilman Mr. Brian Johnson. So want and what is in favor. And councilmembers Herbold and Burgess opposed. Councilmember O'Brien. Thank you. So, I mean, just give a quick overview of the legislation before us. We've had a lot of discussion\u2026   \"})}),/*#__PURE__*/e(\"p\",{children:\"Let's construct a prompt in three parts: \"}),/*#__PURE__*/t(\"ul\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"the persona we want used\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"the directive we want the LLM to follow\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"the structure of the response we are looking for\"})})]}),/*#__PURE__*/e(\"h6\",{children:\"Example Prompt\"}),/*#__PURE__*/e(\"div\",{className:\"framer-text-module\",style:{height:\"auto\",width:\"100%\"},children:/*#__PURE__*/e(n,{componentIdentifier:\"module:pVk4QsoHxASnVtUBp6jr/TbhpORLndv1iOkZzyo83/CodeBlock.js:default\",children:t=>/*#__PURE__*/e(s,{...t,code:'`You are a City Manager with significant AI/LLM skills.\\nYou are tasked with summarizing the key points from a\\nmeeting and responding in a structured manner. You have\\na strong understanding of the meeting\\'s context and the\\nattendees. You also follow rules very closely.\\n\\nProvide a summary of the meeting in under 50 words.\\nYour response MUST be in the following JSON format.\\nContent you add should not have special characters \\nor line breaks.\\n\\n{\\n  \"actions\": LIST_OF_ACTION_ITEMS_FROM_THE_MEETING,\\n  \"short_summary\": SUMMARY_OF_MEETING_IN_UNDER_50_WORDS,\\n  \"attendee_list\": LIST_OF_ATTENDEES\\n}`',language:\"JSX\"})})}),/*#__PURE__*/e(\"h2\",{children:\"Evaluation\"}),/*#__PURE__*/e(\"p\",{children:\"To make sure that the LLM works consistently, we need to evaluate the outcome across a few dimensions:\"}),/*#__PURE__*/t(\"ol\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Setup metrics and assertions on the output\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Expand the inputs to avoid having a built a prompt that only works for one input\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Add an evaluation flow that we can run locally or add to CI\"})})]}),/*#__PURE__*/e(\"p\",{children:\"Each of the items above will get it's own deep dive.  For this article, we will discuss how each of these is assmebled into an evaluation flow.\"}),/*#__PURE__*/e(\"h3\",{children:\"Metrics\"}),/*#__PURE__*/t(\"p\",{children:[\"At Okareo, we call each measurement a \",/*#__PURE__*/e(o,{href:\"https://okareo.com/docs/getting-started/concepts/checks\",motionChild:!0,nodeId:\"wi5ius45d\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(a.a,{children:\"check\"})}),\". For model unit testing locally or in CI, we suggest  checks that are narrowly defined and can be executed on each trio of input, expected, and actual result. After establishing these rapid checks, you will have a baseline for building and improving your use of the LLM.\"]}),/*#__PURE__*/e(\"p\",{children:\"To make sure our summarization output is working, let's establish the following checks per scenario: \"}),/*#__PURE__*/e(\"div\",{className:\"framer-text-module\",style:{height:\"auto\",width:\"100%\"},children:/*#__PURE__*/e(n,{componentIdentifier:\"module:pVk4QsoHxASnVtUBp6jr/TbhpORLndv1iOkZzyo83/CodeBlock.js:default\",children:t=>/*#__PURE__*/e(s,{...t,code:\"/*\\n1. Library - summary_relevance : Score the relevance to the original content.\\n2. Library - summary_consistency : Score the writing consistency of the short_summary\\n3. Custom - summary_relevance : Pass if output is in JSON format with attendee_list, actions, and short_summary properties.\\n4. Custom - attendees.length: Count the number of attendees from the JSON response.\\n5. Custom - actions.length : Count the number of action items from the JSON response.\\n6. Custom - summary.length : Provide the length of the summary so we can see distribution.\\n7. Custom - summary.Under256 :  Pass if the short_summary is under 256 characters.\\n*/\",language:\"JSX\"})})}),/*#__PURE__*/e(\"p\",{children:'The first two checks are standard LLM checks that you can use from the Okareo checks library. The remainder are unique to this model and our expectations. You can create any of the above checks in the Okareo app by clicking \"Create Check\" or through the SDK:         '}),/*#__PURE__*/e(\"h5\",{children:\"Typescript:\"}),/*#__PURE__*/e(\"div\",{className:\"framer-text-module\",style:{height:\"auto\",width:\"100%\"},children:/*#__PURE__*/e(n,{componentIdentifier:\"module:pVk4QsoHxASnVtUBp6jr/TbhpORLndv1iOkZzyo83/CodeBlock.js:default\",children:t=>/*#__PURE__*/e(s,{...t,code:'const isJSONcheck = await okareo.generatecheck({\\n  projectid,\\n  name: \"summary.isJSON\",\\n  description: \"Pass if ouput is in JSON format with attendeelist, actions, and shortsummary properties.\",\\n  outputdatatype: \"bool\",\\n  //int and float are also allowed\\n  requiresscenarioinput:false,\\n  requiresscenario_result:true,\\n});\\nreturn await okareo.upload_check({\\n  project_id,\\n  ...isJSON_check\\n} as UploadEvaluatorProps);',language:\"TypeScript\"})})}),/*#__PURE__*/e(\"h5\",{children:\"Python:\"}),/*#__PURE__*/e(\"div\",{className:\"framer-text-module\",style:{height:\"auto\",width:\"100%\"},children:/*#__PURE__*/e(n,{componentIdentifier:\"module:pVk4QsoHxASnVtUBp6jr/TbhpORLndv1iOkZzyo83/CodeBlock.js:default\",children:t=>/*#__PURE__*/e(s,{...t,code:'import tempfile from okareo_api_client.models.evaluator_spec_request\\nimport EvaluatorSpecRequest description = \"\"\" Pass if ouput is in JSON format with attendee_list, actions, and short_summary properties. \"\"\" \\n\\ngenerate_request = EvaluatorSpecRequest(\\n  description=description,\\n  requires_scenario_input=False,\\n  requires_scenario_result=False,\\n  output_data_type=\"bool\"\\n)\\ngenerated_test = okareo.generate_check(generate_request).generated_code\\n\\ncheck_name = \"summary.isJSON\"\\ntemp_dir = tempfile.gettempdir()\\nfile_path = os.path.join(temp_dir, f\"{check_name}.py\")\\nwith open(file_path, \"w+\") as file:\\n  file.write(generated_test) \\n\\nhas_no_nl_check = okareo.upload_check(\\n  name=check_name,\\n  file_path=file_path,\\n  requires_scenario_input=False,\\n  requires_scenario_result=False\\n)',language:\"Python\"})})}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"br\",{className:\"trailing-break\"})}),/*#__PURE__*/e(\"h3\",{children:\"Scenarios\"}),/*#__PURE__*/e(\"p\",{children:\"There are numerous ways to setup scenarios.  In this case since we have access to meeting transcripts in the public domain, we will simply upload a range of meetings for use in our evaluation.\"}),/*#__PURE__*/e(\"h5\",{children:\"Typescript:\"}),/*#__PURE__*/e(\"div\",{className:\"framer-text-module\",style:{height:\"auto\",width:\"100%\"},children:/*#__PURE__*/e(n,{componentIdentifier:\"module:pVk4QsoHxASnVtUBp6jr/TbhpORLndv1iOkZzyo83/CodeBlock.js:default\",children:t=>/*#__PURE__*/e(s,{...t,code:'import { Okareo } from \"okareo-ts-sdk\";\\nconst okareo = new Okareo({api_key:OKAREO_API_KEY});\\nconst data: any = await okareo.upload_scenario_set({\\n  file_path: \"example_data/seed_data.jsonl\",\\n  scenario_name: \"Uploaded Scenario Set\",\\n  project_id: project_id\\n}); ',language:\"JSX\"})})}),/*#__PURE__*/e(\"h5\",{children:\"Python:\"}),/*#__PURE__*/e(\"div\",{className:\"framer-text-module\",style:{height:\"auto\",width:\"100%\"},children:/*#__PURE__*/e(n,{componentIdentifier:\"module:pVk4QsoHxASnVtUBp6jr/TbhpORLndv1iOkZzyo83/CodeBlock.js:default\",children:t=>/*#__PURE__*/e(s,{...t,code:'from okareo import Okareo\\nokareo = Okareo(\"YOUR API TOKEN\")\\nokareo.upload_scenario_set(\\n  file_path=\\'./evaluation_dataset.jsonl\\', \\n  scenario_name=\"Retrieval Facts Scenario\"\\n)       ',language:\"Python\"})})}),/*#__PURE__*/e(\"p\",{children:\"Using real-world examples is a great way to start.  However, we strongly suggest generating synthetic scenarios with intentional variation in complexity to determine where the model/prompt will break.  In this example, a few scenarios dedicated to number of attendees, number of actions items and meeting complexity would significantly improve general coverage. Okareo can generate these scenarios from the same uploaded examples identified here or from other data sets like real-world usage, PM requirements, etc.\"}),/*#__PURE__*/e(\"h3\",{children:\"Evaluation\"}),/*#__PURE__*/t(\"p\",{children:[\"Time to bring it all together.  When developing a new model, a editing a prompt, it is useful to have a means to verify that is working locally. There are a number of ways to do this with Okareo.  For this article, we are going to use one of the more direct methods called flows.  As part of the Okareo CLI, you can configure flows which are named combinations of models and scenarios.  The flow can then be run directly from the command line using \",/*#__PURE__*/e(\"code\",{children:\"okareo run -f FLOW_NAME\"}),\".  You can also use the same flow definition in GitHub Actions, CircleCI, GitLab or any other bash enabled CI provider.  You can use an Okareo reporter to show the results in your shell or review the results in the app through the \",/*#__PURE__*/e(\"code\",{children:\"app_link\"})]}),/*#__PURE__*/e(\"img\",{alt:\"\",className:\"framer-image\",height:\"573\",src:\"https://framerusercontent.com/images/BX4dvdS53xtKQHPoicHGqCiKxDM.png\",srcSet:\"https://framerusercontent.com/images/BX4dvdS53xtKQHPoicHGqCiKxDM.png?scale-down-to=512 512w,https://framerusercontent.com/images/BX4dvdS53xtKQHPoicHGqCiKxDM.png?scale-down-to=1024 1024w,https://framerusercontent.com/images/BX4dvdS53xtKQHPoicHGqCiKxDM.png 1211w\",style:{aspectRatio:\"1211 / 1147\"},width:\"605\"}),/*#__PURE__*/e(\"h2\",{children:\"All Together Now\"}),/*#__PURE__*/e(\"p\",{children:\"Okareo has a rich set of capabilities to help you evaluate complex model output.  In this case, we looked at evaluation of summarization and meta data from an in-person meeting transcript. Every LLM and model evaluation creates unique challenges specific to the desired behavior. As you are thinking about your use case, start with the desired outcome that you want from the model/prompt.  Describe that outcome and build checks. Then, assemble example scenarios that define the edges of what should and should not work.  When all that is done, add the flow to CI and you can be confident that future modifications to the prompt or the application using it will be protected by your evaluation flow.\"}),/*#__PURE__*/t(\"p\",{children:[\"Building AI into applications is a wild ride. We get it. Let us know what your experiences are with evaluation. And don't hesitate to ask questions. We are here to help: support@okareo.com. In the meantime, Okareo is \",/*#__PURE__*/e(o,{href:\"http://app.okareo.com/\",motionChild:!0,nodeId:\"wi5ius45d\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(a.a,{children:\"free to try\"})}),\". So, don't be shy. Sign up and give Okareo a spin.\"]}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"br\",{className:\"trailing-break\"})})]});export const richText6=/*#__PURE__*/t(i.Fragment,{children:[/*#__PURE__*/e(\"h2\",{children:\"TL;DR\"}),/*#__PURE__*/t(\"ul\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Synthetic data is a key ingredient in building apps that use LLMs.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Better data coverage drives robustness and predictability of an LLM, ensuring it effectively handles your unique task and use case.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Synthetic data touches all phases of development: Describing expected model behavior, Prototyping and Experimenting, Testing and Evaluation, Model Fine-Tuning.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Pre-Production vs. Production use has different roles for synthetic data.\"})})]}),/*#__PURE__*/e(\"h2\",{children:\"Why should you care?\"}),/*#__PURE__*/e(\"p\",{children:\"If you are building an AI app that calls out to an LLM, this will make your life much easier. Once you start mixing LLMs into your app stack, a few basic questions arise:\"}),/*#__PURE__*/t(\"ul\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"How do you select the best LLM to perform your unique task?\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Will it work for all the scenarios that you expect in your app?\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"How stable and reliable will the experience be for your users and use cases?\"})})]}),/*#__PURE__*/e(\"p\",{children:\"There are a ton of benchmarks and leaderboards out there (probably too many) comparing model performance on some predefined, abstract metrics, but how do you know if it will work for you? Change the metric to the one your app cares about, the underlying data, and all of a sudden you are building your own benchmark. Hold on, you need reliability and robustness in your LLM calls, but you don\u2019t have that kind of time. This is where synthetic data comes in.\"}),/*#__PURE__*/e(\"h2\",{children:\"Establishing Baselines\"}),/*#__PURE__*/e(\"p\",{children:\"The key idea to begin with is that you need to choose a set of metrics that capture your unique requirements. Once those metrics are clear, you want to measure baseline performance on them. This contrasts with traditional software engineering, where the main \u2018metric\u2019 was passing expected tests. Establishing baselines is important because LLM output is varied, non-deterministic, and there is never 100% certainty that it is correct. Having baselines and constantly evaluating your metric progress is the only way your app will ever see the light of production. Just like tests in traditional software development, baselines enable quick iterations and evolution of all the moving parts - context fed into the model, prompts, model version, and so on.\"}),/*#__PURE__*/e(\"p\",{children:\"Now, all of these metric baselines are only as good as the range of data they were calculated on. Using a few ad-hoc input samples doesn't represent reality or build confidence. Getting representative data samples, in particular for new AI-powered experiences, is nearly impossible. Synthetic data addresses this problem and is an important tool in the AI engineering toolkit. Synthetic scenarios should be organized by feature and expected model behaviors. Each scenario is paired with baseline metrics. The number of these behavior scenarios and data points within each scenario maps to coverage. As AI feature development matures, expectations and error conditions become clearer and this coverage naturally grows along with confidence in application readiness.\"}),/*#__PURE__*/e(\"h2\",{children:\"Synthetic Data Loop\"}),/*#__PURE__*/e(\"p\",{children:\"Establishing repeatable evaluations in dev pipelines (CI/CD) is a high ROI activity. It is also becoming a standard in LLM app development. Evaluations require sufficient data and clear signals to guide improvement decisions based on metrics movement. Synthetic data expands visibility into model behaviors and, coupled with evaluations, makes these decisions more data-driven.\"}),/*#__PURE__*/e(\"p\",{children:\"The diagram shows how synthetic data applies to various LLM app development stages. Not all AI teams leverage all stages. As teams start with ad-hoc testing and manual evaluations, there are natural opportunities to drive automation. Synthetic data connects and accelerates each stage, maximizing leverage and enabling the complete development loop.\"}),/*#__PURE__*/e(\"img\",{alt:\"\",className:\"framer-image\",height:\"381\",src:\"https://framerusercontent.com/images/sVb18yhFptCbPbITR0QUieg8TqI.png\",srcSet:\"https://framerusercontent.com/images/sVb18yhFptCbPbITR0QUieg8TqI.png?scale-down-to=512 512w,https://framerusercontent.com/images/sVb18yhFptCbPbITR0QUieg8TqI.png 1024w\",style:{aspectRatio:\"1024 / 763\"},width:\"512\"}),/*#__PURE__*/t(\"ol\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Describe Model Behaviors\"}),\" - The loop starts with you identifying the key behaviors your app cares about. Synthetic scenarios are generated for each behavior in iterations until there are sufficient samples and quality. As new features are introduced or gaps identified, this step would need to be repeated.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Iterate & Evaluate\"}),\" - With behaviors and baselines defined, there is a solid foundation for fast iterations on prompts, model context, retrieval parameters. Evaluations here can be automatic, based on user feedback, or human evaluation. The target outcome is that all the quick improvements (e.g., retrieval model, prompt, LLM parameters) are identified and applied.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Distill Failures\"}),\" - Stubborn or intermittent failures will remain from Iterate & Evaluate and are ideal input for this stage. The synthetic generation process is used to better map the triggering conditions of the stubborn failures and to triage their impact. This is accomplished by categorizing failure types, synthetically generating adjacent input variations, and feeding them through the same evaluation criteria. This gives visibility to the persistent failure modes and expands the expected model behavior scenarios.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Fine-Tune Behavior\"}),\" - At a point when cheaper solutions (i.e., prompt tuning) have been exhausted, it is time to fine-tune the underlying model. Synthetic data is leveraged to generate fine-tuning instruction sets with positive and negative samples based on the prior stage of distilled failures.\"]})})]}),/*#__PURE__*/e(\"h2\",{children:\"Let\u2019s Take an Example\"}),/*#__PURE__*/e(\"p\",{children:\"Let\u2019s take an example of building an app while maintaining focus on how to leverage synthetic data. Here, we'll outline how the above stages map at a high level and demonstrate how to implement them in a follow-up blog.\"}),/*#__PURE__*/e(\"p\",{children:\"Let\u2019s say we are building a collaboration app that is agent-based and includes a Meeting Scheduling Agent, Meeting Summarization Agent, and Personalized Doc Retrieval Agent. We are interested in building the user intent detection and routing module that will call out to a model to make decisions on what is the right agent to choose and when to get more information from the user if the intent is not clear. Let\u2019s step through each stage:\"}),/*#__PURE__*/t(\"ul\",{children:[/*#__PURE__*/t(\"li\",{\"data-preset-tag\":\"p\",children:[/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Describe Model Behaviors\"}),\" - You can foresee the following scenarios that capture model behaviors you care about:\"]}),/*#__PURE__*/t(\"ul\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Out of scope, off-topic requests with appropriate guidance to the user.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Robustness related scenarios on jailbreaks, prompt leaks, terse or cryptic input.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Ambiguous or borderline requests that require clarification with the user.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Highly specific scheduling or summarization requests that imply immediate handling by the relevant agent.\"})})]}),/*#__PURE__*/e(\"p\",{children:\"You can start with a few manually crafted examples or sample data from internal dogfooding and use synthetic generation to multiply and create variations within each category. You focus the generation on conversational, informal tone and layer in typical user input issues around sentence fragments, misspellings. You could also create scenario templates and vary key parameters for the behaviors you care about.\"}),/*#__PURE__*/e(\"p\",{children:\"It\u2019s important to think of this stage as a synthetic data pipeline with a data quality evaluation step that filters out low-quality data points. Once the pipeline is set up, you can introduce new generation sources into it, e.g., A/B testing, production.\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"br\",{className:\"trailing-break\"})})]}),/*#__PURE__*/t(\"li\",{\"data-preset-tag\":\"p\",children:[/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Iterate & Evaluate\"}),\" - Here it\u2019s important to identify key metrics that will both guide development quality and product decision-making. For our example app, this could be:\"]}),/*#__PURE__*/t(\"ul\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Standard classification metrics (e.g., Accuracy of selecting the right agent, F1, etc.)\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"The number of dialog turns to successful routing decision.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"User sentiment.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Dialog tone and brevity.\"})})]}),/*#__PURE__*/e(\"p\",{children:\"It\u2019s easy to see how these would be automated and become a standard part of your CI/CD process that runs through a set of behavior scenarios on a regular basis. This step needs to be fast, and automating it gains leverage across the whole system. With chosen metrics and behavior scenarios from the previous step, you can rapidly iterate on the prompt, model version, and other parameters. Baselines ensure you are making progress without fixing one part while breaking another.\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"br\",{className:\"trailing-break\"})})]}),/*#__PURE__*/t(\"li\",{\"data-preset-tag\":\"p\",children:[/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Distill Failures\"}),\" - Based on the evaluation results from the previous stage, you can concentrate your efforts on persistent failures that couldn't be resolved. If you encounter persistent failures with ambiguous user requests and robustness scenarios, you can use specialized synthetic generators to create more variations and examples within each behavior-scenario group. All generated scenarios still pass through evaluations from the previous step to determine if they produce new failure examples. At this stage, synthetic generation would probe the boundaries of particular scenarios and effectively provide automated \u2018red teaming\u2019 of your app.\"]}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"br\",{className:\"trailing-break\"})})]}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Fine-Tune Behavior\"}),\" - A critical aspect of getting this stage right is having high-quality instruction data to tune on. You could argue that all the previous stages have worked together to make this one much easier. Key behaviors have been identified, behavior gaps mapped with synthetic data samples, and data curated by passing through evaluations. What remains is converting this data into behavior-specific instruction sets.\"]})})]}),/*#__PURE__*/e(\"h2\",{children:\"Pre-Production vs. Production\"}),/*#__PURE__*/e(\"p\",{children:\"Interestingly, the same evaluations and metrics you set up to decide if your app is ready for production are also useful once the app is in production. There are no guarantees that the LLM outputs that were \u2018passing\u2019 before will continue to do so. Naturally, production will introduce new, unforeseen scenarios. Evaluations can be used offline against production data to identify new failure types and online to determine if the LLM output is of sufficient quality for use (e.g., displaying LLM-generated code to users).\"}),/*#__PURE__*/e(\"p\",{children:\"In combination with evaluations, synthetic data helps here as well. Coverage is expanded via synthetic generation of new scenarios to include new production failure modes. This will ensure that these expected behaviors are adequately covered in regular evaluations and CI/CD.\"})]});export const richText7=/*#__PURE__*/t(i.Fragment,{children:[/*#__PURE__*/e(\"p\",{children:\"If you are a developer building a large language model (LLM) application, 2023 was the year of the RAG (Retrieval Augmented Generation). 2024 seems to be the year of the Agents, with everything being labeled an \u2018agent\u2019 now. If by now you haven\u2019t heard of RAG, then we are definitely moving in different circles. \"}),/*#__PURE__*/e(\"p\",{children:\"In this case, I wanted to put down a few thoughts on agents and agent evaluations. LLM evaluations are still a hard problem, but they are even harder for complex agent based systems. In the larger developer community consensus is starting to build that evaluations are super important for successful LLM application deployments. More mature teams have implemented evaluations in their development process and the larger community is moving towards standardizing the practice, including as part of CI/CD. In parallel, as it typically happens, agent evaluations are raising the complexity by an order of magnitude.\"}),/*#__PURE__*/e(\"h2\",{children:\"What is an Agent anyway?\"}),/*#__PURE__*/e(\"p\",{children:\"Believe me, I\u2019d prefer not to spend time on this question, but unfortunately, I think it\u2019s necessary to be \u2018grounded\u2019 on this before going further. I only mention this because sometimes I hear the term used to refer to a system prompt with some instructions. So here are a few traits I think separate an agent based LLM system:\"}),/*#__PURE__*/t(\"ul\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Autonomy in interpreting, breaking down and achieving high level goals, with limited to no human guidance\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Independent ability to complete actions in the real world - spin up Kubernetes cluster, schedule a meeting, book airline tickets\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Facility for planning function, invoking tools, and observing environment changes as a result of an action\"})})]}),/*#__PURE__*/e(\"h2\",{children:\"Useful Abstractions for LLM Applications\"}),/*#__PURE__*/e(\"p\",{children:\"As a developer it strikes me how many parallels there are between programming abstractions and what is emerging as building patterns for LLM applications. LLM invocations are used to replace whole modules of an application and are composed together for a rapid development paradigm that is changing how software is built. Not to mention generating code to glue it all together. One can think of foundational LLM as part of your application stack, similar to MEAN (MongoDB, Express. js, Angular, Node. js)  or LAMP (Linux, Apache, MySQL, PHP) stacks. In this context agent pattern is a programming abstraction on top of LLM stack. Instead of trying to stuff all your application logic into a single, unwieldy prompt, you can route to agents with different responsibilities that can also coordinate on more complex tasks.\"}),/*#__PURE__*/t(\"p\",{children:[\"For developers this is not new, putting all your logic into a single prompt doesn\u2019t make sense for many reasons regardless of the context size. We are starting to see many such design patterns starting to emerge on top of LLM stack. I like how Andrew Ng breaks down the \",/*#__PURE__*/e(o,{href:\"https://www.deeplearning.ai/the-batch/how-agents-can-improve-llm-performance/\",motionChild:!0,nodeId:\"wi5ius45d\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(a.a,{children:\"patterns seen so far.\"})})]}),/*#__PURE__*/e(\"p\",{children:\"I see our customers already adopting variations of these patterns in production. Developers modeling agent structure as company hierarchy (worker and manager agents) or team roles (designer, developer, analyst agents) has parallels to typical software abstractions (director/manager/builder modules or classes in programming). Other software abstractions, like mechanical systems (engine, filter, safety system), could work just as well. The complex pattern of multi-agent collaboration isn\u2019t exactly new either. For example, message-passing agents or processes are a well-established programming abstraction in Erlang and other systems. \"}),/*#__PURE__*/e(\"p\",{children:\"Now if we consider each agent pattern and things that could go wrong, it can get messy very quickly\u2026 Applying software development paradigm to building with agents means you need to be able to fix your behavior baselines and have a solid foundation to iterate on. Evaluations are the new unit/functional/end-to-end tests. So how do we evaluate these complex agent patterns?\"}),/*#__PURE__*/e(\"p\",{children:\" \"}),/*#__PURE__*/e(\"img\",{alt:\"\",className:\"framer-image\",height:\"362\",src:\"https://framerusercontent.com/images/tag7UPjL2eUopTzjSCGVUTETuPQ.png\",srcSet:\"https://framerusercontent.com/images/tag7UPjL2eUopTzjSCGVUTETuPQ.png?scale-down-to=512 512w,https://framerusercontent.com/images/tag7UPjL2eUopTzjSCGVUTETuPQ.png?scale-down-to=1024 1024w,https://framerusercontent.com/images/tag7UPjL2eUopTzjSCGVUTETuPQ.png 1992w\",style:{aspectRatio:\"1992 / 725\"},width:\"996\"}),/*#__PURE__*/e(\"h2\",{children:\"Approaches for Agent Evaluations\"}),/*#__PURE__*/e(\"p\",{children:\"Many automatic evaluation (no human evaluators) frameworks to date have focused on single-turn generation, scoring model output based on a given LLM prompt and a reference result. With this approach it is easy to trap entry and exit conditions and substantially easier to reason about what led to erroneous generation. While effective for specific tasks like summarization and sentiment analysis, it is insufficient for evaluating multi-turn agents that are using tools, doing independent planning, and triggering actions with real world implications. Here, we discuss emerging approaches for the latter.\"}),/*#__PURE__*/t(\"ul\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"RAG-based Evals\"}),\" -  RAG is an LLM application building block, an architecture that fits just as well when building LLM agents. Often teams start with a predetermined RAG flow and then evolve it to a more of an agent based control loop as the application gets more complex. With this in mind some evaluation frameworks adopt this incremental approach of agent evaluation as a RAG system. For example, if the agent is focused primarily on information retrieval tasks and also has autonomy to select the right real time data tools to access flight status, IOT sensors, and such. Treating it as a RAG problem allows evaluating each stage along widely used RAG dimensions: context relevance, answer consistency with context, and answer relevance. The wrinkle is model tool selection and that could be treated as a classification problem, mapping user intent.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"LLM Based Dialog Evaluation\"}),\" - Dialog system evaluations have been pursued in academia well before the recent explosion of LLMs. These evaluations are usually performed with the multi-turn dialog history as context and evaluate the quality of the last  generated dialog turn. A recent improvement is using LLM as a Judge evaluator to produce \u2018unified\u2019 scores. The primary focus has been on the quality of the next turn generation in conversational or chatbot contexts, rather than on autonomous systems that perform planning, task breakdown, tool selection, and tool output interpretation. Consequently, the metrics are often linguistic and broad in nature, which is useful in academic settings. \"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"LLM as Simulated User\"}),\" - There are several variations of this implementation, but the main idea is to have an LLM role-play a user, interacting with the agent system under evaluation. The user definition can be as simple as a prompt with instructions on the user's role and a high-level goal to achieve, such as requesting a complex itinerary booking. More advanced implementations could include a dialog scenario generator that uses past dialogs for requested task variety, LLMs to generate user personalities and dialog details, and a simulation agent to drive conversations with these simulated users. Notably, most commercial evaluation frameworks simulate only the user side of the agent interaction and not the environment side (e.g. operating system shell).\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Adversarial Models\"}),\" - This is the 'most emerging' of all the emerging approaches on this list. It borrows from model based red-teaming techniques but applies broader than model security and safety. The idea is to sample potential test scenarios from adversarial LLM and run these through an agent being evaluated. Adversarial model could then further be fine-tuned on failed scenarios. This could apply to any expected agent behavior, outside of safety. The approach typically combines a search function with adversarial model sampling and maps successful and failed outcomes to particular task categories.\"]})})]}),/*#__PURE__*/e(\"h2\",{children:\"Metrics\"}),/*#__PURE__*/e(\"p\",{children:\"Looking at the above approaches, the big question is what metrics could be used to score the evaluation results. Indeed, this is the critical part to make the approaches at all useful. This is also where things get a bit more complicated\u2026 Metrics being used fall into these categories:\"}),/*#__PURE__*/t(\"ul\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"RAG/ Retrieval Metrics\"}),\" - As mentioned, for retrieval oriented Agents that are primarily used to find and synthesize information requested, this could be enough. Obviously it\u2019s limiting for many other agent use cases.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Dialog Evaluation Metrics\"}),\" - In the context of building LLM applications, these could be used as guardrail metrics, as a proxy to overall system health. Other than that, it\u2019s not directly actionable with respect to agent evaluation on particular tasks. For example, getting a Coherence score of 3.5 does not tell you if the agent accomplished a task, it only speaks to the quality of the dialog being generated.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Agent Trajectory Decomposition\"}),\" - Agent trajectory, or the steps agent takes, tools it uses, is another aspect leveraged in agent evaluations. This is more of a white-box scoring method and relies on evaluation sets with predefined tasks and success criteria. In some cases this is taken even further with agent \u2018progress rate metrics\u2019 towards completion, versus simple pass/fail for a given task. Progress rate metrics primarily rely on subjective breakdown of tasks into subtasks that could be measured.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Use Case Specific Metric\"}),\" - On the other end of the spectrum there are metrics that capture the overall LLM application goal or use case. For example, this could be Issue Resolution Rate for a Customer Service Agent. Being able to identify and measure metrics like these is ideal as it aligns to the key application outcome. More detailed metrics, could ladder up to this top level outcome metric. In practice, constructing metrics like this is difficult. In some cases LLM as a Judge method could be used to approximate this.\"]})})]}),/*#__PURE__*/e(\"h3\",{children:\"Metric Issues to Consider:\"}),/*#__PURE__*/t(\"ul\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"For an average 1-2 pizza development team it could be just too much extra effort to manually create enough evaluation data sets for each agent, including success criteria by task.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Having to define agent task and trajectory success criteria ahead of time assumes that the task is known in advance and there is a single successful path to that outcome. This excludes evaluation on production scenarios not seen before. It also ignores that there could be multiple successful paths to the same outcome. For instance, if the task is to add account values stored in a given file it could be done equally well via a python script or shell command. The output could also be captured in different formats. \"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"For any metrics chosen what is often overlooked is measuring cost (API based model costs or compute costs for self hosted) and latency to get to actual task completion. Any agent success measures are not useful outside the context of cost and latency. In real world LLM applications and multi-turn generations, these two can often be prohibitive.\"})})]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"br\",{}),\" I plan to add more references for those interested in going deeper in the next revision. \"]})]});export const richText8=/*#__PURE__*/t(i.Fragment,{children:[/*#__PURE__*/e(\"p\",{children:\"Optimizing a Retrieval-Augmented Generation (RAG) system generally involves three goals:\"}),/*#__PURE__*/t(\"ol\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/e(\"p\",{children:\"Improving the quality, accuracy, and appropriateness of the responses that the RAG system generates.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/e(\"p\",{children:\"Generating responses faster \u2014 this may mean lower latency or higher throughput.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"11px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/e(\"p\",{children:\"Doing it cheaper \u2014 lowering the cost per response or cost of the system overall.\"})})]}),/*#__PURE__*/e(\"p\",{children:\"In this article, we offer a few RAG optimization techniques for all parts of your RAG system.\"}),/*#__PURE__*/e(\"h2\",{children:\"A brief overview of RAG architecture\"}),/*#__PURE__*/e(\"p\",{children:\"A RAG system often consists of three separate components\u2014intent detection and routing, retrieval, and generation\u2014and you need to optimize each of these separately. However, poor performance in an upstream RAG phase has a cascading effect on the overall performance, so earlier phases should not be skipped or glossed over.\"}),/*#__PURE__*/e(\"p\",{children:\"Important note: Your RAG performance metrics are only meaningful if you evaluate them using your own data.\"}),/*#__PURE__*/t(\"p\",{children:[\"As we mention in our article on \",/*#__PURE__*/e(o,{href:\"https://okareo.com/blog/posts/rag-architecture-key-components\",motionChild:!0,nodeId:\"wi5ius45d\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(a.a,{children:\"RAG architecture\"})}),\", there is no single RAG architecture that\u2019s representative of every single RAG system because they can be so different \u2014 but there are a few key components that tend to be present in all RAG systems. You can see how they work together in the following diagram.\"]}),/*#__PURE__*/e(\"img\",{alt:\"Components of a RAG system (that may need to be individually optimized)\",className:\"framer-image\",height:\"800\",src:\"https://framerusercontent.com/images/1tbBXujXPBPm02o9TJBcjjwybQ.png\",srcSet:\"https://framerusercontent.com/images/1tbBXujXPBPm02o9TJBcjjwybQ.png?scale-down-to=1024 762w,https://framerusercontent.com/images/1tbBXujXPBPm02o9TJBcjjwybQ.png 1191w\",style:{aspectRatio:\"1191 / 1600\"},width:\"595\"}),/*#__PURE__*/e(\"p\",{children:\"Let\u2019s see how we can optimize each component individually.\"}),/*#__PURE__*/e(\"h2\",{children:\"General techniques for optimizing your RAG\"}),/*#__PURE__*/t(\"p\",{children:[\"Across all phases, you will likely get a performance improvement by \",/*#__PURE__*/e(\"strong\",{children:\"using the right model\"}),\" for each task and \",/*#__PURE__*/e(\"strong\",{children:\"fine-tuning the model\"}),\" (via domain specific or task specific fine-tuning).\"]}),/*#__PURE__*/t(\"p\",{children:[\"In all cases, \",/*#__PURE__*/e(\"strong\",{children:\"high-quality, clean datasets and scenarios\"}),\" will also help you get better performance, as cleaner data results in lower need for data processing and reduces the likelihood of errors due to data quality.\"]}),/*#__PURE__*/e(\"h2\",{children:\"RAG optimization at the intent detection/routing phase\"}),/*#__PURE__*/e(\"p\",{children:\"At the intent detection and routing phase, you can make the RAG system more optimal by making sure that irrelevant and expensive queries get filtered out and that the routing is as accurate as possible. Here are the specific RAG optimizations you can do in this phase:\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Filter out off-topic, out of scope, or malicious queries \"}),\"before routing them to any downstream system. It's crucial to filter out these irrelevant or harmful queries so that they don\u2019t affect the performance of the system and don\u2019t use resources unnecessarily. Consider that some filtering queries may be cheaper and faster than others.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Consider using a small classification model for intent detection.\"}),\" By using a classification model for this task, rather than a more general, larger model, you can minimize latency and operating costs while still ensuring accurate routing.\"]}),/*#__PURE__*/e(\"img\",{alt:\"Diagram showing how a smaller, less complex model can do intent detection as well as an LLM.\",className:\"framer-image\",height:\"618\",src:\"https://framerusercontent.com/images/5RC5WXX2KGe2QzHJg8L8vc8GqU.jpg\",srcSet:\"https://framerusercontent.com/images/5RC5WXX2KGe2QzHJg8L8vc8GqU.jpg?scale-down-to=512 512w,https://framerusercontent.com/images/5RC5WXX2KGe2QzHJg8L8vc8GqU.jpg?scale-down-to=1024 1024w,https://framerusercontent.com/images/5RC5WXX2KGe2QzHJg8L8vc8GqU.jpg 1600w\",style:{aspectRatio:\"1600 / 1236\"},width:\"800\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Fine-tune the classification model to your specific domains.\"}),\" Once you\u2019re using a suitable model for the intent detection task, you can further improve performance by fine-tuning it on the specific kinds of queries that your users are likely to supply.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Use custom routing logic. \"}),\"If you know that specific queries will get processed faster or more effectively using data from a particular source, you can implement that in your routing logic rather than letting the classification model make that decision every time.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Evaluate your intent detection model\"}),\" to ensure that it is correctly identifying intent. Use specific metrics suitable for evaluating intent detection classification models, such as accuracy, F1 score, precision, and recall.\"]}),/*#__PURE__*/e(\"h3\",{children:\"How Okareo can help with RAG optimizations at the intent detection and routing phase\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Evaluates classification models:\"}),\" Okareo provides tools that evaluate the classification of user intents, ensuring that the system correctly identifies and categorizes requests.\",/*#__PURE__*/e(o,{href:\"https://docs.okareo.ai/docs/guides/classification_overview\",motionChild:!0,nodeId:\"wi5ius45d\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(a.a,{children:\" Learn more about Okareo's classification evaluations.\"})})]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Generates synthetic data for improving test coverage and performance:\"}),\" To enhance the accuracy and reliability of intent detection, Okareo offers synthetic data generation. This helps improve test coverage and overall system performance.\",/*#__PURE__*/e(o,{href:\"https://blog.okareo.ai/blog/posts/synthetic-data\",motionChild:!0,nodeId:\"wi5ius45d\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(a.a,{children:\" Read more on how synthetic data can help.\"})})]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Supports intent detection fine-tuning:\"}),\" Okareo also supports fine-tuning models for intent detection, allowing you to further optimize understanding and routing of specific types of queries.\",/*#__PURE__*/e(o,{href:\"https://okareo.com/blog/posts/bootstrap-finetuning\",motionChild:!0,nodeId:\"wi5ius45d\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(a.a,{children:\" Explore fine-tuning options.\"})})]}),/*#__PURE__*/e(\"h2\",{children:\"RAG optimization at the retrieval phase\"}),/*#__PURE__*/e(\"p\",{children:\"Once the intent is clear, the next step is deciding which data source to use for your retrieval phase. While much attention is on vector databases, your data might be in different relational databases, data lakes, graph databases, or a combination of these, depending on the specific use case. In some cases, you want to route the request to an API or perform a web search. This decision is key, because the quality of the response heavily depends on the relevance and accuracy of the chosen data source.\"}),/*#__PURE__*/e(\"p\",{children:\"Here are a few RAG optimization techniques for the retrieval phase:\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Baseline on BM25:\"}),\" \",/*#__PURE__*/e(o,{href:\"https://en.wikipedia.org/wiki/Okapi_BM25\",motionChild:!0,nodeId:\"wi5ius45d\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(a.a,{children:\"BM25 (Best Matching 25)\"})}),\" is a basic keyword-based ranking algorithm that gives a relevance score based on things like term frequency, document frequency, and document length. This is a very efficient way to narrow down your data and provide an initial benchmark against which your retrieval system can be evaluated. After this initial \",/*#__PURE__*/e(\"strong\",{children:\"retrieval stage\"}),\", you can narrow things down even faster with a \",/*#__PURE__*/e(\"strong\",{children:\"reranking stage\"}),\" that typically uses a similarity search algorithm to rerank documents according to which is most similar to the query.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Choose an appropriate embedding model:\"}),\" Your embedding model is the key component to evaluate in terms of latency, cost, and retrieval performance. It's worth noting that you don't always need a large, expensive model \u2014 many smaller models are quite powerful and can be good enough. Your embedding model is a key part of the reranking stage, so during evaluation, you should compare any performance gains to your BM25 baselines.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Use a hybrid search approach:\"}),\" When deciding between modern sparse vector models (for example, \",/*#__PURE__*/e(o,{href:\"https://arxiv.org/abs/2107.05720\",motionChild:!0,nodeId:\"wi5ius45d\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(a.a,{children:\"SPLADE\"})}),\", which mixes traditional term-matching techniques like that of BM25 with deep learning) and \",/*#__PURE__*/e(o,{href:\"https://www.pinecone.io/learn/series/nlp/dense-vector-embeddings-nlp/\",motionChild:!0,nodeId:\"wi5ius45d\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(a.a,{children:\"dense vector models\"})}),\" like Sentence-BERT, consider that a hybrid setup often works best. This means retrieving documents from both models and combining the top results of each, giving extra weighting to one of the models depending on whether exact matching or semantic similarity are more important. In many cases, a hybrid setup delivers better results for general use cases.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Build evaluations using your own data and your typical queries\"}),\": These could come from production or be synthetically generated based on your seed inputs. You don\u2019t want some random benchmark or dataset that will not show what relevance means for your app.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Balance the footprint of retrieve and reranking models with their performance:\"}),\" You'll need to strike a balance between speed, cost, and accuracy.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Fine-tune retrieve and reranking models\"}),\" on domain-specific data (both synthetic and real-word data).\"]}),/*#__PURE__*/e(\"h3\",{children:\"How Okareo can help with RAG optimization at the retrieval phase\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(o,{href:\"https://docs.okareo.ai/docs/guides/retrieval/retrieval_overview\",motionChild:!0,nodeId:\"wi5ius45d\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(a.a,{children:/*#__PURE__*/e(\"strong\",{children:\"Explore Okareo's retrieval evaluation:\"})})}),\" Due to intrinsic complexity of retrieval, there is a need for performance evaluation dedicated to the retrieval phase of RAG. If you can\u2019t isolate and trap problems here, they cascade into downstream failures.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Optimizing for your data:\"}),\" Okareo provides guidance on selecting an embedding model that fits your data.\",/*#__PURE__*/e(o,{href:\"https://okareo.com/blog/posts/embedding-model-compare\",motionChild:!0,nodeId:\"wi5ius45d\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(a.a,{children:\" Learn more about embedding model selection.\"})})]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Synthetic generation of evaluation data: \"}),\"Synthetically generate evaluation \",/*#__PURE__*/e(o,{href:\"https://docs.okareo.com/docs/getting-started/concepts/scenarios\",motionChild:!0,nodeId:\"wi5ius45d\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(a.a,{children:\"scenarios\"})}),\" (testing data that consists of a series of model inputs paired with its corresponding expected result) based on your own data and typical queries.\"]}),/*#__PURE__*/e(\"h2\",{children:\"RAG optimization at the generation phase\"}),/*#__PURE__*/e(\"p\",{children:\"In this phase, your RAG undertakes complex reasoning and decision-making tasks. These tasks can complicate performance evaluations, as they require the model to not only generate text but also make logical decisions based on the context provided. Some RAG optimization techniques for this phase include:\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Use \"}),/*#__PURE__*/e(o,{href:\"https://www.iguazio.com/glossary/llm-as-a-judge/\",motionChild:!0,nodeId:\"wi5ius45d\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(a.a,{children:/*#__PURE__*/e(\"strong\",{children:\"LLM-as-a-judge\"})})}),/*#__PURE__*/e(\"strong\",{children:\":\"}),\" Using another LLM to evaluate the output of your LLM can enhance the quality of your output without requiring a human in the loop. LLM-as-a-judge uses a reflective model, or \u201Cquality model,\u201D to review the result of your generation model for errors. You can either have one LLM evaluate another, or an LLM can evaluate itself.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Use \"}),/*#__PURE__*/e(o,{href:\"https://medium.com/data-science-in-your-pocket/what-is-reflection-tuning-for-llms-4d4e60c74324\",motionChild:!0,nodeId:\"wi5ius45d\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(a.a,{children:/*#__PURE__*/e(\"strong\",{children:\"reflection tuning\"})})}),/*#__PURE__*/e(\"strong\",{children:\":\"}),\" Reflection tuning is a more advanced version of using LLM-as-a-judge that cycles iteratively between the generation model and reflective model, refining the output until the reflective model is satisfied that it is high quality. A reflective model could be a more powerful or the same model as the generation model, and it takes advantage of LLMs reflective property to find errors and inconsistencies when that is the focus of the prompt.\"]}),/*#__PURE__*/e(\"h3\",{children:\"How Okareo can help with RAG optimization at the generation phase\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(o,{href:\"https://docs.okareo.ai/docs/guides/generation_overview\",motionChild:!0,nodeId:\"wi5ius45d\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(a.a,{children:/*#__PURE__*/e(\"strong\",{children:\"Get started with generation evaluations:\"})})}),\" Evaluate how well your models are generating relevant and accurate outputs based on the context provided.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(o,{href:\"https://docs.okareo.ai/docs/getting-started/concepts/checks\",motionChild:!0,nodeId:\"wi5ius45d\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(a.a,{children:/*#__PURE__*/e(\"strong\",{children:\"Scoring a generative model's output:\"})})}),\" Ensure the output of your generative models meets the quality standards required for your application.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(o,{href:\"https://blog.okareo.ai/blog/posts/llm-evaluation-in-ci\",motionChild:!0,nodeId:\"wi5ius45d\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(a.a,{children:/*#__PURE__*/e(\"strong\",{children:\"Add LLM evaluation to your CI workflow:\"})})}),\" Maintain high-quality output as you develop and deploy your models by integrating your LLM evaluation into your continuous integration (CI) workflow.\"]}),/*#__PURE__*/e(\"h2\",{children:\"What's next?\"}),/*#__PURE__*/t(\"p\",{children:[\"Would you like to get started with optimizing your RAG?\",/*#__PURE__*/e(o,{href:\"https://app.okareo.com/account/sign-up\",motionChild:!0,nodeId:\"wi5ius45d\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(a.a,{children:\" Sign up for Okareo\"})}),\" and follow\",/*#__PURE__*/e(o,{href:\"https://www.okareo.com/docs/getting-started/overview\",motionChild:!0,nodeId:\"wi5ius45d\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(a.a,{children:\" our documentation\"})}),\" to get going.\"]})]});\nexport const __FramerMetadata__ = {\"exports\":{\"richText8\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"richText3\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"richText2\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"richText\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"richText7\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"richText1\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"richText5\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"richText4\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"richText6\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"__FramerMetadata__\":{\"type\":\"variable\"}}}"],
  "mappings": "scAAqZ,IAAMA,EAAsBC,EAAIC,EAAS,CAAC,SAAS,CAAcC,EAAE,IAAI,CAAC,SAAS,2WAA2W,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,6BAA6B,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,SAAS,qBAAqB,CAAC,EAAE,ynBAA2nB,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,iCAAiC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+MAA+M,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,+FAA4GE,EAAEC,EAAE,CAAC,KAAK,+FAA+F,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,MAAM,CAAC,CAAC,CAAC,EAAE,GAAG,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,mMAAmM,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,GAAG,UAAU,eAAe,OAAO,MAAM,IAAI,uEAAuE,OAAO,qWAAqW,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,MAAM,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qHAAqH,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,sBAAsB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6TAA6T,CAAC,EAAeA,EAAE,MAAM,CAAC,UAAU,qBAAqB,MAAM,CAAC,OAAO,OAAO,MAAM,MAAM,EAAE,SAAsBA,EAAEG,EAAE,CAAC,oBAAoB,wEAAwE,SAASC,GAAgBJ,EAAEK,EAAE,CAAC,GAAGD,EAAE,KAAK;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA,IAAiqD,SAAS,QAAQ,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeJ,EAAE,IAAI,CAAC,SAAS,mGAAmG,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,GAAG,UAAU,eAAe,OAAO,MAAM,IAAI,uEAAuE,OAAO,qWAAqW,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,MAAM,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gPAAgP,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,YAAY,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+EAA+E,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,8DAA8D,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,mDAAmD,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,+EAA+E,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2QAA2Q,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0OAA0O,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,gDAAgD,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sSAAsS,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,WAAwBE,EAAEC,EAAE,CAAC,KAAK,yCAAyC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,aAAa,CAAC,CAAC,CAAC,EAAE,wHAAqIF,EAAEC,EAAE,CAAC,KAAK,gDAAgD,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,UAAU,CAAC,CAAC,CAAC,EAAE,qDAAqD,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,UAAU,gBAAgB,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeM,EAAuBR,EAAIC,EAAS,CAAC,SAAS,CAAcC,EAAE,MAAM,CAAC,UAAU,qBAAqB,MAAM,CAAC,iBAAiB,YAAY,YAAY,YAAY,OAAO,OAAO,MAAM,MAAM,EAAE,SAAsBA,EAAEG,EAAE,CAAC,oBAAoB,sEAAsE,SAASC,GAAgBJ,EAAEO,EAAE,CAAC,GAAGH,EAAE,KAAK,MAAM,WAAW,GAAG,UAAU,eAAe,IAAI,kDAAkD,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeN,EAAE,IAAI,CAAC,SAAS,CAAC,qDAAkEE,EAAEC,EAAE,CAAC,KAAK,qDAAqD,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,yDAAyD,CAAC,CAAC,CAAC,EAAE,GAAG,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,oRAAoR,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,wCAAwC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,uEAAuE,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,yEAAyE,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gDAAgD,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAEC,EAAE,CAAC,KAAK,mDAAmD,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,wBAAwB,CAAC,CAAC,CAAC,EAAE,8DAA8D,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAEC,EAAE,CAAC,KAAK,yCAAyC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,SAAS,CAAC,CAAC,CAAC,EAAE,wBAAwB,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAC,qDAAkEE,EAAEC,EAAE,CAAC,KAAK,+GAA+G,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,QAAQ,CAAC,CAAC,CAAC,EAAE,QAAqBF,EAAEC,EAAE,CAAC,KAAK,+GAA+G,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,QAAQ,CAAC,CAAC,CAAC,EAAE,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,UAAU,gBAAgB,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeQ,EAAuBV,EAAIC,EAAS,CAAC,SAAS,CAAcD,EAAE,IAAI,CAAC,SAAS,CAAC,0MAAkNE,EAAEC,EAAE,CAAC,KAAK,iDAAiD,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,gBAAgB,CAAC,CAAC,CAAC,EAAE,OAAoBF,EAAEC,EAAE,CAAC,KAAK,6GAA6G,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,YAAY,CAAC,CAAC,CAAC,EAAE,iRAAiR,CAAC,CAAC,EAAeF,EAAE,MAAM,CAAC,IAAI,GAAG,UAAU,eAAe,OAAO,MAAM,IAAI,sEAAsE,OAAO,oQAAoQ,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,KAAK,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,oCAAoC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yaAA0Z,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sdAAid,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+eAA+e,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sTAA4S,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,oDAAoD,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,WAAW,CAAC,EAAE,sHAAsH,CAAC,CAAC,EAAeA,EAAE,aAAa,CAAC,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAC,kCAA+CE,EAAE,SAAS,CAAC,SAAS,iCAAiC,CAAC,EAAeA,EAAEC,EAAE,CAAC,KAAK,+GAA+G,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAsBF,EAAE,SAAS,CAAC,SAAS,QAAQ,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,SAAS,CAAC,SAAS,OAAO,CAAC,EAAeA,EAAEC,EAAE,CAAC,KAAK,+GAA+G,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAsBF,EAAE,SAAS,CAAC,SAAS,QAAQ,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,SAAS,CAAC,SAAS,GAAG,CAAC,EAAE,sBAAmCA,EAAEC,EAAE,CAAC,KAAK,yCAAyC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,oBAAoB,CAAC,CAAC,CAAC,EAAE,8BAA8B,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,8RAAyR,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wEAAwE,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAC,+DAA4EE,EAAEC,EAAE,CAAC,KAAK,+EAA+E,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,mCAAmC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,sDAAsD,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,8CAA8C,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,mCAAmC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAC,8DAA2EE,EAAEC,EAAE,CAAC,KAAK,yDAAyD,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,kCAAkC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAC,mDAAgEE,EAAEC,EAAE,CAAC,KAAK,yDAAyD,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,6BAA6B,CAAC,CAAC,CAAC,EAAE,wBAAwB,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,6CAA6C,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,qGAAgG,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,4BAA4B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uMAAuM,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+NAA+N,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,GAAG,UAAU,eAAe,OAAO,MAAM,IAAI,uEAAuE,OAAO,qWAAqW,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,MAAM,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uFAAuF,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,2BAA2B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kgBAAye,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,oHAA4HE,EAAEC,EAAE,CAAC,KAAK,wGAAwG,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,wBAAwB,CAAC,CAAC,CAAC,EAAE,qFAAqF,CAAC,CAAC,EAAeF,EAAE,MAAM,CAAC,IAAI,GAAG,UAAU,eAAe,OAAO,MAAM,IAAI,uEAAuE,OAAO,uQAAuQ,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,KAAK,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+DAA+D,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,mDAAmD,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6KAA6K,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,0FAA0F,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,0DAA0D,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,iEAAiE,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kFAAkF,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,GAAG,UAAU,eAAe,OAAO,MAAM,IAAI,sEAAsE,OAAO,iWAAiW,MAAM,CAAC,YAAY,YAAY,EAAE,MAAM,MAAM,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,iDAAiD,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,qBAAkCE,EAAEC,EAAE,CAAC,KAAK,0DAA0D,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,wBAAwB,CAAC,CAAC,CAAC,EAAE,2KAAwLF,EAAEC,EAAE,CAAC,KAAK,+CAA+C,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,eAAe,CAAC,CAAC,CAAC,EAAE,GAAG,CAAC,CAAC,EAAeJ,EAAE,IAAI,CAAC,SAAS,CAAC,yDAAsEE,EAAEC,EAAE,CAAC,KAAK,qDAAqD,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,sDAAsD,CAAC,CAAC,CAAC,EAAE,iHAA8HF,EAAEC,EAAE,CAAC,KAAK,mCAAmC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,MAAM,CAAC,CAAC,CAAC,EAAE,KAAkBF,EAAEC,EAAE,CAAC,KAAK,6DAA6D,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,oBAAoB,CAAC,CAAC,CAAC,EAAE,SAAsBF,EAAEC,EAAE,CAAC,KAAK,mCAAmC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,iBAAiB,CAAC,CAAC,CAAC,EAAE,6CAA6C,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,0CAA0C,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kHAAkH,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,8CAA8C,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,iDAAiD,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,wDAAwD,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4MAAkM,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,GAAG,UAAU,eAAe,OAAO,MAAM,IAAI,uEAAuE,OAAO,qWAAqW,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,MAAM,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2FAA2F,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uIAAuI,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,GAAG,UAAU,eAAe,OAAO,MAAM,IAAI,qEAAqE,OAAO,iQAAiQ,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,KAAK,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gGAAgG,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,oDAAoD,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2QAAsQ,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,kEAAkE,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,iDAAiD,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,wEAA8D,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,GAAG,UAAU,eAAe,OAAO,MAAM,IAAI,sEAAsE,OAAO,iWAAiW,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,MAAM,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oEAAoE,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,gDAAgD,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iHAAiH,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,sCAAsC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2OAAsO,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,GAAG,UAAU,eAAe,OAAO,OAAO,IAAI,qEAAqE,OAAO,kQAAkQ,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,KAAK,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,YAAY,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,maAA8Z,CAAC,CAAC,CAAC,CAAC,EAAeS,EAAuBX,EAAIC,EAAS,CAAC,SAAS,CAAcC,EAAE,IAAI,CAAC,SAAS,yKAAyK,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8TAA8T,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,siBAAsiB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2PAA2P,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,yBAAyB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0SAAqS,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8TAAyT,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,iBAAiB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+UAA+U,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,wDAAqEE,EAAEC,EAAE,CAAC,KAAK,6DAA6D,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAsBF,EAAE,KAAK,CAAC,SAAS,UAAU,CAAC,CAAC,CAAC,CAAC,CAAC,EAAE,6IAA6I,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,6JAAqKE,EAAEC,EAAE,CAAC,KAAK,6DAA6D,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAsBF,EAAE,KAAK,CAAC,SAAS,QAAQ,CAAC,CAAC,CAAC,CAAC,CAAC,EAAE,sHAAmIA,EAAE,KAAK,CAAC,SAAS,aAAa,CAAC,EAAE,OAAoBA,EAAE,KAAK,CAAC,SAAS,WAAW,CAAC,EAAE,GAAG,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,2fAAwgBE,EAAE,OAAO,CAAC,SAAS,4DAA4D,CAAC,EAAE,mFAA8E,CAAC,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,sFAAsF,UAAU,eAAe,OAAO,MAAM,IAAI,uEAAuE,OAAO,qWAAqW,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,MAAM,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,2EAA2E,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,kUAA+UE,EAAE,OAAO,CAAC,SAAS,eAAe,CAAC,EAAE,cAAc,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,oCAAiDE,EAAEC,EAAE,CAAC,KAAK,yCAAyC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,QAAQ,CAAC,CAAC,CAAC,EAAE,QAAqBF,EAAEC,EAAE,CAAC,KAAK,qCAAqC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,QAAQ,CAAC,CAAC,CAAC,EAAE,qCAAqC,CAAC,CAAC,EAAeJ,EAAE,IAAI,CAAC,SAAS,CAAC,gCAA6CE,EAAEC,EAAE,CAAC,KAAK,gDAAgD,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,YAAY,CAAC,CAAC,CAAC,EAAE,aAA0BF,EAAEC,EAAE,CAAC,KAAK,4CAA4C,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,QAAQ,CAAC,CAAC,CAAC,EAAE,qBAAqB,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,gCAAgC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,6RAAqSE,EAAEC,EAAE,CAAC,KAAK,wFAAwF,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,0BAA0B,CAAC,CAAC,CAAC,EAAE,8CAA8C,CAAC,CAAC,EAAeJ,EAAE,KAAK,CAAC,SAAS,CAAcA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAS,CAAcA,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,wBAAwB,CAAC,EAAE,wIAAqJA,EAAEC,EAAE,CAAC,KAAK,kCAAkC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,4BAA4B,CAAC,CAAC,CAAC,EAAE,4MAA4M,CAAC,CAAC,EAAeF,EAAE,MAAM,CAAC,UAAU,qBAAqB,MAAM,CAAC,OAAO,OAAO,MAAM,MAAM,EAAE,SAAsBA,EAAEG,EAAE,CAAC,oBAAoB,wEAAwE,SAASC,GAAgBJ,EAAEK,EAAE,CAAC,GAAGD,EAAE,KAAK;AAAA;AAAA;AAAA;AAAA,8BAAmF,SAAS,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeN,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAS,CAAcA,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,6CAA6C,CAAC,EAAE,8CAA8C,CAAC,CAAC,EAAeA,EAAE,MAAM,CAAC,UAAU,qBAAqB,MAAM,CAAC,OAAO,OAAO,MAAM,MAAM,EAAE,SAAsBA,EAAEG,EAAE,CAAC,oBAAoB,wEAAwE,SAASC,GAAgBJ,EAAEK,EAAE,CAAC,GAAGD,EAAE,KAAK,mCAAmC,SAAS,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeJ,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,UAAU,gBAAgB,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAC,8FAA2GE,EAAE,OAAO,CAAC,SAAS,uBAAuB,CAAC,EAAE,QAAqBA,EAAE,OAAO,CAAC,SAAS,YAAY,CAAC,EAAE,GAAG,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,kCAAkC,CAAC,EAAE,wCAAqDA,EAAE,OAAO,CAAC,SAAS,eAAe,CAAC,EAAE,+GAA+G,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,MAAM,CAAC,UAAU,qBAAqB,MAAM,CAAC,OAAO,OAAO,MAAM,MAAM,EAAE,SAAsBA,EAAEG,EAAE,CAAC,oBAAoB,wEAAwE,SAASC,GAAgBJ,EAAEK,EAAE,CAAC,GAAGD,EAAE,KAAK;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA,MAAqiC,SAAS,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeJ,EAAE,KAAK,CAAC,MAAM,IAAI,SAAsBA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,uBAAuB,CAAC,EAAE,gHAA6HA,EAAEC,EAAE,CAAC,KAAK,4EAA4E,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,eAAe,CAAC,CAAC,CAAC,EAAE,iHAA8HF,EAAE,OAAO,CAAC,SAAS,eAAe,CAAC,EAAE,qHAAqH,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,MAAM,CAAC,UAAU,qBAAqB,MAAM,CAAC,OAAO,OAAO,MAAM,MAAM,EAAE,SAAsBA,EAAEG,EAAE,CAAC,oBAAoB,wEAAwE,SAASC,GAAgBJ,EAAEK,EAAE,CAAC,GAAGD,EAAE,KAAK;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA,EAAwoF,SAAS,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeN,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,4EAA4E,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,UAAU,CAAC,CAAC,CAAC,EAAeA,EAAE,SAAS,CAAC,SAAS,QAAQ,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,MAAM,IAAI,SAAsBA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,yBAAyB,CAAC,EAAE,kBAA+BA,EAAE,OAAO,CAAC,SAAS,YAAY,CAAC,EAAE,6CAA0DA,EAAE,OAAO,CAAC,SAAS,eAAe,CAAC,EAAE,kLAAkL,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,MAAM,CAAC,UAAU,qBAAqB,MAAM,CAAC,OAAO,OAAO,MAAM,MAAM,EAAE,SAAsBA,EAAEG,EAAE,CAAC,oBAAoB,wEAAwE,SAASC,GAAgBJ,EAAEK,EAAE,CAAC,GAAGD,EAAE,KAAK;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA,uBAA0e,SAAS,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeJ,EAAE,KAAK,CAAC,MAAM,IAAI,SAAsBA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,qCAAqC,CAAC,EAAE,gOAAgO,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,MAAM,CAAC,UAAU,qBAAqB,MAAM,CAAC,OAAO,OAAO,MAAM,MAAM,EAAE,SAAsBA,EAAEG,EAAE,CAAC,oBAAoB,wEAAwE,SAASC,GAAgBJ,EAAEK,EAAE,CAAC,GAAGD,EAAE,KAAK;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA,KAAsJ,SAAS,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeJ,EAAE,KAAK,CAAC,MAAM,IAAI,SAAsBF,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAS,CAAcA,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,mBAAmB,CAAC,EAAE,uIAAuI,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,QAAqBE,EAAE,OAAO,CAAC,SAAS,gBAAgB,CAAC,EAAE,yLAAsMA,EAAE,OAAO,CAAC,SAAS,eAAe,CAAC,EAAE,WAAW,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,MAAM,CAAC,UAAU,qBAAqB,MAAM,CAAC,OAAO,OAAO,MAAM,MAAM,EAAE,SAAsBA,EAAEG,EAAE,CAAC,oBAAoB,wEAAwE,SAASC,GAAgBJ,EAAEK,EAAE,CAAC,GAAGD,EAAE,KAAK;AAAA;AAAA;AAAA;AAAA,iBAAkH,SAAS,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeN,EAAE,KAAK,CAAC,MAAM,IAAI,SAAS,CAAcA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAS,CAAcA,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,6CAA6C,CAAC,EAAE,mFAAmF,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAS,CAAcA,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,gDAAgD,CAAC,EAAE,0KAA0K,CAAC,CAAC,EAAeA,EAAE,MAAM,CAAC,UAAU,qBAAqB,MAAM,CAAC,OAAO,OAAO,MAAM,MAAM,EAAE,SAAsBA,EAAEG,EAAE,CAAC,oBAAoB,wEAAwE,SAASC,GAAgBJ,EAAEK,EAAE,CAAC,GAAGD,EAAE,KAAK;AAAA;AAAA;AAAA,IAAwI,SAAS,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeJ,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,4BAA4B,CAAC,EAAE,6KAA6K,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,MAAM,CAAC,UAAU,qBAAqB,MAAM,CAAC,OAAO,OAAO,MAAM,MAAM,EAAE,SAAsBA,EAAEG,EAAE,CAAC,oBAAoB,wEAAwE,SAASC,GAAgBJ,EAAEK,EAAE,CAAC,GAAGD,EAAE,KAAK;AAAA;AAAA;AAAA;AAAA,GAA+J,SAAS,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeJ,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,8BAA8B,CAAC,EAAE,gDAAgD,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,MAAM,CAAC,UAAU,qBAAqB,MAAM,CAAC,OAAO,OAAO,MAAM,MAAM,EAAE,SAAsBA,EAAEG,EAAE,CAAC,oBAAoB,wEAAwE,SAASC,GAAgBJ,EAAEK,EAAE,CAAC,GAAGD,EAAE,KAAK,oCAAoC,SAAS,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeN,EAAE,IAAI,CAAC,SAAS,CAAC,uDAAoEE,EAAE,OAAO,CAAC,SAAS,gBAAgB,CAAC,EAAE,uLAAuL,CAAC,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,2EAA2E,UAAU,eAAe,OAAO,MAAM,IAAI,uEAAuE,OAAO,uQAAuQ,MAAM,CAAC,YAAY,YAAY,EAAE,MAAM,KAAK,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,uEAAuE,UAAU,eAAe,OAAO,MAAM,IAAI,uEAAuE,OAAO,uQAAuQ,MAAM,CAAC,YAAY,YAAY,EAAE,MAAM,KAAK,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,mFAAmF,UAAU,eAAe,OAAO,MAAM,IAAI,sEAAsE,MAAM,CAAC,YAAY,WAAW,EAAE,MAAM,KAAK,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,0CAA0C,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,4OAAyPE,EAAE,OAAO,CAAC,SAAS,YAAY,CAAC,EAAE,+BAA+B,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6KAA6K,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,gFAA6FE,EAAE,OAAO,CAAC,SAAS,SAAS,CAAC,EAAE,wDAAwD,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,uCAAuC,CAAC,EAAE,0CAAuDA,EAAE,KAAK,CAAC,SAAS,UAAU,CAAC,EAAE,MAAmBA,EAAE,KAAK,CAAC,SAAS,uBAAuB,CAAC,EAAE,MAAmBA,EAAE,KAAK,CAAC,SAAS,SAAS,CAAC,EAAE,cAA2BA,EAAE,KAAK,CAAC,SAAS,uBAAuB,CAAC,EAAE,qBAAkCA,EAAE,OAAO,CAAC,SAAS,mCAAmC,CAAC,EAAE,SAAsBA,EAAE,OAAO,CAAC,SAAS,iBAAiB,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,sBAAsB,CAAC,EAAE,gCAA6CA,EAAE,KAAK,CAAC,SAAS,SAAS,CAAC,EAAE,iBAA8BA,EAAE,KAAK,CAAC,SAAS,+BAA+B,CAAC,EAAE,+DAA4EA,EAAE,OAAO,CAAC,SAAS,4BAA4B,CAAC,EAAE,GAAG,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,kCAA+CE,EAAE,OAAO,CAAC,SAAS,UAAU,CAAC,EAAE,8FAA8F,CAAC,CAAC,EAAeA,EAAE,MAAM,CAAC,UAAU,qBAAqB,MAAM,CAAC,OAAO,OAAO,MAAM,MAAM,EAAE,SAAsBA,EAAEG,EAAE,CAAC,oBAAoB,wEAAwE,SAASC,GAAgBJ,EAAEK,EAAE,CAAC,GAAGD,EAAE,KAAK;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA,2CAAwxB,SAAS,MAAM,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeJ,EAAE,IAAI,CAAC,SAAS,4IAA4I,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,mIAAmI,UAAU,eAAe,OAAO,MAAM,IAAI,sEAAsE,OAAO,uKAAuK,MAAM,CAAC,YAAY,YAAY,EAAE,MAAM,KAAK,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,gDAAgD,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sSAAsS,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,WAAwBE,EAAEC,EAAE,CAAC,KAAK,yCAAyC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,aAAa,CAAC,CAAC,CAAC,EAAE,wJAAqKF,EAAEC,EAAE,CAAC,KAAK,gDAAgD,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,UAAU,CAAC,CAAC,CAAC,EAAE,gDAAgD,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeQ,EAAuBZ,EAAIC,EAAS,CAAC,SAAS,CAAcC,EAAE,IAAI,CAAC,SAAS,mdAAmd,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,+IAA+I,UAAU,eAAe,OAAO,MAAM,IAAI,uEAAuE,OAAO,qWAAqW,MAAM,CAAC,YAAY,YAAY,EAAE,MAAM,MAAM,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAC,uBAAoCE,EAAEC,EAAE,CAAC,KAAK,uEAAuE,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,QAAG,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,uQAAuQ,CAAC,EAAeF,EAAE,KAAK,CAAC,MAAM,CAAC,qBAAqB,OAAO,0BAA0B,QAAQ,sBAAsB,kBAAkB,6BAA6B,MAAM,0BAA0B,MAAM,EAAE,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,qIAAqI,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAC,4IAAyJE,EAAE,KAAK,CAAC,SAAS,qBAAqB,CAAC,EAAE,4TAA4T,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,qQAAqQ,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2YAA2Y,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAC,sBAAmCE,EAAEC,EAAE,CAAC,KAAK,sEAAsE,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,QAAG,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,uNAAuN,CAAC,EAAeF,EAAE,KAAK,CAAC,MAAM,CAAC,qBAAqB,OAAO,0BAA0B,QAAQ,sBAAsB,kBAAkB,6BAA6B,MAAM,0BAA0B,MAAM,EAAE,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,oBAAoB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,uBAAuB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,wBAAwB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,gCAAgC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,kBAAkB,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,8EAA2FE,EAAEC,EAAE,CAAC,KAAK,+DAA+D,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,QAAQ,CAAC,CAAC,CAAC,EAAE,+JAA+J,CAAC,CAAC,EAAeJ,EAAE,KAAK,CAAC,SAAS,CAAC,YAAyBE,EAAEC,EAAE,CAAC,KAAK,4DAA4D,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,QAAG,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,4mBAA4mB,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAC,SAAsBE,EAAEC,EAAE,CAAC,KAAK,yDAAyD,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,QAAG,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeJ,EAAE,IAAI,CAAC,SAAS,CAAC,kBAA+BE,EAAEC,EAAE,CAAC,KAAK,+DAA+D,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,QAAQ,CAAC,CAAC,CAAC,EAAE,2oBAA2oB,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,sCAAsC,CAAC,EAAeF,EAAE,KAAK,CAAC,MAAM,CAAC,qBAAqB,OAAO,0BAA0B,QAAQ,sBAAsB,kBAAkB,6BAA6B,MAAM,0BAA0B,MAAM,EAAE,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,sBAAsB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,sBAAsB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,mBAAmB,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,iDAA8DE,EAAE,SAAS,CAAC,SAAS,QAAQ,CAAC,EAAE,0MAA0M,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,sBAAsB,CAAC,EAAE,ogBAAogB,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,YAAY,CAAC,EAAE,ykBAAykB,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAC,uBAAoCE,EAAEC,EAAE,CAAC,KAAK,uEAAuE,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,QAAG,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,6KAA6K,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,u8BAAu8B,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,iDAAiD,CAAC,EAAE,8UAA8U,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,MAAM,CAAC,qBAAqB,OAAO,0BAA0B,QAAQ,sBAAsB,kBAAkB,6BAA6B,MAAM,0BAA0B,MAAM,EAAE,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,oEAAoE,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,wFAAwF,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,0FAA0F,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAS,CAAcE,EAAE,IAAI,CAAC,SAAS,+FAA+F,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,GAAG,UAAU,eAAe,OAAO,MAAM,IAAI,uEAAuE,OAAO,uQAAuQ,MAAM,CAAC,YAAY,YAAY,EAAE,MAAM,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,gDAAgD,CAAC,EAAE,gjBAAgjB,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,oCAAoC,CAAC,EAAE,i6BAAi6B,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,koBAAmoB,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,2DAA2D,CAAC,EAAE,o1CAAo1C,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,uCAAuC,CAAC,EAAE,shBAAshB,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAC,mBAAgCE,EAAEC,EAAE,CAAC,KAAK,mEAAmE,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,QAAG,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,kaAAka,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yNAAyN,CAAC,CAAC,CAAC,CAAC,EAAeW,EAAuBb,EAAIC,EAAS,CAAC,SAAS,CAAcD,EAAE,IAAI,CAAC,SAAS,CAAC,odAAieE,EAAE,KAAK,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,UAAU,gBAAgB,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,GAAG,UAAU,eAAe,OAAO,MAAM,IAAI,sEAAsE,OAAO,iWAAiW,MAAM,CAAC,YAAY,YAAY,EAAE,MAAM,MAAM,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sZAAsZ,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,0BAA0B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBF,EAAE,OAAO,CAAC,SAAS,CAAC,yNAAsOE,EAAEC,EAAE,CAAC,KAAK,qCAAqC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,QAAQ,CAAC,CAAC,CAAC,EAAE,GAAG,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,kDAAkD,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,sEAAsE,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,gCAAgC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,2BAA2B,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,6BAA6B,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sPAAsP,CAAC,EAAeA,EAAE,aAAa,CAAC,SAAsBA,EAAE,IAAI,CAAC,SAAS,6QAAmQ,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2CAA2C,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,0BAA0B,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,yCAAyC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,kDAAkD,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,gBAAgB,CAAC,EAAeA,EAAE,MAAM,CAAC,UAAU,qBAAqB,MAAM,CAAC,OAAO,OAAO,MAAM,MAAM,EAAE,SAAsBA,EAAEG,EAAE,CAAC,oBAAoB,wEAAwE,SAASC,GAAgBJ,EAAEK,EAAE,CAAC,GAAGD,EAAE,KAAK;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA,KAAomB,SAAS,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeJ,EAAE,KAAK,CAAC,SAAS,YAAY,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wGAAwG,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,4CAA4C,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,kFAAkF,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,6DAA6D,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iJAAiJ,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,SAAS,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,yCAAsDE,EAAEC,EAAE,CAAC,KAAK,0DAA0D,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,EAAE,iRAAiR,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,uGAAuG,CAAC,EAAeA,EAAE,MAAM,CAAC,UAAU,qBAAqB,MAAM,CAAC,OAAO,OAAO,MAAM,MAAM,EAAE,SAAsBA,EAAEG,EAAE,CAAC,oBAAoB,wEAAwE,SAASC,GAAgBJ,EAAEK,EAAE,CAAC,GAAGD,EAAE,KAAK;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA,IAAwoB,SAAS,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeJ,EAAE,IAAI,CAAC,SAAS,6QAA6Q,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,aAAa,CAAC,EAAeA,EAAE,MAAM,CAAC,UAAU,qBAAqB,MAAM,CAAC,OAAO,OAAO,MAAM,MAAM,EAAE,SAAsBA,EAAEG,EAAE,CAAC,oBAAoB,wEAAwE,SAASC,GAAgBJ,EAAEK,EAAE,CAAC,GAAGD,EAAE,KAAK;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA,6BAAgb,SAAS,YAAY,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeJ,EAAE,KAAK,CAAC,SAAS,SAAS,CAAC,EAAeA,EAAE,MAAM,CAAC,UAAU,qBAAqB,MAAM,CAAC,OAAO,OAAO,MAAM,MAAM,EAAE,SAAsBA,EAAEG,EAAE,CAAC,oBAAoB,wEAAwE,SAASC,GAAgBJ,EAAEK,EAAE,CAAC,GAAGD,EAAE,KAAK;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA,GAAsyB,SAAS,QAAQ,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeJ,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,UAAU,gBAAgB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,WAAW,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kMAAkM,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,aAAa,CAAC,EAAeA,EAAE,MAAM,CAAC,UAAU,qBAAqB,MAAM,CAAC,OAAO,OAAO,MAAM,MAAM,EAAE,SAAsBA,EAAEG,EAAE,CAAC,oBAAoB,wEAAwE,SAASC,GAAgBJ,EAAEK,EAAE,CAAC,GAAGD,EAAE,KAAK;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA,MAA+Q,SAAS,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeJ,EAAE,KAAK,CAAC,SAAS,SAAS,CAAC,EAAeA,EAAE,MAAM,CAAC,UAAU,qBAAqB,MAAM,CAAC,OAAO,OAAO,MAAM,MAAM,EAAE,SAAsBA,EAAEG,EAAE,CAAC,oBAAoB,wEAAwE,SAASC,GAAgBJ,EAAEK,EAAE,CAAC,GAAGD,EAAE,KAAK;AAAA;AAAA;AAAA;AAAA;AAAA,UAAgM,SAAS,QAAQ,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeJ,EAAE,IAAI,CAAC,SAAS,ogBAAogB,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,YAAY,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,ocAAidE,EAAE,OAAO,CAAC,SAAS,yBAAyB,CAAC,EAAE,0OAAuPA,EAAE,OAAO,CAAC,SAAS,UAAU,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,GAAG,UAAU,eAAe,OAAO,MAAM,IAAI,uEAAuE,OAAO,uQAAuQ,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,KAAK,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,kBAAkB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6rBAA6rB,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,4NAAyOE,EAAEC,EAAE,CAAC,KAAK,yBAAyB,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,aAAa,CAAC,CAAC,CAAC,EAAE,qDAAqD,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,UAAU,gBAAgB,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeY,EAAuBd,EAAIC,EAAS,CAAC,SAAS,CAAcC,EAAE,KAAK,CAAC,SAAS,OAAO,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,oEAAoE,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,qIAAqI,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,iKAAiK,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,2EAA2E,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,sBAAsB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4KAA4K,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,6DAA6D,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,iEAAiE,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,8EAA8E,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gdAA2c,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,wBAAwB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4vBAAkvB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8vBAA8vB,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,qBAAqB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2XAA2X,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+VAA+V,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,GAAG,UAAU,eAAe,OAAO,MAAM,IAAI,uEAAuE,OAAO,yKAAyK,MAAM,CAAC,YAAY,YAAY,EAAE,MAAM,KAAK,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,0BAA0B,CAAC,EAAE,2RAA2R,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,oBAAoB,CAAC,EAAE,6VAA6V,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,kBAAkB,CAAC,EAAE,4fAA4f,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,oBAAoB,CAAC,EAAE,uRAAuR,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,4BAAuB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kOAA6N,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mcAAyb,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAS,CAAcA,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,0BAA0B,CAAC,EAAE,yFAAyF,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,yEAAyE,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,mFAAmF,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,4EAA4E,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,2GAA2G,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8ZAA8Z,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qQAAgQ,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,UAAU,gBAAgB,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAS,CAAcA,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,oBAAoB,CAAC,EAAE,+JAA0J,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,yFAAyF,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,4DAA4D,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,iBAAiB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,0BAA0B,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qeAAge,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,UAAU,gBAAgB,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAS,CAAcA,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,kBAAkB,CAAC,EAAE,ooBAA0nB,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,UAAU,gBAAgB,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,oBAAoB,CAAC,EAAE,2ZAA2Z,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,+BAA+B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,ohBAA0gB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qRAAqR,CAAC,CAAC,CAAC,CAAC,EAAea,EAAuBf,EAAIC,EAAS,CAAC,SAAS,CAAcC,EAAE,IAAI,CAAC,SAAS,yUAA0T,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,smBAAsmB,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,0BAA0B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6VAAyU,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,2GAA2G,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,kIAAkI,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,4GAA4G,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,0CAA0C,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qzBAAqzB,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,sRAA8RE,EAAEC,EAAE,CAAC,KAAK,gFAAgF,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,uBAAuB,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,qoBAAgoB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4XAAuX,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,GAAG,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,GAAG,UAAU,eAAe,OAAO,MAAM,IAAI,uEAAuE,OAAO,uQAAuQ,MAAM,CAAC,YAAY,YAAY,EAAE,MAAM,KAAK,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,kCAAkC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8lBAA8lB,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,iBAAiB,CAAC,EAAE,u0BAAu0B,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,6BAA6B,CAAC,EAAE,wqBAA8pB,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,uBAAuB,CAAC,EAAE,wuBAAwuB,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,oBAAoB,CAAC,EAAE,6kBAA6kB,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,SAAS,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oSAA+R,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,wBAAwB,CAAC,EAAE,yMAAoM,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,2BAA2B,CAAC,EAAE,wYAAmY,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,gCAAgC,CAAC,EAAE,seAA4d,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,0BAA0B,CAAC,EAAE,ufAAuf,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,4BAA4B,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,qLAAqL,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,wgBAAwgB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,4VAA4V,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,CAAC,EAAE,4FAA4F,CAAC,CAAC,CAAC,CAAC,CAAC,EAAec,EAAuBhB,EAAIC,EAAS,CAAC,SAAS,CAAcC,EAAE,IAAI,CAAC,SAAS,0FAA0F,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBA,EAAE,IAAI,CAAC,SAAS,sGAAsG,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBA,EAAE,IAAI,CAAC,SAAS,sFAAiF,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBA,EAAE,IAAI,CAAC,SAAS,uFAAkF,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+FAA+F,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,sCAAsC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8UAAoU,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4GAA4G,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,mCAAgDE,EAAEC,EAAE,CAAC,KAAK,gEAAgE,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,kBAAkB,CAAC,CAAC,CAAC,EAAE,iRAAuQ,CAAC,CAAC,EAAeF,EAAE,MAAM,CAAC,IAAI,0EAA0E,UAAU,eAAe,OAAO,MAAM,IAAI,sEAAsE,OAAO,wKAAwK,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,KAAK,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iEAA4D,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,4CAA4C,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,uEAAoFE,EAAE,SAAS,CAAC,SAAS,uBAAuB,CAAC,EAAE,sBAAmCA,EAAE,SAAS,CAAC,SAAS,uBAAuB,CAAC,EAAE,sDAAsD,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,iBAA8BE,EAAE,SAAS,CAAC,SAAS,4CAA4C,CAAC,EAAE,iKAAiK,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,wDAAwD,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8QAA8Q,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,2DAA2D,CAAC,EAAE,mSAAyR,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,mEAAmE,CAAC,EAAE,+KAA+K,CAAC,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,+FAA+F,UAAU,eAAe,OAAO,MAAM,IAAI,sEAAsE,OAAO,oQAAoQ,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,KAAK,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,8DAA8D,CAAC,EAAE,sMAAiM,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,4BAA4B,CAAC,EAAE,+OAA+O,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,sCAAsC,CAAC,EAAE,6LAA6L,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,sFAAsF,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,kCAAkC,CAAC,EAAE,mJAAgKA,EAAEC,EAAE,CAAC,KAAK,6DAA6D,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,wDAAwD,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeJ,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,uEAAuE,CAAC,EAAE,0KAAuLA,EAAEC,EAAE,CAAC,KAAK,mDAAmD,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,4CAA4C,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeJ,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,wCAAwC,CAAC,EAAE,0JAAuKA,EAAEC,EAAE,CAAC,KAAK,qDAAqD,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,+BAA+B,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,yCAAyC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0fAA0f,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qEAAqE,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,mBAAmB,CAAC,EAAE,IAAiBA,EAAEC,EAAE,CAAC,KAAK,2CAA2C,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,yBAAyB,CAAC,CAAC,CAAC,EAAE,0TAAuUF,EAAE,SAAS,CAAC,SAAS,iBAAiB,CAAC,EAAE,mDAAgEA,EAAE,SAAS,CAAC,SAAS,iBAAiB,CAAC,EAAE,yHAAyH,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,wCAAwC,CAAC,EAAE,4YAAuY,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,+BAA+B,CAAC,EAAE,oEAAiFA,EAAEC,EAAE,CAAC,KAAK,mCAAmC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,QAAQ,CAAC,CAAC,CAAC,EAAE,gGAA6GF,EAAEC,EAAE,CAAC,KAAK,wEAAwE,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,qBAAqB,CAAC,CAAC,CAAC,EAAE,qWAAqW,CAAC,CAAC,EAAeJ,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,gEAAgE,CAAC,EAAE,wMAAmM,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,gFAAgF,CAAC,EAAE,qEAAqE,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,yCAAyC,CAAC,EAAE,+DAA+D,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,kEAAkE,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAEC,EAAE,CAAC,KAAK,kEAAkE,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAsBF,EAAE,SAAS,CAAC,SAAS,wCAAwC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAE,yNAAoN,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,2BAA2B,CAAC,EAAE,iFAA8FA,EAAEC,EAAE,CAAC,KAAK,wDAAwD,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,8CAA8C,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeJ,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,2CAA2C,CAAC,EAAE,qCAAkDA,EAAEC,EAAE,CAAC,KAAK,kEAAkE,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,WAAW,CAAC,CAAC,CAAC,EAAE,qJAAqJ,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,0CAA0C,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iTAAiT,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,MAAM,CAAC,EAAeA,EAAEC,EAAE,CAAC,KAAK,mDAAmD,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAsBF,EAAE,SAAS,CAAC,SAAS,gBAAgB,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,SAAS,CAAC,SAAS,GAAG,CAAC,EAAE,kVAAwU,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,MAAM,CAAC,EAAeA,EAAEC,EAAE,CAAC,KAAK,iGAAiG,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAsBF,EAAE,SAAS,CAAC,SAAS,mBAAmB,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,SAAS,CAAC,SAAS,GAAG,CAAC,EAAE,0bAA0b,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,mEAAmE,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAEC,EAAE,CAAC,KAAK,yDAAyD,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAsBF,EAAE,SAAS,CAAC,SAAS,0CAA0C,CAAC,CAAC,CAAC,CAAC,CAAC,EAAE,4GAA4G,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAEC,EAAE,CAAC,KAAK,8DAA8D,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAsBF,EAAE,SAAS,CAAC,SAAS,sCAAsC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAE,yGAAyG,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAEC,EAAE,CAAC,KAAK,yDAAyD,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAsBF,EAAE,SAAS,CAAC,SAAS,yCAAyC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAE,wJAAwJ,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,cAAc,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,0DAAuEE,EAAEC,EAAE,CAAC,KAAK,yCAAyC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,qBAAqB,CAAC,CAAC,CAAC,EAAE,cAA2BF,EAAEC,EAAE,CAAC,KAAK,uDAAuD,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,oBAAoB,CAAC,CAAC,CAAC,EAAE,gBAAgB,CAAC,CAAC,CAAC,CAAC,CAAC,EACtooIa,EAAqB,CAAC,QAAU,CAAC,UAAY,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,UAAY,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,UAAY,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,SAAW,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,UAAY,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,UAAY,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,UAAY,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,UAAY,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,UAAY,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,mBAAqB,CAAC,KAAO,UAAU,CAAC,CAAC",
  "names": ["richText", "u", "x", "p", "Link", "motion", "ComponentPresetsConsumer", "t", "CodeBlock_default", "richText1", "Youtube", "richText2", "richText3", "richText4", "richText5", "richText6", "richText7", "richText8", "__FramerMetadata__"]
}