{
  "version": 3,
  "sources": ["ssg:https://framerusercontent.com/modules/YVgynkVt96a0ES9Gk8yz/e2OiwG21hMIpRxQ1FbTF/eo4RAmtig-27.js"],
  "sourcesContent": ["import{jsx as e,jsxs as t}from\"react/jsx-runtime\";import{Link as a}from\"framer\";import{motion as n}from\"framer-motion\";import*as i from\"react\";export const richText=/*#__PURE__*/t(i.Fragment,{children:[/*#__PURE__*/t(\"h5\",{children:[\"Read the recent case study to learn how Toloka effectively \",/*#__PURE__*/e(a,{href:\"https://toloka.ai/blog/toloka-helps-servicenow-increase-evaluation-throughput-multiple-times/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!0,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"scaled human evaluation\"})}),\" by processing thousands of multi-turn dialogues daily, demonstrating significant improvements in throughput and efficiency. By leveraging a large pool of domain experts, Toloka\u2019s approach optimizes the quality and speed of evaluations necessary for training, fine-tuning, and validating. This framework ensures consistency and accuracy in assessment and supports the dynamic needs of high-throughput environments, making it ideal for ongoing, intensive model evaluations.\"]}),/*#__PURE__*/e(\"h2\",{children:\"Introduction\"}),/*#__PURE__*/e(\"p\",{children:\"As advanced Natural Language Processing (NLP) models, Large Language Models (LLMs) are the talk of the town these days given their uncanny ability to generate human-like text by leveraging large quantities of pre-existing language data and multifaceted machine learning algorithms. Think GPT-3, BERT, and XLNet. However, as we become increasingly dependent on these models, finding ways to effectively evaluate them and safeguard against potential threats is more imperative than ever.\"}),/*#__PURE__*/e(\"p\",{children:\"In this article, we cover in detail the importance of evaluating large language models to improve their performance and why human insight is critical to this process. While much of the information out there indicates that incorporating human input can be costly, with the help of Toloka\u2019s crowd contributors, it can be done much faster and in a more cost-effective way.\"}),/*#__PURE__*/e(\"h2\",{children:\"The importance of evaluating LLMs\"}),/*#__PURE__*/e(\"p\",{children:\"As large language models become more deeply embedded in machine learning, having thorough evaluation methods and comprehensive regulation frameworks in place can help ensure that AI is serving everyone\u2019s best interests.\"}),/*#__PURE__*/e(\"p\",{children:\"The number of large language models on the market has increased exponentially, but there\u2019s no set standard by which to evaluate them. As a result, it\u2019s becoming significantly more challenging to both evaluate models and determine which one is the best \u2014 and safest. That\u2019s why we need a reliable, comprehensive evaluation framework by which to precisely assess the quality of large language models. A standardized framework will not only help regulators evaluate the safety, accuracy, and reliability of a model, but it will also hold developers accountable when releasing these models \u2014 rather than simply slapping a disclaimer on their product. Moreover, large language model users will be able to see whether they need to fine-tune a model on supplementary data.\"}),/*#__PURE__*/e(\"img\",{alt:\"The importance of evaluating LLMs\",className:\"framer-image\",height:\"620\",src:\"https://framerusercontent.com/images/wBqAtJFnUTDUONiloaOYcOe8l3w.jpeg\",srcSet:\"https://framerusercontent.com/images/wBqAtJFnUTDUONiloaOYcOe8l3w.jpeg?scale-down-to=512 512w,https://framerusercontent.com/images/wBqAtJFnUTDUONiloaOYcOe8l3w.jpeg?scale-down-to=1024 1024w,https://framerusercontent.com/images/wBqAtJFnUTDUONiloaOYcOe8l3w.jpeg?scale-down-to=2048 2048w,https://framerusercontent.com/images/wBqAtJFnUTDUONiloaOYcOe8l3w.jpeg 2100w\",style:{aspectRatio:\"2100 / 1240\"},width:\"1050\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"em\",{children:\"Source: \"}),/*#__PURE__*/e(a,{href:\"https://www.analyticsvidhya.com/blog/2023/05/how-to-evaluate-a-large-language-model-llm/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!0,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:/*#__PURE__*/e(\"em\",{children:\"https://www.analyticsvidhya.com/blog/2023/05/how-to-evaluate-a-large-language-model-llm/\"})})})]}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"br\",{className:\"trailing-break\"})}),/*#__PURE__*/e(\"h3\",{children:\"Why evaluate LLMs?\"}),/*#__PURE__*/e(\"p\",{children:\"It\u2019s vital to evaluate large language models to assess their quality and usefulness in different applications. We\u2019ve outlined some real-life examples of why it\u2019s important to evaluate large language models:\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"Assessing performance\"})}),/*#__PURE__*/e(\"p\",{children:\"A company must choose between several models for its foundational enterprise generative model based on relevance, accuracy, fluency, and more. The given LLMs must be assessed according to their ability to generate text and respond to input.\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"Comparing models\"})}),/*#__PURE__*/e(\"p\",{children:\"A company selects and fine-tunes a model for better performance on industry-specific tasks by carrying out a comparative evaluation of LLMs to choose the one that best suits their needs.\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"Detecting and preventing bias\"})}),/*#__PURE__*/e(\"p\",{children:\"By having a holistic evaluation framework, companies can work to detect and eliminate bias found in large language model outputs and training data to create fairer outcomes.\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"Building user trust\"})}),/*#__PURE__*/e(\"p\",{children:\"Evaluating user feedback and trust in the answers provided by LLMs is paramount to building reliable systems that are aligned with user expectations and societal norms.\"}),/*#__PURE__*/e(\"h2\",{children:\"Methods for evaluating LLMs\"}),/*#__PURE__*/e(\"p\",{children:\"Evaluating an LLM\u2019s performance includes measuring features such as language fluency, coherence, context comprehension, speech recognition, fact-based accuracy, and its aptitude to generate pertinent and significant responses. Metrics like perplexity, BLEU score, and human evaluations can be used to assess and compare language model performance.\"}),/*#__PURE__*/e(\"p\",{children:\"This process requires a closer look at the following factors:\"}),/*#__PURE__*/t(\"ol\",{style:{\"--list-style-type\":\"unset\"},children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Authenticity and accuracy\"}),\" \u2014 How close a model\u2019s outputs are to the correct answers or expected results. Precision, recall, and F1 scores point to a model\u2019s accuracy level.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Speed\"}),\" \u2014 The speed at which a model can produce outputs is essential, especially in cases where rapid response times are critical.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Grammar and legibility\"}),\" \u2014 LLMs need to be able to generate text that is readable with correct grammar usage and sentence structure.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Fairness and equality\"}),\" \u2014 Making sure a model doesn\u2019t have any bias toward specific groups and doesn\u2019t generate discriminatory outcomes based on race, gender, and other factors.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Robustness\"}),\" \u2014 A model\u2019s resistance to adversarial attacks and its ability to perform well across various conditions.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Explainability\"}),\" \u2014 LLMs must substantiate their predictions and outputs to build trust among users and provide accountability. They need to be able to provide sources and references.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Safety and responsibility\"}),\" \u2014 Guardrails are essential for AI models to make them safer and more reliable (more on this later).\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Understanding context\"}),\" \u2014 The model needs to be able to understand the context surrounding a given question since that same question can be asked in different ways.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Text operations\"}),\" \u2014 LLMs should be able to carry out text classification, summarization, translation, and other basic functions.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Generalization\"}),\" \u2014 How well a model manages unseen data or scenarios.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Intelligence Quotient (IQ)\"}),\" \u2014 Just as humans are evaluated based on their IQs, so too can machines be.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Emotional Quotient (EQ)\"}),\" \u2014 The same applies to emotional intelligence.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Versatility\"}),\" \u2014 The number of domains, industries, and languages a model can work with.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Real-time updates\"}),\" \u2014 Models that are updated in real-time can produce enhanced outputs.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Cost\"}),\" \u2014 Cost needs to be factored in when considering the development and application of these models.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Consistency\"}),\" \u2014 Comparable prompts should ideally generate identical or similar responses from models.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Prompt Engineering\"}),\" \u2014 The engineering needed to obtain the best response can also be leveraged to compare two models.\"]})})]}),/*#__PURE__*/e(\"p\",{children:\"By ensuring the points above are taken into account in our evaluation of LLMs, we can maximize the potential of these formidable models.\"}),/*#__PURE__*/e(\"p\",{children:\"Here are five of the most common methods used to evaluate LLMs today:\"}),/*#__PURE__*/t(\"ol\",{style:{\"--list-style-type\":\"unset\"},children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Perplexity\"}),\" \u2014 calculates how well a model can predict a sample of text where lower values equal higher performance.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Human evaluation\"}),\" \u2014 human evaluators help to review the overall quality of a language model\u2019s output based on certain criteria such as relevance, fluency, and coherence.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Bilingual Evaluation Understudy (BLEU)\"}),\" \u2014 is a metric comparing generated text with a reference one.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Recall-Oriented Understudy for Gissing Evaluation (ROUGE)\"}),\" \u2014 a metric that shows how well a model generates summaries compared to reference summaries.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Diversity\"}),\" \u2014 metrics that evaluate the multiplicity and individuality of responses generated by a model, where higher scores equal better outputs.\"]})})]}),/*#__PURE__*/e(\"h3\",{children:\"Zero-shot evaluation\"}),/*#__PURE__*/e(\"p\",{children:\"Moreover, zero-shot evaluation can now be used for large language models. A cost-effective and well-known way to assess the performance of LLMs, zero-shot evaluation measures the probability that a trained model will produce a particular set of tokens without needing any labeled training data.\"}),/*#__PURE__*/e(\"img\",{alt:\"Zero-shot evaluation\",className:\"framer-image\",height:\"200\",src:\"https://framerusercontent.com/images/TeXIZ4GUWXINbdyAotTtVzkoFM.jpeg\",srcSet:\"https://framerusercontent.com/images/TeXIZ4GUWXINbdyAotTtVzkoFM.jpeg?scale-down-to=512 512w,https://framerusercontent.com/images/TeXIZ4GUWXINbdyAotTtVzkoFM.jpeg?scale-down-to=1024 1024w,https://framerusercontent.com/images/TeXIZ4GUWXINbdyAotTtVzkoFM.jpeg 1252w\",style:{aspectRatio:\"1252 / 400\"},width:\"626\"}),/*#__PURE__*/e(\"h2\",{children:\"Challenges with current model evaluation techniques\"}),/*#__PURE__*/e(\"p\",{children:\"Multiple frameworks have been developed to evaluate LLMs, such as Big Bench, GLUE Benchmark, SuperGLUE Benchmark, and others, each of them focusing on its own domain. For example, Big Bench takes into account generalization capabilities when evaluating LLMs, GLUE Benchmark considers grammar, paraphrasing, text similarity, inference, and several other factors, while SuperGLUE Benchmark looks at Natural Language Understanding, reasoning, reading comprehension, and how well an LLM understands complex sentences beyond training data, among other considerations. See the table below for an overview of the current evaluation frameworks and the factors they consider when evaluating LLMs.\"}),/*#__PURE__*/e(\"img\",{alt:\"\",className:\"framer-image\",height:\"778\",src:\"https://framerusercontent.com/images/NujCTBSe3R9onIJkcg42c65AD24.png\",srcSet:\"https://framerusercontent.com/images/NujCTBSe3R9onIJkcg42c65AD24.png?scale-down-to=1024 906w,https://framerusercontent.com/images/NujCTBSe3R9onIJkcg42c65AD24.png 1378w\",style:{aspectRatio:\"1378 / 1556\"},width:\"689\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"em\",{children:\"Source: \"}),/*#__PURE__*/e(a,{href:\"https://www.analyticsvidhya.com/blog/2023/05/how-to-evaluate-a-large-language-model-llm/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:/*#__PURE__*/e(\"em\",{children:\"Analytics Vidhya, Table of the Major Existing Evaluation Frameworks\"})})})]}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"br\",{className:\"trailing-break\"})}),/*#__PURE__*/e(\"p\",{children:\"Apart from OpenAI\u2019s Moderation API, most of the ones available today don\u2019t take safety into account as a factor for their evaluation results. And not one of them is inclusive enough to be self-reliant.\"}),/*#__PURE__*/e(\"p\",{children:\"There are still quite a lot of factors to overcome when it comes to evaluating large language models. Some of the most frequent challenges include:\"}),/*#__PURE__*/t(\"ul\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Over-dependence on perplexity\"}),\" \u2014 For all the good that perplexity does, it still has its limitations, namely, it can\u2019t determine context understanding, relevance, or consistency.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Human subjectivity\"}),\" \u2014 Despite the fact that human input is critical for assessing LLMs, humans are nevertheless susceptible to subconscious bias and differing views.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Insufficient reference data\"}),\" \u2014 Getting a hold of quality reference data poses a significant challenge, especially when you\u2019re seeking multiple responses for tasks.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Limited diversity metrics\"}),\" \u2014 Current evaluation methods focus solely on accuracy and applicability of LLM outputs rather than diversity and creativity.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Inapplicability to real-life cases\"}),\" \u2014 The real world is filled with complexities and challenges that current evaluation methods are not equipped to handle.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Adverse attacks\"}),\" \u2014 LLMs are easy targets for deception, misuse, and manipulation via insidiously constructed input, and current methods don\u2019t factor these attacks in.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Data leakage\"}),\" \u2014 Commonly occurs from the test set of datasets into the training set; details of a dataset can also go missing.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Coverage issues\"}),\" \u2014 Evaluation datasets don\u2019t always have wide-ranging coverage of all the ways in which a task can be evaluated, which could show up as accuracy, variability, sample size, or robustness issues.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"False correlations or duplicate test samples\"}),\" \u2014 The evaluation sets for many tasks often contain a \u201Cshortcut\u201D solution, which can create a false illusion that a test sample is a reliable representation upon which to evaluate a particular task.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Partitioning and phrasing\"}),\" \u2014 Including evaluation dataset splits that come with an alternate application of the same issue.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Random seed\"}),\" \u2014 Upon which neural network outputs depend; this could lead to inaccurate results.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Precision versus recall\"}),\" \u2014 False positives and false negatives are not identical for each task and can be harmful in certain contexts.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Unexplained decisions\"}),\" \u2014 When it comes to getting rid of or keeping data, being able to explain the given thresholds is essential.\"]})})]}),/*#__PURE__*/e(\"h2\",{children:\"Best practices for assessing LLMs\"}),/*#__PURE__*/e(\"p\",{children:\"The development of large language models has forever changed the field of Natural Language Processing and the fulfillment of natural language tasks, but we still need a comprehensive and standardized evaluation framework to assess the quality, accuracy, and reliability of these models. Current frameworks offer valuable insights, but they lack a uniform, more comprehensive evaluation approach and don\u2019t factor in safety concerns.\"}),/*#__PURE__*/e(\"h3\",{children:\"Setting up an LLM\"}),/*#__PURE__*/e(\"p\",{children:\"Before we dive into the best practices for evaluating LLMs, let\u2019s take a look at the five steps required for setting up an LLM:\"}),/*#__PURE__*/t(\"ol\",{style:{\"--list-style-type\":\"unset\"},children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Selecting benchmarks\"}),\" \u2014 Depicting real-life cases on an extensive range of language-related challenges such as language modeling, machine translation, and so on.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Preparing datasets\"}),\" \u2014 These must be large enough to obtain a high-quality and objective evaluation, including training, validation, and test sets.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Training and fine-tuning models\"}),\" \u2014 For example, pre-training on a large text corpus and then fine-tuning on benchmark datasets specific to a certain task.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Evaluating models\"}),\" \u2014 On benchmark tasks using predefined evaluation metrics, the results of which illustrate the overall performance of the models.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Performing a comparative analysis\"}),\" \u2014 On the overall performance of LLMs and models trained for each benchmark task according to specific metrics.\"]})})]}),/*#__PURE__*/e(\"h3\",{children:\"Best practices\"}),/*#__PURE__*/e(\"p\",{children:\"As mentioned above, a dependable evaluation framework should factor in authenticity, speed, context, and more, which will help developers release LLMs responsibly. Collaboration among key stakeholders and regulators is key to creating a reliable and comprehensive evaluation framework.\"}),/*#__PURE__*/e(\"p\",{children:\"In the meantime, we've outlined several best practices for circumventing some of the challenges of evaluating large language models:\"}),/*#__PURE__*/t(\"ul\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Including varied metrics\"}),\" \u2014 Additional metrics, such as fluency, relevance, context, coherence, and diversity, lead to a more well-rounded review of large language model output.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Improving human evaluation\"}),\" \u2014 Creating guidelines, standards, and benchmarks can help make human evaluation more reliable, consistent, and objective. Crowdsourcing can be particularly helpful in this case for obtaining diverse feedback on large projects.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Incorporating diverse reference data\"}),\" \u2014 Reference data that comes from multiple sources encompassing diverse views can improve evaluation methods for model outputs.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Implementing diversity indicators\"}),\" \u2014 Incorporating these metrics can prompt models to develop strategies that generate diverse responses, from which uniqueness can be evaluated.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Using real-life scenarios\"}),\" \u2014 Incorporating real-world use cases and tasks, as well as industry- or domain-specific benchmarks, can provide a more realistic evaluation.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Assessing robustness\"}),\" \u2014 Evaluation methods that assess a model\u2019s robustness against adversarial attacks can add to the safety and dependability of LLMs.\"]})})]}),/*#__PURE__*/e(\"h2\",{children:\"Human input in LLM evaluation\"}),/*#__PURE__*/e(\"p\",{children:\"Human input plays a key role in making large language models safer and more reliable. One of the main ways in which human input is valuable is in the establishment and enforcement of guardrails.\"}),/*#__PURE__*/e(\"h3\",{children:\"What are guardrails and how do they apply to AI?\"}),/*#__PURE__*/e(\"p\",{children:\"While AI has the potential to simplify and improve our lives in many ways, it also comes with inherent risks and some real dangers, which cannot be ignored. That\u2019s where guardrails come into play. In short, guardrails are essentially defenses \u2014 policies, strategies, mechanisms \u2014 that are put into place to ensure the ethical and responsible application of AI-based technologies. Guardrails are designed to preclude misuse, defend user privacy, and encourage transparency and equality. Without them, it could be a world gone mad!\"}),/*#__PURE__*/e(\"p\",{children:\"With technology evolving at the speed of light, let\u2019s take a closer look at how guardrails work and why they\u2019re so important in effectively managing AI.\"}),/*#__PURE__*/e(\"h3\",{children:\"How guardrails work\"}),/*#__PURE__*/e(\"p\",{children:\"Given the nature of the risks posed by AI \u2014 such as ethical and privacy concerns, bias and prejudice, as well as environmental and computational costs \u2014 guardrails work to prevent many of these challenges by establishing a set of guidelines and controls. Imagine if someone were to take unauthorized medical advice from an AI system. The results could be disastrous, and even deadly! That\u2019s why building controls into AI systems that determine acceptable behavior and responses is so critical. We need them to ensure that AI technologies work within our societal norms and standards.\"}),/*#__PURE__*/e(\"p\",{children:\"Below are some examples of guardrails:\"}),/*#__PURE__*/t(\"ul\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Mechanisms that protect individual privacy and confidentiality of data\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Procedures that avert biased results\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Policies that regulate AI systems in accordance with ethical and legal benchmarks\"})})]}),/*#__PURE__*/e(\"p\",{children:\"Transparency and accountability go hand-in-hand when it comes to the importance of guardrails in AI systems. You need transparency to ensure that responses generated by these systems can be explained and mistakes can be found and fixed. Guardrails can also ensure that human input is taken into account in regard to important decisions in areas like medical care and self-driving cars. Basically, the point of guardrails is to make sure that AI technologies are being used for the betterment of humanity and are serving us in helpful and positive ways without posing significant risks, harm, or danger.\"}),/*#__PURE__*/e(\"h3\",{children:\"Why they\u2019re so important in evaluation metrics\"}),/*#__PURE__*/e(\"p\",{children:\"As AI systems continue to develop, they are endowed with an ever-increasing amount of responsibility and tasked with more important decisions. That means they have the power to make a greater impact, but also to inflict more damage if misused.\"}),/*#__PURE__*/e(\"p\",{children:\"Here are some of the key benefits of guardrails:\"}),/*#__PURE__*/t(\"ul\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Protect against the misuse of AI technologies\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Promote equality and prevent bias\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Preserve public trust in AI technologies\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Comply with legal and regulatory standards\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Facilitate human oversight in AI systems\"})})]}),/*#__PURE__*/e(\"p\",{children:\"Not only do guardrails ensure the safe, responsible, and ethical use of AI technologies, but they can also be used to detect and eliminate bias, promote openness and accountability, comply with local laws, and most importantly, implement human oversight to ensure that AI doesn\u2019t replace human decision-making.\"}),/*#__PURE__*/e(\"h3\",{children:\"How guardrails can guarantee human input\"}),/*#__PURE__*/e(\"p\",{children:\"By making sure that humans are involved in key decision-making processes, guardrails can help guarantee that AI technologies stay under our control and that their outputs follow our societal norms and values. That\u2019s why keeping humans in the loop (HITL) \u2014 where both humans and machines work together to create machine learning models \u2014 is imperative to making sure that AI doesn\u2019t go off the rails. HITL ensures that people are involved at every stage of the algorithm cycle, including training, tuning, and testing. As a result, AI outputs become safer and more reliable.\"}),/*#__PURE__*/e(\"h3\",{children:\"Overcoming the challenges of deploying guardrails\"}),/*#__PURE__*/e(\"p\",{children:\"Implementing guardrails is no easy task, however, there are a couple of ways to overcome the technical complexities involved in this process, namely:\"}),/*#__PURE__*/t(\"ul\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Adversarial training\"}),\" \u2014 Training an AI model on a series of contradictory examples meant to fool the model into making mistakes, thereby teaching it to detect rigged inputs.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Differential privacy\"}),\" \u2014 Adding noise (data that\u2019s been corrupted, distorted, has a low signal-to-noise ratio, or contains a lot of meaningless information) to training data to hide individual data points, which protects confidential information as well as the privacy of individuals.\"]})})]}),/*#__PURE__*/e(\"p\",{children:\"What\u2019s more, the actual process of setting up guardrails poses a challenge in itself. You need predefined roles and responsibilities within your company, along with board oversight and tools for recording and surveying AI system outputs, not to mention legal and regulatory compliance measures. All this requires continual monitoring, assessment, and tweaking \u2014 it\u2019s by no means an easy or static process! As AI evolves, so too will the guardrails needed to keep us safe.\"}),/*#__PURE__*/e(\"h2\",{children:\"Crowdsourcing in model evaluation\"}),/*#__PURE__*/e(\"p\",{children:\"Human evaluation can be time-consuming and expensive, particularly when it comes to large-scale evaluations. That\u2019s where crowdsourcing comes into play.\"}),/*#__PURE__*/e(\"p\",{children:\"As mentioned above, crowdsourcing can be particularly helpful in improving human evaluation by obtaining diverse feedback on large projects. Human evaluation via crowdsourcing involves crowd participants evaluating the output or performance of a large language model within a given context. By leveraging the capabilities of millions of crowd contributors, you can obtain qualitative feedback and identify subtle nuances that traditional LLM-based evaluation might otherwise miss.\"}),/*#__PURE__*/e(\"p\",{children:\"Overall, crowdsourcing can be a great solution for evaluating large language model projects. At Toloka, we help clients solve real-life business challenges and offer best-in-class solutions for fine-tuning and evaluating language models. We help clients monitor quality in production applications and obtain unbiased feedback for model improvement using the collective power of the crowd.\"}),/*#__PURE__*/e(\"h2\",{children:\"Key takeaways\"}),/*#__PURE__*/e(\"p\",{children:\"Large language models are revolutionizing many sectors from medicine and banking to academia and the media. As these models become increasingly sophisticated, regulatory oversight is critical. Data privacy, accountability, transparency, and elimination of bias all need to be taken into account. LLMs must also be able to substantiate and provide sources for their outputs and decisions in order to build public trust. As such, public involvement can be an effective way to ensure that these systems comply with societal norms and expectations.\"}),/*#__PURE__*/e(\"p\",{children:\"To unlock the full potential of such a language model, we need to incorporate suitable evaluation methods that cover accuracy, fairness, robustness, explainability, and generalization. That way, we can make the most of their strengths and successfully address their weaknesses.\"}),/*#__PURE__*/e(\"p\",{children:\"We invite you to read through our blog to learn more about large language models and how Toloka can help in their evaluation process.\"}),/*#__PURE__*/e(\"h2\",{children:\"About Toloka\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"em\",{children:\"Toloka is a European company based in Amsterdam, the Netherlands that provides data for Generative AI development. Toloka empowers businesses to build high quality, safe, and responsible AI. We are the trusted data partner for all stages of AI development from training to evaluation. Toloka has over a decade of experience supporting clients with its unique methodology and optimal combination of machine learning technology and human expertise, offering the highest quality and scalability in the market.\"})})]});export const richText1=/*#__PURE__*/t(i.Fragment,{children:[/*#__PURE__*/t(\"p\",{children:[\"We at the Toloka ML team continue researching and comparing approaches to the text classification problem under different conditions, and here we present another experiment particularly focusing on the performance of different NLP models when applied to extra-small datasets. In our previous articles, we provided a \",/*#__PURE__*/e(a,{href:\"https://medium.com/toloka/choosing-the-best-architecture-for-your-text-classification-task-aee30ecc7870\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"brief overview of potential solutions\"})}),\" and \",/*#__PURE__*/e(a,{href:\"https://www.kdnuggets.com/2023/04/best-architecture-text-classification-task-benchmarking-options.html\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"compared classical models with large language models (LLMs) for a specific task\"})}),'. However, those comparisons were based on a \"regular\" dataset containing a sufficient number of data points to build a reliable classifier. In real-world scenarios, one may encounter situations where limited data is available, or human labeling has not yet been performed. Intuitively, LLMs such as GPT-3 or ChatGPT might outperform smaller models due to their extensive \"knowledge\u201D. ']}),/*#__PURE__*/e(\"p\",{children:\"To investigate this hypothesis, we will create an artificially small dataset by extracting a portion of a larger one and compare several approaches. We will fine-tune the RoBERTa base model, employ ChatGPT for few-shot classification, and also fine-tune the GPT-3 Babbage model.\"}),/*#__PURE__*/e(\"h2\",{children:\"The Dataset\"}),/*#__PURE__*/t(\"p\",{children:[\"To evaluate the comprehension capabilities of various models, we selected a multiclass dataset comprising scientific article abstracts, with the task of determining each article's domain. We opted for the \",/*#__PURE__*/e(a,{href:\"https://paperswithcode.com/dataset/web-of-science-dataset\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"WOS-11967\"})}),\" [1] dataset, which contains 11,967 documents with 35 categories which include 7 parent categories: medical domain, psychology, computer science, biochemistry, electrical engineering, civil sciences, and mechanical engineering. We sampled 10,000 data points and focused solely on the parent categories for our analysis.\"]}),/*#__PURE__*/e(\"p\",{children:\"While the dataset is not perfectly balanced, the class distribution is reasonably proportional, allowing for the possibility of achieving satisfactory results across all classes. The class distribution is illustrated in the image below.\"}),/*#__PURE__*/e(\"img\",{alt:\"The class distribution of the sample of the WOS-11967 dataset\",className:\"framer-image\",height:\"775\",src:\"https://framerusercontent.com/images/ZcDDe23hnleaC8iUDyhTAQGHnU.webp\",srcSet:\"https://framerusercontent.com/images/ZcDDe23hnleaC8iUDyhTAQGHnU.webp?scale-down-to=512 512w,https://framerusercontent.com/images/ZcDDe23hnleaC8iUDyhTAQGHnU.webp?scale-down-to=1024 1024w,https://framerusercontent.com/images/ZcDDe23hnleaC8iUDyhTAQGHnU.webp?scale-down-to=2048 2048w,https://framerusercontent.com/images/ZcDDe23hnleaC8iUDyhTAQGHnU.webp 2560w\",style:{aspectRatio:\"2560 / 1551\"},width:\"1280\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"em\",{children:\"The class distribution of the sample of the \"}),/*#__PURE__*/e(a,{href:\"https://paperswithcode.com/dataset/web-of-science-dataset\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!0,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:/*#__PURE__*/e(\"em\",{children:\"WOS-11967\"})})}),/*#__PURE__*/e(\"em\",{children:\" dataset\"})]}),/*#__PURE__*/e(\"p\",{children:\"Upon manual analysis, we observed that determining the domain of some abstracts is relatively straightforward, while in other cases, the task becomes more challenging. For instance, computer science articles may discuss mathematical topics, or psychology articles might employ numerous medical and biochemical terms and abbreviations, making it difficult to distinguish them from biochemistry or medical domains. The abstracts also vary significantly in length, with a mean of 274 tokens (ChatGPT tokens) and a standard deviation of 115 tokens.\"}),/*#__PURE__*/e(\"p\",{children:\"To simulate scenarios involving extra-small datasets, we performed a train-test split on the corpora, allocating a small number of samples to the training set. We repeated this process three times with different training set sizes to evaluate the performance changes in the models based on the available training data. We created three splits for our experiment: WOS-11967-s200 (200 samples in the training set, 9,800 samples in the test set), WOS-11967-s500 (500 / 9,500), and WOS-11967-s2000 (2,000 / 8,000).\"}),/*#__PURE__*/e(\"p\",{children:\"Now, let's examine the results obtained using different models to tackle these problems.\"}),/*#__PURE__*/e(\"h2\",{children:\"Regular fine-tuning with RoBERTa\"}),/*#__PURE__*/t(\"p\",{children:[\"For our baseline, we selected the \",/*#__PURE__*/e(a,{href:\"https://huggingface.co/roberta-base\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"RoBERTa base\"})}),\" model [2] and fine-tuned it on these 3 datasets. and fine-tuned it on the three datasets mentioned earlier. We used the same hyperparameter configuration for each run (batch size of 32, learning rate of 3e-5, linear scheduler with warmup, and 256-token window), along with early stopping to prevent overfitting. We obtained the following results:\"]}),/*#__PURE__*/e(\"img\",{alt:\"Regular fine-tuning with RoBERTa\",className:\"framer-image\",height:\"310\",src:\"https://framerusercontent.com/images/C8ogfTUTERczeVutUkzy26q5b4.png\",srcSet:\"https://framerusercontent.com/images/C8ogfTUTERczeVutUkzy26q5b4.png?scale-down-to=512 512w,https://framerusercontent.com/images/C8ogfTUTERczeVutUkzy26q5b4.png?scale-down-to=1024 1024w,https://framerusercontent.com/images/C8ogfTUTERczeVutUkzy26q5b4.png?scale-down-to=2048 2048w,https://framerusercontent.com/images/C8ogfTUTERczeVutUkzy26q5b4.png 2098w\",style:{aspectRatio:\"2098 / 620\"},width:\"1049\"}),/*#__PURE__*/e(\"p\",{children:\"We can clearly see that 200 samples are insufficient for extracting all the necessary patterns and information to accurately classify the abstracts. The lower macro-average F1 score also indicates that the model underperforms on under-represented classes like Mechanical Engineering, suggesting that having only a few samples from a particular class is inadequate. As anticipated, the model's performance improves as the amount of available data increases, ultimately resulting in a fairly robust performance for multiclass classification across seven classes.\"}),/*#__PURE__*/e(\"h2\",{children:\"Few-shot with ChatGPT\"}),/*#__PURE__*/e(\"p\",{children:\"The second approach we explored was few-shot classification using ChatGPT. This method differs significantly from traditional classification, as it doesn't involve training a model per se; instead, we engineer the input prompt to achieve optimal performance. However, it's impossible to feed all 200 samples into the model due to its 4096-token context size limit. So, given the measurements above, we could only present around 14 abstracts to the model, and that number is further reduced when considering the tokens used for instructions and delimiters.\"}),/*#__PURE__*/t(\"p\",{children:[\"Initially, we employed the \",/*#__PURE__*/e(\"code\",{children:\"system\"}),' role for instructions and provided a single example per class to guide the model\\'s response. We simplified the class names to single tokens while retaining their meaning, making it easier for the model to select the appropriate category and limiting the output to a single token. For instance, \"Biochemistry\" became \"Bio,\" and \"Computer Science\" became \"Computer.\" Additionally, we restricted token generation by providing a list of classes to choose from and instructing the model to return the \"Unknown\" token if it is unsure about the category.']}),/*#__PURE__*/e(\"p\",{children:'Unfortunately, the performance achieved with this method was inferior compared to the RoBERTa model trained on just 200 samples. We also noticed that the model\\'s classification ability heavily depends on the supplied prompt. Modifying a single sentence could either improve or worsen the metrics. In some cases, ChatGPT missed categories despite explicit instructions not to do so (which could be a drawback of our prompt formulation). In a few edge cases, it produced categories not listed in the instruction but actually described the articles\\' domains, such as \"Math\" or \"Chemistry.\" It\\'s unclear whether these flaws should be attributed to the model or the dataset, but according to the validation set, these categories can be effectively corrected using simple rules, like changing all instances of \"Math\" to \"Computer.\"'}),/*#__PURE__*/e(\"p\",{children:\"In pursuit of improved metrics, we attempted to utilize as much data as possible (since we still can't feed all 200 samples into the model). We devised a two-stage process: first, we asked the model to identify similarities between abstracts from a specific domain and generate summaries; second, we incorporated these summaries into the instruction to provide the model with insights about the classes and features identified by the model itself in the first stage. Essentially, this approach allowed us to feed more training data samples into the model, and it worked \u2013 we boosted metrics by approximately 10%. The following is the prompt we used to generate these summaries:\"}),/*#__PURE__*/e(\"img\",{alt:\"The prompt for ChatGPT used to extract meaningful information about article domains\",className:\"framer-image\",height:\"930\",src:\"https://framerusercontent.com/images/3e7gF5rRaNjChWQFsJKH0dI5tsc.webp\",srcSet:\"https://framerusercontent.com/images/3e7gF5rRaNjChWQFsJKH0dI5tsc.webp?scale-down-to=512 512w,https://framerusercontent.com/images/3e7gF5rRaNjChWQFsJKH0dI5tsc.webp?scale-down-to=1024 1024w,https://framerusercontent.com/images/3e7gF5rRaNjChWQFsJKH0dI5tsc.webp?scale-down-to=2048 2048w,https://framerusercontent.com/images/3e7gF5rRaNjChWQFsJKH0dI5tsc.webp 2560w\",style:{aspectRatio:\"2560 / 1861\"},width:\"1280\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"em\",{children:\"The prompt for ChatGPT used to extract meaningful information about article domains\"})}),/*#__PURE__*/e(\"p\",{children:'For each domain, we supplied approximately 7-8 abstracts, resulting in a total of around 63 distinct abstracts used to prepare the classification prompt (8 abstracts per 7 classes to build summaries, and 7 abstracts provided as examples in the actual prompt). Nevertheless, we instructed the model to respond with \"Unknown\" if uncertain about the class, and observed in the validation set that the majority of \"Unknown\" responses corresponded to Computer Science articles. We subsequently replaced all \"Unknown\" instances with the \"Computer\" class.'}),/*#__PURE__*/e(\"p\",{children:\"The resulting classification prompt looked as follows:\"}),/*#__PURE__*/e(\"img\",{alt:\"The final prompt for ChatGPT used to classify article abstracts\",className:\"framer-image\",height:\"2303\",src:\"https://framerusercontent.com/images/t5rOZQi4FVCxqFXOkYfPJbsYsyY.webp\",srcSet:\"https://framerusercontent.com/images/t5rOZQi4FVCxqFXOkYfPJbsYsyY.webp?scale-down-to=1024 569w,https://framerusercontent.com/images/t5rOZQi4FVCxqFXOkYfPJbsYsyY.webp?scale-down-to=2048 1138w,https://framerusercontent.com/images/t5rOZQi4FVCxqFXOkYfPJbsYsyY.webp?scale-down-to=4096 2276w,https://framerusercontent.com/images/t5rOZQi4FVCxqFXOkYfPJbsYsyY.webp 2560w\",style:{aspectRatio:\"2560 / 4607\"},width:\"1280\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"em\",{children:\"The final prompt for ChatGPT used to classify article abstracts\"})}),/*#__PURE__*/e(\"p\",{children:\"Once again, performance was heavily influenced by the prompt and the samples provided as examples. The model also generated several categories outside the target list, requiring manual adjustments based on the validation set. This approach yielded the following results:\"}),/*#__PURE__*/e(\"img\",{alt:\"This approach yielded the following results:\",className:\"framer-image\",height:\"190\",src:\"https://framerusercontent.com/images/JJMZr7QFVkOY69jLevhC9EwbZB8.png\",srcSet:\"https://framerusercontent.com/images/JJMZr7QFVkOY69jLevhC9EwbZB8.png?scale-down-to=512 512w,https://framerusercontent.com/images/JJMZr7QFVkOY69jLevhC9EwbZB8.png?scale-down-to=1024 1024w,https://framerusercontent.com/images/JJMZr7QFVkOY69jLevhC9EwbZB8.png?scale-down-to=2048 2048w,https://framerusercontent.com/images/JJMZr7QFVkOY69jLevhC9EwbZB8.png 2100w\",style:{aspectRatio:\"2100 / 380\"},width:\"1050\"}),/*#__PURE__*/t(\"p\",{children:[\"The performance was notably better than fine-tuning a RoBERTa model on 200 samples, and it also required fewer samples. However, as the availability of labeled data increased, RoBERTa began to outperform this approach, even with just 500 samples. We believe that further performance improvements are possible through proper prompt engineering. Some useful tips and tricks can be found in resources such as the \",/*#__PURE__*/e(a,{href:\"https://www.promptingguide.ai/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"Prompting Guide\"})}),\".\"]}),/*#__PURE__*/e(\"h2\",{children:\"Fine-tuning a GPT-3 model\"}),/*#__PURE__*/t(\"p\",{children:[\"For our final approach, we fine-tuned the GPT-3 Babbage model on the three datasets. We adhered to the dataset preparation recommendations outlined in the \",/*#__PURE__*/e(a,{href:\"https://platform.openai.com/docs/guides/fine-tuning/preparing-your-dataset\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"OpenAI guide\"})}),\" and opted for the default hyperparameters without making any specific adjustments. The training process for each dataset took approximately 20 minutes, yielding the following results:\"]}),/*#__PURE__*/e(\"img\",{alt:\"Fine-tuning a GPT-3 model\",className:\"framer-image\",height:\"311\",src:\"https://framerusercontent.com/images/jPHvfSuqnxsyIOenab9CBTXW6NA.png\",srcSet:\"https://framerusercontent.com/images/jPHvfSuqnxsyIOenab9CBTXW6NA.png?scale-down-to=512 512w,https://framerusercontent.com/images/jPHvfSuqnxsyIOenab9CBTXW6NA.png?scale-down-to=1024 1024w,https://framerusercontent.com/images/jPHvfSuqnxsyIOenab9CBTXW6NA.png?scale-down-to=2048 2048w,https://framerusercontent.com/images/jPHvfSuqnxsyIOenab9CBTXW6NA.png 2098w\",style:{aspectRatio:\"2098 / 622\"},width:\"1049\"}),/*#__PURE__*/t(\"p\",{children:[\"The fine-tuned GPT-3 model delivered impressive results even on the smallest dataset, surpassing both RoBERTa and ChatGPT. As the amount of training data increased, the performance gap between RoBERTa and the tuned GPT-3 model narrowed, raising questions about the resources and feasibility of using either option. We discussed the advantages and disadvantages of both approaches in our \",/*#__PURE__*/e(a,{href:\"https://medium.com/toloka/choosing-the-best-architecture-for-your-text-classification-task-aee30ecc7870\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"previous articles\"})}),\".\"]}),/*#__PURE__*/e(\"p\",{children:\"The key takeaway is that you shouldn\u2019t approach this problem with only the performance in mind. There are other factors like latency, budget, availability, along with legal and privacy concerns. OpenAI models are available via API and bill you on a per-token basis, while your own small model or even your own LLM can be deployed directly on a virtual machine or a kubernetes cluster, and you\u2019ll be paying only for the hardware time. The latter approach requires necessary engineering skills though, and might not fit every business.\"}),/*#__PURE__*/e(\"h2\",{children:\"Conclusions\"}),/*#__PURE__*/t(\"p\",{children:['The experiment demonstrates that our initial intuition is reflected in reality \u2013 larger models trained on more extensive data perform significantly better on extra-small datasets. With proper prompt engineering and few-shot techniques, it is possible to achieve favorable results. However, this performance difference diminishes as the dataset size increases. As illustrated in our article, \"',/*#__PURE__*/e(a,{href:\"https://www.kdnuggets.com/2023/04/best-architecture-text-classification-task-benchmarking-options.html\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"Best Architecture for Your Text Classification Task: Benchmarking Your Options\"})}),\",\u201D a domain-adapted RoBERTa model can sometimes outperform generic large language models (LLMs).\"]}),/*#__PURE__*/e(\"h2\",{children:\"Reference\"}),/*#__PURE__*/t(\"ol\",{style:{\"--framer-font-size\":\"18px\",\"--framer-text-alignment\":\"start\",\"--framer-text-color\":\"rgb(30, 33, 38)\",\"--framer-text-stroke-width\":\"0px\",\"--framer-text-transform\":\"none\",\"--list-style-type\":\"unset\"},children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[\"Kowsari K, Brown DE, Heidarysafa M, Jafari Meimandi K, Gerber MS, Barnes LE. HDLTex: Hierarchical Deep Learning for Text Classification. In: \",/*#__PURE__*/e(\"em\",{children:\"Machine Learning and Applications (ICMLA), 2017 16th IEEE International Conference On\"}),\". IEEE; 2017.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[\"Liu Y, Ott M, Goyal N, et al. RoBERTa: A Robustly Optimized BERT Pretraining Approach. \",/*#__PURE__*/e(\"em\",{children:\"CoRR\"}),\". 2019;abs/1907.11692. \",/*#__PURE__*/e(a,{href:\"http://arxiv.org/abs/1907.11692\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"http://arxiv.org/abs/1907.11692\"})})]})})]})]});export const richText2=/*#__PURE__*/t(i.Fragment,{children:[/*#__PURE__*/e(\"p\",{children:\"Automated data labeling has greatly reduced the workload of machine learning practitioners. They are the ones who truly understand that the manual labeling process is among the most tedious and labor-intensive processes in creating AI products. Collecting, preparing, and labeling generally takes up to 80 percent of the whole project.\"}),/*#__PURE__*/e(\"p\",{children:\"An automated data labeling pipeline permits human labelers to drastically minimize the time it takes to label data. Although its principal benefit is speed, auto-labeling is often not compatible with all kinds of tasks.\"}),/*#__PURE__*/e(\"p\",{children:\"Frequently it is essential to rely on human beings to achieve the finest outcome. For instance, crowdsourcing may be an effective and relatively inexpensive alternative. Below we will discuss reasons why automatic labeling is beneficial, and when it is more appropriate to turn to another type of labeling, such as human labeling via crowdsourcing.\"}),/*#__PURE__*/e(\"h2\",{children:\"Labeled data\"}),/*#__PURE__*/e(\"p\",{children:\"Labeled data is the cornerstone of all machine learning. Today, high-quality ML models cannot be generated without training data, which is comprised of datasets with labels. An important note is that labels or tags are most often assigned to the data by humans, but there are exceptions, for instance in some techniques of automated labeling, which will be additionally discussed further on.\"}),/*#__PURE__*/e(\"h2\",{children:\"Data labeling process\"}),/*#__PURE__*/e(\"p\",{children:\"The necessity for people labeling data arose as ML technology became ubiquitous. Algorithms are supposed to learn from the training dataset, and it is often impossible to obtain it until a human creates it.\"}),/*#__PURE__*/e(\"p\",{children:\"How does data labeling work? Commonly the data labeling process consists of the following:\"}),/*#__PURE__*/t(\"ul\",{style:{\"--framer-font-size\":\"18px\",\"--framer-text-alignment\":\"start\",\"--framer-text-color\":\"rgb(30, 33, 38)\",\"--framer-text-stroke-width\":\"0px\",\"--framer-text-transform\":\"none\"},children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(a,{href:\"https://toloka.ai/data-labeling-platform/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"Data collection\"})}),\" and data preparation\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Data labeling, acquiring ground truth \u2013 the target variables that the model is supposed to foresee\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Quality control. The data has to be of high quality, precise, and coherent\"})})]}),/*#__PURE__*/e(\"h2\",{children:\"Manual labeling\"}),/*#__PURE__*/e(\"p\",{children:\"Automated labeling is commonly different from manual labeling. Such a way of labeling assumes that the labeling specialist makes all the necessary labels manually. For instance, outlining an object in a bounding box on a photo or video, or assigning text to an audio recording when designing a natural language processing application.\"}),/*#__PURE__*/e(\"p\",{children:\"Manual labeling allows for better control of the data labeling process. Human labelers are more likely to have a deeper comprehension of the objective and skills to interpret the data presented to them. Human-processed data is apt to propose higher quality labels and more accurate data than those produced by automated tools. Such quality is essential in complex projects such as processing medical images or videos for the development of autonomous vehicles, which will determine the safety of the people on the road.\"}),/*#__PURE__*/e(\"h2\",{children:\"How Does Automated Data Labeling Work? \u200D\"}),/*#__PURE__*/e(\"p\",{children:\"Automatic labeling refers to a feature of data annotation mechanisms that employ artificial intelligence to refine, annotate, or auto-label datasets. Having this characteristic supports manual labeling, thus saving time and costs on data annotation for ML.\"}),/*#__PURE__*/e(\"p\",{children:\"Artificial intelligence training relies on huge quantities of high-quality data. More data can certainly be beneficial in many situations, as it can provide a more complete and accurate understanding of a particular subject. The data quality is also decisive since not all data is equally valuable.\"}),/*#__PURE__*/e(\"p\",{children:\"Auto labeling algorithm assumes that the pre-trained program will accomplish everything on its own, without human input. However, auto-labeling is not about having a machine doing all the work for you. Different types of automated labeling involve humans in one way or another. Humans come up with algorithms that could teach machines to identify and label the required items. They also collect the data or place data labels on the data to automate data labeling or perform QA checks of the system, or a combination of some of the above.\"}),/*#__PURE__*/e(\"p\",{children:\"Automatic data annotation provides a straightforward, fast, and up-to-date approach to processing data using AI itself. The software tools possessing this feature are backing up human efforts while saving time and money on data labeling. However, it still is unable to completely replace manual labeling and it has its drawbacks, which we will discuss further on.\"}),/*#__PURE__*/e(\"h2\",{children:\"ML approaches that enable auto labeling\"}),/*#__PURE__*/e(\"p\",{children:\"Deciding which type of ML that enables auto-labeling is most efficient for your particular project varies depending on your objectives, project requirements, the field of work you are in, and the type of data you are handling, among several other important aspects. Similarly, there is no single classification of approaches to automated data labeling. Here are a few of the most notable ML methods that allow the creation of auto labeling tools:\"}),/*#__PURE__*/e(\"h3\",{children:\"Supervised Learning\"}),/*#__PURE__*/e(\"p\",{children:\"Supervised learning is an ML technique, where a model is learning from labeled data. In other words, the algorithm memorizes the examples given to it. One of the techniques for a data set to be annotated automatically and evolve into training data involves data labeling professional uploading the relevant information to an ML tool, which already can qualitatively label the data. Supervised learning is employed to develop such applications.\"}),/*#__PURE__*/e(\"p\",{children:\"This method for automated labeling may be called extremely reliable since supervised learning requires vast amounts of human pre-labeled data to start the learning process in the first place. As a result, specialists create software that can automatically assign labels to new data.\"}),/*#__PURE__*/e(\"p\",{children:\"Frequently, to automatically label photos, video, or audio for a particular area of use, such as autonomous vehicles, highly specialized professionals are brought in to help with the labeling, which makes the software based on such data even more reliable. But again, creating such a platform to auto-label data requires a tremendous amount of resources, including trained professionals and hours of work. Not every team that builds ML tools can afford that.\"}),/*#__PURE__*/e(\"h3\",{children:\"Reinforcement learning\"}),/*#__PURE__*/e(\"p\",{children:\"In RL, experts typically train a model called an agent. This agent cooperates with the environment through actions in it. The method is founded on the concept that a functioning algorithm has to be rewarded for its achievements and punished for its failures. As a response, the environment communicates some reward in exchange for the right agent's action, as well as its new state. Thus, the agent accumulates knowledge about the rewards received for their actions in certain states. The agent's goal is to maximize the total reward received over a period of time. The agent starts to forecast outcomes that are appropriate to the specific context through trial and error.\"}),/*#__PURE__*/e(\"p\",{children:\"RL algorithms, in contrast to the majority of widely known machine learning algorithms that make predictions, require interaction with the outside world they are put in, which requires analyzing data over some period of time. Doing so helps such models meet their main goals. RL algorithm's purpose consists in maximizing their gain during a given period of time. These types of algorithms most closely resemble the mechanisms of conditioned response in people.\"}),/*#__PURE__*/e(\"h3\",{children:\"Unsupervised Learning\"}),/*#__PURE__*/e(\"p\",{children:\"In supervised learning, a computer has a specific manager telling it how to do the right thing and what to choose. That is, the supervisor has already labeled all raw data in advance, and the machine learns from the specific cases.\"}),/*#__PURE__*/e(\"p\",{children:\"Unsupervised Learning is the complete opposite of supervised learning, first of all, due to the fact that the learning system has no supervisor or labeled data. Unsupervised learning algorithms work autonomously, without any manual intervention. It evolves its own mechanisms to produce the correct results. A human doesn't need to label the data sets in this type of machine learning. It figures it out on its own.\"}),/*#__PURE__*/e(\"p\",{children:\"In unsupervised learning, the algorithm is not informed of the end goal or given any templates, just presented with data sets. Shared characteristics of the data items are recognized automatically. However, these systems still require a bit of a human workforce to validate the output variables.\"}),/*#__PURE__*/e(\"p\",{children:\"In addition, labeled datasets generated in this manner are often unreliable and difficult to interpret. In unsupervised learning, it is harder to deduce the precision of the algorithms since there are no true answers or labels in the given data.\"}),/*#__PURE__*/e(\"h3\",{children:\"Active learning\"}),/*#__PURE__*/e(\"p\",{children:\"Active learning is an area of ML where a program collaborates with a data provider, which is capable of annotating the requested data. There are occasions when there is a great deal of unlabeled data and getting manually labeled data is costly. ML algorithms in this case may actively request labels from the user (data provider). Typically, a person or even a group of individuals represent a data provider. Active learning aims at achieving the best attainable quality of the model, employing as few additional examples as possible.\"}),/*#__PURE__*/e(\"p\",{children:\"Active learning involves the data labelers first submitting a small subset of labeled data to the system for the model to learn from this training set. Then it questions a human by determining unlabeled data regarding which it has no certainty so that a person could respond by creating labels for those instances. The model then updates itself once again by repeating the process until a fairly high degree of accuracy has been achieved. By ensuring that the human trains the model in an iterative manner, the model may be refined in less time, with much less labeled data.\"}),/*#__PURE__*/e(\"p\",{children:\"However, the most significant impediment of active learning comes from the fact that sometimes it is difficult to predict the reliability of the algorithm's results, even despite the fact of using only the most high-quality training assets.\"}),/*#__PURE__*/e(\"h2\",{children:\"When do we require automated data labeling?\"}),/*#__PURE__*/e(\"p\",{children:\"The applications of automatic data labeling may vary widely, but commonly they are in one of the following categories:\"}),/*#__PURE__*/t(\"ol\",{style:{\"--framer-font-size\":\"18px\",\"--framer-text-alignment\":\"start\",\"--framer-text-color\":\"rgb(30, 33, 38)\",\"--framer-text-stroke-width\":\"0px\",\"--framer-text-transform\":\"none\",\"--list-style-type\":\"unset\"},children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Automated labeling for Natural Language Processing (NLP). NLP refers to a subfield of computer science and AI that is concerned with ways in which machines analyze natural or human languages. NLP allows ML algorithms to be applied to word processing, voice recognition, and speech transcription;\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Audio labeling process for the purposes such as recognizing not only human speech but also sounds such as dogs barking or vehicle sounds;\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Image/video labeling for computer vision (CV) purposes. The goal of computer vision models is to make computers recognize patterns in visual data similar to how people do it. For example, CV technology makes it possible to create and improve self-driving cars.\"})})]}),/*#__PURE__*/e(\"h2\",{children:\"Why you can't always use auto labeling\"}),/*#__PURE__*/e(\"p\",{children:\"Automated data labeling is still not appropriate for the vast majority of machine learning projects. For instance, it is not suitable for collecting ground truth data. Since ground truth is the ideal expected result, auto labeling cannot always provide 100% correct results. The output of such a system requires constant human review to evaluate model performance and quality.\"}),/*#__PURE__*/e(\"p\",{children:\"The labeling team follows the automated process by monitoring, fixing, and adding labels, which may increase the time required to do labeling projects, as compared to manual data labeling. There are exceptions and edge cases where the system will not be able to assign an auto label and only humans can help it do so.\"}),/*#__PURE__*/e(\"p\",{children:\"It is never absolutely clear how the automatic labeling system will perform. In some cases, it provides a solid baseline for getting the job done and decreases the time it takes to accomplish projects. But in other cases, it produces low-quality results, particularly when there are edge cases, which increases the time required to complete machine learning projects.\"}),/*#__PURE__*/e(\"h2\",{children:\"Why crowdsourcing is a good alternative to automated data labeling?\"}),/*#__PURE__*/e(\"p\",{children:\"Automated data labeling is clearly an advanced and functional method of labeling, but as already mentioned earlier, it is still at such a stage of development that it is not always able to completely replace humans.\"}),/*#__PURE__*/e(\"p\",{children:\"Crowdsourcing on the other hand allows separating a large-scale task into many smaller tasks and assigning them to different people. This makes it convenient to collect and label vast amounts of data.\"}),/*#__PURE__*/e(\"p\",{children:\"Involving a large number of people in data labeling is especially crucial when the output values require certain knowledge that is available only to human beings and which is most likely to be the most complicated for automatic algorithms to grasp.\"}),/*#__PURE__*/e(\"p\",{children:\"The labor-intensive process of qualitative data labeling can be accelerated by crowdsourcing. Due to the fact that a large talent pool carries out labeling tasks all at once, it requires much less time than the process completed by internal staff.\"}),/*#__PURE__*/e(\"p\",{children:\"Labeling data automatically also provides a relatively quick way of labeling, but as previously mentioned, it does not perform well on all types of data and not in all situations. Occasionally, not only may it not speed up the labeling process, but it may also slow it down.\"}),/*#__PURE__*/e(\"p\",{children:\"The human approach to labeling that crowdsourcing enables is not only more accurate but also quicker, requiring fewer or no subsequent revisions, since contributors who are unable to complete tasks properly are more likely to be eliminated from the project. Moreover, hiring experts via crowdsourcing platforms is likely to be much cheaper than preparing your own automated labeling systems.\"}),/*#__PURE__*/e(\"h2\",{children:\"Types of jobs on crowdsourcing platforms\"}),/*#__PURE__*/e(\"p\",{children:\"Crowdsourcing offers several basic ways to optimize the creation of high-quality datasets. Thousands of users on crowdsourcing platforms assist in improving existing algorithms, developing new ones, keeping the data up-to-date, and collecting new real data. Essentially, they are engaged in manual data labeling, solving these and many other tasks:\"}),/*#__PURE__*/t(\"ul\",{style:{\"--framer-font-size\":\"18px\",\"--framer-text-alignment\":\"start\",\"--framer-text-color\":\"rgb(30, 33, 38)\",\"--framer-text-stroke-width\":\"0px\",\"--framer-text-transform\":\"none\"},children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(a,{href:\"https://toloka.ai/search-relevance/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"evaluation of the site's relevance\"})})})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(a,{href:\"https://toloka.ai/ml/computer-vision/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"object detection on photos\"})})})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(a,{href:\"https://toloka.ai/ml/computer-vision/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"semantic segmentation of images\"})})})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(a,{href:\"https://toloka.ai/image-data/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"image classification\"})})})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(a,{href:\"https://toloka.ai/image-data/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"image segmentation\"})}),\" and so on\"]})})]}),/*#__PURE__*/e(\"h2\",{children:\"Summary\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(a,{href:\"https://toloka.ai/data-labeling-platform/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"Data labeling\"})}),\" and preparation take a lot of energy and resources when developing ML projects. However, automated data labeling exists for this purpose, which allows labeling experts to reduce human error and the time prescribed to process data sets.\"]}),/*#__PURE__*/e(\"p\",{children:\"But this type of data labeling does not always work correctly and often cannot solve the task with 100% accuracy, or may even slow down the project implementation. A great alternative to this type of labeling is employing crowdsourcing platforms that offer a huge selection of labeling tools, as well as a large selection of human labelers with various skills that can accurately and quickly.\"})]});export const richText3=/*#__PURE__*/t(i.Fragment,{children:[/*#__PURE__*/e(\"h2\",{children:\"About the client\"}),/*#__PURE__*/e(\"p\",{children:\"The client\u2019s conversational voice assistant can search online, play music, control smart home devices, and chat with the user. With an average of over 30 million users per month and 75\u2013100 minutes of daily average use time, the assistant plays a vital role in helping people with everyday requests. \"}),/*#__PURE__*/e(\"h2\",{children:\"\u0421hallenge\"}),/*#__PURE__*/e(\"p\",{children:\"Continuous improvement of the voice assistant focuses on areas like on-time activation, accurate speech recognition even in situations with background noise, and continuing or ending conversations appropriately.\"}),/*#__PURE__*/e(\"p\",{children:\"In addition, the service was looking to expand into new markets and languages, where they would need to retrain the voice assistant from scratch. Directly translating requests and responses to a new language isn\u2019t accurate enough to support natural conversations \u2014 linguistic nuances get in the way.\"}),/*#__PURE__*/e(\"p\",{children:\"Human input is crucial for collecting training data and evaluating current output from the voice assistant. The client needed large volumes of human-labeled data for multiple scenarios, including speech-to-text, request classification, answer relevance evaluation, and audio recording.\"}),/*#__PURE__*/e(\"p\",{children:\"The client\u2019s primary business goals included:\"}),/*#__PURE__*/t(\"ul\",{style:{\"--framer-font-size\":\"18px\",\"--framer-text-alignment\":\"start\",\"--framer-text-color\":\"rgb(30, 33, 38)\",\"--framer-text-stroke-width\":\"0px\",\"--framer-text-transform\":\"none\"},children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Refining the relevance of answers in multiple languages for better user satisfaction.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Expanding into new markets with functionality in new languages.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Boosting the integral accuracy of responses in Arabic, a new language for the assistant.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Tracking metrics to measure quality in labeled data on a daily basis.\"})})]}),/*#__PURE__*/e(\"p\",{children:\"The Arabic language presented a significant challenge for the voice assistant. Writing is very different from speaking, particularly for Arabic, with multiple dialects to consider and variations in vocabulary and pronunciation. There are also no vowels in Arabic, which complicates reading and articulation, making human input even more critical.\"}),/*#__PURE__*/e(\"h2\",{children:\"Solution\"}),/*#__PURE__*/e(\"p\",{children:\"The client leveraged Toloka\u2019s crowd to help train the voice assistant to perform various tasks in multiple languages.\"}),/*#__PURE__*/e(\"p\",{children:\"Evaluation of the assistant\u2019s responses used binary classification to label which responses were \u201Cgood\u201D or \u201Ccomplete\u201D.\"}),/*#__PURE__*/e(\"img\",{alt:'Evaluation of the assistant\u2019s responses used binary classification to label which responses were \u201Cgood\u201D or \u201Ccomplete\"',className:\"framer-image\",height:\"706\",src:\"https://framerusercontent.com/images/VeqdsprqmOSZbDSCaCBteUpaDJ8.webp\",srcSet:\"https://framerusercontent.com/images/VeqdsprqmOSZbDSCaCBteUpaDJ8.webp?scale-down-to=512 512w,https://framerusercontent.com/images/VeqdsprqmOSZbDSCaCBteUpaDJ8.webp?scale-down-to=1024 1024w,https://framerusercontent.com/images/VeqdsprqmOSZbDSCaCBteUpaDJ8.webp?scale-down-to=2048 2048w,https://framerusercontent.com/images/VeqdsprqmOSZbDSCaCBteUpaDJ8.webp 2560w\",style:{aspectRatio:\"2560 / 1413\"},width:\"1280\"}),/*#__PURE__*/e(\"p\",{children:\"The crowd provided vital input for speech recognition, request classification, and answer relevance in different languages. They also made audio recordings of spoken language, including Arabic. The client used the data collected to train models, track metrics, and evaluate quality.\"}),/*#__PURE__*/e(\"p\",{children:\"Here\u2019s what a typical workflow looked like:\"}),/*#__PURE__*/e(\"p\",{children:\"The client used existing scripts of interactions with the voice assistant, like setting an alarm, turning on music, or chatting about the weather.\"}),/*#__PURE__*/e(\"img\",{alt:\"The client used existing scripts of interactions with the voice assistant, like setting an alarm, turning on music, or chatting about the weather\",className:\"framer-image\",height:\"706\",src:\"https://framerusercontent.com/images/Buj0hd1Vmc3i4oPu4g6z40Kw.webp\",srcSet:\"https://framerusercontent.com/images/Buj0hd1Vmc3i4oPu4g6z40Kw.webp?scale-down-to=512 512w,https://framerusercontent.com/images/Buj0hd1Vmc3i4oPu4g6z40Kw.webp?scale-down-to=1024 1024w,https://framerusercontent.com/images/Buj0hd1Vmc3i4oPu4g6z40Kw.webp?scale-down-to=2048 2048w,https://framerusercontent.com/images/Buj0hd1Vmc3i4oPu4g6z40Kw.webp 2560w\",style:{aspectRatio:\"2560 / 1413\"},width:\"1280\"}),/*#__PURE__*/e(\"p\",{children:\"They created questions for the Toloka crowd and translated instructions to Arabic, asking Tolokers to pretend they are talking to the voice assistant in a particular scenario (for instance, setting an alarm for weekdays only).\"}),/*#__PURE__*/e(\"img\",{alt:\"They created questions for the Toloka crowd and translated instructions to Arabic\",className:\"framer-image\",height:\"706\",src:\"https://framerusercontent.com/images/B9x0FWZUBGVxNvGF3hKpS8xVzvU.webp\",srcSet:\"https://framerusercontent.com/images/B9x0FWZUBGVxNvGF3hKpS8xVzvU.webp?scale-down-to=512 512w,https://framerusercontent.com/images/B9x0FWZUBGVxNvGF3hKpS8xVzvU.webp?scale-down-to=1024 1024w,https://framerusercontent.com/images/B9x0FWZUBGVxNvGF3hKpS8xVzvU.webp?scale-down-to=2048 2048w,https://framerusercontent.com/images/B9x0FWZUBGVxNvGF3hKpS8xVzvU.webp 2560w\",style:{aspectRatio:\"2560 / 1413\"},width:\"1280\"}),/*#__PURE__*/e(\"p\",{children:\"Arabic-speaking Tolokers followed the instructions and recorded voice requests on their phones. The audio recordings were given to other Tolokers to transcribe into text and classify by intent during manual review.\"}),/*#__PURE__*/e(\"img\",{alt:\"The resulting data was fed to speech recognition models\",className:\"framer-image\",height:\"706\",src:\"https://framerusercontent.com/images/WPL0jHsm8SoSW7LEGDgsggiI9C8.webp\",srcSet:\"https://framerusercontent.com/images/WPL0jHsm8SoSW7LEGDgsggiI9C8.webp?scale-down-to=512 512w,https://framerusercontent.com/images/WPL0jHsm8SoSW7LEGDgsggiI9C8.webp?scale-down-to=1024 1024w,https://framerusercontent.com/images/WPL0jHsm8SoSW7LEGDgsggiI9C8.webp?scale-down-to=2048 2048w,https://framerusercontent.com/images/WPL0jHsm8SoSW7LEGDgsggiI9C8.webp 2560w\",style:{aspectRatio:\"2560 / 1413\"},width:\"1280\"}),/*#__PURE__*/e(\"p\",{children:\"The resulting data was fed to speech recognition models. \"}),/*#__PURE__*/e(\"h2\",{children:\"Results\"}),/*#__PURE__*/e(\"p\",{children:\"With the help of Toloka, the client was able to obtain data for model training, assess current quality, and track metrics for various stages. The client improved the performance of the voice assistant by boosting the accuracy of responses.\"}),/*#__PURE__*/e(\"p\",{children:\"When it comes to new language development, voice assistants have a baseline accuracy of approximately 12%. With human labelers involved at every step of the way, the client was able to build an efficient pipeline for model training and quality assessment. Thanks to the crowd\u2019s input, the assistant\u2019s responses in Arabic ended up being about 62% accurate (two correct responses for every three requests), which was a major step up from the baseline scenario. For comparison, the original version of the voice assistant, which has been under development for many years, has an accuracy rate of approximately 77%.\"}),/*#__PURE__*/e(\"p\",{children:\"In terms of throughput, Toloka processed on average 50,000 unique tasks in Arabic per day, sometimes reaching 400,000 tasks per day. At the peak, there were 970,000 unique tasks.\"}),/*#__PURE__*/e(\"p\",{children:\"With the help of Toloka, the client was able to enter new markets by branching out into new languages and improving metrics across all key areas.\"})]});export const richText4=/*#__PURE__*/t(i.Fragment,{children:[/*#__PURE__*/t(\"p\",{children:[\"Text-to-image models like \",/*#__PURE__*/e(a,{href:\"https://github.com/CompVis/stable-diffusion\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"Stable Diffusion\"})}),\", \",/*#__PURE__*/e(a,{href:\"https://openai.com/product/dall-e-2\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"DALL-E 2\"})}),\", and \",/*#__PURE__*/e(a,{href:\"https://www.midjourney.com/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"Midjourney\"})}),\" can generate impressive, artistic images \u2014 if you\u2019re good at prompt engineering. \"]}),/*#__PURE__*/e(\"p\",{children:\"However, many users simply want to generate aesthetically pleasing images to include in blog posts, presentations, and other materials. So, what about the rest of us who haven\u2019t had the opportunity to learn all the magical keywords, such as \u201Ctrending on Artstation,\u201D and understand how they function in various contexts?\"}),/*#__PURE__*/e(\"p\",{children:\"The Toloka team saw this gap and developed a solution to turn anyone into a prompt engineer. We trained a model to transform simple image descriptions into professional prompts for Stable Diffusion with consistently good results.\"}),/*#__PURE__*/e(\"p\",{children:\"But we wanted our model to do more than just generate prompts \u2014 we wanted to get beautiful prompts that would elicit the most aesthetically pleasing images. To accomplish this, we integrated human preferences by using reinforcement learning from human feedback.\"}),/*#__PURE__*/e(\"p\",{children:\"Furthermore, we provide a user-friendly interface that simplifies the process of generating aesthetically pleasing images. Our interface allows users to input basic image descriptions and receive professionally curated prompts for models such as Stable Diffusion. By incorporating human preferences through reinforcement learning from human feedback, our model ensures that the generated prompts consistently yield visually appealing results. Whether you\u2019re a beginner or an expert, our open-source code, data, and models are freely available, enabling you to replicate our approach or apply it to other text-to-image models with API access. With this comprehensive release, we aim to empower a wider audience to harness the capabilities of text-to-image models and create stunning visuals effortlessly.\"}),/*#__PURE__*/t(\"p\",{children:[\"In this post, we share the technical details and results of our experiments. We have released our code and data, allowing anyone to replicate our approach or apply it to other text-to-image models with API access. You can also use this post as a practical \",/*#__PURE__*/e(\"em\",{children:\"Reinforcement Learning from Human Feedback (RLHF)\"}),\" tutorial to walk you through the steps from data collection to model training.\"]}),/*#__PURE__*/t(\"p\",{children:[\"We \",/*#__PURE__*/e(a,{href:\"https://github.com/Toloka/podvodka\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"release\"})}),\" all the models, code, and data.\"]}),/*#__PURE__*/e(\"h2\",{children:\"Approach\"}),/*#__PURE__*/t(\"p\",{children:[\"Our task is defined as follows: the model takes a human-readable image description, such as \u201Ca cat,\u201D and generates a high-quality prompt, such as \u201Ca cat, trending on Artstation, 4k resolution, cinematic lighting.\u201D This task is relatively simple (definitely not ChatGPT training), so we chose a small yet powerful enough model \u2014 \",/*#__PURE__*/e(a,{href:\"https://huggingface.co/gpt2-large\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"GPT-2 Large\"})}),\". This model has 774M parameters and can be fine-tuned on a single GPU in a couple of hours.\"]}),/*#__PURE__*/e(\"p\",{children:\"So, the model takes the following text as input:\"}),/*#__PURE__*/e(\"img\",{alt:\"So, the model takes the following text as input:\",className:\"framer-image\",height:\"116\",src:\"https://framerusercontent.com/images/StORkIo8lO7qd5jgFXtVAHPHD8.png\",srcSet:\"https://framerusercontent.com/images/StORkIo8lO7qd5jgFXtVAHPHD8.png?scale-down-to=512 512w,https://framerusercontent.com/images/StORkIo8lO7qd5jgFXtVAHPHD8.png?scale-down-to=1024 1024w,https://framerusercontent.com/images/StORkIo8lO7qd5jgFXtVAHPHD8.png?scale-down-to=2048 2048w,https://framerusercontent.com/images/StORkIo8lO7qd5jgFXtVAHPHD8.png 2592w\",style:{aspectRatio:\"2592 / 232\"},width:\"1296\"}),/*#__PURE__*/t(\"p\",{children:[\"We instructed the model to write a prompt after the \",/*#__PURE__*/e(\"em\",{children:\"SEP\"}),\" token since it is an autoregressive language model rather than a seq2seq model.\"]}),/*#__PURE__*/e(\"h2\",{children:\"Overview of Reinforcement Learning from Human Feedback\"}),/*#__PURE__*/e(\"p\",{children:\"Reinforcement Learning from Human Feedback (RLHF) is a technique that allows us to switch from training on an unlabeled text corpus into training to produce human-preferred outputs. This technique has been widely used in the NLP community since it was applied to InstructGPT and ChatGPT.\"}),/*#__PURE__*/e(\"img\",{alt:\"A diagram of training methods from the InstructGPT paper\",className:\"framer-image\",height:\"849\",src:\"https://framerusercontent.com/images/dGkNsYtnU7VLPVu4dYPB6uMeTc.webp\",srcSet:\"https://framerusercontent.com/images/dGkNsYtnU7VLPVu4dYPB6uMeTc.webp?scale-down-to=512 512w,https://framerusercontent.com/images/dGkNsYtnU7VLPVu4dYPB6uMeTc.webp?scale-down-to=1024 1024w,https://framerusercontent.com/images/dGkNsYtnU7VLPVu4dYPB6uMeTc.webp?scale-down-to=2048 2048w,https://framerusercontent.com/images/dGkNsYtnU7VLPVu4dYPB6uMeTc.webp 2560w\",style:{aspectRatio:\"2560 / 1699\"},width:\"1280\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"em\",{children:\"A diagram of training methods from the \"}),/*#__PURE__*/e(a,{href:\"https://arxiv.org/abs/2203.02155\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!0,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:/*#__PURE__*/e(\"em\",{children:\"InstructGPT paper\"})})}),/*#__PURE__*/e(\"em\",{children:\" \"})]}),/*#__PURE__*/e(\"p\",{children:\"According to the ChatGPT blog post, the RLHF pipeline includes three key parts:\"}),/*#__PURE__*/t(\"ol\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Supervised Fine-Tuning (SFT)\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Reward/preference Modeling (RM)\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Reinforcement Learning from Human Feedback (RLHF)\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[\"You can check out the Hugging Face article about \",/*#__PURE__*/e(a,{href:\"https://huggingface.co/blog/stackllama\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"StackLLaMA\"})}),\" that inspired our post.\"]})})]}),/*#__PURE__*/e(\"h2\",{children:\"Step 1: Supervised Fine-Tuning \"}),/*#__PURE__*/t(\"p\",{children:[\"During supervised fine-tuning (SFT), we provide the model with demonstration data to guide its behavior. At this stage, the model is not required to optimize for human preferences. The aim is to learn the desired response format and relevant information contained in the demonstration data. To accomplish this, we collected an unlabeled dataset of text in the desired format, which, in our case, was \",/*#__PURE__*/e(\"em\",{children:\"\u201Cimage_description[SEP]prompt\u201D\"}),\". We then trained the model on this dataset using a standard language modeling task.\"]}),/*#__PURE__*/e(\"p\",{children:\"We faced a challenge when we set out to collect the data because there were no publicly available datasets containing image descriptions and prompts constructed from them. We considered two options.\"}),/*#__PURE__*/e(\"p\",{children:\"First, we could collect a dataset of image descriptions and then create prompts from them. The problem is that it\u2019s not clear how to find image descriptions beyond just a list of objects, and we would need annotators who are experienced at writing prompts for Stable Diffusion, which is a difficult skill to find.\"}),/*#__PURE__*/t(\"p\",{children:[\"The second option was to find a dataset of prompts and then extract image descriptions from them, essentially capturing the user\u2019s intended meaning. This is much easier than the first option because there are open datasets containing Stable Diffusion prompts. We opted to use \",/*#__PURE__*/e(a,{href:\"https://github.com/Toloka/BestPrompts\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"prompts\"})}),\" from Stable Diffusion Discord bots, where users send their prompts and receive a generated image from a bot.\"]}),/*#__PURE__*/t(\"p\",{children:[\"Once we had a dataset of prompts, we needed to extract image descriptions from them. We accomplished this by employing the OpenAI API and asking the \",/*#__PURE__*/e(\"em\",{children:\"text-davinci-003\"}),\" model to extract image descriptions through few-shot learning.\"]}),/*#__PURE__*/e(\"p\",{children:\"We were now ready to begin training the model. To do this, we chose to use a standard example script for LM fine-tuning from HuggingFace Transformers. However, we made several modifications to the script, the most significant of which involved calculating the loss only on prompts, rather than image descriptions. Prior to this change, the model produced severe artifacts, such as starting each generated prompt with the word \u201Cilya.\u201D\"}),/*#__PURE__*/e(\"img\",{alt:\"Validation loss and accuracy\",className:\"framer-image\",height:\"371\",src:\"https://framerusercontent.com/images/OkgJcCTNDiNUBPLQmrAjUwVHGo.webp\",srcSet:\"https://framerusercontent.com/images/OkgJcCTNDiNUBPLQmrAjUwVHGo.webp?scale-down-to=512 512w,https://framerusercontent.com/images/OkgJcCTNDiNUBPLQmrAjUwVHGo.webp?scale-down-to=1024 1024w,https://framerusercontent.com/images/OkgJcCTNDiNUBPLQmrAjUwVHGo.webp?scale-down-to=2048 2048w,https://framerusercontent.com/images/OkgJcCTNDiNUBPLQmrAjUwVHGo.webp 2560w\",style:{aspectRatio:\"2560 / 743\"},width:\"1280\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"em\",{children:\"Validation loss and accuracy\"})}),/*#__PURE__*/e(\"p\",{children:\"The training process takes around 90 minutes to complete on a single NVIDIA A100 80GB GPU.\"}),/*#__PURE__*/e(\"h2\",{children:\"Step 2: Reward Modeling\"}),/*#__PURE__*/e(\"p\",{children:\"Reward modeling (RM) consists of two steps: preference collection and reward modeling itself. During preference collection, we generate several outputs of our model, show them to annotators, and ask them which one they prefer. During reward modeling, we train a model to predict these preferences: the reward model takes a generative model\u2019s output and returns a single value representing how good this output is.\"}),/*#__PURE__*/e(\"h3\",{children:\"Preference Collection\"}),/*#__PURE__*/e(\"p\",{children:\"To gather human preferences, we take each image description and generate three prompts for it. This means that for each image description, we have a total of four prompts, including the original one. For each prompt, we then generate four images using Stable Diffusion 1.5. We then sample pairs of prompts within each image description, take the generated images, and use the Toloka crowdsourcing platform to annotate these pairwise comparisons.\"}),/*#__PURE__*/e(\"p\",{children:\"In the comparison task, annotators are presented with an image description, four images on the left, and four images on the right. They choose which set of images (left or right) looks better.\"}),/*#__PURE__*/e(\"img\",{alt:\"Annotation interface. Example images are generated by the author with Stable Diffusion 1.5\",className:\"framer-image\",height:\"1618\",src:\"https://framerusercontent.com/images/by0hl7wEqKP13Xnkyr1Ivyz3Vg.webp\",srcSet:\"https://framerusercontent.com/images/by0hl7wEqKP13Xnkyr1Ivyz3Vg.webp?scale-down-to=1024 810w,https://framerusercontent.com/images/by0hl7wEqKP13Xnkyr1Ivyz3Vg.webp?scale-down-to=2048 1620w,https://framerusercontent.com/images/by0hl7wEqKP13Xnkyr1Ivyz3Vg.webp 2560w\",style:{aspectRatio:\"2560 / 3236\"},width:\"1280\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"em\",{children:\"Annotation interface. Example images are generated by the author with Stable Diffusion 1.5\"})}),/*#__PURE__*/t(\"p\",{children:[\"We used this process to collect a dataset containing 135,400 preferences for 3,919 prompts. A complete guide for running this annotation process can be found on our \",/*#__PURE__*/e(a,{href:\"https://github.com/Toloka/BestPromptsV2\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"GitHub\"})}),\".\"]}),/*#__PURE__*/e(\"h3\",{children:\"Reward Modeling\"}),/*#__PURE__*/e(\"p\",{children:\"We opted to use a distilroberta-base as our reward model due to its fast training speed and performance similar to other BERT-like models and GPT-2 Large, with a scalar prediction head. This model is designed to take a text input in the form of \u201Cimage description[SEP]prompt\u201D and predict a single number. To implement this, we used AutoModelForSequenceClassification from Transformers and set num_labels=1. However, the training procedure differs from fine-tuning for text classification. We iterated through pairs of prompts for a single image description and predicted two numbers: r\u0302(a)and r\u0302(b), for the left and right prompts, respectively. The predicted probability of the left prompt being better than the right prompt is calculated as\"}),/*#__PURE__*/e(\"img\",{alt:\"The predicted probability of the left prompt being better than the right prompt is calculated as\",className:\"framer-image\",height:\"128\",src:\"https://framerusercontent.com/images/f40A7ito5xdpFXbEpDpA8RquS8.png\",srcSet:\"https://framerusercontent.com/images/f40A7ito5xdpFXbEpDpA8RquS8.png?scale-down-to=512 512w,https://framerusercontent.com/images/f40A7ito5xdpFXbEpDpA8RquS8.png?scale-down-to=1024 1024w,https://framerusercontent.com/images/f40A7ito5xdpFXbEpDpA8RquS8.png?scale-down-to=2048 2048w,https://framerusercontent.com/images/f40A7ito5xdpFXbEpDpA8RquS8.png 2592w\",style:{aspectRatio:\"2592 / 256\"},width:\"1296\"}),/*#__PURE__*/e(\"p\",{children:\"We used a binary cross-entropy loss function where the target is the comparison\u2019s result: 1 if the left prompt is better than the right prompt and 0 otherwise. \"}),/*#__PURE__*/e(\"p\",{children:\"The training process takes approximately 15 minutes on a single NVIDIA A100 80GB. The resulting model achieves an accuracy of 0.63 on the validation set. While this accuracy may appear low, it is adequate for RLHF fine-tuning. This is because the task is subjective and even human annotators may disagree with each other.\"}),/*#__PURE__*/e(\"img\",{alt:\"Loss and accuracy of the reward model\",className:\"framer-image\",height:\"358\",src:\"https://framerusercontent.com/images/Rw1rkoVgIEFUos4XXZlCPbGG8E.webp\",srcSet:\"https://framerusercontent.com/images/Rw1rkoVgIEFUos4XXZlCPbGG8E.webp?scale-down-to=512 512w,https://framerusercontent.com/images/Rw1rkoVgIEFUos4XXZlCPbGG8E.webp?scale-down-to=1024 1024w,https://framerusercontent.com/images/Rw1rkoVgIEFUos4XXZlCPbGG8E.webp?scale-down-to=2048 2048w,https://framerusercontent.com/images/Rw1rkoVgIEFUos4XXZlCPbGG8E.webp 2560w\",style:{aspectRatio:\"2560 / 716\"},width:\"1280\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"em\",{children:\"Loss and accuracy of the reward model\"})}),/*#__PURE__*/e(\"p\",{children:\"We released the reward model weights on HuggingFace Hub.\"}),/*#__PURE__*/e(\"h2\",{children:\"Step 3: Reinforcement Learning from Human Feedback\"}),/*#__PURE__*/t(\"p\",{children:[\"The final step involves training the fine-tuned GPT-2 Large model to maximize the rewards predicted by the reward model. We used the PPO implementation provided by Carper AI\u2019s \",/*#__PURE__*/e(a,{href:\"https://github.com/CarperAI/trlx\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"trlx\"})}),\" library, which is a convenient package for fine-tuning any language model with RL on any reward function.\"]}),/*#__PURE__*/e(\"p\",{children:\"We faced challenges during the hyper-parameter sweep, as the training process is very sensitive to specific hyper-parameter values, particularly the learning rate and init_kl. We found that low values of these parameters worked best.\"}),/*#__PURE__*/e(\"p\",{children:\"It is important to note that both value and policy losses should remain below 1.0 at all times. If one or both losses significantly increase, it indicates that something has gone wrong. We observed the same behavior as described in the StackLLaMA blog post, where the policy loss can sometimes experience extreme spikes.\"}),/*#__PURE__*/e(\"img\",{alt:\"Policy loss spike\",className:\"framer-image\",height:\"712\",src:\"https://framerusercontent.com/images/DQ30HAcHbqhLaMr5zhhvLNI2iaw.webp\",srcSet:\"https://framerusercontent.com/images/DQ30HAcHbqhLaMr5zhhvLNI2iaw.webp?scale-down-to=512 512w,https://framerusercontent.com/images/DQ30HAcHbqhLaMr5zhhvLNI2iaw.webp?scale-down-to=1024 1024w,https://framerusercontent.com/images/DQ30HAcHbqhLaMr5zhhvLNI2iaw.webp?scale-down-to=2048 2048w,https://framerusercontent.com/images/DQ30HAcHbqhLaMr5zhhvLNI2iaw.webp 2560w\",style:{aspectRatio:\"2560 / 1425\"},width:\"1280\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"em\",{children:\"Policy loss spike\"})}),/*#__PURE__*/e(\"p\",{children:\"In our experiments, the value loss consistently decreased, which is a positive sign.\"}),/*#__PURE__*/e(\"img\",{alt:\"Model\u2019s value loss\",className:\"framer-image\",height:\"708\",src:\"https://framerusercontent.com/images/UAaNWTRrrSoWrbNQOIk8he5WA.webp\",srcSet:\"https://framerusercontent.com/images/UAaNWTRrrSoWrbNQOIk8he5WA.webp?scale-down-to=512 512w,https://framerusercontent.com/images/UAaNWTRrrSoWrbNQOIk8he5WA.webp?scale-down-to=1024 1024w,https://framerusercontent.com/images/UAaNWTRrrSoWrbNQOIk8he5WA.webp?scale-down-to=2048 2048w,https://framerusercontent.com/images/UAaNWTRrrSoWrbNQOIk8he5WA.webp 2560w\",style:{aspectRatio:\"2560 / 1417\"},width:\"1280\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"em\",{children:\"Model\u2019s value loss\"})}),/*#__PURE__*/e(\"p\",{children:\"The reward model\u2019s predicted reward significantly increased during the training process. However, this does not necessarily mean that the model is perfect or even improving, as the reward model itself may have imperfections that PPO could exploit.\"}),/*#__PURE__*/e(\"img\",{alt:\"reward/mean\",className:\"framer-image\",height:\"708\",src:\"https://framerusercontent.com/images/HGdqQ31gzIvom5lq2kLbwBqSvls.webp\",srcSet:\"https://framerusercontent.com/images/HGdqQ31gzIvom5lq2kLbwBqSvls.webp?scale-down-to=512 512w,https://framerusercontent.com/images/HGdqQ31gzIvom5lq2kLbwBqSvls.webp?scale-down-to=1024 1024w,https://framerusercontent.com/images/HGdqQ31gzIvom5lq2kLbwBqSvls.webp?scale-down-to=2048 2048w,https://framerusercontent.com/images/HGdqQ31gzIvom5lq2kLbwBqSvls.webp 2560w\",style:{aspectRatio:\"2560 / 1417\"},width:\"1280\"}),/*#__PURE__*/e(\"h2\",{children:\"Visual Results\"}),/*#__PURE__*/e(\"p\",{children:\"Here are some examples of simple prompts and the results generated by Stable Diffusion via our model.\"}),/*#__PURE__*/e(\"img\",{alt:\"an orange boat in sea, generated by SD with our model\",className:\"framer-image\",height:\"853\",src:\"https://framerusercontent.com/images/4lQmN7k5rV6enDId5lTgt46IbM.webp\",srcSet:\"https://framerusercontent.com/images/4lQmN7k5rV6enDId5lTgt46IbM.webp?scale-down-to=512 512w,https://framerusercontent.com/images/4lQmN7k5rV6enDId5lTgt46IbM.webp?scale-down-to=1024 1024w,https://framerusercontent.com/images/4lQmN7k5rV6enDId5lTgt46IbM.webp?scale-down-to=2048 2048w,https://framerusercontent.com/images/4lQmN7k5rV6enDId5lTgt46IbM.webp 2560w\",style:{aspectRatio:\"2560 / 1707\"},width:\"1280\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"em\",{children:\"an orange boat in sea, generated by SD with our model\"})}),/*#__PURE__*/e(\"img\",{alt:\"interior of aliens spacecraft, generated by SD with our model\",className:\"framer-image\",height:\"853\",src:\"https://framerusercontent.com/images/o1GKVprGylKbzV8R8f6jPhAtiA.webp\",srcSet:\"https://framerusercontent.com/images/o1GKVprGylKbzV8R8f6jPhAtiA.webp?scale-down-to=512 512w,https://framerusercontent.com/images/o1GKVprGylKbzV8R8f6jPhAtiA.webp?scale-down-to=1024 1024w,https://framerusercontent.com/images/o1GKVprGylKbzV8R8f6jPhAtiA.webp?scale-down-to=2048 2048w,https://framerusercontent.com/images/o1GKVprGylKbzV8R8f6jPhAtiA.webp 2560w\",style:{aspectRatio:\"2560 / 1707\"},width:\"1280\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"em\",{children:\"interior of aliens spacecraft, generated by SD with our model\"})}),/*#__PURE__*/e(\"img\",{alt:\"a cat portrait, generated by SD with our model\",className:\"framer-image\",height:\"1921\",src:\"https://framerusercontent.com/images/TO2n7R9yB1kSRqCdWNAISl8NRs.webp\",srcSet:\"https://framerusercontent.com/images/TO2n7R9yB1kSRqCdWNAISl8NRs.webp?scale-down-to=1024 682w,https://framerusercontent.com/images/TO2n7R9yB1kSRqCdWNAISl8NRs.webp?scale-down-to=2048 1364w,https://framerusercontent.com/images/TO2n7R9yB1kSRqCdWNAISl8NRs.webp 2560w\",style:{aspectRatio:\"2560 / 3842\"},width:\"1280\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"em\",{children:\"a cat portrait, generated by SD with our model\"})}),/*#__PURE__*/e(\"p\",{children:\"All images in this article were generated by the author using Stable Diffusion 1.5 unless stated otherwise. \"}),/*#__PURE__*/e(\"h2\",{children:\"What\u2019s next\"}),/*#__PURE__*/e(\"p\",{children:\"We have made all the code and data available for the open-source community to build upon our results and create similar solutions for other models like DALL-E 2 and Midjourney.\"}),/*#__PURE__*/t(\"p\",{children:[\"In the meantime, you can experiment with our \",/*#__PURE__*/e(a,{href:\"https://huggingface.co/toloka/gpt2-large-rl-prompt-writing\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"model\"})}),\" on Hugging Face Hub and in our \",/*#__PURE__*/e(a,{href:\"https://huggingface.co/spaces/toloka/sd-prompts\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"Space\"})}),\".\"]}),/*#__PURE__*/e(\"p\",{children:\"We welcome your feedback!\"})]});export const richText5=/*#__PURE__*/t(i.Fragment,{children:[/*#__PURE__*/e(\"h2\",{children:\"About the client\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"The client \u2014 a taxi aggregator \u2014 regularly checks the condition of the cars their drivers use, including vehicle model, branding, license plates, seat belts, cleanliness, damage, and more. \"})}),/*#__PURE__*/e(\"p\",{children:\"With more than 1.5 million cars and over USD 1.5 billion in revenue, the taxi app provides an essential and invaluable service to numerous customers on a daily basis.\"}),/*#__PURE__*/e(\"p\",{children:\"To ensure customer safety and maintain brand image, the taxi aggregator checks the condition of the cars that taxi drivers use. Drivers submit photos of their vehicle via the app, including photos of the exterior and interior of their car, to ensure they\u2019re compliant with the requirements before they are allowed to take orders.\"}),/*#__PURE__*/e(\"p\",{children:\"Vehicle photos are reviewed and tagged for a comprehensive list of criteria, including photo quality, vehicle model and brand, license plate, cleanliness, damage (if any), seatbelts, and more.\"}),/*#__PURE__*/e(\"p\",{children:\"The company has found that photo moderation and quality control directly correlate with the number of customer complaints. For example, when cars aren\u2019t banned for severe damage, the number of complaints increases significantly, impacting customer retention rates.\"}),/*#__PURE__*/e(\"p\",{children:\"The client uses two control methods for photo monitoring:\"}),/*#__PURE__*/t(\"ul\",{style:{\"--framer-font-size\":\"18px\",\"--framer-text-alignment\":\"start\",\"--framer-text-color\":\"rgb(30, 33, 38)\",\"--framer-text-stroke-width\":\"0px\",\"--framer-text-transform\":\"none\"},children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Photo control of cars\"}),\" \u2014 Drivers take vehicle photos via the app and submit them to the service to check for compliance.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Stationary control\"}),\" \u2014 A designated representative evaluates the car in person, fills out a questionnaire, and takes pictures. Then the team follows up and decides whether the car is compliant.\"]})})]}),/*#__PURE__*/e(\"h2\",{children:\"The challenge\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"The client\u2019s goal was to decrease the number of user complaints by improving photo monitoring. They needed to label daily streams of image data with high accuracy.\"})}),/*#__PURE__*/e(\"p\",{children:\"To accommodate large volumes of data, the client was already using machine learning models to classify whether a car is compliant. However, these models weren\u2019t always providing verdicts with a high level of confidence. For more complex cases (such as checking for cleanliness, damage, or branding), model accuracy was not meeting the service\u2019s standards.\"}),/*#__PURE__*/e(\"p\",{children:\"Key business metrics:\"}),/*#__PURE__*/t(\"ul\",{style:{\"--framer-font-size\":\"18px\",\"--framer-text-alignment\":\"start\",\"--framer-text-color\":\"rgb(30, 33, 38)\",\"--framer-text-stroke-width\":\"0px\",\"--framer-text-transform\":\"none\"},children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"The number of rides with customer complaints\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"The number of customer complaints about the car's condition\"})})]}),/*#__PURE__*/e(\"p\",{children:\"Because photo moderation directly affects these metrics, the client prioritized accuracy and speed of labeling over cost.\"}),/*#__PURE__*/e(\"p\",{children:\"Given the complex requirements for approving taxis, the client needed human input to ensure customer safety. In short, the client needed a quick way to assess 1.5 million cars for compliance with comfort and safety standards.\"}),/*#__PURE__*/e(\"h2\",{children:\"Solution\"}),/*#__PURE__*/e(\"p\",{children:\"Toloka\u2019s crowd of human labelers joined the photo moderation pipeline to boost accuracy. They played a crucial role in helping to classify complex cases by filling in the gaps in automatic classification and gathering data for model training and quality improvement.\"}),/*#__PURE__*/e(\"p\",{children:\"The crowd provided verdicts for subjective criteria like cleanliness and damage, where ML models were performing with only 30% accuracy on average. Photos that the models labeled with a low confidence level were sent to Tolokers for evaluation. Tolokers also performed a quality check on the 1% of data that was labeled with high confidence by models.\"}),/*#__PURE__*/e(\"h2\",{children:\"The results\"}),/*#__PURE__*/e(\"p\",{children:\"With the help of Toloka, the client could label larger volumes of data at a cost savings of about five times less per label compared to the alternative internal solution they were using.\"}),/*#__PURE__*/e(\"p\",{children:\"At a significantly lower cost, Toloka's output quality remained high and met the client's expectations for complex classification cases. For relatively subjective evaluations, Toloker accuracy averaged 80-90% compared to 30% for machine learning models. For simple tasks, such as performing a license plate check, accuracy was close to 100%.\"}),/*#__PURE__*/e(\"p\",{children:\"Toloka surpassed the client\u2019s expectations for labeling speed, the ability to scale for large volumes, price per label, and output quality. The taxi app plans on using Toloka for other applications such as gathering data for future model training.\"})]});export const richText6=/*#__PURE__*/t(i.Fragment,{children:[/*#__PURE__*/e(\"p\",{children:\"To be successful and pertinent businesses have to constantly collect feedback from their customers to assess customer satisfaction with a product or service and enhance them if possible. Similarly, the consumers who make a purchase often feel the need to get an independent evaluation of such products.\"}),/*#__PURE__*/e(\"p\",{children:\"With sentiment analysis, an array of texts, comments, messages, survey responses, and any other textual data may be studied to determine their emotional coloring \u2013 positive, negative, or neutral. Sentiment analysis helps companies to further improve their products and services as well as consumers to understand whether they want to buy the product or not. Let's take a look at what sentiment analysis is and why it's so helpful.\"}),/*#__PURE__*/e(\"h2\",{children:\"What is Sentiment analysis?\"}),/*#__PURE__*/e(\"p\",{children:\"Sentiment analysis, also described as opinion mining, is one of the machine learning (ML) domains that allows one to determine the emotional coloring of a particular text or sentence. It can be applied to retrieve textual data and classify it into specific types of sentiment for convenient use.\"}),/*#__PURE__*/e(\"p\",{children:\"Sentiment means the emotional attitude of the creator of a statement towards something, expressed in the text. The sentiment of the entire text may be defined as the general sentiment of its individual elements or sentences. Here are the basic types of sentiment evaluations:\"}),/*#__PURE__*/t(\"ul\",{style:{\"--framer-font-size\":\"18px\",\"--framer-text-alignment\":\"start\",\"--framer-text-color\":\"rgb(30, 33, 38)\",\"--framer-text-stroke-width\":\"0px\",\"--framer-text-transform\":\"none\"},children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"positive sentiment;\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"negative sentiment;\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"neutral sentiment or no emotional coloring. The basic challenge of sentiment analysis involves classifying the polarity of a given text, meaning deciding whether the opinion expressed therein is positive, negative, or neutral. The aggregate of positive and negative words is then analyzed to figure out what type of words prevail in the submitted document or array of texts.\"})})]}),/*#__PURE__*/e(\"p\",{children:\"Natural language processing (NLP) is at the core of the sentiment analysis concept. That is the Artificial Intelligence (AI) subset handling human language, enabling computers to comprehend texts and make use of text processing. Artificial intelligence systems may be tailored to the specific tasks of natural language processing. One of those is sentiment analysis.\"}),/*#__PURE__*/e(\"p\",{children:\"The sentiment score is one of the ways how sentiment can be assessed. Diverse aspects, like the range and sort of emotions conveyed, the intensity of the emotions, as well as the specific context in which they have occurred, are considered in the sentiment score calculation.\"}),/*#__PURE__*/e(\"h2\",{children:\"The main purpose of sentiment analysis\"}),/*#__PURE__*/e(\"p\",{children:\"Sentiment analysis or opinion mining describes a method of content analytics the major purpose of which is to classify a text according to its sentiment by detecting opinions in a text and revealing their properties. It is up to the task at hand to determine which properties will be investigated.\"}),/*#__PURE__*/e(\"p\",{children:\"The sentiment analysis enables the extraction of the author's opinion on a given object from the text data. This opinion may express the author's judgment, view, or evaluation, his or her emotional state. The author, that is, the individual who shares the opinion, may also be the target of the analysis.\"}),/*#__PURE__*/e(\"h2\",{children:\"Approaches to sentiment analysis\"}),/*#__PURE__*/e(\"p\",{children:\"Generally, there are 3 primary approaches to sentiment analysis:\"}),/*#__PURE__*/e(\"h3\",{children:\"Rule-based approach\"}),/*#__PURE__*/e(\"p\",{children:\"This method is based on linguistics, with a set of rules developed by professional language analysts. Such an approach implies the existence of a dedicated vocabulary that contains words or collocations. The rule-based approach identifies, classifies, and evaluates specific keywords based on the pre-defined tone values for such words in the vocabulary.\"}),/*#__PURE__*/e(\"p\",{children:\"A sentiment score and, sometimes, sentiment strength are noted for each word in the vocabulary. The software automatically checks the sentiment score to find positive and negative sentiments and to evaluate whether a sentence is positive, negative, or neutral and then sums up the sentiment score. The final grade is compared with the set boundaries to detect the overall sentiment.\"}),/*#__PURE__*/e(\"h3\",{children:\"Machine Learning-based\"}),/*#__PURE__*/e(\"p\",{children:\"A machine learning algorithm that is more commonly applied in sentiment analysis is supervised learning. Such algorithms employ automatic feature extraction from a text to perform sentiment classification. The essence of the approach lies in the fact that an ML classifier is initially trained on the labeled data and then applied to classify new texts.\"}),/*#__PURE__*/e(\"h3\",{children:\"Hybrid approach\"}),/*#__PURE__*/e(\"p\",{children:\"This type of analysis enables several approaches to be employed simultaneously. For instance, a hybrid approach may operate through a combination of machine learning techniques and rule-based systems. It utilizes the capabilities of both modes to improve speed and accuracy in delivering fine-grained sentiment analysis.\"}),/*#__PURE__*/e(\"h2\",{children:\"Types of sentiment analysis\"}),/*#__PURE__*/e(\"h3\",{children:\"Intent sentiment analysis\"}),/*#__PURE__*/e(\"p\",{children:\"This kind of analysis lets you know whether the customer is eager to make a purchase or is exploring a product. For clients who are keen on buying, the company's specialists arrange special campaigns for them or determine whether it is worth contacting the potential client later with special offers and discounts.\"}),/*#__PURE__*/e(\"h3\",{children:\"Emotion sentiment analysis\"}),/*#__PURE__*/e(\"p\",{children:\"Emotion sentiment analysis involves discovering a person's emotions. Keywords are matched to emotions, yet this is a more challenging task since the computer tries to interpret different human emotions when analyzing sentiment. Emotions in texts do not always have an unambiguous interpretation, because a person who wrote it may, for example, use irony, sarcasm, or multivalent words.\"}),/*#__PURE__*/e(\"h3\",{children:\"Aspect-based sentiment analysis\"}),/*#__PURE__*/e(\"p\",{children:\"Aspect-based sentiment analysis describes a subtype of sentiment analysis designed to identify attitudes about a particular aspect of the underlying subject matter. It includes the extraction of aspects or attributes, the prediction, and classification of sentiment polarity, as well as the aggregation of sentiments.\"}),/*#__PURE__*/e(\"p\",{children:\"Aspect-based sentiment analysis is a process that identifies sentiment in relation to various aspects. Aspects refer to qualities, characteristics, or features of goods or services. Analysis at the aspect level reveals more information about the author's attitude towards the aspects of the object, rather than the object as a whole.\"}),/*#__PURE__*/e(\"p\",{children:\"The aspect retrieval involves the identification of these attributes through customer feedback. Then, experts perform polarity prediction and sentiment classification to decide whether the aspect polarity indicates negative or positive sentiment.\"}),/*#__PURE__*/e(\"h2\",{children:\"Data labeling for sentiment analysis\"}),/*#__PURE__*/e(\"p\",{children:\"Simply put, when performing sentiment analysis, developers usually create their sentiment analysis lexicons. These contain details about which words and phrases are considered positive or negative sentiment.\"}),/*#__PURE__*/e(\"p\",{children:\"The machine learning model for sentiment analysis is trained on massive quantities of carefully selected training data so that they can carry out complex operations such as emotional tone identification, key phrase and feature extraction, word frequency analytics, and overall sentiment detection to provide valuable information.\"}),/*#__PURE__*/e(\"p\",{children:\"Machine learning sentiment analysis mandatorily requires the data to be labeled. A collection of all kinds of labeled content is available in the public domain, but more often than not businesses require their own customized labeling to meet their needs. Crowdsourcing can be a remarkably fast and convenient tool for labeling large amounts of data and building your sentiment analysis solution. For instance, a wide range of performers on Toloka manage to accomplish high-quality labeling in a relatively short period of time.\"}),/*#__PURE__*/e(\"p\",{children:\"For accurate sentiment analysis, the labeling of data is of utmost importance, since this is the material for training and testing the sentiment analysis system. Without accurate data, it will not yield reliable results.\"}),/*#__PURE__*/e(\"h2\",{children:\"How does sentiment analysis work?\"}),/*#__PURE__*/e(\"p\",{children:\"Supervised machine learning methods have become the most commonly applied approaches to sentiment analysis these days. Such methods consist of the initial step of training an ML classifier on previously labeled documents. The model is then applied to new documents.\"}),/*#__PURE__*/e(\"p\",{children:\"To perform sentiment analysis with ML data scientists have to go through the following steps of the process:\"}),/*#__PURE__*/t(\"ol\",{style:{\"--framer-font-size\":\"18px\",\"--framer-text-alignment\":\"start\",\"--framer-text-color\":\"rgb(30, 33, 38)\",\"--framer-text-stroke-width\":\"0px\",\"--framer-text-transform\":\"none\",\"--list-style-type\":\"unset\"},children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Texts collection;\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Text preprocessing or cleanup;\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Data labeling, which consists of the specification of the correct type of sentiment for each text;\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Selection of the classification algorithm and method for training the classifier;\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"The resulting sentiment analysis models are applied to determine the overall sentiment of the new collection of text materials.\"})})]}),/*#__PURE__*/e(\"h2\",{children:\"Sentiment analysis features\"}),/*#__PURE__*/e(\"p\",{children:\"In selecting sentiment analysis tools, there are some attributes worth paying careful attention to that will help you and your business get the complete and most accurate sentiment analysis to understand the public perception of different ideas and concepts or a newly launched product. These are some of the essential sentiment analysis features and strengths that a sentiment analysis tool should have:\"}),/*#__PURE__*/e(\"h3\",{children:\"Precision\"}),/*#__PURE__*/e(\"p\",{children:\"Precision is probably one of the foremost metrics of sentiment analysis. Sentiment analysis tools or platforms have to be capable of accurately assessing manifested sentiments with varied degrees of accuracy. This depends on the level of text sophistication, but once trained on appropriate datasets, high-quality and accurate sentiment analysis should attain as much as 90% of accuracy.\"}),/*#__PURE__*/e(\"h3\",{children:\"Speed and scalability\"}),/*#__PURE__*/e(\"p\",{children:\"Another equally important benchmark for evaluating sentiment analysis tool is its speed and scale. Simply put, sentiment analysis has to handle a great deal of textual data in a relatively short time. In sentiment analysis, speed means the rate at which the software can analyze a piece of content (news article, social media post, online review, video message, survey response, etc.) per second.\"}),/*#__PURE__*/e(\"h3\",{children:\"Personalization\"}),/*#__PURE__*/e(\"p\",{children:\"A flexible sentiment analysis tool also should give users options to adapt its functionality to your industry-specific terminology, item titles, significant assets, and unique semantic subtleties. The type of sentiment analysis you opt for should be appropriate for your individual needs and the demands of the business sector you work in.\"}),/*#__PURE__*/e(\"h2\",{children:\"Sentiment analysis model\"}),/*#__PURE__*/e(\"p\",{children:\"To make sentiment analysis work data scientists basically put instructions in the model. These instructions allow the model to learn. In other words, they build a machine-learning model, embedding a training algorithm in it. Now let's review which algorithms are more frequently employed in the creation of sentiment analysis model.\"}),/*#__PURE__*/e(\"h3\",{children:\"Sentiment analysis algorithms\"}),/*#__PURE__*/e(\"p\",{children:\"Machine learning techniques have expanded considerably in recent years and are still evolving. As mentioned above, supervised learning is widely employed in sentiment analysis. The approach requires sentiment analysis datasets to be preprocessed. Then data scientists feed the system with labeled data, which is further applied to train the classifier, followed by its validation on brand-new texts.\"}),/*#__PURE__*/e(\"p\",{children:\"One of the most accurate linear classifiers, the Support Vector Machine is frequently employed to perform sentiment analysis. Other classic algorithms to analyze sentiment are Naive Bayes Classifier, Decision Tree, and Logistic Regression.\"}),/*#__PURE__*/e(\"p\",{children:\"Currently, various neural networks are more and more often applied to create sentiment analysis tools. For example, recurrent and convolutional networks. The development of the neural network sector, which entailed the introduction of large neural network libraries, has contributed to this.\"}),/*#__PURE__*/e(\"h2\",{children:\"Sentiment analysis challenges\"}),/*#__PURE__*/e(\"h3\",{children:\"Expression of irony or sarcasm\"}),/*#__PURE__*/e(\"p\",{children:\"One major challenge for computers when applying sentiment analysis lies in the difficulty of identifying the exact mood of a text full of sarcasm or irony. This can cause analytical errors and potentially misleading results.\"}),/*#__PURE__*/e(\"h3\",{children:\"Changes in the tone of a word or expression depending on the setting\"}),/*#__PURE__*/e(\"p\",{children:\"The multiple connotations of words that express judgment pose challenges in determining the emotional coloring of a text. Some terms may be neutral in one setting and have a different tone in another setting.\"}),/*#__PURE__*/e(\"h3\",{children:\"Changes in the tone of a word or expression depending on the context\"}),/*#__PURE__*/e(\"p\",{children:\"Another problem of sentiment analysis is related to the situation when the original tone of a word or expression alters depending on the context of use.\"}),/*#__PURE__*/e(\"h3\",{children:\"Quality of the sentiment analysis solution\"}),/*#__PURE__*/e(\"p\",{children:\"The quality of the ML solution is crucial because a poorly trained model can produce unreliable results and damage sentiment analysis performance. Therefore, it is vital to adopt high-quality models that have been trained on diverse and representative training data, as well as to carefully assess the outputs to verify that they are precise and robust.\"}),/*#__PURE__*/e(\"h2\",{children:\"Why is Sentiment Analysis Important?\"}),/*#__PURE__*/e(\"p\",{children:\"The success of a product on the market relies on the customer's perception of it. Marketing specialists, product managers and public relations experts in a company cannot perform 24/7 social media monitoring and read all the existing reviews and comments about their products and services. It's simply not humanly possible. That's when a sentiment analysis tool comes to the rescue of companies. This way businesses survey their customers to build the appropriate image and brand reputation.\"}),/*#__PURE__*/e(\"p\",{children:\"Sentiment analysis helps companies enhance their products and processes with genuine and relevant customer feedback. Sentiment analysis is sometimes applied by customers themselves, mainly when deciding whether or not to purchase a certain product.\"}),/*#__PURE__*/e(\"p\",{children:\"On top of everything else, businesses have to quickly take action in case of potential downturns or emerging trends in today's fast-changing circumstances. Marketing experts rely on sentiment analysis to discover what customers think about a company's brand, products, and services and take immediate action based on their findings. The insight into consumer behavior may be a tremendous asset to companies seeking to highlight each consumer's unique personality and extend their experience with the company's goods.\"}),/*#__PURE__*/e(\"h2\",{children:\"Conclusion\"}),/*#__PURE__*/e(\"p\",{children:\"Sentiment analysis is an essential and useful tool for business, as it gives companies the opportunity to discover what consumers really think about their products and services. Today, thanks to the rapid development of machine learning algorithms, such analysis can be performed quickly and cost-effectively.\"}),/*#__PURE__*/e(\"p\",{children:\"Sentiment analysis may examine a collection of social media posts, articles, documents, and other textual data to assess whether they are positive, negative, or neutral in tone. Experts apply this analysis in a number of fields, ranging from market research and customer feedback review to social media monitoring and political surveys. Sentiment analysis may be applied in customer service applications, such as chatbots and mobile apps. It helps businesses to figure out what customer sentiment is concealed in every complaint, remark, and inquiry, letting businesses grasp deeper insights into their customers' preferences.\"})]});export const richText7=/*#__PURE__*/t(i.Fragment,{children:[/*#__PURE__*/t(\"p\",{children:[\"User-generated content brings both potential benefits and issues to businesses. One of the most difficult challenges for business owners today is how to ensure that their online environment remains secure yet positive through effective content moderation practices. To address this challenge, \",/*#__PURE__*/e(a,{href:\"https://toloka.ai/moderation/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"AI Content Moderation\"})}),\" might be a very advantageous solution, integrating Artificial Intelligence with human expertise in order to manage digital materials more effectively. In this article you will learn about why there\u2019s an urgent need for AI-powered moderation tools as well as its various advantages when implemented into your organization\u2019s operations properly.\"]}),/*#__PURE__*/e(\"h2\",{children:\"Short Summary\"}),/*#__PURE__*/t(\"ul\",{style:{\"--framer-font-size\":\"18px\",\"--framer-text-alignment\":\"start\",\"--framer-text-color\":\"rgb(30, 33, 38)\",\"--framer-text-stroke-width\":\"0px\",\"--framer-text-transform\":\"none\"},children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(a,{href:\"https://toloka.ai/moderation/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"AI-based content moderation\"})}),\" is essential for protecting users from potentially damaging content and moderating an ever-growing volume of user-generated content.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[\"AI techniques such as \",/*#__PURE__*/e(a,{href:\"https://toloka.ai/ml/natural-language-processing/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"NLP\"})}),\", \",/*#__PURE__*/e(a,{href:\"https://toloka.ai/image-data/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"image/video analysis\"})}),\", and \",/*#__PURE__*/e(a,{href:\"https://toloka.ai/audio-data/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,relValues:[],scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"voice recording analysis\"})}),\" enable accurate filtering of various types of online media.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Implementing AI Content Moderation offers enhanced efficiency & scalability, improved accuracy & consistency, cost reduction & risk mitigation.\"})})]}),/*#__PURE__*/e(\"h2\",{children:\"The Growing Need for AI Content Moderation\"}),/*#__PURE__*/e(\"p\",{children:\"To protect users from hazardous material as well as identify patterns that may be undetected by people, AI powered content moderation has become an important tool. It allows platforms to swiftly remove unsuitable posts.\"}),/*#__PURE__*/e(\"p\",{children:\"The use of advanced Content Moderation AI is essential due to the sheer number and diversity of user-submitted information rising continually, making manual monitoring difficult while requiring copious amounts of resources at once.\"}),/*#__PURE__*/e(\"h3\",{children:\"Rise of User-Generated Content\"}),/*#__PURE__*/e(\"p\",{children:\"Content moderation, which often includes user-generated content from various social media platforms and other sites such as images, videos or texts, is made much easier through the utilization of artificial intelligence. AI enables automated responses against habitual wrongdoers with its user-level moderation feature that enhances safety measures while increasing efficiency. Using this technology means that manual moderating efforts become less burdensome by helping to reduce time spent on it tremendously.\"}),/*#__PURE__*/e(\"h3\",{children:\"Brand Reputation and Legal Risks\"}),/*#__PURE__*/e(\"p\",{children:\"Content moderation teams are tasked with eliminating inappropriate content and can benefit from AI-based solutions which swiftly detect said material. This both preserves the platform\u2019s reputation, as well as safeguards its users, while freeing up teams resources for tasks that require more specialized techniques in moderating user generated content. Failure to properly moderate could cause serious repercussions including negative effects on brand image or even result in potential liabilities towards community members.\"}),/*#__PURE__*/e(\"h2\",{children:\"Key Types of AI-Powered Content Moderation\"}),/*#__PURE__*/e(\"p\",{children:\"Content moderation tools, powered by AI technology, are an effective solution for managing a broad range of content. With advanced artificial intelligence techniques these systems can address the specific complexities associated with each type of material. Guaranteeing safe online experiences through active monitoring of online activity.\"}),/*#__PURE__*/e(\"h3\",{children:\"Text Moderation with Natural Language Processing\"}),/*#__PURE__*/e(\"p\",{children:\"Content moderation utilizing natural language processing (NLP) is an essential tool for detecting hate speech, inappropriate user-generated content and other forms of offensive material. This AI technique works by deciphering the underlying text meaning with sentiment analysis and entity recognition data labeling. It helps create sophisticated classifiers which makes it possible to address any ethical considerations that come up when moderating online content efficiently. NLP plays an important role in ensuring accuracy through advanced software systems employed during text moderation - such as knowledge base management, intention detection & more, making sure those viewing user generated materials stay safe from harm caused by this type of activity.\"}),/*#__PURE__*/e(\"h3\",{children:\"Image and Video Analysis\"}),/*#__PURE__*/e(\"p\",{children:\"Content moderation is one of the main functions of AI, which relies on machine learning algorithms, computer vision and text classification to effectively identify any potentially inappropriate content. To ensure user safety through their platforms, tools employed by AI-powered content moderators are equipped with advanced techniques like image processing algorithms as well as visual search approaches. The accuracy level achieved via modern day machines makes it easier for them to quickly filter out dangerous or offensive visuals when conducting video reviews, thus guarding people against malicious material while providing useful insights about incoming data streams across networks or applications online.\"}),/*#__PURE__*/e(\"h3\",{children:\"Voice Recording Analysis\"}),/*#__PURE__*/e(\"p\",{children:\"AI-driven applications are also used for analyzing voice recordings. Determining sentiment and understanding the tone of this data so it can be moderated effectively. These tools enable secure experiences through all forms of media by sifting out offensive content with precision. By utilizing advanced AI powered solutions such as Content Moderation Tools, one can guarantee that only acceptable materials appear in an online space or platform, ensuring brand image safety at its best level possible.\"}),/*#__PURE__*/e(\"h2\",{children:\"The Role of Human Moderators in AI Content Moderation\"}),/*#__PURE__*/e(\"p\",{children:\"It is critical to maintain a safe and positive online experience, which can be achieved through the use of AI content moderation balanced with human oversight. Human moderators bring accuracy and ethics that machines cannot provide. It\u2019s essential to employ both automation as well as people in this process. We will discuss how important it is for these two components (AI & humans) to coexist harmoniously via \u2018the human-in-the-loop\u2019 approach when moderating online content below.\"}),/*#__PURE__*/e(\"h3\",{children:\"Balancing Automation and Human Oversight\"}),/*#__PURE__*/e(\"p\",{children:\"For a safe and successful online experience, the management of content must contain both an automated AI presence as well as individual human intervention. Human moderators are irreplaceable with their knowledge to guarantee accurate moderation is being applied. They can be aided by advances in machine learning for efficient scaling at speed.\"}),/*#__PURE__*/e(\"p\",{children:\"Through combining man-power with artificial intelligence, businesses will have access to more robust methods when it comes to moderating content and sustaining beneficial aspects within the community.\"}),/*#__PURE__*/e(\"h3\",{children:\"Human-in-the-Loop Approach\"}),/*#__PURE__*/e(\"p\",{children:\"So, AI content moderation requires a human-in-the-loop approach where humans and AI systems collaborate to review, refine, and assess decisions. This allows for optimal performance by balancing automation with manual oversight while ensuring accuracy through human feedback. Active learning plays an important role in enhancing the precision of these assessments by incorporating slang into updated language models as well as providing up to date training data that reflects new trends on online platforms, thus implementing reactive strategies essential for improved reaction time from companies when responding or adapting to changing digital landscapes.\"}),/*#__PURE__*/e(\"h2\",{children:\"Advantages of Implementing AI Content Moderation\"}),/*#__PURE__*/e(\"p\",{children:\"In this section we will discuss enhanced efficacy in moderating through AI toolsets including improved accuracy with uniformity, cost saving strategies that reduce liabilities along with mitigating risks associated when dealing with unmoderated data sources like user submissions.\"}),/*#__PURE__*/e(\"h3\",{children:\"Enhanced Efficiency and Scalability\"}),/*#__PURE__*/e(\"p\",{children:\"AI content moderation helps to provide increased efficiency and scalability. It can handle a higher volume of user-generated material while maintaining the same level of quality without engaging human annotators, which allows businesses to maintain their online environment safely. Automation enables human moderators to focus more attention on complex tasks within the moderation process, resulting in improved overall performance for all involved.\"}),/*#__PURE__*/e(\"h3\",{children:\"Improved Accuracy and Consistency\"}),/*#__PURE__*/e(\"p\",{children:\"With its strict adherence to community guidelines, AI content moderation guarantees fairness and uniformity in moderating material. Improved accuracy and consistency in moderated data is another major advantage of employing Artificial Intelligence Content Moderation techniques.\"}),/*#__PURE__*/e(\"h3\",{children:\"Cost Reduction and Risk Mitigation\"}),/*#__PURE__*/e(\"p\",{children:\"By using automated systems instead of human moderators, businesses are able to reduce their spending. It also helps secure the business since it provides real-time analysis which can identify risk areas quickly and accurately before they become an issue. AI makes this entire process much simpler than if done manually as there is no need for constant observation - once set up, AI runs 24/7 ensuring nothing gets through without being caught first!\"}),/*#__PURE__*/e(\"h2\",{children:\"Implementing AI Content Moderation for Your Business\"}),/*#__PURE__*/e(\"p\",{children:\"Selecting the right AI solution for your business\u2019 content moderation needs is an essential step to successful implementation. This involves integration into existing systems and customizing it according to one\u2019s individual requirements. In this article, we will outline the process of selecting such a solution as well as what goes into integrating and personalising it with specifics in mind. By combining these steps, you can ensure that any AI-content moderation operates effectively on behalf of your enterprise.\"}),/*#__PURE__*/e(\"h3\",{children:\"Choosing the Right AI Solution\"}),/*#__PURE__*/e(\"p\",{children:\"When it comes to choosing the appropriate AI technology for content moderation, you should look at different options and decide which one is most suitable for your business needs. Evaluate thoroughly in order to make sure that the solution selected suits your specific content moderation requirements and helps you accomplish what you aim for.\"}),/*#__PURE__*/e(\"p\",{children:\"Considering aspects such as its competency of detecting & deleting prohibited material, capacity of scalability, and compatibility features allowing integration into other existing systems are important factors when making a decision about an AI tool. Cost must be taken into account when assessing these solutions fit for moderating content on behalf of their respective brands\u2019 goals.\"}),/*#__PURE__*/e(\"h3\",{children:\"Integration and Customization\"}),/*#__PURE__*/e(\"p\",{children:\"Integrating and customizing an AI content moderation solution is the key to ensuring that it properly tackles user-generated content. By taking this step, businesses can create a secure environment for their users. The integration process requires incorporating the solution into current systems while customization adjusts its features so as to meet specific needs effectively. Together these measures make sure that moderating online communication goes smoothly and creates a positive experience all around.\"}),/*#__PURE__*/e(\"h2\",{children:\"Summary\"}),/*#__PURE__*/e(\"p\",{children:\"The requirement for AI content moderation is rising and businesses are increasingly leveraging the power of combining Artificial Intelligence with human moderators. These solutions can help manage large volumes of user-generated content, ensuring their brand\u2019s positive reputation while minimizing any potential legal liabilities. With an appropriate selection of AI tools that offer both integration and customization, companies will be able to create a secure digital environment in which customers feel safe. This careful combination of smart technology and experienced people helps them stay on top as they navigate today\u2019s world wide web arena.\"}),/*#__PURE__*/e(\"h2\",{children:\"Frequently Asked Questions\"}),/*#__PURE__*/e(\"h3\",{children:\"What is AI content moderation?\"}),/*#__PURE__*/e(\"p\",{children:\"AI content moderation is a technique used to identify inappropriate and harmful material, achieved through training machine learning algorithms. It utilizes data-driven methods for the purpose of finding patterns that point towards offensive and wrong materials on online platforms. This enables websites to keep their virtual areas secure from any potential dangers presented by such problematic elements. Ultimately it helps maintain safety in digital spaces with aid from Artificial Intelligence based measures like these ones specifically targeting malicious posts/content.\"}),/*#__PURE__*/e(\"h3\",{children:\"Can content moderation be automated?\"}),/*#__PURE__*/e(\"p\",{children:\"Yes, content moderation can be automated. Automated moderation works by relying on a database of predetermined rules and example posts to train the system to identify similar illicit content. It also takes into account a list of prohibited keywords tailored to each specific site.\"}),/*#__PURE__*/e(\"h3\",{children:\"What are the limitations of AI in content moderation?\"}),/*#__PURE__*/e(\"p\",{children:\"Due to the lack of accuracy, transparency and accountability, as well as potential for bias with AI-based content moderation tools, they are best suited for identifying violations quickly rather than replacing human involvement in moderating content.\"}),/*#__PURE__*/e(\"h3\",{children:\"How is AI used in content moderation?\"}),/*#__PURE__*/e(\"p\",{children:\"Using machine learning algorithms, businesses can leverage AI-driven content moderation to effectively track and remove any inappropriate user-generated content from their social media platforms. The accuracy of this process is increasingly improving as advancements in Artificial Intelligence continue to be made. Automation also allows for rapid detection across massive amounts of data generated by users. Thus leading to a faster solution when tackling sensitive topics like online safety.\"})]});\nexport const __FramerMetadata__ = {\"exports\":{\"richText2\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"richText6\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"richText\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"richText5\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"richText1\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"richText7\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"richText3\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"richText4\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"__FramerMetadata__\":{\"type\":\"variable\"}}}"],
  "mappings": "+LAAsJ,IAAMA,EAAsBC,EAAIC,EAAS,CAAC,SAAS,CAAcD,EAAE,KAAK,CAAC,SAAS,CAAC,8DAA2EE,EAAEC,EAAE,CAAC,KAAK,gGAAgG,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,yBAAyB,CAAC,CAAC,CAAC,EAAE,+dAA0d,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,cAAc,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,ueAAue,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wXAAmX,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,mCAAmC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kOAA6N,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wxBAA+vB,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,oCAAoC,UAAU,eAAe,OAAO,MAAM,IAAI,wEAAwE,OAAO,yWAAyW,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,MAAM,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,SAAS,UAAU,CAAC,EAAeA,EAAEC,EAAE,CAAC,KAAK,2FAA2F,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAsBF,EAAE,KAAK,CAAC,SAAS,0FAA0F,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,UAAU,gBAAgB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,oBAAoB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+NAAgN,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,uBAAuB,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kPAAkP,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,kBAAkB,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4LAA4L,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,+BAA+B,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+KAA+K,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,qBAAqB,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0KAA0K,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,6BAA6B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kWAA6V,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+DAA+D,CAAC,EAAeF,EAAE,KAAK,CAAC,MAAM,CAAC,oBAAoB,OAAO,EAAE,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,2BAA2B,CAAC,EAAE,mKAAoJ,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,OAAO,CAAC,EAAE,mIAA8H,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,wBAAwB,CAAC,EAAE,mHAA8G,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,uBAAuB,CAAC,EAAE,2KAA4J,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,YAAY,CAAC,EAAE,qHAA2G,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,gBAAgB,CAAC,EAAE,6KAAwK,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,2BAA2B,CAAC,EAAE,2GAAsG,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,uBAAuB,CAAC,EAAE,oJAA+I,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,iBAAiB,CAAC,EAAE,sHAAiH,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,gBAAgB,CAAC,EAAE,4DAAuD,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,4BAA4B,CAAC,EAAE,kFAA6E,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,yBAAyB,CAAC,EAAE,qDAAgD,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,aAAa,CAAC,EAAE,iFAA4E,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,mBAAmB,CAAC,EAAE,4EAAuE,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,MAAM,CAAC,EAAE,wGAAmG,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,aAAa,CAAC,EAAE,gGAA2F,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,oBAAoB,CAAC,EAAE,yGAAoG,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0IAA0I,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uEAAuE,CAAC,EAAeF,EAAE,KAAK,CAAC,MAAM,CAAC,oBAAoB,OAAO,EAAE,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,YAAY,CAAC,EAAE,+GAA0G,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,kBAAkB,CAAC,EAAE,oKAA0J,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,wCAAwC,CAAC,EAAE,oEAA+D,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,2DAA2D,CAAC,EAAE,mGAA8F,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,WAAW,CAAC,EAAE,+IAA0I,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,sBAAsB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wSAAwS,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,uBAAuB,UAAU,eAAe,OAAO,MAAM,IAAI,uEAAuE,OAAO,uQAAuQ,MAAM,CAAC,YAAY,YAAY,EAAE,MAAM,KAAK,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,qDAAqD,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,irBAAirB,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,GAAG,UAAU,eAAe,OAAO,MAAM,IAAI,uEAAuE,OAAO,0KAA0K,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,KAAK,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,SAAS,UAAU,CAAC,EAAeA,EAAEC,EAAE,CAAC,KAAK,2FAA2F,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAsBF,EAAE,KAAK,CAAC,SAAS,qEAAqE,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,UAAU,gBAAgB,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qNAA2M,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qJAAqJ,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,+BAA+B,CAAC,EAAE,gKAAsJ,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,oBAAoB,CAAC,EAAE,yJAAoJ,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,6BAA6B,CAAC,EAAE,mJAAyI,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,2BAA2B,CAAC,EAAE,oIAA+H,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,oCAAoC,CAAC,EAAE,+HAA0H,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,iBAAiB,CAAC,EAAE,kKAAwJ,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,cAAc,CAAC,EAAE,wHAAmH,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,iBAAiB,CAAC,EAAE,6MAAmM,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,8CAA8C,CAAC,EAAE,uNAAwM,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,2BAA2B,CAAC,EAAE,wGAAmG,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,aAAa,CAAC,EAAE,0FAAqF,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,yBAAyB,CAAC,EAAE,qHAAgH,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,uBAAuB,CAAC,EAAE,mHAA8G,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,mCAAmC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sbAAib,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,mBAAmB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sIAAiI,CAAC,EAAeF,EAAE,KAAK,CAAC,MAAM,CAAC,oBAAoB,OAAO,EAAE,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,sBAAsB,CAAC,EAAE,mJAA8I,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,oBAAoB,CAAC,EAAE,sIAAiI,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,iCAAiC,CAAC,EAAE,iIAA4H,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,mBAAmB,CAAC,EAAE,wIAAmI,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,mCAAmC,CAAC,EAAE,sHAAiH,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,gBAAgB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+RAA+R,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sIAAsI,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,0BAA0B,CAAC,EAAE,+JAA0J,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,4BAA4B,CAAC,EAAE,0OAAqO,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,sCAAsC,CAAC,EAAE,sIAAiI,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,mCAAmC,CAAC,EAAE,sJAAiJ,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,2BAA2B,CAAC,EAAE,oJAA+I,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,sBAAsB,CAAC,EAAE,+IAAqI,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,+BAA+B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oMAAoM,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,kDAAkD,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kiBAAmhB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oKAA0J,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,qBAAqB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wlBAAykB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wCAAwC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,wEAAwE,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,sCAAsC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,mFAAmF,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4lBAA4lB,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,qDAAgD,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qPAAqP,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kDAAkD,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,+CAA+C,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,mCAAmC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,0CAA0C,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,4CAA4C,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,0CAA0C,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6TAAwT,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,0CAA0C,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mlBAA+jB,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,mDAAmD,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uJAAuJ,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,sBAAsB,CAAC,EAAE,+JAA0J,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,sBAAsB,CAAC,EAAE,kRAAwQ,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,weAAyd,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,mCAAmC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+JAA0J,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,keAAke,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sYAAsY,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,eAAe,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kiBAAkiB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uRAAuR,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uIAAuI,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,cAAc,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,SAAS,4fAA4f,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeG,EAAuBL,EAAIC,EAAS,CAAC,SAAS,CAAcD,EAAE,IAAI,CAAC,SAAS,CAAC,+TAA4UE,EAAEC,EAAE,CAAC,KAAK,0GAA0G,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,uCAAuC,CAAC,CAAC,CAAC,EAAE,QAAqBF,EAAEC,EAAE,CAAC,KAAK,yGAAyG,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,iFAAiF,CAAC,CAAC,CAAC,EAAE,wYAAmY,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,wRAAwR,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,aAAa,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,gNAA6NE,EAAEC,EAAE,CAAC,KAAK,4DAA4D,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,WAAW,CAAC,CAAC,CAAC,EAAE,iUAAiU,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,8OAA8O,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,gEAAgE,UAAU,eAAe,OAAO,MAAM,IAAI,uEAAuE,OAAO,qWAAqW,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,MAAM,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,SAAS,8CAA8C,CAAC,EAAeA,EAAEC,EAAE,CAAC,KAAK,4DAA4D,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAsBF,EAAE,KAAK,CAAC,SAAS,WAAW,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,UAAU,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kiBAAkiB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,ggBAAggB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0FAA0F,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,kCAAkC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,qCAAkDE,EAAEC,EAAE,CAAC,KAAK,sCAAsC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,cAAc,CAAC,CAAC,CAAC,EAAE,6VAA6V,CAAC,CAAC,EAAeF,EAAE,MAAM,CAAC,IAAI,mCAAmC,UAAU,eAAe,OAAO,MAAM,IAAI,sEAAsE,OAAO,iWAAiW,MAAM,CAAC,YAAY,YAAY,EAAE,MAAM,MAAM,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kjBAAkjB,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,uBAAuB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6iBAA6iB,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,8BAA2CE,EAAE,OAAO,CAAC,SAAS,QAAQ,CAAC,EAAE,siBAAuiB,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2zBAA8zB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4qBAAuqB,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,sFAAsF,UAAU,eAAe,OAAO,MAAM,IAAI,wEAAwE,OAAO,yWAAyW,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,MAAM,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,SAAS,qFAAqF,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,siBAAsiB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wDAAwD,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,kEAAkE,UAAU,eAAe,OAAO,OAAO,IAAI,wEAAwE,OAAO,0WAA0W,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,MAAM,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,SAAS,iEAAiE,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gRAAgR,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,+CAA+C,UAAU,eAAe,OAAO,MAAM,IAAI,uEAAuE,OAAO,qWAAqW,MAAM,CAAC,YAAY,YAAY,EAAE,MAAM,MAAM,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,6ZAA0aE,EAAEC,EAAE,CAAC,KAAK,iCAAiC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,iBAAiB,CAAC,CAAC,CAAC,EAAE,GAAG,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,2BAA2B,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,8JAA2KE,EAAEC,EAAE,CAAC,KAAK,6EAA6E,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,cAAc,CAAC,CAAC,CAAC,EAAE,0LAA0L,CAAC,CAAC,EAAeF,EAAE,MAAM,CAAC,IAAI,4BAA4B,UAAU,eAAe,OAAO,MAAM,IAAI,uEAAuE,OAAO,qWAAqW,MAAM,CAAC,YAAY,YAAY,EAAE,MAAM,MAAM,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,sYAAmZE,EAAEC,EAAE,CAAC,KAAK,0GAA0G,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,mBAAmB,CAAC,CAAC,CAAC,EAAE,GAAG,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,iiBAAuhB,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,aAAa,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,gZAAwZE,EAAEC,EAAE,CAAC,KAAK,yGAAyG,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,gFAAgF,CAAC,CAAC,CAAC,EAAE,uGAAkG,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,WAAW,CAAC,EAAeF,EAAE,KAAK,CAAC,MAAM,CAAC,qBAAqB,OAAO,0BAA0B,QAAQ,sBAAsB,kBAAkB,6BAA6B,MAAM,0BAA0B,OAAO,oBAAoB,OAAO,EAAE,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAC,gJAA6JE,EAAE,KAAK,CAAC,SAAS,uFAAuF,CAAC,EAAE,eAAe,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAC,0FAAuGE,EAAE,KAAK,CAAC,SAAS,MAAM,CAAC,EAAE,0BAAuCA,EAAEC,EAAE,CAAC,KAAK,kCAAkC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,iCAAiC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeE,EAAuBN,EAAIC,EAAS,CAAC,SAAS,CAAcC,EAAE,IAAI,CAAC,SAAS,iVAAiV,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6NAA6N,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8VAA8V,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,cAAc,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yYAAyY,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,uBAAuB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gNAAgN,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4FAA4F,CAAC,EAAeF,EAAE,KAAK,CAAC,MAAM,CAAC,qBAAqB,OAAO,0BAA0B,QAAQ,sBAAsB,kBAAkB,6BAA6B,MAAM,0BAA0B,MAAM,EAAE,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAEC,EAAE,CAAC,KAAK,4CAA4C,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,iBAAiB,CAAC,CAAC,CAAC,EAAE,uBAAuB,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,yGAAoG,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,4EAA4E,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,iBAAiB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gVAAgV,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,ygBAAygB,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,+CAA0C,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kQAAkQ,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4SAA4S,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2hBAA2hB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6WAA6W,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,yCAAyC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gcAAgc,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,qBAAqB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6bAA6b,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4RAA4R,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4cAA4c,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,wBAAwB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mqBAAmqB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+cAA+c,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,uBAAuB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yOAAyO,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iaAAia,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,ySAAyS,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uPAAuP,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,iBAAiB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,whBAAwhB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gkBAAgkB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kPAAkP,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,6CAA6C,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wHAAwH,CAAC,EAAeF,EAAE,KAAK,CAAC,MAAM,CAAC,qBAAqB,OAAO,0BAA0B,QAAQ,sBAAsB,kBAAkB,6BAA6B,MAAM,0BAA0B,OAAO,oBAAoB,OAAO,EAAE,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,ySAAyS,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,2IAA2I,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,sQAAsQ,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,wCAAwC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0XAA0X,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+TAA+T,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iXAAiX,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,qEAAqE,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yNAAyN,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0MAA0M,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0PAA0P,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yPAAyP,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oRAAoR,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yYAAyY,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,0CAA0C,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8VAA8V,CAAC,EAAeF,EAAE,KAAK,CAAC,MAAM,CAAC,qBAAqB,OAAO,0BAA0B,QAAQ,sBAAsB,kBAAkB,6BAA6B,MAAM,0BAA0B,MAAM,EAAE,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAsBA,EAAEC,EAAE,CAAC,KAAK,sCAAsC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,oCAAoC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAsBA,EAAEC,EAAE,CAAC,KAAK,wCAAwC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,4BAA4B,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAsBA,EAAEC,EAAE,CAAC,KAAK,wCAAwC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,iCAAiC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAsBA,EAAEC,EAAE,CAAC,KAAK,gCAAgC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,sBAAsB,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAEC,EAAE,CAAC,KAAK,gCAAgC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,oBAAoB,CAAC,CAAC,CAAC,EAAE,YAAY,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,SAAS,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAEC,EAAE,CAAC,KAAK,4CAA4C,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,eAAe,CAAC,CAAC,CAAC,EAAE,8OAA8O,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,0YAA0Y,CAAC,CAAC,CAAC,CAAC,EAAeK,EAAuBP,EAAIC,EAAS,CAAC,SAAS,CAAcC,EAAE,KAAK,CAAC,SAAS,kBAAkB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uTAA6S,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,gBAAW,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qNAAqN,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uTAA6S,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+RAA+R,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oDAA+C,CAAC,EAAeF,EAAE,KAAK,CAAC,MAAM,CAAC,qBAAqB,OAAO,0BAA0B,QAAQ,sBAAsB,kBAAkB,6BAA6B,MAAM,0BAA0B,MAAM,EAAE,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,uFAAuF,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,iEAAiE,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,0FAA0F,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,uEAAuE,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4VAA4V,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,UAAU,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4HAAuH,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iJAAwH,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,4IAAwH,UAAU,eAAe,OAAO,MAAM,IAAI,wEAAwE,OAAO,yWAAyW,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,MAAM,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4RAA4R,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kDAA6C,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oJAAoJ,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,oJAAoJ,UAAU,eAAe,OAAO,MAAM,IAAI,qEAAqE,OAAO,6VAA6V,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,MAAM,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oOAAoO,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,oFAAoF,UAAU,eAAe,OAAO,MAAM,IAAI,wEAAwE,OAAO,yWAAyW,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,MAAM,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wNAAwN,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,0DAA0D,UAAU,eAAe,OAAO,MAAM,IAAI,wEAAwE,OAAO,yWAAyW,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,MAAM,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2DAA2D,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,SAAS,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iPAAiP,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+mBAAqmB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oLAAoL,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mJAAmJ,CAAC,CAAC,CAAC,CAAC,EAAeM,EAAuBR,EAAIC,EAAS,CAAC,SAAS,CAAcD,EAAE,IAAI,CAAC,SAAS,CAAC,6BAA0CE,EAAEC,EAAE,CAAC,KAAK,8CAA8C,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,kBAAkB,CAAC,CAAC,CAAC,EAAE,KAAkBF,EAAEC,EAAE,CAAC,KAAK,sCAAsC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,UAAU,CAAC,CAAC,CAAC,EAAE,SAAsBF,EAAEC,EAAE,CAAC,KAAK,8BAA8B,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,YAAY,CAAC,CAAC,CAAC,EAAE,8FAAoF,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,iVAAkU,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uOAAuO,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4QAAuQ,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0yBAAqyB,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,mQAAgRE,EAAE,KAAK,CAAC,SAAS,mDAAmD,CAAC,EAAE,iFAAiF,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,MAAmBE,EAAEC,EAAE,CAAC,KAAK,qCAAqC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,SAAS,CAAC,CAAC,CAAC,EAAE,kCAAkC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,UAAU,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,oWAAwVE,EAAEC,EAAE,CAAC,KAAK,oCAAoC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,aAAa,CAAC,CAAC,CAAC,EAAE,8FAA8F,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,kDAAkD,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,mDAAmD,UAAU,eAAe,OAAO,MAAM,IAAI,sEAAsE,OAAO,iWAAiW,MAAM,CAAC,YAAY,YAAY,EAAE,MAAM,MAAM,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,uDAAoEE,EAAE,KAAK,CAAC,SAAS,KAAK,CAAC,EAAE,kFAAkF,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,wDAAwD,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iSAAiS,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,2DAA2D,UAAU,eAAe,OAAO,MAAM,IAAI,uEAAuE,OAAO,qWAAqW,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,MAAM,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,SAAS,yCAAyC,CAAC,EAAeA,EAAEC,EAAE,CAAC,KAAK,mCAAmC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAsBF,EAAE,KAAK,CAAC,SAAS,mBAAmB,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,GAAG,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iFAAiF,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,8BAA8B,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,iCAAiC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,mDAAmD,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAC,oDAAiEE,EAAEC,EAAE,CAAC,KAAK,yCAAyC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,YAAY,CAAC,CAAC,CAAC,EAAE,0BAA0B,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,iCAAiC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,mZAAgaE,EAAE,KAAK,CAAC,SAAS,0CAAgC,CAAC,EAAE,sFAAsF,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wMAAwM,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gUAA2T,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,4RAAoSE,EAAEC,EAAE,CAAC,KAAK,wCAAwC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,SAAS,CAAC,CAAC,CAAC,EAAE,+GAA+G,CAAC,CAAC,EAAeJ,EAAE,IAAI,CAAC,SAAS,CAAC,wJAAqKE,EAAE,KAAK,CAAC,SAAS,kBAAkB,CAAC,EAAE,iEAAiE,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6bAAmb,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,+BAA+B,UAAU,eAAe,OAAO,MAAM,IAAI,uEAAuE,OAAO,qWAAqW,MAAM,CAAC,YAAY,YAAY,EAAE,MAAM,MAAM,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,SAAS,8BAA8B,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4FAA4F,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,yBAAyB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oaAA+Z,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,uBAAuB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+bAA+b,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kMAAkM,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,6FAA6F,UAAU,eAAe,OAAO,OAAO,IAAI,uEAAuE,OAAO,wQAAwQ,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,MAAM,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,SAAS,4FAA4F,CAAC,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,wKAAqLE,EAAEC,EAAE,CAAC,KAAK,0CAA0C,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,QAAQ,CAAC,CAAC,CAAC,EAAE,GAAG,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,iBAAiB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4vBAAwuB,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,mGAAmG,UAAU,eAAe,OAAO,MAAM,IAAI,sEAAsE,OAAO,iWAAiW,MAAM,CAAC,YAAY,YAAY,EAAE,MAAM,MAAM,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uKAAkK,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mUAAmU,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,wCAAwC,UAAU,eAAe,OAAO,MAAM,IAAI,uEAAuE,OAAO,qWAAqW,MAAM,CAAC,YAAY,YAAY,EAAE,MAAM,MAAM,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,SAAS,uCAAuC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0DAA0D,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,oDAAoD,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,wLAAgME,EAAEC,EAAE,CAAC,KAAK,mCAAmC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,MAAM,CAAC,CAAC,CAAC,EAAE,4GAA4G,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,2OAA2O,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kUAAkU,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,oBAAoB,UAAU,eAAe,OAAO,MAAM,IAAI,wEAAwE,OAAO,yWAAyW,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,MAAM,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,SAAS,mBAAmB,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sFAAsF,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,0BAAqB,UAAU,eAAe,OAAO,MAAM,IAAI,sEAAsE,OAAO,iWAAiW,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,MAAM,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,SAAS,yBAAoB,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8PAAyP,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,cAAc,UAAU,eAAe,OAAO,MAAM,IAAI,wEAAwE,OAAO,yWAAyW,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,MAAM,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,gBAAgB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uGAAuG,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,wDAAwD,UAAU,eAAe,OAAO,MAAM,IAAI,uEAAuE,OAAO,qWAAqW,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,MAAM,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,SAAS,uDAAuD,CAAC,CAAC,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,gEAAgE,UAAU,eAAe,OAAO,MAAM,IAAI,uEAAuE,OAAO,qWAAqW,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,MAAM,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,SAAS,+DAA+D,CAAC,CAAC,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,iDAAiD,UAAU,eAAe,OAAO,OAAO,IAAI,uEAAuE,OAAO,wQAAwQ,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,MAAM,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,SAAS,gDAAgD,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8GAA8G,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,kBAAa,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kLAAkL,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,gDAA6DE,EAAEC,EAAE,CAAC,KAAK,6DAA6D,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,EAAE,mCAAgDF,EAAEC,EAAE,CAAC,KAAK,kDAAkD,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,EAAE,GAAG,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,2BAA2B,CAAC,CAAC,CAAC,CAAC,EAAeO,EAAuBT,EAAIC,EAAS,CAAC,SAAS,CAAcC,EAAE,KAAK,CAAC,SAAS,kBAAkB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,yMAA+L,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wKAAwK,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gVAA2U,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kMAAkM,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+QAA0Q,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2DAA2D,CAAC,EAAeF,EAAE,KAAK,CAAC,MAAM,CAAC,qBAAqB,OAAO,0BAA0B,QAAQ,sBAAsB,kBAAkB,6BAA6B,MAAM,0BAA0B,MAAM,EAAE,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,uBAAuB,CAAC,EAAE,yGAAoG,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,oBAAoB,CAAC,EAAE,oLAA+K,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,eAAe,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,0KAAqK,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+WAAqW,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uBAAuB,CAAC,EAAeF,EAAE,KAAK,CAAC,MAAM,CAAC,qBAAqB,OAAO,0BAA0B,QAAQ,sBAAsB,kBAAkB,6BAA6B,MAAM,0BAA0B,MAAM,EAAE,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,8CAA8C,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,6DAA6D,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2HAA2H,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mOAAmO,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,UAAU,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iRAA4Q,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iWAAiW,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,aAAa,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4LAA4L,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uVAAuV,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8PAAyP,CAAC,CAAC,CAAC,CAAC,EAAeQ,EAAuBV,EAAIC,EAAS,CAAC,SAAS,CAAcC,EAAE,IAAI,CAAC,SAAS,gTAAgT,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qbAAgb,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,6BAA6B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,ySAAyS,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qRAAqR,CAAC,EAAeF,EAAE,KAAK,CAAC,MAAM,CAAC,qBAAqB,OAAO,0BAA0B,QAAQ,sBAAsB,kBAAkB,6BAA6B,MAAM,0BAA0B,MAAM,EAAE,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,qBAAqB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,qBAAqB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,wXAAwX,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gXAAgX,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qRAAqR,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,wCAAwC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2SAA2S,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kTAAkT,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,kCAAkC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kEAAkE,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,qBAAqB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oWAAoW,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gYAAgY,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,wBAAwB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mWAAmW,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,iBAAiB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kUAAkU,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,6BAA6B,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,2BAA2B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4TAA4T,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,4BAA4B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mYAAmY,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,iCAAiC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+TAA+T,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+UAA+U,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wPAAwP,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,sCAAsC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iNAAiN,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2UAA2U,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,ihBAAihB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8NAA8N,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,mCAAmC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2QAA2Q,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8GAA8G,CAAC,EAAeF,EAAE,KAAK,CAAC,MAAM,CAAC,qBAAqB,OAAO,0BAA0B,QAAQ,sBAAsB,kBAAkB,6BAA6B,MAAM,0BAA0B,OAAO,oBAAoB,OAAO,EAAE,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,mBAAmB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,gCAAgC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,oGAAoG,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,mFAAmF,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,iIAAiI,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,6BAA6B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sZAAsZ,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,WAAW,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qYAAqY,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,uBAAuB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8YAA8Y,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,iBAAiB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qVAAqV,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,0BAA0B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8UAA8U,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,+BAA+B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iZAAiZ,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iPAAiP,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qSAAqS,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,+BAA+B,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,gCAAgC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kOAAkO,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,sEAAsE,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kNAAkN,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,sEAAsE,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0JAA0J,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,4CAA4C,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mWAAmW,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,sCAAsC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6eAA6e,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0PAA0P,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sgBAAsgB,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,YAAY,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uTAAuT,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,onBAAonB,CAAC,CAAC,CAAC,CAAC,EAAeS,EAAuBX,EAAIC,EAAS,CAAC,SAAS,CAAcD,EAAE,IAAI,CAAC,SAAS,CAAC,wSAAqTE,EAAEC,EAAE,CAAC,KAAK,gCAAgC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,uBAAuB,CAAC,CAAC,CAAC,EAAE,oWAA0V,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,eAAe,CAAC,EAAeF,EAAE,KAAK,CAAC,MAAM,CAAC,qBAAqB,OAAO,0BAA0B,QAAQ,sBAAsB,kBAAkB,6BAA6B,MAAM,0BAA0B,MAAM,EAAE,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAEC,EAAE,CAAC,KAAK,gCAAgC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,6BAA6B,CAAC,CAAC,CAAC,EAAE,uIAAuI,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAC,yBAAsCE,EAAEC,EAAE,CAAC,KAAK,oDAAoD,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,KAAK,CAAC,CAAC,CAAC,EAAE,KAAkBF,EAAEC,EAAE,CAAC,KAAK,gCAAgC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,sBAAsB,CAAC,CAAC,CAAC,EAAE,SAAsBF,EAAEC,EAAE,CAAC,KAAK,gCAAgC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,UAAU,CAAC,EAAE,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,0BAA0B,CAAC,CAAC,CAAC,EAAE,8DAA8D,CAAC,CAAC,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,iJAAiJ,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,4CAA4C,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6NAA6N,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yOAAyO,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,gCAAgC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,igBAAigB,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,kCAAkC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mhBAA8gB,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,4CAA4C,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qVAAqV,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,kDAAkD,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0vBAA0vB,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,0BAA0B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4sBAA4sB,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,0BAA0B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,ufAAuf,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,uDAAuD,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mfAAoe,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,0CAA0C,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0VAA0V,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0MAA0M,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,4BAA4B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kpBAAkpB,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,kDAAkD,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0RAA0R,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,qCAAqC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mcAAmc,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,mCAAmC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wRAAwR,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,oCAAoC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mcAAmc,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,sDAAsD,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,ihBAAugB,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,gCAAgC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yVAAyV,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yYAAoY,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,+BAA+B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+fAA+f,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,SAAS,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qpBAA2oB,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,4BAA4B,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,gCAAgC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mkBAAmkB,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,sCAAsC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0RAA0R,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,uDAAuD,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4PAA4P,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,uCAAuC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+eAA+e,CAAC,CAAC,CAAC,CAAC,EAC5s1IU,EAAqB,CAAC,QAAU,CAAC,UAAY,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,UAAY,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,SAAW,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,UAAY,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,UAAY,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,UAAY,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,UAAY,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,UAAY,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,mBAAqB,CAAC,KAAO,UAAU,CAAC,CAAC",
  "names": ["richText", "u", "x", "p", "Link", "motion", "richText1", "richText2", "richText3", "richText4", "richText5", "richText6", "richText7", "__FramerMetadata__"]
}