{
  "version": 3,
  "sources": ["ssg:https://framerusercontent.com/modules/YVgynkVt96a0ES9Gk8yz/kOBXWPCp7Mt5VbUuUIlc/eo4RAmtig-13.js"],
  "sourcesContent": ["import{jsx as e,jsxs as t}from\"react/jsx-runtime\";import{Link as a}from\"framer\";import{motion as n}from\"framer-motion\";import*as i from\"react\";export const richText=/*#__PURE__*/t(i.Fragment,{children:[/*#__PURE__*/e(\"p\",{children:\"Pre-trained, general-purpose large languagemodels (LLMs) provide a strong foundation for natural language understanding and generation.\"}),/*#__PURE__*/e(\"p\",{children:\"So, why should you build a domain-specific LLM, and not a general-purpose LLM that is pre-trained on domain-related data?\"}),/*#__PURE__*/e(\"p\",{children:\"In simple terms, fine-tuning LLMs on domain-specific data allows specialists to better harness their models\u2019 capabilities. Fine-tuning LLMs on domain-specific data allows specialists to harness their capabilities and adapt them to the specific requirements and challenges of a specific domain. This results in more accurate, relevant, and efficient language processing solutions. In this article, we discuss why you should tailor your generic language model to be knowledgeable in a specialized area of expertise.\"}),/*#__PURE__*/e(a,{href:\"https://toloka.ai/large-language-models/?utm_source=blog&utm_medium=banner&utm_campaign=sft&utm_content=domainspecificllms\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!0,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{className:\"framer-image\",\"data-preset-tag\":\"img\",children:/*#__PURE__*/e(\"img\",{alt:\"\",className:\"framer-image\",height:\"240\",src:\"https://framerusercontent.com/images/xpUvAvgGNUA6gpXxlqHiClcP1U.png\",srcSet:\"https://framerusercontent.com/images/xpUvAvgGNUA6gpXxlqHiClcP1U.png?scale-down-to=512 512w,https://framerusercontent.com/images/xpUvAvgGNUA6gpXxlqHiClcP1U.png?scale-down-to=1024 1024w,https://framerusercontent.com/images/xpUvAvgGNUA6gpXxlqHiClcP1U.png 1498w\",style:{aspectRatio:\"1498 / 480\"},width:\"749\"})})}),/*#__PURE__*/e(\"h2\",{children:\"What is a Domain Specific LLM?\"}),/*#__PURE__*/e(\"p\",{children:\"A Domain Specific Language Model (LLM) is a type of language model that has been conditioned to excel in a specific domain or field of knowledge.A general-purpose language model is one that has been trained on a wide range of text from various domains. A domain-specific LLM is optimized to understand and generate relevant for a particular subject area.\"}),/*#__PURE__*/e(\"p\",{children:\"For instance, a general language model like GPT-4o can generate text on a variety of topics. Aspecialist may strictly condition a domain-specific language model on data pertaining to legal matters, medical literature, financial reports, or other specialized fields. This specialization allows the model to generate more accurate and contextually appropriate outputs within that domain.\"}),/*#__PURE__*/e(\"p\",{children:\"Pre-trained foundational models are not suitable for real-world business applications. It doesn\u2019t mean that they are useless. General-purpose language models contain a lot of data and can perform a multitude of tasks. However, they lack domain-specific knowledge such as:\"}),/*#__PURE__*/t(\"ul\",{style:{\"--framer-font-size\":\"18px\",\"--framer-text-alignment\":\"start\",\"--framer-text-color\":\"rgb(30, 33, 38)\",\"--framer-text-stroke-width\":\"0px\",\"--framer-text-transform\":\"none\"},children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Product specifications\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Company policies\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Financial data\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Industry-specific business terminology\"})})]}),/*#__PURE__*/e(\"p\",{children:\"Only after being fine-tuned on this specialized domain knowledge can the LLM perform specific tasks required by the industry the model is being used in.\"}),/*#__PURE__*/e(\"h2\",{children:\"Why do you need to tailor your LLM to be domain-specific?\"}),/*#__PURE__*/e(\"p\",{children:\"Many generic large language models can produce high-quality human-like responses. However, the results they give are not always accurate. These models are prone to hallucinations that may be conveyed with such a degree of confidence that a user won\u2019t be able to detect whether it\u2019s true or false. The information they provide should be thoroughly checked before making use of it.\"}),/*#__PURE__*/e(\"p\",{children:\"Unlike domain-specific models, untuned large language models have knowledge of many topics, but their expertise is very superficial. Sometimes, they, make up information about things they have no understanding of such as when their conditioning data creates a limiting or incorrect bias. Essentially, large language models cannot distinguish truth from misinformation or lies as well as most humans can, as they do not possess conscious awareness of the veracity of the information they are processing. They simply predict sequences of words based on their conditioning data.Much like humans who develop biases based on the media they consume and their own experiences, LLMs can only reason based on the data they are fine-tuned with.\"}),/*#__PURE__*/e(\"p\",{children:\"Since many of these models have mastered predicting sequences of words in natural language, false outputs from many generic language models can seem as legitimate as those from reliable human sources., In fields like business, law, and medicine, accurate and truthful information is critical for any issue. , Large language models must be provided with as much reliable information on a particular topic as possible.\"}),/*#__PURE__*/e(\"p\",{children:\"Experts create custom large language models based on foundation models. They train them on specialized data that is verified, current, and relevant to teach models to perform domain-specific tasks. TThe resulting outputs of these models is more accurate because it was fine-tuned with domain-specific context.\"}),/*#__PURE__*/e(\"h2\",{children:\"Benefits of customized LLM\"}),/*#__PURE__*/e(\"p\",{children:\"Domain-specific Language Models (LLMs) offer several benefits that general-purpose language models aren\u2019t engineered for:\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Improved Efficiency.\"}),\" Since domain-specific LLMs are trained on data from a specific field, they can provide quicker and more suitable answers for tasks in that field. Customized LLMs consume less time and fewer steps than general models to obtain the required information. And they don\u2019t require the user to sift through irrelevant details;\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Cost Savings.\"}),\" Cost savings are a big advantage that differentiate domain-specific LLMs from general models. They help organizations reduce costs by making workflows run faster and more efficiently without compromising quality.\"]}),/*#__PURE__*/e(\"p\",{children:\"They do so by:\"}),/*#__PURE__*/t(\"ul\",{style:{\"--framer-font-size\":\"18px\",\"--framer-text-alignment\":\"start\",\"--framer-text-color\":\"rgb(30, 33, 38)\",\"--framer-text-stroke-width\":\"0px\",\"--framer-text-transform\":\"none\"},children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Automating tasks\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Reducing errors,\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Enabling employees to concentrate on more strategic activities that create more value.\"})})]}),/*#__PURE__*/e(\"p\",{children:\"These improvements boost productivity and make an organization\u2019soperations more cost-effective in the long run;\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Higher Accuracy:\"}),\" Domain-specific LLM models can better understand and generate domain-relevant text. During fine-tuning models get better at gathering and producing text that aligns with the respective field. They are trained on field- specific data, which helps them learn the domain\u2019s intricacies, vocabulary, jargon, and context, leading to more accurate results. This helps them give more spot-on results.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Customization and Adaptability:\"}),\" Organizations can fine-tune domain-specific LLMs further to their exact requirements. They can achieve this by training the model on their own proprietary or specialized datasets, and tailoring it to their unique requirements,enhancing its performance in their particular domain.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Specialized Vocabulary:\"}),\" Domain-specific LLMs are better equipped to understand and generate text containing specialized terminology and vocabulary that are unique to the domain. This enables them to produce more contextually appropriate responses.\"]}),/*#__PURE__*/e(\"p\",{children:\"Domain-specific LLMs excel at understanding and creating text with all the special words and phrases unique to a particular field. This means they can give answers that better align with each situation..\"}),/*#__PURE__*/e(\"h2\",{children:\"How to Create a Domain-specific LLM\"}),/*#__PURE__*/e(\"h3\",{children:\"Training a model from scratch or fine-tuning\"}),/*#__PURE__*/e(\"p\",{children:\"There are two ways to create a domain-specific model:\"}),/*#__PURE__*/t(\"ol\",{style:{\"--framer-font-size\":\"18px\",\"--framer-text-alignment\":\"start\",\"--framer-text-color\":\"rgb(30, 33, 38)\",\"--framer-text-stroke-width\":\"0px\",\"--framer-text-transform\":\"none\",\"--list-style-type\":\"unset\"},children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Creating the model from scratch\"}),\" or. This option is more complex, because training a model from scratch (domain-specific pre-training) requires a massive domain-specific training dataset with very high-quality data. The process of creating these models involves self-supervised learning with unlabeled data.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Thoroughly train a foundation model:\"}),\" Many companies do not have the luxury of creating a model from scratch. Doing so demands considerable time and computational resources and a large amount of specialized data. After training the algorithm, data scientists test the model and tune its parameters for better performance. The data quality also directly affects the final result of a model trained from scratch.\"]})})]}),/*#__PURE__*/e(\"p\",{children:\"Additional training of an existing model is much easier than training a model from its inception. Significantly less data is required for training in this scenario, since the model already has a baseline of language knowledge. The model weights gradually change as data scientists use small sets of training data containing specialized knowledge. Pre-existing content that already exists within a pre-trained model is retained.\"}),/*#__PURE__*/t(\"p\",{children:[\"Both approaches can generate high performance levels. However, in some fields, language models pre-trained on domain-specific data demonstrate better \",/*#__PURE__*/e(a,{href:\"http://arxiv.org/pdf/2104.08671\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"results\"})}),\".\"]}),/*#__PURE__*/e(\"h2\",{children:\"Other methods of customizing LLMs for a domain\"}),/*#__PURE__*/e(\"h3\",{children:\"Retrieval augmented generation (RAG)\"}),/*#__PURE__*/e(\"p\",{children:\"RAG first retrieves relevant passages or documents from a knowledge source using a retrieval model. Next, it generates a response based on the retrieved information using a generation model. This approach allows large language models to extract domain-specific knowledge from external real-time databases, ensuring that the generated response is accurate and up-to-date.\"}),/*#__PURE__*/e(\"p\",{children:\"Legal professionals often need to review and analyze large volumes of legal documents, such as contracts, case law, and statutes. With the help of RAG, LLMs can access relevant legal precedents, interpretations, and rulings. They can generate document summaries or analyses to help lawyers and legal researchers understand complex legal issues more efficiently.\"}),/*#__PURE__*/e(\"h3\",{children:\"Prompt engineering\"}),/*#__PURE__*/e(\"p\",{children:\"Prompt engineering involves crafting effective prompts or queries to elicit desired responses, including domain-specific knowledge from a general model. No additional training or architecture modification is required in this approach However, that is the fastest way to guide the model into generating domain-specific outputs..\"}),/*#__PURE__*/e(\"p\",{children:\"To make LLMs generate accurate output, examples of the type of responses expected within the domain can be included in the prompt. These examples can help the model understand the desired style, terminology, and content. For instance, sample sentences or excerpts from documents within the domain may illustrate the type of responses the model should generate.\"}),/*#__PURE__*/e(\"p\",{children:\"These methods can be used together or separately. For example, RAG can complement fine-tuning by updating the model\u2019s knowledge, (such as from regularly updated databases) so the model remains up-to-date with the latest information about the domain. Prompt engineering, on the other hand, can be used alongside RAG to further guide the model's responses by crafting prompts that explicitly specify the desired output and guide the model on how to provide relevant examples.\"}),/*#__PURE__*/e(\"h2\",{children:\"Best practices for training domain-specific language models\"}),/*#__PURE__*/e(\"p\",{children:\"Training domain-specific LLMs involves several best practices to ensure effective performance within the desired domain. Here are some recommended best practices.\"}),/*#__PURE__*/e(\"h3\",{children:\"Involving domain experts\"}),/*#__PURE__*/e(\"p\",{children:\"Involving domain experts throughout the training process allows them to share valuable insights, domain knowledge, and specific recommendations. These are valuable inputs that a person who has not worked in the field (or has no relevant specialized background) would not be able to provide. Collaborating with experts allows the model to capture the domain\u2019s nuances, and deliver results that are more specific and relevant.\"}),/*#__PURE__*/e(\"p\",{children:\"Domain experts have in-depth and contextual knowledge of the subject area, including specialized terminology, industry trends, regulatory requirements, and common areas of concern. Their knowledge is invaluable in guiding training data selection, annotation, and LLM fine-tuning strategies to ensure field relevance and accuracy.\"}),/*#__PURE__*/e(\"p\",{children:\"Experts verify the accuracy and reliability of AI-generated content by cross-referencing authoritative and verifiable sources in the field. This ensures that the content is consistent with established facts, principles, and best practices in the domain.\"}),/*#__PURE__*/t(\"p\",{children:[\"Unlock the expertise of a global network of experts. 15% of our data annotators hold a Doctoral degree in their domain. \",/*#__PURE__*/e(a,{href:\"https://toloka.ai/large-language-models/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"Partner with Toloka\"})}),\" \u2013 we collect essential data to level up your domain-specific model!\"]}),/*#__PURE__*/e(\"h3\",{children:\"Starting with one task before scaling up\"}),/*#__PURE__*/e(\"p\",{children:\"Getting started with a domain-specific LLM does not necessarily require creating a comprehensive model that will fulfill all possible tasks. Start with a specific use case allows ML experts to focus on training an LLM tailored to a specific task or domain. Fulfilling only one task enables an organization to deploya custom model and evaluate its success before considering scaling up.\"}),/*#__PURE__*/e(\"p\",{children:\"Such a smaller scope of work enables relatively quick deployment of a custom model. This allows for real-world testing and evaluation of the model's performance in the intended use case. By gathering feedback and assessing the model's success in addressing the targeted task, data engineers can decide further development and scaling of a large language model.\"}),/*#__PURE__*/e(\"p\",{children:\"In essence, starting small provides the opportunity to iterate, refine, and validate the model before considering broader applications or scaling up to tackle additional tasks or domains. This iterative approach can lead to deployments of domain-specific models being more effective and successful.\"}),/*#__PURE__*/e(\"h3\",{children:\"Use high-quality data for large language models training\"}),/*#__PURE__*/e(\"p\",{children:\"When we talk about training language models for specific tasks, like predicting credit scores for banks, it's important to have high-quality data. If data scientists use data that doesn't represent real world circumstances, the language model might not operate fairly or make good predictions. This can lead to various issues, such as perpetuating existing biases or making decisions that don't align with reality.\"}),/*#__PURE__*/e(\"p\",{children:\"It's crucial for experts to carefully curate and preprocess data to ensure that the language model learns from a diverse and representative sample of real-world examples. Ongoing model performance monitoring and evaluationare essential to identify and address any biases or inaccuracies that may arise.\"}),/*#__PURE__*/e(\"h2\",{children:\"Conclusion\"}),/*#__PURE__*/e(\"p\",{children:\"Building domain-specific LLMs provides organizations with tailored solutions for the nuanced requirements their specialized fields. While pre-trained models provide a strong foundation for processing general language tasks, their generic nature may not produce adequate results for many specific domains.\"}),/*#__PURE__*/e(\"p\",{children:\"Domain-specific LLMs excel in accuracy, relevance, and efficiency, offering several advantages over general-purpose models. These benefits include:\"}),/*#__PURE__*/t(\"ul\",{style:{\"--framer-font-size\":\"18px\",\"--framer-text-alignment\":\"start\",\"--framer-text-color\":\"rgb(30, 33, 38)\",\"--framer-text-stroke-width\":\"0px\",\"--framer-text-transform\":\"none\"},children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Improved efficiency\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Cost savings\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Higher accuracy\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Customizability and adaptability\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Specialized vocabulary understanding.\"})})]}),/*#__PURE__*/e(\"p\",{children:\"With domain-specific LLMs, organizations can obtain faster more accurate responses\u2013tailored to their unique requirements, thereby enhancing productivity and effectiveness throughout their business.\"}),/*#__PURE__*/e(\"p\",{children:\"In essence, the journey towards domain-specific LLMs is a pathway to unlocking the full potential of natural language understanding and generation within specialized domains. It offers tailored solutions that meet the unique needs and challenges of specific industries or subject matter. Organizations around the world continue to recognize the value of leveraging AI-powered language processing in specialized domains. We can anticipate a proliferation of innovative applications and solutions that revolutionize how we communicate, collaborate, and make decisions.\"})]});export const richText1=/*#__PURE__*/t(i.Fragment,{children:[/*#__PURE__*/e(\"p\",{children:\"With the development of machine learning, more and more deep learning models are appearing, with the number of parameters exceeding several hundred billion or even reaching a trillion. Some of the largest models right now include M6-10T, which has 10 trillion parameters, GLaM (Generalist Language Model) by Google, which has 1.2 trillion parameters, and GPT-4, which, according to some sources, has 1.76 trillion parameters.\"}),/*#__PURE__*/e(\"p\",{children:\"Training and deploying such models requires considerable computational power, often necessitating specialized hardware and electricity. In addition, such models demand large amounts of memory to store and process their parameters, which makes them impractical for deployment in resource-limited environments.\"}),/*#__PURE__*/e(\"p\",{children:\"Given the challenges, knowledge distillation of large language models presents a promising solution that makes the benefits of large-scale models more accessible and practical. By enabling smaller and sometimes even more efficient models that retain much of the performance of their larger counterparts, knowledge distillation helps bridge the gap between groundbreaking AI research and real-world applications. In this article, we discuss knowledge distillation training schemes and their use.\"}),/*#__PURE__*/e(\"h2\",{children:\"What is knowledge distillation in deep learning?\"}),/*#__PURE__*/t(\"p\",{children:[\"Knowledge distillation, first introduced in a paper \",/*#__PURE__*/e(a,{href:\"https://arxiv.org/abs/1503.02531?ref=blog.roboflow.com\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"Distilling the Knowledge in a Neural Network\"})}),\", by Geoffrey Hinton and his colleagues, is a technique for training primarily neural network-based machine learning models to transfer knowledge from a teacher model to a student model.\"]}),/*#__PURE__*/e(\"p\",{children:\"A teacher network is a powerful deep-learning model or ensemble of models skilled at capturing complex features and making accurate predictions. Such teacher models are used to train simpler student networks, which can be trained to produce results comparable to those of the teacher network.\"}),/*#__PURE__*/e(\"p\",{children:\"Aside from making accurate predictions, during the learning process, a complex neural network is taught to generate meaningful and helpful representations of data, which can be called knowledge. This notion is at the core of knowledge distillation because the predictions reproduced by the student network through a distillation process are based on these thorough representations of data, or knowledge, stored by the teacher network in its hidden layers.\"}),/*#__PURE__*/e(\"p\",{children:\"Knowledge distillation is necessary to carry out model compression since the initial pre-trained teacher model contains many parameters that require considerable resources to deploy, train, and utilize. Model compression through knowledge distillation helps decrease the memory and computational requirements of a large model without substantially reducing its overall productivity.\"}),/*#__PURE__*/e(\"p\",{children:'In large language models (LLMs), knowledge distillation has facilitated the transfer of more abstract characteristics, such as the model\\'s style, reasoning capabilities, and alignment with human preferences and values. Knowledge distillation techniques go beyond simply copying the outputs of teacher models; they strive to mimic the underlying \"thought processes\" of these models.'}),/*#__PURE__*/e(\"p\",{children:\"Knowledge distillation represents one of the most effective ways to reduce the size of a model as well as its processing speed. As a result of model compression, the intricate and large deep neural network is condensed into a smaller and more simplified one, while preserving the accuracy and performance of the initial model.\"}),/*#__PURE__*/e(\"p\",{children:\"A scaled-down model, trained to replicate the behavior of a heavy and accurate teacher model, achieves similar results to the teacher model, significantly benefiting in terms of size and speed due to its simplified architecture. The student model may even outperform the teacher in some cases, but its accuracy will be slightly poorer.\"}),/*#__PURE__*/e(\"p\",{children:\"Teacher and student models, also known as the teacher-student framework, significantly contribute to the practical deployment of machine learning models by improving efficiency, performance, and generalization.\"}),/*#__PURE__*/e(\"h2\",{children:\"Step-by-step process of knowledge distillation\"}),/*#__PURE__*/e(\"h3\",{children:\"Training teacher model\"}),/*#__PURE__*/e(\"p\",{children:\"At first, the teacher model learns information from a dataset to achieve high accuracy. Such a teacher model is typically represented by a deep neural network that achieves high accuracy by leveraging its enormous capacity to learn complex patterns in the data. Due to its sophistication, it can capture intricate nuances in the data, resulting in accurate predictions.\"}),/*#__PURE__*/e(\"h3\",{children:\"Generating soft labels\"}),/*#__PURE__*/e(\"p\",{children:\"Before predicting a hard target (also known as a label), also referred to as a final output, deep neural networks usually tend to generate intermediate predictions or soft targets (labels). They serve as a principal data source for the student model training process in knowledge distillation.\"}),/*#__PURE__*/e(\"p\",{children:\"So, following the training phase, the teacher model is used to produce soft targets for the training data. Soft targets, unlike hard ones, provide rich information, including the relative probabilities of incorrect classes, which help the student model learn more about the underlying data structure. Hard targets indicate only the correct or highest probability class, whereas soft targets provide a distribution of probabilities across all classes.\"}),/*#__PURE__*/e(\"p\",{children:\"Thanks to the level of detail of the data provided in the soft targets, the student model requires a smaller amount of data than the original teacher model. A higher learning rate is also applied during student model training than the original one.\"}),/*#__PURE__*/e(\"h3\",{children:\"Training student model\"}),/*#__PURE__*/e(\"p\",{children:\"The student model is smaller and sometimes even more efficient than the teacher model. It has fewer parameters, which makes it suitable for implementation in low-resource contexts. The loss function is what the student model tries to minimize during training. It measures how well the student model is doing.\"}),/*#__PURE__*/e(\"p\",{children:\"In knowledge distillation, two loss functions are applied to bring the student model's problem-solving process closer to that of a teacher model. One is a standard loss function, and the other is called distillation loss.\"}),/*#__PURE__*/e(\"p\",{children:\"The standard one in the context of knowledge distillation is called hard loss, which compares the final result of the model with ground truth in supervised learning and with the original data in self-supervised learning. Distillation loss or soft loss compares the student model's soft labels with the teacher model's soft labels using Kullback-Leibler divergence.\"}),/*#__PURE__*/e(\"p\",{children:\"Training the smaller model involves using the teacher\u2019s detailed predictions (soft labels) and the initial dataset to guide the student. The smaller student model learns to make as little difference as possible between its own predictions and the soft labels formed by the teacher network. The student model goes through many iterations (epochs) of training, each time adjusting its parameters to better match the teacher\u2019s predictions and the actual labels.\"}),/*#__PURE__*/e(\"h2\",{children:\"Types of knowledge distillation\"}),/*#__PURE__*/e(\"h3\",{children:\"Response-based knowledge distillation\"}),/*#__PURE__*/e(\"p\",{children:\"Response-based knowledge distillation is a technique that focuses on transferring knowledge from a teacher model to a student model by having the student model mimic the final output layer of the teacher model. This method leverages the rich information contained in the teacher's predictions to train a more efficient student model.\"}),/*#__PURE__*/e(\"p\",{children:\"The student model learns to imitate the teacher model's predictions by minimizing a distillation loss. This ensures that the student captures the nuanced information present in the teacher's outputs. The goal of training is to minimize the loss, which means that during several epochs of training, the small student model will get better at generating the same results as the teacher.\"}),/*#__PURE__*/e(\"h3\",{children:\"Feature-based knowledge distillation\"}),/*#__PURE__*/e(\"p\",{children:\"Feature-based knowledge distillation leverages the internal representations or features learned by a teacher model in its intermediate layers to train a student model. This approach goes beyond focusing solely on the output layer and instead utilizes the rich, hierarchical features captured in the intermediate layers of the teacher model. Intermediate layers of the teacher model extract features crucial for the model's final prediction. These features contain valuable information about the data's structure and patterns.\"}),/*#__PURE__*/e(\"h3\",{children:\"Relation-based knowledge distillation\"}),/*#__PURE__*/e(\"p\",{children:\"Relation-based knowledge distillation is a technique that captures and transfers the relationship knowledge between data samples and layers within the neural network. This method complements response-based and feature-based knowledge distillation by focusing on the interactions and relationships between different features, rather than the features themselves or the final outputs.\"}),/*#__PURE__*/e(\"h2\",{children:\"Knowledge distillation techniques\"}),/*#__PURE__*/t(\"p\",{children:[\"The knowledge distillation approach implies the possibility of training the student on unlabeled data, but to maximize quality it is necessary to combine the teacher's predictions with the labeled data. According to \",/*#__PURE__*/e(a,{href:\"https://arxiv.org/abs/2006.05525\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"Knowledge Distillation: A Survey to* *Qualitatively Distill Knowledge\"})}),\", the number of parameters in the student model architecture should not be decreased by more than double the number of teacher parameters.\"]}),/*#__PURE__*/e(\"p\",{children:\"Knowledge distillation involves transferring knowledge from a larger, well-trained model (teacher) to a smaller, more efficient model (student). There is a model capacity gap between such models that has to be minimized for the student model to generate the same accurate results as the teacher model.\"}),/*#__PURE__*/e(\"p\",{children:\"Several schemes and algorithms are designed to facilitate this process, each with specific methods for improving the student model's performance by leveraging the teacher model's knowledge. There are three primary modes available for training student and teacher models:\"}),/*#__PURE__*/t(\"ul\",{style:{\"--framer-font-size\":\"18px\",\"--framer-text-alignment\":\"start\",\"--framer-text-color\":\"rgb(30, 33, 38)\",\"--framer-text-stroke-width\":\"0px\",\"--framer-text-transform\":\"none\"},children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Offline distillation;\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Online distillation;\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Self-distillation.\"})})]}),/*#__PURE__*/e(\"h3\",{children:\"Offline knowledge distillation\"}),/*#__PURE__*/e(\"p\",{children:\"Offline knowledge distillation refers to the traditional approach where the teacher model is trained first. Then the student model is trained separately using the soft labels generated by the teacher. Knowledge is distilled from a large pre-trained teacher model into a student, while the teacher itself remains unchanged with its weights frozen.\"}),/*#__PURE__*/e(\"h3\",{children:\"Online knowledge distillation\"}),/*#__PURE__*/e(\"p\",{children:\"There may be circumstances when a large pre-trained teacher is not available for a given task, or the teacher model is so huge that there is insufficient storage or processing capacity on the device or in the cloud. Due to this, training is either completely impossible or takes more time than training the desired student model.\"}),/*#__PURE__*/e(\"p\",{children:\"To resolve this issue online knowledge distillation may be used. It refers to a method where the teacher and student models are trained simultaneously, with the student learning from the teacher dynamically during training. Online distillation requires a teacher model of the same architecture as the student.\"}),/*#__PURE__*/e(\"h3\",{children:\"Self-distillation\"}),/*#__PURE__*/e(\"p\",{children:\"Previous knowledge distillation methods included two separate models, whereas self-distillation is a variant of knowledge distillation in which a single model acts as both the teacher and the student.\"}),/*#__PURE__*/e(\"p\",{children:\"In self-distillation, knowledge is transferred from deeper layers of the network to shallow layers of the same network. Shallow classifiers are placed on different layers of the model and removed from that network after the model is trained and functional. Classifiers at deeper layers act as teacher models and train the shallow layers, which, in such cases, represent student models.\"}),/*#__PURE__*/e(\"p\",{children:\"Self-distillation comprises two distinct groups of methods. Since the teacher and student are trained simultaneously, the first group of methods can be referred to as an extension of online knowledge distillation. In this type of self-distillation, the information that is accumulated in the model during training is used to further improve the quality of predictions of the same model. In other words, the model learns from its own predictions generated during earlier stages of training.\"}),/*#__PURE__*/e(\"p\",{children:\"In the second group of methods, experts select a particular neural network architecture and train one model. Then, using knowledge distillation from the previously trained model, they train the same model. This group of self-distillation methods can also be called offline distillation when a new model of the same architecture is obtained from the previously trained model.\"}),/*#__PURE__*/e(\"p\",{children:\"Since the models are identical in their architecture, the teacher cannot provide the student model with new skills, and therefore the final quality of the model will not improve. Hence, this method of self-distillation is applied to explore the knowledge dissociation proposed by Hinton.\"}),/*#__PURE__*/e(\"h2\",{children:\"Knowledge distillation applications\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"Knowledge distillation has no boundaries when it comes to its application areas. It can be utilized in natural language processing (NLP) for LLM compression. For example, DistilBERT is a lightweight compressed counterpart of a larger BERT language model. It reduced the size of the BERT model by 40% while retaining 97% of its performance and being 60% faster.\"}),/*#__PURE__*/e(\"p\",{children:\"DistilBERT was created to address several key challenges associated with large pre-trained language models like BERT. It particularly focuses on the need for efficiency in terms of computational resources and memory usage. The model architecture of DistilBERT is optimized to retain the most important aspects of BERT\u2019s performance while reducing redundant parameters and layers.\"}),/*#__PURE__*/t(\"p\",{children:[\"Apart from NLP, knowledge distillation is employed in the computer vision domain. For example, the paper \",/*#__PURE__*/e(a,{href:\"https://ieeexplore.ieee.org/document/8803717\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"Cogni-Net: Cognitive Feature Learning Through Deep Visual Perception\"})}),\" describes the CogniNet model that employs knowledge distillation in the field of brain signal classification. In this approach, a BiLSTM (Bidirectional Long-Short Term Memory) model, which is adept at handling sequential data, is trained to classify brain EEG (electroencephalogram) signals. This training is enhanced through knowledge distillation, where a pre-trained deep convolutional vision network acts as the teacher model.\"]}),/*#__PURE__*/e(\"p\",{children:\"The distillation process helps the BiLSTM model achieve better accuracy in classifying brain signals. Research demonstrated that leveraging pre-trained vision models to distill knowledge into a recurrent model can effectively bridge the gap between visual perception and brain signal interpretation. The findings highlight the potential of knowledge distillation in enhancing inter-modal learning and understanding complex cognitive processes.\"}),/*#__PURE__*/e(\"h2\",{children:\"Knowledge distillation algorithms\"}),/*#__PURE__*/e(\"p\",{children:\"Several algorithms have been proposed to facilitate the process of knowledge distillation, each with its unique approach and use cases. The following are some prominent algorithms for effective distillation.\"}),/*#__PURE__*/e(\"h3\",{children:\"Attention-based knowledge distillation\"}),/*#__PURE__*/e(\"p\",{children:\"Knowledge distillation involves transferring knowledge from a large, complex teacher model to a smaller, simpler student model. Attention mechanisms can enhance this process by helping the student model learn not just from the teacher model's final outputs but also from the intermediate representations where attention is applied.\"}),/*#__PURE__*/e(\"h3\",{children:\"Adversarial learning\"}),/*#__PURE__*/e(\"p\",{children:\"Adversarial learning enables teacher and student models to capture the underlying data distribution better. Generative Adversarial Networks (GANs) can be used to augment the training dataset by generating additional realistic data samples. This augmented data can help the student model learn better by providing more diverse training examples, thus improving generalization and performance.\"}),/*#__PURE__*/e(\"p\",{children:\"Here\u2019s how adversarial learning and GANs come into play in knowledge distillation. The student model that acts as a generator in GAN tries to create answers (outputs) that look just like the teacher\u2019s answers. A discriminator checks how similar the student\u2019s answers are to the teacher\u2019s. The student gets better by fooling the discriminator into thinking its answers are from the teacher. By using adversarial learning, the student gets better at copying the teacher, making its predictions more accurate over time.\"}),/*#__PURE__*/e(\"h3\",{children:\"Data-free knowledge distillation\"}),/*#__PURE__*/e(\"p\",{children:\"Knowledge Distillation typically involves training a student model using the same data used to train a teacher model. However, this data might not always be available. Being a complex network, the teacher model often requires a vast amount of data for training, which might be impractical to share, or the data used to train the teacher model (such as patient records) cannot be shared due to privacy concerns.\"}),/*#__PURE__*/e(\"p\",{children:\"To address these issues, data-free knowledge distillation techniques have been developed. These methods generate synthetic data that mimics the data distribution of the original training samples used for the teacher model. One such method is the DeepInversion for Object Detection (DIODE) framework, which diverges from traditional GAN-based image synthesis methods. Unlike GANs, which use a generator and discriminator to create realistic images, DeepInversion directly optimizes the images to match the internal feature statistics of the teacher model.\"}),/*#__PURE__*/e(\"h2\",{children:\"Benefits of knowledge distillation\"}),/*#__PURE__*/e(\"p\",{children:\"Knowledge distillation offers several key benefits in the context of machine learning and, more specifically, for complex deep learning models.\"}),/*#__PURE__*/e(\"h3\",{children:\"Reduced model size\"}),/*#__PURE__*/e(\"p\",{children:\"Model compression is perhaps one of the fundamental challenges of knowledge distillation, which made researchers start looking into the necessity of developing such a technology. Knowledge distillation indeed significantly reduces the size of the model.\"}),/*#__PURE__*/e(\"p\",{children:\"Doing so makes it more feasible to deploy student models capable of delivering the same results as original teacher models on devices with limited storage and computational power.\"}),/*#__PURE__*/e(\"p\",{children:\"Small models are also capable of faster inference resulting from knowledge distillation. This is because they can process data more quickly, leading to quicker response times in applications and real-time systems.\"}),/*#__PURE__*/e(\"h3\",{children:\"Resource effficiency\"}),/*#__PURE__*/e(\"p\",{children:\"Training a student model using knowledge distillation requires less data, so it is less resource-intensive than training a large model from scratch. Smaller models also consume less power and memory, which is crucial for applications with limited memory availability. This all leads to reduced computational and energy consumption, i.e., overall lower training costs.\"}),/*#__PURE__*/e(\"p\",{children:\"Knowledge distillation allows the creation of multiple models for various types of tasks without the need for extensive computational resources. Smaller student models can also be scaled more easily across multiple devices or cloud instances.\"}),/*#__PURE__*/e(\"h3\",{children:\"Improved performance\"}),/*#__PURE__*/e(\"p\",{children:\"Distilled models retain much of the accuracy and performance of the larger models, even with significant reductions in size. The student model can capture essential knowledge from the teacher model, often leading to better generalization on unseen data than a previously trained model.\"}),/*#__PURE__*/e(\"h2\",{children:\"Knowledge distillation is an effective way to make sophisticated\"}),/*#__PURE__*/e(\"p\",{children:\"models more accessible\"}),/*#__PURE__*/e(\"p\",{children:\"Knowledge distillation is a powerful and versatile technique for transferring knowledge from complex, high-performing teacher models to smaller, more efficient student models. Leveraging various forms of knowledge\u2014response-based, feature-based, and relation-based\u2014knowledge distillation allows the student model to capture not only the final predictions but also the intricate feature representations and relationships within the data.\"}),/*#__PURE__*/e(\"p\",{children:\"Knowledge distillation solves one of the most pressing challenges: deploying sophisticated models in resource-constrained environments. By transferring knowledge from large, complex models to smaller, more efficient ones, knowledge distillation retains high performance while significantly reducing computational requirements.\"}),/*#__PURE__*/e(\"p\",{children:\"In other words, knowledge distillation is an indispensable tool for employing complex models in resource-constrained environments such as mobile devices, IoT systems, real-time applications, and even on-edge devices with little computing power.\"}),/*#__PURE__*/e(\"p\",{children:\"It allows for democratizing advanced AI, making sophisticated technology accessible across diverse platforms and use cases. As the demand for AI continues to grow, the importance of techniques like knowledge distillation will only increase, enabling new applications that were previously constrained by computational limitations.\"})]});export const richText2=/*#__PURE__*/t(i.Fragment,{children:[/*#__PURE__*/e(\"p\",{children:\"This article discusses how the digital changes brought about by Large Language Models (LLMs) are not experienced by everyone, especially speakers of less common languages, and why this is the case. It proposes solutions on how to narrow the gap between high and low-resource languages and emphasizes the pressing need for more action towards inclusivity, thus creating a more responsible and inclusive AI ecosystem.  \"}),/*#__PURE__*/e(a,{href:\"https://toloka.ai/large-language-models/?utm_source=blog&utm_medium=banner&utm_campaign=sft&utm_content=lowresourcelanguages\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!0,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{className:\"framer-image\",\"data-preset-tag\":\"img\",children:/*#__PURE__*/e(\"img\",{alt:\"\",className:\"framer-image\",height:\"240\",src:\"https://framerusercontent.com/images/xpUvAvgGNUA6gpXxlqHiClcP1U.png\",srcSet:\"https://framerusercontent.com/images/xpUvAvgGNUA6gpXxlqHiClcP1U.png?scale-down-to=512 512w,https://framerusercontent.com/images/xpUvAvgGNUA6gpXxlqHiClcP1U.png?scale-down-to=1024 1024w,https://framerusercontent.com/images/xpUvAvgGNUA6gpXxlqHiClcP1U.png 1498w\",style:{aspectRatio:\"1498 / 480\"},width:\"749\"})})}),/*#__PURE__*/e(\"h2\",{children:\"Low Resource Languages (LRLs) in Large Language Models (LLMs)\"}),/*#__PURE__*/e(\"p\",{children:\"In recent years, the emergence of Large Language Models (LLMs) has brought about significant shifts in the daily routines of consumers. Individuals can now undertake a diverse range of tasks, such as retrieving information, composing text, and refining documents through these powerful language tools. This integration of LLMs into daily life has resulted in notable boosts in productivity, both at work and in personal endeavors.\"}),/*#__PURE__*/e(\"p\",{children:\"However, it\u2019s important to recognize that not all consumers have experienced these benefits equally. Indeed, a considerable number of people around the world who speak less common languages aren\u2019t able to interact with LLMs, primarily due to the inadequacy of language models designed for these specific languages. With 7,000 languages currently spoken in the world, the largest multilingual LLMs have been trained using only fewer than a hundred languages, thus leaving many languages and people completely behind.\"}),/*#__PURE__*/t(\"p\",{children:[\"Supporting non-English languages requires high-quality, abundant data sources, which can be difficult to find and access. And not only do those models perform worse but it also has been reported by \",/*#__PURE__*/e(a,{href:\"https://arxiv.org/pdf/2310.02446.pdf\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"Brown University\"})}),\" that they are more likely to give unethical responses thus making them more vulnerable to malicious attacks.\"]}),/*#__PURE__*/e(\"h2\",{children:\"Why do we have underrepresented languages in LLMs?\"}),/*#__PURE__*/e(\"p\",{children:\"The performance of LLMs tailored for Low Resource Languages (LRL) is hindered by several key challenges. Firstly, the foundation models for many LLMs rely on data scraped from the internet, which often lacks comprehensive coverage of LRLs. The below graph shows a distribution of data across the internet divided into language groups. While more common languages have hundreds of GB of data potentially available for training models, the languages in the tail of the graph only have data available in the range of hundreds of megabytes.\"}),/*#__PURE__*/e(\"img\",{alt:\"The long tail of multilinguality, few high-resource languages, and many sparsely populated languages. - Image originally published in \",className:\"framer-image\",height:\"540\",src:\"https://framerusercontent.com/images/JmTH420A7CkAh0XdlIAVM8OVk6c.jpeg\",srcSet:\"https://framerusercontent.com/images/JmTH420A7CkAh0XdlIAVM8OVk6c.jpeg?scale-down-to=512 512w,https://framerusercontent.com/images/JmTH420A7CkAh0XdlIAVM8OVk6c.jpeg?scale-down-to=1024 1024w,https://framerusercontent.com/images/JmTH420A7CkAh0XdlIAVM8OVk6c.jpeg?scale-down-to=2048 2048w,https://framerusercontent.com/images/JmTH420A7CkAh0XdlIAVM8OVk6c.jpeg 3600w\",style:{aspectRatio:\"3600 / 1080\"},width:\"1800\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"em\",{children:\"The long tail of multilinguality, few high-resource languages, and many sparsely populated languages. - Image originally published in \"}),/*#__PURE__*/e(a,{href:\"https://arxiv.org/pdf/1911.02116.pdf\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!0,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:/*#__PURE__*/e(\"em\",{children:\"https://arxiv.org/pdf/1911.02116.pdf\"})})})]}),/*#__PURE__*/e(\"p\",{children:\"This limitation is further magnified by the absence of fine-tuned instruction datasets for many LRLs. An instruction dataset consists of a question set paired with ideal answers and is a crucial part of LLM training - in this case, in specific languages. This is how the model learns to follow instructions, and without this asset, models are only capable of predicting the next word in the sequence instead of assisting humans with complex questions and problem-solving tasks.\"}),/*#__PURE__*/e(\"p\",{children:\"The above is caused by the fact that LLMs are trained in sequential steps. The first step is to learn the language by reading a large amount of unannotated text which gives the model the ability to predict the next world in the sequence. The second step is tailoring this predictive behavior to follow specific instructions, such as answering questions, writing summaries, or extracting data. This is why fine-tuning datasets is of such importance, as their quality will further determine the ability of LLM to assist users with required tasks.\"}),/*#__PURE__*/e(\"p\",{children:\"In the following section, we will present a method to create a high-quality dataset for Swahili that can be used to fine-tune the LLM for this language. The method can be applied to any low-resource language.\"}),/*#__PURE__*/e(\"h2\",{children:\"Innovative pipeline to gather data for LRLs\"}),/*#__PURE__*/e(\"p\",{children:\"Swahili is a language spoken by over 200 million people across 14 different African countries and is the official national language in Tanzania, Kenya, Uganda, and the Democratic Republic of the Congo. It belongs to the group of low-resource languages and is an example of a language that does not have an out-of-the-box instruction dataset for LLM fine-tuning.\"}),/*#__PURE__*/e(\"img\",{alt:\"Innovative pipeline to gather data for LRLs\",className:\"framer-image\",height:\"994\",src:\"https://framerusercontent.com/images/yobJhUhEt7Ch6NXvn8Z8kYY82Y.jpeg\",srcSet:\"https://framerusercontent.com/images/yobJhUhEt7Ch6NXvn8Z8kYY82Y.jpeg?scale-down-to=512 512w,https://framerusercontent.com/images/yobJhUhEt7Ch6NXvn8Z8kYY82Y.jpeg?scale-down-to=1024 1024w,https://framerusercontent.com/images/yobJhUhEt7Ch6NXvn8Z8kYY82Y.jpeg?scale-down-to=2048 2048w,https://framerusercontent.com/images/yobJhUhEt7Ch6NXvn8Z8kYY82Y.jpeg 3600w\",style:{aspectRatio:\"3600 / 1989\"},width:\"1800\"}),/*#__PURE__*/e(\"p\",{children:\"In general, three approaches exist to create a fine-tuning dataset for a language. The first one is the direct generation of a dataset by assessors, in this case, language experts, which requires developing both questions and ideal answers in the desired language. This can be challenging for the Swahili language because assessors need to be high-level experts and the process is generally expensive.\"}),/*#__PURE__*/e(\"p\",{children:\"Another potential solution is taking an existing instruction dataset in English and translating it into Swahili. This could be done by translators who speak both Swahili and English but this can also be time and resource intensive. An automatic translator could be used, however, this typically results in insufficient or poor-quality results.\"}),/*#__PURE__*/e(\"p\",{children:\"Another solution combines automated translation with human validation, offering a cost-efficient and scalable approach, which is critical to ensure LRL models are accurate, reflect local customs and norms, and are useful to the communities that will be using them. This method utilizes the best available automatic translator from Swahili to English and then asks native Swahili speakers to filter out examples that do not meet quality standards.\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(a,{href:\"https://toloka.ai/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"Toloka\"})}),\" recently conducted a project to develop, where they were able to create an 11,000 fine-tuning dataset for Swahili from the 15,000 original \",/*#__PURE__*/e(a,{href:\"https://huggingface.co/datasets/databricks/databricks-dolly-15k\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"Dolly dataset\"})}),\". Each data point consisting of a prompt and an answer was translated from English to Swahili using automatic translation resulting initially in 15,000 question answers pairs in Swahili. This dataset was further reduced by asking native speakers to remove pairs with low quality thus leaving a fine-tuned Swahili dataset with 11,000 instances. The dataset was then used to improve \",/*#__PURE__*/e(a,{href:\"https://arxiv.org/pdf/2010.11934v3.pdf\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"mT5\"})}),\", one of the top-performing multilingual language models for Swahili, which demonstrated significant performance enhancements for this language. The fine-tuned dataset boosted accuracy and f-score (a measure of predictive performance) for classification tasks, but more importantly, it significantly increased \",/*#__PURE__*/e(a,{href:\"https://huggingface.co/spaces/evaluate-metric/rouge\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"ROUGE\"})}),/*#__PURE__*/e(a,{href:\"https://en.wikipedia.org/wiki/ROUGE_(metric)\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\", or\"})}),\" Recall-Oriented Understudy for Gisting Evaluation, which is a set of metrics used for evaluating automatic summarization and machine translation software in NLP, and \",/*#__PURE__*/e(a,{href:\"https://www.researchgate.net/publication/319123639_chrF_words_helping_character_n-grams\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"chrF++, Character n-gram F-score (chrF),\"})}),\" in generative tasks where the model must respond to open questions. This experiment shows the potential for improving LLM performance in LRLs and therefore opens up a path for building truly multilingual models.\"]}),/*#__PURE__*/e(\"h2\",{children:\"Creating a More Inclusive AI Ecosystem\"}),/*#__PURE__*/t(\"p\",{children:[\"As developers and organizations strive to create a more inclusive AI ecosystem, evaluation becomes even more critical, as does human involvement in training LLMs. Cohere\u2019s recent launch of \",/*#__PURE__*/e(a,{href:\"https://txt.cohere.com/aya/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"Aya\"})}),\", a language model that supports over one hundred languages, including Swahili and other LRLs, exemplifies this commitment. Addressing data scarcity and enhancing model performance for LRLs is an important step to building more inclusive and responsible AI systems that serve diverse linguistic communities worldwide.\"]}),/*#__PURE__*/t(\"p\",{children:[\"Need help with multilingual datasets for SFTs or evaluating models? \",/*#__PURE__*/e(a,{href:\"https://toloka.ai/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"Toloka\"})}),\" can be a great partner for this. Our teams of domain experts ensure high-quality data in multiple languages. \",/*#__PURE__*/e(a,{href:\"https://toloka.ai/talk-to-us/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"Reach out\"})}),\" to us today to get more information.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"em\",{children:\"An original version of this article was published on \"}),/*#__PURE__*/e(a,{href:\"https://hackernoon.com/why-integrating-low-resource-languages-into-llms-is-essential-for-responsible-ai\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:/*#__PURE__*/e(\"em\",{children:\"HackerNoon\"})})})]})]});export const richText3=/*#__PURE__*/t(i.Fragment,{children:[/*#__PURE__*/e(\"p\",{children:\"Large Language Models (LLMs) have revolutionized how we interact with AI, becoming the driving force behind numerous technologies and tools used by millions worldwide. With the widespread use of generative AI, LLM producers must prioritize ethical considerations concerning the text generated by the models. Part of this responsibility is to make sure that the model\u2019s responses are helpful, truthful, and harmless. \"}),/*#__PURE__*/t(\"p\",{children:[\"Making a model adhere to these three principles is referred to as \",/*#__PURE__*/e(\"em\",{children:\"alignment\"}),\", a process that trains and guides the model with good examples and feedback. In practice, this means giving the model thousands of high-quality prompts and completions for Supervised Fine-Tuning (SFT) and scoring an even larger number of model completions for Reinforcement Learning from Human Feedback (RLHF).\"]}),/*#__PURE__*/e(\"p\",{children:\"In this article, we\u2019ll walk you through model alignment based on our own experience as a data partner for several LLM producers.\"}),/*#__PURE__*/e(a,{href:\"https://toloka.ai/large-language-models/?utm_source=blog&utm_medium=banner&utm_campaign=sftrlhf&utm_content=developingllmsthatarehelpful\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!0,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{className:\"framer-image\",\"data-preset-tag\":\"img\",children:/*#__PURE__*/e(\"img\",{alt:\"\",className:\"framer-image\",height:\"240\",src:\"https://framerusercontent.com/images/L1MWCdKVPqiR785bI54iY2TIc9k.png\",srcSet:\"https://framerusercontent.com/images/L1MWCdKVPqiR785bI54iY2TIc9k.png?scale-down-to=512 512w,https://framerusercontent.com/images/L1MWCdKVPqiR785bI54iY2TIc9k.png?scale-down-to=1024 1024w,https://framerusercontent.com/images/L1MWCdKVPqiR785bI54iY2TIc9k.png 1498w\",style:{aspectRatio:\"1498 / 480\"},width:\"749\"})})}),/*#__PURE__*/e(\"h2\",{children:\"LLM alignment\"}),/*#__PURE__*/t(\"p\",{children:[\"Current efforts in aligning LLMs focus on making them helpful, truthful, and harmless, also referred to as \",/*#__PURE__*/e(a,{href:\"https://www.arxiv-vanity.com/papers/2112.00861/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"HHH criteria\"})}),\" (helpful, honest, harmless).\"]}),/*#__PURE__*/t(\"ul\",{style:{\"--framer-font-size\":\"18px\",\"--framer-text-alignment\":\"start\",\"--framer-text-color\":\"rgb(30, 33, 38)\",\"--framer-text-stroke-width\":\"0px\",\"--framer-text-transform\":\"none\"},children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Helpful LLMs generate answers that are accurate, easy to understand, and meet the user\u2019s needs.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Truthfulness ensures answers are accurately sourced and do not include any made-up facts. The model should also explain clearly when it can\u2019t offer a definitive answer.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Harmless LLMs do not offend, reveal sensitive information, or provide content that can lead to dangerous behaviors. The model should not demonstrate bias or discrimination.\"})})]}),/*#__PURE__*/e(\"p\",{children:\"The goal is to optimize all three aspects, but there are instances when they contradict each other and we must prioritize one of them.\"}),/*#__PURE__*/e(\"p\",{children:\"Here is an example of prompt completion that is truthful and harmless, but not helpful:\"}),/*#__PURE__*/e(\"img\",{alt:\"LLM alignment\",className:\"framer-image\",height:\"206\",src:\"https://framerusercontent.com/images/RJZO2913ZJqOIYVOQ08RVsTCnU4.png\",srcSet:\"https://framerusercontent.com/images/RJZO2913ZJqOIYVOQ08RVsTCnU4.png?scale-down-to=512 512w,https://framerusercontent.com/images/RJZO2913ZJqOIYVOQ08RVsTCnU4.png?scale-down-to=1024 1024w,https://framerusercontent.com/images/RJZO2913ZJqOIYVOQ08RVsTCnU4.png?scale-down-to=2048 2048w,https://framerusercontent.com/images/RJZO2913ZJqOIYVOQ08RVsTCnU4.png 2594w\",style:{aspectRatio:\"2594 / 412\"},width:\"1297\"}),/*#__PURE__*/e(\"p\",{children:\"In this case, the model was optimized for harmlessness, so it prioritizes safety over helpfulness. \"}),/*#__PURE__*/e(\"p\",{children:\"But why do we need to align LLMs in the first place?\"}),/*#__PURE__*/e(\"h2\",{children:\"How LLMs work\"}),/*#__PURE__*/e(\"p\",{children:\"Base LLMs are trained on a vast amount of text extracted from the internet and taught to predict the next word given an initial sequence of words. Content produced this way is prone to bias, has a higher probability of being harmful, and may contain misinformation. Additionally, the base models cannot follow the instructions that are required for common NLP tasks. They can predict the next word in a sequence, but they can\u2019t do other NLP tasks such as classification, translation, or summarization \u2014 at least, not yet. This is why we need fine-tuning.\"}),/*#__PURE__*/e(\"p\",{children:\"There are two ways to fine-tune the base models:\"}),/*#__PURE__*/t(\"ul\",{style:{\"--framer-font-size\":\"18px\",\"--framer-text-alignment\":\"start\",\"--framer-text-color\":\"rgb(30, 33, 38)\",\"--framer-text-stroke-width\":\"0px\",\"--framer-text-transform\":\"none\"},children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Supervised Fine-tuning (SFT) teaches the model to follow instructions by providing examples of prompts and completions.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/e(\"p\",{children:\"Reinforcement Learning from Human Feedback (RLHF) uses human experts to rate the model\u2019s responses and reward the model for generating answers that align with human preferences.\"})})]}),/*#__PURE__*/e(\"h2\",{children:\"Datasets for SFT\"}),/*#__PURE__*/t(\"p\",{children:[\"The quest for building aligned LLMs begins with producing quality data for SFT. If you\u2019re like many LLM engineers, you might ignore alignment until the RLHF step \u2014 and it seems to work. For example, OpenAI used this approach in their well-known \",/*#__PURE__*/e(a,{href:\"https://arxiv.org/pdf/2203.02155.pdf\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"InstructGPT paper\"})}),\".\"]}),/*#__PURE__*/t(\"p\",{children:[\"However, recent LIMA (Less Is More for Alignment) \",/*#__PURE__*/e(a,{href:\"https://arxiv.org/abs/2305.11206\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"research\"})}),\" from Meta shows that it is possible to build aligned LLMs with a small number of high-quality prompt completions while omitting RLHF. The paper demonstrates that a LLaMa model fine-tuned with only 1000 samples can achieve performance levels that are comparable to other top-performing LLMs.\"]}),/*#__PURE__*/e(\"p\",{children:\"Does this mean we can forget about RLHF? Probably not, but it shows that focusing on data quality early on has a significant impact on the overall quality when building LLMs.\"}),/*#__PURE__*/e(\"h2\",{children:\"How we guarantee data quality\"}),/*#__PURE__*/t(\"p\",{children:[\"At Toloka we pride ourselves on creating the highest quality data for SFT. As a first step, we develop \",/*#__PURE__*/e(a,{href:\"https://toloka.ai/blog/creative-writing-prompts-for-llms/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"organic datasets of prompts\"})}),\" using crowdsourcing. Qualified annotators follow detailed instructions to write authentic and diverse human prompts.\"]}),/*#__PURE__*/e(\"img\",{alt:\"Prompt generation task performed by annotators\",className:\"framer-image\",height:\"359\",src:\"https://framerusercontent.com/images/cdlOlveMRTeO6sCzlPHfIS9MRY.jpeg\",srcSet:\"https://framerusercontent.com/images/cdlOlveMRTeO6sCzlPHfIS9MRY.jpeg?scale-down-to=512 512w,https://framerusercontent.com/images/cdlOlveMRTeO6sCzlPHfIS9MRY.jpeg?scale-down-to=1024 1024w,https://framerusercontent.com/images/cdlOlveMRTeO6sCzlPHfIS9MRY.jpeg 1236w\",style:{aspectRatio:\"1236 / 718\"},width:\"618\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"em\",{children:\"Prompt generation task performed by annotators\"})}),/*#__PURE__*/e(\"p\",{children:\"The generated prompts are then double-checked by a second line of annotators who ensure they meet the quality standards.\"}),/*#__PURE__*/e(\"img\",{alt:\"Prompt verification task performed by a different annotator\",className:\"framer-image\",height:\"351\",src:\"https://framerusercontent.com/images/drkLzOVuEEWEt9OouBvqKJjiU.jpeg\",srcSet:\"https://framerusercontent.com/images/drkLzOVuEEWEt9OouBvqKJjiU.jpeg?scale-down-to=512 512w,https://framerusercontent.com/images/drkLzOVuEEWEt9OouBvqKJjiU.jpeg?scale-down-to=1024 1024w,https://framerusercontent.com/images/drkLzOVuEEWEt9OouBvqKJjiU.jpeg 1048w\",style:{aspectRatio:\"1048 / 702\"},width:\"524\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"em\",{children:\"Prompt verification task performed by a different annotator\"})}),/*#__PURE__*/t(\"p\",{children:[\"Once we have a set of quality prompts, we deploy domain experts and writers to create prompt completions. These professionals are part of a curated team of \",/*#__PURE__*/e(a,{href:\"https://toloka.ai/blog/meet-the-experts/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"AI Tutors\"})}),\" with advanced writing skills and specific areas of expertise (engineering, coding, ESG, law, medicine, etc). Our AI Tutors complete special training and get continual feedback on their efforts to help them write prompt completions that are helpful, harmless, and truthful.\"]}),/*#__PURE__*/e(\"img\",{alt:\"Prompt completion task given to trained AI Tutors\",className:\"framer-image\",height:\"337\",src:\"https://framerusercontent.com/images/F3AiyNSByXsOQnhKeFVZnObajk.jpeg\",srcSet:\"https://framerusercontent.com/images/F3AiyNSByXsOQnhKeFVZnObajk.jpeg?scale-down-to=512 512w,https://framerusercontent.com/images/F3AiyNSByXsOQnhKeFVZnObajk.jpeg?scale-down-to=1024 1024w,https://framerusercontent.com/images/F3AiyNSByXsOQnhKeFVZnObajk.jpeg 1600w\",style:{aspectRatio:\"1600 / 675\"},width:\"800\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"em\",{children:\"Prompt completion task given to trained AI Tutors\"})}),/*#__PURE__*/e(\"p\",{children:\"All of the prompt completions are then checked by a second line of reviewers who carefully examine each aspect of the\"}),/*#__PURE__*/e(\"img\",{alt:\"Prompt completion verification\",className:\"framer-image\",height:\"265\",src:\"https://framerusercontent.com/images/TNppzCBfsor1rhG2agLeLlPcilo.jpeg\",srcSet:\"https://framerusercontent.com/images/TNppzCBfsor1rhG2agLeLlPcilo.jpeg?scale-down-to=512 512w,https://framerusercontent.com/images/TNppzCBfsor1rhG2agLeLlPcilo.jpeg?scale-down-to=1024 1024w,https://framerusercontent.com/images/TNppzCBfsor1rhG2agLeLlPcilo.jpeg 1600w\",style:{aspectRatio:\"1600 / 530\"},width:\"800\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"em\",{children:\"Prompt completion verification\"})}),/*#__PURE__*/e(\"p\",{children:\"The resulting dataset is aligned to human expectations and ready to feed to the model. This is a scalable approach that allows us to create ethical datasets for SFT.\"}),/*#__PURE__*/e(\"h2\",{children:\"Data for RLHF\"}),/*#__PURE__*/e(\"p\",{children:\"After supervised fine-tuning is complete, the next step is to check the model\u2019s output and see how well it matches human preferences. In RLHF, human annotators rank the LLM\u2019s responses and this data is used to train a Reward Model (RM). But how are such rankings created in practice?\"}),/*#__PURE__*/t(\"p\",{children:[\"The most typical scenario involves taking a large set of prompts (50,000 in the case of \",/*#__PURE__*/e(a,{href:\"https://arxiv.org/abs/2203.02155\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"Instruct GPT\"})}),\") and comparing multiple versions of the model\u2019s response for each prompt.\"]}),/*#__PURE__*/e(\"p\",{children:\"At Toloka, we give AI Tutors a simple interface with the prompt and a pair of responses, and ask them to rate which response is better. We carefully train our annotators to identify the better answer based on explicit criteria. For ambiguous cases, we ask annotators to prioritize the criteria that is most critical (usually harmlessness comes before helpfulness, but it depends on the LLM use case).\"}),/*#__PURE__*/e(\"p\",{children:\"Once all model responses are evaluated, relative rankings are created to train the RM algorithm.\"}),/*#__PURE__*/e(\"p\",{children:\"The side-by-side comparison described above can be further enhanced with more granular details if needed. For example, we can ask annotators to compare the responses separately on helpfulness, harmlessness, and truthfulness. Going even further, we can break down the criteria of harmlessness into subcategories and ask annotators to mark prompts with hate speech, bias, personal info, and so on. We can use this granular data to build custom rankings tailored for specific LLM needs.\"}),/*#__PURE__*/e(\"img\",{alt:\"Example of side by side comparison for RLHF with additional clarifying questions\",className:\"framer-image\",height:\"800\",src:\"https://framerusercontent.com/images/YXttqpnkoUFILuxwXJkcddMYfk.jpeg\",srcSet:\"https://framerusercontent.com/images/YXttqpnkoUFILuxwXJkcddMYfk.jpeg?scale-down-to=1024 860w,https://framerusercontent.com/images/YXttqpnkoUFILuxwXJkcddMYfk.jpeg 1345w\",style:{aspectRatio:\"1345 / 1600\"},width:\"672\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"em\",{children:\"Example of side by side comparison for RLHF with additional clarifying questions\"})}),/*#__PURE__*/e(\"h2\",{children:\"LLM Evaluation\"}),/*#__PURE__*/e(\"p\",{children:\"While many AI teams might stop at RLHF and consider their LLM aligned, model evaluation is a critical step that can\u2019t be overlooked if you are serious about model quality.\"}),/*#__PURE__*/t(\"p\",{children:[\"Basic evaluation can be automated using curated datasets and can serve as the first checkpoint in LLM development, but most models need more than that. You need to know how your LLM responds to real-life queries, and for the highest quality results, we rely on the opinion of \",/*#__PURE__*/e(a,{href:\"https://www.kdnuggets.com/a-better-way-to-evaluate-llms\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"humans\"})}),\".\"]}),/*#__PURE__*/e(\"p\",{children:\"Human evaluation is similar to RLHF ranking. The standard practice involves side-by-side comparison where an annotator is given a prompt and two different completions, and has to decide which one is better. One of the completions is the output of the LLM being evaluated and the other is a reference model that we are comparing it to (e.g. GPT-4).\"}),/*#__PURE__*/e(\"p\",{children:\"We can evaluate all three metrics: truthfulness, helpfulness, harmlessness, or any other aspect that is important for the particular model. After this last step we not only have aligned the model across different metrics but we now have supporting evidence of how it compares with other LLMs on the market.\"}),/*#__PURE__*/e(\"p\",{children:\"In our experience, human evaluation is very effective at pinpointing areas for model improvement. Besides detailed reports and tailored metrics, our clients gain valuable insights and specific recommendations for how to tweak their model output.\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"strong\",{children:\"Human data at scale for responsible AI\"})}),/*#__PURE__*/e(\"p\",{children:\"The quest for developing responsible and aligned Large Language Models rests not just on sophisticated training methodologies \u2014 it is grounded in curated data and continual human oversight.\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(a,{href:\"https://toloka.ai/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"Toloka\"})}),\" helps you take the guesswork out of data collection for SFT, RLHF, and model evaluation. You can trust our experience and expertise to make human-generated data the deciding factor in developing ethical, high-performing LLMs.\"]}),/*#__PURE__*/t(\"p\",{children:[\"We\u2019re always looking for new problems to solve, so if you have an interesting LLM data challenge, we\u2019d love to \",/*#__PURE__*/e(a,{href:\"https://toloka.ai/talk-to-us/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"hear\"})}),\" about it!\"]})]});export const richText4=/*#__PURE__*/t(i.Fragment,{children:[/*#__PURE__*/e(\"p\",{children:\"High-quality training data is a main ingredient in LLM development, as it can vastly improve model performance in practical applications. This data is typically obtained by either traditional data collection methods or their alternatives, such as generative AI models to create synthetic data. Conventional methods include utilizing open-source datasets, closed-source data collections (outsourcing or in-house), web scraping, manual data collection, purchasing data, and so on. These are robust techniques that ensure quality and reliability of the data.\\xa0 \"}),/*#__PURE__*/e(\"p\",{children:\"On the other hand, fine-tuning LLMs can often require domain-specific data that is sometimes easier to supplant with artificially generated data. This is a commonality in domains that have limited data resources. Even though it has its shortcomings and it\u2019s not a unified solution, a routine workaround suggests incorporating synthetic data, in scenarios where traditional data collection can be challenging or costly.\"}),/*#__PURE__*/e(\"p\",{children:\"By mimicking \u201Creal\u201D data, synthetic data lessens the problem of data availability. Generating synthetic data is often more cost-effective and scalable than conventional data collection while ensuring privacy compliance by excluding sensitive information. Hence, it can be a robust addition to the process of fine-tuning LLMs.\"}),/*#__PURE__*/e(\"p\",{children:\"That being said, synthetic data has several drawbacks of its own. For example, if it doesn\u2019t undergo quality control or it\u2019s not a good replica of real-world examples, the trained model may not perform well in practical applications. Human validation of the synthetic data can mitigate this risk and help obtain reliable results. For this reason, we chose a hybrid approach, incorporating both synthetic data and human input.\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"Let\u2019s dive into the process of building a hybrid pipeline for supervised fine-tuning (SFT). You can think of synthetic data generation as a culinary adventure in the digital kitchen of AI development. Just as a chef carefully selects ingredients and applies precise techniques to create a delicious dish, let\u2019s roll up our sleeves to \u201Ccook\u201D a synthetic data pipeline, perfect for SFT. With the right ingredients, techniques, and a pinch of inspiration, you can craft a dataset that satisfies your model\u2019s appetite. Let\u2019s get started!\"}),/*#__PURE__*/e(a,{href:\"https://toloka.ai/large-language-models/?utm_source=blog&utm_medium=banner&utm_campaign=sft&utm_content=craftingsyntheticdata\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!0,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{className:\"framer-image\",\"data-preset-tag\":\"img\",children:/*#__PURE__*/e(\"img\",{alt:\"\",className:\"framer-image\",height:\"240\",src:\"https://framerusercontent.com/images/xpUvAvgGNUA6gpXxlqHiClcP1U.png\",srcSet:\"https://framerusercontent.com/images/xpUvAvgGNUA6gpXxlqHiClcP1U.png?scale-down-to=512 512w,https://framerusercontent.com/images/xpUvAvgGNUA6gpXxlqHiClcP1U.png?scale-down-to=1024 1024w,https://framerusercontent.com/images/xpUvAvgGNUA6gpXxlqHiClcP1U.png 1498w\",style:{aspectRatio:\"1498 / 480\"},width:\"749\"})})}),/*#__PURE__*/e(\"h2\",{children:\"Ingredients\"}),/*#__PURE__*/t(\"ol\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"12px\",\"--framer-text-color\":\"rgb(13, 13, 13)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/e(\"p\",{children:\"A hearty batch of open-source data (raw texts relevant to your domain, sourced from Hugging Face, Kaggle, or the web)\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"12px\",\"--framer-text-color\":\"rgb(13, 13, 13)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/e(\"p\",{children:\"A collection of text-slicing tools (for parsing long texts and chopping them into manageable paragraphs)\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"12px\",\"--framer-text-color\":\"rgb(13, 13, 13)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/e(\"p\",{children:\"A filter for refining your data stew (to keep only the most relevant chunks)\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"12px\",\"--framer-text-color\":\"rgb(13, 13, 13)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/e(\"p\",{children:\"An open-source LLM for data handling (filtering, profiling, generation and transformation). We prefer Mixtral 8x7B, but you can choose your favorite flavor\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"12px\",\"--framer-text-color\":\"rgb(13, 13, 13)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/e(\"p\",{children:\"Web search to fetch fresh data and improve answer generation using web queries crafted from profiles\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"12px\",\"--framer-text-color\":\"rgb(13, 13, 13)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/e(\"p\",{children:\"A splash of LLM-based hot sauce to generate diverse prompts\\xa0\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"12px\",\"--framer-text-color\":\"rgb(13, 13, 13)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/e(\"p\",{children:\"Tools for textual transformation and deduplication to smooth out the texture\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"12px\",\"--framer-text-color\":\"rgb(13, 13, 13)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/e(\"p\",{children:\"Human experts for taste tasting (to evaluate the generated data and add improvements)\"})})]}),/*#__PURE__*/e(\"h2\",{children:\"Cooking the prompts\"}),/*#__PURE__*/e(\"h3\",{children:\"Step 1. Gather your ingredients: collect raw texts\"}),/*#__PURE__*/e(\"p\",{children:\"Start by collecting a set of raw texts that correspond to your domain of interest. These can be open-source datasets already published online or the results of web scraping. Filter each document from the web, and keep the ones relevant to your task. For example, if you need to generate a medical dataset, look for web texts from the medical domain. While doing so, make certain the collected data complies with all legal and ethical standards.\"}),/*#__PURE__*/e(\"p\",{children:\"This step provides the foundational material for generating synthetic data tailored to your needs.\"}),/*#__PURE__*/e(\"p\",{children:\"Let\u2019s illustrate this step with a simple example. Suppose we want to prepare a synthetic pipeline in the domain of physics, more specifically about quarks and gluons. To achieve this, we must gather extensive information and key points about the topic. Our collection would include texts with interesting facts about quarks and gluons, their characteristics, underlying principles, and so on.\"}),/*#__PURE__*/e(\"h3\",{children:\"Step 2. Chop it up: break the texts into chunks\"}),/*#__PURE__*/e(\"p\",{children:\"Slice your long texts into smaller, more digestible paragraphs. Instead of generating one generic prompt from a longer document, divide it into smaller, logical texts and generate separate prompts for each. This step increases the dataset volume and creates a more diverse dataset. Separate parts highlight different aspects of the topic, leading to more varied data and unique prompts.\"}),/*#__PURE__*/e(\"p\",{children:\"Think of this as prepping your vegetables for a stir-fry: you want the perfect size chunks. For your dataset, they should be long enough to be meaningful but short enough to fit several into the context window of the LLM you intend to use later.\"}),/*#__PURE__*/e(\"p\",{children:\"Going back to our physics example, we could divide a text about quarks and gluons into distinct paragraphs: \u201CWhat are quarks and gluons?\u201D, \u201CInteresting facts about quarks and gluons\u201D and \u201CQuarks and gluons in the standard model\u201D.\"}),/*#__PURE__*/e(\"h3\",{children:\"Step 3. Sift carefully: filter texts by relevancy\"}),/*#__PURE__*/e(\"p\",{children:\"Use your filtering tool to sift through the text chunks, keeping only those relevant to your specific needs, such as domain and subdomains. You want to make sure you keep useful information and discard irrelevant noise. For example, we use an LLM to check that each paragraph is informative and relevant to our needs.\"}),/*#__PURE__*/e(\"h3\",{children:\"Step 4. Choose the seasonings: generate text profiles\"}),/*#__PURE__*/e(\"p\",{children:\"Open-source LLMs are ineffective at generating complex prompts directly from raw documents. You can get around this limitation by using profiling. Instruct the LLM to read each of the previously created chunks and generate a profile for it. This involves summarizing the chunk into several sentences, each corresponding to a specific property like topic, subdomain, and so on.\\xa0\\xa0\"}),/*#__PURE__*/t(\"p\",{children:[\"Let's understand this concept through an example, where we generate a profile for a sample document from our physics collection: \",/*#__PURE__*/e(\"br\",{}),\"Generated profile:\",/*#__PURE__*/e(\"br\",{}),\"- Domain: Natural sciences\",/*#__PURE__*/e(\"br\",{}),\"- Subdomain: Physics\",/*#__PURE__*/e(\"br\",{}),\"- Topic: Quarks and Gluons, their properties, and their role in the strong nuclear force\",/*#__PURE__*/e(\"br\",{}),\"- Use Case: Learning about the fundamental particles that make up matter, their interactions, and their study in high-energy physics experiments\\xa0\"]}),/*#__PURE__*/e(\"p\",{children:\"Think of it as identifying the key flavors in your dish \u2014 maybe ginger or peppers? The point is to summarize and categorize the data effectively, which improves the complexity and quality of the prompts. Additionally, based on the profiles, you can control the diversity and distribution of the dataset by selecting the percentage of each domain included in the final dataset.\"}),/*#__PURE__*/e(\"h3\",{children:\"Step 5. Add fresh ingredients: source web documents\"}),/*#__PURE__*/e(\"p\",{children:\"Obtaining fresh documents from the web is essential, as it enhances the scalability and diversity of the final dataset. Once you have identified your key flavors (profiles), it\u2019s time to throw in all the fresh ingredients. You do this in a couple of steps:\"}),/*#__PURE__*/e(\"ol\",{children:/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"12px\",\"--framer-text-color\":\"rgb(13, 13, 13)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/e(\"p\",{children:\"Like writing a shopping list based on a recipe, use the created profiles and a selected LLM to generate web queries.\"})})}),/*#__PURE__*/e(\"p\",{children:\"These are some examples of generated search queries:\"}),/*#__PURE__*/t(\"p\",{children:[\"- what are quarks and gluons\",/*#__PURE__*/e(\"br\",{}),\"- quark properties and types\",/*#__PURE__*/e(\"br\",{}),\"- gluon's role in atoms\",/*#__PURE__*/e(\"br\",{}),\"- history of quark discovery\",/*#__PURE__*/e(\"br\",{}),\"- applications of quark and gluon research in physics\"]}),/*#__PURE__*/t(\"ol\",{start:\"2\",children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"12px\",\"--framer-text-color\":\"rgb(13, 13, 13)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/e(\"p\",{children:\"Using these queries, search the web and find additional relevant documents.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"12px\",\"--framer-text-color\":\"rgb(13, 13, 13)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/e(\"p\",{children:\"Clean and parse these new ingredients and mix them into your source data.\"})})]}),/*#__PURE__*/e(\"p\",{children:\"Repeat these steps until you have enough data. In our projects, we usually need 10 to 20 thousand prompts for a domain, so you can keep this number at the back of your mind.\"}),/*#__PURE__*/e(\"h3\",{children:\"Step 6. Start cooking: generate prompts\"}),/*#__PURE__*/e(\"p\",{children:\"Once you have gathered your ingredients, it\u2019s time to create a balanced dish. Use the profiles created from the source data and instruct the LLM to generate a variety of prompts that cover various aspects of the topic and are suited for the task at hand. Just as different textures and ingredients enhance the richness and flavor of a meal, you should incorporate diverse and detailed prompts to get a more robust dataset.\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"Examples of generated prompts would look like this:\\xa0\"}),/*#__PURE__*/t(\"ul\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"12px\",\"--framer-text-color\":\"rgb(0, 0, 0)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/e(\"p\",{children:\"What role do quarks and gluons play in particle physics, and how do they help form protons and neutrons?\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"12px\",\"--framer-text-color\":\"rgb(13, 13, 13)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/e(\"p\",{children:\"How many types of quarks are there, and what are their names?\"})})]}),/*#__PURE__*/e(\"h3\",{children:\"Step 7. Explore the flavor palette: focus on breadth and diversity\"}),/*#__PURE__*/e(\"p\",{children:\"To evolve the breadth of your prompts with the LLM, generate multiple similar prompts across different topics and use cases to explore the flavor palette. The diversity of prompts is important to the success of the whole pipeline. You want to adapt your SFT dataset to various use cases within a given domain.\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"Toloka implements various automated and manual checks to guarantee that the domain is covered broadly. Here are some examples of how to broaden your collection of prompts:\"}),/*#__PURE__*/t(\"ul\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"12px\",\"--framer-text-color\":\"rgb(14, 14, 14)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/e(\"p\",{children:\"How do electrons interact within an atom, and what role do they play in chemical bonding and the formation of molecules? I\u2019m particularly interested in understanding the differences between ionic and covalent bonds.\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"12px\",\"--framer-text-color\":\"rgb(14, 14, 14)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/e(\"p\",{children:\"How do neutrinos interact with matter given their weak interaction with other particles, what makes them so elusive, and what techniques do scientists use to detect and study them?\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"12px\",\"--framer-text-color\":\"rgb(14, 14, 14)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/e(\"p\",{children:\"What advancements in detector technology and experimental techniques have allowed scientists to study high-energy particle collisions, and what recent discoveries have been made as a result of these experiments?\"})})]}),/*#__PURE__*/e(\"h3\",{children:\"Step 8. Add lexical spice: rephrase monotonous prompts\"}),/*#__PURE__*/e(\"p\",{children:\"After completing the steps above, you\u2019ll probably find that most prompts are lexically identical. For instance, numerous prompts begin with \u201CCould you please explain [topic of interest]?\u201D This results in a monotonous dataset with repetitive language patterns. To avoid this, use your LLM to rephrase your prompts to add lexical diversity, ensuring a rich, complex taste that avoids bland repetition.\"}),/*#__PURE__*/t(\"p\",{children:[\"Let\u2019s illustrate this with an example. For the input \u201CWhat role do quarks and gluons play in particle physics, and how do they help form protons and neutrons?\u201D, you could generate the following prompts:\",/*#__PURE__*/e(\"br\",{}),/*#__PURE__*/e(\"br\",{}),\"- Could someone help me understand the role of quarks and gluons in particle physics? I'm particularly interested in their contribution to the formation of protons and neutrons.\",/*#__PURE__*/e(\"br\",{}),\"- I\u2019m curious about quarks and gluons in particle physics. How do they help create protons and neutrons?\",/*#__PURE__*/e(\"br\",{}),\"- What is the role of quarks and gluons in particle physics, and how do they contribute to the formation of protons and neutrons?\"]}),/*#__PURE__*/e(\"h3\",{children:\"Step 9. Watch the pot: discard bad prompts\"}),/*#__PURE__*/e(\"p\",{children:\"This step is crucial for maintaining the quality of your dish.\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"Let\u2019s say you are generating a recipe for an SFT pipeline on the topic of quarks and gluons, but you notice some odd instructions, such as \u201CA group of flamingoes is called a flamboyance\u201D or \u201CThere are four types of quarks\u201D. The first one is irrelevant to your recipe, while the second one is incorrect and can even ruin your results.\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"It is important to discard any irrelevant or meaningless prompts and only keep the ones that are useful for your task. You can perform automatic prompt filtering using the LLM. This closely resembles step 3, where you filtered raw texts, but now you are focusing on filtering generated prompts.\"}),/*#__PURE__*/e(\"h3\",{children:\"Step 10. Smooth out clumps: minimize repetitive prompts\"}),/*#__PURE__*/e(\"p\",{children:\"You don\u2019t want your data clumping together around the same topics. We recommend running a similarity search to minimize repetition as much as possible. A good approach is to transform the text into embeddings and filter out any closely similar samples. This ensures each prompt in the collection remains distinct and covers relevant domain areas. This step is important for avoiding repetitive samples and enhancing data diversity.\"}),/*#__PURE__*/e(\"h3\",{children:\"Step 11. Bring in the taste testers: add a human feedback loop\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"With any new recipe, you want to have taste testers critique the dish before you serve it. The same goes for data production.\\xa0\"}),/*#__PURE__*/t(\"p\",{children:[\"Despite the improvements synthetic data introduces, you still need human input when developing a robust pipeline for SFT. Integrating human feedback into the process can ensure the quality and scalability of the generated data. To better simulate real-world conditions, human experts validate the synthetic data, usually by accepting, rejecting, or modifying the generated samples/prompts. If the generated prompt is good, it will be accepted; otherwise, it will be modified or discarded altogether. The experts only edit the prompts that are not quite ready for acceptance but are useful and can be easily refined.\\xa0 This \u201Chuman-in-the-loop\u201D approach enhances the reliability and applicability of the data. The Toloka team uses a variety of \",/*#__PURE__*/e(a,{href:\"https://toloka.ai/knowledgebase/quality-control/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"quality control\"})}),\" tools at every step in this process.\"]}),/*#__PURE__*/e(\"p\",{children:\"The ratio of human input versus synthetic data depends on specific criteria for each project. For example, you may opt for a more synthetically generated dataset if budget is a priority. If you\u2019re focusing on quality, you may need more human-generated data. The criteria are based on set thresholds, parameters, and quality control conditions, which help set an optimal tradeoff for each use case.\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"Your prompt starter is ready to serve. You can see a more formal representation of the prompt generation pipeline in the image below.\"}),/*#__PURE__*/e(\"h2\",{children:\"Cooking the answers\"}),/*#__PURE__*/e(\"p\",{children:\"Now that you have a set of prompts, you need to add a pipeline of hearty answers to it. You can reuse parts of the prompt collection recipe here.\"}),/*#__PURE__*/e(\"h3\",{children:\"Step 1. Line up the condiments: use extended web queries\"}),/*#__PURE__*/e(\"p\",{children:\"Whether it\u2019s mayo, ketchup, or mint chutney, everyone appreciates a selection of condiments. But you need depth to improve the quality of answers for your dataset.\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"Generate several queries per prompt and save several top search results to substantially expand the range of your answers. You could extend the above example and generate additional queries that cover more details or aspects. Similar queries to the original prompt, \u201CHow many types of quarks exist, and what are their names?\u201D include:\\xa0\"}),/*#__PURE__*/t(\"ul\",{children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"12px\",\"--framer-text-color\":\"rgb(13, 13, 13)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/e(\"p\",{children:\"Which are the six types of quarks?\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"12px\",\"--framer-text-color\":\"rgb(13, 13, 13)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/e(\"p\",{children:\"How do different types of quarks interact?\"})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",style:{\"--framer-font-size\":\"12px\",\"--framer-text-color\":\"rgb(13, 13, 13)\",\"--framer-text-decoration\":\"none\"},children:/*#__PURE__*/e(\"p\",{children:\"What role do different types of quarks play in particle physics?\"})})]}),/*#__PURE__*/e(\"p\",{children:\"With this approach, you provide more informative and detailed answers, for a more in-depth understanding of the topic. As a result, you expand the diversity and volume of the dataset, which significantly improves the quality of the data and the performance of the model itself.\"}),/*#__PURE__*/e(\"h3\",{children:\"Step 2. Chop-n-RAG: generate synthetic answers\"}),/*#__PURE__*/e(\"p\",{children:\"Just like the second step in the prompt generation pipeline, you need to split the relevant web pages into chunks. Then you can search for the chunks most relevant to a given prompt and add them to the model context (similarly to the RAG approach). If you don\u2019t incorporate RAG, the LLM will generate answers based solely on the information it was trained on.\\xa0\"}),/*#__PURE__*/e(\"p\",{children:\"Retrieval-augmented generation (RAG) uses an information retrieval component that enables the user to first pull relevant data from external sources. The new data is then combined with the original training data, which assists the LLM in creating more accurate and relevant responses. This extra knowledge adds domain-specific information, lowers hallucinations, and improves the overall accuracy of answers.\"}),/*#__PURE__*/e(\"p\",{children:\"Such context-boosted LLM can be further used to get a short list of synthetic answers. For example, for the prompt \u201CHow many types of quarks exist and what are their names?\u201D, the LLM might generate an answer such as\\xa0 \u201CThere are six types of quarks: up, down, charm, strange, top and bottom.\u201D\\xa0\"}),/*#__PURE__*/e(\"h3\",{children:\"Step 3. Add your secret sauce: get expert evaluations and fine-tune\"}),/*#__PURE__*/e(\"p\",{children:\"If you are not satisfied with your synthetic answers, you can recruit knowledgeable experts to evaluate the answers on scales like style, correspondence to the domain, context relevance, etc. This is done similarly to step 11 in prompt generation, where we include humans in the process to validate or edit the answers and assess the overall performance of the pipeline.\"}),/*#__PURE__*/e(\"p\",{children:\"Once you have your evaluations, you can fine-tune the model to the domain and refine the quality of your synthetic answers.\"}),/*#__PURE__*/e(\"p\",{children:\"Like in cooking, the key to AI development success is carefully selecting your ingredients, applying the right techniques, and not being afraid to experiment. Tune on the data while it\u2019s hot, and bon app\\xe9tit!\"}),/*#__PURE__*/e(\"h2\",{children:\"Learn more\\xa0\"}),/*#__PURE__*/t(\"p\",{children:[\"Need help building your SFT pipeline? Reach out to \",/*#__PURE__*/e(a,{href:\"https://toloka.ai/talk-to-us/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"Toloka\"})}),\", and our experts will help you create the perfect recipe!\"]})]});export const richText5=/*#__PURE__*/t(i.Fragment,{children:[/*#__PURE__*/e(\"p\",{children:\"In the landscape of artificial intelligence projects, high quality data is the fundamental fuel that powers innovation and drives meaningful outcomes. Whether it's developing natural language processing systems or enhancing computer vision applications, the quality and nature of data play a pivotal role in determining the success of AI endeavors.\"}),/*#__PURE__*/e(\"p\",{children:\"The world continues to generate an ever-expanding data flow, and it becomes increasingly important to comprehend ways to handle such vast volumes. The proliferation of data presents unprecedented opportunities for problem-solving, but businesses trying to harness this potential need strategies for processing, analyzing, and deriving insights from these immense datasets.\"}),/*#__PURE__*/e(\"img\",{alt:\"Understanding the Difference Between Labeled and Unlabeled Data\",className:\"framer-image\",height:\"406\",src:\"https://framerusercontent.com/images/MNZY3hUkCbUvMX4CiZgvenwjaDI.jpeg\",srcSet:\"https://framerusercontent.com/images/MNZY3hUkCbUvMX4CiZgvenwjaDI.jpeg?scale-down-to=512 512w,https://framerusercontent.com/images/MNZY3hUkCbUvMX4CiZgvenwjaDI.jpeg?scale-down-to=1024 1024w,https://framerusercontent.com/images/MNZY3hUkCbUvMX4CiZgvenwjaDI.jpeg 1200w\",style:{aspectRatio:\"1200 / 812\"},width:\"600\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"em\",{children:\"The amount of data is growing exponentially with some 328.77 million terabytes created, i. e. generated, copied or captured, every day. (Source: \"}),/*#__PURE__*/e(a,{href:\"https://explodingtopics.com/blog/data-generated-per-day\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!0,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:/*#__PURE__*/e(\"em\",{children:\"Exploding Topics\"})})}),/*#__PURE__*/e(\"em\",{children:\")\"})]}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"br\",{className:\"trailing-break\"})}),/*#__PURE__*/e(\"p\",{children:\"Central to this discussion is the distinction between labeled and unlabeled data, each with unique characteristics, implications, and applications. In this article, we delve into the importance of data as the cornerstone of AI projects and explore the nuanced differences between labeled and unlabeled data.\"}),/*#__PURE__*/e(\"h2\",{children:\"Training Datasets for Machine Learning\"}),/*#__PURE__*/e(\"p\",{children:\"Data is the foundation of Machine Learning, as it constitutes a collection of observations or measurements for model training and validation. The efficacy and abundance of data impact the model's performance as we must process considerable volumes of it at every stage of an ML project development.\"}),/*#__PURE__*/e(\"p\",{children:\"Industrial manufacturing, IT, finance, retail, and healthcare are just a few industries with a growing demand for specific training and testing data for new solutions. AI enables the extraction of complex representations via hierarchical learning, necessitating mining meaningful patterns from vast datasets.\"}),/*#__PURE__*/e(\"p\",{children:\"The constant growth of the AI training datasets market clearly illustrates the importance of data. The production of massive data volumes, technical advancements, and the increasing adoption of AI technology across various sectors fuel this growth.\"}),/*#__PURE__*/e(\"img\",{alt:\"Training Datasets for Machine Learning\",className:\"framer-image\",height:\"510\",src:\"https://framerusercontent.com/images/lHBferj3I1irlHptu17rpXK0jH8.jpeg\",srcSet:\"https://framerusercontent.com/images/lHBferj3I1irlHptu17rpXK0jH8.jpeg?scale-down-to=512 512w,https://framerusercontent.com/images/lHBferj3I1irlHptu17rpXK0jH8.jpeg?scale-down-to=1024 1024w,https://framerusercontent.com/images/lHBferj3I1irlHptu17rpXK0jH8.jpeg?scale-down-to=2048 2048w,https://framerusercontent.com/images/lHBferj3I1irlHptu17rpXK0jH8.jpeg 2400w\",style:{aspectRatio:\"2400 / 1020\"},width:\"1200\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"em\",{children:\"Brainy Insights valued the AI training dataset market at $ 1.62 Billion in 2022 and forecasted it to grow to $ 13.75 Billion by 2032. (Source: \"}),/*#__PURE__*/e(a,{href:\"https://www.thebrainyinsights.com/report/ai-training-dataset-market-13562\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!0,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:/*#__PURE__*/e(\"em\",{children:\"The Brainy Insights\"})})}),/*#__PURE__*/e(\"em\",{children:\")\"})]}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"br\",{className:\"trailing-break\"})}),/*#__PURE__*/e(\"p\",{children:\"Data manifests in diverse formats, including numerical and categorical, and derives from multiple origins such as databases, spreadsheets, or APIs. Machine learning algorithms leverage data to distinguish patterns and correlations among input variables and desired outcomes, facilitating predictive or classificatory objectives.\"}),/*#__PURE__*/e(\"h3\",{children:\"How Do Machine Learning Models Work?\"}),/*#__PURE__*/e(\"p\",{children:\"Machine learning begins with the collection and preparation of data from various sources. Numerical, visual, or textual information serves for machine learning models training, enabling them to identify patterns and make predictions. Developers select a machine learning model, supply the data, and allow the model to train itself, with opportunities for human intervention to refine the model's parameters over time.\"}),/*#__PURE__*/e(\"p\",{children:\"The evaluation data, held separately from the training data, tests the accuracy of the trained model with new data. Machine learning can serve descriptive, predictive, or prescriptive functions, depending on the desired outcome.\"}),/*#__PURE__*/e(\"img\",{alt:\"How Do Machine Learning Models Work? 1\",className:\"framer-image\",height:\"350\",src:\"https://framerusercontent.com/images/8XtlTdOjJRhU6Mraa1pQUmn8LuI.jpeg\",srcSet:\"https://framerusercontent.com/images/8XtlTdOjJRhU6Mraa1pQUmn8LuI.jpeg?scale-down-to=512 512w,https://framerusercontent.com/images/8XtlTdOjJRhU6Mraa1pQUmn8LuI.jpeg?scale-down-to=1024 1024w,https://framerusercontent.com/images/8XtlTdOjJRhU6Mraa1pQUmn8LuI.jpeg?scale-down-to=2048 2048w,https://framerusercontent.com/images/8XtlTdOjJRhU6Mraa1pQUmn8LuI.jpeg 2400w\",style:{aspectRatio:\"2400 / 700\"},width:\"1200\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"em\",{children:\"Validation data is meant for frequent evaluation of the model during training, while testing is done when the model is ready to use. (Source: \"}),/*#__PURE__*/e(a,{href:\"https://www.geeksforgeeks.org/ml-introduction-data-machine-learning/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!0,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:/*#__PURE__*/e(\"em\",{children:\"Geeksforgeeks\"})})}),/*#__PURE__*/e(\"em\",{children:\")\"})]}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"br\",{className:\"trailing-break\"})}),/*#__PURE__*/e(\"p\",{children:\"Machine learning encompasses three primary subcategories: supervised, unsupervised, and reinforcement learning, each distinguished by unique methodologies and applications.\"}),/*#__PURE__*/e(\"p\",{children:\"In supervised learning, datasets contain desired outputs, enabling the function to calculate prediction errors. Feedback is provided to adjust the function and refine the mapping based on the disparity between actual and desired outcomes. Conversely, unsupervised learning operates without desired outputs, necessitating the function to segment datasets into distinct classes autonomously.\"}),/*#__PURE__*/e(\"p\",{children:\"Reinforcement learning involves the algorithm learning actions to achieve a goal state, with feedback provided intermittently through reinforcement signals. It\u2019s similar to human learning, where rewards prompt adjustments rather than continuous feedback.\"}),/*#__PURE__*/e(\"img\",{alt:\"How Do Machine Learning Models Work? 2\",className:\"framer-image\",height:\"500\",src:\"https://framerusercontent.com/images/trz3pJ6txCAYDbhwDgTiB7ojEvA.jpeg\",srcSet:\"https://framerusercontent.com/images/trz3pJ6txCAYDbhwDgTiB7ojEvA.jpeg?scale-down-to=512 512w,https://framerusercontent.com/images/trz3pJ6txCAYDbhwDgTiB7ojEvA.jpeg?scale-down-to=1024 1024w,https://framerusercontent.com/images/trz3pJ6txCAYDbhwDgTiB7ojEvA.jpeg?scale-down-to=2048 2048w,https://framerusercontent.com/images/trz3pJ6txCAYDbhwDgTiB7ojEvA.jpeg 2400w\",style:{aspectRatio:\"2400 / 1000\"},width:\"1200\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"em\",{children:\"Three learning models for algorithms. (Source: \"}),/*#__PURE__*/e(a,{href:\"https://developer.ibm.com/articles/cc-models-machine-learning/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!0,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:/*#__PURE__*/e(\"em\",{children:\"IBM Developer\"})})}),/*#__PURE__*/e(\"em\",{children:\")\"})]}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"br\",{className:\"trailing-break\"})}),/*#__PURE__*/e(\"p\",{children:\"Each machine learning category presumes a particular type of training dataset, which directly brings us to the difference between labeled and unlabeled data.\"}),/*#__PURE__*/e(\"h2\",{children:\"What Is Unlabeled Data?\"}),/*#__PURE__*/e(\"p\",{children:\"Unlabeled data comprises elements devoid of specific identifiers. Basically, this term can be applied to any raw data lacking explicit \u2018tags\u2019 or \u2018labels\u2019 denoting their attributes or properties.\"}),/*#__PURE__*/e(\"p\",{children:\"Interpreting such data poses a more significant challenge due to the absence of predefined classifications. However, their significance remains undeniable, particularly in contexts prioritizing exploration over guided analysis.\"}),/*#__PURE__*/e(\"img\",{alt:\"What Is Unlabeled Data?\",className:\"framer-image\",height:\"320\",src:\"https://framerusercontent.com/images/Bq43gGsHz8JTOvn1LbarsHdilI.jpeg\",srcSet:\"https://framerusercontent.com/images/Bq43gGsHz8JTOvn1LbarsHdilI.jpeg?scale-down-to=512 512w,https://framerusercontent.com/images/Bq43gGsHz8JTOvn1LbarsHdilI.jpeg?scale-down-to=1024 1024w,https://framerusercontent.com/images/Bq43gGsHz8JTOvn1LbarsHdilI.jpeg 1200w\",style:{aspectRatio:\"1200 / 640\"},width:\"600\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"em\",{children:\"Unsupervised learning scheme. (Source: \"}),/*#__PURE__*/e(a,{href:\"https://www.g2.com/articles/supervised-vs-unsupervised-learning\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!0,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:/*#__PURE__*/e(\"em\",{children:\"G2\"})})}),/*#__PURE__*/e(\"em\",{children:\")\"})]}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"br\",{className:\"trailing-break\"})}),/*#__PURE__*/e(\"p\",{children:\"ML algorithms can find similarities and differences among unlabeled data points and sort them based on the characteristics they have noted.\"}),/*#__PURE__*/e(\"h3\",{children:\"Unlabeled Data Usage Principles\"}),/*#__PURE__*/e(\"p\",{children:\"Unlabeled data is predominantly used in unsupervised machine learning to uncover hidden patterns and derive valuable insights. In this case, ML models receive only input data without any corresponding output and are allowed to deal with it without any human interference.\"}),/*#__PURE__*/e(\"p\",{children:\"Unsupervised learning is relatively close to the process of human training from their own experience. It comprises different kinds of algorithms, usually categorized in one of the two groups:\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Clustering.\"}),\" This method allows us to find similarities among different objects and group them according to the presence or absence of these similarities.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Association.\"}),\" This method is used to identify relations between different items in a dataset, allowing us to notice which of them tend to occur together.\"]}),/*#__PURE__*/e(\"img\",{alt:\"Unlabeled Data Usage Principles\",className:\"framer-image\",height:\"320\",src:\"https://framerusercontent.com/images/hJqEBAPNfrG46o1kz7p58aFP4DI.jpeg\",srcSet:\"https://framerusercontent.com/images/hJqEBAPNfrG46o1kz7p58aFP4DI.jpeg?scale-down-to=512 512w,https://framerusercontent.com/images/hJqEBAPNfrG46o1kz7p58aFP4DI.jpeg?scale-down-to=1024 1024w,https://framerusercontent.com/images/hJqEBAPNfrG46o1kz7p58aFP4DI.jpeg 1200w\",style:{aspectRatio:\"1200 / 640\"},width:\"600\"}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"em\",{children:\"Market basket analysis is a typical association task. In this case, unsupervised learning algorithms can find what products people buy together more often. (Source: Linoff G., Berry M. Data Mining Techniques: For Marketing, Sales, and Customer Relationship Management)\"})}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"br\",{className:\"trailing-break\"})}),/*#__PURE__*/e(\"h3\",{children:\"Unlabeled Datasets Use Cases\"}),/*#__PURE__*/e(\"p\",{children:\"In business analytics, unlabeled data is pivotal in uncovering hidden connections and informing strategic decision-making. From market segmentation to anomaly detection, businesses leverage unlabeled data across diverse applications to drive innovation and gain a competitive edge with ML-driven insights.\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Market Segmentation:\"}),\" Unlabeled data can help to identify distinct customer segments based on purchasing behavior.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Anomaly Detection:\"}),\" Unlabeled data processing can help to flag unusual patterns or behaviors in operational processes or financial transactions.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Product Recommendation Systems:\"}),\" Unlabeled data is instrumental in personalizing suggestions based on a particular user\u2019s behavior and preferences.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Supply Chain Optimization:\"}),\" Unlabeled data is employed to identify inefficiencies or bottlenecks within a distribution network.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Text Mining and Sentiment Analysis:\"}),\" Analyzing unstructured textual data sources such as customer reviews, social media posts, or support tickets sometimes allow to extract valuable insights.\"]}),/*#__PURE__*/e(\"h3\",{children:\"Challenges in Managing Unlabeled Data\"}),/*#__PURE__*/e(\"p\",{children:\"Despite its potential benefits, working with unlabeled data and unsupervised learning presumes certain limitations. Extracting meaningful value from it may be challenging due to several factors.\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Interpretation Complexity:\"}),\" Unlabeled data lacks explicit guidance, making it difficult to discern meaningful patterns or relationships. This inherent ambiguity requires sophisticated algorithms to uncover actionable insights.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Noise and Irrelevance:\"}),\" Without clear labels to guide the analysis, it may be difficult to outline valuable signals and not get distracted by a curious coincidence. Addressing this challenge requires robust data preprocessing techniques.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Scalability:\"}),\" As datasets continue to grow in size and complexity, organizations need more and more computational resources to process and analyze vast volumes of unlabeled data.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Lack of Ground Truth:\"}),\" Without a definitive benchmark or reference point, businesses may struggle to objectively evaluate the effectiveness of their analytical models.\"]}),/*#__PURE__*/e(\"p\",{children:\"Each of these challenges must be carefully considered at the initial project estimation stage. It\u2019s entirely possible that labeling the collected data may be more effective and cost-efficient for a particular case.\"}),/*#__PURE__*/e(\"h2\",{children:\"What Is Labeled Data?\"}),/*#__PURE__*/e(\"p\",{children:\"Labeled data refers to a data point that has been annotated with one or multiple tags, unveiling its significance or context. At its core, it consists of input-output pairs, where each raw data item is associated with a corresponding output or label. Human judgment is usually pivotal in this process, as annotators assign labels to the data based on their understanding of the problem domain.\"}),/*#__PURE__*/e(\"p\",{children:\"Labeled datasets form the bedrock of supervised machine learning, serving as a crucial resource for training algorithms with enhanced accuracy. Labels provide ground truth information for the ML model to learn from, guiding it towards making precise predictions.\"}),/*#__PURE__*/e(\"img\",{alt:\"What Is Labeled Data?\",className:\"framer-image\",height:\"450\",src:\"https://framerusercontent.com/images/KyXuIdjQMNbGGvidfFtB3vKqw.jpeg\",srcSet:\"https://framerusercontent.com/images/KyXuIdjQMNbGGvidfFtB3vKqw.jpeg?scale-down-to=512 512w,https://framerusercontent.com/images/KyXuIdjQMNbGGvidfFtB3vKqw.jpeg?scale-down-to=1024 1024w,https://framerusercontent.com/images/KyXuIdjQMNbGGvidfFtB3vKqw.jpeg 1200w\",style:{aspectRatio:\"1200 / 900\"},width:\"600\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"em\",{children:\"Cats or dogs? Classic example of a labeled dataset. (Source: \"}),/*#__PURE__*/e(a,{href:\"https://www.tensorflow.org/datasets/catalog/cats_vs_dogs\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!0,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:/*#__PURE__*/e(\"em\",{children:\"TensorFlow\"})})}),/*#__PURE__*/e(\"em\",{children:\")\"})]}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"br\",{className:\"trailing-break\"})}),/*#__PURE__*/e(\"p\",{children:\"Supervised learning, fueled by labeled data, underpins many applications, from object detection for autonomous vehicles to emotion recognition in customers' calls to support lines.\"}),/*#__PURE__*/e(\"h3\",{children:\"Labeled Data Usage Principles\"}),/*#__PURE__*/e(\"p\",{children:\"Data annotation encompasses diverse forms and can be applied to raw data across various formats, such as images, text, video, audio, or numerical data. This structured information enables a machine learning model to understand the representations in the input data and establish connections to the output through formulated rules. Subsequently, the model utilizes these rules to classify new data or generate predictions based on the learned patterns.\"}),/*#__PURE__*/e(\"img\",{alt:\"Labeled Data Usage Principles\",className:\"framer-image\",height:\"200\",src:\"https://framerusercontent.com/images/00qO92VfHgc5OpetwCAM5FVfxag.jpeg\",srcSet:\"https://framerusercontent.com/images/00qO92VfHgc5OpetwCAM5FVfxag.jpeg?scale-down-to=512 512w,https://framerusercontent.com/images/00qO92VfHgc5OpetwCAM5FVfxag.jpeg?scale-down-to=1024 1024w,https://framerusercontent.com/images/00qO92VfHgc5OpetwCAM5FVfxag.jpeg 1200w\",style:{aspectRatio:\"1200 / 400\"},width:\"600\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"em\",{children:\"Data tags can include names, types, and all kinds of characteristic, qualitative or quantitative. (Source: \"}),/*#__PURE__*/e(a,{href:\"https://livebook.manning.com/book/grokking-machine-learning/2-1-what-is-the-difference-between-labelled-and-unlabelled-data-/v-4/1\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!0,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:/*#__PURE__*/e(\"em\",{children:\"Manning\"})})}),/*#__PURE__*/e(\"em\",{children:\")\"})]}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"br\",{className:\"trailing-break\"})}),/*#__PURE__*/e(\"p\",{children:\"Even the same data can be labeled differently for specific purposes. And supervised learning models comprise two distinctive types:\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Regression.\"}),\" Such models use input features to predict continuous outcomes, often representing quantities or measurements. Examples include temperature, height, or sales revenue, where there is a continuum of possible values rather than distinct categories.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Classification.\"}),\" These models apply the learned patterns to assign discrete labels or categories to input data. Thus they predict a state, for example, identify an object or do sentiment analysis.\"]}),/*#__PURE__*/e(\"h3\",{children:\"Labeled Datasets Use Cases\"}),/*#__PURE__*/e(\"p\",{children:\"Labeled data is essential for ML-driven hypothesis testing, automated object or phenomena detection, and modeling of their possible interactions.\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Market Segmentation:\"}),\" Labeled data enables businesses to categorize customers based on predefined characteristics.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Fraud Detection:\"}),\" Labeled transactional data help detect and prevent fraudulent activities. ML models trained on datasets that include instances of known fraudulent behavior can identify suspicious patterns in real time.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Personalized Recommendations:\"}),\" Businesses can enhance customer engagement by analyzing labeled data on user behavior and product attributes.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Inventory Management:\"}),\" Labeling historical sales data with relevant attributes such as seasonality, product categories, and geographical locations, organizations can make data-driven decisions to forecast demand, optimize stock levels, and minimize supply chain disruptions.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Sentiment Analysis:\"}),\" Labeling customer reviews, social media posts, and survey responses with sentiment labels (e.g., positive, negative, neutral) allows brands to leverage machine learning techniques to identify emerging concerns and proactively address customer feedback.\"]}),/*#__PURE__*/e(\"h3\",{children:\"Challenges in Managing Labeled Data\"}),/*#__PURE__*/e(\"p\",{children:\"Accurate data labeling takes time and effort, necessitating thorough comprehension of the specific tags or annotations required for a given project. Besides, various potential issues must be considered when embarking on a supervised ML project.\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Annotation Complexity:\"}),\" Accurate and consistent annotation of vast datasets is challenging, particularly in domains requiring nuanced labeling, such as medical imaging or natural language processing.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Data Labeling Bias:\"}),\" Labeling data can introduce bias through subjective interpretation or skewed representation of certain classes or categories. Mitigating labeling bias requires careful consideration of annotation guidelines, diverse annotator perspectives, and continual monitoring of labeling processes.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Data Labeling Scale:\"}),\" Balancing the need for labeled data with limited resources, time constraints, and budgetary considerations requires strategic planning and timely corrections.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Labeling Consistency and Quality:\"}),\" Achieving uniformity across annotations, especially in collaborative labeling environments, can be challenging. It usually requires inter-annotator agreement measures and automated quality control that are only possible to create with specific experience.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Data Confidentiality:\"}),\" Labeled data often contains sensitive information, raising concerns about privacy and confidentiality. Managing access controls, implementing data anonymization techniques, and adhering to regulatory compliance standards are essential for protecting individual rights and maintaining data integrity.\"]}),/*#__PURE__*/e(\"p\",{children:\"Addressing these challenges requires a holistic approach encompassing robust annotation protocols, advanced tooling for efficient labeling workflows, and ongoing quality assurance measures. Still, effective management of labeled data is indispensable for unlocking the full potential of machine learning applications across various industries.\"}),/*#__PURE__*/e(\"h2\",{children:\"Benefits of Labeled Data\"}),/*#__PURE__*/e(\"p\",{children:'Accurate data labeling enhances the precision and reliability of predictions generated by a machine learning algorithm. It is a crucial aspect of quality assurance and the only way to guarantee the models are trained on reliable and representative data. A properly labeled dataset provides the essential \"ground truth\" against which the performance of subsequent models can be tested and iterated.'}),/*#__PURE__*/e(\"p\",{children:\"Furthermore, data labeling enhances the usability of variables within a model, optimizing its performance and streamlining data processing. For instance, categorical variables may be reclassified as binary to improve model interpretability and efficiency. Aggregating data in such a manner not only reduces the complexity of the model but also enables the incorporation of control variables, further refining the model's predictive capabilities.\"}),/*#__PURE__*/e(\"img\",{alt:\"Benefits of Labeled Data\",className:\"framer-image\",height:\"315\",src:\"https://framerusercontent.com/images/lhRY1rCFfzv90Rn2FngKxgqzk.jpeg\",srcSet:\"https://framerusercontent.com/images/lhRY1rCFfzv90Rn2FngKxgqzk.jpeg?scale-down-to=512 512w,https://framerusercontent.com/images/lhRY1rCFfzv90Rn2FngKxgqzk.jpeg?scale-down-to=1024 1024w,https://framerusercontent.com/images/lhRY1rCFfzv90Rn2FngKxgqzk.jpeg 1200w\",style:{aspectRatio:\"1200 / 630\"},width:\"600\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"em\",{children:\"The importance of data labeling is confirmed by constant growth of data labeling market. (Source: \"}),/*#__PURE__*/e(a,{href:\"https://www.reportsanddata.com/report-detail/data-collection-labeling-market\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!0,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:/*#__PURE__*/e(\"em\",{children:\"Reports and Data\"})})}),/*#__PURE__*/e(\"em\",{children:\")\"})]}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"br\",{className:\"trailing-break\"})}),/*#__PURE__*/e(\"p\",{children:\"Utilizing high-quality labeled data is indispensable for building reliable and effective machine learning models. By ensuring accurate data labeling, organizations can enhance the reliability of predictions, optimize model usability, and drive innovation across diverse domains.\"}),/*#__PURE__*/e(\"h2\",{children:\"Data Labeling Methods\"}),/*#__PURE__*/e(\"p\",{children:\"Data labeling comprises various methods and techniques, each with its advantages and challenges.\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Manual labeling\"}),\" involves human annotators meticulously examining data and assigning appropriate labels. This method offers high accuracy as humans can understand complex contexts, but it can be time-consuming and expensive, especially for large datasets. However, manual labeling remains indispensable for sensitive or nuanced tasks requiring human and sometimes expert judgment.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Automated labeling\"}),\" leverages algorithms to assign labels to data automatically. This scalable approach may be efficient for tasks with well-defined patterns or structures. However, automated labeling lacks human nuance and context comprehension, leading to inaccuracies, especially in complex datasets.\"]}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Hybrid labeling\"}),\" combines manual and automated methods, where humans validate and refine automatically generated labels. This approach aims to balance accuracy and efficiency, utilizing the strengths of both human judgment and machine speed. It requires specific expertise but allows fast labeling of exceptional quality.\"]}),/*#__PURE__*/e(\"img\",{alt:\"Data Labeling Methods 1\",className:\"framer-image\",height:\"458\",src:\"https://framerusercontent.com/images/3eErs4BtNTaIxFTezv76y87p94.jpeg\",srcSet:\"https://framerusercontent.com/images/3eErs4BtNTaIxFTezv76y87p94.jpeg?scale-down-to=512 512w,https://framerusercontent.com/images/3eErs4BtNTaIxFTezv76y87p94.jpeg?scale-down-to=1024 1024w,https://framerusercontent.com/images/3eErs4BtNTaIxFTezv76y87p94.jpeg 1200w\",style:{aspectRatio:\"1200 / 916\"},width:\"600\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"em\",{children:\"Hybrid data labeling pipeline. Image by Sergei Tilga, R&D, Toloka. (\"}),/*#__PURE__*/e(a,{href:\"https://toloka.ai/blog/llms-and-humans-for-data-labeling/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:/*#__PURE__*/e(\"em\",{children:\"Source\"})})}),/*#__PURE__*/e(\"em\",{children:\")\"})]}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"br\",{className:\"trailing-break\"})}),/*#__PURE__*/e(\"p\",{children:\"Organizations can choose between in-house or outsourcing approaches for data labeling. In-house labeling involves establishing a dedicated team within the organization to perform labeling tasks. This offers greater control over the process and ensures data security but requires substantial resources and expertise.\"}),/*#__PURE__*/e(\"p\",{children:\"Outsourcing involves hiring third-party vendors or specialized companies to handle data labeling.\"}),/*#__PURE__*/e(\"img\",{alt:\"Data Labeling Methods 2\",className:\"framer-image\",height:\"240\",src:\"https://framerusercontent.com/images/r6AxzBy7Hq5kq7grWtx6mVn1mdg.jpeg\",srcSet:\"https://framerusercontent.com/images/r6AxzBy7Hq5kq7grWtx6mVn1mdg.jpeg?scale-down-to=512 512w,https://framerusercontent.com/images/r6AxzBy7Hq5kq7grWtx6mVn1mdg.jpeg 768w\",style:{aspectRatio:\"768 / 481\"},width:\"384\"}),/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"em\",{children:\"Typical image labeling task. (Source: \"}),/*#__PURE__*/e(a,{href:\"https://toloka.ai/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:/*#__PURE__*/e(\"em\",{children:\"Toloka.ai\"})})}),/*#__PURE__*/e(\"em\",{children:\")\"})]}),/*#__PURE__*/e(\"p\",{children:/*#__PURE__*/e(\"br\",{className:\"trailing-break\"})}),/*#__PURE__*/e(\"h2\",{children:\"Final Thoughts\"}),/*#__PURE__*/e(\"p\",{children:\"The value of unlabeled and labeled data for machine learning cannot be overstated. While labeling large datasets may seem daunting, particularly in terms of cost, leveraging human annotators through reliable platforms can significantly mitigate expenses. This approach reduces financial burdens and accelerates the labeling process, enabling organizations to focus on model development and optimization.\"}),/*#__PURE__*/e(\"p\",{children:\"For the most common supervised learning tasks, where a labeled dataset is essential for training, the cost of data annotation is justified by the improved performance and accuracy of the resulting model. In contrast, unsupervised learning algorithms can leverage large volumes of unlabeled data, relying on inherent structures and patterns to uncover insights without manual annotation.\"}),/*#__PURE__*/e(\"p\",{children:\"Ultimately, the cost-effectiveness of data labeling must be evaluated in the context of the entire machine-learning pipeline. Organizations can maximize the value of their data assets by strategically allocating resources and leveraging efficient labeling methods.\"})]});export const richText6=/*#__PURE__*/t(i.Fragment,{children:[/*#__PURE__*/e(\"h2\",{children:\"Are LLMs really responsible and accurate?\"}),/*#__PURE__*/e(\"p\",{children:\"In 2022, Beth Stackpole reported for MIT Sloan, that 79% of companies admit their implementation of responsible AI is limited in scale and scope. To create truly responsible AI, an important safety issue must be discussed: LLM (large language model) fluency in low-resource languages. \"}),/*#__PURE__*/t(\"p\",{children:[\"According to a study from \",/*#__PURE__*/e(a,{href:\"https://arxiv.org/pdf/2310.02446.pdf\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"Brown UniversityOpens a new window\"})}),\" , bad actors have found a loophole to influence LLMs to produce unethical results by:\"]}),/*#__PURE__*/e(\"p\",{children:\"Translating unsafe English inputs into low-resource languages[\u2026] which provides actionable items that can get the users towards their harmful goals 79% of the time.\"}),/*#__PURE__*/e(\"p\",{children:\"In this article, we will explore how LLMs are evaluated, why they don't perform equally well in all languages, and how we can improve their performance. We'll discuss the different types of data needed for LLM development and the specific challenges faced when fine-tuning LLMs for languages with limited resources.\"}),/*#__PURE__*/e(\"p\",{children:\"Additionally, we'll examine the importance of human feedback in refining LLMs, especially in languages where expertise is scarce. By involving experts proficient in non-English languages and establishing rigorous evaluation criteria, we aim to narrow the gap between widely spoken languages and those with fewer resources. This approach seeks to create AI systems that are more inclusive and accountable to diverse linguistic communities.\"}),/*#__PURE__*/e(\"h2\",{children:\"Why Do LLMs Not Work Equally for All Languages?\"}),/*#__PURE__*/e(\"p\",{children:\"The effectiveness of LLMs depends directly on the quality of the data on which they are trained. This data comes in three types:\"}),/*#__PURE__*/t(\"ul\",{style:{\"--framer-font-size\":\"18px\",\"--framer-text-alignment\":\"start\",\"--framer-text-color\":\"rgb(30, 33, 38)\",\"--framer-text-stroke-width\":\"0px\",\"--framer-text-transform\":\"none\"},children:[/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Annotated textual data:\"}),\" This is typically gathered from online sources and used for a base model.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Question-answer pairs:\"}),\" These are utilized for Supervised Fine Tuning (SFT), enabling the base model to follow the instructions and comprehend and respond to queries accurately.\"]})}),/*#__PURE__*/e(\"li\",{\"data-preset-tag\":\"p\",children:/*#__PURE__*/t(\"p\",{children:[/*#__PURE__*/e(\"strong\",{children:\"Human feedback on model responses:\"}),\" It is necessary to align the model with human expectations by either using Reinforcement Learning with Human Feedback (RLHF) and Proximal Policy Optimization (PPO), or applying Direct Performance Optimization (DPO) directly.\"]})})]}),/*#__PURE__*/e(\"p\",{children:\"Developing a multilingual LLM requires access to all three data types for each language the LLM aims to comprehend proficiently. While there may be an abundance of data available in English for training LLMs, many languages have limited resources suitable for this purpose. These languages, commonly known as low-resource languages (LRLs), are spoken by significant populations. Swahili, for example, is used by 200 million people across 14 African countries, but it is considered a LRL.\"}),/*#__PURE__*/e(\"p\",{children:\"Furthermore, the scarcity of raw data is compounded by the lack of attention from the Natural Language Processing (NLP) research community toward certain languages. This results in fewer benchmarks and curated datasets available for training NLP algorithms. To create truly multilingual LLMs and inclusive AI, we need to redirect our focus and include low-resource languages despite the initial challenges it may pose.\"}),/*#__PURE__*/e(\"h2\",{children:\"SFT for Low-resource Languages\"}),/*#__PURE__*/e(\"p\",{children:\"More commonly spoken languages like English have abundant datasets comprising thousands of question-answer pairs readily available for fine-tuning LLMs. While these datasets may not yield highly competitive LLM models, they serve as an initial step in the refinement process. On the other hand, many low-resource languages lack out-of-the-box instruction fine-tuning datasets, necessitating their creation even before basic LLM fine-tuning can start.\"}),/*#__PURE__*/e(\"p\",{children:\"Creating a Supervised Fine-Tuning (SFT) dataset requires meticulous planning to include a diverse range of questions, ensuring representation across all desired categories for LLM proficiency. It\u2019s crucial to allocate the right proportion for each skill, such as summarization, generative writing, question-answering, classification, translation, and more. Moreover, securing native speakers proficient in both question-and-answer composition is essential.\"}),/*#__PURE__*/e(\"p\",{children:\"Typically, the creation process of an SFT dataset involves two distinct projects: question generation and prompt completion, both requiring native speakers known as AI tutors or AI trainers. Additionally, if specialization in fields like medicine, finance, or coding is desired, AI tutors with expertise in these areas are necessary. Managing a substantial number of AI tutors is often required to ensure the production of high-quality prompts and answers.\"}),/*#__PURE__*/e(\"p\",{children:\"There are situations for low-resource languages where the above process is not feasible due to the scarcity of native-speaker experts, and alternative solutions must be explored. One viable approach in this scenario is translating English datasets using an automatic translation to a desired language and then refining the output with native speakers. This method reduces the number of annotators required and is more cost-effective.\"}),/*#__PURE__*/e(\"p\",{children:\"It\u2019s important to note that creating a multilingual LLM proficient in one hundred languages requires repeating this process for each language involved.\"}),/*#__PURE__*/e(\"h2\",{children:\"Human Feedback for Low-resource Languages\"}),/*#__PURE__*/e(\"p\",{children:\"In the final stage of training an LLM, it\u2019s essential to align it with human expectations. This process helps to address concerns like minimizing hallucinations and ensuring the model provides useful and harmless responses. Alignment is achieved by seeking feedback from human experts, who compare two potential responses generated by the model and select the preferred one.\"}),/*#__PURE__*/e(\"p\",{children:\"This feedback-gathering process often entails collecting hundreds of thousands or even millions of comparisons, especially for complex models like the Llama 2 model. Depending on the specific abilities targeted for the LLM, experts in relevant fields are needed to identify inappropriate responses. Due to its scarcity of native experts, this can be a challenge for low-resource languages. Because of this, it is common for LLM developers to prioritize gathering feedback for English or a few other widely spoken languages.\"}),/*#__PURE__*/e(\"p\",{children:\"Like the collection of Supervised Fine-Tuning (SFT) data, gathering human feedback to build multilingual LLMs becomes a large-scale annotation project. It involves utilizing AI tutors proficient in various languages and domain experts. Although simpler than collecting data for SFT, it requires significant volumes of annotations and often needs to be continually updated to enhance the safety and alignment of the LLM.\"}),/*#__PURE__*/e(a,{href:\"https://toloka.ai/evaluation/?utm_source=blog&utm_medium=banner&utm_campaign=eval&utm_content=howtoenhance\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!0,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{className:\"framer-image\",\"data-preset-tag\":\"img\",children:/*#__PURE__*/e(\"img\",{alt:\"\",className:\"framer-image\",height:\"240\",src:\"https://framerusercontent.com/images/tMtJladT8xvGMJ2IUos22zqYuv0.png\",srcSet:\"https://framerusercontent.com/images/tMtJladT8xvGMJ2IUos22zqYuv0.png?scale-down-to=512 512w,https://framerusercontent.com/images/tMtJladT8xvGMJ2IUos22zqYuv0.png?scale-down-to=1024 1024w,https://framerusercontent.com/images/tMtJladT8xvGMJ2IUos22zqYuv0.png 1498w\",style:{aspectRatio:\"1498 / 480\"},width:\"749\"})})}),/*#__PURE__*/e(\"h2\",{children:\"Creating Responsible AI\"}),/*#__PURE__*/e(\"p\",{children:\"Bridging the gap between widely spoken languages and low-resource languages is essential to ensure the safety and alignment of LLMs and ultimately create more responsible AI. Although it may pose initial challenges, this disproportion can be addressed by prioritizing the involvement of experts proficient in non-English languages in the creation and evaluation of LLM datasets.\"}),/*#__PURE__*/t(\"p\",{children:[\"Implementing this approach effectively can ensure that there is no tolerance for unethical outcomes, particularly in languages with fewer resources, thereby closing the loophole highlighted by the \",/*#__PURE__*/e(a,{href:\"https://arxiv.org/pdf/2310.02446.pdf\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"Brown University Low-Resource Languages Jailbreak GPT-4 report\"})}),\". Without adequate attention to multilingual focus and evaluation, companies utilizing LLMs face potential legal, competitive, and reputational risks.\"]}),/*#__PURE__*/t(\"p\",{children:[\"To improve LLM evaluation for responsible AI, consider \",/*#__PURE__*/e(a,{href:\"https://toloka.ai/evaluation/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"partnering with Toloka\"})}),\". We specialize in LLM evaluation, SFT creation, and have access to experts in low-resource languages. \",/*#__PURE__*/e(a,{href:\"https://toloka.ai/evaluation/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"Get in touch\"})}),\", and let's work together to build more inclusive and accountable AI systems.\"]}),/*#__PURE__*/t(\"p\",{children:[\"The original version of this article was published on \",/*#__PURE__*/e(a,{href:\"https://www.spiceworks.com/tech/artificial-intelligence/guest-article/enhance-llm-evaluation-for-responsible-ai/\",motionChild:!0,nodeId:\"eo4RAmtig\",openInNewTab:!1,scopeId:\"contentManagement\",smoothScroll:!1,children:/*#__PURE__*/e(n.a,{children:\"Spiceworks\"})}),\".\"]})]});\nexport const __FramerMetadata__ = {\"exports\":{\"richText5\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"richText6\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"richText1\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"richText3\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"richText4\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"richText\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"richText2\":{\"type\":\"variable\",\"annotations\":{\"framerContractVersion\":\"1\"}},\"__FramerMetadata__\":{\"type\":\"variable\"}}}"],
  "mappings": "6JAAsJ,IAAMA,EAAsBC,EAAIC,EAAS,CAAC,SAAS,CAAcC,EAAE,IAAI,CAAC,SAAS,yIAAyI,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2HAA2H,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wgBAAmgB,CAAC,EAAeA,EAAEC,EAAE,CAAC,KAAK,6HAA6H,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,UAAU,eAAe,kBAAkB,MAAM,SAAsBF,EAAE,MAAM,CAAC,IAAI,GAAG,UAAU,eAAe,OAAO,MAAM,IAAI,sEAAsE,OAAO,oQAAoQ,MAAM,CAAC,YAAY,YAAY,EAAE,MAAM,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,gCAAgC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oWAAoW,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mYAAmY,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sRAAiR,CAAC,EAAeF,EAAE,KAAK,CAAC,MAAM,CAAC,qBAAqB,OAAO,0BAA0B,QAAQ,sBAAsB,kBAAkB,6BAA6B,MAAM,0BAA0B,MAAM,EAAE,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,wBAAwB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,kBAAkB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,gBAAgB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,wCAAwC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0JAA0J,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,2DAA2D,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uYAA6X,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,guBAAguB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kaAAka,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uTAAuT,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,4BAA4B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gIAA2H,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,sBAAsB,CAAC,EAAE,uUAAkU,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,eAAe,CAAC,EAAE,uNAAuN,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gBAAgB,CAAC,EAAeF,EAAE,KAAK,CAAC,MAAM,CAAC,qBAAqB,OAAO,0BAA0B,QAAQ,sBAAsB,kBAAkB,6BAA6B,MAAM,0BAA0B,MAAM,EAAE,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,kBAAkB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,kBAAkB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,wFAAwF,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sHAAiH,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,kBAAkB,CAAC,EAAE,gZAA2Y,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,iCAAiC,CAAC,EAAE,0RAA0R,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,yBAAyB,CAAC,EAAE,kOAAkO,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6MAA6M,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,qCAAqC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,8CAA8C,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uDAAuD,CAAC,EAAeF,EAAE,KAAK,CAAC,MAAM,CAAC,qBAAqB,OAAO,0BAA0B,QAAQ,sBAAsB,kBAAkB,6BAA6B,MAAM,0BAA0B,OAAO,oBAAoB,OAAO,EAAE,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,iCAAiC,CAAC,EAAE,qRAAqR,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,sCAAsC,CAAC,EAAE,uXAAuX,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6aAA6a,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,yJAAsKE,EAAEC,EAAE,CAAC,KAAK,kCAAkC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,SAAS,CAAC,CAAC,CAAC,EAAE,GAAG,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,gDAAgD,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,sCAAsC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oXAAoX,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2WAA2W,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,oBAAoB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yUAAyU,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0WAA0W,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,geAA2d,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,6DAA6D,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oKAAoK,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,0BAA0B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+aAA0a,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2UAA2U,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+PAA+P,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,2HAAwIE,EAAEC,EAAE,CAAC,KAAK,2CAA2C,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,qBAAqB,CAAC,CAAC,CAAC,EAAE,2EAAsE,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,0CAA0C,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mYAAmY,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0WAA0W,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4SAA4S,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,0DAA0D,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gaAAga,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gTAAgT,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,YAAY,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kTAAkT,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qJAAqJ,CAAC,EAAeF,EAAE,KAAK,CAAC,MAAM,CAAC,qBAAqB,OAAO,0BAA0B,QAAQ,sBAAsB,kBAAkB,6BAA6B,MAAM,0BAA0B,MAAM,EAAE,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,qBAAqB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,cAAc,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,iBAAiB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,kCAAkC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,uCAAuC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4MAAuM,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wjBAAwjB,CAAC,CAAC,CAAC,CAAC,EAAeG,EAAuBL,EAAIC,EAAS,CAAC,SAAS,CAAcC,EAAE,IAAI,CAAC,SAAS,2aAA2a,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sTAAsT,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gfAAgf,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,kDAAkD,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,uDAAoEE,EAAEC,EAAE,CAAC,KAAK,yDAAyD,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,8CAA8C,CAAC,CAAC,CAAC,EAAE,4LAA4L,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,sSAAsS,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,ycAAyc,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gYAAgY,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+XAAgY,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wUAAwU,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iVAAiV,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oNAAoN,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,gDAAgD,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,wBAAwB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mXAAmX,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,wBAAwB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uSAAuS,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,ocAAoc,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0PAA0P,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,wBAAwB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sTAAsT,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+NAA+N,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8WAA8W,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sdAA4c,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,iCAAiC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,uCAAuC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+UAA+U,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kYAAkY,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,sCAAsC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+gBAA+gB,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,uCAAuC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gYAAgY,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,mCAAmC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,2NAAwOE,EAAEC,EAAE,CAAC,KAAK,mCAAmC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,uEAAuE,CAAC,CAAC,CAAC,EAAE,4IAA4I,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,+SAA+S,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gRAAgR,CAAC,EAAeF,EAAE,KAAK,CAAC,MAAM,CAAC,qBAAqB,OAAO,0BAA0B,QAAQ,sBAAsB,kBAAkB,6BAA6B,MAAM,0BAA0B,MAAM,EAAE,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,uBAAuB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,sBAAsB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,oBAAoB,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,gCAAgC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4VAA4V,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,+BAA+B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2UAA2U,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uTAAuT,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,mBAAmB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0MAA0M,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mYAAmY,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2eAA2e,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wXAAwX,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iSAAiS,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,yCAAyC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0WAA0W,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kYAA6X,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,4GAAyHE,EAAEC,EAAE,CAAC,KAAK,+CAA+C,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,sEAAsE,CAAC,CAAC,CAAC,EAAE,ibAAib,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,6bAA6b,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,mCAAmC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iNAAiN,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,wCAAwC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6UAA6U,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,sBAAsB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yYAAyY,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0hBAAsgB,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,kCAAkC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4ZAA4Z,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4iBAA4iB,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,oCAAoC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iJAAiJ,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,oBAAoB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+PAA+P,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qLAAqL,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uNAAuN,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,sBAAsB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iXAAiX,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oPAAoP,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,sBAAsB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+RAA+R,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,kEAAkE,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wBAAwB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+bAAqb,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wUAAwU,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sPAAsP,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2UAA2U,CAAC,CAAC,CAAC,CAAC,EAAeI,EAAuBN,EAAIC,EAAS,CAAC,SAAS,CAAcC,EAAE,IAAI,CAAC,SAAS,maAAma,CAAC,EAAeA,EAAEC,EAAE,CAAC,KAAK,+HAA+H,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,UAAU,eAAe,kBAAkB,MAAM,SAAsBF,EAAE,MAAM,CAAC,IAAI,GAAG,UAAU,eAAe,OAAO,MAAM,IAAI,sEAAsE,OAAO,oQAAoQ,MAAM,CAAC,YAAY,YAAY,EAAE,MAAM,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,+DAA+D,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gbAAgb,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+gBAAqgB,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,yMAAsNE,EAAEC,EAAE,CAAC,KAAK,uCAAuC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,kBAAkB,CAAC,CAAC,CAAC,EAAE,+GAA+G,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,oDAAoD,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0hBAA0hB,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,yIAAyI,UAAU,eAAe,OAAO,MAAM,IAAI,wEAAwE,OAAO,yWAAyW,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,MAAM,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,SAAS,wIAAwI,CAAC,EAAeA,EAAEC,EAAE,CAAC,KAAK,uCAAuC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAsBF,EAAE,KAAK,CAAC,SAAS,sCAAsC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+dAA+d,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kiBAAkiB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kNAAkN,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,6CAA6C,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2WAA2W,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,8CAA8C,UAAU,eAAe,OAAO,MAAM,IAAI,uEAAuE,OAAO,qWAAqW,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,MAAM,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mZAAmZ,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yVAAyV,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gcAAgc,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAEC,EAAE,CAAC,KAAK,qBAAqB,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,QAAQ,CAAC,CAAC,CAAC,EAAE,+IAA4JF,EAAEC,EAAE,CAAC,KAAK,kEAAkE,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,eAAe,CAAC,CAAC,CAAC,EAAE,gYAA6YF,EAAEC,EAAE,CAAC,KAAK,yCAAyC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,KAAK,CAAC,CAAC,CAAC,EAAE,yTAAsUF,EAAEC,EAAE,CAAC,KAAK,sDAAsD,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,OAAO,CAAC,CAAC,CAAC,EAAeF,EAAEC,EAAE,CAAC,KAAK,+CAA+C,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,MAAM,CAAC,CAAC,CAAC,EAAE,0KAAuLF,EAAEC,EAAE,CAAC,KAAK,0FAA0F,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,0CAA0C,CAAC,CAAC,CAAC,EAAE,sNAAsN,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,wCAAwC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,qMAA6ME,EAAEC,EAAE,CAAC,KAAK,8BAA8B,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,KAAK,CAAC,CAAC,CAAC,EAAE,+TAA+T,CAAC,CAAC,EAAeJ,EAAE,IAAI,CAAC,SAAS,CAAC,uEAAoFE,EAAEC,EAAE,CAAC,KAAK,qBAAqB,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,QAAQ,CAAC,CAAC,CAAC,EAAE,iHAA8HF,EAAEC,EAAE,CAAC,KAAK,gCAAgC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,WAAW,CAAC,CAAC,CAAC,EAAE,uCAAuC,CAAC,CAAC,EAAeJ,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,SAAS,uDAAuD,CAAC,EAAeA,EAAEC,EAAE,CAAC,KAAK,0GAA0G,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAsBF,EAAE,KAAK,CAAC,SAAS,YAAY,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeK,EAAuBP,EAAIC,EAAS,CAAC,SAAS,CAAcC,EAAE,IAAI,CAAC,SAAS,uaAAka,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,qEAAkFE,EAAE,KAAK,CAAC,SAAS,WAAW,CAAC,EAAE,yTAAyT,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uIAAkI,CAAC,EAAeA,EAAEC,EAAE,CAAC,KAAK,2IAA2I,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,UAAU,eAAe,kBAAkB,MAAM,SAAsBF,EAAE,MAAM,CAAC,IAAI,GAAG,UAAU,eAAe,OAAO,MAAM,IAAI,uEAAuE,OAAO,uQAAuQ,MAAM,CAAC,YAAY,YAAY,EAAE,MAAM,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,eAAe,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,8GAA2HE,EAAEC,EAAE,CAAC,KAAK,kDAAkD,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,cAAc,CAAC,CAAC,CAAC,EAAE,+BAA+B,CAAC,CAAC,EAAeJ,EAAE,KAAK,CAAC,MAAM,CAAC,qBAAqB,OAAO,0BAA0B,QAAQ,sBAAsB,kBAAkB,6BAA6B,MAAM,0BAA0B,MAAM,EAAE,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,sGAAiG,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,+KAA0K,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,8KAA8K,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wIAAwI,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yFAAyF,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,gBAAgB,UAAU,eAAe,OAAO,MAAM,IAAI,uEAAuE,OAAO,qWAAqW,MAAM,CAAC,YAAY,YAAY,EAAE,MAAM,MAAM,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qGAAqG,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sDAAsD,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,eAAe,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sjBAA4iB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kDAAkD,CAAC,EAAeF,EAAE,KAAK,CAAC,MAAM,CAAC,qBAAqB,OAAO,0BAA0B,QAAQ,sBAAsB,kBAAkB,6BAA6B,MAAM,0BAA0B,MAAM,EAAE,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,yHAAyH,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBA,EAAE,IAAI,CAAC,SAAS,wLAAmL,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,kBAAkB,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,kQAAqQE,EAAEC,EAAE,CAAC,KAAK,uCAAuC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,mBAAmB,CAAC,CAAC,CAAC,EAAE,GAAG,CAAC,CAAC,EAAeJ,EAAE,IAAI,CAAC,SAAS,CAAC,qDAAkEE,EAAEC,EAAE,CAAC,KAAK,mCAAmC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,UAAU,CAAC,CAAC,CAAC,EAAE,qSAAqS,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,gLAAgL,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,+BAA+B,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,0GAAuHE,EAAEC,EAAE,CAAC,KAAK,4DAA4D,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,6BAA6B,CAAC,CAAC,CAAC,EAAE,uHAAuH,CAAC,CAAC,EAAeF,EAAE,MAAM,CAAC,IAAI,iDAAiD,UAAU,eAAe,OAAO,MAAM,IAAI,uEAAuE,OAAO,uQAAuQ,MAAM,CAAC,YAAY,YAAY,EAAE,MAAM,KAAK,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,SAAS,gDAAgD,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0HAA0H,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,8DAA8D,UAAU,eAAe,OAAO,MAAM,IAAI,sEAAsE,OAAO,oQAAoQ,MAAM,CAAC,YAAY,YAAY,EAAE,MAAM,KAAK,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,SAAS,6DAA6D,CAAC,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,+JAA4KE,EAAEC,EAAE,CAAC,KAAK,2CAA2C,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,WAAW,CAAC,CAAC,CAAC,EAAE,mRAAmR,CAAC,CAAC,EAAeF,EAAE,MAAM,CAAC,IAAI,oDAAoD,UAAU,eAAe,OAAO,MAAM,IAAI,uEAAuE,OAAO,uQAAuQ,MAAM,CAAC,YAAY,YAAY,EAAE,MAAM,KAAK,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,SAAS,mDAAmD,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uHAAuH,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,iCAAiC,UAAU,eAAe,OAAO,MAAM,IAAI,wEAAwE,OAAO,0QAA0Q,MAAM,CAAC,YAAY,YAAY,EAAE,MAAM,KAAK,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,SAAS,gCAAgC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uKAAuK,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,eAAe,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uSAA6R,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,2FAAwGE,EAAEC,EAAE,CAAC,KAAK,mCAAmC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,cAAc,CAAC,CAAC,CAAC,EAAE,iFAA4E,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,kZAAkZ,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kGAAkG,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qeAAqe,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,mFAAmF,UAAU,eAAe,OAAO,MAAM,IAAI,uEAAuE,OAAO,0KAA0K,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,KAAK,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,SAAS,kFAAkF,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,gBAAgB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kLAA6K,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,uRAAoSE,EAAEC,EAAE,CAAC,KAAK,0DAA0D,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,QAAQ,CAAC,CAAC,CAAC,EAAE,GAAG,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,6VAA6V,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oTAAoT,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uPAAuP,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,SAAS,CAAC,SAAS,wCAAwC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oMAA+L,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAEC,EAAE,CAAC,KAAK,qBAAqB,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,QAAQ,CAAC,CAAC,CAAC,EAAE,oOAAoO,CAAC,CAAC,EAAeJ,EAAE,IAAI,CAAC,SAAS,CAAC,4HAA+HE,EAAEC,EAAE,CAAC,KAAK,gCAAgC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,MAAM,CAAC,CAAC,CAAC,EAAE,YAAY,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeI,EAAuBR,EAAIC,EAAS,CAAC,SAAS,CAAcC,EAAE,IAAI,CAAC,SAAS,kjBAAkjB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yaAAoa,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iVAAuU,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,ybAA+a,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qjBAAuhB,CAAC,EAAeA,EAAEC,EAAE,CAAC,KAAK,gIAAgI,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,UAAU,eAAe,kBAAkB,MAAM,SAAsBF,EAAE,MAAM,CAAC,IAAI,GAAG,UAAU,eAAe,OAAO,MAAM,IAAI,sEAAsE,OAAO,oQAAoQ,MAAM,CAAC,YAAY,YAAY,EAAE,MAAM,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,aAAa,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,kBAAkB,2BAA2B,MAAM,EAAE,SAAsBA,EAAE,IAAI,CAAC,SAAS,uHAAuH,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,kBAAkB,2BAA2B,MAAM,EAAE,SAAsBA,EAAE,IAAI,CAAC,SAAS,0GAA0G,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,kBAAkB,2BAA2B,MAAM,EAAE,SAAsBA,EAAE,IAAI,CAAC,SAAS,8EAA8E,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,kBAAkB,2BAA2B,MAAM,EAAE,SAAsBA,EAAE,IAAI,CAAC,SAAS,6JAA6J,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,kBAAkB,2BAA2B,MAAM,EAAE,SAAsBA,EAAE,IAAI,CAAC,SAAS,sGAAsG,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,kBAAkB,2BAA2B,MAAM,EAAE,SAAsBA,EAAE,IAAI,CAAC,SAAS,iEAAiE,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,kBAAkB,2BAA2B,MAAM,EAAE,SAAsBA,EAAE,IAAI,CAAC,SAAS,8EAA8E,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,kBAAkB,2BAA2B,MAAM,EAAE,SAAsBA,EAAE,IAAI,CAAC,SAAS,uFAAuF,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,qBAAqB,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,oDAAoD,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8bAA8b,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oGAAoG,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+YAA0Y,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,iDAAiD,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oYAAoY,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uPAAuP,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qQAAuO,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,mDAAmD,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+TAA+T,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,uDAAuD,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kYAAkY,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,oIAAiJE,EAAE,KAAK,CAAC,CAAC,EAAE,qBAAkCA,EAAE,KAAK,CAAC,CAAC,EAAE,6BAA0CA,EAAE,KAAK,CAAC,CAAC,EAAE,uBAAoCA,EAAE,KAAK,CAAC,CAAC,EAAE,2FAAwGA,EAAE,KAAK,CAAC,CAAC,EAAE,sJAAsJ,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+XAA0X,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,qDAAqD,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uQAAkQ,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAsBA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,kBAAkB,2BAA2B,MAAM,EAAE,SAAsBA,EAAE,IAAI,CAAC,SAAS,sHAAsH,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sDAAsD,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,+BAA4CE,EAAE,KAAK,CAAC,CAAC,EAAE,+BAA4CA,EAAE,KAAK,CAAC,CAAC,EAAE,0BAAuCA,EAAE,KAAK,CAAC,CAAC,EAAE,+BAA4CA,EAAE,KAAK,CAAC,CAAC,EAAE,uDAAuD,CAAC,CAAC,EAAeF,EAAE,KAAK,CAAC,MAAM,IAAI,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,kBAAkB,2BAA2B,MAAM,EAAE,SAAsBA,EAAE,IAAI,CAAC,SAAS,6EAA6E,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,kBAAkB,2BAA2B,MAAM,EAAE,SAAsBA,EAAE,IAAI,CAAC,SAAS,2EAA2E,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+KAA+K,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,yCAAyC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,ibAA4a,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yDAAyD,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,eAAe,2BAA2B,MAAM,EAAE,SAAsBA,EAAE,IAAI,CAAC,SAAS,0GAA0G,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,kBAAkB,2BAA2B,MAAM,EAAE,SAAsBA,EAAE,IAAI,CAAC,SAAS,+DAA+D,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,oEAAoE,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2TAA2T,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6KAA6K,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,kBAAkB,2BAA2B,MAAM,EAAE,SAAsBA,EAAE,IAAI,CAAC,SAAS,8NAAyN,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,kBAAkB,2BAA2B,MAAM,EAAE,SAAsBA,EAAE,IAAI,CAAC,SAAS,sLAAsL,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,kBAAkB,2BAA2B,MAAM,EAAE,SAAsBA,EAAE,IAAI,CAAC,SAAS,qNAAqN,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,wDAAwD,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gaAAiZ,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,4NAA0NE,EAAE,KAAK,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,CAAC,EAAE,oLAAiMA,EAAE,KAAK,CAAC,CAAC,EAAE,gHAAwHA,EAAE,KAAK,CAAC,CAAC,EAAE,mIAAmI,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,4CAA4C,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oEAAoE,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4WAAmV,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wSAAwS,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,yDAAyD,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sbAAib,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,oEAAoE,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mIAAmI,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,qvBAAwvBE,EAAEC,EAAE,CAAC,KAAK,mDAAmD,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,iBAAiB,CAAC,CAAC,CAAC,EAAE,uCAAuC,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,wZAAmZ,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uIAAuI,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,qBAAqB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mJAAmJ,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,0DAA0D,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8KAAyK,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8VAAoV,CAAC,EAAeF,EAAE,KAAK,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,kBAAkB,2BAA2B,MAAM,EAAE,SAAsBA,EAAE,IAAI,CAAC,SAAS,oCAAoC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,kBAAkB,2BAA2B,MAAM,EAAE,SAAsBA,EAAE,IAAI,CAAC,SAAS,4CAA4C,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,MAAM,CAAC,qBAAqB,OAAO,sBAAsB,kBAAkB,2BAA2B,MAAM,EAAE,SAAsBA,EAAE,IAAI,CAAC,SAAS,kEAAkE,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uRAAuR,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,gDAAgD,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kXAA6W,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0ZAA0Z,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,gUAA4S,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,qEAAqE,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oXAAoX,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6HAA6H,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0NAAqN,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,gBAAgB,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,sDAAmEE,EAAEC,EAAE,CAAC,KAAK,gCAAgC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,QAAQ,CAAC,CAAC,CAAC,EAAE,4DAA4D,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeK,EAAuBT,EAAIC,EAAS,CAAC,SAAS,CAAcC,EAAE,IAAI,CAAC,SAAS,8VAA8V,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sXAAsX,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,kEAAkE,UAAU,eAAe,OAAO,MAAM,IAAI,wEAAwE,OAAO,0QAA0Q,MAAM,CAAC,YAAY,YAAY,EAAE,MAAM,KAAK,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,SAAS,mJAAmJ,CAAC,EAAeA,EAAEC,EAAE,CAAC,KAAK,0DAA0D,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAsBF,EAAE,KAAK,CAAC,SAAS,kBAAkB,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,GAAG,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,UAAU,gBAAgB,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qTAAqT,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,wCAAwC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4SAA4S,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sTAAsT,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0PAA0P,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,yCAAyC,UAAU,eAAe,OAAO,MAAM,IAAI,wEAAwE,OAAO,yWAAyW,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,MAAM,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,SAAS,iJAAiJ,CAAC,EAAeA,EAAEC,EAAE,CAAC,KAAK,4EAA4E,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAsBF,EAAE,KAAK,CAAC,SAAS,qBAAqB,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,GAAG,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,UAAU,gBAAgB,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0UAA0U,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,sCAAsC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,maAAma,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sOAAsO,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,yCAAyC,UAAU,eAAe,OAAO,MAAM,IAAI,wEAAwE,OAAO,yWAAyW,MAAM,CAAC,YAAY,YAAY,EAAE,MAAM,MAAM,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,SAAS,gJAAgJ,CAAC,EAAeA,EAAEC,EAAE,CAAC,KAAK,uEAAuE,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAsBF,EAAE,KAAK,CAAC,SAAS,eAAe,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,GAAG,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,UAAU,gBAAgB,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8KAA8K,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,uYAAuY,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qQAAgQ,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,yCAAyC,UAAU,eAAe,OAAO,MAAM,IAAI,wEAAwE,OAAO,yWAAyW,MAAM,CAAC,YAAY,aAAa,EAAE,MAAM,MAAM,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,SAAS,iDAAiD,CAAC,EAAeA,EAAEC,EAAE,CAAC,KAAK,iEAAiE,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAsBF,EAAE,KAAK,CAAC,SAAS,eAAe,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,GAAG,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,UAAU,gBAAgB,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+JAA+J,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,yBAAyB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wNAAoM,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qOAAqO,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,0BAA0B,UAAU,eAAe,OAAO,MAAM,IAAI,uEAAuE,OAAO,uQAAuQ,MAAM,CAAC,YAAY,YAAY,EAAE,MAAM,KAAK,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,SAAS,yCAAyC,CAAC,EAAeA,EAAEC,EAAE,CAAC,KAAK,kEAAkE,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAsBF,EAAE,KAAK,CAAC,SAAS,IAAI,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,GAAG,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,UAAU,gBAAgB,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6IAA6I,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,iCAAiC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iRAAiR,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,iMAAiM,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,aAAa,CAAC,EAAE,gJAAgJ,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,cAAc,CAAC,EAAE,8IAA8I,CAAC,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,kCAAkC,UAAU,eAAe,OAAO,MAAM,IAAI,wEAAwE,OAAO,0QAA0Q,MAAM,CAAC,YAAY,YAAY,EAAE,MAAM,KAAK,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,SAAS,8QAA8Q,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,UAAU,gBAAgB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,8BAA8B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mTAAmT,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,sBAAsB,CAAC,EAAE,+FAA+F,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,oBAAoB,CAAC,EAAE,+HAA+H,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,iCAAiC,CAAC,EAAE,0HAAqH,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,4BAA4B,CAAC,EAAE,sGAAsG,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,qCAAqC,CAAC,EAAE,6JAA6J,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,uCAAuC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oMAAoM,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,4BAA4B,CAAC,EAAE,yMAAyM,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,wBAAwB,CAAC,EAAE,wNAAwN,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,cAAc,CAAC,EAAE,uKAAuK,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,uBAAuB,CAAC,EAAE,mJAAmJ,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6NAAwN,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,uBAAuB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,2YAA2Y,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wQAAwQ,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,wBAAwB,UAAU,eAAe,OAAO,MAAM,IAAI,sEAAsE,OAAO,oQAAoQ,MAAM,CAAC,YAAY,YAAY,EAAE,MAAM,KAAK,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,SAAS,+DAA+D,CAAC,EAAeA,EAAEC,EAAE,CAAC,KAAK,2DAA2D,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAsBF,EAAE,KAAK,CAAC,SAAS,YAAY,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,GAAG,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,UAAU,gBAAgB,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sLAAsL,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,+BAA+B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qcAAqc,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,gCAAgC,UAAU,eAAe,OAAO,MAAM,IAAI,wEAAwE,OAAO,0QAA0Q,MAAM,CAAC,YAAY,YAAY,EAAE,MAAM,KAAK,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,SAAS,6GAA6G,CAAC,EAAeA,EAAEC,EAAE,CAAC,KAAK,qIAAqI,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAsBF,EAAE,KAAK,CAAC,SAAS,SAAS,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,GAAG,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,UAAU,gBAAgB,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qIAAqI,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,aAAa,CAAC,EAAE,uPAAuP,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,iBAAiB,CAAC,EAAE,sLAAsL,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,4BAA4B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mJAAmJ,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,sBAAsB,CAAC,EAAE,+FAA+F,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,kBAAkB,CAAC,EAAE,6MAA6M,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,+BAA+B,CAAC,EAAE,gHAAgH,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,uBAAuB,CAAC,EAAE,8PAA8P,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,qBAAqB,CAAC,EAAE,+PAA+P,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,qCAAqC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,sPAAsP,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,wBAAwB,CAAC,EAAE,kLAAkL,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,qBAAqB,CAAC,EAAE,kSAAkS,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,sBAAsB,CAAC,EAAE,iKAAiK,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,mCAAmC,CAAC,EAAE,kQAAkQ,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,uBAAuB,CAAC,EAAE,8SAA8S,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yVAAyV,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,0BAA0B,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+YAA+Y,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+bAA+b,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,2BAA2B,UAAU,eAAe,OAAO,MAAM,IAAI,sEAAsE,OAAO,oQAAoQ,MAAM,CAAC,YAAY,YAAY,EAAE,MAAM,KAAK,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,SAAS,oGAAoG,CAAC,EAAeA,EAAEC,EAAE,CAAC,KAAK,+EAA+E,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAsBF,EAAE,KAAK,CAAC,SAAS,kBAAkB,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,GAAG,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,UAAU,gBAAgB,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wRAAwR,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,uBAAuB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kGAAkG,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,iBAAiB,CAAC,EAAE,8WAA8W,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,oBAAoB,CAAC,EAAE,8RAA8R,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,iBAAiB,CAAC,EAAE,mTAAmT,CAAC,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,0BAA0B,UAAU,eAAe,OAAO,MAAM,IAAI,uEAAuE,OAAO,uQAAuQ,MAAM,CAAC,YAAY,YAAY,EAAE,MAAM,KAAK,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,SAAS,sEAAsE,CAAC,EAAeA,EAAEC,EAAE,CAAC,KAAK,4DAA4D,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAsBF,EAAE,KAAK,CAAC,SAAS,QAAQ,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,GAAG,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,UAAU,gBAAgB,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6TAA6T,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mGAAmG,CAAC,EAAeA,EAAE,MAAM,CAAC,IAAI,0BAA0B,UAAU,eAAe,OAAO,MAAM,IAAI,wEAAwE,OAAO,0KAA0K,MAAM,CAAC,YAAY,WAAW,EAAE,MAAM,KAAK,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,KAAK,CAAC,SAAS,wCAAwC,CAAC,EAAeA,EAAEC,EAAE,CAAC,KAAK,qBAAqB,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAsBF,EAAE,KAAK,CAAC,SAAS,WAAW,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,GAAG,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAsBA,EAAE,KAAK,CAAC,UAAU,gBAAgB,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,gBAAgB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qZAAqZ,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oYAAoY,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0QAA0Q,CAAC,CAAC,CAAC,CAAC,EAAeQ,EAAuBV,EAAIC,EAAS,CAAC,SAAS,CAAcC,EAAE,KAAK,CAAC,SAAS,2CAA2C,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+RAA+R,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,6BAA0CE,EAAEC,EAAE,CAAC,KAAK,uCAAuC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,oCAAoC,CAAC,CAAC,CAAC,EAAE,wFAAwF,CAAC,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,2KAAsK,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6TAA6T,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,wbAAwb,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,iDAAiD,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,kIAAkI,CAAC,EAAeF,EAAE,KAAK,CAAC,MAAM,CAAC,qBAAqB,OAAO,0BAA0B,QAAQ,sBAAsB,kBAAkB,6BAA6B,MAAM,0BAA0B,MAAM,EAAE,SAAS,CAAcE,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,yBAAyB,CAAC,EAAE,4EAA4E,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,wBAAwB,CAAC,EAAE,4JAA4J,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,kBAAkB,IAAI,SAAsBF,EAAE,IAAI,CAAC,SAAS,CAAcE,EAAE,SAAS,CAAC,SAAS,oCAAoC,CAAC,EAAE,mOAAmO,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,yeAAye,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,oaAAoa,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,gCAAgC,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,ocAAoc,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,+cAA0c,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,0cAA0c,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,mbAAmb,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,8JAAyJ,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,2CAA2C,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6XAAwX,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,6gBAA6gB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,qaAAqa,CAAC,EAAeA,EAAEC,EAAE,CAAC,KAAK,6GAA6G,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,UAAU,eAAe,kBAAkB,MAAM,SAAsBF,EAAE,MAAM,CAAC,IAAI,GAAG,UAAU,eAAe,OAAO,MAAM,IAAI,uEAAuE,OAAO,uQAAuQ,MAAM,CAAC,YAAY,YAAY,EAAE,MAAM,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,EAAeA,EAAE,KAAK,CAAC,SAAS,yBAAyB,CAAC,EAAeA,EAAE,IAAI,CAAC,SAAS,4XAA4X,CAAC,EAAeF,EAAE,IAAI,CAAC,SAAS,CAAC,wMAAqNE,EAAEC,EAAE,CAAC,KAAK,uCAAuC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,gEAAgE,CAAC,CAAC,CAAC,EAAE,wJAAwJ,CAAC,CAAC,EAAeJ,EAAE,IAAI,CAAC,SAAS,CAAC,0DAAuEE,EAAEC,EAAE,CAAC,KAAK,gCAAgC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,wBAAwB,CAAC,CAAC,CAAC,EAAE,0GAAuHF,EAAEC,EAAE,CAAC,KAAK,gCAAgC,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,cAAc,CAAC,CAAC,CAAC,EAAE,+EAA+E,CAAC,CAAC,EAAeJ,EAAE,IAAI,CAAC,SAAS,CAAC,yDAAsEE,EAAEC,EAAE,CAAC,KAAK,mHAAmH,YAAY,GAAG,OAAO,YAAY,aAAa,GAAG,QAAQ,oBAAoB,aAAa,GAAG,SAAsBD,EAAEE,EAAE,EAAE,CAAC,SAAS,YAAY,CAAC,CAAC,CAAC,EAAE,GAAG,CAAC,CAAC,CAAC,CAAC,CAAC,EACxpxIO,EAAqB,CAAC,QAAU,CAAC,UAAY,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,UAAY,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,UAAY,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,UAAY,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,UAAY,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,SAAW,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,UAAY,CAAC,KAAO,WAAW,YAAc,CAAC,sBAAwB,GAAG,CAAC,EAAE,mBAAqB,CAAC,KAAO,UAAU,CAAC,CAAC",
  "names": ["richText", "u", "x", "p", "Link", "motion", "richText1", "richText2", "richText3", "richText4", "richText5", "richText6", "__FramerMetadata__"]
}